docs: measure gemma-4-12b on the workstation against the resident model (V-485)
Both fixtures, run from homesrv across the LAN with the proxy env stripped. Routing: 84.4% full / 93.5% intent-only at p50 329ms through the cascade, against 72.7% / 77.9% at p50 0.80-1.04s for Qwen3-1.7B. Talk: 25/27 against 20/27, with knowledge 9/9. Nudges 15/15. Settles #485's first assumption by measurement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJYcaBiuny9UGSpFweQVQ1
This commit is contained in:
+5
-2
@@ -113,8 +113,11 @@ Then the embedder above, **whisper.cpp** in `mavsttd`, and **piper** in `mavttsd
|
||||
builds an `llm.Pair` in `modelSeam` (`cmd/mavend/voicewire.go`), and routing
|
||||
and replies complete through it. Both are the silent half of the rule. The
|
||||
naming half is not wired. A world question still goes to the resident model
|
||||
through `PhraseQuery`, and the fixture measurement has not been run.
|
||||
Biggest quality delta. A 16GB card runs a 7-14B,
|
||||
through `PhraseQuery`.
|
||||
Measured, `docs/evals/2026-08-02-workstation-gemma4-12b.md`: gemma-4-12b
|
||||
through the cascade scores 84.4% full accuracy at p50 329ms. The resident
|
||||
model scores 72.7% at p50 0.80-1.04s. On the talk fixture it is 25/27
|
||||
against 20/27. Biggest quality delta. A 16GB card runs a 7-14B,
|
||||
which fixes what the 1.7B gets wrong: world knowledge, and the persona the CPT
|
||||
targets. The degradation path is already written and measured, since the
|
||||
classifier scores 68.8% full accuracy at p50 16.6µs on its own.
|
||||
|
||||
Reference in New Issue
Block a user