diff --git a/CLAUDE.md b/CLAUDE.md index 594bbc5..b013500 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -255,11 +255,21 @@ re-run it, start a **second** llama-server on a fixed host port — the resident `--port 0` inside the container and no host process can reach it. **The numbers above are the homesrv floor, not the ceiling.** With the workstation up, routing -completes through `llm.Pair` against gemma-4-12b and scores **84.4% full / 93.5% intent-only at -p50 329ms** — better than the resident model and about 2.5× faster (`docs/evals/2026-08-02-workstation-gemma4-12b.md`, -Vikunja #485). The workstation is never assumed up, so both sets of numbers are live. Judge a +completes through `llm.Pair` against the model mavgpud holds, which is better than the resident +model and about 2.5× faster. gemma-4-12b scored **84.4% full / 93.5% intent-only at p50 329ms** +(`docs/evals/2026-08-02-workstation-gemma4-12b.md`, Vikunja #485). The workstation is never +assumed up, so both sets of numbers are live. Judge a routing change against the classifier and the resident model, since those are what always answer. +**The workstation runs gemma-4-E4B since 2026-08-09** (owner's call), and it is a +step down measured the same day (`docs/evals/2026-08-09-e4b-vs-12b-routing.md`). +Against a same-session 12B control it scores **83.3% full / 89.6% intent-only, +destination 19/33 against 23/33, at p50 294ms against 344ms**. So it costs four +destination cases and buys 50ms. Read destination as the finding: it names nothing +where the 12B names `recall` or `calendar`, which is safe but walks the whole chain. +It also has no MTP and cannot be given any here. The only `gemma4-assistant` +draft on disk is trained against the 12B's hidden states. + **The intended third engine is not a generative model** (owner's call, 05-08-2026, V-546, `docs/plans/18-routing-heads-on-e5-small.md`). Routing has a bounded output space, so it is classification, and the 118M multilingual-e5-small is already resident. Three heads on one diff --git a/deploy/mavgpud.json b/deploy/mavgpud.json index 293b3c9..2738a5c 100644 --- a/deploy/mavgpud.json +++ b/deploy/mavgpud.json @@ -2,9 +2,14 @@ "listen": ":8080", "llama_addr": "127.0.0.1:10000", "llama_bin": "llama-server", + "//llama_args": [ + "E4B carries no MTP tensors, so the speculative flags are gone with the 12B.", + "MTP on this box is a separate gguf of architecture gemma4-assistant with", + "nextn_predict_layers=4, and mtp-gemma-4-12B-it-BF16 is the only one there is.", + "Its head is trained against the 12B's hidden states, so it cannot drive E4B." + ], "llama_args": [ - "-m", "/mnt/D/AI/gemma4/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf", - "-md", "/mnt/D/AI/gemma4/mtp-gemma-4-12B-it-BF16.gguf", + "-m", "/mnt/D/AI/gemma4/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf", "-ngl", "99", "-fa", "on", "-np", "1", @@ -15,9 +20,7 @@ "--batch-size", "2048", "--ubatch-size", "512", "--jinja", - "--chat-template-kwargs", "{\"enable_thinking\":false}", - "--spec-type", "draft-mtp", - "--spec-draft-n-max", "2" + "--chat-template-kwargs", "{\"enable_thinking\":false}" ], "//stt": [ diff --git a/docs/evals/2026-08-09-e4b-vs-12b-routing.md b/docs/evals/2026-08-09-e4b-vs-12b-routing.md new file mode 100644 index 0000000..f0a8fc4 --- /dev/null +++ b/docs/evals/2026-08-09-e4b-vs-12b-routing.md @@ -0,0 +1,51 @@ +# gemma-4-E4B against gemma-4-12B on the routing fixture + +*Measured 2026-08-09 on workpc. The owner asked for the swap. This is what it costs.* + +Both arms ran the same 96-case fixture through `TestLLMRouterBaseline`, minutes +apart, against the same llama-server build and the same mavgpud. The 12B arm is a +control run and not the 2026-08-02 number. That one predates five fixture cases, +the destination labels and a llama.cpp upgrade. + +| | full | intent-only | destination | p50 | p95 | +|---|---|---|---|---|---| +| gemma-4-12B-it-qat-UD-Q4_K_XL, MTP draft | 81/96 (84.4%) | 91.7% | 23/33 (69.7%) | 344ms | 471ms | +| gemma-4-E4B-it-qat-UD-Q4_K_XL | 80/96 (83.3%) | 89.6% | 19/33 (57.6%) | 294ms | 562ms | + +E4B costs one case of full accuracy, two of intent and **four of destination**, +and buys 50ms at p50. Read the destination column as the finding. One case is +three points on 33. So 23 against 19 is outside the noise a single case makes, +and the other two columns are not. + +Both arms produce three false clarifies and one missed clarify, and neither +errored on any case. + +## What E4B loses + +Four of the five destination regressions are the same shape: it names nothing +where the 12B names `recall` or `calendar`. `ru-query-015` ("сколько я прошёл +шагов") goes further and names `self`. Naming nothing is the safe direction, +because `SourceUnknown` walks the whole chain, so these turns are still answered. +They cost latency and they are what a fourth head is meant to fix (V-546). + +Two Russian intent cases regress, both with the interrogative off the front. +`ru-chat-003` ("расскажи анекдот про программистов") goes to `query`. +`ru-fact-003` ("поужинал") goes to `chat`. + +## MTP + +E4B has none, and there is no way to give it any on this box. MTP on workpc is +a separate gguf of architecture `gemma4-assistant` carrying +`nextn_predict_layers=4`, and `mtp-gemma-4-12B-it-BF16.gguf` is the only one on +disk. Its head is trained against the 12B's hidden states, so it cannot drive an +E4B target. Scanning both target ggufs finds no `nextn` tensors in either, so +neither model self-speculates. + +So the 12B arm above ran with speculative decoding and E4B ran without, and E4B +was still faster. + +## Cost on the card + +E4B is 4.2GB against 6.7GB plus a 0.86GB draft. With CW2 resident at 1.6GB that +is 5.8GB of 16GB against 9.2GB. Nothing in Maven needs the difference, so this is +headroom for the owner's own jobs rather than a capability.