claude
7f804b84e7
docs/CLAUDE.md defined four tiers and docs/capabilities/ and docs/architecture/ were neither of them. Both are now declared as generated: rebuilt from a source, never corrected in place. A wrong row in either is a bug in the generator or in one of its hand-written inputs, and editing the output makes the next rebuild silently undo the fix. The eval index gains the 2026-08-19 CPT+SFT measurement and the 2026-08-26 baseline, and marks the 2026-08-13 capability audit superseded. The 2026-08-19 file lands with them: it is the direct evidence that the deployed maven-instruct-b2 routes better than Qwen3-1.7B and cannot hold a Russian sentence, which is the first item on the gaps.md priority list. Its header records Vikunja as returning 503. That reading was wrong and the 2026-08-26 baseline says so, but a dated eval is not edited after the day. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5.1 KiB
The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence
Measured 2026-08-19 on homesrv, CPU. Unfiled: Vikunja has returned 503 since
2026-08-13. Build 5cae33a on master. The deployment was not changed.
The candidate is /mnt/hdd1/llms/maven/maven-model-Q4_K_XL.gguf, built on workpc
from a continued-pretraining stage plus a supervised fine-tune, quantised by
replaying the Unsloth imatrix recipe with 197 per-tensor overrides read out of
the reference gguf. The incumbent is
/mnt/hdd1/llms/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf, the deployed resident model.
Both arms ran on the same homesrv llama-server, same flags, same port, minutes
apart, one model resident at a time. CPU inference throughout, so every latency
figure here is incomparable to any GPU run, including the 116.7 ms the build
side reported. Accuracy is the only number that carries.
The build side's own measurement claimed 88/96 for the candidate. This run reproduces it at 87/96 on the same fixture, so the routing claim stands.
Routing, TestLLMRouterBaseline, 96 cases
| full | intent-only | destination | ru | en | p50 | |
|---|---|---|---|---|---|---|
| maven-model-Q4_K_XL | 87/96 (90.6%) | 94.8% | 24/33 (72.7%) | 72/81 | 15/15 | 1.29s |
| Qwen3-1.7B-UD-Q4_K_XL | 81/96 (84.4%) | 87.5% | 16/33 (48.5%) | 67/81 | 14/15 | 1.22s |
Both ran 0 errors and 1 missed clarify on the cascade. Destination is where the
candidate earns it: eight more correct query sources, 72.7% against 48.5%. That
equals what docs/evals/2026-08-09-e4b-vs-12b-routing.md records for
gemma-4-12B at seven times the parameter count.
The llm-only section of the same run reads 59/96 against 45/96. It bypasses
stage 0 and the daemon slot fillers on purpose. Do not cite it as the routing
number; cascade+llm is the deployed path.
Phrasing, internal/phraser/eval
| nudges | conversational | |
|---|---|---|
| maven-model-Q4_K_XL | 15/15 (100%) | 8/36 (22.2%) |
| maven-model-Q4_K_XL, eos corrected | 15/15 (100%) | 11/36 (30.6%) |
| Qwen3-1.7B-UD-Q4_K_XL | 15/15 (100%) | 22/36 (61.1%) |
Nudges are tied at ceiling. Conversational phrasing falls by half.
Two defects, and they are independent
The gguf carries the wrong eos token. tokenizer.ggml.eos_token_id is
151643 (<|endoftext|>) against the incumbent's 151645 (<|im_end|>), and
bos is 151643 too. Both files carry the identical Qwen3 chat template, and
that template closes every turn with <|im_end|>. The candidate is told to stop
on a token the template never emits, so generation runs through the turn
boundary. The conversion read the base config, not the instruct one.
Re-running with --override-kv tokenizer.ggml.eos_token_id=int:151645 isolates
its cost:
| check | as shipped | eos corrected |
|---|---|---|
| address | 27/36 | 32/36 |
| knowledge path | 3/9 | 7/9 |
| ontopic | 14/36 | 15/36 |
| reply path | 0/9 | 0/9 |
Routing is untouched by this, because GBNF-constrained decoding terminates on the grammar and never reaches the eos token. That is why the defect is invisible in the routing score and fatal in free generation.
The remaining half is not packaging. Corrected, the candidate reaches 11/36
against the incumbent's 22/36. ontopic moves one case and the reply path stays
at 0 of 9. The failures are broken Russian, not persona drift:
Ты не одиноден?!— not a word;одинок.Привёз с собой смайлики-котята и вручила им подарок— masculine and feminine verbs in one clause, wrong case on the noun.Вчера у вас не будет никакого лёг.— yesterday in the future tense, cut off mid-word.Заберите диск от системы или установочный ковш.— "installation bucket".Вам сказал Игоря Валерьевич— a first name declined as accusative.know-hiccupsanswered with weight-loss advice.
What this rules out
- Not a harness mismatch. Both arms ran the identical target, server and fixture, and the candidate reproduced the build side's routing claim to within one case.
- Not the missing
ROLEPLAY_ACTIONandMASC_SELFfixes alone. Those were the build side's stated reason to expect persona failures.femininescores 35/36 corrected. The failures are grammatical, not persona. - Not weights damage alone either. The eos defect is real and costs five address cases and four knowledge cases on its own.
inference: the SFT stage worked and the CPT stage cost general Russian
coherence. Routing is constrained decoding over 7 intents and a destination, and
it improved. Free generation is where the loss shows.
The swap was not made
deploy/mavend.json:19 still reads the incumbent. One phraser.model_path
serves both the router and the phraser and voice.llm_router is true, so
swapping buys 6 routing cases and pays 11 conversational ones. Both ggufs stay
at /mnt/hdd1/llms/maven/ for the re-run.
The two-week replay (scripts/usage-run.py,
docs/evals/2026-08-08-two-weeks.md) was not run. It drives the deployed stack
through POST /api/chat, so it measures whatever mavend has loaded, and the
candidate was never loaded.