claude
7f804b84e7
docs/CLAUDE.md defined four tiers and docs/capabilities/ and docs/architecture/ were neither of them. Both are now declared as generated: rebuilt from a source, never corrected in place. A wrong row in either is a bug in the generator or in one of its hand-written inputs, and editing the output makes the next rebuild silently undo the fix. The eval index gains the 2026-08-19 CPT+SFT measurement and the 2026-08-26 baseline, and marks the 2026-08-13 capability audit superseded. The 2026-08-19 file lands with them: it is the direct evidence that the deployed maven-instruct-b2 routes better than Qwen3-1.7B and cannot hold a Russian sentence, which is the first item on the gaps.md priority list. Its header records Vikunja as returning 503. That reading was wrong and the 2026-08-26 baseline says so, but a dated eval is not edited after the day. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
108 lines
5.1 KiB
Markdown
108 lines
5.1 KiB
Markdown
# The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence
|
|
|
|
*Measured 2026-08-19 on homesrv, CPU. Unfiled: Vikunja has returned 503 since
|
|
2026-08-13. Build `5cae33a` on master. The deployment was not changed.*
|
|
|
|
The candidate is `/mnt/hdd1/llms/maven/maven-model-Q4_K_XL.gguf`, built on workpc
|
|
from a continued-pretraining stage plus a supervised fine-tune, quantised by
|
|
replaying the Unsloth imatrix recipe with 197 per-tensor overrides read out of
|
|
the reference gguf. The incumbent is
|
|
`/mnt/hdd1/llms/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf`, the deployed resident model.
|
|
|
|
Both arms ran on the same homesrv `llama-server`, same flags, same port, minutes
|
|
apart, one model resident at a time. CPU inference throughout, so **every latency
|
|
figure here is incomparable to any GPU run**, including the 116.7 ms the build
|
|
side reported. Accuracy is the only number that carries.
|
|
|
|
The build side's own measurement claimed 88/96 for the candidate. This run
|
|
reproduces it at 87/96 on the same fixture, so the routing claim stands.
|
|
|
|
## Routing, `TestLLMRouterBaseline`, 96 cases
|
|
|
|
| | full | intent-only | destination | ru | en | p50 |
|
|
|---|---|---|---|---|---|---|
|
|
| maven-model-Q4_K_XL | **87/96 (90.6%)** | 94.8% | **24/33 (72.7%)** | 72/81 | 15/15 | 1.29s |
|
|
| Qwen3-1.7B-UD-Q4_K_XL | 81/96 (84.4%) | 87.5% | 16/33 (48.5%) | 67/81 | 14/15 | 1.22s |
|
|
|
|
Both ran 0 errors and 1 missed clarify on the cascade. Destination is where the
|
|
candidate earns it: eight more correct query sources, 72.7% against 48.5%. That
|
|
equals what `docs/evals/2026-08-09-e4b-vs-12b-routing.md` records for
|
|
gemma-4-12B at seven times the parameter count.
|
|
|
|
The `llm-only` section of the same run reads 59/96 against 45/96. It bypasses
|
|
stage 0 and the daemon slot fillers on purpose. **Do not cite it as the routing
|
|
number**; `cascade+llm` is the deployed path.
|
|
|
|
## Phrasing, `internal/phraser/eval`
|
|
|
|
| | nudges | conversational |
|
|
|---|---|---|
|
|
| maven-model-Q4_K_XL | 15/15 (100%) | **8/36 (22.2%)** |
|
|
| maven-model-Q4_K_XL, eos corrected | 15/15 (100%) | 11/36 (30.6%) |
|
|
| Qwen3-1.7B-UD-Q4_K_XL | 15/15 (100%) | 22/36 (61.1%) |
|
|
|
|
Nudges are tied at ceiling. Conversational phrasing falls by half.
|
|
|
|
## Two defects, and they are independent
|
|
|
|
**The gguf carries the wrong eos token.** `tokenizer.ggml.eos_token_id` is
|
|
151643 (`<|endoftext|>`) against the incumbent's 151645 (`<|im_end|>`), and
|
|
`bos` is 151643 too. Both files carry the identical Qwen3 chat template, and
|
|
that template closes every turn with `<|im_end|>`. The candidate is told to stop
|
|
on a token the template never emits, so generation runs through the turn
|
|
boundary. The conversion read the base config, not the instruct one.
|
|
|
|
Re-running with `--override-kv tokenizer.ggml.eos_token_id=int:151645` isolates
|
|
its cost:
|
|
|
|
| check | as shipped | eos corrected |
|
|
|---|---|---|
|
|
| address | 27/36 | 32/36 |
|
|
| knowledge path | 3/9 | 7/9 |
|
|
| ontopic | 14/36 | 15/36 |
|
|
| reply path | 0/9 | 0/9 |
|
|
|
|
Routing is untouched by this, because GBNF-constrained decoding terminates on
|
|
the grammar and never reaches the eos token. That is why the defect is invisible
|
|
in the routing score and fatal in free generation.
|
|
|
|
**The remaining half is not packaging.** Corrected, the candidate reaches 11/36
|
|
against the incumbent's 22/36. `ontopic` moves one case and the reply path stays
|
|
at 0 of 9. The failures are broken Russian, not persona drift:
|
|
|
|
- `Ты не одиноден?!` — not a word; `одинок`.
|
|
- `Привёз с собой смайлики-котята и вручила им подарок` — masculine and feminine
|
|
verbs in one clause, wrong case on the noun.
|
|
- `Вчера у вас не будет никакого лёг.` — yesterday in the future tense, cut off
|
|
mid-word.
|
|
- `Заберите диск от системы или установочный ковш.` — "installation bucket".
|
|
- `Вам сказал Игоря Валерьевич` — a first name declined as accusative.
|
|
- `know-hiccups` answered with weight-loss advice.
|
|
|
|
## What this rules out
|
|
|
|
- **Not a harness mismatch.** Both arms ran the identical target, server and
|
|
fixture, and the candidate reproduced the build side's routing claim to within
|
|
one case.
|
|
- **Not the missing `ROLEPLAY_ACTION` and `MASC_SELF` fixes alone.** Those were
|
|
the build side's stated reason to expect persona failures. `feminine` scores
|
|
35/36 corrected. The failures are grammatical, not persona.
|
|
- **Not weights damage alone either.** The eos defect is real and costs five
|
|
address cases and four knowledge cases on its own.
|
|
|
|
`inference:` the SFT stage worked and the CPT stage cost general Russian
|
|
coherence. Routing is constrained decoding over 7 intents and a destination, and
|
|
it improved. Free generation is where the loss shows.
|
|
|
|
## The swap was not made
|
|
|
|
`deploy/mavend.json:19` still reads the incumbent. One `phraser.model_path`
|
|
serves both the router and the phraser and `voice.llm_router` is true, so
|
|
swapping buys 6 routing cases and pays 11 conversational ones. Both ggufs stay
|
|
at `/mnt/hdd1/llms/maven/` for the re-run.
|
|
|
|
The two-week replay (`scripts/usage-run.py`,
|
|
`docs/evals/2026-08-08-two-weeks.md`) was not run. It drives the deployed stack
|
|
through `POST /api/chat`, so it measures whatever `mavend` has loaded, and the
|
|
candidate was never loaded.
|