diff --git a/docs/CLAUDE.md b/docs/CLAUDE.md index 38ca5a7..27ae5cd 100644 --- a/docs/CLAUDE.md +++ b/docs/CLAUDE.md @@ -11,6 +11,8 @@ The tier is the path, so staleness is visible from the filename. | `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. Indexed in `docs/evals/CLAUDE.md`, which marks each one live or superseded. | forever | | `docs/caveats/` | known limits, one entry per limit, each with a task id and a revisit trigger. Indexed in `docs/caveats/CLAUDE.md`. | until fixed, then deleted | | `docs/plans/` | the plan for one piece of work, frozen once it starts. Indexed in `docs/plans/CLAUDE.md`. | until the work lands | +| `docs/capabilities/` | generated. The capability ledger and its probe harness, regenerated from `docs/spec.md` plus a named eval. **Never hand-edited**, except `domains.yaml`, `probes_field.json` and the scripts, which are its sources. Indexed in `docs/capabilities/README.md`. | until the spec or the measurement moves | +| `docs/architecture/` | generated. The architecture observation and its evidence pack, rebuilt from source by the scripts beside it. Indexed in `docs/architecture/README.md`. | until the shape changes | | `docs/archive/` | dead. Read by nobody by default. | forever | ## Rules for this directory @@ -21,6 +23,10 @@ The tier is the path, so staleness is visible from the filename. correction. Do not append a changelog to it. * A number in prose with no `docs/evals/` file behind it is an opinion. * Fixing something deletes its caveat. It does not edit the eval that found it. +* **A generated tier is rebuilt, never corrected.** A wrong row in + `docs/capabilities/` or `docs/architecture/` is a bug in the generator or in + one of its hand-written inputs. Editing the output makes the next rebuild + silently undo the fix. ## Where a subsystem's reasoning lives diff --git a/docs/evals/2026-08-19-maven-model-cpt-sft.md b/docs/evals/2026-08-19-maven-model-cpt-sft.md new file mode 100644 index 0000000..3dad89d --- /dev/null +++ b/docs/evals/2026-08-19-maven-model-cpt-sft.md @@ -0,0 +1,107 @@ +# The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence + +*Measured 2026-08-19 on homesrv, CPU. Unfiled: Vikunja has returned 503 since +2026-08-13. Build `5cae33a` on master. The deployment was not changed.* + +The candidate is `/mnt/hdd1/llms/maven/maven-model-Q4_K_XL.gguf`, built on workpc +from a continued-pretraining stage plus a supervised fine-tune, quantised by +replaying the Unsloth imatrix recipe with 197 per-tensor overrides read out of +the reference gguf. The incumbent is +`/mnt/hdd1/llms/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf`, the deployed resident model. + +Both arms ran on the same homesrv `llama-server`, same flags, same port, minutes +apart, one model resident at a time. CPU inference throughout, so **every latency +figure here is incomparable to any GPU run**, including the 116.7 ms the build +side reported. Accuracy is the only number that carries. + +The build side's own measurement claimed 88/96 for the candidate. This run +reproduces it at 87/96 on the same fixture, so the routing claim stands. + +## Routing, `TestLLMRouterBaseline`, 96 cases + +| | full | intent-only | destination | ru | en | p50 | +|---|---|---|---|---|---|---| +| maven-model-Q4_K_XL | **87/96 (90.6%)** | 94.8% | **24/33 (72.7%)** | 72/81 | 15/15 | 1.29s | +| Qwen3-1.7B-UD-Q4_K_XL | 81/96 (84.4%) | 87.5% | 16/33 (48.5%) | 67/81 | 14/15 | 1.22s | + +Both ran 0 errors and 1 missed clarify on the cascade. Destination is where the +candidate earns it: eight more correct query sources, 72.7% against 48.5%. That +equals what `docs/evals/2026-08-09-e4b-vs-12b-routing.md` records for +gemma-4-12B at seven times the parameter count. + +The `llm-only` section of the same run reads 59/96 against 45/96. It bypasses +stage 0 and the daemon slot fillers on purpose. **Do not cite it as the routing +number**; `cascade+llm` is the deployed path. + +## Phrasing, `internal/phraser/eval` + +| | nudges | conversational | +|---|---|---| +| maven-model-Q4_K_XL | 15/15 (100%) | **8/36 (22.2%)** | +| maven-model-Q4_K_XL, eos corrected | 15/15 (100%) | 11/36 (30.6%) | +| Qwen3-1.7B-UD-Q4_K_XL | 15/15 (100%) | 22/36 (61.1%) | + +Nudges are tied at ceiling. Conversational phrasing falls by half. + +## Two defects, and they are independent + +**The gguf carries the wrong eos token.** `tokenizer.ggml.eos_token_id` is +151643 (`<|endoftext|>`) against the incumbent's 151645 (`<|im_end|>`), and +`bos` is 151643 too. Both files carry the identical Qwen3 chat template, and +that template closes every turn with `<|im_end|>`. The candidate is told to stop +on a token the template never emits, so generation runs through the turn +boundary. The conversion read the base config, not the instruct one. + +Re-running with `--override-kv tokenizer.ggml.eos_token_id=int:151645` isolates +its cost: + +| check | as shipped | eos corrected | +|---|---|---| +| address | 27/36 | 32/36 | +| knowledge path | 3/9 | 7/9 | +| ontopic | 14/36 | 15/36 | +| reply path | 0/9 | 0/9 | + +Routing is untouched by this, because GBNF-constrained decoding terminates on +the grammar and never reaches the eos token. That is why the defect is invisible +in the routing score and fatal in free generation. + +**The remaining half is not packaging.** Corrected, the candidate reaches 11/36 +against the incumbent's 22/36. `ontopic` moves one case and the reply path stays +at 0 of 9. The failures are broken Russian, not persona drift: + +- `Ты не одиноден?!` — not a word; `одинок`. +- `Привёз с собой смайлики-котята и вручила им подарок` — masculine and feminine + verbs in one clause, wrong case on the noun. +- `Вчера у вас не будет никакого лёг.` — yesterday in the future tense, cut off + mid-word. +- `Заберите диск от системы или установочный ковш.` — "installation bucket". +- `Вам сказал Игоря Валерьевич` — a first name declined as accusative. +- `know-hiccups` answered with weight-loss advice. + +## What this rules out + +- **Not a harness mismatch.** Both arms ran the identical target, server and + fixture, and the candidate reproduced the build side's routing claim to within + one case. +- **Not the missing `ROLEPLAY_ACTION` and `MASC_SELF` fixes alone.** Those were + the build side's stated reason to expect persona failures. `feminine` scores + 35/36 corrected. The failures are grammatical, not persona. +- **Not weights damage alone either.** The eos defect is real and costs five + address cases and four knowledge cases on its own. + +`inference:` the SFT stage worked and the CPT stage cost general Russian +coherence. Routing is constrained decoding over 7 intents and a destination, and +it improved. Free generation is where the loss shows. + +## The swap was not made + +`deploy/mavend.json:19` still reads the incumbent. One `phraser.model_path` +serves both the router and the phraser and `voice.llm_router` is true, so +swapping buys 6 routing cases and pays 11 conversational ones. Both ggufs stay +at `/mnt/hdd1/llms/maven/` for the re-run. + +The two-week replay (`scripts/usage-run.py`, +`docs/evals/2026-08-08-two-weeks.md`) was not run. It drives the deployed stack +through `POST /api/chat`, so it measures whatever `mavend` has loaded, and the +candidate was never loaded. diff --git a/docs/evals/CLAUDE.md b/docs/evals/CLAUDE.md index f95b105..36a1c35 100644 --- a/docs/evals/CLAUDE.md +++ b/docs/evals/CLAUDE.md @@ -45,6 +45,7 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the | [MASSIVE Russian warm-start for the routing heads](2026-08-08-massive-warm-start.md) | live | | [gemma-4-E4B against gemma-4-12B on the routing fixture](2026-08-09-e4b-vs-12b-routing.md) | live | | [The classifier baseline after the tokenizer fix](2026-08-11-classifier-baseline-after-tokenizer-fix.md) | live | +| [The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence](2026-08-19-maven-model-cpt-sft.md) | live | `docs/routing.md` holds the arm table these feed. Cite from there, not from here. @@ -90,6 +91,7 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the | [Talk fixture against the resident model](2026-08-05-talk-fixture-resident.md) | live | | [Talk temperature sweep: Qwen3-1.7B, 4 temperatures times 3 runs](2026-08-05-temperature-sweep.md) | live | | [gemma-4-E4B on the phrasing and talk fixtures](2026-08-09-e4b-phrasing.md) | live | +| [The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence](2026-08-19-maven-model-cpt-sft.md) | live | ## World: search and Kiwix @@ -131,6 +133,13 @@ evidence and is not summarised anywhere else. | measurement | state | | --- | --- | | [Repository deep-audit report](2026-08-10-repo-audit.md) | live | +| [What 39 capabilities actually did on the box](2026-08-13-capability-audit.md) | superseded by 2026-08-26 | +| [Capability baseline: what the deployed Maven actually does](2026-08-26-capability-baseline.md) | live | + +The 2026-08-26 baseline scores all 146 v1 DoD criteria in `docs/spec.md` and is +cited by every `verified` cell in `docs/capabilities/ledger.yaml`. It measures +`maven-instruct-b2-Q4_K_XL`, not the Qwen3-1.7B `CLAUDE.md` names, and it +expires when the model or `deploy/mavend.json` moves. Its open findings live in `docs/caveats/`, one entry each with a revisit trigger. Read the index there, not this file, for what is still broken.