kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Declare the two generated doc tiers and index their evals (V-725)

Browse Source 
docs/CLAUDE.md defined four tiers and docs/capabilities/ and docs/architecture/
were neither of them. Both are now declared as generated: rebuilt from a source,
never corrected in place. A wrong row in either is a bug in the generator or in
one of its hand-written inputs, and editing the output makes the next rebuild
silently undo the fix.

The eval index gains the 2026-08-19 CPT+SFT measurement and the 2026-08-26
baseline, and marks the 2026-08-13 capability audit superseded. The 2026-08-19
file lands with them: it is the direct evidence that the deployed
maven-instruct-b2 routes better than Qwen3-1.7B and cannot hold a Russian
sentence, which is the first item on the gaps.md priority list.

Its header records Vikunja as returning 503. That reading was wrong and the
2026-08-26 baseline says so, but a dated eval is not edited after the day.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
claude
2026-08-26 12:45:54 +04:00
parent bae81b66c8
commit 7f804b84e7
3 changed files with 122 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CLAUDE.md
+6
View File
@@ -11,6 +11,8 @@ The tier is the path, so staleness is visible from the filename.
| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. Indexed in `docs/evals/CLAUDE.md`, which marks each one live or superseded. | forever |
| `docs/caveats/` | known limits, one entry per limit, each with a task id and a revisit trigger. Indexed in `docs/caveats/CLAUDE.md`. | until fixed, then deleted |
| `docs/plans/` | the plan for one piece of work, frozen once it starts. Indexed in `docs/plans/CLAUDE.md`. | until the work lands |
| `docs/capabilities/` | generated. The capability ledger and its probe harness, regenerated from `docs/spec.md` plus a named eval. **Never hand-edited**, except `domains.yaml`, `probes_field.json` and the scripts, which are its sources. Indexed in `docs/capabilities/README.md`. | until the spec or the measurement moves |
| `docs/architecture/` | generated. The architecture observation and its evidence pack, rebuilt from source by the scripts beside it. Indexed in `docs/architecture/README.md`. | until the shape changes |
| `docs/archive/` | dead. Read by nobody by default. | forever |
## Rules for this directory
@@ -21,6 +23,10 @@ The tier is the path, so staleness is visible from the filename.
correction. Do not append a changelog to it.
* A number in prose with no `docs/evals/` file behind it is an opinion.
* Fixing something deletes its caveat. It does not edit the eval that found it.
* **A generated tier is rebuilt, never corrected.** A wrong row in
`docs/capabilities/` or `docs/architecture/` is a bug in the generator or in
one of its hand-written inputs. Editing the output makes the next rebuild
silently undo the fix.
## Where a subsystem's reasoning lives
docs/evals/2026-08-19-maven-model-cpt-sft.md
+107
View File
@@ -0,0 +1,107 @@
# The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence
*Measured 2026-08-19 on homesrv, CPU. Unfiled: Vikunja has returned 503 since
2026-08-13. Build `5cae33a` on master. The deployment was not changed.*
The candidate is `/mnt/hdd1/llms/maven/maven-model-Q4_K_XL.gguf`, built on workpc
from a continued-pretraining stage plus a supervised fine-tune, quantised by
replaying the Unsloth imatrix recipe with 197 per-tensor overrides read out of
the reference gguf. The incumbent is
`/mnt/hdd1/llms/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf`, the deployed resident model.
Both arms ran on the same homesrv `llama-server`, same flags, same port, minutes
apart, one model resident at a time. CPU inference throughout, so **every latency
figure here is incomparable to any GPU run**, including the 116.7 ms the build
side reported. Accuracy is the only number that carries.
The build side's own measurement claimed 88/96 for the candidate. This run
reproduces it at 87/96 on the same fixture, so the routing claim stands.
## Routing, `TestLLMRouterBaseline`, 96 cases
| | full | intent-only | destination | ru | en | p50 |
|---|---|---|---|---|---|---|
| maven-model-Q4_K_XL | **87/96 (90.6%)** | 94.8% | **24/33 (72.7%)** | 72/81 | 15/15 | 1.29s |
| Qwen3-1.7B-UD-Q4_K_XL | 81/96 (84.4%) | 87.5% | 16/33 (48.5%) | 67/81 | 14/15 | 1.22s |
Both ran 0 errors and 1 missed clarify on the cascade. Destination is where the
candidate earns it: eight more correct query sources, 72.7% against 48.5%. That
equals what `docs/evals/2026-08-09-e4b-vs-12b-routing.md` records for
gemma-4-12B at seven times the parameter count.
The `llm-only` section of the same run reads 59/96 against 45/96. It bypasses
stage 0 and the daemon slot fillers on purpose. **Do not cite it as the routing
number**; `cascade+llm` is the deployed path.
## Phrasing, `internal/phraser/eval`
| | nudges | conversational |
|---|---|---|
| maven-model-Q4_K_XL | 15/15 (100%) | **8/36 (22.2%)** |
| maven-model-Q4_K_XL, eos corrected | 15/15 (100%) | 11/36 (30.6%) |
| Qwen3-1.7B-UD-Q4_K_XL | 15/15 (100%) | 22/36 (61.1%) |
Nudges are tied at ceiling. Conversational phrasing falls by half.
## Two defects, and they are independent
**The gguf carries the wrong eos token.** `tokenizer.ggml.eos_token_id` is
151643 (`<|endoftext|>`) against the incumbent's 151645 (`<|im_end|>`), and
`bos` is 151643 too. Both files carry the identical Qwen3 chat template, and
that template closes every turn with `<|im_end|>`. The candidate is told to stop
on a token the template never emits, so generation runs through the turn
boundary. The conversion read the base config, not the instruct one.
Re-running with `--override-kv tokenizer.ggml.eos_token_id=int:151645` isolates
its cost:
| check | as shipped | eos corrected |
|---|---|---|
| address | 27/36 | 32/36 |
| knowledge path | 3/9 | 7/9 |
| ontopic | 14/36 | 15/36 |
| reply path | 0/9 | 0/9 |
Routing is untouched by this, because GBNF-constrained decoding terminates on
the grammar and never reaches the eos token. That is why the defect is invisible
in the routing score and fatal in free generation.
**The remaining half is not packaging.** Corrected, the candidate reaches 11/36
against the incumbent's 22/36. `ontopic` moves one case and the reply path stays
at 0 of 9. The failures are broken Russian, not persona drift:
- `Ты не одиноден?!` — not a word; `одинок`.
- `Привёз с собой смайлики-котята и вручила им подарок` — masculine and feminine
verbs in one clause, wrong case on the noun.
- `Вчера у вас не будет никакого лёг.` — yesterday in the future tense, cut off
mid-word.
- `Заберите диск от системы или установочный ковш.` — "installation bucket".
- `Вам сказал Игоря Валерьевич` — a first name declined as accusative.
- `know-hiccups` answered with weight-loss advice.
## What this rules out
- **Not a harness mismatch.** Both arms ran the identical target, server and
fixture, and the candidate reproduced the build side's routing claim to within
one case.
- **Not the missing `ROLEPLAY_ACTION` and `MASC_SELF` fixes alone.** Those were
the build side's stated reason to expect persona failures. `feminine` scores
35/36 corrected. The failures are grammatical, not persona.
- **Not weights damage alone either.** The eos defect is real and costs five
address cases and four knowledge cases on its own.
`inference:` the SFT stage worked and the CPT stage cost general Russian
coherence. Routing is constrained decoding over 7 intents and a destination, and
it improved. Free generation is where the loss shows.
## The swap was not made
`deploy/mavend.json:19` still reads the incumbent. One `phraser.model_path`
serves both the router and the phraser and `voice.llm_router` is true, so
swapping buys 6 routing cases and pays 11 conversational ones. Both ggufs stay
at `/mnt/hdd1/llms/maven/` for the re-run.
The two-week replay (`scripts/usage-run.py`,
`docs/evals/2026-08-08-two-weeks.md`) was not run. It drives the deployed stack
through `POST /api/chat`, so it measures whatever `mavend` has loaded, and the
candidate was never loaded.
docs/evals/CLAUDE.md
+9
View File
@@ -45,6 +45,7 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the
| [MASSIVE Russian warm-start for the routing heads](2026-08-08-massive-warm-start.md) | live |
| [gemma-4-E4B against gemma-4-12B on the routing fixture](2026-08-09-e4b-vs-12b-routing.md) | live |
| [The classifier baseline after the tokenizer fix](2026-08-11-classifier-baseline-after-tokenizer-fix.md) | live |
| [The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence](2026-08-19-maven-model-cpt-sft.md) | live |
`docs/routing.md` holds the arm table these feed. Cite from there, not from here.
@@ -90,6 +91,7 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the
| [Talk fixture against the resident model](2026-08-05-talk-fixture-resident.md) | live |
| [Talk temperature sweep: Qwen3-1.7B, 4 temperatures times 3 runs](2026-08-05-temperature-sweep.md) | live |
| [gemma-4-E4B on the phrasing and talk fixtures](2026-08-09-e4b-phrasing.md) | live |
| [The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence](2026-08-19-maven-model-cpt-sft.md) | live |
## World: search and Kiwix
@@ -131,6 +133,13 @@ evidence and is not summarised anywhere else.
| measurement | state |
| --- | --- |
| [Repository deep-audit report](2026-08-10-repo-audit.md) | live |
| [What 39 capabilities actually did on the box](2026-08-13-capability-audit.md) | superseded by 2026-08-26 |
| [Capability baseline: what the deployed Maven actually does](2026-08-26-capability-baseline.md) | live |
The 2026-08-26 baseline scores all 146 v1 DoD criteria in `docs/spec.md` and is
cited by every `verified` cell in `docs/capabilities/ledger.yaml`. It measures
`maven-instruct-b2-Q4_K_XL`, not the Qwen3-1.7B `CLAUDE.md` names, and it
expires when the model or `deploy/mavend.json` moves.
Its open findings live in `docs/caveats/`, one entry each with a revisit
trigger. Read the index there, not this file, for what is still broken.