kami/Maven

Files

T

claude 7f804b84e7 Declare the two generated doc tiers and index their evals (V-725)

docs/CLAUDE.md defined four tiers and docs/capabilities/ and docs/architecture/
were neither of them. Both are now declared as generated: rebuilt from a source,
never corrected in place. A wrong row in either is a bug in the generator or in
one of its hand-written inputs, and editing the output makes the next rebuild
silently undo the fix.

The eval index gains the 2026-08-19 CPT+SFT measurement and the 2026-08-26
baseline, and marks the 2026-08-13 capability audit superseded. The 2026-08-19
file lands with them: it is the direct evidence that the deployed
maven-instruct-b2 routes better than Qwen3-1.7B and cannot hold a Russian
sentence, which is the first item on the gaps.md priority list.

Its header records Vikunja as returning 503. That reading was wrong and the
2026-08-26 baseline says so, but a dated eval is not edited after the day.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-26 12:45:54 +04:00

7.9 KiB

Raw Blame History

docs/evals/

One file per measurement, named for the day it was taken. Never edited after that day. A newer number is a new file, never an edit to an old one.

Reasoning does not belong here. It belongs in the subsystem's living doc under docs/. An eval holds the setup, the numbers and what they rule out.

Rules for this directory

Filename is YYYY-MM-DD-<what-was-measured>.md. The date is the day it ran.
The first line is the claim, not the topic. A reader picks a file from this index without opening it, so the H1 has to carry the finding.
Head the file with the date, the task id, the box and the build.
A file that replaces an older number says so. It names the file it replaces. This index then marks that one superseded.
A superseded file is not deleted and not edited. It records what was believed that day. A living doc may still cite it for the run itself.

Reading a number out of here

Check the state column before citing a row. A number in a living doc must cite a live file. The 2026-08-11 classifier baseline exists for that reason. A pair in docs/routing.md went stale unnoticed. Its source predated the encodeWord fix at feabf9f. Nothing failed (V-704).

Routing: the arms and the cascade

measurement	state
Routing evaluation	superseded
Resident model bake-off	live
gemma-4-12b on the workstation, against the resident Qwen3-1.7B	live
Routing from audio: four paths, one fixture	live
Routing with the resident model, re-measured	live
The routing trajectory, and the number that is missing	live
Gemma as a label function, and what it found in the seeds	live
Moving the seed files onto the router prompt's boundaries	live
The first destination number	superseded
The destination, with a model that can name one	live
The routing heads, running in Go	live
Two heads on e5-small, and the first destination the router did not need a model for	live
A slot head, and the corpus that did not exist this morning	live
A clarify head, and a confidence that is not a hardcode	live
MASSIVE Russian warm-start for the routing heads	live
gemma-4-E4B against gemma-4-12B on the routing fixture	live
The classifier baseline after the tokenizer fix	live
The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence	live

docs/routing.md holds the arm table these feed. Cite from there, not from here.

Stage 0, slots and the turn

measurement	state
Five turns retain one referent across fact, query and chat routes	live
Personal boundary, seed scoring vs possession markers	superseded
Three neighbours keep a public noun from opening the personal boundary	superseded
A class-balanced linear head closes every held-out personal-boundary cell	live
Half-past and quarter-to hours	live
Praxis reach at stage 0	live
Alarm verbs reach stage 0	live
Russian acts reach tools	live
The fact parser: closed classes against the substring stems they replaced	live
The reminder completeness rule, measured on the box	live
Nine claimants interrupting a parked reminder	live
The parked clarify ride, bounded and re-measured	live

Ecosystem

measurement	state
Ecosystem reach, measured	live
Ecosystem reach with the resident model as router	live

Recall and memory

measurement	state
Note recall evaluation	superseded
Recall topic veto, what it costs and what it buys	live
Note recall after the e5-small swap	live
Qwen3-1.7B is not safe as a recall-answerability verifier	live

Phrasing and talk

measurement	state
Phrasing evaluation	superseded
Conversational phrasing eval	superseded
Talk fixture against the resident model	live
Talk temperature sweep: Qwen3-1.7B, 4 temperatures times 3 runs	live
gemma-4-E4B on the phrasing and talk fixtures	live
The CPT+SFT Qwen3-1.7B routes better and cannot hold a sentence	live

World: search and Kiwix

measurement	state
Does the ZIM answer when the line is down?	live
Does SearXNG claim a question it cannot answer?	live
Kiwix answered the wrong question, and the fix was not a relevance gate	live

Speech in and out

measurement	state
CrisperWhisper 2.0 in Russian, measured	live
silero-vad against the energy threshold in mavwaked	live
The "Мэйвен" wake word: what it hears and what it invents	live

Runtime and storage

measurement	state
Both ONNX routing gates execute in one process	live
Where the resident model's 7.9GB of RSS goes	live
Does one sqlite connection make reads queue? No	live

Whole-system runs

Each run pairs a write-up with its raw transcript. The transcript is the evidence and is not summarised anywhere else.

measurement	state
A week of talking to Maven, transcript	live
Two weeks of talking to Maven, as a baseline to re-run, transcript	live
The same fortnight, after V-655 merged, transcript	live

Audit

measurement	state
Repository deep-audit report	live
What 39 capabilities actually did on the box	superseded by 2026-08-26
Capability baseline: what the deployed Maven actually does	live

The 2026-08-26 baseline scores all 146 v1 DoD criteria in docs/spec.md and is cited by every verified cell in docs/capabilities/ledger.yaml. It measures maven-instruct-b2-Q4_K_XL, not the Qwen3-1.7B CLAUDE.md names, and it expires when the model or deploy/mavend.json moves.

Its open findings live in docs/caveats/, one entry each with a revisit trigger. Read the index there, not this file, for what is still broken.

7.9 KiB Raw Blame History