diff --git a/CLAUDE.md b/CLAUDE.md index 7262f50..a6c98e8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -433,9 +433,25 @@ not a gap in the labelling. so the fixture scored a grammar set nobody runs. Fixed by V-659, worth 3 points of destination and nothing else. Check that function when adding a grammar. -The model arm is still the follow-up. It lands on V-546. Intent, mood and BIO slot -tags were already three heads on one forward pass of the resident e5-small. -Destination is a fourth head on the same pass. +**The model arm landed the same day** (V-660, +`docs/evals/2026-08-08-destination-model-arm.md`). `routeGrammar` carries a +`source` rule closed over `router.Sources` plus the empty floor, so the model +cannot emit a destination that does not exist. The prompt lists the twelve in +Russian and says `""` is a normal answer to give often. `LLMRouter.Route` reads it +back through `ValidSource` and on `IntentQuery` alone. Against gemma-4-12b on the +workstation the cascade scores destination **24/33 (72.7%)** with intent unmoved +at 84.4%, and **recall goes 0/15 to 14/15**. The resident Qwen3-1.7B is +unmeasured, because it binds `--port 0` inside the container. + +**Stage 0 now costs four destination points.** It did not before. The four cases +the cascade loses and the model alone wins are all calendar. The possessive +agenda rules claim them first and name nothing on purpose. That caution was free +while nothing downstream could name anything either. It is not free now, and the +fix is the owner's call rather than a quiet edit. + +The last arm is V-546. Intent, mood and BIO slot tags were already three heads on +one forward pass of the resident e5-small. Destination is a fourth head on the +same pass, and 72.7% from a 12B teacher is the label source for training it. ## LLM output contract diff --git a/docs/evals/2026-08-08-destination-model-arm.md b/docs/evals/2026-08-08-destination-model-arm.md new file mode 100644 index 0000000..efc69eb --- /dev/null +++ b/docs/evals/2026-08-08-destination-model-arm.md @@ -0,0 +1,66 @@ +# The destination, with a model that can name one + +Measured 2026-08-08 against gemma-4-12b on the workstation, the same 96-case +fixture V-659 built. Covers V-660. + +```sh +no_proxy='*' MAVEN_LLM_URL=http://192.168.1.105:8080 \ + make t PKG=./internal/router/eval/ RUN=TestLLMRouterBaseline V=1 +``` + +## The gap was structural + +V-659 measured the destination at 12/33 on the classifier cascade, with recall +at 0/15. Nothing in `routeSystem` named a `Source` and `routeGrammar` could not +emit one, so the resident model had no string to write. That is the shape V-517 +measured for Praxis reach at 0/12: not a weak model, an absent contract. + +`routeGrammar` now carries a `source` rule closed over `router.Sources` plus the +empty floor. The prompt lists the twelve destinations in Russian and says that +`""` is a normal answer to give often. + +## Result + +| run | intent | destination | +|---|---|---| +| classifier + ONNX (V-659) | 73/96 (76.0%) | 12/33 (36.4%) | +| gemma-4-12b alone | 79/96 intent-only (82.3%) | 26/33 (78.8%) | +| cascade + gemma-4-12b + hash fallback | 81/96 (84.4%) | 24/33 (72.7%) | + +Recall is the move: 0/15 to 14/15. Intent did not shift, which was the +constraint. The prompt is shared, so a destination rule that costs routing +points is not a win. + +The eight llm-only errors are the eight `want_clarify` cases. The model returned +`unknown` on every one, which is correct, and the llm-only harness surfaces a +decline as an error by design. + +## Stage 0 now costs four destination points + +The four cases the cascade loses and the model alone wins are all calendar. The +possessive agenda rules claim them at stage 0 and deliberately name nothing. +"что у меня в списке покупок" matches the same rule. Naming the calendar there +would take the list source off the turn (V-655). + +So a rule written to be careful about the list now blocks a model that would +have named the calendar correctly. Before V-660 that caution was free, because +nothing downstream of stage 0 could name anything either. + +Three ways out, and each costs something. Split the possessive rule so the +calendar-shaped half names its destination. Let a later stage overwrite an empty +destination a grammar left behind, which reverses "a matched value always wins". +Or leave it, on the argument that four points is cheap next to a wrong +destination on a shopping list. This wants the owner's call rather than a quiet +edit. + +## What this does not measure + +The resident Qwen3-1.7B, which is what homesrv runs. It binds `--port 0` inside +the container and no host process can reach it. Scoring it needs a second +llama-server on a fixed port. The workstation is never assumed +up, so the homesrv number is the one that decides whether this ships on by +default. + +The fixture is 33 labelled destinations over twelve values. Recall carries 15 of +them and five destinations carry none at all. A per-destination number below +world, recall, calendar and the floor is not supported by this fixture.