claude
c15c2b7bd2
CLAUDE.md said the destination had no fixture and no accuracy number. It has both now: intent 73/96 and destination 12/33 on the classifier cascade, with the per-destination split, the floor cases and the grammar drift the labelling turned up. Anyone adding a grammar now reads that baselineGrammars mirrors buildRouter and drifts silently when it does not. docs/evals/2026-08-08-massive-warm-start.md was written on the V-655 branch and parked in .task/, which git excludes, so it was one `task start` away from being lost. It is a dated measurement and it belongs under docs/evals whatever branch produced it. Its "destination has no fixture at all" line is now a pointer to the file beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
83 lines
4.0 KiB
Markdown
83 lines
4.0 KiB
Markdown
# The first destination number
|
|
|
|
Measured 2026-08-08 on the classifier cascade with the ONNX multilingual
|
|
embedder, the configuration homesrv runs. `make t PKG=./internal/router/eval/
|
|
RUN=TestONNXBaseline V=1`. Covers V-659, the follow-up V-655 named.
|
|
|
|
## What was measured
|
|
|
|
V-655 split a routing decision in two. The cascade sorts an utterance into one
|
|
of seven intents, and `Decision.Source` then says where the answer lives. The
|
|
first half had a fixture. The second half arrived with none, so twelve
|
|
destinations shipped with no accuracy number.
|
|
|
|
`want_source` is now a field on `eval.Case`. It is a pointer, because the
|
|
destination has three states and a bare string has two. Absent is every intent
|
|
but query, which never reaches `queryWalk`. Present and empty is the
|
|
`SourceUnknown` contract: name nothing and let the daemon walk the chain.
|
|
Present and named is a destination the route must produce.
|
|
|
|
Thirty-three of the ninety-six cases carry one. A destination miss does not
|
|
fail the case, so `Accuracy` and `IntentAccuracy` mean what they meant.
|
|
`SourceAccuracy` is a second number over the labelled cases only.
|
|
|
|
## Result
|
|
|
|
Intent is **73/96 (76.0%)**, against 69/91 (75.8%) before. Four of the five new
|
|
cases pass and no existing case moved.
|
|
|
|
Destination is **12/33 (36.4%)**, and the split is the whole finding.
|
|
|
|
| destination | scored | note |
|
|
|---|---|---|
|
|
| world | 5/5 | `WorldQueryGrammars` names it at stage 0 |
|
|
| the `SourceUnknown` floor | 5/7 | the two misses lost the intent first |
|
|
| calendar | 2/6 | `calendar-query` names it, the possessive agenda rules do not |
|
|
| recall | 0/15 | nothing anywhere names it |
|
|
|
|
Recall is the number to move. Fifteen cases ask about his own words and his own
|
|
facts. The route lands `query` on eleven of them and the destination comes back
|
|
empty every time. Those turns are answered today, because the daemon walks the
|
|
chain in order and the three recall passes are early in it. What is missing is a
|
|
decider that says so, and that is the fourth head on V-546.
|
|
|
|
Two cases labelled the floor lost their intent before a destination was
|
|
possible. A clarify names nothing, so it would satisfy an empty label for free.
|
|
`Score` requires the route to land the case's intent before it credits a
|
|
destination hit, or the floor label would score itself.
|
|
|
|
## Seven cases assert the floor, and six of them cluster
|
|
|
|
The six are homelab operations. `SourceRecall`, `SourceNetwork` and
|
|
`SourceAttention` overlap on every question about the box, because `mavpoll`
|
|
writes its netdata and uptime-kuma observations into the fact store recall
|
|
reads. "почему сервер тормозит" is answerable from all three. Naming one takes
|
|
the other two off the turn.
|
|
|
|
That is a finding about the enum rather than a gap in the labelling. The floor
|
|
is the right answer there and the fixture now says so out loud.
|
|
|
|
## A drift the labelling found
|
|
|
|
`WorldQueryGrammars` went into `buildRouter` with V-655 and never into
|
|
`baselineGrammars`, the fixture's mirror of it. So the fixture was scoring a
|
|
grammar set the daemon does not run. The comment above that function forbids
|
|
exactly that. Adding it moved the destination number from 9/33 to 12/33 and
|
|
moved nothing else.
|
|
|
|
The three cases it recovered are `что такое TCP?`, `сколько будет 17 на 23?`
|
|
and `кто такой Линус Торвальдс?`. All three already routed `query` through
|
|
`NarrativeQueryGrammars`. So the drift was invisible to every number this
|
|
fixture reported, until the destination had one of its own.
|
|
|
|
## What this does not measure
|
|
|
|
The model arm. This is the classifier cascade, which names a destination only
|
|
where a stage 0 rule filled one in. The resident model has no destination in
|
|
its router prompt yet, so 36.4% is a floor and not a comparison.
|
|
|
|
Two pairs of cases are the same utterance. `ru-query-020` and `ru-query-024`
|
|
are both "что дальше?", and `ru-query-021` and `ru-query-025` are both
|
|
"расскажи про битву при Ватерлоо". They differ in tags and note only, so both
|
|
pairs are counted twice here and in every earlier number this fixture reported.
|