claude
e69f1bd0cf
The first 120 floor rows carried one sentence shape, because the generator varies a topic and ambiguity is not a topic. Rotating six shapes takes the floor 3/7 to 6/7 and the destination mean 75.8% to 80.8%. Calendar stays 3/6 at every seed. The possessive agenda rules claim those cases at stage 0 and name nothing, so no label reaches the head. Also corrects the floor-case count in three files: five of the seven are homelab, not six.
85 lines
4.1 KiB
Markdown
85 lines
4.1 KiB
Markdown
# The first destination number
|
|
|
|
Measured 2026-08-08 on the classifier cascade with the ONNX multilingual
|
|
embedder, the configuration homesrv runs. `make t PKG=./internal/router/eval/
|
|
RUN=TestONNXBaseline V=1`. Covers V-659, the follow-up V-655 named.
|
|
|
|
## What was measured
|
|
|
|
V-655 split a routing decision in two. The cascade sorts an utterance into one
|
|
of seven intents, and `Decision.Source` then says where the answer lives. The
|
|
first half had a fixture. The second half arrived with none, so twelve
|
|
destinations shipped with no accuracy number.
|
|
|
|
`want_source` is now a field on `eval.Case`. It is a pointer, because the
|
|
destination has three states and a bare string has two. Absent is every intent
|
|
but query, which never reaches `queryWalk`. Present and empty is the
|
|
`SourceUnknown` contract: name nothing and let the daemon walk the chain.
|
|
Present and named is a destination the route must produce.
|
|
|
|
Thirty-three of the ninety-six cases carry one. A destination miss does not
|
|
fail the case, so `Accuracy` and `IntentAccuracy` mean what they meant.
|
|
`SourceAccuracy` is a second number over the labelled cases only.
|
|
|
|
## Result
|
|
|
|
Intent is **73/96 (76.0%)**, against 69/91 (75.8%) before. Four of the five new
|
|
cases pass and no existing case moved.
|
|
|
|
Destination is **12/33 (36.4%)**, and the split is the whole finding.
|
|
|
|
| destination | scored | note |
|
|
|---|---|---|
|
|
| world | 5/5 | `WorldQueryGrammars` names it at stage 0 |
|
|
| the `SourceUnknown` floor | 5/7 | the two misses lost the intent first |
|
|
| calendar | 2/6 | `calendar-query` names it, the possessive agenda rules do not |
|
|
| recall | 0/15 | nothing anywhere names it |
|
|
|
|
Recall is the number to move. Fifteen cases ask about his own words and his own
|
|
facts. The route lands `query` on eleven of them and the destination comes back
|
|
empty every time. Those turns are answered today, because the daemon walks the
|
|
chain in order and the three recall passes are early in it. What is missing is a
|
|
decider that says so, and that is the fourth head on V-546.
|
|
|
|
Two cases labelled the floor lost their intent before a destination was
|
|
possible. A clarify names nothing, so it would satisfy an empty label for free.
|
|
`Score` requires the route to land the case's intent before it credits a
|
|
destination hit, or the floor label would score itself.
|
|
|
|
## Seven cases assert the floor, and five of them cluster
|
|
|
|
The five are homelab operations. `SourceRecall`, `SourceNetwork` and
|
|
`SourceAttention` overlap on every question about the box, because `mavpoll`
|
|
writes its netdata and uptime-kuma observations into the fact store recall
|
|
reads. "почему сервер тормозит" is answerable from all three. Naming one takes
|
|
the other two off the turn.
|
|
|
|
That is a finding about the enum rather than a gap in the labelling. The floor
|
|
is the right answer there and the fixture now says so out loud.
|
|
|
|
All seven were written by an agent and confirmed by the owner on 08-08-2026.
|
|
|
|
## A drift the labelling found
|
|
|
|
`WorldQueryGrammars` went into `buildRouter` with V-655 and never into
|
|
`baselineGrammars`, the fixture's mirror of it. So the fixture was scoring a
|
|
grammar set the daemon does not run. The comment above that function forbids
|
|
exactly that. Adding it moved the destination number from 9/33 to 12/33 and
|
|
moved nothing else.
|
|
|
|
The three cases it recovered are `что такое TCP?`, `сколько будет 17 на 23?`
|
|
and `кто такой Линус Торвальдс?`. All three already routed `query` through
|
|
`NarrativeQueryGrammars`. So the drift was invisible to every number this
|
|
fixture reported, until the destination had one of its own.
|
|
|
|
## What this does not measure
|
|
|
|
The model arm. This is the classifier cascade, which names a destination only
|
|
where a stage 0 rule filled one in. The resident model has no destination in
|
|
its router prompt yet, so 36.4% is a floor and not a comparison.
|
|
|
|
Two pairs of cases are the same utterance. `ru-query-020` and `ru-query-024`
|
|
are both "что дальше?", and `ru-query-021` and `ru-query-025` are both
|
|
"расскажи про битву при Ватерлоо". They differ in tags and note only, so both
|
|
pairs are counted twice here and in every earlier number this fixture reported.
|