Twenty-eight existing query cases get a want_source and five new ones
arrive with theirs. Every label is the destination that SHOULD claim the
turn, which on the five new cases is not the one that did: they were
observed failing on the box on 2026-08-07, so the fixture fails on the day
it is written.
Seven cases assert the SourceUnknown floor, and six of those are homelab
operations. They cluster because SourceRecall, SourceNetwork and
SourceAttention overlap on every question about the box: mavpoll writes its
netdata and uptime-kuma observations into the fact store recall reads.
Naming one destination there takes the other two off a turn that needs
them. That is a finding about the enum, not a gap in the labelling.
The fixture's grammar mirror had drifted. WorldQueryGrammars went into
buildRouter with V-655 and never into baselineGrammars, so the fixture was
scoring a grammar set the daemon does not run — the exact thing the comment
above that function forbids. Adding it moved the destination number 9/33 to
12/33 and moved nothing else.
Measured classifier+onnx: intent 73/96 (76.0%), was 69/91 (75.8%). Four of
the five new cases pass and no existing case moved. Destination 12/33
(36.4%), and the split is the point. World is 5/5, because a stage 0 rule
names it. Calendar is 2/6, because the possessive agenda rules deliberately
do not. Recall is 0/15, because nothing anywhere names it yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
The fixture measured the first half of a route and stopped. V-655 split a
routing decision in two, and the second half arrived with no fixture, so
Decision.Source had no accuracy number at all.
want_source is a pointer because the destination has three states and a
bare string has two. Absent is every intent but query, which never reaches
queryWalk. Present and empty is the SourceUnknown contract: name nothing
and let the daemon walk the chain, which is right whenever two destinations
can both answer and the utterance does not choose. Present and named is a
destination the route must produce.
A destination miss does not fail the case. It goes in SourceReason, never
in Reasons, so Accuracy and IntentAccuracy stay the numbers they were and
69/91 still means what it meant. SourceAccuracy is the second number, over
the labelled cases only, because a percentage of the whole fixture would be
a percentage of turns that never ask a query source.
A clarified or mis-routed case still counts in the denominator. It named no
destination and that is a miss, not a case to skip, or the denominator drops
every turn the route already lost.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
The onnxruntime .so was already vendored at deps/onnxruntime-linux-x64-1.26.0
— nothing to download. make eval-router now defaults MAVEN_ONNX_LIB there, so
both baselines run by default and only a fresh clone without deps/ falls back
to the hash ratchet alone.
Prod-representative result, deployed 0.55 gate: 28/76 (36.8%), RU 25/61,
EN 3/15, hard 0/11 → 4/11, p50 31ms / p95 71ms. Versus the hash floor's
13/76 at p50 9µs.
The finding is not the accuracy, it's the refusal lane: missed clarifies went
0 → 5 of 6. Better embeddings raise cosine everywhere, so the 0.55 threshold
that used to hold ambiguous utterances back stops holding — "сделай это"
routes to act at 0.847, "бэкап" to chat at 0.755. The gate was implicitly
tuned to the hash floor's low similarities. That is an argument about the
threshold, not about the embedder, and it lands before #320 rather than after.
Also fixes a fixture-model mismatch: ReminderGrammar deliberately skips the
extractor at stage 0 and the daemon's applyAction parses the time downstream
(stage0.go says so). Charging the router for that slot made 4 exact-match wins
read as misses; they are now counted as SlotsDeferred instead. Hash baseline
moves 13/76, ratchet to 0.15.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
#319 asks for a measurement before #320 flips the route decider from the
classifier cascade to the resident model. There was nothing to measure
against: the only routing tests assert single utterances, and the
classifier's seed corpus is its own training set — scoring it there
measures memorisation of frozen centroids, which is the illusion that hid
the weak RU query handling in the first place.
internal/router/eval is a separate package so both paths can be scored
from outside router (including cmd/mavend, where the real llama-server
client lives). The fixture is embedded; the scorer takes a Router
interface, so *router.Router and a bare LLM stage both go through the same
76 cases.
The fixture is a CONTRACT, not a snapshot: cases the cascade fails today
stay in the file and fail loudly. TestFixtureIsHeldOut enforces that no
utterance appears verbatim in models/seeds/*.txt.
Baseline, hash embedder at the deployed 0.55 gate: 9/76 (11.8%), 63 false
clarifies, 0 missed clarifies, p50 9µs. Almost everything falls to the
confidence gate — the documented floor behaviour, not a new bug. The
number worth comparing is TestONNXBaseline's (skipped without
MAVEN_ONNX_LIB); the assertions here are a regression ratchet plus a tight
bound on the dangerous direction: ambiguous utterances must not start
being routed confidently.
Seeding is order-fixed on purpose — a few phrases appear under two intents
and map iteration handed them to a different centroid each run, which made
the score jitter between 9 and 10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
claude
kami