Maven/ROUTING-EVAL-31-07-2026.md

Routing evaluation — 31-07-2026

Settles Vikunja #319 ("measure classifier vs LLM router before flipping"). Everything below is measured against one held-out fixture, not argued from the code.

• Fixture + scorer: internal/router/eval/ (ru_routing_v1.json, 76 cases; eval.go)
• Reproduce: make eval-router (classifier baselines) and MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router (adds the LLM configurations)
• Commits: c7c4422 (fixture), d34fdf4 (ONNX baseline), 46259b4 (LLM baseline)

Why a new fixture

cmd/mavend/eval_scenarios_test.go could not answer #319: it asserts daemon-side safety invariants over already-normalized decisions, so it never exercises routing. And the only utterance corpus that existed — models/seeds/*.txt — is the classifier's own training set. Scoring a nearest-centroid classifier there measures memorisation of frozen centroids, which is exactly the illusion behind voice.go:211's "the classifier handles routing reliably".

TestFixtureIsHeldOut fails the build if any fixture utterance appears verbatim in the seed corpus. The fixture is a contract, not a snapshot: cases the cascade fails today stay in the file and fail loudly.

Results

	classifier+hash	classifier+onnx	llm-only (0.8B)	cascade+llm (0.8B)
intent-only accuracy	17.1%	36.8%	48.7%	50.0%
full accuracy (intent+slots+gate)	17.1%	36.8%	23.7%	32.9%
RU	10/61	25/61	13/61	18/61
EN	3/15	3/15	5/15	7/15
hard tag	0/11	4/11	—	—
false clarify (asked, shouldn't)	63	21	0	2
missed clarify (guessed, shouldn't)	0 / 6	5 / 6	6 / 6	6 / 6
route errors	0	0	2	0
p50 / p95 / max latency	9µs / 14µs	31ms / 71ms	850ms / 1.56s / 3.1s	825ms / 1.20s / 3.0s

classifier+hash is the CI ratchet (deterministic, no model files). classifier+onnx is what homesrv runs today. cascade+llm is the wiring #320 proposes: stage-0 grammar → resident model → classifier as failure floor.

Never compare a hash-embedder run to an ONNX one.

Findings

1. The resident model does route better — 50.0% vs 36.8%

REARCH.md's premise holds; voice.go:211's comment does not. But the classifier is only ~37% correct on held-out utterances, and the model only ~50%. Neither is "reliable". The gap between them is real but both are far from a system you would describe as working.

2. It costs 27× the latency

p50 825ms vs 31ms, p95 1.2s, max 3.0s — on the same llama-server the phraser needs, before any phrasing happens. On the CPU/iGPU deploy target this is a trade, not a free win. The review's second-opinion caution was justified.

3. query→fact ×15 is the dominant LLM failure — and it is a prompt bug

Four times the classifier's ×4 on the same axis. routeSystem's decision order in internal/router/llmrouter.go reads:

3. Сообщает или обновляет текущее состояние/событие → fact
4. Хочет получить информацию → query

Any utterance naming a fact key matches rule 3 first, so a question about past state ("сколько воды я выпил с утра", "сколько раз я ел вчера") is classified as an assertion of that state — and a query becomes a confident wrong write. Reordering query above fact, or adding an explicit interrogative test, is the cheapest accuracy win available and needs no model change.

4. Neither path can refuse — the refusal lane is currently fiction

	missed clarify	why
classifier+hash	0 / 6	cosine never clears 0.55 — refuses by accident
classifier+onnx	5 / 6	better embeddings raise cosine everywhere; the gate stops separating
LLM (any)	6 / 6	llmrouter.go hardcodes Confidence: 1.0, so stage 3 can never fire

The deployed config confidently routes сделай это → act at 0.847, ну это → chat at 0.808, бэкап → chat at 0.755, потом → system at 0.739. сделай это → act with unresolved anaphora is the destructive direction; the daemon's confirm gate is the only thing left.

This is the finding that should block #320. Flipping to the LLM router as-is does not improve the refusal lane — it removes it. Tracked as #359.

5. The 50.0% → 32.9% gap is entirely slots

The LLM path fills neither Fn nor Time: it returns Slots.Text for acts (the verb string, not an allowlist match), and Extractor.Extract never runs on an LLM decision at all. Any flip needs the extractor wired onto the LLM branch or every act and reminder arrives without its arguments.

6. The 2 route errors are a missing RepeatPenalty, not a grammar flaw

Both failures (ru-act-006 "закрой жалюзи", ru-chat-003 "расскажи анекдот про программистов") are the sub-1B repetition loop inside the grammar's text field:

"Закрывание жалюзи — это действие, которое нужно выполнить. Если это не действие, то это сообщение пользователя. Если это не действие, то это сообщение пользователя. …"

It runs to MaxTokens: 128, truncates the JSON mid-string, and parseActions fails → fallback to the classifier. llm.Req already has a RepeatPenalty field added for exactly this ("curbs the sub-1B 'тоже тоже тоже' loop") and LLMRouter.Route does not set it. Two lines.

Note the grammar's string ::= "\"" ([^"\\] | "\\" .)* "\"" is unbounded, so nothing stops a 1000-character text. Worth a length bound as well.

7. Two hypotheses tested and closed

• Thinking mode is a non-issue. Qwen3.5's template defaults thinking = 1, so grammar-constrained JSON lands in reasoning_content with content empty — llm.Client's fallback handles it. A thinking off run scored identically (18/76, 48.7%, same p50). internal/llm deliberately does not grow a chat_template_kwargs field.
• Runaway array repetition does not reproduce. An isolated smoke test with a stripped grammar emitted {"intent":"reminder"} until MaxTokens; under the real routeSystem prompt the few-shot examples anchor it to one object. 2 errors in 76, not 76.

8. Incidental

• ReminderGrammar deliberately skips the extractor at stage 0; the daemon's applyAction parses the time downstream. The scorer counts those as SlotsDeferred rather than misses.
• A local llama-server must bypass http_proxy — this box proxies loopback through a SOCKS bridge that answers 503. noProxyLoopback in the test handles it.
• The onnxruntime .so was already vendored at deps/onnxruntime-linux-x64-1.26.0.

Next steps

Ordered by ratio of value to risk. Nothing here is a decision — #320 stays open.

1. Fix routeSystem's decision order (query above fact, or an explicit interrogative test). Largest single accuracy move, no model change, re-measurable in one command. Expected: most of query→fact ×15.
2. Set RepeatPenalty in LLMRouter.Route and bound the grammar's string length. Removes both route errors.
3. Give the router a refusal signal — #359. Blocks #320.
   • Classifier: the absolute-cosine gate does not survive a better embedder. A margin gate (top1 − top2 > δ) is the likely fix — ambiguous utterances should show flat distributions, which absolute cosine cannot see.
   • LLM: Confidence: 1.0 must go. Either add an unclear intent to the grammar enum, or read logprobs, or gate on the classifier's margin behind the LLM decision.
   • Bar: MissedClarify ≤ 1 without regressing full accuracy below 28/76.
4. Wire Extractor.Extract onto the LLM branch so acts get Fn and reminders get Time. Closes the 50.0% → 32.9% slot gap.
5. Re-measure, then decide #320. At p50 825ms a wholesale swap is probably the wrong shape; the honest candidate is LLM-for-queries with the classifier keeping the fast deterministic paths (stage-0 grammar hits, system, exact acts). That hypothesis is testable against this fixture by scoring a per-intent split.
6. Grow the fixture as failures get understood. 76 cases with ≥5 per intent is enough to rank paths, not enough to trust a 2-point difference. Add cases from real misroutes (CorrectMisroute is already the append-only hook).
7. Second checkpoint when #122 lands. The CPT'd Qwen3-1.7B is the target resident model; the same three configurations should be re-scored against it before it deploys. 0.8B's 50.0% is the floor that checkpoint has to beat, and its latency is the number that decides whether the target is affordable at all.

Open question worth naming

Both paths are under 50%. That is low enough that the interesting question may not be "classifier or model" but whether one-shot classification of a bare utterance is the right frame at all — сделай это, потом, бэкап are unanswerable without dialogue context, and internal/router currently sees none (AnaphoraResolver exists in slots.go but the cascade never calls it). A router that could ask one clarifying question and re-route on the answer would beat both numbers here without a better model.