Maven/ROUTING-EVAL-31-07-2026.md

Routing evaluation — 31-07-2026

Settles Vikunja #319 ("measure classifier vs LLM router before flipping"). Everything below is measured against one held-out fixture, not argued from the code.

• Fixture + scorer: internal/router/eval/ (ru_routing_v1.json, 76 cases; eval.go)
• Reproduce: make eval-router (classifier baselines) and MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router (adds the LLM configurations)
• Commits: c7c4422 (fixture), d34fdf4 (ONNX baseline), 46259b4 (LLM baseline)

Why a new fixture

cmd/mavend/eval_scenarios_test.go could not answer #319: it asserts daemon-side safety invariants over already-normalized decisions, so it never exercises routing. And the only utterance corpus that existed — models/seeds/*.txt — is the classifier's own training set. Scoring a nearest-centroid classifier there measures memorisation of frozen centroids, which is exactly the illusion behind voice.go:211's "the classifier handles routing reliably".

TestFixtureIsHeldOut fails the build if any fixture utterance appears verbatim in the seed corpus. The fixture is a contract, not a snapshot: cases the cascade fails today stay in the file and fail loudly.

Results

	classifier+hash	classifier+onnx	llm-only (0.8B)	cascade+llm (0.8B)
intent-only accuracy	17.1%	36.8%	48.7%	50.0%
full accuracy (intent+slots+gate)	17.1%	36.8%	23.7%	32.9%
RU	10/61	25/61	13/61	18/61
EN	3/15	3/15	5/15	7/15
hard tag	0/11	4/11	—	—
false clarify (asked, shouldn't)	63	21	0	2
missed clarify (guessed, shouldn't)	0 / 6	5 / 6	6 / 6	6 / 6
route errors	0	0	2	0
p50 / p95 / max latency	9µs / 14µs	31ms / 71ms	850ms / 1.56s / 3.1s	825ms / 1.20s / 3.0s

classifier+hash is the CI ratchet (deterministic, no model files). classifier+onnx is what homesrv runs today. cascade+llm is the wiring #320 proposes: stage-0 grammar → resident model → classifier as failure floor.

Never compare a hash-embedder run to an ONNX one.

Re-measured after the prompt fix

The table above is the baseline at commit 46259b4, kept as-is. The prompt fix (query tested before fact, plus repeat_penalty and a bounded grammar string) was then measured on an otherwise idle box — no other eval sharing llama-server, so these latencies are real rather than contention.

	llm-only (0.8B)	cascade+llm (0.8B)	llm-only, thinking off
intent-only accuracy	48.7% → 61.8%	50.0% → 63.2%	67.1%
full accuracy (intent+slots+gate)	23.7% → 38.2%	32.9% → 47.4%	42.1%
route errors	2 → 0	0 → 0	0
p50 / p95 latency	1.08s / 1.55s	1.04s / 1.53s	0.93s / 1.41s

Three things this run settles:

1. The prompt fix holds. An earlier contended run reported 60.5% / 36.8% for llm-only; the quiet run gives 61.8% / 38.2%. Close enough to call the gain real, and the earlier run's 4-5s latency figures were contention, not the model.
2. query→fact fell from ×15 to ×7, and both unparseable replies are gone. Zero route errors in every LLM configuration.
3. note→fact ×4 is real, not noise. It shows up in the quiet run too. The agent that wrote the prompt fix suspected its own change might have caused it by pulling assertive запиши что… phrasings toward fact, and that suspicion stands — all five ru-note-* cases now land on fact. Tracked as Vikunja #375.

The thinking off column above read as the best configuration measured so far (Vikunja #376). It was wrong — see the controlled re-run below. Ignore that column.

Still 6 / 6 missed clarify — the router has no way to say "I don't know" (Vikunja #359). That is unchanged by anything here.

Thinking off — 31-07-2026, controlled re-run (Vikunja #376)

The "thinking off wins by 6 points" observation above does not hold. It was a measurement artefact, and the earlier table's thinking off column should be ignored.

The thinking-off variant was scored by a hand-rolled HTTP client living in the test file instead of llm.Client. That copy did not send repeat_penalty, which the real router does send (routeRepeatPenalty = 1.15). So the two columns differed on two axes at once, and the one that mattered was the penalty, not the thinking mode.

Re-measured with everything else held equal — same fixture, same prompt, same grammar, same sampling, same idle box, the three configurations run back to back and never concurrently:

	llm-only, thinking on	llm-only, thinking off	cascade+llm
intent-only accuracy	59.2% (45/76)	59.2% (45/76)	61.8% (47/76)
full accuracy (intent+slots+gate)	38.2% (29/76)	38.2% (29/76)	57.9% (44/76)
route errors	3	3	0
grammar violations	3 (all 3 route errors)	3 (same 3 cases)	0
missed clarify	5 / 6	5 / 6	5 / 6
p50 latency	836ms	920ms	810ms
p95 latency	1.41s	2.00s	1.31s

Thinking off is not just a tie on the headline numbers — it is identical case for case, with the same confusion matrix and the same three unparseable replies. The latency difference is run-to-run noise on one box, and it points the wrong way here.

The reason is simpler than any accuracy argument: this llama-server build ignores the request-level thinking switch for this model. Probed directly against the running server with chat_template_kwargs.enable_thinking = false, chat_template_kwargs.thinking = false and top-level reasoning_budget = 0 — all three return a byte-identical answer with the thinking trace still in reasoning_content, and the server reports the prompt prefix as cached, meaning the rendered template did not change. There was never anything being turned off, which is also why the numbers match exactly.

Nothing was defaulted. internal/llm still has no chat_template_kwargs field, VoiceConfig has no thinking flag, and deploy/mavend.json is unchanged. The misleading third configuration is removed from internal/router/eval so the table it produced cannot be quoted again.

Two caveats worth saying out loud:

• The fixture is 76 cases. A 6-point difference on 76 cases is roughly 4-5 cases and would not have been worth trusting even if it had reproduced. This one was exactly 0 cases, which is a much easier call.
• This is one server build and one checkpoint (b9351, Qwen3.5-0.8B Q4_K_M). If the #122 checkpoint or a newer llama.cpp does honour the switch, the question reopens — but it reopens as an unmeasured question, not as a 6-point win.

Phrasing was not measured. Whether thinking helps there is still open, and now also blocked on the same "can we even turn it off" question.

Clock and calendar rule — 31-07-2026 (Vikunja #374)

routeSystem never said whether "который час" or "какое число завтра" are system or query, and system→query ×4 showed up in every run. The rule added says: the clock and the calendar date themselves are system; what is written in the calendar or in memory ("что у меня завтра", "какие есть напоминания") stays query; and a time named inside a request ("напомни завтра…") is just a detail of the request, not a reason for system.

That split is not a preference. In cmd/mavend/voice.go only replySystem owns the clock and the date formatter, so a clock question routed to query falls into the embedder + note RAG and answers "не знаю". The agenda, on the other hand, is answered by ParseCalendarDate + CalendarEvents inside the query branch, so that side has to stay query. The rule sits above the question test because every one of these utterances carries a question word and a later rule would never be reached.

The fixture is now 77 cases: one calendar-agenda case was added (ru-query-019 "что у меня стоит в календаре на послезавтра", intent query) specifically so an over-broad system rule cannot pass unnoticed. The clock/date cases (ru-sys-001/002/005, en-sys-001) already existed.

Three runs, same box, back to back, never concurrently:

	baseline	first rule (too broad)	rule as committed
llm-only intent-only	59.2% (45/76)	54.5% (42/77)	59.7% (46/77)
llm-only full	38.2%	35.1%	39.0%
llm-only route errors	3	4	5
llm-only p50	1.09s	0.91s	0.93s
cascade+llm intent-only	61.8% (47/76)	58.4%	62.3% (48/77)
cascade+llm full	57.9%	54.5%	59.7%
cascade+llm route errors	0	0	0
cascade+llm p50	0.91s	0.80s	1.04s

The targeted bug is fixed and the headline number did not move. system→query ×4 is gone in both LLM configurations — the time and date tags go from 0/2 and 0/2 to 2/2 and 2/2 — but the model then over-applies the rule, and query→system ×5 plus reminder→system ×2 appear where they did not exist before. Net accuracy is a wash, inside the noise of a 77-case fixture.

The first attempt is shown because it is the honest history: it said "спрашивает время, дату или день недели → system" with no scope, which swept up reminders, and it cost 3-5 points. It was tightened once, on the reasoning that a rule capturing "напомни завтра в 7" is simply wrong, and not tuned further. The remaining query/reminder → system over-trigger is a new, separate weakness of the sub-1B model and deserves its own task rather than more prompt kneading against a held-out fixture.

The rule is kept. It is correct about what the daemon can answer, and the failure it replaces was silent ("не знаю" to "который час") while the one it introduces is loud.

Findings

1. The resident model does route better — 50.0% vs 36.8%

REARCH.md's premise holds; voice.go:211's comment does not. But the classifier is only ~37% correct on held-out utterances, and the model only ~50%. Neither is "reliable". The gap between them is real but both are far from a system you would describe as working.

2. It costs 27× the latency

p50 825ms vs 31ms, p95 1.2s, max 3.0s — on the same llama-server the phraser needs, before any phrasing happens. On the CPU/iGPU deploy target this is a trade, not a free win. The review's second-opinion caution was justified.

3. query→fact ×15 is the dominant LLM failure — and it is a prompt bug

Four times the classifier's ×4 on the same axis. routeSystem's decision order in internal/router/llmrouter.go reads:

3. Сообщает или обновляет текущее состояние/событие → fact
4. Хочет получить информацию → query

Any utterance naming a fact key matches rule 3 first, so a question about past state ("сколько воды я выпил с утра", "сколько раз я ел вчера") is classified as an assertion of that state — and a query becomes a confident wrong write. Reordering query above fact, or adding an explicit interrogative test, is the cheapest accuracy win available and needs no model change.

4. Neither path can refuse — the refusal lane is currently fiction

	missed clarify	why
classifier+hash	0 / 6	cosine never clears 0.55 — refuses by accident
classifier+onnx	5 / 6	better embeddings raise cosine everywhere; the gate stops separating
LLM (any)	6 / 6	llmrouter.go hardcodes Confidence: 1.0, so stage 3 can never fire

The deployed config confidently routes сделай это → act at 0.847, ну это → chat at 0.808, бэкап → chat at 0.755, потом → system at 0.739. сделай это → act with unresolved anaphora is the destructive direction; the daemon's confirm gate is the only thing left.

This is the finding that should block #320. Flipping to the LLM router as-is does not improve the refusal lane — it removes it. Tracked as #359.

5. The 50.0% → 32.9% gap is entirely slots

The LLM path fills neither Fn nor Time: it returns Slots.Text for acts (the verb string, not an allowlist match), and Extractor.Extract never runs on an LLM decision at all. Any flip needs the extractor wired onto the LLM branch or every act and reminder arrives without its arguments.

6. The 2 route errors are a missing RepeatPenalty, not a grammar flaw

Both failures (ru-act-006 "закрой жалюзи", ru-chat-003 "расскажи анекдот про программистов") are the sub-1B repetition loop inside the grammar's text field:

"Закрывание жалюзи — это действие, которое нужно выполнить. Если это не действие, то это сообщение пользователя. Если это не действие, то это сообщение пользователя. …"

It runs to MaxTokens: 128, truncates the JSON mid-string, and parseActions fails → fallback to the classifier. llm.Req already has a RepeatPenalty field added for exactly this ("curbs the sub-1B 'тоже тоже тоже' loop") and LLMRouter.Route does not set it. Two lines.

Note the grammar's string ::= "\"" ([^"\\] | "\\" .)* "\"" is unbounded, so nothing stops a 1000-character text. Worth a length bound as well.

7. Two hypotheses tested and closed

• Thinking mode is a non-issue. Confirmed twice now, the second time properly — see the controlled re-run section. Grammar-constrained JSON lands in reasoning_content with content empty and llm.Client's fallback handles it; the request-level switch does nothing on this build. internal/llm deliberately does not grow a chat_template_kwargs field.
• Runaway array repetition does not reproduce. An isolated smoke test with a stripped grammar emitted {"intent":"reminder"} until MaxTokens; under the real routeSystem prompt the few-shot examples anchor it to one object. 2 errors in 76, not 76.

8. Incidental

• ReminderGrammar deliberately skips the extractor at stage 0; the daemon's applyAction parses the time downstream. The scorer counts those as SlotsDeferred rather than misses.
• A local llama-server must bypass http_proxy — this box proxies loopback through a SOCKS bridge that answers 503. noProxyLoopback in the test handles it.
• The onnxruntime .so was already vendored at deps/onnxruntime-linux-x64-1.26.0.

Next steps

Ordered by ratio of value to risk. Nothing here is a decision — #320 stays open.

1. Fix routeSystem's decision order (query above fact, or an explicit interrogative test). Largest single accuracy move, no model change, re-measurable in one command. Expected: most of query→fact ×15.
2. Set RepeatPenalty in LLMRouter.Route and bound the grammar's string length. Removes both route errors.
3. Give the router a refusal signal — #359. Blocks #320.
   • Classifier: the absolute-cosine gate does not survive a better embedder. A margin gate (top1 − top2 > δ) is the likely fix — ambiguous utterances should show flat distributions, which absolute cosine cannot see.
   • LLM: Confidence: 1.0 must go. Either add an unclear intent to the grammar enum, or read logprobs, or gate on the classifier's margin behind the LLM decision.
   • Bar: MissedClarify ≤ 1 without regressing full accuracy below 28/76.
4. Wire Extractor.Extract onto the LLM branch so acts get Fn and reminders get Time. Closes the 50.0% → 32.9% slot gap.
5. Re-measure, then decide #320. At p50 825ms a wholesale swap is probably the wrong shape; the honest candidate is LLM-for-queries with the classifier keeping the fast deterministic paths (stage-0 grammar hits, system, exact acts). That hypothesis is testable against this fixture by scoring a per-intent split.
6. Grow the fixture as failures get understood. 76 cases with ≥5 per intent is enough to rank paths, not enough to trust a 2-point difference. Add cases from real misroutes (CorrectMisroute is already the append-only hook).
7. Second checkpoint when #122 lands. The CPT'd Qwen3-1.7B is the target resident model; the same three configurations should be re-scored against it before it deploys. 0.8B's 50.0% is the floor that checkpoint has to beat, and its latency is the number that decides whether the target is affordable at all.

Open question worth naming

Both paths are under 50%. That is low enough that the interesting question may not be "classifier or model" but whether one-shot classification of a bare utterance is the right frame at all — сделай это, потом, бэкап are unanswerable without dialogue context, and internal/router currently sees none (AnaphoraResolver exists in slots.go but the cascade never calls it). A router that could ask one clarifying question and re-route on the answer would beat both numbers here without a better model.