Write up the routing evaluation findings and next steps

Settles the measurement half of Vikunja #319: the resident 0.8B routes
better than the deployed classifier (50.0% vs 36.8% intent-only) at ~27x
the latency, and neither path can refuse an ambiguous utterance.

Records the four-configuration comparison, the per-case evidence behind
each finding (query->fact x15 traced to routeSystem's rule order, the
five missed-clarify cosines, the two text-field repetition truncations),
the two hypotheses that were tested and closed (thinking mode, grammar
array runaway), and a next-steps list mapped to #319/#320/#359.

#320 should not flip as-is: it would remove the refusal lane rather than
improve it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik

ROUTING-EVAL-31-07-2026.md

@@ -0,0 +1,167 @@
+# Routing evaluation — 31-07-2026
+
+Settles Vikunja **#319** ("measure classifier vs LLM router before flipping"). Everything
+below is measured against one held-out fixture, not argued from the code.
+
+- Fixture + scorer: `internal/router/eval/` (`ru_routing_v1.json`, 76 cases; `eval.go`)
+- Reproduce: `make eval-router` (classifier baselines) and
+  `MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router` (adds the LLM configurations)
+- Commits: `c7c4422` (fixture), `d34fdf4` (ONNX baseline), `46259b4` (LLM baseline)
+
+## Why a new fixture
+
+`cmd/mavend/eval_scenarios_test.go` could not answer #319: it asserts daemon-side *safety*
+invariants over already-normalized decisions, so it never exercises routing. And the only
+utterance corpus that existed — `models/seeds/*.txt` — is the classifier's own training set.
+Scoring a nearest-centroid classifier there measures memorisation of frozen centroids, which
+is exactly the illusion behind `voice.go:211`'s "the classifier handles routing reliably".
+
+`TestFixtureIsHeldOut` fails the build if any fixture utterance appears verbatim in the seed
+corpus. The fixture is a **contract, not a snapshot**: cases the cascade fails today stay in
+the file and fail loudly.
+
+## Results
+
+| | classifier+hash | classifier+onnx | llm-only (0.8B) | cascade+llm (0.8B) |
+|---|---|---|---|---|
+| **intent-only accuracy** | 17.1% | 36.8% | 48.7% | **50.0%** |
+| full accuracy (intent+slots+gate) | 17.1% | 36.8% | 23.7% | 32.9% |
+| RU | 10/61 | 25/61 | 13/61 | 18/61 |
+| EN | 3/15 | 3/15 | 5/15 | 7/15 |
+| `hard` tag | 0/11 | 4/11 | — | — |
+| false clarify (asked, shouldn't) | 63 | 21 | 0 | 2 |
+| **missed clarify (guessed, shouldn't)** | **0 / 6** | **5 / 6** | **6 / 6** | **6 / 6** |
+| route errors | 0 | 0 | 2 | 0 |
+| **p50 / p95 / max latency** | 9µs / 14µs | **31ms / 71ms** | 850ms / 1.56s / 3.1s | **825ms / 1.20s / 3.0s** |
+
+`classifier+hash` is the CI ratchet (deterministic, no model files). `classifier+onnx` is what
+homesrv runs today. `cascade+llm` is the wiring #320 proposes: stage-0 grammar → resident
+model → classifier as failure floor.
+
+Never compare a hash-embedder run to an ONNX one.
+
+## Findings
+
+### 1. The resident model does route better — 50.0% vs 36.8%
+
+REARCH.md's premise holds; `voice.go:211`'s comment does not. **But the classifier is only
+~37% correct on held-out utterances, and the model only ~50%.** Neither is "reliable". The
+gap between them is real but both are far from a system you would describe as working.
+
+### 2. It costs 27× the latency
+
+p50 825ms vs 31ms, p95 1.2s, max 3.0s — on the same llama-server the phraser needs, before
+any phrasing happens. On the CPU/iGPU deploy target this is a trade, not a free win. The
+review's second-opinion caution was justified.
+
+### 3. `query→fact ×15` is the dominant LLM failure — and it is a prompt bug
+
+Four times the classifier's `×4` on the same axis. `routeSystem`'s decision order in
+`internal/router/llmrouter.go` reads:
+
+```
+3. Сообщает или обновляет текущее состояние/событие → fact
+4. Хочет получить информацию → query
+```
+
+Any utterance naming a fact key matches rule 3 first, so a *question about* past state
+("сколько воды я выпил с утра", "сколько раз я ел вчера") is classified as an *assertion of*
+that state — and a query becomes a confident wrong write. Reordering query above fact, or
+adding an explicit interrogative test, is the cheapest accuracy win available and needs no
+model change.
+
+### 4. Neither path can refuse — the refusal lane is currently fiction
+
+| | missed clarify | why |
+|---|---|---|
+| classifier+hash | 0 / 6 | cosine never clears 0.55 — refuses by accident |
+| classifier+onnx | 5 / 6 | better embeddings raise cosine everywhere; the gate stops separating |
+| LLM (any) | 6 / 6 | `llmrouter.go` hardcodes `Confidence: 1.0`, so stage 3 can never fire |
+
+The deployed config confidently routes `сделай это` → **act** at 0.847, `ну это` → chat at
+0.808, `бэкап` → chat at 0.755, `потом` → system at 0.739. `сделай это` → act with unresolved
+anaphora is the destructive direction; the daemon's confirm gate is the only thing left.
+
+This is the finding that should block #320. Flipping to the LLM router as-is does not improve
+the refusal lane — it removes it. Tracked as **#359**.
+
+### 5. The 50.0% → 32.9% gap is entirely slots
+
+The LLM path fills neither `Fn` nor `Time`: it returns `Slots.Text` for acts (the verb string,
+not an allowlist match), and `Extractor.Extract` never runs on an LLM decision at all. Any
+flip needs the extractor wired onto the LLM branch or every act and reminder arrives without
+its arguments.
+
+### 6. The 2 route errors are a missing `RepeatPenalty`, not a grammar flaw
+
+Both failures (`ru-act-006` "закрой жалюзи", `ru-chat-003` "расскажи анекдот про
+программистов") are the sub-1B repetition loop *inside* the grammar's `text` field:
+
+> "Закрывание жалюзи — это действие, которое нужно выполнить. Если это не действие, то это
+> сообщение пользователя. Если это не действие, то это сообщение пользователя. …"
+
+It runs to `MaxTokens: 128`, truncates the JSON mid-string, and `parseActions` fails →
+fallback to the classifier. `llm.Req` already has a `RepeatPenalty` field added for exactly
+this ("curbs the sub-1B 'тоже тоже тоже' loop") and `LLMRouter.Route` does not set it. Two
+lines.
+
+Note the grammar's `string ::= "\"" ([^"\\] | "\\" .)* "\""` is unbounded, so nothing stops a
+1000-character `text`. Worth a length bound as well.
+
+### 7. Two hypotheses tested and closed
+
+- **Thinking mode is a non-issue.** Qwen3.5's template defaults `thinking = 1`, so
+  grammar-constrained JSON lands in `reasoning_content` with `content` empty —
+  `llm.Client`'s fallback handles it. A `thinking off` run scored *identically* (18/76,
+  48.7%, same p50). `internal/llm` deliberately does **not** grow a `chat_template_kwargs`
+  field.
+- **Runaway array repetition does not reproduce.** An isolated smoke test with a stripped
+  grammar emitted `{"intent":"reminder"}` until `MaxTokens`; under the real `routeSystem`
+  prompt the few-shot examples anchor it to one object. 2 errors in 76, not 76.
+
+### 8. Incidental
+
+- `ReminderGrammar` deliberately skips the extractor at stage 0; the daemon's `applyAction`
+  parses the time downstream. The scorer counts those as `SlotsDeferred` rather than misses.
+- A local llama-server must bypass `http_proxy` — this box proxies loopback through a SOCKS
+  bridge that answers 503. `noProxyLoopback` in the test handles it.
+- The onnxruntime `.so` was already vendored at `deps/onnxruntime-linux-x64-1.26.0`.
+
+## Next steps
+
+Ordered by ratio of value to risk. Nothing here is a decision — #320 stays open.
+
+1. **Fix `routeSystem`'s decision order** (query above fact, or an explicit interrogative
+   test). Largest single accuracy move, no model change, re-measurable in one command.
+   Expected: most of `query→fact ×15`.
+2. **Set `RepeatPenalty` in `LLMRouter.Route`** and bound the grammar's `string` length.
+   Removes both route errors.
+3. **Give the router a refusal signal — #359.** Blocks #320.
+   - Classifier: the absolute-cosine gate does not survive a better embedder. A **margin**
+     gate (`top1 − top2 > δ`) is the likely fix — ambiguous utterances should show flat
+     distributions, which absolute cosine cannot see.
+   - LLM: `Confidence: 1.0` must go. Either add an `unclear` intent to the grammar enum, or
+     read logprobs, or gate on the classifier's margin *behind* the LLM decision.
+   - Bar: `MissedClarify ≤ 1` without regressing full accuracy below 28/76.
+4. **Wire `Extractor.Extract` onto the LLM branch** so acts get `Fn` and reminders get
+   `Time`. Closes the 50.0% → 32.9% slot gap.
+5. **Re-measure, then decide #320.** At p50 825ms a wholesale swap is probably the wrong
+   shape; the honest candidate is LLM-for-queries with the classifier keeping the fast
+   deterministic paths (stage-0 grammar hits, `system`, exact acts). That hypothesis is
+   testable against this fixture by scoring a per-intent split.
+6. **Grow the fixture** as failures get understood. 76 cases with ≥5 per intent is enough to
+   rank paths, not enough to trust a 2-point difference. Add cases from real misroutes
+   (`CorrectMisroute` is already the append-only hook).
+7. **Second checkpoint when #122 lands.** The CPT'd Qwen3-1.7B is the target resident model;
+   the same three configurations should be re-scored against it before it deploys. 0.8B's
+   50.0% is the floor that checkpoint has to beat, and its latency is the number that decides
+   whether the target is affordable at all.
+
+## Open question worth naming
+
+Both paths are under 50%. That is low enough that the interesting question may not be
+"classifier or model" but whether one-shot classification of a bare utterance is the right
+frame at all — `сделай это`, `потом`, `бэкап` are unanswerable without dialogue context, and
+`internal/router` currently sees none (`AnaphoraResolver` exists in `slots.go` but the
+cascade never calls it). A router that could ask one clarifying question and re-route on the
+answer would beat both numbers here without a better model.