diff --git a/ROUTING-EVAL-31-07-2026.md b/ROUTING-EVAL-31-07-2026.md new file mode 100644 index 0000000..7c95565 --- /dev/null +++ b/ROUTING-EVAL-31-07-2026.md @@ -0,0 +1,167 @@ +# Routing evaluation — 31-07-2026 + +Settles Vikunja **#319** ("measure classifier vs LLM router before flipping"). Everything +below is measured against one held-out fixture, not argued from the code. + +- Fixture + scorer: `internal/router/eval/` (`ru_routing_v1.json`, 76 cases; `eval.go`) +- Reproduce: `make eval-router` (classifier baselines) and + `MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router` (adds the LLM configurations) +- Commits: `c7c4422` (fixture), `d34fdf4` (ONNX baseline), `46259b4` (LLM baseline) + +## Why a new fixture + +`cmd/mavend/eval_scenarios_test.go` could not answer #319: it asserts daemon-side *safety* +invariants over already-normalized decisions, so it never exercises routing. And the only +utterance corpus that existed — `models/seeds/*.txt` — is the classifier's own training set. +Scoring a nearest-centroid classifier there measures memorisation of frozen centroids, which +is exactly the illusion behind `voice.go:211`'s "the classifier handles routing reliably". + +`TestFixtureIsHeldOut` fails the build if any fixture utterance appears verbatim in the seed +corpus. The fixture is a **contract, not a snapshot**: cases the cascade fails today stay in +the file and fail loudly. + +## Results + +| | classifier+hash | classifier+onnx | llm-only (0.8B) | cascade+llm (0.8B) | +|---|---|---|---|---| +| **intent-only accuracy** | 17.1% | 36.8% | 48.7% | **50.0%** | +| full accuracy (intent+slots+gate) | 17.1% | 36.8% | 23.7% | 32.9% | +| RU | 10/61 | 25/61 | 13/61 | 18/61 | +| EN | 3/15 | 3/15 | 5/15 | 7/15 | +| `hard` tag | 0/11 | 4/11 | — | — | +| false clarify (asked, shouldn't) | 63 | 21 | 0 | 2 | +| **missed clarify (guessed, shouldn't)** | **0 / 6** | **5 / 6** | **6 / 6** | **6 / 6** | +| route errors | 0 | 0 | 2 | 0 | +| **p50 / p95 / max latency** | 9µs / 14µs | **31ms / 71ms** | 850ms / 1.56s / 3.1s | **825ms / 1.20s / 3.0s** | + +`classifier+hash` is the CI ratchet (deterministic, no model files). `classifier+onnx` is what +homesrv runs today. `cascade+llm` is the wiring #320 proposes: stage-0 grammar → resident +model → classifier as failure floor. + +Never compare a hash-embedder run to an ONNX one. + +## Findings + +### 1. The resident model does route better — 50.0% vs 36.8% + +REARCH.md's premise holds; `voice.go:211`'s comment does not. **But the classifier is only +~37% correct on held-out utterances, and the model only ~50%.** Neither is "reliable". The +gap between them is real but both are far from a system you would describe as working. + +### 2. It costs 27× the latency + +p50 825ms vs 31ms, p95 1.2s, max 3.0s — on the same llama-server the phraser needs, before +any phrasing happens. On the CPU/iGPU deploy target this is a trade, not a free win. The +review's second-opinion caution was justified. + +### 3. `query→fact ×15` is the dominant LLM failure — and it is a prompt bug + +Four times the classifier's `×4` on the same axis. `routeSystem`'s decision order in +`internal/router/llmrouter.go` reads: + +``` +3. Сообщает или обновляет текущее состояние/событие → fact +4. Хочет получить информацию → query +``` + +Any utterance naming a fact key matches rule 3 first, so a *question about* past state +("сколько воды я выпил с утра", "сколько раз я ел вчера") is classified as an *assertion of* +that state — and a query becomes a confident wrong write. Reordering query above fact, or +adding an explicit interrogative test, is the cheapest accuracy win available and needs no +model change. + +### 4. Neither path can refuse — the refusal lane is currently fiction + +| | missed clarify | why | +|---|---|---| +| classifier+hash | 0 / 6 | cosine never clears 0.55 — refuses by accident | +| classifier+onnx | 5 / 6 | better embeddings raise cosine everywhere; the gate stops separating | +| LLM (any) | 6 / 6 | `llmrouter.go` hardcodes `Confidence: 1.0`, so stage 3 can never fire | + +The deployed config confidently routes `сделай это` → **act** at 0.847, `ну это` → chat at +0.808, `бэкап` → chat at 0.755, `потом` → system at 0.739. `сделай это` → act with unresolved +anaphora is the destructive direction; the daemon's confirm gate is the only thing left. + +This is the finding that should block #320. Flipping to the LLM router as-is does not improve +the refusal lane — it removes it. Tracked as **#359**. + +### 5. The 50.0% → 32.9% gap is entirely slots + +The LLM path fills neither `Fn` nor `Time`: it returns `Slots.Text` for acts (the verb string, +not an allowlist match), and `Extractor.Extract` never runs on an LLM decision at all. Any +flip needs the extractor wired onto the LLM branch or every act and reminder arrives without +its arguments. + +### 6. The 2 route errors are a missing `RepeatPenalty`, not a grammar flaw + +Both failures (`ru-act-006` "закрой жалюзи", `ru-chat-003` "расскажи анекдот про +программистов") are the sub-1B repetition loop *inside* the grammar's `text` field: + +> "Закрывание жалюзи — это действие, которое нужно выполнить. Если это не действие, то это +> сообщение пользователя. Если это не действие, то это сообщение пользователя. …" + +It runs to `MaxTokens: 128`, truncates the JSON mid-string, and `parseActions` fails → +fallback to the classifier. `llm.Req` already has a `RepeatPenalty` field added for exactly +this ("curbs the sub-1B 'тоже тоже тоже' loop") and `LLMRouter.Route` does not set it. Two +lines. + +Note the grammar's `string ::= "\"" ([^"\\] | "\\" .)* "\""` is unbounded, so nothing stops a +1000-character `text`. Worth a length bound as well. + +### 7. Two hypotheses tested and closed + +- **Thinking mode is a non-issue.** Qwen3.5's template defaults `thinking = 1`, so + grammar-constrained JSON lands in `reasoning_content` with `content` empty — + `llm.Client`'s fallback handles it. A `thinking off` run scored *identically* (18/76, + 48.7%, same p50). `internal/llm` deliberately does **not** grow a `chat_template_kwargs` + field. +- **Runaway array repetition does not reproduce.** An isolated smoke test with a stripped + grammar emitted `{"intent":"reminder"}` until `MaxTokens`; under the real `routeSystem` + prompt the few-shot examples anchor it to one object. 2 errors in 76, not 76. + +### 8. Incidental + +- `ReminderGrammar` deliberately skips the extractor at stage 0; the daemon's `applyAction` + parses the time downstream. The scorer counts those as `SlotsDeferred` rather than misses. +- A local llama-server must bypass `http_proxy` — this box proxies loopback through a SOCKS + bridge that answers 503. `noProxyLoopback` in the test handles it. +- The onnxruntime `.so` was already vendored at `deps/onnxruntime-linux-x64-1.26.0`. + +## Next steps + +Ordered by ratio of value to risk. Nothing here is a decision — #320 stays open. + +1. **Fix `routeSystem`'s decision order** (query above fact, or an explicit interrogative + test). Largest single accuracy move, no model change, re-measurable in one command. + Expected: most of `query→fact ×15`. +2. **Set `RepeatPenalty` in `LLMRouter.Route`** and bound the grammar's `string` length. + Removes both route errors. +3. **Give the router a refusal signal — #359.** Blocks #320. + - Classifier: the absolute-cosine gate does not survive a better embedder. A **margin** + gate (`top1 − top2 > δ`) is the likely fix — ambiguous utterances should show flat + distributions, which absolute cosine cannot see. + - LLM: `Confidence: 1.0` must go. Either add an `unclear` intent to the grammar enum, or + read logprobs, or gate on the classifier's margin *behind* the LLM decision. + - Bar: `MissedClarify ≤ 1` without regressing full accuracy below 28/76. +4. **Wire `Extractor.Extract` onto the LLM branch** so acts get `Fn` and reminders get + `Time`. Closes the 50.0% → 32.9% slot gap. +5. **Re-measure, then decide #320.** At p50 825ms a wholesale swap is probably the wrong + shape; the honest candidate is LLM-for-queries with the classifier keeping the fast + deterministic paths (stage-0 grammar hits, `system`, exact acts). That hypothesis is + testable against this fixture by scoring a per-intent split. +6. **Grow the fixture** as failures get understood. 76 cases with ≥5 per intent is enough to + rank paths, not enough to trust a 2-point difference. Add cases from real misroutes + (`CorrectMisroute` is already the append-only hook). +7. **Second checkpoint when #122 lands.** The CPT'd Qwen3-1.7B is the target resident model; + the same three configurations should be re-scored against it before it deploys. 0.8B's + 50.0% is the floor that checkpoint has to beat, and its latency is the number that decides + whether the target is affordable at all. + +## Open question worth naming + +Both paths are under 50%. That is low enough that the interesting question may not be +"classifier or model" but whether one-shot classification of a bare utterance is the right +frame at all — `сделай это`, `потом`, `бэкап` are unanswerable without dialogue context, and +`internal/router` currently sees none (`AnaphoraResolver` exists in `slots.go` but the +cascade never calls it). A router that could ask one clarifying question and re-route on the +answer would beat both numbers here without a better model.