Maven/ROUTING-EVAL-31-07-2026.md

# Routing evaluation — 31-07-2026

Settles Vikunja **#319** ("measure classifier vs LLM router before flipping"). Everything
below is measured against one held-out fixture, not argued from the code.

- Fixture + scorer: `internal/router/eval/` (`ru_routing_v1.json`, 76 cases; `eval.go`)
- Reproduce: `make eval-router` (classifier baselines) and
  `MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router` (adds the LLM configurations)
- Commits: `c7c4422` (fixture), `d34fdf4` (ONNX baseline), `46259b4` (LLM baseline)

## Why a new fixture

`cmd/mavend/eval_scenarios_test.go` could not answer #319: it asserts daemon-side *safety*
invariants over already-normalized decisions, so it never exercises routing. And the only
utterance corpus that existed — `models/seeds/*.txt` — is the classifier's own training set.
Scoring a nearest-centroid classifier there measures memorisation of frozen centroids, which
is exactly the illusion behind `voice.go:211`'s "the classifier handles routing reliably".

`TestFixtureIsHeldOut` fails the build if any fixture utterance appears verbatim in the seed
corpus. The fixture is a **contract, not a snapshot**: cases the cascade fails today stay in
the file and fail loudly.

## Results

| | classifier+hash | classifier+onnx | llm-only (0.8B) | cascade+llm (0.8B) |
|---|---|---|---|---|
| **intent-only accuracy** | 17.1% | 36.8% | 48.7% | **50.0%** |
| full accuracy (intent+slots+gate) | 17.1% | 36.8% | 23.7% | 32.9% |
| RU | 10/61 | 25/61 | 13/61 | 18/61 |
| EN | 3/15 | 3/15 | 5/15 | 7/15 |
| `hard` tag | 0/11 | 4/11 | — | — |
| false clarify (asked, shouldn't) | 63 | 21 | 0 | 2 |
| **missed clarify (guessed, shouldn't)** | **0 / 6** | **5 / 6** | **6 / 6** | **6 / 6** |
| route errors | 0 | 0 | 2 | 0 |
| **p50 / p95 / max latency** | 9µs / 14µs | **31ms / 71ms** | 850ms / 1.56s / 3.1s | **825ms / 1.20s / 3.0s** |

`classifier+hash` is the CI ratchet (deterministic, no model files). `classifier+onnx` is what
homesrv runs today. `cascade+llm` is the wiring #320 proposes: stage-0 grammar → resident
model → classifier as failure floor.

Never compare a hash-embedder run to an ONNX one.

## Findings

### 1. The resident model does route better — 50.0% vs 36.8%

REARCH.md's premise holds; `voice.go:211`'s comment does not. **But the classifier is only
~37% correct on held-out utterances, and the model only ~50%.** Neither is "reliable". The
gap between them is real but both are far from a system you would describe as working.

### 2. It costs 27× the latency

p50 825ms vs 31ms, p95 1.2s, max 3.0s — on the same llama-server the phraser needs, before
any phrasing happens. On the CPU/iGPU deploy target this is a trade, not a free win. The
review's second-opinion caution was justified.

### 3. `query→fact ×15` is the dominant LLM failure — and it is a prompt bug

Four times the classifier's `×4` on the same axis. `routeSystem`'s decision order in
`internal/router/llmrouter.go` reads:

```
3. Сообщает или обновляет текущее состояние/событие → fact
4. Хочет получить информацию → query
```

Any utterance naming a fact key matches rule 3 first, so a *question about* past state
("сколько воды я выпил с утра", "сколько раз я ел вчера") is classified as an *assertion of*
that state — and a query becomes a confident wrong write. Reordering query above fact, or
adding an explicit interrogative test, is the cheapest accuracy win available and needs no
model change.

### 4. Neither path can refuse — the refusal lane is currently fiction

| | missed clarify | why |
|---|---|---|
| classifier+hash | 0 / 6 | cosine never clears 0.55 — refuses by accident |
| classifier+onnx | 5 / 6 | better embeddings raise cosine everywhere; the gate stops separating |
| LLM (any) | 6 / 6 | `llmrouter.go` hardcodes `Confidence: 1.0`, so stage 3 can never fire |

The deployed config confidently routes `сделай это` → **act** at 0.847, `ну это` → chat at
0.808, `бэкап` → chat at 0.755, `потом` → system at 0.739. `сделай это` → act with unresolved
anaphora is the destructive direction; the daemon's confirm gate is the only thing left.

This is the finding that should block #320. Flipping to the LLM router as-is does not improve
the refusal lane — it removes it. Tracked as **#359**.

### 5. The 50.0% → 32.9% gap is entirely slots

The LLM path fills neither `Fn` nor `Time`: it returns `Slots.Text` for acts (the verb string,
not an allowlist match), and `Extractor.Extract` never runs on an LLM decision at all. Any
flip needs the extractor wired onto the LLM branch or every act and reminder arrives without
its arguments.

### 6. The 2 route errors are a missing `RepeatPenalty`, not a grammar flaw

Both failures (`ru-act-006` "закрой жалюзи", `ru-chat-003` "расскажи анекдот про
программистов") are the sub-1B repetition loop *inside* the grammar's `text` field:

> "Закрывание жалюзи — это действие, которое нужно выполнить. Если это не действие, то это
> сообщение пользователя. Если это не действие, то это сообщение пользователя. …"

It runs to `MaxTokens: 128`, truncates the JSON mid-string, and `parseActions` fails →
fallback to the classifier. `llm.Req` already has a `RepeatPenalty` field added for exactly
this ("curbs the sub-1B 'тоже тоже тоже' loop") and `LLMRouter.Route` does not set it. Two
lines.

Note the grammar's `string ::= "\"" ([^"\\] | "\\" .)* "\""` is unbounded, so nothing stops a
1000-character `text`. Worth a length bound as well.

### 7. Two hypotheses tested and closed

- **Thinking mode is a non-issue.** Qwen3.5's template defaults `thinking = 1`, so
  grammar-constrained JSON lands in `reasoning_content` with `content` empty —
  `llm.Client`'s fallback handles it. A `thinking off` run scored *identically* (18/76,
  48.7%, same p50). `internal/llm` deliberately does **not** grow a `chat_template_kwargs`
  field.
- **Runaway array repetition does not reproduce.** An isolated smoke test with a stripped
  grammar emitted `{"intent":"reminder"}` until `MaxTokens`; under the real `routeSystem`
  prompt the few-shot examples anchor it to one object. 2 errors in 76, not 76.

### 8. Incidental

- `ReminderGrammar` deliberately skips the extractor at stage 0; the daemon's `applyAction`
  parses the time downstream. The scorer counts those as `SlotsDeferred` rather than misses.
- A local llama-server must bypass `http_proxy` — this box proxies loopback through a SOCKS
  bridge that answers 503. `noProxyLoopback` in the test handles it.
- The onnxruntime `.so` was already vendored at `deps/onnxruntime-linux-x64-1.26.0`.

## Next steps

Ordered by ratio of value to risk. Nothing here is a decision — #320 stays open.

1. **Fix `routeSystem`'s decision order** (query above fact, or an explicit interrogative
   test). Largest single accuracy move, no model change, re-measurable in one command.
   Expected: most of `query→fact ×15`.
2. **Set `RepeatPenalty` in `LLMRouter.Route`** and bound the grammar's `string` length.
   Removes both route errors.
3. **Give the router a refusal signal — #359.** Blocks #320.
   - Classifier: the absolute-cosine gate does not survive a better embedder. A **margin**
     gate (`top1 − top2 > δ`) is the likely fix — ambiguous utterances should show flat
     distributions, which absolute cosine cannot see.
   - LLM: `Confidence: 1.0` must go. Either add an `unclear` intent to the grammar enum, or
     read logprobs, or gate on the classifier's margin *behind* the LLM decision.
   - Bar: `MissedClarify ≤ 1` without regressing full accuracy below 28/76.
4. **Wire `Extractor.Extract` onto the LLM branch** so acts get `Fn` and reminders get
   `Time`. Closes the 50.0% → 32.9% slot gap.
5. **Re-measure, then decide #320.** At p50 825ms a wholesale swap is probably the wrong
   shape; the honest candidate is LLM-for-queries with the classifier keeping the fast
   deterministic paths (stage-0 grammar hits, `system`, exact acts). That hypothesis is
   testable against this fixture by scoring a per-intent split.
6. **Grow the fixture** as failures get understood. 76 cases with ≥5 per intent is enough to
   rank paths, not enough to trust a 2-point difference. Add cases from real misroutes
   (`CorrectMisroute` is already the append-only hook).
7. **Second checkpoint when #122 lands.** The CPT'd Qwen3-1.7B is the target resident model;
   the same three configurations should be re-scored against it before it deploys. 0.8B's
   50.0% is the floor that checkpoint has to beat, and its latency is the number that decides
   whether the target is affordable at all.

## Open question worth naming

Both paths are under 50%. That is low enough that the interesting question may not be
"classifier or model" but whether one-shot classification of a bare utterance is the right
frame at all — `сделай это`, `потом`, `бэкап` are unanswerable without dialogue context, and
`internal/router` currently sees none (`AnaphoraResolver` exists in `slots.go` but the
cascade never calls it). A router that could ask one clarifying question and re-route on the
answer would beat both numbers here without a better model.