Maven/docs/evals/2026-07-31-routing.md

# Routing evaluation — 31-07-2026

Settles Vikunja **#319** ("measure classifier vs LLM router before flipping"). Everything
below is measured against one held-out fixture, not argued from the code.

- Fixture + scorer: `internal/router/eval/` (`ru_routing_v1.json`, 76 cases; `eval.go`)
- Reproduce: `make eval-router` (classifier baselines) and
  `MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router` (adds the LLM configurations)
- Commits: `c7c4422` (fixture), `d34fdf4` (ONNX baseline), `46259b4` (LLM baseline)

## Why a new fixture

`cmd/mavend/eval_scenarios_test.go` could not answer #319: it asserts daemon-side *safety*
invariants over already-normalized decisions, so it never exercises routing. And the only
utterance corpus that existed — `models/seeds/*.txt` — is the classifier's own training set.
Scoring a nearest-centroid classifier there measures memorisation of frozen centroids, which
is exactly the illusion behind `voice.go:211`'s "the classifier handles routing reliably".

`TestFixtureIsHeldOut` fails the build if any fixture utterance appears verbatim in the seed
corpus. The fixture is a **contract, not a snapshot**: cases the cascade fails today stay in
the file and fail loudly.

## Results

| | classifier+hash | classifier+onnx | llm-only (0.8B) | cascade+llm (0.8B) |
|---|---|---|---|---|
| **intent-only accuracy** | 17.1% | 36.8% | 48.7% | **50.0%** |
| full accuracy (intent+slots+gate) | 17.1% | 36.8% | 23.7% | 32.9% |
| RU | 10/61 | 25/61 | 13/61 | 18/61 |
| EN | 3/15 | 3/15 | 5/15 | 7/15 |
| `hard` tag | 0/11 | 4/11 | — | — |
| false clarify (asked, shouldn't) | 63 | 21 | 0 | 2 |
| **missed clarify (guessed, shouldn't)** | **0 / 6** | **5 / 6** | **6 / 6** | **6 / 6** |
| route errors | 0 | 0 | 2 | 0 |
| **p50 / p95 / max latency** | 9µs / 14µs | **31ms / 71ms** | 850ms / 1.56s / 3.1s | **825ms / 1.20s / 3.0s** |

`classifier+hash` is the CI ratchet (deterministic, no model files). `classifier+onnx` is what
homesrv runs today. `cascade+llm` is the wiring #320 proposes: stage-0 grammar → resident
model → classifier as failure floor.

Never compare a hash-embedder run to an ONNX one.

## Re-measured after the prompt fix

The table above is the **baseline at commit `46259b4`**, kept as-is. The prompt fix (query
tested before fact, plus `repeat_penalty` and a bounded grammar string) was then measured on
an otherwise idle box — no other eval sharing llama-server, so these latencies are real
rather than contention.

| | llm-only (0.8B) | cascade+llm (0.8B) | llm-only, thinking off |
|---|---|---|---|
| **intent-only accuracy** | 48.7% → **61.8%** | 50.0% → **63.2%** | **67.1%** |
| full accuracy (intent+slots+gate) | 23.7% → **38.2%** | 32.9% → **47.4%** | **42.1%** |
| route errors | 2 → **0** | 0 → 0 | **0** |
| p50 / p95 latency | **1.08s / 1.55s** | **1.04s / 1.53s** | **0.93s / 1.41s** |

Three things this run settles:

1. **The prompt fix holds.** An earlier contended run reported 60.5% / 36.8% for llm-only;
   the quiet run gives 61.8% / 38.2%. Close enough to call the gain real, and the earlier
   run's 4-5s latency figures were contention, not the model.
2. **`query→fact` fell from ×15 to ×7**, and both unparseable replies are gone. Zero route
   errors in every LLM configuration.
3. **`note→fact ×4` is real, not noise.** It shows up in the quiet run too. The agent that
   wrote the prompt fix suspected its own change might have caused it by pulling assertive
   `запиши что…` phrasings toward fact, and that suspicion stands — all five `ru-note-*`
   cases now land on fact. Tracked as Vikunja #375.

The `thinking off` column above read as the best configuration measured so far (Vikunja #376).
**It was wrong** — see the controlled re-run below. Ignore that column.

Still `6 / 6` missed clarify — the router has no way to say "I don't know" (Vikunja #359).
That is unchanged by anything here.

## Thinking off — 31-07-2026, controlled re-run (Vikunja #376)

The "thinking off wins by 6 points" observation above **does not hold**. It was a measurement
artefact, and the earlier table's `thinking off` column should be ignored.

The thinking-off variant was scored by a hand-rolled HTTP client living in the test file
instead of `llm.Client`. That copy did not send `repeat_penalty`, which the real router does
send (`routeRepeatPenalty = 1.15`). So the two columns differed on two axes at once, and the
one that mattered was the penalty, not the thinking mode.

Re-measured with everything else held equal — same fixture, same prompt, same grammar, same
sampling, same idle box, the three configurations run back to back and never concurrently:

| | llm-only, thinking on | llm-only, thinking off | cascade+llm |
|---|---|---|---|
| intent-only accuracy | 59.2% (45/76) | 59.2% (45/76) | 61.8% (47/76) |
| full accuracy (intent+slots+gate) | 38.2% (29/76) | 38.2% (29/76) | 57.9% (44/76) |
| route errors | 3 | 3 | 0 |
| grammar violations | 3 (all 3 route errors) | 3 (same 3 cases) | 0 |
| missed clarify | 5 / 6 | 5 / 6 | 5 / 6 |
| p50 latency | 836ms | 920ms | 810ms |
| p95 latency | 1.41s | 2.00s | 1.31s |

Thinking off is not just a tie on the headline numbers — it is identical case for case, with
the same confusion matrix and the same three unparseable replies. The latency difference is
run-to-run noise on one box, and it points the wrong way here.

The reason is simpler than any accuracy argument: **this llama-server build ignores the
request-level thinking switch for this model.** Probed directly against the running server
with `chat_template_kwargs.enable_thinking = false`, `chat_template_kwargs.thinking = false`
and top-level `reasoning_budget = 0` — all three return a byte-identical answer with the
thinking trace still in `reasoning_content`, and the server reports the prompt prefix as
cached, meaning the rendered template did not change. There was never anything being turned
off, which is also why the numbers match exactly.

Nothing was defaulted. `internal/llm` still has no `chat_template_kwargs` field, `VoiceConfig`
has no thinking flag, and `deploy/mavend.json` is unchanged. The misleading third
configuration is removed from `internal/router/eval` so the table it produced cannot be quoted
again.

Two caveats worth saying out loud:

- **The fixture is 76 cases.** A 6-point difference on 76 cases is roughly 4-5 cases and would
  not have been worth trusting even if it had reproduced. This one was exactly 0 cases, which
  is a much easier call.
- **This is one server build and one checkpoint** (`b9351`, Qwen3.5-0.8B Q4_K_M). If the
  #122 checkpoint or a newer llama.cpp does honour the switch, the question reopens — but it
  reopens as an unmeasured question, not as a 6-point win.

Phrasing was **not** measured. Whether thinking helps there is still open, and now also blocked
on the same "can we even turn it off" question.

## Clock and calendar rule — 31-07-2026 (Vikunja #374)

`routeSystem` never said whether "который час" or "какое число завтра" are `system` or
`query`, and `system→query ×4` showed up in every run. The rule added says: the clock and the
calendar date themselves are `system`; what is *written in* the calendar or in memory
("что у меня завтра", "какие есть напоминания") stays `query`; and a time named inside a
request ("напомни завтра…") is just a detail of the request, not a reason for `system`.

That split is not a preference. In `cmd/mavend/voice.go` only `replySystem` owns the clock and
the date formatter, so a clock question routed to `query` falls into the embedder + note RAG
and answers "не знаю". The agenda, on the other hand, is answered by `ParseCalendarDate` +
`CalendarEvents` *inside* the `query` branch, so that side has to stay `query`. The rule sits
above the question test because every one of these utterances carries a question word and a
later rule would never be reached.

The fixture is now 77 cases: one calendar-agenda case was added
(`ru-query-019` "что у меня стоит в календаре на послезавтра", intent `query`) specifically so
an over-broad system rule cannot pass unnoticed. The clock/date cases (`ru-sys-001/002/005`,
`en-sys-001`) already existed.

Three runs, same box, back to back, never concurrently:

| | baseline | first rule (too broad) | rule as committed |
|---|---|---|---|
| llm-only intent-only | 59.2% (45/76) | 54.5% (42/77) | 59.7% (46/77) |
| llm-only full | 38.2% | 35.1% | 39.0% |
| llm-only route errors | 3 | 4 | 5 |
| llm-only p50 | 1.09s | 0.91s | 0.93s |
| cascade+llm intent-only | 61.8% (47/76) | 58.4% | 62.3% (48/77) |
| cascade+llm full | 57.9% | 54.5% | 59.7% |
| cascade+llm route errors | 0 | 0 | 0 |
| cascade+llm p50 | 0.91s | 0.80s | 1.04s |

**The targeted bug is fixed and the headline number did not move.** `system→query ×4` is gone
in both LLM configurations — the `time` and `date` tags go from 0/2 and 0/2 to 2/2 and 2/2 —
but the model then over-applies the rule, and `query→system ×5` plus `reminder→system ×2`
appear where they did not exist before. Net accuracy is a wash, inside the noise of a 77-case
fixture.

The first attempt is shown because it is the honest history: it said "спрашивает время, дату
или день недели → system" with no scope, which swept up reminders, and it cost 3-5 points. It
was tightened once, on the reasoning that a rule capturing "напомни завтра в 7" is simply
wrong, and not tuned further. The remaining `query/reminder → system` over-trigger is a new,
separate weakness of the sub-1B model and deserves its own task rather than more prompt
kneading against a held-out fixture.

The rule is kept. It is correct about what the daemon can answer, and the failure it replaces
was silent ("не знаю" to "который час") while the one it introduces is loud.

## Findings

### 1. The resident model does route better — 50.0% vs 36.8%

docs/rearchitecture.md's premise holds; `voice.go:211`'s comment does not. **But the classifier is only
~37% correct on held-out utterances, and the model only ~50%.** Neither is "reliable". The
gap between them is real but both are far from a system you would describe as working.

### 2. It costs 27× the latency

p50 825ms vs 31ms, p95 1.2s, max 3.0s — on the same llama-server the phraser needs, before
any phrasing happens. On the CPU/iGPU deploy target this is a trade, not a free win. The
review's second-opinion caution was justified.

### 3. `query→fact ×15` is the dominant LLM failure — and it is a prompt bug

Four times the classifier's `×4` on the same axis. `routeSystem`'s decision order in
`internal/router/llmrouter.go` reads:

```
3. Сообщает или обновляет текущее состояние/событие → fact
4. Хочет получить информацию → query
```

Any utterance naming a fact key matches rule 3 first, so a *question about* past state
("сколько воды я выпил с утра", "сколько раз я ел вчера") is classified as an *assertion of*
that state — and a query becomes a confident wrong write. Reordering query above fact, or
adding an explicit interrogative test, is the cheapest accuracy win available and needs no
model change.

### 4. Neither path can refuse — the refusal lane is currently fiction

| | missed clarify | why |
|---|---|---|
| classifier+hash | 0 / 6 | cosine never clears 0.55 — refuses by accident |
| classifier+onnx | 5 / 6 | better embeddings raise cosine everywhere; the gate stops separating |
| LLM (any) | 6 / 6 | `llmrouter.go` hardcodes `Confidence: 1.0`, so stage 3 can never fire |

The deployed config confidently routes `сделай это` → **act** at 0.847, `ну это` → chat at
0.808, `бэкап` → chat at 0.755, `потом` → system at 0.739. `сделай это` → act with unresolved
anaphora is the destructive direction; the daemon's confirm gate is the only thing left.

This is the finding that should block #320. Flipping to the LLM router as-is does not improve
the refusal lane — it removes it. Tracked as **#359**.

### 5. The 50.0% → 32.9% gap is entirely slots

The LLM path fills neither `Fn` nor `Time`: it returns `Slots.Text` for acts (the verb string,
not an allowlist match), and `Extractor.Extract` never runs on an LLM decision at all. Any
flip needs the extractor wired onto the LLM branch or every act and reminder arrives without
its arguments.

### 6. The 2 route errors are a missing `RepeatPenalty`, not a grammar flaw

Both failures (`ru-act-006` "закрой жалюзи", `ru-chat-003` "расскажи анекдот про
программистов") are the sub-1B repetition loop *inside* the grammar's `text` field:

> "Закрывание жалюзи — это действие, которое нужно выполнить. Если это не действие, то это
> сообщение пользователя. Если это не действие, то это сообщение пользователя. …"

It runs to `MaxTokens: 128`, truncates the JSON mid-string, and `parseActions` fails →
fallback to the classifier. `llm.Req` already has a `RepeatPenalty` field added for exactly
this ("curbs the sub-1B 'тоже тоже тоже' loop") and `LLMRouter.Route` does not set it. Two
lines.

Note the grammar's `string ::= "\"" ([^"\\] | "\\" .)* "\""` is unbounded, so nothing stops a
1000-character `text`. Worth a length bound as well.

### 7. Two hypotheses tested and closed

- **Thinking mode is a non-issue.** Confirmed twice now, the second time properly — see the
  controlled re-run section. Grammar-constrained JSON lands in `reasoning_content` with
  `content` empty and `llm.Client`'s fallback handles it; the request-level switch does
  nothing on this build. `internal/llm` deliberately does **not** grow a
  `chat_template_kwargs` field.
- **Runaway array repetition does not reproduce.** An isolated smoke test with a stripped
  grammar emitted `{"intent":"reminder"}` until `MaxTokens`; under the real `routeSystem`
  prompt the few-shot examples anchor it to one object. 2 errors in 76, not 76.

### 8. Incidental

- `ReminderGrammar` deliberately skips the extractor at stage 0; the daemon's `applyAction`
  parses the time downstream. The scorer counts those as `SlotsDeferred` rather than misses.
- A local llama-server must bypass `http_proxy` — this box proxies loopback through a SOCKS
  bridge that answers 503. `noProxyLoopback` in the test handles it.
- The onnxruntime `.so` was already vendored at `deps/onnxruntime-linux-x64-1.26.0`.

## Next steps

Ordered by ratio of value to risk. Nothing here is a decision — #320 stays open.

1. **Fix `routeSystem`'s decision order** (query above fact, or an explicit interrogative
   test). Largest single accuracy move, no model change, re-measurable in one command.
   Expected: most of `query→fact ×15`.
2. **Set `RepeatPenalty` in `LLMRouter.Route`** and bound the grammar's `string` length.
   Removes both route errors.
3. **Give the router a refusal signal — #359.** Blocks #320.
   - Classifier: the absolute-cosine gate does not survive a better embedder. A **margin**
     gate (`top1 − top2 > δ`) is the likely fix — ambiguous utterances should show flat
     distributions, which absolute cosine cannot see.
   - LLM: `Confidence: 1.0` must go. Either add an `unclear` intent to the grammar enum, or
     read logprobs, or gate on the classifier's margin *behind* the LLM decision.
   - Bar: `MissedClarify ≤ 1` without regressing full accuracy below 28/76.
4. **Wire `Extractor.Extract` onto the LLM branch** so acts get `Fn` and reminders get
   `Time`. Closes the 50.0% → 32.9% slot gap.
5. **Re-measure, then decide #320.** At p50 825ms a wholesale swap is probably the wrong
   shape; the honest candidate is LLM-for-queries with the classifier keeping the fast
   deterministic paths (stage-0 grammar hits, `system`, exact acts). That hypothesis is
   testable against this fixture by scoring a per-intent split.
6. **Grow the fixture** as failures get understood. 76 cases with ≥5 per intent is enough to
   rank paths, not enough to trust a 2-point difference. Add cases from real misroutes
   (`CorrectMisroute` is already the append-only hook).
7. **Second checkpoint when #122 lands.** The CPT'd Qwen3-1.7B is the target resident model;
   the same three configurations should be re-scored against it before it deploys. 0.8B's
   50.0% is the floor that checkpoint has to beat, and its latency is the number that decides
   whether the target is affordable at all.

## Open question worth naming

Both paths are under 50%. That is low enough that the interesting question may not be
"classifier or model" but whether one-shot classification of a bare utterance is the right
frame at all — `сделай это`, `потом`, `бэкап` are unanswerable without dialogue context, and
`internal/router` currently sees none (`AnaphoraResolver` exists in `slots.go` but the
cascade never calls it). A router that could ask one clarifying question and re-route on the
answer would beat both numbers here without a better model.