Merge commit '4ca68d2' into overnight-jul31

Vikunja #278 / #250. Keep Qwen: LFM2.5-1.2B loses 8 points of intent
accuracy, all of it Russian, and runs 2.4x slower.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

MODEL-BAKEOFF-31-07-2026.md

@@ -0,0 +1,101 @@
+# Resident model bake-off — 31-07-2026
+
+**Recommendation: keep Qwen3.5-0.8B.** LFM2.5-1.2B is worse at routing (52.6% vs 60.5%
+intent accuracy), and the loss is almost entirely Russian (18/61 vs 22/61 RU, while EN is a
+wash). It is also 2.4× slower. The Thinking variant is far worse again.
+
+Settles Vikunja **#278 / #250**.
+
+- Same fixture and scorer as `ROUTING-EVAL-31-07-2026.md`: `internal/router/eval/`
+  (`ru_routing_v1.json`, 76 held-out cases).
+- Reproduce: `MAVEN_LLM_URL=http://127.0.0.1:<port> make eval-router`
+  (`TestLLMRouterBaseline`). Note: there is no `make eval-models` target.
+- All three models served by the same `llama-server` flags — `-c 2048 -ngl 99 -t 6`, only
+  `-m` and `--port` differ. One server at a time on an otherwise idle box, so latencies are
+  real and not contention.
+- Measured on top of the router prompt fix (`origin/overnight/router-prompt` merged in), so
+  the Qwen column is directly comparable to the numbers already recorded.
+
+## Results
+
+`llm-only` — the model alone. This is the column that measures the model.
+
+| | Qwen3.5-0.8B | LFM2.5-1.2B Instruct | LFM2.5-1.2B Thinking |
+|---|---|---|---|
+| **intent-only accuracy** | **60.5%** | 52.6% | 36.8% |
+| full accuracy (intent+slots+gate) | **36.8%** | 32.9% | 21.1% |
+| **RU** | **22/61** | 18/61 | 10/61 |
+| EN | 6/15 | **7/15** | 6/15 |
+| route errors | 0 | 0 | 0 |
+| **p50 / p95 latency** | **1.05s / 1.71s** | 2.47s / 3.62s | 2.42s / 3.24s |
+| missed clarify | 6 / 6 | 6 / 6 | 6 / 6 |
+
+`cascade+llm` — stage-0 → model → classifier floor, what #320 would actually ship. Same
+ordering.
+
+| | Qwen3.5-0.8B | LFM2.5-1.2B Instruct | LFM2.5-1.2B Thinking |
+|---|---|---|---|
+| intent-only accuracy | **61.8%** | 55.3% | 38.2% |
+| full accuracy | **46.1%** | 42.1% | 30.3% |
+| RU / EN | **27/61** / 8/15 | 23/61 / **9/15** | 15/61 / 8/15 |
+| route errors | 0 | 0 | 0 |
+| p50 / p95 latency | **1.28s / 1.94s** | 2.18s / 2.72s | 2.27s / 3.19s |
+
+Full logs: the three runs are archived in the session scratchpad
+(`qwen08.txt`, `lfm-instruct.txt`, `lfm-thinking.txt`).
+
+## Russian-specific failures — the owner's worry is confirmed
+
+LFM2.5's Russian loss is not spread out. It has one large, specific failure: **it hears
+almost any Russian imperative or short phrase as `reminder`.**
+
+- `перезапусти докер` → reminder (want act)
+- `включи вытяжку` → reminder (want act)
+- `закрой жалюзи` → reminder (want act)
+- `заметка: продлить домен в августе` → reminder (want note)
+- `запиши что кран на кухне снова капает` → reminder (want note)
+- `доброе утро` → reminder (want chat)
+- `спасибо тебе` → reminder (want note/chat)
+- `переходи в тихий режим` → reminder (want system)
+
+That is `note→reminder ×4`, `act→reminder ×4`, `chat→reminder ×2` in one run. Qwen's
+equivalent failure axis is `query→fact ×8`, which is a narrower and already-understood bug.
+
+Two more Russian-side problems worth naming:
+
+1. **Fact keys come back empty or wrong in Russian.** `воды попил наконец`, `поужинал`,
+   `поспал часов пять` and `отметь что я позавтракал овсянкой` all returned an empty key.
+   `сходил в душ` and `отдохнул минут двадцать` both returned `water`. Qwen does not do this.
+2. **It leaked German.** `slept about seven hours` produced the fact key
+   `"7 Stunden geschlafen"`. Grammar-valid, semantically garbage — a sign the multilingual
+   mix is not anchored where Maven needs it.
+
+The claimed tool-calling advantage did not show up here. `act` is the closest thing this
+fixture has to a tool call, and LFM2.5 got it wrong more often than Qwen, mostly by calling
+it a reminder. It also produced no `fn` slot on any act, same as Qwen.
+
+## The Thinking variant
+
+Not viable. 36.8% intent accuracy, 10/61 Russian, and no latency saving over Instruct — the
+thinking trace costs time without buying accuracy on a short enum classification. With the
+`enable_thinking=false` diagnostic it collapsed further to 28.9% with 2 route errors
+(`query→reminder ×12`). Do not pursue.
+
+## Notes
+
+- Nothing crashed, nothing ignored the GBNF grammar, and no model produced unparseable JSON
+  in the shippable configurations. Zero route errors for both Instruct and Thinking in
+  `llm-only` and `cascade+llm`. The problem with LFM2.5 is what it decides, not whether it
+  can emit the contract.
+- The `6 / 6` missed clarify is unchanged across all three models. No model fixes the missing
+  refusal lane — that is `Confidence: 1.0` hardcoded in `llmrouter.go` (Vikunja #359), not a
+  model property.
+- The report labels every configuration `(0.8B)`; that string is hardcoded in the test, not a
+  reflection of which gguf was loaded. Model identity was confirmed per run via `/v1/models`.
+- No Go code was changed for this measurement, and no bug was found that needed one.
+
+## What this does not settle
+
+Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other
+job — that needs its own fixture. But routing is the load-bearing path and Maven is
+Russian-first, so on the evidence here the switch is not worth making.