kami
4ca68d2f3f
Vikunja #278 / #250. Keep Qwen: LFM2.5-1.2B loses 8 points of intent accuracy, all of it Russian, and runs 2.4x slower. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
102 lines
5.2 KiB
Markdown
102 lines
5.2 KiB
Markdown
# Resident model bake-off — 31-07-2026
|
||
|
||
**Recommendation: keep Qwen3.5-0.8B.** LFM2.5-1.2B is worse at routing (52.6% vs 60.5%
|
||
intent accuracy), and the loss is almost entirely Russian (18/61 vs 22/61 RU, while EN is a
|
||
wash). It is also 2.4× slower. The Thinking variant is far worse again.
|
||
|
||
Settles Vikunja **#278 / #250**.
|
||
|
||
- Same fixture and scorer as `ROUTING-EVAL-31-07-2026.md`: `internal/router/eval/`
|
||
(`ru_routing_v1.json`, 76 held-out cases).
|
||
- Reproduce: `MAVEN_LLM_URL=http://127.0.0.1:<port> make eval-router`
|
||
(`TestLLMRouterBaseline`). Note: there is no `make eval-models` target.
|
||
- All three models served by the same `llama-server` flags — `-c 2048 -ngl 99 -t 6`, only
|
||
`-m` and `--port` differ. One server at a time on an otherwise idle box, so latencies are
|
||
real and not contention.
|
||
- Measured on top of the router prompt fix (`origin/overnight/router-prompt` merged in), so
|
||
the Qwen column is directly comparable to the numbers already recorded.
|
||
|
||
## Results
|
||
|
||
`llm-only` — the model alone. This is the column that measures the model.
|
||
|
||
| | Qwen3.5-0.8B | LFM2.5-1.2B Instruct | LFM2.5-1.2B Thinking |
|
||
|---|---|---|---|
|
||
| **intent-only accuracy** | **60.5%** | 52.6% | 36.8% |
|
||
| full accuracy (intent+slots+gate) | **36.8%** | 32.9% | 21.1% |
|
||
| **RU** | **22/61** | 18/61 | 10/61 |
|
||
| EN | 6/15 | **7/15** | 6/15 |
|
||
| route errors | 0 | 0 | 0 |
|
||
| **p50 / p95 latency** | **1.05s / 1.71s** | 2.47s / 3.62s | 2.42s / 3.24s |
|
||
| missed clarify | 6 / 6 | 6 / 6 | 6 / 6 |
|
||
|
||
`cascade+llm` — stage-0 → model → classifier floor, what #320 would actually ship. Same
|
||
ordering.
|
||
|
||
| | Qwen3.5-0.8B | LFM2.5-1.2B Instruct | LFM2.5-1.2B Thinking |
|
||
|---|---|---|---|
|
||
| intent-only accuracy | **61.8%** | 55.3% | 38.2% |
|
||
| full accuracy | **46.1%** | 42.1% | 30.3% |
|
||
| RU / EN | **27/61** / 8/15 | 23/61 / **9/15** | 15/61 / 8/15 |
|
||
| route errors | 0 | 0 | 0 |
|
||
| p50 / p95 latency | **1.28s / 1.94s** | 2.18s / 2.72s | 2.27s / 3.19s |
|
||
|
||
Full logs: the three runs are archived in the session scratchpad
|
||
(`qwen08.txt`, `lfm-instruct.txt`, `lfm-thinking.txt`).
|
||
|
||
## Russian-specific failures — the owner's worry is confirmed
|
||
|
||
LFM2.5's Russian loss is not spread out. It has one large, specific failure: **it hears
|
||
almost any Russian imperative or short phrase as `reminder`.**
|
||
|
||
- `перезапусти докер` → reminder (want act)
|
||
- `включи вытяжку` → reminder (want act)
|
||
- `закрой жалюзи` → reminder (want act)
|
||
- `заметка: продлить домен в августе` → reminder (want note)
|
||
- `запиши что кран на кухне снова капает` → reminder (want note)
|
||
- `доброе утро` → reminder (want chat)
|
||
- `спасибо тебе` → reminder (want note/chat)
|
||
- `переходи в тихий режим` → reminder (want system)
|
||
|
||
That is `note→reminder ×4`, `act→reminder ×4`, `chat→reminder ×2` in one run. Qwen's
|
||
equivalent failure axis is `query→fact ×8`, which is a narrower and already-understood bug.
|
||
|
||
Two more Russian-side problems worth naming:
|
||
|
||
1. **Fact keys come back empty or wrong in Russian.** `воды попил наконец`, `поужинал`,
|
||
`поспал часов пять` and `отметь что я позавтракал овсянкой` all returned an empty key.
|
||
`сходил в душ` and `отдохнул минут двадцать` both returned `water`. Qwen does not do this.
|
||
2. **It leaked German.** `slept about seven hours` produced the fact key
|
||
`"7 Stunden geschlafen"`. Grammar-valid, semantically garbage — a sign the multilingual
|
||
mix is not anchored where Maven needs it.
|
||
|
||
The claimed tool-calling advantage did not show up here. `act` is the closest thing this
|
||
fixture has to a tool call, and LFM2.5 got it wrong more often than Qwen, mostly by calling
|
||
it a reminder. It also produced no `fn` slot on any act, same as Qwen.
|
||
|
||
## The Thinking variant
|
||
|
||
Not viable. 36.8% intent accuracy, 10/61 Russian, and no latency saving over Instruct — the
|
||
thinking trace costs time without buying accuracy on a short enum classification. With the
|
||
`enable_thinking=false` diagnostic it collapsed further to 28.9% with 2 route errors
|
||
(`query→reminder ×12`). Do not pursue.
|
||
|
||
## Notes
|
||
|
||
- Nothing crashed, nothing ignored the GBNF grammar, and no model produced unparseable JSON
|
||
in the shippable configurations. Zero route errors for both Instruct and Thinking in
|
||
`llm-only` and `cascade+llm`. The problem with LFM2.5 is what it decides, not whether it
|
||
can emit the contract.
|
||
- The `6 / 6` missed clarify is unchanged across all three models. No model fixes the missing
|
||
refusal lane — that is `Confidence: 1.0` hardcoded in `llmrouter.go` (Vikunja #359), not a
|
||
model property.
|
||
- The report labels every configuration `(0.8B)`; that string is hardcoded in the test, not a
|
||
reflection of which gguf was loaded. Model identity was confirmed per run via `/v1/models`.
|
||
- No Go code was changed for this measurement, and no bug was found that needed one.
|
||
|
||
## What this does not settle
|
||
|
||
Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other
|
||
job — that needs its own fixture. But routing is the load-bearing path and Maven is
|
||
Russian-first, so on the evidence here the switch is not worth making.
|