Maven/MODEL-BAKEOFF-31-07-2026.md

Resident model bake-off — 31-07-2026

Recommendation: keep Qwen3.5-0.8B. LFM2.5-1.2B is worse at routing (52.6% vs 60.5% intent accuracy), and the loss is almost entirely Russian (18/61 vs 22/61 RU, while EN is a wash). It is also 2.4× slower. The Thinking variant is far worse again.

Settles Vikunja #278 / #250.

• Same fixture and scorer as ROUTING-EVAL-31-07-2026.md: internal/router/eval/ (ru_routing_v1.json, 76 held-out cases).
• Reproduce: MAVEN_LLM_URL=http://127.0.0.1:<port> make eval-router (TestLLMRouterBaseline). Note: there is no make eval-models target.
• All three models served by the same llama-server flags — -c 2048 -ngl 99 -t 6, only -m and --port differ. One server at a time on an otherwise idle box, so latencies are real and not contention.
• Measured on top of the router prompt fix (origin/overnight/router-prompt merged in), so the Qwen column is directly comparable to the numbers already recorded.

Results

llm-only — the model alone. This is the column that measures the model.

	Qwen3.5-0.8B	LFM2.5-1.2B Instruct	LFM2.5-1.2B Thinking
intent-only accuracy	60.5%	52.6%	36.8%
full accuracy (intent+slots+gate)	36.8%	32.9%	21.1%
RU	22/61	18/61	10/61
EN	6/15	7/15	6/15
route errors	0	0	0
p50 / p95 latency	1.05s / 1.71s	2.47s / 3.62s	2.42s / 3.24s
missed clarify	6 / 6	6 / 6	6 / 6

cascade+llm — stage-0 → model → classifier floor, what #320 would actually ship. Same ordering.

	Qwen3.5-0.8B	LFM2.5-1.2B Instruct	LFM2.5-1.2B Thinking
intent-only accuracy	61.8%	55.3%	38.2%
full accuracy	46.1%	42.1%	30.3%
RU / EN	27/61 / 8/15	23/61 / 9/15	15/61 / 8/15
route errors	0	0	0
p50 / p95 latency	1.28s / 1.94s	2.18s / 2.72s	2.27s / 3.19s

Full logs: the three runs are archived in the session scratchpad (qwen08.txt, lfm-instruct.txt, lfm-thinking.txt).

Russian-specific failures — the owner's worry is confirmed

LFM2.5's Russian loss is not spread out. It has one large, specific failure: it hears almost any Russian imperative or short phrase as reminder.

• перезапусти докер → reminder (want act)
• включи вытяжку → reminder (want act)
• закрой жалюзи → reminder (want act)
• заметка: продлить домен в августе → reminder (want note)
• запиши что кран на кухне снова капает → reminder (want note)
• доброе утро → reminder (want chat)
• спасибо тебе → reminder (want note/chat)
• переходи в тихий режим → reminder (want system)

That is note→reminder ×4, act→reminder ×4, chat→reminder ×2 in one run. Qwen's equivalent failure axis is query→fact ×8, which is a narrower and already-understood bug.

Two more Russian-side problems worth naming:

1. Fact keys come back empty or wrong in Russian. воды попил наконец, поужинал, поспал часов пять and отметь что я позавтракал овсянкой all returned an empty key. сходил в душ and отдохнул минут двадцать both returned water. Qwen does not do this.
2. It leaked German. slept about seven hours produced the fact key "7 Stunden geschlafen". Grammar-valid, semantically garbage — a sign the multilingual mix is not anchored where Maven needs it.

The claimed tool-calling advantage did not show up here. act is the closest thing this fixture has to a tool call, and LFM2.5 got it wrong more often than Qwen, mostly by calling it a reminder. It also produced no fn slot on any act, same as Qwen.

The Thinking variant

Not viable. 36.8% intent accuracy, 10/61 Russian, and no latency saving over Instruct — the thinking trace costs time without buying accuracy on a short enum classification. With the enable_thinking=false diagnostic it collapsed further to 28.9% with 2 route errors (query→reminder ×12). Do not pursue.

Notes

• Nothing crashed, nothing ignored the GBNF grammar, and no model produced unparseable JSON in the shippable configurations. Zero route errors for both Instruct and Thinking in llm-only and cascade+llm. The problem with LFM2.5 is what it decides, not whether it can emit the contract.
• The 6 / 6 missed clarify is unchanged across all three models. No model fixes the missing refusal lane — that is Confidence: 1.0 hardcoded in llmrouter.go (Vikunja #359), not a model property.
• The report labels every configuration (0.8B); that string is hardcoded in the test, not a reflection of which gguf was loaded. Model identity was confirmed per run via /v1/models.
• No Go code was changed for this measurement, and no bug was found that needed one.

What this does not settle

Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other job — that needs its own fixture. But routing is the load-bearing path and Maven is Russian-first, so on the evidence here the switch is not worth making.