Maven/docs/evals/2026-07-31-model-bakeoff.md

Resident model bake-off — 31-07-2026

Outcome: the resident model is stock Qwen3-1.7B (UD-Q4_K_XL). Two sweeps ran this evening and the second one changed the answer — read to the end before acting on any table here. Second sweep is the one that holds.

First sweep — LFM2.5-1.2B vs Qwen3.5-0.8B

Verdict, scoped to this pair: keep Qwen3.5-0.8B over LFM2.5-1.2B. LFM2.5-1.2B is worse at routing (52.6% vs 60.5% intent accuracy), and the loss is almost entirely Russian (18/61 vs 22/61 RU, while EN is a wash). It is also 2.4× slower. The Thinking variant is far worse again. This verdict still stands as written — it rejects LFM2.5-1.2B. It is not a recommendation to keep 0.8B as the resident model; the second sweep replaced it with Qwen3-1.7B.

Settles Vikunja #278 / #250.

• Same fixture and scorer as docs/evals/2026-07-31-routing.md: internal/router/eval/ (ru_routing_v1.json, 76 held-out cases).
• Reproduce: MAVEN_LLM_URL=http://127.0.0.1:<port> make eval-router (TestLLMRouterBaseline). (This line used to say there is no make eval-models target. There is one now — start a server with the gguf you want, then make eval-models MAVEN_LLM_URL=http://127.0.0.1:<port>. It runs only the LLM test, since the classifier baselines do not depend on the model.)
• All three models served by the same llama-server flags — -c 2048 -ngl 99 -t 6, only -m and --port differ. One server at a time on an otherwise idle box, so latencies are real and not contention.
• Measured on top of the router prompt fix (origin/overnight/router-prompt merged in), so the Qwen column is directly comparable to the numbers already recorded.

Results

llm-only — the model alone. This is the column that measures the model.

	Qwen3.5-0.8B	LFM2.5-1.2B Instruct	LFM2.5-1.2B Thinking
intent-only accuracy	60.5%	52.6%	36.8%
full accuracy (intent+slots+gate)	36.8%	32.9%	21.1%
RU	22/61	18/61	10/61
EN	6/15	7/15	6/15
route errors	0	0	0
p50 / p95 latency	1.05s / 1.71s	2.47s / 3.62s	2.42s / 3.24s
missed clarify	6 / 6	6 / 6	6 / 6

cascade+llm — stage-0 → model → classifier floor, what #320 would actually ship. Same ordering.

	Qwen3.5-0.8B	LFM2.5-1.2B Instruct	LFM2.5-1.2B Thinking
intent-only accuracy	61.8%	55.3%	38.2%
full accuracy	46.1%	42.1%	30.3%
RU / EN	27/61 / 8/15	23/61 / 9/15	15/61 / 8/15
route errors	0	0	0
p50 / p95 latency	1.28s / 1.94s	2.18s / 2.72s	2.27s / 3.19s

Full logs: the three runs are archived in the session scratchpad (qwen08.txt, lfm-instruct.txt, lfm-thinking.txt).

Russian-specific failures — the owner's worry is confirmed

LFM2.5's Russian loss is not spread out. It has one large, specific failure: it hears almost any Russian imperative or short phrase as reminder.

• перезапусти докер → reminder (want act)
• включи вытяжку → reminder (want act)
• закрой жалюзи → reminder (want act)
• заметка: продлить домен в августе → reminder (want note)
• запиши что кран на кухне снова капает → reminder (want note)
• доброе утро → reminder (want chat)
• спасибо тебе → reminder (want note/chat)
• переходи в тихий режим → reminder (want system)

That is note→reminder ×4, act→reminder ×4, chat→reminder ×2 in one run. Qwen's equivalent failure axis is query→fact ×8, which is a narrower and already-understood bug.

Two more Russian-side problems worth naming:

1. Fact keys come back empty or wrong in Russian. воды попил наконец, поужинал, поспал часов пять and отметь что я позавтракал овсянкой all returned an empty key. сходил в душ and отдохнул минут двадцать both returned water. Qwen does not do this.
2. It leaked German. slept about seven hours produced the fact key "7 Stunden geschlafen". Grammar-valid, semantically garbage — a sign the multilingual mix is not anchored where Maven needs it.

The claimed tool-calling advantage did not show up here. act is the closest thing this fixture has to a tool call, and LFM2.5 got it wrong more often than Qwen, mostly by calling it a reminder. It also produced no fn slot on any act, same as Qwen.

The Thinking variant

Not viable. 36.8% intent accuracy, 10/61 Russian, and no latency saving over Instruct — the thinking trace costs time without buying accuracy on a short enum classification. With the enable_thinking=false diagnostic it collapsed further to 28.9% with 2 route errors (query→reminder ×12). Do not pursue.

Notes

• Nothing crashed, nothing ignored the GBNF grammar, and no model produced unparseable JSON in the shippable configurations. Zero route errors for both Instruct and Thinking in llm-only and cascade+llm. The problem with LFM2.5 is what it decides, not whether it can emit the contract.
• The 6 / 6 missed clarify is unchanged across all three models. No model fixes the missing refusal lane — that is Confidence: 1.0 hardcoded in llmrouter.go (Vikunja #359), not a model property.
• The report labels every configuration (0.8B); that string is hardcoded in the test, not a reflection of which gguf was loaded. Model identity was confirmed per run via /v1/models.
• No Go code was changed for this measurement, and no bug was found that needed one.

What this does not settle

Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other job — that needs its own fixture. But routing is the load-bearing path and Maven is Russian-first, so on the evidence here the switch is not worth making.

---

Second sweep, same evening — five models, and a resident-model change

The sections above compared LFM2.5-1.2B against Qwen3.5-0.8B on routing and concluded "the switch is not worth making". That still holds. This sweep asked a different question — whether a smaller model could work, since LFM2.5's published instruction-following scores beat Qwen3.5-0.8B badly — and answered it, plus found a better resident model by accident.

Outcome: the resident model is now stock Qwen3-1.7B. Sub-500M is a dead end.

Routing — 77 Russian cases, one run each

model	on disk	llm-only (full)	llm-only (intent)	cascade + fallback
LFM2.5-230M-Q8_0	246 MB	23.4%	33.8%	36.4%
LFM2.5-350M-Q8_0	379 MB	2.6%	5.2%	20.8%
Qwen3.5-0.8B-Q4_K_M	527 MB	36.4%	59.7%	61.0%
Qwen3.5-2B-UD-Q4_K_XL	1.34 GB	42.9%	62.3%	63.6%
Qwen3-1.7B-UD-Q4_K_XL (stock)	1.13 GB	44.2%	67.5%	72.7%

Qwen3-1.7B wins every column, including against a model 20% larger than it.

Talk fixture — 27 cases, three runs each, idle box

	Qwen3.5-0.8B	Qwen3-1.7B stock
composite	13, 11, 8	20, 21, 18
address	21, 18, 18	26, 25, 23
feminine	27, 25, 26	26, 27, 26
lang	27, 27, 26	26, 27, 27
ontopic	16, 19, 19	22, 23, 23
canned fallbacks	8, 5, 6	0, 2, 0

This also fills the row docs/evals/2026-07-31-talk.md had to void for contamination: 600ch/1024tok on Qwen3.5-0.8B scores 13, 11, 8.

address is the headline. It sat at 18-22 of 27 on the 0.8B no matter how the prompt was worded — the prompt explicitly forbids "вы" and the model writes вашей, подождите, делаете anyway. That was read as "prompting is out of levers", and it was really "0.8B is out of capacity". The 1.7B mostly holds the constraint.

The fallback column matters too: 5-8 of 27 turns on the 0.8B end in a hardcoded "не знаю.", meaning it failed to emit parseable JSON about a quarter of the time. The 1.7B does that 0-2 times.

Latency — the long tail is not the Thinking block

Stale, corrected 2026-08-02. The p50 figures in this table are contention on a shared llama-server, not the model's cost. The router measures p50 825ms / p95 1.2s / max 3.0s in docs/evals/2026-07-31-routing.md, which says so at line 61. Read this table for the shape of the tail only. Take absolute latency from the routing eval.

	p50	p95
Qwen3.5-0.8B	2.4s, 2.9s, 2.0s	17.4s, 17.6s, 17.4s
Qwen3-1.7B stock	2.7s, 2.6s, 2.8s	16.4s, 6.6s, 3.9s

p50 is flat across a 2× size difference. The first instinct on seeing the 1.7B's 16s p95 was "that is the reasoning trace, cap it" — wrong. The 0.8B's p95 is a consistent 17s and the 1.7B beat it in two of three runs. The tail is shared and lives somewhere else. Do not spend time on /no_think on this evidence.

Sub-500M: not close, and the benchmarks say otherwise for a reason

LFM2.5-350M publishes IFEval 76.96 against Qwen3.5-0.8B's 59.94, and BFCLv3 44.11 against 35.08 — better at instruction-following and structured output, at 2/3 the size. Those numbers are real and they are English. Every benchmark in that table except Multi-IF is English-only.

In Russian, with a 300-token budget and temperature 0:

• 350M, «Столица Франции? Ответь кратко.» → «Сторзит в Париже.» — Сторзит is not a word; it is invented morphology.
• 350M, asked to read back a reminder → a fortune cookie about being attentive and confident. No reminder in it.
• 230M, «Привет, как дела?» → answered in Spanish.

The 230M beating the 350M six-fold on routing (33.8% vs 5.2%) is the other tell: when the larger sibling collapses like that it is format compliance failing, not reasoning.

This is a pretraining gap, not a fine-tuning gap. Teaching Russian to a 350M from near-zero is not an afternoon on a Colab, which was the premise worth checking.

Why this vindicates the 1.7B CPT

Stock Qwen3-1.7B, untrained and unprompted, answers all three probes in fluent correct Russian. What it gets wrong is the persona: «Привет! Я рад, что ты здесь» — рад is masculine and Maven needs рада. That is the right kind of remaining problem, and it is exactly what the CPT (Vikunja #122) is for.

The 1.7B was the correct model choice. What was wrong was treating it as a blocker: stock already beats what was deployed, so it ships now and gets swapped again when the CPT lands.

Caveats

• Routing is one run per model, not three. The gaps between families are far larger than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3 vs 67.5) is not safe to call on one run.
• The routing numbers only reach production once the LLM router is wired on. It is still nil. Resolved the same evening: the LLM router is wired at voice.go:214 behind voice.llm_router, the default is on, and deploy/mavend.json sets it true. These numbers are the production path now. Corrected 2026-08-02: the p50 ≈2.7s in the latency table above WAS a bench artifact. It is contention on the shared llama-server, not the model. docs/evals/2026-07-31-routing.md line 61 says so, and measures the router at p50 825ms / p95 1.2s / max 3.0s. Cite that file for latency, not this one.
• /mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf is a 293 MB truncated download in the wrong directory. Deleted 2026-07-31. The good 1.13 GB copy in qwen3/ is what deploy/mavend.json loads.
• Harness: scratchpad/bakeoff.sh, one server at a time, health-checked before each run, /v1/models recorded per run. Never run two LLM consumers at once — see the contamination note in docs/evals/2026-07-31-talk.md.