Write down the five-model sweep and why the 1.7B won

Numbers behind the resident-model change, plus the answer to "could a 230-350M
model do this instead" — no, and the reason is worth keeping: LFM2.5's published
instruction-following scores beat Qwen3.5-0.8B, and every one of those benchmarks
except Multi-IF is English. In Russian the 350M invents non-words and the 230M
answers in Spanish.

Also fills the row TALK-EVAL-31-07-2026.md had to void for contamination, and
corrects a wrong call I nearly made: the 1.7B's 16s p95 looked like the reasoning
trace, but the 0.8B sits at 17s in every run and the 1.7B beat it twice out of
three. The long tail is shared and is not the Thinking block.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

MODEL-BAKEOFF-31-07-2026.md

@@ -99,3 +99,108 @@ thinking trace costs time without buying accuracy on a short enum classification
 Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other
 job — that needs its own fixture. But routing is the load-bearing path and Maven is
 Russian-first, so on the evidence here the switch is not worth making.
+
+---
+
+# Second sweep, same evening — five models, and a resident-model change
+
+The sections above compared LFM2.5-1.2B against Qwen3.5-0.8B on routing and concluded
+"the switch is not worth making". That still holds. This sweep asked a different
+question — whether a *smaller* model could work, since LFM2.5's published
+instruction-following scores beat Qwen3.5-0.8B badly — and answered it, plus found a
+better resident model by accident.
+
+**Outcome: the resident model is now stock Qwen3-1.7B.** Sub-500M is a dead end.
+
+## Routing — 77 Russian cases, one run each
+
+| model | on disk | llm-only (full) | llm-only (intent) | cascade + fallback |
+|---|---|---|---|---|
+| LFM2.5-230M-Q8_0 | 246 MB | 23.4% | 33.8% | 36.4% |
+| LFM2.5-350M-Q8_0 | 379 MB | 2.6% | **5.2%** | 20.8% |
+| Qwen3.5-0.8B-Q4_K_M | 527 MB | 36.4% | 59.7% | 61.0% |
+| Qwen3.5-2B-UD-Q4_K_XL | 1.34 GB | 42.9% | 62.3% | 63.6% |
+| **Qwen3-1.7B-UD-Q4_K_XL (stock)** | 1.13 GB | **44.2%** | **67.5%** | **72.7%** |
+
+Qwen3-1.7B wins every column, including against a model 20% larger than it.
+
+## Talk fixture — 27 cases, three runs each, idle box
+
+| | Qwen3.5-0.8B | Qwen3-1.7B stock |
+|---|---|---|
+| composite | 13, 11, 8 | **20, 21, 18** |
+| address | 21, 18, 18 | **26, 25, 23** |
+| feminine | 27, 25, 26 | 26, 27, 26 |
+| lang | 27, 27, 26 | 26, 27, 27 |
+| ontopic | 16, 19, 19 | **22, 23, 23** |
+| canned fallbacks | 8, 5, 6 | **0, 2, 0** |
+
+This also fills the row `TALK-EVAL-31-07-2026.md` had to void for contamination:
+**600ch/1024tok on Qwen3.5-0.8B scores 13, 11, 8.**
+
+`address` is the headline. It sat at 18-22 of 27 on the 0.8B no matter how the prompt
+was worded — the prompt explicitly forbids "вы" and the model writes `вашей`,
+`подождите`, `делаете` anyway. That was read as "prompting is out of levers", and it
+was really "0.8B is out of capacity". The 1.7B mostly holds the constraint.
+
+The fallback column matters too: 5-8 of 27 turns on the 0.8B end in a hardcoded
+`"не знаю."`, meaning it failed to emit parseable JSON about a quarter of the time.
+The 1.7B does that 0-2 times.
+
+## Latency — the long tail is not the Thinking block
+
+| | p50 | p95 |
+|---|---|---|
+| Qwen3.5-0.8B | 2.4s, 2.9s, 2.0s | 17.4s, 17.6s, 17.4s |
+| Qwen3-1.7B stock | 2.7s, 2.6s, 2.8s | 16.4s, 6.6s, 3.9s |
+
+p50 is flat across a 2× size difference. The first instinct on seeing the 1.7B's
+16s p95 was "that is the reasoning trace, cap it" — wrong. The 0.8B's p95 is a
+consistent 17s and the 1.7B beat it in two of three runs. The tail is shared and
+lives somewhere else. Do not spend time on `/no_think` on this evidence.
+
+## Sub-500M: not close, and the benchmarks say otherwise for a reason
+
+LFM2.5-350M publishes IFEval 76.96 against Qwen3.5-0.8B's 59.94, and BFCLv3 44.11
+against 35.08 — better at instruction-following and structured output, at 2/3 the
+size. Those numbers are real and they are **English**. Every benchmark in that
+table except Multi-IF is English-only.
+
+In Russian, with a 300-token budget and temperature 0:
+
+- **350M**, «Столица Франции? Ответь кратко.» → *«Сторзит в Париже.»* — `Сторзит` is
+  not a word; it is invented morphology.
+- **350M**, asked to read back a reminder → a fortune cookie about being attentive
+  and confident. No reminder in it.
+- **230M**, «Привет, как дела?» → answered **in Spanish**.
+
+The 230M beating the 350M six-fold on routing (33.8% vs 5.2%) is the other tell:
+when the larger sibling collapses like that it is format compliance failing, not
+reasoning.
+
+This is a pretraining gap, not a fine-tuning gap. Teaching Russian to a 350M from
+near-zero is not an afternoon on a Colab, which was the premise worth checking.
+
+## Why this vindicates the 1.7B CPT
+
+Stock Qwen3-1.7B, untrained and unprompted, answers all three probes in fluent
+correct Russian. What it gets wrong is the persona: *«Привет! Я рад, что ты здесь»*
+— `рад` is masculine and Maven needs `рада`. That is the right kind of remaining
+problem, and it is exactly what the CPT (Vikunja #122) is for.
+
+The 1.7B was the correct model choice. What was wrong was treating it as a
+**blocker**: stock already beats what was deployed, so it ships now and gets
+swapped again when the CPT lands.
+
+## Caveats
+
+- Routing is one run per model, not three. The gaps between families are far larger
+  than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3
+  vs 67.5) is not safe to call on one run.
+- The routing numbers only reach production once the LLM router is wired on. It is
+  still `nil`.
+- `/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download
+  in the wrong directory. The good 1.13 GB copy is in `qwen3/`. Delete the stray one.
+- Harness: `scratchpad/bakeoff.sh`, one server at a time, health-checked before each
+  run, `/v1/models` recorded per run. Never run two LLM consumers at once — see the
+  contamination note in `TALK-EVAL-31-07-2026.md`.