diff --git a/MODEL-BAKEOFF-31-07-2026.md b/MODEL-BAKEOFF-31-07-2026.md index 6b2447b..c6532eb 100644 --- a/MODEL-BAKEOFF-31-07-2026.md +++ b/MODEL-BAKEOFF-31-07-2026.md @@ -99,3 +99,108 @@ thinking trace costs time without buying accuracy on a short enum classification Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other job — that needs its own fixture. But routing is the load-bearing path and Maven is Russian-first, so on the evidence here the switch is not worth making. + +--- + +# Second sweep, same evening — five models, and a resident-model change + +The sections above compared LFM2.5-1.2B against Qwen3.5-0.8B on routing and concluded +"the switch is not worth making". That still holds. This sweep asked a different +question — whether a *smaller* model could work, since LFM2.5's published +instruction-following scores beat Qwen3.5-0.8B badly — and answered it, plus found a +better resident model by accident. + +**Outcome: the resident model is now stock Qwen3-1.7B.** Sub-500M is a dead end. + +## Routing — 77 Russian cases, one run each + +| model | on disk | llm-only (full) | llm-only (intent) | cascade + fallback | +|---|---|---|---|---| +| LFM2.5-230M-Q8_0 | 246 MB | 23.4% | 33.8% | 36.4% | +| LFM2.5-350M-Q8_0 | 379 MB | 2.6% | **5.2%** | 20.8% | +| Qwen3.5-0.8B-Q4_K_M | 527 MB | 36.4% | 59.7% | 61.0% | +| Qwen3.5-2B-UD-Q4_K_XL | 1.34 GB | 42.9% | 62.3% | 63.6% | +| **Qwen3-1.7B-UD-Q4_K_XL (stock)** | 1.13 GB | **44.2%** | **67.5%** | **72.7%** | + +Qwen3-1.7B wins every column, including against a model 20% larger than it. + +## Talk fixture — 27 cases, three runs each, idle box + +| | Qwen3.5-0.8B | Qwen3-1.7B stock | +|---|---|---| +| composite | 13, 11, 8 | **20, 21, 18** | +| address | 21, 18, 18 | **26, 25, 23** | +| feminine | 27, 25, 26 | 26, 27, 26 | +| lang | 27, 27, 26 | 26, 27, 27 | +| ontopic | 16, 19, 19 | **22, 23, 23** | +| canned fallbacks | 8, 5, 6 | **0, 2, 0** | + +This also fills the row `TALK-EVAL-31-07-2026.md` had to void for contamination: +**600ch/1024tok on Qwen3.5-0.8B scores 13, 11, 8.** + +`address` is the headline. It sat at 18-22 of 27 on the 0.8B no matter how the prompt +was worded — the prompt explicitly forbids "вы" and the model writes `вашей`, +`подождите`, `делаете` anyway. That was read as "prompting is out of levers", and it +was really "0.8B is out of capacity". The 1.7B mostly holds the constraint. + +The fallback column matters too: 5-8 of 27 turns on the 0.8B end in a hardcoded +`"не знаю."`, meaning it failed to emit parseable JSON about a quarter of the time. +The 1.7B does that 0-2 times. + +## Latency — the long tail is not the Thinking block + +| | p50 | p95 | +|---|---|---| +| Qwen3.5-0.8B | 2.4s, 2.9s, 2.0s | 17.4s, 17.6s, 17.4s | +| Qwen3-1.7B stock | 2.7s, 2.6s, 2.8s | 16.4s, 6.6s, 3.9s | + +p50 is flat across a 2× size difference. The first instinct on seeing the 1.7B's +16s p95 was "that is the reasoning trace, cap it" — wrong. The 0.8B's p95 is a +consistent 17s and the 1.7B beat it in two of three runs. The tail is shared and +lives somewhere else. Do not spend time on `/no_think` on this evidence. + +## Sub-500M: not close, and the benchmarks say otherwise for a reason + +LFM2.5-350M publishes IFEval 76.96 against Qwen3.5-0.8B's 59.94, and BFCLv3 44.11 +against 35.08 — better at instruction-following and structured output, at 2/3 the +size. Those numbers are real and they are **English**. Every benchmark in that +table except Multi-IF is English-only. + +In Russian, with a 300-token budget and temperature 0: + +- **350M**, «Столица Франции? Ответь кратко.» → *«Сторзит в Париже.»* — `Сторзит` is + not a word; it is invented morphology. +- **350M**, asked to read back a reminder → a fortune cookie about being attentive + and confident. No reminder in it. +- **230M**, «Привет, как дела?» → answered **in Spanish**. + +The 230M beating the 350M six-fold on routing (33.8% vs 5.2%) is the other tell: +when the larger sibling collapses like that it is format compliance failing, not +reasoning. + +This is a pretraining gap, not a fine-tuning gap. Teaching Russian to a 350M from +near-zero is not an afternoon on a Colab, which was the premise worth checking. + +## Why this vindicates the 1.7B CPT + +Stock Qwen3-1.7B, untrained and unprompted, answers all three probes in fluent +correct Russian. What it gets wrong is the persona: *«Привет! Я рад, что ты здесь»* +— `рад` is masculine and Maven needs `рада`. That is the right kind of remaining +problem, and it is exactly what the CPT (Vikunja #122) is for. + +The 1.7B was the correct model choice. What was wrong was treating it as a +**blocker**: stock already beats what was deployed, so it ships now and gets +swapped again when the CPT lands. + +## Caveats + +- Routing is one run per model, not three. The gaps between families are far larger + than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3 + vs 67.5) is not safe to call on one run. +- The routing numbers only reach production once the LLM router is wired on. It is + still `nil`. +- `/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download + in the wrong directory. The good 1.13 GB copy is in `qwen3/`. Delete the stray one. +- Harness: `scratchpad/bakeoff.sh`, one server at a time, health-checked before each + run, `/v1/models` recorded per run. Never run two LLM consumers at once — see the + contamination note in `TALK-EVAL-31-07-2026.md`.