2026-07-31 19:41:52 +02:00
1 changed files with 105 additions and 0 deletions
@@ -99,3 +99,108 @@ thinking trace costs time without buying accuracy on a short enum classification
 Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other
 job — that needs its own fixture. But routing is the load-bearing path and Maven is
 Russian-first, so on the evidence here the switch is not worth making.
+
+---
+
+# Second sweep, same evening — five models, and a resident-model change
+
+The sections above compared LFM2.5-1.2B against Qwen3.5-0.8B on routing and concluded
+"the switch is not worth making". That still holds. This sweep asked a different
+question — whether a *smaller* model could work, since LFM2.5's published
+instruction-following scores beat Qwen3.5-0.8B badly — and answered it, plus found a
+better resident model by accident.
+
+**Outcome: the resident model is now stock Qwen3-1.7B.** Sub-500M is a dead end.
+
+## Routing — 77 Russian cases, one run each
+
+| model | on disk | llm-only (full) | llm-only (intent) | cascade + fallback |
+|---|---|---|---|---|
+| LFM2.5-230M-Q8_0 | 246 MB | 23.4% | 33.8% | 36.4% |
+| LFM2.5-350M-Q8_0 | 379 MB | 2.6% | **5.2%** | 20.8% |
+| Qwen3.5-0.8B-Q4_K_M | 527 MB | 36.4% | 59.7% | 61.0% |
+| Qwen3.5-2B-UD-Q4_K_XL | 1.34 GB | 42.9% | 62.3% | 63.6% |
+| **Qwen3-1.7B-UD-Q4_K_XL (stock)** | 1.13 GB | **44.2%** | **67.5%** | **72.7%** |
+
+Qwen3-1.7B wins every column, including against a model 20% larger than it.
+
+## Talk fixture — 27 cases, three runs each, idle box
+
+| | Qwen3.5-0.8B | Qwen3-1.7B stock |
+|---|---|---|
+| composite | 13, 11, 8 | **20, 21, 18** |
+| address | 21, 18, 18 | **26, 25, 23** |
+| feminine | 27, 25, 26 | 26, 27, 26 |
+| lang | 27, 27, 26 | 26, 27, 27 |
+| ontopic | 16, 19, 19 | **22, 23, 23** |
+| canned fallbacks | 8, 5, 6 | **0, 2, 0** |
+
+This also fills the row `TALK-EVAL-31-07-2026.md` had to void for contamination:
+**600ch/1024tok on Qwen3.5-0.8B scores 13, 11, 8.**
+
+`address` is the headline. It sat at 18-22 of 27 on the 0.8B no matter how the prompt
+was worded — the prompt explicitly forbids "вы" and the model writes `вашей`,
+`подождите`, `делаете` anyway. That was read as "prompting is out of levers", and it
+was really "0.8B is out of capacity". The 1.7B mostly holds the constraint.
+
+The fallback column matters too: 5-8 of 27 turns on the 0.8B end in a hardcoded
+`"не знаю."`, meaning it failed to emit parseable JSON about a quarter of the time.
+The 1.7B does that 0-2 times.
+
+## Latency — the long tail is not the Thinking block
+
+| | p50 | p95 |
+|---|---|---|
+| Qwen3.5-0.8B | 2.4s, 2.9s, 2.0s | 17.4s, 17.6s, 17.4s |
+| Qwen3-1.7B stock | 2.7s, 2.6s, 2.8s | 16.4s, 6.6s, 3.9s |
+
+p50 is flat across a 2× size difference. The first instinct on seeing the 1.7B's
+16s p95 was "that is the reasoning trace, cap it" — wrong. The 0.8B's p95 is a
+consistent 17s and the 1.7B beat it in two of three runs. The tail is shared and
+lives somewhere else. Do not spend time on `/no_think` on this evidence.
+
+## Sub-500M: not close, and the benchmarks say otherwise for a reason
+
+LFM2.5-350M publishes IFEval 76.96 against Qwen3.5-0.8B's 59.94, and BFCLv3 44.11
+against 35.08 — better at instruction-following and structured output, at 2/3 the
+size. Those numbers are real and they are **English**. Every benchmark in that
+table except Multi-IF is English-only.
+
+In Russian, with a 300-token budget and temperature 0:
+
+- **350M**, «Столица Франции? Ответь кратко.» → *«Сторзит в Париже.»* — `Сторзит` is
+  not a word; it is invented morphology.
+- **350M**, asked to read back a reminder → a fortune cookie about being attentive
+  and confident. No reminder in it.
+- **230M**, «Привет, как дела?» → answered **in Spanish**.
+
+The 230M beating the 350M six-fold on routing (33.8% vs 5.2%) is the other tell:
+when the larger sibling collapses like that it is format compliance failing, not
+reasoning.
+
+This is a pretraining gap, not a fine-tuning gap. Teaching Russian to a 350M from
+near-zero is not an afternoon on a Colab, which was the premise worth checking.
+
+## Why this vindicates the 1.7B CPT
+
+Stock Qwen3-1.7B, untrained and unprompted, answers all three probes in fluent
+correct Russian. What it gets wrong is the persona: *«Привет! Я рад, что ты здесь»*
+— `рад` is masculine and Maven needs `рада`. That is the right kind of remaining
+problem, and it is exactly what the CPT (Vikunja #122) is for.
+
+The 1.7B was the correct model choice. What was wrong was treating it as a
+**blocker**: stock already beats what was deployed, so it ships now and gets
+swapped again when the CPT lands.
+
+## Caveats
+
+- Routing is one run per model, not three. The gaps between families are far larger
+  than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3
+  vs 67.5) is not safe to call on one run.
+- The routing numbers only reach production once the LLM router is wired on. It is
+  still `nil`.
+- `/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download
+  in the wrong directory. The good 1.13 GB copy is in `qwen3/`. Delete the stray one.
+- Harness: `scratchpad/bakeoff.sh`, one server at a time, health-checked before each
+  run, `/v1/models` recorded per run. Never run two LLM consumers at once — see the
+  contamination note in `TALK-EVAL-31-07-2026.md`.