From d0afd9d4f685ffc6e0517026f16d99b4969cb7e1 Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 31 Jul 2026 18:58:20 +0400 Subject: [PATCH 1/3] Make Qwen3-1.7B the resident model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stock Qwen3-1.7B, not the CPT'd one — that training is still running. It won on both fixtures we have, measured tonight on an otherwise idle box: routing, 77 RU cases, intent-only: 67.5% vs 59.7% for Qwen3.5-0.8B talk fixture, 27 cases: 20/27 vs 11-17/27 It also beat Qwen3.5-2B, which is 20% larger, on every routing column. Two other things came with it: n_ctx goes 2048 -> 4096. This is a Thinking variant, so reasoning tokens need the room, and 4096 is the context every score above was measured at. Shipping 2048 would ship something nobody measured. The doc now says not to bother with sub-500M models, because I checked and they are not close. LFM2.5-350M routes at 5.2% — worse than guessing among 7 intents — and answers "столица Франции?" with "Сторзит", which is not a word. The 230M replies to Russian in Spanish. Their published IFEval and BFCL numbers are good and they are all English. Note the routing gain needs the LLM router actually wired on to show up. It is still nil, so this commit buys the phrasing improvement today and the routing improvement when that lands. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ --- CLAUDE.md | 17 ++++++++++++++--- deploy/mavend.json | 4 ++-- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9c17503..f297282 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,9 +7,20 @@ talking over unix sockets; one resident small model for routing + phrasing; whis Deploy target is a Ryzen laptop (homesrv) with Vulkan offload to the Vega iGPU (`n_gpu_layers: 99`, compose passes `/dev/dri` + the render gid) — the resident model stays ≤1.7B either way. -**Resident model:** currently **Qwen3.5-0.8B** (`Q4_K_M`), the smallest checkpoint in the gguf -library, picked for CPU/iGPU latency. The **target** is the locally CPT'd **Qwen3-1.7B**; that -training is still in flight (Vikunja #122), so no such gguf exists yet. Model files live in +**Resident model:** currently **Qwen3-1.7B** (`UD-Q4_K_XL`), stock — not yet the CPT'd one. +It replaced Qwen3.5-0.8B on 2026-07-31 because it measured better on both fixtures we have: +67.5% vs 59.7% intent-only on the 77-case RU routing fixture, and 20/27 vs 11-17/27 on the +talk fixture. See `MODEL-BAKEOFF-31-07-2026.md`. It is a Thinking variant, so `n_ctx` is 4096 +— reasoning tokens need the room, and 4096 is what the scores above were measured at. + +The **target** is still the locally CPT'd **Qwen3-1.7B** (Vikunja #122, training in flight). +Stock already speaks good Russian; what it gets wrong is the persona — it writes `я рад`, +masculine, where Maven needs `рада`. That is what the CPT is for. + +**Do not bother with sub-500M models.** LFM2.5-230M and 350M were measured on 2026-07-31 and +both are unusable in Russian: the 350M routes at 5.2% (worse than guessing) and answers +"столица Франции?" with the invented non-word "Сторзит"; the 230M replies to Russian in +Spanish. Their strong published IFEval/BFCL numbers are English-only. Model files live in `/mnt/hdd1/llms`, bind-mounted to `/opt/maven/models/llm` — which **shadows** the repo's `models/llm/`, so the LFM2.5 gguf sitting there is not loaded by anything. Swapping the resident model is a one-line change to `phraser.model_path` in `deploy/mavend.json`. diff --git a/deploy/mavend.json b/deploy/mavend.json index 2c684b4..962e47d 100644 --- a/deploy/mavend.json +++ b/deploy/mavend.json @@ -6,10 +6,10 @@ "state_dir": "/var/lib/maven", "phraser": { - "model_path": "/opt/maven/models/llm/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf", + "model_path": "/opt/maven/models/llm/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf", "bin_path": "llama-server", "n_gpu_layers": 99, - "n_ctx": 2048, + "n_ctx": 4096, "timeout": "60s", "llm_nudges": false }, -- 2.52.0 From 4f59ba78c6bc7c5c658a272eeee7d0b1e5c44b19 Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 31 Jul 2026 19:08:37 +0400 Subject: [PATCH 2/3] Write down the five-model sweep and why the 1.7B won MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Numbers behind the resident-model change, plus the answer to "could a 230-350M model do this instead" — no, and the reason is worth keeping: LFM2.5's published instruction-following scores beat Qwen3.5-0.8B, and every one of those benchmarks except Multi-IF is English. In Russian the 350M invents non-words and the 230M answers in Spanish. Also fills the row TALK-EVAL-31-07-2026.md had to void for contamination, and corrects a wrong call I nearly made: the 1.7B's 16s p95 looked like the reasoning trace, but the 0.8B sits at 17s in every run and the 1.7B beat it twice out of three. The long tail is shared and is not the Thinking block. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ --- MODEL-BAKEOFF-31-07-2026.md | 105 ++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/MODEL-BAKEOFF-31-07-2026.md b/MODEL-BAKEOFF-31-07-2026.md index 6b2447b..c6532eb 100644 --- a/MODEL-BAKEOFF-31-07-2026.md +++ b/MODEL-BAKEOFF-31-07-2026.md @@ -99,3 +99,108 @@ thinking trace costs time without buying accuracy on a short enum classification Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other job — that needs its own fixture. But routing is the load-bearing path and Maven is Russian-first, so on the evidence here the switch is not worth making. + +--- + +# Second sweep, same evening — five models, and a resident-model change + +The sections above compared LFM2.5-1.2B against Qwen3.5-0.8B on routing and concluded +"the switch is not worth making". That still holds. This sweep asked a different +question — whether a *smaller* model could work, since LFM2.5's published +instruction-following scores beat Qwen3.5-0.8B badly — and answered it, plus found a +better resident model by accident. + +**Outcome: the resident model is now stock Qwen3-1.7B.** Sub-500M is a dead end. + +## Routing — 77 Russian cases, one run each + +| model | on disk | llm-only (full) | llm-only (intent) | cascade + fallback | +|---|---|---|---|---| +| LFM2.5-230M-Q8_0 | 246 MB | 23.4% | 33.8% | 36.4% | +| LFM2.5-350M-Q8_0 | 379 MB | 2.6% | **5.2%** | 20.8% | +| Qwen3.5-0.8B-Q4_K_M | 527 MB | 36.4% | 59.7% | 61.0% | +| Qwen3.5-2B-UD-Q4_K_XL | 1.34 GB | 42.9% | 62.3% | 63.6% | +| **Qwen3-1.7B-UD-Q4_K_XL (stock)** | 1.13 GB | **44.2%** | **67.5%** | **72.7%** | + +Qwen3-1.7B wins every column, including against a model 20% larger than it. + +## Talk fixture — 27 cases, three runs each, idle box + +| | Qwen3.5-0.8B | Qwen3-1.7B stock | +|---|---|---| +| composite | 13, 11, 8 | **20, 21, 18** | +| address | 21, 18, 18 | **26, 25, 23** | +| feminine | 27, 25, 26 | 26, 27, 26 | +| lang | 27, 27, 26 | 26, 27, 27 | +| ontopic | 16, 19, 19 | **22, 23, 23** | +| canned fallbacks | 8, 5, 6 | **0, 2, 0** | + +This also fills the row `TALK-EVAL-31-07-2026.md` had to void for contamination: +**600ch/1024tok on Qwen3.5-0.8B scores 13, 11, 8.** + +`address` is the headline. It sat at 18-22 of 27 on the 0.8B no matter how the prompt +was worded — the prompt explicitly forbids "вы" and the model writes `вашей`, +`подождите`, `делаете` anyway. That was read as "prompting is out of levers", and it +was really "0.8B is out of capacity". The 1.7B mostly holds the constraint. + +The fallback column matters too: 5-8 of 27 turns on the 0.8B end in a hardcoded +`"не знаю."`, meaning it failed to emit parseable JSON about a quarter of the time. +The 1.7B does that 0-2 times. + +## Latency — the long tail is not the Thinking block + +| | p50 | p95 | +|---|---|---| +| Qwen3.5-0.8B | 2.4s, 2.9s, 2.0s | 17.4s, 17.6s, 17.4s | +| Qwen3-1.7B stock | 2.7s, 2.6s, 2.8s | 16.4s, 6.6s, 3.9s | + +p50 is flat across a 2× size difference. The first instinct on seeing the 1.7B's +16s p95 was "that is the reasoning trace, cap it" — wrong. The 0.8B's p95 is a +consistent 17s and the 1.7B beat it in two of three runs. The tail is shared and +lives somewhere else. Do not spend time on `/no_think` on this evidence. + +## Sub-500M: not close, and the benchmarks say otherwise for a reason + +LFM2.5-350M publishes IFEval 76.96 against Qwen3.5-0.8B's 59.94, and BFCLv3 44.11 +against 35.08 — better at instruction-following and structured output, at 2/3 the +size. Those numbers are real and they are **English**. Every benchmark in that +table except Multi-IF is English-only. + +In Russian, with a 300-token budget and temperature 0: + +- **350M**, «Столица Франции? Ответь кратко.» → *«Сторзит в Париже.»* — `Сторзит` is + not a word; it is invented morphology. +- **350M**, asked to read back a reminder → a fortune cookie about being attentive + and confident. No reminder in it. +- **230M**, «Привет, как дела?» → answered **in Spanish**. + +The 230M beating the 350M six-fold on routing (33.8% vs 5.2%) is the other tell: +when the larger sibling collapses like that it is format compliance failing, not +reasoning. + +This is a pretraining gap, not a fine-tuning gap. Teaching Russian to a 350M from +near-zero is not an afternoon on a Colab, which was the premise worth checking. + +## Why this vindicates the 1.7B CPT + +Stock Qwen3-1.7B, untrained and unprompted, answers all three probes in fluent +correct Russian. What it gets wrong is the persona: *«Привет! Я рад, что ты здесь»* +— `рад` is masculine and Maven needs `рада`. That is the right kind of remaining +problem, and it is exactly what the CPT (Vikunja #122) is for. + +The 1.7B was the correct model choice. What was wrong was treating it as a +**blocker**: stock already beats what was deployed, so it ships now and gets +swapped again when the CPT lands. + +## Caveats + +- Routing is one run per model, not three. The gaps between families are far larger + than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3 + vs 67.5) is not safe to call on one run. +- The routing numbers only reach production once the LLM router is wired on. It is + still `nil`. +- `/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download + in the wrong directory. The good 1.13 GB copy is in `qwen3/`. Delete the stray one. +- Harness: `scratchpad/bakeoff.sh`, one server at a time, health-checked before each + run, `/v1/models` recorded per run. Never run two LLM consumers at once — see the + contamination note in `TALK-EVAL-31-07-2026.md`. -- 2.52.0 From 533f0acda89e2fce6e49022d958af4c035745fb0 Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 31 Jul 2026 21:17:07 +0400 Subject: [PATCH 3/3] Lead the bake-off with the answer, not the superseded one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The file ran two sweeps and the second one changed the resident model, but the lede still opened with "Recommendation: keep Qwen3.5-0.8B". Anyone landing on the file read the wrong conclusion and had to scroll 100 lines to find that it had been replaced — and it contradicted CLAUDE.md, which already says the resident model is Qwen3-1.7B. Both sweeps are accurate, so nothing is rewritten. The lede now states the outcome and the first sweep's verdict is scoped to what it actually tested: it rejects LFM2.5-1.2B, which still holds. It never was a case for keeping 0.8B as the resident model. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ --- MODEL-BAKEOFF-31-07-2026.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/MODEL-BAKEOFF-31-07-2026.md b/MODEL-BAKEOFF-31-07-2026.md index c6532eb..0a14993 100644 --- a/MODEL-BAKEOFF-31-07-2026.md +++ b/MODEL-BAKEOFF-31-07-2026.md @@ -1,8 +1,18 @@ # Resident model bake-off — 31-07-2026 -**Recommendation: keep Qwen3.5-0.8B.** LFM2.5-1.2B is worse at routing (52.6% vs 60.5% -intent accuracy), and the loss is almost entirely Russian (18/61 vs 22/61 RU, while EN is a -wash). It is also 2.4× slower. The Thinking variant is far worse again. +**Outcome: the resident model is stock Qwen3-1.7B** (`UD-Q4_K_XL`). Two sweeps ran this +evening and the second one changed the answer — read to the end before acting on any table +here. [Second sweep](#second-sweep-same-evening--five-models-and-a-resident-model-change) +is the one that holds. + +## First sweep — LFM2.5-1.2B vs Qwen3.5-0.8B + +**Verdict, scoped to this pair: keep Qwen3.5-0.8B over LFM2.5-1.2B.** LFM2.5-1.2B is worse +at routing (52.6% vs 60.5% intent accuracy), and the loss is almost entirely Russian +(18/61 vs 22/61 RU, while EN is a wash). It is also 2.4× slower. The Thinking variant is +far worse again. This verdict still stands as written — it rejects LFM2.5-1.2B. It is +**not** a recommendation to keep 0.8B as the resident model; the second sweep replaced it +with Qwen3-1.7B. Settles Vikunja **#278 / #250**. -- 2.52.0