CLAUDE.md

@@ -7,9 +7,20 @@ talking over unix sockets; one resident small model for routing + phrasing; whis
 Deploy target is a Ryzen laptop (homesrv) with Vulkan offload to the Vega iGPU (`n_gpu_layers: 99`,
 compose passes `/dev/dri` + the render gid) — the resident model stays ≤1.7B either way.
 
-**Resident model:** currently **Qwen3.5-0.8B** (`Q4_K_M`), the smallest checkpoint in the gguf
-library, picked for CPU/iGPU latency. The **target** is the locally CPT'd **Qwen3-1.7B**; that
-training is still in flight (Vikunja #122), so no such gguf exists yet. Model files live in
+**Resident model:** currently **Qwen3-1.7B** (`UD-Q4_K_XL`), stock — not yet the CPT'd one.
+It replaced Qwen3.5-0.8B on 2026-07-31 because it measured better on both fixtures we have:
+67.5% vs 59.7% intent-only on the 77-case RU routing fixture, and 20/27 vs 11-17/27 on the
+talk fixture. See `MODEL-BAKEOFF-31-07-2026.md`. It is a Thinking variant, so `n_ctx` is 4096
+— reasoning tokens need the room, and 4096 is what the scores above were measured at.
+
+The **target** is still the locally CPT'd **Qwen3-1.7B** (Vikunja #122, training in flight).
+Stock already speaks good Russian; what it gets wrong is the persona — it writes `я рад`,
+masculine, where Maven needs `рада`. That is what the CPT is for.
+
+**Do not bother with sub-500M models.** LFM2.5-230M and 350M were measured on 2026-07-31 and
+both are unusable in Russian: the 350M routes at 5.2% (worse than guessing) and answers
+"столица Франции?" with the invented non-word "Сторзит"; the 230M replies to Russian in
+Spanish. Their strong published IFEval/BFCL numbers are English-only. Model files live in
 `/mnt/hdd1/llms`, bind-mounted to `/opt/maven/models/llm` — which **shadows** the repo's
 `models/llm/`, so the LFM2.5 gguf sitting there is not loaded by anything. Swapping the resident
 model is a one-line change to `phraser.model_path` in `deploy/mavend.json`.

MODEL-BAKEOFF-31-07-2026.md

@@ -1,8 +1,18 @@
 # Resident model bake-off — 31-07-2026
 
-**Recommendation: keep Qwen3.5-0.8B.** LFM2.5-1.2B is worse at routing (52.6% vs 60.5%
-intent accuracy), and the loss is almost entirely Russian (18/61 vs 22/61 RU, while EN is a
-wash). It is also 2.4× slower. The Thinking variant is far worse again.
+**Outcome: the resident model is stock Qwen3-1.7B** (`UD-Q4_K_XL`). Two sweeps ran this
+evening and the second one changed the answer — read to the end before acting on any table
+here. [Second sweep](#second-sweep-same-evening--five-models-and-a-resident-model-change)
+is the one that holds.
+
+## First sweep — LFM2.5-1.2B vs Qwen3.5-0.8B
+
+**Verdict, scoped to this pair: keep Qwen3.5-0.8B over LFM2.5-1.2B.** LFM2.5-1.2B is worse
+at routing (52.6% vs 60.5% intent accuracy), and the loss is almost entirely Russian
+(18/61 vs 22/61 RU, while EN is a wash). It is also 2.4× slower. The Thinking variant is
+far worse again. This verdict still stands as written — it rejects LFM2.5-1.2B. It is
+**not** a recommendation to keep 0.8B as the resident model; the second sweep replaced it
+with Qwen3-1.7B.
 
 Settles Vikunja **#278 / #250**.
 
@@ -99,3 +109,108 @@ thinking trace costs time without buying accuracy on a short enum classification
 Routing only. LFM2.5 might still phrase better, and phrasing is the resident model's other
 job — that needs its own fixture. But routing is the load-bearing path and Maven is
 Russian-first, so on the evidence here the switch is not worth making.
+
+---
+
+# Second sweep, same evening — five models, and a resident-model change
+
+The sections above compared LFM2.5-1.2B against Qwen3.5-0.8B on routing and concluded
+"the switch is not worth making". That still holds. This sweep asked a different
+question — whether a *smaller* model could work, since LFM2.5's published
+instruction-following scores beat Qwen3.5-0.8B badly — and answered it, plus found a
+better resident model by accident.
+
+**Outcome: the resident model is now stock Qwen3-1.7B.** Sub-500M is a dead end.
+
+## Routing — 77 Russian cases, one run each
+
+| model | on disk | llm-only (full) | llm-only (intent) | cascade + fallback |
+|---|---|---|---|---|
+| LFM2.5-230M-Q8_0 | 246 MB | 23.4% | 33.8% | 36.4% |
+| LFM2.5-350M-Q8_0 | 379 MB | 2.6% | **5.2%** | 20.8% |
+| Qwen3.5-0.8B-Q4_K_M | 527 MB | 36.4% | 59.7% | 61.0% |
+| Qwen3.5-2B-UD-Q4_K_XL | 1.34 GB | 42.9% | 62.3% | 63.6% |
+| **Qwen3-1.7B-UD-Q4_K_XL (stock)** | 1.13 GB | **44.2%** | **67.5%** | **72.7%** |
+
+Qwen3-1.7B wins every column, including against a model 20% larger than it.
+
+## Talk fixture — 27 cases, three runs each, idle box
+
+| | Qwen3.5-0.8B | Qwen3-1.7B stock |
+|---|---|---|
+| composite | 13, 11, 8 | **20, 21, 18** |
+| address | 21, 18, 18 | **26, 25, 23** |
+| feminine | 27, 25, 26 | 26, 27, 26 |
+| lang | 27, 27, 26 | 26, 27, 27 |
+| ontopic | 16, 19, 19 | **22, 23, 23** |
+| canned fallbacks | 8, 5, 6 | **0, 2, 0** |
+
+This also fills the row `TALK-EVAL-31-07-2026.md` had to void for contamination:
+**600ch/1024tok on Qwen3.5-0.8B scores 13, 11, 8.**
+
+`address` is the headline. It sat at 18-22 of 27 on the 0.8B no matter how the prompt
+was worded — the prompt explicitly forbids "вы" and the model writes `вашей`,
+`подождите`, `делаете` anyway. That was read as "prompting is out of levers", and it
+was really "0.8B is out of capacity". The 1.7B mostly holds the constraint.
+
+The fallback column matters too: 5-8 of 27 turns on the 0.8B end in a hardcoded
+`"не знаю."`, meaning it failed to emit parseable JSON about a quarter of the time.
+The 1.7B does that 0-2 times.
+
+## Latency — the long tail is not the Thinking block
+
+| | p50 | p95 |
+|---|---|---|
+| Qwen3.5-0.8B | 2.4s, 2.9s, 2.0s | 17.4s, 17.6s, 17.4s |
+| Qwen3-1.7B stock | 2.7s, 2.6s, 2.8s | 16.4s, 6.6s, 3.9s |
+
+p50 is flat across a 2× size difference. The first instinct on seeing the 1.7B's
+16s p95 was "that is the reasoning trace, cap it" — wrong. The 0.8B's p95 is a
+consistent 17s and the 1.7B beat it in two of three runs. The tail is shared and
+lives somewhere else. Do not spend time on `/no_think` on this evidence.
+
+## Sub-500M: not close, and the benchmarks say otherwise for a reason
+
+LFM2.5-350M publishes IFEval 76.96 against Qwen3.5-0.8B's 59.94, and BFCLv3 44.11
+against 35.08 — better at instruction-following and structured output, at 2/3 the
+size. Those numbers are real and they are **English**. Every benchmark in that
+table except Multi-IF is English-only.
+
+In Russian, with a 300-token budget and temperature 0:
+
+- **350M**, «Столица Франции? Ответь кратко.» → *«Сторзит в Париже.»* — `Сторзит` is
+  not a word; it is invented morphology.
+- **350M**, asked to read back a reminder → a fortune cookie about being attentive
+  and confident. No reminder in it.
+- **230M**, «Привет, как дела?» → answered **in Spanish**.
+
+The 230M beating the 350M six-fold on routing (33.8% vs 5.2%) is the other tell:
+when the larger sibling collapses like that it is format compliance failing, not
+reasoning.
+
+This is a pretraining gap, not a fine-tuning gap. Teaching Russian to a 350M from
+near-zero is not an afternoon on a Colab, which was the premise worth checking.
+
+## Why this vindicates the 1.7B CPT
+
+Stock Qwen3-1.7B, untrained and unprompted, answers all three probes in fluent
+correct Russian. What it gets wrong is the persona: *«Привет! Я рад, что ты здесь»*
+— `рад` is masculine and Maven needs `рада`. That is the right kind of remaining
+problem, and it is exactly what the CPT (Vikunja #122) is for.
+
+The 1.7B was the correct model choice. What was wrong was treating it as a
+**blocker**: stock already beats what was deployed, so it ships now and gets
+swapped again when the CPT lands.
+
+## Caveats
+
+- Routing is one run per model, not three. The gaps between families are far larger
+  than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3
+  vs 67.5) is not safe to call on one run.
+- The routing numbers only reach production once the LLM router is wired on. It is
+  still `nil`.
+- `/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download
+  in the wrong directory. The good 1.13 GB copy is in `qwen3/`. Delete the stray one.
+- Harness: `scratchpad/bakeoff.sh`, one server at a time, health-checked before each
+  run, `/v1/models` recorded per run. Never run two LLM consumers at once — see the
+  contamination note in `TALK-EVAL-31-07-2026.md`.

deploy/mavend.json

@@ -6,10 +6,10 @@
   "state_dir": "/var/lib/maven",
 
   "phraser": {
-    "model_path": "/opt/maven/models/llm/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf",
+    "model_path": "/opt/maven/models/llm/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf",
     "bin_path": "llama-server",
     "n_gpu_layers": 99,
-    "n_ctx": 2048,
+    "n_ctx": 4096,
     "timeout": "60s",
     "llm_nudges": false
   },