# Maven RU LoRA — training guide

Fine-tuning a small LLM so Maven speaks correct, in-character Russian (and English)
without breaking Cyrillic. Training scripts live in
`/home/kami/Programs/esp32-whisper-fine-tune/llm`; the model is consumed by this
daemon.

## Two machines — don't conflate them

| Role | Box | Specs | Constraint |
|---|---|---|---|
| **Train** | this workstation | RX 7900 GRE, gfx1100 (RDNA3), **16GB VRAM**, ROCm 7.2 / torch 2.10 HIP, `cuda.is_available()==True` | fits 1.5B bf16 LoRA with huge headroom |
| **Deploy** | homesrv `kami@192.168.1.104` | Ryzen 5 5600U (Zen3 6c/12t), Vega iGPU (Cezanne), **13GB RAM, already swapping** (~8.8GB avail), Ubuntu 24.04 | **CPU-only** inference; caps model at ~1.5B for voice latency |

**Path duality via SSHFS:** `/mnt/server/home/kami/` is an SSHFS mount from homesrv. So when you're in this repo:
- `/home/kami/Programs/` → **local** to workpc
- `/mnt/server/home/kami/` → **remote** homesrv filesystem
- `/mnt/server/home/kami/apps/Maven/` = this repo = remote filesystem accessed locally

**Inference router** (`localhost:6446`) isn't running on either machine. Check for a compose file (`docker compose up -d`) on whichever box has it before assuming it's available.

- The Vega iGPU is **not** worth ROCm inference (gfx90c). Inference is CPU llama.cpp;
  Vulkan can offload a few layers (`-ngl 4`-ish) for a small speedup.
- homesrv swaps already → every ~200MB of model footprint matters. Keep `--mlock` on.
- A 4B model at any quant is ~5–8 tok/s here = too slow for voice. **Stay ≤1.5B.**

## Root cause of broken Russian

Cyrillic breakage is a **base-model + quantization** problem, **not** a missing LoRA:
- Shipped models (`LFM2.5-1.2B-Thinking`, `Nemotron-3-Nano-4B`) are Western, thin-RU,
  run at **Q4_K** → mangled endings, Latin/Cyrillic homoglyph swaps.
- A LoRA **cannot** add spelling knowledge the base lacks. It only steers
  format / persona / mood / tool-schema / language-mirroring.

**Levers, in order of impact:**
1. **RU-native base** — `Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct` (continued RU
   pretraining on Qwen2.5). Same latency class as the current 1.2B, doesn't shatter
   Cyrillic. *(Verify exact HF repo id before download — Vikhr naming shifts.)*
2. **Less aggressive quant** — deploy Q4_K_M first (~1.0GB, matches current
   footprint); bump to Q5_K_M only if Cyrillic still breaks. RU-native base tolerates
   Q4 far better than the Western models did.
3. **LoRA** — format/persona/mood/tools, on top of a base that already spells.

## Output contract: `{response,mood}` (Decision B, 2026-07-11)

All LLM output paths now produce `{"response":"...","mood":"..."}` — the Go side
(`replier_llm.go`, `llmphraser.go`) parses it in all 4 consumers. Fallback logic
preserves backward compat with plain text and the old `{"body","summary"}` format.

| Consumer (file) | Now parses |
|---|---|
| Replier (`cmd/mavend/replier_llm.go`) | `{"response","mood"}` via `parseResponseMood` |
| Nudges (`internal/phraser/llmphraser.go`) | `{"response","mood"}` via `parseResponseMood` → `Body`, `Mood` on struct |
| Reminders (`internal/phraser/llmphraser.go`) | same, summary derived from response |
| Chat / query (`internal/phraser/llmphraser.go`) | `{"response","mood"}` → returns plain response text |

Every training sample's assistant turn must produce `{"response":"...","mood":"..."}`.

## Routing architecture: LLM-as-router (REARCH.md, target)

Target arch = **LLM-as-router** (`REARCH.md`, supersedes the classifier-first
model). One resident model — the **CPT'd Qwen3-1.7B** (RU-CPT run; replaces LFM,
"too meh") — fills **both** router and phraser roles, two call-sites / two contracts:

| Prompt | Contract | Source of truth |
|---|---|---|
| route-prompt | `{"intent":<enum>, key?, value?, text?, verb?}` GBNF-constrained | `internal/router/llmrouter.go` (`routeSystem`+`routeGrammar`) |
| phrase-prompt | `{"response","mood"}` | above section |

7 intents: `fact, reminder, note, query, act, chat, system`. Key rule:
«запомни/запиши» = note, «напомни/не забудь» = reminder. Embedder is **demoted**
from router to a tool (RAG hint), not a threshold gate.

**Status: phase 1 NOT done.** Plumbing exists (`LLMRouter`, `Route` cascade calls
it at `router.go:87`) but `voice.go:209` wires it `nil` — engine OFF, classifier
stopgap still active. Flip `nil` → `NewLLMRouter(qwen)` after CPT finishes. Do NOT
read the current committed code as the intended design — it's the interim stopgap.

Route-training data: `esp32-whisper-fine-tune/llm/gen_route_data.py` relabels real
utterances through the verbatim `routeSystem` into `{intent,...}`. Keep its
`ROUTE_SYSTEM` in sync with the Go const.

## Data defects measured in current corpus

- **Mood collapse:** `neutral 1051, thinking 666, happy 280, confused 206, tired 7`
  (of 2210). Graceful failure (`tired`) is essentially untrained — the most important
  small-assistant behavior. Rebalance to ~10–15% combined `tired`+`confused`.
- **System-prompt drift:** 3 variants in `synthetic_dataset.jsonl` + a 4th in
  `function_calling.jsonl`. Pick **one canonical system prompt = the exact string the
  daemon sends at inference**, normalize all data to it. Train-prompt ≠ deploy-prompt
  is a silent accuracy tax.
- **Dolphin trap:** `dolphin_to_messages` wraps ~3000 free-form paragraph outputs
  under a strict-format system prompt → teaches the model the format is optional.
  Drop it, or reshape via the distiller. Consistency > volume for format LoRAs.
- **`user-*.jsonl`** are bare message fragments (not wrapped in `{"messages":[...]}`);
  the `"messages" in s` filter silently drops them. Wrap/normalize before use.

## Target data recipe (RU-native base — spelling is NOT the job)

Job = output contract + persona + mood mapping + tool schema + language mirroring.

| Bucket | ~Share | Source |
|---|---|---|
| Persona chit-chat, on-contract | 45% | distill (Qwen3-4B / API): topic → canonical prompt → generate → validate (parses, mood∈enum, Cyrillic-clean, length) |
| Graceful failure / clarify | 15% | author unanswerable prompts → `tired`/`confused` (current hole) |
| Tool calls | 20% | expand `function_calling.jsonl` (473) — paraphrase 3–5× RU+EN, vary params |
| Real utterances | 10% | mine Maven dialogue history / logs, relabel |
| English → English | 10% | same buckets, EN in/out, so language-mirroring is trained |

**Hygiene:** dedup near-dup user turns; hold out a *real* (non-distiller) eval set;
route all data through a validator rejecting mixed-script words + invalid JSON — the
same validator is the regression metric (Cyrillic eval).

## Training (ROCm) — `esp32-whisper-fine-tune/llm/train_rocm.py`

Adapted from `train_llama.py`; the ROCm fix is dropping the two CUDA-only pieces:
- **No bitsandbytes** — bf16 full weights (1.5B fits 16GB easily), no 4-bit BnB.
- `optim="adamw_torch_fused"` replaces `paged_adamw_8bit`.
- Base `Vikhr-Qwen-2.5-1.5B`; Qwen ChatML masking (`<|im_start|>assistant` / `<|im_end|>`).
- `attn_implementation="eager"` (flash-attn ROCm wheels flaky).
- `HSA_OVERRIDE_GFX_VERSION=11.0.0` belt-and-suspenders (gfx1100 is officially supported).
- batch 2 × grad-accum 4 (eff. 8), LoRA r16/α32, 3 epochs, grad checkpointing.

## Deploy path (after training)

```bash
# 1. merge adapter → bf16 weights
python -c "from peft import AutoPeftModelForCausalLM; import torch; \
m=AutoPeftModelForCausalLM.from_pretrained('./Vikhr-Qwen-1.5b-ru-lora',torch_dtype=torch.bfloat16); \
m.merge_and_unload().save_pretrained('./Vikhr-merged'); \
from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('./Vikhr-Qwen-1.5b-ru-lora').save_pretrained('./Vikhr-merged')"
# 2. HF → GGUF (in llama.cpp)
python convert_hf_to_gguf.py ./Vikhr-merged --outfile vikhr-maven-f16.gguf --outtype f16
# 3. quantize (start Q4_K_M, test Cyrillic, bump to Q5_K_M if needed)
./llama-quantize vikhr-maven-f16.gguf Vikhr-Maven-1.5B-Q4_K_M.gguf Q4_K_M
# 4. scp to homesrv; point deploy/mavend.json phraser.model_path at it; --mlock, small -ngl on Vulkan
```

## Open items

- [x] **Decide output contract** — **B: `{response,mood}`**, Go side patched (2026-07-11).
- [x] **Base model** — Qwen3-1.7B, RU via **continued pretraining** (not Vikhr). CPT run in progress.
- [x] **Write `gen_data.py` distiller** — ran, produced `persona_train.jsonl` (2045) + eval (107), mood collapse fixed.
- [x] **Write `gen_route_data.py`** — route-schema relabeler (run when router up).
- [ ] **Turn router engine ON** — swap `voice.go:209` `nil` → `NewLLMRouter(qwen)` after CPT (REARCH phase 1).
- [ ] Run `gen_route_data.py`; train route-LoRA (or fold into persona SFT).
- [ ] Improve `routeSystem` prompt for sub-1B disambiguation (awaiting prompt-guy input).
- [ ] Write Cyrillic-validity + JSON eval (extend `llama-eval-test.py`).
- [ ] Normalize `user-*.jsonl` into `{"messages":[...]}`.