Maven/CLAUDE.md

Maven RU LoRA — training guide

Fine-tuning a small LLM so Maven speaks correct, in-character Russian (and English) without breaking Cyrillic. Training scripts live in /home/kami/Programs/esp32-whisper-fine-tune/llm; the model is consumed by this daemon.

Two machines — don't conflate them

Role	Box	Specs	Constraint
Train	this workstation	RX 7900 GRE, gfx1100 (RDNA3), 16GB VRAM, ROCm 7.2 / torch 2.10 HIP, cuda.is_available()==True	fits 1.5B bf16 LoRA with huge headroom
Deploy	homesrv kami@192.168.1.104	Ryzen 5 5600U (Zen3 6c/12t), Vega iGPU (Cezanne), 13GB RAM, already swapping (~8.8GB avail), Ubuntu 24.04	CPU-only inference; caps model at ~1.5B for voice latency

Path duality via SSHFS: /mnt/server/home/kami/ is an SSHFS mount from homesrv. So when you're in this repo:

• /home/kami/Programs/ → local to workpc
• /mnt/server/home/kami/ → remote homesrv filesystem
• /mnt/server/home/kami/apps/Maven/ = this repo = remote filesystem accessed locally

Inference router (localhost:6446) isn't running on either machine. Check for a compose file (docker compose up -d) on whichever box has it before assuming it's available.

• The Vega iGPU is not worth ROCm inference (gfx90c). Inference is CPU llama.cpp; Vulkan can offload a few layers (-ngl 4-ish) for a small speedup.
• homesrv swaps already → every ~200MB of model footprint matters. Keep --mlock on.
• A 4B model at any quant is ~5–8 tok/s here = too slow for voice. Stay ≤1.5B.

Root cause of broken Russian

Cyrillic breakage is a base-model + quantization problem, not a missing LoRA:

• Shipped models (LFM2.5-1.2B-Thinking, Nemotron-3-Nano-4B) are Western, thin-RU, run at Q4_K → mangled endings, Latin/Cyrillic homoglyph swaps.
• A LoRA cannot add spelling knowledge the base lacks. It only steers format / persona / mood / tool-schema / language-mirroring.

Levers, in order of impact:

1. RU-native base — Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct (continued RU pretraining on Qwen2.5). Same latency class as the current 1.2B, doesn't shatter Cyrillic. (Verify exact HF repo id before download — Vikhr naming shifts.)
2. Less aggressive quant — deploy Q4_K_M first (~1.0GB, matches current footprint); bump to Q5_K_M only if Cyrillic still breaks. RU-native base tolerates Q4 far better than the Western models did.
3. LoRA — format/persona/mood/tools, on top of a base that already spells.

Output contract: {response,mood} (Decision B, 2026-07-11)

All LLM output paths now produce {"response":"...","mood":"..."} — the Go side (replier_llm.go, llmphraser.go) parses it in all 4 consumers. Fallback logic preserves backward compat with plain text and the old {"body","summary"} format.

Consumer (file)	Now parses
Replier (cmd/mavend/replier_llm.go)	{"response","mood"} via parseResponseMood
Nudges (internal/phraser/llmphraser.go)	{"response","mood"} via parseResponseMood → Body, Mood on struct
Reminders (internal/phraser/llmphraser.go)	same, summary derived from response
Chat / query (internal/phraser/llmphraser.go)	{"response","mood"} → returns plain response text

Every training sample's assistant turn must produce {"response":"...","mood":"..."}.

Routing architecture: LLM-as-router (REARCH.md, target)

Target arch = LLM-as-router (REARCH.md, supersedes the classifier-first model). One resident model — the CPT'd Qwen3-1.7B (RU-CPT run; replaces LFM, "too meh") — fills both router and phraser roles, two call-sites / two contracts:

Prompt	Contract	Source of truth
route-prompt	{"intent":<enum>, key?, value?, text?, verb?} GBNF-constrained	internal/router/llmrouter.go (routeSystem+routeGrammar)
phrase-prompt	{"response","mood"}	above section

7 intents: fact, reminder, note, query, act, chat, system. Key rule: «запомни/запиши» = note, «напомни/не забудь» = reminder. Embedder is demoted from router to a tool (RAG hint), not a threshold gate.

Status: phase 1 NOT done. Plumbing exists (LLMRouter, Route cascade calls it at router.go:87) but voice.go:209 wires it nil — engine OFF, classifier stopgap still active. Flip nil → NewLLMRouter(qwen) after CPT finishes. Do NOT read the current committed code as the intended design — it's the interim stopgap.

Route-training data: esp32-whisper-fine-tune/llm/gen_route_data.py relabels real utterances through the verbatim routeSystem into {intent,...}. Keep its ROUTE_SYSTEM in sync with the Go const.

Data defects measured in current corpus

• Mood collapse: neutral 1051, thinking 666, happy 280, confused 206, tired 7 (of 2210). Graceful failure (tired) is essentially untrained — the most important small-assistant behavior. Rebalance to ~10–15% combined tired+confused.
• System-prompt drift: 3 variants in synthetic_dataset.jsonl + a 4th in function_calling.jsonl. Pick one canonical system prompt = the exact string the daemon sends at inference, normalize all data to it. Train-prompt ≠ deploy-prompt is a silent accuracy tax.
• Dolphin trap: dolphin_to_messages wraps ~3000 free-form paragraph outputs under a strict-format system prompt → teaches the model the format is optional. Drop it, or reshape via the distiller. Consistency > volume for format LoRAs.
• user-*.jsonl are bare message fragments (not wrapped in {"messages":[...]}); the "messages" in s filter silently drops them. Wrap/normalize before use.

Target data recipe (RU-native base — spelling is NOT the job)

Job = output contract + persona + mood mapping + tool schema + language mirroring.

Bucket	~Share	Source
Persona chit-chat, on-contract	45%	distill (Qwen3-4B / API): topic → canonical prompt → generate → validate (parses, mood∈enum, Cyrillic-clean, length)
Graceful failure / clarify	15%	author unanswerable prompts → tired/confused (current hole)
Tool calls	20%	expand function_calling.jsonl (473) — paraphrase 3–5× RU+EN, vary params
Real utterances	10%	mine Maven dialogue history / logs, relabel
English → English	10%	same buckets, EN in/out, so language-mirroring is trained

Hygiene: dedup near-dup user turns; hold out a real (non-distiller) eval set; route all data through a validator rejecting mixed-script words + invalid JSON — the same validator is the regression metric (Cyrillic eval).

Training (ROCm) — esp32-whisper-fine-tune/llm/train_rocm.py

Adapted from train_llama.py; the ROCm fix is dropping the two CUDA-only pieces:

• No bitsandbytes — bf16 full weights (1.5B fits 16GB easily), no 4-bit BnB.
• optim="adamw_torch_fused" replaces paged_adamw_8bit.
• Base Vikhr-Qwen-2.5-1.5B; Qwen ChatML masking (<|im_start|>assistant / <|im_end|>).
• attn_implementation="eager" (flash-attn ROCm wheels flaky).
• HSA_OVERRIDE_GFX_VERSION=11.0.0 belt-and-suspenders (gfx1100 is officially supported).
• batch 2 × grad-accum 4 (eff. 8), LoRA r16/α32, 3 epochs, grad checkpointing.

Deploy path (after training)

# 1. merge adapter → bf16 weights
python -c "from peft import AutoPeftModelForCausalLM; import torch; \
m=AutoPeftModelForCausalLM.from_pretrained('./Vikhr-Qwen-1.5b-ru-lora',torch_dtype=torch.bfloat16); \
m.merge_and_unload().save_pretrained('./Vikhr-merged'); \
from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('./Vikhr-Qwen-1.5b-ru-lora').save_pretrained('./Vikhr-merged')"
# 2. HF → GGUF (in llama.cpp)
python convert_hf_to_gguf.py ./Vikhr-merged --outfile vikhr-maven-f16.gguf --outtype f16
# 3. quantize (start Q4_K_M, test Cyrillic, bump to Q5_K_M if needed)
./llama-quantize vikhr-maven-f16.gguf Vikhr-Maven-1.5B-Q4_K_M.gguf Q4_K_M
# 4. scp to homesrv; point deploy/mavend.json phraser.model_path at it; --mlock, small -ngl on Vulkan

Open items

• Decide output contract — B: {response,mood}, Go side patched (2026-07-11).
• Base model — Qwen3-1.7B, RU via continued pretraining (not Vikhr). CPT run in progress.
• Write gen_data.py distiller — ran, produced persona_train.jsonl (2045) + eval (107), mood collapse fixed.
• Write gen_route_data.py — route-schema relabeler (run when router up).
• Turn router engine ON — swap voice.go:209 nil → NewLLMRouter(qwen) after CPT (REARCH phase 1).
• Run gen_route_data.py; train route-LoRA (or fold into persona SFT).
• Improve routeSystem prompt for sub-1B disambiguation (awaiting prompt-guy input).
• Write Cyrillic-validity + JSON eval (extend llama-eval-test.py).
• Normalize user-*.jsonl into {"messages":[...]}.