kami
6a5121657a
Daemon side of Decision B: parse {"response","mood"} across the 4 consumers
(replier, nudges, reminders, chat), fall back to legacy formats. Drop the
LLM router — the classifier handles routing; replier/phraser share one
llm.Client (timeout 20s->60s). llm.Client reads reasoning_content when
content is empty (thinking models).
Docs: TTS piper-student plan (OmniVoice teacher -> piper student, from
scratch, phoneme-first). CLAUDE.md training guide.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
122 lines
7.1 KiB
Markdown
122 lines
7.1 KiB
Markdown
# Maven RU LoRA — training guide
|
||
|
||
Fine-tuning a small LLM so Maven speaks correct, in-character Russian (and English)
|
||
without breaking Cyrillic. Training scripts live in
|
||
`/home/kami/Programs/esp32-whisper-fine-tune/llm`; the model is consumed by this
|
||
daemon.
|
||
|
||
## Two machines — don't conflate them
|
||
|
||
| Role | Box | Specs | Constraint |
|
||
|---|---|---|---|
|
||
| **Train** | this workstation | RX 7900 GRE, gfx1100 (RDNA3), **16GB VRAM**, ROCm 7.2 / torch 2.10 HIP, `cuda.is_available()==True` | fits 1.5B bf16 LoRA with huge headroom |
|
||
| **Deploy** | homesrv `kami@192.168.1.104` | Ryzen 5 5600U (Zen3 6c/12t), Vega iGPU (Cezanne), **13GB RAM, already swapping** (~8.8GB avail), Ubuntu 24.04 | **CPU-only** inference; caps model at ~1.5B for voice latency |
|
||
|
||
**Path duality via SSHFS:** `/mnt/server/home/kami/` is an SSHFS mount from homesrv. So when you're in this repo:
|
||
- `/home/kami/Programs/` → **local** to workpc
|
||
- `/mnt/server/home/kami/` → **remote** homesrv filesystem
|
||
- `/mnt/server/home/kami/apps/Maven/` = this repo = remote filesystem accessed locally
|
||
|
||
**Inference router** (`localhost:6446`) isn't running on either machine. Check for a compose file (`docker compose up -d`) on whichever box has it before assuming it's available.
|
||
|
||
- The Vega iGPU is **not** worth ROCm inference (gfx90c). Inference is CPU llama.cpp;
|
||
Vulkan can offload a few layers (`-ngl 4`-ish) for a small speedup.
|
||
- homesrv swaps already → every ~200MB of model footprint matters. Keep `--mlock` on.
|
||
- A 4B model at any quant is ~5–8 tok/s here = too slow for voice. **Stay ≤1.5B.**
|
||
|
||
## Root cause of broken Russian
|
||
|
||
Cyrillic breakage is a **base-model + quantization** problem, **not** a missing LoRA:
|
||
- Shipped models (`LFM2.5-1.2B-Thinking`, `Nemotron-3-Nano-4B`) are Western, thin-RU,
|
||
run at **Q4_K** → mangled endings, Latin/Cyrillic homoglyph swaps.
|
||
- A LoRA **cannot** add spelling knowledge the base lacks. It only steers
|
||
format / persona / mood / tool-schema / language-mirroring.
|
||
|
||
**Levers, in order of impact:**
|
||
1. **RU-native base** — `Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct` (continued RU
|
||
pretraining on Qwen2.5). Same latency class as the current 1.2B, doesn't shatter
|
||
Cyrillic. *(Verify exact HF repo id before download — Vikhr naming shifts.)*
|
||
2. **Less aggressive quant** — deploy Q4_K_M first (~1.0GB, matches current
|
||
footprint); bump to Q5_K_M only if Cyrillic still breaks. RU-native base tolerates
|
||
Q4 far better than the Western models did.
|
||
3. **LoRA** — format/persona/mood/tools, on top of a base that already spells.
|
||
|
||
## Output contract: `{response,mood}` (Decision B, 2026-07-11)
|
||
|
||
All LLM output paths now produce `{"response":"...","mood":"..."}` — the Go side
|
||
(`replier_llm.go`, `llmphraser.go`) parses it in all 4 consumers. Fallback logic
|
||
preserves backward compat with plain text and the old `{"body","summary"}` format.
|
||
|
||
| Consumer (file) | Now parses |
|
||
|---|---|
|
||
| Replier (`cmd/mavend/replier_llm.go`) | `{"response","mood"}` via `parseResponseMood` |
|
||
| Nudges (`internal/phraser/llmphraser.go`) | `{"response","mood"}` via `parseResponseMood` → `Body`, `Mood` on struct |
|
||
| Reminders (`internal/phraser/llmphraser.go`) | same, summary derived from response |
|
||
| Chat / query (`internal/phraser/llmphraser.go`) | `{"response","mood"}` → returns plain response text |
|
||
|
||
Every training sample's assistant turn must produce `{"response":"...","mood":"..."}`.
|
||
|
||
## Data defects measured in current corpus
|
||
|
||
- **Mood collapse:** `neutral 1051, thinking 666, happy 280, confused 206, tired 7`
|
||
(of 2210). Graceful failure (`tired`) is essentially untrained — the most important
|
||
small-assistant behavior. Rebalance to ~10–15% combined `tired`+`confused`.
|
||
- **System-prompt drift:** 3 variants in `synthetic_dataset.jsonl` + a 4th in
|
||
`function_calling.jsonl`. Pick **one canonical system prompt = the exact string the
|
||
daemon sends at inference**, normalize all data to it. Train-prompt ≠ deploy-prompt
|
||
is a silent accuracy tax.
|
||
- **Dolphin trap:** `dolphin_to_messages` wraps ~3000 free-form paragraph outputs
|
||
under a strict-format system prompt → teaches the model the format is optional.
|
||
Drop it, or reshape via the distiller. Consistency > volume for format LoRAs.
|
||
- **`user-*.jsonl`** are bare message fragments (not wrapped in `{"messages":[...]}`);
|
||
the `"messages" in s` filter silently drops them. Wrap/normalize before use.
|
||
|
||
## Target data recipe (RU-native base — spelling is NOT the job)
|
||
|
||
Job = output contract + persona + mood mapping + tool schema + language mirroring.
|
||
|
||
| Bucket | ~Share | Source |
|
||
|---|---|---|
|
||
| Persona chit-chat, on-contract | 45% | distill (Qwen3-4B / API): topic → canonical prompt → generate → validate (parses, mood∈enum, Cyrillic-clean, length) |
|
||
| Graceful failure / clarify | 15% | author unanswerable prompts → `tired`/`confused` (current hole) |
|
||
| Tool calls | 20% | expand `function_calling.jsonl` (473) — paraphrase 3–5× RU+EN, vary params |
|
||
| Real utterances | 10% | mine Maven dialogue history / logs, relabel |
|
||
| English → English | 10% | same buckets, EN in/out, so language-mirroring is trained |
|
||
|
||
**Hygiene:** dedup near-dup user turns; hold out a *real* (non-distiller) eval set;
|
||
route all data through a validator rejecting mixed-script words + invalid JSON — the
|
||
same validator is the regression metric (Cyrillic eval).
|
||
|
||
## Training (ROCm) — `esp32-whisper-fine-tune/llm/train_rocm.py`
|
||
|
||
Adapted from `train_llama.py`; the ROCm fix is dropping the two CUDA-only pieces:
|
||
- **No bitsandbytes** — bf16 full weights (1.5B fits 16GB easily), no 4-bit BnB.
|
||
- `optim="adamw_torch_fused"` replaces `paged_adamw_8bit`.
|
||
- Base `Vikhr-Qwen-2.5-1.5B`; Qwen ChatML masking (`<|im_start|>assistant` / `<|im_end|>`).
|
||
- `attn_implementation="eager"` (flash-attn ROCm wheels flaky).
|
||
- `HSA_OVERRIDE_GFX_VERSION=11.0.0` belt-and-suspenders (gfx1100 is officially supported).
|
||
- batch 2 × grad-accum 4 (eff. 8), LoRA r16/α32, 3 epochs, grad checkpointing.
|
||
|
||
## Deploy path (after training)
|
||
|
||
```bash
|
||
# 1. merge adapter → bf16 weights
|
||
python -c "from peft import AutoPeftModelForCausalLM; import torch; \
|
||
m=AutoPeftModelForCausalLM.from_pretrained('./Vikhr-Qwen-1.5b-ru-lora',torch_dtype=torch.bfloat16); \
|
||
m.merge_and_unload().save_pretrained('./Vikhr-merged'); \
|
||
from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('./Vikhr-Qwen-1.5b-ru-lora').save_pretrained('./Vikhr-merged')"
|
||
# 2. HF → GGUF (in llama.cpp)
|
||
python convert_hf_to_gguf.py ./Vikhr-merged --outfile vikhr-maven-f16.gguf --outtype f16
|
||
# 3. quantize (start Q4_K_M, test Cyrillic, bump to Q5_K_M if needed)
|
||
./llama-quantize vikhr-maven-f16.gguf Vikhr-Maven-1.5B-Q4_K_M.gguf Q4_K_M
|
||
# 4. scp to homesrv; point deploy/mavend.json phraser.model_path at it; --mlock, small -ngl on Vulkan
|
||
```
|
||
|
||
## Open items
|
||
|
||
- [x] **Decide output contract** — **B: `{response,mood}`**, Go side patched (2026-07-11).
|
||
- [ ] Confirm `Vikhrmodels/Vikhr-Qwen-2.5-1.5B-Instruct` repo id on HF.
|
||
- [ ] Write `gen_data.py` distiller (topic → canonical prompt → validate → balanced JSONL w/ mood quotas).
|
||
- [ ] Write Cyrillic-validity + JSON eval (extend `llama-eval-test.py`).
|
||
- [ ] Normalize `user-*.jsonl` into `{"messages":[...]}`.
|