feat: {response,mood} output contract + router removal, TTS piper plan
Daemon side of Decision B: parse {"response","mood"} across the 4 consumers
(replier, nudges, reminders, chat), fall back to legacy formats. Drop the
LLM router — the classifier handles routing; replier/phraser share one
llm.Client (timeout 20s->60s). llm.Client reads reasoning_content when
content is empty (thinking models).
Docs: TTS piper-student plan (OmniVoice teacher -> piper student, from
scratch, phoneme-first). CLAUDE.md training guide.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
kami
@@ -0,0 +1,413 @@
|
||||
# Plan — DIY RU-native base via Continued Pretraining (CPT)
|
||||
|
||||
> **Goal:** Build our *own* RU-native base model instead of using Vikhr. Take a
|
||||
> clean newer base (Qwen3-1.7B-Base), continue-pretrain it on a curated Russian
|
||||
> corpus so it spells Cyrillic natively, then run the existing persona-LoRA →
|
||||
> merge → GGUF → deploy pipeline on top. We own the recipe, the corpus, and the
|
||||
> checkpoint — reproducible, no dependency on a third-party RU repo.
|
||||
>
|
||||
> **This is a big project. The corpus is 80% of the work.** Read the whole file
|
||||
> before starting. Do phases in order. Each phase has a **DONE-CHECK** — do not
|
||||
> advance until it passes.
|
||||
|
||||
---
|
||||
|
||||
## 0. Context an agent picking this up must know
|
||||
|
||||
**Two machines — never conflate (from `CLAUDE.md`):**
|
||||
|
||||
| Role | Box | Specs | Constraint |
|
||||
|---|---|---|---|
|
||||
| **Train** | this workstation | RX 7900 GRE, gfx1100 (RDNA3), **16GB VRAM**, ROCm 7.2 / torch 2.10 HIP, `cuda.is_available()==True` | CPT of 1.7B is *tight* — see Phase 3 VRAM math |
|
||||
| **Deploy** | homesrv `kami@192.168.1.104` | Ryzen 5 5600U CPU, **13GB RAM already swapping** (~8.8GB avail), Ubuntu 24.04 | CPU-only llama.cpp inference; model must stay ≤1.7B and every 200MB matters |
|
||||
|
||||
**Working directories:**
|
||||
- Training scripts & data: `/home/kami/Programs/esp32-whisper-fine-tune/llm/`
|
||||
- existing: `train_rocm.py` (persona LoRA, ChatML), `llama-eval-test.py`, `data/`
|
||||
- **new files from this plan** also go here.
|
||||
- Daemon (consumes final GGUF): `/mnt/server/home/kami/apps/Maven/` (this repo).
|
||||
|
||||
**Output contract (from `CLAUDE.md`, Decision B, locked 2026-07-11):** the daemon
|
||||
parses `{"response":"...","mood":"..."}` in all 4 consumers. This only matters for
|
||||
the **persona LoRA stage (Phase 6)**, NOT for CPT. CPT is raw next-token text, no
|
||||
JSON, no chat format.
|
||||
|
||||
**Why CPT and not just a LoRA (from `CLAUDE.md`):** a LoRA cannot add spelling
|
||||
knowledge the base lacks — it only steers format/persona/mood/tools. CPT changes
|
||||
the base weights themselves, which is the only lever that can add genuine RU
|
||||
language competence. That is exactly why this is more than a LoRA project.
|
||||
|
||||
**ROCm gotchas (inherit from `train_rocm.py`):**
|
||||
- `attn_implementation="eager"` (flash-attn ROCm wheels flaky).
|
||||
- `HSA_OVERRIDE_GFX_VERSION=11.0.0` in env (gfx1100).
|
||||
- **No bitsandbytes** — ROCm build is flaky. This forbids 8-bit Adam; see Phase 3.
|
||||
|
||||
---
|
||||
|
||||
## 1. Pipeline overview (the whole journey)
|
||||
|
||||
```
|
||||
Phase 2 Corpus build → clean packed RU token blocks (.arrow/.bin, ~300M tok)
|
||||
Phase 3 train_cpt.py → CPT'd Qwen3-1.7B-Base (bf16 HF checkpoint)
|
||||
Phase 4 eval → Cyrillic + JSON validator; CPT base vs raw base numbers
|
||||
Phase 5 DECISION GATE → did CPT beat raw? if not, stop / re-corpus
|
||||
Phase 6 persona LoRA → existing train_rocm.py on the CPT'd base (ChatML, {response,mood})
|
||||
Phase 7 merge → GGUF → Q8_0 → deploy to homesrv
|
||||
```
|
||||
|
||||
Build order is **not** the run order: **build the corpus first (Phase 2)** because
|
||||
it gates everything and takes the longest.
|
||||
|
||||
---
|
||||
|
||||
## 2. Corpus — the core deliverable
|
||||
|
||||
**Target: ~300M tokens** (Qwen3 tokenizer), curated. Rationale: <100M won't move
|
||||
the needle vs Qwen3's already-strong multilingual; >1B is weeks of wall-clock on
|
||||
one 16GB card. 300M is the "nudge, don't re-pretrain" sweet spot for 1 epoch.
|
||||
|
||||
**Two token sources, combined:**
|
||||
- **A. Open datasets** (the bulk, real human RU text).
|
||||
- **B. Router-generated synthetic** (fills domain/persona/conversational gaps the
|
||||
open corpora lack — smart-home commands, assistant dialogue, Maven's domain).
|
||||
The user's **router for free inference providers** generates this.
|
||||
|
||||
### 2.1 Corpus mix table (sums to ~300M tokens)
|
||||
|
||||
| # | Bucket | Share | Tokens | Source | Why |
|
||||
|---|---|---|---|---|---|
|
||||
| B1 | Cleaned web RU | 55% | ~165M | `uonlp/CulturaX` (ru) — **already cleaned + deduped** | backbone, broad RU |
|
||||
| B2 | Encyclopedic | 15% | ~45M | `wikimedia/wikipedia` config `20231101.ru` | clean factual RU, correct spelling |
|
||||
| B3 | Books / long-form | 10% | ~30M | `IlyaGusev/librusec` (or subset) | rich morphology, endings |
|
||||
| B4 | Conversational | 8% | ~24M | `Helsinki-NLP/open_subtitles` (ru) | spoken register — matches voice assistant |
|
||||
| B5 | **Router-synthetic** | 10% | ~30M | generated (§2.4) | domain/persona/smart-home RU, absent from open sets |
|
||||
| B6 | Maven real logs | 2% | ~6M | mine daemon dialogue history / logs | exact deploy-domain match |
|
||||
|
||||
**Cap B5 (synthetic) at 10%.** Synthetic-heavy CPT causes model collapse /
|
||||
degeneration. The bulk MUST be real human text. This cap is a hard rule.
|
||||
|
||||
### 2.2 Exact acquisition — open datasets (Bucket A: B1–B4)
|
||||
|
||||
All via HuggingFace `datasets`, streaming to avoid disk blowup. Some are gated
|
||||
(CulturaX, OSCAR) — accept the license on the HF model page while logged in
|
||||
(`huggingface-cli login`) first.
|
||||
|
||||
```python
|
||||
# corpus_fetch.py — run per bucket, writes raw .txt.gz shards to data/cpt_raw/<bucket>/
|
||||
from datasets import load_dataset
|
||||
# B1 CulturaX (gated — accept license first)
|
||||
ds = load_dataset("uonlp/CulturaX", "ru", split="train", streaming=True)
|
||||
# B2 Wikipedia
|
||||
ds = load_dataset("wikimedia/wikipedia", "20231101.ru", split="train", streaming=True)
|
||||
# B3 Books
|
||||
ds = load_dataset("IlyaGusev/librusec", split="train", streaming=True)
|
||||
# B4 Subtitles
|
||||
ds = load_dataset("Helsinki-NLP/open_subtitles", lang1="en", lang2="ru", split="train", streaming=True)
|
||||
```
|
||||
|
||||
Pull enough raw text per bucket to survive filtering (filtering drops ~20–40%);
|
||||
over-pull ~1.5× the target token count, then trim after cleaning. Text field
|
||||
differs per dataset (`text`, `content`, `translation['ru']`) — normalize each to a
|
||||
single `text` string when writing shards.
|
||||
|
||||
**Fallbacks if a source is unavailable:** B1 → `allenai/c4` (ru) or `oscar-corpus/OSCAR-2301` (ru);
|
||||
B3 → `IlyaGusev/gazeta` / Taiga corpus; B4 → `Helsinki-NLP/tatoeba` (ru pairs).
|
||||
|
||||
### 2.3 Cleaning pipeline (mandatory — dirty corpus = dirty base)
|
||||
|
||||
One script `corpus_clean.py`, applied to every raw shard, in this order:
|
||||
|
||||
1. **Unicode fix** — `ftfy.fix_text` (repairs mojibake, normalizes to NFC).
|
||||
2. **Language ID** — fastText `lid.176.bin`; drop any doc where top lang != `ru`
|
||||
or `p(ru) < 0.65`. (Download `lid.176.bin` from fastText releases.)
|
||||
3. **Mixed-script rejection** — regex: reject any *word* containing both Cyrillic
|
||||
and Latin letters (`[а-яё].*[a-z]|[a-z].*[а-яё]` case-insensitive per word).
|
||||
This is the exact homoglyph problem `CLAUDE.md` names. Keep pure-Latin words
|
||||
(URLs/names) but drop homoglyph-contaminated ones.
|
||||
4. **Boilerplate/length** — drop docs <200 chars or >50k chars; strip HTML tags,
|
||||
nav junk, repeated whitespace; drop docs with >30% non-letter chars.
|
||||
5. **Dedup** — MinHash LSH via `datasketch` (`MinHashLSH`, threshold 0.8, 128
|
||||
perms) across ALL buckets combined. Near-dup docs collapse to one. This is the
|
||||
single biggest quality lever — do not skip.
|
||||
6. **PII light-scrub** (optional) — regex phone/email → placeholder, since B6 has
|
||||
real logs.
|
||||
|
||||
Emit cleaned docs as JSONL `{"text": "..."}` in `data/cpt_clean/`.
|
||||
|
||||
**DONE-CHECK 2.3:** run `corpus_stats.py` → per-bucket doc count, char count, and
|
||||
**a 100-doc random sample printed for eyeball review**. No mixed-script words in
|
||||
the sample. If sample looks dirty, fix the filter before proceeding.
|
||||
|
||||
### 2.4 Router-synthetic generation (Bucket B5 — the user's router)
|
||||
|
||||
Purpose: generate ~30M tokens of clean RU text in registers open corpora lack —
|
||||
smart-home/assistant domain, natural conversational RU, Maven-adjacent topics.
|
||||
**This is CPT text (raw prose/dialogue), NOT the `{response,mood}` persona data**
|
||||
(that's Phase 6, a different dataset).
|
||||
|
||||
Protocol (`corpus_synth.py`, calls the free-provider router):
|
||||
1. **Topic seed list** — reuse `data/topics.txt` (already exists, ~46KB of topics)
|
||||
+ add smart-home / daily-assistant / RU-domestic topics.
|
||||
2. **Prompt template** (per topic), ask the router model to produce **long-form
|
||||
natural Russian prose or dialogue** on the topic — NOT JSON, NOT persona, NOT
|
||||
English. e.g. `"Напиши развёрнутый естественный текст на русском языке на тему: {topic}. 400–800 слов, живой разговорный стиль."`
|
||||
3. **Diversify:** vary style knob (formal/casual/dialogue/monologue), vary length,
|
||||
rotate across the router's providers/models so it's not one model's fingerprint
|
||||
(fingerprint uniformity → collapse risk).
|
||||
4. **Route every generation through the SAME cleaning pipeline (§2.3)** — language
|
||||
ID, mixed-script rejection, dedup against the real corpus too. Synthetic that
|
||||
fails Cyrillic cleanliness is exactly what we must not train on.
|
||||
5. **Stop at ~30M tokens.** Enforce the 10% cap.
|
||||
|
||||
**DONE-CHECK 2.4:** synthetic passes the same stats + eyeball sample as 2.3; token
|
||||
count ≤ 10% of total corpus; provider diversity logged.
|
||||
|
||||
### 2.5 Maven real logs (Bucket B6)
|
||||
|
||||
Mine daemon dialogue history / logs for real RU user+assistant utterances. Strip
|
||||
to plain text, run through §2.3. Small (~6M tok) but highest domain value.
|
||||
Source location: check `mavend` logs / any dialogue history store in the repo.
|
||||
|
||||
### 2.6 Tokenize + pack (final corpus artifact)
|
||||
|
||||
`corpus_pack.py`:
|
||||
- Load Qwen3-1.7B-Base tokenizer (`transformers.AutoTokenizer`).
|
||||
- Concatenate all cleaned JSONL, tokenize, append EOS between docs, **pack into
|
||||
fixed 2048-token blocks** (drop the ragged tail). No padding — packing is
|
||||
efficient for CPT.
|
||||
- Save as a HF `datasets` Arrow dir `data/cpt_packed/` (columns: `input_ids`).
|
||||
- Print final total token count — **must be ~300M ± 50M**. Adjust bucket pulls if off.
|
||||
|
||||
**DONE-CHECK 2.6:** `data/cpt_packed/` loads; total tokens logged; a decoded
|
||||
random block reads as clean Russian.
|
||||
|
||||
---
|
||||
|
||||
## 3. `train_cpt.py` — continued pretraining (fits 16GB)
|
||||
|
||||
New file next to `train_rocm.py`. Inherits the ROCm setup from it.
|
||||
|
||||
### 3.1 VRAM reality (why the config is what it is)
|
||||
|
||||
Full-weight CPT of 1.7B with plain Adam ≈ 21–24GB → **does not fit 16GB**. We
|
||||
can't use 8-bit Adam (no bitsandbytes on ROCm). Fitting strategy, in preference
|
||||
order — **try A first, fall back to B**:
|
||||
|
||||
- **A. Full-weight + `adafactor`** — Adafactor has no momentum states, cutting the
|
||||
optimizer-state VRAM roughly in half. Full-weight bf16 + adafactor + gradient
|
||||
checkpointing + batch 1 × grad-accum should fit ~16GB. This is the real
|
||||
"change the base weights" path. **Preferred.**
|
||||
- **B. High-rank DoRA on all linear layers** — if A OOMs or throughput is
|
||||
unbearable. Fits trivially. Weaker for language injection but still adapts more
|
||||
than a small LoRA. Use `r=64, α=128, target all linear`, `use_dora=True`.
|
||||
|
||||
### 3.2 Config
|
||||
|
||||
```python
|
||||
# train_cpt.py essentials
|
||||
import os; os.environ["HSA_OVERRIDE_GFX_VERSION"] = "11.0.0"
|
||||
from transformers import (AutoModelForCausalLM, AutoTokenizer,
|
||||
TrainingArguments, Trainer,
|
||||
DataCollatorForLanguageModeling)
|
||||
|
||||
MODEL = "Qwen/Qwen3-1.7B-Base" # BASE, not Instruct
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL, torch_dtype="bfloat16", attn_implementation="eager")
|
||||
model.gradient_checkpointing_enable()
|
||||
model.config.use_cache = False
|
||||
|
||||
args = TrainingArguments(
|
||||
output_dir="./Qwen3-1.7B-ru-cpt",
|
||||
per_device_train_batch_size=1,
|
||||
gradient_accumulation_steps=16, # eff. batch 16 × 2048 tok
|
||||
num_train_epochs=1, # ONE pass — more risks forgetting
|
||||
learning_rate=1e-5, # LOW — nudging pretrained weights
|
||||
lr_scheduler_type="cosine",
|
||||
warmup_ratio=0.03,
|
||||
optim="adafactor", # path A; no bitsandbytes
|
||||
bf16=True,
|
||||
gradient_checkpointing=True,
|
||||
logging_steps=20,
|
||||
save_steps=500, # checkpoint often — this runs for DAYS
|
||||
save_total_limit=3,
|
||||
report_to="none",
|
||||
)
|
||||
collator = DataCollatorForLanguageModeling(tokenizer, mlm=False) # causal LM
|
||||
```
|
||||
|
||||
- **lr 1e-5 is critical.** CPT lr must be ~10× lower than LoRA lr. High lr =
|
||||
catastrophic forgetting (model forgets English, reasoning, everything). If the
|
||||
eval (Phase 4) shows English/tool ability collapsed, lr was too high.
|
||||
- **1 epoch.** Multiple epochs over 300M tokens overfits and forgets.
|
||||
- **Resume/checkpoint is mandatory** — days-long job; support `--resume_from_checkpoint`.
|
||||
|
||||
### 3.3 Run
|
||||
|
||||
```bash
|
||||
cd /home/kami/Programs/esp32-whisper-fine-tune/llm
|
||||
HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py 2>&1 | tee cpt_run.log
|
||||
```
|
||||
|
||||
**DONE-CHECK 3:** training completes 1 epoch without OOM; loss decreases and
|
||||
plateaus (not NaN, not flat-from-step-0); checkpoint saved to
|
||||
`./Qwen3-1.7B-ru-cpt/`. If OOM → switch to path B (DoRA).
|
||||
|
||||
---
|
||||
|
||||
## 4. Eval — did CPT actually help?
|
||||
|
||||
Extend the existing `llama-eval-test.py` into `eval_cyrillic.py`. This is also an
|
||||
open item in `CLAUDE.md` and is the regression metric for the whole project.
|
||||
|
||||
**Metrics (run on a held-out set of real RU prompts — NOT synthetic, NOT training data):**
|
||||
1. **Cyrillic validity %** — generate on RU prompts; % of outputs with zero
|
||||
mixed-script words and no homoglyph swaps (reuse §2.3 rule 3 as the checker).
|
||||
2. **Perplexity** on a held-out clean RU text set (lower = better RU fit).
|
||||
3. **English-retention check** — a few EN prompts; confirm the model still answers
|
||||
in coherent English (catches catastrophic forgetting).
|
||||
4. **JSON-format sanity** (light, for later) — can it produce `{response,mood}`
|
||||
when asked. (Real test is post-persona-LoRA.)
|
||||
|
||||
**Run it three ways and tabulate:**
|
||||
| Model | Cyrillic valid % | RU PPL | EN retained? |
|
||||
|---|---|---|---|
|
||||
| raw Qwen3-1.7B-Base | (baseline) | | |
|
||||
| our CPT'd base | (must beat raw) | | |
|
||||
|
||||
**DONE-CHECK 4:** table filled with real numbers.
|
||||
|
||||
---
|
||||
|
||||
## 5. DECISION GATE
|
||||
|
||||
- **CPT base beats raw on Cyrillic %/PPL AND English retained** → proceed to Phase 6.
|
||||
- **No improvement** → CPT didn't earn its compute. Options: (a) more/better
|
||||
corpus (usually the fix — dirty or too-small corpus), (b) tune lr, (c) accept
|
||||
raw Qwen3-1.7B as the base and skip CPT. Do NOT proceed to persona LoRA on a
|
||||
CPT base that lost to raw.
|
||||
- **English collapsed** → lr too high or too many epochs; re-run Phase 3 with lower lr.
|
||||
|
||||
---
|
||||
|
||||
## 6. Persona LoRA on the CPT'd base (existing pipeline)
|
||||
|
||||
Now the CPT base is just a better base. Run the **existing** `train_rocm.py`
|
||||
persona/format LoRA on top, with two changes:
|
||||
- Point base model at `./Qwen3-1.7B-ru-cpt/` (the CPT checkpoint), not Vikhr/Qwen2.5.
|
||||
- Confirm ChatML template matches Qwen3 (`<|im_start|>assistant … <|im_end|>`).
|
||||
- The persona **data** is the `{response,mood}` corpus per `CLAUDE.md` Decision B
|
||||
(separate from the CPT corpus): 45% persona chit-chat, 15% graceful failure
|
||||
(`tired`/`confused`), 20% tool calls, 10% real utterances, 10% EN→EN. Mood
|
||||
quotas + validator per `CLAUDE.md`. **This dataset is still an open item** — see
|
||||
`CLAUDE.md` "Target data recipe" and `gen_data.py` open item.
|
||||
|
||||
**DONE-CHECK 6:** LoRA trains; eval (Phase 4) re-run on merged model shows
|
||||
`{response,mood}` parses + Cyrillic clean + moods in enum.
|
||||
|
||||
---
|
||||
|
||||
## 7. Merge → GGUF → quantize → deploy
|
||||
|
||||
Per `CLAUDE.md` deploy path, with **Q8_0 first** (decision this session: start
|
||||
high-precision to remove the Cyrillic-at-Q4 gamble, drop to Q5/Q4 only if homesrv
|
||||
memory chokes):
|
||||
|
||||
```bash
|
||||
# 1. merge LoRA → bf16
|
||||
python -c "from peft import AutoPeftModelForCausalLM; import torch; \
|
||||
m=AutoPeftModelForCausalLM.from_pretrained('./Qwen3-1.7B-ru-lora',torch_dtype=torch.bfloat16); \
|
||||
m.merge_and_unload().save_pretrained('./Qwen3-maven-merged'); \
|
||||
from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('./Qwen3-1.7B-ru-lora').save_pretrained('./Qwen3-maven-merged')"
|
||||
# 2. HF → GGUF (in llama.cpp)
|
||||
python convert_hf_to_gguf.py ./Qwen3-maven-merged --outfile qwen3-maven-f16.gguf --outtype f16
|
||||
# 3. quantize — Q8_0 first (~1.6GB for 1.7B)
|
||||
./llama-quantize qwen3-maven-f16.gguf Qwen3-Maven-1.7B-Q8_0.gguf Q8_0
|
||||
# 4. deploy: scp to homesrv; measure tok/s + swap with --mlock.
|
||||
# If RAM chokes → re-quant Q5_K_M, then Q4_K_M. Point deploy/mavend.json
|
||||
# phraser.model_path at the file; small -ngl on Vulkan for a speedup.
|
||||
```
|
||||
|
||||
**DONE-CHECK 7:** model runs on homesrv at acceptable voice tok/s (measure!),
|
||||
Cyrillic clean in live output, `{response,mood}` parsed by the daemon.
|
||||
|
||||
---
|
||||
|
||||
## 8. File manifest (what this plan creates)
|
||||
|
||||
All in `/home/kami/Programs/esp32-whisper-fine-tune/llm/`:
|
||||
|
||||
| File | Phase | Purpose |
|
||||
|---|---|---|
|
||||
| `corpus_fetch.py` | 2.2 | pull open datasets → `data/cpt_raw/` |
|
||||
| `corpus_synth.py` | 2.4 | router-generate synthetic RU → `data/cpt_raw/synth/` |
|
||||
| `corpus_clean.py` | 2.3 | ftfy + langID + mixed-script + dedup → `data/cpt_clean/` |
|
||||
| `corpus_stats.py` | 2.3 | per-bucket counts + sample dump (DONE-CHECKs) |
|
||||
| `corpus_pack.py` | 2.6 | tokenize + pack 2048 blocks → `data/cpt_packed/` |
|
||||
| `train_cpt.py` | 3 | continued pretraining, adafactor/DoRA |
|
||||
| `eval_cyrillic.py` | 4 | Cyrillic % + PPL + EN-retention (extends `llama-eval-test.py`) |
|
||||
| `data/cpt_packed/` | 2.6 | final corpus artifact (~300M tok) |
|
||||
| `./Qwen3-1.7B-ru-cpt/` | 3 | CPT'd base checkpoint |
|
||||
|
||||
### 8.1 One-time setup
|
||||
|
||||
```bash
|
||||
cd /home/kami/Programs/esp32-whisper-fine-tune/llm
|
||||
|
||||
# python deps (torch/transformers already present from train_rocm.py)
|
||||
pip install datasets ftfy datasketch fasttext openai peft accelerate
|
||||
|
||||
# fastText language-id model → data/lid.176.bin (corpus_clean.py requires it)
|
||||
wget -O data/lid.176.bin \
|
||||
https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin
|
||||
|
||||
# HF login — needed to accept the CulturaX (gated) license
|
||||
huggingface-cli login
|
||||
```
|
||||
|
||||
If `fasttext` wheels fail to build, `pip install fasttext-wheel` is the prebuilt
|
||||
fallback. `lid.176.bin` can also live elsewhere via `export FASTTEXT_LID=/path`.
|
||||
|
||||
### 8.2 Runbook (full run order)
|
||||
|
||||
```bash
|
||||
cd /home/kami/Programs/esp32-whisper-fine-tune/llm
|
||||
|
||||
# ── Phase 2: corpus ──────────────────────────────────────────────
|
||||
python corpus_fetch.py all # 2.2 open datasets
|
||||
ROUTER_BASE_URL="https://your-router/v1" \
|
||||
ROUTER_API_KEY="..." ROUTER_MODELS="m-a,m-b,m-c" \
|
||||
python corpus_synth.py # 2.4 synthetic (≤10%)
|
||||
MAVEN_LOGS=/path/to/dialogue.jsonl python corpus_logs.py # 2.5 optional
|
||||
python corpus_clean.py # 2.3 clean+dedup all
|
||||
python corpus_stats.py # DONE-CHECK 2.3/2.4
|
||||
python corpus_pack.py # 2.6 → data/cpt_packed/
|
||||
|
||||
# ── Phase 3: continued pretraining (days; --resume to continue) ──
|
||||
HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py 2>&1 | tee cpt_run.log
|
||||
# OOM? → CPT_PATH=dora HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py
|
||||
|
||||
# ── Phase 4: eval + gate (Phase 5) ──────────────────────────────
|
||||
python eval_cyrillic.py --model Qwen/Qwen3-1.7B-Base # baseline
|
||||
python eval_cyrillic.py --model ./Qwen3-1.7B-ru-cpt # must beat baseline
|
||||
|
||||
# ── Phase 6-7: persona LoRA → merge → GGUF → Q8_0 → homesrv ──────
|
||||
# train_rocm.py pointed at ./Qwen3-1.7B-ru-cpt ; then §7 deploy block.
|
||||
```
|
||||
|
||||
Third-party deps: `datasets`, `ftfy`, `datasketch`, `fasttext` (+ `lid.176.bin`),
|
||||
`openai`, `peft`, `transformers`, `accelerate`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Hard rules (do not violate)
|
||||
|
||||
1. CPT corpus is **raw text**, no JSON, no ChatML. Persona `{response,mood}` data
|
||||
is a **separate** dataset (Phase 6).
|
||||
2. Synthetic ≤ **10%** of corpus. Bulk must be real human RU.
|
||||
3. CPT lr = **1e-5**, **1 epoch**. Higher = catastrophic forgetting.
|
||||
4. Base = **Qwen3-1.7B-*Base***, not Instruct.
|
||||
5. **Never quantize before merge.** bf16 all the way through, quantize last.
|
||||
6. Every corpus source passes the **same cleaning pipeline** (§2.3), synthetic included.
|
||||
7. Don't advance a phase until its **DONE-CHECK** passes. Phase 5 gate is real —
|
||||
if CPT loses to raw, stop.
|
||||
8. Respect the two-machine split: train on the 7900 GRE, deploy CPU-only on homesrv ≤1.7B.
|
||||
@@ -0,0 +1,249 @@
|
||||
# Plan — Maven TTS: OmniVoice teacher → piper student (from scratch)
|
||||
|
||||
> **Goal:** A tiny, CPU-real-time Russian TTS voice for Maven that replaces the
|
||||
> piper `irina` floor. A heavy zero-shot teacher (**k2-fsa/OmniVoice**) clones/designs
|
||||
> Maven's voice and mass-generates a labelled dataset; we train a **piper** voice
|
||||
> **from scratch** on it. The whole quality game is **phoneme correctness (stress +
|
||||
> homographs)**, not the model — the student aligns perfectly to whatever phonemes
|
||||
> we feed it, so garbage stress in = garbage voice out.
|
||||
>
|
||||
> **Read the whole file before starting. Phases in order. Each has a DONE-CHECK.**
|
||||
|
||||
---
|
||||
|
||||
## 0. Context an agent picking this up must know
|
||||
|
||||
**Two machines (from `CLAUDE.md`):**
|
||||
|
||||
| Role | Box | Constraint |
|
||||
|---|---|---|
|
||||
| **Generate + train** | workstation, RX 7900 GRE gfx1100, 16GB VRAM, ROCm 7.2 | teacher + piper-train both run here; GPU may be busy with the LLM CPT — TTS gen waits its turn, but the phoneme/dataset-prep scripts are CPU/file work and run anytime |
|
||||
| **Deploy** | homesrv, Ryzen 5 5600U CPU, 13GB RAM swapping | piper `.onnx` runs real-time on CPU; replaces the irina floor |
|
||||
|
||||
**Working dirs:**
|
||||
- Teacher + dataset: `/home/kami/Programs/esp32-whisper-fine-tune/tts/` (this is the
|
||||
SSHFS-remote-vs-local split; `/home/kami/Programs/` is **local** to the workstation).
|
||||
- New piper student files: `tts/piper/` (created by this plan).
|
||||
- Daemon that consumes the final voice: `/mnt/server/home/kami/apps/Maven/`.
|
||||
|
||||
**What already exists in `tts/` (reuse, don't rebuild):**
|
||||
- `find_voice.py` — audition teacher voice-designs → save `ref/<N>/maven_reference.{wav,txt}`.
|
||||
- `generate_synthetic_voice.py` — clone the reference, synth per-mood → `dataset/<mood>/NNNN.wav` + `dataset/metadata.csv`.
|
||||
- `homograph_processor.py` — LLM rewrites sentences to remove stress homographs.
|
||||
- `data/*-voice-dataset-list.txt` — the mood transcript lists (neutral/happy/thinking/confused/tired).
|
||||
- `ruaccent` (RUAccent) + `plus_to_acute` — stress-marking, already imported in the gen script.
|
||||
- A **large qwen-generated dataset already exists** in `dataset/` — but it was made by the
|
||||
**old teacher (Qwen3-TTS)** and its text was **not** stress-marked (the `preprocess` call
|
||||
is commented out). See Phase 1 decision: regenerate with OmniVoice + stressed text, or reuse.
|
||||
|
||||
**Decisions locked this session:**
|
||||
1. **Teacher = k2-fsa/OmniVoice** (0.6B on Qwen3-0.6B, 600+ langs incl RU, zero-shot
|
||||
clone + voice-design, RTF 0.025, 24kHz out). Code Apache-2.0; **weights CC-BY-NC**
|
||||
— fine, the teacher never ships, only the piper student deploys.
|
||||
2. **Student = piper**, trained **from scratch** (no warm-start from irina or any
|
||||
piper voice — GPU/patience is free, and warm-start's only benefit was fast
|
||||
convergence we don't need).
|
||||
3. **Phoneme correctness is the priority.** Stress-mark + homograph-resolve every
|
||||
transcript so espeak-ng produces correct RU phonemes, and synth the audio from
|
||||
the **same** stressed text so text↔audio↔phonemes stay locked.
|
||||
|
||||
---
|
||||
|
||||
## 1. Pipeline overview
|
||||
|
||||
```
|
||||
Phase 2 transcripts → stress-marked + homograph-clean mood lists
|
||||
Phase 3 teacher swap → OmniVoice into find_voice.py / generate_synthetic_voice.py
|
||||
Phase 4 generate → dataset/<mood>/NNNN.wav + metadata.csv (24kHz)
|
||||
Phase 5 build_dataset → piper LJSpeech dir: wav 22.05k + metadata (id|stressed_text)
|
||||
Phase 6 check_phonemes → espeak-ng sanity — eyeball stress before training
|
||||
Phase 7 train.sh → piper preprocess → train from scratch → export ONNX
|
||||
Phase 8 deploy → scp .onnx+json to homesrv; point mavend.json; drop irina
|
||||
```
|
||||
|
||||
**Run order = build order here** (unlike the CPT plan). Phase 2 (phonemes) gates
|
||||
quality and is CPU work — do it while the LLM CPT still owns the GPU.
|
||||
|
||||
---
|
||||
|
||||
## 2. Transcripts — stress + homographs (the quality lever)
|
||||
|
||||
The `data/*-voice-dataset-list.txt` files are raw Russian, one sentence per line.
|
||||
Two defects to fix **before** synthesis, because the audio must match the phonemes:
|
||||
|
||||
1. **Homographs** — words whose stress (thus meaning) is ambiguous (`за́мок`/`замо́к`,
|
||||
`сто́ят`/`стоя́т`). `homograph_processor.py` already rewrites these away via a local
|
||||
LLM. Run it per mood list → `*-clean.txt`.
|
||||
2. **Stress marks** — `ruaccent` (`RUAccent().process_all`) inserts `+` before every
|
||||
stressed vowel. This is what makes espeak-ng stress correctly later. The gen script
|
||||
already has `preprocess()` (accent → `plus_to_acute`) but it's **commented out** —
|
||||
the decision is to run stress-marking on the transcripts and keep the `+`-marked
|
||||
form as the canonical text (see Phase 5 for the two acute conventions).
|
||||
|
||||
```bash
|
||||
cd /home/kami/Programs/esp32-whisper-fine-tune/tts
|
||||
# 1. de-homograph each mood list (LLM at localhost:10000 must be up)
|
||||
python homograph_processor.py data/*-voice-dataset-list.txt --all
|
||||
# 2. stress-mark → keep the +VOWEL form as canonical (Phase 5 converts per target)
|
||||
```
|
||||
|
||||
**DONE-CHECK 2:** each mood has a cleaned, `+`-stress-marked list; a 30-line random
|
||||
sample eyeballed — every multisyllable word has exactly one `+`, no homographs left.
|
||||
|
||||
---
|
||||
|
||||
## 3. Teacher swap — OmniVoice into the two gen scripts
|
||||
|
||||
`find_voice.py` and `generate_synthetic_voice.py` currently call `qwen_tts.Qwen3TTSModel`
|
||||
(`generate_voice_design`, `generate_voice_clone`). Swap the teacher to OmniVoice via the
|
||||
thin adapter `tts/piper/omnivoice_tts.py` (created by this plan) so only two call-sites
|
||||
change and the rest of the resume/metadata logic is untouched.
|
||||
|
||||
- `pip install omnivoice` (after torch), on the workstation, **GPU free** (post-CPT).
|
||||
- **Verify the exact OmniVoice API** from its README / `pip show omnivoice` — the adapter
|
||||
is written against the documented surface (ref-audio + transcription clone; attribute
|
||||
voice-design) but the method names/signature **must be confirmed** before running.
|
||||
- Keep synthesis text = the **stressed** transcripts from Phase 2 (re-enable `preprocess`,
|
||||
or pass the pre-stressed lists). Teacher and student then share identical text.
|
||||
|
||||
**DONE-CHECK 3:** `find_voice.py` produces an OmniVoice sample that plays; one clone
|
||||
sample from `generate_synthetic_voice.py --dry-run`-then-real matches the reference timbre.
|
||||
|
||||
---
|
||||
|
||||
## 4. Generate the dataset
|
||||
|
||||
```bash
|
||||
cd /home/kami/Programs/esp32-whisper-fine-tune/tts
|
||||
python find_voice.py # audition → ref/<N>/maven_reference.{wav,txt}
|
||||
python generate_synthetic_voice.py # all moods → dataset/<mood>/NNNN.wav + metadata.csv
|
||||
# resume-safe (.progress.json); --moods / --limit / --dry-run available
|
||||
```
|
||||
|
||||
**Reuse-or-regenerate call:** the existing `dataset/` was made by Qwen3-TTS from
|
||||
**unstressed** text. For phoneme-locked training you want OmniVoice audio from
|
||||
**stressed** text → **regenerate**. (Keeping the old set is only OK if you accept the
|
||||
text↔audio stress mismatch, which is the exact defect this plan exists to kill.)
|
||||
|
||||
**DONE-CHECK 4:** `dataset/metadata.csv` rows all point to existing wavs; total
|
||||
duration logged (aim ≥ ~2–3h across moods for a from-scratch single-speaker voice);
|
||||
5 random wavs play and match their transcript.
|
||||
|
||||
---
|
||||
|
||||
## 5. Build the piper dataset (`tts/piper/build_dataset.py`)
|
||||
|
||||
Piper wants an **LJSpeech-format** dir: `wav/<id>.wav` (its target sample rate) +
|
||||
`metadata.csv` as `id|text`. Our `metadata.csv` is `file_path,text,mood,emotion_id` and
|
||||
wavs are 24kHz. `build_dataset.py`:
|
||||
|
||||
1. Read `dataset/metadata.csv`.
|
||||
2. **Resample 24000 → 22050 Hz** (piper `medium` default), mono, into `piper/dataset/wav/`.
|
||||
3. **Stress → espeak convention:** piper phonemizes via espeak-ng, which reads the
|
||||
**combining acute U+0301 *after* the stressed vowel** (NOT the uppercase form
|
||||
`plus_to_acute` makes for qwen). Convert the `+VOWEL` marks to `vowel+U+0301`.
|
||||
4. Write `piper/dataset/metadata.csv` as `id|stressed_text` (single speaker).
|
||||
|
||||
Self-check (`__main__`): assert `+а` → `а́`, id/text counts match wav count.
|
||||
|
||||
**DONE-CHECK 5:** `piper/dataset/` has N wavs at 22050Hz and a metadata line each;
|
||||
`soxi`/`soundfile` confirms sample rate.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phoneme sanity (`tts/piper/check_phonemes.py`)
|
||||
|
||||
Before burning GPU-days, confirm espeak-ng stresses correctly. For a sample of
|
||||
transcripts, print `espeak-ng -v ru --ipa` output and flag any multisyllable word
|
||||
whose IPA carries no primary-stress mark `ˈ`. This is the cheapest catch for the
|
||||
"sounds bad" failure — wrong stress shows here, not after training.
|
||||
|
||||
Requires `espeak-ng` (`sudo apt install espeak-ng`). **ponytail:** thin wrapper over
|
||||
the espeak CLI, ceiling = it only flags *missing* stress, not *wrong-position* stress
|
||||
(that still needs an ear on Phase 4 audio).
|
||||
|
||||
**DONE-CHECK 6:** sample run shows every content word carrying `ˈ`; obvious homograph
|
||||
words stressed as intended.
|
||||
|
||||
---
|
||||
|
||||
## 7. Train piper from scratch (`tts/piper/train.sh`)
|
||||
|
||||
Install piper-train from the piper repo (`rhasspy/piper`, `src/python`): needs torch +
|
||||
pytorch-lightning. ROCm env like `train_cpt.py` (`HSA_OVERRIDE_GFX_VERSION=11.0.0`).
|
||||
|
||||
```bash
|
||||
# preprocess: text → espeak-ng phonemes → training cache
|
||||
python -m piper_train.preprocess \
|
||||
--language ru --input-dir piper/dataset --output-dir piper/train \
|
||||
--dataset-format ljspeech --single-speaker --sample-rate 22050
|
||||
|
||||
# train FROM SCRATCH (no --resume_from_checkpoint), medium quality
|
||||
HSA_OVERRIDE_GFX_VERSION=11.0.0 python -m piper_train \
|
||||
--dataset-dir piper/train --accelerator gpu --devices 1 \
|
||||
--batch-size 16 --quality medium --precision 32 \
|
||||
--max_epochs 4000 --checkpoint-epochs 100 --validation-split 0.02
|
||||
|
||||
# export best checkpoint → ONNX
|
||||
python -m piper_train.export_onnx piper/train/lightning_logs/version_0/checkpoints/last.ckpt \
|
||||
piper/maven.onnx
|
||||
cp piper/train/config.json piper/maven.onnx.json
|
||||
```
|
||||
|
||||
- **From scratch** = no warm-start (decision 2). Early checkpoints sound broken until
|
||||
MAS alignment settles — expected; that's the patience cost we accepted.
|
||||
- `--quality medium` (22.05k) is the CPU-real-time sweet spot; `high` only if homesrv
|
||||
latency allows (measure).
|
||||
|
||||
**DONE-CHECK 7:** training loss/mel decreases; a mid-run checkpoint synthesizes an
|
||||
intelligible Russian sentence with **correct stress**; final ONNX exported + its `.json`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Deploy to homesrv
|
||||
|
||||
```bash
|
||||
scp piper/maven.onnx piper/maven.onnx.json kami@192.168.1.104:<voices dir>
|
||||
# point deploy/mavend.json at maven.onnx; drop the irina floor
|
||||
```
|
||||
|
||||
Measure CPU tok→audio latency with `--mlock`-class care (homesrv swaps). If too slow,
|
||||
that's a quality/latency knob (medium already chosen), not a re-train.
|
||||
|
||||
**DONE-CHECK 8:** Maven speaks on homesrv in the new voice, real-time enough for
|
||||
conversation, correct Russian stress in live output.
|
||||
|
||||
---
|
||||
|
||||
## 9. File manifest (what this plan creates in `tts/piper/`)
|
||||
|
||||
| File | Phase | Purpose |
|
||||
|---|---|---|
|
||||
| `omnivoice_tts.py` | 3 | thin adapter isolating the OmniVoice API (design + clone) — the single swap point |
|
||||
| `build_dataset.py` | 5 | our `metadata.csv` + 24k wav → piper LJSpeech dir (22.05k, espeak-acute stress) |
|
||||
| `check_phonemes.py` | 6 | espeak-ng stress sanity on a transcript sample |
|
||||
| `train.sh` | 7 | piper preprocess → train-from-scratch → export ONNX (runbook) |
|
||||
|
||||
Reused, unchanged: `find_voice.py`, `generate_synthetic_voice.py` (two call-sites swapped
|
||||
to the adapter), `homograph_processor.py`, `ruaccent`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Hard rules (do not violate)
|
||||
|
||||
1. **Text↔audio↔phonemes locked:** synth the audio from the **same stressed text**
|
||||
espeak-ng later phonemizes. Never train piper on audio whose transcript stress
|
||||
differs from what you feed the preprocessor.
|
||||
2. **Two acute conventions — don't confuse them:** `plus_to_acute` (uppercase vowel)
|
||||
is for the **qwen/OmniVoice** teacher; espeak-ng wants **combining U+0301 after the
|
||||
vowel**. `build_dataset.py` converts.
|
||||
3. **From scratch** — no warm-start (decision 2). Do not `--resume_from_checkpoint`
|
||||
off irina or any piper voice.
|
||||
4. **Teacher weights are CC-BY-NC** — fine for the never-shipped teacher; the deployed
|
||||
piper voice is your own weights. Don't redistribute the OmniVoice-generated dataset
|
||||
as a product.
|
||||
5. **Homographs out first** (Phase 2), then stress-mark. Order matters — rewriting a
|
||||
sentence changes which words need stress.
|
||||
6. Don't advance a phase until its DONE-CHECK passes. Phase 6 (phoneme sanity) is the
|
||||
real gate — a bad phoneme table wastes the whole GPU-days train.
|
||||
```
|
||||
Reference in New Issue
Block a user