# Plan — Maven TTS: OmniVoice teacher → piper student (from scratch) > **Goal:** A tiny, CPU-real-time Russian TTS voice for Maven that replaces the > piper `irina` floor. A heavy zero-shot teacher (**k2-fsa/OmniVoice**) clones/designs > Maven's voice and mass-generates a labelled dataset; we train a **piper** voice > **from scratch** on it. The whole quality game is **phoneme correctness (stress + > homographs)**, not the model — the student aligns perfectly to whatever phonemes > we feed it, so garbage stress in = garbage voice out. > > **Read the whole file before starting. Phases in order. Each has a DONE-CHECK.** --- ## 0. Context an agent picking this up must know **Two machines (from `CLAUDE.md`):** | Role | Box | Constraint | |---|---|---| | **Generate + train** | workstation, RX 7900 GRE gfx1100, 16GB VRAM, ROCm 7.2 | teacher + piper-train both run here; GPU may be busy with the LLM CPT — TTS gen waits its turn, but the phoneme/dataset-prep scripts are CPU/file work and run anytime | | **Deploy** | homesrv, Ryzen 5 5600U CPU, 13GB RAM swapping | piper `.onnx` runs real-time on CPU; replaces the irina floor | **Working dirs:** - Teacher + dataset: `/home/kami/Programs/esp32-whisper-fine-tune/tts/` (this is the SSHFS-remote-vs-local split; `/home/kami/Programs/` is **local** to the workstation). - New piper student files: `tts/piper/` (created by this plan). - Daemon that consumes the final voice: `/mnt/server/home/kami/apps/Maven/`. **What already exists in `tts/` (reuse, don't rebuild):** - `find_voice.py` — audition teacher voice-designs → save `ref//maven_reference.{wav,txt}`. - `generate_synthetic_voice.py` — clone the reference, synth per-mood → `dataset//NNNN.wav` + `dataset/metadata.csv`. - `homograph_processor.py` — LLM rewrites sentences to remove stress homographs. - `data/*-voice-dataset-list.txt` — the mood transcript lists (neutral/happy/thinking/confused/tired). - `ruaccent` (RUAccent) + `plus_to_acute` — stress-marking, already imported in the gen script. - A **large qwen-generated dataset already exists** in `dataset/` — but it was made by the **old teacher (Qwen3-TTS)** and its text was **not** stress-marked (the `preprocess` call is commented out). See Phase 1 decision: regenerate with OmniVoice + stressed text, or reuse. **Decisions locked this session:** 1. **Teacher = k2-fsa/OmniVoice** (0.6B on Qwen3-0.6B, 600+ langs incl RU, zero-shot clone + voice-design, RTF 0.025, 24kHz out). Code Apache-2.0; **weights CC-BY-NC** — fine, the teacher never ships, only the piper student deploys. 2. **Student = piper**, trained **from scratch** (no warm-start from irina or any piper voice — GPU/patience is free, and warm-start's only benefit was fast convergence we don't need). 3. **Phoneme correctness is the priority.** Stress-mark + homograph-resolve every transcript so espeak-ng produces correct RU phonemes, and synth the audio from the **same** stressed text so text↔audio↔phonemes stay locked. --- ## 1. Pipeline overview ``` Phase 2 transcripts → stress-marked + homograph-clean mood lists Phase 3 teacher swap → OmniVoice into find_voice.py / generate_synthetic_voice.py Phase 4 generate → dataset//NNNN.wav + metadata.csv (24kHz) Phase 5 build_dataset → piper LJSpeech dir: wav 22.05k + metadata (id|stressed_text) Phase 6 check_phonemes → espeak-ng sanity — eyeball stress before training Phase 7 train.sh → piper preprocess → train from scratch → export ONNX Phase 8 deploy → scp .onnx+json to homesrv; point mavend.json; drop irina ``` **Run order = build order here** (unlike the CPT plan). Phase 2 (phonemes) gates quality and is CPU work — do it while the LLM CPT still owns the GPU. --- ## 2. Transcripts — stress + homographs (the quality lever) The `data/*-voice-dataset-list.txt` files are raw Russian, one sentence per line. Two defects to fix **before** synthesis, because the audio must match the phonemes: 1. **Homographs** — words whose stress (thus meaning) is ambiguous (`за́мок`/`замо́к`, `сто́ят`/`стоя́т`). `homograph_processor.py` already rewrites these away via a local LLM. Run it per mood list → `*-clean.txt`. 2. **Stress marks** — `ruaccent` (`RUAccent().process_all`) inserts `+` before every stressed vowel. This is what makes espeak-ng stress correctly later. The gen script already has `preprocess()` (accent → `plus_to_acute`) but it's **commented out** — the decision is to run stress-marking on the transcripts and keep the `+`-marked form as the canonical text (see Phase 5 for the two acute conventions). ```bash cd /home/kami/Programs/esp32-whisper-fine-tune/tts # 1. de-homograph each mood list (LLM at localhost:10000 must be up) python homograph_processor.py data/*-voice-dataset-list.txt --all # 2. stress-mark → keep the +VOWEL form as canonical (Phase 5 converts per target) ``` **DONE-CHECK 2:** each mood has a cleaned, `+`-stress-marked list; a 30-line random sample eyeballed — every multisyllable word has exactly one `+`, no homographs left. --- ## 3. Teacher swap — OmniVoice into the two gen scripts `find_voice.py` and `generate_synthetic_voice.py` currently call `qwen_tts.Qwen3TTSModel` (`generate_voice_design`, `generate_voice_clone`). Swap the teacher to OmniVoice via the thin adapter `tts/piper/omnivoice_tts.py` (created by this plan) so only two call-sites change and the rest of the resume/metadata logic is untouched. - `pip install omnivoice` (after torch), on the workstation, **GPU free** (post-CPT). - **Verify the exact OmniVoice API** from its README / `pip show omnivoice` — the adapter is written against the documented surface (ref-audio + transcription clone; attribute voice-design) but the method names/signature **must be confirmed** before running. - Keep synthesis text = the **stressed** transcripts from Phase 2 (re-enable `preprocess`, or pass the pre-stressed lists). Teacher and student then share identical text. **DONE-CHECK 3:** `find_voice.py` produces an OmniVoice sample that plays; one clone sample from `generate_synthetic_voice.py --dry-run`-then-real matches the reference timbre. --- ## 4. Generate the dataset ```bash cd /home/kami/Programs/esp32-whisper-fine-tune/tts python find_voice.py # audition → ref//maven_reference.{wav,txt} python generate_synthetic_voice.py # all moods → dataset//NNNN.wav + metadata.csv # resume-safe (.progress.json); --moods / --limit / --dry-run available ``` **Reuse-or-regenerate call:** the existing `dataset/` was made by Qwen3-TTS from **unstressed** text. For phoneme-locked training you want OmniVoice audio from **stressed** text → **regenerate**. (Keeping the old set is only OK if you accept the text↔audio stress mismatch, which is the exact defect this plan exists to kill.) **DONE-CHECK 4:** `dataset/metadata.csv` rows all point to existing wavs; total duration logged (aim ≥ ~2–3h across moods for a from-scratch single-speaker voice); 5 random wavs play and match their transcript. --- ## 5. Build the piper dataset (`tts/piper/build_dataset.py`) Piper wants an **LJSpeech-format** dir: `wav/.wav` (its target sample rate) + `metadata.csv` as `id|text`. Our `metadata.csv` is `file_path,text,mood,emotion_id` and wavs are 24kHz. `build_dataset.py`: 1. Read `dataset/metadata.csv`. 2. **Resample 24000 → 22050 Hz** (piper `medium` default), mono, into `piper/dataset/wav/`. 3. **Stress → espeak convention:** piper phonemizes via espeak-ng, which reads the **combining acute U+0301 *after* the stressed vowel** (NOT the uppercase form `plus_to_acute` makes for qwen). Convert the `+VOWEL` marks to `vowel+U+0301`. 4. Write `piper/dataset/metadata.csv` as `id|stressed_text` (single speaker). Self-check (`__main__`): assert `+а` → `а́`, id/text counts match wav count. **DONE-CHECK 5:** `piper/dataset/` has N wavs at 22050Hz and a metadata line each; `soxi`/`soundfile` confirms sample rate. --- ## 6. Phoneme sanity (`tts/piper/check_phonemes.py`) Before burning GPU-days, confirm espeak-ng stresses correctly. For a sample of transcripts, print `espeak-ng -v ru --ipa` output and flag any multisyllable word whose IPA carries no primary-stress mark `ˈ`. This is the cheapest catch for the "sounds bad" failure — wrong stress shows here, not after training. Requires `espeak-ng` (`sudo apt install espeak-ng`). **ponytail:** thin wrapper over the espeak CLI, ceiling = it only flags *missing* stress, not *wrong-position* stress (that still needs an ear on Phase 4 audio). **DONE-CHECK 6:** sample run shows every content word carrying `ˈ`; obvious homograph words stressed as intended. --- ## 7. Train piper from scratch (`tts/piper/train.sh`) Install piper-train from the piper repo (`rhasspy/piper`, `src/python`): needs torch + pytorch-lightning. ROCm env like `train_cpt.py` (`HSA_OVERRIDE_GFX_VERSION=11.0.0`). ```bash # preprocess: text → espeak-ng phonemes → training cache python -m piper_train.preprocess \ --language ru --input-dir piper/dataset --output-dir piper/train \ --dataset-format ljspeech --single-speaker --sample-rate 22050 # train FROM SCRATCH (no --resume_from_checkpoint), medium quality HSA_OVERRIDE_GFX_VERSION=11.0.0 python -m piper_train \ --dataset-dir piper/train --accelerator gpu --devices 1 \ --batch-size 16 --quality medium --precision 32 \ --max_epochs 4000 --checkpoint-epochs 100 --validation-split 0.02 # export best checkpoint → ONNX python -m piper_train.export_onnx piper/train/lightning_logs/version_0/checkpoints/last.ckpt \ piper/maven.onnx cp piper/train/config.json piper/maven.onnx.json ``` - **From scratch** = no warm-start (decision 2). Early checkpoints sound broken until MAS alignment settles — expected; that's the patience cost we accepted. - `--quality medium` (22.05k) is the CPU-real-time sweet spot; `high` only if homesrv latency allows (measure). **DONE-CHECK 7:** training loss/mel decreases; a mid-run checkpoint synthesizes an intelligible Russian sentence with **correct stress**; final ONNX exported + its `.json`. --- ## 8. Deploy to homesrv ```bash scp piper/maven.onnx piper/maven.onnx.json kami@192.168.1.104: # point deploy/mavend.json at maven.onnx; drop the irina floor ``` Measure CPU tok→audio latency with `--mlock`-class care (homesrv swaps). If too slow, that's a quality/latency knob (medium already chosen), not a re-train. **DONE-CHECK 8:** Maven speaks on homesrv in the new voice, real-time enough for conversation, correct Russian stress in live output. --- ## 9. File manifest (what this plan creates in `tts/piper/`) | File | Phase | Purpose | |---|---|---| | `omnivoice_tts.py` | 3 | thin adapter isolating the OmniVoice API (design + clone) — the single swap point | | `build_dataset.py` | 5 | our `metadata.csv` + 24k wav → piper LJSpeech dir (22.05k, espeak-acute stress) | | `check_phonemes.py` | 6 | espeak-ng stress sanity on a transcript sample | | `train.sh` | 7 | piper preprocess → train-from-scratch → export ONNX (runbook) | Reused, unchanged: `find_voice.py`, `generate_synthetic_voice.py` (two call-sites swapped to the adapter), `homograph_processor.py`, `ruaccent`. --- ## 10. Hard rules (do not violate) 1. **Text↔audio↔phonemes locked:** synth the audio from the **same stressed text** espeak-ng later phonemizes. Never train piper on audio whose transcript stress differs from what you feed the preprocessor. 2. **Two acute conventions — don't confuse them:** `plus_to_acute` (uppercase vowel) is for the **qwen/OmniVoice** teacher; espeak-ng wants **combining U+0301 after the vowel**. `build_dataset.py` converts. 3. **From scratch** — no warm-start (decision 2). Do not `--resume_from_checkpoint` off irina or any piper voice. 4. **Teacher weights are CC-BY-NC** — fine for the never-shipped teacher; the deployed piper voice is your own weights. Don't redistribute the OmniVoice-generated dataset as a product. 5. **Homographs out first** (Phase 2), then stress-mark. Order matters — rewriting a sentence changes which words need stress. 6. Don't advance a phase until its DONE-CHECK passes. Phase 6 (phoneme sanity) is the real gate — a bad phoneme table wastes the whole GPU-days train. ```