Maven/docs/plans/2026-07-11-tts-piper-student.md

Plan — Maven TTS: OmniVoice teacher → piper student (from scratch)

Goal: A tiny, CPU-real-time Russian TTS voice for Maven that replaces the piper irina floor. A heavy zero-shot teacher (k2-fsa/OmniVoice) clones/designs Maven's voice and mass-generates a labelled dataset; we train a piper voice from scratch on it. The whole quality game is phoneme correctness (stress + homographs), not the model — the student aligns perfectly to whatever phonemes we feed it, so garbage stress in = garbage voice out.

Read the whole file before starting. Phases in order. Each has a DONE-CHECK.

---

0. Context an agent picking this up must know

Two machines (from CLAUDE.md):

Role	Box	Constraint
Generate + train	workstation, RX 7900 GRE gfx1100, 16GB VRAM, ROCm 7.2	teacher + piper-train both run here; GPU may be busy with the LLM CPT — TTS gen waits its turn, but the phoneme/dataset-prep scripts are CPU/file work and run anytime
Deploy	homesrv, Ryzen 5 5600U CPU, 13GB RAM swapping	piper .onnx runs real-time on CPU; replaces the irina floor

Working dirs:

• Teacher + dataset: /home/kami/Programs/esp32-whisper-fine-tune/tts/ (this is the SSHFS-remote-vs-local split; /home/kami/Programs/ is local to the workstation).
• New piper student files: tts/piper/ (created by this plan).
• Daemon that consumes the final voice: /mnt/server/home/kami/apps/Maven/.

What already exists in tts/ (reuse, don't rebuild):

• find_voice.py — audition teacher voice-designs → save ref/<N>/maven_reference.{wav,txt}.
• generate_synthetic_voice.py — clone the reference, synth per-mood → dataset/<mood>/NNNN.wav + dataset/metadata.csv.
• homograph_processor.py — LLM rewrites sentences to remove stress homographs.
• data/*-voice-dataset-list.txt — the mood transcript lists (neutral/happy/thinking/confused/tired).
• ruaccent (RUAccent) + plus_to_acute — stress-marking, already imported in the gen script.
• A large qwen-generated dataset already exists in dataset/ — but it was made by the old teacher (Qwen3-TTS) and its text was not stress-marked (the preprocess call is commented out). See Phase 1 decision: regenerate with OmniVoice + stressed text, or reuse.

Decisions locked this session:

1. Teacher = k2-fsa/OmniVoice (0.6B on Qwen3-0.6B, 600+ langs incl RU, zero-shot clone + voice-design, RTF 0.025, 24kHz out). Code Apache-2.0; weights CC-BY-NC — fine, the teacher never ships, only the piper student deploys.
2. Student = piper, trained from scratch (no warm-start from irina or any piper voice — GPU/patience is free, and warm-start's only benefit was fast convergence we don't need).
3. Phoneme correctness is the priority. Stress-mark + homograph-resolve every transcript so espeak-ng produces correct RU phonemes, and synth the audio from the same stressed text so text↔audio↔phonemes stay locked.

---

1. Pipeline overview

Phase 2  transcripts     →  stress-marked + homograph-clean mood lists
Phase 3  teacher swap     →  OmniVoice into find_voice.py / generate_synthetic_voice.py
Phase 4  generate         →  dataset/<mood>/NNNN.wav + metadata.csv (24kHz)
Phase 5  build_dataset    →  piper LJSpeech dir: wav 22.05k + metadata (id|stressed_text)
Phase 6  check_phonemes   →  espeak-ng sanity — eyeball stress before training
Phase 7  train.sh         →  piper preprocess → train from scratch → export ONNX
Phase 8  deploy           →  scp .onnx+json to homesrv; point mavend.json; drop irina

Run order = build order here (unlike the CPT plan). Phase 2 (phonemes) gates quality and is CPU work — do it while the LLM CPT still owns the GPU.

---

2. Transcripts — stress + homographs (the quality lever)

The data/*-voice-dataset-list.txt files are raw Russian, one sentence per line. Two defects to fix before synthesis, because the audio must match the phonemes:

1. Homographs — words whose stress (thus meaning) is ambiguous (за́мок/замо́к, сто́ят/стоя́т). homograph_processor.py already rewrites these away via a local LLM. Run it per mood list → *-clean.txt.
2. Stress marks — ruaccent (RUAccent().process_all) inserts + before every stressed vowel. This is what makes espeak-ng stress correctly later. The gen script already has preprocess() (accent → plus_to_acute) but it's commented out — the decision is to run stress-marking on the transcripts and keep the +-marked form as the canonical text (see Phase 5 for the two acute conventions).

cd /home/kami/Programs/esp32-whisper-fine-tune/tts
# 1. de-homograph each mood list (LLM at localhost:10000 must be up)
python homograph_processor.py data/*-voice-dataset-list.txt --all
# 2. stress-mark → keep the +VOWEL form as canonical (Phase 5 converts per target)

DONE-CHECK 2: each mood has a cleaned, +-stress-marked list; a 30-line random sample eyeballed — every multisyllable word has exactly one +, no homographs left.

---

3. Teacher swap — OmniVoice into the two gen scripts

find_voice.py and generate_synthetic_voice.py currently call qwen_tts.Qwen3TTSModel (generate_voice_design, generate_voice_clone). Swap the teacher to OmniVoice via the thin adapter tts/piper/omnivoice_tts.py (created by this plan) so only two call-sites change and the rest of the resume/metadata logic is untouched.

• pip install omnivoice (after torch), on the workstation, GPU free (post-CPT).
• Verify the exact OmniVoice API from its README / pip show omnivoice — the adapter is written against the documented surface (ref-audio + transcription clone; attribute voice-design) but the method names/signature must be confirmed before running.
• Keep synthesis text = the stressed transcripts from Phase 2 (re-enable preprocess, or pass the pre-stressed lists). Teacher and student then share identical text.

DONE-CHECK 3: find_voice.py produces an OmniVoice sample that plays; one clone sample from generate_synthetic_voice.py --dry-run-then-real matches the reference timbre.

---

4. Generate the dataset

cd /home/kami/Programs/esp32-whisper-fine-tune/tts
python find_voice.py                         # audition → ref/<N>/maven_reference.{wav,txt}
python generate_synthetic_voice.py           # all moods → dataset/<mood>/NNNN.wav + metadata.csv
#   resume-safe (.progress.json); --moods / --limit / --dry-run available

Reuse-or-regenerate call: the existing dataset/ was made by Qwen3-TTS from unstressed text. For phoneme-locked training you want OmniVoice audio from stressed text → regenerate. (Keeping the old set is only OK if you accept the text↔audio stress mismatch, which is the exact defect this plan exists to kill.)

DONE-CHECK 4: dataset/metadata.csv rows all point to existing wavs; total duration logged (aim ≥ ~2–3h across moods for a from-scratch single-speaker voice); 5 random wavs play and match their transcript.

---

5. Build the piper dataset (tts/piper/build_dataset.py)

Piper wants an LJSpeech-format dir: wav/<id>.wav (its target sample rate) + metadata.csv as id|text. Our metadata.csv is file_path,text,mood,emotion_id and wavs are 24kHz. build_dataset.py:

1. Read dataset/metadata.csv.
2. Resample 24000 → 22050 Hz (piper medium default), mono, into piper/dataset/wav/.
3. Stress → espeak convention: piper phonemizes via espeak-ng, which reads the combining acute U+0301 after the stressed vowel (NOT the uppercase form plus_to_acute makes for qwen). Convert the +VOWEL marks to vowel+U+0301.
4. Write piper/dataset/metadata.csv as id|stressed_text (single speaker).

Self-check (__main__): assert +а → а́, id/text counts match wav count.

DONE-CHECK 5: piper/dataset/ has N wavs at 22050Hz and a metadata line each; soxi/soundfile confirms sample rate.

---

6. Phoneme sanity (tts/piper/check_phonemes.py)

Before burning GPU-days, confirm espeak-ng stresses correctly. For a sample of transcripts, print espeak-ng -v ru --ipa output and flag any multisyllable word whose IPA carries no primary-stress mark ˈ. This is the cheapest catch for the "sounds bad" failure — wrong stress shows here, not after training.

Requires espeak-ng (sudo apt install espeak-ng). ponytail: thin wrapper over the espeak CLI, ceiling = it only flags missing stress, not wrong-position stress (that still needs an ear on Phase 4 audio).

DONE-CHECK 6: sample run shows every content word carrying ˈ; obvious homograph words stressed as intended.

---

7. Train piper from scratch (tts/piper/train.sh)

Install piper-train from the piper repo (rhasspy/piper, src/python): needs torch + pytorch-lightning. ROCm env like train_cpt.py (HSA_OVERRIDE_GFX_VERSION=11.0.0).

# preprocess: text → espeak-ng phonemes → training cache
python -m piper_train.preprocess \
  --language ru --input-dir piper/dataset --output-dir piper/train \
  --dataset-format ljspeech --single-speaker --sample-rate 22050

# train FROM SCRATCH (no --resume_from_checkpoint), medium quality
HSA_OVERRIDE_GFX_VERSION=11.0.0 python -m piper_train \
  --dataset-dir piper/train --accelerator gpu --devices 1 \
  --batch-size 16 --quality medium --precision 32 \
  --max_epochs 4000 --checkpoint-epochs 100 --validation-split 0.02

# export best checkpoint → ONNX
python -m piper_train.export_onnx piper/train/lightning_logs/version_0/checkpoints/last.ckpt \
  piper/maven.onnx
cp piper/train/config.json piper/maven.onnx.json

• From scratch = no warm-start (decision 2). Early checkpoints sound broken until MAS alignment settles — expected; that's the patience cost we accepted.
• --quality medium (22.05k) is the CPU-real-time sweet spot; high only if homesrv latency allows (measure).

DONE-CHECK 7: training loss/mel decreases; a mid-run checkpoint synthesizes an intelligible Russian sentence with correct stress; final ONNX exported + its .json.

---

8. Deploy to homesrv

scp piper/maven.onnx piper/maven.onnx.json kami@192.168.1.104:<voices dir>
# point deploy/mavend.json at maven.onnx; drop the irina floor

Measure CPU tok→audio latency with --mlock-class care (homesrv swaps). If too slow, that's a quality/latency knob (medium already chosen), not a re-train.

DONE-CHECK 8: Maven speaks on homesrv in the new voice, real-time enough for conversation, correct Russian stress in live output.

---

9. File manifest (what this plan creates in tts/piper/)

File	Phase	Purpose
omnivoice_tts.py	3	thin adapter isolating the OmniVoice API (design + clone) — the single swap point
build_dataset.py	5	our metadata.csv + 24k wav → piper LJSpeech dir (22.05k, espeak-acute stress)
check_phonemes.py	6	espeak-ng stress sanity on a transcript sample
train.sh	7	piper preprocess → train-from-scratch → export ONNX (runbook)

Reused, unchanged: find_voice.py, generate_synthetic_voice.py (two call-sites swapped to the adapter), homograph_processor.py, ruaccent.

---

10. Hard rules (do not violate)

1. Text↔audio↔phonemes locked: synth the audio from the same stressed text espeak-ng later phonemizes. Never train piper on audio whose transcript stress differs from what you feed the preprocessor.
2. Two acute conventions — don't confuse them: plus_to_acute (uppercase vowel) is for the qwen/OmniVoice teacher; espeak-ng wants combining U+0301 after the vowel. build_dataset.py converts.
3. From scratch — no warm-start (decision 2). Do not --resume_from_checkpoint off irina or any piper voice.
4. Teacher weights are CC-BY-NC — fine for the never-shipped teacher; the deployed piper voice is your own weights. Don't redistribute the OmniVoice-generated dataset as a product.
5. Homographs out first (Phase 2), then stress-mark. Order matters — rewriting a sentence changes which words need stress.
6. Don't advance a phase until its DONE-CHECK passes. Phase 6 (phoneme sanity) is the real gate — a bad phoneme table wastes the whole GPU-days train.