Maven/docs/plans/2026-07-11-tts-piper-student.md

# Plan — Maven TTS: OmniVoice teacher → piper student (from scratch)

> **Goal:** A tiny, CPU-real-time Russian TTS voice for Maven that replaces the
> piper `irina` floor. A heavy zero-shot teacher (**k2-fsa/OmniVoice**) clones/designs
> Maven's voice and mass-generates a labelled dataset; we train a **piper** voice
> **from scratch** on it. The whole quality game is **phoneme correctness (stress +
> homographs)**, not the model — the student aligns perfectly to whatever phonemes
> we feed it, so garbage stress in = garbage voice out.
>
> **Read the whole file before starting. Phases in order. Each has a DONE-CHECK.**

---

## 0. Context an agent picking this up must know

**Two machines (from `CLAUDE.md`):**

| Role | Box | Constraint |
|---|---|---|
| **Generate + train** | workstation, RX 7900 GRE gfx1100, 16GB VRAM, ROCm 7.2 | teacher + piper-train both run here; GPU may be busy with the LLM CPT — TTS gen waits its turn, but the phoneme/dataset-prep scripts are CPU/file work and run anytime |
| **Deploy** | homesrv, Ryzen 5 5600U CPU, 13GB RAM swapping | piper `.onnx` runs real-time on CPU; replaces the irina floor |

**Working dirs:**
- Teacher + dataset: `/home/kami/Programs/esp32-whisper-fine-tune/tts/` (this is the
  SSHFS-remote-vs-local split; `/home/kami/Programs/` is **local** to the workstation).
- New piper student files: `tts/piper/` (created by this plan).
- Daemon that consumes the final voice: `/mnt/server/home/kami/apps/Maven/`.

**What already exists in `tts/` (reuse, don't rebuild):**
- `find_voice.py` — audition teacher voice-designs → save `ref/<N>/maven_reference.{wav,txt}`.
- `generate_synthetic_voice.py` — clone the reference, synth per-mood → `dataset/<mood>/NNNN.wav` + `dataset/metadata.csv`.
- `homograph_processor.py` — LLM rewrites sentences to remove stress homographs.
- `data/*-voice-dataset-list.txt` — the mood transcript lists (neutral/happy/thinking/confused/tired).
- `ruaccent` (RUAccent) + `plus_to_acute` — stress-marking, already imported in the gen script.
- A **large qwen-generated dataset already exists** in `dataset/` — but it was made by the
  **old teacher (Qwen3-TTS)** and its text was **not** stress-marked (the `preprocess` call
  is commented out). See Phase 1 decision: regenerate with OmniVoice + stressed text, or reuse.

**Decisions locked this session:**
1. **Teacher = k2-fsa/OmniVoice** (0.6B on Qwen3-0.6B, 600+ langs incl RU, zero-shot
   clone + voice-design, RTF 0.025, 24kHz out). Code Apache-2.0; **weights CC-BY-NC**
   — fine, the teacher never ships, only the piper student deploys.
2. **Student = piper**, trained **from scratch** (no warm-start from irina or any
   piper voice — GPU/patience is free, and warm-start's only benefit was fast
   convergence we don't need).
3. **Phoneme correctness is the priority.** Stress-mark + homograph-resolve every
   transcript so espeak-ng produces correct RU phonemes, and synth the audio from
   the **same** stressed text so text↔audio↔phonemes stay locked.

---

## 1. Pipeline overview

```
Phase 2  transcripts     →  stress-marked + homograph-clean mood lists
Phase 3  teacher swap     →  OmniVoice into find_voice.py / generate_synthetic_voice.py
Phase 4  generate         →  dataset/<mood>/NNNN.wav + metadata.csv (24kHz)
Phase 5  build_dataset    →  piper LJSpeech dir: wav 22.05k + metadata (id|stressed_text)
Phase 6  check_phonemes   →  espeak-ng sanity — eyeball stress before training
Phase 7  train.sh         →  piper preprocess → train from scratch → export ONNX
Phase 8  deploy           →  scp .onnx+json to homesrv; point mavend.json; drop irina
```

**Run order = build order here** (unlike the CPT plan). Phase 2 (phonemes) gates
quality and is CPU work — do it while the LLM CPT still owns the GPU.

---

## 2. Transcripts — stress + homographs (the quality lever)

The `data/*-voice-dataset-list.txt` files are raw Russian, one sentence per line.
Two defects to fix **before** synthesis, because the audio must match the phonemes:

1. **Homographs** — words whose stress (thus meaning) is ambiguous (`за́мок`/`замо́к`,
   `сто́ят`/`стоя́т`). `homograph_processor.py` already rewrites these away via a local
   LLM. Run it per mood list → `*-clean.txt`.
2. **Stress marks** — `ruaccent` (`RUAccent().process_all`) inserts `+` before every
   stressed vowel. This is what makes espeak-ng stress correctly later. The gen script
   already has `preprocess()` (accent → `plus_to_acute`) but it's **commented out** —
   the decision is to run stress-marking on the transcripts and keep the `+`-marked
   form as the canonical text (see Phase 5 for the two acute conventions).

```bash
cd /home/kami/Programs/esp32-whisper-fine-tune/tts
# 1. de-homograph each mood list (LLM at localhost:10000 must be up)
python homograph_processor.py data/*-voice-dataset-list.txt --all
# 2. stress-mark → keep the +VOWEL form as canonical (Phase 5 converts per target)
```

**DONE-CHECK 2:** each mood has a cleaned, `+`-stress-marked list; a 30-line random
sample eyeballed — every multisyllable word has exactly one `+`, no homographs left.

---

## 3. Teacher swap — OmniVoice into the two gen scripts

`find_voice.py` and `generate_synthetic_voice.py` currently call `qwen_tts.Qwen3TTSModel`
(`generate_voice_design`, `generate_voice_clone`). Swap the teacher to OmniVoice via the
thin adapter `tts/piper/omnivoice_tts.py` (created by this plan) so only two call-sites
change and the rest of the resume/metadata logic is untouched.

- `pip install omnivoice` (after torch), on the workstation, **GPU free** (post-CPT).
- **Verify the exact OmniVoice API** from its README / `pip show omnivoice` — the adapter
  is written against the documented surface (ref-audio + transcription clone; attribute
  voice-design) but the method names/signature **must be confirmed** before running.
- Keep synthesis text = the **stressed** transcripts from Phase 2 (re-enable `preprocess`,
  or pass the pre-stressed lists). Teacher and student then share identical text.

**DONE-CHECK 3:** `find_voice.py` produces an OmniVoice sample that plays; one clone
sample from `generate_synthetic_voice.py --dry-run`-then-real matches the reference timbre.

---

## 4. Generate the dataset

```bash
cd /home/kami/Programs/esp32-whisper-fine-tune/tts
python find_voice.py                         # audition → ref/<N>/maven_reference.{wav,txt}
python generate_synthetic_voice.py           # all moods → dataset/<mood>/NNNN.wav + metadata.csv
#   resume-safe (.progress.json); --moods / --limit / --dry-run available
```

**Reuse-or-regenerate call:** the existing `dataset/` was made by Qwen3-TTS from
**unstressed** text. For phoneme-locked training you want OmniVoice audio from
**stressed** text → **regenerate**. (Keeping the old set is only OK if you accept the
text↔audio stress mismatch, which is the exact defect this plan exists to kill.)

**DONE-CHECK 4:** `dataset/metadata.csv` rows all point to existing wavs; total
duration logged (aim ≥ ~2–3h across moods for a from-scratch single-speaker voice);
5 random wavs play and match their transcript.

---

## 5. Build the piper dataset (`tts/piper/build_dataset.py`)

Piper wants an **LJSpeech-format** dir: `wav/<id>.wav` (its target sample rate) +
`metadata.csv` as `id|text`. Our `metadata.csv` is `file_path,text,mood,emotion_id` and
wavs are 24kHz. `build_dataset.py`:

1. Read `dataset/metadata.csv`.
2. **Resample 24000 → 22050 Hz** (piper `medium` default), mono, into `piper/dataset/wav/`.
3. **Stress → espeak convention:** piper phonemizes via espeak-ng, which reads the
   **combining acute U+0301 *after* the stressed vowel** (NOT the uppercase form
   `plus_to_acute` makes for qwen). Convert the `+VOWEL` marks to `vowel+U+0301`.
4. Write `piper/dataset/metadata.csv` as `id|stressed_text` (single speaker).

Self-check (`__main__`): assert `+а` → `а́`, id/text counts match wav count.

**DONE-CHECK 5:** `piper/dataset/` has N wavs at 22050Hz and a metadata line each;
`soxi`/`soundfile` confirms sample rate.

---

## 6. Phoneme sanity (`tts/piper/check_phonemes.py`)

Before burning GPU-days, confirm espeak-ng stresses correctly. For a sample of
transcripts, print `espeak-ng -v ru --ipa` output and flag any multisyllable word
whose IPA carries no primary-stress mark `ˈ`. This is the cheapest catch for the
"sounds bad" failure — wrong stress shows here, not after training.

Requires `espeak-ng` (`sudo apt install espeak-ng`). **ponytail:** thin wrapper over
the espeak CLI, ceiling = it only flags *missing* stress, not *wrong-position* stress
(that still needs an ear on Phase 4 audio).

**DONE-CHECK 6:** sample run shows every content word carrying `ˈ`; obvious homograph
words stressed as intended.

---

## 7. Train piper from scratch (`tts/piper/train.sh`)

Install piper-train from the piper repo (`rhasspy/piper`, `src/python`): needs torch +
pytorch-lightning. ROCm env like `train_cpt.py` (`HSA_OVERRIDE_GFX_VERSION=11.0.0`).

```bash
# preprocess: text → espeak-ng phonemes → training cache
python -m piper_train.preprocess \
  --language ru --input-dir piper/dataset --output-dir piper/train \
  --dataset-format ljspeech --single-speaker --sample-rate 22050

# train FROM SCRATCH (no --resume_from_checkpoint), medium quality
HSA_OVERRIDE_GFX_VERSION=11.0.0 python -m piper_train \
  --dataset-dir piper/train --accelerator gpu --devices 1 \
  --batch-size 16 --quality medium --precision 32 \
  --max_epochs 4000 --checkpoint-epochs 100 --validation-split 0.02

# export best checkpoint → ONNX
python -m piper_train.export_onnx piper/train/lightning_logs/version_0/checkpoints/last.ckpt \
  piper/maven.onnx
cp piper/train/config.json piper/maven.onnx.json
```

- **From scratch** = no warm-start (decision 2). Early checkpoints sound broken until
  MAS alignment settles — expected; that's the patience cost we accepted.
- `--quality medium` (22.05k) is the CPU-real-time sweet spot; `high` only if homesrv
  latency allows (measure).

**DONE-CHECK 7:** training loss/mel decreases; a mid-run checkpoint synthesizes an
intelligible Russian sentence with **correct stress**; final ONNX exported + its `.json`.

---

## 8. Deploy to homesrv

```bash
scp piper/maven.onnx piper/maven.onnx.json kami@192.168.1.104:<voices dir>
# point deploy/mavend.json at maven.onnx; drop the irina floor
```

Measure CPU tok→audio latency with `--mlock`-class care (homesrv swaps). If too slow,
that's a quality/latency knob (medium already chosen), not a re-train.

**DONE-CHECK 8:** Maven speaks on homesrv in the new voice, real-time enough for
conversation, correct Russian stress in live output.

---

## 9. File manifest (what this plan creates in `tts/piper/`)

| File | Phase | Purpose |
|---|---|---|
| `omnivoice_tts.py` | 3 | thin adapter isolating the OmniVoice API (design + clone) — the single swap point |
| `build_dataset.py` | 5 | our `metadata.csv` + 24k wav → piper LJSpeech dir (22.05k, espeak-acute stress) |
| `check_phonemes.py` | 6 | espeak-ng stress sanity on a transcript sample |
| `train.sh` | 7 | piper preprocess → train-from-scratch → export ONNX (runbook) |

Reused, unchanged: `find_voice.py`, `generate_synthetic_voice.py` (two call-sites swapped
to the adapter), `homograph_processor.py`, `ruaccent`.

---

## 10. Hard rules (do not violate)

1. **Text↔audio↔phonemes locked:** synth the audio from the **same stressed text**
   espeak-ng later phonemizes. Never train piper on audio whose transcript stress
   differs from what you feed the preprocessor.
2. **Two acute conventions — don't confuse them:** `plus_to_acute` (uppercase vowel)
   is for the **qwen/OmniVoice** teacher; espeak-ng wants **combining U+0301 after the
   vowel**. `build_dataset.py` converts.
3. **From scratch** — no warm-start (decision 2). Do not `--resume_from_checkpoint`
   off irina or any piper voice.
4. **Teacher weights are CC-BY-NC** — fine for the never-shipped teacher; the deployed
   piper voice is your own weights. Don't redistribute the OmniVoice-generated dataset
   as a product.
5. **Homographs out first** (Phase 2), then stress-mark. Order matters — rewriting a
   sentence changes which words need stress.
6. Don't advance a phase until its DONE-CHECK passes. Phase 6 (phoneme sanity) is the
   real gate — a bad phoneme table wastes the whole GPU-days train.
```