# ESP32 Whisper Fine-Tune Project

## Project Overview

This is a **Russian-language AI model fine-tuning pipeline** designed for eventual deployment on an ESP32 device. The project fine-tunes three model families using **LoRA (Low-Rank Adaptation)**:

1. **Whisper** — Speech-to-Text (STT) models (`openai/whisper-small`, `openai/whisper-medium`) fine-tuned on the Common Voice 17.0 Russian dataset.
2. **LLM** — Llama 3.2 models fine-tuned for conversational Russian dialogue with a custom persona ("Maven/Мейвен") — a lively, fast-paced conversational character.
3. **TTS** — Text-to-Speech pipeline using Qwen3-TTS VoiceDesign for synthetic voice generation, with tools for voice discovery and dataset QA.

The project also includes helper utilities for data cleaning and a web-based QA server for reviewing TTS audio datasets.

## Directory Structure

| Directory/File | Purpose |
|---|---|
| `whisper/` | Whisper STT fine-tuning scripts and saved model checkpoints |
| `whisper/main.py` | Trains `whisper-small` with LoRA on Common Voice 17.0 RU |
| `whisper/train.py` | Trains `whisper-medium` with LoRA; custom `WhisperLoraTrainer` class |
| `whisper/check_lora_weights.py` | Inspects LoRA adapter weights |
| `whisper/eval_test.py` | Evaluation/testing script for Whisper models |
| `whisper/sanity_test.py` | Quick sanity check for model loading |
| `whisper/whisper_lora/` | Output directory for whisper-small LoRA checkpoints |
| `whisper/whisper-medium-ru-lora/` | Output directory for whisper-medium LoRA checkpoints |
| `whisper/whisper-small-ru-lora/` | Output directory for whisper-small LoRA checkpoints |
| `whisper/whisper-small/` | Base whisper-small model artifacts |
| `llm/` | LLM fine-tuning scripts for Llama 3.2 and Qwen3 |
| `llm/train_llama.py` | Fine-tunes Llama 3.2 3B with LoRA; supports multiple dataset sources (JSONL, Parquet, HF datasets); custom conversational persona |
| `llm/llama-eval-test.py` | Interactive chat/evaluation script for trained Llama LoRA checkpoints |
| `llm/Llama-3.2-3b-ru-lora/` | Llama 3.2 3B LoRA checkpoint output |
| `llm/Llama-3.2-4b-ru-lora/` | Llama 3.2 4B LoRA checkpoint output |
| `llm/qwen3-4b-2507-ru-lora/` | Qwen3 4B LoRA checkpoint output |
| `llm/data/` | Training data (JSONL files, etc.) |
| `tts/` | Text-to-Speech voice generation and dataset tools |
| `tts/find_voice.py` | Interactive voice discovery tool using Qwen3-TTS VoiceDesign; generates candidate voices and lets you pick one |
| `tts/generate_synthetic_voice.py` | Synthetic voice generation script |
| `tts/generate_synthetic_styles.py` | Generates diverse voice styles |
| `tts/homograph_processor.py` | Homograph processing for TTS text normalization |
| `tts/data/` | TTS dataset files |
| `tts/dataset/` | TTS dataset organization |
| `tts/ref/` | Reference voice audio samples (indexed subdirectories) |
| `ui/tts-qa-srv.py` | Web-based QA server for reviewing TTS audio datasets; FastAPI + Vue-style SPA with keep/delete workflow |
| `helpers/jsonl_cleaner.py` | CLI tool to clean JSONL files of markdown formatting and emojis |
| `clean_tmpfs.sh` | Shell script to clean tmpfs, /dev/shm, and Hugging Face cache artifacts |
| `requirements.txt` | Python dependencies |

## Key Technologies

- **PyTorch** — Deep learning framework
- **Hugging Face Transformers** — Model loading, training, and inference
- **PEFT (LoRA)** — Parameter-Efficient Fine-Tuning
- **Datasets** — Hugging Face Datasets library for data loading and preprocessing
- **bitsandbytes** — 4-bit quantization for efficient training
- **Common Voice 17.0** — Speech dataset (Russian subset)
- **Qwen3-TTS** — Text-to-Speech voice generation
- **FastAPI + Uvicorn** — Web server for TTS QA interface
- **jiwer, librosa, soundfile** — Audio metrics and processing

## Building and Running

### Prerequisites

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

### Whisper Fine-Tuning

```bash
# Train whisper-small with LoRA
cd whisper
python main.py

# Train whisper-medium with LoRA
cd whisper
python train.py

# Evaluate a trained model
python eval_test.py

# Check LoRA weights
python check_lora_weights.py
```

### LLM Fine-Tuning

```bash
# Train Llama 3.2 with LoRA
cd llm
python train_llama.py

# Interactive evaluation
python llama-eval-test.py
```

### TTS Voice Discovery

```bash
cd tts
python find_voice.py
# Interactively generates candidate voices and lets you pick one
# Saves selected voice to ref/<index>/maven_reference.wav
```

### TTS Dataset QA Server

```bash
cd ui
python tts-qa-srv.py --wavs dataset/test --txt data/neutral-voice-dataset-list-clean.txt --port 8765
# Open http://localhost:8765 in browser
# Keyboard shortcuts: k=keep, d=delete, ←/←=navigate, space=play/pause
```

### Data Cleaning

```bash
# Clean JSONL files of markdown and emojis
python helpers/jsonl_cleaner.py input.jsonl output.jsonl
```

### Cache Cleanup

```bash
./clean_tmpfs.sh
```

## Configuration Notes

- **Cache paths** are configured to use `/mnt/D/.cache/huggingface` and `/mnt/D/tmp` in `whisper/train.py` and `llm/train_llama.py`. Adjust these paths for your environment.
- **Dataset**: Whisper models use `fsicoli/common_voice_17_0` (Russian subset).
- **LLM training data**: JSONL files in `llm/data/cleaned.jsonl`, plus optional Parquet and HF dataset sources.
- **Quantization**: LLM training uses 4-bit quantization via bitsandbytes (`nf4`, double quantization, bfloat16 compute).

## Training Hyperparameters

### Whisper
| Parameter | small | medium |
|---|---|---|
| LoRA r | 8 | 32 |
| LoRA alpha | 32 | 64 |
| LoRA dropout | 0.1 | 0.05 |
| Learning rate | 1e-4 | 5e-5 |
| Max steps | 4000 | 4000 |
| Warmup steps | 100 | 500 |
| Batch size (train/eval) | 2/2 | 2/2 |

### LLM (Llama 3.2 3B)
| Parameter | Value |
|---|---|
| LoRA r | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.1 |
| Learning rate | 1e-4 |
| Epochs | 3 |
| Gradient accumulation | 8 |
| Max length | 1024 |
| Optim | paged_adamw_8bit |

## Development Conventions

- All models use **LoRA** for parameter-efficient fine-tuning (only adapter weights are saved).
- Russian language (`ru`) is the primary target for all models.
- The LLM is trained with a custom **persona** ("Maven/Мейвен") — a conversational, energetic character. The system prompt enforces strict JSON output format: `{"response":"...","mood":"..."}`.
- TTS voice discovery uses **Qwen3-TTS VoiceDesign** to generate synthetic voices interactively.
- The QA server uses a **keep/delete workflow** for curating TTS datasets, with state persistence to JSON.
- Dataset preprocessing includes casting audio to 16kHz, removing unnecessary columns, and mapping text through the processor.
- Progress is logged to **TensorBoard** for all training runs.