model-training/QWEN.md

ESP32 Whisper Fine-Tune Project

Project Overview

This is a Russian-language AI model fine-tuning pipeline designed for eventual deployment on an ESP32 device. The project fine-tunes three model families using LoRA (Low-Rank Adaptation):

1. Whisper — Speech-to-Text (STT) models (openai/whisper-small, openai/whisper-medium) fine-tuned on the Common Voice 17.0 Russian dataset.
2. LLM — Llama 3.2 models fine-tuned for conversational Russian dialogue with a custom persona ("Maven/Мейвен") — a lively, fast-paced conversational character.
3. TTS — Text-to-Speech pipeline using Qwen3-TTS VoiceDesign for synthetic voice generation, with tools for voice discovery and dataset QA.

The project also includes helper utilities for data cleaning and a web-based QA server for reviewing TTS audio datasets.

Directory Structure

Directory/File	Purpose
whisper/	Whisper STT fine-tuning scripts and saved model checkpoints
whisper/main.py	Trains whisper-small with LoRA on Common Voice 17.0 RU
whisper/train.py	Trains whisper-medium with LoRA; custom WhisperLoraTrainer class
whisper/check_lora_weights.py	Inspects LoRA adapter weights
whisper/eval_test.py	Evaluation/testing script for Whisper models
whisper/sanity_test.py	Quick sanity check for model loading
whisper/whisper_lora/	Output directory for whisper-small LoRA checkpoints
whisper/whisper-medium-ru-lora/	Output directory for whisper-medium LoRA checkpoints
whisper/whisper-small-ru-lora/	Output directory for whisper-small LoRA checkpoints
whisper/whisper-small/	Base whisper-small model artifacts
llm/	LLM fine-tuning scripts for Llama 3.2 and Qwen3
llm/train_llama.py	Fine-tunes Llama 3.2 3B with LoRA; supports multiple dataset sources (JSONL, Parquet, HF datasets); custom conversational persona
llm/llama-eval-test.py	Interactive chat/evaluation script for trained Llama LoRA checkpoints
llm/Llama-3.2-3b-ru-lora/	Llama 3.2 3B LoRA checkpoint output
llm/Llama-3.2-4b-ru-lora/	Llama 3.2 4B LoRA checkpoint output
llm/qwen3-4b-2507-ru-lora/	Qwen3 4B LoRA checkpoint output
llm/data/	Training data (JSONL files, etc.)
tts/	Text-to-Speech voice generation and dataset tools
tts/find_voice.py	Interactive voice discovery tool using Qwen3-TTS VoiceDesign; generates candidate voices and lets you pick one
tts/generate_synthetic_voice.py	Synthetic voice generation script
tts/generate_synthetic_styles.py	Generates diverse voice styles
tts/homograph_processor.py	Homograph processing for TTS text normalization
tts/data/	TTS dataset files
tts/dataset/	TTS dataset organization
tts/ref/	Reference voice audio samples (indexed subdirectories)
ui/tts-qa-srv.py	Web-based QA server for reviewing TTS audio datasets; FastAPI + Vue-style SPA with keep/delete workflow
helpers/jsonl_cleaner.py	CLI tool to clean JSONL files of markdown formatting and emojis
clean_tmpfs.sh	Shell script to clean tmpfs, /dev/shm, and Hugging Face cache artifacts
requirements.txt	Python dependencies

Key Technologies

• PyTorch — Deep learning framework
• Hugging Face Transformers — Model loading, training, and inference
• PEFT (LoRA) — Parameter-Efficient Fine-Tuning
• Datasets — Hugging Face Datasets library for data loading and preprocessing
• bitsandbytes — 4-bit quantization for efficient training
• Common Voice 17.0 — Speech dataset (Russian subset)
• Qwen3-TTS — Text-to-Speech voice generation
• FastAPI + Uvicorn — Web server for TTS QA interface
• jiwer, librosa, soundfile — Audio metrics and processing

Building and Running

Prerequisites

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Whisper Fine-Tuning

# Train whisper-small with LoRA
cd whisper
python main.py

# Train whisper-medium with LoRA
cd whisper
python train.py

# Evaluate a trained model
python eval_test.py

# Check LoRA weights
python check_lora_weights.py

LLM Fine-Tuning

# Train Llama 3.2 with LoRA
cd llm
python train_llama.py

# Interactive evaluation
python llama-eval-test.py

TTS Voice Discovery

cd tts
python find_voice.py
# Interactively generates candidate voices and lets you pick one
# Saves selected voice to ref/<index>/maven_reference.wav

TTS Dataset QA Server

cd ui
python tts-qa-srv.py --wavs dataset/test --txt data/neutral-voice-dataset-list-clean.txt --port 8765
# Open http://localhost:8765 in browser
# Keyboard shortcuts: k=keep, d=delete, ←/←=navigate, space=play/pause

Data Cleaning

# Clean JSONL files of markdown and emojis
python helpers/jsonl_cleaner.py input.jsonl output.jsonl

Cache Cleanup

./clean_tmpfs.sh

Configuration Notes

• Cache paths are configured to use /mnt/D/.cache/huggingface and /mnt/D/tmp in whisper/train.py and llm/train_llama.py. Adjust these paths for your environment.
• Dataset: Whisper models use fsicoli/common_voice_17_0 (Russian subset).
• LLM training data: JSONL files in llm/data/cleaned.jsonl, plus optional Parquet and HF dataset sources.
• Quantization: LLM training uses 4-bit quantization via bitsandbytes (nf4, double quantization, bfloat16 compute).

Training Hyperparameters

Whisper

Parameter	small	medium
LoRA r	8	32
LoRA alpha	32	64
LoRA dropout	0.1	0.05
Learning rate	1e-4	5e-5
Max steps	4000	4000
Warmup steps	100	500
Batch size (train/eval)	2/2	2/2

LLM (Llama 3.2 3B)

Parameter	Value
LoRA r	16
LoRA alpha	32
LoRA dropout	0.1
Learning rate	1e-4
Epochs	3
Gradient accumulation	8
Max length	1024
Optim	paged_adamw_8bit

Development Conventions

• All models use LoRA for parameter-efficient fine-tuning (only adapter weights are saved).
• Russian language (ru) is the primary target for all models.
• The LLM is trained with a custom persona ("Maven/Мейвен") — a conversational, energetic character. The system prompt enforces strict JSON output format: {"response":"...","mood":"..."}.
• TTS voice discovery uses Qwen3-TTS VoiceDesign to generate synthetic voices interactively.
• The QA server uses a keep/delete workflow for curating TTS datasets, with state persistence to JSON.
• Dataset preprocessing includes casting audio to 16kHz, removing unnecessary columns, and mapping text through the processor.
• Progress is logged to TensorBoard for all training runs.