Maven/docs/plans/2026-07-18-qwen3-resident-training-eval.md

Plan — Qwen3-1.7B resident model: audit, evaluation, SFT and gate

Canonical model plan as of 2026-07-18. Qwen3-1.7B is the one resident router/phraser. Larger reasoners and custom Piper training are deferred until the main Maven features are complete.

Locked architecture

Qwen3-1.7B-Base
  → full-weight Russian CPT
  → raw-vs-CPT decision gate
  → one balanced persona + route LoRA
  → contract evaluation
  → merge → GGUF Q8_0 → homesrv

The route system prompt selects a JSON-array action contract. The persona system prompt selects {response,mood}. Metrics are reported independently so joint-training interference is visible. If one task regresses, use separate adapters before considering a merged multi-task checkpoint.

Current evidence

• Packed corpus: 129,221 × 2,048 = 264,644,608 tokens.
• Materialized character mix: CulturaX 69.407%, Wikipedia 17.991%, books 12.602%. Planned synthetic/log/conversational buckets are absent from this run.
• All 64 sampled Arrow rows had the right length and valid token IDs.
• The audit found 247 CulturaX documents with glued <phone> placeholders and Cyrillic text. This run continues; fix placeholder spacing before any rebuild.
• Full-weight Adafactor training reached checkpoint 1,000/8,077 (12.38%). Loss and gradient norms are finite, so this checkpoint is the full-weight smoke test. Do not launch a competing smoke job.

Machine-readable audit: /home/kami/Programs/esp32-whisper-fine-tune/llm/CPT_CORPUS_AUDIT.json.

Tools

All commands run from /home/kami/Programs/esp32-whisper-fine-tune using its .venv.

# Reproduce the read-only corpus/checkpoint audit.
./.venv/bin/python llm/audit_cpt_corpus.py \
  --output llm/CPT_CORPUS_AUDIT.json

# Fetch pinned, human-written UD RU/EN test sets and their hashes.
./.venv/bin/python llm/prepare_eval_data.py

# Prompt parity and held-out route labels.
./.venv/bin/python llm/check_prompt_parity.py
./.venv/bin/python llm/build_route_eval.py

Baseline and post-CPT evaluation

The evaluation is deterministic and uses token-weighted perplexity. Preserve the raw result; do not regenerate it with different limits when comparing CPT.

./.venv/bin/python llm/eval_cyrillic.py \
  --model Qwen/Qwen3-1.7B-Base --limit 128 --device cuda \
  --output llm/data/eval/raw_qwen3_1.7b.json

# After CPT finishes:
./.venv/bin/python llm/eval_cyrillic.py \
  --model llm/Qwen3-1.7B-ru-cpt --limit 128 --device cuda \
  --output llm/data/eval/cpt_qwen3_1.7b.json

For a publication-quality comparison, repeat both with --limit 0; the quick 128-sentence pair is the operational gate and must use identical arguments.

Exact CPT decision gate

./.venv/bin/python llm/decision_gate.py \
  --raw llm/data/eval/raw_qwen3_1.7b.json \
  --cpt llm/data/eval/cpt_qwen3_1.7b.json

Exit 0 means all conditions passed:

• RU perplexity improves by at least 2%;
• EN perplexity regresses by no more than 10%;
• RU generation validity does not decline;
• deterministic English probes remain English.
• mean repeated 4-gram rate stays below 15%, or within five percentage points of the raw baseline when the baseline itself is worse.

Exit 1 blocks SFT. Inspect the JSON check map before changing a threshold. A threshold change is a documented architecture decision, not a convenient rerun.

Checkpoint 1,000 is an informative interim result: RU PPL improved 4.82% and EN PPL regressed only 2.20%, but repeated 4-grams rose from 8.56% to 25.90%. Therefore it passes the language-loss checks but fails the complete gate. This is not a stop signal at 12.38% of training; it is a regression to watch at the final checkpoint and a reason the gate includes generation degeneration.

Resume CPT

The latest complete checkpoint is selected by Transformers:

cd /home/kami/Programs/esp32-whisper-fine-tune/llm
HSA_OVERRIDE_GFX_VERSION=11.0.0 ../.venv/bin/python train_cpt.py --resume \
  2>&1 | tee -a cpt_run.log

Confirm there is only one train_cpt.py process before resuming. Completion is step 8,077 and must produce the final tokenizer/model files at llm/Qwen3-1.7B-ru-cpt/, not only checkpoint directories.

Joint Qwen3 SFT

Generate route training data first. route_train.jsonl is currently a blocker; the held-out route_eval.jsonl already exists and must never be merged into it.

cd /home/kami/Programs/esp32-whisper-fine-tune
./.venv/bin/python llm/gen_route_data.py
./.venv/bin/python llm/train_rocm.py --check-data
./.venv/bin/python llm/train_rocm.py

The rewritten trainer:

• loads the completed CPT base;
• consumes explicit persona and route train/eval files;
• balances the two tasks by oversampling only within the training set;
• renders Qwen3's own chat template with thinking disabled;
• masks loss before the assistant continuation without hard-coded ChatML IDs;
• supports deterministic seeds, resume, early stopping and best-checkpoint load;
• writes a training manifest.

After training:

./.venv/bin/python llm/eval_contracts.py \
  --base llm/Qwen3-1.7B-ru-cpt \
  --adapter llm/Qwen3-1.7B-maven-sft \
  --output llm/data/eval/qwen3_contracts.json

Initial deploy gates: persona JSON validity ≥99%, route JSON validity ≥99%, route intent sequence exact ≥90%, action intent accuracy ≥95%, slot exact ≥85%. Mood accuracy is diagnostic until the persona evaluation set is manually quality-reviewed; the existing examples contain stale technical answers.

Deferred

• Piper/phoneme training.
• On-demand larger reasoner.
• Model hot-swap and the broad capability plans.

These resume after the main Maven feature set is complete.