Maven/docs/plans/2026-07-11-ru-cpt-base.md

# Plan — DIY RU-native base via Continued Pretraining (CPT)

> **Execution update, 2026-07-18:** corpus packing and full-weight smoke are
> complete; training reached step 1000/8077. The materialized corpus is 264.6M
> tokens from CulturaX/Wikipedia/books only. Evaluation, the exact decision gate
> and Qwen3 joint SFT are now canonical in
> `2026-07-18-qwen3-resident-training-eval.md`.

> **Goal:** Build our *own* RU-native base model instead of using Vikhr. Take a
> clean newer base (Qwen3-1.7B-Base), continue-pretrain it on a curated Russian
> corpus so it spells Cyrillic natively, then run the existing persona-LoRA →
> merge → GGUF → deploy pipeline on top. We own the recipe, the corpus, and the
> checkpoint — reproducible, no dependency on a third-party RU repo.
>
> **This is a big project. The corpus is 80% of the work.** Read the whole file
> before starting. Do phases in order. Each phase has a **DONE-CHECK** — do not
> advance until it passes.

---

## 0. Context an agent picking this up must know

**Two machines — never conflate (from `CLAUDE.md`):**

| Role | Box | Specs | Constraint |
|---|---|---|---|
| **Train** | this workstation | RX 7900 GRE, gfx1100 (RDNA3), **16GB VRAM**, ROCm 7.2 / torch 2.10 HIP, `cuda.is_available()==True` | CPT of 1.7B is *tight* — see Phase 3 VRAM math |
| **Deploy** | homesrv `kami@192.168.1.104` | Ryzen 5 5600U CPU, **13GB RAM already swapping** (~8.8GB avail), Ubuntu 24.04 | CPU-only llama.cpp inference; model must stay ≤1.7B and every 200MB matters |

**Working directories:**
- Training scripts & data: `/home/kami/Programs/esp32-whisper-fine-tune/llm/`
  - existing: `train_rocm.py` (persona LoRA, ChatML), `llama-eval-test.py`, `data/`
  - **new files from this plan** also go here.
- Daemon (consumes final GGUF): `/mnt/server/home/kami/apps/Maven/` (this repo).

**Output contract (from `CLAUDE.md`, Decision B, locked 2026-07-11):** the daemon
parses `{"response":"...","mood":"..."}` in all 4 consumers. This only matters for
the **persona LoRA stage (Phase 6)**, NOT for CPT. CPT is raw next-token text, no
JSON, no chat format.

**Why CPT and not just a LoRA (from `CLAUDE.md`):** a LoRA cannot add spelling
knowledge the base lacks — it only steers format/persona/mood/tools. CPT changes
the base weights themselves, which is the only lever that can add genuine RU
language competence. That is exactly why this is more than a LoRA project.

**ROCm gotchas (inherit from `train_rocm.py`):**
- `attn_implementation="eager"` (flash-attn ROCm wheels flaky).
- `HSA_OVERRIDE_GFX_VERSION=11.0.0` in env (gfx1100).
- **No bitsandbytes** — ROCm build is flaky. This forbids 8-bit Adam; see Phase 3.

---

## 1. Pipeline overview (the whole journey)

```
Phase 2  Corpus build        →  clean packed RU token blocks (.arrow/.bin, ~300M tok)
Phase 3  train_cpt.py        →  CPT'd Qwen3-1.7B-Base (bf16 HF checkpoint)
Phase 4  eval                →  Cyrillic + JSON validator; CPT base vs raw base numbers
Phase 5  DECISION GATE       →  did CPT beat raw? if not, stop / re-corpus
Phase 6  persona LoRA        →  existing train_rocm.py on the CPT'd base (ChatML, {response,mood})
Phase 7  merge → GGUF → Q8_0 →  deploy to homesrv
```

Build order is **not** the run order: **build the corpus first (Phase 2)** because
it gates everything and takes the longest.

---

## 2. Corpus — the core deliverable

**Target: ~300M tokens** (Qwen3 tokenizer), curated. Rationale: <100M won't move
the needle vs Qwen3's already-strong multilingual; >1B is weeks of wall-clock on
one 16GB card. 300M is the "nudge, don't re-pretrain" sweet spot for 1 epoch.

**Two token sources, combined:**
- **A. Open datasets** (the bulk, real human RU text).
- **B. Router-generated synthetic** (fills domain/persona/conversational gaps the
  open corpora lack — smart-home commands, assistant dialogue, Maven's domain).
  The user's **router for free inference providers** generates this.

### 2.1 Corpus mix table (sums to ~300M tokens)

| # | Bucket | Share | Tokens | Source | Why |
|---|---|---|---|---|---|
| B1 | Cleaned web RU | 55% | ~165M | `uonlp/CulturaX` (ru) — **already cleaned + deduped** | backbone, broad RU |
| B2 | Encyclopedic | 15% | ~45M | `wikimedia/wikipedia` config `20231101.ru` | clean factual RU, correct spelling |
| B3 | Books / long-form | 10% | ~30M | `IlyaGusev/librusec` (or subset) | rich morphology, endings |
| B4 | Conversational | 8% | ~24M | `Helsinki-NLP/open_subtitles` (ru) | spoken register — matches voice assistant |
| B5 | **Router-synthetic** | 10% | ~30M | generated (§2.4) | domain/persona/smart-home RU, absent from open sets |
| B6 | Maven real logs | 2% | ~6M | mine daemon dialogue history / logs | exact deploy-domain match |

**Cap B5 (synthetic) at 10%.** Synthetic-heavy CPT causes model collapse /
degeneration. The bulk MUST be real human text. This cap is a hard rule.

### 2.2 Exact acquisition — open datasets (Bucket A: B1–B4)

All via HuggingFace `datasets`, streaming to avoid disk blowup. Some are gated
(CulturaX, OSCAR) — accept the license on the HF model page while logged in
(`huggingface-cli login`) first.

```python
# corpus_fetch.py  — run per bucket, writes raw .txt.gz shards to data/cpt_raw/<bucket>/
from datasets import load_dataset
# B1 CulturaX (gated — accept license first)
ds = load_dataset("uonlp/CulturaX", "ru", split="train", streaming=True)
# B2 Wikipedia
ds = load_dataset("wikimedia/wikipedia", "20231101.ru", split="train", streaming=True)
# B3 Books
ds = load_dataset("IlyaGusev/librusec", split="train", streaming=True)
# B4 Subtitles
ds = load_dataset("Helsinki-NLP/open_subtitles", lang1="en", lang2="ru", split="train", streaming=True)
```

Pull enough raw text per bucket to survive filtering (filtering drops ~20–40%);
over-pull ~1.5× the target token count, then trim after cleaning. Text field
differs per dataset (`text`, `content`, `translation['ru']`) — normalize each to a
single `text` string when writing shards.

**Fallbacks if a source is unavailable:** B1 → `allenai/c4` (ru) or `oscar-corpus/OSCAR-2301` (ru);
B3 → `IlyaGusev/gazeta` / Taiga corpus; B4 → `Helsinki-NLP/tatoeba` (ru pairs).

### 2.3 Cleaning pipeline (mandatory — dirty corpus = dirty base)

One script `corpus_clean.py`, applied to every raw shard, in this order:

1. **Unicode fix** — `ftfy.fix_text` (repairs mojibake, normalizes to NFC).
2. **Language ID** — fastText `lid.176.bin`; drop any doc where top lang != `ru`
   or `p(ru) < 0.65`. (Download `lid.176.bin` from fastText releases.)
3. **Mixed-script rejection** — regex: reject any *word* containing both Cyrillic
   and Latin letters (`[а-яё].*[a-z]|[a-z].*[а-яё]` case-insensitive per word).
   This is the exact homoglyph problem `CLAUDE.md` names. Keep pure-Latin words
   (URLs/names) but drop homoglyph-contaminated ones.
4. **Boilerplate/length** — drop docs <200 chars or >50k chars; strip HTML tags,
   nav junk, repeated whitespace; drop docs with >30% non-letter chars.
5. **Dedup** — MinHash LSH via `datasketch` (`MinHashLSH`, threshold 0.8, 128
   perms) across ALL buckets combined. Near-dup docs collapse to one. This is the
   single biggest quality lever — do not skip.
6. **PII light-scrub** (optional) — regex phone/email → placeholder, since B6 has
   real logs.

Emit cleaned docs as JSONL `{"text": "..."}` in `data/cpt_clean/`.

**DONE-CHECK 2.3:** run `corpus_stats.py` → per-bucket doc count, char count, and
**a 100-doc random sample printed for eyeball review**. No mixed-script words in
the sample. If sample looks dirty, fix the filter before proceeding.

### 2.4 Router-synthetic generation (Bucket B5 — the user's router)

Purpose: generate ~30M tokens of clean RU text in registers open corpora lack —
smart-home/assistant domain, natural conversational RU, Maven-adjacent topics.
**This is CPT text (raw prose/dialogue), NOT the `{response,mood}` persona data**
(that's Phase 6, a different dataset).

Protocol (`corpus_synth.py`, calls the free-provider router):
1. **Topic seed list** — reuse `data/topics.txt` (already exists, ~46KB of topics)
   + add smart-home / daily-assistant / RU-domestic topics.
2. **Prompt template** (per topic), ask the router model to produce **long-form
   natural Russian prose or dialogue** on the topic — NOT JSON, NOT persona, NOT
   English. e.g. `"Напиши развёрнутый естественный текст на русском языке на тему: {topic}. 400–800 слов, живой разговорный стиль."`
3. **Diversify:** vary style knob (formal/casual/dialogue/monologue), vary length,
   rotate across the router's providers/models so it's not one model's fingerprint
   (fingerprint uniformity → collapse risk).
4. **Route every generation through the SAME cleaning pipeline (§2.3)** — language
   ID, mixed-script rejection, dedup against the real corpus too. Synthetic that
   fails Cyrillic cleanliness is exactly what we must not train on.
5. **Stop at ~30M tokens.** Enforce the 10% cap.

**DONE-CHECK 2.4:** synthetic passes the same stats + eyeball sample as 2.3; token
count ≤ 10% of total corpus; provider diversity logged.

### 2.5 Maven real logs (Bucket B6)

Mine daemon dialogue history / logs for real RU user+assistant utterances. Strip
to plain text, run through §2.3. Small (~6M tok) but highest domain value.
Source location: check `mavend` logs / any dialogue history store in the repo.

### 2.6 Tokenize + pack (final corpus artifact)

`corpus_pack.py`:
- Load Qwen3-1.7B-Base tokenizer (`transformers.AutoTokenizer`).
- Concatenate all cleaned JSONL, tokenize, append EOS between docs, **pack into
  fixed 2048-token blocks** (drop the ragged tail). No padding — packing is
  efficient for CPT.
- Save as a HF `datasets` Arrow dir `data/cpt_packed/` (columns: `input_ids`).
- Print final total token count — **must be ~300M ± 50M**. Adjust bucket pulls if off.

**DONE-CHECK 2.6:** `data/cpt_packed/` loads; total tokens logged; a decoded
random block reads as clean Russian.

---

## 3. `train_cpt.py` — continued pretraining (fits 16GB)

New file next to `train_rocm.py`. Inherits the ROCm setup from it.

### 3.1 VRAM reality (why the config is what it is)

Full-weight CPT of 1.7B with plain Adam ≈ 21–24GB → **does not fit 16GB**. We
can't use 8-bit Adam (no bitsandbytes on ROCm). Fitting strategy, in preference
order — **try A first, fall back to B**:

- **A. Full-weight + `adafactor`** — Adafactor has no momentum states, cutting the
  optimizer-state VRAM roughly in half. Full-weight bf16 + adafactor + gradient
  checkpointing + batch 1 × grad-accum should fit ~16GB. This is the real
  "change the base weights" path. **Preferred.**
- **B. High-rank DoRA on all linear layers** — if A OOMs or throughput is
  unbearable. Fits trivially. Weaker for language injection but still adapts more
  than a small LoRA. Use `r=64, α=128, target all linear`, `use_dora=True`.

### 3.2 Config

```python
# train_cpt.py essentials
import os; os.environ["HSA_OVERRIDE_GFX_VERSION"] = "11.0.0"
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          TrainingArguments, Trainer,
                          DataCollatorForLanguageModeling)

MODEL = "Qwen/Qwen3-1.7B-Base"          # BASE, not Instruct
model = AutoModelForCausalLM.from_pretrained(
    MODEL, torch_dtype="bfloat16", attn_implementation="eager")
model.gradient_checkpointing_enable()
model.config.use_cache = False

args = TrainingArguments(
    output_dir="./Qwen3-1.7B-ru-cpt",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,      # eff. batch 16 × 2048 tok
    num_train_epochs=1,                  # ONE pass — more risks forgetting
    learning_rate=1e-5,                  # LOW — nudging pretrained weights
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    optim="adafactor",                   # path A; no bitsandbytes
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=20,
    save_steps=500,                      # checkpoint often — this runs for DAYS
    save_total_limit=3,
    report_to="none",
)
collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)  # causal LM
```

- **lr 1e-5 is critical.** CPT lr must be ~10× lower than LoRA lr. High lr =
  catastrophic forgetting (model forgets English, reasoning, everything). If the
  eval (Phase 4) shows English/tool ability collapsed, lr was too high.
- **1 epoch.** Multiple epochs over 300M tokens overfits and forgets.
- **Resume/checkpoint is mandatory** — days-long job; support `--resume_from_checkpoint`.

### 3.3 Run

```bash
cd /home/kami/Programs/esp32-whisper-fine-tune/llm
HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py 2>&1 | tee cpt_run.log
```

**DONE-CHECK 3:** training completes 1 epoch without OOM; loss decreases and
plateaus (not NaN, not flat-from-step-0); checkpoint saved to
`./Qwen3-1.7B-ru-cpt/`. If OOM → switch to path B (DoRA).

---

## 4. Eval — did CPT actually help?

`eval_cyrillic.py` now writes deterministic machine-readable results using
pinned human-written RU and EN Universal Dependencies test sets.

**Metrics (run on held-out human-written text — NOT synthetic, NOT training data):**
1. **Cyrillic validity %** — generate on RU prompts; % of outputs with zero
   mixed-script words and no homoglyph swaps (reuse §2.3 rule 3 as the checker).
2. **Perplexity** on a held-out clean RU text set (lower = better RU fit).
3. **English-retention check** — a few EN prompts; confirm the model still answers
   in coherent English (catches catastrophic forgetting).
4. **JSON-format sanity** (light, for later) — can it produce `{response,mood}`
   when asked. (Real test is post-persona-LoRA.)

**Run it three ways and tabulate:**
| Model | Cyrillic valid % | RU PPL | EN retained? |
|---|---|---|---|
| raw Qwen3-1.7B-Base | (baseline) | | |
| our CPT'd base | (must beat raw) | | |

**DONE-CHECK 4:** table filled with real numbers.

---

## 5. DECISION GATE

- **CPT base beats raw on Cyrillic %/PPL AND English retained** → proceed to Phase 6.
- **No improvement** → CPT didn't earn its compute. Options: (a) more/better
  corpus (usually the fix — dirty or too-small corpus), (b) tune lr, (c) accept
  raw Qwen3-1.7B as the base and skip CPT. Do NOT proceed to persona LoRA on a
  CPT base that lost to raw.
- **English collapsed** → lr too high or too many epochs; re-run Phase 3 with lower lr.

---

## 6. Persona LoRA on the CPT'd base (existing pipeline)

Now the CPT base is just a better base. The rewritten `train_rocm.py` trains a
balanced joint persona/router adapter and uses Qwen3's own chat template. It
masks the rendered assistant continuation instead of assuming literal ChatML
boundary token IDs.
- The persona **data** is the `{response,mood}` corpus per `CLAUDE.md` Decision B
  (separate from the CPT corpus): 45% persona chit-chat, 15% graceful failure
  (`tired`/`confused`), 20% tool calls, 10% real utterances, 10% EN→EN. Mood
  quotas + validator per `CLAUDE.md`. **This dataset is still an open item** — see
  `CLAUDE.md` "Target data recipe" and `gen_data.py` open item.

**DONE-CHECK 6:** LoRA trains; eval (Phase 4) re-run on merged model shows
`{response,mood}` parses + Cyrillic clean + moods in enum.

---

## 7. Merge → GGUF → quantize → deploy

Per `CLAUDE.md` deploy path, with **Q8_0 first** (decision this session: start
high-precision to remove the Cyrillic-at-Q4 gamble, drop to Q5/Q4 only if homesrv
memory chokes):

```bash
# 1. merge LoRA → bf16
python -c "from peft import AutoPeftModelForCausalLM; import torch; \
m=AutoPeftModelForCausalLM.from_pretrained('./Qwen3-1.7B-ru-lora',torch_dtype=torch.bfloat16); \
m.merge_and_unload().save_pretrained('./Qwen3-maven-merged'); \
from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('./Qwen3-1.7B-ru-lora').save_pretrained('./Qwen3-maven-merged')"
# 2. HF → GGUF (in llama.cpp)
python convert_hf_to_gguf.py ./Qwen3-maven-merged --outfile qwen3-maven-f16.gguf --outtype f16
# 3. quantize — Q8_0 first (~1.6GB for 1.7B)
./llama-quantize qwen3-maven-f16.gguf Qwen3-Maven-1.7B-Q8_0.gguf Q8_0
# 4. deploy: scp to homesrv; measure tok/s + swap with --mlock.
#    If RAM chokes → re-quant Q5_K_M, then Q4_K_M. Point deploy/mavend.json
#    phraser.model_path at the file; small -ngl on Vulkan for a speedup.
```

**DONE-CHECK 7:** model runs on homesrv at acceptable voice tok/s (measure!),
Cyrillic clean in live output, `{response,mood}` parsed by the daemon.

---

## 8. File manifest (what this plan creates)

All in `/home/kami/Programs/esp32-whisper-fine-tune/llm/`:

| File | Phase | Purpose |
|---|---|---|
| `corpus_fetch.py` | 2.2 | pull open datasets → `data/cpt_raw/` |
| `corpus_synth.py` | 2.4 | router-generate synthetic RU → `data/cpt_raw/synth/` |
| `corpus_clean.py` | 2.3 | ftfy + langID + mixed-script + dedup → `data/cpt_clean/` |
| `corpus_stats.py` | 2.3 | per-bucket counts + sample dump (DONE-CHECKs) |
| `corpus_pack.py` | 2.6 | tokenize + pack 2048 blocks → `data/cpt_packed/` |
| `train_cpt.py` | 3 | continued pretraining, adafactor/DoRA |
| `eval_cyrillic.py` | 4 | Cyrillic % + PPL + EN-retention (extends `llama-eval-test.py`) |
| `data/cpt_packed/` | 2.6 | final corpus artifact (~300M tok) |
| `./Qwen3-1.7B-ru-cpt/` | 3 | CPT'd base checkpoint |

### 8.1 One-time setup

```bash
cd /home/kami/Programs/esp32-whisper-fine-tune/llm

# python deps (torch/transformers already present from train_rocm.py)
pip install datasets ftfy datasketch fasttext openai peft accelerate

# fastText language-id model → data/lid.176.bin (corpus_clean.py requires it)
wget -O data/lid.176.bin \
  https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin

# HF login — needed to accept the CulturaX (gated) license
huggingface-cli login
```

If `fasttext` wheels fail to build, `pip install fasttext-wheel` is the prebuilt
fallback. `lid.176.bin` can also live elsewhere via `export FASTTEXT_LID=/path`.

### 8.2 Runbook (full run order)

```bash
cd /home/kami/Programs/esp32-whisper-fine-tune/llm

# ── Phase 2: corpus ──────────────────────────────────────────────
python corpus_fetch.py all                              # 2.2 open datasets
ROUTER_BASE_URL="https://your-router/v1" \
  ROUTER_API_KEY="..." ROUTER_MODELS="m-a,m-b,m-c" \
  python corpus_synth.py                                # 2.4 synthetic (≤10%)
MAVEN_LOGS=/path/to/dialogue.jsonl python corpus_logs.py  # 2.5 optional
python corpus_clean.py                                  # 2.3 clean+dedup all
python corpus_stats.py                                  # DONE-CHECK 2.3/2.4
python corpus_pack.py                                   # 2.6 → data/cpt_packed/

# ── Phase 3: continued pretraining (days; --resume to continue) ──
HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py 2>&1 | tee cpt_run.log
#   OOM? → CPT_PATH=dora HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py

# ── Phase 4: eval + gate (Phase 5) ──────────────────────────────
python eval_cyrillic.py --model Qwen/Qwen3-1.7B-Base    # baseline
python eval_cyrillic.py --model ./Qwen3-1.7B-ru-cpt     # must beat baseline

# ── Phase 6-7: persona LoRA → merge → GGUF → Q8_0 → homesrv ──────
#   train_rocm.py pointed at ./Qwen3-1.7B-ru-cpt ; then §7 deploy block.
```

Third-party deps: `datasets`, `ftfy`, `datasketch`, `fasttext` (+ `lid.176.bin`),
`openai`, `peft`, `transformers`, `accelerate`.

---

## 9. Hard rules (do not violate)

1. CPT corpus is **raw text**, no JSON, no ChatML. Persona `{response,mood}` data
   is a **separate** dataset (Phase 6).
2. Synthetic ≤ **10%** of corpus. Bulk must be real human RU.
3. CPT lr = **1e-5**, **1 epoch**. Higher = catastrophic forgetting.
4. Base = **Qwen3-1.7B-*Base***, not Instruct.
5. **Never quantize before merge.** bf16 all the way through, quantize last.
6. Every corpus source passes the **same cleaning pipeline** (§2.3), synthetic included.
7. Don't advance a phase until its **DONE-CHECK** passes. Phase 5 gate is real —
   if CPT loses to raw, stop.
8. Respect the two-machine split: train on the 7900 GRE, deploy CPU-only on homesrv ≤1.7B.