# Plan — DIY RU-native base via Continued Pretraining (CPT) > **Execution update, 2026-07-18:** corpus packing and full-weight smoke are > complete; training reached step 1000/8077. The materialized corpus is 264.6M > tokens from CulturaX/Wikipedia/books only. Evaluation, the exact decision gate > and Qwen3 joint SFT are now canonical in > `2026-07-18-qwen3-resident-training-eval.md`. > **Goal:** Build our *own* RU-native base model instead of using Vikhr. Take a > clean newer base (Qwen3-1.7B-Base), continue-pretrain it on a curated Russian > corpus so it spells Cyrillic natively, then run the existing persona-LoRA → > merge → GGUF → deploy pipeline on top. We own the recipe, the corpus, and the > checkpoint — reproducible, no dependency on a third-party RU repo. > > **This is a big project. The corpus is 80% of the work.** Read the whole file > before starting. Do phases in order. Each phase has a **DONE-CHECK** — do not > advance until it passes. --- ## 0. Context an agent picking this up must know **Two machines — never conflate (from `CLAUDE.md`):** | Role | Box | Specs | Constraint | |---|---|---|---| | **Train** | this workstation | RX 7900 GRE, gfx1100 (RDNA3), **16GB VRAM**, ROCm 7.2 / torch 2.10 HIP, `cuda.is_available()==True` | CPT of 1.7B is *tight* — see Phase 3 VRAM math | | **Deploy** | homesrv `kami@192.168.1.104` | Ryzen 5 5600U CPU, **13GB RAM already swapping** (~8.8GB avail), Ubuntu 24.04 | CPU-only llama.cpp inference; model must stay ≤1.7B and every 200MB matters | **Working directories:** - Training scripts & data: `/home/kami/Programs/esp32-whisper-fine-tune/llm/` - existing: `train_rocm.py` (persona LoRA, ChatML), `llama-eval-test.py`, `data/` - **new files from this plan** also go here. - Daemon (consumes final GGUF): `/mnt/server/home/kami/apps/Maven/` (this repo). **Output contract (from `CLAUDE.md`, Decision B, locked 2026-07-11):** the daemon parses `{"response":"...","mood":"..."}` in all 4 consumers. This only matters for the **persona LoRA stage (Phase 6)**, NOT for CPT. CPT is raw next-token text, no JSON, no chat format. **Why CPT and not just a LoRA (from `CLAUDE.md`):** a LoRA cannot add spelling knowledge the base lacks — it only steers format/persona/mood/tools. CPT changes the base weights themselves, which is the only lever that can add genuine RU language competence. That is exactly why this is more than a LoRA project. **ROCm gotchas (inherit from `train_rocm.py`):** - `attn_implementation="eager"` (flash-attn ROCm wheels flaky). - `HSA_OVERRIDE_GFX_VERSION=11.0.0` in env (gfx1100). - **No bitsandbytes** — ROCm build is flaky. This forbids 8-bit Adam; see Phase 3. --- ## 1. Pipeline overview (the whole journey) ``` Phase 2 Corpus build → clean packed RU token blocks (.arrow/.bin, ~300M tok) Phase 3 train_cpt.py → CPT'd Qwen3-1.7B-Base (bf16 HF checkpoint) Phase 4 eval → Cyrillic + JSON validator; CPT base vs raw base numbers Phase 5 DECISION GATE → did CPT beat raw? if not, stop / re-corpus Phase 6 persona LoRA → existing train_rocm.py on the CPT'd base (ChatML, {response,mood}) Phase 7 merge → GGUF → Q8_0 → deploy to homesrv ``` Build order is **not** the run order: **build the corpus first (Phase 2)** because it gates everything and takes the longest. --- ## 2. Corpus — the core deliverable **Target: ~300M tokens** (Qwen3 tokenizer), curated. Rationale: <100M won't move the needle vs Qwen3's already-strong multilingual; >1B is weeks of wall-clock on one 16GB card. 300M is the "nudge, don't re-pretrain" sweet spot for 1 epoch. **Two token sources, combined:** - **A. Open datasets** (the bulk, real human RU text). - **B. Router-generated synthetic** (fills domain/persona/conversational gaps the open corpora lack — smart-home commands, assistant dialogue, Maven's domain). The user's **router for free inference providers** generates this. ### 2.1 Corpus mix table (sums to ~300M tokens) | # | Bucket | Share | Tokens | Source | Why | |---|---|---|---|---|---| | B1 | Cleaned web RU | 55% | ~165M | `uonlp/CulturaX` (ru) — **already cleaned + deduped** | backbone, broad RU | | B2 | Encyclopedic | 15% | ~45M | `wikimedia/wikipedia` config `20231101.ru` | clean factual RU, correct spelling | | B3 | Books / long-form | 10% | ~30M | `IlyaGusev/librusec` (or subset) | rich morphology, endings | | B4 | Conversational | 8% | ~24M | `Helsinki-NLP/open_subtitles` (ru) | spoken register — matches voice assistant | | B5 | **Router-synthetic** | 10% | ~30M | generated (§2.4) | domain/persona/smart-home RU, absent from open sets | | B6 | Maven real logs | 2% | ~6M | mine daemon dialogue history / logs | exact deploy-domain match | **Cap B5 (synthetic) at 10%.** Synthetic-heavy CPT causes model collapse / degeneration. The bulk MUST be real human text. This cap is a hard rule. ### 2.2 Exact acquisition — open datasets (Bucket A: B1–B4) All via HuggingFace `datasets`, streaming to avoid disk blowup. Some are gated (CulturaX, OSCAR) — accept the license on the HF model page while logged in (`huggingface-cli login`) first. ```python # corpus_fetch.py — run per bucket, writes raw .txt.gz shards to data/cpt_raw// from datasets import load_dataset # B1 CulturaX (gated — accept license first) ds = load_dataset("uonlp/CulturaX", "ru", split="train", streaming=True) # B2 Wikipedia ds = load_dataset("wikimedia/wikipedia", "20231101.ru", split="train", streaming=True) # B3 Books ds = load_dataset("IlyaGusev/librusec", split="train", streaming=True) # B4 Subtitles ds = load_dataset("Helsinki-NLP/open_subtitles", lang1="en", lang2="ru", split="train", streaming=True) ``` Pull enough raw text per bucket to survive filtering (filtering drops ~20–40%); over-pull ~1.5× the target token count, then trim after cleaning. Text field differs per dataset (`text`, `content`, `translation['ru']`) — normalize each to a single `text` string when writing shards. **Fallbacks if a source is unavailable:** B1 → `allenai/c4` (ru) or `oscar-corpus/OSCAR-2301` (ru); B3 → `IlyaGusev/gazeta` / Taiga corpus; B4 → `Helsinki-NLP/tatoeba` (ru pairs). ### 2.3 Cleaning pipeline (mandatory — dirty corpus = dirty base) One script `corpus_clean.py`, applied to every raw shard, in this order: 1. **Unicode fix** — `ftfy.fix_text` (repairs mojibake, normalizes to NFC). 2. **Language ID** — fastText `lid.176.bin`; drop any doc where top lang != `ru` or `p(ru) < 0.65`. (Download `lid.176.bin` from fastText releases.) 3. **Mixed-script rejection** — regex: reject any *word* containing both Cyrillic and Latin letters (`[а-яё].*[a-z]|[a-z].*[а-яё]` case-insensitive per word). This is the exact homoglyph problem `CLAUDE.md` names. Keep pure-Latin words (URLs/names) but drop homoglyph-contaminated ones. 4. **Boilerplate/length** — drop docs <200 chars or >50k chars; strip HTML tags, nav junk, repeated whitespace; drop docs with >30% non-letter chars. 5. **Dedup** — MinHash LSH via `datasketch` (`MinHashLSH`, threshold 0.8, 128 perms) across ALL buckets combined. Near-dup docs collapse to one. This is the single biggest quality lever — do not skip. 6. **PII light-scrub** (optional) — regex phone/email → placeholder, since B6 has real logs. Emit cleaned docs as JSONL `{"text": "..."}` in `data/cpt_clean/`. **DONE-CHECK 2.3:** run `corpus_stats.py` → per-bucket doc count, char count, and **a 100-doc random sample printed for eyeball review**. No mixed-script words in the sample. If sample looks dirty, fix the filter before proceeding. ### 2.4 Router-synthetic generation (Bucket B5 — the user's router) Purpose: generate ~30M tokens of clean RU text in registers open corpora lack — smart-home/assistant domain, natural conversational RU, Maven-adjacent topics. **This is CPT text (raw prose/dialogue), NOT the `{response,mood}` persona data** (that's Phase 6, a different dataset). Protocol (`corpus_synth.py`, calls the free-provider router): 1. **Topic seed list** — reuse `data/topics.txt` (already exists, ~46KB of topics) + add smart-home / daily-assistant / RU-domestic topics. 2. **Prompt template** (per topic), ask the router model to produce **long-form natural Russian prose or dialogue** on the topic — NOT JSON, NOT persona, NOT English. e.g. `"Напиши развёрнутый естественный текст на русском языке на тему: {topic}. 400–800 слов, живой разговорный стиль."` 3. **Diversify:** vary style knob (formal/casual/dialogue/monologue), vary length, rotate across the router's providers/models so it's not one model's fingerprint (fingerprint uniformity → collapse risk). 4. **Route every generation through the SAME cleaning pipeline (§2.3)** — language ID, mixed-script rejection, dedup against the real corpus too. Synthetic that fails Cyrillic cleanliness is exactly what we must not train on. 5. **Stop at ~30M tokens.** Enforce the 10% cap. **DONE-CHECK 2.4:** synthetic passes the same stats + eyeball sample as 2.3; token count ≤ 10% of total corpus; provider diversity logged. ### 2.5 Maven real logs (Bucket B6) Mine daemon dialogue history / logs for real RU user+assistant utterances. Strip to plain text, run through §2.3. Small (~6M tok) but highest domain value. Source location: check `mavend` logs / any dialogue history store in the repo. ### 2.6 Tokenize + pack (final corpus artifact) `corpus_pack.py`: - Load Qwen3-1.7B-Base tokenizer (`transformers.AutoTokenizer`). - Concatenate all cleaned JSONL, tokenize, append EOS between docs, **pack into fixed 2048-token blocks** (drop the ragged tail). No padding — packing is efficient for CPT. - Save as a HF `datasets` Arrow dir `data/cpt_packed/` (columns: `input_ids`). - Print final total token count — **must be ~300M ± 50M**. Adjust bucket pulls if off. **DONE-CHECK 2.6:** `data/cpt_packed/` loads; total tokens logged; a decoded random block reads as clean Russian. --- ## 3. `train_cpt.py` — continued pretraining (fits 16GB) New file next to `train_rocm.py`. Inherits the ROCm setup from it. ### 3.1 VRAM reality (why the config is what it is) Full-weight CPT of 1.7B with plain Adam ≈ 21–24GB → **does not fit 16GB**. We can't use 8-bit Adam (no bitsandbytes on ROCm). Fitting strategy, in preference order — **try A first, fall back to B**: - **A. Full-weight + `adafactor`** — Adafactor has no momentum states, cutting the optimizer-state VRAM roughly in half. Full-weight bf16 + adafactor + gradient checkpointing + batch 1 × grad-accum should fit ~16GB. This is the real "change the base weights" path. **Preferred.** - **B. High-rank DoRA on all linear layers** — if A OOMs or throughput is unbearable. Fits trivially. Weaker for language injection but still adapts more than a small LoRA. Use `r=64, α=128, target all linear`, `use_dora=True`. ### 3.2 Config ```python # train_cpt.py essentials import os; os.environ["HSA_OVERRIDE_GFX_VERSION"] = "11.0.0" from transformers import (AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer, DataCollatorForLanguageModeling) MODEL = "Qwen/Qwen3-1.7B-Base" # BASE, not Instruct model = AutoModelForCausalLM.from_pretrained( MODEL, torch_dtype="bfloat16", attn_implementation="eager") model.gradient_checkpointing_enable() model.config.use_cache = False args = TrainingArguments( output_dir="./Qwen3-1.7B-ru-cpt", per_device_train_batch_size=1, gradient_accumulation_steps=16, # eff. batch 16 × 2048 tok num_train_epochs=1, # ONE pass — more risks forgetting learning_rate=1e-5, # LOW — nudging pretrained weights lr_scheduler_type="cosine", warmup_ratio=0.03, optim="adafactor", # path A; no bitsandbytes bf16=True, gradient_checkpointing=True, logging_steps=20, save_steps=500, # checkpoint often — this runs for DAYS save_total_limit=3, report_to="none", ) collator = DataCollatorForLanguageModeling(tokenizer, mlm=False) # causal LM ``` - **lr 1e-5 is critical.** CPT lr must be ~10× lower than LoRA lr. High lr = catastrophic forgetting (model forgets English, reasoning, everything). If the eval (Phase 4) shows English/tool ability collapsed, lr was too high. - **1 epoch.** Multiple epochs over 300M tokens overfits and forgets. - **Resume/checkpoint is mandatory** — days-long job; support `--resume_from_checkpoint`. ### 3.3 Run ```bash cd /home/kami/Programs/esp32-whisper-fine-tune/llm HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py 2>&1 | tee cpt_run.log ``` **DONE-CHECK 3:** training completes 1 epoch without OOM; loss decreases and plateaus (not NaN, not flat-from-step-0); checkpoint saved to `./Qwen3-1.7B-ru-cpt/`. If OOM → switch to path B (DoRA). --- ## 4. Eval — did CPT actually help? `eval_cyrillic.py` now writes deterministic machine-readable results using pinned human-written RU and EN Universal Dependencies test sets. **Metrics (run on held-out human-written text — NOT synthetic, NOT training data):** 1. **Cyrillic validity %** — generate on RU prompts; % of outputs with zero mixed-script words and no homoglyph swaps (reuse §2.3 rule 3 as the checker). 2. **Perplexity** on a held-out clean RU text set (lower = better RU fit). 3. **English-retention check** — a few EN prompts; confirm the model still answers in coherent English (catches catastrophic forgetting). 4. **JSON-format sanity** (light, for later) — can it produce `{response,mood}` when asked. (Real test is post-persona-LoRA.) **Run it three ways and tabulate:** | Model | Cyrillic valid % | RU PPL | EN retained? | |---|---|---|---| | raw Qwen3-1.7B-Base | (baseline) | | | | our CPT'd base | (must beat raw) | | | **DONE-CHECK 4:** table filled with real numbers. --- ## 5. DECISION GATE - **CPT base beats raw on Cyrillic %/PPL AND English retained** → proceed to Phase 6. - **No improvement** → CPT didn't earn its compute. Options: (a) more/better corpus (usually the fix — dirty or too-small corpus), (b) tune lr, (c) accept raw Qwen3-1.7B as the base and skip CPT. Do NOT proceed to persona LoRA on a CPT base that lost to raw. - **English collapsed** → lr too high or too many epochs; re-run Phase 3 with lower lr. --- ## 6. Persona LoRA on the CPT'd base (existing pipeline) Now the CPT base is just a better base. The rewritten `train_rocm.py` trains a balanced joint persona/router adapter and uses Qwen3's own chat template. It masks the rendered assistant continuation instead of assuming literal ChatML boundary token IDs. - The persona **data** is the `{response,mood}` corpus per `CLAUDE.md` Decision B (separate from the CPT corpus): 45% persona chit-chat, 15% graceful failure (`tired`/`confused`), 20% tool calls, 10% real utterances, 10% EN→EN. Mood quotas + validator per `CLAUDE.md`. **This dataset is still an open item** — see `CLAUDE.md` "Target data recipe" and `gen_data.py` open item. **DONE-CHECK 6:** LoRA trains; eval (Phase 4) re-run on merged model shows `{response,mood}` parses + Cyrillic clean + moods in enum. --- ## 7. Merge → GGUF → quantize → deploy Per `CLAUDE.md` deploy path, with **Q8_0 first** (decision this session: start high-precision to remove the Cyrillic-at-Q4 gamble, drop to Q5/Q4 only if homesrv memory chokes): ```bash # 1. merge LoRA → bf16 python -c "from peft import AutoPeftModelForCausalLM; import torch; \ m=AutoPeftModelForCausalLM.from_pretrained('./Qwen3-1.7B-ru-lora',torch_dtype=torch.bfloat16); \ m.merge_and_unload().save_pretrained('./Qwen3-maven-merged'); \ from transformers import AutoTokenizer; AutoTokenizer.from_pretrained('./Qwen3-1.7B-ru-lora').save_pretrained('./Qwen3-maven-merged')" # 2. HF → GGUF (in llama.cpp) python convert_hf_to_gguf.py ./Qwen3-maven-merged --outfile qwen3-maven-f16.gguf --outtype f16 # 3. quantize — Q8_0 first (~1.6GB for 1.7B) ./llama-quantize qwen3-maven-f16.gguf Qwen3-Maven-1.7B-Q8_0.gguf Q8_0 # 4. deploy: scp to homesrv; measure tok/s + swap with --mlock. # If RAM chokes → re-quant Q5_K_M, then Q4_K_M. Point deploy/mavend.json # phraser.model_path at the file; small -ngl on Vulkan for a speedup. ``` **DONE-CHECK 7:** model runs on homesrv at acceptable voice tok/s (measure!), Cyrillic clean in live output, `{response,mood}` parsed by the daemon. --- ## 8. File manifest (what this plan creates) All in `/home/kami/Programs/esp32-whisper-fine-tune/llm/`: | File | Phase | Purpose | |---|---|---| | `corpus_fetch.py` | 2.2 | pull open datasets → `data/cpt_raw/` | | `corpus_synth.py` | 2.4 | router-generate synthetic RU → `data/cpt_raw/synth/` | | `corpus_clean.py` | 2.3 | ftfy + langID + mixed-script + dedup → `data/cpt_clean/` | | `corpus_stats.py` | 2.3 | per-bucket counts + sample dump (DONE-CHECKs) | | `corpus_pack.py` | 2.6 | tokenize + pack 2048 blocks → `data/cpt_packed/` | | `train_cpt.py` | 3 | continued pretraining, adafactor/DoRA | | `eval_cyrillic.py` | 4 | Cyrillic % + PPL + EN-retention (extends `llama-eval-test.py`) | | `data/cpt_packed/` | 2.6 | final corpus artifact (~300M tok) | | `./Qwen3-1.7B-ru-cpt/` | 3 | CPT'd base checkpoint | ### 8.1 One-time setup ```bash cd /home/kami/Programs/esp32-whisper-fine-tune/llm # python deps (torch/transformers already present from train_rocm.py) pip install datasets ftfy datasketch fasttext openai peft accelerate # fastText language-id model → data/lid.176.bin (corpus_clean.py requires it) wget -O data/lid.176.bin \ https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin # HF login — needed to accept the CulturaX (gated) license huggingface-cli login ``` If `fasttext` wheels fail to build, `pip install fasttext-wheel` is the prebuilt fallback. `lid.176.bin` can also live elsewhere via `export FASTTEXT_LID=/path`. ### 8.2 Runbook (full run order) ```bash cd /home/kami/Programs/esp32-whisper-fine-tune/llm # ── Phase 2: corpus ────────────────────────────────────────────── python corpus_fetch.py all # 2.2 open datasets ROUTER_BASE_URL="https://your-router/v1" \ ROUTER_API_KEY="..." ROUTER_MODELS="m-a,m-b,m-c" \ python corpus_synth.py # 2.4 synthetic (≤10%) MAVEN_LOGS=/path/to/dialogue.jsonl python corpus_logs.py # 2.5 optional python corpus_clean.py # 2.3 clean+dedup all python corpus_stats.py # DONE-CHECK 2.3/2.4 python corpus_pack.py # 2.6 → data/cpt_packed/ # ── Phase 3: continued pretraining (days; --resume to continue) ── HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py 2>&1 | tee cpt_run.log # OOM? → CPT_PATH=dora HSA_OVERRIDE_GFX_VERSION=11.0.0 python train_cpt.py # ── Phase 4: eval + gate (Phase 5) ────────────────────────────── python eval_cyrillic.py --model Qwen/Qwen3-1.7B-Base # baseline python eval_cyrillic.py --model ./Qwen3-1.7B-ru-cpt # must beat baseline # ── Phase 6-7: persona LoRA → merge → GGUF → Q8_0 → homesrv ────── # train_rocm.py pointed at ./Qwen3-1.7B-ru-cpt ; then §7 deploy block. ``` Third-party deps: `datasets`, `ftfy`, `datasketch`, `fasttext` (+ `lid.176.bin`), `openai`, `peft`, `transformers`, `accelerate`. --- ## 9. Hard rules (do not violate) 1. CPT corpus is **raw text**, no JSON, no ChatML. Persona `{response,mood}` data is a **separate** dataset (Phase 6). 2. Synthetic ≤ **10%** of corpus. Bulk must be real human RU. 3. CPT lr = **1e-5**, **1 epoch**. Higher = catastrophic forgetting. 4. Base = **Qwen3-1.7B-*Base***, not Instruct. 5. **Never quantize before merge.** bf16 all the way through, quantize last. 6. Every corpus source passes the **same cleaning pipeline** (§2.3), synthetic included. 7. Don't advance a phase until its **DONE-CHECK** passes. Phase 5 gate is real — if CPT loses to raw, stop. 8. Respect the two-machine split: train on the 7900 GRE, deploy CPU-only on homesrv ≤1.7B.