Maven/docs/evals/2026-08-08-massive-warm-start.md

MASSIVE Russian warm-start for the routing heads

Measured 2026-08-08 on workpc (Radeon RX 7900 GRE, ROCm). Covers V-546 step 2. Workspace is ~/Programs/embed-training on workpc, scripts train_massive.py, ab_run.py, ab.sh, probe_time.py.

What was trained

Two heads on a copy of multilingual-e5-small: Linear(384, 60) for MASSIVE's own intents over a masked mean pool, Linear(384, 111) per token for BIO slot tags. MASSIVE's label sets verbatim, no alignment to Maven's 7 intents. The intent head is an auxiliary loss that shapes the pooled vector and is thrown away.

Data is amazon-massive-dataset-1.1 pulled from S3. The Hugging Face repo is script-only and datasets 5.0 refuses those, so load_dataset cannot fetch it. ru-RU is 11,514 train, 2,033 dev, 2,974 test, 60 intents, 55 slots, 111 BIO labels. All 16,521 rows survived span alignment: annot_utt re-tokenised to its own utt on every one.

Hyperparameters match train_intent.py, so the two runs differ in data only. Frozen XLM-R vocabulary, body 2e-5, heads 1e-3, batch 32, sequence 64, 10 epochs. MASSIVE's own dev partition selects the epoch, on slot F1 with intent accuracy as tiebreak. Selecting on 60-class intent accuracy would optimise a head that gets deleted.

Result

Epoch 9 of 10 by dev slot F1. Held-out MASSIVE test: intent 86.2%, slot span F1 71.5% (P 68.5, R 74.8). Peak 1.70GB of 17.2GB, about 22 seconds an epoch, under 4 minutes end to end. Dev slot F1 climbed monotonically to epoch 9 and fell at 10, so 10 epochs was the right budget.

Ten slot types sit at 0% test recall. Every one of them has 1 to 7 test instances: alarm_type has 3, drink_type has 1. That is support in MASSIVE's Russian split, not a tagger failure. playlist_name at 6% of 16 is the first real miss.

The intent A/B, and why it settles nothing

train_intent.py was run against both bodies, three seeds by two smoothing settings, on train_v4.jsonl. It is v4 and not v5 because v4 is what sweep2.log measured. ab_run.py strips a --base flag onto the module global, so train_intent.py is unmodified and its baseline stays reproducible. The stock arm reproduced sweep2.log line for line.

Fixture accuracy, 91 cases, one case is 1.1 points:

seed / smooth	stock	warm-started
0 / 0.0	94.0%	92.8%
0 / 0.1	95.2%	92.8%
1 / 0.0	95.2%	94.0%
1 / 0.1	95.2%	97.6%
2 / 0.0	92.8%	94.0%
2 / 0.1	92.8%	96.4%

Mean 94.2% against 94.6%. That is +0.4 points, about a third of one case, and inside seed noise. Spread widened. Stock lands in a 2.4-point band and warm-started in a 4.8-point one. The warm-started arm holds both the best result of the sweep and a tie for the worst. Seed 0 is the bad arm and it fails in a specific way. Its dev peaks at epoch 2 and 3 and never improves, where stock peaks around 7. The dev slice is a quarter of the seed rows. That is small enough that early stopping is fragile when the body arrives already fitted.

The A/B was never the test. Intent had at most 4.8 points of headroom here. MASSIVE was not trained for Maven's intents. Read it as "the warm-start does not cost intent accuracy", nothing more.

The measurement that does mean something

want_time is the one slot Maven's fixture scores, and MASSIVE has time and date. Restricted to those two slot types, F1 is 74.9% over 609 gold spans on the MASSIVE ru test split. Precision is 71.5 and recall 78.7. That beats the 71.5% all-slot figure. Of the 530 test utterances carrying a time or a date, 73.4% get every such span exactly right.

Out of domain matters more, because Maven's traffic is not this corpus. Ten Maven-shaped utterances, none of them in MASSIVE:

utterance	tagged
напомни в 11:00 позвонить маме	time='11:00', relation='маме'
напомни завтра в семь утра выпить таблетки	date='завтра', time='семь утра'
поставь будильник на полседьмого	time='полседьмого'
через двадцать минут напомни про чайник	time='двадцать минут'
напомни в пятницу вечером забрать посылку	date='пятницу', timeofday='вечером'
что у меня сегодня после обеда	date='сегодня', time='после', timeofday='обеда'
запиши что кофе закончился	nothing
что такое TCP	definition_word='TCP'

The first row is the V-572 defect utterance. ReminderGrammar handed the daemon HasTime: false there, and the daemon asked "Когда?" at a sentence that had already said when. полседьмого is a colloquial half-past that no digit pattern catches. запиши что кофе закончился correctly carries nothing, because a note has no time.

Two errors. после обеда split into time='после' plus timeofday='обеда' when it is one span, and через двадцать минут dropped its через. Both are boundary errors on spans the tagger did find.

Unplanned: что такое TCP returned definition_word='TCP'. MASSIVE has a slot for the thing being asked about, which is a SourceWorld signal sitting in a head already trained.

Ten hand-picked utterances are evidence, not a fixture.

What this does not measure

Maven has no span fixture. want_time and want_fn are presence booleans and want_fact_key is an exact string match, so nothing in the repo can score a 71.5% span tagger. Destination got one the same day, at 12/33 on the classifier cascade: see 2026-08-08-destination-fixture.md.

The missing span fixture is why the warm-start stays unjudged against Maven rather than against MASSIVE.

Datasets ruled out

Checked on 2026-08-08 and rejected as label sources:

• MASSIVE's other 50 locales ship in the same tarball and are parallel by id. Co-training on them is free and unmeasured. English was ruled out by the owner on 2026-08-08.
• CLINC150 is reachable as parquet, 150 intents and 1,200 explicit out-of-scope queries, English only. Its value is the labeled out-of-scope set for fitting the energy threshold, not intent labels.
• d0rj/dolphin-ru, roughly 2.8M rows of FLAN-style tasks translated to Russian. No intent, no slots, and not utterances anyone says to an assistant.
• psytechlab/EmpatheticIntents-ru, 24,856 rows of translated EmpatheticDialogues with 32 emotion labels. Maven's mood enum is neutral, happy, thinking, tired, confused and it describes her own reply, not the speaker's emotion. No mapping exists.
• ai-forever/MERA and RussianNLP/russian_super_glue, benchmark harnesses. Rows are prompt templates with {toxic_comment} placeholders.
• ZeroAgency/ru-big-russian-dataset, an LLM-judge quality corpus. Its question and classified_topic columns are a usable Russian out-of-scope pool for threshold fitting. That is the one thing CLINC150 can only supply in English. The questions are long and written, so they belong in the negative set, never in the in-scope query training set.
• No second Russian slot-filling corpus exists. The xSID mirrors are 404, MultiATIS++ has no Russian, SLURP is not on the Hub.