claude
c15c2b7bd2
CLAUDE.md said the destination had no fixture and no accuracy number. It has both now: intent 73/96 and destination 12/33 on the classifier cascade, with the per-destination split, the floor cases and the grammar drift the labelling turned up. Anyone adding a grammar now reads that baselineGrammars mirrors buildRouter and drifts silently when it does not. docs/evals/2026-08-08-massive-warm-start.md was written on the V-655 branch and parked in .task/, which git excludes, so it was one `task start` away from being lost. It is a dated measurement and it belongs under docs/evals whatever branch produced it. Its "destination has no fixture at all" line is now a pointer to the file beside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
143 lines
7.3 KiB
Markdown
143 lines
7.3 KiB
Markdown
# MASSIVE Russian warm-start for the routing heads
|
|
|
|
Measured 2026-08-08 on workpc (Radeon RX 7900 GRE, ROCm). Covers V-546 step 2.
|
|
Workspace is `~/Programs/embed-training` on workpc, scripts `train_massive.py`,
|
|
`ab_run.py`, `ab.sh`, `probe_time.py`.
|
|
|
|
## What was trained
|
|
|
|
Two heads on a copy of multilingual-e5-small: `Linear(384, 60)` for MASSIVE's
|
|
own intents over a masked mean pool, `Linear(384, 111)` per token for BIO slot
|
|
tags. MASSIVE's label sets verbatim, no alignment to Maven's 7 intents. The
|
|
intent head is an auxiliary loss that shapes the pooled vector and is thrown
|
|
away.
|
|
|
|
Data is `amazon-massive-dataset-1.1` pulled from S3. The Hugging Face repo is
|
|
script-only and `datasets` 5.0 refuses those, so `load_dataset` cannot fetch it.
|
|
`ru-RU` is 11,514 train, 2,033 dev, 2,974 test, 60 intents, 55 slots, 111 BIO
|
|
labels. All 16,521 rows survived span alignment: `annot_utt` re-tokenised to its
|
|
own `utt` on every one.
|
|
|
|
Hyperparameters match `train_intent.py`, so the two runs differ in data only.
|
|
Frozen XLM-R vocabulary, body 2e-5, heads 1e-3, batch 32, sequence 64, 10
|
|
epochs. MASSIVE's own dev partition selects the epoch, on slot F1 with intent
|
|
accuracy as tiebreak. Selecting on 60-class intent accuracy would optimise a
|
|
head that gets deleted.
|
|
|
|
## Result
|
|
|
|
Epoch 9 of 10 by dev slot F1. Held-out MASSIVE test: intent 86.2%, slot span
|
|
F1 71.5% (P 68.5, R 74.8). Peak 1.70GB of 17.2GB, about 22 seconds an epoch,
|
|
under 4 minutes end to end. Dev slot F1 climbed monotonically to epoch 9 and
|
|
fell at 10, so 10 epochs was the right budget.
|
|
|
|
Ten slot types sit at 0% test recall. Every one of them has 1 to 7 test
|
|
instances: `alarm_type` has 3, `drink_type` has 1. That is support in MASSIVE's
|
|
Russian split, not a tagger failure. `playlist_name` at 6% of 16 is the first
|
|
real miss.
|
|
|
|
## The intent A/B, and why it settles nothing
|
|
|
|
`train_intent.py` was run against both bodies, three seeds by two smoothing
|
|
settings, on `train_v4.jsonl`. It is v4 and not v5 because v4 is what
|
|
`sweep2.log` measured. `ab_run.py` strips a `--base` flag onto the module global, so
|
|
`train_intent.py` is unmodified and its baseline stays reproducible. The stock
|
|
arm reproduced `sweep2.log` line for line.
|
|
|
|
Fixture accuracy, 91 cases, one case is 1.1 points:
|
|
|
|
| seed / smooth | stock | warm-started |
|
|
|---|---|---|
|
|
| 0 / 0.0 | 94.0% | 92.8% |
|
|
| 0 / 0.1 | 95.2% | 92.8% |
|
|
| 1 / 0.0 | 95.2% | 94.0% |
|
|
| 1 / 0.1 | 95.2% | 97.6% |
|
|
| 2 / 0.0 | 92.8% | 94.0% |
|
|
| 2 / 0.1 | 92.8% | 96.4% |
|
|
|
|
Mean 94.2% against 94.6%. That is +0.4 points, about a third of one case, and
|
|
inside seed noise. Spread widened. Stock lands in a 2.4-point band and
|
|
warm-started in a 4.8-point one. The warm-started arm holds both the best result
|
|
of the sweep and a tie for the worst. Seed 0 is the bad arm and it fails in a
|
|
specific way. Its dev peaks at epoch 2 and 3 and never improves, where stock
|
|
peaks around 7. The dev slice is a quarter of the seed rows. That is small
|
|
enough that early stopping is fragile when the body arrives already fitted.
|
|
|
|
**The A/B was never the test.** Intent had at most 4.8 points of headroom here.
|
|
MASSIVE was not trained for Maven's intents. Read it as "the warm-start does not
|
|
cost intent accuracy", nothing more.
|
|
|
|
## The measurement that does mean something
|
|
|
|
`want_time` is the one slot Maven's fixture scores, and MASSIVE has `time` and
|
|
`date`. Restricted to those two slot types, F1 is 74.9% over 609 gold spans on the
|
|
MASSIVE ru test split. Precision is 71.5 and recall 78.7. That beats the 71.5%
|
|
all-slot figure. Of the 530 test utterances carrying a time or a date, 73.4% get
|
|
every such span exactly right.
|
|
|
|
Out of domain matters more, because Maven's traffic is not this corpus. Ten
|
|
Maven-shaped utterances, none of them in MASSIVE:
|
|
|
|
| utterance | tagged |
|
|
|---|---|
|
|
| `напомни в 11:00 позвонить маме` | `time='11:00'`, `relation='маме'` |
|
|
| `напомни завтра в семь утра выпить таблетки` | `date='завтра'`, `time='семь утра'` |
|
|
| `поставь будильник на полседьмого` | `time='полседьмого'` |
|
|
| `через двадцать минут напомни про чайник` | `time='двадцать минут'` |
|
|
| `напомни в пятницу вечером забрать посылку` | `date='пятницу'`, `timeofday='вечером'` |
|
|
| `что у меня сегодня после обеда` | `date='сегодня'`, `time='после'`, `timeofday='обеда'` |
|
|
| `запиши что кофе закончился` | nothing |
|
|
| `что такое TCP` | `definition_word='TCP'` |
|
|
|
|
The first row is the V-572 defect utterance. `ReminderGrammar` handed the daemon
|
|
`HasTime: false` there, and the daemon asked "Когда?" at a sentence that had
|
|
already said when. `полседьмого` is a colloquial half-past that no digit pattern
|
|
catches. `запиши что кофе закончился` correctly carries nothing, because a note
|
|
has no time.
|
|
|
|
Two errors. `после обеда` split into `time='после'` plus `timeofday='обеда'`
|
|
when it is one span, and `через двадцать минут` dropped its `через`. Both are
|
|
boundary errors on spans the tagger did find.
|
|
|
|
Unplanned: `что такое TCP` returned `definition_word='TCP'`. MASSIVE has a slot
|
|
for the thing being asked about, which is a `SourceWorld` signal sitting in a
|
|
head already trained.
|
|
|
|
Ten hand-picked utterances are evidence, not a fixture.
|
|
|
|
## What this does not measure
|
|
|
|
Maven has no span fixture. `want_time` and `want_fn` are presence booleans and
|
|
`want_fact_key` is an exact string match, so nothing in the repo can score a
|
|
71.5% span tagger. Destination got one the same day, at 12/33 on the classifier
|
|
cascade: see `2026-08-08-destination-fixture.md`.
|
|
|
|
The missing span fixture is why the warm-start stays unjudged against Maven
|
|
rather than against MASSIVE.
|
|
|
|
## Datasets ruled out
|
|
|
|
Checked on 2026-08-08 and rejected as label sources:
|
|
|
|
- **MASSIVE's other 50 locales** ship in the same tarball and are parallel by id.
|
|
Co-training on them is free and unmeasured. English was ruled out by the owner
|
|
on 2026-08-08.
|
|
- **CLINC150** is reachable as parquet, 150 intents and 1,200 explicit
|
|
out-of-scope queries, English only. Its value is the labeled out-of-scope set
|
|
for fitting the energy threshold, not intent labels.
|
|
- **`d0rj/dolphin-ru`**, roughly 2.8M rows of FLAN-style tasks translated to
|
|
Russian. No intent, no slots, and not utterances anyone says to an assistant.
|
|
- **`psytechlab/EmpatheticIntents-ru`**, 24,856 rows of translated
|
|
EmpatheticDialogues with 32 emotion labels. Maven's mood enum is `neutral,
|
|
happy, thinking, tired, confused` and it describes her own reply, not the
|
|
speaker's emotion. No mapping exists.
|
|
- **`ai-forever/MERA`** and **`RussianNLP/russian_super_glue`**, benchmark
|
|
harnesses. Rows are prompt templates with `{toxic_comment}` placeholders.
|
|
- **`ZeroAgency/ru-big-russian-dataset`**, an LLM-judge quality corpus. Its
|
|
`question` and `classified_topic` columns are a usable Russian out-of-scope
|
|
pool for threshold fitting. That is the one thing CLINC150 can only supply in
|
|
English. The questions are long and written, so they belong in the negative
|
|
set, never in the in-scope `query` training set.
|
|
- No second Russian slot-filling corpus exists. The xSID mirrors are 404,
|
|
MultiATIS++ has no Russian, SLURP is not on the Hub.
|