Maven/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md

# Slice 19: from-scratch tiny sequence encoders beat sparse and e5 on the capability-question holdout (seed-stable FA 0.48-0.55 vs 0.667 / ~0.98) yet cap at ~50% accuracy on the leave-generator-out pragmatics split — a pretrained prior is the next step

Date: 2026-09-08 · Task: V-726 (slice 19) · Box: homesrv, Ryzen 5 5600U, 13 GB, CPU-only (this is the production machine) · Build: `cmd/semantic-router-experiment/slice19_*.py`, torch 2.14.0+cpu in `/tmp/mvn-exp-venv`, no GPU.

## 0. Frozen artifacts

```text
development corpus v2 hash: b27fd48f478ca477
original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected)
text normalization: NormalizeMatchText (NFKC, lowercase, whitespace-collapse; punctuation and ё kept)
embedding file: /tmp/mvn-experiment/embeddings.json
total examples: 3025
dev pool: 2490     (796 action / 1694 not_action)
frozen holdout: 535
router-residual: 2943
```

Same data as slices 16-18. Training input per brief §9 is always the
punctuation-stripped text (`re.sub(r"[^\w\s]", " ", c)` + collapse); the `orig`
and `nofinal` variants are evaluation-only stress views.

### Method note: the tokenizer axis was broken once and fixed

The BPE first trained on the *stripped* texts. `tokenizers`' ByteLevel BPE then
silently **drops boundary punctuation at inference** (`сервис,` → the `сервис`
token, comma gone; a comma observed in training keeps its own id 55), so `orig`
vs `strip` differed on only 0.6% of the corpus and the BPE stress axis was a
no-op. Fixed by training both tokenizers on the *natural* dev texts (punctuation
kept, per the normalization line above) while training input remains stripped.
After the fix punctuation fires as real in-vocabulary symbols under stress, and
35.1% of rows differ between `orig` and `strip`. All numbers below are from the
fixed build; the pre-fix run is discarded (its "stress-identical → robust"
reading for BPE models was an artifact).

## 1. What was tested

Three from-scratch, order-sensitive encoders, each in a size ladder, all trained
only on the binary action/not-action label from the dev pool:

| family | A. CharCNN | B. BiGRU | C. TinyTransformer |
| --- | --- | --- | --- |
| input | codepoint ids | subword ids (BPE) | subword ids (BPE) |
| depth | 1D convs (widths 2-5), global max-pool | 1-layer bidirectional GRU, max-pool~final | 2-4 self-attention blocks (PreNorm, GELU FFN 4x), learned pos, max-pool ~ CLS |
| sizes | tiny, medium | tiny, medium, large | small, medium |
| params | 27 k / 149 k | 129 k / 356 k / 1,106 k | 558 k / 2,022 k |

Chosen per the slice-18 handoff: the deciding signal is order and trailing
politeness/modality, which local n-grams provably could not rank (pair 0.571).

## 2. Tokenizers

Trained on the dev pool only.

| tokenizer | vocab | notes |
| --- | --- | --- |
| CharVocab | 73 (id 0 = PAD; OOV → 0) | codepoint ids incl. punctuation and the occasional CJK char |
| BpeVocab | 1,234 / 2,048 cap | byte-level BPE, serialized 52,897 bytes |

Max lengths: char 64, BPE 24. No narrowing to Russian; the corpus is the vocab.

## 3. Training setup

Fixed hyperparameters, no grid search, seed = 42 + fold for grouped CV:

| family | epochs | lr | batch | grad clip | optimiser |
| --- | --- | --- | --- | --- | --- |
| char_cnn | 20 | 1e-3 | 64 | none | AdamW wd 1e-4, BCE-with-logits |
| bigru | 15 | 5e-4 | 64 | 1.0 | AdamW wd 1e-4, BCE-with-logits |
| tiny_transformer | 25 | 5e-4 | 64 | 1.0 | AdamW wd 1e-4, BCE-with-logits |

Grouped 5-fold CV reusing the existing `cv_fold` split; the capability-question
and question families are genuinely held out per fold. Padded sequences,
zero-pad ignored.

## 4. Grouped CV, binary gate (strip input, OOF at threshold 0.5)

| config | PR-AUC | ROC-AUC | action_P | action_R | FA | FA rate |
| --- | --- | --- | --- | --- | --- | --- |
| char_cnn_tiny | 0.814 | 0.908 | 0.808 | 0.672 | 127 | 5.1% |
| char_cnn_medium | **0.838** | 0.917 | **0.816** | 0.706 | 127 | **5.1%** |
| bigru_tiny | 0.813 | 0.889 | 0.726 | 0.716 | 215 | 8.6% |
| bigru_medium | 0.794 | 0.864 | 0.684 | 0.693 | 255 | 10.2% |
| bigru_large | 0.837 | 0.893 | 0.689 | 0.755 | 271 | 10.9% |
| tiny_transformer_small | 0.731 | 0.820 | 0.704 | 0.670 | 224 | 9.0% |
| tiny_transformer_medium | 0.742 | 0.844 | 0.686 | 0.750 | 273 | 11.0% |

The char CNN is the best *aggregate* binary separator (as in slice 18, where the
n-gram representation owned the aggregate boundary). The sequence models trade
aggregate precision for spread-out recall.

### Fold variance (at 0.5)

```text
 config               PR mean   PR min   PR max   FA per fold (n)
 char_cnn_tiny          0.791    0.585    0.967    10, 17, 13, 66, 21
 char_cnn_medium        0.802    0.572    0.974    12, 27, 10, 59, 19
 bigru_tiny             0.803    0.632    0.880    25, 53, 37, 63, 37
 bigru_medium           0.751    0.461    0.930    49, 25, 62, 38, 81
 bigru_large            0.790    0.511    0.921    20, 43, 58, 60, 90
 tiny_transformer_small 0.713    0.529    0.923    64, 17, 61, 23, 59
 tiny_transformer_medium 0.714   0.492    0.900    38, 46, 48, 97, 44
```

Fold 3 (`n=506`, the capability/question-heavy split) is the hard fold for the
CNN/GRU families exactly as in every slice since 16.

## 5. Aggregate comparison against every prior head

| model | extras | PR-AUC | action_P | action_R | FA rate | cap-Q LOFO FA | pair order |
| --- | --- | --- | --- | --- | --- | --- | --- |
| e5 binary linear (C=1.0) * | ~385 | 0.623 | 0.688 | 0.476 | 6.9% | 1.000 | 0.114 |
| e5 binary MLP H=32 * | 12,353 | 0.674 | 0.673 | 0.569 | 8.8% | 0.976 | 0.136 |
| sparse word+char logistic (sl. 18) | 9,403 | 0.838 | 0.875 | 0.485 | 2.2% | 0.667 | 0.571 |
| char_cnn_medium (19) | 149 k | 0.838 | 0.816 | 0.706 | 5.1% | 0.976 | 0.535 |
| bigru_tiny (19) | 129 k | 0.813 | 0.726 | 0.716 | 8.6% | 0.540 | 0.635 |
| tiny_transformer_small (19) | 558 k | 0.731 | 0.704 | 0.670 | 9.0% | 0.421 | 0.763 |

\* e5 rows are recomputed here with fixed single hyperparameters and the slice 17
configuration (`MLP early_stopping=True, validation_fraction=0.15,
n_iter_no_change=10`; logistic C=1.0). FA rates reproduce the published slice
16/17 numbers (6.9% / 8.8%). The published **PR-AUC 0.707** for the linear head
came from the slice 16 C-grid sweep; a fixed-C reproduction lands at 0.623 (and
even the best C on the grid, C=100, reaches only 0.675). The slice 18 table's PR
"0.707 → 0.838 sparse gain" is therefore overstated; the honest sparse-vs-e5 PR
gap is ~0.838 vs 0.62-0.68.

On the **aggregate** boundary the from-scratch sequence models neither beat the
sparse gate's 2.2% FA (5.1-11.0%) nor reach any P ≥ 0.95 operating point (see §6).
Aggregate accuracy was downgraded to a secondary metric from this slice.

## 6. Safety operating curve

None of the seven from-scratch configs has any threshold with P ≥ 0.95 at R > 0
under the grouped OOF (the `ops` field is empty for every config). The sparse
gate's slice-18 operating point (P ≥ 0.95 at R 0.264) does not transfer to any
sequence model. A from-scratch tiny sequence encoder is not a safe standalone
gate out of the box.

## 7. Leave-generator-out: capability questions (the critical split)

Train without the `capability_question` family, evaluate on its 126 rows (0/126
positive — every row here must *not* trip the gate). Threshold 0.5, seed 17:

| config | FA | FA rate | accuracy |
| --- | --- | --- | --- |
| **tiny_transformer_small** | **53** | **0.421** | **0.579** |
| bigru_tiny | 68 | 0.540 | 0.460 |
| tiny_transformer_medium | 69 | 0.548 | 0.452 |
| bigru_medium | 93 | 0.738 | 0.262 |
| char_cnn_tiny | 97 | 0.770 | 0.230 |
| bigru_large | 113 | 0.897 | 0.103 |
| char_cnn_medium | 123 | 0.976 | 0.024 |

Against the sparse gate (0.667 FA → 0.333 accuracy) the leading from-scratch
configs cut the fail rate by a quarter to a third on seed-stable seeds. Against
e5 (1.000 / 0.976) the win is decisive —
the frozen embedder genuinely cannot separate a held-out capability question
from its executable sibling.

### But the single-seed number is not trustworthy

Same leave-generator-out training, three seeds (17 / 41 / 7), cap-Q FA rate:

```text
 tiny_transformer_small  0.603  0.421  0.937   (spread 0.52)
 tiny_transformer_medium 0.548  0.484  0.524   (spread 0.06)
 bigru_tiny              0.690  0.540  0.476   (spread 0.21)
 bigru_medium            0.738  0.389  0.952   (spread 0.56)
```

Only the **transformer-medium is seed-stable**, and it lands at **0.48-0.55 FA
(~50% ± 3 accuracy)**: a genuine but near-chance separation on the exact
generator family the task exists for. The transformer-small's best single seed
(0.421) and the bigru-medium pair-ordering win (§9) are both inside unstable
regimes. The tooling that produced the lucky number is right; the number itself
is luck.

## 8. Full-family LOFO, leading config per architecture

Seed 41, held-out family evaluated in full (accuracy; FA; action recall):

| held-out family | rows | char_cnn_medium | bigru_tiny | tiny_transformer_small |
| --- | --- | --- | --- | --- |
| polite_request | 481 | 0.279 acc / FA 347 / R 1.000 | 0.486 / FA 247 / R 1.000 | **0.975 / FA 12 / R 1.000** |
| modal_request | 223 | R 0.430 (P inf) | R 0.484 (P 1.000) | **R 0.561 (P 1.000)** |
| first_person_request | 833 | **R 0.942** (P 0.991) | R 0.000 (acc 1.000, no FAs) | R 0.646 (P 1.000) |
| reordered_target | 353 | 1.000 / FA 0 | 0.994 / FA 2 | 1.000 / FA 0 |
| question | 123 | 1.000 / FA 0 | 1.000 / FA 0 | 1.000 / FA 0 |

The transformer-small is the only config that survives the **polite_request**
holdout — the family that sounds most like a capability question under voice
("пожалуйста" + soft predicate). The CNN collapses it into "action" (347 of 481
false actions); the GRU nearly so. The same split in reverse: char_cnn owns
first_person recall (short imperatives), which the other two under-recall.

## 9. Paired action/capability ordering

2268 pairs (capability question vs executable sibling on the shared object
noun), ordering accuracy and median margin on grouped OOF, strip text:

| config | ordering | median margin | reversed pairs |
| --- | --- | --- | --- |
| char_cnn_tiny | 0.406 (reversed) | −0.090 | 1347 |
| char_cnn_medium | 0.535 | +0.021 | 1054 |
| bigru_tiny | 0.635 | +0.169 | 828 |
| bigru_medium | **0.866** | +0.598 | 304 |
| bigru_large | 0.724 | +0.282 | 627 |
| tiny_transformer_small | 0.763 | +0.804 | 538 |
| tiny_transformer_medium | 0.648 | +0.019 | 799 |

bigru_medium reaches 0.866 ordering with a healthy +0.598 median margin — the
first head since the slices began that can more often than not put the executable
above its sibling (sparse: 0.571, margin +0.066). Caveat: this is the same
bigru_medium whose LOFO accuracy swings 0.39-0.95 across seeds; margin and
ordering were measured from the grouped-CV models (seed 42+fold), not the same
seed as the LOFO numbers, and the two axes were not measured jointly. The margin
being three times the sparse margin on the *strip* basis is still a real,
sequence-level result. The char CNN's 0.406-0.535 confirms it has no pragmatics
signal at all — it is a shallow aggregate-n-gram reader.

## 10. Punctuation / voice stress

Models trained on stripped text; tokenizers trained on natural text (so
punctuation is a real, seen-at-vocab, unseen-at-training symbol). Stress views
`orig` (full punctuation) / `nofinal` (trailing-only) / `strip`.

Capability-question FA (in-distribution, 126 rows), and pair ordering per view:

```text
 config              capQ_FA orig/nofinal/strip        pairs orig/nofinal/strip
 char_cnn_medium     0.540 / 0.540 / 0.532            0.538 / 0.538 / 0.535
 bigru_tiny          0.230 / 0.270 / 0.310            0.660 / 0.604 / 0.635
 tiny_transformer_small  0.119 / 0.135 / 0.143        0.797 / 0.791 / 0.763
 tiny_transformer_medium 0.262 / 0.262 / 0.286        0.631 / 0.632 / 0.648
```

Punctuation is not load-bearing for any config (all three views within a couple
of points), and the transformer *gains* ordering accuracy when full punctuation
is present (tf_small 0.797 orig > 0.763 strip). The models ignore the `?`/`,`,
which is the safest possible behaviour under ASR, where they cannot be trusted.
The "stress-robust" claim here is real (unlike the pre-fix build): the
punctuation codepoints existed in the boxes' vocabularies and were simply not
used.

## 11. Runtime and size (homesrv CPU, batch 1, p50 inference + tokenisation)

| config | params | fp32 | int8 (=params) | latency p50/p95 | tok | RAM delta over torch |
| --- | --- | --- | --- | --- | --- | --- |
| char_cnn_tiny | 27,297 | 107 KiB | 27 k | 0.28 / 0.42 ms | 3.8 µs | — |
| char_cnn_medium | 149,313 | 583 KiB | 149 k | 0.45 / 0.67 ms | 2.4 µs | ~118 MB |
| bigru_tiny | 129,025 | 504 KiB | 129 k | 0.41 / 0.73 ms | 16.7 µs | ~48 MB |
| bigru_medium | 356,353 | 1.4 MiB | 356 k | 0.52 / 0.82 ms | 15.3 µs | ~63 MB |
| bigru_large | 1,105,921 | 4.2 MiB | 1.1 M | 0.85 / 1.62 ms | 19.5 µs | — |
| tiny_transformer_small | 557,953 | 2.1 MiB | 558 k | 0.89 / 1.30 ms | 24.4 µs | ~89 MB |
| tiny_transformer_medium | 2,021,569 | 7.7 MiB | 2.0 M | 2.57 / 3.78 ms | 17.8 µs | — |

Every config is sub-4 ms end to end on the production CPU; BPE tokenisation is
the dominant term for a single utterance. RAM deltas are fresh-process
high-water marks over a ~225 MB torch baseline (which is itself the item the
deploy would have to absorb — ~chromosome 48 MB at the model level, crane).

## 12. Conclusion

### Verdict: sequence sensitivity is real and useful, from-scratch tiny is not enough

Three findings, one per decision rule:

1. **The critical split improved by half, on stable seeds.** The
   transformer-medium holds capability-question LOFO FA at 0.48-0.55 across
   seeds (sparse gate: 0.667; e5: 1.000/0.976). The transformer-small reaches
   0.421 on its lucky seed. Sequence structure demonstrably carries *some* of
   the pragmatics distinction that local features cannot (pair ordering 0.866
   with +0.598 margin for bigru_medium; tf_small 0.975 accuracy on the held-out
   polite_request family vs 0.279 for the CNN). The slice-18 direction —
   "order and trailing politeness/modality are the deciding signal" — is
   confirmed, not refuted.

2. **It is still a near-coin-flip on the exact unseen-generator case, and the
   variance is the story.** Best seed-stable accuracy is ~52%, and only one of
   seven configs is seed-stable on that metric (the others swing 0.21-0.56 of
   FA rate across three seeds). No config reaches a P ≥ 0.95 gate, and every
   config's aggregate FA rate (5.1-11.0%) is worse than the sparse gate's 2.2%.
   A from-scratch model this size can *see* a difference between the two members
   of a pair from data within the pool, but cannot *generalise it* to a
   generator family it has never seen. It memorises family-ish behaviour; when
   the family is truly new it reverts toward chance.

3. **The next step is a pretrained prior, not a bigger from-scratch model.**
   The distinguishing features — polite softeners, modal auxiliaries, the
   trailing interrogative falling back on world structure — are things a 4 M-8 M
   pretrained encoder (e.g. a small modern multilingual transformer) already
   encodes in its weights, so fine-tuning on the same 2490-row dev pool can be
   expected to move the seed-stable ~0.5 accuracy rather than shuffle the dice
   in a new random draw. Same frozen data, same grouped folds, same 15-section
   metrics — with the seed-stability measurement (§7) carried forward as
   mandatory, since it is what disambiguates a result from a luck draw.

Practical carry-outs: training input stays punctuation-stripped (free stress
robustness, no cost); tokenizers are trained on natural text so the stress axis
is real; never report a single-seed LOFO number without its seed neighbours.

## 13. Commit hash for tooling

`7d31de5` — `cmd/semantic-router-experiment/slice19_models.py`,
`slice19_bpe.py`, `slice19_main.py`; artifacts under `/tmp/mvn-s19/`.