11 KiB
Slice 18: Sparse lexical gate beats every e5 head on the aggregate action boundary, and still collapses on capability questions it has not seen
0. Frozen Artifacts (unchanged from slices 16-17)
development corpus v2 hash: b27fd48f478ca477
original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected)
text normalization: NormalizeMatchText (NFKC, lowercase, whitespace-collapse; punctuation and ё kept)
representation: sparse TF-IDF over RAW TEXT (e5 vectors ignored)
embedding file: /tmp/mvn-experiment/embeddings.json
total examples: 3025
dev pool: 2490
frozen holdout: 535
router-residual: 2943
No ONNX runtime on this box, so no e5 re-embedding; sparse features read the
text field directly. Action class: 796 action vs 1694 not_action in dev pool.
1. What was tested
Three order-sensitive sparse representations, each with an L2-regularised logistic head, under the same grouped 5-fold CV as slices 15-17:
| repr | tokenization | vocab |
|---|---|---|
| word | word 1-2 grams | 1,271 |
| char | Unicode char 3-5 grams | 8,132 |
| both | [word ; char] concatenated | 9,403 |
All fitted on the development corpus only, evaluated by grouped CV with the
existing cv_fold assignment. Hyperparameters fixed (C=1.0, TF-IDF
sublinear_tf, min_df=2) — no grid search, to report the floor.
2. Representation comparison, grouped CV (binary action gate)
| repr | ROC-AUC | PR-AUC | action_P | action_R | FA count | FA rate |
|---|---|---|---|---|---|---|
| word | 0.895 | 0.833 | 0.921 | 0.425 | 29 | 1.2% |
| char | 0.894 | 0.802 | 0.804 | 0.470 | 91 | 3.7% |
| both | 0.909 | 0.838 | 0.875 | 0.485 | 55 | 2.2% |
"both" is the best by PR-AUC and is used for every section below.
Fold variance (both, at 0.5)
fold ROC-AUC PR-AUC action_P action_R FP FN n
------------------------------------------------------------
0 0.962 0.903 0.893 0.347 6 94 476
1 0.926 0.946 0.926 0.687 11 63 341
2 0.936 0.942 1.000 0.367 0 188 707
3 0.901 0.699 0.699 0.674 25 28 506
4 0.879 0.675 0.705 0.456 13 37 460
Fold 2 and fold 3 are the hard ones, as in every slice: fold 3 (the capability / recall-heavy held-out split) sees most of the remaining 25 false actions.
3. Comparison against every e5 baseline (slice 17)
| model | extras | PR-AUC | action_P | action_R | FA rate |
|---|---|---|---|---|---|
| e5 binary linear | ~385 | 0.707 | 0.688 | 0.476 | 6.9% |
| e5 binary MLP H=32 | 12,353 | 0.692 | 0.673 | 0.569 | 8.8% |
| sparse word+char logistic | 9,403 (sparse) | 0.838 | 0.875 | 0.485 | 2.2% |
The sparse lexical head raises binary PR-AUC from 0.707 → 0.838 (e5 linear) and cuts the false-action rate from 6.9% to 2.2%, with comparable recall. On the aggregate action/non-action boundary that Maven guards, a cheap TF-IDF n-gram surface strictly dominates a frozen mean-pooled e5 vector.
4. Safety operating curve (both)
A threshold exists that clears P ≥ 0.95 with materially better recall than e5:
threshold action_P action_R FA count FA rate recall @ P>=0.95
-----------------------------------------------------------------
0.620 0.940 0.355 18 0.0072 0.000
0.655 0.935 0.325 18 0.0072 0.000
0.715 0.959 0.264 9 0.0036 ✓
0.730 0.966 0.247 7 0.0028 ✓
0.745 0.968 0.225 6 0.0024 ✓
...
0.955 1.000 0.004 0 0.0000 ✓
Best recall inside the P ≥ 0.95 region is 0.264 (FA 9, rate 0.36%). That is a real usable operating point for a strict gate — slice 17's e5 MLP had no threshold reaching P ≥ 0.95 at all.
5. Leave-generator-family-out (both, held-out family)
Train without a family, evaluate on that family. The families that exist in the development corpus:
| held-out family | rows | pos/neg | action_P | action_R | FA | accuracy |
|---|---|---|---|---|---|---|
| polite_request | 460 | 127/333 | 0.927 | 1.000 | 10 | 97.8% |
| modal_request | 223 | 223/0 | 1.000 | 0.852 | 0 | 85.2% |
| first_person_request | 791 | 223/568 | 0.995 | 0.906 | 1 | 97.2% |
| reordered_target | 332 | 96/236 | 1.000 | 1.000 | 0 | 100.0% |
| capability_question | 126 | 0/126 | — | — | 84 | 33.3% |
| question | 102 | 0/102 | — | — | 0 | 100.0% |
Request-form families generalize cleanly (0-10 FA). The capability_question
family collapses when held out: 84 of 126 (66.7%) fire as actions. That is the
semantic-pragmatics family, not a surface-form family. negation,
reported_speech, quotation, hypothetical have zero tagged rows in v2 dev,
so they cannot be held out here; they remain a coverage gap for a later corpus.
6. Paired action/capability ordering test (both, no leakage)
The task's core: rank an executable action above its semantically-identical capability-question sibling. Paired by shared object noun (device/entity) plus home domain:
pairs: 2268
ordering accuracy: 0.571 (chance = 0.5)
mean margin (act-cap): +0.074 (tiny)
median margin: +0.066
reversed pairs: 972
0.571 ordering accuracy is barely above chance. Sparse local features see the same verb-object n-grams in both members of a pair and cannot decide which one is executable. This is the specific weakness — the gate that passes the aggregate binary test (above) fails the pairwise pragmatics test.
Sample reversed pairs (capability question scored as more action-like than its executable sibling):
cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне' P=0.527
cap 'ты можешь выключить свет' P=0.726 < act 'выключить свет на кухне' P=0.361
cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне, пожалуйста' P=0.569
7. Punctuation ablation (both)
Training on punctuation-stripped text and evaluating both forms:
trained-stripped, eval punctuated: P=0.993 R=1.000 FA=6 (0.2%)
trained-stripped, eval stripped: P=0.993 R=1.000 FA=6 (0.2%)
Stripping punctuation before training removes the trailing ?/, noise and
cuts false actions from 55 to 6 with perfect recall. The gate does not need
punctuation cues — and under voice input (which the stress protocol simulates)
it must not rely on them. Stripped-input training is the better configuration.
8. Voice-like stress (both, model trained on all text)
all: n=874 capability-question FA 0.0% (42) modal action recall 1.000 (350)
no_final: n=132 capability-question FA 0.0% (21) modal action recall 0.000 (0)
Under the stress protocol the sparse gate keeps every capability question below threshold and every modal/polite action above it, on the in-distribution subset (the training set includes capability questions). This is consistent with the ablation: punctuation is not load-bearing. The honest generalization test remains section 5, where unseen capability questions do break.
9. Six-way probe (both, sparse logistic)
accuracy: 63.1%
macro F1: 0.470
action P: 0.531 action R: 0.923
FA rate: 26.1% (649)
Per-route F1:
action P=0.531 R=0.923 F1=0.674 (n=796)
conversation P=0.250 R=0.011 F1=0.021 (n=93)
knowledge P=0.755 R=0.678 F1=0.715 (n=715)
memory_write P=0.770 R=0.430 F1=0.552 (n=553)
system P=0.759 R=0.376 F1=0.503 (n=226)
uncertain P=0.667 R=0.243 F1=0.356 (n=107)
As a routing head sparse is worse than e5 (six-way macro F1 0.470 vs e5 linear 0.675) — the non-action routes need real semantics, which n-grams do not carry. The win is specific to the binary action gate, not to full routing.
10. Artifact size and latency
repr vocab fp32 model bytes vectorize+fit (2490 rows)
word 1,271 5,084 0.07s
char 8,132 32,528 0.10s
both 9,403 37,612 0.12s
A hashed-n-gram production form (fixed-width, no vocabulary growth, sub-50 kB weights) is clearly feasible; it was not implemented in Go for this slice.
11. Conclusion
Verdict: sparse lexical features are sufficient for the aggregate binary gate, insufficient for the pragmatics boundary
Three findings, one per decision rule in the brief:
-
Sparse is sufficient for the aggregate boundary. PR-AUC 0.838 vs e5's 0.707, false-action rate 2.2% vs 6.9%, and — unlike e5 — a real P ≥ 0.95 operating point with recall 0.264 (FA 9, 0.36%). A simple TF-IDF n-gram head over the raw text beats every frozen e5 head tested on the class that Maven actually guards. This is not e5 vs sparse being close; it is a large, reproducible margin.
-
It is not template leakage — it is semantic-family confusion. Held-out capability questions fail at 84/126 (66.7%), and the pairwise ordering test lands at 0.571 (chance). These are not distinct surface forms leaking into one another; the capability question and its executable sibling share the same verb-object n-grams verbatim. The residual false actions concentrate exactly there (capability_question + capability-ha/tool split groups dominate the decomposition). A representation that held on to per-family surface templates and nothing else would still collapse on these — the two members of each pair are surface-identical apart from the handful of politeness/modal tokens that the n-grams cannot learn to weigh.
-
A sequence encoder is justified for the pragmatics boundary. Order and the trailing politeness/modality are the deciding signal, and local n-grams demonstrably cannot rank them (0.571). The aggregate binary gate is a solved sub-problem that a cheap sparse head holds at FA 2.2%; the open question is whether an order-sensitive encoder separates capability questions from their executable siblings without losing that. That is now the measured, specific target for the next slice, and the paired-ordering test in section 6 is the metric to drive it.
Practical recommendation carried out of this slice: if a sparse gate ships, train it on punctuation-stripped text — it is strictly better (FA 6) and immune to the voice-stress artifact that capped the e5 MLP.
12. Commit hash for tooling
f2b65cd — cmd/semantic-router-experiment/slice18_sparse.py