# Slice 18: Sparse lexical gate beats every e5 head on the aggregate action boundary, and still collapses on capability questions it has not seen ## 0. Frozen Artifacts (unchanged from slices 16-17) ```text development corpus v2 hash: b27fd48f478ca477 original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected) text normalization: NormalizeMatchText (NFKC, lowercase, whitespace-collapse; punctuation and ё kept) representation: sparse TF-IDF over RAW TEXT (e5 vectors ignored) embedding file: /tmp/mvn-experiment/embeddings.json total examples: 3025 dev pool: 2490 frozen holdout: 535 router-residual: 2943 ``` No ONNX runtime on this box, so no e5 re-embedding; sparse features read the `text` field directly. Action class: 796 action vs 1694 not_action in dev pool. ## 1. What was tested Three order-sensitive sparse representations, each with an L2-regularised logistic head, under the same grouped 5-fold CV as slices 15-17: | repr | tokenization | vocab | | --- | --- | --- | | word | word 1-2 grams | 1,271 | | char | Unicode char 3-5 grams | 8,132 | | both | [word ; char] concatenated | 9,403 | All fitted on the *development corpus only*, evaluated by grouped CV with the existing `cv_fold` assignment. Hyperparameters fixed (C=1.0, TF-IDF sublinear_tf, min_df=2) — no grid search, to report the floor. ## 2. Representation comparison, grouped CV (binary action gate) | repr | ROC-AUC | PR-AUC | action_P | action_R | FA count | FA rate | | --- | --- | --- | --- | --- | --- | --- | | word | 0.895 | 0.833 | 0.921 | 0.425 | 29 | 1.2% | | char | 0.894 | 0.802 | 0.804 | 0.470 | 91 | 3.7% | | both | 0.909 | **0.838** | 0.875 | 0.485 | 55 | 2.2% | "both" is the best by PR-AUC and is used for every section below. ### Fold variance (both, at 0.5) ```text fold ROC-AUC PR-AUC action_P action_R FP FN n ------------------------------------------------------------ 0 0.962 0.903 0.893 0.347 6 94 476 1 0.926 0.946 0.926 0.687 11 63 341 2 0.936 0.942 1.000 0.367 0 188 707 3 0.901 0.699 0.699 0.674 25 28 506 4 0.879 0.675 0.705 0.456 13 37 460 ``` Fold 2 and fold 3 are the hard ones, as in every slice: fold 3 (the capability / recall-heavy held-out split) sees most of the remaining 25 false actions. ## 3. Comparison against every e5 baseline (slice 17) | model | extras | PR-AUC | action_P | action_R | FA rate | | --- | --- | --- | --- | --- | --- | | e5 binary linear | ~385 | 0.707 | 0.688 | 0.476 | 6.9% | | e5 binary MLP H=32 | 12,353 | 0.692 | 0.673 | 0.569 | 8.8% | | **sparse word+char logistic** | **9,403 (sparse)** | **0.838** | 0.875 | 0.485 | **2.2%** | The sparse lexical head **raises binary PR-AUC from 0.707 → 0.838** (e5 linear) and cuts the false-action rate from 6.9% to 2.2%, with comparable recall. On the *aggregate* action/non-action boundary that Maven guards, a cheap TF-IDF n-gram surface strictly dominates a frozen mean-pooled e5 vector. ## 4. Safety operating curve (both) A threshold exists that clears P ≥ 0.95 with materially better recall than e5: ```text threshold action_P action_R FA count FA rate recall @ P>=0.95 ----------------------------------------------------------------- 0.620 0.940 0.355 18 0.0072 0.000 0.655 0.935 0.325 18 0.0072 0.000 0.715 0.959 0.264 9 0.0036 ✓ 0.730 0.966 0.247 7 0.0028 ✓ 0.745 0.968 0.225 6 0.0024 ✓ ... 0.955 1.000 0.004 0 0.0000 ✓ ``` Best recall inside the P ≥ 0.95 region is **0.264** (FA 9, rate 0.36%). That is a real usable operating point for a strict gate — slice 17's e5 MLP had *no* threshold reaching P ≥ 0.95 at all. ## 5. Leave-generator-family-out (both, held-out family) Train without a family, evaluate on that family. The families that exist in the development corpus: | held-out family | rows | pos/neg | action_P | action_R | FA | accuracy | | --- | --- | --- | --- | --- | --- | --- | | polite_request | 460 | 127/333 | 0.927 | 1.000 | 10 | 97.8% | | modal_request | 223 | 223/0 | 1.000 | 0.852 | 0 | 85.2% | | first_person_request | 791 | 223/568 | 0.995 | 0.906 | 1 | 97.2% | | reordered_target | 332 | 96/236 | 1.000 | 1.000 | 0 | 100.0% | | **capability_question** | **126** | **0/126** | **—** | **—** | **84** | **33.3%** | | question | 102 | 0/102 | — | — | 0 | 100.0% | Request-form families generalize cleanly (0-10 FA). The **capability_question family collapses when held out: 84 of 126 (66.7%) fire as actions.** That is the semantic-pragmatics family, not a surface-form family. `negation`, `reported_speech`, `quotation`, `hypothetical` have zero tagged rows in v2 dev, so they cannot be held out here; they remain a coverage gap for a later corpus. ## 6. Paired action/capability ordering test (both, no leakage) The task's core: rank an executable action above its semantically-identical capability-question sibling. Paired by shared object noun (device/entity) plus home domain: ```text pairs: 2268 ordering accuracy: 0.571 (chance = 0.5) mean margin (act-cap): +0.074 (tiny) median margin: +0.066 reversed pairs: 972 ``` 0.571 ordering accuracy is barely above chance. Sparse local features see the same verb-object n-grams in both members of a pair and cannot decide which one is executable. This is the *specific* weakness — the gate that passes the aggregate binary test (above) fails the pairwise pragmatics test. Sample reversed pairs (capability question scored as *more* action-like than its executable sibling): ```text cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне' P=0.527 cap 'ты можешь выключить свет' P=0.726 < act 'выключить свет на кухне' P=0.361 cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне, пожалуйста' P=0.569 ``` ## 7. Punctuation ablation (both) Training on punctuation-stripped text and evaluating both forms: ```text trained-stripped, eval punctuated: P=0.993 R=1.000 FA=6 (0.2%) trained-stripped, eval stripped: P=0.993 R=1.000 FA=6 (0.2%) ``` Stripping punctuation *before* training removes the trailing `?`/`,` noise and cuts false actions from 55 to 6 with perfect recall. The gate does not need punctuation cues — and under voice input (which the stress protocol simulates) it must not rely on them. Stripped-input training is the better configuration. ## 8. Voice-like stress (both, model trained on all text) ```text all: n=874 capability-question FA 0.0% (42) modal action recall 1.000 (350) no_final: n=132 capability-question FA 0.0% (21) modal action recall 0.000 (0) ``` Under the stress protocol the sparse gate keeps every capability question below threshold and every modal/polite action above it, on the *in-distribution* subset (the training set includes capability questions). This is consistent with the ablation: punctuation is not load-bearing. The honest generalization test remains section 5, where unseen capability questions do break. ## 9. Six-way probe (both, sparse logistic) ```text accuracy: 63.1% macro F1: 0.470 action P: 0.531 action R: 0.923 FA rate: 26.1% (649) ``` Per-route F1: ```text action P=0.531 R=0.923 F1=0.674 (n=796) conversation P=0.250 R=0.011 F1=0.021 (n=93) knowledge P=0.755 R=0.678 F1=0.715 (n=715) memory_write P=0.770 R=0.430 F1=0.552 (n=553) system P=0.759 R=0.376 F1=0.503 (n=226) uncertain P=0.667 R=0.243 F1=0.356 (n=107) ``` As a *routing* head sparse is worse than e5 (six-way macro F1 0.470 vs e5 linear 0.675) — the non-action routes need real semantics, which n-grams do not carry. The win is specific to the **binary action gate**, not to full routing. ## 10. Artifact size and latency ```text repr vocab fp32 model bytes vectorize+fit (2490 rows) word 1,271 5,084 0.07s char 8,132 32,528 0.10s both 9,403 37,612 0.12s ``` A hashed-n-gram production form (fixed-width, no vocabulary growth, sub-50 kB weights) is clearly feasible; it was not implemented in Go for this slice. ## 11. Conclusion ### Verdict: sparse lexical features are sufficient for the aggregate binary gate, insufficient for the pragmatics boundary Three findings, one per decision rule in the brief: 1. **Sparse is sufficient for the aggregate boundary.** PR-AUC 0.838 vs e5's 0.707, false-action rate 2.2% vs 6.9%, and — unlike e5 — a real P ≥ 0.95 operating point with recall 0.264 (FA 9, 0.36%). A simple TF-IDF n-gram head over the raw text beats every frozen e5 head tested on the class that Maven actually guards. This is not e5 vs sparse being close; it is a large, reproducible margin. 2. **It is not template leakage — it is semantic-family confusion.** Held-out capability questions fail at 84/126 (66.7%), and the pairwise ordering test lands at 0.571 (chance). These are not distinct surface forms leaking into one another; the capability question and its executable sibling share the same verb-object n-grams verbatim. The residual false actions concentrate exactly there (capability_question + capability-ha/tool split groups dominate the decomposition). A representation that held on to per-family surface templates and nothing else would still collapse on these — the two members of each pair *are* surface-identical apart from the handful of politeness/modal tokens that the n-grams cannot learn to weigh. 3. **A sequence encoder is justified for the pragmatics boundary.** Order and the trailing politeness/modality are the deciding signal, and local n-grams demonstrably cannot rank them (0.571). The aggregate binary gate is a solved sub-problem that a cheap sparse head holds at FA 2.2%; the open question is whether an order-sensitive encoder separates capability questions from their executable siblings without losing that. That is now the measured, specific target for the next slice, and the paired-ordering test in section 6 is the metric to drive it. Practical recommendation carried out of this slice: if a sparse gate ships, train it on **punctuation-stripped** text — it is strictly better (FA 6) and immune to the voice-stress artifact that capped the e5 MLP. ## 12. Commit hash for tooling `f2b65cd` — `cmd/semantic-router-experiment/slice18_sparse.py`