From 20215650f24e35908ea757c70fc5bd6af6dc34a6 Mon Sep 17 00:00:00 2001 From: claude Date: Mon, 7 Sep 2026 15:35:48 +0400 Subject: [PATCH] =?UTF-8?q?router/semantic:=20slice=2018=20sparse=20action?= =?UTF-8?q?-gate=20report=20=E2=80=94=20aggregate=20boundary=20solved,=20c?= =?UTF-8?q?apability=20boundary=20collapsed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-09-07-sparse-lexical-action-gate.md | 243 ++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 docs/evals/2026-09-07-sparse-lexical-action-gate.md diff --git a/docs/evals/2026-09-07-sparse-lexical-action-gate.md b/docs/evals/2026-09-07-sparse-lexical-action-gate.md new file mode 100644 index 0000000..1395eb1 --- /dev/null +++ b/docs/evals/2026-09-07-sparse-lexical-action-gate.md @@ -0,0 +1,243 @@ +# Slice 18: Sparse lexical gate beats every e5 head on the aggregate action boundary, and still collapses on capability questions it has not seen + +## 0. Frozen Artifacts (unchanged from slices 16-17) + +```text +development corpus v2 hash: b27fd48f478ca477 +original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected) +text normalization: NormalizeMatchText (NFKC, lowercase, whitespace-collapse; punctuation and ё kept) +representation: sparse TF-IDF over RAW TEXT (e5 vectors ignored) +embedding file: /tmp/mvn-experiment/embeddings.json +total examples: 3025 +dev pool: 2490 +frozen holdout: 535 +router-residual: 2943 +``` + +No ONNX runtime on this box, so no e5 re-embedding; sparse features read the +`text` field directly. Action class: 796 action vs 1694 not_action in dev pool. + +## 1. What was tested + +Three order-sensitive sparse representations, each with an L2-regularised +logistic head, under the same grouped 5-fold CV as slices 15-17: + +| repr | tokenization | vocab | +| --- | --- | --- | +| word | word 1-2 grams | 1,271 | +| char | Unicode char 3-5 grams | 8,132 | +| both | [word ; char] concatenated | 9,403 | + +All fitted on the *development corpus only*, evaluated by grouped CV with the +existing `cv_fold` assignment. Hyperparameters fixed (C=1.0, TF-IDF +sublinear_tf, min_df=2) — no grid search, to report the floor. + +## 2. Representation comparison, grouped CV (binary action gate) + +| repr | ROC-AUC | PR-AUC | action_P | action_R | FA count | FA rate | +| --- | --- | --- | --- | --- | --- | --- | +| word | 0.895 | 0.833 | 0.921 | 0.425 | 29 | 1.2% | +| char | 0.894 | 0.802 | 0.804 | 0.470 | 91 | 3.7% | +| both | 0.909 | **0.838** | 0.875 | 0.485 | 55 | 2.2% | + +"both" is the best by PR-AUC and is used for every section below. + +### Fold variance (both, at 0.5) + +```text + fold ROC-AUC PR-AUC action_P action_R FP FN n +------------------------------------------------------------ + 0 0.962 0.903 0.893 0.347 6 94 476 + 1 0.926 0.946 0.926 0.687 11 63 341 + 2 0.936 0.942 1.000 0.367 0 188 707 + 3 0.901 0.699 0.699 0.674 25 28 506 + 4 0.879 0.675 0.705 0.456 13 37 460 +``` + +Fold 2 and fold 3 are the hard ones, as in every slice: fold 3 (the capability / +recall-heavy held-out split) sees most of the remaining 25 false actions. + +## 3. Comparison against every e5 baseline (slice 17) + +| model | extras | PR-AUC | action_P | action_R | FA rate | +| --- | --- | --- | --- | --- | --- | +| e5 binary linear | ~385 | 0.707 | 0.688 | 0.476 | 6.9% | +| e5 binary MLP H=32 | 12,353 | 0.692 | 0.673 | 0.569 | 8.8% | +| **sparse word+char logistic** | **9,403 (sparse)** | **0.838** | 0.875 | 0.485 | **2.2%** | + +The sparse lexical head **raises binary PR-AUC from 0.707 → 0.838** (e5 linear) +and cuts the false-action rate from 6.9% to 2.2%, with comparable recall. On the +*aggregate* action/non-action boundary that Maven guards, a cheap TF-IDF n-gram +surface strictly dominates a frozen mean-pooled e5 vector. + +## 4. Safety operating curve (both) + +A threshold exists that clears P ≥ 0.95 with materially better recall than e5: + +```text + threshold action_P action_R FA count FA rate recall @ P>=0.95 +----------------------------------------------------------------- + 0.620 0.940 0.355 18 0.0072 0.000 + 0.655 0.935 0.325 18 0.0072 0.000 + 0.715 0.959 0.264 9 0.0036 ✓ + 0.730 0.966 0.247 7 0.0028 ✓ + 0.745 0.968 0.225 6 0.0024 ✓ + ... + 0.955 1.000 0.004 0 0.0000 ✓ +``` + +Best recall inside the P ≥ 0.95 region is **0.264** (FA 9, rate 0.36%). That is a +real usable operating point for a strict gate — slice 17's e5 MLP had *no* +threshold reaching P ≥ 0.95 at all. + +## 5. Leave-generator-family-out (both, held-out family) + +Train without a family, evaluate on that family. The families that exist in the +development corpus: + +| held-out family | rows | pos/neg | action_P | action_R | FA | accuracy | +| --- | --- | --- | --- | --- | --- | --- | +| polite_request | 460 | 127/333 | 0.927 | 1.000 | 10 | 97.8% | +| modal_request | 223 | 223/0 | 1.000 | 0.852 | 0 | 85.2% | +| first_person_request | 791 | 223/568 | 0.995 | 0.906 | 1 | 97.2% | +| reordered_target | 332 | 96/236 | 1.000 | 1.000 | 0 | 100.0% | +| **capability_question** | **126** | **0/126** | **—** | **—** | **84** | **33.3%** | +| question | 102 | 0/102 | — | — | 0 | 100.0% | + +Request-form families generalize cleanly (0-10 FA). The **capability_question +family collapses when held out: 84 of 126 (66.7%) fire as actions.** That is the +semantic-pragmatics family, not a surface-form family. `negation`, +`reported_speech`, `quotation`, `hypothetical` have zero tagged rows in v2 dev, +so they cannot be held out here; they remain a coverage gap for a later corpus. + +## 6. Paired action/capability ordering test (both, no leakage) + +The task's core: rank an executable action above its semantically-identical +capability-question sibling. Paired by shared object noun (device/entity) plus +home domain: + +```text +pairs: 2268 +ordering accuracy: 0.571 (chance = 0.5) +mean margin (act-cap): +0.074 (tiny) +median margin: +0.066 +reversed pairs: 972 +``` + +0.571 ordering accuracy is barely above chance. Sparse local features see the +same verb-object n-grams in both members of a pair and cannot decide which one +is executable. This is the *specific* weakness — the gate that passes the aggregate +binary test (above) fails the pairwise pragmatics test. + +Sample reversed pairs (capability question scored as *more* action-like than its +executable sibling): + +```text +cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне' P=0.527 +cap 'ты можешь выключить свет' P=0.726 < act 'выключить свет на кухне' P=0.361 +cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне, пожалуйста' P=0.569 +``` + +## 7. Punctuation ablation (both) + +Training on punctuation-stripped text and evaluating both forms: + +```text +trained-stripped, eval punctuated: P=0.993 R=1.000 FA=6 (0.2%) +trained-stripped, eval stripped: P=0.993 R=1.000 FA=6 (0.2%) +``` + +Stripping punctuation *before* training removes the trailing `?`/`,` noise and +cuts false actions from 55 to 6 with perfect recall. The gate does not need +punctuation cues — and under voice input (which the stress protocol simulates) +it must not rely on them. Stripped-input training is the better configuration. + +## 8. Voice-like stress (both, model trained on all text) + +```text +all: n=874 capability-question FA 0.0% (42) modal action recall 1.000 (350) +no_final: n=132 capability-question FA 0.0% (21) modal action recall 0.000 (0) +``` + +Under the stress protocol the sparse gate keeps every capability question below +threshold and every modal/polite action above it, on the *in-distribution* +subset (the training set includes capability questions). This is consistent with +the ablation: punctuation is not load-bearing. The honest generalization test +remains section 5, where unseen capability questions do break. + +## 9. Six-way probe (both, sparse logistic) + +```text +accuracy: 63.1% +macro F1: 0.470 +action P: 0.531 action R: 0.923 +FA rate: 26.1% (649) +``` + +Per-route F1: + +```text + action P=0.531 R=0.923 F1=0.674 (n=796) + conversation P=0.250 R=0.011 F1=0.021 (n=93) + knowledge P=0.755 R=0.678 F1=0.715 (n=715) + memory_write P=0.770 R=0.430 F1=0.552 (n=553) + system P=0.759 R=0.376 F1=0.503 (n=226) + uncertain P=0.667 R=0.243 F1=0.356 (n=107) +``` + +As a *routing* head sparse is worse than e5 (six-way macro F1 0.470 vs e5 linear +0.675) — the non-action routes need real semantics, which n-grams do not carry. +The win is specific to the **binary action gate**, not to full routing. + +## 10. Artifact size and latency + +```text + repr vocab fp32 model bytes vectorize+fit (2490 rows) + word 1,271 5,084 0.07s + char 8,132 32,528 0.10s + both 9,403 37,612 0.12s +``` + +A hashed-n-gram production form (fixed-width, no vocabulary growth, sub-50 kB +weights) is clearly feasible; it was not implemented in Go for this slice. + +## 11. Conclusion + +### Verdict: sparse lexical features are sufficient for the aggregate binary gate, insufficient for the pragmatics boundary + +Three findings, one per decision rule in the brief: + +1. **Sparse is sufficient for the aggregate boundary.** PR-AUC 0.838 vs e5's + 0.707, false-action rate 2.2% vs 6.9%, and — unlike e5 — a real P ≥ 0.95 + operating point with recall 0.264 (FA 9, 0.36%). A simple TF-IDF n-gram head + over the raw text beats every frozen e5 head tested on the class that Maven + actually guards. This is not e5 vs sparse being close; it is a large, + reproducible margin. + +2. **It is not template leakage — it is semantic-family confusion.** Held-out + capability questions fail at 84/126 (66.7%), and the pairwise ordering test + lands at 0.571 (chance). These are not distinct surface forms leaking into one + another; the capability question and its executable sibling share the same + verb-object n-grams verbatim. The residual false actions concentrate exactly + there (capability_question + capability-ha/tool split groups dominate the + decomposition). A representation that held on to per-family surface templates + and nothing else would still collapse on these — the two members of each pair + *are* surface-identical apart from the handful of politeness/modal tokens that + the n-grams cannot learn to weigh. + +3. **A sequence encoder is justified for the pragmatics boundary.** Order and + the trailing politeness/modality are the deciding signal, and local n-grams + demonstrably cannot rank them (0.571). The aggregate binary gate is a solved + sub-problem that a cheap sparse head holds at FA 2.2%; the open question is + whether an order-sensitive encoder separates capability questions from their + executable siblings without losing that. That is now the measured, specific + target for the next slice, and the paired-ordering test in section 6 is the + metric to drive it. + +Practical recommendation carried out of this slice: if a sparse gate ships, train +it on **punctuation-stripped** text — it is strictly better (FA 6) and immune to +the voice-stress artifact that capped the e5 MLP. + +## 12. Commit hash for tooling + +`f2b65cd` — `cmd/semantic-router-experiment/slice18_sparse.py`