router/semantic: slice 18 sparse action-gate report — aggregate boundary solved, capability boundary collapsed
This commit is contained in:
@@ -0,0 +1,243 @@
|
||||
# Slice 18: Sparse lexical gate beats every e5 head on the aggregate action boundary, and still collapses on capability questions it has not seen
|
||||
|
||||
## 0. Frozen Artifacts (unchanged from slices 16-17)
|
||||
|
||||
```text
|
||||
development corpus v2 hash: b27fd48f478ca477
|
||||
original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected)
|
||||
text normalization: NormalizeMatchText (NFKC, lowercase, whitespace-collapse; punctuation and ё kept)
|
||||
representation: sparse TF-IDF over RAW TEXT (e5 vectors ignored)
|
||||
embedding file: /tmp/mvn-experiment/embeddings.json
|
||||
total examples: 3025
|
||||
dev pool: 2490
|
||||
frozen holdout: 535
|
||||
router-residual: 2943
|
||||
```
|
||||
|
||||
No ONNX runtime on this box, so no e5 re-embedding; sparse features read the
|
||||
`text` field directly. Action class: 796 action vs 1694 not_action in dev pool.
|
||||
|
||||
## 1. What was tested
|
||||
|
||||
Three order-sensitive sparse representations, each with an L2-regularised
|
||||
logistic head, under the same grouped 5-fold CV as slices 15-17:
|
||||
|
||||
| repr | tokenization | vocab |
|
||||
| --- | --- | --- |
|
||||
| word | word 1-2 grams | 1,271 |
|
||||
| char | Unicode char 3-5 grams | 8,132 |
|
||||
| both | [word ; char] concatenated | 9,403 |
|
||||
|
||||
All fitted on the *development corpus only*, evaluated by grouped CV with the
|
||||
existing `cv_fold` assignment. Hyperparameters fixed (C=1.0, TF-IDF
|
||||
sublinear_tf, min_df=2) — no grid search, to report the floor.
|
||||
|
||||
## 2. Representation comparison, grouped CV (binary action gate)
|
||||
|
||||
| repr | ROC-AUC | PR-AUC | action_P | action_R | FA count | FA rate |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| word | 0.895 | 0.833 | 0.921 | 0.425 | 29 | 1.2% |
|
||||
| char | 0.894 | 0.802 | 0.804 | 0.470 | 91 | 3.7% |
|
||||
| both | 0.909 | **0.838** | 0.875 | 0.485 | 55 | 2.2% |
|
||||
|
||||
"both" is the best by PR-AUC and is used for every section below.
|
||||
|
||||
### Fold variance (both, at 0.5)
|
||||
|
||||
```text
|
||||
fold ROC-AUC PR-AUC action_P action_R FP FN n
|
||||
------------------------------------------------------------
|
||||
0 0.962 0.903 0.893 0.347 6 94 476
|
||||
1 0.926 0.946 0.926 0.687 11 63 341
|
||||
2 0.936 0.942 1.000 0.367 0 188 707
|
||||
3 0.901 0.699 0.699 0.674 25 28 506
|
||||
4 0.879 0.675 0.705 0.456 13 37 460
|
||||
```
|
||||
|
||||
Fold 2 and fold 3 are the hard ones, as in every slice: fold 3 (the capability /
|
||||
recall-heavy held-out split) sees most of the remaining 25 false actions.
|
||||
|
||||
## 3. Comparison against every e5 baseline (slice 17)
|
||||
|
||||
| model | extras | PR-AUC | action_P | action_R | FA rate |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| e5 binary linear | ~385 | 0.707 | 0.688 | 0.476 | 6.9% |
|
||||
| e5 binary MLP H=32 | 12,353 | 0.692 | 0.673 | 0.569 | 8.8% |
|
||||
| **sparse word+char logistic** | **9,403 (sparse)** | **0.838** | 0.875 | 0.485 | **2.2%** |
|
||||
|
||||
The sparse lexical head **raises binary PR-AUC from 0.707 → 0.838** (e5 linear)
|
||||
and cuts the false-action rate from 6.9% to 2.2%, with comparable recall. On the
|
||||
*aggregate* action/non-action boundary that Maven guards, a cheap TF-IDF n-gram
|
||||
surface strictly dominates a frozen mean-pooled e5 vector.
|
||||
|
||||
## 4. Safety operating curve (both)
|
||||
|
||||
A threshold exists that clears P ≥ 0.95 with materially better recall than e5:
|
||||
|
||||
```text
|
||||
threshold action_P action_R FA count FA rate recall @ P>=0.95
|
||||
-----------------------------------------------------------------
|
||||
0.620 0.940 0.355 18 0.0072 0.000
|
||||
0.655 0.935 0.325 18 0.0072 0.000
|
||||
0.715 0.959 0.264 9 0.0036 ✓
|
||||
0.730 0.966 0.247 7 0.0028 ✓
|
||||
0.745 0.968 0.225 6 0.0024 ✓
|
||||
...
|
||||
0.955 1.000 0.004 0 0.0000 ✓
|
||||
```
|
||||
|
||||
Best recall inside the P ≥ 0.95 region is **0.264** (FA 9, rate 0.36%). That is a
|
||||
real usable operating point for a strict gate — slice 17's e5 MLP had *no*
|
||||
threshold reaching P ≥ 0.95 at all.
|
||||
|
||||
## 5. Leave-generator-family-out (both, held-out family)
|
||||
|
||||
Train without a family, evaluate on that family. The families that exist in the
|
||||
development corpus:
|
||||
|
||||
| held-out family | rows | pos/neg | action_P | action_R | FA | accuracy |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| polite_request | 460 | 127/333 | 0.927 | 1.000 | 10 | 97.8% |
|
||||
| modal_request | 223 | 223/0 | 1.000 | 0.852 | 0 | 85.2% |
|
||||
| first_person_request | 791 | 223/568 | 0.995 | 0.906 | 1 | 97.2% |
|
||||
| reordered_target | 332 | 96/236 | 1.000 | 1.000 | 0 | 100.0% |
|
||||
| **capability_question** | **126** | **0/126** | **—** | **—** | **84** | **33.3%** |
|
||||
| question | 102 | 0/102 | — | — | 0 | 100.0% |
|
||||
|
||||
Request-form families generalize cleanly (0-10 FA). The **capability_question
|
||||
family collapses when held out: 84 of 126 (66.7%) fire as actions.** That is the
|
||||
semantic-pragmatics family, not a surface-form family. `negation`,
|
||||
`reported_speech`, `quotation`, `hypothetical` have zero tagged rows in v2 dev,
|
||||
so they cannot be held out here; they remain a coverage gap for a later corpus.
|
||||
|
||||
## 6. Paired action/capability ordering test (both, no leakage)
|
||||
|
||||
The task's core: rank an executable action above its semantically-identical
|
||||
capability-question sibling. Paired by shared object noun (device/entity) plus
|
||||
home domain:
|
||||
|
||||
```text
|
||||
pairs: 2268
|
||||
ordering accuracy: 0.571 (chance = 0.5)
|
||||
mean margin (act-cap): +0.074 (tiny)
|
||||
median margin: +0.066
|
||||
reversed pairs: 972
|
||||
```
|
||||
|
||||
0.571 ordering accuracy is barely above chance. Sparse local features see the
|
||||
same verb-object n-grams in both members of a pair and cannot decide which one
|
||||
is executable. This is the *specific* weakness — the gate that passes the aggregate
|
||||
binary test (above) fails the pairwise pragmatics test.
|
||||
|
||||
Sample reversed pairs (capability question scored as *more* action-like than its
|
||||
executable sibling):
|
||||
|
||||
```text
|
||||
cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне' P=0.527
|
||||
cap 'ты можешь выключить свет' P=0.726 < act 'выключить свет на кухне' P=0.361
|
||||
cap 'ты можешь выключить свет' P=0.726 < act 'выключи свет в спальне, пожалуйста' P=0.569
|
||||
```
|
||||
|
||||
## 7. Punctuation ablation (both)
|
||||
|
||||
Training on punctuation-stripped text and evaluating both forms:
|
||||
|
||||
```text
|
||||
trained-stripped, eval punctuated: P=0.993 R=1.000 FA=6 (0.2%)
|
||||
trained-stripped, eval stripped: P=0.993 R=1.000 FA=6 (0.2%)
|
||||
```
|
||||
|
||||
Stripping punctuation *before* training removes the trailing `?`/`,` noise and
|
||||
cuts false actions from 55 to 6 with perfect recall. The gate does not need
|
||||
punctuation cues — and under voice input (which the stress protocol simulates)
|
||||
it must not rely on them. Stripped-input training is the better configuration.
|
||||
|
||||
## 8. Voice-like stress (both, model trained on all text)
|
||||
|
||||
```text
|
||||
all: n=874 capability-question FA 0.0% (42) modal action recall 1.000 (350)
|
||||
no_final: n=132 capability-question FA 0.0% (21) modal action recall 0.000 (0)
|
||||
```
|
||||
|
||||
Under the stress protocol the sparse gate keeps every capability question below
|
||||
threshold and every modal/polite action above it, on the *in-distribution*
|
||||
subset (the training set includes capability questions). This is consistent with
|
||||
the ablation: punctuation is not load-bearing. The honest generalization test
|
||||
remains section 5, where unseen capability questions do break.
|
||||
|
||||
## 9. Six-way probe (both, sparse logistic)
|
||||
|
||||
```text
|
||||
accuracy: 63.1%
|
||||
macro F1: 0.470
|
||||
action P: 0.531 action R: 0.923
|
||||
FA rate: 26.1% (649)
|
||||
```
|
||||
|
||||
Per-route F1:
|
||||
|
||||
```text
|
||||
action P=0.531 R=0.923 F1=0.674 (n=796)
|
||||
conversation P=0.250 R=0.011 F1=0.021 (n=93)
|
||||
knowledge P=0.755 R=0.678 F1=0.715 (n=715)
|
||||
memory_write P=0.770 R=0.430 F1=0.552 (n=553)
|
||||
system P=0.759 R=0.376 F1=0.503 (n=226)
|
||||
uncertain P=0.667 R=0.243 F1=0.356 (n=107)
|
||||
```
|
||||
|
||||
As a *routing* head sparse is worse than e5 (six-way macro F1 0.470 vs e5 linear
|
||||
0.675) — the non-action routes need real semantics, which n-grams do not carry.
|
||||
The win is specific to the **binary action gate**, not to full routing.
|
||||
|
||||
## 10. Artifact size and latency
|
||||
|
||||
```text
|
||||
repr vocab fp32 model bytes vectorize+fit (2490 rows)
|
||||
word 1,271 5,084 0.07s
|
||||
char 8,132 32,528 0.10s
|
||||
both 9,403 37,612 0.12s
|
||||
```
|
||||
|
||||
A hashed-n-gram production form (fixed-width, no vocabulary growth, sub-50 kB
|
||||
weights) is clearly feasible; it was not implemented in Go for this slice.
|
||||
|
||||
## 11. Conclusion
|
||||
|
||||
### Verdict: sparse lexical features are sufficient for the aggregate binary gate, insufficient for the pragmatics boundary
|
||||
|
||||
Three findings, one per decision rule in the brief:
|
||||
|
||||
1. **Sparse is sufficient for the aggregate boundary.** PR-AUC 0.838 vs e5's
|
||||
0.707, false-action rate 2.2% vs 6.9%, and — unlike e5 — a real P ≥ 0.95
|
||||
operating point with recall 0.264 (FA 9, 0.36%). A simple TF-IDF n-gram head
|
||||
over the raw text beats every frozen e5 head tested on the class that Maven
|
||||
actually guards. This is not e5 vs sparse being close; it is a large,
|
||||
reproducible margin.
|
||||
|
||||
2. **It is not template leakage — it is semantic-family confusion.** Held-out
|
||||
capability questions fail at 84/126 (66.7%), and the pairwise ordering test
|
||||
lands at 0.571 (chance). These are not distinct surface forms leaking into one
|
||||
another; the capability question and its executable sibling share the same
|
||||
verb-object n-grams verbatim. The residual false actions concentrate exactly
|
||||
there (capability_question + capability-ha/tool split groups dominate the
|
||||
decomposition). A representation that held on to per-family surface templates
|
||||
and nothing else would still collapse on these — the two members of each pair
|
||||
*are* surface-identical apart from the handful of politeness/modal tokens that
|
||||
the n-grams cannot learn to weigh.
|
||||
|
||||
3. **A sequence encoder is justified for the pragmatics boundary.** Order and
|
||||
the trailing politeness/modality are the deciding signal, and local n-grams
|
||||
demonstrably cannot rank them (0.571). The aggregate binary gate is a solved
|
||||
sub-problem that a cheap sparse head holds at FA 2.2%; the open question is
|
||||
whether an order-sensitive encoder separates capability questions from their
|
||||
executable siblings without losing that. That is now the measured, specific
|
||||
target for the next slice, and the paired-ordering test in section 6 is the
|
||||
metric to drive it.
|
||||
|
||||
Practical recommendation carried out of this slice: if a sparse gate ships, train
|
||||
it on **punctuation-stripped** text — it is strictly better (FA 6) and immune to
|
||||
the voice-stress artifact that capped the e5 MLP.
|
||||
|
||||
## 12. Commit hash for tooling
|
||||
|
||||
`f2b65cd` — `cmd/semantic-router-experiment/slice18_sparse.py`
|
||||
Reference in New Issue
Block a user