kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

router/semantic: slice 17 nonlinear e5 MLP probe — action gate experiment report

Browse Source
This commit is contained in:
claude
2026-09-07 15:23:46 +04:00
parent e80d45f4b3
commit 9a25945567
1 changed files with 333 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evals/2026-09-07-nonlinear-e5-mlp-probe.md
+333
View File
@@ -0,0 +1,333 @@
# Slice 17: Nonlinear MLP Probe — Action Gate Experiment
## 0. Frozen Artifacts from Slice 16
```text
development corpus v2 hash: b27fd48f478ca477
original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected)
embedder: model_quantized@384/tok2
dimension: 384
pooling: mean-pool + L2-normalize
input template: query: <text>
embedding file: /tmp/mvn-experiment/embeddings.json
total examples: 3025
dev pool: 2490
frozen holdout: 535
router-residual: 2943
```
## 1. Hidden Sizes and Exact Parameter Counts
### Binary action gate: e5[384] → Linear(384→H) → ReLU → Linear(H→1)
H params fp32 bytes int8 bytes
----------------------------------------
8 3,089 12,356 3,089
16 6,177 24,708 6,177
32 12,353 49,412 12,353
64 24,705 98,820 24,705
### Six-way MLP: e5[384] → Linear(384→H) → ReLU → Linear(H→6)
H params fp32 bytes int8 bytes
----------------------------------------
8 3,134 12,536 3,134
16 6,262 25,048 6,262
32 12,518 50,072 12,518
64 25,030 100,120 25,030
## 2. Selected Regularization
Best binary MLP: H=32, weight_decay=0.01
Selected by grouped development CV PR-AUC.
Grid results (binary MLP):
H wd PR-AUC ROC-AUC action_P action_R FA count FA rate
--------------------------------------------------------------------------------
16 0.0 0.422 0.628 0.139 0.215 155 6.2%
16 0.0001 0.422 0.628 0.139 0.215 155 6.2%
16 0.001 0.423 0.628 0.139 0.215 155 6.2%
16 0.01 0.423 0.628 0.139 0.215 155 6.2%
32 0.0 0.691 0.854 0.686 0.610 219 8.8%
32 0.0001 0.691 0.854 0.686 0.610 219 8.8%
32 0.001 0.691 0.854 0.686 0.611 219 8.8%
32 0.01 0.692 0.854 0.686 0.605 220 8.8% *
64 0.0 0.547 0.714 0.309 0.381 195 7.8%
64 0.0001 0.547 0.714 0.309 0.381 195 7.8%
64 0.001 0.547 0.714 0.309 0.381 195 7.8%
64 0.01 0.545 0.713 0.308 0.368 194 7.8%
## 3. Binary MLP OOF Metrics (best: H=32, wd=0.01)
```text
ROC-AUC: 0.854
PR-AUC: 0.692
action precision: 0.686
action recall: 0.605
false-positive: 220
false-negative: 343
false-action rate: 8.8%
total: 2490
mean iters: 64
```
## 4. Fold Variance
Binary MLP (H=32, wd=0.01):
fold ROC-AUC PR-AUC action_P action_R FP FN n
------------------------------------------------------------
0 0.960 0.915 0.935 0.701 7 43 476
1 0.896 0.917 0.840 0.861 33 28 341
2 0.953 0.939 0.988 0.286 1 212 707
3 0.825 0.454 0.440 0.767 84 20 506
4 0.638 0.236 0.228 0.412 95 40 460
Fold variance comparison with linear binary probe:
fold linear FP MLP FP linear FA% MLP FA%
--------------------------------------------------
0 2 7 0.4% 1.5%
1 12 33 3.5% 9.7%
2 0 1 0.0% 0.1%
3 71 84 14.0% 16.6%
4 87 95 18.9% 20.7%
## 5. Safety Operating Curve (best binary MLP)
threshold action_P action_R FA count FA rate coverage
-----------------------------------------------------------------
0.300 0.6224 0.7827 378 0.1518 0.9305
0.325 0.6324 0.7563 350 0.1406 0.9221
0.350 0.6400 0.7349 329 0.1321 0.9153
0.375 0.6462 0.6997 305 0.1225 0.9040
0.400 0.6536 0.6709 283 0.1137 0.8948
0.425 0.6540 0.6482 273 0.1096 0.8876
0.450 0.6631 0.6231 252 0.1012 0.8795
0.475 0.6714 0.5930 231 0.0928 0.8699
0.500 0.6731 0.5691 220 0.0884 0.8622
0.525 0.6801 0.5503 206 0.0827 0.8562
0.550 0.6866 0.5339 194 0.0779 0.8510
0.575 0.6887 0.5113 184 0.0739 0.8438
0.600 0.6968 0.4849 168 0.0675 0.8353
0.625 0.6910 0.4523 161 0.0647 0.8249
0.650 0.6914 0.4334 154 0.0618 0.8189
0.675 0.6930 0.4083 144 0.0578 0.8108
0.700 0.6856 0.3781 138 0.0554 0.8012
0.725 0.6887 0.3530 127 0.0510 0.7932
0.750 0.6870 0.3254 118 0.0474 0.7843
0.775 0.7071 0.3003 99 0.0398 0.7763
0.800 0.7204 0.2751 85 0.0341 0.7683
0.825 0.7159 0.2374 75 0.0301 0.7562
0.850 0.7422 0.2098 58 0.0233 0.7474
0.875 0.7391 0.1709 48 0.0193 0.7349
0.900 0.7518 0.1332 35 0.0141 0.7229
0.925 0.7684 0.0917 22 0.0088 0.7096
0.950 0.7778 0.0528 12 0.0048 0.6972
No threshold achieves action_P ≥ 0.95.
## 6. Capability-Question Boundary Diagnostic
capability_question → false action rate: 91.3% (115/126)
positive modal request → action recall: 0.569 (796 examples)
### Paired capability-question vs action examples
These are the critical diagnostic pairs:
text route predicted action_P FA?
-----------------------------------------------------------------------------------------------
ты можешь выключить свет knowledge action 0.8159 YES
ты можешь выключить жалюзи knowledge action 0.9752 YES
ты можешь выключить вытяжку knowledge action 0.9720 YES
ты можешь выключить вентилятор knowledge action 0.9041 YES
умеешь ли включить свет knowledge action 0.5618 YES
умеешь ли включить жалюзи knowledge action 0.9195 YES
умеешь ли включить вытяжку knowledge action 0.8956 YES
умеешь ли включить вентилятор knowledge action 0.6122 YES
сможешь открыть свет knowledge action 0.7012 YES
сможешь открыть жалюзи knowledge action 0.9633 YES
сможешь открыть вытяжку knowledge action 0.9596 YES
сможешь открыть вентилятор knowledge action 0.8491 YES
ты можешь выключить свет, пожалуйста knowledge action 0.8682 YES
ты можешь выключить жалюзи, пожалуйста knowledge action 0.9778 YES
ты можешь выключить вытяжку, пожалуйста knowledge action 0.9718 YES
...
перезапусти контейнер maven action action 0.8106
перезапусти контейнер nexus action action 0.8002
перезапусти контейнер redis action action 0.6867
перезапусти контейнер postgres action action 0.6016
перезагрузи контейнер maven action action 0.6810
перезагрузи контейнер nexus action action 0.6192
перезагрузи контейнер redis action action 0.5051
перезагрузи контейнер postgres action not_action 0.4449
перезапусти контейнер maven, пожалуйста action action 0.8442
перезапусти контейнер nexus, пожалуйста action action 0.8672
## 7. Voice-Like Stress Results
```text
Total stress-testable pairs: 965
question with ?: 123
question without ?: 242
positive polite/modal action: 350
capability-question pairs: 42
```
Sample affected pairs:
act-stop-091:
original: "останови сервис, пожалуйста"
stressed: "останови сервис пожалуйста"
route: action
act-stop-092:
original: "останови nginx, пожалуйста"
stressed: "останови nginx пожалуйста"
route: action
act-stop-093:
original: "останови bотоbackup, пожалуйста"
stressed: "останови bотоbackup пожалуйста"
route: action
act-stop-094:
original: "выключи сервис, пожалуйста"
stressed: "выключи сервис пожалуйста"
route: action
act-stop-095:
original: "выключи nginx, пожалуйста"
stressed: "выключи nginx пожалуйста"
route: action
act-stop-096:
original: "выключи bотоbackup, пожалуйста"
stressed: "выключи bотоbackup пожалуйста"
route: action
act-stop-097:
original: "можешь останови сервис, пожалуйста"
stressed: "можешь останови сервис пожалуйста"
route: action
act-stop-098:
original: "can you останови сервис, please"
stressed: "can you останови сервис please"
route: action
act-stop-099:
original: "можешь останови nginx, пожалуйста"
stressed: "можешь останови nginx пожалуйста"
route: action
act-stop-100:
original: "can you останови nginx, please"
stressed: "can you останови nginx please"
route: action
Impact assessment:
The MLP's decision boundary must not depend on punctuation artifacts.
If MLP success depends on '?' presence, it will fail under voice input.
## 8. Six-Way MLP Results
Best six-way MLP: H=64, wd=0.001
```text
accuracy: 68.7%
macro F1: 0.681
action precision: 0.639
action recall: 0.830
false-action rate: 15.0%
false-action count: 374
```
Per-route F1:
action P=0.639 R=0.830 F1=0.722 (n=796)
conversation P=0.893 R=0.720 F1=0.798 (n=93)
knowledge P=0.735 R=0.634 F1=0.681 (n=715)
memory_write P=0.743 R=0.591 F1=0.659 (n=553)
system P=0.646 R=0.597 F1=0.621 (n=226)
uncertain P=0.583 R=0.626 F1=0.604 (n=107)
Confusion matrix (rows=expected, cols=predicted):
action conversation knowledge memory_write system uncertain
action 661 3 67 43 7 15
conversation 0 67 9 3 5 9
knowledge 181 0 453 47 33 1
memory_write 132 0 48 327 29 17
system 37 3 34 11 135 6
uncertain 24 2 5 9 0 67
Fold variance (six-way MLP):
fold accuracy macro_f1 action_P action_R FA
-------------------------------------------------------
0 82.1% 0.774 0.871 0.938 20
1 81.5% 0.625 0.838 0.925 36
2 64.2% 0.580 0.722 0.771 88
3 69.0% 0.687 0.425 0.895 104
4 51.7% 0.452 0.212 0.500 126
## 9. Comparison Against All Linear Baselines
| model | params beyond e5 | action P | action R | FA rate | macro F1 |
| ---------------------------- | ------------------ | ---------- | ---------- | ---------- | ---------- |
| 6-way linear | 2,310 | 0.629 | 0.832 | 15.7% | 0.675 |
| binary linear | ~385 | 0.688 | 0.476 | 6.9% | — |
| structural linear | ~2,313 | 0.610 | 0.805 | 15.1% | 0.622 |
| binary MLP (H=32) | 12,353 | 0.673 | 0.569 | 8.8% | — |
| 6-way MLP (H=64) | 25,030 | 0.639 | 0.830 | 15.0% | 0.681 |
## 10. Conclusion
### Verdict: e5 representation inadequate for the action-pragmatics boundary
The tiny nonlinear MLP provides only marginal improvement over linear baselines
and fails the capability-question diagnostic and the safety operating goal.
1. **Binary MLP does not beat binary linear.** PR-AUC 0.692 (MLP, H=32) vs 0.707
(linear). The MLP trades precision for recall (P=0.673/R=0.569) and produces
*more* false actions (220 vs 172) than the linear probe. The nonlinearity does
not unlock a separability that the linear surface was missing — both are pinned
by the same representation.
2. **Capability-question boundary is catastrophically broken for both.** The MLP
fires action on **91.3% (115/126)** of capability questions — the primary
diagnostic fails outright. The linear probe reaches the same place (112/126).
A nonlinear head cannot separate "ты можешь выключить свет?" (knowledge) from
"можешь выключить свет, пожалуйста" (action) because the frozen mean-pooled e5
vector places them on top of each other.
3. **No safe operating region exists.** The MLP never reaches action precision
≥ 0.95 with materially non-zero recall. The best it can do is P=0.778 at
R=0.053 (essentially zero recall). At any threshold that keeps FA count low,
it becomes useless; at any threshold that keeps recall useful, FA rate climbs
past 10%.
4. **Six-way MLP is essentially flat.** F1=0.681 (H=64) vs 0.675 (linear six-way);
FA rate 15.0% vs 15.7%. Per-route F1 is within noise of the linear head. The bad
folds from slice 16 (fold 3 P=0.425, fold 4 P=0.212) are not repaired.
Per the brief's decision rule this is the third case:
> mean-pooled e5 representation is inadequate for Maven's action-pragmatics
> boundary. at that point stop probing e5.
**Stop probing the frozen mean-pooled e5-small representation.** The signal is
not present and not merely nonlinear. A fundamentally different representation or
encoder is required for the action/capability-question pragmatics boundary.
### Capability-question diagnostic
capability_question false-action rate (MLP): 91.3% (115/126)
capability_question false-action rate (linear): 88.9% (112/126)
positive modal action recall (MLP): 0.569
positive modal action recall (linear): 0.476
The MLP does not materially improve the capability-question boundary.
### Six-way MLP vs binary gate
Six-way MLP (F1=0.681) and six-way linear (F1=0.675) are statistically
indistinguishable. Neither the six-way formulation nor the nonlinearity fixes the
boundary. The architecture evidence does not support routing through a single
learned head for Maven's action pragmatics as currently embedded.
## 11. Commit hash for diagnostic tooling
`e80d45f` (slice 17 MLP probe tooling: `cmd/semantic-router-experiment/slice17_mlp.py`)