diff --git a/docs/evals/2026-09-07-nonlinear-e5-mlp-probe.md b/docs/evals/2026-09-07-nonlinear-e5-mlp-probe.md new file mode 100644 index 0000000..b2de769 --- /dev/null +++ b/docs/evals/2026-09-07-nonlinear-e5-mlp-probe.md @@ -0,0 +1,333 @@ +# Slice 17: Nonlinear MLP Probe — Action Gate Experiment + +## 0. Frozen Artifacts from Slice 16 + +```text +development corpus v2 hash: b27fd48f478ca477 +original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected) +embedder: model_quantized@384/tok2 +dimension: 384 +pooling: mean-pool + L2-normalize +input template: query: +embedding file: /tmp/mvn-experiment/embeddings.json +total examples: 3025 +dev pool: 2490 +frozen holdout: 535 +router-residual: 2943 +``` + +## 1. Hidden Sizes and Exact Parameter Counts + +### Binary action gate: e5[384] → Linear(384→H) → ReLU → Linear(H→1) + + H params fp32 bytes int8 bytes +---------------------------------------- + 8 3,089 12,356 3,089 + 16 6,177 24,708 6,177 + 32 12,353 49,412 12,353 + 64 24,705 98,820 24,705 + +### Six-way MLP: e5[384] → Linear(384→H) → ReLU → Linear(H→6) + + H params fp32 bytes int8 bytes +---------------------------------------- + 8 3,134 12,536 3,134 + 16 6,262 25,048 6,262 + 32 12,518 50,072 12,518 + 64 25,030 100,120 25,030 + +## 2. Selected Regularization + +Best binary MLP: H=32, weight_decay=0.01 +Selected by grouped development CV PR-AUC. + +Grid results (binary MLP): + + H wd PR-AUC ROC-AUC action_P action_R FA count FA rate +-------------------------------------------------------------------------------- + 16 0.0 0.422 0.628 0.139 0.215 155 6.2% + 16 0.0001 0.422 0.628 0.139 0.215 155 6.2% + 16 0.001 0.423 0.628 0.139 0.215 155 6.2% + 16 0.01 0.423 0.628 0.139 0.215 155 6.2% + 32 0.0 0.691 0.854 0.686 0.610 219 8.8% + 32 0.0001 0.691 0.854 0.686 0.610 219 8.8% + 32 0.001 0.691 0.854 0.686 0.611 219 8.8% + 32 0.01 0.692 0.854 0.686 0.605 220 8.8% * + 64 0.0 0.547 0.714 0.309 0.381 195 7.8% + 64 0.0001 0.547 0.714 0.309 0.381 195 7.8% + 64 0.001 0.547 0.714 0.309 0.381 195 7.8% + 64 0.01 0.545 0.713 0.308 0.368 194 7.8% + +## 3. Binary MLP OOF Metrics (best: H=32, wd=0.01) + +```text +ROC-AUC: 0.854 +PR-AUC: 0.692 +action precision: 0.686 +action recall: 0.605 +false-positive: 220 +false-negative: 343 +false-action rate: 8.8% +total: 2490 +mean iters: 64 +``` + +## 4. Fold Variance + +Binary MLP (H=32, wd=0.01): + + fold ROC-AUC PR-AUC action_P action_R FP FN n +------------------------------------------------------------ + 0 0.960 0.915 0.935 0.701 7 43 476 + 1 0.896 0.917 0.840 0.861 33 28 341 + 2 0.953 0.939 0.988 0.286 1 212 707 + 3 0.825 0.454 0.440 0.767 84 20 506 + 4 0.638 0.236 0.228 0.412 95 40 460 + +Fold variance comparison with linear binary probe: + fold linear FP MLP FP linear FA% MLP FA% +-------------------------------------------------- + 0 2 7 0.4% 1.5% + 1 12 33 3.5% 9.7% + 2 0 1 0.0% 0.1% + 3 71 84 14.0% 16.6% + 4 87 95 18.9% 20.7% + +## 5. Safety Operating Curve (best binary MLP) + + threshold action_P action_R FA count FA rate coverage +----------------------------------------------------------------- + 0.300 0.6224 0.7827 378 0.1518 0.9305 + 0.325 0.6324 0.7563 350 0.1406 0.9221 + 0.350 0.6400 0.7349 329 0.1321 0.9153 + 0.375 0.6462 0.6997 305 0.1225 0.9040 + 0.400 0.6536 0.6709 283 0.1137 0.8948 + 0.425 0.6540 0.6482 273 0.1096 0.8876 + 0.450 0.6631 0.6231 252 0.1012 0.8795 + 0.475 0.6714 0.5930 231 0.0928 0.8699 + 0.500 0.6731 0.5691 220 0.0884 0.8622 + 0.525 0.6801 0.5503 206 0.0827 0.8562 + 0.550 0.6866 0.5339 194 0.0779 0.8510 + 0.575 0.6887 0.5113 184 0.0739 0.8438 + 0.600 0.6968 0.4849 168 0.0675 0.8353 + 0.625 0.6910 0.4523 161 0.0647 0.8249 + 0.650 0.6914 0.4334 154 0.0618 0.8189 + 0.675 0.6930 0.4083 144 0.0578 0.8108 + 0.700 0.6856 0.3781 138 0.0554 0.8012 + 0.725 0.6887 0.3530 127 0.0510 0.7932 + 0.750 0.6870 0.3254 118 0.0474 0.7843 + 0.775 0.7071 0.3003 99 0.0398 0.7763 + 0.800 0.7204 0.2751 85 0.0341 0.7683 + 0.825 0.7159 0.2374 75 0.0301 0.7562 + 0.850 0.7422 0.2098 58 0.0233 0.7474 + 0.875 0.7391 0.1709 48 0.0193 0.7349 + 0.900 0.7518 0.1332 35 0.0141 0.7229 + 0.925 0.7684 0.0917 22 0.0088 0.7096 + 0.950 0.7778 0.0528 12 0.0048 0.6972 + +No threshold achieves action_P ≥ 0.95. + +## 6. Capability-Question Boundary Diagnostic + +capability_question → false action rate: 91.3% (115/126) +positive modal request → action recall: 0.569 (796 examples) + +### Paired capability-question vs action examples + +These are the critical diagnostic pairs: + +text route predicted action_P FA? +----------------------------------------------------------------------------------------------- +ты можешь выключить свет knowledge action 0.8159 YES +ты можешь выключить жалюзи knowledge action 0.9752 YES +ты можешь выключить вытяжку knowledge action 0.9720 YES +ты можешь выключить вентилятор knowledge action 0.9041 YES +умеешь ли включить свет knowledge action 0.5618 YES +умеешь ли включить жалюзи knowledge action 0.9195 YES +умеешь ли включить вытяжку knowledge action 0.8956 YES +умеешь ли включить вентилятор knowledge action 0.6122 YES +сможешь открыть свет knowledge action 0.7012 YES +сможешь открыть жалюзи knowledge action 0.9633 YES +сможешь открыть вытяжку knowledge action 0.9596 YES +сможешь открыть вентилятор knowledge action 0.8491 YES +ты можешь выключить свет, пожалуйста knowledge action 0.8682 YES +ты можешь выключить жалюзи, пожалуйста knowledge action 0.9778 YES +ты можешь выключить вытяжку, пожалуйста knowledge action 0.9718 YES +... +перезапусти контейнер maven action action 0.8106 +перезапусти контейнер nexus action action 0.8002 +перезапусти контейнер redis action action 0.6867 +перезапусти контейнер postgres action action 0.6016 +перезагрузи контейнер maven action action 0.6810 +перезагрузи контейнер nexus action action 0.6192 +перезагрузи контейнер redis action action 0.5051 +перезагрузи контейнер postgres action not_action 0.4449 +перезапусти контейнер maven, пожалуйста action action 0.8442 +перезапусти контейнер nexus, пожалуйста action action 0.8672 + +## 7. Voice-Like Stress Results + +```text +Total stress-testable pairs: 965 +question with ?: 123 +question without ?: 242 +positive polite/modal action: 350 +capability-question pairs: 42 +``` + +Sample affected pairs: + + act-stop-091: + original: "останови сервис, пожалуйста" + stressed: "останови сервис пожалуйста" + route: action + act-stop-092: + original: "останови nginx, пожалуйста" + stressed: "останови nginx пожалуйста" + route: action + act-stop-093: + original: "останови bотоbackup, пожалуйста" + stressed: "останови bотоbackup пожалуйста" + route: action + act-stop-094: + original: "выключи сервис, пожалуйста" + stressed: "выключи сервис пожалуйста" + route: action + act-stop-095: + original: "выключи nginx, пожалуйста" + stressed: "выключи nginx пожалуйста" + route: action + act-stop-096: + original: "выключи bотоbackup, пожалуйста" + stressed: "выключи bотоbackup пожалуйста" + route: action + act-stop-097: + original: "можешь останови сервис, пожалуйста" + stressed: "можешь останови сервис пожалуйста" + route: action + act-stop-098: + original: "can you останови сервис, please" + stressed: "can you останови сервис please" + route: action + act-stop-099: + original: "можешь останови nginx, пожалуйста" + stressed: "можешь останови nginx пожалуйста" + route: action + act-stop-100: + original: "can you останови nginx, please" + stressed: "can you останови nginx please" + route: action + +Impact assessment: + The MLP's decision boundary must not depend on punctuation artifacts. + If MLP success depends on '?' presence, it will fail under voice input. + +## 8. Six-Way MLP Results + +Best six-way MLP: H=64, wd=0.001 + +```text +accuracy: 68.7% +macro F1: 0.681 +action precision: 0.639 +action recall: 0.830 +false-action rate: 15.0% +false-action count: 374 +``` + +Per-route F1: + + action P=0.639 R=0.830 F1=0.722 (n=796) + conversation P=0.893 R=0.720 F1=0.798 (n=93) + knowledge P=0.735 R=0.634 F1=0.681 (n=715) + memory_write P=0.743 R=0.591 F1=0.659 (n=553) + system P=0.646 R=0.597 F1=0.621 (n=226) + uncertain P=0.583 R=0.626 F1=0.604 (n=107) + +Confusion matrix (rows=expected, cols=predicted): + action conversation knowledge memory_write system uncertain + action 661 3 67 43 7 15 + conversation 0 67 9 3 5 9 + knowledge 181 0 453 47 33 1 + memory_write 132 0 48 327 29 17 + system 37 3 34 11 135 6 + uncertain 24 2 5 9 0 67 + +Fold variance (six-way MLP): + + fold accuracy macro_f1 action_P action_R FA +------------------------------------------------------- + 0 82.1% 0.774 0.871 0.938 20 + 1 81.5% 0.625 0.838 0.925 36 + 2 64.2% 0.580 0.722 0.771 88 + 3 69.0% 0.687 0.425 0.895 104 + 4 51.7% 0.452 0.212 0.500 126 + +## 9. Comparison Against All Linear Baselines + +| model | params beyond e5 | action P | action R | FA rate | macro F1 | +| ---------------------------- | ------------------ | ---------- | ---------- | ---------- | ---------- | +| 6-way linear | 2,310 | 0.629 | 0.832 | 15.7% | 0.675 | +| binary linear | ~385 | 0.688 | 0.476 | 6.9% | — | +| structural linear | ~2,313 | 0.610 | 0.805 | 15.1% | 0.622 | +| binary MLP (H=32) | 12,353 | 0.673 | 0.569 | 8.8% | — | +| 6-way MLP (H=64) | 25,030 | 0.639 | 0.830 | 15.0% | 0.681 | + +## 10. Conclusion + +### Verdict: e5 representation inadequate for the action-pragmatics boundary + +The tiny nonlinear MLP provides only marginal improvement over linear baselines +and fails the capability-question diagnostic and the safety operating goal. + +1. **Binary MLP does not beat binary linear.** PR-AUC 0.692 (MLP, H=32) vs 0.707 + (linear). The MLP trades precision for recall (P=0.673/R=0.569) and produces + *more* false actions (220 vs 172) than the linear probe. The nonlinearity does + not unlock a separability that the linear surface was missing — both are pinned + by the same representation. + +2. **Capability-question boundary is catastrophically broken for both.** The MLP + fires action on **91.3% (115/126)** of capability questions — the primary + diagnostic fails outright. The linear probe reaches the same place (112/126). + A nonlinear head cannot separate "ты можешь выключить свет?" (knowledge) from + "можешь выключить свет, пожалуйста" (action) because the frozen mean-pooled e5 + vector places them on top of each other. + +3. **No safe operating region exists.** The MLP never reaches action precision + ≥ 0.95 with materially non-zero recall. The best it can do is P=0.778 at + R=0.053 (essentially zero recall). At any threshold that keeps FA count low, + it becomes useless; at any threshold that keeps recall useful, FA rate climbs + past 10%. + +4. **Six-way MLP is essentially flat.** F1=0.681 (H=64) vs 0.675 (linear six-way); + FA rate 15.0% vs 15.7%. Per-route F1 is within noise of the linear head. The bad + folds from slice 16 (fold 3 P=0.425, fold 4 P=0.212) are not repaired. + +Per the brief's decision rule this is the third case: + +> mean-pooled e5 representation is inadequate for Maven's action-pragmatics +> boundary. at that point stop probing e5. + +**Stop probing the frozen mean-pooled e5-small representation.** The signal is +not present and not merely nonlinear. A fundamentally different representation or +encoder is required for the action/capability-question pragmatics boundary. + +### Capability-question diagnostic + + capability_question false-action rate (MLP): 91.3% (115/126) + capability_question false-action rate (linear): 88.9% (112/126) + positive modal action recall (MLP): 0.569 + positive modal action recall (linear): 0.476 + + The MLP does not materially improve the capability-question boundary. + +### Six-way MLP vs binary gate + +Six-way MLP (F1=0.681) and six-way linear (F1=0.675) are statistically +indistinguishable. Neither the six-way formulation nor the nonlinearity fixes the +boundary. The architecture evidence does not support routing through a single +learned head for Maven's action pragmatics as currently embedded. + +## 11. Commit hash for diagnostic tooling + +`e80d45f` (slice 17 MLP probe tooling: `cmd/semantic-router-experiment/slice17_mlp.py`)