# Slice 17: Nonlinear MLP Probe — Action Gate Experiment ## 0. Frozen Artifacts from Slice 16 ```text development corpus v2 hash: b27fd48f478ca477 original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected) embedder: model_quantized@384/tok2 dimension: 384 pooling: mean-pool + L2-normalize input template: query: embedding file: /tmp/mvn-experiment/embeddings.json total examples: 3025 dev pool: 2490 frozen holdout: 535 router-residual: 2943 ``` ## 1. Hidden Sizes and Exact Parameter Counts ### Binary action gate: e5[384] → Linear(384→H) → ReLU → Linear(H→1) H params fp32 bytes int8 bytes ---------------------------------------- 8 3,089 12,356 3,089 16 6,177 24,708 6,177 32 12,353 49,412 12,353 64 24,705 98,820 24,705 ### Six-way MLP: e5[384] → Linear(384→H) → ReLU → Linear(H→6) H params fp32 bytes int8 bytes ---------------------------------------- 8 3,134 12,536 3,134 16 6,262 25,048 6,262 32 12,518 50,072 12,518 64 25,030 100,120 25,030 ## 2. Selected Regularization Best binary MLP: H=32, weight_decay=0.01 Selected by grouped development CV PR-AUC. Grid results (binary MLP): H wd PR-AUC ROC-AUC action_P action_R FA count FA rate -------------------------------------------------------------------------------- 16 0.0 0.422 0.628 0.139 0.215 155 6.2% 16 0.0001 0.422 0.628 0.139 0.215 155 6.2% 16 0.001 0.423 0.628 0.139 0.215 155 6.2% 16 0.01 0.423 0.628 0.139 0.215 155 6.2% 32 0.0 0.691 0.854 0.686 0.610 219 8.8% 32 0.0001 0.691 0.854 0.686 0.610 219 8.8% 32 0.001 0.691 0.854 0.686 0.611 219 8.8% 32 0.01 0.692 0.854 0.686 0.605 220 8.8% * 64 0.0 0.547 0.714 0.309 0.381 195 7.8% 64 0.0001 0.547 0.714 0.309 0.381 195 7.8% 64 0.001 0.547 0.714 0.309 0.381 195 7.8% 64 0.01 0.545 0.713 0.308 0.368 194 7.8% ## 3. Binary MLP OOF Metrics (best: H=32, wd=0.01) ```text ROC-AUC: 0.854 PR-AUC: 0.692 action precision: 0.686 action recall: 0.605 false-positive: 220 false-negative: 343 false-action rate: 8.8% total: 2490 mean iters: 64 ``` ## 4. Fold Variance Binary MLP (H=32, wd=0.01): fold ROC-AUC PR-AUC action_P action_R FP FN n ------------------------------------------------------------ 0 0.960 0.915 0.935 0.701 7 43 476 1 0.896 0.917 0.840 0.861 33 28 341 2 0.953 0.939 0.988 0.286 1 212 707 3 0.825 0.454 0.440 0.767 84 20 506 4 0.638 0.236 0.228 0.412 95 40 460 Fold variance comparison with linear binary probe: fold linear FP MLP FP linear FA% MLP FA% -------------------------------------------------- 0 2 7 0.4% 1.5% 1 12 33 3.5% 9.7% 2 0 1 0.0% 0.1% 3 71 84 14.0% 16.6% 4 87 95 18.9% 20.7% ## 5. Safety Operating Curve (best binary MLP) threshold action_P action_R FA count FA rate coverage ----------------------------------------------------------------- 0.300 0.6224 0.7827 378 0.1518 0.9305 0.325 0.6324 0.7563 350 0.1406 0.9221 0.350 0.6400 0.7349 329 0.1321 0.9153 0.375 0.6462 0.6997 305 0.1225 0.9040 0.400 0.6536 0.6709 283 0.1137 0.8948 0.425 0.6540 0.6482 273 0.1096 0.8876 0.450 0.6631 0.6231 252 0.1012 0.8795 0.475 0.6714 0.5930 231 0.0928 0.8699 0.500 0.6731 0.5691 220 0.0884 0.8622 0.525 0.6801 0.5503 206 0.0827 0.8562 0.550 0.6866 0.5339 194 0.0779 0.8510 0.575 0.6887 0.5113 184 0.0739 0.8438 0.600 0.6968 0.4849 168 0.0675 0.8353 0.625 0.6910 0.4523 161 0.0647 0.8249 0.650 0.6914 0.4334 154 0.0618 0.8189 0.675 0.6930 0.4083 144 0.0578 0.8108 0.700 0.6856 0.3781 138 0.0554 0.8012 0.725 0.6887 0.3530 127 0.0510 0.7932 0.750 0.6870 0.3254 118 0.0474 0.7843 0.775 0.7071 0.3003 99 0.0398 0.7763 0.800 0.7204 0.2751 85 0.0341 0.7683 0.825 0.7159 0.2374 75 0.0301 0.7562 0.850 0.7422 0.2098 58 0.0233 0.7474 0.875 0.7391 0.1709 48 0.0193 0.7349 0.900 0.7518 0.1332 35 0.0141 0.7229 0.925 0.7684 0.0917 22 0.0088 0.7096 0.950 0.7778 0.0528 12 0.0048 0.6972 No threshold achieves action_P ≥ 0.95. ## 6. Capability-Question Boundary Diagnostic capability_question → false action rate: 91.3% (115/126) positive modal request → action recall: 0.569 (796 examples) ### Paired capability-question vs action examples These are the critical diagnostic pairs: text route predicted action_P FA? ----------------------------------------------------------------------------------------------- ты можешь выключить свет knowledge action 0.8159 YES ты можешь выключить жалюзи knowledge action 0.9752 YES ты можешь выключить вытяжку knowledge action 0.9720 YES ты можешь выключить вентилятор knowledge action 0.9041 YES умеешь ли включить свет knowledge action 0.5618 YES умеешь ли включить жалюзи knowledge action 0.9195 YES умеешь ли включить вытяжку knowledge action 0.8956 YES умеешь ли включить вентилятор knowledge action 0.6122 YES сможешь открыть свет knowledge action 0.7012 YES сможешь открыть жалюзи knowledge action 0.9633 YES сможешь открыть вытяжку knowledge action 0.9596 YES сможешь открыть вентилятор knowledge action 0.8491 YES ты можешь выключить свет, пожалуйста knowledge action 0.8682 YES ты можешь выключить жалюзи, пожалуйста knowledge action 0.9778 YES ты можешь выключить вытяжку, пожалуйста knowledge action 0.9718 YES ... перезапусти контейнер maven action action 0.8106 перезапусти контейнер nexus action action 0.8002 перезапусти контейнер redis action action 0.6867 перезапусти контейнер postgres action action 0.6016 перезагрузи контейнер maven action action 0.6810 перезагрузи контейнер nexus action action 0.6192 перезагрузи контейнер redis action action 0.5051 перезагрузи контейнер postgres action not_action 0.4449 перезапусти контейнер maven, пожалуйста action action 0.8442 перезапусти контейнер nexus, пожалуйста action action 0.8672 ## 7. Voice-Like Stress Results ```text Total stress-testable pairs: 965 question with ?: 123 question without ?: 242 positive polite/modal action: 350 capability-question pairs: 42 ``` Sample affected pairs: act-stop-091: original: "останови сервис, пожалуйста" stressed: "останови сервис пожалуйста" route: action act-stop-092: original: "останови nginx, пожалуйста" stressed: "останови nginx пожалуйста" route: action act-stop-093: original: "останови bотоbackup, пожалуйста" stressed: "останови bотоbackup пожалуйста" route: action act-stop-094: original: "выключи сервис, пожалуйста" stressed: "выключи сервис пожалуйста" route: action act-stop-095: original: "выключи nginx, пожалуйста" stressed: "выключи nginx пожалуйста" route: action act-stop-096: original: "выключи bотоbackup, пожалуйста" stressed: "выключи bотоbackup пожалуйста" route: action act-stop-097: original: "можешь останови сервис, пожалуйста" stressed: "можешь останови сервис пожалуйста" route: action act-stop-098: original: "can you останови сервис, please" stressed: "can you останови сервис please" route: action act-stop-099: original: "можешь останови nginx, пожалуйста" stressed: "можешь останови nginx пожалуйста" route: action act-stop-100: original: "can you останови nginx, please" stressed: "can you останови nginx please" route: action Impact assessment: The MLP's decision boundary must not depend on punctuation artifacts. If MLP success depends on '?' presence, it will fail under voice input. ## 8. Six-Way MLP Results Best six-way MLP: H=64, wd=0.001 ```text accuracy: 68.7% macro F1: 0.681 action precision: 0.639 action recall: 0.830 false-action rate: 15.0% false-action count: 374 ``` Per-route F1: action P=0.639 R=0.830 F1=0.722 (n=796) conversation P=0.893 R=0.720 F1=0.798 (n=93) knowledge P=0.735 R=0.634 F1=0.681 (n=715) memory_write P=0.743 R=0.591 F1=0.659 (n=553) system P=0.646 R=0.597 F1=0.621 (n=226) uncertain P=0.583 R=0.626 F1=0.604 (n=107) Confusion matrix (rows=expected, cols=predicted): action conversation knowledge memory_write system uncertain action 661 3 67 43 7 15 conversation 0 67 9 3 5 9 knowledge 181 0 453 47 33 1 memory_write 132 0 48 327 29 17 system 37 3 34 11 135 6 uncertain 24 2 5 9 0 67 Fold variance (six-way MLP): fold accuracy macro_f1 action_P action_R FA ------------------------------------------------------- 0 82.1% 0.774 0.871 0.938 20 1 81.5% 0.625 0.838 0.925 36 2 64.2% 0.580 0.722 0.771 88 3 69.0% 0.687 0.425 0.895 104 4 51.7% 0.452 0.212 0.500 126 ## 9. Comparison Against All Linear Baselines | model | params beyond e5 | action P | action R | FA rate | macro F1 | | ---------------------------- | ------------------ | ---------- | ---------- | ---------- | ---------- | | 6-way linear | 2,310 | 0.629 | 0.832 | 15.7% | 0.675 | | binary linear | ~385 | 0.688 | 0.476 | 6.9% | — | | structural linear | ~2,313 | 0.610 | 0.805 | 15.1% | 0.622 | | binary MLP (H=32) | 12,353 | 0.673 | 0.569 | 8.8% | — | | 6-way MLP (H=64) | 25,030 | 0.639 | 0.830 | 15.0% | 0.681 | ## 10. Conclusion ### Verdict: e5 representation inadequate for the action-pragmatics boundary The tiny nonlinear MLP provides only marginal improvement over linear baselines and fails the capability-question diagnostic and the safety operating goal. 1. **Binary MLP does not beat binary linear.** PR-AUC 0.692 (MLP, H=32) vs 0.707 (linear). The MLP trades precision for recall (P=0.673/R=0.569) and produces *more* false actions (220 vs 172) than the linear probe. The nonlinearity does not unlock a separability that the linear surface was missing — both are pinned by the same representation. 2. **Capability-question boundary is catastrophically broken for both.** The MLP fires action on **91.3% (115/126)** of capability questions — the primary diagnostic fails outright. The linear probe reaches the same place (112/126). A nonlinear head cannot separate "ты можешь выключить свет?" (knowledge) from "можешь выключить свет, пожалуйста" (action) because the frozen mean-pooled e5 vector places them on top of each other. 3. **No safe operating region exists.** The MLP never reaches action precision ≥ 0.95 with materially non-zero recall. The best it can do is P=0.778 at R=0.053 (essentially zero recall). At any threshold that keeps FA count low, it becomes useless; at any threshold that keeps recall useful, FA rate climbs past 10%. 4. **Six-way MLP is essentially flat.** F1=0.681 (H=64) vs 0.675 (linear six-way); FA rate 15.0% vs 15.7%. Per-route F1 is within noise of the linear head. The bad folds from slice 16 (fold 3 P=0.425, fold 4 P=0.212) are not repaired. Per the brief's decision rule this is the third case: > mean-pooled e5 representation is inadequate for Maven's action-pragmatics > boundary. at that point stop probing e5. **Stop probing the frozen mean-pooled e5-small representation.** The signal is not present and not merely nonlinear. A fundamentally different representation or encoder is required for the action/capability-question pragmatics boundary. ### Capability-question diagnostic capability_question false-action rate (MLP): 91.3% (115/126) capability_question false-action rate (linear): 88.9% (112/126) positive modal action recall (MLP): 0.569 positive modal action recall (linear): 0.476 The MLP does not materially improve the capability-question boundary. ### Six-way MLP vs binary gate Six-way MLP (F1=0.681) and six-way linear (F1=0.675) are statistically indistinguishable. Neither the six-way formulation nor the nonlinearity fixes the boundary. The architecture evidence does not support routing through a single learned head for Maven's action pragmatics as currently embedded. ## 11. Commit hash for diagnostic tooling `e80d45f` (slice 17 MLP probe tooling: `cmd/semantic-router-experiment/slice17_mlp.py`)