# Slice 16 Diagnostic: Action/Non-Action Boundary Analysis ## 0. Slice 15 Commit Hashes ```text semantic seed type: 07bfcea merge-corpus tool: 6397ea1 e5 embedding cache: f8ec77d sklearn experiment: 9df2239 corpus-factory: 4bb7555 expanded corpus: 87411e5 contract tests: ad3f2d6 experiment reports: cbac8b9 development corpus v2 hash: b27fd48f478ca477 original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected) ``` ## 1. Router-Residual OOF Safety Metrics ```text action precision: 0.620 action recall: 0.826 false-action count: 388 false-action rate: 16.0% uncertain F1: 0.587 total examples: 2418 best C: 10.0 ``` ## 2. False-Action Decomposition by Semantic Family Total false actions: 388 ### By semantic family family count rate ---------------------------------------- memory_write 133 34.3% capability_question 112 28.9% knowledge_general 77 19.8% system 29 7.5% uncertain 26 6.7% question 9 2.3% conversation 2 0.5% ### By held-out split group (top 20) split_group count ------------------------------------- knowledge:capability-ha 70 knowledge:world-def 46 free:remember 45 knowledge:capability-tool 42 note:task 28 system:self-version 27 fact:meal 27 knowledge:homelab-status 15 uncertain:ambiguous-noun 15 note:homelab 11 fact:water 10 note:idea 8 knowledge:deadline 7 uncertain:single-word-verb 7 knowledge:world-explain 6 knowledge:homelab-disk 5 knowledge:task-check 5 uncertain:multi-ambiguous 3 fact:pills 2 system:quiet-on 2 ### By fold fold count ------------ 0 28 1 38 2 92 3 104 4 126 ### Diagnosis - Distinct semantic families contributing false actions: 7 - Distinct held-out split groups: 24 - Largest single family: memory_write (133 false actions) - Fold false-action count CV (std/mean): 0.49 - Broad distribution across families suggests systematic action/non-action overlap ## 3. Paired E5 Geometry cosine(action seed, positive action): mean=0.907 std=0.039 min=0.765 max=0.997 n=38222 cosine(action seed, capability question): mean=0.897 std=0.035 min=0.782 max=0.979 n=796 cosine(action seed, other negative contrast): mean=0.893 std=0.031 min=0.822 max=0.975 n=796 Separation gap (positive - capability): 0.010 **WARNING**: e5 maps action seeds and capability questions nearly on top of each other. The representation itself may not preserve useful separation for this boundary. ## 4. Binary Action Probe (Linear Logistic) ```text ROC-AUC: 0.863 PR-AUC: 0.707 precision: 0.726 recall: 0.538 false-positive: 172 false-negative: 417 total: 2490 best C: 1.0 ``` Per-fold: Fold 0: ROC=0.967 PR=0.937 P=0.978 R=0.618 FP=2 FN=55 n=476 Fold 1: ROC=0.914 PR=0.940 P=0.919 R=0.682 FP=12 FN=64 n=341 Fold 2: ROC=0.962 PR=0.952 P=1.000 R=0.195 FP=0 FN=239 n=707 Fold 3: ROC=0.832 PR=0.472 P=0.482 R=0.767 FP=71 FN=20 n=506 Fold 4: ROC=0.640 PR=0.235 P=0.250 R=0.426 FP=87 FN=39 n=460 ## 5. Cost-Sensitive Linear Action Classification cost action_P action_R FA count FA rate ---------------------------------------------- 1 0.601 0.804 390 15.7% 2 0.562 0.847 481 19.3% 4 0.531 0.891 589 23.5% 8 0.494 0.919 711 28.6% 16 0.461 0.953 847 34.2% ## 6. Expanded Action-Threshold Sweep (30 action groups) threshold action_P action_R FA count coverage -------------------------------------------------- 0.50 0.664 0.736 285 92.9% 0.60 0.687 0.655 229 88.0% 0.70 0.698 0.537 178 82.1% 0.80 0.722 0.407 120 75.6% 0.85 0.714 0.322 99 72.1% 0.90 0.690 0.227 78 68.2% 0.95 0.727 0.115 33 62.8% ## 7. Voice-Like Punctuation Stress Evaluation Total stress-testable pairs: 965 Sample stress pairs: act-stop-091: original: "останови сервис, пожалуйста" stressed: "останови сервис пожалуйста" route: action act-stop-092: original: "останови nginx, пожалуйста" stressed: "останови nginx пожалуйста" route: action act-stop-093: original: "останови bотоbackup, пожалуйста" stressed: "останови bотоbackup пожалуйста" route: action act-stop-094: original: "выключи сервис, пожалуйста" stressed: "выключи сервис пожалуйста" route: action act-stop-095: original: "выключи nginx, пожалуйста" stressed: "выключи nginx пожалуйста" route: action act-stop-096: original: "выключи bотоbackup, пожалуйста" stressed: "выключи bотоbackup пожалуйста" route: action act-stop-097: original: "можешь останови сервис, пожалуйста" stressed: "можешь останови сервис пожалуйста" route: action act-stop-098: original: "can you останови сервис, please" stressed: "can you останови сервис please" route: action act-stop-099: original: "можешь останови nginx, пожалуйста" stressed: "можешь останови nginx пожалуйста" route: action act-stop-100: original: "can you останови nginx, please" stressed: "can you останови nginx please" route: action Impact assessment: Removal of punctuation changes: - trailing '?' removal eliminates the strongest question signal - lowercase normalization removes proper-noun casing cues - whitespace collapse has minimal effect on e5 (subword tokenizer) Production voice punctuation is unreliable; the model must not depend on it. Capability-question pairs total in stress set: 42 Capability-question pairs with trailing '?': 21 Capability-question pairs with comma only: 21 ## 8. E5 + Tiny Structural Features ```text Features: [e5(384) ; IsQuestion(1) ; trailing_?(1) ; prohibition(1)] = 387 dims Macro F1: 0.622 False-action rate: 15.1% False-action count: 373 best C: 10.0 ``` Comparison with pure e5: pure e5: macro_f1=0.616 FA_rate=15.7% e5 + structural bits: macro_f1=0.622 FA_rate=15.1% ## 9. Four-Hypothesis Comparison Table | experiment | action_P | action_R | FA rate | macro F1 | | ----------------------------------- | ---------- | ---------- | ---------- | ---------- | | six-way e5 linear (residual) | 0.620 | 0.826 | 16.0% | 0.663 | | six-way + action threshold | 0.727 | 0.115 | 1.4% | — | | binary linear action probe | 0.726 | 0.538 | 6.9% | — | | e5 + tiny structural features | 0.610 | 0.805 | 15.1% | 0.622 | ## 10. Conservative Interpretation ### Action/capability-question embeddings are nearly indistinguishable The representation itself is suspect for this boundary. Do not claim the boundary is merely nonlinear. A fundamentally different representation or encoder may be needed. **E5 geometry verdict**: action seeds and capability questions are nearly indistinguishable in embedding space. The representation intentionally maps pragmatically different but semantically similar sentences close together. This is a fundamental limitation of the frozen e5 representation for this boundary. **Softmax formulation verdict**: the six-way head produces 388 false actions at 16.0% rate. The binary probe produces 172 false actions. The binary probe has 0.4x the false-action count of the six-way head. This confirms the six-way softmax competition is a significant contributor. ## 11. Commit hash for diagnostic tooling `774c066`