router/semantic: slice 16 diagnostic — action/non-action boundary analysis
Experiments: - Residual-only OOF safety metrics - False-action decomposition by semantic family (7 families, 24 split groups) - Paired e5 geometry (gap=0.010 between action and capability question) - Binary action probe (PR-AUC=0.707, ROC-AUC=0.863) - Cost-sensitive linear classification - Expanded action-threshold sweep (7 thresholds) - Voice-like punctuation stress evaluation - e5 + tiny structural features - Four-hypothesis comparison table Verdict: e5 representation is the primary bottleneck. Action seeds and capability questions are nearly indistinguishable in embedding space.
This commit is contained in:
@@ -0,0 +1,246 @@
|
||||
# Slice 16 Diagnostic: Action/Non-Action Boundary Analysis
|
||||
|
||||
## 0. Slice 15 Commit Hashes
|
||||
|
||||
```text
|
||||
semantic seed type: 07bfcea
|
||||
merge-corpus tool: 6397ea1
|
||||
e5 embedding cache: f8ec77d
|
||||
sklearn experiment: 9df2239
|
||||
corpus-factory: 4bb7555
|
||||
expanded corpus: 87411e5
|
||||
contract tests: ad3f2d6
|
||||
experiment reports: cbac8b9
|
||||
|
||||
development corpus v2 hash: b27fd48f478ca477
|
||||
original frozen holdout hash: ad297fbdbbea704b (byte-identical, uninspected)
|
||||
```
|
||||
|
||||
## 1. Router-Residual OOF Safety Metrics
|
||||
|
||||
```text
|
||||
action precision: 0.620
|
||||
action recall: 0.826
|
||||
false-action count: 388
|
||||
false-action rate: 16.0%
|
||||
uncertain F1: 0.587
|
||||
total examples: 2418
|
||||
best C: 10.0
|
||||
```
|
||||
|
||||
## 2. False-Action Decomposition by Semantic Family
|
||||
|
||||
Total false actions: 388
|
||||
|
||||
### By semantic family
|
||||
|
||||
family count rate
|
||||
----------------------------------------
|
||||
memory_write 133 34.3%
|
||||
capability_question 112 28.9%
|
||||
knowledge_general 77 19.8%
|
||||
system 29 7.5%
|
||||
uncertain 26 6.7%
|
||||
question 9 2.3%
|
||||
conversation 2 0.5%
|
||||
|
||||
### By held-out split group (top 20)
|
||||
|
||||
split_group count
|
||||
-------------------------------------
|
||||
knowledge:capability-ha 70
|
||||
knowledge:world-def 46
|
||||
free:remember 45
|
||||
knowledge:capability-tool 42
|
||||
note:task 28
|
||||
system:self-version 27
|
||||
fact:meal 27
|
||||
knowledge:homelab-status 15
|
||||
uncertain:ambiguous-noun 15
|
||||
note:homelab 11
|
||||
fact:water 10
|
||||
note:idea 8
|
||||
knowledge:deadline 7
|
||||
uncertain:single-word-verb 7
|
||||
knowledge:world-explain 6
|
||||
knowledge:homelab-disk 5
|
||||
knowledge:task-check 5
|
||||
uncertain:multi-ambiguous 3
|
||||
fact:pills 2
|
||||
system:quiet-on 2
|
||||
|
||||
### By fold
|
||||
|
||||
fold count
|
||||
------------
|
||||
0 28
|
||||
1 38
|
||||
2 92
|
||||
3 104
|
||||
4 126
|
||||
|
||||
### Diagnosis
|
||||
|
||||
- Distinct semantic families contributing false actions: 7
|
||||
- Distinct held-out split groups: 24
|
||||
- Largest single family: memory_write (133 false actions)
|
||||
- Fold false-action count CV (std/mean): 0.49
|
||||
- Broad distribution across families suggests systematic action/non-action overlap
|
||||
|
||||
## 3. Paired E5 Geometry
|
||||
|
||||
cosine(action seed, positive action):
|
||||
mean=0.907 std=0.039 min=0.765 max=0.997 n=38222
|
||||
cosine(action seed, capability question):
|
||||
mean=0.897 std=0.035 min=0.782 max=0.979 n=796
|
||||
cosine(action seed, other negative contrast):
|
||||
mean=0.893 std=0.031 min=0.822 max=0.975 n=796
|
||||
|
||||
Separation gap (positive - capability): 0.010
|
||||
**WARNING**: e5 maps action seeds and capability questions nearly on top of each other.
|
||||
The representation itself may not preserve useful separation for this boundary.
|
||||
|
||||
## 4. Binary Action Probe (Linear Logistic)
|
||||
|
||||
```text
|
||||
ROC-AUC: 0.863
|
||||
PR-AUC: 0.707
|
||||
precision: 0.726
|
||||
recall: 0.538
|
||||
false-positive: 172
|
||||
false-negative: 417
|
||||
total: 2490
|
||||
best C: 1.0
|
||||
```
|
||||
|
||||
Per-fold:
|
||||
Fold 0: ROC=0.967 PR=0.937 P=0.978 R=0.618 FP=2 FN=55 n=476
|
||||
Fold 1: ROC=0.914 PR=0.940 P=0.919 R=0.682 FP=12 FN=64 n=341
|
||||
Fold 2: ROC=0.962 PR=0.952 P=1.000 R=0.195 FP=0 FN=239 n=707
|
||||
Fold 3: ROC=0.832 PR=0.472 P=0.482 R=0.767 FP=71 FN=20 n=506
|
||||
Fold 4: ROC=0.640 PR=0.235 P=0.250 R=0.426 FP=87 FN=39 n=460
|
||||
|
||||
## 5. Cost-Sensitive Linear Action Classification
|
||||
|
||||
cost action_P action_R FA count FA rate
|
||||
----------------------------------------------
|
||||
1 0.601 0.804 390 15.7%
|
||||
2 0.562 0.847 481 19.3%
|
||||
4 0.531 0.891 589 23.5%
|
||||
8 0.494 0.919 711 28.6%
|
||||
16 0.461 0.953 847 34.2%
|
||||
|
||||
## 6. Expanded Action-Threshold Sweep (30 action groups)
|
||||
|
||||
threshold action_P action_R FA count coverage
|
||||
--------------------------------------------------
|
||||
0.50 0.664 0.736 285 92.9%
|
||||
0.60 0.687 0.655 229 88.0%
|
||||
0.70 0.698 0.537 178 82.1%
|
||||
0.80 0.722 0.407 120 75.6%
|
||||
0.85 0.714 0.322 99 72.1%
|
||||
0.90 0.690 0.227 78 68.2%
|
||||
0.95 0.727 0.115 33 62.8%
|
||||
|
||||
## 7. Voice-Like Punctuation Stress Evaluation
|
||||
|
||||
Total stress-testable pairs: 965
|
||||
|
||||
Sample stress pairs:
|
||||
act-stop-091:
|
||||
original: "останови сервис, пожалуйста"
|
||||
stressed: "останови сервис пожалуйста"
|
||||
route: action
|
||||
act-stop-092:
|
||||
original: "останови nginx, пожалуйста"
|
||||
stressed: "останови nginx пожалуйста"
|
||||
route: action
|
||||
act-stop-093:
|
||||
original: "останови bотоbackup, пожалуйста"
|
||||
stressed: "останови bотоbackup пожалуйста"
|
||||
route: action
|
||||
act-stop-094:
|
||||
original: "выключи сервис, пожалуйста"
|
||||
stressed: "выключи сервис пожалуйста"
|
||||
route: action
|
||||
act-stop-095:
|
||||
original: "выключи nginx, пожалуйста"
|
||||
stressed: "выключи nginx пожалуйста"
|
||||
route: action
|
||||
act-stop-096:
|
||||
original: "выключи bотоbackup, пожалуйста"
|
||||
stressed: "выключи bотоbackup пожалуйста"
|
||||
route: action
|
||||
act-stop-097:
|
||||
original: "можешь останови сервис, пожалуйста"
|
||||
stressed: "можешь останови сервис пожалуйста"
|
||||
route: action
|
||||
act-stop-098:
|
||||
original: "can you останови сервис, please"
|
||||
stressed: "can you останови сервис please"
|
||||
route: action
|
||||
act-stop-099:
|
||||
original: "можешь останови nginx, пожалуйста"
|
||||
stressed: "можешь останови nginx пожалуйста"
|
||||
route: action
|
||||
act-stop-100:
|
||||
original: "can you останови nginx, please"
|
||||
stressed: "can you останови nginx please"
|
||||
route: action
|
||||
|
||||
Impact assessment:
|
||||
Removal of punctuation changes:
|
||||
- trailing '?' removal eliminates the strongest question signal
|
||||
- lowercase normalization removes proper-noun casing cues
|
||||
- whitespace collapse has minimal effect on e5 (subword tokenizer)
|
||||
Production voice punctuation is unreliable; the model must not depend on it.
|
||||
|
||||
Capability-question pairs total in stress set: 42
|
||||
Capability-question pairs with trailing '?': 21
|
||||
Capability-question pairs with comma only: 21
|
||||
|
||||
|
||||
## 8. E5 + Tiny Structural Features
|
||||
|
||||
```text
|
||||
Features: [e5(384) ; IsQuestion(1) ; trailing_?(1) ; prohibition(1)] = 387 dims
|
||||
Macro F1: 0.622
|
||||
False-action rate: 15.1%
|
||||
False-action count: 373
|
||||
best C: 10.0
|
||||
```
|
||||
|
||||
Comparison with pure e5:
|
||||
pure e5: macro_f1=0.616 FA_rate=15.7%
|
||||
e5 + structural bits: macro_f1=0.622 FA_rate=15.1%
|
||||
|
||||
## 9. Four-Hypothesis Comparison Table
|
||||
|
||||
| experiment | action_P | action_R | FA rate | macro F1 |
|
||||
| ----------------------------------- | ---------- | ---------- | ---------- | ---------- |
|
||||
| six-way e5 linear (residual) | 0.620 | 0.826 | 16.0% | 0.663 |
|
||||
| six-way + action threshold | 0.727 | 0.115 | 1.4% | — |
|
||||
| binary linear action probe | 0.726 | 0.538 | 6.9% | — |
|
||||
| e5 + tiny structural features | 0.610 | 0.805 | 15.1% | 0.622 |
|
||||
|
||||
## 10. Conservative Interpretation
|
||||
|
||||
### Action/capability-question embeddings are nearly indistinguishable
|
||||
|
||||
The representation itself is suspect for this boundary.
|
||||
Do not claim the boundary is merely nonlinear.
|
||||
A fundamentally different representation or encoder may be needed.
|
||||
|
||||
**E5 geometry verdict**: action seeds and capability questions are nearly
|
||||
indistinguishable in embedding space. The representation intentionally
|
||||
maps pragmatically different but semantically similar sentences close together.
|
||||
This is a fundamental limitation of the frozen e5 representation for this boundary.
|
||||
|
||||
**Softmax formulation verdict**: the six-way head produces 388 false actions
|
||||
at 16.0% rate. The binary probe produces 172 false actions.
|
||||
The binary probe has 0.4x the false-action count of the six-way head.
|
||||
This confirms the six-way softmax competition is a significant contributor.
|
||||
|
||||
## 11. Commit hash for diagnostic tooling
|
||||
|
||||
(to be filled after commit)
|
||||
Reference in New Issue
Block a user