claude
cbac8b9c4f
- 2026-09-07-linear-e5-router-experiment: 136-example baseline (superseded) - 2026-09-07-expanded-corpus-linear-head-experiment: 3025-example expanded (live) - CLAUDE.md: index updates for both evals
264 lines
11 KiB
Markdown
264 lines
11 KiB
Markdown
# Semantic Router Linear Head Experiment — Report
|
||
|
||
## 1. Exact e5 representation used
|
||
|
||
- **Model**: model_quantized@384/tok2
|
||
- **Checkpoint**: models/embedder/multilingual-e5-small/model_quantized.onnx
|
||
- **Tokenizer**: models/embedder/multilingual-e5-small/tokenizer.json
|
||
- **Dimension**: 384
|
||
- **Pooling**: mean-pool + L2-normalize
|
||
- **Normalization**: L2
|
||
- **Input template**: query: <text>
|
||
|
||
## 2. Development/residual row counts
|
||
|
||
- Total corpus: 136
|
||
- Frozen holdout: 21
|
||
- Development pool: 115
|
||
- Fast-path resolved: 23
|
||
- Router-residual: 113
|
||
|
||
Route distribution (full corpus):
|
||
- action: 22
|
||
- conversation: 7
|
||
- knowledge: 46
|
||
- memory_write: 19
|
||
- system: 7
|
||
- uncertain: 35
|
||
|
||
## 3. Grouped fold composition
|
||
|
||
Folds: 5
|
||
- Fold 0: eval=33 train=82 routes={'action': 4, 'conversation': 1, 'knowledge': 11, 'memory_write': 3, 'system': 1, 'uncertain': 13}
|
||
- Fold 1: eval=16 train=99 routes={'action': 3, 'conversation': 1, 'knowledge': 5, 'memory_write': 4, 'system': 1, 'uncertain': 2}
|
||
- Fold 2: eval=22 train=93 routes={'action': 3, 'conversation': 1, 'knowledge': 7, 'memory_write': 4, 'system': 2, 'uncertain': 5}
|
||
- Fold 3: eval=22 train=93 routes={'action': 4, 'conversation': 1, 'knowledge': 6, 'memory_write': 4, 'system': 2, 'uncertain': 5}
|
||
- Fold 4: eval=22 train=93 routes={'action': 4, 'conversation': 2, 'knowledge': 8, 'memory_write': 2, 'system': 1, 'uncertain': 5}
|
||
|
||
## 4. Selected regularization
|
||
|
||
### Experiment A: All development examples
|
||
- Best C: 100.0
|
||
- Mean accuracy: 65.2% ± 6.2%
|
||
- Mean macro F1: 0.597 ± 0.070
|
||
- Total false actions (CV): 10
|
||
|
||
### Experiment B: Router-residual only
|
||
- Best C: 100.0
|
||
- Mean accuracy: 66.5% ± 7.6%
|
||
- Mean macro F1: 0.475 ± 0.083
|
||
- Total false actions (CV): 0
|
||
|
||
### Stability across folds
|
||
|
||
C=0.01 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%']
|
||
C=0.1 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%']
|
||
C=1.0 acc=44.8%±7.0% f1=0.190±0.049 folds_acc=['36.4%', '37.5%', '50.0%', '45.5%', '54.5%']
|
||
C=10.0 acc=58.3%±4.6% f1=0.403±0.086 folds_acc=['51.5%', '62.5%', '63.6%', '59.1%', '54.5%']
|
||
C=100.0 acc=65.2%±6.2% f1=0.597±0.070 folds_acc=['54.5%', '62.5%', '68.2%', '68.2%', '72.7%']
|
||
|
||
## 5. All-example CV metrics
|
||
|
||
- Accuracy: 64.3%
|
||
- Macro F1: 0.620
|
||
- False-action count: 10
|
||
- False-action rate: 8.7%
|
||
- Action precision: 0.545
|
||
- Action recall: 0.667
|
||
- Uncertain precision: 0.800
|
||
- Uncertain recall: 0.533
|
||
|
||
Per-class metrics:
|
||
action P=0.545 R=0.667 F1=0.600 (n=18)
|
||
conversation P=1.000 R=0.500 F1=0.667 (n=6)
|
||
knowledge P=0.756 R=0.838 F1=0.795 (n=37)
|
||
memory_write P=0.346 R=0.529 F1=0.419 (n=17)
|
||
system P=1.000 R=0.429 F1=0.600 (n=7)
|
||
uncertain P=0.800 R=0.533 F1=0.640 (n=30)
|
||
|
||
Confusion matrix (rows=expected, cols=predicted):
|
||
action conversation knowledge memory_write system uncertain
|
||
action 12 0 0 5 0 1
|
||
conversation 0 3 1 0 0 2
|
||
knowledge 4 0 31 2 0 0
|
||
memory_write 4 0 3 9 0 1
|
||
system 2 0 2 0 3 0
|
||
uncertain 0 0 4 10 0 16
|
||
|
||
## 6. Residual-only CV metrics
|
||
|
||
- Accuracy: 64.6%
|
||
- Macro F1: 0.429
|
||
- False-action count: 0
|
||
- False-action rate: 0.0%
|
||
- Action precision: 0.000
|
||
- Action recall: 0.000
|
||
- Uncertain precision: 0.667
|
||
- Uncertain recall: 0.533
|
||
|
||
Per-class metrics:
|
||
action P=0.000 R=0.000 F1=0.000 (n=5)
|
||
conversation P=1.000 R=0.500 F1=0.667 (n=6)
|
||
knowledge P=0.750 R=0.917 F1=0.825 (n=36)
|
||
memory_write P=0.400 R=0.625 F1=0.488 (n=16)
|
||
system P=0.000 R=0.000 F1=0.000 (n=3)
|
||
uncertain P=0.667 R=0.533 F1=0.593 (n=30)
|
||
|
||
Confusion matrix (rows=expected, cols=predicted):
|
||
action conversation knowledge memory_write system uncertain
|
||
action 0 0 0 2 0 3
|
||
conversation 0 3 1 0 0 2
|
||
knowledge 0 0 33 3 0 0
|
||
memory_write 0 0 4 10 0 2
|
||
system 0 0 2 0 0 1
|
||
uncertain 0 0 4 10 0 16
|
||
|
||
## 7. Legacy-vs-linear comparison
|
||
|
||
### All examples
|
||
metric legacy linear e5 delta
|
||
-------------------------------------------------------
|
||
accuracy 52.2% 64.3% 12.1%
|
||
macro F1 — 0.620 —
|
||
action precision — 0.545 —
|
||
false-action rate 19.9% 8.7% -11.2%
|
||
uncertain F1 0.000 0.640 0.640
|
||
|
||
### Router-residual only
|
||
metric legacy linear e5 delta
|
||
-------------------------------------------------------
|
||
accuracy 40.8% 64.6% 23.8%
|
||
macro F1 — 0.429 —
|
||
false-action rate — 0.0% —
|
||
|
||
## 8. Fold variance
|
||
|
||
All-example CV:
|
||
Fold 0: acc=54.5% f1=0.507 false_action=1
|
||
Fold 1: acc=62.5% f1=0.579 false_action=2
|
||
Fold 2: acc=68.2% f1=0.554 false_action=2
|
||
Fold 3: acc=68.2% f1=0.632 false_action=5
|
||
Fold 4: acc=72.7% f1=0.712 false_action=0
|
||
|
||
Residual-only CV:
|
||
Fold 0: acc=53.6% f1=0.500 false_action=0
|
||
Fold 1: acc=75.0% f1=0.573 false_action=0
|
||
Fold 2: acc=72.2% f1=0.411 false_action=0
|
||
Fold 3: acc=68.4% f1=0.541 false_action=0
|
||
Fold 4: acc=63.2% f1=0.351 false_action=0
|
||
|
||
## 9. False-action repair/new-error analysis
|
||
|
||
Note: Legacy per-example predictions were not available for this experiment.
|
||
The legacy baseline was measured in aggregate in the Go test suite.
|
||
|
||
Learned router false-action cases (out-of-fold):
|
||
en-query-003: 'show me this week's weight' (true=knowledge, proba(action)=0.402)
|
||
ru-query-014: 'я успеваю до дедлайна' (true=knowledge, proba(action)=0.343)
|
||
ru-note-003: 'заметка про настройку vlan на свитче' (true=memory_write, proba(action)=0.335)
|
||
ru-query-005: 'напоминания на завтра есть' (true=knowledge, proba(action)=0.336)
|
||
ru-fact-009: 'отметь что я выпил таблетки утром' (true=memory_write, proba(action)=0.437)
|
||
ru-query-011: 'почему сервер тормозит' (true=knowledge, proba(action)=0.381)
|
||
ru-fact-005: 'поспал часов пять' (true=memory_write, proba(action)=0.557)
|
||
ru-note-006: 'добавь в задачи купить молоко' (true=memory_write, proba(action)=0.739)
|
||
ru-sys-003: 'переходи в тихий режим' (true=system, proba(action)=0.404)
|
||
en-sys-002: 'turn quiet mode back on' (true=system, proba(action)=0.410)
|
||
|
||
## 10. Contrast-family results
|
||
|
||
### Experiment A (all dev)
|
||
family count correct accuracy false_act
|
||
------------------------------------------------------------
|
||
negation 5 3 60.0% 0
|
||
question 5 5 100.0% 0
|
||
reported_speech 6 3 50.0% 0
|
||
quotation 6 4 66.7% 0
|
||
hypothetical 6 3 50.0% 0
|
||
capability_question 6 6 100.0% 0
|
||
|
||
### Experiment B (residual only)
|
||
family count correct accuracy false_act
|
||
------------------------------------------------------------
|
||
negation 5 3 60.0% 0
|
||
question 5 5 100.0% 0
|
||
reported_speech 6 3 50.0% 0
|
||
quotation 6 4 66.7% 0
|
||
hypothetical 6 3 50.0% 0
|
||
capability_question 6 6 100.0% 0
|
||
|
||
## 11. Calibration metrics
|
||
|
||
### Experiment A
|
||
- ECE: 0.116
|
||
- Brier score: 0.418
|
||
- Log loss: 0.832
|
||
|
||
### Experiment B
|
||
- ECE: 0.159
|
||
- Brier score: 0.411
|
||
- Log loss: 0.804
|
||
|
||
## 12. Abstention curves
|
||
|
||
### Experiment A (all dev)
|
||
threshold n_accepted coverage accuracy macro_f1 false_act
|
||
--------------------------------------------------------------
|
||
0.40 98 85.2% 72.4% 0.654 6
|
||
0.50 75 65.2% 84.0% 0.690 2
|
||
0.60 56 48.7% 92.9% 0.851 1
|
||
0.70 42 36.5% 97.6% 0.714 1
|
||
0.80 28 24.3% 100.0% 1.000 0
|
||
0.90 10 8.7% 100.0% 1.000 0
|
||
|
||
### Experiment B (residual only)
|
||
threshold n_accepted coverage accuracy macro_f1 false_act
|
||
--------------------------------------------------------------
|
||
0.40 86 89.6% 67.4% 0.435 0
|
||
0.50 70 72.9% 78.6% 0.583 0
|
||
0.60 56 58.3% 91.1% 0.721 0
|
||
0.70 46 47.9% 95.7% 0.904 0
|
||
0.80 32 33.3% 96.9% 0.880 0
|
||
0.90 12 12.5% 100.0% 1.000 0
|
||
|
||
## 13. Action-threshold curve
|
||
|
||
### Experiment A
|
||
threshold action_P action_R false_act
|
||
----------------------------------------
|
||
0.40 0.667 0.667 6
|
||
0.50 0.846 0.611 2
|
||
0.60 0.857 0.333 1
|
||
0.70 0.750 0.167 1
|
||
0.80 1.000 0.056 0
|
||
0.90 0.000 0.000 0
|
||
|
||
### Experiment B
|
||
threshold action_P action_R false_act
|
||
----------------------------------------
|
||
0.40 0.000 0.000 0
|
||
0.50 0.000 0.000 0
|
||
0.60 0.000 0.000 0
|
||
0.70 0.000 0.000 0
|
||
0.80 0.000 0.000 0
|
||
0.90 0.000 0.000 0
|
||
|
||
## 14. Model artifact size and runtime cost
|
||
|
||
- Trainable parameters: 2310
|
||
- 6 classes × 384 features = 2304 weights
|
||
- 6 bias terms
|
||
- Serialized head size: 9240 bytes (9.0 KB)
|
||
- Additional inference FLOPs: 2304 multiply-accumulates
|
||
- Incremental cost (e5 already computed): ~2304 FLOPs, <1µs
|
||
- Cost if semantic router must trigger its own e5: full ONNX inference (~384 × 128 × 12 = ~590K FLOPs)
|
||
|
||
## 16. Recommendation
|
||
|
||
**need more data**
|
||
|
||
All-example F1 (0.620) is acceptable but residual-only F1 (0.429) drops, suggesting the contrast-family examples are hard for a linear classifier. More contrastive training data may help.
|
||
|
||
## 17. Commit hash for experiment tooling
|
||
|
||
`59a0a08d329fbcbefad4ec858cf8c6cc07014a36`
|