router/semantic: experiment reports (slice 15)
- 2026-09-07-linear-e5-router-experiment: 136-example baseline (superseded) - 2026-09-07-expanded-corpus-linear-head-experiment: 3025-example expanded (live) - CLAUDE.md: index updates for both evals
This commit is contained in:
@@ -0,0 +1,263 @@
|
||||
# Semantic Router Linear Head Experiment — Report
|
||||
|
||||
## 1. Exact e5 representation used
|
||||
|
||||
- **Model**: model_quantized@384/tok2
|
||||
- **Checkpoint**: models/embedder/multilingual-e5-small/model_quantized.onnx
|
||||
- **Tokenizer**: models/embedder/multilingual-e5-small/tokenizer.json
|
||||
- **Dimension**: 384
|
||||
- **Pooling**: mean-pool + L2-normalize
|
||||
- **Normalization**: L2
|
||||
- **Input template**: query: <text>
|
||||
|
||||
## 2. Development/residual row counts
|
||||
|
||||
- Total corpus: 136
|
||||
- Frozen holdout: 21
|
||||
- Development pool: 115
|
||||
- Fast-path resolved: 23
|
||||
- Router-residual: 113
|
||||
|
||||
Route distribution (full corpus):
|
||||
- action: 22
|
||||
- conversation: 7
|
||||
- knowledge: 46
|
||||
- memory_write: 19
|
||||
- system: 7
|
||||
- uncertain: 35
|
||||
|
||||
## 3. Grouped fold composition
|
||||
|
||||
Folds: 5
|
||||
- Fold 0: eval=33 train=82 routes={'action': 4, 'conversation': 1, 'knowledge': 11, 'memory_write': 3, 'system': 1, 'uncertain': 13}
|
||||
- Fold 1: eval=16 train=99 routes={'action': 3, 'conversation': 1, 'knowledge': 5, 'memory_write': 4, 'system': 1, 'uncertain': 2}
|
||||
- Fold 2: eval=22 train=93 routes={'action': 3, 'conversation': 1, 'knowledge': 7, 'memory_write': 4, 'system': 2, 'uncertain': 5}
|
||||
- Fold 3: eval=22 train=93 routes={'action': 4, 'conversation': 1, 'knowledge': 6, 'memory_write': 4, 'system': 2, 'uncertain': 5}
|
||||
- Fold 4: eval=22 train=93 routes={'action': 4, 'conversation': 2, 'knowledge': 8, 'memory_write': 2, 'system': 1, 'uncertain': 5}
|
||||
|
||||
## 4. Selected regularization
|
||||
|
||||
### Experiment A: All development examples
|
||||
- Best C: 100.0
|
||||
- Mean accuracy: 65.2% ± 6.2%
|
||||
- Mean macro F1: 0.597 ± 0.070
|
||||
- Total false actions (CV): 10
|
||||
|
||||
### Experiment B: Router-residual only
|
||||
- Best C: 100.0
|
||||
- Mean accuracy: 66.5% ± 7.6%
|
||||
- Mean macro F1: 0.475 ± 0.083
|
||||
- Total false actions (CV): 0
|
||||
|
||||
### Stability across folds
|
||||
|
||||
C=0.01 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%']
|
||||
C=0.1 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%']
|
||||
C=1.0 acc=44.8%±7.0% f1=0.190±0.049 folds_acc=['36.4%', '37.5%', '50.0%', '45.5%', '54.5%']
|
||||
C=10.0 acc=58.3%±4.6% f1=0.403±0.086 folds_acc=['51.5%', '62.5%', '63.6%', '59.1%', '54.5%']
|
||||
C=100.0 acc=65.2%±6.2% f1=0.597±0.070 folds_acc=['54.5%', '62.5%', '68.2%', '68.2%', '72.7%']
|
||||
|
||||
## 5. All-example CV metrics
|
||||
|
||||
- Accuracy: 64.3%
|
||||
- Macro F1: 0.620
|
||||
- False-action count: 10
|
||||
- False-action rate: 8.7%
|
||||
- Action precision: 0.545
|
||||
- Action recall: 0.667
|
||||
- Uncertain precision: 0.800
|
||||
- Uncertain recall: 0.533
|
||||
|
||||
Per-class metrics:
|
||||
action P=0.545 R=0.667 F1=0.600 (n=18)
|
||||
conversation P=1.000 R=0.500 F1=0.667 (n=6)
|
||||
knowledge P=0.756 R=0.838 F1=0.795 (n=37)
|
||||
memory_write P=0.346 R=0.529 F1=0.419 (n=17)
|
||||
system P=1.000 R=0.429 F1=0.600 (n=7)
|
||||
uncertain P=0.800 R=0.533 F1=0.640 (n=30)
|
||||
|
||||
Confusion matrix (rows=expected, cols=predicted):
|
||||
action conversation knowledge memory_write system uncertain
|
||||
action 12 0 0 5 0 1
|
||||
conversation 0 3 1 0 0 2
|
||||
knowledge 4 0 31 2 0 0
|
||||
memory_write 4 0 3 9 0 1
|
||||
system 2 0 2 0 3 0
|
||||
uncertain 0 0 4 10 0 16
|
||||
|
||||
## 6. Residual-only CV metrics
|
||||
|
||||
- Accuracy: 64.6%
|
||||
- Macro F1: 0.429
|
||||
- False-action count: 0
|
||||
- False-action rate: 0.0%
|
||||
- Action precision: 0.000
|
||||
- Action recall: 0.000
|
||||
- Uncertain precision: 0.667
|
||||
- Uncertain recall: 0.533
|
||||
|
||||
Per-class metrics:
|
||||
action P=0.000 R=0.000 F1=0.000 (n=5)
|
||||
conversation P=1.000 R=0.500 F1=0.667 (n=6)
|
||||
knowledge P=0.750 R=0.917 F1=0.825 (n=36)
|
||||
memory_write P=0.400 R=0.625 F1=0.488 (n=16)
|
||||
system P=0.000 R=0.000 F1=0.000 (n=3)
|
||||
uncertain P=0.667 R=0.533 F1=0.593 (n=30)
|
||||
|
||||
Confusion matrix (rows=expected, cols=predicted):
|
||||
action conversation knowledge memory_write system uncertain
|
||||
action 0 0 0 2 0 3
|
||||
conversation 0 3 1 0 0 2
|
||||
knowledge 0 0 33 3 0 0
|
||||
memory_write 0 0 4 10 0 2
|
||||
system 0 0 2 0 0 1
|
||||
uncertain 0 0 4 10 0 16
|
||||
|
||||
## 7. Legacy-vs-linear comparison
|
||||
|
||||
### All examples
|
||||
metric legacy linear e5 delta
|
||||
-------------------------------------------------------
|
||||
accuracy 52.2% 64.3% 12.1%
|
||||
macro F1 — 0.620 —
|
||||
action precision — 0.545 —
|
||||
false-action rate 19.9% 8.7% -11.2%
|
||||
uncertain F1 0.000 0.640 0.640
|
||||
|
||||
### Router-residual only
|
||||
metric legacy linear e5 delta
|
||||
-------------------------------------------------------
|
||||
accuracy 40.8% 64.6% 23.8%
|
||||
macro F1 — 0.429 —
|
||||
false-action rate — 0.0% —
|
||||
|
||||
## 8. Fold variance
|
||||
|
||||
All-example CV:
|
||||
Fold 0: acc=54.5% f1=0.507 false_action=1
|
||||
Fold 1: acc=62.5% f1=0.579 false_action=2
|
||||
Fold 2: acc=68.2% f1=0.554 false_action=2
|
||||
Fold 3: acc=68.2% f1=0.632 false_action=5
|
||||
Fold 4: acc=72.7% f1=0.712 false_action=0
|
||||
|
||||
Residual-only CV:
|
||||
Fold 0: acc=53.6% f1=0.500 false_action=0
|
||||
Fold 1: acc=75.0% f1=0.573 false_action=0
|
||||
Fold 2: acc=72.2% f1=0.411 false_action=0
|
||||
Fold 3: acc=68.4% f1=0.541 false_action=0
|
||||
Fold 4: acc=63.2% f1=0.351 false_action=0
|
||||
|
||||
## 9. False-action repair/new-error analysis
|
||||
|
||||
Note: Legacy per-example predictions were not available for this experiment.
|
||||
The legacy baseline was measured in aggregate in the Go test suite.
|
||||
|
||||
Learned router false-action cases (out-of-fold):
|
||||
en-query-003: 'show me this week's weight' (true=knowledge, proba(action)=0.402)
|
||||
ru-query-014: 'я успеваю до дедлайна' (true=knowledge, proba(action)=0.343)
|
||||
ru-note-003: 'заметка про настройку vlan на свитче' (true=memory_write, proba(action)=0.335)
|
||||
ru-query-005: 'напоминания на завтра есть' (true=knowledge, proba(action)=0.336)
|
||||
ru-fact-009: 'отметь что я выпил таблетки утром' (true=memory_write, proba(action)=0.437)
|
||||
ru-query-011: 'почему сервер тормозит' (true=knowledge, proba(action)=0.381)
|
||||
ru-fact-005: 'поспал часов пять' (true=memory_write, proba(action)=0.557)
|
||||
ru-note-006: 'добавь в задачи купить молоко' (true=memory_write, proba(action)=0.739)
|
||||
ru-sys-003: 'переходи в тихий режим' (true=system, proba(action)=0.404)
|
||||
en-sys-002: 'turn quiet mode back on' (true=system, proba(action)=0.410)
|
||||
|
||||
## 10. Contrast-family results
|
||||
|
||||
### Experiment A (all dev)
|
||||
family count correct accuracy false_act
|
||||
------------------------------------------------------------
|
||||
negation 5 3 60.0% 0
|
||||
question 5 5 100.0% 0
|
||||
reported_speech 6 3 50.0% 0
|
||||
quotation 6 4 66.7% 0
|
||||
hypothetical 6 3 50.0% 0
|
||||
capability_question 6 6 100.0% 0
|
||||
|
||||
### Experiment B (residual only)
|
||||
family count correct accuracy false_act
|
||||
------------------------------------------------------------
|
||||
negation 5 3 60.0% 0
|
||||
question 5 5 100.0% 0
|
||||
reported_speech 6 3 50.0% 0
|
||||
quotation 6 4 66.7% 0
|
||||
hypothetical 6 3 50.0% 0
|
||||
capability_question 6 6 100.0% 0
|
||||
|
||||
## 11. Calibration metrics
|
||||
|
||||
### Experiment A
|
||||
- ECE: 0.116
|
||||
- Brier score: 0.418
|
||||
- Log loss: 0.832
|
||||
|
||||
### Experiment B
|
||||
- ECE: 0.159
|
||||
- Brier score: 0.411
|
||||
- Log loss: 0.804
|
||||
|
||||
## 12. Abstention curves
|
||||
|
||||
### Experiment A (all dev)
|
||||
threshold n_accepted coverage accuracy macro_f1 false_act
|
||||
--------------------------------------------------------------
|
||||
0.40 98 85.2% 72.4% 0.654 6
|
||||
0.50 75 65.2% 84.0% 0.690 2
|
||||
0.60 56 48.7% 92.9% 0.851 1
|
||||
0.70 42 36.5% 97.6% 0.714 1
|
||||
0.80 28 24.3% 100.0% 1.000 0
|
||||
0.90 10 8.7% 100.0% 1.000 0
|
||||
|
||||
### Experiment B (residual only)
|
||||
threshold n_accepted coverage accuracy macro_f1 false_act
|
||||
--------------------------------------------------------------
|
||||
0.40 86 89.6% 67.4% 0.435 0
|
||||
0.50 70 72.9% 78.6% 0.583 0
|
||||
0.60 56 58.3% 91.1% 0.721 0
|
||||
0.70 46 47.9% 95.7% 0.904 0
|
||||
0.80 32 33.3% 96.9% 0.880 0
|
||||
0.90 12 12.5% 100.0% 1.000 0
|
||||
|
||||
## 13. Action-threshold curve
|
||||
|
||||
### Experiment A
|
||||
threshold action_P action_R false_act
|
||||
----------------------------------------
|
||||
0.40 0.667 0.667 6
|
||||
0.50 0.846 0.611 2
|
||||
0.60 0.857 0.333 1
|
||||
0.70 0.750 0.167 1
|
||||
0.80 1.000 0.056 0
|
||||
0.90 0.000 0.000 0
|
||||
|
||||
### Experiment B
|
||||
threshold action_P action_R false_act
|
||||
----------------------------------------
|
||||
0.40 0.000 0.000 0
|
||||
0.50 0.000 0.000 0
|
||||
0.60 0.000 0.000 0
|
||||
0.70 0.000 0.000 0
|
||||
0.80 0.000 0.000 0
|
||||
0.90 0.000 0.000 0
|
||||
|
||||
## 14. Model artifact size and runtime cost
|
||||
|
||||
- Trainable parameters: 2310
|
||||
- 6 classes × 384 features = 2304 weights
|
||||
- 6 bias terms
|
||||
- Serialized head size: 9240 bytes (9.0 KB)
|
||||
- Additional inference FLOPs: 2304 multiply-accumulates
|
||||
- Incremental cost (e5 already computed): ~2304 FLOPs, <1µs
|
||||
- Cost if semantic router must trigger its own e5: full ONNX inference (~384 × 128 × 12 = ~590K FLOPs)
|
||||
|
||||
## 16. Recommendation
|
||||
|
||||
**need more data**
|
||||
|
||||
All-example F1 (0.620) is acceptable but residual-only F1 (0.429) drops, suggesting the contrast-family examples are hard for a linear classifier. More contrastive training data may help.
|
||||
|
||||
## 17. Commit hash for experiment tooling
|
||||
|
||||
`59a0a08d329fbcbefad4ec858cf8c6cc07014a36`
|
||||
Reference in New Issue
Block a user