Maven/docs/evals/2026-09-07-linear-e5-router-experiment.md

Semantic Router Linear Head Experiment — Report

1. Exact e5 representation used

• Model: model_quantized@384/tok2
• Checkpoint: models/embedder/multilingual-e5-small/model_quantized.onnx
• Tokenizer: models/embedder/multilingual-e5-small/tokenizer.json
• Dimension: 384
• Pooling: mean-pool + L2-normalize
• Normalization: L2
• Input template: query:

2. Development/residual row counts

• Total corpus: 136
• Frozen holdout: 21
• Development pool: 115
• Fast-path resolved: 23
• Router-residual: 113

Route distribution (full corpus):

• action: 22
• conversation: 7
• knowledge: 46
• memory_write: 19
• system: 7
• uncertain: 35

3. Grouped fold composition

Folds: 5

• Fold 0: eval=33 train=82 routes={'action': 4, 'conversation': 1, 'knowledge': 11, 'memory_write': 3, 'system': 1, 'uncertain': 13}
• Fold 1: eval=16 train=99 routes={'action': 3, 'conversation': 1, 'knowledge': 5, 'memory_write': 4, 'system': 1, 'uncertain': 2}
• Fold 2: eval=22 train=93 routes={'action': 3, 'conversation': 1, 'knowledge': 7, 'memory_write': 4, 'system': 2, 'uncertain': 5}
• Fold 3: eval=22 train=93 routes={'action': 4, 'conversation': 1, 'knowledge': 6, 'memory_write': 4, 'system': 2, 'uncertain': 5}
• Fold 4: eval=22 train=93 routes={'action': 4, 'conversation': 2, 'knowledge': 8, 'memory_write': 2, 'system': 1, 'uncertain': 5}

4. Selected regularization

Experiment A: All development examples

• Best C: 100.0
• Mean accuracy: 65.2% ± 6.2%
• Mean macro F1: 0.597 ± 0.070
• Total false actions (CV): 10

Experiment B: Router-residual only

• Best C: 100.0
• Mean accuracy: 66.5% ± 7.6%
• Mean macro F1: 0.475 ± 0.083
• Total false actions (CV): 0

Stability across folds

C=0.01 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%'] C=0.1 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%'] C=1.0 acc=44.8%±7.0% f1=0.190±0.049 folds_acc=['36.4%', '37.5%', '50.0%', '45.5%', '54.5%'] C=10.0 acc=58.3%±4.6% f1=0.403±0.086 folds_acc=['51.5%', '62.5%', '63.6%', '59.1%', '54.5%'] C=100.0 acc=65.2%±6.2% f1=0.597±0.070 folds_acc=['54.5%', '62.5%', '68.2%', '68.2%', '72.7%']

5. All-example CV metrics

• Accuracy: 64.3%
• Macro F1: 0.620
• False-action count: 10
• False-action rate: 8.7%
• Action precision: 0.545
• Action recall: 0.667
• Uncertain precision: 0.800
• Uncertain recall: 0.533

Per-class metrics: action P=0.545 R=0.667 F1=0.600 (n=18) conversation P=1.000 R=0.500 F1=0.667 (n=6) knowledge P=0.756 R=0.838 F1=0.795 (n=37) memory_write P=0.346 R=0.529 F1=0.419 (n=17) system P=1.000 R=0.429 F1=0.600 (n=7) uncertain P=0.800 R=0.533 F1=0.640 (n=30)

Confusion matrix (rows=expected, cols=predicted): action conversation knowledge memory_write system uncertain action 12 0 0 5 0 1 conversation 0 3 1 0 0 2 knowledge 4 0 31 2 0 0 memory_write 4 0 3 9 0 1 system 2 0 2 0 3 0 uncertain 0 0 4 10 0 16

6. Residual-only CV metrics

• Accuracy: 64.6%
• Macro F1: 0.429
• False-action count: 0
• False-action rate: 0.0%
• Action precision: 0.000
• Action recall: 0.000
• Uncertain precision: 0.667
• Uncertain recall: 0.533

Per-class metrics: action P=0.000 R=0.000 F1=0.000 (n=5) conversation P=1.000 R=0.500 F1=0.667 (n=6) knowledge P=0.750 R=0.917 F1=0.825 (n=36) memory_write P=0.400 R=0.625 F1=0.488 (n=16) system P=0.000 R=0.000 F1=0.000 (n=3) uncertain P=0.667 R=0.533 F1=0.593 (n=30)

Confusion matrix (rows=expected, cols=predicted): action conversation knowledge memory_write system uncertain action 0 0 0 2 0 3 conversation 0 3 1 0 0 2 knowledge 0 0 33 3 0 0 memory_write 0 0 4 10 0 2 system 0 0 2 0 0 1 uncertain 0 0 4 10 0 16

7. Legacy-vs-linear comparison

All examples

metric legacy linear e5 delta

accuracy 52.2% 64.3% 12.1% macro F1 — 0.620 — action precision — 0.545 — false-action rate 19.9% 8.7% -11.2% uncertain F1 0.000 0.640 0.640

Router-residual only

metric legacy linear e5 delta

accuracy 40.8% 64.6% 23.8% macro F1 — 0.429 — false-action rate — 0.0% —

8. Fold variance

All-example CV: Fold 0: acc=54.5% f1=0.507 false_action=1 Fold 1: acc=62.5% f1=0.579 false_action=2 Fold 2: acc=68.2% f1=0.554 false_action=2 Fold 3: acc=68.2% f1=0.632 false_action=5 Fold 4: acc=72.7% f1=0.712 false_action=0

Residual-only CV: Fold 0: acc=53.6% f1=0.500 false_action=0 Fold 1: acc=75.0% f1=0.573 false_action=0 Fold 2: acc=72.2% f1=0.411 false_action=0 Fold 3: acc=68.4% f1=0.541 false_action=0 Fold 4: acc=63.2% f1=0.351 false_action=0

9. False-action repair/new-error analysis

Note: Legacy per-example predictions were not available for this experiment. The legacy baseline was measured in aggregate in the Go test suite.

Learned router false-action cases (out-of-fold): en-query-003: 'show me this week's weight' (true=knowledge, proba(action)=0.402) ru-query-014: 'я успеваю до дедлайна' (true=knowledge, proba(action)=0.343) ru-note-003: 'заметка про настройку vlan на свитче' (true=memory_write, proba(action)=0.335) ru-query-005: 'напоминания на завтра есть' (true=knowledge, proba(action)=0.336) ru-fact-009: 'отметь что я выпил таблетки утром' (true=memory_write, proba(action)=0.437) ru-query-011: 'почему сервер тормозит' (true=knowledge, proba(action)=0.381) ru-fact-005: 'поспал часов пять' (true=memory_write, proba(action)=0.557) ru-note-006: 'добавь в задачи купить молоко' (true=memory_write, proba(action)=0.739) ru-sys-003: 'переходи в тихий режим' (true=system, proba(action)=0.404) en-sys-002: 'turn quiet mode back on' (true=system, proba(action)=0.410)

10. Contrast-family results

Experiment A (all dev)

family count correct accuracy false_act

negation 5 3 60.0% 0 question 5 5 100.0% 0 reported_speech 6 3 50.0% 0 quotation 6 4 66.7% 0 hypothetical 6 3 50.0% 0 capability_question 6 6 100.0% 0

Experiment B (residual only)

family count correct accuracy false_act

negation 5 3 60.0% 0 question 5 5 100.0% 0 reported_speech 6 3 50.0% 0 quotation 6 4 66.7% 0 hypothetical 6 3 50.0% 0 capability_question 6 6 100.0% 0

11. Calibration metrics

Experiment A

• ECE: 0.116
• Brier score: 0.418
• Log loss: 0.832

Experiment B

• ECE: 0.159
• Brier score: 0.411
• Log loss: 0.804

12. Abstention curves

Experiment A (all dev)

threshold n_accepted coverage accuracy macro_f1 false_act

  0.40          98      85.2%      72.4%      0.654          6
  0.50          75      65.2%      84.0%      0.690          2
  0.60          56      48.7%      92.9%      0.851          1
  0.70          42      36.5%      97.6%      0.714          1
  0.80          28      24.3%     100.0%      1.000          0
  0.90          10       8.7%     100.0%      1.000          0

Experiment B (residual only)

threshold n_accepted coverage accuracy macro_f1 false_act

  0.40          86      89.6%      67.4%      0.435          0
  0.50          70      72.9%      78.6%      0.583          0
  0.60          56      58.3%      91.1%      0.721          0
  0.70          46      47.9%      95.7%      0.904          0
  0.80          32      33.3%      96.9%      0.880          0
  0.90          12      12.5%     100.0%      1.000          0

13. Action-threshold curve

Experiment A

threshold action_P action_R false_act

  0.40      0.667      0.667          6
  0.50      0.846      0.611          2
  0.60      0.857      0.333          1
  0.70      0.750      0.167          1
  0.80      1.000      0.056          0
  0.90      0.000      0.000          0

Experiment B

threshold action_P action_R false_act

  0.40      0.000      0.000          0
  0.50      0.000      0.000          0
  0.60      0.000      0.000          0
  0.70      0.000      0.000          0
  0.80      0.000      0.000          0
  0.90      0.000      0.000          0

14. Model artifact size and runtime cost

• Trainable parameters: 2310
  • 6 classes × 384 features = 2304 weights
  • 6 bias terms
• Serialized head size: 9240 bytes (9.0 KB)
• Additional inference FLOPs: 2304 multiply-accumulates
• Incremental cost (e5 already computed): ~2304 FLOPs, <1µs
• Cost if semantic router must trigger its own e5: full ONNX inference (~384 × 128 × 12 = ~590K FLOPs)

16. Recommendation

need more data

All-example F1 (0.620) is acceptable but residual-only F1 (0.429) drops, suggesting the contrast-family examples are hard for a linear classifier. More contrastive training data may help.

17. Commit hash for experiment tooling

59a0a08d329fbcbefad4ec858cf8c6cc07014a36