# Semantic Router Linear Head Experiment — Report ## 1. Exact e5 representation used - **Model**: model_quantized@384/tok2 - **Checkpoint**: models/embedder/multilingual-e5-small/model_quantized.onnx - **Tokenizer**: models/embedder/multilingual-e5-small/tokenizer.json - **Dimension**: 384 - **Pooling**: mean-pool + L2-normalize - **Normalization**: L2 - **Input template**: query: ## 2. Development/residual row counts - Total corpus: 136 - Frozen holdout: 21 - Development pool: 115 - Fast-path resolved: 23 - Router-residual: 113 Route distribution (full corpus): - action: 22 - conversation: 7 - knowledge: 46 - memory_write: 19 - system: 7 - uncertain: 35 ## 3. Grouped fold composition Folds: 5 - Fold 0: eval=33 train=82 routes={'action': 4, 'conversation': 1, 'knowledge': 11, 'memory_write': 3, 'system': 1, 'uncertain': 13} - Fold 1: eval=16 train=99 routes={'action': 3, 'conversation': 1, 'knowledge': 5, 'memory_write': 4, 'system': 1, 'uncertain': 2} - Fold 2: eval=22 train=93 routes={'action': 3, 'conversation': 1, 'knowledge': 7, 'memory_write': 4, 'system': 2, 'uncertain': 5} - Fold 3: eval=22 train=93 routes={'action': 4, 'conversation': 1, 'knowledge': 6, 'memory_write': 4, 'system': 2, 'uncertain': 5} - Fold 4: eval=22 train=93 routes={'action': 4, 'conversation': 2, 'knowledge': 8, 'memory_write': 2, 'system': 1, 'uncertain': 5} ## 4. Selected regularization ### Experiment A: All development examples - Best C: 100.0 - Mean accuracy: 65.2% ± 6.2% - Mean macro F1: 0.597 ± 0.070 - Total false actions (CV): 10 ### Experiment B: Router-residual only - Best C: 100.0 - Mean accuracy: 66.5% ± 7.6% - Mean macro F1: 0.475 ± 0.083 - Total false actions (CV): 0 ### Stability across folds C=0.01 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%'] C=0.1 acc=32.0%±3.0% f1=0.081±0.006 folds_acc=['33.3%', '31.2%', '31.8%', '27.3%', '36.4%'] C=1.0 acc=44.8%±7.0% f1=0.190±0.049 folds_acc=['36.4%', '37.5%', '50.0%', '45.5%', '54.5%'] C=10.0 acc=58.3%±4.6% f1=0.403±0.086 folds_acc=['51.5%', '62.5%', '63.6%', '59.1%', '54.5%'] C=100.0 acc=65.2%±6.2% f1=0.597±0.070 folds_acc=['54.5%', '62.5%', '68.2%', '68.2%', '72.7%'] ## 5. All-example CV metrics - Accuracy: 64.3% - Macro F1: 0.620 - False-action count: 10 - False-action rate: 8.7% - Action precision: 0.545 - Action recall: 0.667 - Uncertain precision: 0.800 - Uncertain recall: 0.533 Per-class metrics: action P=0.545 R=0.667 F1=0.600 (n=18) conversation P=1.000 R=0.500 F1=0.667 (n=6) knowledge P=0.756 R=0.838 F1=0.795 (n=37) memory_write P=0.346 R=0.529 F1=0.419 (n=17) system P=1.000 R=0.429 F1=0.600 (n=7) uncertain P=0.800 R=0.533 F1=0.640 (n=30) Confusion matrix (rows=expected, cols=predicted): action conversation knowledge memory_write system uncertain action 12 0 0 5 0 1 conversation 0 3 1 0 0 2 knowledge 4 0 31 2 0 0 memory_write 4 0 3 9 0 1 system 2 0 2 0 3 0 uncertain 0 0 4 10 0 16 ## 6. Residual-only CV metrics - Accuracy: 64.6% - Macro F1: 0.429 - False-action count: 0 - False-action rate: 0.0% - Action precision: 0.000 - Action recall: 0.000 - Uncertain precision: 0.667 - Uncertain recall: 0.533 Per-class metrics: action P=0.000 R=0.000 F1=0.000 (n=5) conversation P=1.000 R=0.500 F1=0.667 (n=6) knowledge P=0.750 R=0.917 F1=0.825 (n=36) memory_write P=0.400 R=0.625 F1=0.488 (n=16) system P=0.000 R=0.000 F1=0.000 (n=3) uncertain P=0.667 R=0.533 F1=0.593 (n=30) Confusion matrix (rows=expected, cols=predicted): action conversation knowledge memory_write system uncertain action 0 0 0 2 0 3 conversation 0 3 1 0 0 2 knowledge 0 0 33 3 0 0 memory_write 0 0 4 10 0 2 system 0 0 2 0 0 1 uncertain 0 0 4 10 0 16 ## 7. Legacy-vs-linear comparison ### All examples metric legacy linear e5 delta ------------------------------------------------------- accuracy 52.2% 64.3% 12.1% macro F1 — 0.620 — action precision — 0.545 — false-action rate 19.9% 8.7% -11.2% uncertain F1 0.000 0.640 0.640 ### Router-residual only metric legacy linear e5 delta ------------------------------------------------------- accuracy 40.8% 64.6% 23.8% macro F1 — 0.429 — false-action rate — 0.0% — ## 8. Fold variance All-example CV: Fold 0: acc=54.5% f1=0.507 false_action=1 Fold 1: acc=62.5% f1=0.579 false_action=2 Fold 2: acc=68.2% f1=0.554 false_action=2 Fold 3: acc=68.2% f1=0.632 false_action=5 Fold 4: acc=72.7% f1=0.712 false_action=0 Residual-only CV: Fold 0: acc=53.6% f1=0.500 false_action=0 Fold 1: acc=75.0% f1=0.573 false_action=0 Fold 2: acc=72.2% f1=0.411 false_action=0 Fold 3: acc=68.4% f1=0.541 false_action=0 Fold 4: acc=63.2% f1=0.351 false_action=0 ## 9. False-action repair/new-error analysis Note: Legacy per-example predictions were not available for this experiment. The legacy baseline was measured in aggregate in the Go test suite. Learned router false-action cases (out-of-fold): en-query-003: 'show me this week's weight' (true=knowledge, proba(action)=0.402) ru-query-014: 'я успеваю до дедлайна' (true=knowledge, proba(action)=0.343) ru-note-003: 'заметка про настройку vlan на свитче' (true=memory_write, proba(action)=0.335) ru-query-005: 'напоминания на завтра есть' (true=knowledge, proba(action)=0.336) ru-fact-009: 'отметь что я выпил таблетки утром' (true=memory_write, proba(action)=0.437) ru-query-011: 'почему сервер тормозит' (true=knowledge, proba(action)=0.381) ru-fact-005: 'поспал часов пять' (true=memory_write, proba(action)=0.557) ru-note-006: 'добавь в задачи купить молоко' (true=memory_write, proba(action)=0.739) ru-sys-003: 'переходи в тихий режим' (true=system, proba(action)=0.404) en-sys-002: 'turn quiet mode back on' (true=system, proba(action)=0.410) ## 10. Contrast-family results ### Experiment A (all dev) family count correct accuracy false_act ------------------------------------------------------------ negation 5 3 60.0% 0 question 5 5 100.0% 0 reported_speech 6 3 50.0% 0 quotation 6 4 66.7% 0 hypothetical 6 3 50.0% 0 capability_question 6 6 100.0% 0 ### Experiment B (residual only) family count correct accuracy false_act ------------------------------------------------------------ negation 5 3 60.0% 0 question 5 5 100.0% 0 reported_speech 6 3 50.0% 0 quotation 6 4 66.7% 0 hypothetical 6 3 50.0% 0 capability_question 6 6 100.0% 0 ## 11. Calibration metrics ### Experiment A - ECE: 0.116 - Brier score: 0.418 - Log loss: 0.832 ### Experiment B - ECE: 0.159 - Brier score: 0.411 - Log loss: 0.804 ## 12. Abstention curves ### Experiment A (all dev) threshold n_accepted coverage accuracy macro_f1 false_act -------------------------------------------------------------- 0.40 98 85.2% 72.4% 0.654 6 0.50 75 65.2% 84.0% 0.690 2 0.60 56 48.7% 92.9% 0.851 1 0.70 42 36.5% 97.6% 0.714 1 0.80 28 24.3% 100.0% 1.000 0 0.90 10 8.7% 100.0% 1.000 0 ### Experiment B (residual only) threshold n_accepted coverage accuracy macro_f1 false_act -------------------------------------------------------------- 0.40 86 89.6% 67.4% 0.435 0 0.50 70 72.9% 78.6% 0.583 0 0.60 56 58.3% 91.1% 0.721 0 0.70 46 47.9% 95.7% 0.904 0 0.80 32 33.3% 96.9% 0.880 0 0.90 12 12.5% 100.0% 1.000 0 ## 13. Action-threshold curve ### Experiment A threshold action_P action_R false_act ---------------------------------------- 0.40 0.667 0.667 6 0.50 0.846 0.611 2 0.60 0.857 0.333 1 0.70 0.750 0.167 1 0.80 1.000 0.056 0 0.90 0.000 0.000 0 ### Experiment B threshold action_P action_R false_act ---------------------------------------- 0.40 0.000 0.000 0 0.50 0.000 0.000 0 0.60 0.000 0.000 0 0.70 0.000 0.000 0 0.80 0.000 0.000 0 0.90 0.000 0.000 0 ## 14. Model artifact size and runtime cost - Trainable parameters: 2310 - 6 classes × 384 features = 2304 weights - 6 bias terms - Serialized head size: 9240 bytes (9.0 KB) - Additional inference FLOPs: 2304 multiply-accumulates - Incremental cost (e5 already computed): ~2304 FLOPs, <1µs - Cost if semantic router must trigger its own e5: full ONNX inference (~384 × 128 × 12 = ~590K FLOPs) ## 16. Recommendation **need more data** All-example F1 (0.620) is acceptable but residual-only F1 (0.429) drops, suggesting the contrast-family examples are hard for a linear classifier. More contrastive training data may help. ## 17. Commit hash for experiment tooling `59a0a08d329fbcbefad4ec858cf8c6cc07014a36`