From bb6bd8efb996c4f0a226f582b4703e99c01dd945 Mon Sep 17 00:00:00 2001 From: claude Date: Tue, 8 Sep 2026 01:27:03 +0400 Subject: [PATCH] =?UTF-8?q?router/semantic:=20slice=2022=20report=20?= =?UTF-8?q?=E2=80=94=20linear=20e5=20at=20parity=20with=20the=20deployed?= =?UTF-8?q?=20cascade,=20cross-family=20transfer=20absent,=20143=20illegal?= =?UTF-8?q?=20actions,=20levers=20are=20deterministic?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...09-08-slice22-residual-nonaction-router.md | 315 ++++++++++++++++++ docs/evals/CLAUDE.md | 1 + 2 files changed, 316 insertions(+) create mode 100644 docs/evals/2026-09-08-slice22-residual-nonaction-router.md diff --git a/docs/evals/2026-09-08-slice22-residual-nonaction-router.md b/docs/evals/2026-09-08-slice22-residual-nonaction-router.md new file mode 100644 index 0000000..d05cec3 --- /dev/null +++ b/docs/evals/2026-09-08-slice22-residual-nonaction-router.md @@ -0,0 +1,315 @@ +# On the frozen 1652-row five-way residual non-action task the deployed trained cascade (routing heads + e5 centroid classifier, LLM out) already matches a fresh linear head on the same frozen embeddings — 0.715 vs 0.712 macro-F1 — while the deterministic hash-classifier floor resolves only 10.9% of the pool (acc 0.171, 89.1% clarify); cross-family generalization is structurally absent (leave-one-family-out macro-F1 0.150; fact holdout acc 0.018; system/conversation/uncertain holdouts 0.000), the trained cascade calls an executable action on 143 residual non-action rows (8.7%; 91 of them knowledge), and the five-way head swallows 94% of the 766 action OOD rows into non-action buckets at argmax — a new representation or a new head are not justified; the levers are deterministic + +Date: 2026-09-08 · Task: slice 22 (brief after the accepted slice 21, task/725) · Box: workpc, Arch, RX 7900 GRE, 32 GB · Build: `cmd/semantic-router-experiment/slice22` (Go 1.25.12, `-mode legacy`/`-mode heads`) + `slice22_emit.py` + `slice22_main.py` in the frozen venv `/tmp/mvn-exp-venv` (sklearn 1.9.0). ONNX for the heads run via `MAVEN_ONNX_LIB` pointing at the venv's `libonnxruntime.so.1.29.0`. + +## 0. Frozen artifacts and population + +```text +corpus (3266 → frozen 3025): dev pool 2490, byte-identical to slices 16-21 +dev residual (fast_path_resolved=false): 2418 +residual non-action pool (this slice): 1652 + knowledge 715 / memory_write 553 / system 184 / conversation 93 / uncertain 107 + 53 family_id = 53 split_group (no row differs); folds {0:317, 1:140, 2:410, 3:408, 4:377} +action rows held out as OOD probes: 766 +frozen embeddings: model_quantized@384/tok2, mean-pool + L2, 'query: ' prefix +``` + +Artifacts under `/tmp/mvn-s22/`: `pool.json` (1652 rows, `idx` = position in dev +pool dev-order, so embeddings align by index with `slice19.load_dev`), +`ood.json` (766), `legacy.json` (mode `legacy`), `legacy_heads.json` (mode +`heads`), `results.json` (all sections below). All three JSON outputs are +reproducible: the Go harness is deterministic, and the Python sections carry +`random_state=42`. + +## 1. What is being asked + +When the fast path misses, the execution-frame guard passes, and no executable +action is warranted, the utterance is one of five residual non-action +semantics: **conversation, knowledge, memory_write, system, uncertain**. This +slice measures whether the deployed e5-small embeddings (384-d, frozen) with a +linear softmax head suffice for that five-way decision, against three floors +and two deployed baselines. The executable-action population is out of scope +for the head (closed in slices 18-21); its rows are OOD probes only. + +Decision criteria carried from the brief, judged in §13: +e5-linear sufficient / sparse sufficient / corpus-taxonomy problem / new +representation justified. + +## 2. Method and baselines + +Five-way projection rules (the daemon's behaviour, `project()` in +`legacy_main.go`): `chat → conversation`, `query → knowledge`, +`fact/note → memory_write`, `system → system`; `act/reminder` on a trusted +non-action row is recorded **verbatim as `action` with +`illegal_action_prediction=true`** and never softly re-mapped to `uncertain`; +`Clarify` and route errors become `uncertain`. In the metrics of §4-§5 the 143 +`action` projections count as errors (predicted `uncertain`), so the reported +scores already carry the price of the false-action leak. + +Three routers measured (all: stage-0 grammars, `StubDateTimeParser`, +`DefaultFactParser`, act allowlist of 18 verbs, threshold 0.55): + +```text +legacy buildMinimalRouter(): StageZeroGrammars + HashEmbedder(1024) + nearest-centroid classifier, seeds models/seeds (331 examples) + → the deterministic floor, no model, no ONNX +heads grammars + RouterHeads (router_heads.onnx, 0.6 decline) + e5 ONNX + embedder nearest-centroid classifier (same seeds) + → the deployed cascade with the LLM stage out (docs/routing.md + pickLLMRouter → classifier); the production offline behaviour +``` + +The two modes were smoke-verified to diverge on the same 40-row slice (the +NN heads fire at softmax 0.94-0.98 on residual rows the hash floor clarifies at +0.16-0.24), and the full-population files were checked for internal +consistency (producer/stage/stage-0 counts add to 1652) before use. Any number +in this report comes from `results.json`, not from a transcript. + +## 3. Stage-0 grammar hit under the live rule set: corpus drift + +185 of the 1652 "residual" rows are resolved by the **current** stage-0 +grammars (`producer=grammar`): by intent `query 157 / note 14 / chat 7 / +system 6 / reminder 1`, by truth-route `knowledge 157 / memory_write 12 / +system 6 / conversation 2 / uncertain 8`. `corpus fast_path_resolved` was set +by a corpus-side mirror (`cmd/corpus-factory/main.go`), not by the live rule +set; a grammar landed (or the mirror froze) after the corpus was built. The +pool remains 1652 for the primary metrics — dropping the 185 would not move +any conclusion below — and this drift is reported separately as +`grammar_drift: hits 185 / grammar-pure 1467`. The mirror should be reconciled +with the live grammar set. + +## 4. Baseline A — the deterministic floor (`legacy`) + +```text +acc 0.1707 macro-F1 0.1202 illegal_action_prediction 3 +conversation P 0.000 R 0.000 F1 0.000 n=93 +knowledge P 1.000 R 0.220 F1 0.360 n=715 +memory_write P 0.857 R 0.022 F1 0.042 n=553 +system P 1.000 R 0.033 F1 0.063 n=184 +uncertain P 0.073 R 1.000 F1 0.135 n=107 +``` + +The hash classifier fires above 0.55 on 180 rows (10.9%); the other 1472 +(89.1%) are below-threshold `uncertain`. The three illegal predictions are all +`uncertain:incomplete-reminder` ("напомни", "напомни, пожалуйста", "я хочу +напомни") — the reminder grammar fires and the daemon would drive an action +probe on a row the corpus trusted as non-action. This is the boundary class +the deterministic floor cannot price, and it is small. + +## 5. Baseline B — the deployed cascade with the LLM out (`heads`) + +```text +acc 0.7724 macro-F1 0.7150 illegal_action_prediction 143 +conversation P 0.550 R 0.828 F1 0.661 n=93 +knowledge P 0.858 R 0.744 F1 0.797 n=715 +memory_write P 0.940 R 0.825 F1 0.879 n=553 +system P 0.924 R 0.658 F1 0.768 n=184 +uncertain P 0.326 R 0.841 F1 0.470 n=107 +``` + +Producer mix: NN-heads 1176, e5-classifier 291, grammar 185. The NN heads name +`act` on 159 rows and `reminder` on 18; **143 of those land on trusted +non-action rows** (`illegal`), split `knowledge 91 / memory_write 46 / system 2 / +uncertain 4`. By family the biggest offenders are `knowledge:capability-ha` 45, +`knowledge:capability-tool` 38 (i.e. 83 capability-question rows — the slice-21 +population — called executable), `mw-free-remember` 35, `kq-homelab-disk` 5, +`mw-note-task` 5. Producer split of the illegal 143: classifier 102 (conf +0.89-0.91), NN-heads 40, grammar 1. Example rows: "хватает ли места на диске" +→ `act` 0.90; "я хочу покажи шаги за неделю" → `reminder` 0.91; "машинное +обучение опиши" → `act` 0.89. The trained components cannot refuse: their +largest directional error is toward execution, which is exactly the boundary +the deterministic stage-0 + execution-frame-guard arm owns. + +This is the same shape the fresh head shows in §8: confidence 0.89-0.91 is not +a refusal signal either. + +## 6. The fresh e5-linear head (the ask) + +Out-of-fold, grouped by the pool's split folds, C grid {0.1, 1, 10}: + +```text +C=0.1 acc 0.6096 macro-F1 0.2728 +C=1.0 acc 0.6949 macro-F1 0.5974 +C=10.0 acc 0.7222 macro-F1 0.7117 ← used everywhere below +``` + +Per-fold OOF (C=10): fold 0 acc 0.782 / macro 0.783; fold 1 (n=140) 0.736 / +0.605; fold 2 0.705 / 0.576; fold 3 0.713 / 0.710; fold 4 0.695 / 0.592; +mean acc 0.726 ± 0.031, mean macro-F1 0.653 ± 0.080. + +```text +OOF per class (C=10): +conversation P 0.840 R 0.731 F1 0.782 n=93 +knowledge P 0.713 R 0.761 F1 0.736 n=715 +memory_write P 0.720 R 0.749 F1 0.734 n=553 +system P 0.687 R 0.500 F1 0.579 n=184 +uncertain P 0.758 R 0.701 F1 0.728 n=107 +``` + +The fresh head and the deployed cascade land at the same operating point: +**OOF 0.712 vs deployed 0.715 macro-F1, 0.722 vs 0.772 accuracy.** The linear +head on the deployed embeddings does not beat what is already wired, and the +accuracy gap is explained by the 143 illegal rows the head cannot make (it has +no action class) — a structural, not a representational, difference. + +## 7. Floors + +```text +majority (all knowledge) acc 0.4328 macro-F1 0.1208 +centroid (cosine, in-fold means) acc 0.6731 macro-F1 0.6494 +sparse word+char TF-IDF logistic acc 0.6247 macro-F1 0.4463 vocab 7079 +e5-linear (the ask) acc 0.7222 macro-F1 0.7117 +``` + +Sparse is below e5-linear on both axes despite a 7079-term vocabulary; the +lexical signal is present but orders of magnitude weaker than the embedding. +Centroid cosine is the closest floor and e5-linear beats it by +0.062 macro-F1. + +## 8. Cross-family generalization is absent + +Route-family leave-*-out on the frozen 53 families. Each group below is a +single-route family set, so the route is literally absent from training when +the family is removed: + +```text +family (route) n acc macro-F1 dominant prediction +capability (kno) 126 0.2619 0.0830 knowledge (F1 0.42) +world (kno) 143 0.7622 0.1730 knowledge (F1 0.87) +calendar (kno) 92 0.6304 0.1547 knowledge (F1 0.77) +recall (kno) 114 0.7632 0.1731 knowledge (F1 0.87) +fact (mwr) 389 0.0180 0.0071 memory_write (F1 0.04) +note (mwr) 104 0.2212 0.0724 memory_write (F1 0.36) +remember (mwr) 60 0.2000 0.0667 memory_write (F1 0.33) +system (sys) 184 0.0000 0.0000 (nothing) +conversation (conv) 93 0.0000 0.0000 (nothing) +uncertain (unc) 107 0.0000 0.0000 (nothing) +leave-one-family-out over all 53: acc mean 0.682, macro-F1 mean 0.149 +``` + +The world/calendar/recall rows read as "good acc" only because knowledge is the +majority: the head's macro-F1 collapses (0.15-0.17). On fact, correct +predictions fall to 1.8%; on system, conversation and uncertain to exactly +zero. Within-family OOF macro-F1 is 0.71 (all routes present), and removing one +family drops it to 0.15. The head is a per-family-template memoriser: it +transfers **no route semantics** to an unseen template family. This is the +ceiling of the frozen representation, not a hyperparameter problem. + +## 9. knowledge vs memory_write, matched pairs + +`build_pairs` pairs every recall/knowledge row against every write row that +shares a corpus-justified lexeme, and asks whether the model puts the higher +probability on the truth side (memory_write for write, knowledge for recall): + +```text +lexeme pairs mw_over_k order_acc mean_margin +"вод" 480 0.863 +0.298 +"задач" 720 0.735 +0.122 +"dns/сервер/vlan" 180 0.106 -0.228 ← homelab notes vs homelab status +``` + +The fact/recall contrast ("воды попил" vs "сколько воды") separates well. The +**collision is the note-keep vs report-family**: `note:homelab` and +`note:task` write rows are ranked *below* their `knowledge:homelab-status` and +`knowledge:task-check` partners 89% of the time, with a negative margin. The +frozen embeddings put both sides of "дом" as one cluster; the head cannot +resolve intent direction there. This is the concrete, smallest instance of the +§8 collapse and the strongest single argument that a deterministic +note-vs-recall disambiguator (subject class, scheduling verbs, report vs +record verbs) would buy more than any representation retrain. + +## 10. uncertain as an explicit class + +```text +OOF: uncertain P 0.758 / R 0.701 / F1 0.728 (n=107) +pulled out of uncertain (true uncertain, mislabelled): knowledge 14, + memory_write 11, conversation 7, system 0 +pulled into uncertain (not uncertain, predicted uncertain): conversation 9, + knowledge 7, memory_write 5, system 3 +``` + +`uncertain` is taught and recalled (R 0.70), and the pulls are small and +mirrored — the "ask instead" bucket behaves like a class, which is what the +daemon needs it to be only if the ask threshold is real (§11). + +## 11. Calibration, abstention, and the action OOD + +Confidence (max-softmax of the OOF head): correct-mean 0.766, wrong-mean 0.612, +ECE 0.0275, log-loss 0.7365, macro-Brier 0.0781. The head is well calibrated +on-pool. + +```text +threshold (max-softmax) coverage accuracy macro-F1 +0.10 1.000 0.722 0.712 +0.28 0.996 0.725 0.715 +0.46 0.907 0.756 0.757 +0.64 0.676 0.822 0.829 +0.82 0.379 0.887 0.913 +``` + +**Action OOD probes** — the 766 frozen action rows, scored by the same five-way +head, are the wrong-population test the head must refuse: + +```text +argmax class: knowledge 370 (48.3%), memory_write 347 (45.3%), + uncertain 38 (5.0%), conversation 11, system 0 +conf mean 0.653, conf median 0.644, conf > 0.9 on 46 rows (6.0%) +``` + +94.6% of action utterances are named a resolvable non-action route at argmax, +and 6% land above the pool's own 0.82 high-confidence line. A five-way head +replacing the current cascade would turn action rows into confident +knowledge/write decisions. Two consequences: (a) the head must stay behind the +deterministic act/reminder + execution-frame-guard arm and never precede it — +the current cascade order already does that; (b) it cannot double as an +action-refuser by thresholding, so the reminder/act grammar family stays the +only refusal mechanism, and its 143-row miss rate (§5) is the real open +boundary for the trained arm, priced at 8.7% of residual non-action. + +## 12. Artifact cost + +Linear head: 1925 params (5×384 + 5 bias), 7.52 KiB fp32, incremental +decision latency mean 190 µs / p50 165 µs (pure head, Python timing on the +box; the shared cost is the e5 forward the routing-heads path already pays). +The head itself is negligible; the representation forward is not the lever. + +## 13. Verdict on the decision criteria + +| criterion | verdict | +| --- | --- | +| e5-linear suffices | **No.** In-fold parity with what is deployed (0.712 vs 0.715 macro-F1), but it transfers nothing across template families (LOO macro-F1 0.149) and cannot abstain on action rows (94.6% swallowed). | +| sparse suffices | **No.** 0.6247 / 0.4463, below the linear head on both axes. | +| corpus-taxonomy problem | **Partly.** 185 rows the corpus called residual are resolved by the live grammar set today (mirror staleness), and the two largest illegal-action families are capability questions — a corpus-trust boundary, not a representation one. | +| new representation justified | **No on this evidence.** The ceiling is family-lexicon transfer, not embedding capacity: replacing the frozen e5 changes nothing measured; the floors show the same collapse. A new representation would need to demonstrate the §8 holdouts moving before it earns a retrain. | + +## 14. Operational implications + +* **Keep the cascade order.** Stage-0 grammars and the execution-frame guard + (slice 21) are the only two mechanisms that keep the 8.7% false-action leak + from reaching tools; neither the linear head nor the deployed NN heads can + be promoted to the action boundary. +* **Reconcile the corpus fast-path mirror** against the live grammar set (185 + drifting rows). Purely deterministic, zero-model, and removes a class of + "legacy understates the router" arguments. +* **Add a deterministic note-vs-recall disambiguator** for the homelab/task + lexical clusters (the 0.106 ordering in §9): subject class, report verbs vs + record verbs, date/scheduling adverbs. Expected lever: most of the + memory_write recall gap without touching embeddings. +* **Watch the classifier's confident `act` calls** (conf 0.89-0.91 on + capability-question and free-remember rows, §5). The "хватает ли места на + диске" → `act` 0.90 row is a slice-21 capability question that the trained + arm still prices as executable; the deterministic guard is the only thing + standing between it and an action probe. +* **Do not threshold the uncertain bucket for action refusals.** The 38/766 + uncertain picks on OOD action rows show an explicit uncertain class is near + useless as a guard; the numbers are confidence, not category. + +## 15. Commit hashes + +* Tooling (emit step, Go harness both modes, Python experiment): `29f74dd` + (`router/semantic: slice 22 residual non-action router — emit, harness, + experiment`). +* Report + eval index row: this file with its `docs/evals/CLAUDE.md` entry. +* Artifacts under `/tmp/mvn-s22/` (not committed; reproducible by + `slice22_emit.py`, then `go run ./cmd/semantic-router-experiment/slice22/ + -mode legacy` and `-mode heads` with `MAVEN_ONNX_LIB`, then + `slice22_main.py`). \ No newline at end of file diff --git a/docs/evals/CLAUDE.md b/docs/evals/CLAUDE.md index 53db613..5f96dc8 100644 --- a/docs/evals/CLAUDE.md +++ b/docs/evals/CLAUDE.md @@ -54,6 +54,7 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the | [From-scratch tiny sequence pragmatics specialist (slice 19)](2026-09-08-tiny-sequence-pragmatics-specialist.md) | live | | [Pretrained rubert-tiny pragmatics baseline (slice 20)](2026-09-07-pretrained-rubert-tiny-pragmatics-baseline.md) | live | | [Deterministic execution-frame guard (slice 21)](2026-09-07-execution-frame-guard.md) | live | +| [Five-way residual non-action router: linear e5 vs the deployed cascade (slice 22)](2026-09-08-slice22-residual-nonaction-router.md) | live | `docs/routing.md` holds the arm table these feed. Cite from there, not from here.