diff --git a/docs/evals/2026-09-08-slice23-fast-path-metadata-reconciliation.md b/docs/evals/2026-09-08-slice23-fast-path-metadata-reconciliation.md new file mode 100644 index 0000000..59508e4 --- /dev/null +++ b/docs/evals/2026-09-08-slice23-fast-path-metadata-reconciliation.md @@ -0,0 +1,190 @@ +# Rebuilding corpus fast-path metadata from the real router moves the slice-22 1652-row five-way numbers by ≤2pp (heads 0.772→0.751 acc, e5-linear 0.722/0.712→0.720/0.731 macro-F1) and every aggregate slice-22 conclusion survives, so the learned non-action router investigation is closed as run; the residual pool is 1509 non-action + 720 action OOD rows, and the deterministic hash-classifier floor collapses to acc 0.066 because its earlier 0.171 was almost entirely the 185 rows the stale mirror had mislabelled as residual while the real grammars already resolve them + +Date: 2026-09-08 · Task: slice 23 (fast-path metadata reconciliation; postscript to the accepted slice 22, task/725) · Box: workpc, Arch, RX 7900 GRE, 32 GB · Build: `internal/router/semantic/fastpath.go` + `cmd/semantic-router-experiment/slice23/` (Go 1.25.12) + `slice23_emit.py`/`slice23_main.py` in the frozen venv `/tmp/mvn-exp-venv` (sklearn 1.9.0); ONNX via `MAVEN_ONNX_LIB` pointing at the venv's `libonnxruntime.so.1.29.0`. + +Replaces the corrected-population numbers in [slice 22](2026-09-08-slice22-residual-nonaction-router.md). The slice-22 file keeps its role for the run itself; its 1652-row pool is no longer the current measurement population. + +## 0. What changed and what did not + +The corpus's `fast_path_resolved` flag was stamped by a hand-written regex mirror +(`classifyFastPath` in `cmd/corpus-factory/main.go`) that drifted from the +stage-0 grammars. Slice 22 noticed 185 residual rows resolving at runtime. This +slice makes the flag a derivation from the real router and re-measures. + +```text +corpus 3025 rows, dev pool 2490, frozen holdout 535 unchanged +dataset_hash b27fd48f478ca477cab1e59773bb353a unchanged (texts) +fast_path_resolved 82 → 271 (dev 72 → 261, frozen 10, preserved) +residual 2943 → 2754 (dev 2418 → 2229, frozen 525, preserved) +residual non-action pool 1652 → 1509 + knowledge 715→558 / memory_write 553→541 / system 184→220 / conversation 93→91 / uncertain 107→99 +action OOD probes 766 → 720 +residual dev pool folds {0:317,1:140,2:410,3:408,4:377} → {0:247,1:111,2:373,3:386,4:392} +``` + +The rebuild changed **exactly the derived fields on exactly the 277 dev rows** +that disagree with the router: 0 rows differ in text, route, source, source_id, +split_group, or the non-`router_residual` tags. The 22 stale frozen rows (see §1) +are preserved verbatim per the merge rule — reported, never silently rewritten. + +## 1. The drift, decomposed (299 disagreements across 3025 corpus rows) + +`cmd/semantic-router-experiment/slice23` runs `DeriveFastPath` (the production +`TryFastPath` over `StageZeroGrammars` with the experiment act allowlist) on +every row and attributes the winner. Output: `/tmp/mvn-s23/drift.json`. + +```text +mirror_missed (stored residual, runtime resolves) 249 + dev 233 = 185 non-action + 48 action + dev non-action by grammar: calendar-query 66, definition-query 30, + narrative-query 26, task-capture 12, agenda-query 9, task-list-query 9, + plan-day-query 8, ambiguous-fragment 7, time-query 6, rest-of-day 5, + event-time 4, possession-statement 2, reminder-wakeword 1 + frozen 16 (possession 6, implicit-elapsed 3, wakeword-act 2, + 5 singles) +claimed_fast_now_miss (stored fast, runtime declines) 50 + dev 44 (system 42, action 2) + frozen 6 (memory_write "note: " rows) + every one has ≥1 stage-0 grammar matching its shape and declining the build; + none is a "no grammar matches" row +``` + +The **185 dev non-action `mirror_missed` rows are byte-identical in count and +route split** to the slice-22 harness's `grammar_hits_in_pool: 185` +(knowledge 157, memory_write 12, system 6, conversation 2, uncertain 8) — a +strong cross-check that `DeriveFastPath` equals what the slice-22 harness +measured through `Route()`. + +Root cause, two mechanisms: + +1. **Grammar coverage grew after the corpus froze.** `calendar-query` (67), + `definition-query` (31), `narrative-query` (27), `task-status` (24), + `agenda-query` (9), `possession-statement` (8), `plan-day-query` (8), + `event-time-query` (4) and the rest are current stage-0 rules with no + counterpart in the mirror. The mirror ever only knew a fixed verb list. +2. **The mirror's verb lists over-stamped.** The 50 `claimed_fast_now_miss` + rows ("покажи uptime", "note: кран на кухне капает", "what's the date + сегодня") matched `toolAliasRe`/`captureVerbRe`/`timeQueryRe`, but no real + grammar accepts them: rules like `task-list-query`/`list-capture` use a + catch-all `(?s)^\s*(.+)$` pattern with the real gate in `Build`, which + declines. The mirror had no concept of a build gate. + +The frozen 22 (16 + 6) are the same two mechanisms on holdout families, +including three `ru_routing_v1` rows ("что у меня стоит в календаре на +послезавтра", "расскажи про битву при Ватерлоо", "кто такой Линус Торвальдс?") +now resolved by calendar/narrative/definition grammars. + +## 2. The authoritative derivation + +`internal/router/semantic/fastpath.go` (non-test) owns: + +- `ExperimentActVerbs()` — the 18 verbs formerly duplicated in + `helpers_test.go`, `legacy_build.go` and `heads_main.go`; now one non-test + list, so propagation can never drift from measurement. +- `DeriveFastPath(text) FastPathOutcome{Matched, Grammar}` — builds the exact + router the legacy baseline measures (`router.StageZeroGrammars` + the + experiment act matcher + `StubDateTimeParser`/`DefaultFactParser`) and runs + `TryFastPath(NormalizedInput{Text, MatchText: NormalizeMatchText(Text)})`. + Grammar attribution replays the ordered first-accept walk the router performs, + including wake-stripped alternates and build-declined fall-through. + +Consumers: + +- `cmd/corpus-factory` stamps `fast_path_resolved`/`router_residual` from + `DeriveFastPath`; the regex mirror is deleted. +- `cmd/merge-corpus` validates every v2 row against the derivation and **fails + on a stale value**; frozen holdout drift is reported only. +- `internal/router/semantic/fastpath_invariant_test.go` asserts every dev row + satisfies `fast_path_resolved == DeriveFastPath(text).Matched`; frozen rows + are exempt by the merge rule. + +## 3. Rebuild proofs + +```text +factory v2: 3008 rows, text-set identical old↔new + diff = exactly the fast-path fields on 296 rows; 0 other diffs +merge: v2 dev rows 2490 all validated against the router (0 stale) + frozen 535 preserved; 22 drift rows reported, none rewritten +v1 embedded: 3025 rows, text-set identical; dataset_hash unchanged + diff = fast-path fields on the 277 dev rows; 0 other diffs +embeddings: regenerated (fast_path_count 82 → 271, residual_count 2943 → 2754) + all 3025 vectors differ by ≤ ~0.02 max-abs from the slice-22 file + (onnxruntime build noise); on the same 1509 pool the e5-linear + score moves only −0.0014 acc / −0.0002 macro-F1 (C=10), i.e. the + population effect, not the embedding noise, drives every delta +``` + +## 4. Baseline deltas on the corrected population + +| measure | slice 22 (n=1652) | slice 23 (n=1509) | +| --- | --- | --- | +| legacy hash floor | 0.1707 / 0.1202 (illegal 3) | **0.0663 / 0.0265 (illegal 2)** | +| deployed cascade (heads) | 0.7724 / 0.7150 (illegal 143) | 0.7515 / 0.7065 (illegal 142) | +| e5-linear C=10 | 0.7222 / 0.7117 | 0.7203 / 0.7310 | +| centroid (cosine) | 0.6731 / 0.6494 | 0.6753 / 0.6494 | +| sparse word+char (both) | 0.6247 / 0.4463 (vocab 6951) | 0.5991 / 0.4588 | +| majority | 0.4328 / 0.1208 | 0.3698 / 0.1080 | +| leave-one-family-out | 0.6824 / 0.1499, 53 fm | 0.6863 / 0.1519, 53 fm | +| fact holdout | 0.0180 (n 389) | 0.0077 (n 389) | +| world holdout | 0.7622 (n 143) | 0.6552 (n 87) | +| recall holdout | 0.7632 (n 114) | 0.5667 (n 90) | +| calendar holdout | 0.6304 (n 92) | 0.5185 (n 27) | +| K/MW water pairs | 0.863 (480), +0.298 | 0.850 (480), +0.275 | +| K/MW homelab pairs | 0.106 (180), −0.228 | 0.072 (180), −0.247 | +| K/MW task pairs | 0.735 (720), +0.122 | 0.403 (216), −0.095 | +| uncertain OOF P/R/F1 | 0.758 / 0.701 / 0.728 | 0.734 / 0.697 / 0.715 | +| action OOD swallowed | knowledge 370, mw 347 of 766 (conf>0.9: 0.060) | knowledge 364, mw 312 of 720 (conf>0.9: 0.056) | +| grammar hits in pool | 185 / 1467 pure | **0 / 1509 pure** | + +The hash floor's collapse is explained structurally: the 185 removed rows were +nearly all the hash classifier's correct knowledge hits (correct predictions +282 → 98, a one-to-one loss with the drift), so the deterministic floor "resolves" +the same rows the real grammars already own. + +## 5. The decisions requested + +1. **Drift decomposition** — §1; 299 = 249 + 50; the slice-22 185 reproduces exactly. +2. **Root cause** — §1; coverage growth + build-gate over-stamp, neither visible to a regex mirror. +3. **Authoritative derivation** — §2; `DeriveFastPath` on the standard experiment router; duplicated act list removed. +4. **Corpus-builder change** — §2-§3; factory stamps from the router, merge validates (fail on stale) + reports frozen drift, invariant test enforces it. +5. **Corrected counts** — §0; fast 82→271, residual 2943→2754, pool 1652→1509 non-action + 720 OOD; dataset hash unchanged proves only the derived field moved. +6. **Baseline deltas** — §4; all ≤2pp on the head scores; hash floor collapses because its output was the drifted rows. + +## 6. Conclusion survival and the gate + +Every aggregate slice-22 conclusion is unchanged on the corrected population: + +- **e5-linear ≈ deployed cascade.** e5 0.7203 vs heads 0.7515 acc (slice 22: + 0.7222 vs 0.7724); macro-F1 e5 0.7310 now *exceeds* heads 0.7065. The ≈3pp + gap stands; a head on frozen e5 embeddings adds nothing a linear probe lacks. +- **Cross-family transfer is structurally absent.** LOFO macro-F1 0.1519; + fact holdout 0.0077; system/conversation/uncertain holdouts 0.0000. +- **K/MW is a deterministic semantic defect.** Homelab pairs still order + backwards (0.072, margin −0.247); task pairs now order backwards too + (0.403, margin −0.095, pairs 720→216 as the calendar/task queries leave the + pool). Water stays right (0.850). No learned head fixes this at the margin. +- **The deterministic floor is not a baseline.** acc 0.0663 ≈ predicting + uncertain for everything. +- **Action OOD is still swallowed.** 676 of 720 (93.9%) land in + knowledge/memory_write at argmax; conf>0.9 5.6%. + +Decision gate from the brief: conclusions **materially the same**. The learned +non-action-router investigation is **closed**; no new representation or head is +justified. Next slice: the note-vs-recall disambiguator (the one genuine residual +deterministic defect the K/MW numbers keep surfacing). + +## Artifacts + +Re-run (corrected population): + +```sh +go run ./cmd/semantic-router-experiment/slice23/ -out /tmp/mvn-s23/drift.json +go build -o /tmp/mvn-s23/s23 ./cmd/semantic-router-experiment/slice22/ +MAVEN_ONNX_LIB=…/libonnxruntime.so.1.29.0 /tmp/mvn-s23/s23 -mode legacy -pool /tmp/mvn-s23/pool.json -out /tmp/mvn-s23/legacy.json +MAVEN_ONNX_LIB=… /tmp/mvn-s23/s23 -mode heads -pool /tmp/mvn-s23/pool.json -out /tmp/mvn-s23/legacy_heads.json +python3 slice23_emit.py # → /tmp/mvn-s23/{pool,ood,stats}.json (1509/720) +MAVEN_ONNX_LIB=… python3 slice23_main.py # → /tmp/mvn-s23/results.json +``` + +Corpus rebuild (proofs in §3): `corpus-factory -out /tmp/corpus_v2.json`, then +`merge-corpus -v1 corpus_v1.json -v2 /tmp/corpus_v2.json -out corpus_v1.json` +(identity of the change verified against `/tmp/mvn-s23/corpus_v1.old.json`). +Embedding regeneration (noise isolation against `/tmp/mvn-s23/embeddings.old.json`). \ No newline at end of file diff --git a/docs/evals/CLAUDE.md b/docs/evals/CLAUDE.md index 5f96dc8..e1b7a34 100644 --- a/docs/evals/CLAUDE.md +++ b/docs/evals/CLAUDE.md @@ -54,7 +54,8 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the | [From-scratch tiny sequence pragmatics specialist (slice 19)](2026-09-08-tiny-sequence-pragmatics-specialist.md) | live | | [Pretrained rubert-tiny pragmatics baseline (slice 20)](2026-09-07-pretrained-rubert-tiny-pragmatics-baseline.md) | live | | [Deterministic execution-frame guard (slice 21)](2026-09-07-execution-frame-guard.md) | live | -| [Five-way residual non-action router: linear e5 vs the deployed cascade (slice 22)](2026-09-08-slice22-residual-nonaction-router.md) | live | +| [Five-way residual non-action router: linear e5 vs the deployed cascade (slice 22)](2026-09-08-slice22-residual-nonaction-router.md) | superseded by 2026-09-08-slice23 | +| [Fast-path metadata rebuilt from the real router: the slice-22 numbers, corrected (slice 23)](2026-09-08-slice23-fast-path-metadata-reconciliation.md) | live | `docs/routing.md` holds the arm table these feed. Cite from there, not from here.