kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

router/semantic: slice 23 report — corrected population keeps every slice-22 conclusion, hash floor collapses to 0.066, pool 1509 non-action + 720 OOD

Browse Source
This commit is contained in:
claude
2026-09-08 04:02:03 +04:00
parent be71ac406b
commit 1a309b283b
2 changed files with 192 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evals/2026-09-08-slice23-fast-path-metadata-reconciliation.md
+190
View File
@@ -0,0 +1,190 @@
# Rebuilding corpus fast-path metadata from the real router moves the slice-22 1652-row five-way numbers by ≤2pp (heads 0.772→0.751 acc, e5-linear 0.722/0.712→0.720/0.731 macro-F1) and every aggregate slice-22 conclusion survives, so the learned non-action router investigation is closed as run; the residual pool is 1509 non-action + 720 action OOD rows, and the deterministic hash-classifier floor collapses to acc 0.066 because its earlier 0.171 was almost entirely the 185 rows the stale mirror had mislabelled as residual while the real grammars already resolve them
Date: 2026-09-08 · Task: slice 23 (fast-path metadata reconciliation; postscript to the accepted slice 22, task/725) · Box: workpc, Arch, RX 7900 GRE, 32 GB · Build: `internal/router/semantic/fastpath.go` + `cmd/semantic-router-experiment/slice23/` (Go 1.25.12) + `slice23_emit.py`/`slice23_main.py` in the frozen venv `/tmp/mvn-exp-venv` (sklearn 1.9.0); ONNX via `MAVEN_ONNX_LIB` pointing at the venv's `libonnxruntime.so.1.29.0`.
Replaces the corrected-population numbers in [slice 22](2026-09-08-slice22-residual-nonaction-router.md). The slice-22 file keeps its role for the run itself; its 1652-row pool is no longer the current measurement population.
## 0. What changed and what did not
The corpus's `fast_path_resolved` flag was stamped by a hand-written regex mirror
(`classifyFastPath` in `cmd/corpus-factory/main.go`) that drifted from the
stage-0 grammars. Slice 22 noticed 185 residual rows resolving at runtime. This
slice makes the flag a derivation from the real router and re-measures.
```text
corpus 3025 rows, dev pool 2490, frozen holdout 535 unchanged
dataset_hash b27fd48f478ca477cab1e59773bb353a unchanged (texts)
fast_path_resolved 82 → 271 (dev 72 → 261, frozen 10, preserved)
residual 2943 → 2754 (dev 2418 → 2229, frozen 525, preserved)
residual non-action pool 1652 → 1509
knowledge 715→558 / memory_write 553→541 / system 184→220 / conversation 93→91 / uncertain 107→99
action OOD probes 766 → 720
residual dev pool folds {0:317,1:140,2:410,3:408,4:377} → {0:247,1:111,2:373,3:386,4:392}
```
The rebuild changed **exactly the derived fields on exactly the 277 dev rows**
that disagree with the router: 0 rows differ in text, route, source, source_id,
split_group, or the non-`router_residual` tags. The 22 stale frozen rows (see §1)
are preserved verbatim per the merge rule — reported, never silently rewritten.
## 1. The drift, decomposed (299 disagreements across 3025 corpus rows)
`cmd/semantic-router-experiment/slice23` runs `DeriveFastPath` (the production
`TryFastPath` over `StageZeroGrammars` with the experiment act allowlist) on
every row and attributes the winner. Output: `/tmp/mvn-s23/drift.json`.
```text
mirror_missed (stored residual, runtime resolves) 249
dev 233 = 185 non-action + 48 action
dev non-action by grammar: calendar-query 66, definition-query 30,
narrative-query 26, task-capture 12, agenda-query 9, task-list-query 9,
plan-day-query 8, ambiguous-fragment 7, time-query 6, rest-of-day 5,
event-time 4, possession-statement 2, reminder-wakeword 1
frozen 16 (possession 6, implicit-elapsed 3, wakeword-act 2, + 5 singles)
claimed_fast_now_miss (stored fast, runtime declines) 50
dev 44 (system 42, action 2) + frozen 6 (memory_write "note: " rows)
every one has ≥1 stage-0 grammar matching its shape and declining the build;
none is a "no grammar matches" row
```
The **185 dev non-action `mirror_missed` rows are byte-identical in count and
route split** to the slice-22 harness's `grammar_hits_in_pool: 185`
(knowledge 157, memory_write 12, system 6, conversation 2, uncertain 8) — a
strong cross-check that `DeriveFastPath` equals what the slice-22 harness
measured through `Route()`.
Root cause, two mechanisms:
1. **Grammar coverage grew after the corpus froze.** `calendar-query` (67),
`definition-query` (31), `narrative-query` (27), `task-status` (24),
`agenda-query` (9), `possession-statement` (8), `plan-day-query` (8),
`event-time-query` (4) and the rest are current stage-0 rules with no
counterpart in the mirror. The mirror ever only knew a fixed verb list.
2. **The mirror's verb lists over-stamped.** The 50 `claimed_fast_now_miss`
rows ("покажи uptime", "note: кран на кухне капает", "what's the date
сегодня") matched `toolAliasRe`/`captureVerbRe`/`timeQueryRe`, but no real
grammar accepts them: rules like `task-list-query`/`list-capture` use a
catch-all `(?s)^\s*(.+)$` pattern with the real gate in `Build`, which
declines. The mirror had no concept of a build gate.
The frozen 22 (16 + 6) are the same two mechanisms on holdout families,
including three `ru_routing_v1` rows ("что у меня стоит в календаре на
послезавтра", "расскажи про битву при Ватерлоо", "кто такой Линус Торвальдс?")
now resolved by calendar/narrative/definition grammars.
## 2. The authoritative derivation
`internal/router/semantic/fastpath.go` (non-test) owns:
- `ExperimentActVerbs()` — the 18 verbs formerly duplicated in
`helpers_test.go`, `legacy_build.go` and `heads_main.go`; now one non-test
list, so propagation can never drift from measurement.
- `DeriveFastPath(text) FastPathOutcome{Matched, Grammar}` — builds the exact
router the legacy baseline measures (`router.StageZeroGrammars` + the
experiment act matcher + `StubDateTimeParser`/`DefaultFactParser`) and runs
`TryFastPath(NormalizedInput{Text, MatchText: NormalizeMatchText(Text)})`.
Grammar attribution replays the ordered first-accept walk the router performs,
including wake-stripped alternates and build-declined fall-through.
Consumers:
- `cmd/corpus-factory` stamps `fast_path_resolved`/`router_residual` from
`DeriveFastPath`; the regex mirror is deleted.
- `cmd/merge-corpus` validates every v2 row against the derivation and **fails
on a stale value**; frozen holdout drift is reported only.
- `internal/router/semantic/fastpath_invariant_test.go` asserts every dev row
satisfies `fast_path_resolved == DeriveFastPath(text).Matched`; frozen rows
are exempt by the merge rule.
## 3. Rebuild proofs
```text
factory v2: 3008 rows, text-set identical old↔new
diff = exactly the fast-path fields on 296 rows; 0 other diffs
merge: v2 dev rows 2490 all validated against the router (0 stale)
frozen 535 preserved; 22 drift rows reported, none rewritten
v1 embedded: 3025 rows, text-set identical; dataset_hash unchanged
diff = fast-path fields on the 277 dev rows; 0 other diffs
embeddings: regenerated (fast_path_count 82 → 271, residual_count 2943 → 2754)
all 3025 vectors differ by ≤ ~0.02 max-abs from the slice-22 file
(onnxruntime build noise); on the same 1509 pool the e5-linear
score moves only −0.0014 acc / −0.0002 macro-F1 (C=10), i.e. the
population effect, not the embedding noise, drives every delta
```
## 4. Baseline deltas on the corrected population
| measure | slice 22 (n=1652) | slice 23 (n=1509) |
| --- | --- | --- |
| legacy hash floor | 0.1707 / 0.1202 (illegal 3) | **0.0663 / 0.0265 (illegal 2)** |
| deployed cascade (heads) | 0.7724 / 0.7150 (illegal 143) | 0.7515 / 0.7065 (illegal 142) |
| e5-linear C=10 | 0.7222 / 0.7117 | 0.7203 / 0.7310 |
| centroid (cosine) | 0.6731 / 0.6494 | 0.6753 / 0.6494 |
| sparse word+char (both) | 0.6247 / 0.4463 (vocab 6951) | 0.5991 / 0.4588 |
| majority | 0.4328 / 0.1208 | 0.3698 / 0.1080 |
| leave-one-family-out | 0.6824 / 0.1499, 53 fm | 0.6863 / 0.1519, 53 fm |
| fact holdout | 0.0180 (n 389) | 0.0077 (n 389) |
| world holdout | 0.7622 (n 143) | 0.6552 (n 87) |
| recall holdout | 0.7632 (n 114) | 0.5667 (n 90) |
| calendar holdout | 0.6304 (n 92) | 0.5185 (n 27) |
| K/MW water pairs | 0.863 (480), +0.298 | 0.850 (480), +0.275 |
| K/MW homelab pairs | 0.106 (180), −0.228 | 0.072 (180), −0.247 |
| K/MW task pairs | 0.735 (720), +0.122 | 0.403 (216), −0.095 |
| uncertain OOF P/R/F1 | 0.758 / 0.701 / 0.728 | 0.734 / 0.697 / 0.715 |
| action OOD swallowed | knowledge 370, mw 347 of 766 (conf>0.9: 0.060) | knowledge 364, mw 312 of 720 (conf>0.9: 0.056) |
| grammar hits in pool | 185 / 1467 pure | **0 / 1509 pure** |
The hash floor's collapse is explained structurally: the 185 removed rows were
nearly all the hash classifier's correct knowledge hits (correct predictions
282 → 98, a one-to-one loss with the drift), so the deterministic floor "resolves"
the same rows the real grammars already own.
## 5. The decisions requested
1. **Drift decomposition** — §1; 299 = 249 + 50; the slice-22 185 reproduces exactly.
2. **Root cause** — §1; coverage growth + build-gate over-stamp, neither visible to a regex mirror.
3. **Authoritative derivation** — §2; `DeriveFastPath` on the standard experiment router; duplicated act list removed.
4. **Corpus-builder change** — §2-§3; factory stamps from the router, merge validates (fail on stale) + reports frozen drift, invariant test enforces it.
5. **Corrected counts** — §0; fast 82→271, residual 2943→2754, pool 1652→1509 non-action + 720 OOD; dataset hash unchanged proves only the derived field moved.
6. **Baseline deltas** — §4; all ≤2pp on the head scores; hash floor collapses because its output was the drifted rows.
## 6. Conclusion survival and the gate
Every aggregate slice-22 conclusion is unchanged on the corrected population:
- **e5-linear ≈ deployed cascade.** e5 0.7203 vs heads 0.7515 acc (slice 22:
0.7222 vs 0.7724); macro-F1 e5 0.7310 now *exceeds* heads 0.7065. The ≈3pp
gap stands; a head on frozen e5 embeddings adds nothing a linear probe lacks.
- **Cross-family transfer is structurally absent.** LOFO macro-F1 0.1519;
fact holdout 0.0077; system/conversation/uncertain holdouts 0.0000.
- **K/MW is a deterministic semantic defect.** Homelab pairs still order
backwards (0.072, margin −0.247); task pairs now order backwards too
(0.403, margin −0.095, pairs 720→216 as the calendar/task queries leave the
pool). Water stays right (0.850). No learned head fixes this at the margin.
- **The deterministic floor is not a baseline.** acc 0.0663 ≈ predicting
uncertain for everything.
- **Action OOD is still swallowed.** 676 of 720 (93.9%) land in
knowledge/memory_write at argmax; conf>0.9 5.6%.
Decision gate from the brief: conclusions **materially the same**. The learned
non-action-router investigation is **closed**; no new representation or head is
justified. Next slice: the note-vs-recall disambiguator (the one genuine residual
deterministic defect the K/MW numbers keep surfacing).
## Artifacts
Re-run (corrected population):
```sh
go run ./cmd/semantic-router-experiment/slice23/ -out /tmp/mvn-s23/drift.json
go build -o /tmp/mvn-s23/s23 ./cmd/semantic-router-experiment/slice22/
MAVEN_ONNX_LIB=…/libonnxruntime.so.1.29.0 /tmp/mvn-s23/s23 -mode legacy -pool /tmp/mvn-s23/pool.json -out /tmp/mvn-s23/legacy.json
MAVEN_ONNX_LIB=… /tmp/mvn-s23/s23 -mode heads -pool /tmp/mvn-s23/pool.json -out /tmp/mvn-s23/legacy_heads.json
python3 slice23_emit.py # → /tmp/mvn-s23/{pool,ood,stats}.json (1509/720)
MAVEN_ONNX_LIB=… python3 slice23_main.py # → /tmp/mvn-s23/results.json
```
Corpus rebuild (proofs in §3): `corpus-factory -out /tmp/corpus_v2.json`, then
`merge-corpus -v1 corpus_v1.json -v2 /tmp/corpus_v2.json -out corpus_v1.json`
(identity of the change verified against `/tmp/mvn-s23/corpus_v1.old.json`).
Embedding regeneration (noise isolation against `/tmp/mvn-s23/embeddings.old.json`).
docs/evals/CLAUDE.md
+2 -1
View File
@@ -54,7 +54,8 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the
| [From-scratch tiny sequence pragmatics specialist (slice 19)](2026-09-08-tiny-sequence-pragmatics-specialist.md) | live |
| [Pretrained rubert-tiny pragmatics baseline (slice 20)](2026-09-07-pretrained-rubert-tiny-pragmatics-baseline.md) | live |
| [Deterministic execution-frame guard (slice 21)](2026-09-07-execution-frame-guard.md) | live |
| [Five-way residual non-action router: linear e5 vs the deployed cascade (slice 22)](2026-09-08-slice22-residual-nonaction-router.md) | live |
| [Five-way residual non-action router: linear e5 vs the deployed cascade (slice 22)](2026-09-08-slice22-residual-nonaction-router.md) | superseded by 2026-09-08-slice23 |
| [Fast-path metadata rebuilt from the real router: the slice-22 numbers, corrected (slice 23)](2026-09-08-slice23-fast-path-metadata-reconciliation.md) | live |
`docs/routing.md` holds the arm table these feed. Cite from there, not from here.