190 lines
12 KiB
Markdown
190 lines
12 KiB
Markdown
# Rebuilding corpus fast-path metadata from the real router moves the slice-22 1652-row five-way numbers by ≤2pp (heads 0.772→0.751 acc, e5-linear 0.722/0.712→0.720/0.731 macro-F1) and every aggregate slice-22 conclusion survives, so the learned non-action router investigation is closed as run; the residual pool is 1509 non-action + 720 action OOD rows, and the deterministic hash-classifier floor collapses to acc 0.066 because its earlier 0.171 was almost entirely the 185 rows the stale mirror had mislabelled as residual while the real grammars already resolve them
|
||
|
||
Date: 2026-09-08 · Task: slice 23 (fast-path metadata reconciliation; postscript to the accepted slice 22, task/725) · Box: workpc, Arch, RX 7900 GRE, 32 GB · Build: `internal/router/semantic/fastpath.go` + `cmd/semantic-router-experiment/slice23/` (Go 1.25.12) + `slice23_emit.py`/`slice23_main.py` in the frozen venv `/tmp/mvn-exp-venv` (sklearn 1.9.0); ONNX via `MAVEN_ONNX_LIB` pointing at the venv's `libonnxruntime.so.1.29.0`.
|
||
|
||
Replaces the corrected-population numbers in [slice 22](2026-09-08-slice22-residual-nonaction-router.md). The slice-22 file keeps its role for the run itself; its 1652-row pool is no longer the current measurement population.
|
||
|
||
## 0. What changed and what did not
|
||
|
||
The corpus's `fast_path_resolved` flag was stamped by a hand-written regex mirror
|
||
(`classifyFastPath` in `cmd/corpus-factory/main.go`) that drifted from the
|
||
stage-0 grammars. Slice 22 noticed 185 residual rows resolving at runtime. This
|
||
slice makes the flag a derivation from the real router and re-measures.
|
||
|
||
```text
|
||
corpus 3025 rows, dev pool 2490, frozen holdout 535 unchanged
|
||
dataset_hash b27fd48f478ca477cab1e59773bb353a unchanged (texts)
|
||
fast_path_resolved 82 → 271 (dev 72 → 261, frozen 10, preserved)
|
||
residual 2943 → 2754 (dev 2418 → 2229, frozen 525, preserved)
|
||
residual non-action pool 1652 → 1509
|
||
knowledge 715→558 / memory_write 553→541 / system 184→220 / conversation 93→91 / uncertain 107→99
|
||
action OOD probes 766 → 720
|
||
residual dev pool folds {0:317,1:140,2:410,3:408,4:377} → {0:247,1:111,2:373,3:386,4:392}
|
||
```
|
||
|
||
The rebuild changed **exactly the derived fields on exactly the 277 dev rows**
|
||
that disagree with the router: 0 rows differ in text, route, source, source_id,
|
||
split_group, or the non-`router_residual` tags. The 22 stale frozen rows (see §1)
|
||
are preserved verbatim per the merge rule — reported, never silently rewritten.
|
||
|
||
## 1. The drift, decomposed (299 disagreements across 3025 corpus rows)
|
||
|
||
`cmd/semantic-router-experiment/slice23` runs `DeriveFastPath` (the production
|
||
`TryFastPath` over `StageZeroGrammars` with the experiment act allowlist) on
|
||
every row and attributes the winner. Output: `/tmp/mvn-s23/drift.json`.
|
||
|
||
```text
|
||
mirror_missed (stored residual, runtime resolves) 249
|
||
dev 233 = 185 non-action + 48 action
|
||
dev non-action by grammar: calendar-query 66, definition-query 30,
|
||
narrative-query 26, task-capture 12, agenda-query 9, task-list-query 9,
|
||
plan-day-query 8, ambiguous-fragment 7, time-query 6, rest-of-day 5,
|
||
event-time 4, possession-statement 2, reminder-wakeword 1
|
||
frozen 16 (possession 6, implicit-elapsed 3, wakeword-act 2, + 5 singles)
|
||
claimed_fast_now_miss (stored fast, runtime declines) 50
|
||
dev 44 (system 42, action 2) + frozen 6 (memory_write "note: " rows)
|
||
every one has ≥1 stage-0 grammar matching its shape and declining the build;
|
||
none is a "no grammar matches" row
|
||
```
|
||
|
||
The **185 dev non-action `mirror_missed` rows are byte-identical in count and
|
||
route split** to the slice-22 harness's `grammar_hits_in_pool: 185`
|
||
(knowledge 157, memory_write 12, system 6, conversation 2, uncertain 8) — a
|
||
strong cross-check that `DeriveFastPath` equals what the slice-22 harness
|
||
measured through `Route()`.
|
||
|
||
Root cause, two mechanisms:
|
||
|
||
1. **Grammar coverage grew after the corpus froze.** `calendar-query` (67),
|
||
`definition-query` (31), `narrative-query` (27), `task-status` (24),
|
||
`agenda-query` (9), `possession-statement` (8), `plan-day-query` (8),
|
||
`event-time-query` (4) and the rest are current stage-0 rules with no
|
||
counterpart in the mirror. The mirror ever only knew a fixed verb list.
|
||
2. **The mirror's verb lists over-stamped.** The 50 `claimed_fast_now_miss`
|
||
rows ("покажи uptime", "note: кран на кухне капает", "what's the date
|
||
сегодня") matched `toolAliasRe`/`captureVerbRe`/`timeQueryRe`, but no real
|
||
grammar accepts them: rules like `task-list-query`/`list-capture` use a
|
||
catch-all `(?s)^\s*(.+)$` pattern with the real gate in `Build`, which
|
||
declines. The mirror had no concept of a build gate.
|
||
|
||
The frozen 22 (16 + 6) are the same two mechanisms on holdout families,
|
||
including three `ru_routing_v1` rows ("что у меня стоит в календаре на
|
||
послезавтра", "расскажи про битву при Ватерлоо", "кто такой Линус Торвальдс?")
|
||
now resolved by calendar/narrative/definition grammars.
|
||
|
||
## 2. The authoritative derivation
|
||
|
||
`internal/router/semantic/fastpath.go` (non-test) owns:
|
||
|
||
- `ExperimentActVerbs()` — the 18 verbs formerly duplicated in
|
||
`helpers_test.go`, `legacy_build.go` and `heads_main.go`; now one non-test
|
||
list, so propagation can never drift from measurement.
|
||
- `DeriveFastPath(text) FastPathOutcome{Matched, Grammar}` — builds the exact
|
||
router the legacy baseline measures (`router.StageZeroGrammars` + the
|
||
experiment act matcher + `StubDateTimeParser`/`DefaultFactParser`) and runs
|
||
`TryFastPath(NormalizedInput{Text, MatchText: NormalizeMatchText(Text)})`.
|
||
Grammar attribution replays the ordered first-accept walk the router performs,
|
||
including wake-stripped alternates and build-declined fall-through.
|
||
|
||
Consumers:
|
||
|
||
- `cmd/corpus-factory` stamps `fast_path_resolved`/`router_residual` from
|
||
`DeriveFastPath`; the regex mirror is deleted.
|
||
- `cmd/merge-corpus` validates every v2 row against the derivation and **fails
|
||
on a stale value**; frozen holdout drift is reported only.
|
||
- `internal/router/semantic/fastpath_invariant_test.go` asserts every dev row
|
||
satisfies `fast_path_resolved == DeriveFastPath(text).Matched`; frozen rows
|
||
are exempt by the merge rule.
|
||
|
||
## 3. Rebuild proofs
|
||
|
||
```text
|
||
factory v2: 3008 rows, text-set identical old↔new
|
||
diff = exactly the fast-path fields on 296 rows; 0 other diffs
|
||
merge: v2 dev rows 2490 all validated against the router (0 stale)
|
||
frozen 535 preserved; 22 drift rows reported, none rewritten
|
||
v1 embedded: 3025 rows, text-set identical; dataset_hash unchanged
|
||
diff = fast-path fields on the 277 dev rows; 0 other diffs
|
||
embeddings: regenerated (fast_path_count 82 → 271, residual_count 2943 → 2754)
|
||
all 3025 vectors differ by ≤ ~0.02 max-abs from the slice-22 file
|
||
(onnxruntime build noise); on the same 1509 pool the e5-linear
|
||
score moves only −0.0014 acc / −0.0002 macro-F1 (C=10), i.e. the
|
||
population effect, not the embedding noise, drives every delta
|
||
```
|
||
|
||
## 4. Baseline deltas on the corrected population
|
||
|
||
| measure | slice 22 (n=1652) | slice 23 (n=1509) |
|
||
| --- | --- | --- |
|
||
| legacy hash floor | 0.1707 / 0.1202 (illegal 3) | **0.0663 / 0.0265 (illegal 2)** |
|
||
| deployed cascade (heads) | 0.7724 / 0.7150 (illegal 143) | 0.7515 / 0.7065 (illegal 142) |
|
||
| e5-linear C=10 | 0.7222 / 0.7117 | 0.7203 / 0.7310 |
|
||
| centroid (cosine) | 0.6731 / 0.6494 | 0.6753 / 0.6494 |
|
||
| sparse word+char (both) | 0.6247 / 0.4463 (vocab 6951) | 0.5991 / 0.4588 |
|
||
| majority | 0.4328 / 0.1208 | 0.3698 / 0.1080 |
|
||
| leave-one-family-out | 0.6824 / 0.1499, 53 fm | 0.6863 / 0.1519, 53 fm |
|
||
| fact holdout | 0.0180 (n 389) | 0.0077 (n 389) |
|
||
| world holdout | 0.7622 (n 143) | 0.6552 (n 87) |
|
||
| recall holdout | 0.7632 (n 114) | 0.5667 (n 90) |
|
||
| calendar holdout | 0.6304 (n 92) | 0.5185 (n 27) |
|
||
| K/MW water pairs | 0.863 (480), +0.298 | 0.850 (480), +0.275 |
|
||
| K/MW homelab pairs | 0.106 (180), −0.228 | 0.072 (180), −0.247 |
|
||
| K/MW task pairs | 0.735 (720), +0.122 | 0.403 (216), −0.095 |
|
||
| uncertain OOF P/R/F1 | 0.758 / 0.701 / 0.728 | 0.734 / 0.697 / 0.715 |
|
||
| action OOD swallowed | knowledge 370, mw 347 of 766 (conf>0.9: 0.060) | knowledge 364, mw 312 of 720 (conf>0.9: 0.056) |
|
||
| grammar hits in pool | 185 / 1467 pure | **0 / 1509 pure** |
|
||
|
||
The hash floor's collapse is explained structurally: the 185 removed rows were
|
||
nearly all the hash classifier's correct knowledge hits (correct predictions
|
||
282 → 98, a one-to-one loss with the drift), so the deterministic floor "resolves"
|
||
the same rows the real grammars already own.
|
||
|
||
## 5. The decisions requested
|
||
|
||
1. **Drift decomposition** — §1; 299 = 249 + 50; the slice-22 185 reproduces exactly.
|
||
2. **Root cause** — §1; coverage growth + build-gate over-stamp, neither visible to a regex mirror.
|
||
3. **Authoritative derivation** — §2; `DeriveFastPath` on the standard experiment router; duplicated act list removed.
|
||
4. **Corpus-builder change** — §2-§3; factory stamps from the router, merge validates (fail on stale) + reports frozen drift, invariant test enforces it.
|
||
5. **Corrected counts** — §0; fast 82→271, residual 2943→2754, pool 1652→1509 non-action + 720 OOD; dataset hash unchanged proves only the derived field moved.
|
||
6. **Baseline deltas** — §4; all ≤2pp on the head scores; hash floor collapses because its output was the drifted rows.
|
||
|
||
## 6. Conclusion survival and the gate
|
||
|
||
Every aggregate slice-22 conclusion is unchanged on the corrected population:
|
||
|
||
- **e5-linear ≈ deployed cascade.** e5 0.7203 vs heads 0.7515 acc (slice 22:
|
||
0.7222 vs 0.7724); macro-F1 e5 0.7310 now *exceeds* heads 0.7065. The ≈3pp
|
||
gap stands; a head on frozen e5 embeddings adds nothing a linear probe lacks.
|
||
- **Cross-family transfer is structurally absent.** LOFO macro-F1 0.1519;
|
||
fact holdout 0.0077; system/conversation/uncertain holdouts 0.0000.
|
||
- **K/MW is a deterministic semantic defect.** Homelab pairs still order
|
||
backwards (0.072, margin −0.247); task pairs now order backwards too
|
||
(0.403, margin −0.095, pairs 720→216 as the calendar/task queries leave the
|
||
pool). Water stays right (0.850). No learned head fixes this at the margin.
|
||
- **The deterministic floor is not a baseline.** acc 0.0663 ≈ predicting
|
||
uncertain for everything.
|
||
- **Action OOD is still swallowed.** 676 of 720 (93.9%) land in
|
||
knowledge/memory_write at argmax; conf>0.9 5.6%.
|
||
|
||
Decision gate from the brief: conclusions **materially the same**. The learned
|
||
non-action-router investigation is **closed**; no new representation or head is
|
||
justified. Next slice: the note-vs-recall disambiguator (the one genuine residual
|
||
deterministic defect the K/MW numbers keep surfacing).
|
||
|
||
## Artifacts
|
||
|
||
Re-run (corrected population):
|
||
|
||
```sh
|
||
go run ./cmd/semantic-router-experiment/slice23/ -out /tmp/mvn-s23/drift.json
|
||
go build -o /tmp/mvn-s23/s23 ./cmd/semantic-router-experiment/slice22/
|
||
MAVEN_ONNX_LIB=…/libonnxruntime.so.1.29.0 /tmp/mvn-s23/s23 -mode legacy -pool /tmp/mvn-s23/pool.json -out /tmp/mvn-s23/legacy.json
|
||
MAVEN_ONNX_LIB=… /tmp/mvn-s23/s23 -mode heads -pool /tmp/mvn-s23/pool.json -out /tmp/mvn-s23/legacy_heads.json
|
||
python3 slice23_emit.py # → /tmp/mvn-s23/{pool,ood,stats}.json (1509/720)
|
||
MAVEN_ONNX_LIB=… python3 slice23_main.py # → /tmp/mvn-s23/results.json
|
||
```
|
||
|
||
Corpus rebuild (proofs in §3): `corpus-factory -out /tmp/corpus_v2.json`, then
|
||
`merge-corpus -v1 corpus_v1.json -v2 /tmp/corpus_v2.json -out corpus_v1.json`
|
||
(identity of the change verified against `/tmp/mvn-s23/corpus_v1.old.json`).
|
||
Embedding regeneration (noise isolation against `/tmp/mvn-s23/embeddings.old.json`). |