diff --git a/CLAUDE.md b/CLAUDE.md index 2efe82c..1cf2a82 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -140,10 +140,11 @@ it is deliberate.** It guesses, so naming `SourceWorld` drops it. **Only a stage 0 grammar may drop it** (owner's call, V-666). `queryWalk` reads `Decision.SourceAnchored` for the source marked `boundary: true` and no other. -Judge a routing change against the classifier (76.0% intent, 36.4% destination) -and the resident model (80.2% intent), since those always answer. The fixture -has grown from 77 cases to 96, so a number compares only to another number on -the same fixture. +Judge a routing change against the classifier and the resident model, since +those always answer. **Their scores live in `docs/routing.md`, never here.** A +pair copied into this file goes stale silently. The fixture has changed size +more than once, so a number compares only to another number on the same +fixture. ## Language: model output and Russian diff --git a/docs/evals/2026-08-11-classifier-baseline-after-tokenizer-fix.md b/docs/evals/2026-08-11-classifier-baseline-after-tokenizer-fix.md new file mode 100644 index 0000000..892742e --- /dev/null +++ b/docs/evals/2026-08-11-classifier-baseline-after-tokenizer-fix.md @@ -0,0 +1,71 @@ +# The classifier baseline after the tokenizer fix + +Measured 2026-08-11 on master at `2cf8b7e`, the classifier cascade with the ONNX +multilingual embedder, the configuration homesrv runs. `make t +PKG=./internal/router/eval/ RUN=TestONNXBaseline V=1`. Covers V-704. + +## Result + +Intent is **72/96 (75.0%)**. Destination is **11/33 (33.3%)**. + +`docs/evals/2026-08-08-destination-fixture.md` recorded 73/96 (76.0%) and 12/33 +(36.4%) three days earlier, and `CLAUDE.md` and `docs/routing.md` both carried +that pair until today. Same fixture, same 33 labelled cases, so the two pairs +are comparable and one case moved. + +| | 2026-08-08 | today | +|---|---|---| +| intent | 73/96 (76.0%) | 72/96 (75.0%) | +| destination | 12/33 (36.4%) | 11/33 (33.3%) | +| world | 5/5 | 5/5 | +| the `SourceUnknown` floor | 5/7 | 4/7 | +| calendar | 2/6 | 2/6 | +| recall | 0/15 | 0/15 | + +## One case moved, and it cost both numbers + +`en-query-002`, "how long since the last backup finished". It routed `query` on +08-08 and routes `fact` today, at 0.870. + +The case carries `want_source` present and empty, the floor label. `Score` +credits a destination only after the route lands the intent. So the case paid +once on the intent and once on the destination. Two numbers moved by one case, +which is why the drift read as two regressions. + +## The commit + +`feabf9f`, "The tokenizer read every long word backwards" (V-664), landed +2026-08-08 after the destination measurement. + +Bisected by running `TestONNXBaseline` at each commit: + +| commit | intent | destination | +|---|---|---| +| `b6eaa70` V-659, the 08-08 measurement | 73/96 | 12/33 | +| `e470435` V-661 | 73/96 | 12/33 | +| `a886217` V-663 | 73/96 | 12/33 | +| `50c6637` merge #205, parent of the fix | 73/96 | 12/33 | +| **`feabf9f` V-664, the tokenizer fix** | **72/96** | **11/33** | +| `c938148` V-666 | 72/96 | 11/33 | +| `2cf8b7e` master today | 72/96 | 11/33 | + +Nothing between `a886217` and `feabf9f` touches routing except that commit and +one merge, and the merge holds the old pair. + +## What this means for the pair + +The higher pair is not a target to restore. Before `feabf9f`, `encodeWord` +emitted the characters of every long word in reverse. So the 76.0% and 36.4% +were scored on embeddings from a broken encoder. One case landed right under the +defect and lands wrong under the fix. + +So 75.0% and 33.3% is the honest floor, and `en-query-002` is not a regression to +chase. It is a case the corrected encoder gets wrong, worth the same attention as +the other 23 intent misses and no more. + +## What this does not measure + +The routing heads row and the two gemma rows in `docs/routing.md` were not +re-run. The heads commits all land after `feabf9f`, so that row was already +measured against the corrected tokenizer. The gemma rows are the model arm and +were not checked either way. diff --git a/docs/routing.md b/docs/routing.md index 4cdfdc4..2d1f29f 100644 --- a/docs/routing.md +++ b/docs/routing.md @@ -1,6 +1,6 @@ # Routing -*Last verified: 2026-08-11 @ 25ed201* +*Last verified: 2026-08-11 @ 2cf8b7e* How an utterance becomes a `Decision`, why each stage exists, and what every stage has measured. `CLAUDE.md` carries the rules an agent must not break. This @@ -60,7 +60,7 @@ classifier and the resident model, since those are what always answer. | Arm | Intent | Destination | p50 | Measured | |---|---|---|---|---| -| classifier + ONNX | 76.0% (73/96) | 36.4% (12/33) | 16.6µs | 2026-08-08 | +| classifier + ONNX | 75.0% (72/96) | 33.3% (11/33) | 20.6ms | 2026-08-11 | | resident Qwen3-1.7B, cascade | 80.2% | not measured | 1.19s | 2026-08-05 | | routing heads, cascade | 96.9% | 75.8% | 27.9ms | 2026-08-08 | | gemma-4-12b, cascade | 84.4% | 72.7% | 329ms | 2026-08-02 | @@ -72,7 +72,15 @@ another number on the same fixture. Sources: `docs/evals/2026-08-02-workstation-gemma4-12b.md`, `docs/evals/2026-08-09-e4b-vs-12b-routing.md`, `docs/evals/2026-08-08-routing-heads-in-go.md`, -`docs/evals/2026-08-08-destination-fixture.md`. +`docs/evals/2026-08-11-classifier-baseline-after-tokenizer-fix.md`. + +The classifier row read 76.0% and 36.4% until 2026-08-11, from +`docs/evals/2026-08-08-destination-fixture.md`. That pair predates `feabf9f`, +which fixed `encodeWord` emitting long words in reverse, so it was scored on +embeddings from a broken encoder. The fix cost one case, `en-query-002`, and that +case carries the floor destination label, so it moved both numbers at once. The +row also carried 16.6µs, which came from the classifier measured without the +embedder. 20.6ms is the ONNX arm's own p50. The resident model alone scores 37.4% full against 61.5% intent-only. The gap is slots and not routing. It routes `reminder` and leaves the time to the daemon, @@ -201,7 +209,7 @@ are `docs/evals/2026-08-08-routing-heads-two-head.md`, Read the best destination run as one seed and not a headline. One case is 3 points on a fixture this small. Head intent accuracy is **not** comparable to the -cascade's 76.0% and 84.4%. A softmax has no clarify class, so the head's fixture +cascade's 75.0% and 84.4%. A softmax has no clarify class, so the head's fixture is 88 cases and not 96. Recall is 15/15 and world is 5/5.