72 lines
2.7 KiB
Markdown
72 lines
2.7 KiB
Markdown
# The classifier baseline after the tokenizer fix
|
|
|
|
Measured 2026-08-11 on master at `2cf8b7e`, the classifier cascade with the ONNX
|
|
multilingual embedder, the configuration homesrv runs. `make t
|
|
PKG=./internal/router/eval/ RUN=TestONNXBaseline V=1`. Covers V-704.
|
|
|
|
## Result
|
|
|
|
Intent is **72/96 (75.0%)**. Destination is **11/33 (33.3%)**.
|
|
|
|
`docs/evals/2026-08-08-destination-fixture.md` recorded 73/96 (76.0%) and 12/33
|
|
(36.4%) three days earlier, and `CLAUDE.md` and `docs/routing.md` both carried
|
|
that pair until today. Same fixture, same 33 labelled cases, so the two pairs
|
|
are comparable and one case moved.
|
|
|
|
| | 2026-08-08 | today |
|
|
|---|---|---|
|
|
| intent | 73/96 (76.0%) | 72/96 (75.0%) |
|
|
| destination | 12/33 (36.4%) | 11/33 (33.3%) |
|
|
| world | 5/5 | 5/5 |
|
|
| the `SourceUnknown` floor | 5/7 | 4/7 |
|
|
| calendar | 2/6 | 2/6 |
|
|
| recall | 0/15 | 0/15 |
|
|
|
|
## One case moved, and it cost both numbers
|
|
|
|
`en-query-002`, "how long since the last backup finished". It routed `query` on
|
|
08-08 and routes `fact` today, at 0.870.
|
|
|
|
The case carries `want_source` present and empty, the floor label. `Score`
|
|
credits a destination only after the route lands the intent. So the case paid
|
|
once on the intent and once on the destination. Two numbers moved by one case,
|
|
which is why the drift read as two regressions.
|
|
|
|
## The commit
|
|
|
|
`feabf9f`, "The tokenizer read every long word backwards" (V-664), landed
|
|
2026-08-08 after the destination measurement.
|
|
|
|
Bisected by running `TestONNXBaseline` at each commit:
|
|
|
|
| commit | intent | destination |
|
|
|---|---|---|
|
|
| `b6eaa70` V-659, the 08-08 measurement | 73/96 | 12/33 |
|
|
| `e470435` V-661 | 73/96 | 12/33 |
|
|
| `a886217` V-663 | 73/96 | 12/33 |
|
|
| `50c6637` merge #205, parent of the fix | 73/96 | 12/33 |
|
|
| **`feabf9f` V-664, the tokenizer fix** | **72/96** | **11/33** |
|
|
| `c938148` V-666 | 72/96 | 11/33 |
|
|
| `2cf8b7e` master today | 72/96 | 11/33 |
|
|
|
|
Nothing between `a886217` and `feabf9f` touches routing except that commit and
|
|
one merge, and the merge holds the old pair.
|
|
|
|
## What this means for the pair
|
|
|
|
The higher pair is not a target to restore. Before `feabf9f`, `encodeWord`
|
|
emitted the characters of every long word in reverse. So the 76.0% and 36.4%
|
|
were scored on embeddings from a broken encoder. One case landed right under the
|
|
defect and lands wrong under the fix.
|
|
|
|
So 75.0% and 33.3% is the honest floor, and `en-query-002` is not a regression to
|
|
chase. It is a case the corrected encoder gets wrong, worth the same attention as
|
|
the other 23 intent misses and no more.
|
|
|
|
## What this does not measure
|
|
|
|
The routing heads row and the two gemma rows in `docs/routing.md` were not
|
|
re-run. The heads commits all land after `feabf9f`, so that row was already
|
|
measured against the corrected tokenizer. The gemma rows are the model arm and
|
|
were not checked either way.
|