kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Correct the classifier baseline after tokenizer repair (V-704)

Browse Source
This commit is contained in:
claude
2026-08-13 01:27:13 +04:00
parent 2cf8b7e1b5
commit 8035a317d2
3 changed files with 88 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/routing.md
+12 -4
View File
@@ -1,6 +1,6 @@
# Routing
*Last verified: 2026-08-11 @ 25ed201*
*Last verified: 2026-08-11 @ 2cf8b7e*
How an utterance becomes a `Decision`, why each stage exists, and what every
stage has measured. `CLAUDE.md` carries the rules an agent must not break. This
@@ -60,7 +60,7 @@ classifier and the resident model, since those are what always answer.
| Arm | Intent | Destination | p50 | Measured |
|---|---|---|---|---|
| classifier + ONNX | 76.0% (73/96) | 36.4% (12/33) | 16.6µs | 2026-08-08 |
| classifier + ONNX | 75.0% (72/96) | 33.3% (11/33) | 20.6ms | 2026-08-11 |
| resident Qwen3-1.7B, cascade | 80.2% | not measured | 1.19s | 2026-08-05 |
| routing heads, cascade | 96.9% | 75.8% | 27.9ms | 2026-08-08 |
| gemma-4-12b, cascade | 84.4% | 72.7% | 329ms | 2026-08-02 |
@@ -72,7 +72,15 @@ another number on the same fixture. Sources:
`docs/evals/2026-08-02-workstation-gemma4-12b.md`,
`docs/evals/2026-08-09-e4b-vs-12b-routing.md`,
`docs/evals/2026-08-08-routing-heads-in-go.md`,
`docs/evals/2026-08-08-destination-fixture.md`.
`docs/evals/2026-08-11-classifier-baseline-after-tokenizer-fix.md`.
The classifier row read 76.0% and 36.4% until 2026-08-11, from
`docs/evals/2026-08-08-destination-fixture.md`. That pair predates `feabf9f`,
which fixed `encodeWord` emitting long words in reverse, so it was scored on
embeddings from a broken encoder. The fix cost one case, `en-query-002`, and that
case carries the floor destination label, so it moved both numbers at once. The
row also carried 16.6µs, which came from the classifier measured without the
embedder. 20.6ms is the ONNX arm's own p50.
The resident model alone scores 37.4% full against 61.5% intent-only. The gap is
slots and not routing. It routes `reminder` and leaves the time to the daemon,
@@ -201,7 +209,7 @@ are `docs/evals/2026-08-08-routing-heads-two-head.md`,
Read the best destination run as one seed and not a headline. One case is 3
points on a fixture this small. Head intent accuracy is **not** comparable to the
cascade's 76.0% and 84.4%. A softmax has no clarify class, so the head's fixture
cascade's 75.0% and 84.4%. A softmax has no clarify class, so the head's fixture
is 88 cases and not 96.
Recall is 15/15 and world is 5/5.