Record what the routing heads score in Go (V-664)
Two defects were found on the way: the tokenizer read every long word backwards, and the clarify head was discarded below the intent threshold. Both numbers are in the doc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
This commit is contained in:
@@ -0,0 +1,146 @@
|
||||
# The routing heads, running in Go
|
||||
|
||||
Date: 2026-08-08. Vikunja V-664.
|
||||
Weights: `router_heads.onnx`, fp32, exported from `heads.pt` on workpc.
|
||||
Fixture: `internal/router/eval/ru_routing_v1.json`, 96 cases, 33 carrying a destination.
|
||||
Runner: `make t PKG=./internal/router/eval/ RUN=TestONNXRoutingHeads`.
|
||||
|
||||
The four heads of V-661 ran nowhere. This is the number they score through the
|
||||
Go cascade. Same fixture and same grammars as `TestONNXBaseline`, and only the
|
||||
middle stage varies.
|
||||
|
||||
## Headline
|
||||
|
||||
| | classifier + ONNX | heads + classifier | gemma-4-12b cascade |
|
||||
|---|---|---|---|
|
||||
| intent | 75.0% (72/96) | **96.9% (93/96)** | 84.4% |
|
||||
| destination | 33.3% (11/33) | **75.8% (25/33)** | 72.7% |
|
||||
| false clarify | 0 | 1 | 2 |
|
||||
| missed clarify | 8 | 1 | 1 |
|
||||
| p50 | 24.5ms | 27.9ms | 329ms |
|
||||
|
||||
A 118M encoder beats the 12B teacher it was distilled from. It wins on both
|
||||
halves of the route, at a twelfth of the latency. The workstation stays the
|
||||
better phraser and is no longer the better router.
|
||||
|
||||
The p50 is not the heads. Most of it is the classifier's own embedder pass on
|
||||
the turns the heads decline, plus process warm-up on the first case. The heads'
|
||||
own forward pass measures 7.3ms on workpc.
|
||||
|
||||
## Two defects were in the way, and the first was not in the heads
|
||||
|
||||
**The tokenizer read every long word backwards.** `encodeWord` backtracks the
|
||||
Viterbi path from the end of a word and prepends each piece. That puts them back
|
||||
in reading order, and a second reverse after the loop undid it. So
|
||||
`query: вода` tokenized to `[0 12 1294 41 12489 2]` where the reference
|
||||
tokenizer gives `[0 41 1294 12 12489 2]`.
|
||||
|
||||
It was found here and only here. The heads were trained through transformers and
|
||||
are read through the hand-written tokenizer. So a mismatch shows up as a score
|
||||
far below what Python measured on the same weights. Nothing else in the suite
|
||||
compares the two.
|
||||
|
||||
Measured on the recall fixture, same 27 cases either way:
|
||||
|
||||
| | reversed | fixed |
|
||||
|---|---|---|
|
||||
| recall@1 | 70.4% (19/27) | **77.8% (21/27)** |
|
||||
| recall@3 | 85.2% (23/27) | **96.3% (26/27)** |
|
||||
| answered after gate | 63.0% | 66.7% |
|
||||
| wrong note on top | 8 | 6 |
|
||||
| false recall | 0/5 | 1/5 |
|
||||
|
||||
The classifier barely moved, 76.0% to 75.0%, and destination 36.4% to 33.3%.
|
||||
Both are one case on 96 and neither is a finding. Seeds and queries were mangled
|
||||
the same way, so cosine survived it. Recall is where it cost, because a stored
|
||||
passage and a live query are different lengths and break differently.
|
||||
|
||||
The one new false recall is the honest cost and it is not being hidden. A
|
||||
sharper embedder scores every candidate higher, including the ones that should
|
||||
have stayed under the gate. That is the same trade `2026-08-04-recall-e5-small.md`
|
||||
recorded when e5-small replaced MiniLM.
|
||||
|
||||
The embedder id now carries a tokenizer revision, `model_quantized@384/tok2`.
|
||||
Stored vectors were written under rev 1 and no longer sit in the same space as a
|
||||
query embedded now. The model file's name never moved, so nothing would have
|
||||
triggered `ReembedAll`. On the box the marker fired on start, and the re-embed
|
||||
rewrote 65 notes and 19 facts in 5 seconds.
|
||||
|
||||
**The clarify head was being thrown away.** It was read only when the intent head
|
||||
cleared its own threshold. That cost 6 of the 8 ambiguous cases. `вода` reads as intent
|
||||
`act` at 0.233 and clarify at 0.983. Burying that handed the turn to the
|
||||
classifier, which routed it confidently and never asked. The clarify head answers
|
||||
a different question, which is whether there is enough here to act on at all. So
|
||||
it decides on its own and decides first.
|
||||
|
||||
| | intent-gated | clarify decides first |
|
||||
|---|---|---|
|
||||
| intent | 90.6% | 96.9% |
|
||||
| missed clarify | 7 | 1 |
|
||||
| false clarify | 0 | 1 |
|
||||
|
||||
## The threshold is measured, not chosen
|
||||
|
||||
Max softmax over the intent head, on the 88 cases carrying an intent:
|
||||
|
||||
| threshold | kept | accuracy kept | wrong kept | right dropped |
|
||||
|---|---|---|---|---|
|
||||
| 0.5 | 84 | 96.4% | 3 | 2 |
|
||||
| **0.6** | **81** | **97.5%** | **2** | **4** |
|
||||
| 0.7 | 75 | 97.3% | 2 | 10 |
|
||||
| 0.8 | 64 | 96.9% | 2 | 21 |
|
||||
| 0.9 | 46 | 100.0% | 0 | 37 |
|
||||
|
||||
0.6 is the knee. Every value from 0.7 to 0.85 drops right answers and keeps the
|
||||
same two wrong ones. 0.9 is the only value that clears them, and it costs 37
|
||||
correct routes to do it.
|
||||
|
||||
## Quantization was measured and rejected
|
||||
|
||||
| build | size | intent | destination | p50 |
|
||||
|---|---|---|---|---|
|
||||
| fp32 | 470MB | 83/88 (94.3%) | 28/33 (84.8%) | 7.3ms |
|
||||
| int8 | 118MB | 79/88 (89.8%) | 26/33 (78.8%) | 4.0ms |
|
||||
| fp16 | 235MB | will not load | — | — |
|
||||
|
||||
Python numbers, on the heads alone rather than through the cascade. int8 costs
|
||||
4.5 points of intent and 6 of destination to save 3ms. The cascade around it has
|
||||
a p50 over a second when the resident model answers. The fp16 graph is broken:
|
||||
`convert_float_to_float16` leaves a Cast node emitting float16 where the graph
|
||||
expects float, and onnxruntime refuses the session. It was not worth fixing.
|
||||
|
||||
The exporter also had to be told to write one file. It splits weights into a
|
||||
`.onnx.data` sidecar by default. This onnxruntime resolves that path against the
|
||||
process working directory rather than the model. A split graph loads from one
|
||||
directory only.
|
||||
|
||||
## What is still wrong
|
||||
|
||||
**Four of the eight destination misses are calendar.** Training cannot move them.
|
||||
The possessive agenda rules claim those cases at stage 0 and name nothing on
|
||||
purpose. That caution was free while nothing downstream could name anything
|
||||
either. It has now cost four points in three separate measurements. The call is
|
||||
the owner's and it is still open.
|
||||
|
||||
**The slot head is exported and not read.** Slots come from the stage-2
|
||||
extractor. Mapping BIO tags back to text needs character offsets the unigram
|
||||
tokenizer does not keep, which is its own piece of work.
|
||||
|
||||
**`поужинал` is a false clarify**, which is the same defect `thinSingleToken`
|
||||
was narrowed for on 2026-08-01, arriving now from a different direction.
|
||||
|
||||
## On the box
|
||||
|
||||
Deployed to homesrv the same day. `voice: routing heads loaded` on start, and
|
||||
`/trace` shows `routing-heads` winning or thinning every turn. The resident model
|
||||
and the classifier are both marked never asked. Live probes:
|
||||
|
||||
```text
|
||||
что такое TCP? -> kiwix a real definition
|
||||
кто такой Линус Торвальдс? -> kiwix a real answer
|
||||
во сколько я лёг вчера -> personal не нашла у тебя такой записи
|
||||
вода -> thinned to clarify at 0.233 / 0.983
|
||||
```
|
||||
|
||||
A missing or broken weights file logs and leaves the heads nil, which is
|
||||
byte-for-byte the cascade that shipped before this.
|
||||
Reference in New Issue
Block a user