From 83e168f326c51fbb09bebf66adfbc3fb53b1d84f Mon Sep 17 00:00:00 2001 From: claude Date: Sat, 8 Aug 2026 22:32:48 +0400 Subject: [PATCH] Record what the routing heads score in Go (V-664) Two defects were found on the way: the tokenizer read every long word backwards, and the clarify head was discarded below the intent threshold. Both numbers are in the doc. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN --- docs/evals/2026-08-08-routing-heads-in-go.md | 146 +++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 docs/evals/2026-08-08-routing-heads-in-go.md diff --git a/docs/evals/2026-08-08-routing-heads-in-go.md b/docs/evals/2026-08-08-routing-heads-in-go.md new file mode 100644 index 0000000..aea3a6a --- /dev/null +++ b/docs/evals/2026-08-08-routing-heads-in-go.md @@ -0,0 +1,146 @@ +# The routing heads, running in Go + +Date: 2026-08-08. Vikunja V-664. +Weights: `router_heads.onnx`, fp32, exported from `heads.pt` on workpc. +Fixture: `internal/router/eval/ru_routing_v1.json`, 96 cases, 33 carrying a destination. +Runner: `make t PKG=./internal/router/eval/ RUN=TestONNXRoutingHeads`. + +The four heads of V-661 ran nowhere. This is the number they score through the +Go cascade. Same fixture and same grammars as `TestONNXBaseline`, and only the +middle stage varies. + +## Headline + +| | classifier + ONNX | heads + classifier | gemma-4-12b cascade | +|---|---|---|---| +| intent | 75.0% (72/96) | **96.9% (93/96)** | 84.4% | +| destination | 33.3% (11/33) | **75.8% (25/33)** | 72.7% | +| false clarify | 0 | 1 | 2 | +| missed clarify | 8 | 1 | 1 | +| p50 | 24.5ms | 27.9ms | 329ms | + +A 118M encoder beats the 12B teacher it was distilled from. It wins on both +halves of the route, at a twelfth of the latency. The workstation stays the +better phraser and is no longer the better router. + +The p50 is not the heads. Most of it is the classifier's own embedder pass on +the turns the heads decline, plus process warm-up on the first case. The heads' +own forward pass measures 7.3ms on workpc. + +## Two defects were in the way, and the first was not in the heads + +**The tokenizer read every long word backwards.** `encodeWord` backtracks the +Viterbi path from the end of a word and prepends each piece. That puts them back +in reading order, and a second reverse after the loop undid it. So +`query: вода` tokenized to `[0 12 1294 41 12489 2]` where the reference +tokenizer gives `[0 41 1294 12 12489 2]`. + +It was found here and only here. The heads were trained through transformers and +are read through the hand-written tokenizer. So a mismatch shows up as a score +far below what Python measured on the same weights. Nothing else in the suite +compares the two. + +Measured on the recall fixture, same 27 cases either way: + +| | reversed | fixed | +|---|---|---| +| recall@1 | 70.4% (19/27) | **77.8% (21/27)** | +| recall@3 | 85.2% (23/27) | **96.3% (26/27)** | +| answered after gate | 63.0% | 66.7% | +| wrong note on top | 8 | 6 | +| false recall | 0/5 | 1/5 | + +The classifier barely moved, 76.0% to 75.0%, and destination 36.4% to 33.3%. +Both are one case on 96 and neither is a finding. Seeds and queries were mangled +the same way, so cosine survived it. Recall is where it cost, because a stored +passage and a live query are different lengths and break differently. + +The one new false recall is the honest cost and it is not being hidden. A +sharper embedder scores every candidate higher, including the ones that should +have stayed under the gate. That is the same trade `2026-08-04-recall-e5-small.md` +recorded when e5-small replaced MiniLM. + +The embedder id now carries a tokenizer revision, `model_quantized@384/tok2`. +Stored vectors were written under rev 1 and no longer sit in the same space as a +query embedded now. The model file's name never moved, so nothing would have +triggered `ReembedAll`. On the box the marker fired on start, and the re-embed +rewrote 65 notes and 19 facts in 5 seconds. + +**The clarify head was being thrown away.** It was read only when the intent head +cleared its own threshold. That cost 6 of the 8 ambiguous cases. `вода` reads as intent +`act` at 0.233 and clarify at 0.983. Burying that handed the turn to the +classifier, which routed it confidently and never asked. The clarify head answers +a different question, which is whether there is enough here to act on at all. So +it decides on its own and decides first. + +| | intent-gated | clarify decides first | +|---|---|---| +| intent | 90.6% | 96.9% | +| missed clarify | 7 | 1 | +| false clarify | 0 | 1 | + +## The threshold is measured, not chosen + +Max softmax over the intent head, on the 88 cases carrying an intent: + +| threshold | kept | accuracy kept | wrong kept | right dropped | +|---|---|---|---|---| +| 0.5 | 84 | 96.4% | 3 | 2 | +| **0.6** | **81** | **97.5%** | **2** | **4** | +| 0.7 | 75 | 97.3% | 2 | 10 | +| 0.8 | 64 | 96.9% | 2 | 21 | +| 0.9 | 46 | 100.0% | 0 | 37 | + +0.6 is the knee. Every value from 0.7 to 0.85 drops right answers and keeps the +same two wrong ones. 0.9 is the only value that clears them, and it costs 37 +correct routes to do it. + +## Quantization was measured and rejected + +| build | size | intent | destination | p50 | +|---|---|---|---|---| +| fp32 | 470MB | 83/88 (94.3%) | 28/33 (84.8%) | 7.3ms | +| int8 | 118MB | 79/88 (89.8%) | 26/33 (78.8%) | 4.0ms | +| fp16 | 235MB | will not load | — | — | + +Python numbers, on the heads alone rather than through the cascade. int8 costs +4.5 points of intent and 6 of destination to save 3ms. The cascade around it has +a p50 over a second when the resident model answers. The fp16 graph is broken: +`convert_float_to_float16` leaves a Cast node emitting float16 where the graph +expects float, and onnxruntime refuses the session. It was not worth fixing. + +The exporter also had to be told to write one file. It splits weights into a +`.onnx.data` sidecar by default. This onnxruntime resolves that path against the +process working directory rather than the model. A split graph loads from one +directory only. + +## What is still wrong + +**Four of the eight destination misses are calendar.** Training cannot move them. +The possessive agenda rules claim those cases at stage 0 and name nothing on +purpose. That caution was free while nothing downstream could name anything +either. It has now cost four points in three separate measurements. The call is +the owner's and it is still open. + +**The slot head is exported and not read.** Slots come from the stage-2 +extractor. Mapping BIO tags back to text needs character offsets the unigram +tokenizer does not keep, which is its own piece of work. + +**`поужинал` is a false clarify**, which is the same defect `thinSingleToken` +was narrowed for on 2026-08-01, arriving now from a different direction. + +## On the box + +Deployed to homesrv the same day. `voice: routing heads loaded` on start, and +`/trace` shows `routing-heads` winning or thinning every turn. The resident model +and the classifier are both marked never asked. Live probes: + +```text +что такое TCP? -> kiwix a real definition +кто такой Линус Торвальдс? -> kiwix a real answer +во сколько я лёг вчера -> personal не нашла у тебя такой записи +вода -> thinned to clarify at 0.233 / 0.983 +``` + +A missing or broken weights file logs and leaves the heads nil, which is +byte-for-byte the cascade that shipped before this.