Score the routing fixture with the ONNX embedder (Vikunja #319)

The onnxruntime .so was already vendored at deps/onnxruntime-linux-x64-1.26.0
— nothing to download. make eval-router now defaults MAVEN_ONNX_LIB there, so
both baselines run by default and only a fresh clone without deps/ falls back
to the hash ratchet alone.

Prod-representative result, deployed 0.55 gate: 28/76 (36.8%), RU 25/61,
EN 3/15, hard 0/11 → 4/11, p50 31ms / p95 71ms. Versus the hash floor's
13/76 at p50 9µs.

The finding is not the accuracy, it's the refusal lane: missed clarifies went
0 → 5 of 6. Better embeddings raise cosine everywhere, so the 0.55 threshold
that used to hold ambiguous utterances back stops holding — "сделай это"
routes to act at 0.847, "бэкап" to chat at 0.755. The gate was implicitly
tuned to the hash floor's low similarities. That is an argument about the
threshold, not about the embedder, and it lands before #320 rather than after.

Also fixes a fixture-model mismatch: ReminderGrammar deliberately skips the
extractor at stage 0 and the daemon's applyAction parses the time downstream
(stage0.go says so). Charging the router for that slot made 4 exact-match wins
read as misses; they are now counted as SlotsDeferred instead. Hash baseline
moves 13/76, ratchet to 0.15.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
This commit is contained in:
kami
2026-07-31 00:34:29 +04:00
parent c7c44229a2
commit d34fdf40aa
3 changed files with 36 additions and 11 deletions
+13 -5
View File
@@ -142,13 +142,13 @@ func TestClassifierBaseline(t *testing.T) {
}
t.Log("\n" + rep.String() + rep.Failures())
// 0.10 is under the observed 0.118, not a target. Almost every case here
// 0.15 is under the observed 0.171, not a target. Almost every case here
// falls to clarify because the hash embedder's cosine never clears the
// 0.55 gate on paraphrases — which is the documented floor behaviour
// (AGENTS.md: "Russian recall rarely clears the confidence gate"), not a
// bug this fixture is asking anyone to fix. The number worth moving is
// TestONNXBaseline's.
const floor = 0.10
const floor = 0.15
if rep.Accuracy() < floor {
t.Errorf("accuracy %.3f below ratchet %.2f — routing regressed", rep.Accuracy(), floor)
}
@@ -162,10 +162,18 @@ func TestClassifierBaseline(t *testing.T) {
// TestONNXBaseline — the number that actually belongs in Vikunja #319: the
// deployed cascade with the multilingual ONNX embedder, the configuration
// homesrv runs. Skipped unless the runtime is present, because the .so is a
// gitignored ~200MB download and CI has the hash ratchet above instead.
// homesrv runs. Opt-in via MAVEN_ONNX_LIB because deps/ is gitignored, so the
// runtime is not guaranteed to exist on a fresh clone; CI has the hash ratchet
// above instead. `make eval-router` defaults the variable to the vendored
// deps/onnxruntime-linux-x64-1.26.0 copy.
//
// MAVEN_ONNX_LIB=/usr/local/lib/libonnxruntime.so go test -run ONNXBaseline ./internal/router/eval/
// Measured 2026-07-31 on deps/onnxruntime-linux-x64-1.26.0: 28/76 (36.8%),
// 21 false clarifies, 5 MISSED clarifies, p50 ~30-70ms. The missed clarifies
// are the finding — every one of the six ambiguous utterances scores higher
// cosine under ONNX than under the hash floor, so the 0.55 gate that held them
// back stops holding: "сделай это" routes to act at 0.847. A better embedder
// made the refusal lane worse, which is an argument about the gate, not the
// embedder.
//
// Reports rather than asserts: the score is an input to the #320 flip decision,
// and a threshold invented here would just be a second opinion about the same