Record the measured confidence-gate numbers

This commit is contained in:
kami
2026-07-31 23:23:37 +04:00
parent 9b8bdf73cc
commit a2031a31d1
+10 -2
View File
@@ -93,8 +93,16 @@ cascade at p50 ≈2.7s. Accuracy roughly doubled, latency is ~90× worse, and th
accepted deliberately. `Confidence: 1.0` used to be hardcoded in `llmrouter.go`, so the LLM
path could never ask for clarification (6/6 refusal cases missed on the fixture) — Vikunja
#359. Fixed 31-07-2026 with structural signal (single-token utterance, keyless fact, act with
no allowlisted fn) feeding the same stage-3 gate the classifier path already had; the fixture
re-run to confirm the 6/6 moves is still outstanding, see `gateLLMDecision` in `router.go`.
no allowlisted fn) feeding the same stage-3 gate the classifier path already had — see
`gateLLMDecision` in `router.go`. Note the second half of that bug: the LLM branch never
consulted `r.threshold` at all, so a correct low confidence would have been discarded anyway.
Re-measured on the fixture after the fix: **missed clarify 6/6 → 1**, at the cost of 3 false
clarifies and 2.6pt of full accuracy (72.7% → 70.1%, intent-only 67.5% → 74.0%). Two of the
three false clarifies are acts the model mis-routed and the gate caught — asking beats wrongly
executing, so the fixture and the daemon disagree about what is correct there. The third,
`"поужинал"`, is a real defect: **the single-token rule is an English intuition and does not
transfer to Russian**, where one word is routinely a whole sentence. Narrow or drop it.
## LLM output contract