From a2031a31d1c08c6bd433c8cac6cdda41acc8416b Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 31 Jul 2026 23:23:37 +0400 Subject: [PATCH] Record the measured confidence-gate numbers --- CLAUDE.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b8ec08b..acd1408 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -93,8 +93,16 @@ cascade at p50 ≈2.7s. Accuracy roughly doubled, latency is ~90× worse, and th accepted deliberately. `Confidence: 1.0` used to be hardcoded in `llmrouter.go`, so the LLM path could never ask for clarification (6/6 refusal cases missed on the fixture) — Vikunja #359. Fixed 31-07-2026 with structural signal (single-token utterance, keyless fact, act with -no allowlisted fn) feeding the same stage-3 gate the classifier path already had; the fixture -re-run to confirm the 6/6 moves is still outstanding, see `gateLLMDecision` in `router.go`. +no allowlisted fn) feeding the same stage-3 gate the classifier path already had — see +`gateLLMDecision` in `router.go`. Note the second half of that bug: the LLM branch never +consulted `r.threshold` at all, so a correct low confidence would have been discarded anyway. + +Re-measured on the fixture after the fix: **missed clarify 6/6 → 1**, at the cost of 3 false +clarifies and 2.6pt of full accuracy (72.7% → 70.1%, intent-only 67.5% → 74.0%). Two of the +three false clarifies are acts the model mis-routed and the gate caught — asking beats wrongly +executing, so the fixture and the daemon disagree about what is correct there. The third, +`"поужинал"`, is a real defect: **the single-token rule is an English intuition and does not +transfer to Russian**, where one word is routinely a whole sentence. Narrow or drop it. ## LLM output contract