Record the measured confidence-gate numbers
This commit is contained in:
@@ -93,8 +93,16 @@ cascade at p50 ≈2.7s. Accuracy roughly doubled, latency is ~90× worse, and th
|
|||||||
accepted deliberately. `Confidence: 1.0` used to be hardcoded in `llmrouter.go`, so the LLM
|
accepted deliberately. `Confidence: 1.0` used to be hardcoded in `llmrouter.go`, so the LLM
|
||||||
path could never ask for clarification (6/6 refusal cases missed on the fixture) — Vikunja
|
path could never ask for clarification (6/6 refusal cases missed on the fixture) — Vikunja
|
||||||
#359. Fixed 31-07-2026 with structural signal (single-token utterance, keyless fact, act with
|
#359. Fixed 31-07-2026 with structural signal (single-token utterance, keyless fact, act with
|
||||||
no allowlisted fn) feeding the same stage-3 gate the classifier path already had; the fixture
|
no allowlisted fn) feeding the same stage-3 gate the classifier path already had — see
|
||||||
re-run to confirm the 6/6 moves is still outstanding, see `gateLLMDecision` in `router.go`.
|
`gateLLMDecision` in `router.go`. Note the second half of that bug: the LLM branch never
|
||||||
|
consulted `r.threshold` at all, so a correct low confidence would have been discarded anyway.
|
||||||
|
|
||||||
|
Re-measured on the fixture after the fix: **missed clarify 6/6 → 1**, at the cost of 3 false
|
||||||
|
clarifies and 2.6pt of full accuracy (72.7% → 70.1%, intent-only 67.5% → 74.0%). Two of the
|
||||||
|
three false clarifies are acts the model mis-routed and the gate caught — asking beats wrongly
|
||||||
|
executing, so the fixture and the daemon disagree about what is correct there. The third,
|
||||||
|
`"поужинал"`, is a real defect: **the single-token rule is an English intuition and does not
|
||||||
|
transfer to Russian**, where one word is routinely a whole sentence. Narrow or drop it.
|
||||||
|
|
||||||
## LLM output contract
|
## LLM output contract
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user