Give LLM-routed decisions a real confidence so clarify can fire (#359)
Confidence was hardcoded to 1.0 for every LLM decision, and the LLM branch
in Router.Route returned straight from fillSlots without ever touching the
stage-3 threshold gate — so the LLM path could not produce a Clarify no
matter what confidence a model reported. That is why all 6 want_clarify
cases in the 77-case RU fixture were missed by every model in the bake-off.
Fix reads structural signal instead of changing the (parity-locked) router
prompt: a single-token utterance ("вода", "бэкап") is flagged thin evidence
in llmrouter.go; a fact left keyless or an act that never resolves to an
allowlisted fn, checked after fillSlots so the deterministic parsers get
first crack, is flagged in router.go's new gateLLMDecision. Anything below
config.DefaultRouterThreshold (0.55) now sets Clarify=true through the same
path the classifier already uses.
Added unit tests with a stubbed Completer proving both directions: thin
cases clarify, clean multi-word/resolved-slot cases stay confident. The
77-case fixture re-run against a live llama-server is still needed to
confirm the 6/6 moves — not done here, no llama-server on this box.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
This commit is contained in:
@@ -118,6 +118,39 @@ const routeRepeatPenalty = 1.15
|
||||
// Route return ok=false so the caller drops to the classifier cascade.
|
||||
const routeIntentUnknown = "unknown"
|
||||
|
||||
// llmFullConfidence / llmThinConfidence — Vikunja #359. Confidence used to be
|
||||
// hardcoded to 1.0 for every LLM decision, so the stage-3 gate in router.go
|
||||
// never had anything to bite on and the LLM path could never produce a
|
||||
// Clarify: on the 77-case RU fixture, 6/6 want_clarify cases were missed by
|
||||
// EVERY model in the 31-07-2026 bake-off (0.8B through 2B) — proof this was a
|
||||
// code bug, not a capability ceiling.
|
||||
//
|
||||
// The fix does not touch the prompt (routeSystem is under
|
||||
// llm/check_prompt_parity.py in the training workspace; changing its text
|
||||
// creates a parity break that has to be fixed there too — see Vikunja #362).
|
||||
// Instead it reads structural signal that is already free:
|
||||
// - a single-token utterance is thin evidence for anything a grammar
|
||||
// didn't already catch at stage 0 — "вода" and "бэкап" alone don't say
|
||||
// fact-vs-query or act-vs-report;
|
||||
// - a fact with no key, or an act that never resolves to an allowlisted fn
|
||||
// (checked in router.go, after slot-fill has had its say), is a decision
|
||||
// with a hole in the one slot that makes it actionable.
|
||||
//
|
||||
// A model self-reporting confidence in the JSON was considered and rejected:
|
||||
// a sub-2B is not calibrated (nothing stops it saying "confident" on exactly
|
||||
// the cases it gets wrong today), and true logprobs would need a response
|
||||
// field internal/llm.Client's Complete does not currently return — see
|
||||
// internal/llm/client.go.
|
||||
//
|
||||
// llmThinConfidence sits below config.DefaultRouterThreshold (0.55) so the
|
||||
// existing stage-3 gate in Router.Route treats it exactly like a low-scoring
|
||||
// classifier result — same lane, same daemon-side clarify machinery
|
||||
// (cmd/mavend/clarify.go), no new consumer to build.
|
||||
const (
|
||||
llmFullConfidence = 1.0
|
||||
llmThinConfidence = 0.3
|
||||
)
|
||||
|
||||
type routeAction struct {
|
||||
Intent string `json:"intent"`
|
||||
Key string `json:"key"`
|
||||
@@ -160,7 +193,15 @@ func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time)
|
||||
if a.Intent == routeIntentUnknown {
|
||||
return Decision{}, false, nil
|
||||
}
|
||||
d := Decision{Utterance: utterance, Stage: 1, Confidence: 1.0}
|
||||
d := Decision{Utterance: utterance, Stage: 1, Confidence: llmFullConfidence}
|
||||
// A single-token utterance is thin evidence: the model had nothing to
|
||||
// disambiguate on ("вода" is a fact-or-query coin flip, "бэкап" an
|
||||
// act-or-report one) and stage 0 would already have won on anything
|
||||
// that pattern-matches cleanly. Flag it now; router.go's stage-3 gate
|
||||
// (Router.Route) decides whether that trips Clarify.
|
||||
if len(strings.Fields(utterance)) <= 1 {
|
||||
d.Confidence = llmThinConfidence
|
||||
}
|
||||
switch Intent(a.Intent) {
|
||||
case IntentFact:
|
||||
d.Intent = IntentFact
|
||||
|
||||
Reference in New Issue
Block a user