Give LLM-routed decisions a real confidence so clarify can fire (#359)

Confidence was hardcoded to 1.0 for every LLM decision, and the LLM branch
in Router.Route returned straight from fillSlots without ever touching the
stage-3 threshold gate — so the LLM path could not produce a Clarify no
matter what confidence a model reported. That is why all 6 want_clarify
cases in the 77-case RU fixture were missed by every model in the bake-off.

Fix reads structural signal instead of changing the (parity-locked) router
prompt: a single-token utterance ("вода", "бэкап") is flagged thin evidence
in llmrouter.go; a fact left keyless or an act that never resolves to an
allowlisted fn, checked after fillSlots so the deterministic parsers get
first crack, is flagged in router.go's new gateLLMDecision. Anything below
config.DefaultRouterThreshold (0.55) now sets Clarify=true through the same
path the classifier already uses.

Added unit tests with a stubbed Completer proving both directions: thin
cases clarify, clean multi-word/resolved-slot cases stay confident. The
77-case fixture re-run against a live llama-server is still needed to
confirm the 6/6 moves — not done here, no llama-server on this box.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
This commit is contained in:
kami
2026-07-31 23:07:32 +04:00
parent 73d13f1ea6
commit f0f7ebc9b2
4 changed files with 174 additions and 3 deletions
+30
View File
@@ -89,6 +89,7 @@ func (r *Router) Route(ctx context.Context, utterance string, now time.Time) (De
if d, ok, err := r.llm.Route(ctx, utterance, now); err == nil && ok {
d.Utterance = utterance
r.fillSlots(ctx, &d, now)
r.gateLLMDecision(&d)
return d, nil
} else if err != nil {
log.Printf("router: llm route fell back to classifier: %v", err)
@@ -152,6 +153,35 @@ func (r *Router) fillSlots(ctx context.Context, d *Decision, now time.Time) {
// Stage stays 1: it says who decided the route, and that was the LLM.
}
// gateLLMDecision — stage 3 for the LLM path (Vikunja #359). This used to be
// the classifier's job alone (see the threshold check at the bottom of
// Route): the LLM branch returned straight from fillSlots and never touched
// r.threshold at all, so a hardcoded Confidence: 1.0 in llmrouter.go could
// never gate. Two more structural holes are checked here, after fillSlots
// has had a chance to fill them from the deterministic parsers — checking
// before fillSlots would flag e.g. every keyless fact the fact parser goes
// on to resolve (TestLLMFactGetsKeyFromParser):
// - a fact with no key even after the parser tried — nothing to write, or
// worse, a confident write under the wrong key;
// - an act that never resolved to an allowlisted fn — a confident guess
// here means either silently doing nothing or, if the daemon is lax,
// running something never on the allowlist. Don't guess; ask.
//
// Anything below threshold gets the exact same Clarify=true treatment the
// classifier path already produces — same field, same daemon-side consumer
// (cmd/mavend/clarify.go), nothing new to wire.
func (r *Router) gateLLMDecision(d *Decision) {
if d.Intent == IntentFact && !d.Slots.HasKey && d.Confidence > llmThinConfidence {
d.Confidence = llmThinConfidence
}
if d.Intent == IntentAct && !d.Slots.HasFn && d.Confidence > llmThinConfidence {
d.Confidence = llmThinConfidence
}
if d.Confidence < r.threshold {
d.Clarify = true
}
}
// CorrectMisroute — the user corrected a bad classification. Appends a new
// example for the corrected intent (append-only — grows the classifier, no
// retrain). Same shape as nudges.outcome tuning cooldowns: more reliable over