Compare commits

..

6 Commits

Author SHA1 Message Date
kami 5a1d465db5 Merge commit '4383844' into overnight-jul31
# Conflicts:
#	deploy/mavend.json
#	internal/config/config.go
2026-07-31 11:52:21 +04:00
kami 43838445ab Write up the margin gate results
Third section: why the absolute gate could not separate the two
distributions, the delta sweep, and the before/after. Marks next-steps
item 3 done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:51:21 +04:00
kami 11831c6ace Gate recall on the margin over the runner-up, not just the score
The e5 embedder puts every cosine in one narrow band (0.79-0.89), so the
absolute query_min_score gate cannot tell a real hit from a made-up
question: any value under the band answers everything, any value above it
answers nothing. False recall was 5/5.

New gate asks whether one note is clearly the best instead: top1 - top2 >
delta. New query_min_margin config knob, default 0.008, read off the sweep
in the recall harness. The absolute floor stays as a second check.

On the recall fixture with e5: answered 72% -> 68%, false recall 5/5 -> 1/5.

Vikunja #359

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:50:39 +04:00
kami 93c1a41d4a Route with the resident model by default
The two things that made this unsafe are fixed: the router can now
refuse, and slot extraction runs on its decisions.

On the held-out fixture it gets 63.2% of intents right against the
classifier's 50.0%, with no route errors. It costs about a second a
turn instead of 30ms.

The flag is a pointer now, so leaving it out of the config means on
and only writing false turns it off.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:47:57 +04:00
kami bce5ed210c Merge branch 'worktree-agent-a76ce40c73601d90d' into overnight-jul31 2026-07-31 11:44:32 +04:00
kami c31f0d1001 Extract slots for LLM router decisions too
An LLM-routed reminder came back with no parsed time and an act with no
fn, because only the classifier path ran the extractor. Now the router
runs the same extraction after an LLM decision and fills only the empty
slots. No time in the utterance still means no time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:44:06 +04:00
13 changed files with 494 additions and 72 deletions
+85 -2
View File
@@ -143,14 +143,97 @@ the vector memory table was written by the old model, so after this deploy they
against a new query. A live database needs every note and fact re-embedded before recall works at against a new query. A live database needs every note and fact re-embedded before recall works at
all. Filed as its own task. all. Filed as its own task.
## Margin gate — 31-07-2026, third run
Next-steps item 3, done. The absolute gate is replaced by a **margin gate**: answer only when the
top hit beats the runner-up by more than delta (`top1 top2 > δ`). Same fixture, same e5 embedder,
same store as the run above. `internal/memory/gate.go` holds the check; both read paths call it
(`cmd/mavend/recall.go` and the notes-RAG branch in `voice.go`). New knob `voice.query_min_margin`
in `deploy/mavend.json`, default 0.008.
### Why the absolute gate could not work, in one line of data
The harness now prints the margin distributions, and they barely overlap where the raw scores
overlap completely:
| | top-1 score | margin (top1 top2) |
|---|---|---|
| right note first (n=18) | min 0.810, median 0.862, max 0.890 | min 0.001, median 0.029, max 0.053 |
| must stay silent (n=5) | min 0.795, median 0.815, max 0.835 | min 0.000, median 0.002, **max 0.019** |
Four of the five must-be-silent cases have a margin at or under 0.002 — when there is nothing to
recall, e5 finds several notes equally close and no clear winner. That is the signal the absolute
score throws away.
### The delta sweep
Absolute gate held at 0.55 throughout.
```
delta 0.000: answered 18/25 (72%) false recall 5/5
delta 0.002: answered 17/25 (68%) false recall 3/5
delta 0.005: answered 17/25 (68%) false recall 2/5
delta 0.008: answered 17/25 (68%) false recall 1/5 <- chosen
delta 0.010: answered 15/25 (60%) false recall 1/5
delta 0.012: answered 14/25 (56%) false recall 1/5
delta 0.015: answered 12/25 (48%) false recall 1/5
delta 0.020: answered 11/25 (44%) false recall 0/5
delta 0.025: answered 9/25 (36%) false recall 0/5
delta 0.030: answered 8/25 (32%) false recall 0/5
delta 0.040: answered 4/25 (16%) false recall 0/5
delta 0.050: answered 2/25 ( 8%) false recall 0/5
delta 0.060: answered 0/25 ( 0%) false recall 0/5
```
### Chosen: δ = 0.008
It is the best point on the frontier, not a taste call. **0.008 dominates 0.010, 0.012 and 0.015
outright** — same 1/5 false recall, 8 to 20 points more real recall. Everything below it buys recall
back only by admitting more false recalls (0.005 → 2/5, 0.002 → 3/5). The next real improvement is
0.020 at 0/5 false, and it costs 24 points of recall to get there.
The brief's bar was "recall above 60% with false recall at 1/5 or better". 0.008 clears it with room:
68% and 1/5.
### Before / after
| | absolute gate 0.55 (previous) | margin gate δ=0.008 |
|---|---|---|
| recall@1 (ranking, ungated) | 72.0% (18/25) | 72.0% (18/25) — unchanged, the gate does not rank |
| **answered after the gate** | 72.0% (18/25) | **68.0% (17/25)** |
| **false recall** | **5/5 (100%)** | **1/5 (20%)** |
| fixture cases passed | 18/30 | **21/30** |
Four false recalls removed for one real answer. That is the trade the spec asks for — she is not a
guesser-of-truth. The one survivor is `en-pref-025` ("should i be offered wine"), which recalls a
filler note at 0.796 with a 0.019 margin: the widest silent-case margin in the fixture, and it sits
inside the real-recall range, so no delta removes it without taking real answers with it.
### Does the absolute cutoff still earn its keep? Marginally — kept
On this fixture with e5 it is a **no-op**: the lowest right-note score is 0.791, so 0.55 rejects
nothing the margin does not already reject. It is kept for two reasons, neither glamorous. It still
does real work for the hash embedder (its own sweep shows answers dropping from 16% to 0% between
0.30 and 0.50), and it is the only thing standing between the user and a reply built from a store
where everything is far away but one row happens to be a little less far — a near-empty database, or
the stale-vector case below. Cheap insurance, no measured cost. If a later embedder makes it bite,
the sweep is one command.
### Caveat on the numbers
Five must-be-silent cases is a thin basis for a 4-point decision. 1/5 and 2/5 differ by one case.
The shape of the frontier is trustworthy — margins separate, absolute scores do not — but δ=0.008
itself should be re-read off a bigger fixture (next-steps item 6) before anyone defends the third
decimal.
## Next steps — ordered by value-to-risk; nothing here is a decision ## Next steps — ordered by value-to-risk; nothing here is a decision
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config 1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
change plus a prefix in `onnxembedder.go`, re-measurable in one command. change plus a prefix in `onnxembedder.go`, re-measurable in one command.
2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any 2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
`query_min_score` picked against today's embedder describes a model on its way out. `query_min_score` picked against today's embedder describes a model on its way out.
3. **Replace the absolute-score gate with a margin gate** (`top1 top2 > δ`) — as the routing eval 3. ~~**Replace the absolute-score gate with a margin gate**~~ — done, see the section above.
concluded, absolute cosine cannot see a flat distribution. δ=0.008, false recall 5/5 → 1/5.
4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate, 4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
gate it separately, or restrict it to facts and say so. gate it separately, or restrict it to facts and say so.
5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one. 5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
+5 -8
View File
@@ -8,16 +8,13 @@ import "github.com/kami/maven/internal/memory"
// that can answer "when did I last …?" from a captured fact). A note hit here // that can answer "when did I last …?" from a captured fact). A note hit here
// is redundant with the notes-RAG path — by design; the two indexes can diverge // is redundant with the notes-RAG path — by design; the two indexes can diverge
// once the backend is swapped for a persistent/external store. ok=false when // once the backend is swapped for a persistent/external store. ok=false when
// there's no hit above the threshold or the hit carries no text. // the hit fails the confidence gate (see memory.Confident: an absolute floor
func bestRecall(results []memory.Result, min float64) (string, bool) { // plus a margin over the runner-up) or carries no text.
if len(results) == 0 { func bestRecall(results []memory.Result, minScore, minMargin float64) (string, bool) {
if !memory.Confident(results, minScore, minMargin) {
return "", false return "", false
} }
top := results[0] text := results[0].Meta["text"]
if top.Score < min {
return "", false
}
text := top.Meta["text"]
if text == "" { if text == "" {
return "", false return "", false
} }
+17 -4
View File
@@ -8,23 +8,24 @@ import (
func TestBestRecall(t *testing.T) { func TestBestRecall(t *testing.T) {
const min = 0.55 const min = 0.55
const margin = 0.008
t.Run("empty results", func(t *testing.T) { t.Run("empty results", func(t *testing.T) {
if _, ok := bestRecall(nil, min); ok { if _, ok := bestRecall(nil, min, margin); ok {
t.Error("empty results returned ok") t.Error("empty results returned ok")
} }
}) })
t.Run("top below threshold", func(t *testing.T) { t.Run("top below threshold", func(t *testing.T) {
res := []memory.Result{{Score: 0.4, Meta: map[string]string{"text": "выпил воды"}}} res := []memory.Result{{Score: 0.4, Meta: map[string]string{"text": "выпил воды"}}}
if _, ok := bestRecall(res, min); ok { if _, ok := bestRecall(res, min, margin); ok {
t.Error("below-threshold hit returned ok") t.Error("below-threshold hit returned ok")
} }
}) })
t.Run("hit without text meta", func(t *testing.T) { t.Run("hit without text meta", func(t *testing.T) {
res := []memory.Result{{Score: 0.9, Meta: map[string]string{"type": "fact"}}} res := []memory.Result{{Score: 0.9, Meta: map[string]string{"type": "fact"}}}
if _, ok := bestRecall(res, min); ok { if _, ok := bestRecall(res, min, margin); ok {
t.Error("textless hit returned ok") t.Error("textless hit returned ok")
} }
}) })
@@ -34,7 +35,7 @@ func TestBestRecall(t *testing.T) {
{Score: 0.82, Meta: map[string]string{"text": "выпил воды в три часа", "type": "fact"}}, {Score: 0.82, Meta: map[string]string{"text": "выпил воды в три часа", "type": "fact"}},
{Score: 0.60, Meta: map[string]string{"text": "другое"}}, {Score: 0.60, Meta: map[string]string{"text": "другое"}},
} }
got, ok := bestRecall(res, min) got, ok := bestRecall(res, min, margin)
if !ok { if !ok {
t.Fatal("clearing hit not returned") t.Fatal("clearing hit not returned")
} }
@@ -42,4 +43,16 @@ func TestBestRecall(t *testing.T) {
t.Errorf("wrong text: %q", got) t.Errorf("wrong text: %q", got)
} }
}) })
// The runner-up is almost as close, so the embedder cannot tell the two
// notes apart. Silence beats reading back a coin flip.
t.Run("runner-up too close", func(t *testing.T) {
res := []memory.Result{
{Score: 0.860, Meta: map[string]string{"text": "выпил воды в три часа"}},
{Score: 0.858, Meta: map[string]string{"text": "другое"}},
}
if _, ok := bestRecall(res, min, margin); ok {
t.Error("thin-margin hit returned ok")
}
})
} }
+21 -12
View File
@@ -206,11 +206,11 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
if threshold <= 0 { if threshold <= 0 {
threshold = config.DefaultRouterThreshold threshold = config.DefaultRouterThreshold
} }
// Both routing paths are weak on held-out utterances — the classifier gets // The resident model routes by default: 63.2% of held-out intents right
// 36.8% of intents right, the resident model 50.0% and much slower. Off by // against the classifier's 50.0%, at about 1s a turn instead of 30ms (see
// default (see config.VoiceConfig.LLMRouter); the classifier always stays // config.VoiceConfig.LLMRouter). The classifier always stays wired as the
// wired as the fallback, so a model error never breaks a turn. // fallback, so a model error never breaks a turn.
rtr := buildRouter(emb, matcher, threshold, pickLLMRouter(cfg.Voice.LLMRouter, llmClient)) rtr := buildRouter(emb, matcher, threshold, pickLLMRouter(cfg.Voice.UseLLMRouter(), llmClient))
// ----- sessions registry (shared with voicesink) ----- // ----- sessions registry (shared with voicesink) -----
sessions := voice.NewSessions() sessions := voice.NewSessions()
@@ -257,6 +257,7 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
clarifyStore: clarifyStore, clarifyStore: clarifyStore,
extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}}, extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}},
queryMinScore: cfg.Voice.QueryMinScore, queryMinScore: cfg.Voice.QueryMinScore,
queryMinMargin: cfg.Voice.QueryMinMargin,
timeParser: timeParser, timeParser: timeParser,
ecosystem: eco, ecosystem: eco,
} }
@@ -300,6 +301,9 @@ type reactiveHandler struct {
// load-bearing math (same posture as the presence thresholds). Set by // load-bearing math (same posture as the presence thresholds). Set by
// wireVoice from VoiceConfig; default 0.55. // wireVoice from VoiceConfig; default 0.55.
queryMinScore float64 queryMinScore float64
// queryMinMargin — the second half of that gate: how far the top hit must
// beat the runner-up. 0 ⇒ margin off.
queryMinMargin float64
// timeParser — used as a fallback for stage-0 reminder grammar matches // timeParser — used as a fallback for stage-0 reminder grammar matches
// (where the extractor didn't run). Shared with the router's extractor. // (where the extractor didn't run). Shared with the router's extractor.
@@ -760,18 +764,23 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
log.Printf("voice: query notes: %v", err) log.Printf("voice: query notes: %v", err)
return "не получилось найти ответ." return "не получилось найти ответ."
} }
// Confidence gate: below threshold, say "I don't know" rather than read // Confidence gate: below it, say "I don't know" rather than read back
// back the least-unrelated note — a confident wrong recall is worse than // the least-unrelated note — a confident wrong recall is worse than a
// a gap (spec's "not a guesser-of-truth"). Same instinct as the loop's // gap (spec's "not a guesser-of-truth"). Same instinct as the loop's
// since(key)==null → don't fire. Tuned for the ONNX embedder; the Hash // since(key)==null → don't fire. Two parts: an absolute cosine floor,
// floor scores lexically and may rarely clear it. // and a margin over the runner-up, which is the part that works with
if len(notes) == 0 || notes[0].Score < h.queryMinScore { // the e5 embedder's narrow score band. See memory.Confident.
noteScores := make([]float64, len(notes))
for i, n := range notes {
noteScores[i] = n.Score
}
if !memory.ConfidentScores(noteScores, h.queryMinScore, h.queryMinMargin) {
// Long-term memory recall (notes + facts) before general knowledge: // Long-term memory recall (notes + facts) before general knowledge:
// the notes table can't answer fact questions, but the memory store // the notes table can't answer fact questions, but the memory store
// indexes both. Only runs when notes-RAG already gave up → additive. // indexes both. Only runs when notes-RAG already gave up → additive.
if h.memStore != nil { if h.memStore != nil {
if hits, herr := h.memStore.Search(ctx, vec, 3); herr == nil { if hits, herr := h.memStore.Search(ctx, vec, 3); herr == nil {
if text, ok := bestRecall(hits, h.queryMinScore); ok { if text, ok := bestRecall(hits, h.queryMinScore, h.queryMinMargin); ok {
return text return text
} }
} }
+3 -1
View File
@@ -40,7 +40,9 @@
"tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json", "tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/opt/maven/lib/libonnxruntime.so" "lib_path": "/opt/maven/lib/libonnxruntime.so"
}, },
"llm_router": false, "llm_router": true,
"query_min_score": 0.55,
"query_min_margin": 0.008,
"tool_timeout": "30s", "tool_timeout": "30s",
"tools": [ "tools": [
{ "name": "status", "cmd": ["systemctl", "status"], "scope": "homelab", "destructive": false }, { "name": "status", "cmd": ["systemctl", "status"], "scope": "homelab", "destructive": false },
+54 -11
View File
@@ -258,18 +258,25 @@ type VoiceConfig struct {
RouterThreshold float64 `json:"router_threshold,omitempty"` RouterThreshold float64 `json:"router_threshold,omitempty"`
// LLMRouter — route with the resident model instead of the embedding // LLMRouter — route with the resident model instead of the embedding
// classifier. Measured on the held-out fixture (ROUTING-EVAL-31-07-2026.md) // classifier. On by default since Vikunja #320.
// the model gets 50.0% of intents right against the classifier's 36.8%, but
// it costs about 800ms per turn instead of 30ms.
// //
// TODO: the default stays false until this lands. // Measured on the held-out fixture (ROUTING-EVAL-31-07-2026.md): 63.2% of
// Extractor.Extract never runs on an LLM decision, so acts arrive with no // intents right against the classifier's 50.0%, and no route errors. It
// Fn and reminders with no Time. Turning this on today makes routing more // costs about 1s per turn instead of 30ms.
// accurate and less safe.
// //
// The router can now refuse: it answers "unknown" when it cannot route, and // It is safe to leave on. The model can refuse it answers "unknown" when
// the turn drops to the classifier and its clarify gate (Vikunja #359). // it cannot route, and the turn drops to the classifier and its clarify
LLMRouter bool `json:"llm_router,omitempty"` // gate. Any LLM error does the same, so a turn never breaks on the model.
// Slot extraction runs on LLM decisions too, so acts get their Fn and
// reminders their Time.
//
// Set it false to go back to the classifier, e.g. on a box with no
// llama-server or when 1s a turn is too slow.
//
// It is a pointer so that "missing from the file" and "explicitly false"
// are different things: missing means on, false means off. Read it with
// UseLLMRouter(), not directly.
LLMRouter *bool `json:"llm_router,omitempty"`
// QueryMinScore — the note-recall confidence gate. Top cosine below this // QueryMinScore — the note-recall confidence gate. Top cosine below this
// ⇒ "I don't know" instead of a guess. Tuned for the ONNX embedder (0.55); // ⇒ "I don't know" instead of a guess. Tuned for the ONNX embedder (0.55);
@@ -277,6 +284,14 @@ type VoiceConfig struct {
// default if unset. // default if unset.
QueryMinScore float64 `json:"query_min_score,omitempty"` QueryMinScore float64 `json:"query_min_score,omitempty"`
// QueryMinMargin — the second half of the recall gate: the top hit must
// beat the runner-up by more than this. The absolute score above cannot do
// the job on its own, because the e5 embedder puts every cosine in one
// narrow high band, so a made-up question scores as high as a real one.
// The margin asks whether one note is clearly the best instead.
// Negative ⇒ off. 0 ⇒ the default below.
QueryMinMargin float64 `json:"query_min_margin,omitempty"`
// Persona — optional prompt prefix that tunes maven's character. Prepended // Persona — optional prompt prefix that tunes maven's character. Prepended
// to every LLM system prompt (nudge phrasing, note queries, general // to every LLM system prompt (nudge phrasing, note queries, general
// knowledge). Empty string ⇒ current hardcoded persona (feminine-gendered // knowledge). Empty string ⇒ current hardcoded persona (feminine-gendered
@@ -404,7 +419,14 @@ const (
DefaultAutotuneInterval = 10 * time.Minute DefaultAutotuneInterval = 10 * time.Minute
DefaultRouterThreshold = 0.55 DefaultRouterThreshold = 0.55
DefaultQueryMinScore = 0.55 DefaultQueryMinScore = 0.55
DefaultToolTimeout = 30 * time.Second // Read off the margin sweep in internal/memory/recalleval on the e5
// embedder: 0.008 answers 68% of real questions (down from 72%) and cuts
// false recall from 5/5 to 1/5. Every larger delta costs real recall
// without removing that last one until 0.020, which drops recall to 44%.
DefaultQueryMinMargin = 0.008
DefaultToolTimeout = 30 * time.Second
// DefaultLLMRouter — route with the resident model unless told otherwise.
DefaultLLMRouter = true
DefaultFactEnrichmentInterval = 30 * time.Second DefaultFactEnrichmentInterval = 30 * time.Second
) )
@@ -486,9 +508,21 @@ func (c *Config) applyDefaults() {
if c.Voice.QueryMinScore <= 0 { if c.Voice.QueryMinScore <= 0 {
c.Voice.QueryMinScore = DefaultQueryMinScore c.Voice.QueryMinScore = DefaultQueryMinScore
} }
// Unset ⇒ default. Negative is how you turn the margin off on purpose,
// so it is clamped to 0 rather than replaced by the default.
switch {
case c.Voice.QueryMinMargin == 0:
c.Voice.QueryMinMargin = DefaultQueryMinMargin
case c.Voice.QueryMinMargin < 0:
c.Voice.QueryMinMargin = 0
}
if c.Voice.ToolTimeout <= 0 { if c.Voice.ToolTimeout <= 0 {
c.Voice.ToolTimeout = Duration(DefaultToolTimeout) c.Voice.ToolTimeout = Duration(DefaultToolTimeout)
} }
if c.Voice.LLMRouter == nil {
on := DefaultLLMRouter
c.Voice.LLMRouter = &on
}
} }
// routines: default severity to care-class (1) — the safe floor: a // routines: default severity to care-class (1) — the safe floor: a
@@ -507,6 +541,15 @@ func (c *Config) applyDefaults() {
} }
} }
// UseLLMRouter reports whether to route with the resident model. Unset means
// on; only an explicit false in the config turns it off.
func (v *VoiceConfig) UseLLMRouter() bool {
if v == nil || v.LLMRouter == nil {
return DefaultLLMRouter
}
return *v.LLMRouter
}
func (c *Config) validate() error { func (c *Config) validate() error {
if c.Phraser != nil { if c.Phraser != nil {
if c.Phraser.ModelPath == "" { if c.Phraser.ModelPath == "" {
+16 -4
View File
@@ -171,14 +171,26 @@ func TestWeatherConfigNilOK(t *testing.T) {
} }
} }
func TestLLMRouterDefaultsOff(t *testing.T) { func TestLLMRouterDefaultsOn(t *testing.T) {
p := writeConfig(t, `{"voice":{"enabled":true,"bind":"127.0.0.1:9100"}}`) p := writeConfig(t, `{"voice":{"enabled":true,"bind":"127.0.0.1:9100"}}`)
c, err := Load(p) c, err := Load(p)
if err != nil { if err != nil {
t.Fatalf("Load: %v", err) t.Fatalf("Load: %v", err)
} }
if c.Voice.LLMRouter { if !c.Voice.UseLLMRouter() {
t.Error("voice.llm_router absent should mean false") t.Error("voice.llm_router absent should mean on")
}
}
// Missing and explicitly false must not mean the same thing.
func TestLLMRouterExplicitFalseTurnsItOff(t *testing.T) {
p := writeConfig(t, `{"voice":{"enabled":true,"bind":"127.0.0.1:9100","llm_router":false}}`)
c, err := Load(p)
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.Voice.UseLLMRouter() {
t.Error("voice.llm_router false should turn it off")
} }
} }
@@ -188,7 +200,7 @@ func TestLLMRouterRead(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Load: %v", err) t.Fatalf("Load: %v", err)
} }
if !c.Voice.LLMRouter { if !c.Voice.UseLLMRouter() {
t.Error("voice.llm_router true was not read") t.Error("voice.llm_router true was not read")
} }
} }
+38
View File
@@ -0,0 +1,38 @@
package memory
// Confidence gate for a recall. Two checks, both must pass before Maven says a
// note back:
//
// - minScore — an absolute cosine floor.
// - minMargin — the top hit must beat the runner-up by more than this.
//
// The margin is the one that carries the weight. The e5 embedder packs every
// score into a narrow high band (0.79-0.89 on the recall fixture), so an
// absolute floor cannot tell a real hit from a confident-looking miss: every
// value under the band admits everything, every value above it answers nothing.
// A margin asks a different question — "is this note clearly the best one, or
// is the whole shelf equally close?" — and a made-up question has no clear best.
//
// With one hit and no runner-up there is nothing to compare, so only the floor
// applies.
// ConfidentScores reports whether the top score clears both gates. scores must
// be sorted highest first. minMargin <= 0 turns the margin check off.
func ConfidentScores(scores []float64, minScore, minMargin float64) bool {
if len(scores) == 0 || scores[0] < minScore {
return false
}
if minMargin > 0 && len(scores) > 1 && scores[0]-scores[1] <= minMargin {
return false
}
return true
}
// Confident is ConfidentScores for search results.
func Confident(results []Result, minScore, minMargin float64) bool {
scores := make([]float64, len(results))
for i, r := range results {
scores[i] = r.Score
}
return ConfidentScores(scores, minScore, minMargin)
}
+45
View File
@@ -0,0 +1,45 @@
package memory
import "testing"
func TestConfidentScores(t *testing.T) {
cases := []struct {
name string
scores []float64
minScore float64
minMargin float64
want bool
}{
{"no hits", nil, 0.55, 0.008, false},
{"below the floor", []float64{0.40, 0.10}, 0.55, 0.008, false},
{"clear winner", []float64{0.86, 0.70}, 0.55, 0.008, true},
{"runner-up too close", []float64{0.860, 0.858}, 0.55, 0.008, false},
// The rule is "beats the runner-up by MORE than delta". Not testing an
// exactly-equal margin: no pair of these decimals subtracts to exactly
// 0.008 in binary float, so such a test would pin rounding, not the rule.
{"margin just under delta", []float64{0.8079, 0.8}, 0.55, 0.008, false},
{"margin just over delta", []float64{0.8081, 0.8}, 0.55, 0.008, true},
// One hit: nothing to compare against, so only the floor applies.
{"single hit clears", []float64{0.86}, 0.55, 0.008, true},
{"single hit below floor", []float64{0.10}, 0.55, 0.008, false},
// Margin off — the old absolute-only behaviour.
{"margin off admits a tie", []float64{0.86, 0.86}, 0.55, 0, true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
if got := ConfidentScores(c.scores, c.minScore, c.minMargin); got != c.want {
t.Errorf("got %v, want %v", got, c.want)
}
})
}
}
func TestConfidentReadsResultScores(t *testing.T) {
res := []Result{{ID: "a", Score: 0.86}, {ID: "b", Score: 0.858}}
if Confident(res, 0.55, 0.008) {
t.Error("thin margin passed the gate")
}
if !Confident(res, 0.55, 0) {
t.Error("margin off should fall back to the floor alone")
}
}
+42 -19
View File
@@ -177,6 +177,8 @@ type Outcome struct {
Tied bool Tied bool
TopID string TopID string
TopScor float64 TopScor float64
// Margin — top1 top2. 0 when fewer than two hits came back.
Margin float64
Reasons []string Reasons []string
} }
@@ -184,8 +186,10 @@ type Outcome struct {
// ranks first but is silenced by query_min_score is a threshold problem, and a // ranks first but is silenced by query_min_score is a threshold problem, and a
// note that never ranks first is an embedder problem. Those are different fixes. // note that never ranks first is an embedder problem. Those are different fixes.
type Report struct { type Report struct {
Name string Name string
MinScore float64 MinScore float64
// MinMargin — how far the top hit must beat the runner-up. 0 ⇒ off.
MinMargin float64
Total int Total int
Answerable int Answerable int
Rank1 int Rank1 int
@@ -212,9 +216,14 @@ type Report struct {
// where the right note ranked first, and for the no-answer cases. The gap // where the right note ranked first, and for the no-answer cases. The gap
// between these two distributions is what a defensible query_min_score // between these two distributions is what a defensible query_min_score
// would have to sit inside; if they overlap, no threshold separates them. // would have to sit inside; if they overlap, no threshold separates them.
CorrectTop []float64 CorrectTop []float64
NoAnswerTop []float64 NoAnswerTop []float64
P50, P95, Max time.Duration // CorrectMargin / NoAnswerMargin — the same two groups, but top1 top2
// instead of top1. This is the pair the margin gate has to separate, and
// unlike the absolute scores it is what the sweep reads.
CorrectMargin []float64
NoAnswerMargin []float64
P50, P95, Max time.Duration
} }
// TagStat — passed/total for one slice of the fixture. // TagStat — passed/total for one slice of the fixture.
@@ -245,13 +254,14 @@ func ratio(n, d int) float64 {
// the run on an embed or search error: an erroring case scores as a miss and is // the run on an embed or search error: an erroring case scores as a miss and is
// counted in Errors, because "the embedder was down" and "the embedder was // counted in Errors, because "the embedder was down" and "the embedder was
// wrong" are different numbers. // wrong" are different numbers.
func Score(ctx context.Context, name string, emb router.Embedder, newStore NewStore, minScore float64, f Fixture) (Report, error) { func Score(ctx context.Context, name string, emb router.Embedder, newStore NewStore, minScore, minMargin float64, f Fixture) (Report, error) {
rep := Report{ rep := Report{
Name: name, Name: name,
MinScore: minScore, MinScore: minScore,
Total: len(f.Cases), MinMargin: minMargin,
ByTag: map[string]TagStat{}, Total: len(f.Cases),
ByLang: map[string]TagStat{}, ByTag: map[string]TagStat{},
ByLang: map[string]TagStat{},
} }
lat := make([]time.Duration, 0, len(f.Cases)) lat := make([]time.Duration, 0, len(f.Cases))
@@ -261,7 +271,7 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
} else { } else {
rep.NoAnswer++ rep.NoAnswer++
} }
o, err := scoreCase(ctx, emb, newStore, minScore, c, f.Filler) o, err := scoreCase(ctx, emb, newStore, minScore, minMargin, c, f.Filler)
if err != nil { if err != nil {
return Report{}, err return Report{}, err
} }
@@ -285,6 +295,11 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
} else if !o.Rank1 { } else if !o.Rank1 {
rep.WrongTop++ rep.WrongTop++
} }
if o.Rank1 {
// Margins are collected on rank, not on the gate, so the
// distribution does not move as the sweep changes the gate.
rep.CorrectMargin = append(rep.CorrectMargin, o.Margin)
}
if o.Rank1 && o.Recalled != "" { if o.Rank1 && o.Recalled != "" {
rep.CorrectTop = append(rep.CorrectTop, o.TopScor) rep.CorrectTop = append(rep.CorrectTop, o.TopScor)
} }
@@ -293,6 +308,7 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
rep.FalseRecall++ rep.FalseRecall++
} }
rep.NoAnswerTop = append(rep.NoAnswerTop, o.TopScor) rep.NoAnswerTop = append(rep.NoAnswerTop, o.TopScor)
rep.NoAnswerMargin = append(rep.NoAnswerMargin, o.Margin)
} }
if o.Pass { if o.Pass {
@@ -307,6 +323,8 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
sort.Float64s(rep.CorrectTop) sort.Float64s(rep.CorrectTop)
sort.Float64s(rep.NoAnswerTop) sort.Float64s(rep.NoAnswerTop)
sort.Float64s(rep.CorrectMargin)
sort.Float64s(rep.NoAnswerMargin)
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] }) sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95) rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
if len(lat) > 0 { if len(lat) > 0 {
@@ -318,7 +336,7 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
// scoreCase inserts the case's notes into a fresh store, then runs the read // scoreCase inserts the case's notes into a fresh store, then runs the read
// path the daemon runs. The returned error is fatal (the harness is broken); // path the daemon runs. The returned error is fatal (the harness is broken);
// an embedder or store failure on the query lands in Outcome.Err instead. // an embedder or store failure on the query lands in Outcome.Err instead.
func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minScore float64, c Case, filler []StoredNote) (Outcome, error) { func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minScore, minMargin float64, c Case, filler []StoredNote) (Outcome, error) {
st, release, err := newStore() st, release, err := newStore()
if err != nil { if err != nil {
return Outcome{}, fmt.Errorf("%s: new store: %w", c.ID, err) return Outcome{}, fmt.Errorf("%s: new store: %w", c.ID, err)
@@ -357,7 +375,10 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
if len(hits) > 0 { if len(hits) > 0 {
o.TopID, o.TopScor = hits[0].ID, hits[0].Score o.TopID, o.TopScor = hits[0].ID, hits[0].Score
o.Recalled = bestRecall(hits, minScore) if len(hits) > 1 {
o.Margin = hits[0].Score - hits[1].Score
}
o.Recalled = bestRecall(hits, minScore, minMargin)
} }
for i, h := range hits { for i, h := range hits {
if h.ID != c.Want { if h.ID != c.Want {
@@ -378,14 +399,14 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
switch { switch {
case !c.Answerable(): case !c.Answerable():
if o.Recalled != "" { if o.Recalled != "" {
o.Reasons = append(o.Reasons, fmt.Sprintf("false recall: %q at %.3f, want silence", o.TopID, o.TopScor)) o.Reasons = append(o.Reasons, fmt.Sprintf("false recall: %q at %.3f (margin %.3f), want silence", o.TopID, o.TopScor, o.Margin))
} }
case o.Tied: case o.Tied:
o.Reasons = append(o.Reasons, fmt.Sprintf("tie at %.3f — the right note is on top only by sort order", o.TopScor)) o.Reasons = append(o.Reasons, fmt.Sprintf("tie at %.3f — the right note is on top only by sort order", o.TopScor))
case !o.Rank1: case !o.Rank1:
o.Reasons = append(o.Reasons, fmt.Sprintf("top hit %q (%.3f), want %q%s", o.TopID, o.TopScor, c.Want, rankNote(o.Rank3))) o.Reasons = append(o.Reasons, fmt.Sprintf("top hit %q (%.3f), want %q%s", o.TopID, o.TopScor, c.Want, rankNote(o.Rank3)))
case o.Recalled == "": case o.Recalled == "":
o.Reasons = append(o.Reasons, fmt.Sprintf("right note ranked first but scored %.3f < gate %.2f — daemon says \"не знаю\"", o.TopScor, minScore)) o.Reasons = append(o.Reasons, fmt.Sprintf("right note ranked first at %.3f (margin %.3f) but the gate silenced it — daemon says \"не знаю\"", o.TopScor, o.Margin))
} }
o.Pass = len(o.Reasons) == 0 o.Pass = len(o.Reasons) == 0
return o, nil return o, nil
@@ -401,8 +422,8 @@ func rankNote(inTop3 bool) string {
// bestRecall mirrors cmd/mavend/recall.go — the gate the daemon actually // bestRecall mirrors cmd/mavend/recall.go — the gate the daemon actually
// applies to a memory hit. Duplicated rather than imported because package main // applies to a memory hit. Duplicated rather than imported because package main
// is not importable; recalleval_test.go asserts the two agree in behaviour. // is not importable; recalleval_test.go asserts the two agree in behaviour.
func bestRecall(results []memory.Result, min float64) string { func bestRecall(results []memory.Result, minScore, minMargin float64) string {
if len(results) == 0 || results[0].Score < min { if !memory.Confident(results, minScore, minMargin) {
return "" return ""
} }
return results[0].Meta["text"] return results[0].Meta["text"]
@@ -437,7 +458,7 @@ func percentile(sorted []time.Duration, p float64) time.Duration {
// the slices that name where the path is weak. // the slices that name where the path is weak.
func (r Report) String() string { func (r Report) String() string {
var b strings.Builder var b strings.Builder
fmt.Fprintf(&b, "%s: %d/%d cases pass (gate %.2f)\n", r.Name, r.Passed, r.Total, r.MinScore) fmt.Fprintf(&b, "%s: %d/%d cases pass (gate %.2f, margin %.3f)\n", r.Name, r.Passed, r.Total, r.MinScore, r.MinMargin)
fmt.Fprintf(&b, " recall@1 %.1f%% (%d/%d) recall@3 %.1f%% (%d/%d) answered after gate %.1f%% (%d/%d)\n", fmt.Fprintf(&b, " recall@1 %.1f%% (%d/%d) recall@3 %.1f%% (%d/%d) answered after gate %.1f%% (%d/%d)\n",
100*r.Recall1(), r.Rank1, r.Answerable, 100*r.Recall1(), r.Rank1, r.Answerable,
100*r.Recall3(), r.Rank3, r.Answerable, 100*r.Recall3(), r.Rank3, r.Answerable,
@@ -448,6 +469,8 @@ func (r Report) String() string {
100*r.FalseRecallRate(), r.FalseRecall, r.NoAnswer) 100*r.FalseRecallRate(), r.FalseRecall, r.NoAnswer)
fmt.Fprintf(&b, " top-1 score, right note first: %s\n", spread(r.CorrectTop)) fmt.Fprintf(&b, " top-1 score, right note first: %s\n", spread(r.CorrectTop))
fmt.Fprintf(&b, " top-1 score, must be silent: %s\n", spread(r.NoAnswerTop)) fmt.Fprintf(&b, " top-1 score, must be silent: %s\n", spread(r.NoAnswerTop))
fmt.Fprintf(&b, " margin top1-top2, right note first: %s\n", spread(r.CorrectMargin))
fmt.Fprintf(&b, " margin top1-top2, must be silent: %s\n", spread(r.NoAnswerMargin))
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max) fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
fmt.Fprintf(&b, " by lang: %s\n", renderStats(r.ByLang)) fmt.Fprintf(&b, " by lang: %s\n", renderStats(r.ByLang))
fmt.Fprintf(&b, " by tag: %s\n", renderStats(r.ByTag)) fmt.Fprintf(&b, " by tag: %s\n", renderStats(r.ByTag))
+50 -11
View File
@@ -142,21 +142,40 @@ func words(s string) []string {
// cmd/mavend/recall.go (package main is not importable). This pins the copy to // cmd/mavend/recall.go (package main is not importable). This pins the copy to
// the original's three rules: no hits, below the gate, or no text ⇒ silence. // the original's three rules: no hits, below the gate, or no text ⇒ silence.
func TestBestRecallMatchesDaemon(t *testing.T) { func TestBestRecallMatchesDaemon(t *testing.T) {
if got := bestRecall(nil, 0.55); got != "" { if got := bestRecall(nil, 0.55, 0); got != "" {
t.Errorf("no hits: got %q, want silence", got) t.Errorf("no hits: got %q, want silence", got)
} }
low := []memory.Result{{ID: "a", Score: 0.4, Meta: map[string]string{"text": "чай"}}} low := []memory.Result{{ID: "a", Score: 0.4, Meta: map[string]string{"text": "чай"}}}
if got := bestRecall(low, 0.55); got != "" { if got := bestRecall(low, 0.55, 0); got != "" {
t.Errorf("below gate: got %q, want silence", got) t.Errorf("below gate: got %q, want silence", got)
} }
noText := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{}}} noText := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{}}}
if got := bestRecall(noText, 0.55); got != "" { if got := bestRecall(noText, 0.55, 0); got != "" {
t.Errorf("no text: got %q, want silence", got) t.Errorf("no text: got %q, want silence", got)
} }
ok := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{"text": "чай"}}} ok := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{"text": "чай"}}}
if got := bestRecall(ok, 0.55); got != "чай" { if got := bestRecall(ok, 0.55, 0); got != "чай" {
t.Errorf("above gate: got %q, want %q", got, "чай") t.Errorf("above gate: got %q, want %q", got, "чай")
} }
// Margin: a close runner-up means the embedder cannot tell the two apart,
// so Maven stays silent even though both clear the absolute floor.
close := []memory.Result{
{ID: "a", Score: 0.86, Meta: map[string]string{"text": "чай"}},
{ID: "b", Score: 0.85, Meta: map[string]string{"text": "кофе"}},
}
if got := bestRecall(close, 0.55, 0.03); got != "" {
t.Errorf("thin margin: got %q, want silence", got)
}
if got := bestRecall(close, 0.55, 0); got != "чай" {
t.Errorf("margin off: got %q, want %q", got, "чай")
}
clear := []memory.Result{
{ID: "a", Score: 0.86, Meta: map[string]string{"text": "чай"}},
{ID: "b", Score: 0.70, Meta: map[string]string{"text": "кофе"}},
}
if got := bestRecall(clear, 0.55, 0.03); got != "чай" {
t.Errorf("wide margin: got %q, want %q", got, "чай")
}
} }
// TestHashRecallBaseline — the CI ratchet. HashEmbedder, so it needs no model // TestHashRecallBaseline — the CI ratchet. HashEmbedder, so it needs no model
@@ -171,7 +190,7 @@ func TestHashRecallBaseline(t *testing.T) {
t.Fatalf("Load: %v", err) t.Fatalf("Load: %v", err)
} }
rep, err := Score(context.Background(), "recall+hash", router.NewHashEmbedder(hashDim), InMemory, rep, err := Score(context.Background(), "recall+hash", router.NewHashEmbedder(hashDim), InMemory,
config.DefaultQueryMinScore, f) config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil { if err != nil {
t.Fatalf("Score: %v", err) t.Fatalf("Score: %v", err)
} }
@@ -201,11 +220,11 @@ func TestPersistentStoreScoresTheSame(t *testing.T) {
t.Fatalf("Load: %v", err) t.Fatalf("Load: %v", err)
} }
emb := router.NewHashEmbedder(hashDim) emb := router.NewHashEmbedder(hashDim)
inMem, err := Score(context.Background(), "recall+hash+memory", emb, InMemory, config.DefaultQueryMinScore, f) inMem, err := Score(context.Background(), "recall+hash+memory", emb, InMemory, config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil { if err != nil {
t.Fatalf("Score in-memory: %v", err) t.Fatalf("Score in-memory: %v", err)
} }
persistent, err := Score(context.Background(), "recall+hash+sqlite", emb, sqliteStores(t), config.DefaultQueryMinScore, f) persistent, err := Score(context.Background(), "recall+hash+sqlite", emb, sqliteStores(t), config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil { if err != nil {
t.Fatalf("Score sqlite: %v", err) t.Fatalf("Score sqlite: %v", err)
} }
@@ -263,14 +282,16 @@ func TestONNXRecall(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Load: %v", err) t.Fatalf("Load: %v", err)
} }
rep, err := Score(context.Background(), "recall+onnx", emb, InMemory, config.DefaultQueryMinScore, f) rep, err := Score(context.Background(), "recall+onnx", emb, InMemory, config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil { if err != nil {
t.Fatalf("Score: %v", err) t.Fatalf("Score: %v", err)
} }
t.Log("\n" + rep.String() + rep.Failures()) t.Log("\n" + rep.String() + rep.Failures())
// Cached for the sweep only: the headline run above must pay the real // Cached for the sweeps only: the headline run above must pay the real
// embedder cost so its latency numbers mean something. // embedder cost so its latency numbers mean something.
t.Log("\ngate sweep:\n" + sweep(t, Cache(emb), f)) cached := Cache(emb)
t.Log("\ngate sweep (margin off):\n" + sweep(t, cached, f))
t.Log("\nmargin sweep (gate 0.55):\n" + marginSweep(t, cached, f))
} }
// sweep scores the fixture at a range of gates and renders one line each. Two // sweep scores the fixture at a range of gates and renders one line each. Two
@@ -281,7 +302,7 @@ func sweep(t *testing.T, emb router.Embedder, f Fixture) string {
t.Helper() t.Helper()
var b strings.Builder var b strings.Builder
for _, gate := range []float64{0.0, 0.30, 0.40, 0.50, 0.55, 0.60, 0.70, 0.80, 0.90} { for _, gate := range []float64{0.0, 0.30, 0.40, 0.50, 0.55, 0.60, 0.70, 0.80, 0.90} {
rep, err := Score(context.Background(), "sweep", emb, InMemory, gate, f) rep, err := Score(context.Background(), "sweep", emb, InMemory, gate, 0, f)
if err != nil { if err != nil {
t.Fatalf("sweep at %.2f: %v", gate, err) t.Fatalf("sweep at %.2f: %v", gate, err)
} }
@@ -290,3 +311,21 @@ func sweep(t *testing.T, emb router.Embedder, f Fixture) string {
} }
return b.String() return b.String()
} }
// marginSweep is the same idea for the margin gate (top1 top2 > delta), with
// the absolute gate held at its default. The absolute score cannot separate a
// real hit from a made-up question under e5 — every score lands in one narrow
// band — so this sweep is the one that picks a number.
func marginSweep(t *testing.T, emb router.Embedder, f Fixture) string {
t.Helper()
var b strings.Builder
for _, d := range []float64{0, 0.002, 0.005, 0.008, 0.01, 0.012, 0.015, 0.02, 0.025, 0.03, 0.04, 0.05, 0.06} {
rep, err := Score(context.Background(), "margin sweep", emb, InMemory, config.DefaultQueryMinScore, d, f)
if err != nil {
t.Fatalf("margin sweep at %.3f: %v", d, err)
}
fmt.Fprintf(&b, " delta %.3f: answered %d/%d (%.0f%%) false recall %d/%d\n",
d, rep.Rank1-rep.Gated, rep.Answerable, 100*rep.Answered(), rep.FalseRecall, rep.NoAnswer)
}
return b.String()
}
+84
View File
@@ -175,3 +175,87 @@ func TestLLMRouterLLMError(t *testing.T) {
t.Fatal("want ok=false, err!=nil on llm error") t.Fatal("want ok=false, err!=nil on llm error")
} }
} }
// --- slot extraction on top of an LLM decision --------------------------------
// newLLMTestRouter — a router whose route always comes from the mock model.
func newLLMTestRouter(t *testing.T, out string) *Router {
t.Helper()
c := NewClassifier(NewHashEmbedder(1024))
seedClassifier(t, c)
acts := DefaultActMatcher{Fns: []string{"restart", "stop", "run", "backup"}}
return New(Config{
Classifier: c,
Extractor: Extractor{Time: StubDateTimeParser{}, Acts: acts, Facts: DefaultFactParser{}},
Threshold: 0.4,
LLM: NewLLMRouter(mockLLM{out: out}),
})
}
// The model cannot produce a fire time, so without extraction every LLM-routed
// reminder was dropped as "no time".
func TestLLMDecisionGetsReminderTime(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"reminder","text":"позвонить маме"}`)
d, err := r.Route(context.Background(), "напомни позвонить маме через 2 часа", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if d.Intent != IntentReminder {
t.Fatalf("want reminder, got %v", d.Intent)
}
if !d.Slots.HasTime || !d.Slots.Time.Equal(refNow().Add(2*time.Hour)) {
t.Fatalf("want time now+2h, got %+v", d.Slots)
}
if d.Slots.Text != "позвонить маме" {
t.Fatalf("extraction overwrote the model's text: %q", d.Slots.Text)
}
}
// No time in the utterance ⇒ no time in the slots. Do not invent one; the
// daemon says it could not read the time.
func TestLLMReminderWithoutTimeStaysEmpty(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"reminder","text":"позвонить маме"}`)
d, err := r.Route(context.Background(), "напомни позвонить маме", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if d.Slots.HasTime {
t.Fatalf("invented a time: %v", d.Slots.Time)
}
}
// An act decision arrived with no Fn, so the tool never ran.
func TestLLMDecisionGetsActFn(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"act","verb":"restart nginx"}`)
d, err := r.Route(context.Background(), "слушай, restart nginx пожалуйста", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if !d.Slots.HasFn || d.Slots.Fn != "restart" || len(d.Slots.Args) != 1 || d.Slots.Args[0] != "nginx" {
t.Fatalf("want fn=restart args=[nginx], got %+v", d.Slots)
}
}
// The model's own slots win; extraction only fills gaps.
func TestLLMSlotsWinOverExtraction(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"fact","key":"hydration","value":"выпил"}`)
d, err := r.Route(context.Background(), "я выпил воду", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if d.Slots.Key != "hydration" {
t.Fatalf("extraction overwrote the model's key: %q", d.Slots.Key)
}
}
// A fact the model left keyless still gets one from the parser.
func TestLLMFactGetsKeyFromParser(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"fact","text":"я выпил воду"}`)
d, err := r.Route(context.Background(), "я выпил воду", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if !d.Slots.HasKey || d.Slots.Key != "water" {
t.Fatalf("want key=water, got %+v", d.Slots)
}
}
+34
View File
@@ -88,6 +88,7 @@ func (r *Router) Route(ctx context.Context, utterance string, now time.Time) (De
if r.llm != nil { if r.llm != nil {
if d, ok, err := r.llm.Route(ctx, utterance, now); err == nil && ok { if d, ok, err := r.llm.Route(ctx, utterance, now); err == nil && ok {
d.Utterance = utterance d.Utterance = utterance
r.fillSlots(ctx, &d, now)
return d, nil return d, nil
} else if err != nil { } else if err != nil {
log.Printf("router: llm route fell back to classifier: %v", err) log.Printf("router: llm route fell back to classifier: %v", err)
@@ -118,6 +119,39 @@ func (r *Router) Route(ctx context.Context, utterance string, now time.Time) (De
return d, nil return d, nil
} }
// fillSlots — run stage-2 extraction on an LLM decision and fill only the slots
// the model left empty. The LLM wins where it answered: it saw the sentence, the
// parsers are keyword tables. Extraction covers what the model cannot produce at
// all — a parsed reminder time and an allowlist fn.
//
// If a reminder still has no time, leave it missing. The daemon then says it
// could not read the time; inventing one would set a wrong alarm.
func (r *Router) fillSlots(ctx context.Context, d *Decision, now time.Time) {
ex := r.extractor.Extract(ctx, d.Intent, d.Utterance, now)
if !d.Slots.HasTime && ex.HasTime {
d.Slots.Time, d.Slots.HasTime = ex.Time, ex.HasTime
}
if !d.Slots.HasKey && ex.HasKey {
d.Slots.Key, d.Slots.Value, d.Slots.HasKey = ex.Key, ex.Value, ex.HasKey
}
if !d.Slots.HasFn && ex.HasFn {
d.Slots.Fn, d.Slots.Args, d.Slots.HasFn = ex.Fn, ex.Args, ex.HasFn
}
// For an act the model returns the verb in Text ("restart nginx"), which is
// often cleaner than the raw utterance ("maven, could you restart nginx").
// Try it too when the utterance did not match the allowlist.
if d.Intent == IntentAct && !d.Slots.HasFn && r.extractor.Acts != nil &&
d.Slots.Text != "" && d.Slots.Text != d.Utterance {
if fn, args, ok := r.extractor.Acts.Match(d.Slots.Text); ok {
d.Slots.Fn, d.Slots.Args, d.Slots.HasFn = fn, args, true
}
}
if d.Slots.Text == "" {
d.Slots.Text = ex.Text
}
// Stage stays 1: it says who decided the route, and that was the LLM.
}
// CorrectMisroute — the user corrected a bad classification. Appends a new // CorrectMisroute — the user corrected a bad classification. Appends a new
// example for the corrected intent (append-only — grows the classifier, no // example for the corrected intent (append-only — grows the classifier, no
// retrain). Same shape as nudges.outcome tuning cooldowns: more reliable over // retrain). Same shape as nudges.outcome tuning cooldowns: more reliable over