Merge: a reminder said whole is not asked about (#214)
V-572. "напомни в 11:00 позвонить маме" answered "Когда?". ReminderGrammar builds its slots by hand and the extractor never ran over a stage 0 decision, so HasTime was false however clearly the hour was spoken, and missingFor read the silence as absence. fillMatchedSlots in internal/router/router.go now runs the stage 2 extractor over every stage 0 decision and fills only what the grammar left empty. A matched value always wins. The LLM path had the same hole and the same fix, so both share one function rather than ten grammars re-implementing extraction. Slots.Text is deliberately not filled. A grammar that left Text empty meant it: agendaQueryBuild hands the query chain the sentence itself. Filling it would also make SlotText unaskable, which is the bug V-383 fixed on the LLM side. Enabled for all ten grammars and inert for nine. Extract fills Time for a reminder, Fn for an act and Key for a fact, and nothing for query, system, note or chat. Benchmarked at 20000x with the real date parser: every stage 0 shape stays inside the noise, and the reminder rule gains, because actionReminder was already running that same parse one layer down. TestONNXBaseline 64/91 before and after, no case regressed. The fixture's own "slots deferred to daemon" line went 6 to 0. On the box: "хорошо, напомню сегодня в 11:00." Conflict in internal/router/router.go resolved by hand: V-564's grammar-outcome note and V-572's slot fill both belong, fill first. Full -race suite green. --no-verify: the pre-commit hook refuses master, and the owner asked for straight-to-master merges for this unattended run.
This commit is contained in:
@@ -162,6 +162,19 @@ on in deploy** — this section used to say it was wired `nil`, which stopped be
|
|||||||
Cascade order: `stage0.go` exact-match fast-path → LLM router (when non-nil) → classifier
|
Cascade order: `stage0.go` exact-match fast-path → LLM router (when non-nil) → classifier
|
||||||
fallback. Any LLM error falls through to the classifier so a turn never breaks on the model.
|
fallback. Any LLM error falls through to the classifier so a turn never breaks on the model.
|
||||||
|
|
||||||
|
**A stage-0 decision is slot-extracted too, since 06-08-2026** (V-572). `fillMatchedSlots`
|
||||||
|
in `router.go` runs the stage-2 extractor over whatever a grammar built and fills only the
|
||||||
|
slots it left empty — a matched value always wins, because the rule read a literal pattern
|
||||||
|
and the extractor guesses. It did not run before, so `ReminderGrammar` handed the daemon
|
||||||
|
`HasTime: false` for "напомни в 11:00 позвонить маме" and `missingFor` read the silence as
|
||||||
|
absence and asked "Когда?". It applies to every grammar and is inert for all but the
|
||||||
|
reminder: `Extract` fills Time, Fn and Key and nothing else, and the query, clock, agenda,
|
||||||
|
feed, list, task and narrative rules all emit intents with no such slot. Benchmarked at
|
||||||
|
20000x, a stage-0 query costs 3.7µs against 3.9µs before. **`Slots.Text` is deliberately not
|
||||||
|
filled** — a grammar that left it empty meant it, and `agendaQueryBuild` hands the query
|
||||||
|
chain the utterance itself. Fixture unchanged at 64/91, with "slots deferred to daemon"
|
||||||
|
6 → 0.
|
||||||
|
|
||||||
Measured on the 77-case RU fixture. **Re-measured 2026-08-02: the classifier scores 68.8%
|
Measured on the 77-case RU fixture. **Re-measured 2026-08-02: the classifier scores 68.8%
|
||||||
full accuracy at p50 16.6µs**, not the 36.8% at p50 31ms that stood here from
|
full accuracy at p50 16.6µs**, not the 36.8% at p50 31ms that stood here from
|
||||||
`docs/evals/2026-07-31-model-bakeoff.md`. That older figure predates the stage 0 rules and the
|
`docs/evals/2026-07-31-model-bakeoff.md`. That older figure predates the stage 0 rules and the
|
||||||
|
|||||||
@@ -90,6 +90,9 @@ func (r *Router) Route(ctx context.Context, utterance string, now time.Time) (De
|
|||||||
continue // grammar matched shape but not content → fall through
|
continue // grammar matched shape but not content → fall through
|
||||||
}
|
}
|
||||||
d.Utterance = utterance
|
d.Utterance = utterance
|
||||||
|
// The grammar decided the intent; the extractor fills the slots it did
|
||||||
|
// not match (V-572). See fillMatchedSlots for why every grammar gets it.
|
||||||
|
r.fillMatchedSlots(ctx, &d, now)
|
||||||
r.noteGrammarOutcomes(ctx, i+1, declinedBuild, g.Name, d.Intent)
|
r.noteGrammarOutcomes(ctx, i+1, declinedBuild, g.Name, d.Intent)
|
||||||
return d, nil
|
return d, nil
|
||||||
}
|
}
|
||||||
@@ -176,14 +179,37 @@ func (r *Router) Route(ctx context.Context, utterance string, now time.Time) (De
|
|||||||
return d, nil
|
return d, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// fillSlots — run stage-2 extraction on an LLM decision and fill only the slots
|
// fillMatchedSlots — run stage-2 extraction over a decision some earlier
|
||||||
// the model left empty. The LLM wins where it answered: it saw the sentence, the
|
// claimant produced, and fill only the slots that claimant left empty. A
|
||||||
// parsers are keyword tables. Extraction covers what the model cannot produce at
|
// matched value always wins: the claimant read the sentence, the extractor
|
||||||
// all — a parsed reminder time and an allowlist fn.
|
// guesses from keyword tables.
|
||||||
|
//
|
||||||
|
// Shared by the stage-0 grammars and the LLM router, which had the same hole
|
||||||
|
// for the same reason. A grammar asserts an intent at confidence 1.0 and says
|
||||||
|
// nothing about the slots, so "напомни в 11:00 позвонить маме" arrived with
|
||||||
|
// HasTime false however plainly the hour was spoken, and the daemon read the
|
||||||
|
// silence as absence and asked "Когда?" (V-572). The alternative was ten
|
||||||
|
// grammars each re-implementing extraction.
|
||||||
|
//
|
||||||
|
// It is applied to every stage-0 decision rather than to a chosen few, because
|
||||||
|
// for every intent but reminder it is inert: Extract fills Time for a reminder,
|
||||||
|
// Fn for an act and Key for a fact, and nothing at all for query, system, note
|
||||||
|
// or chat, which is what the query, clock, agenda, feed, list, task and
|
||||||
|
// narrative rules emit. The act rules — wakeword-act and the Praxis ones —
|
||||||
|
// already carry an Fn or they do not match, so there is nothing left for the
|
||||||
|
// matcher to fill. The reminder rule is the one that gains, and its time parse
|
||||||
|
// is a cost the daemon was already paying one layer down in actionReminder.
|
||||||
|
//
|
||||||
|
// Slots.Text is deliberately NOT filled here. Extract sets it to the raw
|
||||||
|
// utterance, and a grammar that left it empty meant it: agendaQueryBuild hands
|
||||||
|
// the query chain the utterance itself, and narrativeQueryBuild's Text is the
|
||||||
|
// topic, not the sentence.
|
||||||
//
|
//
|
||||||
// If a reminder still has no time, leave it missing. The daemon then says it
|
// If a reminder still has no time, leave it missing. The daemon then says it
|
||||||
// could not read the time; inventing one would set a wrong alarm.
|
// could not read the time; inventing one would set a wrong alarm.
|
||||||
func (r *Router) fillSlots(ctx context.Context, d *Decision, now time.Time) {
|
// Returns what the extractor read, so a caller that wants more of it does not
|
||||||
|
// pay for a second extraction — the reminder parser is the expensive one.
|
||||||
|
func (r *Router) fillMatchedSlots(ctx context.Context, d *Decision, now time.Time) Slots {
|
||||||
ex := r.extractor.Extract(ctx, d.Intent, d.Utterance, now)
|
ex := r.extractor.Extract(ctx, d.Intent, d.Utterance, now)
|
||||||
if !d.Slots.HasTime && ex.HasTime {
|
if !d.Slots.HasTime && ex.HasTime {
|
||||||
d.Slots.Time, d.Slots.HasTime = ex.Time, ex.HasTime
|
d.Slots.Time, d.Slots.HasTime = ex.Time, ex.HasTime
|
||||||
@@ -194,6 +220,14 @@ func (r *Router) fillSlots(ctx context.Context, d *Decision, now time.Time) {
|
|||||||
if !d.Slots.HasFn && ex.HasFn {
|
if !d.Slots.HasFn && ex.HasFn {
|
||||||
d.Slots.Fn, d.Slots.Args, d.Slots.HasFn = ex.Fn, ex.Args, ex.HasFn
|
d.Slots.Fn, d.Slots.Args, d.Slots.HasFn = ex.Fn, ex.Args, ex.HasFn
|
||||||
}
|
}
|
||||||
|
return ex
|
||||||
|
}
|
||||||
|
|
||||||
|
// fillSlots — fillMatchedSlots for an LLM decision, plus the two backfills that
|
||||||
|
// only make sense there. The LLM wins where it answered: it saw the sentence,
|
||||||
|
// the parsers are keyword tables.
|
||||||
|
func (r *Router) fillSlots(ctx context.Context, d *Decision, now time.Time) {
|
||||||
|
ex := r.fillMatchedSlots(ctx, d, now)
|
||||||
// For an act the model returns the verb in Text ("restart nginx"), which is
|
// For an act the model returns the verb in Text ("restart nginx"), which is
|
||||||
// often cleaner than the raw utterance ("maven, could you restart nginx").
|
// often cleaner than the raw utterance ("maven, could you restart nginx").
|
||||||
// Try it too when the utterance did not match the allowlist.
|
// Try it too when the utterance did not match the allowlist.
|
||||||
|
|||||||
@@ -121,6 +121,73 @@ func TestStage0GrammarFiresThroughCyrillicWakeWord(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestStage0ReminderCarriesTheHourHeSaid — "напомни в 11:00 позвонить маме" is
|
||||||
|
// the commonest reminder there is, and it used to reach the daemon with HasTime
|
||||||
|
// false, because ReminderGrammar builds its slots by hand and the router ran no
|
||||||
|
// extraction over a stage-0 decision. The daemon read the silence as absence and
|
||||||
|
// asked "Когда?" about an hour he had just said (V-572).
|
||||||
|
func TestStage0ReminderCarriesTheHourHeSaid(t *testing.T) {
|
||||||
|
r := newTestRouter(t, 0.0)
|
||||||
|
r.grammars = append(r.grammars, ReminderGrammar())
|
||||||
|
|
||||||
|
d, err := r.Route(context.Background(), "напомни в 11:00 позвонить маме", refNow())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("route: %v", err)
|
||||||
|
}
|
||||||
|
if d.Stage != 0 || d.Intent != IntentReminder {
|
||||||
|
t.Fatalf("want stage0 reminder, got %+v", d)
|
||||||
|
}
|
||||||
|
if !d.Slots.HasTime {
|
||||||
|
t.Fatalf("the hour was spoken, so the slot must be filled: %+v", d.Slots)
|
||||||
|
}
|
||||||
|
if got, want := d.Slots.Time.Format("15:04"), "11:00"; got != want {
|
||||||
|
t.Errorf("fire time = %s, want %s", got, want)
|
||||||
|
}
|
||||||
|
// The subject is the grammar's, not the extractor's: Slots.Text is what she
|
||||||
|
// says at the hour, and Extract would have overwritten it with the sentence.
|
||||||
|
if d.Slots.Text != "в 11:00 позвонить маме" {
|
||||||
|
t.Errorf("Text = %q, want the grammar's capture", d.Slots.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStage0MatchedSlotBeatsTheExtractor — a grammar that matched a literal
|
||||||
|
// pattern outranks a parser that guessed. The wake-word act names its fn from
|
||||||
|
// the remainder after the wake token; extraction over the raw utterance must not
|
||||||
|
// be able to replace it.
|
||||||
|
func TestStage0MatchedSlotBeatsTheExtractor(t *testing.T) {
|
||||||
|
r := newTestRouter(t, 0.0)
|
||||||
|
d, err := r.Route(context.Background(), "maven, restart nginx", refNow())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("route: %v", err)
|
||||||
|
}
|
||||||
|
if d.Slots.Fn != "restart" || len(d.Slots.Args) != 1 || d.Slots.Args[0] != "nginx" {
|
||||||
|
t.Fatalf("matched fn was overwritten: %+v", d.Slots)
|
||||||
|
}
|
||||||
|
if d.Slots.Text != "restart nginx" {
|
||||||
|
t.Errorf("Text = %q, want the grammar's remainder", d.Slots.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStage0QueryKeepsAnEmptyText — agendaQueryBuild deliberately leaves Text
|
||||||
|
// empty so the query chain reads the utterance itself. Extraction fills Time,
|
||||||
|
// Key and Fn and never Text, or every stage-0 query would start carrying the
|
||||||
|
// whole sentence in a slot that means something narrower.
|
||||||
|
func TestStage0QueryKeepsAnEmptyText(t *testing.T) {
|
||||||
|
r := newTestRouter(t, 0.0)
|
||||||
|
r.grammars = append(r.grammars, AgendaQueryGrammars()...)
|
||||||
|
|
||||||
|
d, err := r.Route(context.Background(), "что у меня сегодня", refNow())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("route: %v", err)
|
||||||
|
}
|
||||||
|
if d.Stage != 0 || d.Intent != IntentQuery {
|
||||||
|
t.Fatalf("want stage0 query, got %+v", d)
|
||||||
|
}
|
||||||
|
if d.Slots.Text != "" {
|
||||||
|
t.Errorf("Text = %q, want it left empty", d.Slots.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ----------------------------- stage 1 ---------------------------------------
|
// ----------------------------- stage 1 ---------------------------------------
|
||||||
|
|
||||||
func TestStage1ClassifiesAct(t *testing.T) {
|
func TestStage1ClassifiesAct(t *testing.T) {
|
||||||
|
|||||||
@@ -88,11 +88,12 @@ func DefaultGrammars(actMatcher ActMatcher) []Grammar {
|
|||||||
// non-reminder time queries toward it, and the verb+action overlap pushes
|
// non-reminder time queries toward it, and the verb+action overlap pushes
|
||||||
// actual reminders toward fact — a double contamination. Stage 0 fixes both.
|
// actual reminders toward fact — a double contamination. Stage 0 fixes both.
|
||||||
//
|
//
|
||||||
// The grammar captures the part after "напомни"/"remind me" into Slots.Text
|
// The grammar captures the part after "напомни"/"remind me" into Slots.Text —
|
||||||
// so the daemon's time parser can extract the fire time from it. The grammar
|
// what she says at the hour. The grammar itself does NOT parse time; that is
|
||||||
// itself does NOT parse time — that's the extractor's job (stage 2), but
|
// the extractor's job, and since V-572 the router runs the extractor over a
|
||||||
// stage 0 skips the extractor. The daemon's applyAction fallback calls the
|
// stage-0 decision too (fillMatchedSlots in router.go). Before that it did not,
|
||||||
// time parser for stage-0 reminders that arrive without HasTime.
|
// so "напомни в 11:00 позвонить маме" reached the daemon with HasTime false and
|
||||||
|
// was asked "Когда?" about an hour he had just said.
|
||||||
func ReminderGrammar() Grammar {
|
func ReminderGrammar() Grammar {
|
||||||
return Grammar{
|
return Grammar{
|
||||||
Name: "reminder-wakeword",
|
Name: "reminder-wakeword",
|
||||||
|
|||||||
Reference in New Issue
Block a user