diff --git a/docs/evals/2026-08-06-alarm-verbs-reach-stage-0.md b/docs/evals/2026-08-06-alarm-verbs-reach-stage-0.md new file mode 100644 index 0000000..acac981 --- /dev/null +++ b/docs/evals/2026-08-06-alarm-verbs-reach-stage-0.md @@ -0,0 +1,51 @@ +# Alarm verbs reach stage 0 + +**06-08-2026. V-627.** Measured with `TestONNXBaseline`, 91-case RU routing fixture, +classifier plus the ONNX embedder. No LLM arm in this run. + +## What was wrong + +`lexicon.ReminderVerbs` held five words and none of them named an alarm. `ReminderGrammar` +in `internal/router/stage0.go` did not read the set at all: it carried the literal +`напомни|remind me`. So no part of the cascade recognised `разбуди`. + +Three fixture cases ride on that. Under the classifier they went to fact and act at high +confidence, so the failure was never a near miss: + +- `ru-rem-005` "разбуди меня в 6:30" to fact at 0.918 +- `ru-rem-009` "разбуди меня полвосьмого" to act at 0.941 +- `en-rem-002` "wake me at 6:15" to fact at 0.899 + +Found while training the V-546 intent head, where the same three cases went to system. The +head reads a spoken time with no known verb in front of it as a clock question. The +classifier was making the same mistake in its own way. + +## The change + +Four alarm imperatives and bare `wake` join `reminder_verbs`. `ReminderGrammar` builds its +alternation from the set, longest alternative first, and eats an optional `мне`, `меня` or +`me` before the body. + +Longest-first is load-bearing. Go's alternation is leftmost-first rather than longest-match, +so `напомнить` listed after `напомни` would never match. + +## Result + +**66/91 to 69/91, 72.5% to 75.8% full.** Three cases gained, none lost. + +All three are the alarms above, and each now carries its time slot, which it did not before. +Clarify counts unchanged at 0 false and 8 missed. The two remaining system failures, +`какое число завтра` and `какой день недели послезавтра`, failed at baseline too. + +## What this does not fix + +The lexicon addition on its own moved nothing. Measured before touching the grammar: +**66/91**, exactly the baseline. Every consumer of `reminder_verbs` reads it after a reminder +route already exists. A verb that cannot win the route is a verb nobody asks about. The +grammar was the whole change. + +Lemma matching in `isReminderVerb` now covers `разбудил` as well as `разбуди`, because one +lemma holds both. That is the trap `cmd/mavend/quiet_toggle.go` documents for `говори`. It +is tolerable here and not in the quiet toggle. `isReminderVerb` runs only on an utterance +already routed to reminder, and it decides where the subject starts. A quiet match flips a +daemon-wide setting from any channel. diff --git a/docs/evals/2026-08-06-seeds-to-prompt-boundary.md b/docs/evals/2026-08-06-seeds-to-prompt-boundary.md new file mode 100644 index 0000000..6eb9553 --- /dev/null +++ b/docs/evals/2026-08-06-seeds-to-prompt-boundary.md @@ -0,0 +1,58 @@ +# Moving the seed files onto the router prompt's boundaries + +**06-08-2026. V-626.** Measured with `TestONNXBaseline`, 91-case RU routing fixture, +classifier plus the ONNX embedder. No LLM arm in this run. + +`docs/evals/2026-08-06-seed-labels-vs-router-prompt.md` found three intent boundaries where +`models/seeds` and `routeSystem` disagree. This applies two of them and rejects the third, +because the third was measured and it costs a case. + +## Baseline + +**64/91, 70.3% full.** Latency p50 22.9ms. + +## What moved + +**Sensor and host state, system to query. 26 lines.** `какая температура воздуха`, +`сколько памяти занято`, `какой статус сервисов`. The prompt restricts system to the clock, +the calendar date and the assistant itself, which is the V-374 edit of 31-07-2026. + +**World questions, chat to query. 8 lines.** `почему небо голубое`, `why is the sky blue`, +`как работает интернет`. Only the genuine world-knowledge lines. An opener about herself +stays in chat. `как тебя зовут` is a question word by rule 4 and about the assistant by +rule 8. The rules are ordered and rule 4 fires first, which reads wrong. That is a prompt +question rather than a seed question. + +`system.txt` goes from 43 lines to 17 and `query.txt` from 64 to 98. + +## Result + +**66/91, 72.5% full.** Two cases gained, none lost. + +- `en-sys-002` "turn quiet mode back on", quiet 2/3 to 3/3 +- `ru-query-011` "почему сервер тормозит", homelab 5/6 to 6/6 + +Clarify counts unchanged at 0 false and 8 missed. The eight missed clarifies are the +`ambiguous` tag and this change does not touch them. `TestONNXRecall`, `TestONNXTopics`, +`TestONNXPersonalBoundary` and `TestONNXClaimConfidenceDistribution` all pass. + +Thinning system to 17 lines did not hurt it. The two remaining system failures, +`какое число завтра` and `какой день недели послезавтра`, both failed at baseline too. + +## The third boundary, measured and rejected + +`reminder.txt` holds eight bare verbs: `поставь напоминание`, `создай напоминание`, +`set a reminder`. Rule 9 of the prompt calls an utterance with no named subject unknown. +By the prompt they do not belong in a reminder seed set. + +Dropping them scores **65/91**, one below keeping them. `ru-rem-004` "поставь напоминание +через полчаса" falls from reminder to fact, because the centroid loses the phrase the +utterance is built from. + +So the seed file and the prompt are not stale against each other here. They have different +jobs. A prompt classifies one utterance and can say it cannot. A nearest-neighbour centroid +is a shape to be near, and a bare verb phrase is part of that shape. The eight lines stay. + +That distinction matters past this file. V-546 trains a classification head on labeled +utterances rather than a centroid, and the head is the prompt's kind of thing. These eight +lines are seed data and not training data. diff --git a/internal/lexicon/lexicon_ru_v1.json b/internal/lexicon/lexicon_ru_v1.json index 8629c96..f958601 100644 --- a/internal/lexicon/lexicon_ru_v1.json +++ b/internal/lexicon/lexicon_ru_v1.json @@ -173,10 +173,11 @@ ] }, "reminder_verbs": { - "note": "The imperatives that mean \"remind me\", in the forms he speaks. The same kind of set as capture_verbs and decided the same way: it is her vocabulary, not a discovery about Russian (Vikunja #530).", + "note": "The imperatives that mean \"remind me\", in the forms he speaks. The same kind of set as capture_verbs and decided the same way: it is her vocabulary, not a discovery about Russian (Vikunja #530). The alarm verbs joined them in V-627. \"разбуди меня в 6:30\" is a reminder that fires at the hour he gets up, and the set knew no form of it, so an alarm reached IntentReminder only by resembling one to the embedder.", "words": [ "напомни", "напомните", "напомнить", "напоминай", - "remind" + "разбуди", "разбудите", "разбудить", "буди", + "remind", "wake" ] }, "half_hour": { diff --git a/internal/router/stage0.go b/internal/router/stage0.go index 8faa4f0..1a4c117 100644 --- a/internal/router/stage0.go +++ b/internal/router/stage0.go @@ -2,6 +2,7 @@ package router import ( "regexp" + "sort" "strings" "unicode" @@ -94,10 +95,41 @@ func DefaultGrammars(actMatcher ActMatcher) []Grammar { // stage-0 decision too (fillMatchedSlots in router.go). Before that it did not, // so "напомни в 11:00 позвонить маме" reached the daemon with HasTime false and // was asked "Когда?" about an hour he had just said. +// +// The verb alternation is built from lexicon.ReminderVerbs rather than written +// out (V-627). The literal here knew "напомни" and "remind me" and nothing +// else, so "разбуди меня в 6:30" never reached stage 0 — and it does not reach +// IntentReminder further down either, where the classifier calls it fact at +// 0.918. An alarm is a reminder that fires at the hour he gets up, and the +// verb that names one is her vocabulary, so it belongs in the lexicon with the +// rest of it. +// +// Longest-first ordering matters: Go's regexp alternation is leftmost-first, +// not longest-match, so "напомнить" listed after "напомни" would never match. +var reminderVerbPattern = regexp.MustCompile( + `(?i)^\s*(?:` + longestFirstAlternation(lexicon.ReminderVerbs()) + + `)\s*(?:мне|меня|me)?[\s,:]+(.+)$`) + +// longestFirstAlternation joins a word set into a regexp alternation, longest +// alternative first, with every member escaped. +func longestFirstAlternation(set []string) string { + out := make([]string, 0, len(set)) + for _, w := range set { + out = append(out, regexp.QuoteMeta(w)) + } + sort.Slice(out, func(i, j int) bool { + if len(out[i]) != len(out[j]) { + return len(out[i]) > len(out[j]) + } + return out[i] < out[j] + }) + return strings.Join(out, "|") +} + func ReminderGrammar() Grammar { return Grammar{ Name: "reminder-wakeword", - Pattern: regexp.MustCompile(`(?i)^\s*(?:напомни|remind me)[\s,:]+(.+)$`), + Pattern: reminderVerbPattern, Build: func(m []string) (Decision, bool) { rest := strings.TrimSpace(m[1]) if rest == "" { diff --git a/models/seeds/chat.txt b/models/seeds/chat.txt index 8639afa..ce7d2a2 100644 --- a/models/seeds/chat.txt +++ b/models/seeds/chat.txt @@ -8,12 +8,9 @@ как прошёл день расскажи про себя ты мне нравишься -почему небо голубое о чём поговорим у тебя есть чувства -что такое любовь расскажи историю -как работает интернет шутка анекдот пошути @@ -24,8 +21,6 @@ что нового думаешь о чём-то расскажи про космос -почему трава зелёная -откуда берётся дождь что было интересного сегодня как тебя зовут сколько тебе лет @@ -37,7 +32,4 @@ i'm bored what's up tell me a joke do you have feelings -what is love tell me about yourself -why is the sky blue -how does the internet work diff --git a/models/seeds/query.txt b/models/seeds/query.txt index f74aaec..7d1339b 100644 --- a/models/seeds/query.txt +++ b/models/seeds/query.txt @@ -62,3 +62,37 @@ what did I note about the garden будет дождь погода на сегодня weather in london +сколько времени осталось до вечера +какая температура воздуха +сколько человек дома +кто сейчас дома +есть ли кто дома +кто дома сейчас +все ли дома +сколько памяти занято +какая загрузка процессора +сколько свободного места на диске +какой ip адрес у сервера +всё ли работает +сколько сервер работает без перезагрузки +когда сервер запускался +какая версия софта +сколько аптайм +какой статус сервисов +все ли сервисы работают +что с интернетом +когда последний раз перезагружался +сколько трафика сегодня +какая скорость интернета +сколько процессов запущено +как загрузка системы +сколько оперативной памяти свободно +какая температура процессора +почему небо голубое +что такое любовь +как работает интернет +почему трава зелёная +откуда берётся дождь +what is love +why is the sky blue +how does the internet work diff --git a/models/seeds/system.txt b/models/seeds/system.txt index aba46a3..885074f 100644 --- a/models/seeds/system.txt +++ b/models/seeds/system.txt @@ -5,36 +5,10 @@ который час в Москве сколько сейчас времени который час у нас -сколько времени осталось до вечера какой сегодня день недели -какая температура воздуха -сколько человек дома -кто сейчас дома -есть ли кто дома -кто дома сейчас -все ли дома -сколько памяти занято -какая загрузка процессора -сколько свободного места на диске -какой ip адрес у сервера как дела у сервера -всё ли работает -сколько сервер работает без перезагрузки -когда сервер запускался -какая версия софта -сколько аптайм -какой статус сервисов -все ли сервисы работают -что с интернетом интернет работает -когда последний раз перезагружался -сколько трафика сегодня -какая скорость интернета загрузка сети -сколько процессов запущено -как загрузка системы -сколько оперативной памяти свободно -какая температура процессора тихий режим тихо не шуми @@ -48,4 +22,4 @@ quiet mode on quiet mode off quiet on -quiet off \ No newline at end of file +quiet off