From e7537d032e6f3dbc7881c050d792d9101fa5c52f Mon Sep 17 00:00:00 2001 From: claude Date: Thu, 6 Aug 2026 13:26:23 +0400 Subject: [PATCH 1/2] move the seed files onto the router prompt's intent boundaries (V-626) The classifier learns models/seeds and the router is prompted with routeSystem, and they held different definitions on 80 lines. Sensor and host state was system in the seeds and is query in the prompt, which is the V-374 edit the seeds never received. World questions were chat, written before external search could answer them. 64/91 to 66/91 on the fixture. en-sys-002 and ru-query-011 gain, nothing regresses, clarify counts unchanged. The third disagreement is measured and rejected. Dropping the eight bare reminder verbs scores 65, because a centroid is a shape to be near and the bare verb phrase is part of that shape. A seed file and a prompt have different jobs there. --- .../2026-08-06-seeds-to-prompt-boundary.md | 58 +++++++++++++++++++ models/seeds/chat.txt | 8 --- models/seeds/query.txt | 34 +++++++++++ models/seeds/system.txt | 28 +-------- 4 files changed, 93 insertions(+), 35 deletions(-) create mode 100644 docs/evals/2026-08-06-seeds-to-prompt-boundary.md diff --git a/docs/evals/2026-08-06-seeds-to-prompt-boundary.md b/docs/evals/2026-08-06-seeds-to-prompt-boundary.md new file mode 100644 index 0000000..6eb9553 --- /dev/null +++ b/docs/evals/2026-08-06-seeds-to-prompt-boundary.md @@ -0,0 +1,58 @@ +# Moving the seed files onto the router prompt's boundaries + +**06-08-2026. V-626.** Measured with `TestONNXBaseline`, 91-case RU routing fixture, +classifier plus the ONNX embedder. No LLM arm in this run. + +`docs/evals/2026-08-06-seed-labels-vs-router-prompt.md` found three intent boundaries where +`models/seeds` and `routeSystem` disagree. This applies two of them and rejects the third, +because the third was measured and it costs a case. + +## Baseline + +**64/91, 70.3% full.** Latency p50 22.9ms. + +## What moved + +**Sensor and host state, system to query. 26 lines.** `какая температура воздуха`, +`сколько памяти занято`, `какой статус сервисов`. The prompt restricts system to the clock, +the calendar date and the assistant itself, which is the V-374 edit of 31-07-2026. + +**World questions, chat to query. 8 lines.** `почему небо голубое`, `why is the sky blue`, +`как работает интернет`. Only the genuine world-knowledge lines. An opener about herself +stays in chat. `как тебя зовут` is a question word by rule 4 and about the assistant by +rule 8. The rules are ordered and rule 4 fires first, which reads wrong. That is a prompt +question rather than a seed question. + +`system.txt` goes from 43 lines to 17 and `query.txt` from 64 to 98. + +## Result + +**66/91, 72.5% full.** Two cases gained, none lost. + +- `en-sys-002` "turn quiet mode back on", quiet 2/3 to 3/3 +- `ru-query-011` "почему сервер тормозит", homelab 5/6 to 6/6 + +Clarify counts unchanged at 0 false and 8 missed. The eight missed clarifies are the +`ambiguous` tag and this change does not touch them. `TestONNXRecall`, `TestONNXTopics`, +`TestONNXPersonalBoundary` and `TestONNXClaimConfidenceDistribution` all pass. + +Thinning system to 17 lines did not hurt it. The two remaining system failures, +`какое число завтра` and `какой день недели послезавтра`, both failed at baseline too. + +## The third boundary, measured and rejected + +`reminder.txt` holds eight bare verbs: `поставь напоминание`, `создай напоминание`, +`set a reminder`. Rule 9 of the prompt calls an utterance with no named subject unknown. +By the prompt they do not belong in a reminder seed set. + +Dropping them scores **65/91**, one below keeping them. `ru-rem-004` "поставь напоминание +через полчаса" falls from reminder to fact, because the centroid loses the phrase the +utterance is built from. + +So the seed file and the prompt are not stale against each other here. They have different +jobs. A prompt classifies one utterance and can say it cannot. A nearest-neighbour centroid +is a shape to be near, and a bare verb phrase is part of that shape. The eight lines stay. + +That distinction matters past this file. V-546 trains a classification head on labeled +utterances rather than a centroid, and the head is the prompt's kind of thing. These eight +lines are seed data and not training data. diff --git a/models/seeds/chat.txt b/models/seeds/chat.txt index 8639afa..ce7d2a2 100644 --- a/models/seeds/chat.txt +++ b/models/seeds/chat.txt @@ -8,12 +8,9 @@ как прошёл день расскажи про себя ты мне нравишься -почему небо голубое о чём поговорим у тебя есть чувства -что такое любовь расскажи историю -как работает интернет шутка анекдот пошути @@ -24,8 +21,6 @@ что нового думаешь о чём-то расскажи про космос -почему трава зелёная -откуда берётся дождь что было интересного сегодня как тебя зовут сколько тебе лет @@ -37,7 +32,4 @@ i'm bored what's up tell me a joke do you have feelings -what is love tell me about yourself -why is the sky blue -how does the internet work diff --git a/models/seeds/query.txt b/models/seeds/query.txt index f74aaec..7d1339b 100644 --- a/models/seeds/query.txt +++ b/models/seeds/query.txt @@ -62,3 +62,37 @@ what did I note about the garden будет дождь погода на сегодня weather in london +сколько времени осталось до вечера +какая температура воздуха +сколько человек дома +кто сейчас дома +есть ли кто дома +кто дома сейчас +все ли дома +сколько памяти занято +какая загрузка процессора +сколько свободного места на диске +какой ip адрес у сервера +всё ли работает +сколько сервер работает без перезагрузки +когда сервер запускался +какая версия софта +сколько аптайм +какой статус сервисов +все ли сервисы работают +что с интернетом +когда последний раз перезагружался +сколько трафика сегодня +какая скорость интернета +сколько процессов запущено +как загрузка системы +сколько оперативной памяти свободно +какая температура процессора +почему небо голубое +что такое любовь +как работает интернет +почему трава зелёная +откуда берётся дождь +what is love +why is the sky blue +how does the internet work diff --git a/models/seeds/system.txt b/models/seeds/system.txt index aba46a3..885074f 100644 --- a/models/seeds/system.txt +++ b/models/seeds/system.txt @@ -5,36 +5,10 @@ который час в Москве сколько сейчас времени который час у нас -сколько времени осталось до вечера какой сегодня день недели -какая температура воздуха -сколько человек дома -кто сейчас дома -есть ли кто дома -кто дома сейчас -все ли дома -сколько памяти занято -какая загрузка процессора -сколько свободного места на диске -какой ip адрес у сервера как дела у сервера -всё ли работает -сколько сервер работает без перезагрузки -когда сервер запускался -какая версия софта -сколько аптайм -какой статус сервисов -все ли сервисы работают -что с интернетом интернет работает -когда последний раз перезагружался -сколько трафика сегодня -какая скорость интернета загрузка сети -сколько процессов запущено -как загрузка системы -сколько оперативной памяти свободно -какая температура процессора тихий режим тихо не шуми @@ -48,4 +22,4 @@ quiet mode on quiet mode off quiet on -quiet off \ No newline at end of file +quiet off -- 2.52.0 From 1f8e9f21ce43436ec2488002b6dc822b0126dbb8 Mon Sep 17 00:00:00 2001 From: claude Date: Thu, 6 Aug 2026 15:04:08 +0400 Subject: [PATCH 2/2] an alarm verb reaches stage 0, and the reminder grammar reads the lexicon (V-627) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit reminder_verbs held five words and none named an alarm, and ReminderGrammar did not read the set anyway — it carried the literal напомни|remind me. So no part of the cascade recognised разбуди, and the three alarm cases in the fixture went to fact and act at over 0.89. The lexicon addition alone moved nothing, measured at 66/91. Every consumer reads the set after a reminder route already exists. Building the grammar's alternation from the set is what scored: 66/91 to 69/91, three cases gained, none lost, and each alarm now carries its time slot. Longest-first ordering in the alternation is load-bearing. Go's regexp alternation is leftmost-first, so напомнить after напомни would never match. Found while training the V-546 intent head, where the same three cases went to system. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0117tgnmbgZpHVV3XSNw8Qua --- .../2026-08-06-alarm-verbs-reach-stage-0.md | 51 +++++++++++++++++++ internal/lexicon/lexicon_ru_v1.json | 5 +- internal/router/stage0.go | 34 ++++++++++++- 3 files changed, 87 insertions(+), 3 deletions(-) create mode 100644 docs/evals/2026-08-06-alarm-verbs-reach-stage-0.md diff --git a/docs/evals/2026-08-06-alarm-verbs-reach-stage-0.md b/docs/evals/2026-08-06-alarm-verbs-reach-stage-0.md new file mode 100644 index 0000000..acac981 --- /dev/null +++ b/docs/evals/2026-08-06-alarm-verbs-reach-stage-0.md @@ -0,0 +1,51 @@ +# Alarm verbs reach stage 0 + +**06-08-2026. V-627.** Measured with `TestONNXBaseline`, 91-case RU routing fixture, +classifier plus the ONNX embedder. No LLM arm in this run. + +## What was wrong + +`lexicon.ReminderVerbs` held five words and none of them named an alarm. `ReminderGrammar` +in `internal/router/stage0.go` did not read the set at all: it carried the literal +`напомни|remind me`. So no part of the cascade recognised `разбуди`. + +Three fixture cases ride on that. Under the classifier they went to fact and act at high +confidence, so the failure was never a near miss: + +- `ru-rem-005` "разбуди меня в 6:30" to fact at 0.918 +- `ru-rem-009` "разбуди меня полвосьмого" to act at 0.941 +- `en-rem-002` "wake me at 6:15" to fact at 0.899 + +Found while training the V-546 intent head, where the same three cases went to system. The +head reads a spoken time with no known verb in front of it as a clock question. The +classifier was making the same mistake in its own way. + +## The change + +Four alarm imperatives and bare `wake` join `reminder_verbs`. `ReminderGrammar` builds its +alternation from the set, longest alternative first, and eats an optional `мне`, `меня` or +`me` before the body. + +Longest-first is load-bearing. Go's alternation is leftmost-first rather than longest-match, +so `напомнить` listed after `напомни` would never match. + +## Result + +**66/91 to 69/91, 72.5% to 75.8% full.** Three cases gained, none lost. + +All three are the alarms above, and each now carries its time slot, which it did not before. +Clarify counts unchanged at 0 false and 8 missed. The two remaining system failures, +`какое число завтра` and `какой день недели послезавтра`, failed at baseline too. + +## What this does not fix + +The lexicon addition on its own moved nothing. Measured before touching the grammar: +**66/91**, exactly the baseline. Every consumer of `reminder_verbs` reads it after a reminder +route already exists. A verb that cannot win the route is a verb nobody asks about. The +grammar was the whole change. + +Lemma matching in `isReminderVerb` now covers `разбудил` as well as `разбуди`, because one +lemma holds both. That is the trap `cmd/mavend/quiet_toggle.go` documents for `говори`. It +is tolerable here and not in the quiet toggle. `isReminderVerb` runs only on an utterance +already routed to reminder, and it decides where the subject starts. A quiet match flips a +daemon-wide setting from any channel. diff --git a/internal/lexicon/lexicon_ru_v1.json b/internal/lexicon/lexicon_ru_v1.json index 8629c96..f958601 100644 --- a/internal/lexicon/lexicon_ru_v1.json +++ b/internal/lexicon/lexicon_ru_v1.json @@ -173,10 +173,11 @@ ] }, "reminder_verbs": { - "note": "The imperatives that mean \"remind me\", in the forms he speaks. The same kind of set as capture_verbs and decided the same way: it is her vocabulary, not a discovery about Russian (Vikunja #530).", + "note": "The imperatives that mean \"remind me\", in the forms he speaks. The same kind of set as capture_verbs and decided the same way: it is her vocabulary, not a discovery about Russian (Vikunja #530). The alarm verbs joined them in V-627. \"разбуди меня в 6:30\" is a reminder that fires at the hour he gets up, and the set knew no form of it, so an alarm reached IntentReminder only by resembling one to the embedder.", "words": [ "напомни", "напомните", "напомнить", "напоминай", - "remind" + "разбуди", "разбудите", "разбудить", "буди", + "remind", "wake" ] }, "half_hour": { diff --git a/internal/router/stage0.go b/internal/router/stage0.go index 8faa4f0..1a4c117 100644 --- a/internal/router/stage0.go +++ b/internal/router/stage0.go @@ -2,6 +2,7 @@ package router import ( "regexp" + "sort" "strings" "unicode" @@ -94,10 +95,41 @@ func DefaultGrammars(actMatcher ActMatcher) []Grammar { // stage-0 decision too (fillMatchedSlots in router.go). Before that it did not, // so "напомни в 11:00 позвонить маме" reached the daemon with HasTime false and // was asked "Когда?" about an hour he had just said. +// +// The verb alternation is built from lexicon.ReminderVerbs rather than written +// out (V-627). The literal here knew "напомни" and "remind me" and nothing +// else, so "разбуди меня в 6:30" never reached stage 0 — and it does not reach +// IntentReminder further down either, where the classifier calls it fact at +// 0.918. An alarm is a reminder that fires at the hour he gets up, and the +// verb that names one is her vocabulary, so it belongs in the lexicon with the +// rest of it. +// +// Longest-first ordering matters: Go's regexp alternation is leftmost-first, +// not longest-match, so "напомнить" listed after "напомни" would never match. +var reminderVerbPattern = regexp.MustCompile( + `(?i)^\s*(?:` + longestFirstAlternation(lexicon.ReminderVerbs()) + + `)\s*(?:мне|меня|me)?[\s,:]+(.+)$`) + +// longestFirstAlternation joins a word set into a regexp alternation, longest +// alternative first, with every member escaped. +func longestFirstAlternation(set []string) string { + out := make([]string, 0, len(set)) + for _, w := range set { + out = append(out, regexp.QuoteMeta(w)) + } + sort.Slice(out, func(i, j int) bool { + if len(out[i]) != len(out[j]) { + return len(out[i]) > len(out[j]) + } + return out[i] < out[j] + }) + return strings.Join(out, "|") +} + func ReminderGrammar() Grammar { return Grammar{ Name: "reminder-wakeword", - Pattern: regexp.MustCompile(`(?i)^\s*(?:напомни|remind me)[\s,:]+(.+)$`), + Pattern: reminderVerbPattern, Build: func(m []string) (Decision, bool) { rest := strings.TrimSpace(m[1]) if rest == "" { -- 2.52.0