Merge pull request 'reminder_verbs has no alarm verb, so an alarm never routes (V-627)' (#180) from task/627-reminder-verbs-has-no-alarm-verb-so-an-a into master
This commit was merged in pull request #180.
This commit is contained in:
@@ -0,0 +1,51 @@
|
|||||||
|
# Alarm verbs reach stage 0
|
||||||
|
|
||||||
|
**06-08-2026. V-627.** Measured with `TestONNXBaseline`, 91-case RU routing fixture,
|
||||||
|
classifier plus the ONNX embedder. No LLM arm in this run.
|
||||||
|
|
||||||
|
## What was wrong
|
||||||
|
|
||||||
|
`lexicon.ReminderVerbs` held five words and none of them named an alarm. `ReminderGrammar`
|
||||||
|
in `internal/router/stage0.go` did not read the set at all: it carried the literal
|
||||||
|
`напомни|remind me`. So no part of the cascade recognised `разбуди`.
|
||||||
|
|
||||||
|
Three fixture cases ride on that. Under the classifier they went to fact and act at high
|
||||||
|
confidence, so the failure was never a near miss:
|
||||||
|
|
||||||
|
- `ru-rem-005` "разбуди меня в 6:30" to fact at 0.918
|
||||||
|
- `ru-rem-009` "разбуди меня полвосьмого" to act at 0.941
|
||||||
|
- `en-rem-002` "wake me at 6:15" to fact at 0.899
|
||||||
|
|
||||||
|
Found while training the V-546 intent head, where the same three cases went to system. The
|
||||||
|
head reads a spoken time with no known verb in front of it as a clock question. The
|
||||||
|
classifier was making the same mistake in its own way.
|
||||||
|
|
||||||
|
## The change
|
||||||
|
|
||||||
|
Four alarm imperatives and bare `wake` join `reminder_verbs`. `ReminderGrammar` builds its
|
||||||
|
alternation from the set, longest alternative first, and eats an optional `мне`, `меня` or
|
||||||
|
`me` before the body.
|
||||||
|
|
||||||
|
Longest-first is load-bearing. Go's alternation is leftmost-first rather than longest-match,
|
||||||
|
so `напомнить` listed after `напомни` would never match.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
**66/91 to 69/91, 72.5% to 75.8% full.** Three cases gained, none lost.
|
||||||
|
|
||||||
|
All three are the alarms above, and each now carries its time slot, which it did not before.
|
||||||
|
Clarify counts unchanged at 0 false and 8 missed. The two remaining system failures,
|
||||||
|
`какое число завтра` and `какой день недели послезавтра`, failed at baseline too.
|
||||||
|
|
||||||
|
## What this does not fix
|
||||||
|
|
||||||
|
The lexicon addition on its own moved nothing. Measured before touching the grammar:
|
||||||
|
**66/91**, exactly the baseline. Every consumer of `reminder_verbs` reads it after a reminder
|
||||||
|
route already exists. A verb that cannot win the route is a verb nobody asks about. The
|
||||||
|
grammar was the whole change.
|
||||||
|
|
||||||
|
Lemma matching in `isReminderVerb` now covers `разбудил` as well as `разбуди`, because one
|
||||||
|
lemma holds both. That is the trap `cmd/mavend/quiet_toggle.go` documents for `говори`. It
|
||||||
|
is tolerable here and not in the quiet toggle. `isReminderVerb` runs only on an utterance
|
||||||
|
already routed to reminder, and it decides where the subject starts. A quiet match flips a
|
||||||
|
daemon-wide setting from any channel.
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
# Moving the seed files onto the router prompt's boundaries
|
||||||
|
|
||||||
|
**06-08-2026. V-626.** Measured with `TestONNXBaseline`, 91-case RU routing fixture,
|
||||||
|
classifier plus the ONNX embedder. No LLM arm in this run.
|
||||||
|
|
||||||
|
`docs/evals/2026-08-06-seed-labels-vs-router-prompt.md` found three intent boundaries where
|
||||||
|
`models/seeds` and `routeSystem` disagree. This applies two of them and rejects the third,
|
||||||
|
because the third was measured and it costs a case.
|
||||||
|
|
||||||
|
## Baseline
|
||||||
|
|
||||||
|
**64/91, 70.3% full.** Latency p50 22.9ms.
|
||||||
|
|
||||||
|
## What moved
|
||||||
|
|
||||||
|
**Sensor and host state, system to query. 26 lines.** `какая температура воздуха`,
|
||||||
|
`сколько памяти занято`, `какой статус сервисов`. The prompt restricts system to the clock,
|
||||||
|
the calendar date and the assistant itself, which is the V-374 edit of 31-07-2026.
|
||||||
|
|
||||||
|
**World questions, chat to query. 8 lines.** `почему небо голубое`, `why is the sky blue`,
|
||||||
|
`как работает интернет`. Only the genuine world-knowledge lines. An opener about herself
|
||||||
|
stays in chat. `как тебя зовут` is a question word by rule 4 and about the assistant by
|
||||||
|
rule 8. The rules are ordered and rule 4 fires first, which reads wrong. That is a prompt
|
||||||
|
question rather than a seed question.
|
||||||
|
|
||||||
|
`system.txt` goes from 43 lines to 17 and `query.txt` from 64 to 98.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
**66/91, 72.5% full.** Two cases gained, none lost.
|
||||||
|
|
||||||
|
- `en-sys-002` "turn quiet mode back on", quiet 2/3 to 3/3
|
||||||
|
- `ru-query-011` "почему сервер тормозит", homelab 5/6 to 6/6
|
||||||
|
|
||||||
|
Clarify counts unchanged at 0 false and 8 missed. The eight missed clarifies are the
|
||||||
|
`ambiguous` tag and this change does not touch them. `TestONNXRecall`, `TestONNXTopics`,
|
||||||
|
`TestONNXPersonalBoundary` and `TestONNXClaimConfidenceDistribution` all pass.
|
||||||
|
|
||||||
|
Thinning system to 17 lines did not hurt it. The two remaining system failures,
|
||||||
|
`какое число завтра` and `какой день недели послезавтра`, both failed at baseline too.
|
||||||
|
|
||||||
|
## The third boundary, measured and rejected
|
||||||
|
|
||||||
|
`reminder.txt` holds eight bare verbs: `поставь напоминание`, `создай напоминание`,
|
||||||
|
`set a reminder`. Rule 9 of the prompt calls an utterance with no named subject unknown.
|
||||||
|
By the prompt they do not belong in a reminder seed set.
|
||||||
|
|
||||||
|
Dropping them scores **65/91**, one below keeping them. `ru-rem-004` "поставь напоминание
|
||||||
|
через полчаса" falls from reminder to fact, because the centroid loses the phrase the
|
||||||
|
utterance is built from.
|
||||||
|
|
||||||
|
So the seed file and the prompt are not stale against each other here. They have different
|
||||||
|
jobs. A prompt classifies one utterance and can say it cannot. A nearest-neighbour centroid
|
||||||
|
is a shape to be near, and a bare verb phrase is part of that shape. The eight lines stay.
|
||||||
|
|
||||||
|
That distinction matters past this file. V-546 trains a classification head on labeled
|
||||||
|
utterances rather than a centroid, and the head is the prompt's kind of thing. These eight
|
||||||
|
lines are seed data and not training data.
|
||||||
@@ -173,10 +173,11 @@
|
|||||||
]
|
]
|
||||||
},
|
},
|
||||||
"reminder_verbs": {
|
"reminder_verbs": {
|
||||||
"note": "The imperatives that mean \"remind me\", in the forms he speaks. The same kind of set as capture_verbs and decided the same way: it is her vocabulary, not a discovery about Russian (Vikunja #530).",
|
"note": "The imperatives that mean \"remind me\", in the forms he speaks. The same kind of set as capture_verbs and decided the same way: it is her vocabulary, not a discovery about Russian (Vikunja #530). The alarm verbs joined them in V-627. \"разбуди меня в 6:30\" is a reminder that fires at the hour he gets up, and the set knew no form of it, so an alarm reached IntentReminder only by resembling one to the embedder.",
|
||||||
"words": [
|
"words": [
|
||||||
"напомни", "напомните", "напомнить", "напоминай",
|
"напомни", "напомните", "напомнить", "напоминай",
|
||||||
"remind"
|
"разбуди", "разбудите", "разбудить", "буди",
|
||||||
|
"remind", "wake"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"half_hour": {
|
"half_hour": {
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package router
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"regexp"
|
"regexp"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"unicode"
|
"unicode"
|
||||||
|
|
||||||
@@ -94,10 +95,41 @@ func DefaultGrammars(actMatcher ActMatcher) []Grammar {
|
|||||||
// stage-0 decision too (fillMatchedSlots in router.go). Before that it did not,
|
// stage-0 decision too (fillMatchedSlots in router.go). Before that it did not,
|
||||||
// so "напомни в 11:00 позвонить маме" reached the daemon with HasTime false and
|
// so "напомни в 11:00 позвонить маме" reached the daemon with HasTime false and
|
||||||
// was asked "Когда?" about an hour he had just said.
|
// was asked "Когда?" about an hour he had just said.
|
||||||
|
//
|
||||||
|
// The verb alternation is built from lexicon.ReminderVerbs rather than written
|
||||||
|
// out (V-627). The literal here knew "напомни" and "remind me" and nothing
|
||||||
|
// else, so "разбуди меня в 6:30" never reached stage 0 — and it does not reach
|
||||||
|
// IntentReminder further down either, where the classifier calls it fact at
|
||||||
|
// 0.918. An alarm is a reminder that fires at the hour he gets up, and the
|
||||||
|
// verb that names one is her vocabulary, so it belongs in the lexicon with the
|
||||||
|
// rest of it.
|
||||||
|
//
|
||||||
|
// Longest-first ordering matters: Go's regexp alternation is leftmost-first,
|
||||||
|
// not longest-match, so "напомнить" listed after "напомни" would never match.
|
||||||
|
var reminderVerbPattern = regexp.MustCompile(
|
||||||
|
`(?i)^\s*(?:` + longestFirstAlternation(lexicon.ReminderVerbs()) +
|
||||||
|
`)\s*(?:мне|меня|me)?[\s,:]+(.+)$`)
|
||||||
|
|
||||||
|
// longestFirstAlternation joins a word set into a regexp alternation, longest
|
||||||
|
// alternative first, with every member escaped.
|
||||||
|
func longestFirstAlternation(set []string) string {
|
||||||
|
out := make([]string, 0, len(set))
|
||||||
|
for _, w := range set {
|
||||||
|
out = append(out, regexp.QuoteMeta(w))
|
||||||
|
}
|
||||||
|
sort.Slice(out, func(i, j int) bool {
|
||||||
|
if len(out[i]) != len(out[j]) {
|
||||||
|
return len(out[i]) > len(out[j])
|
||||||
|
}
|
||||||
|
return out[i] < out[j]
|
||||||
|
})
|
||||||
|
return strings.Join(out, "|")
|
||||||
|
}
|
||||||
|
|
||||||
func ReminderGrammar() Grammar {
|
func ReminderGrammar() Grammar {
|
||||||
return Grammar{
|
return Grammar{
|
||||||
Name: "reminder-wakeword",
|
Name: "reminder-wakeword",
|
||||||
Pattern: regexp.MustCompile(`(?i)^\s*(?:напомни|remind me)[\s,:]+(.+)$`),
|
Pattern: reminderVerbPattern,
|
||||||
Build: func(m []string) (Decision, bool) {
|
Build: func(m []string) (Decision, bool) {
|
||||||
rest := strings.TrimSpace(m[1])
|
rest := strings.TrimSpace(m[1])
|
||||||
if rest == "" {
|
if rest == "" {
|
||||||
|
|||||||
@@ -8,12 +8,9 @@
|
|||||||
как прошёл день
|
как прошёл день
|
||||||
расскажи про себя
|
расскажи про себя
|
||||||
ты мне нравишься
|
ты мне нравишься
|
||||||
почему небо голубое
|
|
||||||
о чём поговорим
|
о чём поговорим
|
||||||
у тебя есть чувства
|
у тебя есть чувства
|
||||||
что такое любовь
|
|
||||||
расскажи историю
|
расскажи историю
|
||||||
как работает интернет
|
|
||||||
шутка
|
шутка
|
||||||
анекдот
|
анекдот
|
||||||
пошути
|
пошути
|
||||||
@@ -24,8 +21,6 @@
|
|||||||
что нового
|
что нового
|
||||||
думаешь о чём-то
|
думаешь о чём-то
|
||||||
расскажи про космос
|
расскажи про космос
|
||||||
почему трава зелёная
|
|
||||||
откуда берётся дождь
|
|
||||||
что было интересного сегодня
|
что было интересного сегодня
|
||||||
как тебя зовут
|
как тебя зовут
|
||||||
сколько тебе лет
|
сколько тебе лет
|
||||||
@@ -37,7 +32,4 @@ i'm bored
|
|||||||
what's up
|
what's up
|
||||||
tell me a joke
|
tell me a joke
|
||||||
do you have feelings
|
do you have feelings
|
||||||
what is love
|
|
||||||
tell me about yourself
|
tell me about yourself
|
||||||
why is the sky blue
|
|
||||||
how does the internet work
|
|
||||||
|
|||||||
@@ -62,3 +62,37 @@ what did I note about the garden
|
|||||||
будет дождь
|
будет дождь
|
||||||
погода на сегодня
|
погода на сегодня
|
||||||
weather in london
|
weather in london
|
||||||
|
сколько времени осталось до вечера
|
||||||
|
какая температура воздуха
|
||||||
|
сколько человек дома
|
||||||
|
кто сейчас дома
|
||||||
|
есть ли кто дома
|
||||||
|
кто дома сейчас
|
||||||
|
все ли дома
|
||||||
|
сколько памяти занято
|
||||||
|
какая загрузка процессора
|
||||||
|
сколько свободного места на диске
|
||||||
|
какой ip адрес у сервера
|
||||||
|
всё ли работает
|
||||||
|
сколько сервер работает без перезагрузки
|
||||||
|
когда сервер запускался
|
||||||
|
какая версия софта
|
||||||
|
сколько аптайм
|
||||||
|
какой статус сервисов
|
||||||
|
все ли сервисы работают
|
||||||
|
что с интернетом
|
||||||
|
когда последний раз перезагружался
|
||||||
|
сколько трафика сегодня
|
||||||
|
какая скорость интернета
|
||||||
|
сколько процессов запущено
|
||||||
|
как загрузка системы
|
||||||
|
сколько оперативной памяти свободно
|
||||||
|
какая температура процессора
|
||||||
|
почему небо голубое
|
||||||
|
что такое любовь
|
||||||
|
как работает интернет
|
||||||
|
почему трава зелёная
|
||||||
|
откуда берётся дождь
|
||||||
|
what is love
|
||||||
|
why is the sky blue
|
||||||
|
how does the internet work
|
||||||
|
|||||||
+1
-27
@@ -5,36 +5,10 @@
|
|||||||
который час в Москве
|
который час в Москве
|
||||||
сколько сейчас времени
|
сколько сейчас времени
|
||||||
который час у нас
|
который час у нас
|
||||||
сколько времени осталось до вечера
|
|
||||||
какой сегодня день недели
|
какой сегодня день недели
|
||||||
какая температура воздуха
|
|
||||||
сколько человек дома
|
|
||||||
кто сейчас дома
|
|
||||||
есть ли кто дома
|
|
||||||
кто дома сейчас
|
|
||||||
все ли дома
|
|
||||||
сколько памяти занято
|
|
||||||
какая загрузка процессора
|
|
||||||
сколько свободного места на диске
|
|
||||||
какой ip адрес у сервера
|
|
||||||
как дела у сервера
|
как дела у сервера
|
||||||
всё ли работает
|
|
||||||
сколько сервер работает без перезагрузки
|
|
||||||
когда сервер запускался
|
|
||||||
какая версия софта
|
|
||||||
сколько аптайм
|
|
||||||
какой статус сервисов
|
|
||||||
все ли сервисы работают
|
|
||||||
что с интернетом
|
|
||||||
интернет работает
|
интернет работает
|
||||||
когда последний раз перезагружался
|
|
||||||
сколько трафика сегодня
|
|
||||||
какая скорость интернета
|
|
||||||
загрузка сети
|
загрузка сети
|
||||||
сколько процессов запущено
|
|
||||||
как загрузка системы
|
|
||||||
сколько оперативной памяти свободно
|
|
||||||
какая температура процессора
|
|
||||||
тихий режим
|
тихий режим
|
||||||
тихо
|
тихо
|
||||||
не шуми
|
не шуми
|
||||||
@@ -48,4 +22,4 @@
|
|||||||
quiet mode on
|
quiet mode on
|
||||||
quiet mode off
|
quiet mode off
|
||||||
quiet on
|
quiet on
|
||||||
quiet off
|
quiet off
|
||||||
|
|||||||
Reference in New Issue
Block a user