Merge pull request 'QA: Voice session quality polish' (#171) from task/287-qa-voice-session-quality-polish into master
This commit was merged in pull request #171.
This commit is contained in:
+116
-3
@@ -1,6 +1,6 @@
|
||||
# QA plan: checking Maven properly
|
||||
|
||||
*Last verified: 2026-08-04 @ a4d5155. Living doc: correct it in place, do not append.*
|
||||
*Last verified: 2026-08-04 @ 8d816f4. Living doc: correct it in place, do not append.*
|
||||
|
||||
Written 2026-08-01, after the 35-PR stack landed and the box came back up.
|
||||
Refreshed 2026-08-02 against the live list, after PRs #85-#90.
|
||||
@@ -88,8 +88,71 @@ session quality), **321** steps 3-5 (quiet mode), **288** (STT golden audio).
|
||||
**288 is not blocked.** The fixtures are committed under `cmd/mavsttd/testdata/`
|
||||
and `make test-stt-golden` runs today. This plan said otherwise until 02-08-2026.
|
||||
|
||||
Steps 1 and 3-6 were run on 02-08-2026 and pass. Steps 2 and 7-9 still need a
|
||||
person at the box, because they need a microphone or a nudge to arrive.
|
||||
Steps 1 and 3-6 were run on 02-08-2026 and pass.
|
||||
|
||||
**Step 2 no longer needs a person, and step 9 has a number now** (04-08-2026).
|
||||
`POST /api/ptt` takes raw PCM16 16kHz mono and answers with audio plus an
|
||||
`X-Reply-Text` header, so the committed STT fixtures stand in for a microphone:
|
||||
|
||||
```sh
|
||||
tail -c +45 cmd/mavsttd/testdata/ru_query.wav > /tmp/q.pcm
|
||||
curl -s --noproxy '*' -D /tmp/h -o /tmp/reply.pcm -X POST \
|
||||
http://127.0.0.1:9201/api/ptt --data-binary @/tmp/q.pcm \
|
||||
-H 'Content-Type: application/octet-stream' -m 180
|
||||
```
|
||||
|
||||
That covers audio in → STT → router → phrasing → TTS audio out. It leaves only
|
||||
browser microphone capture needing a person, and the wake path needing a machine.
|
||||
Do not post `en_act.wav` without deciding first: it is a mutating act.
|
||||
|
||||
**Steps 7 and 8 still cannot run, but 15 is no longer the reason** (04-08-2026).
|
||||
The desk presence poster is installed on workpc. It is a `maven-desk` systemd
|
||||
user timer on a 60s cadence, gated by hypridle at 120s idle. `desk_active` facts
|
||||
now arrive, and the first landed at 18:43.
|
||||
|
||||
What blocks the two steps now is that no rule wants to fire. `/trace` shows all
|
||||
five at `predicate`, none inert:
|
||||
|
||||
| rule | sev | why it is false |
|
||||
|---|---|---|
|
||||
| water | 1 | needs ≥3h since the last `water` fact; step 2's `ru_fact` wrote one |
|
||||
| meal | 1 | needs ≥6h since a `meal` fact; none exists |
|
||||
| break | 2 | needs both `desk_active` and a `break` fact; `break` has never been written |
|
||||
| service_down | 4 | no kuma monitor is down |
|
||||
| netdata_critical | 3 | nothing critical |
|
||||
|
||||
So the honest way to run step 8 is to wait three hours after the last `water`
|
||||
fact, or to write one antedated. Do not read the water rule's silence as a defect.
|
||||
|
||||
**The sev4 telegram reach works** (04-08-2026). Resuming a paused kuma monitor
|
||||
for paperless, which is genuinely down, put a real `service_down` through the
|
||||
whole path with presence away:
|
||||
|
||||
```
|
||||
23:03 voicesink: no live voice session for service_down, falling through to away channels
|
||||
/notifications: 19:03 | service_down | telegram | pending | Сервис перестал отвечать.
|
||||
04.08 23:03 | nudge | service_down | telegram | sent | 23:03
|
||||
```
|
||||
|
||||
`ChannelsFor(Sev4, Away)` returned telegram, the send succeeded, and the row
|
||||
holds at `pending` because sev4 repeats until acked. The 15:51 row shows the
|
||||
same rule reaching `acted` earlier, so the ack path works too.
|
||||
|
||||
The body was `Сервис перестал отвечать.`, which names no service. That is a bug
|
||||
and it is deterministic, filed as **534**. `nudgeValues` fills `{service}` from
|
||||
`State.Fact("service_down")`, an exact key mavpoll stopped writing when
|
||||
per-monitor facts landed. Nine of the ten templates carry `{service}`, so all
|
||||
nine are rejected as unfillable. The one nameless variant is left as the only
|
||||
usable one, every time. The stub and LLM phrasers both call `loop.DownServices`
|
||||
and get it right. The template path is the one that runs.
|
||||
|
||||
**Presence itself has a real defect, filed as 532.** `SavePresenceState` has no
|
||||
caller outside tests, so the singleton row is never written. The gate is fine,
|
||||
because it reads the bucket `GatherState` computes in memory each tick. Two
|
||||
things follow. Hysteresis is dead, because `lastBucket` is always cold-start `Away`
|
||||
and the 0.30-0.55 hold band never applies. And every presence readout lies:
|
||||
`/dash` shows `away — score 0.00 (never)` with fresh `desk_active` facts arriving
|
||||
every 60s. Do not trust that number while checking anything else here.
|
||||
|
||||
Steps 1 and 3-6 do not need a browser. `POST /api/chat` takes a form-encoded
|
||||
`text=` field and a cookie jar, and answers with the rendered `/chat` page:
|
||||
@@ -112,6 +175,33 @@ turns look misaligned when they are not.
|
||||
back. This covers browser mic to STT to core to TTS as one path. It does
|
||||
**not** cover the wake word or the voice-activity gate, and no step here
|
||||
does — see below.
|
||||
**Passes below the browser** (04-08-2026, three fixtures through `/api/ptt`):
|
||||
HTTP 200, `audio/l16;rate=16000;channels=1`, and real speech back. `ru_query`
|
||||
answered `на 04.08.2026 ничего нет.` in 3.82s of audio at RMS 3865, `ru_fact`
|
||||
answered `отметила: water = выпил`, `ru_reminder` answered `хорошо, напомню.`
|
||||
at `intent=reminder`.
|
||||
**Passes in the browser too** (04-08-2026), and it needed no person. Headless
|
||||
Chrome takes a fake microphone, so the whole browser half runs unattended:
|
||||
|
||||
```sh
|
||||
chrome --headless=new --remote-debugging-port=9333 --remote-allow-origins='*' \
|
||||
--use-fake-device-for-media-stream --use-fake-ui-for-media-stream \
|
||||
--use-file-for-fake-audio-capture=cmd/mavsttd/testdata/ru_query.wav%noloop
|
||||
```
|
||||
|
||||
Then drive it over the debug protocol: click `#btn`, wait, click again, read
|
||||
`#status` and `#log`. That covers `getUserMedia`, `MediaRecorder`, the webm
|
||||
decode and the hand-written resample to 16k Int16. It logged
|
||||
`sending 188160 bytes`, which is 5.88s at 16k mono, and got the reply back.
|
||||
|
||||
**The button is on `/`, not `/dash`.** `handleVoice` serves it at the root
|
||||
(`main.go:332`). `/dash` is the presence and fact dashboard and carries no
|
||||
`#btn`. This step said `/dash` until 04-08-2026.
|
||||
|
||||
One defect fell out, filed as **533**. The reply logged as
|
||||
`на+04.08.2026+ничего+нет.` The header is escaped with `url.QueryEscape`,
|
||||
which writes a space as `+`, then decoded with `decodeURIComponent`, which
|
||||
leaves `+` alone. Transcript only, the audio is fine.
|
||||
3. Say `тихий режим`. Expect `тихий режим включён. буду реже напоминать.` **Passes.**
|
||||
4. Say `выключи тихий режим`. Expect `тихий режим выключен.` Negation must win. **Passes.**
|
||||
5. Say `в комнате тихо`. Quiet mode must NOT flip. Confirm on `/history` that no
|
||||
@@ -132,6 +222,29 @@ turns look misaligned when they are not.
|
||||
**First evidence, in text** (02-08-2026): nothing breaks, but answers wander
|
||||
and stitch unrelated topics. Asked whether he should move flats, she opened
|
||||
with the weather. That is 287, and it is a phrasing problem, not a loop problem.
|
||||
**The slowness now has a cause and a number** (04-08-2026). A spoken turn
|
||||
takes 32 to 34 seconds. One phrasing call is 30.0s of that. STT is 1.0s
|
||||
and routing is under 10ms. Both interactive calls decoded exactly 512 tokens,
|
||||
which is the phrasing cap. Both were truncated, to produce a reply of
|
||||
under 25 characters.
|
||||
The cause is `responseGrammar`, not the model. Its last rule is
|
||||
`ws ::= [ \t\n]*`, and `*` is unbounded, so the model emits `{` and then
|
||||
satisfies `ws` with whitespace until `max_tokens` stops it. Reproduced on a
|
||||
second server: at `repeat_penalty` 1.0 it runs to 512 and returns
|
||||
`finish_reason=length`, at 1.3 it stops at 24. Bounding the rule to
|
||||
`[ \t\n]{0,4}` gives a clean stop at 33 tokens three times out of three with
|
||||
no penalty at all.
|
||||
Only some callers are exposed. `internal/llm.Req` sends `repeat_penalty` and
|
||||
the replier sets it to 1.3, so that path is protected by accident. `chatReq`
|
||||
in the phraser sends no penalty, so `PhraseChat`, `PhraseQuery`,
|
||||
`PhraseNudge` and `PhraseReminder` all run at the default 1.0. Filed as
|
||||
**531**.
|
||||
Two guesses were wrong on the way and are recorded so nobody repeats them.
|
||||
It is not reasoning tokens: the probe returned `reasoning_content` of length
|
||||
0, and the grammar constrains output from the first token. It is not the
|
||||
`--cache-ram 512` limit either: that is MiB of prompt cache and the 512 that
|
||||
was hit is a token count.
|
||||
The wandering is a second thing and stays on 287.
|
||||
|
||||
**The wake path cannot be checked here, and that is now the decision rather
|
||||
than a gap.** `mavwaked` and `mavenclient` appear in no compose file and run as
|
||||
|
||||
@@ -94,7 +94,7 @@ func (t *NudgeTemplates) PhraseNudge(_ context.Context, c loop.Candidate) (deliv
|
||||
// template fits it uses the plain per-rule fallback.
|
||||
func (t *NudgeTemplates) Nudge(c loop.Candidate) (body, mood string) {
|
||||
rule := c.Rule.Name
|
||||
family := t.family(rule)
|
||||
family := t.pluralFamily(t.family(rule), c)
|
||||
set, ok := t.file.Rules[family]
|
||||
if !ok {
|
||||
return fallbackNudge(c), "neutral"
|
||||
@@ -155,6 +155,25 @@ func (t *NudgeTemplates) family(rule string) string {
|
||||
return "default"
|
||||
}
|
||||
|
||||
// pluralFamily swaps in the plural wording when {service} will hold a list.
|
||||
// Russian agrees the verb with the subject, so one set of templates cannot
|
||||
// serve both: "Сервис paperless не отвечает" and "Сервисы nginx, paperless не
|
||||
// отвечают" differ in the noun, the verb and the adjective. Filling a list into
|
||||
// the singular text is the kind of near-miss that reads as machine-written.
|
||||
//
|
||||
// Only service_down has a plural form today. A family with no "_many" set in
|
||||
// the file is returned unchanged, so adding one is a data change.
|
||||
func (t *NudgeTemplates) pluralFamily(family string, c loop.Candidate) string {
|
||||
if len(loop.DownServices(c.State)) < 2 {
|
||||
return family
|
||||
}
|
||||
many := family + "_many"
|
||||
if _, ok := t.file.Rules[many]; ok {
|
||||
return many
|
||||
}
|
||||
return family
|
||||
}
|
||||
|
||||
// placeholderRE — the {name} slots a template may use.
|
||||
var placeholderRE = regexp.MustCompile(`\{([a-z]+)\}`)
|
||||
|
||||
@@ -165,17 +184,25 @@ func nudgeValues(c loop.Candidate) map[string]string {
|
||||
vals := map[string]string{}
|
||||
rule := c.Rule.Name
|
||||
|
||||
// {service} — one fact per kuma monitor, keyed "service_down:<name>", so
|
||||
// the name lives in the key SUFFIX and there is no fact called plain
|
||||
// "service_down" to read. loop.DownServices is the same helper the rule
|
||||
// fired on, which is what stops the message naming a service that is up.
|
||||
// This used to read c.State.Fact(rule) — the pre-per-monitor aggregate —
|
||||
// and so never filled, leaving the one nameless variant as the only
|
||||
// fillable template every time (Vikunja #534).
|
||||
if down := loop.DownServices(c.State); len(down) > 0 {
|
||||
vals["service"] = strings.Join(down, ", ")
|
||||
}
|
||||
// {since} — only at hour scale. Below an hour the phrase would be minutes,
|
||||
// and none of the templates read well with "сорок минут".
|
||||
// and none of the templates read well with "сорок минут". service_down has
|
||||
// no {since} to offer: its facts are keyed by monitor, and the rule is
|
||||
// edge-triggered, so it fires on the transition rather than hours later.
|
||||
if d, ok := c.State.Since(rule); ok && d >= time.Hour {
|
||||
if s := ruSinceWords(d); s != "" {
|
||||
vals["since"] = s
|
||||
}
|
||||
}
|
||||
// {service} — the aggregate fact's key carries the service name.
|
||||
if f, ok := c.State.Fact(rule); ok && f.Key != "" && f.Key != rule {
|
||||
vals["service"] = f.Key
|
||||
}
|
||||
// {what} — the Russian suffix of "routine:таблетки" / "morning:утро".
|
||||
if i := strings.IndexByte(rule, ':'); i > 0 && i+1 < len(rule) {
|
||||
vals["what"] = rule[i+1:]
|
||||
|
||||
@@ -11,6 +11,102 @@ import (
|
||||
"github.com/kami/maven/internal/store"
|
||||
)
|
||||
|
||||
// downCand builds a service_down candidate the way a tick actually does it:
|
||||
// one fact per kuma monitor under the prefix, carrying the source and value
|
||||
// loop.DownServices checks. The old cand() shape wrote a single fact keyed
|
||||
// plain "service_down", which mavpoll stopped producing, and that is why the
|
||||
// tests passed through the whole of #534.
|
||||
func downCand(names ...string) loop.Candidate {
|
||||
now := time.Date(2026, 7, 31, 21, 40, 0, 0, time.UTC)
|
||||
st := loop.State{Now: now, Facts: map[string]store.Fact{}}
|
||||
for _, n := range names {
|
||||
key := loop.ServiceDownPrefix + n
|
||||
st.Facts[key] = store.Fact{
|
||||
Key: key, Ts: now.Add(-3 * time.Minute),
|
||||
Source: loop.ServiceDownSource, Value: `"down"`,
|
||||
}
|
||||
}
|
||||
return loop.Candidate{
|
||||
Rule: loop.Rule{Name: "service_down", Severity: loop.Sev4},
|
||||
Severity: loop.Sev4, State: st,
|
||||
}
|
||||
}
|
||||
|
||||
// The nudge he reads on telegram must name what broke. It is a sev4 that
|
||||
// reaches him away from the box, so "a service is down" costs him a trip to
|
||||
// kuma to learn anything at all.
|
||||
func TestNudgeNamesTheDownService(t *testing.T) {
|
||||
// Lowercased before matching: a name that opens the sentence is
|
||||
// capitalized by capitalizeFirst, which is wanted.
|
||||
nt := newTestTemplates(t, 5)
|
||||
for i := 0; i < 40; i++ {
|
||||
body, _ := nt.Nudge(downCand("paperless"))
|
||||
if !strings.Contains(strings.ToLower(body), "paperless") {
|
||||
t.Fatalf("body does not name the service: %q", body)
|
||||
}
|
||||
}
|
||||
// Two down: both named, in the key order the rule itself uses.
|
||||
for i := 0; i < 40; i++ {
|
||||
body, _ := nt.Nudge(downCand("nginx", "paperless"))
|
||||
low := strings.ToLower(body)
|
||||
if !strings.Contains(low, "nginx") || !strings.Contains(low, "paperless") {
|
||||
t.Fatalf("body drops a service: %q", body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Russian agrees the verb with the subject, so a list of services cannot go
|
||||
// into the singular sentence. One down takes the singular set, two or more
|
||||
// take service_down_many.
|
||||
func TestNudgeAgreesWithTheServiceCount(t *testing.T) {
|
||||
nt := newTestTemplates(t, 9)
|
||||
// "упал " keeps its trailing space: "упали" starts with "упал", and the
|
||||
// plural must not read as the singular by prefix.
|
||||
singular := []string{"не отвечает", "недоступен", "лежит", "упал "}
|
||||
plural := []string{"не отвечают", "недоступны", "лежат", "упали"}
|
||||
|
||||
for i := 0; i < 60; i++ {
|
||||
body, _ := nt.Nudge(downCand("paperless"))
|
||||
if !containsAny(body, singular) {
|
||||
t.Fatalf("one down, no singular verb: %q", body)
|
||||
}
|
||||
if containsAny(body, plural) {
|
||||
t.Fatalf("one down, plural wording: %q", body)
|
||||
}
|
||||
}
|
||||
for i := 0; i < 60; i++ {
|
||||
body, _ := nt.Nudge(downCand("nginx", "paperless"))
|
||||
if !containsAny(body, plural) {
|
||||
t.Fatalf("two down, no plural verb: %q", body)
|
||||
}
|
||||
if containsAny(body, singular) {
|
||||
t.Fatalf("two down, singular wording: %q", body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func containsAny(s string, subs []string) bool {
|
||||
for _, sub := range subs {
|
||||
if strings.Contains(s, sub) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Nothing down means no template fits, and the fallback answers rather than
|
||||
// the picker inventing a name.
|
||||
func TestNudgeServiceDownWithoutFacts(t *testing.T) {
|
||||
nt := newTestTemplates(t, 5)
|
||||
body, mood := nt.Nudge(downCand())
|
||||
if body != "Сервис не отвечает." {
|
||||
t.Fatalf("fallback body %q", body)
|
||||
}
|
||||
if mood != "neutral" {
|
||||
t.Fatalf("mood %q", mood)
|
||||
}
|
||||
}
|
||||
|
||||
// cand builds a candidate the way a tick would.
|
||||
func cand(rule string, sinceMin int, factKey string) loop.Candidate {
|
||||
now := time.Date(2026, 7, 31, 21, 40, 0, 0, time.UTC)
|
||||
@@ -36,7 +132,7 @@ func newTestTemplates(t *testing.T, seed int64) *NudgeTemplates {
|
||||
|
||||
func TestNudgeTemplatesLoad(t *testing.T) {
|
||||
nt := newTestTemplates(t, 1)
|
||||
for _, rule := range []string{"water", "meal", "break", "service_down", "netdata_critical", "routine", "morning", "default"} {
|
||||
for _, rule := range []string{"water", "meal", "break", "service_down", "service_down_many", "netdata_critical", "routine", "morning", "default"} {
|
||||
set, ok := nt.file.Rules[rule]
|
||||
if !ok {
|
||||
t.Errorf("no templates for %q", rule)
|
||||
@@ -46,8 +142,12 @@ func TestNudgeTemplatesLoad(t *testing.T) {
|
||||
t.Errorf("%s: only %d variants", rule, len(set.Variants))
|
||||
}
|
||||
// Every rule needs one variant that needs no value, or a candidate
|
||||
// without context has nothing to say. routine and morning are exempt:
|
||||
// they always carry a name and must always say it.
|
||||
// without context has nothing to say. routine, morning and
|
||||
// service_down are exempt: they always carry a name and must always
|
||||
// say it. service_down's predicate cannot fire without a down fact,
|
||||
// so loop.DownServices always has something to fill {service} with,
|
||||
// and the nameless variant it used to carry was the bug (#534) —
|
||||
// {service} never filled, so that variant was the only fillable one.
|
||||
plain := 0
|
||||
seen := map[string]bool{}
|
||||
for _, v := range set.Variants {
|
||||
@@ -59,7 +159,7 @@ func TestNudgeTemplatesLoad(t *testing.T) {
|
||||
}
|
||||
seen[v] = true
|
||||
}
|
||||
if plain == 0 && rule != "routine" && rule != "morning" {
|
||||
if plain == 0 && rule != "routine" && rule != "morning" && !strings.HasPrefix(rule, "service_down") {
|
||||
t.Errorf("%s: every variant needs a placeholder value", rule)
|
||||
}
|
||||
}
|
||||
@@ -102,8 +202,8 @@ func TestNudgeNoLeftoverPlaceholders(t *testing.T) {
|
||||
cand("water", 0, ""), // no duration
|
||||
cand("water", 30, ""), // under an hour
|
||||
cand("water", 200, ""), // hours
|
||||
cand("service_down", 3, "vaultwarden"),
|
||||
cand("service_down", 3, ""), // no service name
|
||||
downCand("vaultwarden"),
|
||||
downCand(), // nothing down: the fallback answers
|
||||
cand("routine:таблетки", 0, ""),
|
||||
cand("morning:утро", 0, ""),
|
||||
cand("unknown_rule", 0, ""),
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
"Hand-written Russian nudges. Edit the wording here, no Go changes needed.",
|
||||
"Rules: she is feminine about herself, he is a man addressed as ты. Never вы/вас/ваш, never plural imperatives (выпейте), never он/его about him.",
|
||||
"One short sentence. No questions, no emoji, no pet names, no emotional support.",
|
||||
"Placeholders: {since} how long it has been (only used when it is at least an hour), {service} the service name, {what} the routine name. A variant whose placeholder has no value is skipped, so every rule needs at least one variant with no placeholder. The exception is routine and morning: those only exist for rules like routine:таблетки that always carry a name, and a routine nudge that drops the name is useless.",
|
||||
"Placeholders: {since} how long it has been (only used when it is at least an hour), {service} the service name, {what} the routine name. A variant whose placeholder has no value is skipped, so every rule needs at least one variant with no placeholder. The exception is routine, morning and service_down: those only exist for rules that always carry a name, and one that drops the name is useless.",
|
||||
"A rule may carry a second set named <rule>_many, used when {service} holds more than one name. Russian agrees the verb with the subject, so the plural needs its own wording rather than a list dropped into the singular sentence. Only service_down has one.",
|
||||
"mood must be one of: neutral, happy, thinking, tired, confused."
|
||||
],
|
||||
"rules": {
|
||||
@@ -62,13 +63,24 @@
|
||||
"{service} не отвечает, сервис нужно поднимать.",
|
||||
"Сервис {service} недоступен.",
|
||||
"Проверь {service}: сервис не отвечает.",
|
||||
"Сервис перестал отвечать.",
|
||||
"Сервис {service} лежит, нужно смотреть.",
|
||||
"{service} не отвечает уже {since}.",
|
||||
"Мониторинг сообщает: {service} лежит.",
|
||||
"Сервис {service} не отвечает, посмотри логи."
|
||||
]
|
||||
},
|
||||
"service_down_many": {
|
||||
"mood": "neutral",
|
||||
"variants": [
|
||||
"Сервисы {service} не отвечают.",
|
||||
"{service} упали — сервисы не отвечают.",
|
||||
"{service} не отвечают, сервисы нужно поднимать.",
|
||||
"Сервисы {service} недоступны.",
|
||||
"Проверь {service}: сервисы не отвечают.",
|
||||
"Сервисы {service} лежат, нужно смотреть.",
|
||||
"Мониторинг сообщает: {service} лежат.",
|
||||
"Сервисы {service} не отвечают, посмотри логи."
|
||||
]
|
||||
},
|
||||
"netdata_critical": {
|
||||
"mood": "neutral",
|
||||
"variants": [
|
||||
|
||||
Reference in New Issue
Block a user