docs: session 1 and 2 results, and the classifier baseline was wrong (V-459)

Ran sessions 1 and 2 on the live box.

Session 1 steps 1 and 3-6 pass. Steps 2 and 7-9 need a person at the box.
POST /api/chat is drivable with form encoding and a cookie jar, so the text
half needs no browser.

Session 2 confirms the deploy matches the bench at 72.7% full accuracy, and
contradicts two recorded numbers. The classifier scores 68.8% at p50 16.6us,
not 36.8% at 31ms. Router latency measured under contention again.

Also: 319 item 2 point 2 closes on the recall margin sweep, CheckFeminine has
a false positive on second-person masculine verbs, and the wake path cannot be
checked because mavwaked and mavenclient are deployed nowhere.
This commit is contained in:
2026-08-02 14:24:01 +04:00
parent af9d2133dc
commit 2b8d0f74fa
2 changed files with 90 additions and 18 deletions
+83 -13
View File
@@ -1,6 +1,6 @@
# QA plan: checking Maven properly
*Last verified: 2026-08-02 @ 5c05163. Living doc: correct it in place, do not append.*
*Last verified: 2026-08-02 @ af9d213. Living doc: correct it in place, do not append.*
Written 2026-08-01, after the 35-PR stack landed and the box came back up.
Refreshed 2026-08-02 against the live list, after PRs #85-#90.
@@ -43,19 +43,37 @@ it works. Do this first.
Closes or advances: **44** (conversation), **45** (text chat), **287** (voice
session quality), **321** steps 3-5 (quiet mode), **288** (STT fixtures).
Steps 1 and 3-6 were run on 02-08-2026 and pass. Steps 2 and 7-9 still need a
person at the box, because they need a microphone or a nudge to arrive.
Steps 1 and 3-6 do not need a browser. `POST /api/chat` takes a form-encoded
`text=` field and a cookie jar, and answers with the rendered `/chat` page:
```sh
curl -s --noproxy '*' -c jar -b jar -L -X POST \
http://127.0.0.1:9201/api/chat --data-urlencode 'text=привет'
```
Parse the whole page, not the last text node. The page carries nav and footer
text. A naive tail of the Cyrillic nodes returns the wrong string, which makes
turns look misaligned when they are not.
1. Open `http://127.0.0.1:9201/chat` and hold a short conversation in Russian.
Watch for three things: she answers in feminine forms (`рада`, `поняла`), she
says `ты` and never `вы`, and no pet names appear.
**Passes** (02-08-2026, five turns): `я рада`, `поняла`, `помогла`,
`проверила`, `записала`, `грустна`, `ты` throughout, no pet names.
2. Press push-to-talk on `/dash`. Say `привет`. Confirm a spoken reply comes
back. This is the only check that covers mic to STT to core to TTS to
speaker as one path. It is also the path the eleven-day outage most likely
broke.
3. Say `тихий режим`. Expect `тихий режим включён. буду реже напоминать.`
4. Say `выключи тихий режим`. Expect `тихий режим выключен.` Negation must win.
3. Say `тихий режим`. Expect `тихий режим включён. буду реже напоминать.` **Passes.**
4. Say `выключи тихий режим`. Expect `тихий режим выключен.` Negation must win. **Passes.**
5. Say `в комнате тихо`. Quiet mode must NOT flip. Confirm on `/history` that no
`quiet_hours` fact was written.
`quiet_hours` fact was written. **Passes**: no row written. She answers `пока
не умею отвечать на этот вопрос.`, so it lands on `IntentSystem` with no arm.
6. Say `включи режим тишины`, then `сделай потише`. Both must flip quiet mode
on. These are the noun form and the comparative, added 01-08-2026.
on. These are the noun form and the comparative, added 01-08-2026. **Both pass.**
7. Wait for a nudge, then say `потом` within twenty minutes. Expect `хорошо,
вернусь к этому позже.` and the nudge row on `/notifications` reading
`snoozed`. Say `потом` again with nothing pending: it must route as an
@@ -66,6 +84,15 @@ session quality), **321** steps 3-5 (quiet mode), **288** (STT fixtures).
`готово`. Expect `отлично, отметила.` and the same outcome.
9. Note anything where she is slow, cuts off, or talks over herself. That is
287's whole content and it has no written acceptance criteria yet.
**First evidence, in text** (02-08-2026): nothing breaks, but answers wander
and stitch unrelated topics. Asked whether he should move flats, she opened
with the weather. That is 287, and it is a phrasing problem, not a loop problem.
**The wake path cannot be checked as deployed.** `mavwaked` and `mavenclient`
appear in no compose file and run as no host process. Step 2 covers only
push-to-talk, from `/dash` through mavsttd and mavttsd. Wake word and VAD
are untested by construction. Decide whether they belong in compose or on a
client machine, and say which in the deploy docs.
**319's single-token bug is fixed** (01-08-2026). Single-word Russian utterances no longer come
back as `не совсем поняла — можешь переформулировать?`. `привет` and `поужинал`
@@ -104,13 +131,43 @@ make eval-recall
A large miss against 72.7% means the deploy differs from the bench harness.
**Run on 02-08-2026 @ af9d213. The deploy matches the bench.** `eval-models`
scored 56 of 77: 72.7% full, 77.9% intent-only, 2 false clarifies and 1 missed.
That is the recorded figure to the decimal, and calendar sat at 2 of 2, so the
stage 0 agenda rules hold. `eval-phrasing` scored 21 of 27 on the talk fixture
against a recorded 20, and the 15 nudge templates passed every check.
Two numbers in this repo were wrong, and both flattered the resident model.
- **The classifier is not 36.8% and not 31ms.** `make eval-router` reports
`classifier+onnx: 53/77 (68.8% full)` at p50 16.6µs. The figure repeated here
and in `CLAUDE.md` predates the stage 0 rules and the seed additions. Both now
score inside that baseline. The accuracy gap the router buys is
roughly 4 points, not 36. Re-argue the trade on the real numbers.
- **Router latency was measured under contention again.** p50 1.126s, p95 1.58s,
max 3.24s, against a recorded p50 825ms. The resident model was serving the
daemon on the same iGPU throughout. Do not record this as a regression, and do
not record it as a measurement either. Stop the stack before timing the router.
`classifier+hash` scores 19.5%, which is the no-ONNX degraded path and is not the
failure floor the deploy uses. Do not quote it as the classifier baseline.
Then three things to decide while the numbers are in front of you:
- **319's gate recalibration is done.** 359 gave the LLM path a real confidence
signal. `thinSingleToken` was narrowed on 01-08-2026, and agenda questions moved
to stage 0. Missed clarify sits at 1 of 6 and false clarifies at 2. What is left
on 319 is item 2 point 2: re-derive `QueryMinMargin`, or show the distribution
the hand-picked `0.008` came from.
- **319 is done.** 359 gave the LLM path a real confidence signal.
`thinSingleToken` was narrowed on 01-08-2026, and agenda questions moved to
stage 0. Missed clarify sits at 1 of 6 and false clarifies at 2. Item 2 point 2
closed on 02-08-2026: the `make eval-recall` margin sweep is the distribution
that was asked for, and `0.008` sits at the knee.
| delta | answered | false recall |
|---|---|---|
| 0.005 | 18/27 | 2/5 |
| **0.008** | **18/27** | **1/5** |
| 0.010 | 16/27 | 1/5 |
It removes four of five false recalls at no cost in answers, and the next step
costs two answers for nothing. The hand-picked value survives on evidence.
- **278's real ask** is making the eval lab routine rather than building it. It
is built. Decide whether it runs on a timer, on every merge, or on demand, and
the task can close.
@@ -124,6 +181,18 @@ needs a `StartupTimeout` field on `Config` rather than a test-only hack. While y
are on the box, time a cold 1.7B load off spinning disk. If it runs near 60s, the
default is too tight and the field earns itself twice.
Warm, it is nowhere near. A second llama-server answered `/health` 1.8s after
launch at `n_ctx 4096` on 02-08-2026. That is page cache, so it does not settle
the question. A cold read needs a cache drop, which needs root.
**`CheckFeminine` has a false positive.** On 02-08-2026 it failed
`query-notes-do-not-answer` for `ты заплатил`, calling it masculine
self-reference. Masculine second person is correct, because the owner is male.
The check matches a masculine
past-tense verb before `за` without confirming the subject is `я`. Fix it in
`internal/phraser/eval/checks.go` before trusting a phrasing score to the case.
The real talk-fixture score on that run is 22 of 27, not 21.
Item 4 of **320** needs a permission I do not have. Kill the `llama-server`
pid under `maven-mavend-1`, post a turn, and confirm it still completes
through the classifier. Either grant it or run it yourself. It is the only
@@ -231,8 +300,9 @@ where they land, so the board stops reading as 50 things Maven owes.
1. Session 1. If the voice loop is broken, nothing else matters.
2. The `-require-stepup` and Kuma decisions. Five minutes, and it unblocks **16**.
3. Session 2. The numbers tell you whether the router is worth its ~27× latency.
The 90× figure that stood here came from a 2.7s p50 that was contention, not
the model. Real p50 is 825ms against the classifier's 31ms.
3. Session 2. **Run on 02-08-2026.** The numbers came back worse for the router
than the docs claimed. The classifier is 68.8%, not 36.8%, and 16.6µs, not
31ms. The router buys about 4 points of accuracy for four orders of magnitude
of latency. Whether that still earns its place is now an open question.
4. Housekeeping. Cheap, and it makes the remaining backlog honest.
5. Session 3, split whichever way suits you.