From 2b8d0f74fa57c178c67d2eaa359a7d6ba71d26ba Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 2 Aug 2026 14:24:01 +0400 Subject: [PATCH] docs: session 1 and 2 results, and the classifier baseline was wrong (V-459) Ran sessions 1 and 2 on the live box. Session 1 steps 1 and 3-6 pass. Steps 2 and 7-9 need a person at the box. POST /api/chat is drivable with form encoding and a cookie jar, so the text half needs no browser. Session 2 confirms the deploy matches the bench at 72.7% full accuracy, and contradicts two recorded numbers. The classifier scores 68.8% at p50 16.6us, not 36.8% at 31ms. Router latency measured under contention again. Also: 319 item 2 point 2 closes on the recall margin sweep, CheckFeminine has a false positive on second-person masculine verbs, and the wake path cannot be checked because mavwaked and mavenclient are deployed nowhere. --- CLAUDE.md | 12 ++++--- docs/qa.md | 96 ++++++++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 90 insertions(+), 18 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index bce03d3..fb22297 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -127,11 +127,13 @@ on in deploy** — this section used to say it was wired `nil`, which stopped be Cascade order: `stage0.go` exact-match fast-path → LLM router (when non-nil) → classifier fallback. Any LLM error falls through to the classifier so a turn never breaks on the model. -Measured on the 77-case RU fixture (`docs/evals/2026-07-31-model-bakeoff.md`): the classifier scores -36.8% full accuracy at p50 31ms; Qwen3-1.7B scores 67.5% intent-only / 72.7% through the -cascade at p50 ≈825ms. Accuracy roughly doubled, latency is ~27× worse, and that trade was -accepted deliberately. **The ≈2.7s figure that stood here until 2026-08-02 was contention, -not the model.** See `docs/evals/2026-07-31-routing.md` line 61, which measures the LLM router at +Measured on the 77-case RU fixture. **Re-measured 2026-08-02: the classifier scores 68.8% +full accuracy at p50 16.6µs**, not the 36.8% at p50 31ms that stood here from +`docs/evals/2026-07-31-model-bakeoff.md`. That older figure predates the stage 0 rules and the +seed additions, both of which now score inside the classifier baseline. Qwen3-1.7B scores +77.9% intent-only / 72.7% through the cascade. So the router buys about 4 points of accuracy, +not a doubling, and the trade is worth re-arguing rather than assuming. **The ≈2.7s figure +that stood here until 2026-08-02 was contention, not the model.** See `docs/evals/2026-07-31-routing.md` line 61, which measures the LLM router at p50 825ms / p95 1.2s / max 3.0s and the full cascade at p50 0.80-1.04s. Do not plan latency work off the bakeoff table. `Confidence: 1.0` used to be hardcoded in `llmrouter.go`, so the LLM path could never ask for clarification (6/6 refusal cases missed on the fixture) — Vikunja diff --git a/docs/qa.md b/docs/qa.md index f187fa7..f4cbfd4 100644 --- a/docs/qa.md +++ b/docs/qa.md @@ -1,6 +1,6 @@ # QA plan: checking Maven properly -*Last verified: 2026-08-02 @ 5c05163. Living doc: correct it in place, do not append.* +*Last verified: 2026-08-02 @ af9d213. Living doc: correct it in place, do not append.* Written 2026-08-01, after the 35-PR stack landed and the box came back up. Refreshed 2026-08-02 against the live list, after PRs #85-#90. @@ -43,19 +43,37 @@ it works. Do this first. Closes or advances: **44** (conversation), **45** (text chat), **287** (voice session quality), **321** steps 3-5 (quiet mode), **288** (STT fixtures). +Steps 1 and 3-6 were run on 02-08-2026 and pass. Steps 2 and 7-9 still need a +person at the box, because they need a microphone or a nudge to arrive. + +Steps 1 and 3-6 do not need a browser. `POST /api/chat` takes a form-encoded +`text=` field and a cookie jar, and answers with the rendered `/chat` page: + +```sh +curl -s --noproxy '*' -c jar -b jar -L -X POST \ + http://127.0.0.1:9201/api/chat --data-urlencode 'text=привет' +``` + +Parse the whole page, not the last text node. The page carries nav and footer +text. A naive tail of the Cyrillic nodes returns the wrong string, which makes +turns look misaligned when they are not. + 1. Open `http://127.0.0.1:9201/chat` and hold a short conversation in Russian. Watch for three things: she answers in feminine forms (`рада`, `поняла`), she says `ты` and never `вы`, and no pet names appear. + **Passes** (02-08-2026, five turns): `я рада`, `поняла`, `помогла`, + `проверила`, `записала`, `грустна`, `ты` throughout, no pet names. 2. Press push-to-talk on `/dash`. Say `привет`. Confirm a spoken reply comes back. This is the only check that covers mic to STT to core to TTS to speaker as one path. It is also the path the eleven-day outage most likely broke. -3. Say `тихий режим`. Expect `тихий режим включён. буду реже напоминать.` -4. Say `выключи тихий режим`. Expect `тихий режим выключен.` Negation must win. +3. Say `тихий режим`. Expect `тихий режим включён. буду реже напоминать.` **Passes.** +4. Say `выключи тихий режим`. Expect `тихий режим выключен.` Negation must win. **Passes.** 5. Say `в комнате тихо`. Quiet mode must NOT flip. Confirm on `/history` that no - `quiet_hours` fact was written. + `quiet_hours` fact was written. **Passes**: no row written. She answers `пока + не умею отвечать на этот вопрос.`, so it lands on `IntentSystem` with no arm. 6. Say `включи режим тишины`, then `сделай потише`. Both must flip quiet mode - on. These are the noun form and the comparative, added 01-08-2026. + on. These are the noun form and the comparative, added 01-08-2026. **Both pass.** 7. Wait for a nudge, then say `потом` within twenty minutes. Expect `хорошо, вернусь к этому позже.` and the nudge row on `/notifications` reading `snoozed`. Say `потом` again with nothing pending: it must route as an @@ -66,6 +84,15 @@ session quality), **321** steps 3-5 (quiet mode), **288** (STT fixtures). `готово`. Expect `отлично, отметила.` and the same outcome. 9. Note anything where she is slow, cuts off, or talks over herself. That is 287's whole content and it has no written acceptance criteria yet. + **First evidence, in text** (02-08-2026): nothing breaks, but answers wander + and stitch unrelated topics. Asked whether he should move flats, she opened + with the weather. That is 287, and it is a phrasing problem, not a loop problem. + +**The wake path cannot be checked as deployed.** `mavwaked` and `mavenclient` +appear in no compose file and run as no host process. Step 2 covers only +push-to-talk, from `/dash` through mavsttd and mavttsd. Wake word and VAD +are untested by construction. Decide whether they belong in compose or on a +client machine, and say which in the deploy docs. **319's single-token bug is fixed** (01-08-2026). Single-word Russian utterances no longer come back as `не совсем поняла — можешь переформулировать?`. `привет` and `поужинал` @@ -104,13 +131,43 @@ make eval-recall A large miss against 72.7% means the deploy differs from the bench harness. +**Run on 02-08-2026 @ af9d213. The deploy matches the bench.** `eval-models` +scored 56 of 77: 72.7% full, 77.9% intent-only, 2 false clarifies and 1 missed. +That is the recorded figure to the decimal, and calendar sat at 2 of 2, so the +stage 0 agenda rules hold. `eval-phrasing` scored 21 of 27 on the talk fixture +against a recorded 20, and the 15 nudge templates passed every check. + +Two numbers in this repo were wrong, and both flattered the resident model. + +- **The classifier is not 36.8% and not 31ms.** `make eval-router` reports + `classifier+onnx: 53/77 (68.8% full)` at p50 16.6µs. The figure repeated here + and in `CLAUDE.md` predates the stage 0 rules and the seed additions. Both now + score inside that baseline. The accuracy gap the router buys is + roughly 4 points, not 36. Re-argue the trade on the real numbers. +- **Router latency was measured under contention again.** p50 1.126s, p95 1.58s, + max 3.24s, against a recorded p50 825ms. The resident model was serving the + daemon on the same iGPU throughout. Do not record this as a regression, and do + not record it as a measurement either. Stop the stack before timing the router. + +`classifier+hash` scores 19.5%, which is the no-ONNX degraded path and is not the +failure floor the deploy uses. Do not quote it as the classifier baseline. + Then three things to decide while the numbers are in front of you: -- **319's gate recalibration is done.** 359 gave the LLM path a real confidence - signal. `thinSingleToken` was narrowed on 01-08-2026, and agenda questions moved - to stage 0. Missed clarify sits at 1 of 6 and false clarifies at 2. What is left - on 319 is item 2 point 2: re-derive `QueryMinMargin`, or show the distribution - the hand-picked `0.008` came from. +- **319 is done.** 359 gave the LLM path a real confidence signal. + `thinSingleToken` was narrowed on 01-08-2026, and agenda questions moved to + stage 0. Missed clarify sits at 1 of 6 and false clarifies at 2. Item 2 point 2 + closed on 02-08-2026: the `make eval-recall` margin sweep is the distribution + that was asked for, and `0.008` sits at the knee. + + | delta | answered | false recall | + |---|---|---| + | 0.005 | 18/27 | 2/5 | + | **0.008** | **18/27** | **1/5** | + | 0.010 | 16/27 | 1/5 | + + It removes four of five false recalls at no cost in answers, and the next step + costs two answers for nothing. The hand-picked value survives on evidence. - **278's real ask** is making the eval lab routine rather than building it. It is built. Decide whether it runs on a timer, on every merge, or on demand, and the task can close. @@ -124,6 +181,18 @@ needs a `StartupTimeout` field on `Config` rather than a test-only hack. While y are on the box, time a cold 1.7B load off spinning disk. If it runs near 60s, the default is too tight and the field earns itself twice. +Warm, it is nowhere near. A second llama-server answered `/health` 1.8s after +launch at `n_ctx 4096` on 02-08-2026. That is page cache, so it does not settle +the question. A cold read needs a cache drop, which needs root. + +**`CheckFeminine` has a false positive.** On 02-08-2026 it failed +`query-notes-do-not-answer` for `ты заплатил`, calling it masculine +self-reference. Masculine second person is correct, because the owner is male. +The check matches a masculine +past-tense verb before `за` without confirming the subject is `я`. Fix it in +`internal/phraser/eval/checks.go` before trusting a phrasing score to the case. +The real talk-fixture score on that run is 22 of 27, not 21. + Item 4 of **320** needs a permission I do not have. Kill the `llama-server` pid under `maven-mavend-1`, post a turn, and confirm it still completes through the classifier. Either grant it or run it yourself. It is the only @@ -231,8 +300,9 @@ where they land, so the board stops reading as 50 things Maven owes. 1. Session 1. If the voice loop is broken, nothing else matters. 2. The `-require-stepup` and Kuma decisions. Five minutes, and it unblocks **16**. -3. Session 2. The numbers tell you whether the router is worth its ~27× latency. - The 90× figure that stood here came from a 2.7s p50 that was contention, not - the model. Real p50 is 825ms against the classifier's 31ms. +3. Session 2. **Run on 02-08-2026.** The numbers came back worse for the router + than the docs claimed. The classifier is 68.8%, not 36.8%, and 16.6µs, not + 31ms. The router buys about 4 points of accuracy for four orders of magnitude + of latency. Whether that still earns its place is now an open question. 4. Housekeeping. Cheap, and it makes the remaining backlog honest. 5. Session 3, split whichever way suits you.