claude
2b8d0f74fa
Ran sessions 1 and 2 on the live box. Session 1 steps 1 and 3-6 pass. Steps 2 and 7-9 need a person at the box. POST /api/chat is drivable with form encoding and a cookie jar, so the text half needs no browser. Session 2 confirms the deploy matches the bench at 72.7% full accuracy, and contradicts two recorded numbers. The classifier scores 68.8% at p50 16.6us, not 36.8% at 31ms. Router latency measured under contention again. Also: 319 item 2 point 2 closes on the recall margin sweep, CheckFeminine has a false positive on second-person masculine verbs, and the wake path cannot be checked because mavwaked and mavenclient are deployed nowhere.
309 lines
15 KiB
Markdown
309 lines
15 KiB
Markdown
# QA plan: checking Maven properly
|
|
|
|
*Last verified: 2026-08-02 @ af9d213. Living doc: correct it in place, do not append.*
|
|
|
|
Written 2026-08-01, after the 35-PR stack landed and the box came back up.
|
|
Refreshed 2026-08-02 against the live list, after PRs #85-#90.
|
|
|
|
42 of the 50 open Vikunja tasks are `QA:` tasks. They are verification work, not
|
|
build work. Most sat unverifiable while Maven was down for 11 days. That
|
|
blocker is gone.
|
|
|
|
The plan as written on 2026-08-01 named 40 task numbers. Ten open `QA:` tasks were
|
|
missing and two of the named ones had closed. Every open task now appears below,
|
|
the eight non-QA ones in the last two sections.
|
|
|
|
This plan orders them by what unblocks what. Do sessions 1 and 2 first. Almost everything
|
|
downstream assumes the voice loop works, and nobody has confirmed that since
|
|
the redeploy.
|
|
|
|
---
|
|
|
|
## Before you start
|
|
|
|
Two things bite anyone running these checks on homesrv.
|
|
|
|
**curl needs `--noproxy '*'`.** The shell exports `http_proxy=http://127.0.0.1:18080`.
|
|
Without the flag, every local check returns 503 from the proxy and looks like a
|
|
dead service. This cost me a false regression report today.
|
|
|
|
**The database is not readable with sqlite3.** Four older QA steps say
|
|
`docker compose exec mavend sqlite3 /data/maven.db "select ..."`. That cannot
|
|
work: the container has no `sqlite3` binary, and the store is AES-256-GCM at
|
|
rest with a tmpfs working copy. Read state through mavweb instead, at
|
|
`/history`, `/trace`, `/routines` and `/dash`.
|
|
|
|
---
|
|
|
|
## Session 1: the voice loop (half a day)
|
|
|
|
Nothing here has been confirmed since the redeploy, and everything else assumes
|
|
it works. Do this first.
|
|
|
|
Closes or advances: **44** (conversation), **45** (text chat), **287** (voice
|
|
session quality), **321** steps 3-5 (quiet mode), **288** (STT fixtures).
|
|
|
|
Steps 1 and 3-6 were run on 02-08-2026 and pass. Steps 2 and 7-9 still need a
|
|
person at the box, because they need a microphone or a nudge to arrive.
|
|
|
|
Steps 1 and 3-6 do not need a browser. `POST /api/chat` takes a form-encoded
|
|
`text=` field and a cookie jar, and answers with the rendered `/chat` page:
|
|
|
|
```sh
|
|
curl -s --noproxy '*' -c jar -b jar -L -X POST \
|
|
http://127.0.0.1:9201/api/chat --data-urlencode 'text=привет'
|
|
```
|
|
|
|
Parse the whole page, not the last text node. The page carries nav and footer
|
|
text. A naive tail of the Cyrillic nodes returns the wrong string, which makes
|
|
turns look misaligned when they are not.
|
|
|
|
1. Open `http://127.0.0.1:9201/chat` and hold a short conversation in Russian.
|
|
Watch for three things: she answers in feminine forms (`рада`, `поняла`), she
|
|
says `ты` and never `вы`, and no pet names appear.
|
|
**Passes** (02-08-2026, five turns): `я рада`, `поняла`, `помогла`,
|
|
`проверила`, `записала`, `грустна`, `ты` throughout, no pet names.
|
|
2. Press push-to-talk on `/dash`. Say `привет`. Confirm a spoken reply comes
|
|
back. This is the only check that covers mic to STT to core to TTS to
|
|
speaker as one path. It is also the path the eleven-day outage most likely
|
|
broke.
|
|
3. Say `тихий режим`. Expect `тихий режим включён. буду реже напоминать.` **Passes.**
|
|
4. Say `выключи тихий режим`. Expect `тихий режим выключен.` Negation must win. **Passes.**
|
|
5. Say `в комнате тихо`. Quiet mode must NOT flip. Confirm on `/history` that no
|
|
`quiet_hours` fact was written. **Passes**: no row written. She answers `пока
|
|
не умею отвечать на этот вопрос.`, so it lands on `IntentSystem` with no arm.
|
|
6. Say `включи режим тишины`, then `сделай потише`. Both must flip quiet mode
|
|
on. These are the noun form and the comparative, added 01-08-2026. **Both pass.**
|
|
7. Wait for a nudge, then say `потом` within twenty minutes. Expect `хорошо,
|
|
вернусь к этому позже.` and the nudge row on `/notifications` reading
|
|
`snoozed`. Say `потом` again with nothing pending: it must route as an
|
|
ordinary utterance, not be swallowed.
|
|
8. Wait for the water nudge, then say `выпил воды`. Expect the ordinary fact
|
|
reply and nothing extra. She must not congratulate you. Check
|
|
`/notifications`: the row reads `acted`. Then trigger another nudge and say
|
|
`готово`. Expect `отлично, отметила.` and the same outcome.
|
|
9. Note anything where she is slow, cuts off, or talks over herself. That is
|
|
287's whole content and it has no written acceptance criteria yet.
|
|
**First evidence, in text** (02-08-2026): nothing breaks, but answers wander
|
|
and stitch unrelated topics. Asked whether he should move flats, she opened
|
|
with the weather. That is 287, and it is a phrasing problem, not a loop problem.
|
|
|
|
**The wake path cannot be checked as deployed.** `mavwaked` and `mavenclient`
|
|
appear in no compose file and run as no host process. Step 2 covers only
|
|
push-to-talk, from `/dash` through mavsttd and mavttsd. Wake word and VAD
|
|
are untested by construction. Decide whether they belong in compose or on a
|
|
client machine, and say which in the deploy docs.
|
|
|
|
**319's single-token bug is fixed** (01-08-2026). Single-word Russian utterances no longer come
|
|
back as `не совсем поняла — можешь переформулировать?`. `привет` and `поужинал`
|
|
both pass now: `thinSingleToken` spares social singles and any token carrying a
|
|
verb ending, and only thins a bare nominal like `вода`. A one-word utterance that
|
|
still gets clarified in this session is a new case for the lexicon, not the old bug.
|
|
|
|
---
|
|
|
|
## Session 2: measurement (half a day, mostly waiting)
|
|
|
|
Closes or advances: **320** items 2-4, **278** (make the eval lab routine).
|
|
Also **248** (memory evaluation), **319** (the margin gate) and **323** (the
|
|
startup timeout arm).
|
|
|
|
The resident llama-server cannot be reached by the eval harness. It binds
|
|
`--host 127.0.0.1 --port 0` inside the container, so the port is kernel-assigned
|
|
and never published. Start a second one on a fixed port instead:
|
|
|
|
```sh
|
|
llama-server -m /mnt/hdd1/llms/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf \
|
|
--host 127.0.0.1 --port 18100 -c 4096 -ngl 99 --no-webui
|
|
```
|
|
|
|
`-c 4096` matters. The recorded numbers were measured at that context size, and
|
|
a mismatch invalidates the comparison.
|
|
|
|
Then:
|
|
|
|
```sh
|
|
make eval-models MAVEN_LLM_URL=http://127.0.0.1:18100 # want ~72.7% cascade
|
|
make eval-router # classifier baseline
|
|
MAVEN_LLM_URL=http://127.0.0.1:18100 make eval-phrasing # persona checks, slow
|
|
make eval-recall
|
|
```
|
|
|
|
A large miss against 72.7% means the deploy differs from the bench harness.
|
|
|
|
**Run on 02-08-2026 @ af9d213. The deploy matches the bench.** `eval-models`
|
|
scored 56 of 77: 72.7% full, 77.9% intent-only, 2 false clarifies and 1 missed.
|
|
That is the recorded figure to the decimal, and calendar sat at 2 of 2, so the
|
|
stage 0 agenda rules hold. `eval-phrasing` scored 21 of 27 on the talk fixture
|
|
against a recorded 20, and the 15 nudge templates passed every check.
|
|
|
|
Two numbers in this repo were wrong, and both flattered the resident model.
|
|
|
|
- **The classifier is not 36.8% and not 31ms.** `make eval-router` reports
|
|
`classifier+onnx: 53/77 (68.8% full)` at p50 16.6µs. The figure repeated here
|
|
and in `CLAUDE.md` predates the stage 0 rules and the seed additions. Both now
|
|
score inside that baseline. The accuracy gap the router buys is
|
|
roughly 4 points, not 36. Re-argue the trade on the real numbers.
|
|
- **Router latency was measured under contention again.** p50 1.126s, p95 1.58s,
|
|
max 3.24s, against a recorded p50 825ms. The resident model was serving the
|
|
daemon on the same iGPU throughout. Do not record this as a regression, and do
|
|
not record it as a measurement either. Stop the stack before timing the router.
|
|
|
|
`classifier+hash` scores 19.5%, which is the no-ONNX degraded path and is not the
|
|
failure floor the deploy uses. Do not quote it as the classifier baseline.
|
|
|
|
Then three things to decide while the numbers are in front of you:
|
|
|
|
- **319 is done.** 359 gave the LLM path a real confidence signal.
|
|
`thinSingleToken` was narrowed on 01-08-2026, and agenda questions moved to
|
|
stage 0. Missed clarify sits at 1 of 6 and false clarifies at 2. Item 2 point 2
|
|
closed on 02-08-2026: the `make eval-recall` margin sweep is the distribution
|
|
that was asked for, and `0.008` sits at the knee.
|
|
|
|
| delta | answered | false recall |
|
|
|---|---|---|
|
|
| 0.005 | 18/27 | 2/5 |
|
|
| **0.008** | **18/27** | **1/5** |
|
|
| 0.010 | 16/27 | 1/5 |
|
|
|
|
It removes four of five false recalls at no cost in answers, and the next step
|
|
costs two answers for nothing. The hand-picked value survives on evidence.
|
|
- **278's real ask** is making the eval lab routine rather than building it. It
|
|
is built. Decide whether it runs on a timer, on every merge, or on demand, and
|
|
the task can close.
|
|
- **248** is the memory evaluation loop. It ships, it writes notes, and it cannot
|
|
speak. `make eval-recall` covers the retrieval half. The open question is whether
|
|
a written evaluation nobody reads is worth the tick.
|
|
|
|
**323 is down to one check.** PR #90 covered the spawn path and took phraser
|
|
coverage to 76.9%. Only the 60s startup timeout arm is untested, because testing it
|
|
needs a `StartupTimeout` field on `Config` rather than a test-only hack. While you
|
|
are on the box, time a cold 1.7B load off spinning disk. If it runs near 60s, the
|
|
default is too tight and the field earns itself twice.
|
|
|
|
Warm, it is nowhere near. A second llama-server answered `/health` 1.8s after
|
|
launch at `n_ctx 4096` on 02-08-2026. That is page cache, so it does not settle
|
|
the question. A cold read needs a cache drop, which needs root.
|
|
|
|
**`CheckFeminine` has a false positive.** On 02-08-2026 it failed
|
|
`query-notes-do-not-answer` for `ты заплатил`, calling it masculine
|
|
self-reference. Masculine second person is correct, because the owner is male.
|
|
The check matches a masculine
|
|
past-tense verb before `за` without confirming the subject is `я`. Fix it in
|
|
`internal/phraser/eval/checks.go` before trusting a phrasing score to the case.
|
|
The real talk-fixture score on that run is 22 of 27, not 21.
|
|
|
|
Item 4 of **320** needs a permission I do not have. Kill the `llama-server`
|
|
pid under `maven-mavend-1`, post a turn, and confirm it still completes
|
|
through the classifier. Either grant it or run it yourself. It is the only
|
|
check that the failure floor catches a mid-session model death.
|
|
|
|
---
|
|
|
|
## Session 3: the interaction batch (a day, or five sittings)
|
|
|
|
These need real use rather than a command, grouped by what one sitting covers.
|
|
|
|
**Morning and delivery** (**280**, **281**, **128**, **282**, **283**, **285**):
|
|
open `/morning`, walk the seven required behaviours, then check the four
|
|
interruption outcomes and the digest gap. **282** needs the `desk_active` script
|
|
enabled on the desk PC first, which is **15** and needs you at that machine.
|
|
**283** is the event intake envelope every reach shares, so a delivery check
|
|
exercises it whether you name it or not. **285** is not verification: the bridge
|
|
framework works and the remaining ask is more adapters. Decide which reach comes
|
|
next, or park it.
|
|
|
|
**Query sources** (**258**, **286**): ask her something the RSS feeds answer and
|
|
something only a ZIM answers, with the search block on. Live search leads and the
|
|
ZIMs are the fallback since 02-08-2026. A ZIM answer to a current-events question
|
|
means the search leg failed silently. **286**'s remaining half is doc and
|
|
git ingestion, which is build work, not a check.
|
|
|
|
**Tasks and calendar** (**129**, **130**, **127**, **126**, **246**): capture a
|
|
task by voice, confirm it lands, check prioritisation ordering is not nonsense.
|
|
**246** (mail reader) also exercises the `IngestMail` rung that moved to
|
|
`AuthWrite` this morning.
|
|
|
|
**Routines and patterns** (**43**, **46**, **247**, **254**): these need history
|
|
to detect against. If the database is thin after the outage, they may have
|
|
nothing to propose, which is not a failure. Check `/routines` before
|
|
concluding anything.
|
|
|
|
**Ecosystem** (**272**, **273**, **276**): nexus, hexis and praxis are wired and
|
|
logged clean at boot. **276** is the degraded-mode suite, which means taking
|
|
siblings down on purpose. Worth doing while you are already in there.
|
|
|
|
**Operations** (**249**, **250**): these bite hardest if they are broken, and
|
|
nobody has pulled either lever on this box. Roll **249** forward and back once,
|
|
then swap `phraser.model_path` and confirm **250** reloads without a restart. Do
|
|
this sitting last, because both checks can take the box down.
|
|
|
|
---
|
|
|
|
## Housekeeping (one sitting, no box needed)
|
|
|
|
Six QA tasks will not close no matter how long they sit, because they are
|
|
gated on something that does not exist:
|
|
|
|
- **125** zenmoney: needs a token you have not minted.
|
|
- **256** Home Assistant: needs HA configured.
|
|
- **257** Bluetooth: BLOCKED, no bluez on the box. Says so in the title.
|
|
- **288** STT golden audio: needs fixtures generated.
|
|
- **14** cold-start unlock: the `-wrapped-key-file` seam exists, the passkey to L3
|
|
half does not. `lockedAPI` was deleted as dead code in PR #50, so there is
|
|
nothing to verify.
|
|
- **284** replayable full-system simulator: nothing is built. This one is a
|
|
design task wearing a `QA:` prefix.
|
|
|
|
Relabel these so they stop reading as backlog. They are not verification work
|
|
that is pending, they are work that has not started.
|
|
|
|
Same treatment for the five plan-only tasks (**251** MCP, **252** vision,
|
|
**253** hearing, **255** speaker recognition, **259** crawler). A `QA:` prefix on
|
|
a plan is misleading.
|
|
|
|
---
|
|
|
|
## Needs you specifically
|
|
|
|
Not QA. These are blocked on a decision or a credential only you have.
|
|
|
|
| # | what |
|
|
|---|---|
|
|
| 16 | Create the Kuma API key. `-kuma-key uk5_mavpoll-key` in `docker-compose.yml` is still the placeholder. |
|
|
| 15 | Deploy `desk_active` on the desk PC. Blocks **282**. |
|
|
| 122 | Finish the CPT run for Qwen3-1.7B. The persona fix depends on it. |
|
|
| 355 | Deploy the Hexis auth change. Was blocked on Maven being under construction, which it no longer is. The client half is vendored and wired. |
|
|
| 357 | Decide whether entity-existence validation is the permanent target guard or whether blessing lands in Nexus. |
|
|
| 275 | Hexis native API and MCP parity. |
|
|
| — | Decide on `-require-stepup`. Making it the default needs WebAuthn configured first, or it locks you out of your own admin surfaces. |
|
|
|
|
317 and 354 closed on 01-08-2026. The step-up gate now covers `POST /api/chat` and
|
|
`/routines`, and the nginx template is locked down with a `maven.<domain>` block for
|
|
mavweb. The `-require-stepup` default is still your call.
|
|
|
|
---
|
|
|
|
## Not this repo
|
|
|
|
Two open tasks sit on the Maven board and are not Maven work. Move them or note
|
|
where they land, so the board stops reading as 50 things Maven owes.
|
|
|
|
- **358** replace the rowid execution cursor with a real seq column. This is Hexis,
|
|
and it must land before any execution retention or pruning does.
|
|
- **362** mirror the router prompt reorder into the relabelling prompt. This is the
|
|
training workspace, enforced by `llm/check_prompt_parity.py` there, not here.
|
|
|
|
---
|
|
|
|
## Suggested order
|
|
|
|
1. Session 1. If the voice loop is broken, nothing else matters.
|
|
2. The `-require-stepup` and Kuma decisions. Five minutes, and it unblocks **16**.
|
|
3. Session 2. **Run on 02-08-2026.** The numbers came back worse for the router
|
|
than the docs claimed. The classifier is 68.8%, not 36.8%, and 16.6µs, not
|
|
31ms. The router buys about 4 points of accuracy for four orders of magnitude
|
|
of latency. Whether that still earns its place is now an open question.
|
|
4. Housekeeping. Cheap, and it makes the remaining backlog honest.
|
|
5. Session 3, split whichever way suits you.
|