claude
93987f2dfc
Seventeen markdown files at the repo root, twelve of them dated one-shot reports sitting next to CLAUDE.md. That is why stale docs read as current: nothing in the path said which was which. Root now keeps CLAUDE.md and AGENTS.md. Living docs move under docs/ and carry a Last verified line. Dated measurements move to docs/evals/ ISO-prefixed, and are never edited after the day, so a newer number is a new file. The senior review moves to docs/archive/. Every reference was rewritten across markdown, Go comments, the Makefile and the recall fixture. The touched Go packages still build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
187 lines
8.6 KiB
Markdown
187 lines
8.6 KiB
Markdown
# QA plan: checking Maven properly
|
|
|
|
*Last verified: 2026-08-02 @ 7079a24. Living doc: correct it in place, do not append.*
|
|
|
|
Written 2026-08-01, after the 35-PR stack landed and the box came back up.
|
|
|
|
44 of the 50 open Vikunja tasks are `QA:` tasks. They are verification work, not
|
|
build work. Most sat unverifiable while Maven was down for 11 days. That
|
|
blocker is gone.
|
|
|
|
This plan orders them by what unblocks what. Do sessions 1 and 2 first. Almost everything
|
|
downstream assumes the voice loop works, and nobody has confirmed that since
|
|
the redeploy.
|
|
|
|
---
|
|
|
|
## Before you start
|
|
|
|
Two things bite anyone running these checks on homesrv.
|
|
|
|
**curl needs `--noproxy '*'`.** The shell exports `http_proxy=http://127.0.0.1:18080`.
|
|
Without the flag, every local check returns 503 from the proxy and looks like a
|
|
dead service. This cost me a false regression report today.
|
|
|
|
**The database is not readable with sqlite3.** Four older QA steps say
|
|
`docker compose exec mavend sqlite3 /data/maven.db "select ..."`. That cannot
|
|
work: the container has no `sqlite3` binary, and the store is AES-256-GCM at
|
|
rest with a tmpfs working copy. Read state through mavweb instead, at
|
|
`/history`, `/trace`, `/routines` and `/dash`.
|
|
|
|
---
|
|
|
|
## Session 1: the voice loop (half a day)
|
|
|
|
Nothing here has been confirmed since the redeploy, and everything else assumes
|
|
it works. Do this first.
|
|
|
|
Closes or advances: **44** (conversation), **45** (text chat), **287** (voice
|
|
session quality), **321** steps 3-5 (quiet mode), **288** (STT fixtures).
|
|
|
|
1. Open `http://127.0.0.1:9201/chat` and hold a short conversation in Russian.
|
|
Watch for three things: she answers in feminine forms (`рада`, `поняла`), she
|
|
says `ты` and never `вы`, and no pet names appear.
|
|
2. Press push-to-talk on `/dash`. Say `привет`. Confirm a spoken reply comes
|
|
back. This is the only check that covers mic to STT to core to TTS to
|
|
speaker as one path. It is also the path the eleven-day outage most likely
|
|
broke.
|
|
3. Say `тихий режим`. Expect `тихий режим включён. буду реже напоминать.`
|
|
4. Say `выключи тихий режим`. Expect `тихий режим выключен.` Negation must win.
|
|
5. Say `в комнате тихо`. Quiet mode must NOT flip. Confirm on `/history` that no
|
|
`quiet_hours` fact was written.
|
|
6. Say `включи режим тишины`, then `сделай потише`. Both must flip quiet mode
|
|
on. These are the noun form and the comparative, added 01-08-2026.
|
|
7. Wait for a nudge, then say `потом` within twenty minutes. Expect `хорошо,
|
|
вернусь к этому позже.` and the nudge row on `/notifications` reading
|
|
`snoozed`. Say `потом` again with nothing pending: it must route as an
|
|
ordinary utterance, not be swallowed.
|
|
8. Wait for the water nudge, then say `выпил воды`. Expect the ordinary fact
|
|
reply and nothing extra — she must not congratulate you. Check
|
|
`/notifications`: the row reads `acted`. Then trigger another nudge and say
|
|
`готово`; expect `отлично, отметила.` and the same outcome.
|
|
9. Note anything where she is slow, cuts off, or talks over herself. That is
|
|
287's whole content and it has no written acceptance criteria yet.
|
|
|
|
**319 is fixed** (01-08-2026). Single-word Russian utterances no longer come
|
|
back as `не совсем поняла — можешь переформулировать?`. `привет` and `поужинал`
|
|
both pass now: `thinSingleToken` spares social singles and any token carrying a
|
|
verb ending, and only thins a bare nominal like `вода`. If a one-word utterance
|
|
still gets clarified during the smoke test, that is a new case for the lexicon,
|
|
not the old bug.
|
|
|
|
---
|
|
|
|
## Session 2: measurement (half a day, mostly waiting)
|
|
|
|
Closes or advances: **320** items 2-4, **278** (make the eval lab routine),
|
|
**319** (gate recalibration).
|
|
|
|
The resident llama-server cannot be reached by the eval harness. It binds
|
|
`--host 127.0.0.1 --port 0` inside the container, so the port is kernel-assigned
|
|
and never published. Start a second one on a fixed port instead:
|
|
|
|
```sh
|
|
llama-server -m /mnt/hdd1/llms/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf \
|
|
--host 127.0.0.1 --port 18100 -c 4096 -ngl 99 --no-webui
|
|
```
|
|
|
|
`-c 4096` matters. The recorded numbers were measured at that context size, and
|
|
a mismatch invalidates the comparison.
|
|
|
|
Then:
|
|
|
|
```sh
|
|
make eval-models MAVEN_LLM_URL=http://127.0.0.1:18100 # want ~72.7% cascade
|
|
make eval-router # classifier baseline
|
|
MAVEN_LLM_URL=http://127.0.0.1:18100 make eval-phrasing # persona checks, slow
|
|
make eval-recall
|
|
```
|
|
|
|
A large miss against 72.7% means the deploy differs from the bench harness.
|
|
|
|
Two things to decide while the numbers are in front of you:
|
|
|
|
- **319's gate recalibration.** The single-token rule needs narrowing or
|
|
dropping. This needs your judgement, not a threshold sweep. The fixture and the
|
|
daemon disagree about what is correct on two of the three false clarifies.
|
|
- **278's real ask** is making the eval lab routine rather than building it. It
|
|
is built. Decide whether it runs on a timer, on every merge, or on demand, and
|
|
the task can close.
|
|
|
|
Item 4 of **320** needs a permission I do not have. Kill the `llama-server`
|
|
pid under `maven-mavend-1`, post a turn, and confirm it still completes
|
|
through the classifier. Either grant it or run it yourself. It is the only
|
|
check that the failure floor catches a mid-session model death.
|
|
|
|
---
|
|
|
|
## Session 3: the interaction batch (a day, or three sittings)
|
|
|
|
These need real use rather than a command, grouped by what one sitting covers.
|
|
|
|
**Morning and delivery** (**280**, **281**, **128**, **282**): open `/morning`,
|
|
walk the seven required behaviours, then check the four interruption outcomes
|
|
and the digest gap. **282** needs the `desk_active` script enabled on the desk
|
|
PC first, which is **15** and needs you at that machine.
|
|
|
|
**Tasks and calendar** (**129**, **130**, **127**, **126**, **246**): capture a
|
|
task by voice, confirm it lands, check prioritisation ordering is not nonsense.
|
|
**246** (mail reader) also exercises the `IngestMail` rung that moved to
|
|
`AuthWrite` this morning.
|
|
|
|
**Routines and patterns** (**43**, **46**, **247**, **254**): these need history
|
|
to detect against. If the database is thin after the outage, they may have
|
|
nothing to propose, which is not a failure. Check `/routines` before
|
|
concluding anything.
|
|
|
|
**Ecosystem** (**272**, **273**, **276**): nexus, hexis and praxis are wired and
|
|
logged clean at boot. **276** is the degraded-mode suite, which means taking
|
|
siblings down on purpose. Worth doing while you are already in there.
|
|
|
|
---
|
|
|
|
## Housekeeping (one sitting, no box needed)
|
|
|
|
Four QA tasks will not close no matter how long they sit, because they are
|
|
gated on something that does not exist:
|
|
|
|
- **125** zenmoney: needs a token you have not minted.
|
|
- **256** Home Assistant: needs HA configured.
|
|
- **257** Bluetooth: BLOCKED, no bluez on the box. Says so in the title.
|
|
- **288** STT golden audio: needs fixtures generated.
|
|
|
|
Relabel these so they stop reading as backlog. They are not verification work
|
|
that is pending, they are work that has not started.
|
|
|
|
Same treatment for the five plan-only tasks (**251** MCP, **252** vision,
|
|
**253** hearing, **255** speaker recognition, **259** crawler). A `QA:` prefix on
|
|
a plan is misleading.
|
|
|
|
---
|
|
|
|
## Needs you specifically
|
|
|
|
Not QA. These are blocked on a decision or a credential only you have.
|
|
|
|
| # | what |
|
|
|---|---|
|
|
| 16 | Create the Kuma API key. `-kuma-key uk5_mavpoll-key` in `docker-compose.yml` is still the placeholder. |
|
|
| 15 | Deploy `desk_active` on the desk PC. Blocks **282**. |
|
|
| 122 | Finish the CPT run for Qwen3-1.7B. The persona fix depends on it. |
|
|
| 355 | Deploy the Hexis auth change. Was blocked on Maven being under construction, which it no longer is. The client half is vendored and wired. |
|
|
| 357 | Decide whether entity-existence validation is the permanent target guard or whether blessing lands in Nexus. |
|
|
| 275 | Hexis native API and MCP parity. |
|
|
| — | Decide on `-require-stepup`. Making it the default needs WebAuthn configured first, or it locks you out of your own admin surfaces. See **317**. |
|
|
| — | Three nginx sites bind wildcard `:80` (`acme.conf`, `matrix`, `panel`), so the ecosystem's bind-level protection is not in effect and `allow`/`deny` is carrying it alone. See **354**. |
|
|
|
|
---
|
|
|
|
## Suggested order
|
|
|
|
1. Session 1. If the voice loop is broken, nothing else matters.
|
|
2. The `-require-stepup` and Kuma decisions. Five minutes, unblocks **317** fully
|
|
and **16**.
|
|
3. Session 2. The numbers tell you whether the router is worth its 90x latency.
|
|
4. Housekeeping. Cheap, and it makes the remaining backlog honest.
|
|
5. Session 3, split whichever way suits you.
|