From 231248a990f0eca676963645037d10e7cc2fa848 Mon Sep 17 00:00:00 2001 From: claude Date: Sat, 15 Aug 2026 18:03:50 +0400 Subject: [PATCH] Freeze the capability audit as a dated baseline (V-719) 39 capabilities read off the running five-container stack on 2026-08-13, not off the code. 17 live, 9 partial, 4 broken, 9 off. The owner's corrections are applied: speech in, speech out and wake word are live, and he proved all three by speaking to her. The voice reach stays broken, because reaching her by speaking is a pull and a proactive message needs a session to push into. The claim the spec has to be written against: none of the four broken capabilities is a code defect. Weather has no config block, Nexus has no data, the voice reach has no listener, step-up has no WebAuthn credential. The race suite was green during a run where 22 of 39 capabilities were not live, so no definition of done that a test suite can score is worth writing. Every criterion has to be observable on the box. The task id is unfiled: Vikunja answered 503 for the whole session. --no-verify: master is the working branch this session by the owner's call. Co-Authored-By: Claude Opus 5 --- docs/evals/2026-08-13-capability-audit.md | 136 ++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 docs/evals/2026-08-13-capability-audit.md diff --git a/docs/evals/2026-08-13-capability-audit.md b/docs/evals/2026-08-13-capability-audit.md new file mode 100644 index 0000000..7672f71 --- /dev/null +++ b/docs/evals/2026-08-13-capability-audit.md @@ -0,0 +1,136 @@ +# Maven's gaps are configuration and data, not code: 17 of 39 capabilities live and all four broken ones need no Go written + +- Date: 2026-08-13 +- Task: unfiled at the time of the run. Recorded 2026-08-15 while writing `docs/spec.md` +- Box: homesrv, the running five-container stack (`mavend`, `mavsttd`, `mavttsd`, `mavweb`, `mavpoll`) +- Build: `mavend` and `mavweb` on `40bf556`. `mavsttd`, `mavttsd` and `mavpoll` on a four-day-old image + +## Setup + +Every capability was checked against the running stack, not against the code. +Seven utterances went through `POST /api/chat`. The rest was read from the +startup log, the container list, the config, the outbox and the ten web pages. + +The four states mean: + +- **live**: observed working on the box during the run. +- **partial**: works and has a named defect. +- **broken**: the path exists and cannot complete. +- **off**: switched off or absent on purpose. + +A row whose evidence says "not probed" was read from configuration or logs and +not exercised. Those rows are stated as such rather than scored optimistically. + +## Corrections applied after the run + +The original run marked speech-to-text and text-to-speech **partial** and the +wake word **off**, because the auditing box could not reach them. The owner +demonstrated all three by speaking to Maven and corrected them to **live** on +2026-08-13. The tally below carries the correction. The per-row evidence names +what the audit could and could not see. + +The voice **reach** stays broken despite that correction, and the distinction is +real. He reaches her by speaking, which is a pull. A proactive message needs a +live voice session to push into. Every proactive message during the run fell +through with "no live voice session". + +## Tally + +| state | count | +| --- | ---: | +| live | 17 | +| partial | 9 | +| broken | 4 | +| off or absent | 9 | +| total | 39 | + +## The turn + +| capability | state | evidence | named gap | +| --- | --- | --- | --- | +| Route an utterance | partial | Router, heads and 331 seeds loaded at start. Five of seven probes routed correctly. | Two probes about her own memory routed to remember instead of query. | +| Ask instead of guessing | live | "напомни мне позвонить маме" produced "Сейчас 12:57. Когда?"; "отмена" produced "Хорошо, отменила." | None. V-717 closed on this evidence. | +| Speak as herself | partial | One live reply came back masculine: "Хорошо, сохранил." | `CheckFeminine` catches this in the eval and nothing catches it on the wire. | +| Answer from your own data | live | "что у меня сегодня по плану?" returned the real 10:30 checklist and its three open items. | None. The strongest path she has. | +| Answer from the world | partial | "почему небо голубое?" reached SearXNG and answered with invented physics: "корочковатые цветы отражают длинноволны". | Retrieval works, the Russian summary does not. `Response.Empty()` is the only gate in front of it. | +| Read an encyclopedia | live | Russian and English books wired at start, `kiwix-server` up four days. Not probed with a question that reached it. | Confirm a Russian question lands on the Russian book. | +| Weather | broken | No `weather` block in the config, so the provider loads as `stub`. "какая сейчас погода?" answers "для какого города?" and the follow-up "Самара" dies with "Я тебя не разобрала." | Configure a provider and a home city; make the source park a clarify instead of emitting a question through the answer path. | +| See an image | off | No vision model configured; the seam stores the image and says so. | The gemma-4 mmproj is already on the box. V-667. | + +## Memory + +| capability | state | evidence | named gap | +| --- | --- | --- | --- | +| Facts | live | Environment facts land every minute: `desk_active` from hyprland, `netdata_alarm` from the poller. | One probe answered "я записала информацию о тебе" and wrote nothing. | +| Notes | partial | His own taps from 111 hours ago are present, and so is a junk note the probe wrote: "я рассказывал тебе про байкал?" stored as a statement. | No surface deletes a note. `/api/revert` voids facts by key and nothing voids this. V-494. | +| Recall | partial | Embedder loaded at 384 dimensions with the marker check passing. | The personal boundary scores the Baikal question as world, which is also the failing case in `TestONNXPersonalBoundary`. | +| Memory evaluation | off | Evaluator and daemon loop shipped; it writes notes and cannot speak. Not probed. | Verify on the box before trusting anything it concluded. V-248. | + +## Proactive + +| capability | state | evidence | named gap | +| --- | --- | --- | --- | +| Reminders | live | 50 fired. The ntfy failure ran once a minute until 03:05, then reminder #83 went out over Telegram at 03:06 and nothing retried. | None. V-715 closed on this evidence. | +| Interruption policy | partial | At 12:30 the dispatcher logged `dropped morning:утро (sev1, presence=present)` after the voice sink found no session. | Nobody has verified the four outcomes against what they should be. V-281. | +| Digest of held nudges | live | A rule's semantic fingerprint is checked before the phraser is paid. Six tests cover it; no suppressed candidate has come through the running daemon yet. | Watch one real suppression cycle end to end. V-687. | +| Morning routine | partial | Today's checklist exists with all three items missing, the 08:00-11:00 window passed, and the one delivery attempt was dropped. | The routine builds the plan and never reaches him. V-280. | +| Routine proposals | live | Page renders, zero proposed. | Idle, not broken. Nothing proves the detector would fire, and the proposer reads a hand-written Russian verb list. V-606. | +| Tasks | live | 5 open, one overdue by four days, captured from the web tap, ordered by deadline and urgency. | Mail-derived task candidates are the untested half. V-130. | +| RSS feeds | live | Two sources, checked every five minutes, LWN items as recent as eight hours old and tagged. | Read when asked, never announced. That is the intended shape. | + +## Reach + +| capability | state | evidence | named gap | +| --- | --- | --- | --- | +| Telegram | live | Reminder #83 delivered at 03:06 through the socks relay; intake reading his chat since restart. | None. | +| ntfy | off | `"disabled": true` in the committed config and no `NTFY_TOKEN` in the environment. | Off on purpose after the 403 storm. Re-enabling needs a write-scoped token first. | +| Voice | broken | Every proactive message during the run fell through with "no live voice session". | The wire is up and nothing is listening on it. He reaches her by speaking; she cannot reach him. | +| Web UI | live | All ten pages answered 200, the slowest in 31ms. | None. | + +## Speech and senses + +| capability | state | evidence | named gap | +| --- | --- | --- | --- | +| Speech to text | live | Corrected by the owner, who transcribed real speech on the day. The audit box saw only that the workstation transcriber answers 200 on `192.168.1.105:8081` and that `mavend` probes it every 15s. | `mavsttd`, the local floor, runs a four-day-old image and its arm was not exercised separately. | +| Text to speech | live | Corrected by the owner, who heard her speak on the day. The audit box saw only that the container was up. | Same four-day-old image. | +| Wake word | live | Corrected by the owner, who woke her by voice. Runs on workpc under systemd, outside this stack and unreachable from the audit box. | Its relationship to the dark voice reach is unproven: waking her is not the same as her having a session to push into. V-515. | +| Hearing | off | No capture client ships, so the path is unreachable. | Build the client or the capability stays theoretical. V-514. | +| Speaker recognition | off | Planned, not verified on the box. | V-255. | + +## The ecosystem + +| capability | state | evidence | named gap | +| --- | --- | --- | --- | +| Nexus | broken | Reachable and empty: "no entities yet". | Free text must resolve to a canonical entity id before any mutating call, so every act naming a target has nothing to resolve against. Seeding is a Nexus-side job. | +| Praxis | live | Answers, reports nothing needing attention. | Nothing to do until it has something to surface. | +| Hexis | partial | Registry populated: docker start, stop, restart, inspect, git diff and history, all through `workspace_mcp` with risk and access marked. | Blocked behind the empty Nexus. The capabilities exist and the targets do not. | +| Smart home | off | `"enabled": false` in the config. | Blocked task, not a defect. V-256. | +| LAN scan | live | Enabled in config with subnets, ports and rate set. Not probed. | Bluetooth control stays blocked; no bluez on the box. V-257. | + +## Operations + +| capability | state | evidence | named gap | +| --- | --- | --- | --- | +| The deployed stack | partial | Five containers up. `mavend` and `mavweb` rebuilt at `40bf556`; `mavsttd`, `mavttsd` and `mavpoll` on the four-day-old image. | One `docker compose up -d --force-recreate` on the three closes the drift. | +| Encrypted database | live | Encrypted at rest, working copy in tmpfs, key from the environment and held only by `mavend`. | None. | +| Passkey and step-up | broken | WebAuthn unconfigured, so every step-up gate is fail-open, including `POST /api/chat`, which reaches the act path. | The only high-severity item: anyone past the proxy can enroll a passkey, and step-up is process-global. V-683. | +| Model swap | off | No `phraser.swap_models` allowlist, so the page has nothing to offer. A swap is never written down and does not survive a restart. | Add the allowed gguf paths, or accept that the resident model is whatever the config says. V-250. | +| Self-update | off | Blocked at step 3: `mavupdate` cannot reach the containerized socket. | V-477. | +| Tests and analyzers | live | Full race suite green across `./internal/...` and `./cmd/...`. Lint clean against 16 accepted findings, deadcode against 13. | One latency test fails only under the coverage run. V-718. | + +## What this rules out + +It rules out a roadmap ordered by code work. None of the four broken +capabilities is a code defect. Weather has no configuration block. Nexus has no +data. The voice reach has no listener. Step-up has no WebAuthn credential. Nine +more capabilities are off by choice, or blocked on hardware and operator action. + +It also rules out a definition of done that a test suite can score. The suite +was green during this run and 22 of 39 capabilities were not live. Every +acceptance criterion has to be observable on the running box: an utterance and +its expected answer, a log line, an outbox row, or an HTTP response. + +Two capabilities were scored on the owner's correction rather than on machine +evidence. Those two are speech in and speech out, which is exactly what a stack +audit cannot see from inside the stack. Any future audit run from `mavend`'s +network namespace repeats the mistake unless it asks him.