From b6666196c17d99c31c455055200a50a0c02bafa2 Mon Sep 17 00:00:00 2001 From: claude Date: Wed, 26 Aug 2026 12:26:28 +0400 Subject: [PATCH] Plan the capability ledger and the empirical baseline (V-725) Three sessions on one causal order: the spec says what should happen, the empirical run says what actually happens, code and architecture explain why, priority says what to fix. The predecessor audit had a green suite while 22 of 39 capabilities were not live, which is the failure mode this order exists to stop. Co-Authored-By: Claude Opus 5 --- .../26-capability-ledger-and-baseline.md | 276 ++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 docs/plans/26-capability-ledger-and-baseline.md diff --git a/docs/plans/26-capability-ledger-and-baseline.md b/docs/plans/26-capability-ledger-and-baseline.md new file mode 100644 index 0000000..761fcc9 --- /dev/null +++ b/docs/plans/26-capability-ledger-and-baseline.md @@ -0,0 +1,276 @@ +# 26. The capability ledger and the empirical baseline + +Unfiled, no task id. Written 2026-08-26 against `5cae33a`. + +Frozen once session 1 starts. Changes of mind go to the handoff, not back here. + +## The question this answers + +How much of the Maven described by `docs/spec.md` exists today, and where does +the current architecture support or obstruct that target. + +## What this is not + +- Not a redesign and not another architecture pass. +- No fixes. This pass produces the map used to decide what to build next. +- `docs/spec.md` is not edited. It is the authoritative statement of intent. +- A package, type or function existing is not proof a capability works. + +## The causal order + +```text +spec says what should happen + ↓ +the empirical run says what actually happens + ↓ +code and architecture explain why + ↓ +priority says what to fix +``` + +The reverse order is what the capability spec was written to prevent. Its +predecessor audit had a green suite while 22 of 39 capabilities were not live. + +## The target Maven + +A persistent personal agent. Request and response is one input mode, not the +shape of the system. The target is drawn around capabilities, never packages. + +Eight domains: + +| domain | holds | +| --- | --- | +| perception/context | presence, device state, time, environment, current activity, conversation state | +| memory/model of you | facts, preferences, relationships, history, commitments, notes, routines, inferred context | +| attention | what matters now, unfinished things, reminders, deadlines, anomalies, things worth surfacing | +| deliberation | interpret requests, resolve ambiguity, connect current events to past context, decide to act, query, ask or wait | +| initiative | nudges, follow-ups, missed-task recovery, opportunistic suggestions, background checks | +| action | local tools, home automation, other services, information retrieval, document and email workflows | +| interaction | voice, web, notifications, telegram or matrix, with continuity across surfaces | +| governance | permissions, confidence, confirmation, privacy, reversibility, non-autonomous propose-then-enable | + +A ninth bucket, `operations`, holds what the spec files under Operations. It is +infrastructure, not agent behavior, and a forced fit would hide that. + +## The five vertical slices + +The slices the field probes exercise. Each is end to end, never one turn. + +1. **reminder**: create, clarify time, correct, persist, fire, acknowledge. +2. **fact/note**: record, correct or supersede, retrieve later, answer with current truth. +3. **query**: understand, choose source, preserve follow-up context, answer. +4. **action**: understand target, confirm only when needed, execute, report the actual result. +5. **proactive**: detect the condition, decide whether to interrupt, deliver on the right channel. + +## Three sessions + +The baseline is perishable. It measures one model on one config, so a config +change invalidates it. The viewer is presentation over data that does not exist +yet. + +| session | produces | +| --- | --- | +| 1 | `docs/capabilities/ledger.yaml` target side, then the empirical baseline as a dated eval | +| 2 | implementation mapping, `docs/capabilities/invariants.md`, `docs/capabilities/gaps.md`, the ranked list | +| 3 | the `capabilities` mode in the architecture viewer | + +## Session 1, step 1: the skeleton + +Mechanical extraction from `docs/spec.md` into `docs/capabilities/ledger.yaml`. +No implementation judgment, no verification status, no ranking. + +Preserved per capability: id, section, `v1` or `deferred` scope, state +references, every DoD criterion, scenario names, and the findings the spec +already carries. + +Added at extraction time: + +- `domain`, at most two, primary first. More than two means it is two capabilities. +- A stable criterion id, `#<4 hex of the criterion text>`. A + positional id shifts when the spec gains a criterion, and the frozen eval + would then cite the wrong one. + +The 51-row capability-to-domain table goes to the owner once, before any probe +runs. + +The five deferred capabilities enter the skeleton and are not probed. Their +criteria read `untested`, reason `deferred past v1`. + +## Session 1, step 2: the empirical baseline + +### Where it runs + +Against the live `maven-mavend-1`, then `mavend -wipe -confirm-wipe` after the +run. A dry run without the confirm flag comes first and its row counts are the +eval's first artifact. + +The live resident model is `maven-instruct-b2-Q4_K_XL`, not the Qwen3-1.7B that +`CLAUDE.md` still names. `deploy/mavend.json` carries the switch uncommitted. +Every number in the baseline is attributable to b2. + +### Transport + +`POST /api/chat` on `127.0.0.1:9201` drives a real turn through the deployed +stack. The reply comes back in the redirect `Location` as `?q=…&r=…&t=`. +No new binary and no audio needed, and it covers the web reach. + +### Readback + +`cmd/e2eprobe`, built in a one-off `golang:1.25-trixie` container and copied in +with `docker cp`. Both images are trixie, so glibc matches. It gives typed IPC +JSON for `decisions`, `facts`, `notes`, `reminders`, `nudges`, `tools` and +`delivery-attempts`. + +Not the sqlite file. The plaintext copy at `/dev/shm/maven-plain.db` bypasses the +contract the ledger exists to measure. Not the mavweb pages either: HTML is a +second-hand rendering. + +### Two probe origins + +- `dod:`, derived from the ledger. Only these set a DoD verdict. +- `field:`, the owner's real week. 25 multi-turn probes, drafted from the + five slices, the three ugly conversations verbatim, the five existing + scenarios, and what the box is configured for. + +A failing `field` probe becomes an unresolved product question in +`invariants.md` where the spec never defined the behavior. It becomes a +missing-criterion finding against `docs/spec.md` where the spec should have +covered it. Neither edits the spec in this pass. + +### Three evidence kinds + +| kind | what it is | may set `pass` | +| --- | --- | --- | +| `live` | `e2eprobe` and `/api/chat` against the deployed build, real model, real store rows | yes | +| `replay` | the scenario harness, model faked, clock controlled | no | +| `simulated` | faked clock, real code path, no deployed process | no | + +A green `replay` with no live probe reads `verified: untested`, with the replay +recorded as implementation evidence. The scenario harness scripts both `route` +and `reply`, so a pass proves the wiring around the model and not the turn. + +`simulated` is allowed only where the trigger is anchored to a wall-clock hour +or date the pass cannot reach. The morning digest hour, the 14-day trace +retention sweep, a weekly routine. Everything else uses a compressed live +horizon, because `TickInterval` defaults to 60s and no override is configured. + +### Reaches + +`reachable` is per reach. Inbound: web, ipc. Outbound: `ntfysink`, +`telegramsink`, `voicesink`. + +One delivery per outbound reach, three notifications total, each carrying a fixed +marker so a probe is not read as a real nudge. `reachable: pass` needs the +`delivery-attempts` row plus the owner confirming arrival. `voicesink` needs a +connected client on workpc, and records `blocked, no listener` when there is +none. + +### Per-criterion verdict + +Every DoD criterion gets `pass`, `fail`, `blocked`, `untested` or `unknown`, and +a reason distinguishing why it is not passing: code missing, wiring missing, +configuration missing, deployment missing, external dependency unavailable, +scenario missing, scenario fails, or implementation exists with no runtime proof. + +Where a scenario covers a criterion, the exact step or assertion is named. Where +none exists, the spec's `(to write)` state stands. No evidence is invented. + +Every generated claim carries `path:line`, or runtime or scenario evidence. +`unknown` where evidence is insufficient. + +### Stop condition + +Session 1 ends when every v1 DoD criterion carries a verdict with evidence +attached, and the eval is written. No mapping, no gaps, no ranking. + +A baseline that stops halfway leaves a ledger that looks measured and is not. + +## Session 2: the explanation + +Implementation mapping onto the ledger, as independent dimensions, never +collapsed into one `implemented` boolean: `designed`, `code_present`, `wired`, +`configured`, `deployed`, `reachable`, `verified`. + +`docs/capabilities/invariants.md` extracts the cross-cutting rules the 51 +capabilities imply and do not state. Continuity across turns and across +reaches, memory and correction semantics, current context and presence, +proactive attention, interruption policy, clarification and follow-up ownership, +degradation and honesty, authority and confirmation, privacy boundaries, +learning from outcomes, capability composition, persistence across restart. + +Each invariant is marked `explicit`, `implied` or `unresolved`, with evidence. +Nothing desired is invented where the sources do not define it. An unresolved +invariant is a product question, not a defect. + +`docs/capabilities/gaps.md` compares responsibilities, never package names. +Classes: capability missing, capability partial, capability exists but +unreachable, capability exists but unverified, duplicated mechanism, missing +shared mechanism, current architecture conflicts with target behavior, and +architecture concern with no current product impact. + +Every architecture concern names the capability or invariant it affects. One +affecting none is marked non-blocking cleanup explicitly. + +### Prioritization + +One final list, ranked: + +1. Prevents intended everyday use today. +2. Makes existing behavior incorrect or unreliable. +3. Blocks multiple capabilities. +4. Prevents verification. +5. Architectural cleanup with no present user impact. + +An unwired or unreachable future defect never outranks a live user-visible +failure because its architecture is ugly. + +## Session 3: the viewer + +A `capabilities` mode in the architecture viewer. The primary view is a matrix of +capability against `designed`, `coded`, `wired`, `configured`, `deployed`, +`reachable`, `verified`. + +Clicking a capability shows the target DoD, implementation evidence, +verification evidence, affected components, scenarios, blockers and unresolved +product questions. + +An `invariants` view shows which components participate in each cross-cutting +rule. Target, implementation and runtime verification are distinguished visually. + +## Where the artifacts live + +`docs/capabilities/` becomes a declared tier in `docs/CLAUDE.md`: generated, +regenerated from `docs/spec.md` plus a named eval, never hand-edited. The same +row legitimises `docs/architecture/`, which is currently an undeclared +directory. + +The empirical run lands as one dated eval, `docs/evals/2026-08-26-capability-baseline.md`, +frozen on the day. Every `verified` cell in the ledger cites it by path, so no +status is an opinion. + +## Decided, do not re-ask + +- Three sessions, split on the causal order above. +- Target side of the ledger first, with no implementation or verification status. +- Two verified columns plus `simulated`. Only `live` sets `pass`. +- Probes run against the live container, with the store wiped afterwards. +- Row counts are read with a `mavend -wipe` dry run before anything else. +- Readback is `e2eprobe`, not sqlite and not the web pages. +- Both probe origins, `dod:` and `field:`, tagged. +- Deferred capabilities get the skeleton only. +- All three outbound reaches get one real delivery. +- Domain is a second axis beside the spec section. +- No fixes land in this pass. + +## Findings recorded while planning + +Live and reproduced, not inferred. They belong to the baseline, not to this plan. + +- `POST /api/chat` with `что ты помнишь обо мне?` answered + `Я не знаю вас или как вы себя называете`. Formal `вас` and `вы` on the wire, + where `CLAUDE.md` requires informal singular `ты`. The phrasing eval passes. +- Vikunja was never down. `vikunja-mcp` publishes `127.0.0.1:9100` only, so the + LAN address never answers from workpc. Three sessions read a refused + connection as an outage and filed nothing. +- The deployed resident model is `maven-instruct-b2-Q4_K_XL`. `CLAUDE.md` still + names Qwen3-1.7B.