Plan the capability ledger and the empirical baseline (V-725)
Three sessions on one causal order: the spec says what should happen, the empirical run says what actually happens, code and architecture explain why, priority says what to fix. The predecessor audit had a green suite while 22 of 39 capabilities were not live, which is the failure mode this order exists to stop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,276 @@
|
||||
# 26. The capability ledger and the empirical baseline
|
||||
|
||||
Unfiled, no task id. Written 2026-08-26 against `5cae33a`.
|
||||
|
||||
Frozen once session 1 starts. Changes of mind go to the handoff, not back here.
|
||||
|
||||
## The question this answers
|
||||
|
||||
How much of the Maven described by `docs/spec.md` exists today, and where does
|
||||
the current architecture support or obstruct that target.
|
||||
|
||||
## What this is not
|
||||
|
||||
- Not a redesign and not another architecture pass.
|
||||
- No fixes. This pass produces the map used to decide what to build next.
|
||||
- `docs/spec.md` is not edited. It is the authoritative statement of intent.
|
||||
- A package, type or function existing is not proof a capability works.
|
||||
|
||||
## The causal order
|
||||
|
||||
```text
|
||||
spec says what should happen
|
||||
↓
|
||||
the empirical run says what actually happens
|
||||
↓
|
||||
code and architecture explain why
|
||||
↓
|
||||
priority says what to fix
|
||||
```
|
||||
|
||||
The reverse order is what the capability spec was written to prevent. Its
|
||||
predecessor audit had a green suite while 22 of 39 capabilities were not live.
|
||||
|
||||
## The target Maven
|
||||
|
||||
A persistent personal agent. Request and response is one input mode, not the
|
||||
shape of the system. The target is drawn around capabilities, never packages.
|
||||
|
||||
Eight domains:
|
||||
|
||||
| domain | holds |
|
||||
| --- | --- |
|
||||
| perception/context | presence, device state, time, environment, current activity, conversation state |
|
||||
| memory/model of you | facts, preferences, relationships, history, commitments, notes, routines, inferred context |
|
||||
| attention | what matters now, unfinished things, reminders, deadlines, anomalies, things worth surfacing |
|
||||
| deliberation | interpret requests, resolve ambiguity, connect current events to past context, decide to act, query, ask or wait |
|
||||
| initiative | nudges, follow-ups, missed-task recovery, opportunistic suggestions, background checks |
|
||||
| action | local tools, home automation, other services, information retrieval, document and email workflows |
|
||||
| interaction | voice, web, notifications, telegram or matrix, with continuity across surfaces |
|
||||
| governance | permissions, confidence, confirmation, privacy, reversibility, non-autonomous propose-then-enable |
|
||||
|
||||
A ninth bucket, `operations`, holds what the spec files under Operations. It is
|
||||
infrastructure, not agent behavior, and a forced fit would hide that.
|
||||
|
||||
## The five vertical slices
|
||||
|
||||
The slices the field probes exercise. Each is end to end, never one turn.
|
||||
|
||||
1. **reminder**: create, clarify time, correct, persist, fire, acknowledge.
|
||||
2. **fact/note**: record, correct or supersede, retrieve later, answer with current truth.
|
||||
3. **query**: understand, choose source, preserve follow-up context, answer.
|
||||
4. **action**: understand target, confirm only when needed, execute, report the actual result.
|
||||
5. **proactive**: detect the condition, decide whether to interrupt, deliver on the right channel.
|
||||
|
||||
## Three sessions
|
||||
|
||||
The baseline is perishable. It measures one model on one config, so a config
|
||||
change invalidates it. The viewer is presentation over data that does not exist
|
||||
yet.
|
||||
|
||||
| session | produces |
|
||||
| --- | --- |
|
||||
| 1 | `docs/capabilities/ledger.yaml` target side, then the empirical baseline as a dated eval |
|
||||
| 2 | implementation mapping, `docs/capabilities/invariants.md`, `docs/capabilities/gaps.md`, the ranked list |
|
||||
| 3 | the `capabilities` mode in the architecture viewer |
|
||||
|
||||
## Session 1, step 1: the skeleton
|
||||
|
||||
Mechanical extraction from `docs/spec.md` into `docs/capabilities/ledger.yaml`.
|
||||
No implementation judgment, no verification status, no ranking.
|
||||
|
||||
Preserved per capability: id, section, `v1` or `deferred` scope, state
|
||||
references, every DoD criterion, scenario names, and the findings the spec
|
||||
already carries.
|
||||
|
||||
Added at extraction time:
|
||||
|
||||
- `domain`, at most two, primary first. More than two means it is two capabilities.
|
||||
- A stable criterion id, `<capability-slug>#<4 hex of the criterion text>`. A
|
||||
positional id shifts when the spec gains a criterion, and the frozen eval
|
||||
would then cite the wrong one.
|
||||
|
||||
The 51-row capability-to-domain table goes to the owner once, before any probe
|
||||
runs.
|
||||
|
||||
The five deferred capabilities enter the skeleton and are not probed. Their
|
||||
criteria read `untested`, reason `deferred past v1`.
|
||||
|
||||
## Session 1, step 2: the empirical baseline
|
||||
|
||||
### Where it runs
|
||||
|
||||
Against the live `maven-mavend-1`, then `mavend -wipe -confirm-wipe` after the
|
||||
run. A dry run without the confirm flag comes first and its row counts are the
|
||||
eval's first artifact.
|
||||
|
||||
The live resident model is `maven-instruct-b2-Q4_K_XL`, not the Qwen3-1.7B that
|
||||
`CLAUDE.md` still names. `deploy/mavend.json` carries the switch uncommitted.
|
||||
Every number in the baseline is attributable to b2.
|
||||
|
||||
### Transport
|
||||
|
||||
`POST /api/chat` on `127.0.0.1:9201` drives a real turn through the deployed
|
||||
stack. The reply comes back in the redirect `Location` as `?q=…&r=…&t=<trace id>`.
|
||||
No new binary and no audio needed, and it covers the web reach.
|
||||
|
||||
### Readback
|
||||
|
||||
`cmd/e2eprobe`, built in a one-off `golang:1.25-trixie` container and copied in
|
||||
with `docker cp`. Both images are trixie, so glibc matches. It gives typed IPC
|
||||
JSON for `decisions`, `facts`, `notes`, `reminders`, `nudges`, `tools` and
|
||||
`delivery-attempts`.
|
||||
|
||||
Not the sqlite file. The plaintext copy at `/dev/shm/maven-plain.db` bypasses the
|
||||
contract the ledger exists to measure. Not the mavweb pages either: HTML is a
|
||||
second-hand rendering.
|
||||
|
||||
### Two probe origins
|
||||
|
||||
- `dod:<criterion id>`, derived from the ledger. Only these set a DoD verdict.
|
||||
- `field:<slice>`, the owner's real week. 25 multi-turn probes, drafted from the
|
||||
five slices, the three ugly conversations verbatim, the five existing
|
||||
scenarios, and what the box is configured for.
|
||||
|
||||
A failing `field` probe becomes an unresolved product question in
|
||||
`invariants.md` where the spec never defined the behavior. It becomes a
|
||||
missing-criterion finding against `docs/spec.md` where the spec should have
|
||||
covered it. Neither edits the spec in this pass.
|
||||
|
||||
### Three evidence kinds
|
||||
|
||||
| kind | what it is | may set `pass` |
|
||||
| --- | --- | --- |
|
||||
| `live` | `e2eprobe` and `/api/chat` against the deployed build, real model, real store rows | yes |
|
||||
| `replay` | the scenario harness, model faked, clock controlled | no |
|
||||
| `simulated` | faked clock, real code path, no deployed process | no |
|
||||
|
||||
A green `replay` with no live probe reads `verified: untested`, with the replay
|
||||
recorded as implementation evidence. The scenario harness scripts both `route`
|
||||
and `reply`, so a pass proves the wiring around the model and not the turn.
|
||||
|
||||
`simulated` is allowed only where the trigger is anchored to a wall-clock hour
|
||||
or date the pass cannot reach. The morning digest hour, the 14-day trace
|
||||
retention sweep, a weekly routine. Everything else uses a compressed live
|
||||
horizon, because `TickInterval` defaults to 60s and no override is configured.
|
||||
|
||||
### Reaches
|
||||
|
||||
`reachable` is per reach. Inbound: web, ipc. Outbound: `ntfysink`,
|
||||
`telegramsink`, `voicesink`.
|
||||
|
||||
One delivery per outbound reach, three notifications total, each carrying a fixed
|
||||
marker so a probe is not read as a real nudge. `reachable: pass` needs the
|
||||
`delivery-attempts` row plus the owner confirming arrival. `voicesink` needs a
|
||||
connected client on workpc, and records `blocked, no listener` when there is
|
||||
none.
|
||||
|
||||
### Per-criterion verdict
|
||||
|
||||
Every DoD criterion gets `pass`, `fail`, `blocked`, `untested` or `unknown`, and
|
||||
a reason distinguishing why it is not passing: code missing, wiring missing,
|
||||
configuration missing, deployment missing, external dependency unavailable,
|
||||
scenario missing, scenario fails, or implementation exists with no runtime proof.
|
||||
|
||||
Where a scenario covers a criterion, the exact step or assertion is named. Where
|
||||
none exists, the spec's `(to write)` state stands. No evidence is invented.
|
||||
|
||||
Every generated claim carries `path:line`, or runtime or scenario evidence.
|
||||
`unknown` where evidence is insufficient.
|
||||
|
||||
### Stop condition
|
||||
|
||||
Session 1 ends when every v1 DoD criterion carries a verdict with evidence
|
||||
attached, and the eval is written. No mapping, no gaps, no ranking.
|
||||
|
||||
A baseline that stops halfway leaves a ledger that looks measured and is not.
|
||||
|
||||
## Session 2: the explanation
|
||||
|
||||
Implementation mapping onto the ledger, as independent dimensions, never
|
||||
collapsed into one `implemented` boolean: `designed`, `code_present`, `wired`,
|
||||
`configured`, `deployed`, `reachable`, `verified`.
|
||||
|
||||
`docs/capabilities/invariants.md` extracts the cross-cutting rules the 51
|
||||
capabilities imply and do not state. Continuity across turns and across
|
||||
reaches, memory and correction semantics, current context and presence,
|
||||
proactive attention, interruption policy, clarification and follow-up ownership,
|
||||
degradation and honesty, authority and confirmation, privacy boundaries,
|
||||
learning from outcomes, capability composition, persistence across restart.
|
||||
|
||||
Each invariant is marked `explicit`, `implied` or `unresolved`, with evidence.
|
||||
Nothing desired is invented where the sources do not define it. An unresolved
|
||||
invariant is a product question, not a defect.
|
||||
|
||||
`docs/capabilities/gaps.md` compares responsibilities, never package names.
|
||||
Classes: capability missing, capability partial, capability exists but
|
||||
unreachable, capability exists but unverified, duplicated mechanism, missing
|
||||
shared mechanism, current architecture conflicts with target behavior, and
|
||||
architecture concern with no current product impact.
|
||||
|
||||
Every architecture concern names the capability or invariant it affects. One
|
||||
affecting none is marked non-blocking cleanup explicitly.
|
||||
|
||||
### Prioritization
|
||||
|
||||
One final list, ranked:
|
||||
|
||||
1. Prevents intended everyday use today.
|
||||
2. Makes existing behavior incorrect or unreliable.
|
||||
3. Blocks multiple capabilities.
|
||||
4. Prevents verification.
|
||||
5. Architectural cleanup with no present user impact.
|
||||
|
||||
An unwired or unreachable future defect never outranks a live user-visible
|
||||
failure because its architecture is ugly.
|
||||
|
||||
## Session 3: the viewer
|
||||
|
||||
A `capabilities` mode in the architecture viewer. The primary view is a matrix of
|
||||
capability against `designed`, `coded`, `wired`, `configured`, `deployed`,
|
||||
`reachable`, `verified`.
|
||||
|
||||
Clicking a capability shows the target DoD, implementation evidence,
|
||||
verification evidence, affected components, scenarios, blockers and unresolved
|
||||
product questions.
|
||||
|
||||
An `invariants` view shows which components participate in each cross-cutting
|
||||
rule. Target, implementation and runtime verification are distinguished visually.
|
||||
|
||||
## Where the artifacts live
|
||||
|
||||
`docs/capabilities/` becomes a declared tier in `docs/CLAUDE.md`: generated,
|
||||
regenerated from `docs/spec.md` plus a named eval, never hand-edited. The same
|
||||
row legitimises `docs/architecture/`, which is currently an undeclared
|
||||
directory.
|
||||
|
||||
The empirical run lands as one dated eval, `docs/evals/2026-08-26-capability-baseline.md`,
|
||||
frozen on the day. Every `verified` cell in the ledger cites it by path, so no
|
||||
status is an opinion.
|
||||
|
||||
## Decided, do not re-ask
|
||||
|
||||
- Three sessions, split on the causal order above.
|
||||
- Target side of the ledger first, with no implementation or verification status.
|
||||
- Two verified columns plus `simulated`. Only `live` sets `pass`.
|
||||
- Probes run against the live container, with the store wiped afterwards.
|
||||
- Row counts are read with a `mavend -wipe` dry run before anything else.
|
||||
- Readback is `e2eprobe`, not sqlite and not the web pages.
|
||||
- Both probe origins, `dod:` and `field:`, tagged.
|
||||
- Deferred capabilities get the skeleton only.
|
||||
- All three outbound reaches get one real delivery.
|
||||
- Domain is a second axis beside the spec section.
|
||||
- No fixes land in this pass.
|
||||
|
||||
## Findings recorded while planning
|
||||
|
||||
Live and reproduced, not inferred. They belong to the baseline, not to this plan.
|
||||
|
||||
- `POST /api/chat` with `что ты помнишь обо мне?` answered
|
||||
`Я не знаю вас или как вы себя называете`. Formal `вас` and `вы` on the wire,
|
||||
where `CLAUDE.md` requires informal singular `ты`. The phrasing eval passes.
|
||||
- Vikunja was never down. `vikunja-mcp` publishes `127.0.0.1:9100` only, so the
|
||||
LAN address never answers from workpc. Three sessions read a refused
|
||||
connection as an outage and filed nothing.
|
||||
- The deployed resident model is `maven-instruct-b2-Q4_K_XL`. `CLAUDE.md` still
|
||||
names Qwen3-1.7B.
|
||||
Reference in New Issue
Block a user