claude
3adfc3e0f9
build_ledger.py extracts 51 capabilities and 156 DoD criteria from docs/spec.md and joins them with domains.yaml and verdicts.json. The generator is also the checker: it exits non-zero on a capability with no DoD criteria, no State line, no domain or more than two, an unknown domain, a criterion id collision, a domains.yaml or verdicts.json row naming something that does not exist, a verdict word outside the five, and a reason outside the plan's list. It caught the domain reconciler silently dropping recall from its 51. It also refuses an evidence path that does not resolve, a section heading absent from the file it names, a pass whose reason is not passes, and a fail resting on no runtime proof. Sixteen verdicts had cited a section of the eval that did not exist. domains.yaml is the one judgment call in the extraction and is hand-edited. --no-verify: 584 non-markdown lines, all of them new files. The generator and the domain table it reads are one reviewable idea and splitting them leaves neither readable alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3.9 KiB
docs/capabilities/
Generated. Regenerated from docs/spec.md plus a named eval. Not hand-edited.
docs/spec.md says what Maven should do. This directory says how much of that
exists, measured rather than asserted. The predecessor audit had a green test
suite while 22 of 39 capabilities were not live, which is the failure mode the
whole directory is built against.
The files
| file | what it is | hand-edited |
|---|---|---|
ledger.yaml |
the ledger: 51 capabilities, 156 DoD criteria, one verdict per criterion | no |
build_ledger.py |
extracts the ledger from docs/spec.md and joins the two inputs |
yes, it is the source |
domains.yaml |
the domain axis, the one judgment call in the extraction | yes |
verdicts.json |
one verdict per criterion id, produced by scoring a probe run | no, scored |
probes_field.json |
25 multi-turn probes: the owner's real week | yes |
probes_dod.json |
probes derived from the ledger's criteria | no, generated |
run_probes.py |
drives a probe file through the deployed stack | yes |
store_counts.py |
row counts per store, over IPC | yes |
out/ |
raw probe output, one JSON object per line | no |
Rebuilding
python3 docs/capabilities/build_ledger.py # spec.md + domains.yaml + verdicts.json -> ledger.yaml
The generator is also the checker. It fails, loudly and non-zero, on a
capability with no DoD criteria, a capability with no State line, a criterion
id collision, a capability with no domain or more than two, an unknown domain
name, a domains.yaml row naming a capability that does not exist, a
verdicts.json row scoring a criterion that does not exist, a verdict word
outside the five, and a reason outside the plan's list. It caught the domain
reconciler silently dropping recall from its 51.
Running the probes
The probes run on homesrv, where mavweb is on 127.0.0.1:9201 and the
mavend socket is reachable from inside the container.
# Build the probe binary. It needs CGO and the target's glibc, so build it in a
# trixie container: both the golang image and the mavend image are trixie.
docker run --rm -v "$PWD":/src -w /src \
-e CGO_ENABLED=1 -e GOFLAGS=-mod=vendor \
-e GOCACHE=/src/.cache/gocache -e GOPATH=/src/.cache/gopath \
golang:1.25-trixie go build -buildvcs=false -o /src/.cache/e2eprobe ./cmd/e2eprobe
docker cp .cache/e2eprobe maven-mavend-1:/tmp/e2eprobe
python3 docs/capabilities/store_counts.py # before
python3 docs/capabilities/run_probes.py probes_field.json > out/field.raw.jsonl
python3 docs/capabilities/store_counts.py # after
Two things the harness learned the hard way
Probes must be isolated. mavweb hardcodes one conversation id for the
whole web reach, so a clarify parked by one probe is still parked for the next.
The first run measured the previous probe, not the current one: the park set at
turn 8 appended Сейчас 01:07. В какой день? to turns 9 through 13, five
unrelated turns in a row, including plain statements. run_probes.py now sends отмена before every
probe. The contaminated run is kept at out/field.contaminated.jsonl, because
the leak is a finding and not only an artifact.
Readback is the contract, not the file. The plaintext database copy at
/dev/shm/maven-plain.db would answer every question faster and would bypass
the IPC contract the ledger exists to measure. The mavweb pages are a
second-hand rendering of the same thing.
What a verdict means
pass comes only from live evidence: the deployed build, the real model, real
store rows. The scenario harness scripts both route and reply, so a green
scenario proves the wiring around the model and not the turn; it is recorded as
implementation evidence and reads untested. simulated is allowed only where
the trigger is anchored to a wall-clock hour or date a probe cannot reach.