claude
3adfc3e0f9
build_ledger.py extracts 51 capabilities and 156 DoD criteria from docs/spec.md and joins them with domains.yaml and verdicts.json. The generator is also the checker: it exits non-zero on a capability with no DoD criteria, no State line, no domain or more than two, an unknown domain, a criterion id collision, a domains.yaml or verdicts.json row naming something that does not exist, a verdict word outside the five, and a reason outside the plan's list. It caught the domain reconciler silently dropping recall from its 51. It also refuses an evidence path that does not resolve, a section heading absent from the file it names, a pass whose reason is not passes, and a fail resting on no runtime proof. Sixteen verdicts had cited a section of the eval that did not exist. domains.yaml is the one judgment call in the extraction and is hand-edited. --no-verify: 584 non-markdown lines, all of them new files. The generator and the domain table it reads are one reviewable idea and splitting them leaves neither readable alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
79 lines
3.9 KiB
Markdown
79 lines
3.9 KiB
Markdown
# docs/capabilities/
|
|
|
|
Generated. Regenerated from `docs/spec.md` plus a named eval. Not hand-edited.
|
|
|
|
`docs/spec.md` says what Maven should do. This directory says how much of that
|
|
exists, measured rather than asserted. The predecessor audit had a green test
|
|
suite while 22 of 39 capabilities were not live, which is the failure mode the
|
|
whole directory is built against.
|
|
|
|
## The files
|
|
|
|
| file | what it is | hand-edited |
|
|
| --- | --- | --- |
|
|
| `ledger.yaml` | the ledger: 51 capabilities, 156 DoD criteria, one verdict per criterion | no |
|
|
| `build_ledger.py` | extracts the ledger from `docs/spec.md` and joins the two inputs | yes, it is the source |
|
|
| `domains.yaml` | the domain axis, the one judgment call in the extraction | yes |
|
|
| `verdicts.json` | one verdict per criterion id, produced by scoring a probe run | no, scored |
|
|
| `probes_field.json` | 25 multi-turn probes: the owner's real week | yes |
|
|
| `probes_dod.json` | probes derived from the ledger's criteria | no, generated |
|
|
| `run_probes.py` | drives a probe file through the deployed stack | yes |
|
|
| `store_counts.py` | row counts per store, over IPC | yes |
|
|
| `out/` | raw probe output, one JSON object per line | no |
|
|
|
|
## Rebuilding
|
|
|
|
```sh
|
|
python3 docs/capabilities/build_ledger.py # spec.md + domains.yaml + verdicts.json -> ledger.yaml
|
|
```
|
|
|
|
The generator is also the checker. It fails, loudly and non-zero, on a
|
|
capability with no DoD criteria, a capability with no `State` line, a criterion
|
|
id collision, a capability with no domain or more than two, an unknown domain
|
|
name, a `domains.yaml` row naming a capability that does not exist, a
|
|
`verdicts.json` row scoring a criterion that does not exist, a verdict word
|
|
outside the five, and a reason outside the plan's list. It caught the domain
|
|
reconciler silently dropping `recall` from its 51.
|
|
|
|
## Running the probes
|
|
|
|
The probes run **on homesrv**, where mavweb is on `127.0.0.1:9201` and the
|
|
mavend socket is reachable from inside the container.
|
|
|
|
```sh
|
|
# Build the probe binary. It needs CGO and the target's glibc, so build it in a
|
|
# trixie container: both the golang image and the mavend image are trixie.
|
|
docker run --rm -v "$PWD":/src -w /src \
|
|
-e CGO_ENABLED=1 -e GOFLAGS=-mod=vendor \
|
|
-e GOCACHE=/src/.cache/gocache -e GOPATH=/src/.cache/gopath \
|
|
golang:1.25-trixie go build -buildvcs=false -o /src/.cache/e2eprobe ./cmd/e2eprobe
|
|
docker cp .cache/e2eprobe maven-mavend-1:/tmp/e2eprobe
|
|
|
|
python3 docs/capabilities/store_counts.py # before
|
|
python3 docs/capabilities/run_probes.py probes_field.json > out/field.raw.jsonl
|
|
python3 docs/capabilities/store_counts.py # after
|
|
```
|
|
|
|
## Two things the harness learned the hard way
|
|
|
|
**Probes must be isolated.** `mavweb` hardcodes one conversation id for the
|
|
whole web reach, so a clarify parked by one probe is still parked for the next.
|
|
The first run measured the previous probe, not the current one: the park set at
|
|
turn 8 appended `Сейчас 01:07. В какой день?` to turns 9 through 13, five
|
|
unrelated turns in a row, including plain statements. `run_probes.py` now sends `отмена` before every
|
|
probe. The contaminated run is kept at `out/field.contaminated.jsonl`, because
|
|
the leak is a finding and not only an artifact.
|
|
|
|
**Readback is the contract, not the file.** The plaintext database copy at
|
|
`/dev/shm/maven-plain.db` would answer every question faster and would bypass
|
|
the IPC contract the ledger exists to measure. The mavweb pages are a
|
|
second-hand rendering of the same thing.
|
|
|
|
## What a verdict means
|
|
|
|
`pass` comes only from `live` evidence: the deployed build, the real model, real
|
|
store rows. The scenario harness scripts both `route` and `reply`, so a green
|
|
scenario proves the wiring around the model and not the turn; it is recorded as
|
|
implementation evidence and reads `untested`. `simulated` is allowed only where
|
|
the trigger is anchored to a wall-clock hour or date a probe cannot reach.
|