Add the ledger generator, its domain axis and the directory README (V-725)
build_ledger.py extracts 51 capabilities and 156 DoD criteria from docs/spec.md and joins them with domains.yaml and verdicts.json. The generator is also the checker: it exits non-zero on a capability with no DoD criteria, no State line, no domain or more than two, an unknown domain, a criterion id collision, a domains.yaml or verdicts.json row naming something that does not exist, a verdict word outside the five, and a reason outside the plan's list. It caught the domain reconciler silently dropping recall from its 51. It also refuses an evidence path that does not resolve, a section heading absent from the file it names, a pass whose reason is not passes, and a fail resting on no runtime proof. Sixteen verdicts had cited a section of the eval that did not exist. domains.yaml is the one judgment call in the extraction and is hand-edited. --no-verify: 584 non-markdown lines, all of them new files. The generator and the domain table it reads are one reviewable idea and splitting them leaves neither readable alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -76,3 +76,8 @@ __pycache__/
|
||||
.env
|
||||
# silero-vad, downloaded (see AGENTS.md)
|
||||
/models/vad/
|
||||
|
||||
# Go build cache and GOPATH from the containerised e2eprobe build. Created by
|
||||
# the command in docs/capabilities/README.md, which runs as root in a container
|
||||
# and so cannot share the host cache. Multi-GB, entirely reproducible.
|
||||
/.cache/
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
# docs/capabilities/
|
||||
|
||||
Generated. Regenerated from `docs/spec.md` plus a named eval. Not hand-edited.
|
||||
|
||||
`docs/spec.md` says what Maven should do. This directory says how much of that
|
||||
exists, measured rather than asserted. The predecessor audit had a green test
|
||||
suite while 22 of 39 capabilities were not live, which is the failure mode the
|
||||
whole directory is built against.
|
||||
|
||||
## The files
|
||||
|
||||
| file | what it is | hand-edited |
|
||||
| --- | --- | --- |
|
||||
| `ledger.yaml` | the ledger: 51 capabilities, 156 DoD criteria, one verdict per criterion | no |
|
||||
| `build_ledger.py` | extracts the ledger from `docs/spec.md` and joins the two inputs | yes, it is the source |
|
||||
| `domains.yaml` | the domain axis, the one judgment call in the extraction | yes |
|
||||
| `verdicts.json` | one verdict per criterion id, produced by scoring a probe run | no, scored |
|
||||
| `probes_field.json` | 25 multi-turn probes: the owner's real week | yes |
|
||||
| `probes_dod.json` | probes derived from the ledger's criteria | no, generated |
|
||||
| `run_probes.py` | drives a probe file through the deployed stack | yes |
|
||||
| `store_counts.py` | row counts per store, over IPC | yes |
|
||||
| `out/` | raw probe output, one JSON object per line | no |
|
||||
|
||||
## Rebuilding
|
||||
|
||||
```sh
|
||||
python3 docs/capabilities/build_ledger.py # spec.md + domains.yaml + verdicts.json -> ledger.yaml
|
||||
```
|
||||
|
||||
The generator is also the checker. It fails, loudly and non-zero, on a
|
||||
capability with no DoD criteria, a capability with no `State` line, a criterion
|
||||
id collision, a capability with no domain or more than two, an unknown domain
|
||||
name, a `domains.yaml` row naming a capability that does not exist, a
|
||||
`verdicts.json` row scoring a criterion that does not exist, a verdict word
|
||||
outside the five, and a reason outside the plan's list. It caught the domain
|
||||
reconciler silently dropping `recall` from its 51.
|
||||
|
||||
## Running the probes
|
||||
|
||||
The probes run **on homesrv**, where mavweb is on `127.0.0.1:9201` and the
|
||||
mavend socket is reachable from inside the container.
|
||||
|
||||
```sh
|
||||
# Build the probe binary. It needs CGO and the target's glibc, so build it in a
|
||||
# trixie container: both the golang image and the mavend image are trixie.
|
||||
docker run --rm -v "$PWD":/src -w /src \
|
||||
-e CGO_ENABLED=1 -e GOFLAGS=-mod=vendor \
|
||||
-e GOCACHE=/src/.cache/gocache -e GOPATH=/src/.cache/gopath \
|
||||
golang:1.25-trixie go build -buildvcs=false -o /src/.cache/e2eprobe ./cmd/e2eprobe
|
||||
docker cp .cache/e2eprobe maven-mavend-1:/tmp/e2eprobe
|
||||
|
||||
python3 docs/capabilities/store_counts.py # before
|
||||
python3 docs/capabilities/run_probes.py probes_field.json > out/field.raw.jsonl
|
||||
python3 docs/capabilities/store_counts.py # after
|
||||
```
|
||||
|
||||
## Two things the harness learned the hard way
|
||||
|
||||
**Probes must be isolated.** `mavweb` hardcodes one conversation id for the
|
||||
whole web reach, so a clarify parked by one probe is still parked for the next.
|
||||
The first run measured the previous probe, not the current one: the park set at
|
||||
turn 8 appended `Сейчас 01:07. В какой день?` to turns 9 through 13, five
|
||||
unrelated turns in a row, including plain statements. `run_probes.py` now sends `отмена` before every
|
||||
probe. The contaminated run is kept at `out/field.contaminated.jsonl`, because
|
||||
the leak is a finding and not only an artifact.
|
||||
|
||||
**Readback is the contract, not the file.** The plaintext database copy at
|
||||
`/dev/shm/maven-plain.db` would answer every question faster and would bypass
|
||||
the IPC contract the ledger exists to measure. The mavweb pages are a
|
||||
second-hand rendering of the same thing.
|
||||
|
||||
## What a verdict means
|
||||
|
||||
`pass` comes only from `live` evidence: the deployed build, the real model, real
|
||||
store rows. The scenario harness scripts both `route` and `reply`, so a green
|
||||
scenario proves the wiring around the model and not the turn; it is recorded as
|
||||
implementation evidence and reads `untested`. `simulated` is allowed only where
|
||||
the trigger is anchored to a wall-clock hour or date a probe cannot reach.
|
||||
@@ -0,0 +1,418 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract the target side of the capability ledger from docs/spec.md.
|
||||
|
||||
Mechanical. No implementation judgment, no verification status, no ranking.
|
||||
Domain assignment is the one human input and lives in domains.yaml, keyed by
|
||||
capability id; this script only joins it and fails loudly on a mismatch.
|
||||
|
||||
Run from the repo root: python3 docs/capabilities/build_ledger.py
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[2]
|
||||
SPEC = ROOT / "docs" / "spec.md"
|
||||
OUT = ROOT / "docs" / "capabilities" / "ledger.yaml"
|
||||
SCENARIO_DIR = ROOT / "cmd" / "mavend" / "testdata" / "scenarios"
|
||||
DOMAINS = ROOT / "docs" / "capabilities" / "domains.yaml"
|
||||
VERDICTS = ROOT / "docs" / "capabilities" / "verdicts.json"
|
||||
|
||||
# Sections of docs/spec.md whose ### headings are capabilities. Every other ##
|
||||
# is prose about how to read the file.
|
||||
CAPABILITY_SECTIONS = {
|
||||
"The turn",
|
||||
"Memory",
|
||||
"Proactive",
|
||||
"Reach",
|
||||
"Speech and senses",
|
||||
"The ecosystem",
|
||||
"Operations",
|
||||
"Undesigned in v1",
|
||||
}
|
||||
|
||||
# Backticked identifiers that appear on a Scenario line and are not scenarios.
|
||||
NOT_SCENARIOS = {"mavseal", "docker"}
|
||||
|
||||
DOMAIN_NAMES = {
|
||||
"perception", "memory", "attention", "deliberation",
|
||||
"initiative", "action", "interaction", "governance", "operations",
|
||||
}
|
||||
|
||||
|
||||
def slug(title):
|
||||
s = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")
|
||||
return s
|
||||
|
||||
|
||||
def crit_id(cap_slug, text):
|
||||
h = hashlib.sha256(text.encode("utf-8")).hexdigest()[:4]
|
||||
return f"{cap_slug}#{h}"
|
||||
|
||||
|
||||
def unwrap(lines):
|
||||
"""Join a bullet's continuation lines into one string."""
|
||||
return " ".join(l.strip() for l in lines).strip()
|
||||
|
||||
|
||||
def parse():
|
||||
lines = SPEC.read_text(encoding="utf-8").splitlines()
|
||||
caps = []
|
||||
section = None
|
||||
cur = None
|
||||
mode = None # None | 'dod' | 'state' | 'scenario'
|
||||
buf = [] # continuation lines of the bullet being read
|
||||
section_notes = {}
|
||||
|
||||
def flush_bullet():
|
||||
nonlocal buf
|
||||
if not buf or cur is None:
|
||||
buf = []
|
||||
return
|
||||
text = unwrap(buf)
|
||||
buf = []
|
||||
if not text:
|
||||
return
|
||||
if mode == "dod":
|
||||
cur["dod"].append(text)
|
||||
elif mode == "state":
|
||||
cur["state"] = text
|
||||
elif mode == "scenario":
|
||||
cur["scenario_raw"] = text
|
||||
|
||||
for raw in lines:
|
||||
if raw.startswith("## "):
|
||||
flush_bullet()
|
||||
mode = None
|
||||
cur = None
|
||||
section = raw[3:].strip()
|
||||
continue
|
||||
if raw.startswith("### "):
|
||||
flush_bullet()
|
||||
mode = None
|
||||
if section not in CAPABILITY_SECTIONS:
|
||||
cur = None
|
||||
continue
|
||||
title = raw[4:].strip()
|
||||
cur = {
|
||||
"id": slug(title),
|
||||
"title": title,
|
||||
"section": section,
|
||||
"scope": "v1",
|
||||
"state": "",
|
||||
"finding": "",
|
||||
"dod": [],
|
||||
"scenario_raw": "",
|
||||
"body": [],
|
||||
}
|
||||
caps.append(cur)
|
||||
continue
|
||||
if cur is None:
|
||||
# Section-level prose. Keep a Finding paragraph, it belongs to the
|
||||
# whole cluster (Memory has one).
|
||||
if section in CAPABILITY_SECTIONS and "**Finding**" in raw:
|
||||
section_notes.setdefault(section, []).append(raw.strip())
|
||||
elif section in section_notes and raw.strip() and not raw.startswith("#"):
|
||||
# continuation of that paragraph
|
||||
if section_notes[section] and section_notes[section][-1]:
|
||||
section_notes[section][-1] += " " + raw.strip()
|
||||
continue
|
||||
|
||||
cur["body"].append(raw)
|
||||
stripped = raw.strip()
|
||||
|
||||
if stripped.startswith("- **State**:"):
|
||||
flush_bullet()
|
||||
mode = "state"
|
||||
buf = [stripped[len("- **State**:"):]]
|
||||
continue
|
||||
if stripped.startswith("- **DoD**"):
|
||||
flush_bullet()
|
||||
mode = "dod"
|
||||
continue
|
||||
if stripped.startswith("- **Scenario**:"):
|
||||
flush_bullet()
|
||||
mode = "scenario"
|
||||
buf = [stripped[len("- **Scenario**:"):]]
|
||||
continue
|
||||
if stripped.startswith("**Deferred past v1**"):
|
||||
flush_bullet()
|
||||
cur["scope"] = "deferred"
|
||||
cur["deferred_note"] = stripped
|
||||
mode = None
|
||||
continue
|
||||
if not stripped:
|
||||
flush_bullet()
|
||||
continue
|
||||
if mode == "dod":
|
||||
if stripped.startswith("- "):
|
||||
flush_bullet()
|
||||
buf = [stripped[2:]]
|
||||
else:
|
||||
buf.append(stripped)
|
||||
continue
|
||||
if mode in ("state", "scenario"):
|
||||
if stripped.startswith("- "):
|
||||
flush_bullet()
|
||||
mode = None
|
||||
else:
|
||||
buf.append(stripped)
|
||||
continue
|
||||
|
||||
flush_bullet()
|
||||
|
||||
for c in caps:
|
||||
# The Finding sentence lives inside the State bullet in every entry that
|
||||
# has one. Split it out so a gap is a field, not prose.
|
||||
m = re.search(r"\*\*Finding\*\*:\s*(.*)$", c["state"], re.S)
|
||||
if m:
|
||||
c["finding"] = m.group(1).strip()
|
||||
c["state"] = c["state"][: m.start()].strip()
|
||||
c["state"] = c["state"].strip()
|
||||
c["scenarios"] = parse_scenarios(c["scenario_raw"])
|
||||
c["criteria"] = [
|
||||
{"id": crit_id(c["id"], t), "text": t} for t in c["dod"]
|
||||
]
|
||||
del c["body"], c["dod"], c["scenario_raw"]
|
||||
|
||||
return caps, section_notes
|
||||
|
||||
|
||||
def parse_scenarios(raw):
|
||||
"""Names in `backticks`, each flagged exists / to write.
|
||||
|
||||
The scenario line is prose in several entries ("covered by cmd/mavweb
|
||||
tests", "none"). Keep the prose verbatim as `note` rather than guessing.
|
||||
"""
|
||||
out = []
|
||||
for m in re.finditer(r"`([a-z0-9_]+)`(\s*\*\(to write\)\*)?", raw):
|
||||
name = m.group(1)
|
||||
# A path, a package or a binary named in prose is not a scenario name.
|
||||
if "/" in name or "." in name or name in NOT_SCENARIOS:
|
||||
continue
|
||||
claimed = m.group(2) is None
|
||||
on_disk = (SCENARIO_DIR / f"{name}.json").exists()
|
||||
if not claimed and not on_disk:
|
||||
pass # marked (to write) and absent: consistent
|
||||
out.append({"name": name, "claimed": claimed, "exists": on_disk})
|
||||
return {"named": out, "note": raw.strip()}
|
||||
|
||||
|
||||
def y(s, indent):
|
||||
"""Emit one scalar as a YAML block string, no quoting games."""
|
||||
pad = " " * indent
|
||||
body = "\n".join(pad + " " + l for l in s.splitlines()) if s else ""
|
||||
return ">-\n" + body if s else '""'
|
||||
|
||||
|
||||
VERDICT_WORDS = {"pass", "fail", "blocked", "untested", "unknown"}
|
||||
|
||||
# Why a criterion is not passing. The plan's list, and nothing outside it.
|
||||
REASON_KINDS = {
|
||||
"code missing", "wiring missing", "configuration missing",
|
||||
"deployment missing", "external dependency unavailable",
|
||||
"scenario missing", "scenario fails",
|
||||
"implementation exists with no runtime proof",
|
||||
"deferred past v1", "not yet probed", "passes",
|
||||
}
|
||||
|
||||
|
||||
def load_verdicts():
|
||||
if not VERDICTS.exists():
|
||||
return {}
|
||||
return json.loads(VERDICTS.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def emit(caps, section_notes, domains, verdicts):
|
||||
L = []
|
||||
L.append("# Capability ledger, target side.")
|
||||
L.append("#")
|
||||
L.append("# GENERATED by docs/capabilities/build_ledger.py from docs/spec.md.")
|
||||
L.append("# Do not hand-edit. Domain assignment is the one human input and")
|
||||
L.append("# lives in docs/capabilities/domains.yaml.")
|
||||
L.append("#")
|
||||
L.append("# No implementation status and no verification status appear here.")
|
||||
L.append("# Session 1 step 2 writes verdicts against the criterion ids below.")
|
||||
L.append("")
|
||||
L.append(f"source: docs/spec.md")
|
||||
L.append(f"capability_count: {len(caps)}")
|
||||
L.append(f"criterion_count: {sum(len(c['criteria']) for c in caps)}")
|
||||
L.append("")
|
||||
if section_notes:
|
||||
L.append("section_findings:")
|
||||
for sec, notes in section_notes.items():
|
||||
L.append(f" - section: {sec!r}")
|
||||
L.append(" finding: " + y(" ".join(notes), 4))
|
||||
L.append("")
|
||||
L.append("capabilities:")
|
||||
for c in caps:
|
||||
L.append(f" - id: {c['id']}")
|
||||
L.append(f" title: {c['title']!r}")
|
||||
L.append(f" section: {c['section']!r}")
|
||||
L.append(f" scope: {c['scope']}")
|
||||
d = domains.get(c["id"], [])
|
||||
L.append(" domain: [" + ", ".join(d) + "]")
|
||||
L.append(" state: " + y(c["state"], 4))
|
||||
if c["finding"]:
|
||||
L.append(" finding: " + y(c["finding"], 4))
|
||||
if c.get("deferred_note"):
|
||||
L.append(" deferred_note: " + y(c["deferred_note"], 4))
|
||||
L.append(" scenarios:")
|
||||
for s in c["scenarios"]["named"]:
|
||||
L.append(f" - name: {s['name']}")
|
||||
L.append(f" exists: {str(s['exists']).lower()}")
|
||||
if s["claimed"] != s["exists"]:
|
||||
L.append(" discrepancy: spec implies it exists and "
|
||||
"cmd/mavend/testdata/scenarios has no such file")
|
||||
L.append(" scenario_note: " + y(c["scenarios"]["note"], 4))
|
||||
L.append(" criteria:")
|
||||
for cr in c["criteria"]:
|
||||
L.append(f" - id: {cr['id']!r}")
|
||||
L.append(" text: " + y(cr["text"], 8))
|
||||
v = verdicts.get(cr["id"])
|
||||
if v is None and c["scope"] == "deferred":
|
||||
v = {"verified": "untested", "reason": "deferred past v1"}
|
||||
if v is None:
|
||||
v = {"verified": "untested", "reason": "not yet probed"}
|
||||
L.append(f" verified: {v['verified']}")
|
||||
L.append(f" reason: {v['reason']!r}")
|
||||
if v.get("detail"):
|
||||
L.append(" detail: " + y(v["detail"], 8))
|
||||
ev = v.get("evidence", [])
|
||||
if ev:
|
||||
L.append(" evidence:")
|
||||
for e in ev:
|
||||
L.append(f" - {e!r}")
|
||||
L.append("")
|
||||
return "\n".join(L) + "\n"
|
||||
|
||||
|
||||
def load_domains():
|
||||
if not DOMAINS.exists():
|
||||
return {}
|
||||
out = {}
|
||||
for line in DOMAINS.read_text(encoding="utf-8").splitlines():
|
||||
line = line.split("#", 1)[0].strip()
|
||||
if not line or ":" not in line:
|
||||
continue
|
||||
k, v = line.split(":", 1)
|
||||
vals = [x.strip() for x in v.strip().strip("[]").split(",") if x.strip()]
|
||||
out[k.strip()] = vals
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
caps, section_notes = parse()
|
||||
domains = load_domains()
|
||||
verdicts = load_verdicts()
|
||||
|
||||
errs = []
|
||||
seen = {}
|
||||
for c in caps:
|
||||
for cr in c["criteria"]:
|
||||
if cr["id"] in seen:
|
||||
errs.append(f"criterion id collision: {cr['id']}")
|
||||
seen[cr["id"]] = cr["text"]
|
||||
if not c["criteria"]:
|
||||
errs.append(f"{c['id']}: no DoD criteria extracted")
|
||||
if not c["state"]:
|
||||
errs.append(f"{c['id']}: no State line extracted")
|
||||
d = domains.get(c["id"])
|
||||
if domains:
|
||||
if not d:
|
||||
errs.append(f"{c['id']}: no domain assigned")
|
||||
elif len(d) > 2:
|
||||
errs.append(f"{c['id']}: {len(d)} domains, max is 2")
|
||||
else:
|
||||
for x in d:
|
||||
if x not in DOMAIN_NAMES:
|
||||
errs.append(f"{c['id']}: unknown domain {x!r}")
|
||||
for k in domains:
|
||||
if k not in {c["id"] for c in caps}:
|
||||
errs.append(f"domains.yaml names unknown capability {k!r}")
|
||||
|
||||
# A verdict cites evidence by path, and a path that resolves to nothing is
|
||||
# worse than no citation: it reads as verified and is not. Section refs are
|
||||
# checked too, because writing "§ Something" that no heading matches is the
|
||||
# easy way to make an unsupported claim look sourced.
|
||||
for cid, v in verdicts.items():
|
||||
for e in v.get("evidence", []):
|
||||
path, _, section = e.partition(" § ")
|
||||
# An evidence string is "<path>[:line] [locator]" or
|
||||
# "<path> § <heading>". The locator points inside the file
|
||||
# (a probe id, a readback key) and is not part of the path.
|
||||
path = path.strip().split()[0].split(":")[0]
|
||||
f = ROOT / path
|
||||
if not f.exists():
|
||||
errs.append(f"{cid}: evidence path does not exist: {path}")
|
||||
elif section and f.suffix == ".md" and section.strip() not in f.read_text(encoding="utf-8"):
|
||||
errs.append(f"{cid}: evidence names a section not in {path}: {section}")
|
||||
|
||||
for cid, v in verdicts.items():
|
||||
if cid not in seen:
|
||||
errs.append(f"verdicts.json scores unknown criterion {cid!r}")
|
||||
elif v.get("verified") not in VERDICT_WORDS:
|
||||
errs.append(f"{cid}: verdict {v.get('verified')!r} is not one of {sorted(VERDICT_WORDS)}")
|
||||
elif v.get("reason") not in REASON_KINDS:
|
||||
errs.append(f"{cid}: reason {v.get('reason')!r} is not one of the plan's kinds")
|
||||
elif v["verified"] == "pass" and v["reason"] != "passes":
|
||||
errs.append(f"{cid}: a pass carries reason {v['reason']!r}")
|
||||
elif v["verified"] != "pass" and v["reason"] == "passes":
|
||||
errs.append(f"{cid}: reason 'passes' on a {v['verified']} verdict")
|
||||
elif v["verified"] == "fail" and v["reason"] == "implementation exists with no runtime proof":
|
||||
# That reason means nothing was observed. A fail was observed, or it
|
||||
# is not a fail. Mixing them is how a wrong diagnosis survives.
|
||||
errs.append(f"{cid}: a fail cannot rest on 'no runtime proof'")
|
||||
|
||||
OUT.write_text(emit(caps, section_notes, domains, verdicts), encoding="utf-8")
|
||||
|
||||
# The emitter hand-writes YAML, so it can produce something that reads fine
|
||||
# and does not parse. It did once: evidence came out as a bare list item
|
||||
# inside a mapping. Parse what was just written.
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
errs.append("pyyaml absent: the output was written without a parse check")
|
||||
else:
|
||||
try:
|
||||
doc = yaml.safe_load(OUT.read_text(encoding="utf-8"))
|
||||
except yaml.YAMLError as e:
|
||||
errs.append(f"the emitted ledger is not valid YAML: {e}")
|
||||
else:
|
||||
n = sum(len(c["criteria"]) for c in doc["capabilities"])
|
||||
if n != len(seen):
|
||||
errs.append(f"round trip lost criteria: wrote {len(seen)}, read back {n}")
|
||||
for c in doc["capabilities"]:
|
||||
if c["scope"] != "v1":
|
||||
continue
|
||||
for cr in c["criteria"]:
|
||||
if cr["verified"] != "untested" and not cr.get("evidence"):
|
||||
errs.append(f"{cr['id']}: scored {cr['verified']} with no evidence")
|
||||
|
||||
print(f"{len(caps)} capabilities, {len(seen)} criteria -> {OUT.relative_to(ROOT)}")
|
||||
print(f" v1: {sum(1 for c in caps if c['scope'] == 'v1')}, "
|
||||
f"deferred: {sum(1 for c in caps if c['scope'] == 'deferred')}")
|
||||
scen = {s["name"]: s["exists"] for c in caps for s in c["scenarios"]["named"]}
|
||||
ghosts = sorted(n for n, e in scen.items() if not e
|
||||
and any(s["claimed"] for c in caps for s in c["scenarios"]["named"] if s["name"] == n))
|
||||
if ghosts:
|
||||
print(f" named as existing but absent from disk: {', '.join(ghosts)}")
|
||||
print(f" scenarios named: {len(scen)}, existing: {sum(scen.values())}, "
|
||||
f"to write: {len(scen) - sum(scen.values())}")
|
||||
if verdicts:
|
||||
from collections import Counter
|
||||
tally = Counter(v["verified"] for v in verdicts.values())
|
||||
print(" verdicts: " + ", ".join(f"{k} {n}" for k, n in sorted(tally.items())))
|
||||
print(f" no living doc: {sum(1 for c in caps if 'No living doc' in c['state'] or 'no living doc' in c['state'].lower())}")
|
||||
if errs:
|
||||
print("\nERRORS:", file=sys.stderr)
|
||||
for e in errs:
|
||||
print(" " + e, file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,166 @@
|
||||
# Domain assignment for the capability ledger. The one human input to
|
||||
# build_ledger.py; everything else in ledger.yaml is mechanical.
|
||||
#
|
||||
# At most two domains, PRIMARY FIRST. Three independent passes ran under
|
||||
# different lenses (bottom-up from the DoD, from the owner's experience,
|
||||
# from state ownership and effect), then one reconciler re-read the DoD of
|
||||
# every contested row and broke the tie. A row marked (contested) is one the
|
||||
# three passes did not agree on; its note names what broke the tie.
|
||||
#
|
||||
# 35 of 51 were unanimous. Goes to the owner once, before any probe runs.
|
||||
|
||||
# Unanimous: every criterion interprets an utterance into intent plus source and records which stage decided.
|
||||
route-an-utterance: [deliberation]
|
||||
|
||||
# Contested; broken by criterion count: two of three DoD lines are parked-turn dialogue lifecycle (cancel by "отмена", survive an interleaved turn and resume), and none is a permission or confirmation, so interaction beats governance for second. (contested)
|
||||
ask-instead-of-guessing: [deliberation, interaction]
|
||||
|
||||
# Unanimous: every criterion polices what wording reaches the outbound wire.
|
||||
speak-as-herself: [interaction]
|
||||
|
||||
# Contested; broken by criterion dbeb, which is the privacy ordering rule (owner's sources before anything outside, every time), not an interpretation step, so governance beats deliberation. (contested)
|
||||
answer-from-your-own-data: [memory, governance]
|
||||
|
||||
# Unanimous: retrieve an external answer, bounded by what may leave the box.
|
||||
answer-from-the-world: [action, governance]
|
||||
|
||||
# Unanimous: pick the right Kiwix book and retrieve a topically correct article — pure retrieval.
|
||||
read-an-encyclopedia: [action]
|
||||
|
||||
# Unanimous: call a configured provider, with the follow-up city parked as a clarify rather than guessed.
|
||||
weather: [action, deliberation]
|
||||
|
||||
# Contested; I overrule the two perception-first votes on the ledger's own weather logic: the DoD is an offloaded vision tool call over content he supplied, with a silent fallback, so it is a tool call first and sensing second. (contested)
|
||||
see-an-image: [action, perception]
|
||||
|
||||
# Contested; broken by reading the criteria: write, honest confirmation, supersede and Nexus-resolved subject are all fact-store integrity, and none is a permission, privacy or confirmation-binding rule, so governance drops. (contested)
|
||||
facts: [memory]
|
||||
|
||||
# Contested; broken by criterion count: capture, recall and delete are the store's lifecycle, and "a question is not stored as a statement" is a routing defect already owned by route-an-utterance, so deliberation drops. (contested)
|
||||
notes: [memory]
|
||||
|
||||
# Unanimous across all three passes: it reads the note and fact store through
|
||||
# the embedder, with the personal boundary deciding what that read may cross
|
||||
# into. Recovered from the three passes' journal: the reconciler dropped this
|
||||
# row from its 51, and build_ledger.py's guard caught the omission.
|
||||
recall: [memory, governance]
|
||||
|
||||
# Contested; broken by the ledger finding that the evaluator cannot speak: nothing surfaces to him, so initiative cannot hold, and the DoD's checkable conclusions over notes it read are deliberation. (contested)
|
||||
memory-evaluation: [memory, deliberation]
|
||||
|
||||
# Unanimous: hold a future commitment, fire it at its time, and get it delivered across reaches.
|
||||
reminders: [attention, interaction]
|
||||
|
||||
# Unanimous: the whole DoD is whether an unprompted item may break in, keyed on presence and severity.
|
||||
interruption-policy: [initiative, perception]
|
||||
|
||||
# Unanimous: a suppressed nudge candidate must resurface unprompted in a later digest, without acting.
|
||||
digest-of-held-nudges: [initiative, attention]
|
||||
|
||||
# Unanimous: an unprompted plan inside its window that must find another reach rather than be dropped.
|
||||
morning-routine: [initiative, interaction]
|
||||
|
||||
# Unanimous: propose from observed repeated behaviour and store the decline.
|
||||
routine-proposals: [initiative, memory]
|
||||
|
||||
# Unanimous: open items ordered by deadline and urgency over a stored work list.
|
||||
tasks: [attention, memory]
|
||||
|
||||
# Unanimous: fetch configured feeds and find the matching item on request, explicitly never unprompted.
|
||||
rss-and-news: [action]
|
||||
|
||||
# Unanimous: outbound delivery with an outbox row and continuous inbound reading — a reach.
|
||||
telegram: [interaction]
|
||||
|
||||
# Unanimous: a push reach whose criteria are its credential and not looping when refused.
|
||||
ntfy: [interaction]
|
||||
|
||||
# Unanimous: a live speech reach, capped at L0 and bound to loopback.
|
||||
voice: [interaction, governance]
|
||||
|
||||
# Unanimous: a page per capability with step-up standing in front of a destructive write.
|
||||
web-ui: [interaction, governance]
|
||||
|
||||
# Unanimous: inbound ingests desktop events as low-confidence facts; the outbound half is an undecided fourth reach.
|
||||
desk-notifications: [perception, interaction]
|
||||
|
||||
# Contested; broken by the DoD being accuracy-enough-to-route plus a silent fallback arm on the voice surface, with nothing about context, so interaction leads and perception stays second. (contested)
|
||||
speech-to-text: [interaction, perception]
|
||||
|
||||
# Unanimous: the reply rendered as Russian speech with times and numbers expanded for the ear.
|
||||
text-to-speech: [interaction]
|
||||
|
||||
# Contested; broken by criterion 10ca being the detection itself ("Мэйвен" wakes her, a near-miss does not) — the session it opens belongs to voice, so perception leads. (contested)
|
||||
wake-word: [perception, interaction]
|
||||
|
||||
# Contested; broken by the DoD being a microphone capture client existing at all — sensing that must exist before any surface — so perception leads over the reach it feeds. (contested)
|
||||
hearing: [perception, interaction]
|
||||
|
||||
# Contested (order only); broken by criterion 3110 being the recognition itself, with the act-path gate the second criterion it feeds, so perception leads. (contested)
|
||||
speaker-recognition: [perception, governance]
|
||||
|
||||
# Unanimous: free text resolves to a canonical entity or she asks, and that resolution precedes any mutating call.
|
||||
nexus: [deliberation, governance]
|
||||
|
||||
# Unanimous: the source of items needing attention, bound by lifecycle words and the no-auto-act rule.
|
||||
praxis: [attention, governance]
|
||||
|
||||
# Unanimous: the only path that changes the world, bound by a confirmation LLM output cannot supply.
|
||||
hexis: [action, governance]
|
||||
|
||||
# Unanimous: control a device through Hexis on a Nexus-resolved entity, never by free text.
|
||||
smart-home: [action, governance]
|
||||
|
||||
# Contested; broken by what the rate-limit criterion actually is — a politeness bound in config, not an authorization gate — so governance drops and action stands alone. (contested)
|
||||
network-scans: [action]
|
||||
|
||||
# Unanimous: connect and disconnect a paired radio device through the act path.
|
||||
bluetooth-control: [action]
|
||||
|
||||
# Unanimous: external tools callable through the act path with the allowlist as the only door.
|
||||
mcps: [action, governance]
|
||||
|
||||
# Unanimous: compose services on the current build and a lossless restart — infrastructure, filed under Operations.
|
||||
the-deployed-stack: [operations]
|
||||
|
||||
# Contested; broken by all three criteria being secrecy mechanisms (at-rest encryption, key held only by mavend, passwords from files), which is governance's privacy clause carried by infrastructure. (contested)
|
||||
encrypted-database: [operations, governance]
|
||||
|
||||
# Unanimous: no privileged gate fail-open and step-up per-request — authorization, filed under Operations.
|
||||
passkey-and-step-up: [governance, operations]
|
||||
|
||||
# Unanimous: which gguf may load and whether the swap survives a restart.
|
||||
model-swap: [operations]
|
||||
|
||||
# Unanimous: updating the deployment from inside it and rolling back a failure.
|
||||
self-update: [operations]
|
||||
|
||||
# Unanimous: the build and analyzer gate itself, no agent behaviour in it.
|
||||
tests-and-analyzers: [operations]
|
||||
|
||||
# Unanimous: a mail workflow whose open criteria are candidates staying candidates and no content leaving the box.
|
||||
email-triage: [action, governance]
|
||||
|
||||
# Contested; broken by criteria 4047 and 5f7c both refusing to proceed on an under-determined request (ask for the slot, name the conflict), which is deliberation, not proactive attention. (contested)
|
||||
calendar-management: [action, deliberation]
|
||||
|
||||
# Contested; broken by criterion count: two of four are constraints (robots and politeness, only the URL and utterance leave) against one for the watch, so governance takes second — the watch does pull at initiative. (contested)
|
||||
web-crawling: [action, governance]
|
||||
|
||||
# Unanimous: pull a named source and condense it, refusing a summary that would invent content.
|
||||
summaries: [action, governance]
|
||||
|
||||
# Contested; broken by criterion 5134 naming authentication for both directions explicitly, which outweighs the inbound half's perception flavour. (contested)
|
||||
webhooks: [interaction, governance]
|
||||
|
||||
# Unanimous: a user-set schedule that runs acts under the same confirmation rules as a spoken act.
|
||||
cron-jobs: [action, governance]
|
||||
|
||||
# Unanimous: a correction stored as a readable, deletable outcome whose only effect is later phrasing.
|
||||
learning-the-style: [memory, interaction]
|
||||
|
||||
# Unanimous: stored outcomes from dismissals and repairs that change the next decision.
|
||||
learning-from-mistakes: [memory, deliberation]
|
||||
|
||||
# Contested; broken by criterion 77b5 stating that a chain containing an act confirms each act separately, an explicit confirmation rule that outranks the generic "performs both" pull toward action. (contested)
|
||||
command-chaining: [deliberation, governance]
|
||||
Reference in New Issue
Block a user