Measure the registry against the run it describes

ARCHITECTURE.md step 0. eval/chapter-truth.json holds the 38 occurrences the
user walked crop by crop on the 19:44 run of 2026-08-12, and audit_registry.py
now prints purity per cluster and fragmentation per person against it. All six
baseline numbers reproduce.

Rows key on page-space geometry, purity is a share, and fragmentation is a
count of ids, so nothing in the file names a panel_id or a character_id. The
fifth cycle re-crops and calls /characters/reset, and the file survives both.
That was the ordering trap in the handoff.

NEXT.md said 2 of woman B's 9 crops were really woman A and never said which.
They are panel_order 31 and 33, identified from p030 and p032.

Four fixes to the audit itself, all pre-existing:

- 20 characters counted where 14 are live and 6 are merge losers kept on purpose
- the assignment spread keyed on name, so the two Seonhos summed into one line
- the default worked example was panel_index 7, a panel vision skips. NEXT.md's
  "panel 7" is panel_order 7, one lower
- nothing about skipped panels. 41 of 116 are skip=True, four checked and all
  four correct, and they hold 28 of the chapter's 122 dialogue lines

That last count is the measured case for an offscreen speaker_ref kind: 23% of
dialogue sits on panels with no character to attribute to.

Checks: audit_registry.py --selftest covers the IoU match, the greedy tie-break
and the purity maths with no database. ruff check . exits 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-13 23:32:05 +04:00
parent a67f1c2501
commit bc19f9e9fd
8 changed files with 1001 additions and 14 deletions
+3
View File
@@ -54,3 +54,6 @@ still live belongs in `caveats/`.
| [A name is a word set, not a string](identity-naming.md#alias-grouping) | closed, GPU pending |
| [A confident caption names a character on its own](identity-naming.md#caption-is-strong) | closed, GPU pending |
| [A name belongs to one character](identity-naming.md#one-name-one-character) | closed, GPU pending |
| [The truth file keys on page-space geometry](measurement-spine.md#truth-is-page-geometry) | closed |
| [Purity is a share, so it never names an id](measurement-spine.md#purity-is-id-free) | closed |
| [The truth is 38 assignments, and holds no dialogue yet](measurement-spine.md#truth-scope-is-38) | open |
+56
View File
@@ -0,0 +1,56 @@
# The measurement spine
`eval/chapter-truth.json` plus the purity and fragmentation section of `audit_registry.py`. This is
`ARCHITECTURE.md` step 0, and nothing below it in that build order is measurable without it.
## The truth file keys on page-space geometry {#truth-is-page-geometry}
**Closed, 2026-08-13.** Reproduces the baseline on the 19:44 run of 2026-08-12.
A truth row cannot name a `panel_id` or a `character_id`. A re-crop remints panel ids, and
`/characters/reset` remints character ids. The fifth cycle does both, so a truth file keyed on either
would die the moment it was first used.
Each row carries `[x1, y1, x2, y2]` in page space, which is the panel's `bbox` offset plus the
detection box. The page strip is the fetched artifact and does not change when panels are re-sliced, so
the key survives. `audit_registry.py:_score` matches a row to an assignment on the same page at
`IOU_MIN = 0.5`, greedily, best overlap first, each row and each assignment used once.
The 19:44 `panel_order`, `local_id` and `character_id` are kept per row under `from_19_44`, for
debugging only. Nothing matches on them.
## Purity is a share, so it never names an id {#purity-is-id-free}
**Closed, 2026-08-13.**
Purity of a cluster is the largest share of one true person inside it. The lead's 16 assignments hold
14 of him, so 0.88, and the truth file never has to say which id he was. Fragmentation of a person is
the number of clusters holding their occurrences. Woman A has 15 occurrences over 2 ids, so 2.
So both numbers survive a reset. The definitions in `ARCHITECTURE.md` were not explicit about that.
A cluster is reported under its dominant person. Every other person in it prints as a `wrong:` line,
which is where the photograph and the chibi show up on the lead.
The `baseline` block in the truth file holds the 19:44 numbers, and each printed line says
`= baseline` or the delta. That makes step 0's DoD a check rather than an eyeball.
## The truth is 38 assignments, and holds no dialogue yet {#truth-scope-is-38}
**Open, 2026-08-13.** The character half is done, the dialogue half is not written.
The eyeball pass covered three characters, 38 of the run's 60 assignments. The other 10 characters were
never checked and carry no truth row. The audit prints them as out of scope rather than as correct.
Anything claiming chapter-wide accuracy off this file is overreaching.
The prose in `NEXT.md` left two rows unnamed. They were identified on 2026-08-13 by re-cropping `p030`
and `p032`: `panel_order` 31 and 33 on woman B's id are both woman A. That closes the recorded
arithmetic of 7 correct and 2 wrong, and it is why woman A's fragmentation is 2.
One judgement is recorded in the file rather than hidden. The chibi of woman B at `panel_order` 28
counts as her, because it is a stylised rendering of the subject. The chibi of the lead at
`panel_order` 20 does not, because it is a drawing inside the scene. `not_people` names the second kind.
`ARCHITECTURE.md` also asks for 30 dialogue lines with their true speaker, typed
`visible | offscreen | narrator | unknown`. `dialogue` is an empty array. No such pass exists, because
`NEXT.md` records no per-line speaker truth. Writing it means watching the chapter panel by panel. The
audit prints nothing about speakers until it is filled.