manga-recap-pipeline/ARCHITECTURE.md

ARCHITECTURE

The target shape of the pipeline. Written 2026-08-13 from the user's design, rewritten the same day under the no-Magi constraint. This is not what the code does. NEXT.md holds the live state and AUDIT.md holds the current pipeline. Every section ends with what exists today and what would make it done. The gap stays legible and testable without reading both.

The governing principle:

Do not make the next panel understand the previous panel. Make it understand the current world state produced by all previous panels.

Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below follows from that split.

Ruled out

Read this before proposing any of it again.

ruled out	why	who decided
Magi, Magiv2, Magiv3 as a detector or as pair heads	project constraint, not a measurement	user, 2026-08-13
A trained t2c pair model, or any learned head	no labelled pages exist and no labeller is planned	follows from the above
Crop-to-crop cosine as a link signal	measured: two men reach 0.93, one man reaches 0.96, no threshold exists (caveats/audit-open.md#cosine-not-identity)	2026-08-12

The consequence runs through the whole document. Every structure below comes out of a gemma prompt field, or out of plain Python over gemma's output. Nothing below is trained. A learned score in the original design becomes a hand-weighted sum. The weights are read off the labelled chapter, and the DoD is the accuracy number rather than the mechanism.

The measurement spine

Nothing below can be called done without this, and it does not exist yet.

The only ground truth in the project is the eyeball pass over the 19:44 run of 2026-08-12. It lives in prose in NEXT.md. Write it to eval/chapter-truth.json against chapter 7c944dd4-e972-42c7-ba60-9f6939548e80, scoped to what was already checked by eye rather than to all 119 detections:

• the three characters walked crop by crop, each assignment marked as the real person or not
• 30 dialogue lines with their true speaker, typed as visible | offscreen | narrator | unknown

The baseline it records, from that run:

character	assignments	correct	purity	note
the lead	16	14	0.88	plus a photograph at order 17 and a chibi at order 20
character_2b1b12a1	13	13	1.00	a main character the registry never named
character_f0d4e901	9	7	0.78	the other 2 are 2b1b12a1

Woman A is 2b1b12a1. She has 15 occurrences split across 2 ids, so her fragmentation is 2.

audit_registry.py already walks panels, reads identity_assignments and counts per character. Extend it to print purity and fragmentation against the truth file. Do not write an eval harness.

Done when: audit_registry.py <chapter> prints purity per labelled character and fragmentation per labelled person, and reproduces the table above on the 19:44 run.

1. The page is a region graph, not a list of panels

page
 ├─ regions
 │   ├─ panel
 │   ├─ inset_panel
 │   ├─ embedded_art
 │   ├─ text
 │   ├─ tail
 │   └─ character_occurrence
 │
 └─ edges
     ├─ contains(region, region)
     ├─ reads_before(text, text)
     ├─ tail_of(tail, text)
     ├─ points_to(tail, character)
     ├─ spoken_by(text, character)
     └─ same_identity(character, character)

A flat set of panels cannot express a television inside a room. That is the defect the current pipeline shows most often.

With no detector to train, the graph comes out of the detection prompt. Gemma already returns a box per character and per text. Two fields per detection buy most of the graph with no new model:

plane   = story | screen | photo | poster | drawing | flashback | dream
species = human | animal | object

plane is the containment edge in disguise. A detection whose plane is not story sits inside embedded art, and that is the fact every stage below needs. species is a separate axis and exists because vision boxes cats as people and dresses them.

Tail regions stay unbuilt. The det/seg heads exist and are unused (caveats/speaker-attribution.md#tail-is-not-geometry), and section 3 says why they are not the first thing to spend on.

Today: the crop stage emits a flat panel list with a bbox each, plus context_fragments, a non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no containment edge, no region type, no plane and no species.

Done when: every detection carries plane and species. On the labelled chapter, order 17 and order 20 of the lead are not story, none of his 14 correct crops is demoted, and p081 and p108 are animal. Measured by audit_registry.py, which already reads the vision blob per panel.

2. Identity exists independently of names

occurrence c42
    -> identity char_07
         name = null
         aliases = []

char_07.name may be filled later, or stay null forever and display as unknown character #7. The occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster. Three levels, never collapsed into one.

Today: already true. identity_assignments is the occurrence, characters owns the identity, name is nullable and downstream already falls back to an anonymous display.

Done when: already done. No work item. The clustering is section 4 and the naming is decisions/identity-naming.md.

3. Speaker attribution is a scored edge, not a procedure

Do not write find bubble -> find tail -> nearest character. Score every plausible edge:

score(text, character) =
    w1 * gemma_answer
  + w2 * spatial_evidence
  + w3 * same_panel
  + w4 * same_plane
  + w5 * conversation_continuity
  + w6 * character_activity_prior

The original design put a learned t2c head in the first term and called it load-bearing. No labelled pages exist, so that term does not. Gemma's answer becomes one term of six rather than the whole procedure. The geometry terms overrule it when they agree against it. The weights are constants read off the 30 labelled lines. Six numbers in a module, not a training run.

Geometry carries normalized relative position, distance, overlap, same-panel and containment depth. Tail direction is absent until a tail region exists, and it is not the first thing to build. conversation continuity is free, and turn-taking is the strongest prior for a tail-less bubble.

Then the cases fall out of one mechanism instead of four:

case	what carries it
bubble with a tail	gemma plus spatial, usually decisive
bubble with no tail	conversation continuity plus spatial
speaker outside the panel	recent identities plus an offscreen candidate
narration	the narrator candidate
nothing resolves	unknown speaker

A dialogue line must not be required to resolve to a visible character. That is a failure mode, not a safeguard. The speaker type is a union:

speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown

Today: speaker_ref is already a typed union of character_id | name | unknown | narrator (decisions/audit-phase1.md#speaker-ref-is-canonical). offscreen is the missing arm. Attribution is a prompt to gemma over a window of panels, with no geometry term at all.

Done when: audit_speakers.py reports accuracy over the 30 labelled lines, split by true type, and the scored version beats the recorded gemma-window baseline. Two numbers must move the right way, and both are reported. Correct assignments go up. Forced errors go down, where forced means a line given a visible character while the truth is offscreen, narrator or unknown. Record the baseline before touching the code.

4. Character recognition is occurrence, then identity, then name

character detection
       ↓
occurrence embeddings
       ↓
pairwise same_identity scores
       ↓
chapter-wide constrained clustering
       ↓
char_001, char_002, ...
       ↓
optional name claim
       ↓
name or unknown

Three rules the current code gets wrong.

The embedding is not the character crop alone. The crop embedding measures scene, not person, which is why two men reach 0.93. Combine the face or head crop with the person crop instead of replacing one with the other. face_detect already finds the face and pairs it for has_face, so the face box is free. This is the queued experiment in NEXT.md item 1: re-embed the same 22 detections and recompute the matrix.

Cluster chapter-wide, not page by page. tracklets.link_tracklets groups within an 8-panel window.

Two characters in the same panel may be one person. Seven things break that rule. Mirrors, photographs, flashbacks, insets, screens, imagined scenes, repeated action drawings. Make it a weak cannot-link, and only between detections on the same plane.

That last rule has an ordering trap. Same-panel co-presence is currently a hard constraint and it is load-bearing precisely because cosine cannot separate people. Weakening it before the embedding improves will regress purity. The dependency is the embedding fix, not the plane field alone.

Today: the embedding is the person box only (caveats/audit-open.md#cosine-not-identity). Clustering is greedy and local. tracklets.cannot_link treats same-panel co-presence as hard. Naming is db.add_name_claim, corroboration over name_claims.

Done when: no labelled character holds more than one wrong assignment, and woman A's fragmentation is

1. Baseline is 2 wrong, 0 wrong, 2 wrong, and fragmentation 2. The bar is stated in errors rather than in a purity ratio on purpose. The three characters hold 16, 13 and 9 assignments. At those counts any ratio above 0.94 means zero tolerated errors, and the ratio hides that.

The face-plus-person embedding lands first and carries its own smaller check. On the 22 measured detections, the highest different-person pair must fall below the lowest same-person pair.

5. The narrative plane is what stops art-in-art

The page is a hierarchical scene graph:

page
└── panel A                     plane=story
    ├── character c1
    ├── text t1
    └── television/poster       plane=screen
        ├── character c2
        └── text t2

Perfect classification is not the point. The output that matters is one predicate:

same_narrative_plane(a, b)

It has exactly two consumers, and they are the reason the field is worth adding at all:

• section 3, as the same_plane term. A real character beside a poster does not get the poster person's line.
• section 4, as the guard that makes the weak cannot-link safe.

Today: nothing models this, and it is the whole of the remaining identity error on the lead. On the 19:44 run his 16 assignments were 14 correct plus a photograph of another man and a chibi drawing. Both are art inside a panel.

Done when: section 1's DoD, plus both consumers wired, plus section 4's purity DoD holds with the cannot-link demoted to weak. If purity regresses when the constraint is weakened, the embedding is not ready and the demotion reverts.

6. Narrative understanding is carried state, not a per-panel description

Today each stage reads its predecessor's blob for one panel or one beat. recent, a rolling list of the last few dialogue lines, is the only carried state. That is the root of the invented narration.

The version worth building is one record per scene, inherited forward:

scene_31:
  location: school_rooftop
  time: evening
  participants: {char_03: present, char_07: present, char_11: offscreen}
  narrative_mode: present
  last_speaker: char_07
  addressee: char_02

A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character absent for one panel has not left. A panel with no background has not changed location. A tail-less line keeps the offscreen participant as a candidate. A close-up still belongs to the scene. last_speaker and participants are what section 3's continuity term reads.

A panel produces a delta against that record, not another standalone prose interpretation:

panel 142:
- character_07 enters room_03
- character_02 is already present
- character_07 says "..."

Classify the transition, not just the panel:

CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START
FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE

EMBEDDED_SCENE is plane != story at scene granularity, and is what stops a television's contents mutating the room around it.

Scene state is written by a resolver, never by the vision model. The path is observation -> resolver -> state transition. The resolver may reject an impossible update. It is plain Python over gemma's per-panel delta, and it is where the constraint lives.

Today: none of it exists. Chapter boundaries are a reset. Narration asserts things no panel shows (NEXT.md item 6).

Done when: a scene record carries location, participants and narrative_mode across panels. A character absent from one panel stays a participant. On the next full run the four invented-fact timestamps do not recur. Those are 0:43, 2:03, 2:05 and 2:15, and they are the regression list. The correctness verifier passed 116/116 over them because it checks quotes and names, never invented claims. So the check is a re-watch of those four points, not a stage counter.

Not building yet

Each of these was in the original design. Each is deferred with a trigger, not dropped.

deferred	trigger to revisit
Facts, hypotheses and unknowns as separate records with confidences	when scene state exists and narration still asserts unshown claims
The seven-check consistency checker	when a scene record exists for it to check against
Unresolved references that survive and back-propagate	when a second chapter of the same manga runs
Chapter checkpoints and the two-memory split	when a second chapter of the same manga runs

One reason covers all four. They sit on an identity layer still wrong on 2 of the lead's 16 crops. State machinery over wrong identity produces confidently wrong state.

7. Build order

Detection, vision and character embeddings already exist. The order below is chosen so each step is falsifiable by the step's own DoD before the next one starts.

0. eval/chapter-truth.json + purity and fragmentation in audit_registry.py
1. plane + species per detection                      -> section 1 DoD
2. face-plus-person embedding                         -> section 4 embedding check
3. chapter-wide clustering, weak cannot-link on plane  -> section 4 purity DoD
4. scene record carried forward                       -> section 6 DoD
5. scored speaker edge, offscreen arm                 -> section 3 DoD
6. tail regions from the unused det/seg heads         -> only if 5 misses its DoD

Steps 1 and 2 are independent and can land together. Step 3 depends on 2, which is the ordering trap in section 4. Step 5 depends on 4, because the continuity term reads the scene record. Step 6 is conditional on purpose: build a tail detector only after the cheap terms have been measured and found insufficient.

The VLM keeps judging ambiguous edges. What changes is that it stops rediscovering every character and dialogue relationship from raw pixels on every panel. The chapter graph carries the answer forward.

Sources

Three ideas come from published comics-transcription work. Detection and association as graph generation, the text-to-character pair head, and the character bank of exemplar images plus names. The formulation is kept. The models are ruled out, see Ruled out above. No source in this document is a runtime dependency.