manga-recap-pipeline/ARCHITECTURE.md

# ARCHITECTURE

The target shape of the pipeline, written 2026-08-13 from the user's design. This is **not** what the code
does. `NEXT.md` holds the live state and `AUDIT.md` holds the current pipeline. Every section here ends
with what exists today, so the gap is legible without reading both.

The governing principle:

> Do not make the next panel understand the previous panel. Make it understand the current world state
> produced by all previous panels.

Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below
follows from that split.

## 1. The page is a region graph, not a list of panels

```
page
 ├─ regions
 │   ├─ panel
 │   ├─ inset_panel
 │   ├─ embedded_art
 │   ├─ text
 │   ├─ tail
 │   └─ character_occurrence
 │
 └─ edges
     ├─ contains(region, region)
     ├─ reads_before(text, text)
     ├─ tail_of(tail, text)
     ├─ points_to(tail, character)
     ├─ spoken_by(text, character)
     └─ same_identity(character, character)
```

A flat set of panels cannot express a television inside a room. That is the defect the current pipeline
shows most often.

**Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a
non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no
containment edge, no tail region and no region type.

## 2. Identity exists independently of names

```
occurrence c42
    -> identity char_07
         name = null
         aliases = []
```

`char_07.name` may be filled later, or stay null forever and display as `unknown character #7`. The
occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster.
Three levels, never collapsed into one.

**Today:** the schema already has this split. `identity_assignments` is the occurrence,
`characters` owns the identity, `name` is nullable and downstream already falls back to an anonymous
display. What is missing is the clustering, not the separation. See section 4.

## 3. Speaker attribution is a scored graph edge, not a procedure

Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge:

```
score(text, character) =
    learned_t2c_score
  + tail_evidence
  + spatial_evidence
  + same_panel
  + dialogue_continuity
  + character_activity_prior
  + identity_context
```

`learned_t2c_score` is the load-bearing term: a pair classifier over the whole page, the text object's
visual feature and the character object's visual feature. Magi's text-character head does exactly this.
It can start as a tiny MLP:

```
t2c(text_embedding, character_embedding, page_context, geometry_features) -> p(speaker)
```

with geometry carrying normalized relative position, distance, overlap, same-panel, containment depth and
tail direction.

Then the cases fall out of one mechanism instead of four:

| case | what carries it |
| --- | --- |
| bubble with a tail | `t2c` + tail, usually decisive |
| bubble with no tail | `t2c` + spatial and context |
| speaker outside the panel | recent identities + an offscreen candidate |
| narration | the narrator candidate |
| nothing resolves | unknown speaker |

**A dialogue line must not be required to resolve to a visible character.** That is a failure mode, not a
safeguard. The speaker type is a union:

```
speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown
```

**Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator`
(`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a
prompt to gemma over a window of panels, with no geometry term at all. The `det`/`seg` tail heads exist
and are unused (`caveats/speaker-attribution.md#tail-is-not-geometry`).

## 4. Character recognition is occurrence, then identity, then name

```
character detection
       ↓
occurrence embeddings
       ↓
pairwise same_identity probabilities
       ↓
chapter-wide constrained clustering
       ↓
char_001, char_002, ...
       ↓
optional character-bank lookup
       ↓
name or unknown
```

Two rules that the current code gets wrong.

**The embedding is not the character crop alone.** Combine four signals: the character crop, the face or
head crop, the full-body crop, and a contextual object feature. Magiv2 combines detected object features
with a separate crop-embedding model.

**Cluster chapter-wide, not page by page.**

**Two characters in the same panel may be one person.** Mirrors, photographs, flashbacks, insets,
screens, imagined scenes and repeated action drawings all break that rule. Make it a weak cannot-link,
and only when the two are on the same narrative plane.

**Today:** the embedding is the person box only, which is measurably the wrong signal
(`caveats/audit-open.md#cosine-not-identity`). Clustering is greedy and local: `tracklets.link_tracklets`
groups within an 8-panel window. `tracklets.cannot_link` treats same-panel co-presence as a **hard**
constraint, which is exactly the correction above. Naming is `db.add_name_claim`, corroboration over
`name_claims`.

## 5. The art-in-art problem needs a narrative plane

Treat the page as a hierarchical scene graph:

```
page
└── panel A                     depth=0
    ├── character c1
    ├── text t1
    └── television/poster       depth=1, type=embedded_art
        ├── character c2
        └── text t2
```

Speaker candidates normally come from the same `scene_depth`. Otherwise a real character standing beside a
poster of a drawn person can be given the poster person's line.

A region classifier predicts a type:

```
story_scene | inset_story_panel | flashback | screen | photo | poster | illustration | decorative
```

Perfect classification is not the point. The output that matters is one probability:

```
same_narrative_plane(a, b)
```

which then enters the association score in section 3 and the cannot-link in section 4.

**Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the
19:44 run of 2026-08-12 his 16 assignments were 14 correct plus a photograph of another man and a chibi
drawing. Both are art inside a panel. Vision also boxes cats as people and dresses them (`p081`, `p108`).

## 6. Narrative understanding is a state machine, not a per-panel description

```
story_state
├─ entities        (characters, locations, important objects)
├─ scenes
├─ timeline
├─ relationships
├─ unresolved_threads
├─ facts
└─ hypotheses
```

A panel produces a **delta**, not another standalone prose interpretation:

```
panel 142:
- character_07 enters room_03
- character_02 is already present
- character_07 says "..."
- object_12 changes owner: 02 -> 07
- possible flashback begins
```

### Facts, hypotheses and unknowns are different records

```
fact:        source=panel_142  confidence=0.99  character_07 is visible
hypothesis:  confidence=0.64                    character_07 is angry
unknown:                                        who caused the explosion
```

A later panel strengthens, replaces or invalidates a hypothesis without rewriting history.

### Scene state is explicit and inherited

```
scene_31:
  location: school_rooftop
  time: evening
  participants: {char_03: present, char_07: present, char_11: offscreen}
  pov: null
  narrative_mode: present
  parent_scene: null
```

A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character
absent for one panel has not left. A panel with no background has not changed location. A tail-less line
keeps the offscreen participant as a candidate. A close-up still belongs to the scene.

### Classify the transition, not just the panel

```
CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START
FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE
```

`EMBEDDED_SCENE` is what stops a television's contents mutating the room around it:

```
scene_12 present
 ├─ panel 101
 ├─ panel 102
 └─ embedded scene_13 [television]
      ├─ panel-like region
      └─ char_19
```

### Character state is written by a resolver, never by the vision model

```
char_07:
  known_names: [...]
  currently_at: room_03
  status: alive
  appearance_state: {clothes: school_uniform, injured: true}
  relationships: {char_02: friend?}
  last_seen: panel_142
```

The path is `observation -> resolver -> state transition`, and the resolver may reject an impossible
update.

### Conversation state is its own record

```
conversation_18:
  scene: scene_31
  participants: [char_02, char_07]
  last_speaker: char_07
  addressee: char_02
  topic: missing_key
```

This is the strongest available prior for a tail-less bubble. Given `A: where did you put it? / ... /
A: don't lie.`, turn-taking assigns the middle line with no visual evidence at all.

### An unresolved reference survives instead of being forced

```
unknown_04:
  type: person
  descriptions: ["the man from yesterday", "silhouette in panel_58"]
  candidate_ids: {char_12: 0.55, char_19: 0.22}
```

Chapter 6 may reveal `unknown_04 == char_12`, and that identity back-propagates through the graph. The
same applies to unnamed characters, pronouns, disguised characters, mysterious objects and unseen
speakers.

### Two memories

- **Working narrative state**: the current scene and the recent ones, in detail.
- **Canonical long-term memory**: compressed facts, not chapter summaries. `char_07 learned that char_02
  betrayed the group.` `object_04 is held by char_11.` `char_03 does not know char_07 survived.`

### A chapter boundary is a checkpoint, not a reset

```
chapter_checkpoint:
  persistent_entity_changes / relationship_changes / location and status changes
  newly established facts / unresolved questions / active plot threads / final scene state
```

Chapter `n+1` starts from that. The detailed panel graph may be kept forever. Only five things load into
the model: the current scene, the previous scene, the relevant character records, the active threads, and
retrieved old facts.

### A consistency checker runs after each scene and chapter

Seven checks. A dead character appearing normally. A character knowing a fact before learning it. An
object owned by two people at once. A flashback never closed. A location jump with no transition. A
speaker who was neither present nor offscreen. A name that conflicts with the identity graph. The model
proposes corrections. The graph stays the source of truth.

**Today:** none of this exists. Each stage reads its predecessor's blob for one panel or one beat.
`recent` is a rolling list of the last few dialogue lines and is the only carried state. Chapter
boundaries are a reset. There is no fact-versus-hypothesis distinction anywhere, which is why narration
asserts things no panel shows (`NEXT.md` item 6).

## 7. The staged version worth building

Do not recreate Magi's monolithic network first. Detection, vision and character embeddings already exist,
so stage it:

```
page
 ↓
region detector          panels / nested regions / texts / characters / tails
 ↓
object feature extraction
 ↓
three pair models        character↔character (identity)
                         text→character      (speaker)
                         text→tail           (bubble structure)
 ↓
chapter graph
 ↓
global character clustering
 ↓
optional naming
 ↓
ocr + reading order
 ↓
dialogue stream
```

The VLM then judges only the ambiguous graph edges. It no longer rediscovers every character and dialogue
relationship from raw pixels on every panel. Magi formulates detection and association as graph
generation, which is why it beats a crop, OCR and nearest-character pipeline here.

## What to take from this before the rewrite

Three items are cheap against the current code and pay immediately. They are entered in `NEXT.md`, not
here.

1. **`same_narrative_plane`, as a per-detection field.** Vision already returns per-panel boxes. Add a
   `plane` or `depth` to a detection, set when the model says the figure sits inside a screen, poster,
   photo or drawing. That buys the containment edge with no detector. It is the whole of the remaining
   identity error on the lead, and it feeds every stage below.
2. **Same-panel co-presence becomes a weak cannot-link.** `tracklets.cannot_link` currently makes it hard.
   It needs item 1 first, because the plane is what makes the weak version safe.
3. **`offscreen` as a fourth `speaker_ref` kind.** The union already exists, the arm does not.

## Sources

Magi and Magiv2 for the detection-and-association-as-graph-generation formulation, the text-character
pair head, and the character bank of exemplar images plus names. Magiv3 for panels, texts, characters and
tails with their associations, and for character grounding between textual descriptions and detected
character regions.