kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite the architecture under the no-Magi constraint

Browse Source 
Magi and any learned head are ruled out, so every structure comes out of a
gemma prompt field or plain Python over gemma's output. Each section now
carries what exists today and a DoD.

The build order starts at step 0, the measurement spine: eval/chapter-truth.json
plus purity and fragmentation printed by audit_registry.py. Nothing below it is
measurable without that file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kami
2026-08-13 22:28:57 +04:00
parent dae9dbc93a
commit 12d4ed8534
2 changed files with 216 additions and 225 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+207 -222
View File
@@ -1,8 +1,9 @@
# ARCHITECTURE
The target shape of the pipeline, written 2026-08-13 from the user's design. This is **not** what the code
does. `NEXT.md` holds the live state and `AUDIT.md` holds the current pipeline. Every section here ends
with what exists today, so the gap is legible without reading both.
The target shape of the pipeline. Written 2026-08-13 from the user's design, rewritten the same day under
the no-Magi constraint. This is **not** what the code does. `NEXT.md` holds the live state and `AUDIT.md`
holds the current pipeline. Every section ends with what exists today and what would make it done. The
gap stays legible and testable without reading both.
The governing principle:
@@ -12,6 +13,49 @@ The governing principle:
Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below
follows from that split.
## Ruled out
Read this before proposing any of it again.
| ruled out | why | who decided |
| --- | --- | --- |
| Magi, Magiv2, Magiv3 as a detector or as pair heads | project constraint, not a measurement | user, 2026-08-13 |
| A trained `t2c` pair model, or any learned head | no labelled pages exist and no labeller is planned | follows from the above |
| Crop-to-crop cosine as a link signal | measured: two men reach 0.93, one man reaches 0.96, no threshold exists (`caveats/audit-open.md#cosine-not-identity`) | 2026-08-12 |
The consequence runs through the whole document. **Every structure below comes out of a gemma prompt
field, or out of plain Python over gemma's output.** Nothing below is trained. A learned score
in the original design becomes a hand-weighted sum. The weights are read off the labelled chapter, and
the DoD is the accuracy number rather than the mechanism.
## The measurement spine
Nothing below can be called done without this, and it does not exist yet.
The only ground truth in the project is the eyeball pass over the 19:44 run of 2026-08-12. It lives in
prose in `NEXT.md`. Write it to `eval/chapter-truth.json` against chapter
`7c944dd4-e972-42c7-ba60-9f6939548e80`, scoped to what was already checked by eye rather than to all 119
detections:
- the three characters walked crop by crop, each assignment marked as the real person or not
- 30 dialogue lines with their true speaker, typed as `visible | offscreen | narrator | unknown`
The baseline it records, from that run:
| character | assignments | correct | purity | note |
| --- | --- | --- | --- | --- |
| the lead | 16 | 14 | 0.88 | plus a photograph at `order 17` and a chibi at `order 20` |
| `character_2b1b12a1` | 13 | 13 | 1.00 | a main character the registry never named |
| `character_f0d4e901` | 9 | 7 | 0.78 | the other 2 are `2b1b12a1` |
Woman A is `2b1b12a1`. She has 15 occurrences split across 2 ids, so her fragmentation is 2.
`audit_registry.py` already walks panels, reads `identity_assignments` and counts per character. Extend
it to print purity and fragmentation against the truth file. Do not write an eval harness.
**Done when:** `audit_registry.py <chapter>` prints purity per labelled character and fragmentation per
labelled person, and reproduces the table above on the 19:44 run.
## 1. The page is a region graph, not a list of panels
```
@@ -36,9 +80,29 @@ page
A flat set of panels cannot express a television inside a room. That is the defect the current pipeline
shows most often.
With no detector to train, the graph comes out of the detection prompt. Gemma already returns a box per
character and per text. Two fields per detection buy most of the graph with no new model:
```
plane = story | screen | photo | poster | drawing | flashback | dream
species = human | animal | object
```
`plane` is the containment edge in disguise. A detection whose `plane` is not `story` sits inside embedded
art, and that is the fact every stage below needs. `species` is a separate axis and exists because vision
boxes cats as people and dresses them.
Tail regions stay unbuilt. The `det`/`seg` heads exist and are unused
(`caveats/speaker-attribution.md#tail-is-not-geometry`), and section 3 says why they are not the first
thing to spend on.
**Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a
non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no
containment edge, no tail region and no region type.
containment edge, no region type, no `plane` and no `species`.
**Done when:** every detection carries `plane` and `species`. On the labelled chapter, `order 17` and
`order 20` of the lead are not `story`, none of his 14 correct crops is demoted, and `p081` and `p108` are
`animal`. Measured by `audit_registry.py`, which already reads the vision blob per panel.
## 2. Identity exists independently of names
@@ -53,43 +117,42 @@ occurrence c42
occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster.
Three levels, never collapsed into one.
**Today:** the schema already has this split. `identity_assignments` is the occurrence,
`characters` owns the identity, `name` is nullable and downstream already falls back to an anonymous
display. What is missing is the clustering, not the separation. See section 4.
**Today:** already true. `identity_assignments` is the occurrence, `characters` owns the identity, `name`
is nullable and downstream already falls back to an anonymous display.
## 3. Speaker attribution is a scored graph edge, not a procedure
**Done when:** already done. No work item. The clustering is section 4 and the naming is
`decisions/identity-naming.md`.
## 3. Speaker attribution is a scored edge, not a procedure
Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge:
```
score(text, character) =
learned_t2c_score
+ tail_evidence
+ spatial_evidence
+ same_panel
+ dialogue_continuity
+ character_activity_prior
+ identity_context
w1 * gemma_answer
+ w2 * spatial_evidence
+ w3 * same_panel
+ w4 * same_plane
+ w5 * conversation_continuity
+ w6 * character_activity_prior
```
`learned_t2c_score` is the load-bearing term: a pair classifier over the whole page, the text object's
visual feature and the character object's visual feature. Magi's text-character head does exactly this.
It can start as a tiny MLP:
The original design put a learned `t2c` head in the first term and called it load-bearing. No labelled
pages exist, so that term does not. **Gemma's answer becomes one term of six rather than the whole
procedure.** The geometry terms overrule it when they agree against it. The weights are constants read off
the 30 labelled lines. Six numbers in a module, not a training run.
```
t2c(text_embedding, character_embedding, page_context, geometry_features) -> p(speaker)
```
with geometry carrying normalized relative position, distance, overlap, same-panel, containment depth and
tail direction.
Geometry carries normalized relative position, distance, overlap, same-panel and containment depth. Tail
direction is absent until a tail region exists, and it is not the first thing to build. `conversation
continuity` is free, and turn-taking is the strongest prior for a tail-less bubble.
Then the cases fall out of one mechanism instead of four:
| case | what carries it |
| --- | --- |
| bubble with a tail | `t2c` + tail, usually decisive |
| bubble with no tail | `t2c` + spatial and context |
| speaker outside the panel | recent identities + an offscreen candidate |
| bubble with a tail | gemma plus spatial, usually decisive |
| bubble with no tail | conversation continuity plus spatial |
| speaker outside the panel | recent identities plus an offscreen candidate |
| narration | the narrator candidate |
| nothing resolves | unknown speaker |
@@ -102,8 +165,13 @@ speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown
**Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator`
(`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a
prompt to gemma over a window of panels, with no geometry term at all. The `det`/`seg` tail heads exist
and are unused (`caveats/speaker-attribution.md#tail-is-not-geometry`).
prompt to gemma over a window of panels, with no geometry term at all.
**Done when:** `audit_speakers.py` reports accuracy over the 30 labelled lines, split by true type, and
the scored version beats the recorded gemma-window baseline. Two numbers must move the right way, and both
are reported. Correct assignments go up. **Forced** errors go down, where forced means a line given a
visible character while the truth is `offscreen`, `narrator` or `unknown`. Record the baseline before
touching the code.
## 4. Character recognition is occurrence, then identity, then name
@@ -112,257 +180,174 @@ character detection
↓
occurrence embeddings
↓
pairwise same_identity probabilities
pairwise same_identity scores
↓
chapter-wide constrained clustering
↓
char_001, char_002, ...
↓
optional character-bank lookup
optional name claim
↓
name or unknown
```
Two rules that the current code gets wrong.
Three rules the current code gets wrong.
**The embedding is not the character crop alone.** Combine four signals: the character crop, the face or
head crop, the full-body crop, and a contextual object feature. Magiv2 combines detected object features
with a separate crop-embedding model.
**The embedding is not the character crop alone.** The crop embedding measures scene, not person, which is
why two men reach 0.93. Combine the face or head crop with the person crop instead of replacing one with
the other. `face_detect` already finds the face and pairs it for `has_face`, so the face box is free. This
is the queued experiment in `NEXT.md` item 1: re-embed the same 22 detections and recompute the matrix.
**Cluster chapter-wide, not page by page.**
**Cluster chapter-wide, not page by page.** `tracklets.link_tracklets` groups within an 8-panel window.
**Two characters in the same panel may be one person.** Mirrors, photographs, flashbacks, insets,
screens, imagined scenes and repeated action drawings all break that rule. Make it a weak cannot-link,
and only when the two are on the same narrative plane.
**Two characters in the same panel may be one person.** Seven things break that rule. Mirrors,
photographs, flashbacks, insets, screens, imagined scenes, repeated action drawings. Make it a weak
cannot-link, and only between detections on the same `plane`.
**Today:** the embedding is the person box only, which is measurably the wrong signal
(`caveats/audit-open.md#cosine-not-identity`). Clustering is greedy and local: `tracklets.link_tracklets`
groups within an 8-panel window. `tracklets.cannot_link` treats same-panel co-presence as a **hard**
constraint, which is exactly the correction above. Naming is `db.add_name_claim`, corroboration over
`name_claims`.
That last rule has an ordering trap. Same-panel co-presence is currently a **hard** constraint and it is
load-bearing precisely because cosine cannot separate people. Weakening it before the embedding improves
will regress purity. The dependency is the embedding fix, not the `plane` field alone.
## 5. The art-in-art problem needs a narrative plane
**Today:** the embedding is the person box only (`caveats/audit-open.md#cosine-not-identity`). Clustering
is greedy and local. `tracklets.cannot_link` treats same-panel co-presence as hard. Naming is
`db.add_name_claim`, corroboration over `name_claims`.
Treat the page as a hierarchical scene graph:
**Done when:** no labelled character holds more than one wrong assignment, and woman A's fragmentation is
1. Baseline is 2 wrong, 0 wrong, 2 wrong, and fragmentation 2. The bar is stated in errors rather than in
a purity ratio on purpose. The three characters hold 16, 13 and 9 assignments. At those counts any ratio
above 0.94 means zero tolerated errors, and the ratio hides that.
The face-plus-person embedding lands first and carries its own smaller check. On the 22 measured
detections, the highest different-person pair must fall below the lowest same-person pair.
## 5. The narrative plane is what stops art-in-art
The page is a hierarchical scene graph:
```
page
└── panel A depth=0
└── panel A plane=story
├── character c1
├── text t1
└── television/poster depth=1, type=embedded_art
└── television/poster plane=screen
├── character c2
└── text t2
```
Speaker candidates normally come from the same `scene_depth`. Otherwise a real character standing beside a
poster of a drawn person can be given the poster person's line.
A region classifier predicts a type:
```
story_scene | inset_story_panel | flashback | screen | photo | poster | illustration | decorative
```
Perfect classification is not the point. The output that matters is one probability:
Perfect classification is not the point. The output that matters is one predicate:
```
same_narrative_plane(a, b)
```
which then enters the association score in section 3 and the cannot-link in section 4.
It has exactly two consumers, and they are the reason the field is worth adding at all:
- section 3, as the `same_plane` term. A real character beside a poster does not get the poster person's
line.
- section 4, as the guard that makes the weak cannot-link safe.
**Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the
19:44 run of 2026-08-12 his 16 assignments were 14 correct plus a photograph of another man and a chibi
drawing. Both are art inside a panel. Vision also boxes cats as people and dresses them (`p081`, `p108`).
19:44 run his 16 assignments were 14 correct plus a photograph of another man and a chibi drawing. Both
are art inside a panel.
## 6. Narrative understanding is a state machine, not a per-panel description
**Done when:** section 1's DoD, plus both consumers wired, plus section 4's purity DoD holds with the
cannot-link demoted to weak. If purity regresses when the constraint is weakened, the embedding is not
ready and the demotion reverts.
```
story_state
├─ entities (characters, locations, important objects)
├─ scenes
├─ timeline
├─ relationships
├─ unresolved_threads
├─ facts
└─ hypotheses
```
## 6. Narrative understanding is carried state, not a per-panel description
A panel produces a **delta**, not another standalone prose interpretation:
Today each stage reads its predecessor's blob for one panel or one beat. `recent`, a rolling list of the
last few dialogue lines, is the only carried state. That is the root of the invented narration.
```
panel 142:
- character_07 enters room_03
- character_02 is already present
- character_07 says "..."
- object_12 changes owner: 02 -> 07
- possible flashback begins
```
### Facts, hypotheses and unknowns are different records
```
fact: source=panel_142 confidence=0.99 character_07 is visible
hypothesis: confidence=0.64 character_07 is angry
unknown: who caused the explosion
```
A later panel strengthens, replaces or invalidates a hypothesis without rewriting history.
### Scene state is explicit and inherited
The version worth building is one record per scene, inherited forward:
```
scene_31:
location: school_rooftop
time: evening
participants: {char_03: present, char_07: present, char_11: offscreen}
pov: null
narrative_mode: present
parent_scene: null
last_speaker: char_07
addressee: char_02
```
A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character
absent for one panel has not left. A panel with no background has not changed location. A tail-less line
keeps the offscreen participant as a candidate. A close-up still belongs to the scene.
keeps the offscreen participant as a candidate. A close-up still belongs to the scene. `last_speaker` and
`participants` are what section 3's continuity term reads.
### Classify the transition, not just the panel
A panel produces a **delta** against that record, not another standalone prose interpretation:
```
panel 142:
- character_07 enters room_03
- character_02 is already present
- character_07 says "..."
```
Classify the transition, not just the panel:
```
CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START
FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE
```
`EMBEDDED_SCENE` is what stops a television's contents mutating the room around it:
`EMBEDDED_SCENE` is `plane != story` at scene granularity, and is what stops a television's contents
mutating the room around it.
**Scene state is written by a resolver, never by the vision model.** The path is
`observation -> resolver -> state transition`. The resolver may reject an impossible update. It is plain
Python over gemma's per-panel delta, and it is where the constraint lives.
**Today:** none of it exists. Chapter boundaries are a reset. Narration asserts things no panel shows
(`NEXT.md` item 6).
**Done when:** a scene record carries location, participants and `narrative_mode` across panels. A
character absent from one panel stays a participant. On the next full run the four invented-fact
timestamps do not recur. Those are 0:43, 2:03, 2:05 and 2:15, and they are the regression list. The
correctness verifier passed 116/116 over them because it checks quotes and names, never invented claims.
So the check is a re-watch of those four points, not a stage counter.
### Not building yet
Each of these was in the original design. Each is deferred with a trigger, not dropped.
| deferred | trigger to revisit |
| --- | --- |
| Facts, hypotheses and unknowns as separate records with confidences | when scene state exists and narration still asserts unshown claims |
| The seven-check consistency checker | when a scene record exists for it to check against |
| Unresolved references that survive and back-propagate | when a second chapter of the same manga runs |
| Chapter checkpoints and the two-memory split | when a second chapter of the same manga runs |
One reason covers all four. They sit on an identity layer still wrong on 2 of the lead's 16 crops. State
machinery over wrong identity produces confidently wrong state.
## 7. Build order
Detection, vision and character embeddings already exist. The order below is chosen so each step is
falsifiable by the step's own DoD before the next one starts.
```
scene_12 present
├─ panel 101
├─ panel 102
└─ embedded scene_13 [television]
├─ panel-like region
└─ char_19
0. eval/chapter-truth.json + purity and fragmentation in audit_registry.py
1. plane + species per detection -> section 1 DoD
2. face-plus-person embedding -> section 4 embedding check
3. chapter-wide clustering, weak cannot-link on plane -> section 4 purity DoD
4. scene record carried forward -> section 6 DoD
5. scored speaker edge, offscreen arm -> section 3 DoD
6. tail regions from the unused det/seg heads -> only if 5 misses its DoD
```
### Character state is written by a resolver, never by the vision model
Steps 1 and 2 are independent and can land together. Step 3 depends on 2, which is the ordering trap in
section 4. Step 5 depends on 4, because the continuity term reads the scene record. Step 6 is
conditional on purpose: build a tail detector only after the cheap terms have been measured and found
insufficient.
```
char_07:
known_names: [...]
currently_at: room_03
status: alive
appearance_state: {clothes: school_uniform, injured: true}
relationships: {char_02: friend?}
last_seen: panel_142
```
The path is `observation -> resolver -> state transition`, and the resolver may reject an impossible
update.
### Conversation state is its own record
```
conversation_18:
scene: scene_31
participants: [char_02, char_07]
last_speaker: char_07
addressee: char_02
topic: missing_key
```
This is the strongest available prior for a tail-less bubble. Given `A: where did you put it? / ... /
A: don't lie.`, turn-taking assigns the middle line with no visual evidence at all.
### An unresolved reference survives instead of being forced
```
unknown_04:
type: person
descriptions: ["the man from yesterday", "silhouette in panel_58"]
candidate_ids: {char_12: 0.55, char_19: 0.22}
```
Chapter 6 may reveal `unknown_04 == char_12`, and that identity back-propagates through the graph. The
same applies to unnamed characters, pronouns, disguised characters, mysterious objects and unseen
speakers.
### Two memories
- **Working narrative state**: the current scene and the recent ones, in detail.
- **Canonical long-term memory**: compressed facts, not chapter summaries. `char_07 learned that char_02
betrayed the group.` `object_04 is held by char_11.` `char_03 does not know char_07 survived.`
### A chapter boundary is a checkpoint, not a reset
```
chapter_checkpoint:
persistent_entity_changes / relationship_changes / location and status changes
newly established facts / unresolved questions / active plot threads / final scene state
```
Chapter `n+1` starts from that. The detailed panel graph may be kept forever. Only five things load into
the model: the current scene, the previous scene, the relevant character records, the active threads, and
retrieved old facts.
### A consistency checker runs after each scene and chapter
Seven checks. A dead character appearing normally. A character knowing a fact before learning it. An
object owned by two people at once. A flashback never closed. A location jump with no transition. A
speaker who was neither present nor offscreen. A name that conflicts with the identity graph. The model
proposes corrections. The graph stays the source of truth.
**Today:** none of this exists. Each stage reads its predecessor's blob for one panel or one beat.
`recent` is a rolling list of the last few dialogue lines and is the only carried state. Chapter
boundaries are a reset. There is no fact-versus-hypothesis distinction anywhere, which is why narration
asserts things no panel shows (`NEXT.md` item 6).
## 7. The staged version worth building
Do not recreate Magi's monolithic network first. Detection, vision and character embeddings already exist,
so stage it:
```
page
↓
region detector panels / nested regions / texts / characters / tails
↓
object feature extraction
↓
three pair models character↔character (identity)
text→character (speaker)
text→tail (bubble structure)
↓
chapter graph
↓
global character clustering
↓
optional naming
↓
ocr + reading order
↓
dialogue stream
```
The VLM then judges only the ambiguous graph edges. It no longer rediscovers every character and dialogue
relationship from raw pixels on every panel. Magi formulates detection and association as graph
generation, which is why it beats a crop, OCR and nearest-character pipeline here.
## What to take from this before the rewrite
Three items are cheap against the current code and pay immediately. They are entered in `NEXT.md`, not
here.
1. **`same_narrative_plane`, as a per-detection field.** Vision already returns per-panel boxes. Add a
`plane` or `depth` to a detection, set when the model says the figure sits inside a screen, poster,
photo or drawing. That buys the containment edge with no detector. It is the whole of the remaining
identity error on the lead, and it feeds every stage below.
2. **Same-panel co-presence becomes a weak cannot-link.** `tracklets.cannot_link` currently makes it hard.
It needs item 1 first, because the plane is what makes the weak version safe.
3. **`offscreen` as a fourth `speaker_ref` kind.** The union already exists, the arm does not.
The VLM keeps judging ambiguous edges. What changes is that it stops rediscovering every character and
dialogue relationship from raw pixels on every panel. The chapter graph carries the answer forward.
## Sources
Magi and Magiv2 for the detection-and-association-as-graph-generation formulation, the text-character
pair head, and the character bank of exemplar images plus names. Magiv3 for panels, texts, characters and
tails with their associations, and for character grounding between textual descriptions and detected
character regions.
Three ideas come from published comics-transcription work. Detection and association as graph generation,
the text-to-character pair head, and the character bank of exemplar images plus names. The
formulation is kept. The models are ruled out, see **Ruled out** above. No source in this document is a
runtime dependency.
NEXT.md
+9 -3
View File
@@ -71,9 +71,15 @@ contain the right person. Coverage is still the `has_face` gate plus those refus
face touching the bottom edge of one fragment with a textless fragment below. No evidence has been
gathered yet on how often this chapter does it.
1c. **Three items from `ARCHITECTURE.md` are cheap against the current code.** A `plane` field per
detection for art-in-art, same-panel co-presence demoted to a weak cannot-link once the plane exists,
and `offscreen` as a fourth `speaker_ref` kind.
1c. **`ARCHITECTURE.md` was rewritten on 2026-08-13 under the no-Magi constraint.** Every section now
carries a DoD. Its build order starts at step 0, `eval/chapter-truth.json` plus purity and
fragmentation printed by `audit_registry.py`. Nothing below it is measurable without that file.
Steps 1 and 2 are independent and cheap: `plane` and `species` per detection, and the
face-plus-person embedding already queued in item 1 above. The weak cannot-link is step 3 and depends
on step 2, not on `plane` alone. Co-presence as a hard constraint is load-bearing while cosine cannot
separate people, so demoting it early regresses purity. `offscreen` as a fourth `speaker_ref` kind is
still cheap and independent.
2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0
degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`).