kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Wire the caption merge, and write the target architecture down

Browse Source 
merge_faceless_captions had been written and never called; both crop
endpoints called the non-destructive context_fragment_links instead, with no
decision recording that choice. Wiring it changes panel count and every panel
index, so the chapter needs a re-crop with the panels prefix cleared first --
crop_webtoon skips an upload when the key already exists, which is right for a
resume and silently wrong after a slicing change. Noted at the line.

It does not cover the head-in-one-shot body-in-the-next split that prompted
the question. _merge_plan only folds a fragment that has text and no face.

ARCHITECTURE.md is the target shape from the user's design, with what exists
against each section today. Nothing in it is built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kami
2026-08-13 00:39:57 +04:00
parent a386e9d910
commit 97cb4831f9
7 changed files with 495 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+368
View File
@@ -0,0 +1,368 @@
# ARCHITECTURE
The target shape of the pipeline, written 2026-08-13 from the user's design. This is **not** what the code
does. `NEXT.md` holds the live state and `AUDIT.md` holds the current pipeline. Every section here ends
with what exists today, so the gap is legible without reading both.
The governing principle:
> Do not make the next panel understand the previous panel. Make it understand the current world state
> produced by all previous panels.
Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below
follows from that split.
## 1. The page is a region graph, not a list of panels
```
page
├─ regions
│ ├─ panel
│ ├─ inset_panel
│ ├─ embedded_art
│ ├─ text
│ ├─ tail
│ └─ character_occurrence
│
└─ edges
├─ contains(region, region)
├─ reads_before(text, text)
├─ tail_of(tail, text)
├─ points_to(tail, character)
├─ spoken_by(text, character)
└─ same_identity(character, character)
```
A flat set of panels cannot express a television inside a room. That is the defect the current pipeline
shows most often.
**Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a
non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no
containment edge, no tail region and no region type.
## 2. Identity exists independently of names
```
occurrence c42
-> identity char_07
name = null
aliases = []
```
`char_07.name` may be filled later, or stay null forever and display as `unknown character #7`. The
occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster.
Three levels, never collapsed into one.
**Today:** the schema already has this split. `identity_assignments` is the occurrence,
`characters` owns the identity, `name` is nullable and downstream already falls back to an anonymous
display. What is missing is the clustering, not the separation. See section 4.
## 3. Speaker attribution is a scored graph edge, not a procedure
Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge:
```
score(text, character) =
learned_t2c_score
+ tail_evidence
+ spatial_evidence
+ same_panel
+ dialogue_continuity
+ character_activity_prior
+ identity_context
```
`learned_t2c_score` is the load-bearing term: a pair classifier over the whole page, the text object's
visual feature and the character object's visual feature. Magi's text-character head does exactly this.
It can start as a tiny MLP:
```
t2c(text_embedding, character_embedding, page_context, geometry_features) -> p(speaker)
```
with geometry carrying normalized relative position, distance, overlap, same-panel, containment depth and
tail direction.
Then the cases fall out of one mechanism instead of four:
| case | what carries it |
| --- | --- |
| bubble with a tail | `t2c` + tail, usually decisive |
| bubble with no tail | `t2c` + spatial and context |
| speaker outside the panel | recent identities + an offscreen candidate |
| narration | the narrator candidate |
| nothing resolves | unknown speaker |
**A dialogue line must not be required to resolve to a visible character.** That is a failure mode, not a
safeguard. The speaker type is a union:
```
speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown
```
**Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator`
(`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a
prompt to gemma over a window of panels, with no geometry term at all. The `det`/`seg` tail heads exist
and are unused (`caveats/speaker-attribution.md#tail-is-not-geometry`).
## 4. Character recognition is occurrence, then identity, then name
```
character detection
↓
occurrence embeddings
↓
pairwise same_identity probabilities
↓
chapter-wide constrained clustering
↓
char_001, char_002, ...
↓
optional character-bank lookup
↓
name or unknown
```
Two rules that the current code gets wrong.
**The embedding is not the character crop alone.** Combine four signals: the character crop, the face or
head crop, the full-body crop, and a contextual object feature. Magiv2 combines detected object features
with a separate crop-embedding model.
**Cluster chapter-wide, not page by page.**
**Two characters in the same panel may be one person.** Mirrors, photographs, flashbacks, insets,
screens, imagined scenes and repeated action drawings all break that rule. Make it a weak cannot-link,
and only when the two are on the same narrative plane.
**Today:** the embedding is the person box only, which is measurably the wrong signal
(`caveats/audit-open.md#cosine-not-identity`). Clustering is greedy and local: `tracklets.link_tracklets`
groups within an 8-panel window. `tracklets.cannot_link` treats same-panel co-presence as a **hard**
constraint, which is exactly the correction above. Naming is `db.add_name_claim`, corroboration over
`name_claims`.
## 5. The art-in-art problem needs a narrative plane
Treat the page as a hierarchical scene graph:
```
page
└── panel A depth=0
├── character c1
├── text t1
└── television/poster depth=1, type=embedded_art
├── character c2
└── text t2
```
Speaker candidates normally come from the same `scene_depth`. Otherwise a real character standing beside a
poster of a drawn person can be given the poster person's line.
A region classifier predicts a type:
```
story_scene | inset_story_panel | flashback | screen | photo | poster | illustration | decorative
```
Perfect classification is not the point. The output that matters is one probability:
```
same_narrative_plane(a, b)
```
which then enters the association score in section 3 and the cannot-link in section 4.
**Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the
19:44 run of 2026-08-12 his 16 assignments were 14 correct plus a photograph of another man and a chibi
drawing. Both are art inside a panel. Vision also boxes cats as people and dresses them (`p081`, `p108`).
## 6. Narrative understanding is a state machine, not a per-panel description
```
story_state
├─ entities (characters, locations, important objects)
├─ scenes
├─ timeline
├─ relationships
├─ unresolved_threads
├─ facts
└─ hypotheses
```
A panel produces a **delta**, not another standalone prose interpretation:
```
panel 142:
- character_07 enters room_03
- character_02 is already present
- character_07 says "..."
- object_12 changes owner: 02 -> 07
- possible flashback begins
```
### Facts, hypotheses and unknowns are different records
```
fact: source=panel_142 confidence=0.99 character_07 is visible
hypothesis: confidence=0.64 character_07 is angry
unknown: who caused the explosion
```
A later panel strengthens, replaces or invalidates a hypothesis without rewriting history.
### Scene state is explicit and inherited
```
scene_31:
location: school_rooftop
time: evening
participants: {char_03: present, char_07: present, char_11: offscreen}
pov: null
narrative_mode: present
parent_scene: null
```
A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character
absent for one panel has not left. A panel with no background has not changed location. A tail-less line
keeps the offscreen participant as a candidate. A close-up still belongs to the scene.
### Classify the transition, not just the panel
```
CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START
FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE
```
`EMBEDDED_SCENE` is what stops a television's contents mutating the room around it:
```
scene_12 present
├─ panel 101
├─ panel 102
└─ embedded scene_13 [television]
├─ panel-like region
└─ char_19
```
### Character state is written by a resolver, never by the vision model
```
char_07:
known_names: [...]
currently_at: room_03
status: alive
appearance_state: {clothes: school_uniform, injured: true}
relationships: {char_02: friend?}
last_seen: panel_142
```
The path is `observation -> resolver -> state transition`, and the resolver may reject an impossible
update.
### Conversation state is its own record
```
conversation_18:
scene: scene_31
participants: [char_02, char_07]
last_speaker: char_07
addressee: char_02
topic: missing_key
```
This is the strongest available prior for a tail-less bubble. Given `A: where did you put it? / ... /
A: don't lie.`, turn-taking assigns the middle line with no visual evidence at all.
### An unresolved reference survives instead of being forced
```
unknown_04:
type: person
descriptions: ["the man from yesterday", "silhouette in panel_58"]
candidate_ids: {char_12: 0.55, char_19: 0.22}
```
Chapter 6 may reveal `unknown_04 == char_12`, and that identity back-propagates through the graph. The
same applies to unnamed characters, pronouns, disguised characters, mysterious objects and unseen
speakers.
### Two memories
- **Working narrative state**: the current scene and the recent ones, in detail.
- **Canonical long-term memory**: compressed facts, not chapter summaries. `char_07 learned that char_02
betrayed the group.` `object_04 is held by char_11.` `char_03 does not know char_07 survived.`
### A chapter boundary is a checkpoint, not a reset
```
chapter_checkpoint:
persistent_entity_changes / relationship_changes / location and status changes
newly established facts / unresolved questions / active plot threads / final scene state
```
Chapter `n+1` starts from that. The detailed panel graph may be kept forever. Only five things load into
the model: the current scene, the previous scene, the relevant character records, the active threads, and
retrieved old facts.
### A consistency checker runs after each scene and chapter
Seven checks. A dead character appearing normally. A character knowing a fact before learning it. An
object owned by two people at once. A flashback never closed. A location jump with no transition. A
speaker who was neither present nor offscreen. A name that conflicts with the identity graph. The model
proposes corrections. The graph stays the source of truth.
**Today:** none of this exists. Each stage reads its predecessor's blob for one panel or one beat.
`recent` is a rolling list of the last few dialogue lines and is the only carried state. Chapter
boundaries are a reset. There is no fact-versus-hypothesis distinction anywhere, which is why narration
asserts things no panel shows (`NEXT.md` item 6).
## 7. The staged version worth building
Do not recreate Magi's monolithic network first. Detection, vision and character embeddings already exist,
so stage it:
```
page
↓
region detector panels / nested regions / texts / characters / tails
↓
object feature extraction
↓
three pair models character↔character (identity)
text→character (speaker)
text→tail (bubble structure)
↓
chapter graph
↓
global character clustering
↓
optional naming
↓
ocr + reading order
↓
dialogue stream
```
The VLM then judges only the ambiguous graph edges. It no longer rediscovers every character and dialogue
relationship from raw pixels on every panel. Magi formulates detection and association as graph
generation, which is why it beats a crop, OCR and nearest-character pipeline here.
## What to take from this before the rewrite
Three items are cheap against the current code and pay immediately. They are entered in `NEXT.md`, not
here.
1. **`same_narrative_plane`, as a per-detection field.** Vision already returns per-panel boxes. Add a
`plane` or `depth` to a detection, set when the model says the figure sits inside a screen, poster,
photo or drawing. That buys the containment edge with no detector. It is the whole of the remaining
identity error on the lead, and it feeds every stage below.
2. **Same-panel co-presence becomes a weak cannot-link.** `tracklets.cannot_link` currently makes it hard.
It needs item 1 first, because the plane is what makes the weak version safe.
3. **`offscreen` as a fourth `speaker_ref` kind.** The union already exists, the arm does not.
## Sources
Magi and Magiv2 for the detection-and-association-as-graph-generation formulation, the text-character
pair head, and the character bank of exemplar images plus names. Magiv3 for panels, texts, characters and
tails with their associations, and for character grounding between textual descriptions and detected
character regions.
CLAUDE.md
+1
View File
@@ -11,6 +11,7 @@ Goal, invariants, and working rules. Read this first.
| `caveats/` | every known limit and its revisit trigger, indexed in `caveats/CLAUDE.md` |
| `AGENTS.md` | commands, with the traps beside them |
| `AUDIT.md` | the 2026-08-11 pipeline audit, the source of the roadmap |
| `ARCHITECTURE.md` | the target shape of the pipeline, and what exists against it today |
| `spec-v3.md` | current quality and look work, marked DONE/TODO per item |
Do not restate a finding here. Point at the decision.
JOURNAL.md
+33
View File
@@ -723,3 +723,36 @@ Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestr
`/characters/reset` asked for it.
Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed.
## 2026-08-13 — the dialogue stage names nobody, and why
Ran `dialogue` 116/116 in 5m57s on the fourth cycle's registry, to see whether the fixed identity lets the
existing `name_claims` path name the female lead. It does not, and the six claims it produced name three
separate defects.
```
p040 character_2b1b12a1 "Choi Haeseon" caption 1.00 -> NOT promoted
p010 character_b1dd5659 "Lim Seonho" caption 1.00 -> conflict flag
p047 character_b1dd5659 "Seonho" address 0.90 -> conflict flag
p011 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
p026 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
p110 character_028d4a49 "Haeseon" address 1.00 -> already named
```
All three are fixed in `db.add_name_claim` and recorded in `decisions/identity-naming.md`: alias grouping,
a confident caption as strong evidence, and one name per character. `test_name_binding.py` replays these
six claims, 121 tests pass, and each new assert was confirmed to fail with its fix disabled.
Also wired `merge_faceless_captions` into both crop endpoints. It had been written and never called; both
endpoints called the non-destructive `context_fragment_links` instead, and no decision recorded that
choice. It does not cover the head-in-one-shot, body-in-the-next split that prompted the question, because
a body fragment has no text and `_merge_plan` only folds a fragment that has text and no face.
Found while wiring it: `crop_webtoon` skips an upload when the key exists, which is right for a resume and
silently wrong after a slicing change. Documented at the line and in `NEXT.md`.
Wrote `ARCHITECTURE.md` from the user's design: region graph, occurrence/identity/name, speaker as a scored
graph edge with a typed union, narrative plane for art-in-art, and a persistent story state machine. Every
section carries what exists against it today. Nothing in it is built.
Nothing ran on a GPU after the dialogue stage.
NEXT.md
+31 -17
View File
@@ -39,28 +39,42 @@ contain the right person. Coverage is still the `has_face` gate plus those refus
## Next
1. **Deploy the two tracklet fixes and run the fourth cycle.** Neither has touched a GPU. Rebuild the
orchestrator image, reset the registry, run vision/identity/reconcile, then re-check the lead's crops.
Expect roughly 30 tracklets over 64 crops instead of 12, so identity goes from about 1m25s to 3 or 4
minutes. Watch the lead's assignment count against 36.
1. **Re-crop the chapter and run the fifth cycle.** Four changes are written and tested since the fourth
cycle, and none has touched a GPU.
Neither fix is sufficient. Bare hair colour still links different men, and the cat still joins its
neighbours. Do not add a crop-to-crop cosine to close that: measured on this run's 22 embeddings,
different people reach 0.93 while the same person reaches 0.96, so no threshold exists
(`caveats/audit-open.md#cosine-not-identity`).
- `merge_faceless_captions` is wired into both crop endpoints. It was written, never called, and
`context_fragment_links` was called instead. A stranded caption fragment now vstacks into the
face-bearing fragment it belongs to, so panel count and every panel index change.
- three naming fixes in `db.add_name_claim` (`decisions/identity-naming.md`): alias grouping, a caption
as strong evidence, and a name held by another character refusing to promote onto a second one.
Agreed next step after the cycle, chosen by the user and not started: **stop letting cosine pick the
gallery.** There are 9 live characters. `run_stage_identity` builds `union_cands` from the members'
cosine top-k shortlists, so a metric that cannot separate people decides who is even considered. Send
the live cast instead, gender-gated, capped and logged when truncated. Note two traps found while
reading it: `/vision/resolve` sends up to 3 reference images per candidate
(`worker_vision.py:1057`), so 9 candidates is 27 images plus the query and needs a cap; and only a crop
with a non-empty cosine shortlist enters `shortlists` at all, so an empty top-k currently drops the crop
from resolution entirely.
**Clear `s3://panels/<manga>/<chapter>/panels/` before re-cropping.** `crop_webtoon` skips the upload
when the key exists, so a re-crop after a slicing change silently keeps the previous run's images.
Wiring the merge is a slicing change. Everything downstream is invalidated by it, so this is a full
re-run and not a stage rerun.
Expected: fewer than 116 panels, `2b1b12a1` named `Choi Haeseon` from the p040 caption, the lead's
`conflicting-name-claims` flag gone, and the green-dress woman no longer named `Seonho` but carrying a
`name-already-taken` flag instead.
Not fixed by any of it. Bare hair colour still links different men. Do not add a crop-to-crop cosine to
close that. Measured on 22 embeddings, different people reach 0.93 and the same person reaches 0.96, so
no threshold exists (`caveats/audit-open.md#cosine-not-identity`).
Then, separately, test embedding the FACE box rather than the person box. `face_detect` already finds
the face and pairs it for `has_face`. That is the likely root cause of cosine measuring scene instead of
person, and the test is to re-embed these same 22 detections and recompute the matrix.
person. The test is to re-embed these same 22 detections and recompute the matrix.
1b. **The head/body split that started the crop question is NOT fixed.** The wired merge only folds a
fragment that has text and no face. A body fragment carries no dialogue, so `_merge_plan` leaves it
solo and it becomes its own panel and its own shot. Finding it needs a different signal, most likely a
face touching the bottom edge of one fragment with a textless fragment below. No evidence has been
gathered yet on how often this chapter does it.
1c. **Three items from `ARCHITECTURE.md` are cheap against the current code.** A `plane` field per
detection for art-in-art, same-panel co-presence demoted to a weak cannot-link once the plane exists,
and `offscreen` as a fourth `speaker_ref` kind.
2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0
degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`).
3. **Vision boxes animals as people and dresses them.** `p081` and `p108` are cats, described
decisions/CLAUDE.md
+6 -2
View File
@@ -48,5 +48,9 @@ still live belongs in `caveats/`.
| [A roster name is a guess, so it never reaches detection](identity-bbox.md#roster-does-not-name) | closed |
| [`merged_into` is exactly one hop deep](identity-bbox.md#merge-chains-flatten) | closed |
| [`_bbox_to_pixels` orders the corners, because the model sometimes swaps them](identity-bbox.md#bbox-corners-ordered) | closed |
| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed, GPU pending |
| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed, GPU pending |
| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed |
| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed |
| [The gallery is the live cast, not cosine's top-k](identity-bbox.md#cast-is-the-gallery) | closed |
| [A name is a word set, not a string](identity-naming.md#alias-grouping) | closed, GPU pending |
| [A confident caption names a character on its own](identity-naming.md#caption-is-strong) | closed, GPU pending |
| [A name belongs to one character](identity-naming.md#one-name-one-character) | closed, GPU pending |
decisions/identity-naming.md
+51
View File
@@ -0,0 +1,51 @@
# Naming a character
How a discovered name reaches `characters.name`. The mechanism is `db.add_name_claim`, fed by the dialogue
stage through `service._absorb`.
## A name is a word set, not a string {#alias-grouping}
**Closed, 2026-08-13, not yet run on a GPU.**
Claims grouped on the casefolded name, so `Lim Seonho` from a p010 caption and `Seonho` from a p047 address
counted as two names for one character. `len(grouped) > 1` fired, a `conflicting-name-claims` flag was
filed, and promotion was blocked permanently on evidence that in fact corroborated.
`alias_groups` groups two names when one's word set contains the other's, and keeps the longer as
canonical. `Seonho` and `Lim Seonho` are one name and the registry stores `Lim Seonho`. `Seonho` and
`Haeseon` are still two, so a real conflict still flags.
## A confident caption names a character on its own {#caption-is-strong}
**Closed, 2026-08-13, not yet run on a GPU.**
Promotion needed two distinct panels, or one `self_intro` or `name_tag` claim above 0.9. On the 2026-08-12
chapter the only claim naming a main character was `Choi Haeseon`, a caption at p040 at confidence 1.0, and
it was discarded. She held 13 correct crops and stayed anonymous through the whole pipeline.
A caption is the narration naming the person it is drawn beside, which is how a webtoon introduces its
cast, so it joins `STRONG_EVIDENCE`. This is the loosest of the three changes and is only safe because of
the next one.
## A name belongs to one character {#one-name-one-character}
**Closed, 2026-08-13, not yet run on a GPU.**
Being addressed by name identifies the addressee. Choosing which drawn body that is fails often. Two panels
addressed `Seonho`, the dialogue model pointed `target_local_id` at the woman standing beside him, and two
independent claims promoted her. The registry then held a female `Seonho` over 9 crops beside the lead's
`LIM SEONHO`.
A promotion now checks every other live character of the same manga first, by alias group. A collision
refuses the promotion and files a `name-already-taken` flag carrying both ids. The collision is itself
evidence that either the addressee or the identity cluster is wrong, so it is worth surfacing rather than
resolving silently.
Deliberately not built: no attempt to decide WHICH character deserves the name. That needs the addressee
fixed, which is `ARCHITECTURE.md` section 3.
## Checks
`test_name_binding.py` replays the six real claims from the 2026-08-12 chapter. Each new assert was
confirmed to fail with its fix disabled: the caption test with `STRONG_EVIDENCE` reverted, the alias test
with casefold grouping restored. The taken-name test asserts a flag kind that only the new branch emits.
worker_crop.py
+5 -2
View File
@@ -240,12 +240,15 @@ async def crop_webtoon(data: WebtoonInput):
tag = uuid.uuid4().hex[:8]
locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)]
strip = restitch(locals_)
crops = slice_webtoon(strip)
crops = merge_faceless_captions(slice_webtoon(strip))
context_links = context_fragment_links(crops)
panels = []
for idx, (crop_img, bbox) in enumerate(crops):
uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png"
# slicing is deterministic, so on a resume the same idx -> same key; skip re-upload.
# TRAP: that holds only while the PLAN is unchanged. Edit slice_webtoon or the merge pass and a
# re-crop silently keeps the previous run's images under the same keys, because every one of them
# already exists. Clear the s3://panels/<manga>/<chapter>/panels/ prefix before re-cropping.
if not transport.exists(uri):
out = f"{SHM}/wt_{tag}_p{idx:03d}.png"
cv2.imwrite(out, crop_img)
@@ -266,7 +269,7 @@ async def crop(data: CropInput):
raise HTTPException(400, f"page not readable: {data.page_uri}")
h, w = img.shape[:2]
webtoon = h / w >= WEBTOON_RATIO
crops = slice_webtoon(img) if webtoon else kumiko_panels(local, data.rtl)
crops = merge_faceless_captions(slice_webtoon(img)) if webtoon else kumiko_panels(local, data.rtl)
context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))}
ambiguous = flag_overlaps(crops, data.page_index)