diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..2b9adad --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,368 @@ +# ARCHITECTURE + +The target shape of the pipeline, written 2026-08-13 from the user's design. This is **not** what the code +does. `NEXT.md` holds the live state and `AUDIT.md` holds the current pipeline. Every section here ends +with what exists today, so the gap is legible without reading both. + +The governing principle: + +> Do not make the next panel understand the previous panel. Make it understand the current world state +> produced by all previous panels. + +Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below +follows from that split. + +## 1. The page is a region graph, not a list of panels + +``` +page + ├─ regions + │ ├─ panel + │ ├─ inset_panel + │ ├─ embedded_art + │ ├─ text + │ ├─ tail + │ └─ character_occurrence + │ + └─ edges + ├─ contains(region, region) + ├─ reads_before(text, text) + ├─ tail_of(tail, text) + ├─ points_to(tail, character) + ├─ spoken_by(text, character) + └─ same_identity(character, character) +``` + +A flat set of panels cannot express a television inside a room. That is the defect the current pipeline +shows most often. + +**Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a +non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no +containment edge, no tail region and no region type. + +## 2. Identity exists independently of names + +``` +occurrence c42 + -> identity char_07 + name = null + aliases = [] +``` + +`char_07.name` may be filled later, or stay null forever and display as `unknown character #7`. The +occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster. +Three levels, never collapsed into one. + +**Today:** the schema already has this split. `identity_assignments` is the occurrence, +`characters` owns the identity, `name` is nullable and downstream already falls back to an anonymous +display. What is missing is the clustering, not the separation. See section 4. + +## 3. Speaker attribution is a scored graph edge, not a procedure + +Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge: + +``` +score(text, character) = + learned_t2c_score + + tail_evidence + + spatial_evidence + + same_panel + + dialogue_continuity + + character_activity_prior + + identity_context +``` + +`learned_t2c_score` is the load-bearing term: a pair classifier over the whole page, the text object's +visual feature and the character object's visual feature. Magi's text-character head does exactly this. +It can start as a tiny MLP: + +``` +t2c(text_embedding, character_embedding, page_context, geometry_features) -> p(speaker) +``` + +with geometry carrying normalized relative position, distance, overlap, same-panel, containment depth and +tail direction. + +Then the cases fall out of one mechanism instead of four: + +| case | what carries it | +| --- | --- | +| bubble with a tail | `t2c` + tail, usually decisive | +| bubble with no tail | `t2c` + spatial and context | +| speaker outside the panel | recent identities + an offscreen candidate | +| narration | the narrator candidate | +| nothing resolves | unknown speaker | + +**A dialogue line must not be required to resolve to a visible character.** That is a failure mode, not a +safeguard. The speaker type is a union: + +``` +speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown +``` + +**Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator` +(`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a +prompt to gemma over a window of panels, with no geometry term at all. The `det`/`seg` tail heads exist +and are unused (`caveats/speaker-attribution.md#tail-is-not-geometry`). + +## 4. Character recognition is occurrence, then identity, then name + +``` +character detection + ↓ +occurrence embeddings + ↓ +pairwise same_identity probabilities + ↓ +chapter-wide constrained clustering + ↓ +char_001, char_002, ... + ↓ +optional character-bank lookup + ↓ +name or unknown +``` + +Two rules that the current code gets wrong. + +**The embedding is not the character crop alone.** Combine four signals: the character crop, the face or +head crop, the full-body crop, and a contextual object feature. Magiv2 combines detected object features +with a separate crop-embedding model. + +**Cluster chapter-wide, not page by page.** + +**Two characters in the same panel may be one person.** Mirrors, photographs, flashbacks, insets, +screens, imagined scenes and repeated action drawings all break that rule. Make it a weak cannot-link, +and only when the two are on the same narrative plane. + +**Today:** the embedding is the person box only, which is measurably the wrong signal +(`caveats/audit-open.md#cosine-not-identity`). Clustering is greedy and local: `tracklets.link_tracklets` +groups within an 8-panel window. `tracklets.cannot_link` treats same-panel co-presence as a **hard** +constraint, which is exactly the correction above. Naming is `db.add_name_claim`, corroboration over +`name_claims`. + +## 5. The art-in-art problem needs a narrative plane + +Treat the page as a hierarchical scene graph: + +``` +page +└── panel A depth=0 + ├── character c1 + ├── text t1 + └── television/poster depth=1, type=embedded_art + ├── character c2 + └── text t2 +``` + +Speaker candidates normally come from the same `scene_depth`. Otherwise a real character standing beside a +poster of a drawn person can be given the poster person's line. + +A region classifier predicts a type: + +``` +story_scene | inset_story_panel | flashback | screen | photo | poster | illustration | decorative +``` + +Perfect classification is not the point. The output that matters is one probability: + +``` +same_narrative_plane(a, b) +``` + +which then enters the association score in section 3 and the cannot-link in section 4. + +**Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the +19:44 run of 2026-08-12 his 16 assignments were 14 correct plus a photograph of another man and a chibi +drawing. Both are art inside a panel. Vision also boxes cats as people and dresses them (`p081`, `p108`). + +## 6. Narrative understanding is a state machine, not a per-panel description + +``` +story_state +├─ entities (characters, locations, important objects) +├─ scenes +├─ timeline +├─ relationships +├─ unresolved_threads +├─ facts +└─ hypotheses +``` + +A panel produces a **delta**, not another standalone prose interpretation: + +``` +panel 142: +- character_07 enters room_03 +- character_02 is already present +- character_07 says "..." +- object_12 changes owner: 02 -> 07 +- possible flashback begins +``` + +### Facts, hypotheses and unknowns are different records + +``` +fact: source=panel_142 confidence=0.99 character_07 is visible +hypothesis: confidence=0.64 character_07 is angry +unknown: who caused the explosion +``` + +A later panel strengthens, replaces or invalidates a hypothesis without rewriting history. + +### Scene state is explicit and inherited + +``` +scene_31: + location: school_rooftop + time: evening + participants: {char_03: present, char_07: present, char_11: offscreen} + pov: null + narrative_mode: present + parent_scene: null +``` + +A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character +absent for one panel has not left. A panel with no background has not changed location. A tail-less line +keeps the offscreen participant as a candidate. A close-up still belongs to the scene. + +### Classify the transition, not just the panel + +``` +CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START +FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE +``` + +`EMBEDDED_SCENE` is what stops a television's contents mutating the room around it: + +``` +scene_12 present + ├─ panel 101 + ├─ panel 102 + └─ embedded scene_13 [television] + ├─ panel-like region + └─ char_19 +``` + +### Character state is written by a resolver, never by the vision model + +``` +char_07: + known_names: [...] + currently_at: room_03 + status: alive + appearance_state: {clothes: school_uniform, injured: true} + relationships: {char_02: friend?} + last_seen: panel_142 +``` + +The path is `observation -> resolver -> state transition`, and the resolver may reject an impossible +update. + +### Conversation state is its own record + +``` +conversation_18: + scene: scene_31 + participants: [char_02, char_07] + last_speaker: char_07 + addressee: char_02 + topic: missing_key +``` + +This is the strongest available prior for a tail-less bubble. Given `A: where did you put it? / ... / +A: don't lie.`, turn-taking assigns the middle line with no visual evidence at all. + +### An unresolved reference survives instead of being forced + +``` +unknown_04: + type: person + descriptions: ["the man from yesterday", "silhouette in panel_58"] + candidate_ids: {char_12: 0.55, char_19: 0.22} +``` + +Chapter 6 may reveal `unknown_04 == char_12`, and that identity back-propagates through the graph. The +same applies to unnamed characters, pronouns, disguised characters, mysterious objects and unseen +speakers. + +### Two memories + +- **Working narrative state**: the current scene and the recent ones, in detail. +- **Canonical long-term memory**: compressed facts, not chapter summaries. `char_07 learned that char_02 + betrayed the group.` `object_04 is held by char_11.` `char_03 does not know char_07 survived.` + +### A chapter boundary is a checkpoint, not a reset + +``` +chapter_checkpoint: + persistent_entity_changes / relationship_changes / location and status changes + newly established facts / unresolved questions / active plot threads / final scene state +``` + +Chapter `n+1` starts from that. The detailed panel graph may be kept forever. Only five things load into +the model: the current scene, the previous scene, the relevant character records, the active threads, and +retrieved old facts. + +### A consistency checker runs after each scene and chapter + +Seven checks. A dead character appearing normally. A character knowing a fact before learning it. An +object owned by two people at once. A flashback never closed. A location jump with no transition. A +speaker who was neither present nor offscreen. A name that conflicts with the identity graph. The model +proposes corrections. The graph stays the source of truth. + +**Today:** none of this exists. Each stage reads its predecessor's blob for one panel or one beat. +`recent` is a rolling list of the last few dialogue lines and is the only carried state. Chapter +boundaries are a reset. There is no fact-versus-hypothesis distinction anywhere, which is why narration +asserts things no panel shows (`NEXT.md` item 6). + +## 7. The staged version worth building + +Do not recreate Magi's monolithic network first. Detection, vision and character embeddings already exist, +so stage it: + +``` +page + ↓ +region detector panels / nested regions / texts / characters / tails + ↓ +object feature extraction + ↓ +three pair models character↔character (identity) + text→character (speaker) + text→tail (bubble structure) + ↓ +chapter graph + ↓ +global character clustering + ↓ +optional naming + ↓ +ocr + reading order + ↓ +dialogue stream +``` + +The VLM then judges only the ambiguous graph edges. It no longer rediscovers every character and dialogue +relationship from raw pixels on every panel. Magi formulates detection and association as graph +generation, which is why it beats a crop, OCR and nearest-character pipeline here. + +## What to take from this before the rewrite + +Three items are cheap against the current code and pay immediately. They are entered in `NEXT.md`, not +here. + +1. **`same_narrative_plane`, as a per-detection field.** Vision already returns per-panel boxes. Add a + `plane` or `depth` to a detection, set when the model says the figure sits inside a screen, poster, + photo or drawing. That buys the containment edge with no detector. It is the whole of the remaining + identity error on the lead, and it feeds every stage below. +2. **Same-panel co-presence becomes a weak cannot-link.** `tracklets.cannot_link` currently makes it hard. + It needs item 1 first, because the plane is what makes the weak version safe. +3. **`offscreen` as a fourth `speaker_ref` kind.** The union already exists, the arm does not. + +## Sources + +Magi and Magiv2 for the detection-and-association-as-graph-generation formulation, the text-character +pair head, and the character bank of exemplar images plus names. Magiv3 for panels, texts, characters and +tails with their associations, and for character grounding between textual descriptions and detected +character regions. diff --git a/CLAUDE.md b/CLAUDE.md index 666e431..34e92fc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,6 +11,7 @@ Goal, invariants, and working rules. Read this first. | `caveats/` | every known limit and its revisit trigger, indexed in `caveats/CLAUDE.md` | | `AGENTS.md` | commands, with the traps beside them | | `AUDIT.md` | the 2026-08-11 pipeline audit, the source of the roadmap | +| `ARCHITECTURE.md` | the target shape of the pipeline, and what exists against it today | | `spec-v3.md` | current quality and look work, marked DONE/TODO per item | Do not restate a finding here. Point at the decision. diff --git a/JOURNAL.md b/JOURNAL.md index c23cf3a..f4c2a40 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -723,3 +723,36 @@ Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestr `/characters/reset` asked for it. Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed. + +## 2026-08-13 — the dialogue stage names nobody, and why + +Ran `dialogue` 116/116 in 5m57s on the fourth cycle's registry, to see whether the fixed identity lets the +existing `name_claims` path name the female lead. It does not, and the six claims it produced name three +separate defects. + +``` +p040 character_2b1b12a1 "Choi Haeseon" caption 1.00 -> NOT promoted +p010 character_b1dd5659 "Lim Seonho" caption 1.00 -> conflict flag +p047 character_b1dd5659 "Seonho" address 0.90 -> conflict flag +p011 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED +p026 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED +p110 character_028d4a49 "Haeseon" address 1.00 -> already named +``` + +All three are fixed in `db.add_name_claim` and recorded in `decisions/identity-naming.md`: alias grouping, +a confident caption as strong evidence, and one name per character. `test_name_binding.py` replays these +six claims, 121 tests pass, and each new assert was confirmed to fail with its fix disabled. + +Also wired `merge_faceless_captions` into both crop endpoints. It had been written and never called; both +endpoints called the non-destructive `context_fragment_links` instead, and no decision recorded that +choice. It does not cover the head-in-one-shot, body-in-the-next split that prompted the question, because +a body fragment has no text and `_merge_plan` only folds a fragment that has text and no face. + +Found while wiring it: `crop_webtoon` skips an upload when the key exists, which is right for a resume and +silently wrong after a slicing change. Documented at the line and in `NEXT.md`. + +Wrote `ARCHITECTURE.md` from the user's design: region graph, occurrence/identity/name, speaker as a scored +graph edge with a typed union, narrative plane for art-in-art, and a persistent story state machine. Every +section carries what exists against it today. Nothing in it is built. + +Nothing ran on a GPU after the dialogue stage. diff --git a/NEXT.md b/NEXT.md index 05c198d..b61b241 100644 --- a/NEXT.md +++ b/NEXT.md @@ -39,28 +39,42 @@ contain the right person. Coverage is still the `has_face` gate plus those refus ## Next -1. **Deploy the two tracklet fixes and run the fourth cycle.** Neither has touched a GPU. Rebuild the - orchestrator image, reset the registry, run vision/identity/reconcile, then re-check the lead's crops. - Expect roughly 30 tracklets over 64 crops instead of 12, so identity goes from about 1m25s to 3 or 4 - minutes. Watch the lead's assignment count against 36. +1. **Re-crop the chapter and run the fifth cycle.** Four changes are written and tested since the fourth + cycle, and none has touched a GPU. - Neither fix is sufficient. Bare hair colour still links different men, and the cat still joins its - neighbours. Do not add a crop-to-crop cosine to close that: measured on this run's 22 embeddings, - different people reach 0.93 while the same person reaches 0.96, so no threshold exists - (`caveats/audit-open.md#cosine-not-identity`). + - `merge_faceless_captions` is wired into both crop endpoints. It was written, never called, and + `context_fragment_links` was called instead. A stranded caption fragment now vstacks into the + face-bearing fragment it belongs to, so panel count and every panel index change. + - three naming fixes in `db.add_name_claim` (`decisions/identity-naming.md`): alias grouping, a caption + as strong evidence, and a name held by another character refusing to promote onto a second one. - Agreed next step after the cycle, chosen by the user and not started: **stop letting cosine pick the - gallery.** There are 9 live characters. `run_stage_identity` builds `union_cands` from the members' - cosine top-k shortlists, so a metric that cannot separate people decides who is even considered. Send - the live cast instead, gender-gated, capped and logged when truncated. Note two traps found while - reading it: `/vision/resolve` sends up to 3 reference images per candidate - (`worker_vision.py:1057`), so 9 candidates is 27 images plus the query and needs a cap; and only a crop - with a non-empty cosine shortlist enters `shortlists` at all, so an empty top-k currently drops the crop - from resolution entirely. + **Clear `s3://panels///panels/` before re-cropping.** `crop_webtoon` skips the upload + when the key exists, so a re-crop after a slicing change silently keeps the previous run's images. + Wiring the merge is a slicing change. Everything downstream is invalidated by it, so this is a full + re-run and not a stage rerun. + + Expected: fewer than 116 panels, `2b1b12a1` named `Choi Haeseon` from the p040 caption, the lead's + `conflicting-name-claims` flag gone, and the green-dress woman no longer named `Seonho` but carrying a + `name-already-taken` flag instead. + + Not fixed by any of it. Bare hair colour still links different men. Do not add a crop-to-crop cosine to + close that. Measured on 22 embeddings, different people reach 0.93 and the same person reaches 0.96, so + no threshold exists (`caveats/audit-open.md#cosine-not-identity`). Then, separately, test embedding the FACE box rather than the person box. `face_detect` already finds the face and pairs it for `has_face`. That is the likely root cause of cosine measuring scene instead of - person, and the test is to re-embed these same 22 detections and recompute the matrix. + person. The test is to re-embed these same 22 detections and recompute the matrix. + +1b. **The head/body split that started the crop question is NOT fixed.** The wired merge only folds a + fragment that has text and no face. A body fragment carries no dialogue, so `_merge_plan` leaves it + solo and it becomes its own panel and its own shot. Finding it needs a different signal, most likely a + face touching the bottom edge of one fragment with a textless fragment below. No evidence has been + gathered yet on how often this chapter does it. + +1c. **Three items from `ARCHITECTURE.md` are cheap against the current code.** A `plane` field per + detection for art-in-art, same-panel co-presence demoted to a weak cannot-link once the plane exists, + and `offscreen` as a fourth `speaker_ref` kind. + 2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0 degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`). 3. **Vision boxes animals as people and dresses them.** `p081` and `p108` are cats, described diff --git a/decisions/CLAUDE.md b/decisions/CLAUDE.md index 73278ab..918048e 100644 --- a/decisions/CLAUDE.md +++ b/decisions/CLAUDE.md @@ -48,5 +48,9 @@ still live belongs in `caveats/`. | [A roster name is a guess, so it never reaches detection](identity-bbox.md#roster-does-not-name) | closed | | [`merged_into` is exactly one hop deep](identity-bbox.md#merge-chains-flatten) | closed | | [`_bbox_to_pixels` orders the corners, because the model sometimes swaps them](identity-bbox.md#bbox-corners-ordered) | closed | -| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed, GPU pending | -| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed, GPU pending | +| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed | +| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed | +| [The gallery is the live cast, not cosine's top-k](identity-bbox.md#cast-is-the-gallery) | closed | +| [A name is a word set, not a string](identity-naming.md#alias-grouping) | closed, GPU pending | +| [A confident caption names a character on its own](identity-naming.md#caption-is-strong) | closed, GPU pending | +| [A name belongs to one character](identity-naming.md#one-name-one-character) | closed, GPU pending | diff --git a/decisions/identity-naming.md b/decisions/identity-naming.md new file mode 100644 index 0000000..ea36ddd --- /dev/null +++ b/decisions/identity-naming.md @@ -0,0 +1,51 @@ +# Naming a character + +How a discovered name reaches `characters.name`. The mechanism is `db.add_name_claim`, fed by the dialogue +stage through `service._absorb`. + +## A name is a word set, not a string {#alias-grouping} + +**Closed, 2026-08-13, not yet run on a GPU.** + +Claims grouped on the casefolded name, so `Lim Seonho` from a p010 caption and `Seonho` from a p047 address +counted as two names for one character. `len(grouped) > 1` fired, a `conflicting-name-claims` flag was +filed, and promotion was blocked permanently on evidence that in fact corroborated. + +`alias_groups` groups two names when one's word set contains the other's, and keeps the longer as +canonical. `Seonho` and `Lim Seonho` are one name and the registry stores `Lim Seonho`. `Seonho` and +`Haeseon` are still two, so a real conflict still flags. + +## A confident caption names a character on its own {#caption-is-strong} + +**Closed, 2026-08-13, not yet run on a GPU.** + +Promotion needed two distinct panels, or one `self_intro` or `name_tag` claim above 0.9. On the 2026-08-12 +chapter the only claim naming a main character was `Choi Haeseon`, a caption at p040 at confidence 1.0, and +it was discarded. She held 13 correct crops and stayed anonymous through the whole pipeline. + +A caption is the narration naming the person it is drawn beside, which is how a webtoon introduces its +cast, so it joins `STRONG_EVIDENCE`. This is the loosest of the three changes and is only safe because of +the next one. + +## A name belongs to one character {#one-name-one-character} + +**Closed, 2026-08-13, not yet run on a GPU.** + +Being addressed by name identifies the addressee. Choosing which drawn body that is fails often. Two panels +addressed `Seonho`, the dialogue model pointed `target_local_id` at the woman standing beside him, and two +independent claims promoted her. The registry then held a female `Seonho` over 9 crops beside the lead's +`LIM SEONHO`. + +A promotion now checks every other live character of the same manga first, by alias group. A collision +refuses the promotion and files a `name-already-taken` flag carrying both ids. The collision is itself +evidence that either the addressee or the identity cluster is wrong, so it is worth surfacing rather than +resolving silently. + +Deliberately not built: no attempt to decide WHICH character deserves the name. That needs the addressee +fixed, which is `ARCHITECTURE.md` section 3. + +## Checks + +`test_name_binding.py` replays the six real claims from the 2026-08-12 chapter. Each new assert was +confirmed to fail with its fix disabled: the caption test with `STRONG_EVIDENCE` reverted, the alias test +with casefold grouping restored. The taken-name test asserts a flag kind that only the new branch emits. diff --git a/worker_crop.py b/worker_crop.py index 5c5fa48..e0ec4ee 100644 --- a/worker_crop.py +++ b/worker_crop.py @@ -240,12 +240,15 @@ async def crop_webtoon(data: WebtoonInput): tag = uuid.uuid4().hex[:8] locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)] strip = restitch(locals_) - crops = slice_webtoon(strip) + crops = merge_faceless_captions(slice_webtoon(strip)) context_links = context_fragment_links(crops) panels = [] for idx, (crop_img, bbox) in enumerate(crops): uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png" # slicing is deterministic, so on a resume the same idx -> same key; skip re-upload. + # TRAP: that holds only while the PLAN is unchanged. Edit slice_webtoon or the merge pass and a + # re-crop silently keeps the previous run's images under the same keys, because every one of them + # already exists. Clear the s3://panels///panels/ prefix before re-cropping. if not transport.exists(uri): out = f"{SHM}/wt_{tag}_p{idx:03d}.png" cv2.imwrite(out, crop_img) @@ -266,7 +269,7 @@ async def crop(data: CropInput): raise HTTPException(400, f"page not readable: {data.page_uri}") h, w = img.shape[:2] webtoon = h / w >= WEBTOON_RATIO - crops = slice_webtoon(img) if webtoon else kumiko_panels(local, data.rtl) + crops = merge_faceless_captions(slice_webtoon(img)) if webtoon else kumiko_panels(local, data.rtl) context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))} ambiguous = flag_overlaps(crops, data.page_index)