From 12d4ed8534a5061b08e67b65abddf2a74dc143ed Mon Sep 17 00:00:00 2001 From: kami Date: Thu, 13 Aug 2026 22:28:57 +0400 Subject: [PATCH] Rewrite the architecture under the no-Magi constraint Magi and any learned head are ruled out, so every structure comes out of a gemma prompt field or plain Python over gemma's output. Each section now carries what exists today and a DoD. The build order starts at step 0, the measurement spine: eval/chapter-truth.json plus purity and fragmentation printed by audit_registry.py. Nothing below it is measurable without that file. Co-Authored-By: Claude Opus 5 --- ARCHITECTURE.md | 429 +++++++++++++++++++++++------------------------- NEXT.md | 12 +- 2 files changed, 216 insertions(+), 225 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 2b9adad..67def2e 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,8 +1,9 @@ # ARCHITECTURE -The target shape of the pipeline, written 2026-08-13 from the user's design. This is **not** what the code -does. `NEXT.md` holds the live state and `AUDIT.md` holds the current pipeline. Every section here ends -with what exists today, so the gap is legible without reading both. +The target shape of the pipeline. Written 2026-08-13 from the user's design, rewritten the same day under +the no-Magi constraint. This is **not** what the code does. `NEXT.md` holds the live state and `AUDIT.md` +holds the current pipeline. Every section ends with what exists today and what would make it done. The +gap stays legible and testable without reading both. The governing principle: @@ -12,6 +13,49 @@ The governing principle: Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below follows from that split. +## Ruled out + +Read this before proposing any of it again. + +| ruled out | why | who decided | +| --- | --- | --- | +| Magi, Magiv2, Magiv3 as a detector or as pair heads | project constraint, not a measurement | user, 2026-08-13 | +| A trained `t2c` pair model, or any learned head | no labelled pages exist and no labeller is planned | follows from the above | +| Crop-to-crop cosine as a link signal | measured: two men reach 0.93, one man reaches 0.96, no threshold exists (`caveats/audit-open.md#cosine-not-identity`) | 2026-08-12 | + +The consequence runs through the whole document. **Every structure below comes out of a gemma prompt +field, or out of plain Python over gemma's output.** Nothing below is trained. A learned score +in the original design becomes a hand-weighted sum. The weights are read off the labelled chapter, and +the DoD is the accuracy number rather than the mechanism. + +## The measurement spine + +Nothing below can be called done without this, and it does not exist yet. + +The only ground truth in the project is the eyeball pass over the 19:44 run of 2026-08-12. It lives in +prose in `NEXT.md`. Write it to `eval/chapter-truth.json` against chapter +`7c944dd4-e972-42c7-ba60-9f6939548e80`, scoped to what was already checked by eye rather than to all 119 +detections: + +- the three characters walked crop by crop, each assignment marked as the real person or not +- 30 dialogue lines with their true speaker, typed as `visible | offscreen | narrator | unknown` + +The baseline it records, from that run: + +| character | assignments | correct | purity | note | +| --- | --- | --- | --- | --- | +| the lead | 16 | 14 | 0.88 | plus a photograph at `order 17` and a chibi at `order 20` | +| `character_2b1b12a1` | 13 | 13 | 1.00 | a main character the registry never named | +| `character_f0d4e901` | 9 | 7 | 0.78 | the other 2 are `2b1b12a1` | + +Woman A is `2b1b12a1`. She has 15 occurrences split across 2 ids, so her fragmentation is 2. + +`audit_registry.py` already walks panels, reads `identity_assignments` and counts per character. Extend +it to print purity and fragmentation against the truth file. Do not write an eval harness. + +**Done when:** `audit_registry.py ` prints purity per labelled character and fragmentation per +labelled person, and reproduces the table above on the 19:44 run. + ## 1. The page is a region graph, not a list of panels ``` @@ -36,9 +80,29 @@ page A flat set of panels cannot express a television inside a room. That is the defect the current pipeline shows most often. +With no detector to train, the graph comes out of the detection prompt. Gemma already returns a box per +character and per text. Two fields per detection buy most of the graph with no new model: + +``` +plane = story | screen | photo | poster | drawing | flashback | dream +species = human | animal | object +``` + +`plane` is the containment edge in disguise. A detection whose `plane` is not `story` sits inside embedded +art, and that is the fact every stage below needs. `species` is a separate axis and exists because vision +boxes cats as people and dresses them. + +Tail regions stay unbuilt. The `det`/`seg` heads exist and are unused +(`caveats/speaker-attribution.md#tail-is-not-geometry`), and section 3 says why they are not the first +thing to spend on. + **Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no -containment edge, no tail region and no region type. +containment edge, no region type, no `plane` and no `species`. + +**Done when:** every detection carries `plane` and `species`. On the labelled chapter, `order 17` and +`order 20` of the lead are not `story`, none of his 14 correct crops is demoted, and `p081` and `p108` are +`animal`. Measured by `audit_registry.py`, which already reads the vision blob per panel. ## 2. Identity exists independently of names @@ -53,43 +117,42 @@ occurrence c42 occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster. Three levels, never collapsed into one. -**Today:** the schema already has this split. `identity_assignments` is the occurrence, -`characters` owns the identity, `name` is nullable and downstream already falls back to an anonymous -display. What is missing is the clustering, not the separation. See section 4. +**Today:** already true. `identity_assignments` is the occurrence, `characters` owns the identity, `name` +is nullable and downstream already falls back to an anonymous display. -## 3. Speaker attribution is a scored graph edge, not a procedure +**Done when:** already done. No work item. The clustering is section 4 and the naming is +`decisions/identity-naming.md`. + +## 3. Speaker attribution is a scored edge, not a procedure Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge: ``` score(text, character) = - learned_t2c_score - + tail_evidence - + spatial_evidence - + same_panel - + dialogue_continuity - + character_activity_prior - + identity_context + w1 * gemma_answer + + w2 * spatial_evidence + + w3 * same_panel + + w4 * same_plane + + w5 * conversation_continuity + + w6 * character_activity_prior ``` -`learned_t2c_score` is the load-bearing term: a pair classifier over the whole page, the text object's -visual feature and the character object's visual feature. Magi's text-character head does exactly this. -It can start as a tiny MLP: +The original design put a learned `t2c` head in the first term and called it load-bearing. No labelled +pages exist, so that term does not. **Gemma's answer becomes one term of six rather than the whole +procedure.** The geometry terms overrule it when they agree against it. The weights are constants read off +the 30 labelled lines. Six numbers in a module, not a training run. -``` -t2c(text_embedding, character_embedding, page_context, geometry_features) -> p(speaker) -``` - -with geometry carrying normalized relative position, distance, overlap, same-panel, containment depth and -tail direction. +Geometry carries normalized relative position, distance, overlap, same-panel and containment depth. Tail +direction is absent until a tail region exists, and it is not the first thing to build. `conversation +continuity` is free, and turn-taking is the strongest prior for a tail-less bubble. Then the cases fall out of one mechanism instead of four: | case | what carries it | | --- | --- | -| bubble with a tail | `t2c` + tail, usually decisive | -| bubble with no tail | `t2c` + spatial and context | -| speaker outside the panel | recent identities + an offscreen candidate | +| bubble with a tail | gemma plus spatial, usually decisive | +| bubble with no tail | conversation continuity plus spatial | +| speaker outside the panel | recent identities plus an offscreen candidate | | narration | the narrator candidate | | nothing resolves | unknown speaker | @@ -102,8 +165,13 @@ speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown **Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator` (`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a -prompt to gemma over a window of panels, with no geometry term at all. The `det`/`seg` tail heads exist -and are unused (`caveats/speaker-attribution.md#tail-is-not-geometry`). +prompt to gemma over a window of panels, with no geometry term at all. + +**Done when:** `audit_speakers.py` reports accuracy over the 30 labelled lines, split by true type, and +the scored version beats the recorded gemma-window baseline. Two numbers must move the right way, and both +are reported. Correct assignments go up. **Forced** errors go down, where forced means a line given a +visible character while the truth is `offscreen`, `narrator` or `unknown`. Record the baseline before +touching the code. ## 4. Character recognition is occurrence, then identity, then name @@ -112,257 +180,174 @@ character detection ↓ occurrence embeddings ↓ -pairwise same_identity probabilities +pairwise same_identity scores ↓ chapter-wide constrained clustering ↓ char_001, char_002, ... ↓ -optional character-bank lookup +optional name claim ↓ name or unknown ``` -Two rules that the current code gets wrong. +Three rules the current code gets wrong. -**The embedding is not the character crop alone.** Combine four signals: the character crop, the face or -head crop, the full-body crop, and a contextual object feature. Magiv2 combines detected object features -with a separate crop-embedding model. +**The embedding is not the character crop alone.** The crop embedding measures scene, not person, which is +why two men reach 0.93. Combine the face or head crop with the person crop instead of replacing one with +the other. `face_detect` already finds the face and pairs it for `has_face`, so the face box is free. This +is the queued experiment in `NEXT.md` item 1: re-embed the same 22 detections and recompute the matrix. -**Cluster chapter-wide, not page by page.** +**Cluster chapter-wide, not page by page.** `tracklets.link_tracklets` groups within an 8-panel window. -**Two characters in the same panel may be one person.** Mirrors, photographs, flashbacks, insets, -screens, imagined scenes and repeated action drawings all break that rule. Make it a weak cannot-link, -and only when the two are on the same narrative plane. +**Two characters in the same panel may be one person.** Seven things break that rule. Mirrors, +photographs, flashbacks, insets, screens, imagined scenes, repeated action drawings. Make it a weak +cannot-link, and only between detections on the same `plane`. -**Today:** the embedding is the person box only, which is measurably the wrong signal -(`caveats/audit-open.md#cosine-not-identity`). Clustering is greedy and local: `tracklets.link_tracklets` -groups within an 8-panel window. `tracklets.cannot_link` treats same-panel co-presence as a **hard** -constraint, which is exactly the correction above. Naming is `db.add_name_claim`, corroboration over -`name_claims`. +That last rule has an ordering trap. Same-panel co-presence is currently a **hard** constraint and it is +load-bearing precisely because cosine cannot separate people. Weakening it before the embedding improves +will regress purity. The dependency is the embedding fix, not the `plane` field alone. -## 5. The art-in-art problem needs a narrative plane +**Today:** the embedding is the person box only (`caveats/audit-open.md#cosine-not-identity`). Clustering +is greedy and local. `tracklets.cannot_link` treats same-panel co-presence as hard. Naming is +`db.add_name_claim`, corroboration over `name_claims`. -Treat the page as a hierarchical scene graph: +**Done when:** no labelled character holds more than one wrong assignment, and woman A's fragmentation is +1. Baseline is 2 wrong, 0 wrong, 2 wrong, and fragmentation 2. The bar is stated in errors rather than in +a purity ratio on purpose. The three characters hold 16, 13 and 9 assignments. At those counts any ratio +above 0.94 means zero tolerated errors, and the ratio hides that. + +The face-plus-person embedding lands first and carries its own smaller check. On the 22 measured +detections, the highest different-person pair must fall below the lowest same-person pair. + +## 5. The narrative plane is what stops art-in-art + +The page is a hierarchical scene graph: ``` page -└── panel A depth=0 +└── panel A plane=story ├── character c1 ├── text t1 - └── television/poster depth=1, type=embedded_art + └── television/poster plane=screen ├── character c2 └── text t2 ``` -Speaker candidates normally come from the same `scene_depth`. Otherwise a real character standing beside a -poster of a drawn person can be given the poster person's line. - -A region classifier predicts a type: - -``` -story_scene | inset_story_panel | flashback | screen | photo | poster | illustration | decorative -``` - -Perfect classification is not the point. The output that matters is one probability: +Perfect classification is not the point. The output that matters is one predicate: ``` same_narrative_plane(a, b) ``` -which then enters the association score in section 3 and the cannot-link in section 4. +It has exactly two consumers, and they are the reason the field is worth adding at all: + +- section 3, as the `same_plane` term. A real character beside a poster does not get the poster person's + line. +- section 4, as the guard that makes the weak cannot-link safe. **Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the -19:44 run of 2026-08-12 his 16 assignments were 14 correct plus a photograph of another man and a chibi -drawing. Both are art inside a panel. Vision also boxes cats as people and dresses them (`p081`, `p108`). +19:44 run his 16 assignments were 14 correct plus a photograph of another man and a chibi drawing. Both +are art inside a panel. -## 6. Narrative understanding is a state machine, not a per-panel description +**Done when:** section 1's DoD, plus both consumers wired, plus section 4's purity DoD holds with the +cannot-link demoted to weak. If purity regresses when the constraint is weakened, the embedding is not +ready and the demotion reverts. -``` -story_state -├─ entities (characters, locations, important objects) -├─ scenes -├─ timeline -├─ relationships -├─ unresolved_threads -├─ facts -└─ hypotheses -``` +## 6. Narrative understanding is carried state, not a per-panel description -A panel produces a **delta**, not another standalone prose interpretation: +Today each stage reads its predecessor's blob for one panel or one beat. `recent`, a rolling list of the +last few dialogue lines, is the only carried state. That is the root of the invented narration. -``` -panel 142: -- character_07 enters room_03 -- character_02 is already present -- character_07 says "..." -- object_12 changes owner: 02 -> 07 -- possible flashback begins -``` - -### Facts, hypotheses and unknowns are different records - -``` -fact: source=panel_142 confidence=0.99 character_07 is visible -hypothesis: confidence=0.64 character_07 is angry -unknown: who caused the explosion -``` - -A later panel strengthens, replaces or invalidates a hypothesis without rewriting history. - -### Scene state is explicit and inherited +The version worth building is one record per scene, inherited forward: ``` scene_31: location: school_rooftop time: evening participants: {char_03: present, char_07: present, char_11: offscreen} - pov: null narrative_mode: present - parent_scene: null + last_speaker: char_07 + addressee: char_02 ``` A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character absent for one panel has not left. A panel with no background has not changed location. A tail-less line -keeps the offscreen participant as a candidate. A close-up still belongs to the scene. +keeps the offscreen participant as a candidate. A close-up still belongs to the scene. `last_speaker` and +`participants` are what section 3's continuity term reads. -### Classify the transition, not just the panel +A panel produces a **delta** against that record, not another standalone prose interpretation: + +``` +panel 142: +- character_07 enters room_03 +- character_02 is already present +- character_07 says "..." +``` + +Classify the transition, not just the panel: ``` CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE ``` -`EMBEDDED_SCENE` is what stops a television's contents mutating the room around it: +`EMBEDDED_SCENE` is `plane != story` at scene granularity, and is what stops a television's contents +mutating the room around it. + +**Scene state is written by a resolver, never by the vision model.** The path is +`observation -> resolver -> state transition`. The resolver may reject an impossible update. It is plain +Python over gemma's per-panel delta, and it is where the constraint lives. + +**Today:** none of it exists. Chapter boundaries are a reset. Narration asserts things no panel shows +(`NEXT.md` item 6). + +**Done when:** a scene record carries location, participants and `narrative_mode` across panels. A +character absent from one panel stays a participant. On the next full run the four invented-fact +timestamps do not recur. Those are 0:43, 2:03, 2:05 and 2:15, and they are the regression list. The +correctness verifier passed 116/116 over them because it checks quotes and names, never invented claims. +So the check is a re-watch of those four points, not a stage counter. + +### Not building yet + +Each of these was in the original design. Each is deferred with a trigger, not dropped. + +| deferred | trigger to revisit | +| --- | --- | +| Facts, hypotheses and unknowns as separate records with confidences | when scene state exists and narration still asserts unshown claims | +| The seven-check consistency checker | when a scene record exists for it to check against | +| Unresolved references that survive and back-propagate | when a second chapter of the same manga runs | +| Chapter checkpoints and the two-memory split | when a second chapter of the same manga runs | + +One reason covers all four. They sit on an identity layer still wrong on 2 of the lead's 16 crops. State +machinery over wrong identity produces confidently wrong state. + +## 7. Build order + +Detection, vision and character embeddings already exist. The order below is chosen so each step is +falsifiable by the step's own DoD before the next one starts. ``` -scene_12 present - ├─ panel 101 - ├─ panel 102 - └─ embedded scene_13 [television] - ├─ panel-like region - └─ char_19 +0. eval/chapter-truth.json + purity and fragmentation in audit_registry.py +1. plane + species per detection -> section 1 DoD +2. face-plus-person embedding -> section 4 embedding check +3. chapter-wide clustering, weak cannot-link on plane -> section 4 purity DoD +4. scene record carried forward -> section 6 DoD +5. scored speaker edge, offscreen arm -> section 3 DoD +6. tail regions from the unused det/seg heads -> only if 5 misses its DoD ``` -### Character state is written by a resolver, never by the vision model +Steps 1 and 2 are independent and can land together. Step 3 depends on 2, which is the ordering trap in +section 4. Step 5 depends on 4, because the continuity term reads the scene record. Step 6 is +conditional on purpose: build a tail detector only after the cheap terms have been measured and found +insufficient. -``` -char_07: - known_names: [...] - currently_at: room_03 - status: alive - appearance_state: {clothes: school_uniform, injured: true} - relationships: {char_02: friend?} - last_seen: panel_142 -``` - -The path is `observation -> resolver -> state transition`, and the resolver may reject an impossible -update. - -### Conversation state is its own record - -``` -conversation_18: - scene: scene_31 - participants: [char_02, char_07] - last_speaker: char_07 - addressee: char_02 - topic: missing_key -``` - -This is the strongest available prior for a tail-less bubble. Given `A: where did you put it? / ... / -A: don't lie.`, turn-taking assigns the middle line with no visual evidence at all. - -### An unresolved reference survives instead of being forced - -``` -unknown_04: - type: person - descriptions: ["the man from yesterday", "silhouette in panel_58"] - candidate_ids: {char_12: 0.55, char_19: 0.22} -``` - -Chapter 6 may reveal `unknown_04 == char_12`, and that identity back-propagates through the graph. The -same applies to unnamed characters, pronouns, disguised characters, mysterious objects and unseen -speakers. - -### Two memories - -- **Working narrative state**: the current scene and the recent ones, in detail. -- **Canonical long-term memory**: compressed facts, not chapter summaries. `char_07 learned that char_02 - betrayed the group.` `object_04 is held by char_11.` `char_03 does not know char_07 survived.` - -### A chapter boundary is a checkpoint, not a reset - -``` -chapter_checkpoint: - persistent_entity_changes / relationship_changes / location and status changes - newly established facts / unresolved questions / active plot threads / final scene state -``` - -Chapter `n+1` starts from that. The detailed panel graph may be kept forever. Only five things load into -the model: the current scene, the previous scene, the relevant character records, the active threads, and -retrieved old facts. - -### A consistency checker runs after each scene and chapter - -Seven checks. A dead character appearing normally. A character knowing a fact before learning it. An -object owned by two people at once. A flashback never closed. A location jump with no transition. A -speaker who was neither present nor offscreen. A name that conflicts with the identity graph. The model -proposes corrections. The graph stays the source of truth. - -**Today:** none of this exists. Each stage reads its predecessor's blob for one panel or one beat. -`recent` is a rolling list of the last few dialogue lines and is the only carried state. Chapter -boundaries are a reset. There is no fact-versus-hypothesis distinction anywhere, which is why narration -asserts things no panel shows (`NEXT.md` item 6). - -## 7. The staged version worth building - -Do not recreate Magi's monolithic network first. Detection, vision and character embeddings already exist, -so stage it: - -``` -page - ↓ -region detector panels / nested regions / texts / characters / tails - ↓ -object feature extraction - ↓ -three pair models character↔character (identity) - text→character (speaker) - text→tail (bubble structure) - ↓ -chapter graph - ↓ -global character clustering - ↓ -optional naming - ↓ -ocr + reading order - ↓ -dialogue stream -``` - -The VLM then judges only the ambiguous graph edges. It no longer rediscovers every character and dialogue -relationship from raw pixels on every panel. Magi formulates detection and association as graph -generation, which is why it beats a crop, OCR and nearest-character pipeline here. - -## What to take from this before the rewrite - -Three items are cheap against the current code and pay immediately. They are entered in `NEXT.md`, not -here. - -1. **`same_narrative_plane`, as a per-detection field.** Vision already returns per-panel boxes. Add a - `plane` or `depth` to a detection, set when the model says the figure sits inside a screen, poster, - photo or drawing. That buys the containment edge with no detector. It is the whole of the remaining - identity error on the lead, and it feeds every stage below. -2. **Same-panel co-presence becomes a weak cannot-link.** `tracklets.cannot_link` currently makes it hard. - It needs item 1 first, because the plane is what makes the weak version safe. -3. **`offscreen` as a fourth `speaker_ref` kind.** The union already exists, the arm does not. +The VLM keeps judging ambiguous edges. What changes is that it stops rediscovering every character and +dialogue relationship from raw pixels on every panel. The chapter graph carries the answer forward. ## Sources -Magi and Magiv2 for the detection-and-association-as-graph-generation formulation, the text-character -pair head, and the character bank of exemplar images plus names. Magiv3 for panels, texts, characters and -tails with their associations, and for character grounding between textual descriptions and detected -character regions. +Three ideas come from published comics-transcription work. Detection and association as graph generation, +the text-to-character pair head, and the character bank of exemplar images plus names. The +formulation is kept. The models are ruled out, see **Ruled out** above. No source in this document is a +runtime dependency. diff --git a/NEXT.md b/NEXT.md index b61b241..64ba814 100644 --- a/NEXT.md +++ b/NEXT.md @@ -71,9 +71,15 @@ contain the right person. Coverage is still the `has_face` gate plus those refus face touching the bottom edge of one fragment with a textless fragment below. No evidence has been gathered yet on how often this chapter does it. -1c. **Three items from `ARCHITECTURE.md` are cheap against the current code.** A `plane` field per - detection for art-in-art, same-panel co-presence demoted to a weak cannot-link once the plane exists, - and `offscreen` as a fourth `speaker_ref` kind. +1c. **`ARCHITECTURE.md` was rewritten on 2026-08-13 under the no-Magi constraint.** Every section now + carries a DoD. Its build order starts at step 0, `eval/chapter-truth.json` plus purity and + fragmentation printed by `audit_registry.py`. Nothing below it is measurable without that file. + + Steps 1 and 2 are independent and cheap: `plane` and `species` per detection, and the + face-plus-person embedding already queued in item 1 above. The weak cannot-link is step 3 and depends + on step 2, not on `plane` alone. Co-presence as a hard constraint is load-bearing while cosine cannot + separate people, so demoting it early regresses purity. `offscreen` as a fourth `speaker_ref` kind is + still cheap and independent. 2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0 degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`).