Wire the caption merge, and write the target architecture down
merge_faceless_captions had been written and never called; both crop endpoints called the non-destructive context_fragment_links instead, with no decision recording that choice. Wiring it changes panel count and every panel index, so the chapter needs a re-crop with the panels prefix cleared first -- crop_webtoon skips an upload when the key already exists, which is right for a resume and silently wrong after a slicing change. Noted at the line. It does not cover the head-in-one-shot body-in-the-next split that prompted the question. _merge_plan only folds a fragment that has text and no face. ARCHITECTURE.md is the target shape from the user's design, with what exists against each section today. Nothing in it is built. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+368
@@ -0,0 +1,368 @@
|
|||||||
|
# ARCHITECTURE
|
||||||
|
|
||||||
|
The target shape of the pipeline, written 2026-08-13 from the user's design. This is **not** what the code
|
||||||
|
does. `NEXT.md` holds the live state and `AUDIT.md` holds the current pipeline. Every section here ends
|
||||||
|
with what exists today, so the gap is legible without reading both.
|
||||||
|
|
||||||
|
The governing principle:
|
||||||
|
|
||||||
|
> Do not make the next panel understand the previous panel. Make it understand the current world state
|
||||||
|
> produced by all previous panels.
|
||||||
|
|
||||||
|
Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below
|
||||||
|
follows from that split.
|
||||||
|
|
||||||
|
## 1. The page is a region graph, not a list of panels
|
||||||
|
|
||||||
|
```
|
||||||
|
page
|
||||||
|
├─ regions
|
||||||
|
│ ├─ panel
|
||||||
|
│ ├─ inset_panel
|
||||||
|
│ ├─ embedded_art
|
||||||
|
│ ├─ text
|
||||||
|
│ ├─ tail
|
||||||
|
│ └─ character_occurrence
|
||||||
|
│
|
||||||
|
└─ edges
|
||||||
|
├─ contains(region, region)
|
||||||
|
├─ reads_before(text, text)
|
||||||
|
├─ tail_of(tail, text)
|
||||||
|
├─ points_to(tail, character)
|
||||||
|
├─ spoken_by(text, character)
|
||||||
|
└─ same_identity(character, character)
|
||||||
|
```
|
||||||
|
|
||||||
|
A flat set of panels cannot express a television inside a room. That is the defect the current pipeline
|
||||||
|
shows most often.
|
||||||
|
|
||||||
|
**Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a
|
||||||
|
non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no
|
||||||
|
containment edge, no tail region and no region type.
|
||||||
|
|
||||||
|
## 2. Identity exists independently of names
|
||||||
|
|
||||||
|
```
|
||||||
|
occurrence c42
|
||||||
|
-> identity char_07
|
||||||
|
name = null
|
||||||
|
aliases = []
|
||||||
|
```
|
||||||
|
|
||||||
|
`char_07.name` may be filled later, or stay null forever and display as `unknown character #7`. The
|
||||||
|
occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster.
|
||||||
|
Three levels, never collapsed into one.
|
||||||
|
|
||||||
|
**Today:** the schema already has this split. `identity_assignments` is the occurrence,
|
||||||
|
`characters` owns the identity, `name` is nullable and downstream already falls back to an anonymous
|
||||||
|
display. What is missing is the clustering, not the separation. See section 4.
|
||||||
|
|
||||||
|
## 3. Speaker attribution is a scored graph edge, not a procedure
|
||||||
|
|
||||||
|
Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge:
|
||||||
|
|
||||||
|
```
|
||||||
|
score(text, character) =
|
||||||
|
learned_t2c_score
|
||||||
|
+ tail_evidence
|
||||||
|
+ spatial_evidence
|
||||||
|
+ same_panel
|
||||||
|
+ dialogue_continuity
|
||||||
|
+ character_activity_prior
|
||||||
|
+ identity_context
|
||||||
|
```
|
||||||
|
|
||||||
|
`learned_t2c_score` is the load-bearing term: a pair classifier over the whole page, the text object's
|
||||||
|
visual feature and the character object's visual feature. Magi's text-character head does exactly this.
|
||||||
|
It can start as a tiny MLP:
|
||||||
|
|
||||||
|
```
|
||||||
|
t2c(text_embedding, character_embedding, page_context, geometry_features) -> p(speaker)
|
||||||
|
```
|
||||||
|
|
||||||
|
with geometry carrying normalized relative position, distance, overlap, same-panel, containment depth and
|
||||||
|
tail direction.
|
||||||
|
|
||||||
|
Then the cases fall out of one mechanism instead of four:
|
||||||
|
|
||||||
|
| case | what carries it |
|
||||||
|
| --- | --- |
|
||||||
|
| bubble with a tail | `t2c` + tail, usually decisive |
|
||||||
|
| bubble with no tail | `t2c` + spatial and context |
|
||||||
|
| speaker outside the panel | recent identities + an offscreen candidate |
|
||||||
|
| narration | the narrator candidate |
|
||||||
|
| nothing resolves | unknown speaker |
|
||||||
|
|
||||||
|
**A dialogue line must not be required to resolve to a visible character.** That is a failure mode, not a
|
||||||
|
safeguard. The speaker type is a union:
|
||||||
|
|
||||||
|
```
|
||||||
|
speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown
|
||||||
|
```
|
||||||
|
|
||||||
|
**Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator`
|
||||||
|
(`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a
|
||||||
|
prompt to gemma over a window of panels, with no geometry term at all. The `det`/`seg` tail heads exist
|
||||||
|
and are unused (`caveats/speaker-attribution.md#tail-is-not-geometry`).
|
||||||
|
|
||||||
|
## 4. Character recognition is occurrence, then identity, then name
|
||||||
|
|
||||||
|
```
|
||||||
|
character detection
|
||||||
|
↓
|
||||||
|
occurrence embeddings
|
||||||
|
↓
|
||||||
|
pairwise same_identity probabilities
|
||||||
|
↓
|
||||||
|
chapter-wide constrained clustering
|
||||||
|
↓
|
||||||
|
char_001, char_002, ...
|
||||||
|
↓
|
||||||
|
optional character-bank lookup
|
||||||
|
↓
|
||||||
|
name or unknown
|
||||||
|
```
|
||||||
|
|
||||||
|
Two rules that the current code gets wrong.
|
||||||
|
|
||||||
|
**The embedding is not the character crop alone.** Combine four signals: the character crop, the face or
|
||||||
|
head crop, the full-body crop, and a contextual object feature. Magiv2 combines detected object features
|
||||||
|
with a separate crop-embedding model.
|
||||||
|
|
||||||
|
**Cluster chapter-wide, not page by page.**
|
||||||
|
|
||||||
|
**Two characters in the same panel may be one person.** Mirrors, photographs, flashbacks, insets,
|
||||||
|
screens, imagined scenes and repeated action drawings all break that rule. Make it a weak cannot-link,
|
||||||
|
and only when the two are on the same narrative plane.
|
||||||
|
|
||||||
|
**Today:** the embedding is the person box only, which is measurably the wrong signal
|
||||||
|
(`caveats/audit-open.md#cosine-not-identity`). Clustering is greedy and local: `tracklets.link_tracklets`
|
||||||
|
groups within an 8-panel window. `tracklets.cannot_link` treats same-panel co-presence as a **hard**
|
||||||
|
constraint, which is exactly the correction above. Naming is `db.add_name_claim`, corroboration over
|
||||||
|
`name_claims`.
|
||||||
|
|
||||||
|
## 5. The art-in-art problem needs a narrative plane
|
||||||
|
|
||||||
|
Treat the page as a hierarchical scene graph:
|
||||||
|
|
||||||
|
```
|
||||||
|
page
|
||||||
|
└── panel A depth=0
|
||||||
|
├── character c1
|
||||||
|
├── text t1
|
||||||
|
└── television/poster depth=1, type=embedded_art
|
||||||
|
├── character c2
|
||||||
|
└── text t2
|
||||||
|
```
|
||||||
|
|
||||||
|
Speaker candidates normally come from the same `scene_depth`. Otherwise a real character standing beside a
|
||||||
|
poster of a drawn person can be given the poster person's line.
|
||||||
|
|
||||||
|
A region classifier predicts a type:
|
||||||
|
|
||||||
|
```
|
||||||
|
story_scene | inset_story_panel | flashback | screen | photo | poster | illustration | decorative
|
||||||
|
```
|
||||||
|
|
||||||
|
Perfect classification is not the point. The output that matters is one probability:
|
||||||
|
|
||||||
|
```
|
||||||
|
same_narrative_plane(a, b)
|
||||||
|
```
|
||||||
|
|
||||||
|
which then enters the association score in section 3 and the cannot-link in section 4.
|
||||||
|
|
||||||
|
**Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the
|
||||||
|
19:44 run of 2026-08-12 his 16 assignments were 14 correct plus a photograph of another man and a chibi
|
||||||
|
drawing. Both are art inside a panel. Vision also boxes cats as people and dresses them (`p081`, `p108`).
|
||||||
|
|
||||||
|
## 6. Narrative understanding is a state machine, not a per-panel description
|
||||||
|
|
||||||
|
```
|
||||||
|
story_state
|
||||||
|
├─ entities (characters, locations, important objects)
|
||||||
|
├─ scenes
|
||||||
|
├─ timeline
|
||||||
|
├─ relationships
|
||||||
|
├─ unresolved_threads
|
||||||
|
├─ facts
|
||||||
|
└─ hypotheses
|
||||||
|
```
|
||||||
|
|
||||||
|
A panel produces a **delta**, not another standalone prose interpretation:
|
||||||
|
|
||||||
|
```
|
||||||
|
panel 142:
|
||||||
|
- character_07 enters room_03
|
||||||
|
- character_02 is already present
|
||||||
|
- character_07 says "..."
|
||||||
|
- object_12 changes owner: 02 -> 07
|
||||||
|
- possible flashback begins
|
||||||
|
```
|
||||||
|
|
||||||
|
### Facts, hypotheses and unknowns are different records
|
||||||
|
|
||||||
|
```
|
||||||
|
fact: source=panel_142 confidence=0.99 character_07 is visible
|
||||||
|
hypothesis: confidence=0.64 character_07 is angry
|
||||||
|
unknown: who caused the explosion
|
||||||
|
```
|
||||||
|
|
||||||
|
A later panel strengthens, replaces or invalidates a hypothesis without rewriting history.
|
||||||
|
|
||||||
|
### Scene state is explicit and inherited
|
||||||
|
|
||||||
|
```
|
||||||
|
scene_31:
|
||||||
|
location: school_rooftop
|
||||||
|
time: evening
|
||||||
|
participants: {char_03: present, char_07: present, char_11: offscreen}
|
||||||
|
pov: null
|
||||||
|
narrative_mode: present
|
||||||
|
parent_scene: null
|
||||||
|
```
|
||||||
|
|
||||||
|
A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character
|
||||||
|
absent for one panel has not left. A panel with no background has not changed location. A tail-less line
|
||||||
|
keeps the offscreen participant as a candidate. A close-up still belongs to the scene.
|
||||||
|
|
||||||
|
### Classify the transition, not just the panel
|
||||||
|
|
||||||
|
```
|
||||||
|
CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START
|
||||||
|
FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE
|
||||||
|
```
|
||||||
|
|
||||||
|
`EMBEDDED_SCENE` is what stops a television's contents mutating the room around it:
|
||||||
|
|
||||||
|
```
|
||||||
|
scene_12 present
|
||||||
|
├─ panel 101
|
||||||
|
├─ panel 102
|
||||||
|
└─ embedded scene_13 [television]
|
||||||
|
├─ panel-like region
|
||||||
|
└─ char_19
|
||||||
|
```
|
||||||
|
|
||||||
|
### Character state is written by a resolver, never by the vision model
|
||||||
|
|
||||||
|
```
|
||||||
|
char_07:
|
||||||
|
known_names: [...]
|
||||||
|
currently_at: room_03
|
||||||
|
status: alive
|
||||||
|
appearance_state: {clothes: school_uniform, injured: true}
|
||||||
|
relationships: {char_02: friend?}
|
||||||
|
last_seen: panel_142
|
||||||
|
```
|
||||||
|
|
||||||
|
The path is `observation -> resolver -> state transition`, and the resolver may reject an impossible
|
||||||
|
update.
|
||||||
|
|
||||||
|
### Conversation state is its own record
|
||||||
|
|
||||||
|
```
|
||||||
|
conversation_18:
|
||||||
|
scene: scene_31
|
||||||
|
participants: [char_02, char_07]
|
||||||
|
last_speaker: char_07
|
||||||
|
addressee: char_02
|
||||||
|
topic: missing_key
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the strongest available prior for a tail-less bubble. Given `A: where did you put it? / ... /
|
||||||
|
A: don't lie.`, turn-taking assigns the middle line with no visual evidence at all.
|
||||||
|
|
||||||
|
### An unresolved reference survives instead of being forced
|
||||||
|
|
||||||
|
```
|
||||||
|
unknown_04:
|
||||||
|
type: person
|
||||||
|
descriptions: ["the man from yesterday", "silhouette in panel_58"]
|
||||||
|
candidate_ids: {char_12: 0.55, char_19: 0.22}
|
||||||
|
```
|
||||||
|
|
||||||
|
Chapter 6 may reveal `unknown_04 == char_12`, and that identity back-propagates through the graph. The
|
||||||
|
same applies to unnamed characters, pronouns, disguised characters, mysterious objects and unseen
|
||||||
|
speakers.
|
||||||
|
|
||||||
|
### Two memories
|
||||||
|
|
||||||
|
- **Working narrative state**: the current scene and the recent ones, in detail.
|
||||||
|
- **Canonical long-term memory**: compressed facts, not chapter summaries. `char_07 learned that char_02
|
||||||
|
betrayed the group.` `object_04 is held by char_11.` `char_03 does not know char_07 survived.`
|
||||||
|
|
||||||
|
### A chapter boundary is a checkpoint, not a reset
|
||||||
|
|
||||||
|
```
|
||||||
|
chapter_checkpoint:
|
||||||
|
persistent_entity_changes / relationship_changes / location and status changes
|
||||||
|
newly established facts / unresolved questions / active plot threads / final scene state
|
||||||
|
```
|
||||||
|
|
||||||
|
Chapter `n+1` starts from that. The detailed panel graph may be kept forever. Only five things load into
|
||||||
|
the model: the current scene, the previous scene, the relevant character records, the active threads, and
|
||||||
|
retrieved old facts.
|
||||||
|
|
||||||
|
### A consistency checker runs after each scene and chapter
|
||||||
|
|
||||||
|
Seven checks. A dead character appearing normally. A character knowing a fact before learning it. An
|
||||||
|
object owned by two people at once. A flashback never closed. A location jump with no transition. A
|
||||||
|
speaker who was neither present nor offscreen. A name that conflicts with the identity graph. The model
|
||||||
|
proposes corrections. The graph stays the source of truth.
|
||||||
|
|
||||||
|
**Today:** none of this exists. Each stage reads its predecessor's blob for one panel or one beat.
|
||||||
|
`recent` is a rolling list of the last few dialogue lines and is the only carried state. Chapter
|
||||||
|
boundaries are a reset. There is no fact-versus-hypothesis distinction anywhere, which is why narration
|
||||||
|
asserts things no panel shows (`NEXT.md` item 6).
|
||||||
|
|
||||||
|
## 7. The staged version worth building
|
||||||
|
|
||||||
|
Do not recreate Magi's monolithic network first. Detection, vision and character embeddings already exist,
|
||||||
|
so stage it:
|
||||||
|
|
||||||
|
```
|
||||||
|
page
|
||||||
|
↓
|
||||||
|
region detector panels / nested regions / texts / characters / tails
|
||||||
|
↓
|
||||||
|
object feature extraction
|
||||||
|
↓
|
||||||
|
three pair models character↔character (identity)
|
||||||
|
text→character (speaker)
|
||||||
|
text→tail (bubble structure)
|
||||||
|
↓
|
||||||
|
chapter graph
|
||||||
|
↓
|
||||||
|
global character clustering
|
||||||
|
↓
|
||||||
|
optional naming
|
||||||
|
↓
|
||||||
|
ocr + reading order
|
||||||
|
↓
|
||||||
|
dialogue stream
|
||||||
|
```
|
||||||
|
|
||||||
|
The VLM then judges only the ambiguous graph edges. It no longer rediscovers every character and dialogue
|
||||||
|
relationship from raw pixels on every panel. Magi formulates detection and association as graph
|
||||||
|
generation, which is why it beats a crop, OCR and nearest-character pipeline here.
|
||||||
|
|
||||||
|
## What to take from this before the rewrite
|
||||||
|
|
||||||
|
Three items are cheap against the current code and pay immediately. They are entered in `NEXT.md`, not
|
||||||
|
here.
|
||||||
|
|
||||||
|
1. **`same_narrative_plane`, as a per-detection field.** Vision already returns per-panel boxes. Add a
|
||||||
|
`plane` or `depth` to a detection, set when the model says the figure sits inside a screen, poster,
|
||||||
|
photo or drawing. That buys the containment edge with no detector. It is the whole of the remaining
|
||||||
|
identity error on the lead, and it feeds every stage below.
|
||||||
|
2. **Same-panel co-presence becomes a weak cannot-link.** `tracklets.cannot_link` currently makes it hard.
|
||||||
|
It needs item 1 first, because the plane is what makes the weak version safe.
|
||||||
|
3. **`offscreen` as a fourth `speaker_ref` kind.** The union already exists, the arm does not.
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
Magi and Magiv2 for the detection-and-association-as-graph-generation formulation, the text-character
|
||||||
|
pair head, and the character bank of exemplar images plus names. Magiv3 for panels, texts, characters and
|
||||||
|
tails with their associations, and for character grounding between textual descriptions and detected
|
||||||
|
character regions.
|
||||||
@@ -11,6 +11,7 @@ Goal, invariants, and working rules. Read this first.
|
|||||||
| `caveats/` | every known limit and its revisit trigger, indexed in `caveats/CLAUDE.md` |
|
| `caveats/` | every known limit and its revisit trigger, indexed in `caveats/CLAUDE.md` |
|
||||||
| `AGENTS.md` | commands, with the traps beside them |
|
| `AGENTS.md` | commands, with the traps beside them |
|
||||||
| `AUDIT.md` | the 2026-08-11 pipeline audit, the source of the roadmap |
|
| `AUDIT.md` | the 2026-08-11 pipeline audit, the source of the roadmap |
|
||||||
|
| `ARCHITECTURE.md` | the target shape of the pipeline, and what exists against it today |
|
||||||
| `spec-v3.md` | current quality and look work, marked DONE/TODO per item |
|
| `spec-v3.md` | current quality and look work, marked DONE/TODO per item |
|
||||||
|
|
||||||
Do not restate a finding here. Point at the decision.
|
Do not restate a finding here. Point at the decision.
|
||||||
|
|||||||
+33
@@ -723,3 +723,36 @@ Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestr
|
|||||||
`/characters/reset` asked for it.
|
`/characters/reset` asked for it.
|
||||||
|
|
||||||
Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed.
|
Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed.
|
||||||
|
|
||||||
|
## 2026-08-13 — the dialogue stage names nobody, and why
|
||||||
|
|
||||||
|
Ran `dialogue` 116/116 in 5m57s on the fourth cycle's registry, to see whether the fixed identity lets the
|
||||||
|
existing `name_claims` path name the female lead. It does not, and the six claims it produced name three
|
||||||
|
separate defects.
|
||||||
|
|
||||||
|
```
|
||||||
|
p040 character_2b1b12a1 "Choi Haeseon" caption 1.00 -> NOT promoted
|
||||||
|
p010 character_b1dd5659 "Lim Seonho" caption 1.00 -> conflict flag
|
||||||
|
p047 character_b1dd5659 "Seonho" address 0.90 -> conflict flag
|
||||||
|
p011 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
|
||||||
|
p026 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
|
||||||
|
p110 character_028d4a49 "Haeseon" address 1.00 -> already named
|
||||||
|
```
|
||||||
|
|
||||||
|
All three are fixed in `db.add_name_claim` and recorded in `decisions/identity-naming.md`: alias grouping,
|
||||||
|
a confident caption as strong evidence, and one name per character. `test_name_binding.py` replays these
|
||||||
|
six claims, 121 tests pass, and each new assert was confirmed to fail with its fix disabled.
|
||||||
|
|
||||||
|
Also wired `merge_faceless_captions` into both crop endpoints. It had been written and never called; both
|
||||||
|
endpoints called the non-destructive `context_fragment_links` instead, and no decision recorded that
|
||||||
|
choice. It does not cover the head-in-one-shot, body-in-the-next split that prompted the question, because
|
||||||
|
a body fragment has no text and `_merge_plan` only folds a fragment that has text and no face.
|
||||||
|
|
||||||
|
Found while wiring it: `crop_webtoon` skips an upload when the key exists, which is right for a resume and
|
||||||
|
silently wrong after a slicing change. Documented at the line and in `NEXT.md`.
|
||||||
|
|
||||||
|
Wrote `ARCHITECTURE.md` from the user's design: region graph, occurrence/identity/name, speaker as a scored
|
||||||
|
graph edge with a typed union, narrative plane for art-in-art, and a persistent story state machine. Every
|
||||||
|
section carries what exists against it today. Nothing in it is built.
|
||||||
|
|
||||||
|
Nothing ran on a GPU after the dialogue stage.
|
||||||
|
|||||||
@@ -39,28 +39,42 @@ contain the right person. Coverage is still the `has_face` gate plus those refus
|
|||||||
|
|
||||||
## Next
|
## Next
|
||||||
|
|
||||||
1. **Deploy the two tracklet fixes and run the fourth cycle.** Neither has touched a GPU. Rebuild the
|
1. **Re-crop the chapter and run the fifth cycle.** Four changes are written and tested since the fourth
|
||||||
orchestrator image, reset the registry, run vision/identity/reconcile, then re-check the lead's crops.
|
cycle, and none has touched a GPU.
|
||||||
Expect roughly 30 tracklets over 64 crops instead of 12, so identity goes from about 1m25s to 3 or 4
|
|
||||||
minutes. Watch the lead's assignment count against 36.
|
|
||||||
|
|
||||||
Neither fix is sufficient. Bare hair colour still links different men, and the cat still joins its
|
- `merge_faceless_captions` is wired into both crop endpoints. It was written, never called, and
|
||||||
neighbours. Do not add a crop-to-crop cosine to close that: measured on this run's 22 embeddings,
|
`context_fragment_links` was called instead. A stranded caption fragment now vstacks into the
|
||||||
different people reach 0.93 while the same person reaches 0.96, so no threshold exists
|
face-bearing fragment it belongs to, so panel count and every panel index change.
|
||||||
(`caveats/audit-open.md#cosine-not-identity`).
|
- three naming fixes in `db.add_name_claim` (`decisions/identity-naming.md`): alias grouping, a caption
|
||||||
|
as strong evidence, and a name held by another character refusing to promote onto a second one.
|
||||||
|
|
||||||
Agreed next step after the cycle, chosen by the user and not started: **stop letting cosine pick the
|
**Clear `s3://panels/<manga>/<chapter>/panels/` before re-cropping.** `crop_webtoon` skips the upload
|
||||||
gallery.** There are 9 live characters. `run_stage_identity` builds `union_cands` from the members'
|
when the key exists, so a re-crop after a slicing change silently keeps the previous run's images.
|
||||||
cosine top-k shortlists, so a metric that cannot separate people decides who is even considered. Send
|
Wiring the merge is a slicing change. Everything downstream is invalidated by it, so this is a full
|
||||||
the live cast instead, gender-gated, capped and logged when truncated. Note two traps found while
|
re-run and not a stage rerun.
|
||||||
reading it: `/vision/resolve` sends up to 3 reference images per candidate
|
|
||||||
(`worker_vision.py:1057`), so 9 candidates is 27 images plus the query and needs a cap; and only a crop
|
Expected: fewer than 116 panels, `2b1b12a1` named `Choi Haeseon` from the p040 caption, the lead's
|
||||||
with a non-empty cosine shortlist enters `shortlists` at all, so an empty top-k currently drops the crop
|
`conflicting-name-claims` flag gone, and the green-dress woman no longer named `Seonho` but carrying a
|
||||||
from resolution entirely.
|
`name-already-taken` flag instead.
|
||||||
|
|
||||||
|
Not fixed by any of it. Bare hair colour still links different men. Do not add a crop-to-crop cosine to
|
||||||
|
close that. Measured on 22 embeddings, different people reach 0.93 and the same person reaches 0.96, so
|
||||||
|
no threshold exists (`caveats/audit-open.md#cosine-not-identity`).
|
||||||
|
|
||||||
Then, separately, test embedding the FACE box rather than the person box. `face_detect` already finds
|
Then, separately, test embedding the FACE box rather than the person box. `face_detect` already finds
|
||||||
the face and pairs it for `has_face`. That is the likely root cause of cosine measuring scene instead of
|
the face and pairs it for `has_face`. That is the likely root cause of cosine measuring scene instead of
|
||||||
person, and the test is to re-embed these same 22 detections and recompute the matrix.
|
person. The test is to re-embed these same 22 detections and recompute the matrix.
|
||||||
|
|
||||||
|
1b. **The head/body split that started the crop question is NOT fixed.** The wired merge only folds a
|
||||||
|
fragment that has text and no face. A body fragment carries no dialogue, so `_merge_plan` leaves it
|
||||||
|
solo and it becomes its own panel and its own shot. Finding it needs a different signal, most likely a
|
||||||
|
face touching the bottom edge of one fragment with a textless fragment below. No evidence has been
|
||||||
|
gathered yet on how often this chapter does it.
|
||||||
|
|
||||||
|
1c. **Three items from `ARCHITECTURE.md` are cheap against the current code.** A `plane` field per
|
||||||
|
detection for art-in-art, same-panel co-presence demoted to a weak cannot-link once the plane exists,
|
||||||
|
and `offscreen` as a fourth `speaker_ref` kind.
|
||||||
|
|
||||||
2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0
|
2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0
|
||||||
degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`).
|
degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`).
|
||||||
3. **Vision boxes animals as people and dresses them.** `p081` and `p108` are cats, described
|
3. **Vision boxes animals as people and dresses them.** `p081` and `p108` are cats, described
|
||||||
|
|||||||
+6
-2
@@ -48,5 +48,9 @@ still live belongs in `caveats/`.
|
|||||||
| [A roster name is a guess, so it never reaches detection](identity-bbox.md#roster-does-not-name) | closed |
|
| [A roster name is a guess, so it never reaches detection](identity-bbox.md#roster-does-not-name) | closed |
|
||||||
| [`merged_into` is exactly one hop deep](identity-bbox.md#merge-chains-flatten) | closed |
|
| [`merged_into` is exactly one hop deep](identity-bbox.md#merge-chains-flatten) | closed |
|
||||||
| [`_bbox_to_pixels` orders the corners, because the model sometimes swaps them](identity-bbox.md#bbox-corners-ordered) | closed |
|
| [`_bbox_to_pixels` orders the corners, because the model sometimes swaps them](identity-bbox.md#bbox-corners-ordered) | closed |
|
||||||
| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed, GPU pending |
|
| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed |
|
||||||
| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed, GPU pending |
|
| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed |
|
||||||
|
| [The gallery is the live cast, not cosine's top-k](identity-bbox.md#cast-is-the-gallery) | closed |
|
||||||
|
| [A name is a word set, not a string](identity-naming.md#alias-grouping) | closed, GPU pending |
|
||||||
|
| [A confident caption names a character on its own](identity-naming.md#caption-is-strong) | closed, GPU pending |
|
||||||
|
| [A name belongs to one character](identity-naming.md#one-name-one-character) | closed, GPU pending |
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Naming a character
|
||||||
|
|
||||||
|
How a discovered name reaches `characters.name`. The mechanism is `db.add_name_claim`, fed by the dialogue
|
||||||
|
stage through `service._absorb`.
|
||||||
|
|
||||||
|
## A name is a word set, not a string {#alias-grouping}
|
||||||
|
|
||||||
|
**Closed, 2026-08-13, not yet run on a GPU.**
|
||||||
|
|
||||||
|
Claims grouped on the casefolded name, so `Lim Seonho` from a p010 caption and `Seonho` from a p047 address
|
||||||
|
counted as two names for one character. `len(grouped) > 1` fired, a `conflicting-name-claims` flag was
|
||||||
|
filed, and promotion was blocked permanently on evidence that in fact corroborated.
|
||||||
|
|
||||||
|
`alias_groups` groups two names when one's word set contains the other's, and keeps the longer as
|
||||||
|
canonical. `Seonho` and `Lim Seonho` are one name and the registry stores `Lim Seonho`. `Seonho` and
|
||||||
|
`Haeseon` are still two, so a real conflict still flags.
|
||||||
|
|
||||||
|
## A confident caption names a character on its own {#caption-is-strong}
|
||||||
|
|
||||||
|
**Closed, 2026-08-13, not yet run on a GPU.**
|
||||||
|
|
||||||
|
Promotion needed two distinct panels, or one `self_intro` or `name_tag` claim above 0.9. On the 2026-08-12
|
||||||
|
chapter the only claim naming a main character was `Choi Haeseon`, a caption at p040 at confidence 1.0, and
|
||||||
|
it was discarded. She held 13 correct crops and stayed anonymous through the whole pipeline.
|
||||||
|
|
||||||
|
A caption is the narration naming the person it is drawn beside, which is how a webtoon introduces its
|
||||||
|
cast, so it joins `STRONG_EVIDENCE`. This is the loosest of the three changes and is only safe because of
|
||||||
|
the next one.
|
||||||
|
|
||||||
|
## A name belongs to one character {#one-name-one-character}
|
||||||
|
|
||||||
|
**Closed, 2026-08-13, not yet run on a GPU.**
|
||||||
|
|
||||||
|
Being addressed by name identifies the addressee. Choosing which drawn body that is fails often. Two panels
|
||||||
|
addressed `Seonho`, the dialogue model pointed `target_local_id` at the woman standing beside him, and two
|
||||||
|
independent claims promoted her. The registry then held a female `Seonho` over 9 crops beside the lead's
|
||||||
|
`LIM SEONHO`.
|
||||||
|
|
||||||
|
A promotion now checks every other live character of the same manga first, by alias group. A collision
|
||||||
|
refuses the promotion and files a `name-already-taken` flag carrying both ids. The collision is itself
|
||||||
|
evidence that either the addressee or the identity cluster is wrong, so it is worth surfacing rather than
|
||||||
|
resolving silently.
|
||||||
|
|
||||||
|
Deliberately not built: no attempt to decide WHICH character deserves the name. That needs the addressee
|
||||||
|
fixed, which is `ARCHITECTURE.md` section 3.
|
||||||
|
|
||||||
|
## Checks
|
||||||
|
|
||||||
|
`test_name_binding.py` replays the six real claims from the 2026-08-12 chapter. Each new assert was
|
||||||
|
confirmed to fail with its fix disabled: the caption test with `STRONG_EVIDENCE` reverted, the alias test
|
||||||
|
with casefold grouping restored. The taken-name test asserts a flag kind that only the new branch emits.
|
||||||
+5
-2
@@ -240,12 +240,15 @@ async def crop_webtoon(data: WebtoonInput):
|
|||||||
tag = uuid.uuid4().hex[:8]
|
tag = uuid.uuid4().hex[:8]
|
||||||
locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)]
|
locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)]
|
||||||
strip = restitch(locals_)
|
strip = restitch(locals_)
|
||||||
crops = slice_webtoon(strip)
|
crops = merge_faceless_captions(slice_webtoon(strip))
|
||||||
context_links = context_fragment_links(crops)
|
context_links = context_fragment_links(crops)
|
||||||
panels = []
|
panels = []
|
||||||
for idx, (crop_img, bbox) in enumerate(crops):
|
for idx, (crop_img, bbox) in enumerate(crops):
|
||||||
uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png"
|
uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png"
|
||||||
# slicing is deterministic, so on a resume the same idx -> same key; skip re-upload.
|
# slicing is deterministic, so on a resume the same idx -> same key; skip re-upload.
|
||||||
|
# TRAP: that holds only while the PLAN is unchanged. Edit slice_webtoon or the merge pass and a
|
||||||
|
# re-crop silently keeps the previous run's images under the same keys, because every one of them
|
||||||
|
# already exists. Clear the s3://panels/<manga>/<chapter>/panels/ prefix before re-cropping.
|
||||||
if not transport.exists(uri):
|
if not transport.exists(uri):
|
||||||
out = f"{SHM}/wt_{tag}_p{idx:03d}.png"
|
out = f"{SHM}/wt_{tag}_p{idx:03d}.png"
|
||||||
cv2.imwrite(out, crop_img)
|
cv2.imwrite(out, crop_img)
|
||||||
@@ -266,7 +269,7 @@ async def crop(data: CropInput):
|
|||||||
raise HTTPException(400, f"page not readable: {data.page_uri}")
|
raise HTTPException(400, f"page not readable: {data.page_uri}")
|
||||||
h, w = img.shape[:2]
|
h, w = img.shape[:2]
|
||||||
webtoon = h / w >= WEBTOON_RATIO
|
webtoon = h / w >= WEBTOON_RATIO
|
||||||
crops = slice_webtoon(img) if webtoon else kumiko_panels(local, data.rtl)
|
crops = merge_faceless_captions(slice_webtoon(img)) if webtoon else kumiko_panels(local, data.rtl)
|
||||||
context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))}
|
context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))}
|
||||||
ambiguous = flag_overlaps(crops, data.page_index)
|
ambiguous = flag_overlaps(crops, data.page_index)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user