kami
a9d64fe80a
Somebody watched chapter.mp4 for the first time. Two failures came out of it that no stage counter could see. chapter.mp4 is video 436.39s over audio 363.67s, so narration finishes 72.7s before the picture. The 49 clips are clean: all 25fps, video and audio agree to 0.03s, summing to 363.6s. A per-round probe puts the loss in the final round of _assemble_batched, which turns 359s of video into 100s while the audio survives. Round 0 is correct. Round 1 differs by holding a 7th input, the leftover clip that skips encoding, so the tree mixes concat output, xfade output and a raw clip. Not fixed. worker_render.py gains an FPS constant, fps normalization in the xfade branch to match concat, _stream_dur, and a self-check that compares video against audio instead of asserting the file is non-empty. That old check is how a 20% sync failure shipped. The fps inconsistency is real but not proven to be the shipped cause. Pinning -r on the output was tried and reverted: it drops frames to force CFR, which the concat branch comment already warned about. Panel 7 checked against the art has zero correct identity bindings out of two, and Seonho, the one character who matters, is unbound. bbox values are consumed as absolute pixels; on a 900x1650 panel that puts all six boxes in the top third, two inside a speech balloon. Identity therefore embeds crops of balloon edges and window frames, which is how confidence 0.9 lands on the wrong person. The colleague has no name in the story and was labelled Choi Haeseon; that row holds 25 of 26 assignments, so it is the label the pipeline stamps on any unnamed woman. Four caveats added. Two earlier claims are withdrawn in place: rescaling bbox by 1000 does not make the boxes correct, and the constraint is not 16 nameless rows needing names. Cast profiles already exist, since all 53 rows populate ref_image_uris and embedding_uri, but they are enrolled from the wrong crops. worker_render.py self-check passes. No pipeline ran. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
151 lines
9.1 KiB
Markdown
151 lines
9.1 KiB
Markdown
# speaker-attribution
|
|
|
|
Limits found by cross-checking the 2026-08-11 chapter run against the panel images.
|
|
|
|
## Nothing attributes a speaker in a multi-character panel {#tail-is-not-geometry}
|
|
|
|
The false `tail` label is gone (`decisions/speaker-attribution.md#no-fake-tail`). What replaced it is a
|
|
refusal, not an answer: with two or more characters present, every speech line now returns `unknown`.
|
|
On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back
|
|
to a `generic-handle`. That is the honest floor, and it is not the fix.
|
|
|
|
The measurement that forced it, on job `778297bc`, chapter `7c944dd4`. Three two-character panels were
|
|
checked against the art. All three are wrong, each with the two speakers swapped:
|
|
|
|
| panel key | line | truth | pipeline |
|
|
| --- | --- | --- | --- |
|
|
| `p010.png` | "…definitely an Egen guy, Seonho!" | the woman | Seonho, the person addressed |
|
|
| `p010.png` | "Y-you think so?" | Seonho | Choi Haeseon |
|
|
| `p012.png` | "Want me to send you the link?" | the woman | the man |
|
|
| `p059.png` | "If team leader Choi says it, it must be true." | the man | Choi Haeseon |
|
|
|
|
The last row needs no image: the line refers to Choi in the third person and is attributed to Choi.
|
|
|
|
The grounded path exists and almost never fires. Only 2 of 81 speech lines got `som_face`, because
|
|
attribution marks need `face_detect` boxes that survive `_pair_faces_to_present`, and these webtoon
|
|
close-ups rarely produce them. Inference, not measured: the face detector was not instrumented.
|
|
|
|
`worker_vision.py:356` already carries the `ponytail:` note that multi-character attribution needs
|
|
per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads carry balloon fill and
|
|
tail tips and are unused.
|
|
|
|
**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric
|
|
(`ROADMAP.md`). The 30% read on this run counted attributions the sample says are wrong. The next run
|
|
will read lower and will be the first honest number. Raising it means binding a balloon to a speaker
|
|
by tail geometry, using the unused `det`/`seg` heads.
|
|
|
|
## One character id covers two different women {#identity-over-merge}
|
|
|
|
`character_afa7623b` is stored as "black bob, white sweater" and is assigned both to that person
|
|
(`p059.png`) and to the brown-bob green-top coworker (`p010.png`, `p012.png`). It took 25 of the 26
|
|
identity assignments in the chapter, against 113 detected people. Coverage is 23%.
|
|
|
|
**Revisit trigger:** any work on the identity Tier-2 decider. A single id absorbing a whole chapter is
|
|
the signature to watch for.
|
|
|
|
## The character registry carries five weeks of wrong names {#registry-pollution}
|
|
|
|
The registry holds 53 characters for manga `ef105a86`, 41 of them unnamed, with "Kei" three times and
|
|
"Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only
|
|
Haeseon and Seonho do. `/stage/clear` leaves the per-manga registry intact by design, so every rerun
|
|
inherits the whole pile.
|
|
|
|
Duplicate rows also make a correct name unresolvable. This manga holds `Choi Haeseon`, `Seonho` with
|
|
aliases `["Lim Seonho", "Seonho"]`, and a separate `Lim Seonho`. An answer of "Lim Seonho" matches two
|
|
rows, so `normalize_speaker` returns candidates and raises `ambiguous-speaker` instead of binding. The
|
|
pipeline read the name correctly and still cannot name the speaker.
|
|
|
|
**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the
|
|
registry to a chapter or add a reviewed reset. Merging the duplicate rows needs the reversible-merge
|
|
design first (`caveats/audit-open.md#destructive-reconcile`).
|
|
|
|
## One invented word still halts the chapter {#one-word-halts-chapter}
|
|
|
|
The multi-word name failure is fixed (`decisions/speaker-attribution.md#multiword-cast-names`). The
|
|
blast radius it exposed is not. `run_stage_script` retries a rejected beat once, then raises, so a
|
|
single unsupported token ends the run at that beat. On job `778297bc` one of the 29 lost beats cited
|
|
`['Blur']`, an onomatopoeia the model invented. The verifier was right, and the whole chapter still
|
|
stopped.
|
|
|
|
**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer
|
|
is to flag the beat for review and continue, which is `#136` gate work, not a verifier change.
|
|
|
|
## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space}
|
|
|
|
Vision's `bbox` values are stored and consumed as absolute pixels. On panel
|
|
`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of
|
|
the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own
|
|
dimensions, four of the six fit their subjects tightly.
|
|
|
|
Two places in the code assert pixels, and the art contradicts both:
|
|
|
|
- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
|
|
- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
|
|
|
|
Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces
|
|
against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9
|
|
from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed
|
|
alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at
|
|
`worker_vision.py:57` reads the same numbers and was not checked.
|
|
|
|
Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with
|
|
nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already
|
|
calls the box "coarse, imprecise".
|
|
|
|
Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can
|
|
be judged while the crops are wrong.
|
|
|
|
## Identity cannot say "a person with no name" {#no-anonymous-identity}
|
|
|
|
The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9.
|
|
Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline
|
|
stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which
|
|
is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing
|
|
line.
|
|
|
|
This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an
|
|
unparseable model answer. The missing rule: never attach a name to a detection that carries no name
|
|
evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the
|
|
colleague" every time.
|
|
|
|
`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from
|
|
cosine clearing the threshold on a wrong crop. Whether the Tier-2 gemma resolver can answer "none of
|
|
these" was not verified.
|
|
|
|
Revisit trigger: immediately after the `bbox` space is settled.
|
|
|
|
## Vision does not separate a background extra from cast {#extras-as-cast}
|
|
|
|
Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground
|
|
and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame
|
|
with nobody in it. All six reach identity as equal candidates.
|
|
|
|
Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly
|
|
extras, so it measured nothing useful and should not be quoted again.
|
|
|
|
Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return.
|
|
|
|
## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set}
|
|
|
|
`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the
|
|
cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are
|
|
crops of balloon edges, window frames and background extras rather than of faces.
|
|
|
|
The visual comparison people reach for as the fix is **already implemented**, so do not build it again.
|
|
`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images
|
|
per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair
|
|
and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for
|
|
NONE when unsure. `choice: 0` becomes a new character and an out-of-range index becomes `unresolved`. The
|
|
`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`,
|
|
so the references reach the model.
|
|
|
|
That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon
|
|
edge against references enrolled from window frames and background extras, then sometimes answers "same".
|
|
Nothing gates enrollment on the crop holding a face.
|
|
|
|
Who pays: every later match, because the reference set defines what a character looks like. Fixing
|
|
`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place.
|
|
|
|
Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the
|
|
existing `ref_image_uris` and `embedding_uri` values as invalid.
|