kami
8113bdfc8b
The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.
collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.
Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.
The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
158 lines
9.6 KiB
Markdown
158 lines
9.6 KiB
Markdown
# speaker-attribution
|
|
|
|
Limits found by cross-checking the 2026-08-11 chapter run against the panel images.
|
|
|
|
## Nothing attributes a speaker in a multi-character panel {#tail-is-not-geometry}
|
|
|
|
The false `tail` label is gone (`decisions/speaker-attribution.md#no-fake-tail`). What replaced it is a
|
|
refusal, not an answer: with two or more characters present, every speech line now returns `unknown`.
|
|
On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back
|
|
to a `generic-handle`. That is the honest floor, and it is not the fix.
|
|
|
|
The measurement that forced it, on job `778297bc`, chapter `7c944dd4`. Three two-character panels were
|
|
checked against the art. All three are wrong, each with the two speakers swapped:
|
|
|
|
| panel key | line | truth | pipeline |
|
|
| --- | --- | --- | --- |
|
|
| `p010.png` | "…definitely an Egen guy, Seonho!" | the woman | Seonho, the person addressed |
|
|
| `p010.png` | "Y-you think so?" | Seonho | Choi Haeseon |
|
|
| `p012.png` | "Want me to send you the link?" | the woman | the man |
|
|
| `p059.png` | "If team leader Choi says it, it must be true." | the man | Choi Haeseon |
|
|
|
|
The last row needs no image: the line refers to Choi in the third person and is attributed to Choi.
|
|
|
|
The grounded path exists and almost never fires. Only 2 of 81 speech lines got `som_face`, because
|
|
attribution marks need `face_detect` boxes that survive `_pair_faces_to_present`, and these webtoon
|
|
close-ups rarely produce them. Inference, not measured: the face detector was not instrumented.
|
|
|
|
`worker_vision.py:356` already carries the `ponytail:` note that multi-character attribution needs
|
|
per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads carry balloon fill and
|
|
tail tips and are unused.
|
|
|
|
**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric
|
|
(`ROADMAP.md`). The 30% read on this run counted attributions the sample says are wrong. The next run
|
|
will read lower and will be the first honest number. Raising it means binding a balloon to a speaker
|
|
by tail geometry, using the unused `det`/`seg` heads.
|
|
|
|
## One character id covers two different women {#identity-over-merge}
|
|
|
|
`character_afa7623b` is stored as "black bob, white sweater" and is assigned both to that person
|
|
(`p059.png`) and to the brown-bob green-top coworker (`p010.png`, `p012.png`). It took 25 of the 26
|
|
identity assignments in the chapter, against 113 detected people. Coverage is 23%.
|
|
|
|
**Revisit trigger:** any work on the identity Tier-2 decider. A single id absorbing a whole chapter is
|
|
the signature to watch for.
|
|
|
|
## The character registry carries five weeks of wrong names {#registry-pollution}
|
|
|
|
The registry holds 53 characters for manga `ef105a86`, 41 of them unnamed, with "Kei" three times and
|
|
"Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only
|
|
Haeseon and Seonho do. `/stage/clear` leaves the per-manga registry intact by design, so every rerun
|
|
inherits the whole pile.
|
|
|
|
Duplicate rows also make a correct name unresolvable. This manga holds `Choi Haeseon`, `Seonho` with
|
|
aliases `["Lim Seonho", "Seonho"]`, and a separate `Lim Seonho`. An answer of "Lim Seonho" matches two
|
|
rows, so `normalize_speaker` returns candidates and raises `ambiguous-speaker` instead of binding. The
|
|
pipeline read the name correctly and still cannot name the speaker.
|
|
|
|
**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the
|
|
registry to a chapter or add a reviewed reset. Merging the duplicate rows needs the reversible-merge
|
|
design first (`caveats/audit-open.md#destructive-reconcile`).
|
|
|
|
## One invented word still halts the chapter {#one-word-halts-chapter}
|
|
|
|
The multi-word name failure is fixed (`decisions/speaker-attribution.md#multiword-cast-names`). The
|
|
blast radius it exposed is not. `run_stage_script` retries a rejected beat once, then raises, so a
|
|
single unsupported token ends the run at that beat. On job `778297bc` one of the 29 lost beats cited
|
|
`['Blur']`, an onomatopoeia the model invented. The verifier was right, and the whole chapter still
|
|
stopped.
|
|
|
|
**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer
|
|
is to flag the beat for review and continue, which is `#136` gate work, not a verifier change.
|
|
|
|
## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space}
|
|
|
|
**Resolved 2026-08-12, `decisions/identity-bbox.md#bbox-is-normalized`.** The space is gemma's 0-1000
|
|
grid, proven over all 113 detections, and `/vision` now converts to pixels before returning. The face
|
|
pairing at `worker_vision.py:57` was reading the same numbers against real pixel face boxes, so it is
|
|
fixed by the same change. What is left of this entry is the consequence. Every stored assignment,
|
|
embedding and `ref_image_uris` came from a wrong crop. Identity has to re-run before any of it means
|
|
anything. The rest below is kept as the record of how it read before.
|
|
|
|
Vision's `bbox` values are stored and consumed as absolute pixels. On panel
|
|
`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of
|
|
the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own
|
|
dimensions, four of the six fit their subjects tightly.
|
|
|
|
Two places in the code assert pixels, and the art contradicts both:
|
|
|
|
- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
|
|
- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
|
|
|
|
Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces
|
|
against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9
|
|
from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed
|
|
alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at
|
|
`worker_vision.py:57` reads the same numbers and was not checked.
|
|
|
|
Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with
|
|
nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already
|
|
calls the box "coarse, imprecise".
|
|
|
|
Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can
|
|
be judged while the crops are wrong.
|
|
|
|
## Identity cannot say "a person with no name" {#no-anonymous-identity}
|
|
|
|
The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9.
|
|
Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline
|
|
stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which
|
|
is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing
|
|
line.
|
|
|
|
This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an
|
|
unparseable model answer. The missing rule: never attach a name to a detection that carries no name
|
|
evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the
|
|
colleague" every time.
|
|
|
|
`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from
|
|
cosine clearing the threshold on a wrong crop. Whether the Tier-2 gemma resolver can answer "none of
|
|
these" was not verified.
|
|
|
|
Revisit trigger: immediately after the `bbox` space is settled.
|
|
|
|
## Vision does not separate a background extra from cast {#extras-as-cast}
|
|
|
|
Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground
|
|
and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame
|
|
with nobody in it. All six reach identity as equal candidates.
|
|
|
|
Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly
|
|
extras, so it measured nothing useful and should not be quoted again.
|
|
|
|
Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return.
|
|
|
|
## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set}
|
|
|
|
`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the
|
|
cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are
|
|
crops of balloon edges, window frames and background extras rather than of faces.
|
|
|
|
The visual comparison people reach for as the fix is **already implemented**, so do not build it again.
|
|
`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images
|
|
per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair
|
|
and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for
|
|
NONE when unsure. `choice: 0` becomes a new character and an out-of-range index becomes `unresolved`. The
|
|
`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`,
|
|
so the references reach the model.
|
|
|
|
That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon
|
|
edge against references enrolled from window frames and background extras, then sometimes answers "same".
|
|
Nothing gates enrollment on the crop holding a face.
|
|
|
|
Who pays: every later match, because the reference set defines what a character looks like. Fixing
|
|
`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place.
|
|
|
|
Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the
|
|
existing `ref_image_uris` and `embedding_uri` values as invalid.
|