Files

T

kami 8113bdfc8b Fix the real A/V gap: a stream copy across mixed frame rates

The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.

collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.

Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.

The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-12 12:16:22 +04:00

9.6 KiB

Raw Blame History

speaker-attribution

Limits found by cross-checking the 2026-08-11 chapter run against the panel images.

Nothing attributes a speaker in a multi-character panel

The false tail label is gone (decisions/speaker-attribution.md#no-fake-tail). What replaced it is a refusal, not an answer: with two or more characters present, every speech line now returns unknown. On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back to a generic-handle. That is the honest floor, and it is not the fix.

The measurement that forced it, on job 778297bc, chapter 7c944dd4. Three two-character panels were checked against the art. All three are wrong, each with the two speakers swapped:

panel key	line	truth	pipeline
`p010.png`	"…definitely an Egen guy, Seonho!"	the woman	Seonho, the person addressed
`p010.png`	"Y-you think so?"	Seonho	Choi Haeseon
`p012.png`	"Want me to send you the link?"	the woman	the man
`p059.png`	"If team leader Choi says it, it must be true."	the man	Choi Haeseon

The last row needs no image: the line refers to Choi in the third person and is attributed to Choi.

The grounded path exists and almost never fires. Only 2 of 81 speech lines got som_face, because attribution marks need face_detect boxes that survive _pair_faces_to_present, and these webtoon close-ups rarely produce them. Inference, not measured: the face detector was not instrumented.

worker_vision.py:356 already carries the ponytail: note that multi-character attribution needs per-balloon geometry. bubble_detect.py:9 records that the det/seg heads carry balloon fill and tail tips and are unused.

Revisit trigger: the share of narrated lines with a named speaker is the Phase 1 headline metric (ROADMAP.md). The 30% read on this run counted attributions the sample says are wrong. The next run will read lower and will be the first honest number. Raising it means binding a balloon to a speaker by tail geometry, using the unused det/seg heads.

One character id covers two different women

character_afa7623b is stored as "black bob, white sweater" and is assigned both to that person (p059.png) and to the brown-bob green-top coworker (p010.png, p012.png). It took 25 of the 26 identity assignments in the chapter, against 113 detected people. Coverage is 23%.

Revisit trigger: any work on the identity Tier-2 decider. A single id absorbing a whole chapter is the signature to watch for.

The character registry carries five weeks of wrong names

The registry holds 53 characters for manga ef105a86, 41 of them unnamed, with "Kei" three times and "Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only Haeseon and Seonho do. /stage/clear leaves the per-manga registry intact by design, so every rerun inherits the whole pile.

Duplicate rows also make a correct name unresolvable. This manga holds Choi Haeseon, Seonho with aliases ["Lim Seonho", "Seonho"], and a separate Lim Seonho. An answer of "Lim Seonho" matches two rows, so normalize_speaker returns candidates and raises ambiguous-speaker instead of binding. The pipeline read the name correctly and still cannot name the speaker.

Revisit trigger: before any run that is meant to produce a clean baseline. Either scope the registry to a chapter or add a reviewed reset. Merging the duplicate rows needs the reversible-merge design first (caveats/audit-open.md#destructive-reconcile).

One invented word still halts the chapter

The multi-word name failure is fixed (decisions/speaker-attribution.md#multiword-cast-names). The blast radius it exposed is not. run_stage_script retries a rejected beat once, then raises, so a single unsupported token ends the run at that beat. On job 778297bc one of the 29 lost beats cited ['Blur'], an onomatopoeia the model invented. The verifier was right, and the whole chapter still stopped.

Revisit trigger: the next unsupported-proper-noun halt that is a true positive. The likely answer is to flag the beat for review and continue, which is #136 gate work, not a verifier change.

Every `bbox` is read in the wrong coordinate space

Resolved 2026-08-12, decisions/identity-bbox.md#bbox-is-normalized. The space is gemma's 0-1000 grid, proven over all 113 detections, and /vision now converts to pixels before returning. The face pairing at worker_vision.py:57 was reading the same numbers against real pixel face boxes, so it is fixed by the same change. What is left of this entry is the consequence. Every stored assignment, embedding and ref_image_uris came from a wrong crop. Identity has to re-run before any of it means anything. The rest below is kept as the record of how it read before.

Vision's bbox values are stored and consumed as absolute pixels. On panel 7c944dd4-e972-42c7-ba60-9f6939548e80_p007 (crop 900x1650) all six boxes then land in the top third of the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own dimensions, four of the six fit their subjects tightly.

Two places in the code assert pixels, and the art contradicts both:

worker_vision.py:271, prompt text: pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)
worker_identity.py:91, comment: vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)

Who pays: identity embeds _crop_bbox(img, ch["bbox"]) at worker_identity.py:200, so it matches faces against crops of balloons and window frames. On panel 7 that produced Choi Haeseon at confidence 0.9 from a crop of a balloon edge and Lim Seonho at 0.9 from an empty window frame. Blank crops embed alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at worker_vision.py:57 reads the same numbers and was not checked.

Rescaling is necessary and not sufficient. After scaling, person_1 still sits on a window frame with nobody in it, and person_6 clips its subject and runs onto the frame. worker_vision.py:38 already calls the box "coarse, imprecise".

Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of bbox can be judged while the crops are wrong.

Identity cannot say "a person with no name"

The colleague on panel 7 has no name in the story. She was assigned Choi Haeseon at confidence 0.9. Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing line.

This is the same shape as invariant 6 in CLAUDE.md, which forbids minting a character from an unparseable model answer. The missing rule: never attach a name to a detection that carries no name evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the colleague" every time.

match() at worker_identity.py:69 does abstain, returning None below threshold, so the 0.9 came from cosine clearing the threshold on a wrong crop. Whether the Tier-2 gemma resolver can answer "none of these" was not verified.

Revisit trigger: immediately after the bbox space is settled.

Vision does not separate a background extra from cast

Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground and the unnamed colleague. Three are background office extras, and one (person_1) is a window frame with nobody in it. All six reach identity as equal candidates.

Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly extras, so it measured nothing useful and should not be quoted again.

Revisit trigger: with #no-anonymous-identity, since both change what identity is allowed to return.

Cast reference profiles are enrolled from wrong crops

characters carries ref_image_uris and embedding_uri, and all 53 rows have both populated. So the cast-profile mechanism exists. It is enrolled through #bbox-wrong-space, so the stored references are crops of balloon edges, window frames and background extras rather than of faces.

The visual comparison people reach for as the fix is already implemented, so do not build it again. /vision/resolve at worker_vision.py:963 sends the query crop plus up to 3 labelled reference images per candidate. build_resolve_prompt already tells the model to judge face shape first, to treat hair and outfit as secondary, that two people sharing a hair colour are not the same, and to answer 0 for NONE when unsure. choice: 0 becomes a new character and an out-of-range index becomes unresolved. The ref_image_uris column is republished as reference_image_uris at worker_identity.py:152 and :161, so the references reach the model.

That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon edge against references enrolled from window frames and background extras, then sometimes answers "same". Nothing gates enrollment on the crop holding a face.

Who pays: every later match, because the reference set defines what a character looks like. Fixing #bbox-wrong-space without re-enrolling leaves the poisoned references in place.

Revisit trigger: as soon as #bbox-wrong-space lands, re-enroll from corrected crops and treat the existing ref_image_uris and embedding_uri values as invalid.

9.6 KiB Raw Blame History