manga-recap-pipeline/HANDOFF.md

HANDOFF, 2026-08-12

Live state is in NEXT.md. This file is only what this session did.

Asked

Get up to speed from the previous handoff. Then: the user watched chapter.mp4 for the first time and read out 19 timestamped defects. Mid-session, what about characters. Then, write the handoff.

Result

Somebody finally watched the video. That single act found more than four sessions of measuring did. The recorded metrics said script 116/116 and "9 named speech lines". Both were true. Both measured the wrong thing, because the 9 names are the wrong name.

Two hard numbers came out of it.

1. The chapter is 20% out of sync. chapter.mp4 is video 436.39s over audio 363.67s. The narration ends 72.7 seconds before the picture. The gap accumulates, which is why everything after 2:54 goes sideways. The 49 clips are clean: video and audio agree to 0.03s and sum to 363.6s. Assembly adds 72.7s of video and no audio.
2. Identity binds names to the wrong people, and to people who have no name. Walking panel 7 against the art found zero correct bindings out of two, plus the one character who matters left unbound. See Panel 7, walked against the art, which supersedes the earlier reading of this.

Nothing is committed. worker_render.py is edited in the working tree. The fix is not verified.

The user's 19 notes, grouped by cause

cause	timestamps	what is wrong
one row absorbed every identity	0:20, 1:07, 1:51, 1:59, end	anyone identified comes out Choi Haeseon
the MC has no name	0:44, 0:50, 1:02, 1:23, 1:45	falls back to "the worker", "someone", "she"
gender read off the wrongly bound row	1:45 "she admits", end "as he waves"	Choi is f, so "he" means a nameless m row got the line
narration invents facts	0:43 "results", 2:03, 2:05, 2:15 "long shift"	the verifier checks quotes and names, not invented claims
vision reads art-within-art as scene	1:35 chibi on a monitor as "a man holding a drink", 1:59 "pointing towards the screen"	panel-in-panel and screen content taken as reality
a beat carries nothing	0:35-0:37	no content worth narrating
no parallax, so a still holds	2:24-2:52, 28s static	layers wrote nothing (caveats/audit-open.md#layers-writes-nothing)
transition quality	2:52-2:54 slide "too sharp and laggy"	push is slideleft at 0.4s. Retest after the sync fix
A/V drift	everything after 2:54	the 72.7s gap above

Panel 7, walked against the art

This is the load-bearing finding of the session. The user pulled up the panel and checked every detection by eye. Read this before touching identity. It contradicts what the earlier sessions recorded, and it contradicts two theories I floated today before the user corrected them.

Panel 7c944dd4-e972-42c7-ba60-9f6939548e80_p007, crop s3://panels/.../panels/p006.png, 900x1650. A wide establishing shot of an office seen through a window. Vision emitted 6 characters.

detection	vision said	the art shows	identity assigned
person_5	m, short black, yellow sweater, sitting	Seonho, foreground, yellow plaid, headphones, back to camera. The character who matters	nothing
person_6	f, short brown, white shirt, sitting	the colleague, green dress, ponytail. She has no name in the story	Choi Haeseon at 0.9
person_2	m, short brown, green sweater, sitting	a background extra, seated beyond the next window pane	Lim Seonho at 0.9
person_1	m, short black, suit, standing	nobody. A window frame	nothing
person_3	m, short black, blue sweater, standing	background extra	nothing
person_4	m, short black, grey sweater, standing	background extra	nothing

Zero of the two bindings are right, and the one character who matters got nothing. Both wrong binds carry confidence 0.9.

Three separate defects stack here.

1. The stored bbox coordinate space is wrong. Consumed as absolute pixels, all six boxes land in the top third of a 1650px-tall panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own dimensions, person_2, person_3, person_4 and person_5 fit their subjects tightly. So the numbers are not pixels. Two places assert that they are:

• worker_vision.py:271 prompt text: pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)
• worker_identity.py:91 comment: vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)

Everything reading bbox is therefore cropping the wrong region. worker_identity.py:200 embeds _crop_bbox(img, ch["bbox"]), so Choi Haeseon at 0.9 was matched on a crop of the speech balloon's edge and Lim Seonho at 0.9 on a crop of empty window frame. Wrong crops are mostly blank white, which embed alike, which is a plausible mechanism for one row absorbing 25 assignments.

Do not treat rescaling as the fix. I claimed that and the user disproved it in one screenshot. After scaling, person_1 still sits on a window frame with nobody in it, and person_6 is offset, clipping the woman and running onto the dark frame. worker_vision.py:38 already calls the box "coarse, imprecise". Scaling buys roughly-right boxes, not right ones.

2. Vision has no concept of extra versus cast. Four of the six detections are background extras or nothing at all. They are handed to identity as candidates on equal footing with the two people who carry the scene. That also means the "113 detected people" figure that framed the whole roadmap was never the right denominator, so "26 of 113 carry an identity" measured nothing useful.

3. Identity mints a name onto a person who has none. The colleague has no name in the story. She was labelled Choi Haeseon at 0.9. Across the chapter Choi Haeseon holds 25 of 26 assignments, so in practice that row is the label this pipeline stamps on any unnamed woman. This is the direct cause of the user's 0:20 note, "Choi Haeseon when there's no Choi in the frame, it's the colleague", and of the gender flips, since gender is read off whichever row got bound.

This is a cousin of invariant 6 in CLAUDE.md, which forbids minting a character from an unparseable model answer. The missing rule is the same shape: never attach a name to a detection that carries no name evidence. An unnamed recurring person needs a stable anonymous identity so the narration can call her "the colleague" every time, rather than being forced onto a named row.

Partly checked, not finished: match() in worker_identity.py:69 does abstain, returning None below threshold, so the 0.9 came from cosine clearing the threshold on a garbage crop. Whether the Tier-2 gemma resolver can answer "none of these" was not verified. Check that first.

Descriptions are not trustworthy either

person_6 is "white shirt" for a woman in a green dress. person_5 is "yellow sweater" for yellow plaid, which is close enough. Any downstream rule keyed on appearance text inherits this.

Registry, measured

characters is keyed by manga_id, not chapter. Two manga share the table. d7104032 has 34 rows with 9 named. This chapter's ef105a86 has 19 rows with 3 named. The Kei, Zen, Kanade, Rico and K3 rows belong to the other manga, so they are not polluting this chapter.

This manga's 19 rows, all status=confirmed, all first_seen_panel=NULL:

• named: Choi Haeseon (f), Seonho (m, aliases ["Lim Seonho","Seonho"]), Lim Seonho (m)
• 16 rows with name=NULL and an empty alias list

Seonho and Lim Seonho are the same person in two rows. Seonho's alias list contains the other row's name. That is why either spelling matches two rows and binds nothing.

Assignments across the whole table, for scale:

character	name	assignments
character_afa7623b	Choi Haeseon	25
character_bb79cfb4	Kanade	25
character_cfd34340	Rico	19
character_6f491712	Lim Seonho	1

In this chapter only Choi Haeseon (25) and Lim Seonho (1) appear.

Read those 25 together with panel 7. Choi Haeseon is not a character who appears 25 times. It is the row that absorbs any unnamed woman. The 16 nameless rows are not a backlog of people waiting for names. Some of them are background extras that should never have become rows, and at least one of them, the colleague, is a real recurring person who correctly has no name and needs to keep it.

An earlier draft of this file said "naming is the ceiling, 16 of 19 rows need names". That was wrong and it is corrected here. The ceiling is that identity cannot say "person, no name" and cannot tell an extra from cast.

The A/V bug, located but not fixed

Reproduced on the 49 real clips with repro.py and probe.py (see Open). Per-round probe with ASSEMBLE_BATCH=8 and 6 fade_black boundaries spread across batches:

r0 g0 n=8 XFADE  in v= 49.44 a= 49.44 -> out v= 48.56 a= 48.64   lost_v=+0.88 lost_a=+0.80
r0 g1 n=8 XFADE  in v= 47.04 a= 47.04 -> out v= 46.16 a= 46.23   lost_v=+0.88 lost_a=+0.81
r0 g2 n=8 XFADE  in v= 48.00 a= 48.00 -> out v= 47.12 a= 47.19   lost_v=+0.88 lost_a=+0.81
r0 g3 n=8 XFADE  in v=100.80 a=100.80 -> out v= 99.92 a= 99.96   lost_v=+0.88 lost_a=+0.84
r0 g4 n=8 concat in v= 64.18 a= 64.16 -> out v= 64.24 a= 64.26   lost_v=-0.06 lost_a=-0.10
r0 g5 n=8 XFADE  in v= 44.00 a= 44.00 -> out v= 43.12 a= 43.17   lost_v=+0.88 lost_a=+0.83
r1 g0 n=7 XFADE  in v=359.29 a=359.60 -> out v= 99.96 a=358.79   lost_v=+259.33 lost_a=+0.81

Round 0 is correct. Each group loses only the xfade overlap. Round 1 loses 259s of video against 0.8s of audio. Its output video is 99.96s at 2499 frames, almost exactly the frame count of input n3 on its own, the 99.92s intermediate. The final video appears to carry the frames of one input.

The round-1 filtergraph is arithmetically correct, so this is ffmpeg behaviour, not offset math:

[n0][n1]xfade=transition=fade:duration=0.050:offset=48.510[v1]
[v1][n2]xfade=transition=fade:duration=0.050:offset=94.620[v2]
[v2][n3]xfade=transition=fadeblack:duration=0.600:offset=141.140[v3]
[v3][n4]xfade=transition=fade:duration=0.050:offset=241.010[v4]
[v4][n5]xfade=transition=fade:duration=0.050:offset=305.200[v5]
[v5][n6]xfade=transition=fade:duration=0.050:offset=348.270[v6]

The strongest clue, found last

Re-running that chain by hand over only the 6 round-0 intermediates gives a correct 348.24s at 8708 frames, rc 0, no warnings. Round 1 collapses with 7 inputs, not 6.

The 7th input is the leftover 49th clip. With 49 clips and batch 8, round 0 makes 6 groups of 8 and one group of 1, and _assemble_batched passes a lone group through un-encoded:

if len(group) == 1 and not final_round:
    next_items.append(group[0])

So the final xfade mixes 6 encoded intermediates with 1 raw clip. That passthrough is a third path next to concat and xfade, and it is the prime suspect. Start here tomorrow. Confirm it by running probe.py with 48 clips instead of 49, which removes the leftover entirely.

What was tried and what it cost

Two paths exist in _assemble_once. A concat branch handles cut-only batches. An xfade branch handles batches with a real transition. They disagreed on frame rate. The concat branch forced fps=30 while the xfade branch normalized nothing, and clips are 25fps. 436.39 / 363.63 = 1.2001, exactly 30/25, which is what sent me down this path.

Working-tree changes to worker_render.py, all uncommitted:

• new FPS = 25 constant. The three hardcoded 25s and the one 30 now reference it
• the xfade branch normalizes every input with setsar=1,fps={FPS} into [n{i}] labels, matching what the concat branch already did
• new _stream_dur(path, kind). _audio_dur probes format=duration, which is max(video, audio), so it hides A/V drift by construction
• the __main__ xfade self-check now assembles 4 clips through _assemble_batched with ASSEMBLE_BATCH=2 and asserts abs(video - audio) < 0.25. It previously asserted only getsize(out) > 0, which is why this shipped

A dead end worth not repeating. I also pinned -r FPS on both output encodes. That made it worse. The chapter collapsed to exactly 100.00s at 2500 frames, because forcing CFR on irregular input timestamps drops frames. The comment already sitting at the concat branch warns about this. Both -r flags were removed again. The in-graph fps= filter is the right normalization. The output -r is not.

The fps inconsistency is real and worth keeping fixed. It is not proven to be the cause of the shipped 72.7s gap. The scene graphs hold 356 cut against 6 fade_black, so round 2 of the real run most likely stayed on the concat branch, where no mixing occurs. Treat the fps work as necessary and insufficient.

The user's own hypothesis, which is the recommended direction

Two guesses, both worth following:

1. the assembly is wrong in an ffmpeg sense
2. there are two different paths, and they get mixed when there should only be one

Guess 2 matches the code. concat, xfade and the single-item passthrough are three paths, and _assemble_batched feeds the output of one into the input of another. Collapse it to one path. Normalize every input, then xfade every boundary, with cut as a 0.05s fade. acrossfade shortens audio by the same amount that xfade shortens video, so A/V stays locked. The chapter then comes out about 2.4s shorter than the sum of the clips, with both streams agreeing. That removes the branch interaction instead of tuning it.

Measured

Job 778297bc-e7ce-439d-91b5-8a027060d17f, chapter 7c944dd4-e972-42c7-ba60-9f6939548e80.

• job status=completed, every stage at its unit count, finished 2026-08-11T20:08:16Z
• it still carries error: "partial: 112/116 completed" (caveats/audit-open.md#stale-job-error)
• chapter.mp4 50MiB, video 436.392s, audio 363.675s, r_frame_rate=25/1, avg_frame_rate=63372800/2792909 which is 22.69, nb_frames=9902
• 49 clips, every one 25/1 exactly, sum video 363.63s, sum audio 363.60s. No clip has the two differing by more than 0.05s
• scene-graph transitions: cut 356, fade_black 6
• worker_render.py __main__ self-check passes on the current working tree
• workers up in tmux manga-workers, 9 windows

Open

• Finish the round-1 diagnosis. The _assemble_batched tree turns 359s of video into 100s. This is the worst defect found and it reproduces offline in about two minutes with no GPU. Suspect the 7th passthrough input first.

  .venv/bin/python <scratchpad>/probe.py    # the per-round loss table above
  .venv/bin/python <scratchpad>/repro.py    # end-to-end verdict
  

  The scratchpad is session-scoped and will be gone. Re-download the clips first:

  /usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/
  

  Both scripts are worth committing next session. They are the only check that has ever caught this.

• Do not trust format=duration. It returns max(video, audio), so every existing duration assert in worker_render.py is blind to drift. _stream_dur exists now. The other asserts still use _audio_dur.

• Identity, in the order the panel 7 evidence implies. First, settle the bbox coordinate space and fix every consumer, since nothing else can be judged while crops are wrong. Second, let identity abstain and hold a stable anonymous identity, so the colleague stays "the colleague". Third, separate extra from cast so extras never reach identity. Only then merge Seonho into Lim Seonho and split character_afa7623b, which still needs the reversible-merge design (caveats/audit-open.md#destructive-reconcile).

• Verify the bbox space before changing anything. Two independent claims in the code say pixels, and the art says otherwise. Confirm what the model was told and what it returns, rather than trusting either comment. Then check whether the crop is the only consumer, or whether SoM marker placement and the face-pairing in worker_vision.py:57 read the same numbers.

• Re-derive the panel 7 overlay when needed. It took one mc cat of the crop plus a Pillow script, and it found more than any query did:

  /usr/bin/ssh kami@192.168.1.104 'mc cat homesrv/panels/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/panels/p006.png' > p007.png
  

  Draw each bbox twice, once as pixels and once divided by 1000, then look at it.

• mc aliases on homesrv: use homesrv or mio, not local, which returns Access Denied. rfs is the empty rustfs.

• Plain ssh is the kitty ssh kitten and refuses non-interactive stdin. Use /usr/bin/ssh.

• cp is aliased to cp -i in this shell and hangs on overwrite. Use /usr/bin/cp -f.

• The Bash tool's default timeout is 120s no matter what timeout the command itself carries. Pass the tool's own timeout or background the run. Otherwise a restore step after a mutation test never runs, which left a deliberately broken worker_render.py on disk once this session.