Files

T

kami 8113bdfc8b Fix the real A/V gap: a stream copy across mixed frame rates

The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.

collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.

Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.

The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-12 12:16:22 +04:00

6.4 KiB

Raw Blame History

HANDOFF, 2026-08-12 (second session of the day)

Live state is in NEXT.md. This file is only what this session did. The previous handoff's content is now in JOURNAL.md and in decisions/.

Asked

Get up to speed from the previous handoff. Then: rebuild the chapter, and what about characters.

Result

Two things landed. The A/V gap is fixed and the rebuilt chapter.mp4 is in the bucket. The bbox coordinate space is settled with measurements instead of comments, and converted at the boundary.

One correction to carry forward. The first fix of the day named the wrong cause. Commit 1457556 claimed the xfade offset drift was the shipped 72.7s gap. The rebuild came out byte-identical to the broken file, which disproved it. Both are real defects. Only the second one shipped.

The chapter, rebuilt

s3://video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/chapter.mp4

before  v=436.392  a=363.675  nb_frames=9902  avg_frame_rate=22.69   gap +72.72
after   v=364.120  a=364.122  nb_frames=9101  r_frame_rate=25/1      gap  -0.002

Cause: assemble sends an all-cut chapter down a concat demuxer with -c copy. That path writes the output in the first input's time_base and reinterprets every later packet in it. 14 of the 49 clips are 30/1 at 1/15360, the other 35 are 25/1 at 1/12800, so those 14 play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd hardcoded -r 30. decisions/chapter-assembly.md#mixed-rate-stream-copy.

Fixes: collage_cmd emits -r FPS. assemble probes _fps_of across the clips and routes mixed rates through _assemble_batched, whose branches both normalize with fps={FPS}.

A second, latent defect on the transition path was fixed and committed separately. _xfade_chain took offsets from format=duration, which is max(video, audio). The accumulator crept past the end of its input, and ffmpeg discarded whole clips at rc 0 with nothing on stderr. decisions/chapter-assembly.md#offsets-from-min-stream.

The single-item passthrough theory from the previous handoff is dead, recorded void at decisions/chapter-assembly.md#passthrough-innocent. The one-path rewrite it recommended is not needed.

Still open here. The 14 clips in the bucket are still 30fps. Assembly normalizes them, so the chapter is correct, but the fast stream-copy path stays off for this chapter until render re-runs. Nobody has watched the rebuilt video yet. The 2:52 slide transition and the 28s static hold from 2:24 were both supposed to be re-judged after the sync fix.

Characters: the `bbox` space, settled

All 113 detections, straight from /review/identity:

test	result
boxes with `x2` past the 900px panel width	47 of 113
boxes with `y2` past 1000, on panels 1257 to 2307px tall	0 of 113
boxes clamped at exactly 1000	21 in x, 5 in y
coordinate range over every box	`[0, 1000]`

Gemma's native 0-1000 grid. Not pixels. worker_vision.py prompt text and the old worker_identity.py:91 comment both claimed pixels and both were wrong.

/vision now calls _bbox_to_pixels(characters, w, h) before returning. Four consumers are fixed at once: _crop_bbox in identity, _pair_faces_to_present, the set-of-mark boxes, and the review UI's client-side crop. The pairing one was comparing real pixel face boxes against 0-1000 character boxes, which is the likely mechanism behind 7 unknown out of 7 som_face lines. decisions/identity-bbox.md#bbox-is-normalized.

Checked by eye on panel 7, not just asserted. Five of six converted boxes land on their subject. That includes person_5, who is Seonho in the foreground with headphones and carried no identity. person_1 still frames an empty window mullion, which is caveats/speaker-attribution.md#extras-as-cast.

Converted boxes for panel 7, for whoever redraws the overlay:

person_1 [226, 414, 286, 553]   window mullion, nobody
person_2 [ 34, 558, 106, 749]   background extra, was assigned Lim Seonho
person_3 [428, 384, 494, 533]   background extra
person_4 [498, 389, 561, 549]   background extra
person_5 [460, 657, 631, 939]   Seonho, foreground. was assigned nothing
person_6 [646, 591, 767, 794]   the colleague, no name in the story. was assigned Choi Haeseon at 0.9

The registry is unchanged and still wrong. Every stored box, embedding and ref_image_uris was enrolled from the wrong space. vision and identity have to re-run before any of it means anything, and that is GPU work nobody authorized. caveats/speaker-attribution.md#bbox-wrong-space is marked resolved with the rerun pending.

Checks

Every self-check runs from the repo root and passes:

.venv/bin/python worker_render.py     # about 4 minutes, real ffmpeg
.venv/bin/python worker_vision.py
.venv/bin/python worker_identity.py

Three checks were added, because the existing ones passed all the way through both shipped defects:

_fps_of(collage clip) == "25/1", on a real collage encode. This is the one that would have caught the mixed-rate bug at the source.
three clips whose audio outlasts their video by 0.4s, assembled through the xfade branch. Mutation tested by restoring _audio_dur: fires with video=1.80 audio=3.56 expected=3.56.
_bbox_to_pixels against panel 7's real person_5 box, asserting the result covers the lower half of a 1650px panel, which a raw grid value cannot.

_check_assembled now runs after every encode on both paths, because ffmpeg returns 0 while dropping whole inputs.

Next command

Watch the rebuilt chapter before anything else. That is what found every real defect so far.

/usr/bin/ssh kami@192.168.1.104 'mc cat homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/chapter.mp4' > chapter.mp4

Then, with a go-ahead, the vision and identity rerun in NEXT.md item 1.

Traps confirmed again this session

tmux session manga-workers was gone and both systemd units were inactive. ./start_workers.sh starts 9 windows. The render worker is window 9 and must be restarted by hand to pick up an edit.
Plain ssh is the kitty ssh kitten and refuses non-interactive stdin. Use /usr/bin/ssh.
cd $dir && .venv/bin/python fails, because the venv path is relative to the repo root. Use the absolute interpreter path when the working directory is elsewhere.
The whole assembly investigation ran offline on 49 downloaded clips with no GPU and no orchestrator. Re-download with the command in NEXT.md.

6.4 KiB Raw Blame History