Files

T

kami 8113bdfc8b Fix the real A/V gap: a stream copy across mixed frame rates

The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.

collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.

Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.

The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-12 12:16:22 +04:00

5.8 KiB

Raw Permalink Blame History

chapter-assembly

Settled questions about _assemble_batched / _assemble_once / _xfade_chain in worker_render.py.

The shipped 72.7s gap was a stream copy across mixed frame rates

Closed, 2026-08-12.

assemble routes an all-cut chapter to a concat demuxer with -c copy. That path writes the output with the first input's time_base and reinterprets every later packet in it.

14 of this chapter's 49 clips came off collage_cmd, which hardcoded -r 30. They carry r_frame_rate=30/1 and time_base=1/15360. The other 35 are 25/1 at 1/12800. Copied into the first clip's timebase, those 14 play 15360/12800 = 1.2 times too long while their audio is untouched. That is the 1.2001 ratio, and the whole of video 436.39s over audio 363.67s.

Reproduced offline by running the same -c copy concat over the 49 real clips. Duration 436.392031 and nb_frames 9902, identical to the shipped file. Seconds to run, no GPU.

Two changes hold it closed:

collage_cmd emits -r FPS like every other clip path, and the __main__ self-check asserts _fps_of(clip) == "25/1" on a real collage encode.
assemble probes r_frame_rate across the clips and sends mixed rates through _assemble_batched, whose branches both normalize with fps={FPS}. Only a single shared rate keeps the stream copy.

Verified end to end. The rebuilt chapter.mp4 is video 364.120s against audio 364.122s at 25/1.

An earlier version of this file, and commit 1457556, blamed #offsets-from-min-stream below for the shipped gap. That was wrong. The rebuild came out byte-identical to the broken file, which proved the xfade tree never ran for this chapter. The entry below is a real defect and stays closed on its own evidence. It was not this one.

xfade offsets are computed from `min(video, audio)`, never `format=duration`

Closed, 2026-08-12. A real latent defect on the transition path. Not the cause of the shipped gap, see #mixed-rate-stream-copy above.

_xfade_chain accumulates cum += dur[i] - td and hands each boundary offset=cum-td. That offset is an assertion about where input i-1 still has frames. It fed on _audio_dur, which probes format=duration, which is max(video, audio). A rendered clip's audio outlasts its video by about a frame. So every boundary pushed the accumulator further ahead of the picture.

Once the accumulated overshoot exceeds the transition width, the xfade window starts after the last frame of input i-1. ffmpeg emits the transition and then silently discards input i and every clip downstream of it. rc 0, no warning on stderr, output file present and playable. Measured on a group of 8 real clips, chain truncated at each stage:

k=7 out= 52.52   correct
k=8 out= 52.52   the last xfade contributed nothing
    [v6][n7]xfade=duration=0.050:offset=52.500   <- [v6] is only 52.52s long, so 0.02s of margin

This is the whole cause of the shipped chapter being video 436.39s over audio 363.67s. It reproduces with synthetic clips in about four minutes and needs no GPU.

Two changes hold it closed:

_assemble_once probes min(_stream_dur(p, "v"), _stream_dur(p, "a")). The minimum, because either stream running long breaks a different half of the graph.
_xfade_chain floors every input to a whole frame count. It applies trim and atrim to both streams, so the accumulator tracks the real timeline instead of estimating it. Transition widths are quantized to frames for the same reason.

Verified over the 49 real clips of chapter 7c944dd4. The round that previously turned 359s of video into 100s now loses 0.85s. The chapter comes out video 358.76s against audio 358.76s, agreeing to the frame.

Forbidden from here: _audio_dur in anything that positions a filter. It is fine for "how long is this clip roughly", nothing else.

Assembly verifies its own output instead of trusting ffmpeg's exit code

Closed, 2026-08-12.

Both assembly branches drop stream time without failing. xfade discards inputs as above. The concat demuxer with -vsync cfr drops video frames to force a constant rate. Neither is an error to ffmpeg.

So _assemble_once calls _check_assembled(out, expect) after every encode. It compares the output's video stream against the predicted timeline and against its own audio stream. It raises when either is off by more than ASSEMBLE_TOL_S, which is 0.5s. That tolerance covers frame boundaries and aac padding. A dropped input is off by whole seconds.

Without this the failure stays invisible until somebody watches the video. That is how a 50MiB chapter with 72.7s of silent picture reached the bucket while every stage counter read success.

The single-item passthrough is not the bug

Void, 2026-08-12. Cited in HANDOFF.md for 2026-08-11 as the prime suspect and must not be cited again.

With 49 clips and ASSEMBLE_BATCH=8, round 0 makes six groups of 8 plus a leftover group of 1, which _assemble_batched carries forward un-encoded. The theory was that mixing that raw clip with six encoded intermediates broke round 1. The instrumented run disproves it: two round-0 groups of 8 fresh clips collapse on their own, before any passthrough exists.

n=8 XFADE  in v=  63.52 a=  63.60 -> out v=  52.52 a=  62.77   lost_v= +11.00
n=8 XFADE  in v=  58.12 a=  58.20 -> out v=  12.04 a=  56.83   lost_v= +46.08

Round 1's 259s loss was the cascade. Its inputs already held 309s of video against 364s of audio, and durs read the audio.

Consequence for the plan: collapsing concat, xfade and the passthrough into one path was the recommended fix in NEXT.md and is not needed. Three paths are fine once each one positions filters on a real timeline. The tree keeps the bounded memory it was built for.

5.8 KiB Raw Permalink Blame History