kami
8113bdfc8b
The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.
collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.
Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.
The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
108 lines
5.8 KiB
Markdown
108 lines
5.8 KiB
Markdown
# chapter-assembly
|
|
|
|
Settled questions about `_assemble_batched` / `_assemble_once` / `_xfade_chain` in `worker_render.py`.
|
|
|
|
## The shipped 72.7s gap was a stream copy across mixed frame rates {#mixed-rate-stream-copy}
|
|
|
|
**Closed, 2026-08-12.**
|
|
|
|
`assemble` routes an all-`cut` chapter to a `concat` demuxer with `-c copy`. That path writes the output
|
|
with the **first** input's `time_base` and reinterprets every later packet in it.
|
|
|
|
14 of this chapter's 49 clips came off `collage_cmd`, which hardcoded `-r 30`. They carry
|
|
`r_frame_rate=30/1` and `time_base=1/15360`. The other 35 are `25/1` at `1/12800`. Copied into the first
|
|
clip's timebase, those 14 play `15360/12800 = 1.2` times too long while their audio is untouched. That is
|
|
the 1.2001 ratio, and the whole of video 436.39s over audio 363.67s.
|
|
|
|
Reproduced offline by running the same `-c copy` concat over the 49 real clips. Duration 436.392031 and
|
|
`nb_frames` 9902, identical to the shipped file. Seconds to run, no GPU.
|
|
|
|
Two changes hold it closed:
|
|
|
|
* `collage_cmd` emits `-r FPS` like every other clip path, and the `__main__` self-check asserts
|
|
`_fps_of(clip) == "25/1"` on a real collage encode.
|
|
* `assemble` probes `r_frame_rate` across the clips and sends mixed rates through `_assemble_batched`,
|
|
whose branches both normalize with `fps={FPS}`. Only a single shared rate keeps the stream copy.
|
|
|
|
Verified end to end. The rebuilt `chapter.mp4` is video 364.120s against audio 364.122s at `25/1`.
|
|
|
|
**An earlier version of this file, and commit `1457556`, blamed `#offsets-from-min-stream` below for the
|
|
shipped gap. That was wrong.** The rebuild came out byte-identical to the broken file, which proved the
|
|
xfade tree never ran for this chapter. The entry below is a real defect and stays closed on its own
|
|
evidence. It was not this one.
|
|
|
|
## xfade offsets are computed from `min(video, audio)`, never `format=duration` {#offsets-from-min-stream}
|
|
|
|
**Closed, 2026-08-12.** A real latent defect on the transition path. Not the cause of the shipped gap,
|
|
see `#mixed-rate-stream-copy` above.
|
|
|
|
`_xfade_chain` accumulates `cum += dur[i] - td` and hands each boundary `offset=cum-td`. That offset is
|
|
an assertion about where input `i-1` still has frames. It fed on `_audio_dur`, which probes
|
|
`format=duration`, which is `max(video, audio)`. A rendered clip's audio outlasts its video by about a
|
|
frame. So every boundary pushed the accumulator further ahead of the picture.
|
|
|
|
Once the accumulated overshoot exceeds the transition width, the xfade window starts after the last frame
|
|
of input `i-1`. ffmpeg emits the transition and then **silently discards input `i` and every clip
|
|
downstream of it**. `rc 0`, no warning on stderr, output file present and playable. Measured on a group of
|
|
8 real clips, chain truncated at each stage:
|
|
|
|
```
|
|
k=7 out= 52.52 correct
|
|
k=8 out= 52.52 the last xfade contributed nothing
|
|
[v6][n7]xfade=duration=0.050:offset=52.500 <- [v6] is only 52.52s long, so 0.02s of margin
|
|
```
|
|
|
|
This is the whole cause of the shipped chapter being video 436.39s over audio 363.67s. It reproduces with
|
|
synthetic clips in about four minutes and needs no GPU.
|
|
|
|
Two changes hold it closed:
|
|
|
|
* `_assemble_once` probes `min(_stream_dur(p, "v"), _stream_dur(p, "a"))`. The minimum, because either
|
|
stream running long breaks a different half of the graph.
|
|
* `_xfade_chain` floors every input to a whole frame count. It applies `trim` and `atrim` to both streams,
|
|
so the accumulator tracks the real timeline instead of estimating it. Transition widths are quantized to
|
|
frames for the same reason.
|
|
|
|
Verified over the 49 real clips of chapter `7c944dd4`. The round that previously turned 359s of video into
|
|
100s now loses 0.85s. The chapter comes out video 358.76s against audio 358.76s, agreeing to the frame.
|
|
|
|
Forbidden from here: `_audio_dur` in anything that positions a filter. It is fine for "how long is this
|
|
clip roughly", nothing else.
|
|
|
|
## Assembly verifies its own output instead of trusting ffmpeg's exit code {#check-assembled}
|
|
|
|
**Closed, 2026-08-12.**
|
|
|
|
Both assembly branches drop stream time without failing. xfade discards inputs as above. The concat
|
|
demuxer with `-vsync cfr` drops video frames to force a constant rate. Neither is an error to ffmpeg.
|
|
|
|
So `_assemble_once` calls `_check_assembled(out, expect)` after every encode. It compares the output's
|
|
video stream against the predicted timeline and against its own audio stream. It raises when either is off
|
|
by more than `ASSEMBLE_TOL_S`, which is 0.5s. That tolerance covers frame boundaries and aac padding. A
|
|
dropped input is off by whole seconds.
|
|
|
|
Without this the failure stays invisible until somebody watches the video. That is how a 50MiB chapter
|
|
with 72.7s of silent picture reached the bucket while every stage counter read success.
|
|
|
|
## The single-item passthrough is not the bug {#passthrough-innocent}
|
|
|
|
**Void, 2026-08-12.** Cited in `HANDOFF.md` for 2026-08-11 as the prime suspect and must not be cited
|
|
again.
|
|
|
|
With 49 clips and `ASSEMBLE_BATCH=8`, round 0 makes six groups of 8 plus a leftover group of 1, which
|
|
`_assemble_batched` carries forward un-encoded. The theory was that mixing that raw clip with six encoded
|
|
intermediates broke round 1. The instrumented run disproves it: **two round-0 groups of 8 fresh clips
|
|
collapse on their own**, before any passthrough exists.
|
|
|
|
```
|
|
n=8 XFADE in v= 63.52 a= 63.60 -> out v= 52.52 a= 62.77 lost_v= +11.00
|
|
n=8 XFADE in v= 58.12 a= 58.20 -> out v= 12.04 a= 56.83 lost_v= +46.08
|
|
```
|
|
|
|
Round 1's 259s loss was the cascade. Its inputs already held 309s of video against 364s of audio, and
|
|
`durs` read the audio.
|
|
|
|
Consequence for the plan: collapsing `concat`, `xfade` and the passthrough into one path was the
|
|
recommended fix in `NEXT.md` and is **not needed**. Three paths are fine once each one positions filters
|
|
on a real timeline. The tree keeps the bounded memory it was built for.
|