Fix the real A/V gap: a stream copy across mixed frame rates

The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.

collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.

Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.

The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 12:16:22 +04:00
parent 1457556ce3
commit 8113bdfc8b
12 changed files with 329 additions and 285 deletions
+37
View File
@@ -269,3 +269,40 @@ back: the new check fires with `video=1.80 audio=3.56 expected=3.56`.
Not done: `s3://video/.../chapter.mp4` is still the broken 436s file. Rebuilding it means clearing the
`assemble` stage and resuming, which is CPU-only and was not run.
## 2026-08-12, the chapter rebuilt, and the bbox space settled
**The rebuild came out byte-identical to the broken file.** Clearing `assemble` and resuming produced
video 436.392031s over audio 363.674667s and `nb_frames` 9902 again, which proved the xfade fix committed
earlier today never runs for this chapter. With all-`cut` transitions `assemble` takes the `else` branch,
a `concat` demuxer with `-c copy`.
Reproduced that path offline in seconds and got the shipped numbers exactly. The cause is mixed frame
rates: 14 of the 49 clips are `r_frame_rate=30/1` at `time_base=1/15360`, the other 35 are `25/1` at
`1/12800`. `-c copy` writes the output in the first input's timebase, so those 14 play `15360/12800 = 1.2`
too long with their audio untouched. `collage_cmd` hardcoded `-r 30`, which yesterday's `FPS` sweep
missed. `decisions/chapter-assembly.md#mixed-rate-stream-copy`.
Fixed `collage_cmd` to emit `-r FPS`, and made `assemble` probe `r_frame_rate` across the clips and route
mixed rates through the re-encoding tree. Rebuilt:
```
before v=436.392 a=363.675 nb_frames=9902 avg_frame_rate=22.69
after v=364.120 a=364.122 nb_frames=9101 r=25/1
```
The 14 clips in the bucket are still 30fps. Assembly normalizes them, so the chapter is correct without
re-rendering, but the fast stream-copy path stays disabled for this chapter until `render` re-runs.
**The `bbox` space is 0-1000, not pixels.** Pulled all 113 detections from `/review/identity` and
measured: 47 boxes have `x2` past the 900px panel width, none has `y2` past 1000 on panels 1257 to 2307px
tall, 21 clamp at exactly 1000 in x, and the whole range is `[0, 1000]`. `/vision` now converts to pixels
before returning, so identity crops, gated face pairing, the set-of-mark boxes and the review UI all read
pixels (`decisions/identity-bbox.md#bbox-is-normalized`).
Checked by eye the way the user did. Drew the converted boxes on panel 7: five of six land on their
subject, including `person_5`, who is Seonho in the foreground with headphones and carried no identity.
`person_1` still frames an empty window mullion, which is the extra-versus-cast caveat, not this one.
Not done: `vision` and `identity` have not re-run, so every box, embedding and `ref_image_uris` in the
registry is still from the wrong space. That rerun is GPU work and was not started.