Fix the real A/V gap: a stream copy across mixed frame rates

The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.

collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.

Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.

The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 12:16:22 +04:00
parent 1457556ce3
commit 8113bdfc8b
12 changed files with 329 additions and 285 deletions
+18 -21
View File
@@ -4,44 +4,41 @@ Updated 2026-08-12. What this session did is in `HANDOFF.md`.
## State
The chapter runs end to end and the output is **not watchable**. That is now measured, not guessed.
Assembly is fixed and verified offline. The shipped `chapter.mp4` has not been rebuilt yet.
The chapter runs end to end. The A/V sync defect is fixed and `chapter.mp4` is rebuilt: video 364.120s
against audio 364.122s at `25/1`. The identity defects are still in the output.
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels,
`status=completed`, finished 2026-08-11T20:08:16Z. `s3://video/` holds 49 clips and a 50MiB
`chapter.mp4`. The user watched it and read out 19 defects. They are grouped by cause in `HANDOFF.md`.
Two numbers set the agenda:
One number sets the agenda:
- The shipped `chapter.mp4` is video 436.39s over audio 363.67s. Cause found and fixed, see below.
- Panel 7 checked against the art has **zero correct identity bindings** out of two, and the one
character who matters is unbound. `HANDOFF.md#panel-7-walked-against-the-art` has the table.
character who matters is unbound. `HANDOFF.md#panel-7-walked-against-the-art` has the table. The
coordinate cause is fixed. The registry built on it is not.
## Next
1. **Rebuild `chapter.mp4`.** Assembly is fixed in `worker_render.py`. Verified over the 49 real clips of
this chapter: video 358.76s against audio 358.76s, agreeing to the frame. The cause was `_xfade_chain`
taking offsets from `format=duration`, which is `max(video, audio)`. The accumulator drifted past the
end of its input, and ffmpeg silently discarded whole clips at `rc 0`
(`decisions/chapter-assembly.md#offsets-from-min-stream`). The single-item passthrough was innocent
and the one-path rewrite is not needed (`decisions/chapter-assembly.md#passthrough-innocent`).
1. **Re-run vision and identity.** The `bbox` space is settled and converted at `/vision`
(`decisions/identity-bbox.md#bbox-is-normalized`). Every stored box, embedding and `ref_image_uris` in
the registry came from the wrong space. The fix changes nothing until those stages run again.
This is GPU work and needs the user's go-ahead. Clear `vision` and everything downstream of it, or
accept that the boxes in the database stay normalized while new ones are pixels.
What is left is to clear the `assemble` stage and resume, then watch the result. That is CPU-only
ffmpeg, no GPU, but it needs the user's go-ahead.
Watch two things on the rerun. Whether `som_face` still returns `unknown` on every face, since gated
pairing was comparing pixel face boxes against 0-1000 character boxes. And whether `Choi Haeseon` still
absorbs every unnamed woman, which is item (b) below and independent of the crops.
Smaller follow-on: nine other `_audio_dur` calls in `worker_render.py` measure finished clips with
Smaller follow-on: nine `_audio_dur` calls in `worker_render.py` measure finished clips with
`format=duration`. So the durations reported to the orchestrator are blind to per-clip drift.
They position no filter, so invariant 9 does not cover them. Worth converting to `_stream_dur`.
2. **Fix identity, in this order.** Panel 7 is the worked example and
`HANDOFF.md#panel-7-walked-against-the-art` carries the evidence. Do not start at the registry.
a. **Settle the `bbox` coordinate space.** Consumed as pixels, all six boxes on panel 7 land in the
top third of the panel, two inside a speech balloon. Divided by 1000 they mostly land on their
subjects. `worker_vision.py:271` and `worker_identity.py:91` both assert pixels, and the art says
otherwise. Identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so today it
matches faces against crops of balloons and window frames. Nothing downstream can be judged until
this is right. Rescaling alone is **not** the fix: after scaling, one box still sits on an empty
window frame and another clips its subject.
a. ~~Settle the `bbox` coordinate space.~~ **Done 2026-08-12**, proven over all 113 detections and
checked by eye on panel 7, where five of six converted boxes land on their subject
(`decisions/identity-bbox.md#bbox-is-normalized`). `person_1` still frames an empty window mullion,
which is (c).
b. **Let identity abstain and stay abstained.** The colleague has no name in the story and was
labelled `Choi Haeseon` at 0.9. An unnamed recurring person needs a stable anonymous identity so
narration says "the colleague" every time. `match()` already returns `None` below threshold. Check