From a9d64fe80a3fb5dbef5424fa17ba1738171be2d6 Mon Sep 17 00:00:00 2001 From: kami Date: Wed, 12 Aug 2026 01:04:51 +0400 Subject: [PATCH] Record panel 7 against the art, and the A/V gap it hid Somebody watched chapter.mp4 for the first time. Two failures came out of it that no stage counter could see. chapter.mp4 is video 436.39s over audio 363.67s, so narration finishes 72.7s before the picture. The 49 clips are clean: all 25fps, video and audio agree to 0.03s, summing to 363.6s. A per-round probe puts the loss in the final round of _assemble_batched, which turns 359s of video into 100s while the audio survives. Round 0 is correct. Round 1 differs by holding a 7th input, the leftover clip that skips encoding, so the tree mixes concat output, xfade output and a raw clip. Not fixed. worker_render.py gains an FPS constant, fps normalization in the xfade branch to match concat, _stream_dur, and a self-check that compares video against audio instead of asserting the file is non-empty. That old check is how a 20% sync failure shipped. The fps inconsistency is real but not proven to be the shipped cause. Pinning -r on the output was tried and reverted: it drops frames to force CFR, which the concat branch comment already warned about. Panel 7 checked against the art has zero correct identity bindings out of two, and Seonho, the one character who matters, is unbound. bbox values are consumed as absolute pixels; on a 900x1650 panel that puts all six boxes in the top third, two inside a speech balloon. Identity therefore embeds crops of balloon edges and window frames, which is how confidence 0.9 lands on the wrong person. The colleague has no name in the story and was labelled Choi Haeseon; that row holds 25 of 26 assignments, so it is the label the pipeline stamps on any unnamed woman. Four caveats added. Two earlier claims are withdrawn in place: rescaling bbox by 1000 does not make the boxes correct, and the constraint is not 16 nameless rows needing names. Cast profiles already exist, since all 53 rows populate ref_image_uris and embedding_uri, but they are enrolled from the wrong crops. worker_render.py self-check passes. No pipeline ran. Co-Authored-By: Claude Opus 5 --- HANDOFF.md | 321 +++++++++++++++++++++++++++------ JOURNAL.md | 59 ++++++ NEXT.md | 157 +++++++++++----- caveats/CLAUDE.md | 4 + caveats/speaker-attribution.md | 79 ++++++++ worker_render.py | 57 +++++- 6 files changed, 563 insertions(+), 114 deletions(-) diff --git a/HANDOFF.md b/HANDOFF.md index c59bdd5..154a66e 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,88 +1,291 @@ -# HANDOFF, 2026-08-11 late session +# HANDOFF, 2026-08-12 Live state is in `NEXT.md`. This file is only what this session did. ## Asked -Continue from the previous handoff: fix speaker attribution, then rerun and re-read the named-speaker -share. Mid-session: what happens to unidentified people in a two-person panel, can a known character be -mis-named, and keep watch during the run. +Get up to speed from the previous handoff. Then: the user watched `chapter.mp4` for the first time and +read out 19 timestamped defects. Mid-session, what about characters. Then, write the handoff. ## Result -The named-speaker share is **9 of 95 speech lines, 9%**. The previous 30% counted fake tails. All 9 binds -are `Choi Haeseon`, the over-merged row. `script` passed 116/116 for the first time. +**Somebody finally watched the video.** That single act found more than four sessions of measuring did. +The recorded metrics said `script` 116/116 and "9 named speech lines". Both were true. Both measured the +wrong thing, because the 9 names are the *wrong* name. -**Identity, not attribution, is now the constraint.** 26 of 113 detected people carry an identity. 25 of -the 26 are that one row. This chapter caps near 23% named even with perfect balloon binding. +Two hard numbers came out of it. -Six defects, four found by measuring the run rather than by reading code. Each has a decision entry: +1. **The chapter is 20% out of sync.** `chapter.mp4` is video 436.39s over audio 363.67s. The narration + ends 72.7 seconds before the picture. The gap accumulates, which is why everything after 2:54 goes + sideways. The 49 clips are clean: video and audio agree to 0.03s and sum to 363.6s. Assembly adds + 72.7s of video and no audio. +2. **Identity binds names to the wrong people, and to people who have no name.** Walking panel 7 against + the art found zero correct bindings out of two, plus the one character who matters left unbound. See + `Panel 7, walked against the art`, which supersedes the earlier reading of this. -| # | defect | entry | +Nothing is committed. `worker_render.py` is edited in the working tree. The fix is **not** verified. + +## The user's 19 notes, grouped by cause + +| cause | timestamps | what is wrong | | --- | --- | --- | -| 1 | `tail` stamped on a model guess at confidence 1.0 | `decisions/speaker-attribution.md#no-fake-tail` | -| 2 | a two-word cast name always failed the script verifier | `#multiword-cast-names` | -| 3 | `/stage/clear dialogue` deleted nothing and reported success | `decisions/storage-layout.md#clear-vision-blob` | -| 4 | gemma's speaker answer echoed the prompt label, became a name | `#prompt-label-answers` | -| 5 | `som_face` stamped on a mark that paired to nobody | `#unpaired-mark` | -| 6 | one two-letter interjection halted the chapter at 112/116 | `#interjection-false-positive` | +| one row absorbed every identity | 0:20, 1:07, 1:51, 1:59, end | anyone identified comes out `Choi Haeseon` | +| the MC has no name | 0:44, 0:50, 1:02, 1:23, 1:45 | falls back to "the worker", "someone", "she" | +| gender read off the wrongly bound row | 1:45 "she admits", end "as he waves" | `Choi` is `f`, so "he" means a nameless `m` row got the line | +| narration invents facts | 0:43 "results", 2:03, 2:05, 2:15 "long shift" | the verifier checks quotes and names, not invented claims | +| vision reads art-within-art as scene | 1:35 chibi on a monitor as "a man holding a drink", 1:59 "pointing towards the screen" | panel-in-panel and screen content taken as reality | +| a beat carries nothing | 0:35-0:37 | no content worth narrating | +| no parallax, so a still holds | 2:24-2:52, 28s static | `layers` wrote nothing (`caveats/audit-open.md#layers-writes-nothing`) | +| transition quality | 2:52-2:54 slide "too sharp and laggy" | `push` is `slideleft` at 0.4s. Retest after the sync fix | +| A/V drift | everything after 2:54 | the 72.7s gap above | -## Changed +## Panel 7, walked against the art -Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`, commits `e8941d8 8071137 a965077` -plus docs: +This is the load-bearing finding of the session. The user pulled up the panel and checked every +detection by eye. **Read this before touching identity.** It contradicts what the earlier sessions +recorded, and it contradicts two theories I floated today before the user corrected them. -- `worker_vision.py`: `_annotate_speaker_methods`, `_apply_speaker_labels`, new `_present_keys` -- `decisions/speaker-attribution.md` (new, 5 sections), `decisions/storage-layout.md#clear-vision-blob`, - `caveats/speaker-attribution.md`, `caveats/audit-open.md#dishonest-clearing`, both indexes, - `JOURNAL.md`, `NEXT.md` +Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, crop `s3://panels/.../panels/p006.png`, 900x1650. +A wide establishing shot of an office seen through a window. Vision emitted 6 characters. -Orchestrator, `/mnt/server/home/kami/docker-apps`, commits `b18b6b4 603d388 db8d7c5 ccc3a8e` plus the -interjection commit: +| detection | vision said | the art shows | identity assigned | +| --- | --- | --- | --- | +| `person_5` | m, short black, **yellow sweater**, sitting | **Seonho**, foreground, yellow plaid, headphones, back to camera. The character who matters | **nothing** | +| `person_6` | f, short brown, **white shirt**, sitting | the **colleague**, green dress, ponytail. She has **no name** in the story | `Choi Haeseon` at **0.9** | +| `person_2` | m, short brown, green sweater, sitting | a background extra, seated beyond the next window pane | `Lim Seonho` at **0.9** | +| `person_1` | m, short black, suit, standing | **nobody. A window frame** | nothing | +| `person_3` | m, short black, blue sweater, standing | background extra | nothing | +| `person_4` | m, short black, grey sweater, standing | background extra | nothing | -- `correctness.py`: tokenized `allowed`, `_ID_SHAPED` guard in `normalize_speaker`, interjection - stopwords, short-quote grounding skip -- `db.py`: `_STAGE_VISION_KEYS` and the strip pass in `clear_stage_data` -- `test_script_verify.py`, `test_correctness.py`, `test_db.py`: one case each +**Zero of the two bindings are right, and the one character who matters got nothing.** Both wrong binds +carry confidence 0.9. + +Three separate defects stack here. + +**1. The stored bbox coordinate space is wrong.** Consumed as absolute pixels, all six boxes land in the +top third of a 1650px-tall panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against +the panel's own dimensions, `person_2`, `person_3`, `person_4` and `person_5` fit their subjects tightly. +So the numbers are not pixels. Two places assert that they are: + +- `worker_vision.py:271` prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)` +- `worker_identity.py:91` comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)` + +Everything reading `bbox` is therefore cropping the wrong region. `worker_identity.py:200` embeds +`_crop_bbox(img, ch["bbox"])`, so `Choi Haeseon` at 0.9 was matched on a crop of the speech balloon's +edge and `Lim Seonho` at 0.9 on a crop of empty window frame. Wrong crops are mostly blank white, which +embed alike, which is a plausible mechanism for one row absorbing 25 assignments. + +**Do not treat rescaling as the fix.** I claimed that and the user disproved it in one screenshot. +After scaling, `person_1` still sits on a window frame with nobody in it, and `person_6` is offset, +clipping the woman and running onto the dark frame. `worker_vision.py:38` already calls the box +"coarse, imprecise". Scaling buys roughly-right boxes, not right ones. + +**2. Vision has no concept of extra versus cast.** Four of the six detections are background extras or +nothing at all. They are handed to identity as candidates on equal footing with the two people who +carry the scene. That also means the "113 detected people" figure that framed the whole roadmap was +never the right denominator, so "26 of 113 carry an identity" measured nothing useful. + +**3. Identity mints a name onto a person who has none.** The colleague has no name in the story. She was +labelled `Choi Haeseon` at 0.9. Across the chapter `Choi Haeseon` holds 25 of 26 assignments, so in +practice that row is the label this pipeline stamps on any unnamed woman. This is the direct cause of +the user's 0:20 note, "Choi Haeseon when there's no Choi in the frame, it's the colleague", and of the +gender flips, since gender is read off whichever row got bound. + +This is a cousin of invariant 6 in `CLAUDE.md`, which forbids minting a character from an unparseable +model answer. The missing rule is the same shape: **never attach a name to a detection that carries no +name evidence.** An unnamed recurring person needs a stable anonymous identity so the narration can +call her "the colleague" every time, rather than being forced onto a named row. + +Partly checked, not finished: `match()` in `worker_identity.py:69` does abstain, returning `None` below +threshold, so the 0.9 came from cosine clearing the threshold on a garbage crop. Whether the Tier-2 +gemma resolver can answer "none of these" was not verified. Check that first. + +### Descriptions are not trustworthy either + +`person_6` is "white shirt" for a woman in a green dress. `person_5` is "yellow sweater" for yellow +plaid, which is close enough. Any downstream rule keyed on appearance text inherits this. + +## Registry, measured + +`characters` is keyed by `manga_id`, not chapter. Two manga share the table. `d7104032` has 34 rows with +9 named. This chapter's `ef105a86` has 19 rows with 3 named. The `Kei`, `Zen`, `Kanade`, `Rico` and `K3` +rows belong to the other manga, so they are not polluting this chapter. + +This manga's 19 rows, all `status=confirmed`, all `first_seen_panel=NULL`: + +- named: `Choi Haeseon` (f), `Seonho` (m, aliases `["Lim Seonho","Seonho"]`), `Lim Seonho` (m) +- 16 rows with `name=NULL` and an empty alias list + +`Seonho` and `Lim Seonho` are the same person in two rows. `Seonho`'s alias list contains the other +row's name. That is why either spelling matches two rows and binds nothing. + +Assignments across the whole table, for scale: + +| character | name | assignments | +| --- | --- | --- | +| `character_afa7623b` | Choi Haeseon | 25 | +| `character_bb79cfb4` | Kanade | 25 | +| `character_cfd34340` | Rico | 19 | +| `character_6f491712` | Lim Seonho | 1 | + +In this chapter only `Choi Haeseon` (25) and `Lim Seonho` (1) appear. + +Read those 25 together with panel 7. `Choi Haeseon` is not a character who appears 25 times. It is the +row that absorbs any unnamed woman. The 16 nameless rows are not a backlog of people waiting for names. +Some of them are background extras that should never have become rows, and at least one of them, the +colleague, is a real recurring person who correctly has no name and needs to keep it. + +An earlier draft of this file said "naming is the ceiling, 16 of 19 rows need names". **That was wrong** +and it is corrected here. The ceiling is that identity cannot say "person, no name" and cannot tell an +extra from cast. + +## The A/V bug, located but not fixed + +Reproduced on the 49 real clips with `repro.py` and `probe.py` (see Open). Per-round probe with +`ASSEMBLE_BATCH=8` and 6 `fade_black` boundaries spread across batches: + +``` +r0 g0 n=8 XFADE in v= 49.44 a= 49.44 -> out v= 48.56 a= 48.64 lost_v=+0.88 lost_a=+0.80 +r0 g1 n=8 XFADE in v= 47.04 a= 47.04 -> out v= 46.16 a= 46.23 lost_v=+0.88 lost_a=+0.81 +r0 g2 n=8 XFADE in v= 48.00 a= 48.00 -> out v= 47.12 a= 47.19 lost_v=+0.88 lost_a=+0.81 +r0 g3 n=8 XFADE in v=100.80 a=100.80 -> out v= 99.92 a= 99.96 lost_v=+0.88 lost_a=+0.84 +r0 g4 n=8 concat in v= 64.18 a= 64.16 -> out v= 64.24 a= 64.26 lost_v=-0.06 lost_a=-0.10 +r0 g5 n=8 XFADE in v= 44.00 a= 44.00 -> out v= 43.12 a= 43.17 lost_v=+0.88 lost_a=+0.83 +r1 g0 n=7 XFADE in v=359.29 a=359.60 -> out v= 99.96 a=358.79 lost_v=+259.33 lost_a=+0.81 +``` + +Round 0 is correct. Each group loses only the xfade overlap. **Round 1 loses 259s of video against 0.8s +of audio.** Its output video is 99.96s at 2499 frames, almost exactly the frame count of input `n3` on +its own, the 99.92s intermediate. The final video appears to carry the frames of one input. + +The round-1 filtergraph is arithmetically correct, so this is ffmpeg behaviour, not offset math: + +``` +[n0][n1]xfade=transition=fade:duration=0.050:offset=48.510[v1] +[v1][n2]xfade=transition=fade:duration=0.050:offset=94.620[v2] +[v2][n3]xfade=transition=fadeblack:duration=0.600:offset=141.140[v3] +[v3][n4]xfade=transition=fade:duration=0.050:offset=241.010[v4] +[v4][n5]xfade=transition=fade:duration=0.050:offset=305.200[v5] +[v5][n6]xfade=transition=fade:duration=0.050:offset=348.270[v6] +``` + +### The strongest clue, found last + +Re-running that chain by hand over only the **6** round-0 intermediates gives a correct 348.24s at 8708 +frames, `rc 0`, no warnings. Round 1 collapses with **7** inputs, not 6. + +The 7th input is the leftover 49th clip. With 49 clips and batch 8, round 0 makes 6 groups of 8 and one +group of 1, and `_assemble_batched` passes a lone group through un-encoded: + +```python +if len(group) == 1 and not final_round: + next_items.append(group[0]) +``` + +So the final xfade mixes 6 encoded intermediates with 1 raw clip. That passthrough is a **third** path +next to `concat` and `xfade`, and it is the prime suspect. Start here tomorrow. Confirm it by running +`probe.py` with 48 clips instead of 49, which removes the leftover entirely. + +### What was tried and what it cost + +Two paths exist in `_assemble_once`. A `concat` branch handles cut-only batches. An `xfade` branch +handles batches with a real transition. They disagreed on frame rate. The concat branch forced `fps=30` +while the xfade branch normalized nothing, and clips are 25fps. `436.39 / 363.63 = 1.2001`, exactly +`30/25`, which is what sent me down this path. + +Working-tree changes to `worker_render.py`, all uncommitted: + +- new `FPS = 25` constant. The three hardcoded `25`s and the one `30` now reference it +- the xfade branch normalizes every input with `setsar=1,fps={FPS}` into `[n{i}]` labels, matching what + the concat branch already did +- new `_stream_dur(path, kind)`. `_audio_dur` probes `format=duration`, which is `max(video, audio)`, so + it hides A/V drift by construction +- the `__main__` xfade self-check now assembles 4 clips through `_assemble_batched` with + `ASSEMBLE_BATCH=2` and asserts `abs(video - audio) < 0.25`. It previously asserted only + `getsize(out) > 0`, which is why this shipped + +**A dead end worth not repeating.** I also pinned `-r FPS` on both output encodes. That made it worse. +The chapter collapsed to exactly 100.00s at 2500 frames, because forcing CFR on irregular input +timestamps drops frames. The comment already sitting at the concat branch warns about this. Both `-r` +flags were removed again. The in-graph `fps=` filter is the right normalization. The output `-r` is not. + +The fps inconsistency is real and worth keeping fixed. It is **not** proven to be the cause of the +shipped 72.7s gap. The scene graphs hold 356 `cut` against 6 `fade_black`, so round 2 of the real run +most likely stayed on the concat branch, where no mixing occurs. Treat the fps work as necessary and +insufficient. + +### The user's own hypothesis, which is the recommended direction + +Two guesses, both worth following: + +1. the assembly is wrong in an ffmpeg sense +2. there are two different paths, and they get mixed when there should only be one + +Guess 2 matches the code. `concat`, `xfade` and the single-item passthrough are three paths, and +`_assemble_batched` feeds the output of one into the input of another. **Collapse it to one path.** +Normalize every input, then xfade every boundary, with `cut` as a 0.05s fade. `acrossfade` shortens +audio by the same amount that xfade shortens video, so A/V stays locked. The chapter then comes out +about 2.4s shorter than the sum of the clips, with both streams agreeing. That removes the branch +interaction instead of tuning it. ## Measured -Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels. +Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`. -- 113 orchestrator tests pass. `worker_vision.py` self-check passes. -- `dialogue` 116/116, `direct` 116/116, `scene` 116/116, `script` 116/116. -- speech lines 95, named 9. Multi-character panels 0 of 40 by design. Single-character 9 of 55. -- `speaker_method`: `unknown` 47, `model_solo` 28, `solo_prior` 8, `som_face` 7, `turn_taking` 5. -- unresolved name refs 3, all `brown ponytail, green dress`, from a neighbouring panel in the same - 8-panel window. Was 24 of 51 before the fix. -- identity assignments 26, of which `character_afa762` "Choi Haeseon" holds 25. -- registry duplicates that block a correct bind: `seonho` matches 2 rows, `lim seonho` matches 2 rows. -- buckets during `tts`: `panels` 116, `raw` 79, `manga` 491, `audio` 1, `layers` 0, `video` 0. +- job `status=completed`, every stage at its unit count, finished `2026-08-11T20:08:16Z` +- it still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`) +- `chapter.mp4` 50MiB, video 436.392s, audio 363.675s, `r_frame_rate=25/1`, + `avg_frame_rate=63372800/2792909` which is 22.69, `nb_frames=9902` +- 49 clips, every one `25/1` exactly, sum video 363.63s, sum audio 363.60s. No clip has the two + differing by more than 0.05s +- scene-graph transitions: `cut` 356, `fade_black` 6 +- `worker_render.py` `__main__` self-check passes on the current working tree +- workers up in tmux `manga-workers`, 9 windows ## Open -- **The run finished.** Every stage completed at 2026-08-11T20:08:16Z: `tts` 116/116, `layers` 116/116, - `render` 116/116, `assemble` 1/1. `s3://video/` holds 49 clips and a 50MiB `chapter.mp4` under - `ef105a86-.../7c944dd4-.../`, `s3://audio/` 49 objects at 32MiB. Nobody has watched the video. - - Two honesty defects at the finish, recorded not fixed. `layers` reported `completed 116/116` with an - empty bucket, so no clip has parallax (`caveats/audit-open.md#layers-writes-nothing`). The completed - job still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`). - - Read the state, or clear a stage and resume: +- **Finish the round-1 diagnosis.** The `_assemble_batched` tree turns 359s of video into 100s. This is + the worst defect found and it reproduces offline in about two minutes with no GPU. Suspect the 7th + passthrough input first. ```bash - /usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"\"}'" - /usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" + .venv/bin/python /probe.py # the per-round loss table above + .venv/bin/python /repro.py # end-to-end verdict ``` - Note: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`. + The scratchpad is session-scoped and will be gone. **Re-download the clips first:** -- Defects 5 and 6 landed after `dialogue` had already run, so this run's 7 `som_face` lines are still - labelled from the unpaired-mark path. The next dialogue pass fixes that. No name was affected. -- The audit script is now `audit_speakers.py` in this repo. It reads `/data/manga.db`, so it runs inside - the container: `docker cp audit_speakers.py manga-orchestrator:/tmp/ && docker exec - manga-orchestrator python3 /tmp/audit_speakers.py`. -- The 3 cross-panel unresolved refs need the window's whole present-list, not one panel's. -- Workers were restarted twice this session and are running in tmux `manga-workers`. Nothing watches - them, and nothing watches the homesrv containers. + ```bash + /usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/ + ``` + + Both scripts are worth committing next session. They are the only check that has ever caught this. + +- **Do not trust `format=duration`.** It returns `max(video, audio)`, so every existing duration assert + in `worker_render.py` is blind to drift. `_stream_dur` exists now. The other asserts still use + `_audio_dur`. +- **Identity, in the order the panel 7 evidence implies.** First, settle the `bbox` coordinate space and + fix every consumer, since nothing else can be judged while crops are wrong. Second, let identity + abstain and hold a stable anonymous identity, so the colleague stays "the colleague". Third, separate + extra from cast so extras never reach identity. Only then merge `Seonho` into `Lim Seonho` and split + `character_afa7623b`, which still needs the reversible-merge design + (`caveats/audit-open.md#destructive-reconcile`). +- **Verify the bbox space before changing anything.** Two independent claims in the code say pixels, and + the art says otherwise. Confirm what the model was told and what it returns, rather than trusting + either comment. Then check whether the crop is the only consumer, or whether SoM marker placement and + the face-pairing in `worker_vision.py:57` read the same numbers. +- Re-derive the panel 7 overlay when needed. It took one `mc cat` of the crop plus a Pillow script, and + it found more than any query did: + + ```bash + /usr/bin/ssh kami@192.168.1.104 'mc cat homesrv/panels/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/panels/p006.png' > p007.png + ``` + + Draw each `bbox` twice, once as pixels and once divided by 1000, then look at it. +- `mc` aliases on homesrv: use `homesrv` or `mio`, not `local`, which returns Access Denied. `rfs` is + the empty rustfs. +- Plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`. +- `cp` is aliased to `cp -i` in this shell and hangs on overwrite. Use `/usr/bin/cp -f`. +- The Bash tool's default timeout is 120s no matter what `timeout` the command itself carries. Pass the + tool's own timeout or background the run. Otherwise a restore step after a mutation test never runs, + which left a deliberately broken `worker_render.py` on disk once this session. diff --git a/JOURNAL.md b/JOURNAL.md index 60dbba9..0f380a1 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -169,3 +169,62 @@ layers (`decisions/storage-layout.md#bucket-per-artifact`). Two honesty defects surfaced at the finish, both recorded rather than fixed. `layers` reported `completed 116/116` with an empty bucket, and the completed job still carries `error: "partial: 112/116 completed"` from the failure three resumes earlier. + +## 2026-08-12 — the video got watched + +No pipeline ran. The user watched `chapter.mp4` for the first time and read out 19 timestamped defects. +That found more than the previous four sessions of measuring, because the recorded metrics were all +measuring whether code ran rather than whether the result was right. + +Two measurements came out of it. First, `chapter.mp4` is video 436.39s over audio 363.67s, so the +narration finishes 72.7s before the picture and the gap accumulates. The 49 clips are clean: every one is +25fps exactly, video and audio agree to 0.03s, and they sum to 363.6s. Assembly adds 72.7s of video and +no audio. Second, this manga holds 19 character rows of which 3 carry a name, and `Choi Haeseon` holds 25 +of the chapter's 26 identity assignments. That is why the video calls the colleague Choi, never names the +MC, and flips gender. + +The A/V bug was narrowed with a per-round probe over the 49 real clips. Round 0 of `_assemble_batched` is +correct, losing only the xfade overlap per group. Round 1 turns 359s of video into 100s while the audio +survives at 358.79s. The round-1 filtergraph is arithmetically correct, and re-running the same chain by +hand over only the 6 encoded intermediates gives a correct 348.24s with no warnings. Round 1 differs by +holding a 7th input: the leftover 49th clip, which `_assemble_batched` passes through un-encoded. That +passthrough is a third path beside `concat` and `xfade` and is the prime suspect. + +`worker_render.py` gained an `FPS = 25` constant, fps normalization in the xfade branch to match the +concat branch, a `_stream_dur` helper, and a self-check that compares video against audio rather than +asserting the file is non-empty. The old check only asserted `getsize(out) > 0`, which is how a 20% sync +failure shipped. Pinning `-r FPS` on the output encodes was tried and reverted: it collapsed the chapter +to exactly 100.00s by dropping frames to force CFR, which the comment at the concat branch already +warned about. None of it is committed and none of it fixes the chapter yet. + +The fps inconsistency between the two branches is real but not proven to be the shipped cause. The scene +graphs hold 356 `cut` against 6 `fade_black`, so the real run's final round most likely stayed on the +concat branch where no mixing happens. + +### Panel 7, checked against the art + +The same day, the user pulled up panel 7 and checked every detection by eye. It overturned the framing +this file carried an hour earlier, and it overturned two theories I proposed before being corrected. + +Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, a wide establishing shot of an office through a +window, crop 900x1650. Vision emitted 6 characters. Zero of the two identity bindings are correct and the +one character who matters is unbound. `person_5`, described as "yellow sweater", is Seonho in the +foreground and got no identity. `person_6` is the colleague, who has no name in the story, and was +assigned `Choi Haeseon` at 0.9. `person_2` is a background extra and was assigned `Lim Seonho` at 0.9. +`person_1` is a window frame with nobody in it. `person_3` and `person_4` are background extras. + +Three defects stack, recorded as `caveats/speaker-attribution.md#bbox-wrong-space`, +`#no-anonymous-identity` and `#extras-as-cast`. The `bbox` values are consumed as absolute pixels, and on +this panel that puts all six boxes in the top third with two inside a speech balloon. Divided by 1000 +four of the six fit tightly. Identity therefore embedded crops of balloon edges and window frames, which +is how a 0.9 confidence lands on the wrong person. Blank crops embed alike, a plausible mechanism for one +row absorbing 25 of 26 assignments. + +Two claims I made and had to withdraw. First, that rescaling by 1000 makes the boxes correct: after +scaling, `person_1` still sits on an empty window frame and `person_6` clips its subject, and the +descriptions are unreliable anyway, since `person_6` reads "white shirt" for a green dress. Second, that +the constraint is 16 nameless rows needing names. The opposite is true. The pipeline mints names onto +people who have none, and at least one nameless row is a real recurring person who should stay nameless. + +The "26 of 113 detected people carry an identity" figure that framed the roadmap counted mostly +background extras. It should not be quoted again. diff --git a/NEXT.md b/NEXT.md index 5daa5a3..4ef352b 100644 --- a/NEXT.md +++ b/NEXT.md @@ -1,85 +1,148 @@ # NEXT -Updated 2026-08-11. Replaces the old `HANDOFF.md`. +Updated 2026-08-12. What this session did is in `HANDOFF.md`. ## State -Audit Phase 1 is implemented and green. Nothing is half-finished. +The chapter runs end to end and the output is **not watchable**. That is now measured, not guessed. -Changed on workpc: `worker_scene.py`, `worker_script.py`, `worker_vision.py`, `session_manager.py`. -Changed on homesrv (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`): `tracklets.py`, -`correctness.py`, `db.py`, `service.py`, `session_proxy.py`, `test_script_verify.py`, -`test_name_binding.py`. +Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels, +`status=completed`, finished 2026-08-11T20:08:16Z. `s3://video/` holds 49 clips and a 50MiB +`chapter.mp4`. The user watched it and read out 19 defects. They are grouped by cause in `HANDOFF.md`. -What landed and why: `decisions/audit-phase1.md`. What was left open: `caveats/audit-open.md`. +Two numbers set the agenda: -Verification: CPU-only self-checks and the orchestrator test suite. 108 orchestrator tests pass -(`test_api.py` is excluded on workpc because fastapi is not installed in this venv). No GPU work ran -and no pipeline ran, so none of this is confirmed against a real chapter. +- `chapter.mp4` is video 436.39s over audio 363.67s. The narration finishes 72.7s before the picture. +- Panel 7 checked against the art has **zero correct identity bindings** out of two, and the one + character who matters is unbound. `HANDOFF.md#panel-7-walked-against-the-art` has the table. -The orchestrator half is committed as `1c60710` in `/mnt/server/home/kami/docker-apps` and the -container is rebuilt and serving. `94bd4d8` in the same repo repins minio to its amd64 digest, which -the rebuild exposed. +Uncommitted work sits in the tree: `worker_render.py` has an `FPS = 25` constant, fps normalization in +the xfade branch, a new `_stream_dur`, and a self-check that compares video against audio. The +self-check passes. It does **not** yet fix the chapter. Details and one dead end in `HANDOFF.md`. ## Next -The named-speaker share is 9%, 9 of 95 speech lines, and that number is real. See `JOURNAL.md` for the -six defects behind the old 30%. Everything below is measured on job `778297bc`, not inferred. +1. **Fix chapter assembly.** `_assemble_batched` turns 359s of video into 100s while the audio survives. + It reproduces offline in two minutes, no GPU. Round 0 is correct and round 1 collapses. Round 1 is + the only round holding a raw clip that skipped encoding, so suspect the single-item passthrough + first. The recommended shape is one path, not three: normalize every input, then xfade every + boundary, treating `cut` as a 0.05s fade. `acrossfade` and `xfade` shorten audio and video equally, + so the streams stay locked. `HANDOFF.md` holds the per-round table, the filtergraph, and the repro + commands. Nothing downstream is worth judging until this lands. +2. **Fix identity, in this order.** Panel 7 is the worked example and + `HANDOFF.md#panel-7-walked-against-the-art` carries the evidence. Do not start at the registry. -That job now runs end to end: `chapter.mp4`, 50MiB, 49 clips. Nobody has watched it. Two stages lie about -it (`caveats/audit-open.md#layers-writes-nothing`, `#stale-job-error`). + a. **Settle the `bbox` coordinate space.** Consumed as pixels, all six boxes on panel 7 land in the + top third of the panel, two inside a speech balloon. Divided by 1000 they mostly land on their + subjects. `worker_vision.py:271` and `worker_identity.py:91` both assert pixels, and the art says + otherwise. Identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so today it + matches faces against crops of balloons and window frames. Nothing downstream can be judged until + this is right. Rescaling alone is **not** the fix: after scaling, one box still sits on an empty + window frame and another clips its subject. + b. **Let identity abstain and stay abstained.** The colleague has no name in the story and was + labelled `Choi Haeseon` at 0.9. An unnamed recurring person needs a stable anonymous identity so + narration says "the colleague" every time. `match()` already returns `None` below threshold. Check + whether the Tier-2 gemma resolver can answer "none of these"; that was not verified. + c. **Separate extra from cast.** Four of the six detections on panel 7 are background extras or + nothing at all, and all six reach identity as equal candidates. + d. Only then merge `Seonho` into `Lim Seonho` and split `character_afa7623b`, which still needs the + reversible-merge design (`caveats/audit-open.md#destructive-reconcile`), not a patch. -**Identity is the constraint now, not attribution.** 26 of 113 detected people carry an identity. 25 of -those 26 are the single over-merged row (`caveats/speaker-attribution.md#identity-over-merge`). That caps -this chapter near 23% named even with perfect balloon binding. Work identity before geometry. + **Cast profiles already exist. Do not rebuild them.** The user asked whether the main cast could get a + profile built from reference frames and reused. `characters` already carries `ref_image_uris` and + `embedding_uri`, and all 53 rows have both populated. The mechanism is not missing, it is enrolled + from the wrong crops, so today it stores references to balloon edges and window frames. Step (a) is + what makes it work. Three things are genuinely absent and are the smaller follow-on: -1. Split the over-merged character row and dedupe the registry. `Lim Seonho` and `Seonho` are separate - rows with overlapping aliases, so either name matches two rows and binds nothing. Needs the - reversible-merge design (`caveats/audit-open.md#destructive-reconcile`) rather than a patch. -2. Bind a balloon to a speaker by tail geometry, using the unused `det`/`seg` heads - (`caveats/speaker-attribution.md#tail-is-not-geometry`). Until then multi-character panels have no - speaker at all, which is 40 of 95 lines here. -3. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3 - unresolved refs are a description belonging to a neighbouring panel in the same 8-panel call. -4. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work + - no quality gate on enrollment, so nothing checks that a reference crop holds a face at all + - nothing re-enrolls a reference set once it is written, so the wrong crops persist + + **The visual "is this them?" check is already built. Do not write it again.** `/vision/resolve` at + `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images per candidate. + `build_resolve_prompt` tells the model to judge face shape first, to treat hair and outfit as + secondary, that two people sharing a hair colour are not the same, and to answer `0` for NONE when + unsure. `choice: 0` becomes a new character, an out-of-range index becomes `unresolved`, and + `ref_image_uris` is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`. The + mechanism, the prompt and the abstain path are all correct. They are fed crops of the wrong region, + which is step (a). + - no human gate to name, merge or split the clusters. The user wants this as a minor adjustment on + top, not as the mechanism. The `gates` table and the review gates from [#136] are the place to hang + it + + The chibi at 1:35 will survive all of this. He genuinely is brown hair plus a yellow shirt, so a + profile match is correct on appearance and wrong on reality. That needs item 4 below, plus requiring + a real face before a crop can enroll. +3. **Stop the narration inventing facts.** 0:43, 2:03, 2:05 and 2:15 assert things no panel shows. The + correctness verifier passed 116/116 because it checks quotes and names, never invented claims. +4. **Teach vision that art inside a panel is not the scene.** A chibi on a monitor became "a man holding + a drink" at 1:35. A colleague pointing into the distance became "pointing towards the screen" at + 1:59. +5. **`layers` writes nothing** and reports `completed 116/116`, so no clip has parallax and a still + holds for 28s from 2:24 (`caveats/audit-open.md#layers-writes-nothing`). +6. **Clear the stale job error.** The completed job still carries `error: "partial: 112/116 completed"` + (`caveats/audit-open.md#stale-job-error`). +7. Balloon-to-speaker geometry via the unused `det`/`seg` heads + (`caveats/speaker-attribution.md#tail-is-not-geometry`) is now behind item 2. With no name to attach, + geometry buys nothing. +8. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3 + unresolved refs describe a neighbouring panel in the same 8-panel call. +9. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work (`caveats/audit-open.md#sqlite-locking`). -## The 2026-08-11 chapter run +## Lesson worth keeping -Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 -panels. It reached `scene` 116/116 and then failed in `script` at 87/116 on the verifier bug fixed -above. `tts layers render assemble` never ran. Read where it got to, clear the failed stage, resume: +Every metric recorded before this session said the pipeline was fine or nearly fine. `script` 116/116, +"9 named speech lines", `layers` 116/116, `assemble` 1/1. Watching two and a half minutes of output +found a 20% sync failure, a cast that is 84% anonymous, invented narration, and a stage that writes +nothing while reporting success. Stage counters measure whether code ran. They say nothing about whether +the result is correct. Watch the output before trusting a number. + +## Running the pieces ```bash -ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" -ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"script\"}'" -ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +./start_workers.sh # session_manager + 9 workers, each a uvicorn in a tmux window +tmux attach -t manga-workers # per-worker logs +.venv/bin/python worker_render.py # self-check, runs real ffmpeg, about 4 minutes ``` -That resume narrates the attributions the old `tail` label produced. Clearing back to `dialogue` -instead re-runs the GPU stages and produces the honest metric. +Read the state, or clear a stage and resume: -Numbers and the quality read are in `JOURNAL.md` and `caveats/speaker-attribution.md`. +```bash +/usr/bin/ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"\"}'" +/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +``` + +Traps: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin, so use `/usr/bin/ssh`. +`mc` aliases on homesrv are `homesrv` and `mio`. `local` returns Access Denied and `rfs` is the empty +rustfs. `cp` is aliased to `cp -i` and hangs on overwrite, so use `/usr/bin/cp -f`. + +Re-fixing assembly needs the real clips, which the session scratchpad no longer holds: + +```bash +/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/ +``` ## Storage and viewer, tasks #116/#117 -[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had -been dead since 2026-07-19 on an arm64 digest pin. Details in `JOURNAL.md`. +[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had been +dead since 2026-07-19 on an arm64 digest pin. -[#116] is closer but not cut over. Artifacts now split one bucket per class +[#116] is closer but not cut over. Artifacts split one bucket per class (`decisions/storage-layout.md#bucket-per-artifact`), and both MinIO and `rustfs` hold all six buckets. `rustfs` on `127.0.0.1:9010/9011` is still empty and nothing is repointed, so MinIO serves every read and write. Remaining: `mc mirror` the live buckets, verify counts and sizes, then decide on cutover (`decisions/storage-layout.md#rustfs-staged`). -Two containers on homesrv had been dead for two weeks and are now running. `manga-fetch` is the one -`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them. +Two containers on homesrv had been dead for two weeks and now run. `manga-fetch` is the one +`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them, and +nothing watches the workers. ## Open questions -Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for. They are the -speaker contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only -[#203] existed and is now closed by `decisions/audit-phase1.md#unlocked-model-load`. +Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for: the speaker +contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only [#203] +existed and is closed by `decisions/audit-phase1.md#unlocked-model-load`. Three audit items are deliberately not done and are recorded as caveats rather than silently dropped: honest stage clearing, ComfyUI under the session mutex, and reversible identity merges. Each needs a diff --git a/caveats/CLAUDE.md b/caveats/CLAUDE.md index cc8beac..6836615 100644 --- a/caveats/CLAUDE.md +++ b/caveats/CLAUDE.md @@ -38,3 +38,7 @@ a complaint, so give it one or drop it. | [One invented word still halts the chapter](speaker-attribution.md#one-word-halts-chapter) | 2026-08-11 run | | [A completed job keeps the error from an earlier failure](audit-open.md#stale-job-error) | 2026-08-11 run | | [`layers` reports success on an empty bucket](audit-open.md#layers-writes-nothing) | 2026-08-11 run | +| [Every `bbox` is read in the wrong coordinate space](speaker-attribution.md#bbox-wrong-space) | 2026-08-12 panel 7 | +| [Identity cannot say "a person with no name"](speaker-attribution.md#no-anonymous-identity) | 2026-08-12 panel 7 | +| [Vision does not separate a background extra from cast](speaker-attribution.md#extras-as-cast) | 2026-08-12 panel 7 | +| [Cast reference profiles are enrolled from wrong crops](speaker-attribution.md#poisoned-reference-set) | 2026-08-12 panel 7 | diff --git a/caveats/speaker-attribution.md b/caveats/speaker-attribution.md index 3a6ff10..8709925 100644 --- a/caveats/speaker-attribution.md +++ b/caveats/speaker-attribution.md @@ -69,3 +69,82 @@ stopped. **Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer is to flag the beat for review and continue, which is `#136` gate work, not a verifier change. + +## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space} + +Vision's `bbox` values are stored and consumed as absolute pixels. On panel +`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of +the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own +dimensions, four of the six fit their subjects tightly. + +Two places in the code assert pixels, and the art contradicts both: + +- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)` +- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)` + +Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces +against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9 +from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed +alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at +`worker_vision.py:57` reads the same numbers and was not checked. + +Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with +nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already +calls the box "coarse, imprecise". + +Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can +be judged while the crops are wrong. + +## Identity cannot say "a person with no name" {#no-anonymous-identity} + +The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9. +Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline +stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which +is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing +line. + +This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an +unparseable model answer. The missing rule: never attach a name to a detection that carries no name +evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the +colleague" every time. + +`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from +cosine clearing the threshold on a wrong crop. Whether the Tier-2 gemma resolver can answer "none of +these" was not verified. + +Revisit trigger: immediately after the `bbox` space is settled. + +## Vision does not separate a background extra from cast {#extras-as-cast} + +Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground +and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame +with nobody in it. All six reach identity as equal candidates. + +Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly +extras, so it measured nothing useful and should not be quoted again. + +Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return. + +## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set} + +`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the +cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are +crops of balloon edges, window frames and background extras rather than of faces. + +The visual comparison people reach for as the fix is **already implemented**, so do not build it again. +`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images +per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair +and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for +NONE when unsure. `choice: 0` becomes a new character and an out-of-range index becomes `unresolved`. The +`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`, +so the references reach the model. + +That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon +edge against references enrolled from window frames and background extras, then sometimes answers "same". +Nothing gates enrollment on the crop holding a face. + +Who pays: every later match, because the reference set defines what a character looks like. Fixing +`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place. + +Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the +existing `ref_image_uris` and `embedding_uri` values as invalid. diff --git a/worker_render.py b/worker_render.py index 6ddaf82..4a15687 100644 --- a/worker_render.py +++ b/worker_render.py @@ -116,8 +116,25 @@ def _audio_dur(path: str) -> float: return 0.0 +def _stream_dur(path: str, kind: str) -> float: + """duration of one stream. `format=duration` is max(video,audio) and so hides A/V drift.""" + r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", f"{kind}:0", + "-show_entries", "stream=duration", "-of", "default=nk=1:nw=1", path], + capture_output=True, text=True) + try: + return float(r.stdout.strip()) + except ValueError: + return 0.0 + + ZMAX, ZPAN = 1.15, 1.18 # ken-burns zoom ceiling; constant zoom that gives pans room to travel +# Every clip and every assembly stage MUST agree on this. xfade does not resample: it reinterprets the +# second input's frames at the first input's rate, so a 30fps input joined onto a 25fps one plays 1.2x +# too slow with the audio untouched -- the video ends minutes long and the narration runs ahead of the +# picture. That is exactly what a 25fps clip pipeline plus a `fps=30` concat branch produced. +FPS = 25 + def _motion(camera: dict, frames: int) -> str: """#8 content-aware motion: map the vision `camera` block to a zoompan z/x/y expression. @@ -155,7 +172,7 @@ def _motion(camera: dict, frames: int) -> str: x, y = f"{xc}+(iw*0.03)*sin(6.283*on/{T})", f"{yc}+(ih*0.03)*cos(6.283*on/{T})" else: # zoom_in (default ken burns) z, x, y = f"1+{ZMAX-1:.3f}*on/{T}", xc, yc - return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps=25" + return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps={FPS}" def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict = None, @@ -163,8 +180,7 @@ def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict """ffmpeg: still panel over a blurred fill of itself + content-aware motion, burned subs, 9:16. #5 blurred bg replaces black bars: one copy scaled to COVER + blurred, the fitted panel on top. #13 pad seconds of trailing silence (last frame held) give the panel a beat before the next.""" - fps = 25 - frames = max(1, int((dur + pad) * fps)) # hold the last frame through the pad + frames = max(1, int((dur + pad) * FPS)) # hold the last frame through the pad # overlay's W/H/w/h are ffmpeg's main/overlay dims -- kept literal (no f-string braces). fc = ( f"[0:v]split=2[bg][fg];" @@ -483,7 +499,7 @@ def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list burned subtitle (in `ass`) spans the whole beat. 185: per-image screen-time is content-weighted (see _beat_slices), equal split when weights are absent. pure -> testable without S3.""" cameras = cameras or [] - n, fps = len(imgs), 25 + n, fps = len(imgs), FPS slices = _beat_slices(D, n, weights) # 185: content-weighted, equal-split fallback # one frame per image (no -loop): zoompan d=frames expands that single frame to exactly `frames` # output frames = seg seconds. looping instead would feed many frames and zoompan multiplies each. @@ -675,12 +691,15 @@ def _xfade_chain(durs: list, trans: list): # accumulator would run BACKWARDS (cum += dur - td), swallowing every later clip into a frozen # overlap near the middle. Floor to a small positive length so the timeline stays monotonic. durs = [d if (d and d > 0.1) else 0.1 for d in durs] - parts, vlast, alast, cum = [], "[0:v]", "[0:a]", durs[0] + # Normalize every input to FPS/SAR before it reaches xfade, exactly as the concat branch does. Both + # branches feed the same tree, so an un-normalized xfade input is what stretched the chapter 1.2x. + parts = [f"[{i}:v]setsar=1,fps={FPS}[n{i}]" for i in range(len(durs))] + vlast, alast, cum = "[n0]", "[0:a]", durs[0] for i in range(1, len(durs)): name, td = XFADE.get(trans[i - 1] if i - 1 < len(trans) else "cut", XFADE["cut"]) td = max(0.05, min(td, durs[i - 1] - 0.05, durs[i] - 0.05)) # overlap fits in both clips off = max(cum - td, 0) - parts.append(f"{vlast}[{i}:v]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]") + parts.append(f"{vlast}[n{i}]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]") parts.append(f"{alast}[{i}:a]acrossfade=d={td:.3f}[a{i}]") vlast, alast, cum = f"[v{i}]", f"[a{i}]", cum + durs[i] - td return ";".join(parts), vlast, alast @@ -713,7 +732,7 @@ def _assemble_once(inputs: list[str], trans: list[str], out: str): cmd = ["ffmpeg", "-y"] for p in inputs: cmd += ["-i", p] - pre = "".join(f"[{i}:v]setsar=1,fps=30[v{i}];" for i in range(n)) + pre = "".join(f"[{i}:v]setsar=1,fps={FPS}[v{i}];" for i in range(n)) fg = pre + "".join(f"[v{i}][{i}:a]" for i in range(n)) + f"concat=n={n}:v=1:a=1[v][a]" cmd += ["-filter_complex", fg, "-map", "[v]", "-map", "[a]", "-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p", @@ -945,7 +964,29 @@ if __name__ == "__main__": "-map", vmap, "-map", amap, "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", out], check=True, capture_output=True) assert os.path.getsize(out) > 0 - for p in (img, c0, c1): + # A/V drift: the tree must not stretch video. The concat branch normalizes fps and the xfade + # branch used not to, so a chapter mixing both played 1.2x slow with the audio untouched and the + # narration ran ahead of the picture. Batch=2 over 4 clips forces BOTH branches plus a second + # round -- the shipped bug's exact shape. Compare the streams, not the file size. + c2, c3 = f"{SHM}/c2.mp4", f"{SHM}/c3.mp4" + subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c2, 1.0), check=True, capture_output=True) + subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c3, 1.0), check=True, capture_output=True) + _saved_batch, cl = ASSEMBLE_BATCH, [] + globals()["ASSEMBLE_BATCH"] = 2 + try: + # ["crossfade","crossfade","cut","cut"] with batch=2 is the one shape that mixes branches: + # round 1 sends [c0,c1] through xfade and [c2,c3] through concat, then round 2 xfades those + # two intermediates together. A cut-only first group would keep both on the concat branch and + # the rates would agree by accident, which is why this case has to be spelled out. + _assemble_batched([c0, c1, c2, c3], ["crossfade", "crossfade", "cut", "cut"], + out, "selfck", cl) + finally: + globals()["ASSEMBLE_BATCH"] = _saved_batch + for p in cl: + if os.path.exists(p): os.remove(p) + vd, ad = _stream_dur(out, "v"), _stream_dur(out, "a") + assert abs(vd - ad) < 0.25, f"A/V drift: video {vd:.2f}s vs audio {ad:.2f}s" + for p in (img, c0, c1, c2, c3): os.remove(p) # #6 composite: 2 panels + 2 audios -> one stacked clip; duration = sum, subs timed per row. a2 = f"{SHM}/a2.wav"