kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record panel 7 against the art, and the A/V gap it hid

Browse Source 
Somebody watched chapter.mp4 for the first time. Two failures came out of it
that no stage counter could see.

chapter.mp4 is video 436.39s over audio 363.67s, so narration finishes 72.7s
before the picture. The 49 clips are clean: all 25fps, video and audio agree
to 0.03s, summing to 363.6s. A per-round probe puts the loss in the final
round of _assemble_batched, which turns 359s of video into 100s while the
audio survives. Round 0 is correct. Round 1 differs by holding a 7th input,
the leftover clip that skips encoding, so the tree mixes concat output, xfade
output and a raw clip. Not fixed.

worker_render.py gains an FPS constant, fps normalization in the xfade branch
to match concat, _stream_dur, and a self-check that compares video against
audio instead of asserting the file is non-empty. That old check is how a 20%
sync failure shipped. The fps inconsistency is real but not proven to be the
shipped cause. Pinning -r on the output was tried and reverted: it drops
frames to force CFR, which the concat branch comment already warned about.

Panel 7 checked against the art has zero correct identity bindings out of two,
and Seonho, the one character who matters, is unbound. bbox values are
consumed as absolute pixels; on a 900x1650 panel that puts all six boxes in
the top third, two inside a speech balloon. Identity therefore embeds crops of
balloon edges and window frames, which is how confidence 0.9 lands on the
wrong person. The colleague has no name in the story and was labelled Choi
Haeseon; that row holds 25 of 26 assignments, so it is the label the pipeline
stamps on any unnamed woman.

Four caveats added. Two earlier claims are withdrawn in place: rescaling bbox
by 1000 does not make the boxes correct, and the constraint is not 16 nameless
rows needing names. Cast profiles already exist, since all 53 rows populate
ref_image_uris and embedding_uri, but they are enrolled from the wrong crops.

worker_render.py self-check passes. No pipeline ran.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kami
2026-08-12 01:04:51 +04:00
parent 447c68c8aa
commit a9d64fe80a
6 changed files with 563 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files
HANDOFF.md
+262 -59
View File
@@ -1,88 +1,291 @@
# HANDOFF, 2026-08-11 late session
# HANDOFF, 2026-08-12
Live state is in `NEXT.md`. This file is only what this session did.
## Asked
Continue from the previous handoff: fix speaker attribution, then rerun and re-read the named-speaker
share. Mid-session: what happens to unidentified people in a two-person panel, can a known character be
mis-named, and keep watch during the run.
Get up to speed from the previous handoff. Then: the user watched `chapter.mp4` for the first time and
read out 19 timestamped defects. Mid-session, what about characters. Then, write the handoff.
## Result
The named-speaker share is **9 of 95 speech lines, 9%**. The previous 30% counted fake tails. All 9 binds
are `Choi Haeseon`, the over-merged row. `script` passed 116/116 for the first time.
**Somebody finally watched the video.** That single act found more than four sessions of measuring did.
The recorded metrics said `script` 116/116 and "9 named speech lines". Both were true. Both measured the
wrong thing, because the 9 names are the *wrong* name.
**Identity, not attribution, is now the constraint.** 26 of 113 detected people carry an identity. 25 of
the 26 are that one row. This chapter caps near 23% named even with perfect balloon binding.
Two hard numbers came out of it.
Six defects, four found by measuring the run rather than by reading code. Each has a decision entry:
1. **The chapter is 20% out of sync.** `chapter.mp4` is video 436.39s over audio 363.67s. The narration
ends 72.7 seconds before the picture. The gap accumulates, which is why everything after 2:54 goes
sideways. The 49 clips are clean: video and audio agree to 0.03s and sum to 363.6s. Assembly adds
72.7s of video and no audio.
2. **Identity binds names to the wrong people, and to people who have no name.** Walking panel 7 against
the art found zero correct bindings out of two, plus the one character who matters left unbound. See
`Panel 7, walked against the art`, which supersedes the earlier reading of this.
| # | defect | entry |
Nothing is committed. `worker_render.py` is edited in the working tree. The fix is **not** verified.
## The user's 19 notes, grouped by cause
| cause | timestamps | what is wrong |
| --- | --- | --- |
| 1 | `tail` stamped on a model guess at confidence 1.0 | `decisions/speaker-attribution.md#no-fake-tail` |
| 2 | a two-word cast name always failed the script verifier | `#multiword-cast-names` |
| 3 | `/stage/clear dialogue` deleted nothing and reported success | `decisions/storage-layout.md#clear-vision-blob` |
| 4 | gemma's speaker answer echoed the prompt label, became a name | `#prompt-label-answers` |
| 5 | `som_face` stamped on a mark that paired to nobody | `#unpaired-mark` |
| 6 | one two-letter interjection halted the chapter at 112/116 | `#interjection-false-positive` |
| one row absorbed every identity | 0:20, 1:07, 1:51, 1:59, end | anyone identified comes out `Choi Haeseon` |
| the MC has no name | 0:44, 0:50, 1:02, 1:23, 1:45 | falls back to "the worker", "someone", "she" |
| gender read off the wrongly bound row | 1:45 "she admits", end "as he waves" | `Choi` is `f`, so "he" means a nameless `m` row got the line |
| narration invents facts | 0:43 "results", 2:03, 2:05, 2:15 "long shift" | the verifier checks quotes and names, not invented claims |
| vision reads art-within-art as scene | 1:35 chibi on a monitor as "a man holding a drink", 1:59 "pointing towards the screen" | panel-in-panel and screen content taken as reality |
| a beat carries nothing | 0:35-0:37 | no content worth narrating |
| no parallax, so a still holds | 2:24-2:52, 28s static | `layers` wrote nothing (`caveats/audit-open.md#layers-writes-nothing`) |
| transition quality | 2:52-2:54 slide "too sharp and laggy" | `push` is `slideleft` at 0.4s. Retest after the sync fix |
| A/V drift | everything after 2:54 | the 72.7s gap above |
## Changed
## Panel 7, walked against the art
Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`, commits `e8941d8 8071137 a965077`
plus docs:
This is the load-bearing finding of the session. The user pulled up the panel and checked every
detection by eye. **Read this before touching identity.** It contradicts what the earlier sessions
recorded, and it contradicts two theories I floated today before the user corrected them.
- `worker_vision.py`: `_annotate_speaker_methods`, `_apply_speaker_labels`, new `_present_keys`
- `decisions/speaker-attribution.md` (new, 5 sections), `decisions/storage-layout.md#clear-vision-blob`,
`caveats/speaker-attribution.md`, `caveats/audit-open.md#dishonest-clearing`, both indexes,
`JOURNAL.md`, `NEXT.md`
Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, crop `s3://panels/.../panels/p006.png`, 900x1650.
A wide establishing shot of an office seen through a window. Vision emitted 6 characters.
Orchestrator, `/mnt/server/home/kami/docker-apps`, commits `b18b6b4 603d388 db8d7c5 ccc3a8e` plus the
interjection commit:
| detection | vision said | the art shows | identity assigned |
| --- | --- | --- | --- |
| `person_5` | m, short black, **yellow sweater**, sitting | **Seonho**, foreground, yellow plaid, headphones, back to camera. The character who matters | **nothing** |
| `person_6` | f, short brown, **white shirt**, sitting | the **colleague**, green dress, ponytail. She has **no name** in the story | `Choi Haeseon` at **0.9** |
| `person_2` | m, short brown, green sweater, sitting | a background extra, seated beyond the next window pane | `Lim Seonho` at **0.9** |
| `person_1` | m, short black, suit, standing | **nobody. A window frame** | nothing |
| `person_3` | m, short black, blue sweater, standing | background extra | nothing |
| `person_4` | m, short black, grey sweater, standing | background extra | nothing |
- `correctness.py`: tokenized `allowed`, `_ID_SHAPED` guard in `normalize_speaker`, interjection
stopwords, short-quote grounding skip
- `db.py`: `_STAGE_VISION_KEYS` and the strip pass in `clear_stage_data`
- `test_script_verify.py`, `test_correctness.py`, `test_db.py`: one case each
**Zero of the two bindings are right, and the one character who matters got nothing.** Both wrong binds
carry confidence 0.9.
Three separate defects stack here.
**1. The stored bbox coordinate space is wrong.** Consumed as absolute pixels, all six boxes land in the
top third of a 1650px-tall panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against
the panel's own dimensions, `person_2`, `person_3`, `person_4` and `person_5` fit their subjects tightly.
So the numbers are not pixels. Two places assert that they are:
- `worker_vision.py:271` prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
- `worker_identity.py:91` comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
Everything reading `bbox` is therefore cropping the wrong region. `worker_identity.py:200` embeds
`_crop_bbox(img, ch["bbox"])`, so `Choi Haeseon` at 0.9 was matched on a crop of the speech balloon's
edge and `Lim Seonho` at 0.9 on a crop of empty window frame. Wrong crops are mostly blank white, which
embed alike, which is a plausible mechanism for one row absorbing 25 assignments.
**Do not treat rescaling as the fix.** I claimed that and the user disproved it in one screenshot.
After scaling, `person_1` still sits on a window frame with nobody in it, and `person_6` is offset,
clipping the woman and running onto the dark frame. `worker_vision.py:38` already calls the box
"coarse, imprecise". Scaling buys roughly-right boxes, not right ones.
**2. Vision has no concept of extra versus cast.** Four of the six detections are background extras or
nothing at all. They are handed to identity as candidates on equal footing with the two people who
carry the scene. That also means the "113 detected people" figure that framed the whole roadmap was
never the right denominator, so "26 of 113 carry an identity" measured nothing useful.
**3. Identity mints a name onto a person who has none.** The colleague has no name in the story. She was
labelled `Choi Haeseon` at 0.9. Across the chapter `Choi Haeseon` holds 25 of 26 assignments, so in
practice that row is the label this pipeline stamps on any unnamed woman. This is the direct cause of
the user's 0:20 note, "Choi Haeseon when there's no Choi in the frame, it's the colleague", and of the
gender flips, since gender is read off whichever row got bound.
This is a cousin of invariant 6 in `CLAUDE.md`, which forbids minting a character from an unparseable
model answer. The missing rule is the same shape: **never attach a name to a detection that carries no
name evidence.** An unnamed recurring person needs a stable anonymous identity so the narration can
call her "the colleague" every time, rather than being forced onto a named row.
Partly checked, not finished: `match()` in `worker_identity.py:69` does abstain, returning `None` below
threshold, so the 0.9 came from cosine clearing the threshold on a garbage crop. Whether the Tier-2
gemma resolver can answer "none of these" was not verified. Check that first.
### Descriptions are not trustworthy either
`person_6` is "white shirt" for a woman in a green dress. `person_5` is "yellow sweater" for yellow
plaid, which is close enough. Any downstream rule keyed on appearance text inherits this.
## Registry, measured
`characters` is keyed by `manga_id`, not chapter. Two manga share the table. `d7104032` has 34 rows with
9 named. This chapter's `ef105a86` has 19 rows with 3 named. The `Kei`, `Zen`, `Kanade`, `Rico` and `K3`
rows belong to the other manga, so they are not polluting this chapter.
This manga's 19 rows, all `status=confirmed`, all `first_seen_panel=NULL`:
- named: `Choi Haeseon` (f), `Seonho` (m, aliases `["Lim Seonho","Seonho"]`), `Lim Seonho` (m)
- 16 rows with `name=NULL` and an empty alias list
`Seonho` and `Lim Seonho` are the same person in two rows. `Seonho`'s alias list contains the other
row's name. That is why either spelling matches two rows and binds nothing.
Assignments across the whole table, for scale:
| character | name | assignments |
| --- | --- | --- |
| `character_afa7623b` | Choi Haeseon | 25 |
| `character_bb79cfb4` | Kanade | 25 |
| `character_cfd34340` | Rico | 19 |
| `character_6f491712` | Lim Seonho | 1 |
In this chapter only `Choi Haeseon` (25) and `Lim Seonho` (1) appear.
Read those 25 together with panel 7. `Choi Haeseon` is not a character who appears 25 times. It is the
row that absorbs any unnamed woman. The 16 nameless rows are not a backlog of people waiting for names.
Some of them are background extras that should never have become rows, and at least one of them, the
colleague, is a real recurring person who correctly has no name and needs to keep it.
An earlier draft of this file said "naming is the ceiling, 16 of 19 rows need names". **That was wrong**
and it is corrected here. The ceiling is that identity cannot say "person, no name" and cannot tell an
extra from cast.
## The A/V bug, located but not fixed
Reproduced on the 49 real clips with `repro.py` and `probe.py` (see Open). Per-round probe with
`ASSEMBLE_BATCH=8` and 6 `fade_black` boundaries spread across batches:
```
r0 g0 n=8 XFADE in v= 49.44 a= 49.44 -> out v= 48.56 a= 48.64 lost_v=+0.88 lost_a=+0.80
r0 g1 n=8 XFADE in v= 47.04 a= 47.04 -> out v= 46.16 a= 46.23 lost_v=+0.88 lost_a=+0.81
r0 g2 n=8 XFADE in v= 48.00 a= 48.00 -> out v= 47.12 a= 47.19 lost_v=+0.88 lost_a=+0.81
r0 g3 n=8 XFADE in v=100.80 a=100.80 -> out v= 99.92 a= 99.96 lost_v=+0.88 lost_a=+0.84
r0 g4 n=8 concat in v= 64.18 a= 64.16 -> out v= 64.24 a= 64.26 lost_v=-0.06 lost_a=-0.10
r0 g5 n=8 XFADE in v= 44.00 a= 44.00 -> out v= 43.12 a= 43.17 lost_v=+0.88 lost_a=+0.83
r1 g0 n=7 XFADE in v=359.29 a=359.60 -> out v= 99.96 a=358.79 lost_v=+259.33 lost_a=+0.81
```
Round 0 is correct. Each group loses only the xfade overlap. **Round 1 loses 259s of video against 0.8s
of audio.** Its output video is 99.96s at 2499 frames, almost exactly the frame count of input `n3` on
its own, the 99.92s intermediate. The final video appears to carry the frames of one input.
The round-1 filtergraph is arithmetically correct, so this is ffmpeg behaviour, not offset math:
```
[n0][n1]xfade=transition=fade:duration=0.050:offset=48.510[v1]
[v1][n2]xfade=transition=fade:duration=0.050:offset=94.620[v2]
[v2][n3]xfade=transition=fadeblack:duration=0.600:offset=141.140[v3]
[v3][n4]xfade=transition=fade:duration=0.050:offset=241.010[v4]
[v4][n5]xfade=transition=fade:duration=0.050:offset=305.200[v5]
[v5][n6]xfade=transition=fade:duration=0.050:offset=348.270[v6]
```
### The strongest clue, found last
Re-running that chain by hand over only the **6** round-0 intermediates gives a correct 348.24s at 8708
frames, `rc 0`, no warnings. Round 1 collapses with **7** inputs, not 6.
The 7th input is the leftover 49th clip. With 49 clips and batch 8, round 0 makes 6 groups of 8 and one
group of 1, and `_assemble_batched` passes a lone group through un-encoded:
```python
if len(group) == 1 and not final_round:
next_items.append(group[0])
```
So the final xfade mixes 6 encoded intermediates with 1 raw clip. That passthrough is a **third** path
next to `concat` and `xfade`, and it is the prime suspect. Start here tomorrow. Confirm it by running
`probe.py` with 48 clips instead of 49, which removes the leftover entirely.
### What was tried and what it cost
Two paths exist in `_assemble_once`. A `concat` branch handles cut-only batches. An `xfade` branch
handles batches with a real transition. They disagreed on frame rate. The concat branch forced `fps=30`
while the xfade branch normalized nothing, and clips are 25fps. `436.39 / 363.63 = 1.2001`, exactly
`30/25`, which is what sent me down this path.
Working-tree changes to `worker_render.py`, all uncommitted:
- new `FPS = 25` constant. The three hardcoded `25`s and the one `30` now reference it
- the xfade branch normalizes every input with `setsar=1,fps={FPS}` into `[n{i}]` labels, matching what
the concat branch already did
- new `_stream_dur(path, kind)`. `_audio_dur` probes `format=duration`, which is `max(video, audio)`, so
it hides A/V drift by construction
- the `__main__` xfade self-check now assembles 4 clips through `_assemble_batched` with
`ASSEMBLE_BATCH=2` and asserts `abs(video - audio) < 0.25`. It previously asserted only
`getsize(out) > 0`, which is why this shipped
**A dead end worth not repeating.** I also pinned `-r FPS` on both output encodes. That made it worse.
The chapter collapsed to exactly 100.00s at 2500 frames, because forcing CFR on irregular input
timestamps drops frames. The comment already sitting at the concat branch warns about this. Both `-r`
flags were removed again. The in-graph `fps=` filter is the right normalization. The output `-r` is not.
The fps inconsistency is real and worth keeping fixed. It is **not** proven to be the cause of the
shipped 72.7s gap. The scene graphs hold 356 `cut` against 6 `fade_black`, so round 2 of the real run
most likely stayed on the concat branch, where no mixing occurs. Treat the fps work as necessary and
insufficient.
### The user's own hypothesis, which is the recommended direction
Two guesses, both worth following:
1. the assembly is wrong in an ffmpeg sense
2. there are two different paths, and they get mixed when there should only be one
Guess 2 matches the code. `concat`, `xfade` and the single-item passthrough are three paths, and
`_assemble_batched` feeds the output of one into the input of another. **Collapse it to one path.**
Normalize every input, then xfade every boundary, with `cut` as a 0.05s fade. `acrossfade` shortens
audio by the same amount that xfade shortens video, so A/V stays locked. The chapter then comes out
about 2.4s shorter than the sum of the clips, with both streams agreeing. That removes the branch
interaction instead of tuning it.
## Measured
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels.
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`.
- 113 orchestrator tests pass. `worker_vision.py` self-check passes.
- `dialogue` 116/116, `direct` 116/116, `scene` 116/116, `script` 116/116.
- speech lines 95, named 9. Multi-character panels 0 of 40 by design. Single-character 9 of 55.
- `speaker_method`: `unknown` 47, `model_solo` 28, `solo_prior` 8, `som_face` 7, `turn_taking` 5.
- unresolved name refs 3, all `brown ponytail, green dress`, from a neighbouring panel in the same
8-panel window. Was 24 of 51 before the fix.
- identity assignments 26, of which `character_afa762` "Choi Haeseon" holds 25.
- registry duplicates that block a correct bind: `seonho` matches 2 rows, `lim seonho` matches 2 rows.
- buckets during `tts`: `panels` 116, `raw` 79, `manga` 491, `audio` 1, `layers` 0, `video` 0.
- job `status=completed`, every stage at its unit count, finished `2026-08-11T20:08:16Z`
- it still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`)
- `chapter.mp4` 50MiB, video 436.392s, audio 363.675s, `r_frame_rate=25/1`,
`avg_frame_rate=63372800/2792909` which is 22.69, `nb_frames=9902`
- 49 clips, every one `25/1` exactly, sum video 363.63s, sum audio 363.60s. No clip has the two
differing by more than 0.05s
- scene-graph transitions: `cut` 356, `fade_black` 6
- `worker_render.py` `__main__` self-check passes on the current working tree
- workers up in tmux `manga-workers`, 9 windows
## Open
- **The run finished.** Every stage completed at 2026-08-11T20:08:16Z: `tts` 116/116, `layers` 116/116,
`render` 116/116, `assemble` 1/1. `s3://video/` holds 49 clips and a 50MiB `chapter.mp4` under
`ef105a86-.../7c944dd4-.../`, `s3://audio/` 49 objects at 32MiB. Nobody has watched the video.
Two honesty defects at the finish, recorded not fixed. `layers` reported `completed 116/116` with an
empty bucket, so no clip has parallax (`caveats/audit-open.md#layers-writes-nothing`). The completed
job still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`).
Read the state, or clear a stage and resume:
- **Finish the round-1 diagnosis.** The `_assemble_batched` tree turns 359s of video into 100s. This is
the worst defect found and it reproduces offline in about two minutes with no GPU. Suspect the 7th
passthrough input first.
```bash
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"<stage>\"}'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
.venv/bin/python <scratchpad>/probe.py # the per-round loss table above
.venv/bin/python <scratchpad>/repro.py # end-to-end verdict
```
Note: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`.
The scratchpad is session-scoped and will be gone. **Re-download the clips first:**
- Defects 5 and 6 landed after `dialogue` had already run, so this run's 7 `som_face` lines are still
labelled from the unpaired-mark path. The next dialogue pass fixes that. No name was affected.
- The audit script is now `audit_speakers.py` in this repo. It reads `/data/manga.db`, so it runs inside
the container: `docker cp audit_speakers.py manga-orchestrator:/tmp/ && docker exec
manga-orchestrator python3 /tmp/audit_speakers.py`.
- The 3 cross-panel unresolved refs need the window's whole present-list, not one panel's.
- Workers were restarted twice this session and are running in tmux `manga-workers`. Nothing watches
them, and nothing watches the homesrv containers.
```bash
/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/
```
Both scripts are worth committing next session. They are the only check that has ever caught this.
- **Do not trust `format=duration`.** It returns `max(video, audio)`, so every existing duration assert
in `worker_render.py` is blind to drift. `_stream_dur` exists now. The other asserts still use
`_audio_dur`.
- **Identity, in the order the panel 7 evidence implies.** First, settle the `bbox` coordinate space and
fix every consumer, since nothing else can be judged while crops are wrong. Second, let identity
abstain and hold a stable anonymous identity, so the colleague stays "the colleague". Third, separate
extra from cast so extras never reach identity. Only then merge `Seonho` into `Lim Seonho` and split
`character_afa7623b`, which still needs the reversible-merge design
(`caveats/audit-open.md#destructive-reconcile`).
- **Verify the bbox space before changing anything.** Two independent claims in the code say pixels, and
the art says otherwise. Confirm what the model was told and what it returns, rather than trusting
either comment. Then check whether the crop is the only consumer, or whether SoM marker placement and
the face-pairing in `worker_vision.py:57` read the same numbers.
- Re-derive the panel 7 overlay when needed. It took one `mc cat` of the crop plus a Pillow script, and
it found more than any query did:
```bash
/usr/bin/ssh kami@192.168.1.104 'mc cat homesrv/panels/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/panels/p006.png' > p007.png
```
Draw each `bbox` twice, once as pixels and once divided by 1000, then look at it.
- `mc` aliases on homesrv: use `homesrv` or `mio`, not `local`, which returns Access Denied. `rfs` is
the empty rustfs.
- Plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`.
- `cp` is aliased to `cp -i` in this shell and hangs on overwrite. Use `/usr/bin/cp -f`.
- The Bash tool's default timeout is 120s no matter what `timeout` the command itself carries. Pass the
tool's own timeout or background the run. Otherwise a restore step after a mutation test never runs,
which left a deliberately broken `worker_render.py` on disk once this session.
JOURNAL.md
+59
View File
@@ -169,3 +169,62 @@ layers (`decisions/storage-layout.md#bucket-per-artifact`).
Two honesty defects surfaced at the finish, both recorded rather than fixed. `layers` reported
`completed 116/116` with an empty bucket, and the completed job still carries
`error: "partial: 112/116 completed"` from the failure three resumes earlier.
## 2026-08-12 — the video got watched
No pipeline ran. The user watched `chapter.mp4` for the first time and read out 19 timestamped defects.
That found more than the previous four sessions of measuring, because the recorded metrics were all
measuring whether code ran rather than whether the result was right.
Two measurements came out of it. First, `chapter.mp4` is video 436.39s over audio 363.67s, so the
narration finishes 72.7s before the picture and the gap accumulates. The 49 clips are clean: every one is
25fps exactly, video and audio agree to 0.03s, and they sum to 363.6s. Assembly adds 72.7s of video and
no audio. Second, this manga holds 19 character rows of which 3 carry a name, and `Choi Haeseon` holds 25
of the chapter's 26 identity assignments. That is why the video calls the colleague Choi, never names the
MC, and flips gender.
The A/V bug was narrowed with a per-round probe over the 49 real clips. Round 0 of `_assemble_batched` is
correct, losing only the xfade overlap per group. Round 1 turns 359s of video into 100s while the audio
survives at 358.79s. The round-1 filtergraph is arithmetically correct, and re-running the same chain by
hand over only the 6 encoded intermediates gives a correct 348.24s with no warnings. Round 1 differs by
holding a 7th input: the leftover 49th clip, which `_assemble_batched` passes through un-encoded. That
passthrough is a third path beside `concat` and `xfade` and is the prime suspect.
`worker_render.py` gained an `FPS = 25` constant, fps normalization in the xfade branch to match the
concat branch, a `_stream_dur` helper, and a self-check that compares video against audio rather than
asserting the file is non-empty. The old check only asserted `getsize(out) > 0`, which is how a 20% sync
failure shipped. Pinning `-r FPS` on the output encodes was tried and reverted: it collapsed the chapter
to exactly 100.00s by dropping frames to force CFR, which the comment at the concat branch already
warned about. None of it is committed and none of it fixes the chapter yet.
The fps inconsistency between the two branches is real but not proven to be the shipped cause. The scene
graphs hold 356 `cut` against 6 `fade_black`, so the real run's final round most likely stayed on the
concat branch where no mixing happens.
### Panel 7, checked against the art
The same day, the user pulled up panel 7 and checked every detection by eye. It overturned the framing
this file carried an hour earlier, and it overturned two theories I proposed before being corrected.
Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, a wide establishing shot of an office through a
window, crop 900x1650. Vision emitted 6 characters. Zero of the two identity bindings are correct and the
one character who matters is unbound. `person_5`, described as "yellow sweater", is Seonho in the
foreground and got no identity. `person_6` is the colleague, who has no name in the story, and was
assigned `Choi Haeseon` at 0.9. `person_2` is a background extra and was assigned `Lim Seonho` at 0.9.
`person_1` is a window frame with nobody in it. `person_3` and `person_4` are background extras.
Three defects stack, recorded as `caveats/speaker-attribution.md#bbox-wrong-space`,
`#no-anonymous-identity` and `#extras-as-cast`. The `bbox` values are consumed as absolute pixels, and on
this panel that puts all six boxes in the top third with two inside a speech balloon. Divided by 1000
four of the six fit tightly. Identity therefore embedded crops of balloon edges and window frames, which
is how a 0.9 confidence lands on the wrong person. Blank crops embed alike, a plausible mechanism for one
row absorbing 25 of 26 assignments.
Two claims I made and had to withdraw. First, that rescaling by 1000 makes the boxes correct: after
scaling, `person_1` still sits on an empty window frame and `person_6` clips its subject, and the
descriptions are unreliable anyway, since `person_6` reads "white shirt" for a green dress. Second, that
the constraint is 16 nameless rows needing names. The opposite is true. The pipeline mints names onto
people who have none, and at least one nameless row is a real recurring person who should stay nameless.
The "26 of 113 detected people carry an identity" figure that framed the roadmap counted mostly
background extras. It should not be quoted again.
NEXT.md
+110 -47
View File
@@ -1,85 +1,148 @@
# NEXT
Updated 2026-08-11. Replaces the old `HANDOFF.md`.
Updated 2026-08-12. What this session did is in `HANDOFF.md`.
## State
Audit Phase 1 is implemented and green. Nothing is half-finished.
The chapter runs end to end and the output is **not watchable**. That is now measured, not guessed.
Changed on workpc: `worker_scene.py`, `worker_script.py`, `worker_vision.py`, `session_manager.py`.
Changed on homesrv (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`): `tracklets.py`,
`correctness.py`, `db.py`, `service.py`, `session_proxy.py`, `test_script_verify.py`,
`test_name_binding.py`.
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels,
`status=completed`, finished 2026-08-11T20:08:16Z. `s3://video/` holds 49 clips and a 50MiB
`chapter.mp4`. The user watched it and read out 19 defects. They are grouped by cause in `HANDOFF.md`.
What landed and why: `decisions/audit-phase1.md`. What was left open: `caveats/audit-open.md`.
Two numbers set the agenda:
Verification: CPU-only self-checks and the orchestrator test suite. 108 orchestrator tests pass
(`test_api.py` is excluded on workpc because fastapi is not installed in this venv). No GPU work ran
and no pipeline ran, so none of this is confirmed against a real chapter.
- `chapter.mp4` is video 436.39s over audio 363.67s. The narration finishes 72.7s before the picture.
- Panel 7 checked against the art has **zero correct identity bindings** out of two, and the one
character who matters is unbound. `HANDOFF.md#panel-7-walked-against-the-art` has the table.
The orchestrator half is committed as `1c60710` in `/mnt/server/home/kami/docker-apps` and the
container is rebuilt and serving. `94bd4d8` in the same repo repins minio to its amd64 digest, which
the rebuild exposed.
Uncommitted work sits in the tree: `worker_render.py` has an `FPS = 25` constant, fps normalization in
the xfade branch, a new `_stream_dur`, and a self-check that compares video against audio. The
self-check passes. It does **not** yet fix the chapter. Details and one dead end in `HANDOFF.md`.
## Next
The named-speaker share is 9%, 9 of 95 speech lines, and that number is real. See `JOURNAL.md` for the
six defects behind the old 30%. Everything below is measured on job `778297bc`, not inferred.
1. **Fix chapter assembly.** `_assemble_batched` turns 359s of video into 100s while the audio survives.
It reproduces offline in two minutes, no GPU. Round 0 is correct and round 1 collapses. Round 1 is
the only round holding a raw clip that skipped encoding, so suspect the single-item passthrough
first. The recommended shape is one path, not three: normalize every input, then xfade every
boundary, treating `cut` as a 0.05s fade. `acrossfade` and `xfade` shorten audio and video equally,
so the streams stay locked. `HANDOFF.md` holds the per-round table, the filtergraph, and the repro
commands. Nothing downstream is worth judging until this lands.
2. **Fix identity, in this order.** Panel 7 is the worked example and
`HANDOFF.md#panel-7-walked-against-the-art` carries the evidence. Do not start at the registry.
That job now runs end to end: `chapter.mp4`, 50MiB, 49 clips. Nobody has watched it. Two stages lie about
it (`caveats/audit-open.md#layers-writes-nothing`, `#stale-job-error`).
a. **Settle the `bbox` coordinate space.** Consumed as pixels, all six boxes on panel 7 land in the
top third of the panel, two inside a speech balloon. Divided by 1000 they mostly land on their
subjects. `worker_vision.py:271` and `worker_identity.py:91` both assert pixels, and the art says
otherwise. Identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so today it
matches faces against crops of balloons and window frames. Nothing downstream can be judged until
this is right. Rescaling alone is **not** the fix: after scaling, one box still sits on an empty
window frame and another clips its subject.
b. **Let identity abstain and stay abstained.** The colleague has no name in the story and was
labelled `Choi Haeseon` at 0.9. An unnamed recurring person needs a stable anonymous identity so
narration says "the colleague" every time. `match()` already returns `None` below threshold. Check
whether the Tier-2 gemma resolver can answer "none of these"; that was not verified.
c. **Separate extra from cast.** Four of the six detections on panel 7 are background extras or
nothing at all, and all six reach identity as equal candidates.
d. Only then merge `Seonho` into `Lim Seonho` and split `character_afa7623b`, which still needs the
reversible-merge design (`caveats/audit-open.md#destructive-reconcile`), not a patch.
**Identity is the constraint now, not attribution.** 26 of 113 detected people carry an identity. 25 of
those 26 are the single over-merged row (`caveats/speaker-attribution.md#identity-over-merge`). That caps
this chapter near 23% named even with perfect balloon binding. Work identity before geometry.
**Cast profiles already exist. Do not rebuild them.** The user asked whether the main cast could get a
profile built from reference frames and reused. `characters` already carries `ref_image_uris` and
`embedding_uri`, and all 53 rows have both populated. The mechanism is not missing, it is enrolled
from the wrong crops, so today it stores references to balloon edges and window frames. Step (a) is
what makes it work. Three things are genuinely absent and are the smaller follow-on:
1. Split the over-merged character row and dedupe the registry. `Lim Seonho` and `Seonho` are separate
rows with overlapping aliases, so either name matches two rows and binds nothing. Needs the
reversible-merge design (`caveats/audit-open.md#destructive-reconcile`) rather than a patch.
2. Bind a balloon to a speaker by tail geometry, using the unused `det`/`seg` heads
(`caveats/speaker-attribution.md#tail-is-not-geometry`). Until then multi-character panels have no
speaker at all, which is 40 of 95 lines here.
3. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3
unresolved refs are a description belonging to a neighbouring panel in the same 8-panel call.
4. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work
- no quality gate on enrollment, so nothing checks that a reference crop holds a face at all
- nothing re-enrolls a reference set once it is written, so the wrong crops persist
**The visual "is this them?" check is already built. Do not write it again.** `/vision/resolve` at
`worker_vision.py:963` sends the query crop plus up to 3 labelled reference images per candidate.
`build_resolve_prompt` tells the model to judge face shape first, to treat hair and outfit as
secondary, that two people sharing a hair colour are not the same, and to answer `0` for NONE when
unsure. `choice: 0` becomes a new character, an out-of-range index becomes `unresolved`, and
`ref_image_uris` is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`. The
mechanism, the prompt and the abstain path are all correct. They are fed crops of the wrong region,
which is step (a).
- no human gate to name, merge or split the clusters. The user wants this as a minor adjustment on
top, not as the mechanism. The `gates` table and the review gates from [#136] are the place to hang
it
The chibi at 1:35 will survive all of this. He genuinely is brown hair plus a yellow shirt, so a
profile match is correct on appearance and wrong on reality. That needs item 4 below, plus requiring
a real face before a crop can enroll.
3. **Stop the narration inventing facts.** 0:43, 2:03, 2:05 and 2:15 assert things no panel shows. The
correctness verifier passed 116/116 because it checks quotes and names, never invented claims.
4. **Teach vision that art inside a panel is not the scene.** A chibi on a monitor became "a man holding
a drink" at 1:35. A colleague pointing into the distance became "pointing towards the screen" at
1:59.
5. **`layers` writes nothing** and reports `completed 116/116`, so no clip has parallax and a still
holds for 28s from 2:24 (`caveats/audit-open.md#layers-writes-nothing`).
6. **Clear the stale job error.** The completed job still carries `error: "partial: 112/116 completed"`
(`caveats/audit-open.md#stale-job-error`).
7. Balloon-to-speaker geometry via the unused `det`/`seg` heads
(`caveats/speaker-attribution.md#tail-is-not-geometry`) is now behind item 2. With no name to attach,
geometry buys nothing.
8. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3
unresolved refs describe a neighbouring panel in the same 8-panel call.
9. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work
(`caveats/audit-open.md#sqlite-locking`).
## The 2026-08-11 chapter run
## Lesson worth keeping
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116
panels. It reached `scene` 116/116 and then failed in `script` at 87/116 on the verifier bug fixed
above. `tts layers render assemble` never ran. Read where it got to, clear the failed stage, resume:
Every metric recorded before this session said the pipeline was fine or nearly fine. `script` 116/116,
"9 named speech lines", `layers` 116/116, `assemble` 1/1. Watching two and a half minutes of output
found a 20% sync failure, a cast that is 84% anonymous, invented narration, and a stage that writes
nothing while reporting success. Stage counters measure whether code ran. They say nothing about whether
the result is correct. Watch the output before trusting a number.
## Running the pieces
```bash
ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"script\"}'"
ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
./start_workers.sh # session_manager + 9 workers, each a uvicorn in a tmux window
tmux attach -t manga-workers # per-worker logs
.venv/bin/python worker_render.py # self-check, runs real ffmpeg, about 4 minutes
```
That resume narrates the attributions the old `tail` label produced. Clearing back to `dialogue`
instead re-runs the GPU stages and produces the honest metric.
Read the state, or clear a stage and resume:
Numbers and the quality read are in `JOURNAL.md` and `caveats/speaker-attribution.md`.
```bash
/usr/bin/ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"<stage>\"}'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
```
Traps: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin, so use `/usr/bin/ssh`.
`mc` aliases on homesrv are `homesrv` and `mio`. `local` returns Access Denied and `rfs` is the empty
rustfs. `cp` is aliased to `cp -i` and hangs on overwrite, so use `/usr/bin/cp -f`.
Re-fixing assembly needs the real clips, which the session scratchpad no longer holds:
```bash
/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/
```
## Storage and viewer, tasks #116/#117
[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had
been dead since 2026-07-19 on an arm64 digest pin. Details in `JOURNAL.md`.
[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had been
dead since 2026-07-19 on an arm64 digest pin.
[#116] is closer but not cut over. Artifacts now split one bucket per class
[#116] is closer but not cut over. Artifacts split one bucket per class
(`decisions/storage-layout.md#bucket-per-artifact`), and both MinIO and `rustfs` hold all six buckets.
`rustfs` on `127.0.0.1:9010/9011` is still empty and nothing is repointed, so MinIO serves every read
and write. Remaining: `mc mirror` the live buckets, verify counts and sizes, then decide on cutover
(`decisions/storage-layout.md#rustfs-staged`).
Two containers on homesrv had been dead for two weeks and are now running. `manga-fetch` is the one
`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them.
Two containers on homesrv had been dead for two weeks and now run. `manga-fetch` is the one
`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them, and
nothing watches the workers.
## Open questions
Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for. They are the
speaker contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only
[#203] existed and is now closed by `decisions/audit-phase1.md#unlocked-model-load`.
Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for: the speaker
contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only [#203]
existed and is closed by `decisions/audit-phase1.md#unlocked-model-load`.
Three audit items are deliberately not done and are recorded as caveats rather than silently dropped:
honest stage clearing, ComfyUI under the session mutex, and reversible identity merges. Each needs a
caveats/CLAUDE.md
+4
View File
@@ -38,3 +38,7 @@ a complaint, so give it one or drop it.
| [One invented word still halts the chapter](speaker-attribution.md#one-word-halts-chapter) | 2026-08-11 run |
| [A completed job keeps the error from an earlier failure](audit-open.md#stale-job-error) | 2026-08-11 run |
| [`layers` reports success on an empty bucket](audit-open.md#layers-writes-nothing) | 2026-08-11 run |
| [Every `bbox` is read in the wrong coordinate space](speaker-attribution.md#bbox-wrong-space) | 2026-08-12 panel 7 |
| [Identity cannot say "a person with no name"](speaker-attribution.md#no-anonymous-identity) | 2026-08-12 panel 7 |
| [Vision does not separate a background extra from cast](speaker-attribution.md#extras-as-cast) | 2026-08-12 panel 7 |
| [Cast reference profiles are enrolled from wrong crops](speaker-attribution.md#poisoned-reference-set) | 2026-08-12 panel 7 |
caveats/speaker-attribution.md
+79
View File
@@ -69,3 +69,82 @@ stopped.
**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer
is to flag the beat for review and continue, which is `#136` gate work, not a verifier change.
## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space}
Vision's `bbox` values are stored and consumed as absolute pixels. On panel
`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of
the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own
dimensions, four of the six fit their subjects tightly.
Two places in the code assert pixels, and the art contradicts both:
- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces
against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9
from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed
alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at
`worker_vision.py:57` reads the same numbers and was not checked.
Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with
nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already
calls the box "coarse, imprecise".
Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can
be judged while the crops are wrong.
## Identity cannot say "a person with no name" {#no-anonymous-identity}
The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9.
Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline
stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which
is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing
line.
This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an
unparseable model answer. The missing rule: never attach a name to a detection that carries no name
evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the
colleague" every time.
`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from
cosine clearing the threshold on a wrong crop. Whether the Tier-2 gemma resolver can answer "none of
these" was not verified.
Revisit trigger: immediately after the `bbox` space is settled.
## Vision does not separate a background extra from cast {#extras-as-cast}
Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground
and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame
with nobody in it. All six reach identity as equal candidates.
Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly
extras, so it measured nothing useful and should not be quoted again.
Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return.
## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set}
`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the
cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are
crops of balloon edges, window frames and background extras rather than of faces.
The visual comparison people reach for as the fix is **already implemented**, so do not build it again.
`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images
per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair
and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for
NONE when unsure. `choice: 0` becomes a new character and an out-of-range index becomes `unresolved`. The
`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`,
so the references reach the model.
That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon
edge against references enrolled from window frames and background extras, then sometimes answers "same".
Nothing gates enrollment on the crop holding a face.
Who pays: every later match, because the reference set defines what a character looks like. Fixing
`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place.
Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the
existing `ref_image_uris` and `embedding_uri` values as invalid.
worker_render.py
+49 -8
View File
@@ -116,8 +116,25 @@ def _audio_dur(path: str) -> float:
return 0.0
def _stream_dur(path: str, kind: str) -> float:
"""duration of one stream. `format=duration` is max(video,audio) and so hides A/V drift."""
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", f"{kind}:0",
"-show_entries", "stream=duration", "-of", "default=nk=1:nw=1", path],
capture_output=True, text=True)
try:
return float(r.stdout.strip())
except ValueError:
return 0.0
ZMAX, ZPAN = 1.15, 1.18 # ken-burns zoom ceiling; constant zoom that gives pans room to travel
# Every clip and every assembly stage MUST agree on this. xfade does not resample: it reinterprets the
# second input's frames at the first input's rate, so a 30fps input joined onto a 25fps one plays 1.2x
# too slow with the audio untouched -- the video ends minutes long and the narration runs ahead of the
# picture. That is exactly what a 25fps clip pipeline plus a `fps=30` concat branch produced.
FPS = 25
def _motion(camera: dict, frames: int) -> str:
"""#8 content-aware motion: map the vision `camera` block to a zoompan z/x/y expression.
@@ -155,7 +172,7 @@ def _motion(camera: dict, frames: int) -> str:
x, y = f"{xc}+(iw*0.03)*sin(6.283*on/{T})", f"{yc}+(ih*0.03)*cos(6.283*on/{T})"
else: # zoom_in (default ken burns)
z, x, y = f"1+{ZMAX-1:.3f}*on/{T}", xc, yc
return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps=25"
return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps={FPS}"
def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict = None,
@@ -163,8 +180,7 @@ def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict
"""ffmpeg: still panel over a blurred fill of itself + content-aware motion, burned subs, 9:16.
#5 blurred bg replaces black bars: one copy scaled to COVER + blurred, the fitted panel on top.
#13 pad seconds of trailing silence (last frame held) give the panel a beat before the next."""
fps = 25
frames = max(1, int((dur + pad) * fps)) # hold the last frame through the pad
frames = max(1, int((dur + pad) * FPS)) # hold the last frame through the pad
# overlay's W/H/w/h are ffmpeg's main/overlay dims -- kept literal (no f-string braces).
fc = (
f"[0:v]split=2[bg][fg];"
@@ -483,7 +499,7 @@ def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list
burned subtitle (in `ass`) spans the whole beat. 185: per-image screen-time is content-weighted
(see _beat_slices), equal split when weights are absent. pure -> testable without S3."""
cameras = cameras or []
n, fps = len(imgs), 25
n, fps = len(imgs), FPS
slices = _beat_slices(D, n, weights) # 185: content-weighted, equal-split fallback
# one frame per image (no -loop): zoompan d=frames expands that single frame to exactly `frames`
# output frames = seg seconds. looping instead would feed many frames and zoompan multiplies each.
@@ -675,12 +691,15 @@ def _xfade_chain(durs: list, trans: list):
# accumulator would run BACKWARDS (cum += dur - td), swallowing every later clip into a frozen
# overlap near the middle. Floor to a small positive length so the timeline stays monotonic.
durs = [d if (d and d > 0.1) else 0.1 for d in durs]
parts, vlast, alast, cum = [], "[0:v]", "[0:a]", durs[0]
# Normalize every input to FPS/SAR before it reaches xfade, exactly as the concat branch does. Both
# branches feed the same tree, so an un-normalized xfade input is what stretched the chapter 1.2x.
parts = [f"[{i}:v]setsar=1,fps={FPS}[n{i}]" for i in range(len(durs))]
vlast, alast, cum = "[n0]", "[0:a]", durs[0]
for i in range(1, len(durs)):
name, td = XFADE.get(trans[i - 1] if i - 1 < len(trans) else "cut", XFADE["cut"])
td = max(0.05, min(td, durs[i - 1] - 0.05, durs[i] - 0.05)) # overlap fits in both clips
off = max(cum - td, 0)
parts.append(f"{vlast}[{i}:v]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]")
parts.append(f"{vlast}[n{i}]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]")
parts.append(f"{alast}[{i}:a]acrossfade=d={td:.3f}[a{i}]")
vlast, alast, cum = f"[v{i}]", f"[a{i}]", cum + durs[i] - td
return ";".join(parts), vlast, alast
@@ -713,7 +732,7 @@ def _assemble_once(inputs: list[str], trans: list[str], out: str):
cmd = ["ffmpeg", "-y"]
for p in inputs:
cmd += ["-i", p]
pre = "".join(f"[{i}:v]setsar=1,fps=30[v{i}];" for i in range(n))
pre = "".join(f"[{i}:v]setsar=1,fps={FPS}[v{i}];" for i in range(n))
fg = pre + "".join(f"[v{i}][{i}:a]" for i in range(n)) + f"concat=n={n}:v=1:a=1[v][a]"
cmd += ["-filter_complex", fg, "-map", "[v]", "-map", "[a]",
"-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p",
@@ -945,7 +964,29 @@ if __name__ == "__main__":
"-map", vmap, "-map", amap, "-c:v", "libx264", "-pix_fmt", "yuv420p",
"-c:a", "aac", out], check=True, capture_output=True)
assert os.path.getsize(out) > 0
for p in (img, c0, c1):
# A/V drift: the tree must not stretch video. The concat branch normalizes fps and the xfade
# branch used not to, so a chapter mixing both played 1.2x slow with the audio untouched and the
# narration ran ahead of the picture. Batch=2 over 4 clips forces BOTH branches plus a second
# round -- the shipped bug's exact shape. Compare the streams, not the file size.
c2, c3 = f"{SHM}/c2.mp4", f"{SHM}/c3.mp4"
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c2, 1.0), check=True, capture_output=True)
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c3, 1.0), check=True, capture_output=True)
_saved_batch, cl = ASSEMBLE_BATCH, []
globals()["ASSEMBLE_BATCH"] = 2
try:
# ["crossfade","crossfade","cut","cut"] with batch=2 is the one shape that mixes branches:
# round 1 sends [c0,c1] through xfade and [c2,c3] through concat, then round 2 xfades those
# two intermediates together. A cut-only first group would keep both on the concat branch and
# the rates would agree by accident, which is why this case has to be spelled out.
_assemble_batched([c0, c1, c2, c3], ["crossfade", "crossfade", "cut", "cut"],
out, "selfck", cl)
finally:
globals()["ASSEMBLE_BATCH"] = _saved_batch
for p in cl:
if os.path.exists(p): os.remove(p)
vd, ad = _stream_dur(out, "v"), _stream_dur(out, "a")
assert abs(vd - ad) < 0.25, f"A/V drift: video {vd:.2f}s vs audio {ad:.2f}s"
for p in (img, c0, c1, c2, c3):
os.remove(p)
# #6 composite: 2 panels + 2 audios -> one stacked clip; duration = sum, subs timed per row.
a2 = f"{SHM}/a2.wav"