Record panel 7 against the art, and the A/V gap it hid

Somebody watched chapter.mp4 for the first time. Two failures came out of it
that no stage counter could see.

chapter.mp4 is video 436.39s over audio 363.67s, so narration finishes 72.7s
before the picture. The 49 clips are clean: all 25fps, video and audio agree
to 0.03s, summing to 363.6s. A per-round probe puts the loss in the final
round of _assemble_batched, which turns 359s of video into 100s while the
audio survives. Round 0 is correct. Round 1 differs by holding a 7th input,
the leftover clip that skips encoding, so the tree mixes concat output, xfade
output and a raw clip. Not fixed.

worker_render.py gains an FPS constant, fps normalization in the xfade branch
to match concat, _stream_dur, and a self-check that compares video against
audio instead of asserting the file is non-empty. That old check is how a 20%
sync failure shipped. The fps inconsistency is real but not proven to be the
shipped cause. Pinning -r on the output was tried and reverted: it drops
frames to force CFR, which the concat branch comment already warned about.

Panel 7 checked against the art has zero correct identity bindings out of two,
and Seonho, the one character who matters, is unbound. bbox values are
consumed as absolute pixels; on a 900x1650 panel that puts all six boxes in
the top third, two inside a speech balloon. Identity therefore embeds crops of
balloon edges and window frames, which is how confidence 0.9 lands on the
wrong person. The colleague has no name in the story and was labelled Choi
Haeseon; that row holds 25 of 26 assignments, so it is the label the pipeline
stamps on any unnamed woman.

Four caveats added. Two earlier claims are withdrawn in place: rescaling bbox
by 1000 does not make the boxes correct, and the constraint is not 16 nameless
rows needing names. Cast profiles already exist, since all 53 rows populate
ref_image_uris and embedding_uri, but they are enrolled from the wrong crops.

worker_render.py self-check passes. No pipeline ran.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-12 01:04:51 +04:00
parent 447c68c8aa
commit a9d64fe80a
6 changed files with 563 additions and 114 deletions
+262 -59
View File
@@ -1,88 +1,291 @@
# HANDOFF, 2026-08-11 late session
# HANDOFF, 2026-08-12
Live state is in `NEXT.md`. This file is only what this session did.
## Asked
Continue from the previous handoff: fix speaker attribution, then rerun and re-read the named-speaker
share. Mid-session: what happens to unidentified people in a two-person panel, can a known character be
mis-named, and keep watch during the run.
Get up to speed from the previous handoff. Then: the user watched `chapter.mp4` for the first time and
read out 19 timestamped defects. Mid-session, what about characters. Then, write the handoff.
## Result
The named-speaker share is **9 of 95 speech lines, 9%**. The previous 30% counted fake tails. All 9 binds
are `Choi Haeseon`, the over-merged row. `script` passed 116/116 for the first time.
**Somebody finally watched the video.** That single act found more than four sessions of measuring did.
The recorded metrics said `script` 116/116 and "9 named speech lines". Both were true. Both measured the
wrong thing, because the 9 names are the *wrong* name.
**Identity, not attribution, is now the constraint.** 26 of 113 detected people carry an identity. 25 of
the 26 are that one row. This chapter caps near 23% named even with perfect balloon binding.
Two hard numbers came out of it.
Six defects, four found by measuring the run rather than by reading code. Each has a decision entry:
1. **The chapter is 20% out of sync.** `chapter.mp4` is video 436.39s over audio 363.67s. The narration
ends 72.7 seconds before the picture. The gap accumulates, which is why everything after 2:54 goes
sideways. The 49 clips are clean: video and audio agree to 0.03s and sum to 363.6s. Assembly adds
72.7s of video and no audio.
2. **Identity binds names to the wrong people, and to people who have no name.** Walking panel 7 against
the art found zero correct bindings out of two, plus the one character who matters left unbound. See
`Panel 7, walked against the art`, which supersedes the earlier reading of this.
| # | defect | entry |
Nothing is committed. `worker_render.py` is edited in the working tree. The fix is **not** verified.
## The user's 19 notes, grouped by cause
| cause | timestamps | what is wrong |
| --- | --- | --- |
| 1 | `tail` stamped on a model guess at confidence 1.0 | `decisions/speaker-attribution.md#no-fake-tail` |
| 2 | a two-word cast name always failed the script verifier | `#multiword-cast-names` |
| 3 | `/stage/clear dialogue` deleted nothing and reported success | `decisions/storage-layout.md#clear-vision-blob` |
| 4 | gemma's speaker answer echoed the prompt label, became a name | `#prompt-label-answers` |
| 5 | `som_face` stamped on a mark that paired to nobody | `#unpaired-mark` |
| 6 | one two-letter interjection halted the chapter at 112/116 | `#interjection-false-positive` |
| one row absorbed every identity | 0:20, 1:07, 1:51, 1:59, end | anyone identified comes out `Choi Haeseon` |
| the MC has no name | 0:44, 0:50, 1:02, 1:23, 1:45 | falls back to "the worker", "someone", "she" |
| gender read off the wrongly bound row | 1:45 "she admits", end "as he waves" | `Choi` is `f`, so "he" means a nameless `m` row got the line |
| narration invents facts | 0:43 "results", 2:03, 2:05, 2:15 "long shift" | the verifier checks quotes and names, not invented claims |
| vision reads art-within-art as scene | 1:35 chibi on a monitor as "a man holding a drink", 1:59 "pointing towards the screen" | panel-in-panel and screen content taken as reality |
| a beat carries nothing | 0:35-0:37 | no content worth narrating |
| no parallax, so a still holds | 2:24-2:52, 28s static | `layers` wrote nothing (`caveats/audit-open.md#layers-writes-nothing`) |
| transition quality | 2:52-2:54 slide "too sharp and laggy" | `push` is `slideleft` at 0.4s. Retest after the sync fix |
| A/V drift | everything after 2:54 | the 72.7s gap above |
## Changed
## Panel 7, walked against the art
Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`, commits `e8941d8 8071137 a965077`
plus docs:
This is the load-bearing finding of the session. The user pulled up the panel and checked every
detection by eye. **Read this before touching identity.** It contradicts what the earlier sessions
recorded, and it contradicts two theories I floated today before the user corrected them.
- `worker_vision.py`: `_annotate_speaker_methods`, `_apply_speaker_labels`, new `_present_keys`
- `decisions/speaker-attribution.md` (new, 5 sections), `decisions/storage-layout.md#clear-vision-blob`,
`caveats/speaker-attribution.md`, `caveats/audit-open.md#dishonest-clearing`, both indexes,
`JOURNAL.md`, `NEXT.md`
Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, crop `s3://panels/.../panels/p006.png`, 900x1650.
A wide establishing shot of an office seen through a window. Vision emitted 6 characters.
Orchestrator, `/mnt/server/home/kami/docker-apps`, commits `b18b6b4 603d388 db8d7c5 ccc3a8e` plus the
interjection commit:
| detection | vision said | the art shows | identity assigned |
| --- | --- | --- | --- |
| `person_5` | m, short black, **yellow sweater**, sitting | **Seonho**, foreground, yellow plaid, headphones, back to camera. The character who matters | **nothing** |
| `person_6` | f, short brown, **white shirt**, sitting | the **colleague**, green dress, ponytail. She has **no name** in the story | `Choi Haeseon` at **0.9** |
| `person_2` | m, short brown, green sweater, sitting | a background extra, seated beyond the next window pane | `Lim Seonho` at **0.9** |
| `person_1` | m, short black, suit, standing | **nobody. A window frame** | nothing |
| `person_3` | m, short black, blue sweater, standing | background extra | nothing |
| `person_4` | m, short black, grey sweater, standing | background extra | nothing |
- `correctness.py`: tokenized `allowed`, `_ID_SHAPED` guard in `normalize_speaker`, interjection
stopwords, short-quote grounding skip
- `db.py`: `_STAGE_VISION_KEYS` and the strip pass in `clear_stage_data`
- `test_script_verify.py`, `test_correctness.py`, `test_db.py`: one case each
**Zero of the two bindings are right, and the one character who matters got nothing.** Both wrong binds
carry confidence 0.9.
Three separate defects stack here.
**1. The stored bbox coordinate space is wrong.** Consumed as absolute pixels, all six boxes land in the
top third of a 1650px-tall panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against
the panel's own dimensions, `person_2`, `person_3`, `person_4` and `person_5` fit their subjects tightly.
So the numbers are not pixels. Two places assert that they are:
- `worker_vision.py:271` prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
- `worker_identity.py:91` comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
Everything reading `bbox` is therefore cropping the wrong region. `worker_identity.py:200` embeds
`_crop_bbox(img, ch["bbox"])`, so `Choi Haeseon` at 0.9 was matched on a crop of the speech balloon's
edge and `Lim Seonho` at 0.9 on a crop of empty window frame. Wrong crops are mostly blank white, which
embed alike, which is a plausible mechanism for one row absorbing 25 assignments.
**Do not treat rescaling as the fix.** I claimed that and the user disproved it in one screenshot.
After scaling, `person_1` still sits on a window frame with nobody in it, and `person_6` is offset,
clipping the woman and running onto the dark frame. `worker_vision.py:38` already calls the box
"coarse, imprecise". Scaling buys roughly-right boxes, not right ones.
**2. Vision has no concept of extra versus cast.** Four of the six detections are background extras or
nothing at all. They are handed to identity as candidates on equal footing with the two people who
carry the scene. That also means the "113 detected people" figure that framed the whole roadmap was
never the right denominator, so "26 of 113 carry an identity" measured nothing useful.
**3. Identity mints a name onto a person who has none.** The colleague has no name in the story. She was
labelled `Choi Haeseon` at 0.9. Across the chapter `Choi Haeseon` holds 25 of 26 assignments, so in
practice that row is the label this pipeline stamps on any unnamed woman. This is the direct cause of
the user's 0:20 note, "Choi Haeseon when there's no Choi in the frame, it's the colleague", and of the
gender flips, since gender is read off whichever row got bound.
This is a cousin of invariant 6 in `CLAUDE.md`, which forbids minting a character from an unparseable
model answer. The missing rule is the same shape: **never attach a name to a detection that carries no
name evidence.** An unnamed recurring person needs a stable anonymous identity so the narration can
call her "the colleague" every time, rather than being forced onto a named row.
Partly checked, not finished: `match()` in `worker_identity.py:69` does abstain, returning `None` below
threshold, so the 0.9 came from cosine clearing the threshold on a garbage crop. Whether the Tier-2
gemma resolver can answer "none of these" was not verified. Check that first.
### Descriptions are not trustworthy either
`person_6` is "white shirt" for a woman in a green dress. `person_5` is "yellow sweater" for yellow
plaid, which is close enough. Any downstream rule keyed on appearance text inherits this.
## Registry, measured
`characters` is keyed by `manga_id`, not chapter. Two manga share the table. `d7104032` has 34 rows with
9 named. This chapter's `ef105a86` has 19 rows with 3 named. The `Kei`, `Zen`, `Kanade`, `Rico` and `K3`
rows belong to the other manga, so they are not polluting this chapter.
This manga's 19 rows, all `status=confirmed`, all `first_seen_panel=NULL`:
- named: `Choi Haeseon` (f), `Seonho` (m, aliases `["Lim Seonho","Seonho"]`), `Lim Seonho` (m)
- 16 rows with `name=NULL` and an empty alias list
`Seonho` and `Lim Seonho` are the same person in two rows. `Seonho`'s alias list contains the other
row's name. That is why either spelling matches two rows and binds nothing.
Assignments across the whole table, for scale:
| character | name | assignments |
| --- | --- | --- |
| `character_afa7623b` | Choi Haeseon | 25 |
| `character_bb79cfb4` | Kanade | 25 |
| `character_cfd34340` | Rico | 19 |
| `character_6f491712` | Lim Seonho | 1 |
In this chapter only `Choi Haeseon` (25) and `Lim Seonho` (1) appear.
Read those 25 together with panel 7. `Choi Haeseon` is not a character who appears 25 times. It is the
row that absorbs any unnamed woman. The 16 nameless rows are not a backlog of people waiting for names.
Some of them are background extras that should never have become rows, and at least one of them, the
colleague, is a real recurring person who correctly has no name and needs to keep it.
An earlier draft of this file said "naming is the ceiling, 16 of 19 rows need names". **That was wrong**
and it is corrected here. The ceiling is that identity cannot say "person, no name" and cannot tell an
extra from cast.
## The A/V bug, located but not fixed
Reproduced on the 49 real clips with `repro.py` and `probe.py` (see Open). Per-round probe with
`ASSEMBLE_BATCH=8` and 6 `fade_black` boundaries spread across batches:
```
r0 g0 n=8 XFADE in v= 49.44 a= 49.44 -> out v= 48.56 a= 48.64 lost_v=+0.88 lost_a=+0.80
r0 g1 n=8 XFADE in v= 47.04 a= 47.04 -> out v= 46.16 a= 46.23 lost_v=+0.88 lost_a=+0.81
r0 g2 n=8 XFADE in v= 48.00 a= 48.00 -> out v= 47.12 a= 47.19 lost_v=+0.88 lost_a=+0.81
r0 g3 n=8 XFADE in v=100.80 a=100.80 -> out v= 99.92 a= 99.96 lost_v=+0.88 lost_a=+0.84
r0 g4 n=8 concat in v= 64.18 a= 64.16 -> out v= 64.24 a= 64.26 lost_v=-0.06 lost_a=-0.10
r0 g5 n=8 XFADE in v= 44.00 a= 44.00 -> out v= 43.12 a= 43.17 lost_v=+0.88 lost_a=+0.83
r1 g0 n=7 XFADE in v=359.29 a=359.60 -> out v= 99.96 a=358.79 lost_v=+259.33 lost_a=+0.81
```
Round 0 is correct. Each group loses only the xfade overlap. **Round 1 loses 259s of video against 0.8s
of audio.** Its output video is 99.96s at 2499 frames, almost exactly the frame count of input `n3` on
its own, the 99.92s intermediate. The final video appears to carry the frames of one input.
The round-1 filtergraph is arithmetically correct, so this is ffmpeg behaviour, not offset math:
```
[n0][n1]xfade=transition=fade:duration=0.050:offset=48.510[v1]
[v1][n2]xfade=transition=fade:duration=0.050:offset=94.620[v2]
[v2][n3]xfade=transition=fadeblack:duration=0.600:offset=141.140[v3]
[v3][n4]xfade=transition=fade:duration=0.050:offset=241.010[v4]
[v4][n5]xfade=transition=fade:duration=0.050:offset=305.200[v5]
[v5][n6]xfade=transition=fade:duration=0.050:offset=348.270[v6]
```
### The strongest clue, found last
Re-running that chain by hand over only the **6** round-0 intermediates gives a correct 348.24s at 8708
frames, `rc 0`, no warnings. Round 1 collapses with **7** inputs, not 6.
The 7th input is the leftover 49th clip. With 49 clips and batch 8, round 0 makes 6 groups of 8 and one
group of 1, and `_assemble_batched` passes a lone group through un-encoded:
```python
if len(group) == 1 and not final_round:
next_items.append(group[0])
```
So the final xfade mixes 6 encoded intermediates with 1 raw clip. That passthrough is a **third** path
next to `concat` and `xfade`, and it is the prime suspect. Start here tomorrow. Confirm it by running
`probe.py` with 48 clips instead of 49, which removes the leftover entirely.
### What was tried and what it cost
Two paths exist in `_assemble_once`. A `concat` branch handles cut-only batches. An `xfade` branch
handles batches with a real transition. They disagreed on frame rate. The concat branch forced `fps=30`
while the xfade branch normalized nothing, and clips are 25fps. `436.39 / 363.63 = 1.2001`, exactly
`30/25`, which is what sent me down this path.
Working-tree changes to `worker_render.py`, all uncommitted:
- new `FPS = 25` constant. The three hardcoded `25`s and the one `30` now reference it
- the xfade branch normalizes every input with `setsar=1,fps={FPS}` into `[n{i}]` labels, matching what
the concat branch already did
- new `_stream_dur(path, kind)`. `_audio_dur` probes `format=duration`, which is `max(video, audio)`, so
it hides A/V drift by construction
- the `__main__` xfade self-check now assembles 4 clips through `_assemble_batched` with
`ASSEMBLE_BATCH=2` and asserts `abs(video - audio) < 0.25`. It previously asserted only
`getsize(out) > 0`, which is why this shipped
**A dead end worth not repeating.** I also pinned `-r FPS` on both output encodes. That made it worse.
The chapter collapsed to exactly 100.00s at 2500 frames, because forcing CFR on irregular input
timestamps drops frames. The comment already sitting at the concat branch warns about this. Both `-r`
flags were removed again. The in-graph `fps=` filter is the right normalization. The output `-r` is not.
The fps inconsistency is real and worth keeping fixed. It is **not** proven to be the cause of the
shipped 72.7s gap. The scene graphs hold 356 `cut` against 6 `fade_black`, so round 2 of the real run
most likely stayed on the concat branch, where no mixing occurs. Treat the fps work as necessary and
insufficient.
### The user's own hypothesis, which is the recommended direction
Two guesses, both worth following:
1. the assembly is wrong in an ffmpeg sense
2. there are two different paths, and they get mixed when there should only be one
Guess 2 matches the code. `concat`, `xfade` and the single-item passthrough are three paths, and
`_assemble_batched` feeds the output of one into the input of another. **Collapse it to one path.**
Normalize every input, then xfade every boundary, with `cut` as a 0.05s fade. `acrossfade` shortens
audio by the same amount that xfade shortens video, so A/V stays locked. The chapter then comes out
about 2.4s shorter than the sum of the clips, with both streams agreeing. That removes the branch
interaction instead of tuning it.
## Measured
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels.
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`.
- 113 orchestrator tests pass. `worker_vision.py` self-check passes.
- `dialogue` 116/116, `direct` 116/116, `scene` 116/116, `script` 116/116.
- speech lines 95, named 9. Multi-character panels 0 of 40 by design. Single-character 9 of 55.
- `speaker_method`: `unknown` 47, `model_solo` 28, `solo_prior` 8, `som_face` 7, `turn_taking` 5.
- unresolved name refs 3, all `brown ponytail, green dress`, from a neighbouring panel in the same
8-panel window. Was 24 of 51 before the fix.
- identity assignments 26, of which `character_afa762` "Choi Haeseon" holds 25.
- registry duplicates that block a correct bind: `seonho` matches 2 rows, `lim seonho` matches 2 rows.
- buckets during `tts`: `panels` 116, `raw` 79, `manga` 491, `audio` 1, `layers` 0, `video` 0.
- job `status=completed`, every stage at its unit count, finished `2026-08-11T20:08:16Z`
- it still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`)
- `chapter.mp4` 50MiB, video 436.392s, audio 363.675s, `r_frame_rate=25/1`,
`avg_frame_rate=63372800/2792909` which is 22.69, `nb_frames=9902`
- 49 clips, every one `25/1` exactly, sum video 363.63s, sum audio 363.60s. No clip has the two
differing by more than 0.05s
- scene-graph transitions: `cut` 356, `fade_black` 6
- `worker_render.py` `__main__` self-check passes on the current working tree
- workers up in tmux `manga-workers`, 9 windows
## Open
- **The run finished.** Every stage completed at 2026-08-11T20:08:16Z: `tts` 116/116, `layers` 116/116,
`render` 116/116, `assemble` 1/1. `s3://video/` holds 49 clips and a 50MiB `chapter.mp4` under
`ef105a86-.../7c944dd4-.../`, `s3://audio/` 49 objects at 32MiB. Nobody has watched the video.
Two honesty defects at the finish, recorded not fixed. `layers` reported `completed 116/116` with an
empty bucket, so no clip has parallax (`caveats/audit-open.md#layers-writes-nothing`). The completed
job still carries `error: "partial: 112/116 completed"` (`caveats/audit-open.md#stale-job-error`).
Read the state, or clear a stage and resume:
- **Finish the round-1 diagnosis.** The `_assemble_batched` tree turns 359s of video into 100s. This is
the worst defect found and it reproduces offline in about two minutes with no GPU. Suspect the 7th
passthrough input first.
```bash
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"<stage>\"}'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
.venv/bin/python <scratchpad>/probe.py # the per-round loss table above
.venv/bin/python <scratchpad>/repro.py # end-to-end verdict
```
Note: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`.
The scratchpad is session-scoped and will be gone. **Re-download the clips first:**
- Defects 5 and 6 landed after `dialogue` had already run, so this run's 7 `som_face` lines are still
labelled from the unpaired-mark path. The next dialogue pass fixes that. No name was affected.
- The audit script is now `audit_speakers.py` in this repo. It reads `/data/manga.db`, so it runs inside
the container: `docker cp audit_speakers.py manga-orchestrator:/tmp/ && docker exec
manga-orchestrator python3 /tmp/audit_speakers.py`.
- The 3 cross-panel unresolved refs need the window's whole present-list, not one panel's.
- Workers were restarted twice this session and are running in tmux `manga-workers`. Nothing watches
them, and nothing watches the homesrv containers.
```bash
/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/
```
Both scripts are worth committing next session. They are the only check that has ever caught this.
- **Do not trust `format=duration`.** It returns `max(video, audio)`, so every existing duration assert
in `worker_render.py` is blind to drift. `_stream_dur` exists now. The other asserts still use
`_audio_dur`.
- **Identity, in the order the panel 7 evidence implies.** First, settle the `bbox` coordinate space and
fix every consumer, since nothing else can be judged while crops are wrong. Second, let identity
abstain and hold a stable anonymous identity, so the colleague stays "the colleague". Third, separate
extra from cast so extras never reach identity. Only then merge `Seonho` into `Lim Seonho` and split
`character_afa7623b`, which still needs the reversible-merge design
(`caveats/audit-open.md#destructive-reconcile`).
- **Verify the bbox space before changing anything.** Two independent claims in the code say pixels, and
the art says otherwise. Confirm what the model was told and what it returns, rather than trusting
either comment. Then check whether the crop is the only consumer, or whether SoM marker placement and
the face-pairing in `worker_vision.py:57` read the same numbers.
- Re-derive the panel 7 overlay when needed. It took one `mc cat` of the crop plus a Pillow script, and
it found more than any query did:
```bash
/usr/bin/ssh kami@192.168.1.104 'mc cat homesrv/panels/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/panels/p006.png' > p007.png
```
Draw each `bbox` twice, once as pixels and once divided by 1000, then look at it.
- `mc` aliases on homesrv: use `homesrv` or `mio`, not `local`, which returns Access Denied. `rfs` is
the empty rustfs.
- Plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`.
- `cp` is aliased to `cp -i` in this shell and hangs on overwrite. Use `/usr/bin/cp -f`.
- The Bash tool's default timeout is 120s no matter what `timeout` the command itself carries. Pass the
tool's own timeout or background the run. Otherwise a restore step after a mutation test never runs,
which left a deliberately broken `worker_render.py` on disk once this session.