manga-recap-pipeline/NEXT.md at 8113bdfc8b9030dfd2dbf8f327e66c4b2e58b2cc

Files

T

kami 8113bdfc8b Fix the real A/V gap: a stream copy across mixed frame rates

The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.

collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.

Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.

The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-12 12:16:22 +04:00

9.5 KiB

Raw Blame History

Updated 2026-08-12. What this session did is in HANDOFF.md.

State

The chapter runs end to end. The A/V sync defect is fixed and chapter.mp4 is rebuilt: video 364.120s against audio 364.122s at 25/1. The identity defects are still in the output.

Job 778297bc-e7ce-439d-91b5-8a027060d17f, chapter 7c944dd4-e972-42c7-ba60-9f6939548e80, 116 panels, status=completed, finished 2026-08-11T20:08:16Z. s3://video/ holds 49 clips and a 50MiB chapter.mp4. The user watched it and read out 19 defects. They are grouped by cause in HANDOFF.md.

One number sets the agenda:

Panel 7 checked against the art has zero correct identity bindings out of two, and the one character who matters is unbound. HANDOFF.md#panel-7-walked-against-the-art has the table. The coordinate cause is fixed. The registry built on it is not.

Re-run vision and identity. The bbox space is settled and converted at /vision (decisions/identity-bbox.md#bbox-is-normalized). Every stored box, embedding and ref_image_uris in the registry came from the wrong space. The fix changes nothing until those stages run again. This is GPU work and needs the user's go-ahead. Clear vision and everything downstream of it, or accept that the boxes in the database stay normalized while new ones are pixels.

Watch two things on the rerun. Whether som_face still returns unknown on every face, since gated pairing was comparing pixel face boxes against 0-1000 character boxes. And whether Choi Haeseon still absorbs every unnamed woman, which is item (b) below and independent of the crops.

Smaller follow-on: nine _audio_dur calls in worker_render.py measure finished clips with format=duration. So the durations reported to the orchestrator are blind to per-clip drift. They position no filter, so invariant 9 does not cover them. Worth converting to _stream_dur.
Fix identity, in this order. Panel 7 is the worked example and HANDOFF.md#panel-7-walked-against-the-art carries the evidence. Do not start at the registry.

a. ~~Settle the bbox coordinate space.~~ Done 2026-08-12, proven over all 113 detections and checked by eye on panel 7, where five of six converted boxes land on their subject (decisions/identity-bbox.md#bbox-is-normalized). person_1 still frames an empty window mullion, which is (c). b. Let identity abstain and stay abstained. The colleague has no name in the story and was labelled Choi Haeseon at 0.9. An unnamed recurring person needs a stable anonymous identity so narration says "the colleague" every time. match() already returns None below threshold. Check whether the Tier-2 gemma resolver can answer "none of these"; that was not verified. c. Separate extra from cast. Four of the six detections on panel 7 are background extras or nothing at all, and all six reach identity as equal candidates. d. Only then merge Seonho into Lim Seonho and split character_afa7623b, which still needs the reversible-merge design (caveats/audit-open.md#destructive-reconcile), not a patch.

Cast profiles already exist. Do not rebuild them. The user asked whether the main cast could get a profile built from reference frames and reused. characters already carries ref_image_uris and embedding_uri, and all 53 rows have both populated. The mechanism is not missing, it is enrolled from the wrong crops, so today it stores references to balloon edges and window frames. Step (a) is what makes it work. Three things are genuinely absent and are the smaller follow-on:
- no quality gate on enrollment, so nothing checks that a reference crop holds a face at all
- nothing re-enrolls a reference set once it is written, so the wrong crops persist
The visual "is this them?" check is already built. Do not write it again. /vision/resolve at worker_vision.py:963 sends the query crop plus up to 3 labelled reference images per candidate. build_resolve_prompt tells the model to judge face shape first, to treat hair and outfit as secondary, that two people sharing a hair colour are not the same, and to answer 0 for NONE when unsure. choice: 0 becomes a new character, an out-of-range index becomes unresolved, and ref_image_uris is republished as reference_image_uris at worker_identity.py:152 and :161. The mechanism, the prompt and the abstain path are all correct. They are fed crops of the wrong region, which is step (a).
- no human gate to name, merge or split the clusters. The user wants this as a minor adjustment on top, not as the mechanism. The gates table and the review gates from [#136] are the place to hang it
The chibi at 1:35 will survive all of this. He genuinely is brown hair plus a yellow shirt, so a profile match is correct on appearance and wrong on reality. That needs item 4 below, plus requiring a real face before a crop can enroll.
Stop the narration inventing facts. 0:43, 2:03, 2:05 and 2:15 assert things no panel shows. The correctness verifier passed 116/116 because it checks quotes and names, never invented claims.
Teach vision that art inside a panel is not the scene. A chibi on a monitor became "a man holding a drink" at 1:35. A colleague pointing into the distance became "pointing towards the screen" at 1:59.
layers writes nothing and reports completed 116/116, so no clip has parallax and a still holds for 28s from 2:24 (caveats/audit-open.md#layers-writes-nothing).
Clear the stale job error. The completed job still carries error: "partial: 112/116 completed" (caveats/audit-open.md#stale-job-error).
Balloon-to-speaker geometry via the unused det/seg heads (caveats/speaker-attribution.md#tail-is-not-geometry) is now behind item 2. With no name to attach, geometry buys nothing.
Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3 unresolved refs describe a neighbouring panel in the same 8-panel call.
Start Phase 2 from ROADMAP.md. Set SQLite busy_timeout before any concurrency work (caveats/audit-open.md#sqlite-locking).

Lesson worth keeping

Every metric recorded before this session said the pipeline was fine or nearly fine. script 116/116, "9 named speech lines", layers 116/116, assemble 1/1. Watching two and a half minutes of output found a 20% sync failure, a cast that is 84% anonymous, invented narration, and a stage that writes nothing while reporting success. Stage counters measure whether code ran. They say nothing about whether the result is correct. Watch the output before trusting a number.

Running the pieces

./start_workers.sh              # session_manager + 9 workers, each a uvicorn in a tmux window
tmux attach -t manga-workers    # per-worker logs
.venv/bin/python worker_render.py    # self-check, runs real ffmpeg, about 4 minutes

Read the state, or clear a stage and resume:

/usr/bin/ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"<stage>\"}'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"

Traps: plain ssh is the kitty ssh kitten and refuses non-interactive stdin, so use /usr/bin/ssh. mc aliases on homesrv are homesrv and mio. local returns Access Denied and rfs is the empty rustfs. cp is aliased to cp -i and hangs on overwrite, so use /usr/bin/cp -f.

Re-fixing assembly needs the real clips, which the session scratchpad no longer holds:

/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/

Storage and viewer, tasks #116/#117

[#117] is done. stowage serves the manga buckets. It was never a MinIO problem: the container had been dead since 2026-07-19 on an arm64 digest pin.

[#116] is closer but not cut over. Artifacts split one bucket per class (decisions/storage-layout.md#bucket-per-artifact), and both MinIO and rustfs hold all six buckets. rustfs on 127.0.0.1:9010/9011 is still empty and nothing is repointed, so MinIO serves every read and write. Remaining: mc mirror the live buckets, verify counts and sizes, then decide on cutover (decisions/storage-layout.md#rustfs-staged).

Two containers on homesrv had been dead for two weeks and now run. manga-fetch is the one /job/create needs. manga-web is what manga.kvmx.ru proxies to on 8083. Nothing watches them, and nothing watches the workers.

Open questions

Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for: the speaker contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only [#203] existed and is closed by decisions/audit-phase1.md#unlocked-model-load.

Three audit items are deliberately not done and are recorded as caveats rather than silently dropped: honest stage clearing, ComfyUI under the session mutex, and reversible identity merges. Each needs a design decision, not a patch.

Carried over from the reconstruction: .venv needs the ROCm torch wheel reinstalled, and dots.tts/, legacy/, RESUME_SPEC.md, pipeline-design-notes.md, spec-v2.md are unrecoverable.

9.5 KiB Raw Blame History

NEXT