manga-recap-pipeline/spec-v3.md

this is a v3 delta spec: it lists what changes on top of the working v2 pipeline (see spec-v2.md). v2 is validated end-to-end (a full chapter renders to one narrated mp4). v3 is about quality of the narration and the look of the video, plus panel curation. structured for hand-off to an agent.

manga recap v3 — quality + look

0. what v2 does today (baseline)

• one narrated chapter.mp4 per chapter: fit-on-black panels, uniform ken-burns zoom, burned subtitles, hard cuts, single default TTS voice.
• identity assigns a stable character_id per body (siglip2), but characters are unnamed, so the script prints "Person A/B". the script worker sees one panel at a time (no memory).
• every panel is narrated (no curation).

---

A. cross-panel identity & continuity [biggest quality lever] [DONE]

SHIPPED:

• descriptive frozen labels: worker_scene emits label per character (name -> frozen registry appearance via _describe, "the one with black hair and glasses" -> Person X fallback); worker_script _name_map consumes it. no "Person A" when appearance is known, char_id preserved.
• chapter_context populated: service.py _chapter_context feeds a cast+entities roster to every script call so the narrator has memory (kills "Unknown voice", item 11).
• name acquisition + backfill (item 12): vision emits name only when a char is explicitly named; service.py scene stage binds it via set_character_name (first name wins, only if unnamed); script stage resolves display names from the FINAL registry (names_by_id) so a name learned in ANY panel backfills to every panel's narration. ceiling: vision name-inference can misattribute; first confident name sticks. upgrade path = roster seed for higher precision. STILL TODO (small, deferred):
• true per-panel story-state (only what's been seen so far) for "establishing shot once / don't re-narrate repeats" (the C-todo) — currently whole-manga roster.

--- original plan --- one feature: carry a running story-state through a chapter so narration has memory. covers former items 4, 11, 12.

• stable anchor, mutable label. the character_id is never lost (already stable via siglip2). only its display label changes:
  • name known -> the name.
  • name unknown -> a descriptive phrase ("guy with black hair and glasses", "girl with short hair holding a phone") derived ONCE from vision's appearance data and frozen per character_id, reused across all panels. never regenerate per-panel (labels would drift).
• name acquisition + backfill. when dialogue addresses a character by name and speaker attribution is confident, bind name -> character_id and switch the label to the name from then on (optionally backfill earlier references).
• story-state threaded per chapter. accumulate {cast seen (id,label,name?), entities/props seen (the cat)} and feed it into scene+script (populate the currently-empty chapter_context). this is what lets the narrator say "he calls out to his cat" instead of leaking a bare name (item 12), and removes "Unknown voice" (item 11).
• optional roster seed (not required). a per-manga character list (names, maybe ref art) makes naming + cross-panel matching more reliable than dialogue inference. graceful-degrade without it; add when available.

touches: worker_vision (appearance -> label), worker_scene / orchestrator (story-state store + speaker continuity), worker_script (consume context), orchestrator characters table (label, name).

B. narration voice & style (items 2, 3) [DONE]

• less descriptive gemma (item 2): DONE. prompt now demands TIGHT recap (1-2 sentences, ~35 words, "no purple prose"); max_tokens 512->160, temp 0.7->0.6 (worker_script).
• one fixed voice (item 3): DONE. dots.tts samples a RANDOM speaker per call with no reference, so the voice drifted. worker_tts now bootstraps one seeded reference clip (~/.cache/manga-tts/ narrator_ref.wav) and clones from it every synth. delete the file to reroll; swap REF_WAV/REF_TEXT for a hand-picked voice.
• multi-voice per character: future, not v3 (user: "later") — same mechanism, per-character ref clips.

C. panel curation (item 10) [DONE — basic]

• skip panels with no narrative content (pure scenery / transition art); show an establishing shot once, don't re-narrate repeats. vision flags "scenery-only / skip". big pacing + cost win; do early because it shrinks every downstream stage.
• SHIPPED: vision emits skip:true for no-narrative panels (worker_vision prompt+schema); the script stage honors it (service.py run_stage_script) -> no narration -> cascades to no audio, no clip. Single chokepoint, no new db columns (skip rides in the vision result_json blob).
• STILL TODO (needs cross-panel memory, folds into A): "establishing shot once, don't re-narrate repeats" — requires dedup of repeated scenery across panels. deferred to story-state.

D. render / look overhaul (items 1, 5, 6, 7, 8)

all in worker_render.py (self-contained ffmpeg builder), except #6.

• #1 dynamic clip duration — DONE. _audio_dur probes narration length via ffprobe; scene_timing kept only as fallback. no more 4.0s sawtooth / vanishing subs.
• #5 blurred-panel background fill — DONE. filter_complex: cover-scaled + boxblur copy behind the fitted panel (replaces black bars).
• #7 subtitles — DONE. boxed caption: bold white on translucent black box (BorderStyle 3), lower-center, DejaVu Sans 52. (user chose boxed over outline/minimal.)
• #8 content-aware motion — DONE. vision picks a camera.effect per panel from the fixed vocab (+ to=[x,y] for dolly_to_subject); it rides the vision result_json -> scene_graph.camera -> render. worker_render _motion(camera, frames) maps the vocab to a zoompan z/x/y expression (static/zoom/pan/dolly/shake/orbit); default zoom_in = old ken burns. orchestrator render stage reads scene_graph.camera. self-checked (ffmpeg renders pan + dolly clips).
• #6 multi-panel composites + non-generic transitions [the ambitious one, sequenced last]
  • non-generic transitions — DONE. vision emits transition (cut|crossfade|dissolve| fade_black|fade_white|wipe_left|wipe_right|push) per panel -> scene_graph.transition -> orchestrator assemble collects a per-clip list -> worker_render _xfade_chain builds an xfade+acrossfade graph (audio kept in sync, overlap clamped to clip length). all-cut/empty keeps the fast stream-copy concat (no re-encode); any real transition re-encodes the chapter. self-checked (two clips fade_white into one mp4). transition = OUT of that panel.
  • multi-panel composites — DONE (basic). grouping source = new pass (grouping.py, deterministic): consecutive SMALL panels (area < 0.7*page-max, same source page, capped at MAX_GROUP=3) share ONE shot. render + assemble both call plan_groups so their grouping matches. a composite = vertical stack of the members on a W×H still; each panel keeps its own script+tts (per-panel stages unchanged); the audios play in sequence while a moving highlight marks the active row ("swap the front panel while audio plays"). clip keyed by the group LEADER; assemble takes one clip per group and the group's exit transition = last member's. solo groups render exactly as before, so blast radius is only genuinely-small adjacent panels. knobs: MAX_GROUP, SMALL_FRAC. ceilings: vertical-stack layout only (no grid/overlay/front-swap animation beyond the highlight), size+adjacency grouping (no narrative-beat grouping), black gutters (no blurred fill in composites). upgrade paths noted in code.

D2. beat pacing, timed cues, subtitle redesign, collage (vikunja 185/187/188/183)

scene-level narration made the beat (not the panel) the render unit; these refine how a beat plays. all in worker_render.py + collage.py, orchestrator run_stage_render.

• 185 content-aware panel timing — DONE. a beat's one narration used to split EQUALLY across its member panels (D/n). now the orchestrator scores each member from its scene graph (dialogue amount, action density, camera emphasis: _beat_weights) and passes bounded weights to render; worker _beat_slices(D, n, weights) distributes screen-time by weight with a per-panel floor (no flashing) and a deterministic equal-split fallback (degenerate/short weights). self-checked.
• 187 timed subtitle cues — DONE. the whole narration paragraph used to show from frame 1, leaking later-panel lines early. cue_plan splits the flowing narration into sentence cues, times them on the SAME [0,D] timeline as the panel slices (length-weighted + readable floor, reused _beat_slices), maps each cue to the panel on screen at its start, and clamps a cue never to precede its panel. per-cue ASS events (not one paragraph). cue plan returned + persisted to clips.cues_json for the review UI / collage. no-TTS-alignment fallback = punctuation/length estimate. self-checked ("Hold up." can't appear before its panel).
• 188 subtitle redesign — DONE. _ass_multi now takes mode (off|minimal|boxed, env SUB_MODE, default minimal) + orientation presets (portrait/landscape safe-areas). smaller type (40/32 vs 52), minimal = thin outline + soft shadow (no box), boxed = restrained ~50% box. _wrap2 caps every cue at ≤2 short lines (ellipsis truncation) so it never regrows into a paragraph. _sub_align dodges the subject vertically using the director focus point (camera.to.y low -> caption to top). ceiling: no real face/bubble CV yet — camera focus is the only subject signal wired.
• 183 animated manga collage — DONE (core + wired behind COLLAGE flag). collage.py plan_layout is a pure deterministic planner: ordered panel aspects + RTL + dominant + frame -> template (centered_wide / vertical_pair / stacked_wides / strip_over_dominant / supporting_left_dominant_right / quad), aspect-correct resting rects (no stretch), entrance vectors (RTL slide / dominant scale), z-order, hold/transition timing. worker_render collage_cmd renders it: blurred plate of the dominant panel + sharp aspect-fit panels with restrained drop shadows, non-dominant panels sliding into place over ~0.45s, crisp holds; reuses 185 slices + 187 cues + 188 subtitles. orchestrator run_stage_render calls it when COLLAGE=1 (else the ken-burns montage). ceilings (QA-tuned against the reference video, no still to eyeball here): transition-only directional/radial MOTION BLUR not applied (clean slide/scale); per-panel hold DRIFT omitted (static hold); beat-replace "streak" rides the assemble xfade. planner + renderer both self-checked; visual acceptance is the QA run.

D3. identity tracklets + review gates (vikunja 160/136)

• 160 tracklet spine — DONE (code, QA-pending). the Tier-2 gemma decider (identity phase-2) used to resolve identity once PER CROP, so one person got adjudicated repeatedly and could get two answers on two panels. tracklets.py link_tracklets (pure, union-find, self-checked) first LINKS the per-panel siglip shortlists into per-person tracklets over a locality window — hard gender gate + appearance-token overlap + shared-candidate — then the resolver runs gemma ONCE per tracklet against the UNION of the members' shortlists (the person's gallery, consensus-ordered) and applies that one character_id to every crop in the tracklet. fewer gemma calls, one answer per person. ceiling: gemma NONE (cosine over-merged onto an existing char) still isn't split to a fresh id — needs a 3rd siglip re-embed pass; add if the NONE-rate is high.
• 136 review gates — DONE (code, QA-pending), off by default (GATES=1). two human sign-off gates in the pipeline loop: gate script pauses BEFORE tts/render, gate audiovisual pauses BEFORE assemble. When enabled and unapproved the job goes awaiting_review and the loop returns; /review/approve records the gate (gates table) and relaunches _run_pipeline (skips completed stages → resumes at the gated stage). Gate-1: /review/scripts shows narration per panel grouped by beat + cheap textual flags (review_flags.py: generic-handle / repeated / empty, self-checked; semantic flags — spoilers, altered meaning, attribution — left to the human eye, ceiling noted). Gate-2: /review/audio serves the per-panel wav so the reviewer actually listens; /review/retts re-synthesizes ONE panel's TTS + re-renders just its beat clip (delete_clip forces it) and clears the AV gate. web UI (index.html review tab): gate-approve buttons + status, per-panel audio player, flags badge, re-synthesize button. reuses existing /review/panels + /review/script + /review/preview. ceiling: no panel-to-cue drag-edit UI (cues persisted in clips.cues_json, edit endpoint not built); flags are textual only.

E. reference

generic manga-recap youtube style (single narrator, static panels w/ slow pan/zoom, minimal text, dramatic pacing) — but with less generic transitions (feeds #6). #6 goes beyond the reference.

---

sequencing (proposed)

1. C panel curation — shrinks everything downstream first.
2. A identity/continuity — most changes how it feels (names + memory).
3. B voice+style — small, high impact.
4. D basics (#1,#5,#7,#8) — most changes how it looks, low risk.
5. D#6 compositing — last; highest risk/effort, isolated to render.

direction schema (decided — enables #8 and #6)

narration (text->TTS) and direction (camera/transition/layers->render) are separate concerns from the SAME scene graph. VISION picks the effect (it sees the panel + has char bboxes for focus points) and emits a direction block into the scene_graph JSON; RENDER just executes it. fixed vocab:

• camera.effect: static | zoom_in | zoom_out | pan_left | pan_right | pan_up | pan_down | dolly_to_subject | orbit | shake | hold (+ from/to [x,y] normalized, easing)
• transition.type: cut | crossfade | fade_black | fade_white | wipe_left | wipe_right | push | dissolve (+ duration)
• layers: {artifact, motion} (parallax, when the depth renderer is wired) not built yet; lands before #8 (needs camera) and #6 (needs transition).

open design points

• roster seed format if/when available.
• B#3: which fixed dots.tts reference voice to pin for the whole video (needs a pick).

---

v3.1 delta — identity is broken; add human-in-loop review

evidence (last job, 116-panel Teto X Egen, from manga.db)

• 53 character rows for ~8 actual people. "Seonho" stored 5× (ids 85ad36b0/811bf0d7/0033ff7f/ bbddf20c/90269899), Egen Girl 2×. ~40 rows have a single identity_assignment (one-time NPCs).
• root cause: siglip2 over-splits. same person across panels (pose/expression/B&W line art) doesn't clear cosine 0.85, so nearly every appearance spawns a new character_id.
• cascade: names never propagate (panel 10 names id A, panel 51's Seonho is id C -> names_by_id can't map it -> narration falls back to "the woman with straight dark hair", flips gender to "her"). re-introductions + DB/S3 bloat all follow from the split.
• concrete bug: worker_identity.py create call hardcodes "name": None — vision reads the name but identity discards it at registration. db.create_character is a blind INSERT, zero dedup.

E1. pipeline decomposition [DECIDED — the vision mega-call splits]

Reason: one gemma4 call doing 4 jobs (detect + dialogue + direction + scene) does each worse, and there was nowhere to run a real cross-panel merge. Split the jobs; insert a merge stage where the model actually compares two crops.

Final stage order (OCR removed — gemma4 transcribes bubble text image-natively; a roster prepass now runs after crop, see below):

fetch → crop → roster → vision → identity → reconcile → dialogue → direct → scene → script → tts → layers → render → assemble

• vision (per panel, image call) — character DETECT only: local_id, bbox, appearance (hair/clothing/features), gender (m|f|unknown), emotion, action. plus skip + scene (location/time). the only stage that must precede identity (identity crops its bboxes).
• identity (per panel) — siglip assigns local_id → character_id. on a collision (siglip ambiguous OR hair+clothing appearance overlap OR name collision) it records a merge-candidate pair instead of silently minting a twin. still creates provisional ids; reconcile prunes them.
• reconcile (NEW, once per chapter, after all panels) — the second vision pass. for each flagged candidate pair, POST worker_vision /vision/same with the two ref_image_uris crops -> gemma4 "same character? {yes|no, confidence}". confirmed -> db.merge_characters(loser->keeper): repoint identity_assignments, union name/aliases/ref_image_uris, delete loser row. loser ref crops are PRESERVED on the keeper as gallery views (not deleted from S3 — deleting them left the keeper's own uris dangling). cast is CLEAN + named from here on. only flagged pairs get a call. lives as run_stage_reconcile in service.py + /vision/same on worker_vision (reuses warm gemma4).
• dialogue (per panel, image call) — gemma reads bubble text straight from the panel image (no external OCR): bubble classify (speech/thought/shout/narration/sfx), merge split bubbles, text cleanup (sentence case, fix names/proper nouns, sfx literal), speaker attribution (speaker=local_id + confidence), entities. prompt gets the CLEAN named cast present in the panel as context -> better attribution ("Seonho (m) vs Haeseon (f)").
• direct (per panel, off scene understanding) — camera.effect (+to) and transition from the fixed direction vocab (unchanged schema, section D/direction-schema). runs after identity so it can weight who's present. may glance at the image or work from the scene graph — cheap.

new workers/endpoints: worker_vision becomes detect + /vision/same; new worker_dialogue.py and worker_direct.py (or /dialogue + /direct endpoints — same warm gemma4, decide at build like reconcile). db: STAGES gains reconcile,dialogue,direct; run_stage_* for each; session_proxy call_* wrappers. the old omni-vision prompt/schema is partitioned across the four prompts.

F. identity overhaul [rides on E1]

Goal: 53 rows -> ~12; named recurring cast stays one id; NPCs never persist.

• F1 merge = the reconcile stage (see E1). the fix for the hardcoded "name": None bug: vision/ identity now carry name+gender+appearance into create_character so provisional rows are named/typed; reconcile does the actual dedup with the model comparing crops (not a blind string match). db.merge_characters + candidate-flagging in identity.
• F2 gender field. add gender TEXT to characters schema. vision emits gender. worker_script _name_map feeds it to the model ("Seonho (he)") for correct pronouns — kills "her bold personality" flips even when the name resolves.
• F3 confirm-before-persist. hold unnamed provisional embeddings in a per-session cache; only write a DB row + S3 crop/npy when an embedding is seen >=2x OR is named. the ~40 one-timers never touch the bucket. session_id already flows into identity. (complements reconcile: reconcile merges the twins that WERE created; F3 stops most from being created at all.)
• F4 centroid + re-rank matching. store multiple refs per char, match against the accumulated centroid not one arbitrary embedding; break siglip ties by name/gender agreement. ceiling: perfect auto-ID on B&W is not achievable — reconcile+F4 get ~80%, the UI (G) closes the rest. don't over-invest.
• DECIDED: start clean. no migration script. wipe manga.db + the minio manga bucket before the first v3.1 run (destructive ops step, run at build time). fixed logic applies to fresh data.

G. review / edit UI (replaces the auto CoT-strip idea, item C-bleed)

Rationale: the human reviews the final video and knows exactly what's wrong and where. Give them a synced editor instead of chasing every failure through the pipeline. ~80% of infra already exists in the orchestrator: static/index.html dashboard, /job/video (streams mp4), /db/tables + /db/table, /stage/run + /stage/clear (re-render), set_character_name.

• timestamp -> panel is trivial: clips concat in panel_order, so panel start offset = cumsum of prior panels' audio.duration. store the offset at assemble time (or compute on the fly) as an offset table the page binary-searches on <video> timeupdate.
• view: <video> + side panel that, for the current panel, shows editable: script text, assigned characters (name / gender / appearance), camera.effect, transition.
• edits -> small endpoints (most exist): add set_gender, merge_characters(ids[]->keep), set_script(panel_id,text), set_camera(panel_id,effect). merge_characters is where the Seonho mess gets fixed in seconds — and the merge is ground truth that feeds F.
• re-render: "re-render from stage X" button = existing /stage/clear + /stage/run on the affected panels/clips. effect preview = dropdown of _motion vocab + XFADE map, re-render one clip.
• lazy ceiling: no DAW timeline, no non-destructive edit graph. "video + synced editable form + re-render button" over existing data/endpoints. ~1 day. highest-leverage addition: turns every future quality bug into a 30-second manual fix.

v3.1 sequencing

1. E1 decomposition — split vision into vision/dialogue/direct, add reconcile + direct + dialogue to STAGES with stage runners. Foundation; F1 lives inside it.
   • 1a. vision -> detect-only (+ gender, F2). 1b. dialogue stage. 1c. direct stage.
   • 1d. identity candidate-flagging + db.merge_characters. 1e. reconcile stage + /vision/same.
2. F3 (confirm-before-persist) — kills NPC bloat.
3. G (review UI) — builds on the cleaner DB; closes the residual reconcile/F4 can't.
4. F4 (centroid/re-rank) — only if G shows auto-ID still too noisy to be worth hand-fixing.

Build order note: each of 1a–1e is independently deployable/testable (workpc workers reload via tmux; orchestrator needs a container rebuild). Do a full clean re-run after 1e before starting G.

D#6 revisited — camera-traversal composite [DONE 2026-07-14]

the vstack composite was the wrong look; target = camera moving across panels in their REAL page positions (motion-comic page traversal, partial-neighbor reveals). geometry already exists: panels store bbox + page_index + panel_order, source page is retained. composite feeds a constant 9:16 crop-window that HOLDS on each member then quick-eases (0.5s) to the next (traverse_cmd, _pan_expr in worker_render.py). page uri is reconstructed from page_key(...) (deterministic — no schema change). vstack kept as the fallback when bbox/page_uri are missing. orchestrator TODO: run_stage_render composite branch must pass each member's bbox + one page_uri into call_render_composite.

E2 — chapter roster / context pre-pass [SPEC — not built]

problem. every stage sees one panel at a time, so nobody has a chapter-level view. symptoms in the wild: the MC is "the man in the yellow shirt" at panel 7 but named at 10 (name learned too late to backfill cleanly); unnamed extras get re-described every panel; bubble-only panels (p005) have no cast to attribute a line to; the narrator's tone/naming drifts. _chapter_context today is just the DB registry accumulated so far — partial and late.

idea. one cheap gemma4 pass up front that reads the pages (text image-natively) and produces a chapter view that grounds the per-panel stages. it does NOT replace vision/identity — it's coarse context only, threaded back as NAME hints (never identity evidence).

stage. new roster stage, placed after crop, before vision. names surface in captions/ dialogue on the page images themselves (e.g. "Lim Seonho, Designer at Everyday"). one artifact per chapter, stored on chapters.roster_json, resumable (skip if present), non-blocking on failure.

method (implemented). one multi-image /roster call over an even spread of ~6 chapter pages (ROSTER_SAMPLE_PAGES) -> {premise, characters:[{name,aliases,gender,species,description}]}. threaded into: vision detect (known_characters hints, registry wins), dialogue (off-panel cast for bubble-only panels), script (_chapter_context premise). ceiling: sampled pages; widen the sample if recall bites.

output (stored per chapter, e.g. a chapter_roster blob):

{"premise": "3-5 sentence chapter setup, present tense, no spoilers ahead",
 "roster": [{"name":"Lim Seonho","aliases":["Seonho"],"gender":"m",
             "appearance":"short brown hair, glasses, yellow shirt",
             "role":"designer at Everyday; the MC"}]}

consumers (all already have the hook, just feed them the roster instead of the thin registry):

• vision (detect): pass roster as known_characters so detections carry a provisional name.
• identity/reconcile: seed name labels — a face gets its roster name on FIRST sighting, killing "named at 10, described at 7". embeddings stay authoritative for who-is-who; roster only supplies the label when appearance matches.
• dialogue: roster = the known cast for off-panel attribution (the p005 case now has names/handles to point at instead of "".)
• script: seed brief from premise at panel 1 (no cold start); roster names -> names_by_id + introduced from the start (stable handles, no re-description).

caveats. roster names are a COARSE read → HINTS, not ground truth: apply a roster name only when a detected/identified character's appearance matches a roster entry; never override an embedding match. one extra pass per chapter (latency/cost) — acceptable, it's once per chapter, warm session.