manga-recap-pipeline/manga-recap-pipeline-spec.md

Manga/Manhwa Recap Pipeline — Technical Spec

Status: draft v1 Target: fully local, single-GPU, resumable batch pipeline. No publishing (copyright out of scope). Core constraint: no new PyTorch / ROCm wheels. Existing llama.cpp (gemma) stays. Everything else runs on onnxruntime + opencv + numpy + scikit-learn.

---

1. Goal

Given raw manga pages or manhwa strips for a series, automatically produce a narrated recap video with burned-in subtitles, with minimal human involvement (one-time character-bank setup per series).

This spec replaces a dependency on the magi/magiv2 model by decomposing its subtasks into swappable, low-dependency components.

---

2. Design principles

• Dependency isolation. gemma runs on llama.cpp (unchanged). All CV/ML runs via ONNX Runtime. Detection models are small enough to run on CPU EP; ROCm EP optional. No transformers, no second torch install.
• Specialists over generalist. Do not ask one VLM to detect + identify + attribute. Each subtask is a dumb specialized component; a clean data bus (JSON files) connects them.
• Resumable. Every stage reads its input artifact from disk and writes its output artifact to disk. A stage is skipped if its output exists and is newer than its input (unless --force).
• Supervised shortcut for identity. Replace magi's unsupervised character clustering with a per-series character bank (exemplar crops + names). Simpler, more robust, and how magiv2 gets names anyway.
• Fail loud, fail per-panel. Low-confidence panels are flagged, not silently guessed. Confidence thresholds are config, not magic numbers in code.

---

3. Architecture overview

ingest → [1] panels → [2] detect → [3] identity → [4] text
                                                      │
                          [5] speaker-bind ←──────────┤
                                                      │
                          [6] filter essential ←──────┘
                                    │
                          [7] transcript (ordered)
                                    │
              ┌─────────────────────┴───────────────────┐
        [8] scene-action (gemma)                   (transcript.json)
              └─────────────────────┬───────────────────┘
                                    │
                          [9] script gen (LLM, chapter + rolling summary)
                                    │
        [10] TTS (dots.tts) ──► [11] visual assembly (ffmpeg) ──► [12] subs (faster-whisper) ──► [13] mux

Stages 2–7 are the magi replacement. Stages 8–13 are the existing recap backend.

---

4. Filesystem layout

work/
  <series>/
    source/            # input pages/strips
    ch<NN>/
      panels/          # panel crops, ordered
      panels.json      # [1]
      detections.json  # [2]
      identities.json  # [3]
      texts.json       # [4]
      transcript.json  # [5][6][7] merged
      script.json      # [9]
      audio/           # [10] wav per segment
      video.mp4        # [13] final
    bank/
      bank.json        # character bank (per series, hand-built once)
      crops/           # exemplar images

Artifact-per-stage = resumability. Delete an artifact to re-run that stage forward.

---

5. Stage specs

[1] Panel extraction

• Manga (page-based): kumiko → ordered panel polygons. Pure OpenCV.
• Manhwa (vertical strip): slice at horizontal whitespace bands. OpenCV: row-wise background uniformity → cut points. Produces pseudo-panels.
• In: source/* · Out: panels/, panels.json
• Dep: kumiko, opencv. No torch.

// panels.json
{
  "type": "manga",            // or "manhwa"
  "reading_order": "rtl",     // rtl | ltr | ttb
  "panels": [
    { "id": "p001", "page": 1, "bbox": [x,y,w,h], "file": "panels/p001.png", "order": 0 }
  ]
}

[2] Detection (text + balloons + characters)

• Text + balloons: comic-text-detector (ships ONNX). Returns text regions + balloon masks. This is what kills the OCR problem — you never OCR a full page again.
• Character boxes: YOLOv8 anime face/person model exported to ONNX. CPU-fine.
• In: panels/ · Out: detections.json
• Dep: onnxruntime, opencv, numpy.

// detections.json  (per panel)
{
  "p001": {
    "balloons": [
      { "id": "b0", "mask_poly": [[x,y],...], "centroid": [x,y], "bbox": [x,y,w,h] }
    ],
    "text_regions": [
      { "id": "t0", "bbox": [x,y,w,h], "in_balloon": "b0" }   // null if floating (sfx/sign)
    ],
    "chars": [
      { "id": "c0", "bbox": [x,y,w,h], "crop": "..." }
    ]
  }
}

[3] Character identity (bank match)

• Embed every chars[*] crop with SigLIP or an anime ArcFace model (ONNX).
• Cosine-match against bank.json entries. Above threshold → assign name; else "unknown".
• Bank is built once per series by hand (10 min): a few exemplar crops + a name each. This is the only required human touch.
• In: detections.json, bank/ · Out: identities.json
• Dep: onnxruntime, numpy, sklearn (cosine / nearest-neighbour).
• Note: this is the accuracy-critical stage. Generic embeddings + same-face syndrome = the weakest link vs magi. The bank is what rescues it. Do NOT attempt unsupervised clustering as primary — it merges lookalikes.

// bank.json
{ "characters": [
    { "name": "Aria", "exemplars": ["bank/crops/aria_0.png","bank/crops/aria_1.png"] }
]}
// identities.json  (per panel)
{ "p001": { "c0": { "name": "Aria", "score": 0.82 }, "c1": { "name": "unknown", "score": 0.41 } } }

[4] Text extraction

• For each in_balloon text region: crop the balloon, feed the clean crop to gemma (llama.cpp) → read text. Isolated crops read far better than full pages.
• In: detections.json, panels/ · Out: texts.json
• Dep: llama.cpp (existing). No torch.

// texts.json
{ "p001": { "t0": "We can't stay here." } }

[5] Speaker binding

Two paths — start geometry, fall back to gemma on low confidence.

• A — tail geometry (default, pure CV): from the balloon mask, find the tail = sharpest protrusion off the centroid. Vector centroid→tip; nearest chars[*] box along that ray = speaker. Hand-rolled magiv2 tail logic.
• B — set-of-mark + gemma (fallback): draw numbered boxes on chars + balloons on the panel image, ask gemma "balloon 2 → face #3 or #5?". Grounding via drawn marks beats free-form spatial reasoning. No new deps.
• Trigger fallback when: >2 candidate chars, ambiguous/absent tail, or geometry confidence < threshold.
• In: detections.json, identities.json, texts.json · Out: merged into transcript.json.

[6] Essential vs non-essential filter

• Pure geometry: text_region.in_balloon != null → dialogue. Floating on raw art → sfx/sign → drop. Kills "THUD" and street-sign garbage with no classifier.

[7] Transcript assembly (reading order)

• Panel order from panels.json. Within panel: manga = sort (top → right-to-left); manhwa = top-down.
• Emit ordered speaker+line list.

// transcript.json
{ "chapter": 12, "lines": [
    { "panel": "p001", "speaker": "Aria", "line": "We can't stay here.", "conf": 0.79, "flagged": false }
]}

[8] Scene-action description

• gemma describes physical action per panel ("she draws a sword") — the one thing magi does NOT do and gemma is good at. Runs in parallel with 2–7.
• Out: scene field per panel, merged for the script stage.

[9] Script generation

• LLM input = one chapter of transcript + scene-actions + a ~500-token rolling summary of prior chapters. Do NOT token-max the context; attention degrades mid-window and coherence/attribution drop.
• Prompt: compress and narrate, not transcribe. Output narration segments each mapped to source panel IDs (needed for visual timing).
• Out: script.json

// script.json
{ "segments": [
    { "id": "s0", "text": "Cornered in the ruins, Aria makes her choice...", "panels": ["p001","p002"] }
]}

[10] TTS

• dots.tts, wav per segment. Keep durations — they drive visual timing.

[11] Visual assembly

• ffmpeg ken-burns (pan/zoom) per panel. Panel display time = proportional to its segment's audio length. moviepy to orchestrate or raw filtergraphs for lean/fast.

[12] Subtitles

• faster-whisper on generated audio → timestamped SRT → burn in with ffmpeg. Easier than aligning from the script side.

[13] Mux

• Combine video + audio + burned subs → video.mp4.

---

6. Dependency matrix

Stage	Tooling	New torch?
1 panels	kumiko, opencv	no
2 detect	comic-text-detector (onnx), yolo-anime (onnx)	no
3 identity	siglip/arcface (onnx), sklearn	no
4 text	llama.cpp gemma	no
5 bind	numpy geometry + gemma fallback	no
6 filter	numpy geometry	no
7 order	numpy	no
8 scene	llama.cpp gemma	no
9 script	local LLM	no
10 tts	dots.tts	(its own env)
11 video	ffmpeg, moviepy	no
12 subs	faster-whisper	(ctranslate2, not torch)
13 mux	ffmpeg	no

Net: the magi replacement (2–7) adds only ONNX Runtime + a couple of onnx model files. No ROCm wheel churn.

---

7. Orchestration

• Bash driver calls python stage scripts; artifacts handed off as files.
• Each stage: stage_N.py --series X --chapter NN [--force].
• Skip logic: if output exists and mtime(output) > mtime(input) and not --force, skip.
• Model loading: sequence stages so you never hold two large models in VRAM at once (gemma vs detectors vs faster-whisper). One card handles all, serially.
• Do NOT wrap in a Spring service. This is a batch job, not request/response — a service is pure added attack surface and state for zero benefit.

---

8. Hardware notes

• gemma (mmproj) is the VRAM heavyweight; detectors are CPU-viable.
• Sequence model loads; never co-resident. Target: single decent GPU, staged.
• Detection/embedding on CPU EP is fine and frees VRAM for gemma.

---

9. Known risks / caveats

• Re-ID accuracy is the weak link vs magi's end-to-end association. Mitigation: character bank (mandatory), tune cosine threshold per series, flag unknown rather than guess.
• Manhwa layout: comic-text-detector and the anime detectors are manga/anime-trained. Vertical webtoon art + non-japanese layout = degraded results. Expect tuning on the whitespace slicer and lower binding confidence.
• Tail geometry fails on off-panel speakers and thought bubbles. Fallback to gemma SoM; if still ambiguous, flag the line.
• Rolling summary drift: long series accumulate summary error. Periodically re-anchor the summary from a canonical synopsis if available.

---

10. Pre-build check (do this first)

Before building any of this: magi's HF weights may load through your existing ROCm torch directly, ignoring its pinned requirements.txt (the version pain is usually the wrapper deps, not torch). 5-minute test. If it instantiates and runs, you skip this entire rebuild. If it OOMs or the arch won't load, decompose per this spec.