From ff6a512630ec22925637f045d6b03921fc3d3482 Mon Sep 17 00:00:00 2001 From: kami Date: Tue, 11 Aug 2026 02:42:41 +0400 Subject: [PATCH] Reconstruct repo from Claude Code + codex transcripts Working tree (including .git) was lost to an rm. Rebuilt by replaying Write/Edit/ Read/attachment events from 25 Claude sessions and 22 successful codex apply_patch blocks into one timestamp-ordered timeline. Verified against ground truth recorded in the transcripts: wc -l on 10 files and ls -l on 5 files at 2026-07-18T13:13:44Z both match exactly; 18 files are byte-identical to their newest ~/.claude/file-history blob. See HANDOFF.md for sources, gaps, and how to rebuild .venv. Co-Authored-By: Claude Opus 5 --- .gitignore | 6 + AGENTS.md | 23 + CLAUDE.md | 67 ++ HANDOFF.md | 75 +++ attic/char-recognition.md | 29 + attic/plan-workpc.md | 575 ++++++++++++++++++ attic/worker_ocr.py | 99 +++ attic/worker_parse.py | 125 ++++ bubble_detect.py | 110 ++++ collage.py | 169 ++++++ face_detect.py | 74 +++ identity-rework-task.md | 67 ++ manga-recap-pipeline-spec.md | 248 ++++++++ plan.md | 414 +++++++++++++ requirements.txt | 15 + scripts/analyze_video_frames.sh | 95 +++ scripts/pick_tts_voice.py | 118 ++++ session_manager.py | 238 ++++++++ spec-correctness.md | 25 + spec-v3.md | 348 +++++++++++ start_workers.sh | 44 ++ systemd/install.sh | 47 ++ test_vision_parse.py | 47 ++ transport.py | 216 +++++++ worker_crop.py | 359 +++++++++++ worker_identity.py | 322 ++++++++++ worker_layers.py | 99 +++ worker_render.py | 996 ++++++++++++++++++++++++++++++ worker_scene.py | 170 ++++++ worker_script.py | 300 +++++++++ worker_tts.py | 231 +++++++ worker_vision.py | 1008 +++++++++++++++++++++++++++++++ 32 files changed, 6759 insertions(+) create mode 100644 .gitignore create mode 100644 AGENTS.md create mode 100644 CLAUDE.md create mode 100644 HANDOFF.md create mode 100644 attic/char-recognition.md create mode 100644 attic/plan-workpc.md create mode 100644 attic/worker_ocr.py create mode 100644 attic/worker_parse.py create mode 100644 bubble_detect.py create mode 100644 collage.py create mode 100644 face_detect.py create mode 100644 identity-rework-task.md create mode 100644 manga-recap-pipeline-spec.md create mode 100644 plan.md create mode 100644 requirements.txt create mode 100755 scripts/analyze_video_frames.sh create mode 100644 scripts/pick_tts_voice.py create mode 100644 session_manager.py create mode 100644 spec-correctness.md create mode 100644 spec-v3.md create mode 100755 start_workers.sh create mode 100755 systemd/install.sh create mode 100644 test_vision_parse.py create mode 100644 transport.py create mode 100644 worker_crop.py create mode 100644 worker_identity.py create mode 100644 worker_layers.py create mode 100644 worker_render.py create mode 100644 worker_scene.py create mode 100644 worker_script.py create mode 100644 worker_tts.py create mode 100644 worker_vision.py diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..5906d58 --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +.venv/ +__pycache__/ +*.pyc +dots.tts/ +/dev/shm/ +*.gguf diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..f43fcb5 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,23 @@ +# Repository guidance + +## Video analysis + +When diagnosing render motion, transitions, timing, or visual artifacts, use +`scripts/analyze_video_frames.sh` instead of manually seeking through the video. + +```bash +scripts/analyze_video_frames.sh VIDEO [OUTPUT_DIR] \ + --window START:DURATION [--window START:DURATION ...] +``` + +The script creates: + +- timestamped overview contact sheets sampled at 2 fps; +- scene-change frames and a `scene-timestamps.txt` index; +- optional dense 10-fps contact sheets for specified transition windows. + +Inspect the overview first, then request dense windows around transitions or artifacts. Sampling and +scene sensitivity can be adjusted with `OVERVIEW_FPS`, `WINDOW_FPS`, and `SCENE_THRESHOLD`. + +The generated files are analysis artifacts and belong in `/tmp` or another requested output directory; +do not commit them to the repository. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..9461ab6 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,67 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +The **workpc compute half** of a manga→narrated-video pipeline. This repo holds stateless +GPU/CPU workers only. State, job scheduling, and stage orchestration live in a **separate +homesrv orchestrator repo** (`/mnt/server/home/kami/apps/Maven/` region) — not here. The two +communicate over a fixed HTTP contract; never add sqlite or durable state to a worker. + +Machine split (memorize): workers run on **workpc** (RX 7900 GRE, ROCm). MinIO + orchestrator +run on **homesrv** (`192.168.1.104`, CPU-only). Data dir `/mnt/server/home/kami/` is an SSHFS +mount of homesrv. + +## Run / test + +```bash +./start_workers.sh # dev: session_manager + 9 workers, each a uvicorn in a tmux window +tmux attach -t manga-workers # watch logs; per-worker window +sudo systemd/install.sh # production: one systemd unit per process (User=kami) + +python session_manager.py # each module has a __main__ self-check (assert-based, no framework) +python transport.py # run these to verify a file after editing it +python test_vision_parse.py # the one standalone pytest-free test +``` + +There is no lint/build step. `.venv` is the ROCm torch env; workers import `transport` by module name. + +## Architecture + +**Workers are stateless HTTP stages.** Each `worker_*.py` is a FastAPI app on a fixed port. It +pulls inputs from MinIO by URI to local disk (`/dev/shm`), does one stage, pushes outputs back, +returns URIs. No cross-request memory. Ports: crop 8000, vision 8002, identity 8003, scene 8004, +script 8005, tts 8006, layers 8007, render 8008, **session_manager 8095**. + +**`transport.py`** — shared MinIO client (`get`/`put`/`put_bytes`/`exists`) + `install_logging(app, name)` +(one log line per request with panel id). URIs are `s3://bucket/key`. Import it in every worker. + +**`session_manager.py`** — the **GPU mutex**. Only one warm model at a time on the single local GPU. +Two model kinds: +- *subprocess* (`gemma4`): it spawns/health-waits/terminates `llama-server`, and a supervisor + respawns it in-place if it crashes mid-session (keeps the same session_id + port). +- *in-process* (`siglip2`, `dots`): returns `port=None`; the **worker** loads the transformers model + itself and must expose `/unload` so `/session/close` can free the ~5GB VRAM (the mutex alone + can't reclaim it). Leases have a TTL + heartbeat; a reaper force-closes stale ones. + +A GPU worker's flow: `/session/open {model}` → (409 if busy) → do work against the returned port or +its own resident model → `/session/close`. CPU workers (crop) take no session. + +## Conventions + +- **`ponytail:` comments** mark deliberate simplifications and name the upgrade path — respect them, + don't "fix" them without reason. +- Any non-trivial logic gets ONE runnable check in `__main__` (assert-based `demo`/self-check), not a + test suite. Follow that pattern; run the file to verify. +- The HTTP contract with the orchestrator is load-bearing and shared across repos — changing a + worker's request/response shape means reconciling the orchestrator too (see commit history: + "reconcile worker contracts"). + +## Specs & docs + +- `spec-v3.md` — current quality/look work (narration voice, panel curation, render overhaul), marked + DONE/TODO per item. `legacy/` holds the old single-repo workers. +- `AGENTS.md` — use `scripts/analyze_video_frames.sh` (not manual seeking) to diagnose render output; + its frames are `/tmp` artifacts, never commit them. +- `collage.py` (behind `COLLAGE` flag) and the render worker carry the animated-layout planners. diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..f9ece08 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,75 @@ +# HANDOFF: repo reconstructed from agent transcripts (2026-08-11) + +## What was asked +The working copy at `/home/kami/Programs/n8n-worker` was deleted with `rm`, including `.git`. +This repo is the workpc GPU-compute half of the manga pipeline. The ask was to rebuild it from +the transcripts on disk, at minimum the transport layer and the pipeline flow. + +## Sources used +No backup and no remote survived. `~/.local/share/Trash` is empty. No copy exists under +`/home/kami`, `/mnt/D`, or `/mnt/server`. Gitea +(`/mnt/server/mnt/hdd2/gitea/git/repositories/`) holds only correx, hexis, maven, +model-training, muzick, orchestra, and test-e2e. There is no `n8n-worker` mirror. + +Reconstruction replays three histories into one timeline, ordered by timestamp: + +1. Claude Code transcripts, 25 sessions in + `~/.claude/projects/-home-kami-Programs-n8n-worker/*.jsonl`. Contributes Write and Edit + content, full-file Read results, and `@`-mention attachments. +2. Codex rollouts, 5 sessions with `cwd=/home/kami/Programs/n8n-worker` under + `~/.codex/sessions/2026/07/{13,15,16,18}/rollout-*.jsonl`. Contributes 26 `apply_patch` + blocks. The 4 that the log marks `Script failed` are skipped. These carry the 2026-07-18 + correctness work (`_dialogue_envelope`, `_normalize_claims`, `_annotate_speaker_methods`, + `SOM_ATTRIBUTION=1`) that exists in no Claude transcript. +3. `~/.claude/file-history//@vN` pre-edit blobs, used as cross-checks only. + +Scripts live in +`/tmp/claude-1000/-home-kami-Programs-n8n-worker/6a35d5d8-9f44-4412-955c-9cf088737ad5/scratchpad/`. +`replay2.py` is the merged replay. `codex.py` extracts and applies codex patches. `backups.py` +compares against file-history. `recon_final/_log.json` holds the per-file event log. + +## Verification +- Replayed to 2026-07-18T13:13:44Z, line counts match the `wc -l` recorded in the transcript at + that moment, for all 10 files it covered. crop 326, identity 314, layers 99, render 996, + scene 159, script 300, tts 231, vision 871, session_manager 238, transport 216. +- Byte sizes match the `ls -l` recorded by the same command. AGENTS.md 867, CLAUDE.md 3678, + manga-recap-pipeline-spec.md 11413, plan.md 16511, spec-v3.md 26170. `.gitignore` is 53. +- 18 files are byte-identical to their newest file-history blob. The 4 that differ (render, + script, tts, spec-v3) differ only by edits made after that blob was taken. +- `python3 -m py_compile *.py scripts/*.py attic/*.py` passes. `bash -n` passes on all 3 shell + scripts. Runtime self-checks such as `python worker_vision.py` were NOT run, because `.venv/` + is gone. + +## Recovered (live tree) +Workers `worker_{crop,vision,identity,scene,script,tts,layers,render}.py`, the GPU mutex +`session_manager.py`, `transport.py`, helpers `bubble_detect.py`, `face_detect.py`, +`collage.py`, `test_vision_parse.py`, `start_workers.sh`, `systemd/install.sh`, +`scripts/{analyze_video_frames.sh,pick_tts_voice.py}`, `requirements.txt`, `.gitignore`, +and the docs `AGENTS.md`, `CLAUDE.md`, `plan.md`, `spec-v3.md`, `spec-correctness.md`, +`manga-recap-pipeline-spec.md`, `identity-rework-task.md`. + +`attic/` holds files that had been deleted from the tree before the `rm`. They are kept but not +live: `worker_ocr.py`, `worker_parse.py`, `plan-workpc.md`, `char-recognition.md`. + +## Still open +- `.venv/` is gone. Rebuild with + `python -m venv .venv && .venv/bin/pip install -r requirements.txt`. The ROCm torch wheel is + not pinned in requirements.txt, so install it the way workpc had it. +- `dots.tts/` (gitignored external checkout) and `legacy/` (pre-rewrite workers, moved there + 2026-07-13) are not recoverable from transcripts. Re-clone dots.tts if TTS is needed. +- `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable. Neither agent + ever read them in full. All three were already deleted from the working tree as of 2026-07-17. +- Git history is gone. A fresh `git init` plus one commit replaces it. Old commits are not + recoverable. +- Two replay gaps were left unpatched. One Edit MISS in `worker_render.py` on 2026-07-14, and one + orphan Edit against `attic/plan-workpc.md` on 2026-07-13. Both files were later replaced by a + full-file snapshot, so the final content stays anchored. +- Edits made outside Claude and codex after 2026-07-18T14:35Z (the last recorded write) cannot be + detected. `unknown:` whether any exist. + +## Next command +``` +cd /home/kami/Programs/n8n-worker +python -m venv .venv && .venv/bin/pip install -r requirements.txt +.venv/bin/python worker_vision.py # per-file __main__ self-checks, then ./start_workers.sh +``` diff --git a/attic/char-recognition.md b/attic/char-recognition.md new file mode 100644 index 0000000..c39ed08 --- /dev/null +++ b/attic/char-recognition.md @@ -0,0 +1,29 @@ +# Character name recognition — open problem + +The script stage writes narration that refers to characters generically ("a young man", "a woman with dark hair") because the vision describe prompt has no way to attach names to identities. Fixing this is a multi-stage problem, not a prompt tweak. + +## The core loop + +OCR alone doesn't help — finding "Teto" in a speech bubble gives us a string but no visual binding. The vision model needs to see the image to tie a name to an appearance. That means: + +1. OCR each crop for character names/introductions (pytesseract, cpu, cheap) +2. For any crop where OCR finds a new name, re-send the same image to gemma4 with: + *"A character just spoke the name 'Teto'. Describe their distinguishing visual features."* +3. Store a name→description map: `{"Teto": "young girl, short dark hair, red scarf"}` +4. Inject known names into subsequent describe prompts: + *"Known characters in this chapter: Teto (young girl, short dark hair, red scarf). Describe this panel using their names if they appear."* + +## Cost per chapter + +Each named panel costs an extra gemma4 call (the identity grounding). For a typical 40-page chapter with ~200 crops and maybe 8-15 named references, that's 8-15 extra vision calls — exactly as expensive as the main describe call since it loads the same model. + +## Implement hints + +- Add `work_dir/characters.json` as the persistent name→description store, created/reused per chapter +- Move the identity grounding call into a helper in pipeline_worker: `srv.identify(crop_path, name)` +- Inject names into `DESCRIBE_PROMPT` at runtime (f-string or str.replace) so known names propagate forward +- `pytesseract` is zero-model-load CPU, can run on homesrv or workpc — cheap enough to run on every crop + +## Not built + +No code exists for this. The pipeline currently has no ocr step, no identity store, and no dynamic prompt injection. The script prompt just gets whatever `scene_text` and `tone` the vision stage produces. diff --git a/attic/plan-workpc.md b/attic/plan-workpc.md new file mode 100644 index 0000000..34bac5f --- /dev/null +++ b/attic/plan-workpc.md @@ -0,0 +1,575 @@ +# plan — workpc (compute + transport) + +workpc owns: GPU, model lifecycle, per-item worker endpoints, the session manager that enforces one-warm-model-at-a-time. workpc holds NO durable state — every artifact goes to homesrv (minio) before a stage is considered done. sqlite lives on homesrv; workpc workers are stateless HTTP endpoints that read inputs by URI and write outputs to minio. + +this plan assumes the homesrv plan (sqlite schema, minio buckets, orchestrator service, n8n workflow) lands in parallel. workpc can't be fully tested without homesrv's transport layer, but each worker can be developed and self-checked in isolation against a local minio + sqlite stub. + +--- + +## phase 0 — repo hygiene + +- `git init` in `/home/kami/Programs/n8n-worker/`, initial commit of current state on a `legacy/` branch. main branch starts clean. +- `.gitignore`: `.venv/`, `__pycache__/`, `dots.tts/` (submodule-ish, has its own repo), any local model paths. +- move existing workers into `legacy/` for reference, don't delete — the new code is a rewrite, not an edit. + +## phase 1 — session manager (the GPU mutex) + +new file: `session_manager.py`. FastAPI on `127.0.0.1:8095`. + +this is the single most important new component on workpc. it owns: +- which model is currently loaded (gemma4 / siglip2 / dots.tts / comfyui-via-external) +- the llama-server subprocess lifecycle (start, health-wait, teardown) +- a GPU mutex — only one model resident at a time, enforced via a lock +- session lease tracking (who opened what, when, for timeout/reaper logic) + +### endpoints + +``` +POST /session/open {model: "gemma4"|"siglip2"|"dots", ttl: 3600} + -> {session_id, port} # port is the warm server's localhost port + 409 if another model is resident (caller retries or queues) + +POST /session/close {session_id} + -> {ok: true} # tears down the subprocess, releases GPU + +GET /session/active + -> {model, session_id, opened_at, port} | null + +POST /session/heartbeat {session_id} + -> {ok: true} # extends the lease; stale sessions get reaped +``` + +### model registry + +hardcoded map in the file (not config — these paths are workpc-specific): + +```python +MODELS = { + "gemma4": { + "binary": "/mnt/D/AI/llama-cpp/llama.cpp/build/bin/llama-server", + "model": "/mnt/D/AI/gemma4/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf", + "mmproj": "/mnt/D/AI/gemma4/mmproj-F16.gguf", + "port": 8090, "ctx": 4096, "args_extra": ["--jinja", "--reasoning-budget", "0"], + }, + "siglip2": { + "binary": "transformers", # in-process, not subprocess + "model": "google/siglip2-so400m-patch16-384", + "port": None, # no http server, direct python call + }, + "dots": { + "binary": "transformers", # in-process + "model": "rednote-hilab/dots.tts-base", + }, +} +``` + +siglip2 and dots load in-process (transformers, the rocm torch build already on workpc). gemma4 stays subprocess (llama-server, the working mtmd path). the session manager handles both kinds. + +### reaper + +background thread: any session without a heartbeat in `ttl` seconds gets force-closed. prevents a crashed n8n node from holding the GPU forever. + +### systemd + +system unit `/etc/systemd/system/manga-session.service`, `User=kami`, `ExecStart` via the workpc venv (`python -m uvicorn session_manager:app --host 127.0.0.1 --port 8095`). `--system-site-packages` venv to reuse the rocm torch build. + +### self-check + +`python session_manager.py`: +- open gemma4 session → active → open second → 409 → close first → open second → ok +- heartbeat: open with ttl=2, sleep 5, assert reaped via `/session/active` +- (gemma4 subprocess not actually started in self-check — mock the binary path to `/bin/sleep`) + +--- + +## phase 2 — transport layer (minio client + sqlite status reporter) + +new file: `transport.py`. shared by all workers. two concerns: + +### 2a. minio artifact client + +```python +def put(local_path: str, uri: str) -> str: + """upload local file to minio at uri (e.g. 'manga/{mid}/{cid}/panels/p001.png'), + return the full s3-style uri. raises on failure.""" + +def get(uri: str, local_path: str) -> str: + """download uri to local_path, return local_path. workers pull inputs to /dev/shm + before processing, push outputs back.""" + +def exists(uri: str) -> bool: + """HEAD check. used for skip-if-exists at the worker level (defense in depth — + the orchestrator on homesrv is the real skip authority via sqlite).""" + +def put_bytes(data: bytes, uri: str) -> str: + """for generated artifacts that don't touch disk (embeddings, json).""" +``` + +minio endpoint: `192.168.1.104:9000` (homesrv). creds via env, defaults match existing `storage.py`. + +### 2b. sqlite status reporter + +workers don't own state, but they need to report progress so n8n/the orchestrator can poll. two options: + +**option A (chosen): workers don't touch sqlite directly.** they return progress in their HTTP response, the orchestrator on homesrv writes to sqlite. workers are pure compute — input URI in, output URI + metadata out. keeps the "workpc = compute only" boundary clean. + +**option B (rejected): workers write to sqlite over the nginx-exposed endpoint.** adds a network-sqlite dependency to every worker, couples them to homesrv's db schema. rejected. + +so `transport.py` is just minio. sqlite stays homesrv-side. workers are stateless. + +### self-check + +`python transport.py`: put a tempfile, get it back, assert bytes match, exists() true, missing uri exists() false. mock minio client for offline test. + +--- + +## phase 3 — worker: OCR (stage 3) + +new file: `worker_ocr.py`. FastAPI on `127.0.0.1:8001`. + +### input + +```json +POST /ocr +{ + "panel_uri": "s3://manga/{mid}/{cid}/panels/p001.png", + "job_id": "...", + "panel_id": "p001" +} +``` + +### process + +1. `transport.get(panel_uri, /dev/shm/ocr_input.png)` +2. pytesseract.image_to_data with bbox output (`--psm 11` sparse, manga bubbles are isolated) +3. filter low-confidence (<0.5), dedupe overlapping boxes +4. cluster text lines (group words on same row within x-threshold) +5. return structured result + +### output + +```json +{ + "panel_id": "p001", + "texts": [ + {"id": "t001", "content": "...", "bbox": [x,y,w,h], "confidence": 0.97} + ] +} +``` + +the orchestrator (homesrv) takes this, writes to sqlite `ocr_results` table, and feeds it forward to vision + scene graph. worker doesn't persist anything — it returns the json, homesrv stores it. + +### self-check + +synthetic image with two text blocks, assert two texts returned with sane bboxes. no minio in self-check (local path variant of the endpoint for testing). + +--- + +## phase 4 — worker: vision (stage 4, reworked) + +new file: `worker_vision.py`. FastAPI on `127.0.0.1:8002`. + +### input + +```json +POST /vision +{ + "panel_uri": "...", + "ocr_texts": [...], // from stage 3, injected into prompt + "known_characters": [ // from identity store, injected into prompt + {"name": "Gojo", "description": "white hair, blindfold, black coat"} + ], + "panel_id": "p001", + "session_id": "..." // gemma4 session, opened by orchestrator via session manager +} +``` + +### process + +1. `transport.get(panel_uri, /dev/shm/vision_input.png)` +2. build prompt: structured output request + OCR text + known characters +3. call gemma4 at `localhost:8090/v1/chat/completions` (the session manager's warm server) +4. parse structured json from response (gemma4 with `--jinja` can do json mode; enforce via prompt + json.loads with retry on parse failure) + +### output (matches spec §4) + +```json +{ + "panel_id": "p001", + "characters": [ + {"local_id": "person_1", "appearance": {"hair": "...", "clothing": "...", "features": [...]}, "emotion": "...", "action": "..."} + ], + "scene": {"location": "...", "time": "..."} +} +``` + +### prompt design (the hard part) + +prompt must produce parseable json. structure: + +``` +You are analyzing a manga panel. Known characters in this story: Gojo (white hair, blindfold, black coat). + +Text found in speech bubbles: "..." , "..." + +Respond in this exact JSON format: +{ + "characters": [{"local_id": "person_1", "appearance": {"hair": "", "clothing": "", "features": []}, "emotion": "", "action": ""}], + "scene": {"location": "", "time": ""} +} +``` + +gemma4's reasoning output (`<|channel>thought...`) must be stripped before json parse — reuse `_strip_thought` from legacy code. + +### self-check + +mock the gemma4 call (return canned json), assert parse + structured output. real gemma4 test is integration (needs session manager + model). + +--- + +## phase 5 — worker: character identity (stage 5) + +new file: `worker_identity.py`. FastAPI on `127.0.0.1:8003`. + +this is the biggest new worker. it does NOT load a model itself — it uses the siglip2 session (opened via session manager) for embeddings, and queries homesrv's sqlite (via an orchestrator endpoint, not direct sqlite) for known characters. + +### input + +```json +POST /identity/resolve +{ + "panel_uri": "...", + "panel_id": "p001", + "vision_characters": [ // from stage 4 + {"local_id": "person_1", "appearance": {...}, "bbox_in_panel": [...]} + ], + "manga_id": "...", + "session_id": "..." // siglip2 session +} +``` + +### process + +1. `transport.get(panel_uri, /dev/shm/ident_input.png)` +2. for each vision character: + a. crop the character region from the panel (bbox from vision stage — **open question: does vision return character bboxes? currently spec §4 doesn't include them. needs adding.**) + b. compute siglip2 image embedding of the crop + c. query orchestrator: `GET /characters/known?manga_id=...` → list of known characters with their reference embedding URIs + d. for each known character: `transport.get(embedding_uri, /dev/shm/ref.npy)`, cosine similarity vs the new crop embedding + e. if best match > threshold (start 0.85, tune per-manga): assign `character_id` + f. if below threshold: create unknown candidate → `POST /characters/create` to orchestrator, get new `character_id`, upload the crop as its first reference image + +### output + +```json +{ + "panel_id": "p001", + "assignments": [ + {"local_id": "person_1", "character_id": "character_001", "confidence": 0.96} + ], + "new_characters": ["character_007"] +} +``` + +### threshold + matching strategy + +- cosine similarity on siglip2 image embeddings, top-1 match +- threshold per-manga, stored in sqlite `manga_config` table (default 0.85) +- if multiple known characters are close (within 0.05 of each other), pick the best but flag low-confidence for the scene graph to handle gracefully (use "a person" in narration rather than a wrong name) + +### self-check + +mock siglip2 (return deterministic vectors), mock orchestrator (return known chars), assert: high-similarity → assign, low → create new. no real model in self-check. + +--- + +## phase 6 — worker: scene graph (stage 6) + +new file: `worker_scene.py`. FastAPI on `127.0.0.1:8004`. + +this is a join stage, not a model stage. no GPU needed. it could run on homesrv, but keeping it on workpc for uniformity (and it's cheap enough to not need the session manager). + +### input + +```json +POST /scene/build +{ + "panel_id": "p001", + "panel_uri": "...", + "ocr_texts": [...], // stage 3 + "vision_result": {...}, // stage 4 + "identity_assignments": [...], // stage 5 + "characters_registry": [...] // known characters with names, for speaker attribution +} +``` + +### process + +1. join vision characters with identity assignments → characters with real names +2. **speaker attribution** (the hard sub-problem): + - for each OCR text block, find the nearest vision character bbox (spatial heuristic) + - if text contains a name that ISN'T the nearest character (e.g. "Teto, come here" nearest to person_2), the speaker is likely person_2 addressing Teto + - if text is a question/exclamation with no names, speaker = nearest character + - if ambiguous, mark `speaker: null` (narration will quote without attribution) + - **this is heuristic and will be wrong sometimes — accept it for v2, flag for v3 improvement with a dedicated model** +3. build scene graph + +### output (matches spec §6) + +```json +{ + "panel_id": "p001", + "characters": [{"id": "character_001", "name": "Gojo Satoru"}], + "dialogue": [{"speaker": "character_001", "text": "..."}], + "action": "character approaches enemy" +} +``` + +### self-check + +synthetic inputs with known spatial layout, assert speaker attribution picks nearest character. test the name-exception case. + +--- + +## phase 7 — worker: script (stage 7, reworked) + +new file: `worker_script.py`. FastAPI on `127.0.0.1:8005`. + +### input + +```json +POST /script +{ + "scene_graph": {...}, // stage 6 + "chapter_context": "...", // prior scenes summary, for continuity + "panel_id": "p001", + "session_id": "..." // gemma4 session (reused from vision stage if still open) +} +``` + +### process + +1. build prompt from scene graph: characters (with names), dialogue (quoted), action, scene setting +2. call gemma4 (warm session) +3. return narration text + +### output + +```json +{"panel_id": "p001", "text": "..."} +``` + +### prompt design + +``` +Write narration for this manga panel. Use the characters' real names. Quote dialogue naturally. One paragraph. + +Characters present: Gojo Satoru (white hair, blindfold) +Dialogue: Gojo says "..." +Action: Gojo approaches the enemy +Setting: street, night + +Narration: +``` + +### self-check + +mock gemma4, assert prompt construction includes character names + dialogue. real test is integration. + +--- + +## phase 8 — worker: TTS (stage 8, reworked) + +new file: `worker_tts.py`. FastAPI on `127.0.0.1:8006`. + +reuses the dots.tts code from legacy `render_worker.py` but decoupled from layers. + +### input + +```json +POST /tts +{ + "text": "...", + "speaker": "narrator", // for now always narrator; multi-voice is v3 + "panel_id": "p001", + "session_id": "..." // dots session +} +``` + +### output + +```json +{ + "audio_uri": "s3://manga/{mid}/{cid}/audio/p001.wav", + "duration": 4.2 +} +``` + +worker generates audio locally, `transport.put` to minio, returns the uri. no local persistence. + +### self-check + +mock the tts model (write a sine wave), assert upload + uri return. real test is integration. + +--- + +## phase 9 — worker: layers (stage 9 part 1) + +new file: `worker_layers.py`. FastAPI on `127.0.0.1:8007`. + +wraps the comfyui layered workflow from legacy `render_worker.py`. comfyui runs as its own external process (not via session manager — it has its own server lifecycle). + +### input + +```json +POST /layers +{ + "panel_uri": "...", + "panel_id": "p001", + "num_layers": 4, + "prompt": "..." +} +``` + +### output + +```json +{ + "layer_uris": ["s3://manga/{mid}/{cid}/layers/p001/0.png", ...] +} +``` + +### self-check + +mock comfyui (return canned image urls), assert upload + uri list. real test needs comfyui running. + +--- + +## phase 10 — worker: render (stage 9 part 2, the new part) + +new file: `worker_render.py`. FastAPI on `127.0.0.1:8008`. + +this is the **new** rendering work — camera movement, transitions, subtitles, effects. spec §9. this is the most open-ended phase. + +### input + +```json +POST /render/scene +{ + "panel_uri": "...", + "layer_uris": [...], // from stage 9 part 1 (optional) + "audio_uri": "...", + "narration_text": "...", // for subtitles + "panel_id": "p001", + "scene_timing": {"start": 0.0, "end": 4.2} +} +``` + +### process + +1. pull inputs to /dev/shm +2. if layers present: parallax motion (subtle pan/zoom on separated depth layers) +3. if no layers: ken burns effect (slow zoom/pan on static panel) +4. burn subtitles from narration_text (ass format, styled) +5. transitions between scenes: crossfade (handled at the chapter assembly level, not per-scene — see below) +6. encode scene clip with ffmpeg + +### output + +```json +{"clip_uri": "s3://manga/{mid}/{cid}/clips/p001.mp4"} +``` + +### chapter assembly (separate endpoint) + +```json +POST /render/assemble +{ + "clip_uris": [...], // ordered + "chapter_id": "..." +} +``` + +concat with crossfade transitions between clips, output final `chapter.mp4`, upload to minio. + +### self-check + +synthetic images + silent audio, assert clip produced + uploaded. real motion test is visual (manual review). + +--- + +## phase 12 — worker: crop (stage 2, runs first) + +new file: `worker_crop.py`. FastAPI on `127.0.0.1:8000`. + +the panel detector. stage 2 in the pipeline (before OCR) but written last — it's the one worker the homesrv plan referenced without a contract here. no GPU (kumiko panel detection is cpu/opencv), no session manager. one page in, N panel crops out. + +### input + +```json +POST /crop +{ + "page_uri": "s3://manga/{mid}/{cid}/pages/p_000.png", + "manga_id": "...", + "chapter_id": "...", + "page_index": 0, + "job_id": "..." +} +``` + +### process + +1. `transport.get(page_uri, /dev/shm/crop_input.png)` +2. kumiko panel detection (reuse the legacy kumiko path) → ordered list of panel bboxes in reading order (right-to-left, top-to-bottom for manga) +3. for each detected panel: crop from the page, `transport.put` to `manga/{mid}/{cid}/panels/pg{page_index:03d}_p{panel_index:02d}.png` +4. return the panel URIs + bboxes **in reading order** (panel_index is the within-page order; the orchestrator assigns the flat chapter-wide `panel_id`/reading order) + +### output + +```json +{ + "page_index": 0, + "panels": [ + {"panel_index": 0, "uri": "s3://manga/{mid}/{cid}/panels/pg000_p00.png", "bbox": [x, y, w, h]} + ] +} +``` + +`bbox` is `[x, y, w, h]` on the source page (matches the homesrv `panels.bbox` column). the orchestrator is the skip authority: it calls `/crop` once per page and skips pages that already have panel rows (defense-in-depth `transport.exists` in the worker is optional). + +### self-check + +synthetic 2-panel page image, assert two panels returned in reading order with sane bboxes + uploaded URIs. no real kumiko model needed if the detector is opencv-only; otherwise mock the detector. + +--- + +## phase 11 — startup script + +rewrite `start_workers.sh` to launch: +- session_manager (8095) +- worker_crop (8000) +- worker_ocr (8001) +- worker_vision (8002) +- worker_identity (8003) +- worker_scene (8004) +- worker_script (8005) +- worker_tts (8006) +- worker_layers (8007) +- worker_render (8008) + +each as a background uvicorn process in a tmux session. session manager starts first, workers wait for it. + +systemd units for each (system services, `User=kami`) — preferred over tmux for production, tmux for dev. provide both. + +--- + +## open questions deferred to homesrv plan + +- exact sqlite schema (characters table, ocr_results, vision_results, scene_graphs, scripts, audio, clips, jobs, stages) — defined in homesrv plan +- orchestrator endpoint contracts (`/characters/known`, `/characters/create`, stage status writes) — defined in homesrv plan +- n8n workflow shape (which nodes, how session open/close maps to stage boundaries) — defined in homesrv plan +- minio bucket layout (`manga` single bucket with key prefixes, or per-manga buckets?) — defined in homesrv plan, workpc transport.py just takes URIs + +## open question flagged for your decision + +- **vision stage character bboxes**: spec §4 doesn't include character bounding boxes in the vision output, but the identity worker (phase 5) needs them to crop character regions for siglip2 embeddings. add `bbox` to each character in the vision output? this is a spec amendment. my recommendation: yes, add it — gemma4 can produce bboxes if asked, and it's necessary for the identity crop. flag this in the homesrv plan too since it touches the spec. \ No newline at end of file diff --git a/attic/worker_ocr.py b/attic/worker_ocr.py new file mode 100644 index 0000000..d7371dd --- /dev/null +++ b/attic/worker_ocr.py @@ -0,0 +1,99 @@ +# worker_ocr.py — stage 3 text extraction. FastAPI :8001. cpu (easyocr), no session. +# panel in -> text blocks with bboxes + confidence out. orchestrator persists to sqlite. +# easyocr (not tesseract): it reads stylized manga lettering far better -- recovers whole +# lines tesseract garbles or drops. runs on GPU (~0.4s/page warm) by default; the OCR stage +# runs before any LLM session opens so it doesn't contend with the resident model. set +# OCR_GPU=0 to force CPU (~3s/page). GPU needs MIOPEN_FIND_MODE=FAST in the env or the first +# ROCm run spends ~60s in MIOpen's exhaustive kernel search -- the launcher sets it. +import os, uuid +from fastapi import FastAPI +from pydantic import BaseModel +import transport + +app = FastAPI() +SHM = "/dev/shm" +MIN_CONF = 0.3 # easyocr line confidence floor +OCR_GPU = os.environ.get("OCR_GPU", "1") == "1" + +_reader = None # easyocr.Reader, lazy-loaded on first request + + +def _get_reader(): + global _reader + if _reader is None: + import easyocr + _reader = easyocr.Reader(["en"], gpu=OCR_GPU, verbose=False) + return _reader + + +def _detections_to_texts(detections): + """easyocr readtext output [(box_pts, text, conf)] -> our text blocks with xywh bboxes. + box_pts is 4 corner [x,y] points. drops low-confidence and art-noise (<2 letters). + casing is left as-is (mixed) -- the vision stage re-cases from the image anyway.""" + texts = [] + for i, (box, txt, conf) in enumerate(detections): + txt = txt.strip() + if conf < MIN_CONF or sum(c.isalpha() for c in txt) < 2: + continue + xs = [p[0] for p in box]; ys = [p[1] for p in box] + x, y = int(min(xs)), int(min(ys)) + texts.append({ + "id": f"t{i+1:03d}", + "content": txt, + "bbox": [x, y, int(max(xs)) - x, int(max(ys)) - y], + "confidence": round(float(conf), 3), + }) + return texts + + +def ocr_image(path: str): + return _detections_to_texts(_get_reader().readtext(path, detail=1, paragraph=False)) + + +class OCRInput(BaseModel): + panel_uri: str + job_id: str = "" + panel_id: str = "" + + +@app.post("/ocr") +async def ocr(data: OCRInput): + local = transport.get(data.panel_uri, f"{SHM}/ocr_{uuid.uuid4().hex[:8]}.png") + texts = ocr_image(local) + os.remove(local) + return {"panel_id": data.panel_id, "texts": texts} + + +@app.post("/unload") +async def unload(): + """free the resident easyocr reader (~1-2GB) once the OCR stage is done, before gemma4 loads. + ocr isn't session-managed, so the orchestrator calls this at stage end.""" + global _reader + was = _reader is not None + _reader = None + import gc; gc.collect() + try: + import torch; torch.cuda.empty_cache() + except Exception: + pass + return {"ok": True, "unloaded": was} + + +@app.get("/health") +async def health(): + return {"status": "ok"} + + +if __name__ == "__main__": + # self-check: detection->text-block conversion (pure, no model needed). + dets = [ + ([[10, 10], [110, 10], [110, 40], [10, 40]], "HELLO", 0.9), # kept + ([[10, 200], [70, 200], [70, 230], [10, 230]], "WORLD", 0.8), # kept + ([[5, 5], [13, 5], [13, 13], [5, 13]], "=", 0.9), # art-noise: <2 letters + ([[0, 0], [50, 0], [50, 20], [0, 20]], "REAL", 0.1), # below MIN_CONF + ] + texts = _detections_to_texts(dets) + assert [t["content"] for t in texts] == ["HELLO", "WORLD"], texts + assert texts[0]["bbox"] == [10, 10, 100, 30], texts[0]["bbox"] + assert texts[1]["confidence"] == 0.8 + print("worker_ocr self-check ok") diff --git a/attic/worker_parse.py b/attic/worker_parse.py new file mode 100644 index 0000000..08b177e --- /dev/null +++ b/attic/worker_parse.py @@ -0,0 +1,125 @@ +# worker_parse.py — manga parse (paged manga only). FastAPI :8009. GPU, session-guarded ("magi"). +# Magi v2 chapter-wide pass: panel detection + reading order + OCR in one shot. Replaces the +# crop+ocr stages for paged manga; downstream ocr stage no-ops because rows are pre-populated. +# webtoons do NOT come here — they use worker_crop /crop/webtoon. See manga-two-repo-split memory. +import os, uuid +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel +import cv2 +import numpy as np +import transport + +app = FastAPI() +SHM = "/dev/shm" +MAGI_MODEL = "ragavsachdeva/magiv2" +_model = None + + +def _load_magi(): + global _model + if _model is None: + import torch + from transformers import AutoModel + _model = AutoModel.from_pretrained(MAGI_MODEL, trust_remote_code=True).cuda().eval() + _model._torch = torch + return _model + + +class ParseInput(BaseModel): + page_uris: list # all pages of the chapter, in order + manga_id: str + chapter_id: str + session_id: str = "" # magi GPU lease (opened by orchestrator) + job_id: str = "" + + +def _center_in(box, panel) -> bool: + x1, y1, x2, y2 = box + px1, py1, px2, py2 = panel + cx, cy = (x1 + x2) / 2, (y1 + y2) / 2 + return px1 <= cx <= px2 and py1 <= cy <= py2 + + +def assemble_panels(pages, results, manga_id, chapter_id, put): + """Flatten Magi's per-page output into chapter-order panels with their OCR. + `pages`: RGB np arrays. `results`: per-page dicts (Magi keys). `put(np_crop, uri)` uploads. + Text is assigned to the panel whose box contains the text-box center; SFX (non-essential) + is dropped so narration isn't polluted. bbox converted [x1,y1,x2,y2] -> [x,y,w,h].""" + out, gidx = [], 0 + for img, res in zip(pages, results): + panels = res.get("panels", []) + texts = res.get("texts", []) + ocr = res.get("ocr", []) + essential = res.get("is_essential_text", [True] * len(texts)) + for p in panels: + x1, y1, x2, y2 = (int(v) for v in p) + uri = f"s3://manga/{manga_id}/{chapter_id}/panels/p{gidx:03d}.png" + if not transport.exists(uri): # deterministic per gidx -> resumable + put(img[y1:y2, x1:x2], uri) + ocr_texts = [] + for ti, tb in enumerate(texts): + if ti < len(ocr) and essential[ti] and _center_in(tb, p): + tx1, ty1, tx2, ty2 = (int(v) for v in tb) + ocr_texts.append({"text_id": f"t{ti}", "content": ocr[ti], + "bbox": [tx1, ty1, tx2 - tx1, ty2 - ty1], "confidence": 1.0}) + out.append({"panel_index": gidx, "uri": uri, + "bbox": [x1, y1, x2 - x1, y2 - y1], "ocr": ocr_texts}) + gidx += 1 + return out + + +def _put_crop(np_rgb, uri): + tmp = f"{SHM}/parse_{uuid.uuid4().hex[:8]}.png" + cv2.imwrite(tmp, cv2.cvtColor(np_rgb, cv2.COLOR_RGB2BGR)) + transport.put(tmp, uri) + os.remove(tmp) + + +@app.post("/parse") +async def parse(data: ParseInput): + tag = uuid.uuid4().hex[:8] + pages = [] + for i, u in enumerate(data.page_uris): + local = transport.get(u, f"{SHM}/pp_{tag}_{i:03d}.png") + img = cv2.imread(local) + if img is None: + raise HTTPException(400, f"page not readable: {u}") + pages.append(cv2.cvtColor(img, cv2.COLOR_BGR2RGB)) + os.remove(local) + model = _load_magi() + # ponytail: empty character bank in v1 — identity stays with the downstream siglip stage; + # feed a real bank (known-char ref crops + names) here to get Magi speaker association. + bank = {"images": [], "names": []} + with model._torch.no_grad(): + results = model.do_chapter_wide_prediction(pages, bank, use_tqdm=False, do_ocr=True) + panels = assemble_panels(pages, results, data.manga_id, data.chapter_id, _put_crop) + return {"panels": panels} + + +@app.get("/health") +async def health(): + return {"status": "ok"} + + +if __name__ == "__main__": + # self-check: model-free. Fake a 2-page Magi result and assert panel flattening, + # chapter-order indexing, text->panel containment, SFX drop, and bbox conversion. + transport.exists = lambda *a, **k: False # no minio in self-check + stored = {} + pages = [np.zeros((100, 100, 3), np.uint8), np.zeros((100, 100, 3), np.uint8)] + results = [ + {"panels": [[0, 0, 50, 100], [50, 0, 100, 100]], # page 0: two panels + "texts": [[10, 10, 20, 20], [60, 10, 70, 20]], # one text in each + "ocr": ["HELLO", "BOOM"], "is_essential_text": [True, False]}, # BOOM = SFX, dropped + {"panels": [[0, 0, 100, 100]], # page 1: one panel + "texts": [[5, 5, 15, 15]], "ocr": ["WORLD"], "is_essential_text": [True]}, + ] + panels = assemble_panels(pages, results, "m", "c", lambda img, uri: stored.__setitem__(uri, img.shape)) + assert [p["panel_index"] for p in panels] == [0, 1, 2], "chapter-order index" + assert panels[0]["ocr"][0]["content"] == "HELLO" + assert panels[1]["ocr"] == [], "SFX text dropped from panel 1" + assert panels[2]["ocr"][0]["content"] == "WORLD" + assert panels[0]["bbox"] == [0, 0, 50, 100], "xyxy->xywh" + assert panels[0]["ocr"][0]["bbox"] == [10, 10, 10, 10] + assert stored, "crops uploaded via put" + print("worker_parse self-check ok") diff --git a/bubble_detect.py b/bubble_detect.py new file mode 100644 index 0000000..d0cf8d3 --- /dev/null +++ b/bubble_detect.py @@ -0,0 +1,110 @@ +# bubble_detect.py — comic-text-detector ONNX -> per-panel TEXT-REGION boxes, for set-of-mark +# speaker attribution (see spec: decompose the VLM's implicit tail-tracing into grounded marks). +# Runs on the CPU EP so it never contends with gemma for the GPU. The model's `blk` head is a +# YOLO-style text-line detector; validated pixel-accurate on real panels + synthetic bubbles. +# +# We use ONLY the text-region boxes: a drawn, numbered region is the attribution anchor, and because +# region #k lives on panel N's own image it structurally prevents the window from smearing a line onto +# a neighbouring panel (the p108 bleed) or inventing dialogue that isn't drawn (the p108 phantom line). +# ponytail: full balloon-fill mask + tail-tip geometry are also in the `det`/`seg` heads but noisy and +# webtoon bubbles are often tailless anyway — add them only if region marks prove insufficient. +import os +import numpy as np +import cv2 + +MODEL = os.environ.get("CTD_MODEL", + os.path.join(os.path.dirname(__file__), "models", "comictextdetector.pt.onnx")) +CONF = float(os.environ.get("CTD_CONF", "0.20")) # blk head confidences run low (~0.55 max); tune per title +_sess = None + + +def _load(): + global _sess + if _sess is None: + import onnxruntime as ort + _sess = ort.InferenceSession(MODEL, providers=["CPUExecutionProvider"]) + return _sess + + +def _letterbox(img, sz=1024): + """resize keeping aspect ratio, pad to sz*sz with 114-grey (the model was trained letterboxed; + a plain squash distorts tall webtoon panels and wrecks the masks). returns canvas + inverse params.""" + h, w = img.shape[:2] + r = min(sz / h, sz / w) + nh, nw = int(round(h * r)), int(round(w * r)) + canvas = np.full((sz, sz, 3), 114, np.uint8) + py, px = (sz - nh) // 2, (sz - nw) // 2 + canvas[py:py + nh, px:px + nw] = cv2.resize(img, (nw, nh)) + return canvas, r, px, py + + +def detect_text_regions(img, conf: float = None) -> list: + """img: BGR ndarray (one panel). -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels, + reading order (top->bottom, left->right). Empty list on no detections (caller falls back to the + holistic prompt). Load + inference are lazy so importing this never touches the GPU or the model.""" + conf = CONF if conf is None else conf + h, w = img.shape[:2] + canvas, r, px, py = _letterbox(img) + x = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB).astype(np.float32).transpose(2, 0, 1)[None] / 255.0 + blk = _load().run(None, {"images": x})[0][0] # [N,7] = cx,cy,w,h,conf,cls0,cls1 in 1024 space + m = blk[:, 4] > conf + if not m.any(): + return [] + b = blk[m] + rects = [[float(cx - bw / 2), float(cy - bh / 2), float(bw), float(bh)] for cx, cy, bw, bh in b[:, :4]] + idx = cv2.dnn.NMSBoxes(rects, b[:, 4].tolist(), conf, 0.5) + if len(idx) == 0: + return [] + out = [] + for i in np.array(idx).flatten(): + cx, cy, bw, bh = b[i, :4] + x1, y1 = max(0, int((cx - bw / 2 - px) / r)), max(0, int((cy - bh / 2 - py) / r)) + x2, y2 = min(w, int((cx + bw / 2 - px) / r)), min(h, int((cy + bh / 2 - py) / r)) + if x2 > x1 and y2 > y1: + out.append({"bbox": [x1, y1, x2, y2], "conf": round(float(b[i, 4]), 3)}) + out.sort(key=lambda d: (d["bbox"][1], d["bbox"][0])) + return out + + +def draw_region_marks(img, regions: list): + """draw a numbered red box (#1..#N) over each text region on a COPY of img; return the marked + image. the number is what gemma cites in set-of-mark attribution ("region 2 -> P1").""" + vis = img.copy() + for n, reg in enumerate(regions, 1): + x1, y1, x2, y2 = reg["bbox"] + cv2.rectangle(vis, (x1, y1), (x2, y2), (0, 0, 255), 2) + cv2.rectangle(vis, (x1, max(0, y1 - 22)), (x1 + 26, y1), (0, 0, 255), -1) + cv2.putText(vis, str(n), (x1 + 3, y1 - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2) + return vis + + +def draw_face_marks(img, faces: list): + """draw a labelled green box for each character face/body, IN PLACE, return img. faces: + [{"label": "P1", "bbox": [x1,y1,x2,y2]}]. gemma names the label (P1/P2) as the region's speaker, + which the worker maps back to that character's local_id.""" + for f in faces: + x1, y1, x2, y2 = f["bbox"] + cv2.rectangle(img, (x1, y1), (x2, y2), (0, 170, 0), 2) + cv2.rectangle(img, (x1, y1), (min(x2, x1 + 40), y1 + 22), (0, 170, 0), -1) + cv2.putText(img, f["label"], (x1 + 3, y1 + 17), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2) + return img + + +if __name__ == "__main__": + # self-check: the real ONNX model must find the text inside a synthetic speech bubble, and the + # returned box must overlap the text we drew. Exercises letterbox + decode + NMS end to end. + panel = np.full((600, 400, 3), 200, np.uint8) + cv2.ellipse(panel, (200, 200), (150, 90), 0, 0, 360, (255, 255, 255), -1) + cv2.ellipse(panel, (200, 200), (150, 90), 0, 0, 360, (0, 0, 0), 3) + cv2.putText(panel, "HELLO", (110, 215), cv2.FONT_HERSHEY_SIMPLEX, 1.8, (0, 0, 0), 5) + regs = detect_text_regions(panel) + assert regs, "no text region detected in synthetic bubble" + # at least one box should contain the drawn text centre (~200,200) + hit = any(x1 <= 200 <= x2 and y1 <= 200 <= y2 for (x1, y1, x2, y2) in (r["bbox"] for r in regs)) + assert hit, f"no region covers the text centre: {regs}" + marked = draw_region_marks(panel, regs) + assert marked.shape == panel.shape and marked is not panel + # face marks: labelled box drawn in place; sample the left border low enough to miss the bubble + draw_face_marks(marked, [{"label": "P1", "bbox": [50, 180, 350, 500]}]) + assert tuple(int(v) for v in marked[490, 51]) == (0, 170, 0), f"face mark not green: {marked[490, 51]}" + print(f"bubble_detect self-check ok ({len(regs)} region(s), top conf {regs[0]['conf']})") diff --git a/collage.py b/collage.py new file mode 100644 index 0000000..50fc01a --- /dev/null +++ b/collage.py @@ -0,0 +1,169 @@ +# collage.py — task 183 deterministic layout planner for animated manga collages. +# Pure geometry, no ffmpeg/IO: ordered panels (+ aspects, RTL, emphasis, frame) -> a layout state +# (template, aspect-correct resting rectangles, entrance vectors, z-order, hold/transition timing). +# worker_render turns a state into a clip; this module decides WHERE everything sits so the decision +# is testable without rendering. Coordinates are pixels in the output frame, origin top-left. +# +# ponytail: a bounded template set (n=1..4) covering the reference's common resting layouts, not all 7. +# Add templates when a fixture proves a missing one is needed — the fit/RTL/entrance machinery is shared. + +MARGIN = 0.035 # outer + inter-panel gap as a fraction of the frame's short side +HOLD_S = 3.0 # reference holds run ~2-4s; #185 overrides per-beat from narration length +TRANS_S = 0.45 # reference transitions run ~0.3-0.6s + + +def _fit(aspect, slot): + """Aspect-correct rect centered inside slot (x,y,w,h). aspect=w/h. No stretch: letterbox to fit.""" + sx, sy, sw, sh = slot + if aspect >= sw / sh: # panel wider than slot -> width-bound + w = sw; h = w / aspect + else: # taller than slot -> height-bound + h = sh; w = h * aspect + return (sx + (sw - w) / 2.0, sy + (sh - h) / 2.0, w, h) + + +def _slots(template, n, W, H, g): + """Slot rectangles (pre-fit) for a template, in READING order (slot[0] = read first). + g = gap in pixels. Slots tile the safe area; _fit later letterboxes each panel inside its slot.""" + x0, y0 = g, g + fw, fh = W - 2 * g, H - 2 * g + if template == "centered_wide": + return [(x0, y0, fw, fh)] + if template == "vertical_pair": # two columns + cw = (fw - g) / 2.0 + return [(x0, y0, cw, fh), (x0 + cw + g, y0, cw, fh)] + if template == "stacked_wides": # two rows + rh = (fh - g) / 2.0 + return [(x0, y0, fw, rh), (x0, y0 + rh + g, fw, rh)] + if template == "strip_over_dominant": # thin top strip (reads first), big bottom + sh = fh * 0.32 + return [(x0, y0, fw, sh), (x0, y0 + sh + g, fw, fh - sh - g)] + if template == "supporting_left_dominant_right": # small stack left, one dominant right + lw = fw * 0.34 + k = max(1, n - 1) + rh = (fh - (k - 1) * g) / k + left = [(x0, y0 + i * (rh + g), lw, rh) for i in range(k)] + return left + [(x0 + lw + g, y0, fw - lw - g, fh)] + if template == "quad": # 2x2 + cw, rh = (fw - g) / 2.0, (fh - g) / 2.0 + return [(x0, y0, cw, rh), (x0 + cw + g, y0, cw, rh), + (x0, y0 + rh + g, cw, rh), (x0 + cw + g, y0 + rh + g, cw, rh)] + raise ValueError(f"unknown template {template}") + + +def _pick_template(aspects): + """Choose a resting template from panel count and aspect ratios (w/h).""" + n = len(aspects) + if n <= 1: + return "centered_wide" + if n == 2: + if all(a < 0.9 for a in aspects): # two tall panels -> side by side + return "vertical_pair" + if all(a > 1.15 for a in aspects): # two wide panels -> stacked + return "stacked_wides" + return "vertical_pair" + if n == 3: + return "supporting_left_dominant_right" + return "quad" # 4 (planner caps callers at 4) + + +def plan_layout(aspects, rtl=True, active=0, frame=(1080, 1920)): + """Deterministic collage layout for one beat. + aspects: panel width/height ratios in reading order (len 1..4). + rtl: right-to-left reading (manga) -> reading-first panel takes the RIGHTmost horizontal slot. + active: index of the dominant/emphasized panel (gets the largest slot where a template has one). + returns dict: template, rects[(x,y,w,h)] aligned to input panel order, entrances[(dx,dy)] pixel + offset a panel starts at before sliding to rest, z_order, emphasis, hold_s, transition_s. + Rects/entrances are indexed to match the INPUT panel order (not slot order).""" + n = len(aspects) + if n == 0: + return {"template": "empty", "rects": [], "entrances": [], "z_order": [], + "emphasis": 0, "hold_s": HOLD_S, "transition_s": TRANS_S} + W, H = frame + g = int(MARGIN * min(W, H)) + template = _pick_template(aspects) + slots = _slots(template, n, W, H, g) + + # Map input panels (reading order) to slots. Templates whose first slots are a horizontal run + # honor RTL by reversing that run so the reading-first panel lands on the right. + order = list(range(n)) + if rtl and template in ("vertical_pair", "quad"): + if template == "vertical_pair": + order = [1, 0] + else: # quad: reverse each row + order = [1, 0, 3, 2][:n] + # dominant panel takes the dominant slot when the template has a distinguished one (last slot). + if template in ("strip_over_dominant", "supporting_left_dominant_right") and 0 <= active < n: + rest = [i for i in range(n) if i != active] + order = rest + [active] # dominant slot is the last one in _slots order + + rects = [None] * n + for slot_i, panel_i in enumerate(order): + rects[panel_i] = _fit(aspects[panel_i], slots[slot_i]) + + # entrances: reading-direction slide for horizontal templates; dominant scales in place (dy=0,dx=0). + edge = W if rtl else -W # RTL panels enter from the right (+x), LTR from left + entrances = [] + for i in range(n): + if i == active and template in ("centered_wide", "strip_over_dominant", + "supporting_left_dominant_right"): + entrances.append((0, 0)) # emphasized panel resolves by scale, not slide + elif template == "stacked_wides": + entrances.append((0, -H if i == 0 else H)) # rows drop/rise into place + else: + entrances.append((edge, 0)) + # supporting panels sit above the dominant in z so their shadow reads; dominant drawn first (back). + z_order = sorted(range(n), key=lambda i: 0 if i == active else 1) + return {"template": template, "rects": rects, "entrances": entrances, "z_order": z_order, + "emphasis": active, "hold_s": HOLD_S, "transition_s": TRANS_S} + + +if __name__ == "__main__": + W, H = 1080, 1920 + + def _inside(r): + x, y, w, h = r + return x >= -1 and y >= -1 and x + w <= W + 1 and y + h <= H + 1 + + # 1) single panel -> centered_wide, aspect preserved, inside frame. + p = plan_layout([1.5], frame=(W, H)) + assert p["template"] == "centered_wide" and len(p["rects"]) == 1 + x, y, w, h = p["rects"][0] + assert abs(w / h - 1.5) < 1e-3 and _inside(p["rects"][0]) + + # 2) two tall panels -> vertical_pair; RTL puts reading-first (panel 0) on the RIGHT. + p = plan_layout([0.6, 0.6], rtl=True, frame=(W, H)) + assert p["template"] == "vertical_pair" + assert p["rects"][0][0] > p["rects"][1][0], "RTL: panel 0 is rightmost" + # LTR flips it. + q = plan_layout([0.6, 0.6], rtl=False, frame=(W, H)) + assert q["rects"][0][0] < q["rects"][1][0], "LTR: panel 0 is leftmost" + + # 3) two wide panels -> stacked_wides; panel 0 on top. + p = plan_layout([1.6, 1.6], frame=(W, H)) + assert p["template"] == "stacked_wides" and p["rects"][0][1] < p["rects"][1][1] + + # 4) three panels, active=2 -> dominant takes the large right slot (widest rect). + p = plan_layout([0.7, 0.7, 1.3], active=2, frame=(W, H)) + assert p["template"] == "supporting_left_dominant_right" + dom = p["rects"][2][2] * p["rects"][2][3] + assert all(dom >= p["rects"][i][2] * p["rects"][i][3] for i in (0, 1)), "active is dominant area" + assert all(_inside(r) for r in p["rects"]) + + # 5) aspect never stretched: fitted rect ratio == input aspect for every panel/template. + for asp in ([1.5], [0.6, 0.6], [1.6, 1.6], [0.7, 0.7, 1.3], [1.0, 1.0, 1.0, 1.0]): + pl = plan_layout(asp, frame=(W, H)) + for a, r in zip(asp, pl["rects"]): + assert abs(r[2] / r[3] - a) < 1e-3, (asp, a, r) + + # 6) quad RTL reverses each row; all four rects disjoint-ish (tile the frame). + p = plan_layout([1, 1, 1, 1], rtl=True, frame=(W, H)) + assert p["template"] == "quad" and len(p["rects"]) == 4 + assert p["rects"][0][0] > p["rects"][1][0], "top row RTL: panel0 right of panel1" + + # 7) entrances: RTL horizontal-slide panels start off the right edge; deterministic. + p = plan_layout([0.6, 0.6], rtl=True, frame=(W, H)) + assert any(dx > 0 for dx, _ in p["entrances"]), "RTL entrance from right" + assert plan_layout([0.6, 0.6], frame=(W, H)) == plan_layout([0.6, 0.6], frame=(W, H)) + + print("collage self-check ok") diff --git a/face_detect.py b/face_detect.py new file mode 100644 index 0000000..c065688 --- /dev/null +++ b/face_detect.py @@ -0,0 +1,74 @@ +# face_detect.py — anime face detector (YOLOv8, ONNX) -> real character-face boxes for set-of-mark +# speaker attribution. The vision DETECT stage returns gemma-guessed bboxes which are too imprecise to +# draw a mark on; this gives grounded boxes the same way bubble_detect gives grounded text regions. +# Model: deepghs/anime_face_detection face_detect_v1.4_s (single class "face"). CPU EP -> no GPU contention. +# ponytail: face boxes only (not full body). Attribution just needs "which face said this"; add a +# person head only if off-panel/back-turned speakers need body grounding. +import os +import numpy as np +import cv2 + +MODEL = os.environ.get("FACE_MODEL", + os.path.join(os.path.dirname(__file__), "models", "anime_face_v1.4_s.onnx")) +CONF = float(os.environ.get("FACE_CONF", "0.30")) +_sess = None + + +def _load(): + global _sess + if _sess is None: + import onnxruntime as ort + _sess = ort.InferenceSession(MODEL, providers=["CPUExecutionProvider"]) + return _sess + + +def _letterbox(img, sz=640): + h, w = img.shape[:2] + r = min(sz / h, sz / w) + nh, nw = int(round(h * r)), int(round(w * r)) + canvas = np.full((sz, sz, 3), 114, np.uint8) + py, px = (sz - nh) // 2, (sz - nw) // 2 + canvas[py:py + nh, px:px + nw] = cv2.resize(img, (nw, nh)) + return canvas, r, px, py + + +def detect_faces(img, conf: float = None) -> list: + """img: BGR ndarray. -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels, sorted + reading order. Empty on no faces. Lazy load so import never touches the model.""" + conf = CONF if conf is None else conf + h, w = img.shape[:2] + canvas, r, px, py = _letterbox(img) + x = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB).astype(np.float32).transpose(2, 0, 1)[None] / 255.0 + out = _load().run(None, {"images": x})[0][0].T # (8400,5) = cx,cy,w,h,conf in 640 space + m = out[:, 4] > conf + if not m.any(): + return [] + b = out[m] + rects = [[float(cx - bw / 2), float(cy - bh / 2), float(bw), float(bh)] for cx, cy, bw, bh in b[:, :4]] + idx = cv2.dnn.NMSBoxes(rects, b[:, 4].tolist(), conf, 0.45) + if len(idx) == 0: + return [] + res = [] + for i in np.array(idx).flatten(): + cx, cy, bw, bh = b[i, :4] + x1, y1 = max(0, int((cx - bw / 2 - px) / r)), max(0, int((cy - bh / 2 - py) / r)) + x2, y2 = min(w, int((cx + bw / 2 - px) / r)), min(h, int((cy + bh / 2 - py) / r)) + if x2 > x1 and y2 > y1: + res.append({"bbox": [x1, y1, x2, y2], "conf": round(float(b[i, 4]), 3)}) + res.sort(key=lambda d: (d["bbox"][1], d["bbox"][0])) + return res + + +if __name__ == "__main__": + # self-check needs a real anime face; use a page from the sample series if present, else skip loudly. + import glob + pages = glob.glob("/mnt/server/mnt/hdd2/manga-raw/*/*.png") + if not pages: + print("face_detect: no sample page available, skipping live check") + else: + faces = detect_faces(cv2.imread(sorted(pages)[13])) + assert faces, "no face detected on a page that should have characters" + for f in faces: + x1, y1, x2, y2 = f["bbox"] + assert x2 > x1 and y2 > y1 and f["conf"] >= CONF + print(f"face_detect self-check ok ({len(faces)} face(s), top conf {faces[0]['conf']})") diff --git a/identity-rework-task.md b/identity-rework-task.md new file mode 100644 index 0000000..0260e3e --- /dev/null +++ b/identity-rework-task.md @@ -0,0 +1,67 @@ +# Rework manga character identity: tracklet spine + gemma resolver + +Replace siglip-cosine as the primary identity signal. Priority: high. + +## Problem +Identity quality plateaus because **siglip2 cosine is the wrong instrument**: it's a +semantic encoder ("man, office, manga panel"), not an instance re-id model. Different +people in the same setting score high; the same person across scenes scores low. No +threshold / gate / gallery fixes what the vector *means* — they only shave ~10% off the +error. Symptoms on the test chapter: +- male-brown colleague labeled as female-black **Choi Haeseon** +- speech bubble mis-attributed (MC credited, colleague actually speaking) +- "MC becomes Choi Haeseon" at the end (reconcile over-merge) + +## Already done (guards against *new* contamination, not a fix for existing data) +- Gender gate in `worker_identity.match()` candidate filter + `_pending_match` + (opposite decided genders never match). +- Reconcile gender gate in `service.run_stage_reconcile` (skip pair if decided genders differ). +- `db.create_character` name-dedup: skip fold if decided genders conflict + (vision-hallucinated name on a wrong-gender crop). +- **NOTE:** existing roster is already polluted; guards only stop NEW contamination. + Needs `/stage/clear identity reconcile scene script` + re-run to benefit. + +## Target architecture — two-tier (from multi-object tracking) +**Tier 1 — tracklets (local, greedy, high-precision).** +Link per-panel detections into per-person tracklets across a 5–10 panel window. Local +association is the reliable regime (same page, adjacent shot, stable appearance, often the +same speaker). Don't touch global identity yet. A tracklet = ordered crops for one person → +carries an embedding gallery + merged attributes + associated speaker/dialogue. + +**Tier 2 — resolve tracklet → global character (deliberate, max evidence).** +Once a tracklet is stable, resolve identity ONCE per person per scene, using a whole gallery +of views + gemma's opinion — not greedily per-crop. Multi-view (front/side/angry/crying) +falls out for free (a tracklet accumulates poses as it spans panels). + +## Signal fusion — keep ORDINAL, not a 9-weight learned sum +No labeled data to tune weights → a hand-weighted 9-signal cost is a worse treadmill. +Make it ordinal: +- **HARD GATE:** gender, species → block impossible matches. +- **DECIDER:** gemma "same person?" vs a text character-sheet (reuse reconcile `/same`). +- **STRONG FEATURE:** hair color+style, name / honorific / alias from dialogue + (Korean honorifics = gold anchors). +- **TIEBREAK:** embedding gallery max-sim (K~5), dialogue-speaker continuity, scene co-presence. + +Only build a learned/weighted scorer AFTER labeling a couple chapters. + +## Attributes to add (cherry-picked) +- Yes / cheap / high-value: honorifics, aliases, hair color/style, gender, species. +- Skip for now (noisy, marginal): age, body-shape, eye color — add only when a specific + mislabel needs them. + +## Reuses existing infra (rewiring, not greenfield) +- `/direct/window` = windowed multi-image gemma calls in reading order → tracklet window. +- reconcile `/same` = pairwise "same person?" gemma call → tier-2 adjudicator. +- appearance attrs, gender, dialogue+speaker already in the scene graph. + +## Suggested sequencing +1. **Tier-2 first** (~1 day, most of the win): gemma resolver vs text character-sheet, + gender-gated, gallery max-sim as tiebreak. Swaps cosine-decider for gemma-decider. +2. **Tier-1 tracklet spine:** link detections → tracklets, carry gallery + merged attrs, + resolve once per tracklet. Robust across poses; contains errors to a tracklet not a crop. +3. Add dialogue-continuity / scene-co-presence tiebreaks. +4. (Later, if labeled data exists) learned association cost. + +## Open question +Confirm "**gemma decides, tracklets give it evidence**" is the intended core +(vs the fusion score being the core). diff --git a/manga-recap-pipeline-spec.md b/manga-recap-pipeline-spec.md new file mode 100644 index 0000000..9571fea --- /dev/null +++ b/manga-recap-pipeline-spec.md @@ -0,0 +1,248 @@ +# Manga/Manhwa Recap Pipeline — Technical Spec + +**Status:** draft v1 +**Target:** fully local, single-GPU, resumable batch pipeline. No publishing (copyright out of scope). +**Core constraint:** no new PyTorch / ROCm wheels. Existing llama.cpp (gemma) stays. Everything else runs on `onnxruntime + opencv + numpy + scikit-learn`. + +--- + +## 1. Goal + +Given raw manga pages or manhwa strips for a series, automatically produce a narrated recap video with burned-in subtitles, with minimal human involvement (one-time character-bank setup per series). + +This spec replaces a dependency on the `magi`/`magiv2` model by **decomposing its subtasks** into swappable, low-dependency components. + +--- + +## 2. Design principles + +- **Dependency isolation.** gemma runs on llama.cpp (unchanged). All CV/ML runs via ONNX Runtime. Detection models are small enough to run on CPU EP; ROCm EP optional. No `transformers`, no second torch install. +- **Specialists over generalist.** Do not ask one VLM to detect + identify + attribute. Each subtask is a dumb specialized component; a clean data bus (JSON files) connects them. +- **Resumable.** Every stage reads its input artifact from disk and writes its output artifact to disk. A stage is skipped if its output exists and is newer than its input (unless `--force`). +- **Supervised shortcut for identity.** Replace magi's unsupervised character clustering with a per-series character bank (exemplar crops + names). Simpler, more robust, and how magiv2 gets names anyway. +- **Fail loud, fail per-panel.** Low-confidence panels are flagged, not silently guessed. Confidence thresholds are config, not magic numbers in code. + +--- + +## 3. Architecture overview + +``` +ingest → [1] panels → [2] detect → [3] identity → [4] text + │ + [5] speaker-bind ←──────────┤ + │ + [6] filter essential ←──────┘ + │ + [7] transcript (ordered) + │ + ┌─────────────────────┴───────────────────┐ + [8] scene-action (gemma) (transcript.json) + └─────────────────────┬───────────────────┘ + │ + [9] script gen (LLM, chapter + rolling summary) + │ + [10] TTS (dots.tts) ──► [11] visual assembly (ffmpeg) ──► [12] subs (faster-whisper) ──► [13] mux +``` + +Stages 2–7 are the magi replacement. Stages 8–13 are the existing recap backend. + +--- + +## 4. Filesystem layout + +``` +work/ + / + source/ # input pages/strips + ch/ + panels/ # panel crops, ordered + panels.json # [1] + detections.json # [2] + identities.json # [3] + texts.json # [4] + transcript.json # [5][6][7] merged + script.json # [9] + audio/ # [10] wav per segment + video.mp4 # [13] final + bank/ + bank.json # character bank (per series, hand-built once) + crops/ # exemplar images +``` + +Artifact-per-stage = resumability. Delete an artifact to re-run that stage forward. + +--- + +## 5. Stage specs + +### [1] Panel extraction +- **Manga (page-based):** `kumiko` → ordered panel polygons. Pure OpenCV. +- **Manhwa (vertical strip):** slice at horizontal whitespace bands. OpenCV: row-wise background uniformity → cut points. Produces pseudo-panels. +- **In:** `source/*` · **Out:** `panels/`, `panels.json` +- **Dep:** kumiko, opencv. No torch. + +```jsonc +// panels.json +{ + "type": "manga", // or "manhwa" + "reading_order": "rtl", // rtl | ltr | ttb + "panels": [ + { "id": "p001", "page": 1, "bbox": [x,y,w,h], "file": "panels/p001.png", "order": 0 } + ] +} +``` + +### [2] Detection (text + balloons + characters) +- **Text + balloons:** `comic-text-detector` (ships ONNX). Returns text regions + balloon masks. This is what kills the OCR problem — you never OCR a full page again. +- **Character boxes:** YOLOv8 anime face/person model exported to ONNX. CPU-fine. +- **In:** `panels/` · **Out:** `detections.json` +- **Dep:** onnxruntime, opencv, numpy. + +```jsonc +// detections.json (per panel) +{ + "p001": { + "balloons": [ + { "id": "b0", "mask_poly": [[x,y],...], "centroid": [x,y], "bbox": [x,y,w,h] } + ], + "text_regions": [ + { "id": "t0", "bbox": [x,y,w,h], "in_balloon": "b0" } // null if floating (sfx/sign) + ], + "chars": [ + { "id": "c0", "bbox": [x,y,w,h], "crop": "..." } + ] + } +} +``` + +### [3] Character identity (bank match) +- Embed every `chars[*]` crop with SigLIP or an anime ArcFace model (ONNX). +- Cosine-match against `bank.json` entries. Above threshold → assign name; else `"unknown"`. +- **Bank is built once per series by hand** (10 min): a few exemplar crops + a name each. This is the only required human touch. +- **In:** `detections.json`, `bank/` · **Out:** `identities.json` +- **Dep:** onnxruntime, numpy, sklearn (cosine / nearest-neighbour). +- **Note:** this is the accuracy-critical stage. Generic embeddings + same-face syndrome = the weakest link vs magi. The bank is what rescues it. Do NOT attempt unsupervised clustering as primary — it merges lookalikes. + +```jsonc +// bank.json +{ "characters": [ + { "name": "Aria", "exemplars": ["bank/crops/aria_0.png","bank/crops/aria_1.png"] } +]} +// identities.json (per panel) +{ "p001": { "c0": { "name": "Aria", "score": 0.82 }, "c1": { "name": "unknown", "score": 0.41 } } } +``` + +### [4] Text extraction +- For each `in_balloon` text region: crop the balloon, feed the clean crop to gemma (llama.cpp) → read text. Isolated crops read far better than full pages. +- **In:** `detections.json`, `panels/` · **Out:** `texts.json` +- **Dep:** llama.cpp (existing). No torch. + +```jsonc +// texts.json +{ "p001": { "t0": "We can't stay here." } } +``` + +### [5] Speaker binding +Two paths — start geometry, fall back to gemma on low confidence. + +- **A — tail geometry (default, pure CV):** from the balloon mask, find the tail = sharpest protrusion off the centroid. Vector centroid→tip; nearest `chars[*]` box along that ray = speaker. Hand-rolled magiv2 tail logic. +- **B — set-of-mark + gemma (fallback):** draw numbered boxes on chars + balloons on the panel image, ask gemma "balloon 2 → face #3 or #5?". Grounding via drawn marks beats free-form spatial reasoning. No new deps. +- **Trigger fallback when:** >2 candidate chars, ambiguous/absent tail, or geometry confidence < threshold. +- **In:** `detections.json`, `identities.json`, `texts.json` · **Out:** merged into `transcript.json`. + +### [6] Essential vs non-essential filter +- Pure geometry: `text_region.in_balloon != null` → dialogue. Floating on raw art → sfx/sign → **drop**. Kills "THUD" and street-sign garbage with no classifier. + +### [7] Transcript assembly (reading order) +- Panel order from `panels.json`. Within panel: manga = sort (top → right-to-left); manhwa = top-down. +- Emit ordered speaker+line list. + +```jsonc +// transcript.json +{ "chapter": 12, "lines": [ + { "panel": "p001", "speaker": "Aria", "line": "We can't stay here.", "conf": 0.79, "flagged": false } +]} +``` + +### [8] Scene-action description +- gemma describes physical action per panel ("she draws a sword") — the one thing magi does NOT do and gemma is good at. Runs in parallel with 2–7. +- **Out:** `scene` field per panel, merged for the script stage. + +### [9] Script generation +- LLM input = **one chapter of transcript + scene-actions + a ~500-token rolling summary** of prior chapters. Do NOT token-max the context; attention degrades mid-window and coherence/attribution drop. +- Prompt: compress and narrate, not transcribe. Output narration segments each mapped to source panel IDs (needed for visual timing). +- **Out:** `script.json` + +```jsonc +// script.json +{ "segments": [ + { "id": "s0", "text": "Cornered in the ruins, Aria makes her choice...", "panels": ["p001","p002"] } +]} +``` + +### [10] TTS +- `dots.tts`, wav per segment. Keep durations — they drive visual timing. + +### [11] Visual assembly +- ffmpeg ken-burns (pan/zoom) per panel. Panel display time = proportional to its segment's audio length. moviepy to orchestrate or raw filtergraphs for lean/fast. + +### [12] Subtitles +- `faster-whisper` on generated audio → timestamped SRT → burn in with ffmpeg. Easier than aligning from the script side. + +### [13] Mux +- Combine video + audio + burned subs → `video.mp4`. + +--- + +## 6. Dependency matrix + +| Stage | Tooling | New torch? | +|---|---|---| +| 1 panels | kumiko, opencv | no | +| 2 detect | comic-text-detector (onnx), yolo-anime (onnx) | no | +| 3 identity | siglip/arcface (onnx), sklearn | no | +| 4 text | llama.cpp gemma | no | +| 5 bind | numpy geometry + gemma fallback | no | +| 6 filter | numpy geometry | no | +| 7 order | numpy | no | +| 8 scene | llama.cpp gemma | no | +| 9 script | local LLM | no | +| 10 tts | dots.tts | (its own env) | +| 11 video | ffmpeg, moviepy | no | +| 12 subs | faster-whisper | (ctranslate2, not torch) | +| 13 mux | ffmpeg | no | + +Net: the magi replacement (2–7) adds **only ONNX Runtime + a couple of onnx model files**. No ROCm wheel churn. + +--- + +## 7. Orchestration + +- Bash driver calls python stage scripts; artifacts handed off as files. +- Each stage: `stage_N.py --series X --chapter NN [--force]`. +- Skip logic: if output exists and `mtime(output) > mtime(input)` and not `--force`, skip. +- Model loading: sequence stages so you never hold two large models in VRAM at once (gemma vs detectors vs faster-whisper). One card handles all, serially. +- Do NOT wrap in a Spring service. This is a batch job, not request/response — a service is pure added attack surface and state for zero benefit. + +--- + +## 8. Hardware notes + +- gemma (mmproj) is the VRAM heavyweight; detectors are CPU-viable. +- Sequence model loads; never co-resident. Target: single decent GPU, staged. +- Detection/embedding on CPU EP is fine and frees VRAM for gemma. + +--- + +## 9. Known risks / caveats + +- **Re-ID accuracy** is the weak link vs magi's end-to-end association. Mitigation: character bank (mandatory), tune cosine threshold per series, flag `unknown` rather than guess. +- **Manhwa layout:** comic-text-detector and the anime detectors are manga/anime-trained. Vertical webtoon art + non-japanese layout = degraded results. Expect tuning on the whitespace slicer and lower binding confidence. +- **Tail geometry** fails on off-panel speakers and thought bubbles. Fallback to gemma SoM; if still ambiguous, flag the line. +- **Rolling summary drift:** long series accumulate summary error. Periodically re-anchor the summary from a canonical synopsis if available. + +--- + +## 10. Pre-build check (do this first) + +Before building any of this: magi's HF weights may load through your **existing** ROCm torch directly, ignoring its pinned `requirements.txt` (the version pain is usually the wrapper deps, not torch). 5-minute test. If it instantiates and runs, you skip this entire rebuild. If it OOMs or the arch won't load, decompose per this spec. diff --git a/plan.md b/plan.md new file mode 100644 index 0000000..f814dc4 --- /dev/null +++ b/plan.md @@ -0,0 +1,414 @@ +# Pipeline correctness plan + +## Goal + +Make character identity, naming, dialogue transcription, speaker attribution, scene grouping, and +script generation fail loudly instead of silently turning uncertain model output into canonical story +facts. + +The desired end state is an evidence-bearing pipeline in which every name, speaker, quote, action, +and scene boundary can be traced to its source panel and confidence. Operational batching must never +create narrative boundaries. + +## Scope and repository boundary + +This worker repository implements the model and media stages. Scheduling, database mutations, +tracklet reconciliation, backfills, beat aggregation, and review gates live in the separate homesrv +orchestrator repository. Worker contract changes below must therefore be implemented and tested in +both repositories. + +The current intended flow is: + +```text +crop/order panels + -> character detection + -> identity shortlist and tracklets + -> identity adjudication/reconciliation + -> grounded dialogue transcription + -> speaker resolution + -> scene graph assembly + -> scene/beat grouping + -> narration and verification + -> TTS/render +``` + +## Current strengths + +- Vision, dialogue, identity, direction, and script generation are separate focused passes. +- Identity uses gender-gated SigLIP shortlists and has a Gemma Tier-2 resolver. +- Unnamed one-off characters are held until a second sighting instead of always polluting the + permanent registry. +- Dialogue has a short-window mode for cross-panel turn-taking. +- Set-of-mark text and face grounding exists. +- Character-detection parse failures are distinguished from editorial `skip` decisions. +- Panel ordering and suspicious overlap checks exist. +- Direction and script stages support multi-panel beats. + +## P0: stop silent corruption + +These changes should land before further prompt tuning or visual polish. + +### 1. Introduce typed, canonical speaker references + +Problem: dialogue is allowed to return either a panel-local ID or an off-panel character name, but +scene assembly only maps local IDs. A valid off-panel name can therefore become `None` and be narrated +as "Someone." + +Use an explicit speaker representation at the dialogue boundary: + +```json +{ + "speaker": { + "kind": "local_id|character_id|name|unknown|narrator", + "value": "person_1" + } +} +``` + +Normalize this to `character_id` before building a scene graph. Name lookup must support canonical +names and aliases, reject ambiguous duplicate-name matches, and retain unresolved names as unresolved +rather than discarding them. + +Acceptance criteria: + +- An off-panel named speaker resolves to the correct canonical character. +- An unknown speaker remains unknown and is never assigned to the main character by default. +- Duplicate/ambiguous names produce a review flag, not an arbitrary selection. +- Scene graphs never receive raw `person_N` or free-form name values as canonical speakers. + +### 2. Preserve confidence, method, and provenance end to end + +Problem: dialogue confidence and identity ambiguity are currently discarded by scene assembly. A weak +turn-taking guess is narrated exactly like a clearly grounded bubble tail. + +Every dialogue line should retain at least: + +```json +{ + "dialogue_id": "p012_r03", + "panel_id": "p012", + "region_id": "r03", + "text": "...", + "type": "speech", + "speaker_id": "character_x", + "speaker_confidence": 0.72, + "speaker_method": "tail|som_face|turn_taking|solo_prior|manual", + "identity_confidence": 0.91, + "flags": [] +} +``` + +Acceptance criteria: + +- Confidence and method survive dialogue -> scene -> script review artifacts. +- Low-confidence attribution can be rendered neutrally without asserting a name. +- Configurable thresholds decide accept/review/reject; prompts do not decide policy. +- Manual corrections are distinguishable from model output and are never overwritten on resume. + +### 3. Make all technical failures fail loudly + +Problem: dialogue JSON failure and missing window results currently become empty dialogue, which is +indistinguishable from a genuinely silent panel. + +All model endpoints should return explicit status metadata: + +```json +{ + "status": "ok|partial|failed", + "parse_failed": false, + "expected_items": 3, + "returned_items": 3, + "warnings": [] +} +``` + +Acceptance criteria: + +- Parse failure cannot mark a stage complete. +- A window response missing one requested panel becomes `partial` and queues only the missing work. +- Empty dialogue is accepted only when the model completed successfully and grounded detection found + no relevant text. +- Script/TTS/render cannot run for panels or beats with unresolved upstream technical failures. + +### 4. Remove the unconditional solo-character speaker assignment + +Problem: one visible character does not prove they own a bubble; reaction panels often show the +listener while an off-panel character speaks. + +Treat sole presence as a prior only. It may raise a score, but it must not overwrite `unknown` without +tail/region evidence or strong conversation continuity. + +Acceptance criteria: + +- A one-face reaction panel can retain an off-panel speaker. +- Solo attribution records its evidence and confidence. +- Tests cover visible speaker, visible listener/off-panel speaker, thought bubble, and narration box. + +## P1: ground transcription and identity + +### 5. Make region-grounded dialogue the primary path + +Set-of-mark text/face grounding currently exists but is disabled by default. Calibrate it on real +titles, then make it the normal path with a clearly reported holistic fallback. + +Required validation: + +- One output record per detected dialogue/caption region. +- No region can be consumed twice. +- Every line remains attached to the panel containing its region. +- Unmarked text invented by the VLM is rejected or flagged. +- Tail/face evidence is recorded separately from conversation-flow inference. +- Detector failure and zero detected regions are distinct states. + +Do not treat all non-Latin text as inherently decorative. Language filtering should be configurable +per title/source; otherwise legitimate original-language dialogue can be deleted. + +### 6. Separate name claims from canonical names + +Problem: detection can emit a name and identity immediately persists a named unmatched character. +This lets one visual-language-model guess become permanent identity state and bypass the two-sighting +guard. + +Represent naming as evidence: + +```json +{ + "claim_id": "...", + "panel_id": "p012", + "name": "Seonho", + "target_local_id": "person_2", + "evidence_type": "address|self_intro|caption|name_tag|roster_hint", + "confidence": 0.84 +} +``` + +Canonical naming should require corroboration or review. Roster names are hints, never identity +evidence. A name claim alone must not create or merge a canonical character. + +Acceptance criteria: + +- Addressed-person and speaking-person tests cannot swap the name. +- Nearby chart/UI labels cannot become canonical names. +- Conflicting name claims are surfaced. +- A later confirmed name safely backfills earlier appearances and narration previews. + +### 7. Strengthen character identity evidence + +Current risks include context-heavy body crops, outfit changes, partial occlusion, a global threshold, +and a Tier-2 resolver that compares the crop mainly against generated text descriptions. + +Implement: + +- Face-first embeddings, with body/outfit evidence as a secondary channel. +- Multiple reference exemplars per canonical character. +- Tracklet aggregation across adjacent/overlapping windows before committing identity. +- Per-title calibrated thresholds. +- A real open-set result: known match, new character, or unresolved. +- Candidate reference images in Tier-2 comparison, not only prose descriptions. +- A same-panel exclusivity constraint: two distinct visible people cannot resolve to one identity + unless reflection/duplicate-art evidence explicitly allows it. +- Alias and duplicate-name collision checks. + +Acceptance criteria: + +- Outfit changes do not create a new identity when face evidence is strong. +- Similar-looking simultaneous characters do not collapse into one identity. +- Weak crops remain unresolved rather than minting or merging characters. +- Identity decisions expose all contributing evidence and thresholds. + +## P1: preserve panel semantics across solo and batch processing + +### 8. Stop destructively merging faceless webtoon fragments + +Problem: vertically stacking a text-only fragment with a nearby face changes the visual evidence and +can manufacture speaker proximity. It also collapses source gaps and weakens source-coordinate +traceability. + +Keep fragments separate and add contextual links instead: + +```json +{ + "fragment_id": "f12", + "source_bbox": [0, 1200, 800, 300], + "context_fragments": ["f11", "f13"], + "link_reason": "adjacent_text_without_face" +} +``` + +The dialogue window can see linked fragments while retaining their boundaries and coordinates. + +Acceptance criteria: + +- Source pixels and coordinates remain reconstructable. +- A caption/monologue is not assigned merely because a face fragment is adjacent. +- Fragment linkage can be reviewed and changed without recropping the chapter. + +### 9. Reconcile overlapping dialogue windows + +Operational windows should overlap by at least one or two panels. Results in the overlap must be +reconciled using region IDs, transcription similarity, and attribution confidence. + +Acceptance criteria: + +- Changing the window size does not change the final transcript for stable fixtures. +- No duplicate lines appear from overlaps. +- Conflicting overlap predictions produce a flag and retain both candidates for review. +- Conversation history is scene-aware and resets only at a confirmed boundary. + +### 10. Prevent batch boundaries from becoming scene boundaries + +The first panel of a direction window currently becomes a new scene even when it continues the prior +window. Add `continues_previous` or an equivalent cross-window edge, then run a final boundary +reconciliation pass over compact ordered beat summaries. + +Validate grouping as an exact partition: + +- Every input panel appears exactly once. +- Groups are contiguous and ordered. +- There are no duplicate or unknown panel IDs. +- Missing/invalid grouping fails or falls back with a warning. + +Acceptance criteria: + +- The same chapter grouped with different operational window sizes produces equivalent scene + boundaries. +- A conversation crossing a window boundary remains one beat. +- Non-contiguous model groups are rejected rather than silently applied. + +## P2: make narration auditable + +### 11. Use an ordered, evidence-bearing beat schema + +A multi-panel script request should contain ordered member panels rather than a flattened scene graph +plus `panel_count`: + +```json +{ + "beat_id": "b07", + "panels": [ + {"panel_id": "p20", "dialogue_ids": ["p20_r1"], "actions": [...]}, + {"panel_id": "p21", "dialogue_ids": [], "actions": [...]} + ] +} +``` + +This preserves action and dialogue chronology and gives render/subtitle stages exact mappings. + +### 12. Add a deterministic script verifier before TTS + +Verify generated narration against the beat artifact: + +- Every used character name exists in the canonical cast map. +- Direct quotes fuzzy-match source dialogue and retain the correct speaker. +- No panel, dialogue line, or action is moved out of chronological order. +- Unsupported proper nouns and character names are rejected. +- Low-confidence facts are not asserted as certain. +- Each sentence/cue maps to one or more source panel/dialogue IDs. + +Failed verification should regenerate once with structured feedback, then enter script review. It +must not proceed automatically to TTS. + +### 13. Make review gates evidence-focused + +Prioritize review items by impact: + +1. Duplicate/conflicting canonical names. +2. Identity collisions or unresolved recurring tracklets. +3. Low-confidence or contradictory speaker assignments. +4. Missing/partial dialogue regions. +5. Cross-window scene-boundary conflicts. +6. Script verifier failures. + +The reviewer should see the source panel, marked text region, candidate speaker/identity evidence, +confidence, and downstream narration affected by the decision. + +## Test plan + +Create a small checked-in fixture corpus with expected structured artifacts. Avoid testing only prompt +strings; test complete contracts and invariants. + +Required fixture cases: + +- Two similar characters speaking back and forth. +- One visible listener with an off-panel speaker. +- Bubble-only panel continuing a prior speaker. +- Name used to address someone other than the speaker. +- Self-introduction and caption-based naming. +- Same character in different clothing and lighting. +- Two simultaneous similar-looking characters. +- Thought, shout, narration, sign, UI text, and SFX classification. +- Sentence split across bubbles without crossing panel ownership. +- Webtoon text fragment between two face fragments. +- Scene and conversation crossing an operational window boundary. +- Malformed/truncated JSON and a model response omitting one panel. +- Multi-panel beat with interleaved action and dialogue chronology. + +Contract/invariant tests should assert: + +- Stable panel and region IDs across resume. +- No raw local ID or unresolved free-form name reaches the script as a canonical identity. +- No successful empty transcript when text regions were detected but not accounted for. +- No permanent character mutation from a single unsupported name claim. +- No duplicated panel/dialogue membership across beats. +- No TTS request before all blocking correctness gates pass. + +Add end-to-end golden tests for: + +```text +panels -> detections -> tracklets -> transcript -> beats -> script claims +``` + +The golden artifact should compare structured facts, not exact prose wording. + +## Observability and evaluation + +Track correctness metrics per title and pipeline version: + +- Character-ID pairwise precision/recall over recurring characters. +- Canonical-name accuracy and unresolved-name rate. +- Dialogue region recall and hallucinated-line rate. +- Speaker-attribution accuracy, split by tail-grounded, solo prior, and off-panel inference. +- Scene-boundary precision/recall. +- Script unsupported-claim and wrong-name rates. +- Percentage of panels/beats sent to review. +- Result stability when operational batch/window sizes change. + +Log model/prompt version, detector version, thresholds, window membership, and evidence method with +each decision so regressions are reproducible. + +## Suggested implementation sequence + +1. Define versioned shared schemas for dialogue lines, speaker references, name claims, identities, + panels, and beats. +2. Update worker and orchestrator contracts to preserve confidence/provenance and explicit failure + states. +3. Fix speaker normalization and remove the unconditional solo assignment. +4. Add fail-loud dialogue completeness checks. +5. Enable/calibrate region-grounded transcription and create dialogue fixtures. +6. Separate name claims from persistence; add canonical-name conflict handling. +7. Implement stronger tracklet evidence and image-based Tier-2 identity comparison. +8. Replace destructive webtoon fragment merging with contextual links. +9. Add overlapping-window reconciliation and cross-window scene edges. +10. Introduce ordered beat artifacts and the script verifier. +11. Build evidence-focused review screens and quality dashboards. +12. Re-run the golden corpus after every prompt/model/threshold change. + +## Immediate repository hygiene + +- Fix the stale `worker_script.py` self-check: implementation expects a ~25-word solo budget while + the assertion still expects 35 words. +- Add tests proving dialogue parse failure is not treated as successful empty dialogue. +- Add tests for named off-panel speaker normalization and the visible-listener/off-panel-speaker case. +- Document the configured dialogue and direction window sizes in the orchestrator. +- Confirm that crop warnings, identity ambiguity, backfills, and reconciliation confidence are + persisted and honored by review gates rather than merely logged. + +## Definition of done + +This correctness pass is complete when: + +- Every narrated name, quote, speaker, action, and scene boundary is traceable to source evidence. +- Low-confidence and failed work cannot silently reach TTS/render. +- Identity and scene results are stable across resume and operational window-size changes. +- The fixture corpus passes end to end in both worker and orchestrator repositories. +- Remaining uncertainty is explicitly reviewable rather than hidden behind polished narration. diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..39b9212 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,15 @@ +fastapi +uvicorn[standard] +python-multipart +requests +minio +numpy +opencv-python +kumiko +onnxruntime # set-of-mark: comic-text-detector text regions + anime-face boxes (CPU EP); bubble_detect.py, face_detect.py +# GPU / in-process models (rocm torch build already on workpc): +torch +transformers +accelerate +soundfile +# system deps (not pip): ffmpeg, comfyui (external server) diff --git a/scripts/analyze_video_frames.sh b/scripts/analyze_video_frames.sh new file mode 100755 index 0000000..ad8e1bb --- /dev/null +++ b/scripts/analyze_video_frames.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +# Turn a video into timestamped contact sheets and scene-change frames for visual analysis. +# Optional --window START:DURATION arguments create dense 10-fps sheets around transitions. +set -euo pipefail + +usage() { + cat <<'EOF' +Usage: + scripts/analyze_video_frames.sh VIDEO [OUTPUT_DIR] [--window START:DURATION ...] + +Examples: + scripts/analyze_video_frames.sh chapter.mp4 + scripts/analyze_video_frames.sh chapter.mp4 /tmp/chapter-analysis \ + --window 5.8:2.2 --window 15.8:2.5 + +Outputs: + overview-001.jpg ... 2-fps timestamped contact sheets + scenes/scene-001.jpg ... frames selected by scene-change score + scene-timestamps.txt scene-change timestamps from FFmpeg showinfo + window-START.jpg 10-fps contact sheet for each requested window + +Environment overrides: + OVERVIEW_FPS=2 SCENE_THRESHOLD=0.18 WINDOW_FPS=10 +EOF +} + +if [ "$#" -lt 1 ] || [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then + usage + exit $([ "$#" -lt 1 ] && echo 1 || echo 0) +fi + +video=$1 +shift +if [ ! -f "$video" ]; then + echo "video not found: $video" >&2 + exit 1 +fi + +if [ "$#" -gt 0 ] && [[ $1 != --* ]]; then + output_dir=$1 + shift +else + base=$(basename "$video") + output_dir="/tmp/${base%.*}-frames" +fi + +overview_fps=${OVERVIEW_FPS:-2} +scene_threshold=${SCENE_THRESHOLD:-0.18} +window_fps=${WINDOW_FPS:-10} +windows=() +while [ "$#" -gt 0 ]; do + case "$1" in + --window) + [ "$#" -ge 2 ] || { echo "--window needs START:DURATION" >&2; exit 2; } + windows+=("$2") + shift 2 + ;; + *) + echo "unknown argument: $1" >&2 + usage >&2 + exit 2 + ;; + esac +done + +mkdir -p "$output_dir/scenes" + +# Six-by-six sheets keep individual frames readable while allowing videos of arbitrary length: +# tile emits additional overview-NNN images after every 36 sampled frames. +ffmpeg -hide_banner -loglevel error -y -i "$video" \ + -vf "fps=${overview_fps},scale=320:-1,drawtext=text='%{pts\\:hms}':x=8:y=8:fontsize=20:fontcolor=yellow:borderw=2,tile=6x6:padding=4:margin=4" \ + -fps_mode vfr "$output_dir/overview-%03d.jpg" + +# Keep stderr because showinfo reports the original timestamps there. +ffmpeg -hide_banner -y -i "$video" \ + -vf "select='gt(scene,${scene_threshold})',showinfo" -fps_mode vfr \ + "$output_dir/scenes/scene-%03d.jpg" 2>"$output_dir/scene-showinfo.log" || true +sed -n 's/.*pts_time:\([^ ]*\).*/\1/p' "$output_dir/scene-showinfo.log" \ + >"$output_dir/scene-timestamps.txt" + +for window in "${windows[@]}"; do + start=${window%%:*} + duration=${window#*:} + if [ "$start" = "$window" ] || [ -z "$start" ] || [ -z "$duration" ]; then + echo "invalid window '$window'; expected START:DURATION" >&2 + exit 2 + fi + safe_start=${start//./_} + # A 5x5 sheet covers 2.5 seconds at the default 10 fps. Longer windows naturally emit more sheets. + ffmpeg -hide_banner -loglevel error -y -ss "$start" -t "$duration" -i "$video" \ + -vf "fps=${window_fps},scale=320:-1,drawtext=text='%{pts\\:hms}':x=6:y=6:fontsize=18:fontcolor=yellow:borderw=2,tile=5x5:padding=3:margin=3" \ + -fps_mode vfr "$output_dir/window-${safe_start}-%03d.jpg" +done + +echo "video analysis frames: $output_dir" diff --git a/scripts/pick_tts_voice.py b/scripts/pick_tts_voice.py new file mode 100644 index 0000000..5027f94 --- /dev/null +++ b/scripts/pick_tts_voice.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +"""Interactively generate and choose a dots.tts narrator reference.""" + +import argparse +import os +from pathlib import Path +import shutil +import subprocess +import sys +import time + + +DEFAULT_TEXT = "The story continues as our hero steps forward into the unknown." +DEFAULT_MODEL = "rednote-hilab/dots.tts-base" +DEFAULT_VOICE_DIR = Path("~/.cache/manga-tts").expanduser() + + +def load_model(model_name: str): + # Match worker_tts.py's workaround for the tokenizer bundled with dots.tts. + import transformers + + original = transformers.AutoTokenizer.from_pretrained.__func__ + transformers.AutoTokenizer.from_pretrained = classmethod( + lambda cls, *args, **kwargs: original( + cls, *args, **{"fix_mistral_regex": True, **kwargs} + ) + ) + from dots_tts.runtime import DotsTtsRuntime + + return DotsTtsRuntime.from_pretrained(model_name, precision="bfloat16") + + +def write_wav(result, path: Path) -> None: + import numpy as np + import soundfile as sf + + audio = result["audio"] + if hasattr(audio, "detach"): + audio = audio.detach().cpu().numpy() + sf.write(path, np.asarray(audio, dtype="float32").squeeze(), + result["sample_rate"], subtype="PCM_16") + + +def play(path: Path) -> None: + players = ( + ("ffplay", "-nodisp", "-autoexit", "-loglevel", "error"), + ("aplay", "-q"), + ("paplay",), + ) + for command in players: + if shutil.which(command[0]): + subprocess.run([*command, str(path)], check=False) + return + print(f"No ffplay, aplay, or paplay found; listen manually: {path}") + + +def choice() -> str: + while True: + answer = input("[p]ick / p[a]rk / [s]kip: ").strip().lower() + aliases = {"p": "pick", "pick": "pick", "a": "park", "park": "park", + "s": "skip", "skip": "skip"} + if answer in aliases: + return aliases[answer] + print("Enter p, a, or s.") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--text", default=os.environ.get("VOICE_REF_TEXT", DEFAULT_TEXT), + help="sample text to speak") + parser.add_argument("--model", default=DEFAULT_MODEL) + parser.add_argument("--voice-dir", type=Path, default=DEFAULT_VOICE_DIR) + args = parser.parse_args() + + voice_dir = args.voice_dir.expanduser() + parked_dir = voice_dir / "parked" + voice_dir.mkdir(parents=True, exist_ok=True) + parked_dir.mkdir(parents=True, exist_ok=True) + candidate = voice_dir / "voice_candidate.wav" + selected = voice_dir / "narrator_ref.wav" + + print(f"Loading {args.model} ...") + model = load_model(args.model) + print(f"Reference text: {args.text!r}") + + try: + while True: + print("\nGenerating a new voice ...") + write_wav(model.generate(text=args.text), candidate) + play(candidate) + + action = choice() + if action == "pick": + os.replace(candidate, selected) + (voice_dir / "narrator_ref.txt").write_text(args.text + "\n") + print(f"Selected: {selected}") + return 0 + if action == "park": + stamp = time.strftime("%Y%m%d-%H%M%S") + parked = parked_dir / f"voice-{stamp}.wav" + suffix = 2 + while parked.exists(): + parked = parked_dir / f"voice-{stamp}-{suffix}.wav" + suffix += 1 + os.replace(candidate, parked) + parked.with_suffix(".txt").write_text(args.text + "\n") + print(f"Parked: {parked}") + else: + candidate.unlink(missing_ok=True) + print("Skipped.") + except (KeyboardInterrupt, EOFError): + candidate.unlink(missing_ok=True) + print("\nStopped without changing the selected voice.") + return 130 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/session_manager.py b/session_manager.py new file mode 100644 index 0000000..6d20ca8 --- /dev/null +++ b/session_manager.py @@ -0,0 +1,238 @@ +# session_manager.py — the GPU mutex. one warm model at a time on workpc. +# owns: model registry, llama-server subprocess lifecycle, session leases + reaper. +# workpc holds no durable state; this only guards the single local GPU. +# +# subprocess models (gemma4): this process starts/stops llama-server and health-waits. +# in-process models (siglip2, dots): no server — the lease just reserves the GPU; the +# worker loads the transformers model itself after /session/open returns (port=None). +# the guarantee that matters is the mutex: a second open() gets 409 until the first closes. +import os, time, uuid, threading, subprocess +import requests +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel + +app = FastAPI() + +MODELS = { + "gemma4": { + "binary": "/mnt/D/AI/llama-cpp/llama.cpp/build/bin/llama-server", + "model": "/mnt/D/AI/gemma4/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf", + "mmproj": "/mnt/D/AI/gemma4/mmproj-F16.gguf", + "port": 8090, "ctx": 32768, # 12B fits 32k KV unquantized on 16GB ROCm; do NOT -ctk/-ctv quant (leaks on ROCm) + # MTP speculative decoding: the ~430M gemma-4 draft head drafts up to 4 tokens the 12B verifies. + # needs `--spec-type draft-mtp` to actually engage (without it llama-server loads the draft but + # silently runs no speculation). ~42% accept on our JSON stages -> faster decode; ~0.9GB VRAM. + "args_extra": ["--jinja", "--reasoning-budget", "0", + "-md", "/mnt/D/AI/gemma4/mtp-gemma-4-12b-it-BF16.gguf", "-ngld", "99", + "--spec-type", "draft-mtp", "--spec-draft-n-max", "4"], + }, + # in-process models: the worker loads them and holds ~5GB resident. `worker` is the port whose + # /unload frees that GPU memory on close -- the mutex alone can't reclaim it (leaked across runs, + # starving the next model on open). subprocess models (gemma4) free via terminate instead. + "siglip2": {"binary": "transformers", "model": "google/siglip2-so400m-patch16-384", "port": None, "worker": 8003}, + "dots": {"binary": "transformers", "model": "rednote-hilab/dots.tts-base", "port": None, "worker": 8006}, +} + +SKIP_HEALTH = False # self-check flips this; real runs health-wait the llama-server + +_lock = threading.Lock() +# the single resident session, or None. {model, session_id, opened_at, port, ttl, last_beat, proc} +_active = None + + +class OpenReq(BaseModel): + model: str + ttl: int = 3600 + +class SessionReq(BaseModel): + session_id: str + + +def _health_wait(port: int, timeout: int = 300): + deadline = time.time() + timeout + while time.time() < deadline: + try: + if requests.get(f"http://127.0.0.1:{port}/health", timeout=2).status_code == 200: + return + except requests.RequestException: + pass + time.sleep(1) + raise TimeoutError(f"llama-server on :{port} never became healthy") + + +def _start_subprocess(cfg): + """spawn llama-server for a subprocess-type model, return the Popen once healthy.""" + cmd = [cfg["binary"], "-m", cfg["model"], "--port", str(cfg["port"]), + "-c", str(cfg.get("ctx", 4096)), "-ngl", "99"] + if cfg.get("mmproj"): + cmd += ["--mmproj", cfg["mmproj"]] + cmd += cfg.get("args_extra", []) + # keep llama-server's stderr — it prints the assert / CUDA error / context-overflow that kills it. + # DEVNULL here is why crashes showed up as "no error in the logs". append so a respawn keeps history. + log = open(f"/tmp/llama-server-{cfg['port']}.log", "a") + proc = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT) + if not SKIP_HEALTH and cfg["port"]: + _health_wait(cfg["port"]) + return proc + + +def _teardown(sess): + proc = sess.get("proc") + if proc: + if proc.poll() is None: + proc.terminate() + try: + proc.wait(timeout=15) # wait() even on an already-dead proc, else it stays a zombie + except subprocess.TimeoutExpired: + proc.kill() + # in-process models live in the worker, not here -- ask it to free the GPU (best-effort; + # if the worker is down the VRAM is already gone with it). + worker = sess.get("worker") + if worker: + try: + requests.post(f"http://127.0.0.1:{worker}/unload", timeout=30) + except requests.RequestException: + pass + + +@app.post("/session/open") +def open_session(req: OpenReq): + if req.model not in MODELS: + raise HTTPException(400, f"unknown model {req.model}") + global _active + cfg = MODELS[req.model] + session_id = uuid.uuid4().hex + with _lock: + if _active is not None: + raise HTTPException(409, f"{_active['model']} resident (session {_active['session_id']})") + # ponytail: claim the slot before the slow spawn/health-wait so /active and /close + # for other sessions aren't blocked behind it; only the actual subprocess start runs unlocked. + _active = { + "model": req.model, "session_id": session_id, "opened_at": time.time(), + "port": cfg.get("port"), "ttl": req.ttl, "last_beat": time.time(), "proc": None, + "worker": cfg.get("worker"), + } + proc = _start_subprocess(cfg) if cfg["binary"] != "transformers" else None + with _lock: + if _active is not None and _active["session_id"] == session_id: + _active["proc"] = proc + return {"session_id": session_id, "port": cfg.get("port")} + + +@app.post("/session/close") +def close_session(req: SessionReq): + with _lock: + global _active + if _active is None or _active["session_id"] != req.session_id: + raise HTTPException(404, "no such active session") + _teardown(_active) + _active = None + return {"ok": True} + + +@app.get("/session/active") +def active(): + with _lock: + if _active is None: + return None + return {k: _active[k] for k in ("model", "session_id", "opened_at", "port")} + + +@app.post("/session/heartbeat") +def heartbeat(req: SessionReq): + with _lock: + if _active is None or _active["session_id"] != req.session_id: + raise HTTPException(404, "no such active session") + _active["last_beat"] = time.time() + return {"ok": True} + + +def _reap_once(): + with _lock: + global _active + if _active and time.time() - _active["last_beat"] > _active["ttl"]: + _teardown(_active) + _active = None + return True + return False + + +def _supervise_once(): + """Respawn a crashed subprocess model in place. llama-server can OOM/die mid-session; the worker + talks to the port directly, so if we don't bring it back every remaining panel of the stage fails + with 'connection refused'. Restarting under the SAME session keeps the lease + port valid so the + orchestrator never has to re-open. Returns True if a respawn happened.""" + with _lock: + global _active + sess = _active + if not sess: + return False + proc = sess.get("proc") + if proc is None or proc.poll() is None: + return False # in-process model, or subprocess still alive + try: + proc.wait(timeout=1) # reap the zombie + except subprocess.TimeoutExpired: + pass + print(f"[supervise] gemma subprocess died (rc={proc.returncode}) session {sess['session_id']}; " + f"respawning", flush=True) + try: + sess["proc"] = _start_subprocess(MODELS[sess["model"]]) + sess["last_beat"] = time.time() # don't count the downtime against the TTL reaper + return True + except Exception as e: + print(f"[supervise] respawn failed: {e}; clearing session so the mutex frees", flush=True) + _active = None + return False + + +def _reaper(): + while True: + time.sleep(1) + if _reap_once(): + continue + _supervise_once() + + +@app.on_event("startup") +def _startup(): + threading.Thread(target=_reaper, daemon=True).start() + + +if __name__ == "__main__": + # self-check: mutex (409 on second open), close releases, reaper force-closes a stale lease. + SKIP_HEALTH = True + MODELS["gemma4"] = {"binary": "/bin/sleep", "model": "5", "port": None, "args_extra": []} + + s1 = open_session(OpenReq(model="gemma4", ttl=3600)) + assert active()["model"] == "gemma4" + try: + open_session(OpenReq(model="siglip2")) + assert False, "second open must 409" + except HTTPException as e: + assert e.status_code == 409 + close_session(SessionReq(session_id=s1["session_id"])) + assert active() is None + + s2 = open_session(OpenReq(model="gemma4", ttl=2)) + _active["last_beat"] -= 5 # simulate a lease that missed its heartbeat window + assert _reap_once() is True + assert active() is None + + # supervisor: a crashed subprocess is respawned in place, keeping the same session_id + port. + s4 = open_session(OpenReq(model="gemma4", ttl=3600)) + old_proc = _active["proc"] + old_proc.terminate(); old_proc.wait() # simulate the llama-server crashing mid-session + assert _supervise_once() is True + assert active()["session_id"] == s4["session_id"] # lease survived + assert _active["proc"] is not old_proc and _active["proc"].poll() is None # fresh live proc + close_session(SessionReq(session_id=s4["session_id"])) + assert active() is None + + # in-process model: close must POST /unload to the worker so its resident VRAM is freed. + calls = [] + requests.post = lambda url, **kw: calls.append(url) or type("R", (), {"status_code": 200})() + s3 = open_session(OpenReq(model="siglip2", ttl=3600)) + close_session(SessionReq(session_id=s3["session_id"])) + assert any("/unload" in u and ":8003" in u for u in calls), calls + print("session_manager self-check ok") diff --git a/spec-correctness.md b/spec-correctness.md new file mode 100644 index 0000000..a8ee25c --- /dev/null +++ b/spec-correctness.md @@ -0,0 +1,25 @@ +# Correctness status — worker side + +Audit of `plan.md` against the actual code (2026-07-18). Most of the plan is **implemented**; this file +records only what's verified done and the remaining calibration work, so nobody re-does finished work. Orchestrator +counterpart: `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/spec-correctness.md`. + +### #5 region-grounded dialogue — code done, calibration ongoing +Set-of-mark is now the **default path** (`SOM_ATTRIBUTION="1"`, `worker_vision.py:31`; the "off by +default" comment at `:22` is stale — fix it). Residual is per-title threshold calibration on real +chapters, not a code gap. Detector-failure vs zero-regions are already distinct (`_set_of_mark`). + +## Verified done (was in plan/earlier drafts as TODO — do not redo) +- #10 cross-window direction continuity: the grouping pass emits `continues_previous`; the first row + carries it through `/direct/window`, and a true edge forces `new_scene=false`. The self-check covers + both a continuing window and the chapter-start guard. +- Script self-check (`python worker_script.py` passes). #2 confidence + `speaker_method` reach the scene + graph (`worker_scene.py:80-81`). H3/#3 fail-loud: `/dialogue` and `/dialogue/window` return a status + envelope (`_dialogue_envelope`) and skip dropped panels instead of manufacturing silent-empty. +- #4 solo assignment is a 0.7-confidence prior with `speaker_method=solo_prior`, not unconditional. +- #6 typed name claims emitted (`_normalize_claims`), roster is hints-only. +- #7 Tier-2 `/vision/resolve` sends candidate **reference images** (≤3 each), judges face-first, returns + open-set `state: known|new|unresolved`. + +- #8 webtoon fragments remain independent and retain source bboxes; `context_fragment_links` emits + contextual links instead of the crop endpoints destructively stacking them. diff --git a/spec-v3.md b/spec-v3.md new file mode 100644 index 0000000..e0dbe3d --- /dev/null +++ b/spec-v3.md @@ -0,0 +1,348 @@ +this is a v3 delta spec: it lists what changes on top of the working v2 pipeline (see spec-v2.md). +v2 is validated end-to-end (a full chapter renders to one narrated mp4). v3 is about quality of the +narration and the look of the video, plus panel curation. structured for hand-off to an agent. + +# manga recap v3 — quality + look + +## 0. what v2 does today (baseline) +- one narrated `chapter.mp4` per chapter: fit-on-black panels, uniform ken-burns zoom, burned + subtitles, hard cuts, single default TTS voice. +- identity assigns a stable `character_id` per body (siglip2), but characters are unnamed, so the + script prints "Person A/B". the script worker sees one panel at a time (no memory). +- every panel is narrated (no curation). + +--- + +## A. cross-panel identity & continuity [biggest quality lever] [DONE] +SHIPPED: +- descriptive frozen labels: worker_scene emits `label` per character (name -> frozen registry + appearance via `_describe`, "the one with black hair and glasses" -> Person X fallback); + worker_script `_name_map` consumes it. no "Person A" when appearance is known, char_id preserved. +- chapter_context populated: service.py `_chapter_context` feeds a cast+entities roster to every + script call so the narrator has memory (kills "Unknown voice", item 11). +- name acquisition + backfill (item 12): vision emits `name` only when a char is explicitly named; + service.py scene stage binds it via `set_character_name` (first name wins, only if unnamed); + script stage resolves display names from the FINAL registry (`names_by_id`) so a name learned in + ANY panel backfills to every panel's narration. ceiling: vision name-inference can misattribute; + first confident name sticks. upgrade path = roster seed for higher precision. +STILL TODO (small, deferred): +- true per-panel story-state (only what's been seen so far) for "establishing shot once / don't + re-narrate repeats" (the C-todo) — currently whole-manga roster. + +--- original plan --- +one feature: carry a running **story-state** through a chapter so narration has memory. covers +former items 4, 11, 12. + +- **stable anchor, mutable label.** the `character_id` is never lost (already stable via siglip2). + only its *display label* changes: + - name known -> the name. + - name unknown -> a **descriptive phrase** ("guy with black hair and glasses", "girl with short + hair holding a phone") derived ONCE from vision's appearance data and frozen per character_id, + reused across all panels. never regenerate per-panel (labels would drift). +- **name acquisition + backfill.** when dialogue addresses a character by name and speaker + attribution is confident, bind name -> character_id and switch the label to the name from then on + (optionally backfill earlier references). +- **story-state threaded per chapter.** accumulate {cast seen (id,label,name?), entities/props seen + (the cat)} and feed it into scene+script (populate the currently-empty `chapter_context`). this is + what lets the narrator say "he calls out to his cat" instead of leaking a bare name (item 12), and + removes "Unknown voice" (item 11). +- **optional roster seed (not required).** a per-manga character list (names, maybe ref art) makes + naming + cross-panel matching more reliable than dialogue inference. graceful-degrade without it; + add when available. + +touches: worker_vision (appearance -> label), worker_scene / orchestrator (story-state store + +speaker continuity), worker_script (consume context), orchestrator characters table (label, name). + +## B. narration voice & style (items 2, 3) [DONE] +- **less descriptive gemma** (item 2): DONE. prompt now demands TIGHT recap (1-2 sentences, ~35 + words, "no purple prose"); max_tokens 512->160, temp 0.7->0.6 (worker_script). +- **one fixed voice** (item 3): DONE. dots.tts samples a RANDOM speaker per call with no reference, + so the voice drifted. worker_tts now bootstraps one seeded reference clip (`~/.cache/manga-tts/ + narrator_ref.wav`) and clones from it every synth. delete the file to reroll; swap REF_WAV/REF_TEXT + for a hand-picked voice. +- **multi-voice per character: future**, not v3 (user: "later") — same mechanism, per-character ref clips. + +## C. panel curation (item 10) [DONE — basic] +- skip panels with no narrative content (pure scenery / transition art); show an establishing shot + once, don't re-narrate repeats. vision flags "scenery-only / skip". big pacing + cost win; do early + because it shrinks every downstream stage. +- SHIPPED: vision emits `skip:true` for no-narrative panels (worker_vision prompt+schema); the + script stage honors it (service.py run_stage_script) -> no narration -> cascades to no audio, no + clip. Single chokepoint, no new db columns (skip rides in the vision result_json blob). +- STILL TODO (needs cross-panel memory, folds into A): "establishing shot once, don't re-narrate + repeats" — requires dedup of repeated scenery across panels. deferred to story-state. + +## D. render / look overhaul (items 1, 5, 6, 7, 8) +all in worker_render.py (self-contained ffmpeg builder), except #6. +- **#1 dynamic clip duration** — DONE. `_audio_dur` probes narration length via ffprobe; scene_timing + kept only as fallback. no more 4.0s sawtooth / vanishing subs. +- **#5 blurred-panel background fill** — DONE. filter_complex: cover-scaled + boxblur copy behind the + fitted panel (replaces black bars). +- **#7 subtitles** — DONE. boxed caption: bold white on translucent black box (BorderStyle 3), + lower-center, DejaVu Sans 52. (user chose boxed over outline/minimal.) +- **#8 content-aware motion** — DONE. vision picks a camera.effect per panel from the fixed vocab + (+ to=[x,y] for dolly_to_subject); it rides the vision result_json -> scene_graph.camera -> + render. worker_render `_motion(camera, frames)` maps the vocab to a zoompan z/x/y expression + (static/zoom/pan/dolly/shake/orbit); default zoom_in = old ken burns. orchestrator render stage + reads scene_graph.camera. self-checked (ffmpeg renders pan + dolly clips). +- **#6 multi-panel composites + non-generic transitions** [the ambitious one, sequenced last] + - **non-generic transitions — DONE.** vision emits `transition` (cut|crossfade|dissolve| + fade_black|fade_white|wipe_left|wipe_right|push) per panel -> scene_graph.transition -> + orchestrator assemble collects a per-clip list -> worker_render `_xfade_chain` builds an + xfade+acrossfade graph (audio kept in sync, overlap clamped to clip length). all-cut/empty + keeps the fast stream-copy concat (no re-encode); any real transition re-encodes the chapter. + self-checked (two clips fade_white into one mp4). transition = OUT of that panel. + - **multi-panel composites — DONE (basic).** grouping source = new pass (`grouping.py`, + deterministic): consecutive SMALL panels (area < 0.7*page-max, same source page, capped at + MAX_GROUP=3) share ONE shot. render + assemble both call plan_groups so their grouping matches. + a composite = vertical stack of the members on a W×H still; each panel keeps its own script+tts + (per-panel stages unchanged); the audios play in sequence while a moving highlight marks the + active row ("swap the front panel while audio plays"). clip keyed by the group LEADER; assemble + takes one clip per group and the group's exit transition = last member's. solo groups render + exactly as before, so blast radius is only genuinely-small adjacent panels. + knobs: MAX_GROUP, SMALL_FRAC. ceilings: vertical-stack layout only (no grid/overlay/front-swap + animation beyond the highlight), size+adjacency grouping (no narrative-beat grouping), black + gutters (no blurred fill in composites). upgrade paths noted in code. + +## D2. beat pacing, timed cues, subtitle redesign, collage (vikunja 185/187/188/183) +scene-level narration made the beat (not the panel) the render unit; these refine how a beat plays. +all in worker_render.py + collage.py, orchestrator run_stage_render. +- **185 content-aware panel timing — DONE.** a beat's one narration used to split EQUALLY across its + member panels (`D/n`). now the orchestrator scores each member from its scene graph (dialogue amount, + action density, camera emphasis: `_beat_weights`) and passes bounded weights to render; worker + `_beat_slices(D, n, weights)` distributes screen-time by weight with a per-panel floor (no flashing) + and a deterministic equal-split fallback (degenerate/short weights). self-checked. +- **187 timed subtitle cues — DONE.** the whole narration paragraph used to show from frame 1, leaking + later-panel lines early. `cue_plan` splits the flowing narration into sentence cues, times them on the + SAME [0,D] timeline as the panel slices (length-weighted + readable floor, reused `_beat_slices`), + maps each cue to the panel on screen at its start, and clamps a cue never to precede its panel. per-cue + ASS events (not one paragraph). cue plan returned + persisted to `clips.cues_json` for the review UI / + collage. no-TTS-alignment fallback = punctuation/length estimate. self-checked ("Hold up." can't + appear before its panel). +- **188 subtitle redesign — DONE.** `_ass_multi` now takes mode (off|minimal|boxed, env `SUB_MODE`, + default minimal) + orientation presets (portrait/landscape safe-areas). smaller type (40/32 vs 52), + minimal = thin outline + soft shadow (no box), boxed = restrained ~50% box. `_wrap2` caps every cue at + ≤2 short lines (ellipsis truncation) so it never regrows into a paragraph. `_sub_align` dodges the + subject vertically using the director focus point (camera.to.y low -> caption to top). ceiling: no + real face/bubble CV yet — camera focus is the only subject signal wired. +- **183 animated manga collage — DONE (core + wired behind `COLLAGE` flag).** `collage.py plan_layout` + is a pure deterministic planner: ordered panel aspects + RTL + dominant + frame -> template + (centered_wide / vertical_pair / stacked_wides / strip_over_dominant / supporting_left_dominant_right + / quad), aspect-correct resting rects (no stretch), entrance vectors (RTL slide / dominant scale), + z-order, hold/transition timing. `worker_render collage_cmd` renders it: blurred plate of the dominant + panel + sharp aspect-fit panels with restrained drop shadows, non-dominant panels sliding into place + over ~0.45s, crisp holds; reuses 185 slices + 187 cues + 188 subtitles. orchestrator run_stage_render + calls it when `COLLAGE=1` (else the ken-burns montage). ceilings (QA-tuned against the reference video, + no still to eyeball here): transition-only directional/radial MOTION BLUR not applied (clean + slide/scale); per-panel hold DRIFT omitted (static hold); beat-replace "streak" rides the assemble + xfade. planner + renderer both self-checked; visual acceptance is the QA run. + +## D3. identity tracklets + review gates (vikunja 160/136) +- **160 tracklet spine — DONE (code, QA-pending).** the Tier-2 gemma decider (identity phase-2) used to + resolve identity once PER CROP, so one person got adjudicated repeatedly and could get two answers on + two panels. `tracklets.py link_tracklets` (pure, union-find, self-checked) first LINKS the per-panel + siglip shortlists into per-person tracklets over a locality window — hard gender gate + appearance-token + overlap + shared-candidate — then the resolver runs gemma ONCE per tracklet against the UNION of the + members' shortlists (the person's gallery, consensus-ordered) and applies that one character_id to every + crop in the tracklet. fewer gemma calls, one answer per person. ceiling: gemma NONE (cosine over-merged + onto an existing char) still isn't split to a fresh id — needs a 3rd siglip re-embed pass; add if the + NONE-rate is high. +- **136 review gates — DONE (code, QA-pending), off by default (`GATES=1`).** two human sign-off gates in + the pipeline loop: gate `script` pauses BEFORE tts/render, gate `audiovisual` pauses BEFORE assemble. + When enabled and unapproved the job goes `awaiting_review` and the loop returns; `/review/approve` + records the gate (`gates` table) and relaunches `_run_pipeline` (skips completed stages → resumes at the + gated stage). Gate-1: `/review/scripts` shows narration per panel grouped by beat + cheap textual flags + (`review_flags.py`: generic-handle / repeated / empty, self-checked; semantic flags — spoilers, altered + meaning, attribution — left to the human eye, ceiling noted). Gate-2: `/review/audio` serves the per-panel + wav so the reviewer actually listens; `/review/retts` re-synthesizes ONE panel's TTS + re-renders just its + beat clip (delete_clip forces it) and clears the AV gate. web UI (index.html review tab): gate-approve + buttons + status, per-panel audio player, flags badge, re-synthesize button. reuses existing + /review/panels + /review/script + /review/preview. ceiling: no panel-to-cue drag-edit UI (cues persisted + in clips.cues_json, edit endpoint not built); flags are textual only. + +## E. reference +generic manga-recap youtube style (single narrator, static panels w/ slow pan/zoom, minimal text, +dramatic pacing) — but with **less generic transitions** (feeds #6). #6 goes beyond the reference. + +--- + +## sequencing (proposed) +1. **C panel curation** — shrinks everything downstream first. +2. **A identity/continuity** — most changes how it *feels* (names + memory). +3. **B voice+style** — small, high impact. +4. **D basics (#1,#5,#7,#8)** — most changes how it *looks*, low risk. +5. **D#6 compositing** — last; highest risk/effort, isolated to render. + +## direction schema (decided — enables #8 and #6) +narration (text->TTS) and direction (camera/transition/layers->render) are separate concerns from +the SAME scene graph. VISION picks the effect (it sees the panel + has char bboxes for focus points) +and emits a direction block into the scene_graph JSON; RENDER just executes it. fixed vocab: +- camera.effect: static | zoom_in | zoom_out | pan_left | pan_right | pan_up | pan_down | + dolly_to_subject | orbit | shake | hold (+ from/to [x,y] normalized, easing) +- transition.type: cut | crossfade | fade_black | fade_white | wipe_left | wipe_right | push | dissolve + (+ duration) +- layers: {artifact, motion} (parallax, when the depth renderer is wired) +not built yet; lands before #8 (needs camera) and #6 (needs transition). + +## open design points +- roster seed format if/when available. +- B#3: which fixed dots.tts reference voice to pin for the whole video (needs a pick). + +--- + +# v3.1 delta — identity is broken; add human-in-loop review + +## evidence (last job, 116-panel Teto X Egen, from manga.db) +- **53 character rows for ~8 actual people.** "Seonho" stored 5× (ids 85ad36b0/811bf0d7/0033ff7f/ + bbddf20c/90269899), Egen Girl 2×. ~40 rows have a single identity_assignment (one-time NPCs). +- **root cause: siglip2 over-splits.** same person across panels (pose/expression/B&W line art) + doesn't clear cosine 0.85, so nearly every appearance spawns a new character_id. +- **cascade:** names never propagate (panel 10 names id A, panel 51's Seonho is id C -> `names_by_id` + can't map it -> narration falls back to "the woman with straight dark hair", flips gender to "her"). + re-introductions + DB/S3 bloat all follow from the split. +- **concrete bug:** `worker_identity.py` create call hardcodes `"name": None` — vision reads the name + but identity discards it at registration. `db.create_character` is a blind INSERT, zero dedup. + +## E1. pipeline decomposition [DECIDED — the vision mega-call splits] +Reason: one gemma4 call doing 4 jobs (detect + dialogue + direction + scene) does each worse, and +there was nowhere to run a real cross-panel merge. Split the jobs; insert a merge stage where the +model actually compares two crops. + +**Final stage order** (OCR removed — gemma4 transcribes bubble text image-natively; a `roster` +prepass now runs after crop, see below): +``` +fetch → crop → roster → vision → identity → reconcile → dialogue → direct → scene → script → tts → layers → render → assemble +``` +- **`vision`** (per panel, image call) — character DETECT only: `local_id`, `bbox`, `appearance` + (hair/clothing/features), `gender` (m|f|unknown), `emotion`, `action`. plus `skip` + `scene` + (location/time). the only stage that must precede identity (identity crops its bboxes). +- **`identity`** (per panel) — siglip assigns `local_id → character_id`. on a collision (siglip + ambiguous OR hair+clothing appearance overlap OR name collision) it records a **merge-candidate + pair** instead of silently minting a twin. still creates provisional ids; reconcile prunes them. +- **`reconcile`** (NEW, once per chapter, after all panels) — the second vision pass. for each + flagged candidate pair, `POST worker_vision /vision/same` with the two `ref_image_uris` crops -> + gemma4 "same character? {yes|no, confidence}". confirmed -> `db.merge_characters(loser->keeper)`: + repoint `identity_assignments`, union `name`/`aliases`/`ref_image_uris`, delete loser row. loser + ref crops are PRESERVED on the keeper as gallery views (not deleted from S3 — deleting them left the + keeper's own uris dangling). cast is CLEAN + named from here on. only flagged pairs get a call. + lives as `run_stage_reconcile` in service.py + `/vision/same` on worker_vision (reuses warm gemma4). +- **`dialogue`** (per panel, image call) — gemma reads bubble text straight from the panel image (no + external OCR): bubble classify (speech/thought/shout/narration/sfx), merge split bubbles, text + cleanup (sentence case, fix names/proper nouns, sfx literal), speaker attribution + (`speaker=local_id` + `confidence`), entities. prompt gets the CLEAN + named cast present in the panel as context -> better attribution ("Seonho (m) vs Haeseon (f)"). +- **`direct`** (per panel, off scene understanding) — `camera.effect` (+`to`) and `transition` from + the fixed direction vocab (unchanged schema, section D/direction-schema). runs after identity so it + can weight who's present. may glance at the image or work from the scene graph — cheap. + +new workers/endpoints: `worker_vision` becomes detect + `/vision/same`; new `worker_dialogue.py` and +`worker_direct.py` (or `/dialogue` + `/direct` endpoints — same warm gemma4, decide at build like +reconcile). db: `STAGES` gains `reconcile`,`dialogue`,`direct`; `run_stage_*` for each; session_proxy +call_* wrappers. the old omni-`vision` prompt/schema is partitioned across the four prompts. + +## F. identity overhaul [rides on E1] +Goal: 53 rows -> ~12; named recurring cast stays one id; NPCs never persist. + +- **F1 merge = the `reconcile` stage** (see E1). the fix for the hardcoded `"name": None` bug: vision/ + identity now carry `name`+`gender`+`appearance` into `create_character` so provisional rows are + named/typed; reconcile does the actual dedup with the model comparing crops (not a blind string + match). `db.merge_characters` + candidate-flagging in identity. +- **F2 gender field.** add `gender TEXT` to characters schema. `vision` emits `gender`. worker_script + `_name_map` feeds it to the model ("Seonho (he)") for correct pronouns — kills "her bold + personality" flips even when the name resolves. +- **F3 confirm-before-persist.** hold unnamed provisional embeddings in a per-session cache; only + write a DB row + S3 crop/npy when an embedding is seen >=2x OR is named. the ~40 one-timers never + touch the bucket. session_id already flows into identity. (complements reconcile: reconcile merges + the twins that WERE created; F3 stops most from being created at all.) +- **F4 centroid + re-rank matching.** store multiple refs per char, match against the accumulated + centroid not one arbitrary embedding; break siglip ties by name/gender agreement. ceiling: perfect + auto-ID on B&W is not achievable — reconcile+F4 get ~80%, the UI (G) closes the rest. don't + over-invest. +- **DECIDED: start clean.** no migration script. wipe manga.db + the minio `manga` bucket before the + first v3.1 run (destructive ops step, run at build time). fixed logic applies to fresh data. + +## G. review / edit UI (replaces the auto CoT-strip idea, item C-bleed) +Rationale: the human reviews the final video and knows exactly what's wrong and where. Give them a +synced editor instead of chasing every failure through the pipeline. ~80% of infra already exists in +the orchestrator: `static/index.html` dashboard, `/job/video` (streams mp4), `/db/tables` + `/db/table`, +`/stage/run` + `/stage/clear` (re-render), `set_character_name`. + +- **timestamp -> panel is trivial:** clips concat in `panel_order`, so panel start offset = cumsum of + prior panels' `audio.duration`. store the offset at assemble time (or compute on the fly) as an + offset table the page binary-searches on `