kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Reconstruct repo from Claude Code + codex transcripts

Browse Source 
Working tree (including .git) was lost to an rm. Rebuilt by replaying Write/Edit/
Read/attachment events from 25 Claude sessions and 22 successful codex apply_patch
blocks into one timestamp-ordered timeline.

Verified against ground truth recorded in the transcripts: wc -l on 10 files and
ls -l on 5 files at 2026-07-18T13:13:44Z both match exactly; 18 files are
byte-identical to their newest ~/.claude/file-history blob.

See HANDOFF.md for sources, gaps, and how to rebuild .venv.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kami
2026-08-11 02:42:41 +04:00
commit ff6a512630
32 changed files with 6759 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+6
View File
@@ -0,0 +1,6 @@
.venv/
__pycache__/
*.pyc
dots.tts/
/dev/shm/
*.gguf
AGENTS.md
+23
View File
@@ -0,0 +1,23 @@
# Repository guidance
## Video analysis
When diagnosing render motion, transitions, timing, or visual artifacts, use
`scripts/analyze_video_frames.sh` instead of manually seeking through the video.
```bash
scripts/analyze_video_frames.sh VIDEO [OUTPUT_DIR] \
--window START:DURATION [--window START:DURATION ...]
```
The script creates:
- timestamped overview contact sheets sampled at 2 fps;
- scene-change frames and a `scene-timestamps.txt` index;
- optional dense 10-fps contact sheets for specified transition windows.
Inspect the overview first, then request dense windows around transitions or artifacts. Sampling and
scene sensitivity can be adjusted with `OVERVIEW_FPS`, `WINDOW_FPS`, and `SCENE_THRESHOLD`.
The generated files are analysis artifacts and belong in `/tmp` or another requested output directory;
do not commit them to the repository.
CLAUDE.md
+67
View File
@@ -0,0 +1,67 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
The **workpc compute half** of a manga→narrated-video pipeline. This repo holds stateless
GPU/CPU workers only. State, job scheduling, and stage orchestration live in a **separate
homesrv orchestrator repo** (`/mnt/server/home/kami/apps/Maven/` region) — not here. The two
communicate over a fixed HTTP contract; never add sqlite or durable state to a worker.
Machine split (memorize): workers run on **workpc** (RX 7900 GRE, ROCm). MinIO + orchestrator
run on **homesrv** (`192.168.1.104`, CPU-only). Data dir `/mnt/server/home/kami/` is an SSHFS
mount of homesrv.
## Run / test
```bash
./start_workers.sh # dev: session_manager + 9 workers, each a uvicorn in a tmux window
tmux attach -t manga-workers # watch logs; per-worker window
sudo systemd/install.sh # production: one systemd unit per process (User=kami)
python session_manager.py # each module has a __main__ self-check (assert-based, no framework)
python transport.py # run these to verify a file after editing it
python test_vision_parse.py # the one standalone pytest-free test
```
There is no lint/build step. `.venv` is the ROCm torch env; workers import `transport` by module name.
## Architecture
**Workers are stateless HTTP stages.** Each `worker_*.py` is a FastAPI app on a fixed port. It
pulls inputs from MinIO by URI to local disk (`/dev/shm`), does one stage, pushes outputs back,
returns URIs. No cross-request memory. Ports: crop 8000, vision 8002, identity 8003, scene 8004,
script 8005, tts 8006, layers 8007, render 8008, **session_manager 8095**.
**`transport.py`** — shared MinIO client (`get`/`put`/`put_bytes`/`exists`) + `install_logging(app, name)`
(one log line per request with panel id). URIs are `s3://bucket/key`. Import it in every worker.
**`session_manager.py`** — the **GPU mutex**. Only one warm model at a time on the single local GPU.
Two model kinds:
- *subprocess* (`gemma4`): it spawns/health-waits/terminates `llama-server`, and a supervisor
respawns it in-place if it crashes mid-session (keeps the same session_id + port).
- *in-process* (`siglip2`, `dots`): returns `port=None`; the **worker** loads the transformers model
itself and must expose `/unload` so `/session/close` can free the ~5GB VRAM (the mutex alone
can't reclaim it). Leases have a TTL + heartbeat; a reaper force-closes stale ones.
A GPU worker's flow: `/session/open {model}` → (409 if busy) → do work against the returned port or
its own resident model → `/session/close`. CPU workers (crop) take no session.
## Conventions
- **`ponytail:` comments** mark deliberate simplifications and name the upgrade path — respect them,
don't "fix" them without reason.
- Any non-trivial logic gets ONE runnable check in `__main__` (assert-based `demo`/self-check), not a
test suite. Follow that pattern; run the file to verify.
- The HTTP contract with the orchestrator is load-bearing and shared across repos — changing a
worker's request/response shape means reconciling the orchestrator too (see commit history:
"reconcile worker contracts").
## Specs & docs
- `spec-v3.md` — current quality/look work (narration voice, panel curation, render overhaul), marked
DONE/TODO per item. `legacy/` holds the old single-repo workers.
- `AGENTS.md` — use `scripts/analyze_video_frames.sh` (not manual seeking) to diagnose render output;
its frames are `/tmp` artifacts, never commit them.
- `collage.py` (behind `COLLAGE` flag) and the render worker carry the animated-layout planners.
HANDOFF.md
+75
View File
@@ -0,0 +1,75 @@
# HANDOFF: repo reconstructed from agent transcripts (2026-08-11)
## What was asked
The working copy at `/home/kami/Programs/n8n-worker` was deleted with `rm`, including `.git`.
This repo is the workpc GPU-compute half of the manga pipeline. The ask was to rebuild it from
the transcripts on disk, at minimum the transport layer and the pipeline flow.
## Sources used
No backup and no remote survived. `~/.local/share/Trash` is empty. No copy exists under
`/home/kami`, `/mnt/D`, or `/mnt/server`. Gitea
(`/mnt/server/mnt/hdd2/gitea/git/repositories/`) holds only correx, hexis, maven,
model-training, muzick, orchestra, and test-e2e. There is no `n8n-worker` mirror.
Reconstruction replays three histories into one timeline, ordered by timestamp:
1. Claude Code transcripts, 25 sessions in
`~/.claude/projects/-home-kami-Programs-n8n-worker/*.jsonl`. Contributes Write and Edit
content, full-file Read results, and `@`-mention attachments.
2. Codex rollouts, 5 sessions with `cwd=/home/kami/Programs/n8n-worker` under
`~/.codex/sessions/2026/07/{13,15,16,18}/rollout-*.jsonl`. Contributes 26 `apply_patch`
blocks. The 4 that the log marks `Script failed` are skipped. These carry the 2026-07-18
correctness work (`_dialogue_envelope`, `_normalize_claims`, `_annotate_speaker_methods`,
`SOM_ATTRIBUTION=1`) that exists in no Claude transcript.
3. `~/.claude/file-history/<session>/<hash>@vN` pre-edit blobs, used as cross-checks only.
Scripts live in
`/tmp/claude-1000/-home-kami-Programs-n8n-worker/6a35d5d8-9f44-4412-955c-9cf088737ad5/scratchpad/`.
`replay2.py` is the merged replay. `codex.py` extracts and applies codex patches. `backups.py`
compares against file-history. `recon_final/_log.json` holds the per-file event log.
## Verification
- Replayed to 2026-07-18T13:13:44Z, line counts match the `wc -l` recorded in the transcript at
that moment, for all 10 files it covered. crop 326, identity 314, layers 99, render 996,
scene 159, script 300, tts 231, vision 871, session_manager 238, transport 216.
- Byte sizes match the `ls -l` recorded by the same command. AGENTS.md 867, CLAUDE.md 3678,
manga-recap-pipeline-spec.md 11413, plan.md 16511, spec-v3.md 26170. `.gitignore` is 53.
- 18 files are byte-identical to their newest file-history blob. The 4 that differ (render,
script, tts, spec-v3) differ only by edits made after that blob was taken.
- `python3 -m py_compile *.py scripts/*.py attic/*.py` passes. `bash -n` passes on all 3 shell
scripts. Runtime self-checks such as `python worker_vision.py` were NOT run, because `.venv/`
is gone.
## Recovered (live tree)
Workers `worker_{crop,vision,identity,scene,script,tts,layers,render}.py`, the GPU mutex
`session_manager.py`, `transport.py`, helpers `bubble_detect.py`, `face_detect.py`,
`collage.py`, `test_vision_parse.py`, `start_workers.sh`, `systemd/install.sh`,
`scripts/{analyze_video_frames.sh,pick_tts_voice.py}`, `requirements.txt`, `.gitignore`,
and the docs `AGENTS.md`, `CLAUDE.md`, `plan.md`, `spec-v3.md`, `spec-correctness.md`,
`manga-recap-pipeline-spec.md`, `identity-rework-task.md`.
`attic/` holds files that had been deleted from the tree before the `rm`. They are kept but not
live: `worker_ocr.py`, `worker_parse.py`, `plan-workpc.md`, `char-recognition.md`.
## Still open
- `.venv/` is gone. Rebuild with
`python -m venv .venv && .venv/bin/pip install -r requirements.txt`. The ROCm torch wheel is
not pinned in requirements.txt, so install it the way workpc had it.
- `dots.tts/` (gitignored external checkout) and `legacy/` (pre-rewrite workers, moved there
2026-07-13) are not recoverable from transcripts. Re-clone dots.tts if TTS is needed.
- `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable. Neither agent
ever read them in full. All three were already deleted from the working tree as of 2026-07-17.
- Git history is gone. A fresh `git init` plus one commit replaces it. Old commits are not
recoverable.
- Two replay gaps were left unpatched. One Edit MISS in `worker_render.py` on 2026-07-14, and one
orphan Edit against `attic/plan-workpc.md` on 2026-07-13. Both files were later replaced by a
full-file snapshot, so the final content stays anchored.
- Edits made outside Claude and codex after 2026-07-18T14:35Z (the last recorded write) cannot be
detected. `unknown:` whether any exist.
## Next command
```
cd /home/kami/Programs/n8n-worker
python -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python worker_vision.py # per-file __main__ self-checks, then ./start_workers.sh
```
attic/char-recognition.md
+29
View File
@@ -0,0 +1,29 @@
# Character name recognition — open problem
The script stage writes narration that refers to characters generically ("a young man", "a woman with dark hair") because the vision describe prompt has no way to attach names to identities. Fixing this is a multi-stage problem, not a prompt tweak.
## The core loop
OCR alone doesn't help — finding "Teto" in a speech bubble gives us a string but no visual binding. The vision model needs to see the image to tie a name to an appearance. That means:
1. OCR each crop for character names/introductions (pytesseract, cpu, cheap)
2. For any crop where OCR finds a new name, re-send the same image to gemma4 with:
*"A character just spoke the name 'Teto'. Describe their distinguishing visual features."*
3. Store a name→description map: `{"Teto": "young girl, short dark hair, red scarf"}`
4. Inject known names into subsequent describe prompts:
*"Known characters in this chapter: Teto (young girl, short dark hair, red scarf). Describe this panel using their names if they appear."*
## Cost per chapter
Each named panel costs an extra gemma4 call (the identity grounding). For a typical 40-page chapter with ~200 crops and maybe 8-15 named references, that's 8-15 extra vision calls — exactly as expensive as the main describe call since it loads the same model.
## Implement hints
- Add `work_dir/characters.json` as the persistent name→description store, created/reused per chapter
- Move the identity grounding call into a helper in pipeline_worker: `srv.identify(crop_path, name)`
- Inject names into `DESCRIBE_PROMPT` at runtime (f-string or str.replace) so known names propagate forward
- `pytesseract` is zero-model-load CPU, can run on homesrv or workpc — cheap enough to run on every crop
## Not built
No code exists for this. The pipeline currently has no ocr step, no identity store, and no dynamic prompt injection. The script prompt just gets whatever `scene_text` and `tone` the vision stage produces.
attic/plan-workpc.md
+575
View File
@@ -0,0 +1,575 @@
# plan — workpc (compute + transport)
workpc owns: GPU, model lifecycle, per-item worker endpoints, the session manager that enforces one-warm-model-at-a-time. workpc holds NO durable state — every artifact goes to homesrv (minio) before a stage is considered done. sqlite lives on homesrv; workpc workers are stateless HTTP endpoints that read inputs by URI and write outputs to minio.
this plan assumes the homesrv plan (sqlite schema, minio buckets, orchestrator service, n8n workflow) lands in parallel. workpc can't be fully tested without homesrv's transport layer, but each worker can be developed and self-checked in isolation against a local minio + sqlite stub.
---
## phase 0 — repo hygiene
- `git init` in `/home/kami/Programs/n8n-worker/`, initial commit of current state on a `legacy/` branch. main branch starts clean.
- `.gitignore`: `.venv/`, `__pycache__/`, `dots.tts/` (submodule-ish, has its own repo), any local model paths.
- move existing workers into `legacy/` for reference, don't delete — the new code is a rewrite, not an edit.
## phase 1 — session manager (the GPU mutex)
new file: `session_manager.py`. FastAPI on `127.0.0.1:8095`.
this is the single most important new component on workpc. it owns:
- which model is currently loaded (gemma4 / siglip2 / dots.tts / comfyui-via-external)
- the llama-server subprocess lifecycle (start, health-wait, teardown)
- a GPU mutex — only one model resident at a time, enforced via a lock
- session lease tracking (who opened what, when, for timeout/reaper logic)
### endpoints
```
POST /session/open {model: "gemma4"|"siglip2"|"dots", ttl: 3600}
-> {session_id, port} # port is the warm server's localhost port
409 if another model is resident (caller retries or queues)
POST /session/close {session_id}
-> {ok: true} # tears down the subprocess, releases GPU
GET /session/active
-> {model, session_id, opened_at, port} | null
POST /session/heartbeat {session_id}
-> {ok: true} # extends the lease; stale sessions get reaped
```
### model registry
hardcoded map in the file (not config — these paths are workpc-specific):
```python
MODELS = {
"gemma4": {
"binary": "/mnt/D/AI/llama-cpp/llama.cpp/build/bin/llama-server",
"model": "/mnt/D/AI/gemma4/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf",
"mmproj": "/mnt/D/AI/gemma4/mmproj-F16.gguf",
"port": 8090, "ctx": 4096, "args_extra": ["--jinja", "--reasoning-budget", "0"],
},
"siglip2": {
"binary": "transformers", # in-process, not subprocess
"model": "google/siglip2-so400m-patch16-384",
"port": None, # no http server, direct python call
},
"dots": {
"binary": "transformers", # in-process
"model": "rednote-hilab/dots.tts-base",
},
}
```
siglip2 and dots load in-process (transformers, the rocm torch build already on workpc). gemma4 stays subprocess (llama-server, the working mtmd path). the session manager handles both kinds.
### reaper
background thread: any session without a heartbeat in `ttl` seconds gets force-closed. prevents a crashed n8n node from holding the GPU forever.
### systemd
system unit `/etc/systemd/system/manga-session.service`, `User=kami`, `ExecStart` via the workpc venv (`python -m uvicorn session_manager:app --host 127.0.0.1 --port 8095`). `--system-site-packages` venv to reuse the rocm torch build.
### self-check
`python session_manager.py`:
- open gemma4 session → active → open second → 409 → close first → open second → ok
- heartbeat: open with ttl=2, sleep 5, assert reaped via `/session/active`
- (gemma4 subprocess not actually started in self-check — mock the binary path to `/bin/sleep`)
---
## phase 2 — transport layer (minio client + sqlite status reporter)
new file: `transport.py`. shared by all workers. two concerns:
### 2a. minio artifact client
```python
def put(local_path: str, uri: str) -> str:
"""upload local file to minio at uri (e.g. 'manga/{mid}/{cid}/panels/p001.png'),
return the full s3-style uri. raises on failure."""
def get(uri: str, local_path: str) -> str:
"""download uri to local_path, return local_path. workers pull inputs to /dev/shm
before processing, push outputs back."""
def exists(uri: str) -> bool:
"""HEAD check. used for skip-if-exists at the worker level (defense in depth —
the orchestrator on homesrv is the real skip authority via sqlite)."""
def put_bytes(data: bytes, uri: str) -> str:
"""for generated artifacts that don't touch disk (embeddings, json)."""
```
minio endpoint: `192.168.1.104:9000` (homesrv). creds via env, defaults match existing `storage.py`.
### 2b. sqlite status reporter
workers don't own state, but they need to report progress so n8n/the orchestrator can poll. two options:
**option A (chosen): workers don't touch sqlite directly.** they return progress in their HTTP response, the orchestrator on homesrv writes to sqlite. workers are pure compute — input URI in, output URI + metadata out. keeps the "workpc = compute only" boundary clean.
**option B (rejected): workers write to sqlite over the nginx-exposed endpoint.** adds a network-sqlite dependency to every worker, couples them to homesrv's db schema. rejected.
so `transport.py` is just minio. sqlite stays homesrv-side. workers are stateless.
### self-check
`python transport.py`: put a tempfile, get it back, assert bytes match, exists() true, missing uri exists() false. mock minio client for offline test.
---
## phase 3 — worker: OCR (stage 3)
new file: `worker_ocr.py`. FastAPI on `127.0.0.1:8001`.
### input
```json
POST /ocr
{
"panel_uri": "s3://manga/{mid}/{cid}/panels/p001.png",
"job_id": "...",
"panel_id": "p001"
}
```
### process
1. `transport.get(panel_uri, /dev/shm/ocr_input.png)`
2. pytesseract.image_to_data with bbox output (`--psm 11` sparse, manga bubbles are isolated)
3. filter low-confidence (<0.5), dedupe overlapping boxes
4. cluster text lines (group words on same row within x-threshold)
5. return structured result
### output
```json
{
"panel_id": "p001",
"texts": [
{"id": "t001", "content": "...", "bbox": [x,y,w,h], "confidence": 0.97}
]
}
```
the orchestrator (homesrv) takes this, writes to sqlite `ocr_results` table, and feeds it forward to vision + scene graph. worker doesn't persist anything — it returns the json, homesrv stores it.
### self-check
synthetic image with two text blocks, assert two texts returned with sane bboxes. no minio in self-check (local path variant of the endpoint for testing).
---
## phase 4 — worker: vision (stage 4, reworked)
new file: `worker_vision.py`. FastAPI on `127.0.0.1:8002`.
### input
```json
POST /vision
{
"panel_uri": "...",
"ocr_texts": [...], // from stage 3, injected into prompt
"known_characters": [ // from identity store, injected into prompt
{"name": "Gojo", "description": "white hair, blindfold, black coat"}
],
"panel_id": "p001",
"session_id": "..." // gemma4 session, opened by orchestrator via session manager
}
```
### process
1. `transport.get(panel_uri, /dev/shm/vision_input.png)`
2. build prompt: structured output request + OCR text + known characters
3. call gemma4 at `localhost:8090/v1/chat/completions` (the session manager's warm server)
4. parse structured json from response (gemma4 with `--jinja` can do json mode; enforce via prompt + json.loads with retry on parse failure)
### output (matches spec §4)
```json
{
"panel_id": "p001",
"characters": [
{"local_id": "person_1", "appearance": {"hair": "...", "clothing": "...", "features": [...]}, "emotion": "...", "action": "..."}
],
"scene": {"location": "...", "time": "..."}
}
```
### prompt design (the hard part)
prompt must produce parseable json. structure:
```
You are analyzing a manga panel. Known characters in this story: Gojo (white hair, blindfold, black coat).
Text found in speech bubbles: "..." , "..."
Respond in this exact JSON format:
{
"characters": [{"local_id": "person_1", "appearance": {"hair": "", "clothing": "", "features": []}, "emotion": "", "action": ""}],
"scene": {"location": "", "time": ""}
}
```
gemma4's reasoning output (`<|channel>thought...<channel|>`) must be stripped before json parse — reuse `_strip_thought` from legacy code.
### self-check
mock the gemma4 call (return canned json), assert parse + structured output. real gemma4 test is integration (needs session manager + model).
---
## phase 5 — worker: character identity (stage 5)
new file: `worker_identity.py`. FastAPI on `127.0.0.1:8003`.
this is the biggest new worker. it does NOT load a model itself — it uses the siglip2 session (opened via session manager) for embeddings, and queries homesrv's sqlite (via an orchestrator endpoint, not direct sqlite) for known characters.
### input
```json
POST /identity/resolve
{
"panel_uri": "...",
"panel_id": "p001",
"vision_characters": [ // from stage 4
{"local_id": "person_1", "appearance": {...}, "bbox_in_panel": [...]}
],
"manga_id": "...",
"session_id": "..." // siglip2 session
}
```
### process
1. `transport.get(panel_uri, /dev/shm/ident_input.png)`
2. for each vision character:
a. crop the character region from the panel (bbox from vision stage — **open question: does vision return character bboxes? currently spec §4 doesn't include them. needs adding.**)
b. compute siglip2 image embedding of the crop
c. query orchestrator: `GET /characters/known?manga_id=...` → list of known characters with their reference embedding URIs
d. for each known character: `transport.get(embedding_uri, /dev/shm/ref.npy)`, cosine similarity vs the new crop embedding
e. if best match > threshold (start 0.85, tune per-manga): assign `character_id`
f. if below threshold: create unknown candidate → `POST /characters/create` to orchestrator, get new `character_id`, upload the crop as its first reference image
### output
```json
{
"panel_id": "p001",
"assignments": [
{"local_id": "person_1", "character_id": "character_001", "confidence": 0.96}
],
"new_characters": ["character_007"]
}
```
### threshold + matching strategy
- cosine similarity on siglip2 image embeddings, top-1 match
- threshold per-manga, stored in sqlite `manga_config` table (default 0.85)
- if multiple known characters are close (within 0.05 of each other), pick the best but flag low-confidence for the scene graph to handle gracefully (use "a person" in narration rather than a wrong name)
### self-check
mock siglip2 (return deterministic vectors), mock orchestrator (return known chars), assert: high-similarity → assign, low → create new. no real model in self-check.
---
## phase 6 — worker: scene graph (stage 6)
new file: `worker_scene.py`. FastAPI on `127.0.0.1:8004`.
this is a join stage, not a model stage. no GPU needed. it could run on homesrv, but keeping it on workpc for uniformity (and it's cheap enough to not need the session manager).
### input
```json
POST /scene/build
{
"panel_id": "p001",
"panel_uri": "...",
"ocr_texts": [...], // stage 3
"vision_result": {...}, // stage 4
"identity_assignments": [...], // stage 5
"characters_registry": [...] // known characters with names, for speaker attribution
}
```
### process
1. join vision characters with identity assignments → characters with real names
2. **speaker attribution** (the hard sub-problem):
- for each OCR text block, find the nearest vision character bbox (spatial heuristic)
- if text contains a name that ISN'T the nearest character (e.g. "Teto, come here" nearest to person_2), the speaker is likely person_2 addressing Teto
- if text is a question/exclamation with no names, speaker = nearest character
- if ambiguous, mark `speaker: null` (narration will quote without attribution)
- **this is heuristic and will be wrong sometimes — accept it for v2, flag for v3 improvement with a dedicated model**
3. build scene graph
### output (matches spec §6)
```json
{
"panel_id": "p001",
"characters": [{"id": "character_001", "name": "Gojo Satoru"}],
"dialogue": [{"speaker": "character_001", "text": "..."}],
"action": "character approaches enemy"
}
```
### self-check
synthetic inputs with known spatial layout, assert speaker attribution picks nearest character. test the name-exception case.
---
## phase 7 — worker: script (stage 7, reworked)
new file: `worker_script.py`. FastAPI on `127.0.0.1:8005`.
### input
```json
POST /script
{
"scene_graph": {...}, // stage 6
"chapter_context": "...", // prior scenes summary, for continuity
"panel_id": "p001",
"session_id": "..." // gemma4 session (reused from vision stage if still open)
}
```
### process
1. build prompt from scene graph: characters (with names), dialogue (quoted), action, scene setting
2. call gemma4 (warm session)
3. return narration text
### output
```json
{"panel_id": "p001", "text": "..."}
```
### prompt design
```
Write narration for this manga panel. Use the characters' real names. Quote dialogue naturally. One paragraph.
Characters present: Gojo Satoru (white hair, blindfold)
Dialogue: Gojo says "..."
Action: Gojo approaches the enemy
Setting: street, night
Narration:
```
### self-check
mock gemma4, assert prompt construction includes character names + dialogue. real test is integration.
---
## phase 8 — worker: TTS (stage 8, reworked)
new file: `worker_tts.py`. FastAPI on `127.0.0.1:8006`.
reuses the dots.tts code from legacy `render_worker.py` but decoupled from layers.
### input
```json
POST /tts
{
"text": "...",
"speaker": "narrator", // for now always narrator; multi-voice is v3
"panel_id": "p001",
"session_id": "..." // dots session
}
```
### output
```json
{
"audio_uri": "s3://manga/{mid}/{cid}/audio/p001.wav",
"duration": 4.2
}
```
worker generates audio locally, `transport.put` to minio, returns the uri. no local persistence.
### self-check
mock the tts model (write a sine wave), assert upload + uri return. real test is integration.
---
## phase 9 — worker: layers (stage 9 part 1)
new file: `worker_layers.py`. FastAPI on `127.0.0.1:8007`.
wraps the comfyui layered workflow from legacy `render_worker.py`. comfyui runs as its own external process (not via session manager — it has its own server lifecycle).
### input
```json
POST /layers
{
"panel_uri": "...",
"panel_id": "p001",
"num_layers": 4,
"prompt": "..."
}
```
### output
```json
{
"layer_uris": ["s3://manga/{mid}/{cid}/layers/p001/0.png", ...]
}
```
### self-check
mock comfyui (return canned image urls), assert upload + uri list. real test needs comfyui running.
---
## phase 10 — worker: render (stage 9 part 2, the new part)
new file: `worker_render.py`. FastAPI on `127.0.0.1:8008`.
this is the **new** rendering work — camera movement, transitions, subtitles, effects. spec §9. this is the most open-ended phase.
### input
```json
POST /render/scene
{
"panel_uri": "...",
"layer_uris": [...], // from stage 9 part 1 (optional)
"audio_uri": "...",
"narration_text": "...", // for subtitles
"panel_id": "p001",
"scene_timing": {"start": 0.0, "end": 4.2}
}
```
### process
1. pull inputs to /dev/shm
2. if layers present: parallax motion (subtle pan/zoom on separated depth layers)
3. if no layers: ken burns effect (slow zoom/pan on static panel)
4. burn subtitles from narration_text (ass format, styled)
5. transitions between scenes: crossfade (handled at the chapter assembly level, not per-scene — see below)
6. encode scene clip with ffmpeg
### output
```json
{"clip_uri": "s3://manga/{mid}/{cid}/clips/p001.mp4"}
```
### chapter assembly (separate endpoint)
```json
POST /render/assemble
{
"clip_uris": [...], // ordered
"chapter_id": "..."
}
```
concat with crossfade transitions between clips, output final `chapter.mp4`, upload to minio.
### self-check
synthetic images + silent audio, assert clip produced + uploaded. real motion test is visual (manual review).
---
## phase 12 — worker: crop (stage 2, runs first)
new file: `worker_crop.py`. FastAPI on `127.0.0.1:8000`.
the panel detector. stage 2 in the pipeline (before OCR) but written last — it's the one worker the homesrv plan referenced without a contract here. no GPU (kumiko panel detection is cpu/opencv), no session manager. one page in, N panel crops out.
### input
```json
POST /crop
{
"page_uri": "s3://manga/{mid}/{cid}/pages/p_000.png",
"manga_id": "...",
"chapter_id": "...",
"page_index": 0,
"job_id": "..."
}
```
### process
1. `transport.get(page_uri, /dev/shm/crop_input.png)`
2. kumiko panel detection (reuse the legacy kumiko path) → ordered list of panel bboxes in reading order (right-to-left, top-to-bottom for manga)
3. for each detected panel: crop from the page, `transport.put` to `manga/{mid}/{cid}/panels/pg{page_index:03d}_p{panel_index:02d}.png`
4. return the panel URIs + bboxes **in reading order** (panel_index is the within-page order; the orchestrator assigns the flat chapter-wide `panel_id`/reading order)
### output
```json
{
"page_index": 0,
"panels": [
{"panel_index": 0, "uri": "s3://manga/{mid}/{cid}/panels/pg000_p00.png", "bbox": [x, y, w, h]}
]
}
```
`bbox` is `[x, y, w, h]` on the source page (matches the homesrv `panels.bbox` column). the orchestrator is the skip authority: it calls `/crop` once per page and skips pages that already have panel rows (defense-in-depth `transport.exists` in the worker is optional).
### self-check
synthetic 2-panel page image, assert two panels returned in reading order with sane bboxes + uploaded URIs. no real kumiko model needed if the detector is opencv-only; otherwise mock the detector.
---
## phase 11 — startup script
rewrite `start_workers.sh` to launch:
- session_manager (8095)
- worker_crop (8000)
- worker_ocr (8001)
- worker_vision (8002)
- worker_identity (8003)
- worker_scene (8004)
- worker_script (8005)
- worker_tts (8006)
- worker_layers (8007)
- worker_render (8008)
each as a background uvicorn process in a tmux session. session manager starts first, workers wait for it.
systemd units for each (system services, `User=kami`) — preferred over tmux for production, tmux for dev. provide both.
---
## open questions deferred to homesrv plan
- exact sqlite schema (characters table, ocr_results, vision_results, scene_graphs, scripts, audio, clips, jobs, stages) — defined in homesrv plan
- orchestrator endpoint contracts (`/characters/known`, `/characters/create`, stage status writes) — defined in homesrv plan
- n8n workflow shape (which nodes, how session open/close maps to stage boundaries) — defined in homesrv plan
- minio bucket layout (`manga` single bucket with key prefixes, or per-manga buckets?) — defined in homesrv plan, workpc transport.py just takes URIs
## open question flagged for your decision
- **vision stage character bboxes**: spec §4 doesn't include character bounding boxes in the vision output, but the identity worker (phase 5) needs them to crop character regions for siglip2 embeddings. add `bbox` to each character in the vision output? this is a spec amendment. my recommendation: yes, add it — gemma4 can produce bboxes if asked, and it's necessary for the identity crop. flag this in the homesrv plan too since it touches the spec.
attic/worker_ocr.py
+99
View File
@@ -0,0 +1,99 @@
# worker_ocr.py — stage 3 text extraction. FastAPI :8001. cpu (easyocr), no session.
# panel in -> text blocks with bboxes + confidence out. orchestrator persists to sqlite.
# easyocr (not tesseract): it reads stylized manga lettering far better -- recovers whole
# lines tesseract garbles or drops. runs on GPU (~0.4s/page warm) by default; the OCR stage
# runs before any LLM session opens so it doesn't contend with the resident model. set
# OCR_GPU=0 to force CPU (~3s/page). GPU needs MIOPEN_FIND_MODE=FAST in the env or the first
# ROCm run spends ~60s in MIOpen's exhaustive kernel search -- the launcher sets it.
import os, uuid
from fastapi import FastAPI
from pydantic import BaseModel
import transport
app = FastAPI()
SHM = "/dev/shm"
MIN_CONF = 0.3 # easyocr line confidence floor
OCR_GPU = os.environ.get("OCR_GPU", "1") == "1"
_reader = None # easyocr.Reader, lazy-loaded on first request
def _get_reader():
global _reader
if _reader is None:
import easyocr
_reader = easyocr.Reader(["en"], gpu=OCR_GPU, verbose=False)
return _reader
def _detections_to_texts(detections):
"""easyocr readtext output [(box_pts, text, conf)] -> our text blocks with xywh bboxes.
box_pts is 4 corner [x,y] points. drops low-confidence and art-noise (<2 letters).
casing is left as-is (mixed) -- the vision stage re-cases from the image anyway."""
texts = []
for i, (box, txt, conf) in enumerate(detections):
txt = txt.strip()
if conf < MIN_CONF or sum(c.isalpha() for c in txt) < 2:
continue
xs = [p[0] for p in box]; ys = [p[1] for p in box]
x, y = int(min(xs)), int(min(ys))
texts.append({
"id": f"t{i+1:03d}",
"content": txt,
"bbox": [x, y, int(max(xs)) - x, int(max(ys)) - y],
"confidence": round(float(conf), 3),
})
return texts
def ocr_image(path: str):
return _detections_to_texts(_get_reader().readtext(path, detail=1, paragraph=False))
class OCRInput(BaseModel):
panel_uri: str
job_id: str = ""
panel_id: str = ""
@app.post("/ocr")
async def ocr(data: OCRInput):
local = transport.get(data.panel_uri, f"{SHM}/ocr_{uuid.uuid4().hex[:8]}.png")
texts = ocr_image(local)
os.remove(local)
return {"panel_id": data.panel_id, "texts": texts}
@app.post("/unload")
async def unload():
"""free the resident easyocr reader (~1-2GB) once the OCR stage is done, before gemma4 loads.
ocr isn't session-managed, so the orchestrator calls this at stage end."""
global _reader
was = _reader is not None
_reader = None
import gc; gc.collect()
try:
import torch; torch.cuda.empty_cache()
except Exception:
pass
return {"ok": True, "unloaded": was}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: detection->text-block conversion (pure, no model needed).
dets = [
([[10, 10], [110, 10], [110, 40], [10, 40]], "HELLO", 0.9), # kept
([[10, 200], [70, 200], [70, 230], [10, 230]], "WORLD", 0.8), # kept
([[5, 5], [13, 5], [13, 13], [5, 13]], "=", 0.9), # art-noise: <2 letters
([[0, 0], [50, 0], [50, 20], [0, 20]], "REAL", 0.1), # below MIN_CONF
]
texts = _detections_to_texts(dets)
assert [t["content"] for t in texts] == ["HELLO", "WORLD"], texts
assert texts[0]["bbox"] == [10, 10, 100, 30], texts[0]["bbox"]
assert texts[1]["confidence"] == 0.8
print("worker_ocr self-check ok")
attic/worker_parse.py
+125
View File
@@ -0,0 +1,125 @@
# worker_parse.py — manga parse (paged manga only). FastAPI :8009. GPU, session-guarded ("magi").
# Magi v2 chapter-wide pass: panel detection + reading order + OCR in one shot. Replaces the
# crop+ocr stages for paged manga; downstream ocr stage no-ops because rows are pre-populated.
# webtoons do NOT come here — they use worker_crop /crop/webtoon. See manga-two-repo-split memory.
import os, uuid
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import cv2
import numpy as np
import transport
app = FastAPI()
SHM = "/dev/shm"
MAGI_MODEL = "ragavsachdeva/magiv2"
_model = None
def _load_magi():
global _model
if _model is None:
import torch
from transformers import AutoModel
_model = AutoModel.from_pretrained(MAGI_MODEL, trust_remote_code=True).cuda().eval()
_model._torch = torch
return _model
class ParseInput(BaseModel):
page_uris: list # all pages of the chapter, in order
manga_id: str
chapter_id: str
session_id: str = "" # magi GPU lease (opened by orchestrator)
job_id: str = ""
def _center_in(box, panel) -> bool:
x1, y1, x2, y2 = box
px1, py1, px2, py2 = panel
cx, cy = (x1 + x2) / 2, (y1 + y2) / 2
return px1 <= cx <= px2 and py1 <= cy <= py2
def assemble_panels(pages, results, manga_id, chapter_id, put):
"""Flatten Magi's per-page output into chapter-order panels with their OCR.
`pages`: RGB np arrays. `results`: per-page dicts (Magi keys). `put(np_crop, uri)` uploads.
Text is assigned to the panel whose box contains the text-box center; SFX (non-essential)
is dropped so narration isn't polluted. bbox converted [x1,y1,x2,y2] -> [x,y,w,h]."""
out, gidx = [], 0
for img, res in zip(pages, results):
panels = res.get("panels", [])
texts = res.get("texts", [])
ocr = res.get("ocr", [])
essential = res.get("is_essential_text", [True] * len(texts))
for p in panels:
x1, y1, x2, y2 = (int(v) for v in p)
uri = f"s3://manga/{manga_id}/{chapter_id}/panels/p{gidx:03d}.png"
if not transport.exists(uri): # deterministic per gidx -> resumable
put(img[y1:y2, x1:x2], uri)
ocr_texts = []
for ti, tb in enumerate(texts):
if ti < len(ocr) and essential[ti] and _center_in(tb, p):
tx1, ty1, tx2, ty2 = (int(v) for v in tb)
ocr_texts.append({"text_id": f"t{ti}", "content": ocr[ti],
"bbox": [tx1, ty1, tx2 - tx1, ty2 - ty1], "confidence": 1.0})
out.append({"panel_index": gidx, "uri": uri,
"bbox": [x1, y1, x2 - x1, y2 - y1], "ocr": ocr_texts})
gidx += 1
return out
def _put_crop(np_rgb, uri):
tmp = f"{SHM}/parse_{uuid.uuid4().hex[:8]}.png"
cv2.imwrite(tmp, cv2.cvtColor(np_rgb, cv2.COLOR_RGB2BGR))
transport.put(tmp, uri)
os.remove(tmp)
@app.post("/parse")
async def parse(data: ParseInput):
tag = uuid.uuid4().hex[:8]
pages = []
for i, u in enumerate(data.page_uris):
local = transport.get(u, f"{SHM}/pp_{tag}_{i:03d}.png")
img = cv2.imread(local)
if img is None:
raise HTTPException(400, f"page not readable: {u}")
pages.append(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
os.remove(local)
model = _load_magi()
# ponytail: empty character bank in v1 — identity stays with the downstream siglip stage;
# feed a real bank (known-char ref crops + names) here to get Magi speaker association.
bank = {"images": [], "names": []}
with model._torch.no_grad():
results = model.do_chapter_wide_prediction(pages, bank, use_tqdm=False, do_ocr=True)
panels = assemble_panels(pages, results, data.manga_id, data.chapter_id, _put_crop)
return {"panels": panels}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: model-free. Fake a 2-page Magi result and assert panel flattening,
# chapter-order indexing, text->panel containment, SFX drop, and bbox conversion.
transport.exists = lambda *a, **k: False # no minio in self-check
stored = {}
pages = [np.zeros((100, 100, 3), np.uint8), np.zeros((100, 100, 3), np.uint8)]
results = [
{"panels": [[0, 0, 50, 100], [50, 0, 100, 100]], # page 0: two panels
"texts": [[10, 10, 20, 20], [60, 10, 70, 20]], # one text in each
"ocr": ["HELLO", "BOOM"], "is_essential_text": [True, False]}, # BOOM = SFX, dropped
{"panels": [[0, 0, 100, 100]], # page 1: one panel
"texts": [[5, 5, 15, 15]], "ocr": ["WORLD"], "is_essential_text": [True]},
]
panels = assemble_panels(pages, results, "m", "c", lambda img, uri: stored.__setitem__(uri, img.shape))
assert [p["panel_index"] for p in panels] == [0, 1, 2], "chapter-order index"
assert panels[0]["ocr"][0]["content"] == "HELLO"
assert panels[1]["ocr"] == [], "SFX text dropped from panel 1"
assert panels[2]["ocr"][0]["content"] == "WORLD"
assert panels[0]["bbox"] == [0, 0, 50, 100], "xyxy->xywh"
assert panels[0]["ocr"][0]["bbox"] == [10, 10, 10, 10]
assert stored, "crops uploaded via put"
print("worker_parse self-check ok")
bubble_detect.py
+110
View File
@@ -0,0 +1,110 @@
# bubble_detect.py — comic-text-detector ONNX -> per-panel TEXT-REGION boxes, for set-of-mark
# speaker attribution (see spec: decompose the VLM's implicit tail-tracing into grounded marks).
# Runs on the CPU EP so it never contends with gemma for the GPU. The model's `blk` head is a
# YOLO-style text-line detector; validated pixel-accurate on real panels + synthetic bubbles.
#
# We use ONLY the text-region boxes: a drawn, numbered region is the attribution anchor, and because
# region #k lives on panel N's own image it structurally prevents the window from smearing a line onto
# a neighbouring panel (the p108 bleed) or inventing dialogue that isn't drawn (the p108 phantom line).
# ponytail: full balloon-fill mask + tail-tip geometry are also in the `det`/`seg` heads but noisy and
# webtoon bubbles are often tailless anyway — add them only if region marks prove insufficient.
import os
import numpy as np
import cv2
MODEL = os.environ.get("CTD_MODEL",
os.path.join(os.path.dirname(__file__), "models", "comictextdetector.pt.onnx"))
CONF = float(os.environ.get("CTD_CONF", "0.20")) # blk head confidences run low (~0.55 max); tune per title
_sess = None
def _load():
global _sess
if _sess is None:
import onnxruntime as ort
_sess = ort.InferenceSession(MODEL, providers=["CPUExecutionProvider"])
return _sess
def _letterbox(img, sz=1024):
"""resize keeping aspect ratio, pad to sz*sz with 114-grey (the model was trained letterboxed;
a plain squash distorts tall webtoon panels and wrecks the masks). returns canvas + inverse params."""
h, w = img.shape[:2]
r = min(sz / h, sz / w)
nh, nw = int(round(h * r)), int(round(w * r))
canvas = np.full((sz, sz, 3), 114, np.uint8)
py, px = (sz - nh) // 2, (sz - nw) // 2
canvas[py:py + nh, px:px + nw] = cv2.resize(img, (nw, nh))
return canvas, r, px, py
def detect_text_regions(img, conf: float = None) -> list:
"""img: BGR ndarray (one panel). -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels,
reading order (top->bottom, left->right). Empty list on no detections (caller falls back to the
holistic prompt). Load + inference are lazy so importing this never touches the GPU or the model."""
conf = CONF if conf is None else conf
h, w = img.shape[:2]
canvas, r, px, py = _letterbox(img)
x = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB).astype(np.float32).transpose(2, 0, 1)[None] / 255.0
blk = _load().run(None, {"images": x})[0][0] # [N,7] = cx,cy,w,h,conf,cls0,cls1 in 1024 space
m = blk[:, 4] > conf
if not m.any():
return []
b = blk[m]
rects = [[float(cx - bw / 2), float(cy - bh / 2), float(bw), float(bh)] for cx, cy, bw, bh in b[:, :4]]
idx = cv2.dnn.NMSBoxes(rects, b[:, 4].tolist(), conf, 0.5)
if len(idx) == 0:
return []
out = []
for i in np.array(idx).flatten():
cx, cy, bw, bh = b[i, :4]
x1, y1 = max(0, int((cx - bw / 2 - px) / r)), max(0, int((cy - bh / 2 - py) / r))
x2, y2 = min(w, int((cx + bw / 2 - px) / r)), min(h, int((cy + bh / 2 - py) / r))
if x2 > x1 and y2 > y1:
out.append({"bbox": [x1, y1, x2, y2], "conf": round(float(b[i, 4]), 3)})
out.sort(key=lambda d: (d["bbox"][1], d["bbox"][0]))
return out
def draw_region_marks(img, regions: list):
"""draw a numbered red box (#1..#N) over each text region on a COPY of img; return the marked
image. the number is what gemma cites in set-of-mark attribution ("region 2 -> P1")."""
vis = img.copy()
for n, reg in enumerate(regions, 1):
x1, y1, x2, y2 = reg["bbox"]
cv2.rectangle(vis, (x1, y1), (x2, y2), (0, 0, 255), 2)
cv2.rectangle(vis, (x1, max(0, y1 - 22)), (x1 + 26, y1), (0, 0, 255), -1)
cv2.putText(vis, str(n), (x1 + 3, y1 - 5), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)
return vis
def draw_face_marks(img, faces: list):
"""draw a labelled green box for each character face/body, IN PLACE, return img. faces:
[{"label": "P1", "bbox": [x1,y1,x2,y2]}]. gemma names the label (P1/P2) as the region's speaker,
which the worker maps back to that character's local_id."""
for f in faces:
x1, y1, x2, y2 = f["bbox"]
cv2.rectangle(img, (x1, y1), (x2, y2), (0, 170, 0), 2)
cv2.rectangle(img, (x1, y1), (min(x2, x1 + 40), y1 + 22), (0, 170, 0), -1)
cv2.putText(img, f["label"], (x1 + 3, y1 + 17), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)
return img
if __name__ == "__main__":
# self-check: the real ONNX model must find the text inside a synthetic speech bubble, and the
# returned box must overlap the text we drew. Exercises letterbox + decode + NMS end to end.
panel = np.full((600, 400, 3), 200, np.uint8)
cv2.ellipse(panel, (200, 200), (150, 90), 0, 0, 360, (255, 255, 255), -1)
cv2.ellipse(panel, (200, 200), (150, 90), 0, 0, 360, (0, 0, 0), 3)
cv2.putText(panel, "HELLO", (110, 215), cv2.FONT_HERSHEY_SIMPLEX, 1.8, (0, 0, 0), 5)
regs = detect_text_regions(panel)
assert regs, "no text region detected in synthetic bubble"
# at least one box should contain the drawn text centre (~200,200)
hit = any(x1 <= 200 <= x2 and y1 <= 200 <= y2 for (x1, y1, x2, y2) in (r["bbox"] for r in regs))
assert hit, f"no region covers the text centre: {regs}"
marked = draw_region_marks(panel, regs)
assert marked.shape == panel.shape and marked is not panel
# face marks: labelled box drawn in place; sample the left border low enough to miss the bubble
draw_face_marks(marked, [{"label": "P1", "bbox": [50, 180, 350, 500]}])
assert tuple(int(v) for v in marked[490, 51]) == (0, 170, 0), f"face mark not green: {marked[490, 51]}"
print(f"bubble_detect self-check ok ({len(regs)} region(s), top conf {regs[0]['conf']})")
collage.py
+169
View File
@@ -0,0 +1,169 @@
# collage.py — task 183 deterministic layout planner for animated manga collages.
# Pure geometry, no ffmpeg/IO: ordered panels (+ aspects, RTL, emphasis, frame) -> a layout state
# (template, aspect-correct resting rectangles, entrance vectors, z-order, hold/transition timing).
# worker_render turns a state into a clip; this module decides WHERE everything sits so the decision
# is testable without rendering. Coordinates are pixels in the output frame, origin top-left.
#
# ponytail: a bounded template set (n=1..4) covering the reference's common resting layouts, not all 7.
# Add templates when a fixture proves a missing one is needed — the fit/RTL/entrance machinery is shared.
MARGIN = 0.035 # outer + inter-panel gap as a fraction of the frame's short side
HOLD_S = 3.0 # reference holds run ~2-4s; #185 overrides per-beat from narration length
TRANS_S = 0.45 # reference transitions run ~0.3-0.6s
def _fit(aspect, slot):
"""Aspect-correct rect centered inside slot (x,y,w,h). aspect=w/h. No stretch: letterbox to fit."""
sx, sy, sw, sh = slot
if aspect >= sw / sh: # panel wider than slot -> width-bound
w = sw; h = w / aspect
else: # taller than slot -> height-bound
h = sh; w = h * aspect
return (sx + (sw - w) / 2.0, sy + (sh - h) / 2.0, w, h)
def _slots(template, n, W, H, g):
"""Slot rectangles (pre-fit) for a template, in READING order (slot[0] = read first).
g = gap in pixels. Slots tile the safe area; _fit later letterboxes each panel inside its slot."""
x0, y0 = g, g
fw, fh = W - 2 * g, H - 2 * g
if template == "centered_wide":
return [(x0, y0, fw, fh)]
if template == "vertical_pair": # two columns
cw = (fw - g) / 2.0
return [(x0, y0, cw, fh), (x0 + cw + g, y0, cw, fh)]
if template == "stacked_wides": # two rows
rh = (fh - g) / 2.0
return [(x0, y0, fw, rh), (x0, y0 + rh + g, fw, rh)]
if template == "strip_over_dominant": # thin top strip (reads first), big bottom
sh = fh * 0.32
return [(x0, y0, fw, sh), (x0, y0 + sh + g, fw, fh - sh - g)]
if template == "supporting_left_dominant_right": # small stack left, one dominant right
lw = fw * 0.34
k = max(1, n - 1)
rh = (fh - (k - 1) * g) / k
left = [(x0, y0 + i * (rh + g), lw, rh) for i in range(k)]
return left + [(x0 + lw + g, y0, fw - lw - g, fh)]
if template == "quad": # 2x2
cw, rh = (fw - g) / 2.0, (fh - g) / 2.0
return [(x0, y0, cw, rh), (x0 + cw + g, y0, cw, rh),
(x0, y0 + rh + g, cw, rh), (x0 + cw + g, y0 + rh + g, cw, rh)]
raise ValueError(f"unknown template {template}")
def _pick_template(aspects):
"""Choose a resting template from panel count and aspect ratios (w/h)."""
n = len(aspects)
if n <= 1:
return "centered_wide"
if n == 2:
if all(a < 0.9 for a in aspects): # two tall panels -> side by side
return "vertical_pair"
if all(a > 1.15 for a in aspects): # two wide panels -> stacked
return "stacked_wides"
return "vertical_pair"
if n == 3:
return "supporting_left_dominant_right"
return "quad" # 4 (planner caps callers at 4)
def plan_layout(aspects, rtl=True, active=0, frame=(1080, 1920)):
"""Deterministic collage layout for one beat.
aspects: panel width/height ratios in reading order (len 1..4).
rtl: right-to-left reading (manga) -> reading-first panel takes the RIGHTmost horizontal slot.
active: index of the dominant/emphasized panel (gets the largest slot where a template has one).
returns dict: template, rects[(x,y,w,h)] aligned to input panel order, entrances[(dx,dy)] pixel
offset a panel starts at before sliding to rest, z_order, emphasis, hold_s, transition_s.
Rects/entrances are indexed to match the INPUT panel order (not slot order)."""
n = len(aspects)
if n == 0:
return {"template": "empty", "rects": [], "entrances": [], "z_order": [],
"emphasis": 0, "hold_s": HOLD_S, "transition_s": TRANS_S}
W, H = frame
g = int(MARGIN * min(W, H))
template = _pick_template(aspects)
slots = _slots(template, n, W, H, g)
# Map input panels (reading order) to slots. Templates whose first slots are a horizontal run
# honor RTL by reversing that run so the reading-first panel lands on the right.
order = list(range(n))
if rtl and template in ("vertical_pair", "quad"):
if template == "vertical_pair":
order = [1, 0]
else: # quad: reverse each row
order = [1, 0, 3, 2][:n]
# dominant panel takes the dominant slot when the template has a distinguished one (last slot).
if template in ("strip_over_dominant", "supporting_left_dominant_right") and 0 <= active < n:
rest = [i for i in range(n) if i != active]
order = rest + [active] # dominant slot is the last one in _slots order
rects = [None] * n
for slot_i, panel_i in enumerate(order):
rects[panel_i] = _fit(aspects[panel_i], slots[slot_i])
# entrances: reading-direction slide for horizontal templates; dominant scales in place (dy=0,dx=0).
edge = W if rtl else -W # RTL panels enter from the right (+x), LTR from left
entrances = []
for i in range(n):
if i == active and template in ("centered_wide", "strip_over_dominant",
"supporting_left_dominant_right"):
entrances.append((0, 0)) # emphasized panel resolves by scale, not slide
elif template == "stacked_wides":
entrances.append((0, -H if i == 0 else H)) # rows drop/rise into place
else:
entrances.append((edge, 0))
# supporting panels sit above the dominant in z so their shadow reads; dominant drawn first (back).
z_order = sorted(range(n), key=lambda i: 0 if i == active else 1)
return {"template": template, "rects": rects, "entrances": entrances, "z_order": z_order,
"emphasis": active, "hold_s": HOLD_S, "transition_s": TRANS_S}
if __name__ == "__main__":
W, H = 1080, 1920
def _inside(r):
x, y, w, h = r
return x >= -1 and y >= -1 and x + w <= W + 1 and y + h <= H + 1
# 1) single panel -> centered_wide, aspect preserved, inside frame.
p = plan_layout([1.5], frame=(W, H))
assert p["template"] == "centered_wide" and len(p["rects"]) == 1
x, y, w, h = p["rects"][0]
assert abs(w / h - 1.5) < 1e-3 and _inside(p["rects"][0])
# 2) two tall panels -> vertical_pair; RTL puts reading-first (panel 0) on the RIGHT.
p = plan_layout([0.6, 0.6], rtl=True, frame=(W, H))
assert p["template"] == "vertical_pair"
assert p["rects"][0][0] > p["rects"][1][0], "RTL: panel 0 is rightmost"
# LTR flips it.
q = plan_layout([0.6, 0.6], rtl=False, frame=(W, H))
assert q["rects"][0][0] < q["rects"][1][0], "LTR: panel 0 is leftmost"
# 3) two wide panels -> stacked_wides; panel 0 on top.
p = plan_layout([1.6, 1.6], frame=(W, H))
assert p["template"] == "stacked_wides" and p["rects"][0][1] < p["rects"][1][1]
# 4) three panels, active=2 -> dominant takes the large right slot (widest rect).
p = plan_layout([0.7, 0.7, 1.3], active=2, frame=(W, H))
assert p["template"] == "supporting_left_dominant_right"
dom = p["rects"][2][2] * p["rects"][2][3]
assert all(dom >= p["rects"][i][2] * p["rects"][i][3] for i in (0, 1)), "active is dominant area"
assert all(_inside(r) for r in p["rects"])
# 5) aspect never stretched: fitted rect ratio == input aspect for every panel/template.
for asp in ([1.5], [0.6, 0.6], [1.6, 1.6], [0.7, 0.7, 1.3], [1.0, 1.0, 1.0, 1.0]):
pl = plan_layout(asp, frame=(W, H))
for a, r in zip(asp, pl["rects"]):
assert abs(r[2] / r[3] - a) < 1e-3, (asp, a, r)
# 6) quad RTL reverses each row; all four rects disjoint-ish (tile the frame).
p = plan_layout([1, 1, 1, 1], rtl=True, frame=(W, H))
assert p["template"] == "quad" and len(p["rects"]) == 4
assert p["rects"][0][0] > p["rects"][1][0], "top row RTL: panel0 right of panel1"
# 7) entrances: RTL horizontal-slide panels start off the right edge; deterministic.
p = plan_layout([0.6, 0.6], rtl=True, frame=(W, H))
assert any(dx > 0 for dx, _ in p["entrances"]), "RTL entrance from right"
assert plan_layout([0.6, 0.6], frame=(W, H)) == plan_layout([0.6, 0.6], frame=(W, H))
print("collage self-check ok")
face_detect.py
+74
View File
@@ -0,0 +1,74 @@
# face_detect.py — anime face detector (YOLOv8, ONNX) -> real character-face boxes for set-of-mark
# speaker attribution. The vision DETECT stage returns gemma-guessed bboxes which are too imprecise to
# draw a mark on; this gives grounded boxes the same way bubble_detect gives grounded text regions.
# Model: deepghs/anime_face_detection face_detect_v1.4_s (single class "face"). CPU EP -> no GPU contention.
# ponytail: face boxes only (not full body). Attribution just needs "which face said this"; add a
# person head only if off-panel/back-turned speakers need body grounding.
import os
import numpy as np
import cv2
MODEL = os.environ.get("FACE_MODEL",
os.path.join(os.path.dirname(__file__), "models", "anime_face_v1.4_s.onnx"))
CONF = float(os.environ.get("FACE_CONF", "0.30"))
_sess = None
def _load():
global _sess
if _sess is None:
import onnxruntime as ort
_sess = ort.InferenceSession(MODEL, providers=["CPUExecutionProvider"])
return _sess
def _letterbox(img, sz=640):
h, w = img.shape[:2]
r = min(sz / h, sz / w)
nh, nw = int(round(h * r)), int(round(w * r))
canvas = np.full((sz, sz, 3), 114, np.uint8)
py, px = (sz - nh) // 2, (sz - nw) // 2
canvas[py:py + nh, px:px + nw] = cv2.resize(img, (nw, nh))
return canvas, r, px, py
def detect_faces(img, conf: float = None) -> list:
"""img: BGR ndarray. -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels, sorted
reading order. Empty on no faces. Lazy load so import never touches the model."""
conf = CONF if conf is None else conf
h, w = img.shape[:2]
canvas, r, px, py = _letterbox(img)
x = cv2.cvtColor(canvas, cv2.COLOR_BGR2RGB).astype(np.float32).transpose(2, 0, 1)[None] / 255.0
out = _load().run(None, {"images": x})[0][0].T # (8400,5) = cx,cy,w,h,conf in 640 space
m = out[:, 4] > conf
if not m.any():
return []
b = out[m]
rects = [[float(cx - bw / 2), float(cy - bh / 2), float(bw), float(bh)] for cx, cy, bw, bh in b[:, :4]]
idx = cv2.dnn.NMSBoxes(rects, b[:, 4].tolist(), conf, 0.45)
if len(idx) == 0:
return []
res = []
for i in np.array(idx).flatten():
cx, cy, bw, bh = b[i, :4]
x1, y1 = max(0, int((cx - bw / 2 - px) / r)), max(0, int((cy - bh / 2 - py) / r))
x2, y2 = min(w, int((cx + bw / 2 - px) / r)), min(h, int((cy + bh / 2 - py) / r))
if x2 > x1 and y2 > y1:
res.append({"bbox": [x1, y1, x2, y2], "conf": round(float(b[i, 4]), 3)})
res.sort(key=lambda d: (d["bbox"][1], d["bbox"][0]))
return res
if __name__ == "__main__":
# self-check needs a real anime face; use a page from the sample series if present, else skip loudly.
import glob
pages = glob.glob("/mnt/server/mnt/hdd2/manga-raw/*/*.png")
if not pages:
print("face_detect: no sample page available, skipping live check")
else:
faces = detect_faces(cv2.imread(sorted(pages)[13]))
assert faces, "no face detected on a page that should have characters"
for f in faces:
x1, y1, x2, y2 = f["bbox"]
assert x2 > x1 and y2 > y1 and f["conf"] >= CONF
print(f"face_detect self-check ok ({len(faces)} face(s), top conf {faces[0]['conf']})")
identity-rework-task.md
+67
View File
@@ -0,0 +1,67 @@
# Rework manga character identity: tracklet spine + gemma resolver
Replace siglip-cosine as the primary identity signal. Priority: high.
## Problem
Identity quality plateaus because **siglip2 cosine is the wrong instrument**: it's a
semantic encoder ("man, office, manga panel"), not an instance re-id model. Different
people in the same setting score high; the same person across scenes scores low. No
threshold / gate / gallery fixes what the vector *means* — they only shave ~10% off the
error. Symptoms on the test chapter:
- male-brown colleague labeled as female-black **Choi Haeseon**
- speech bubble mis-attributed (MC credited, colleague actually speaking)
- "MC becomes Choi Haeseon" at the end (reconcile over-merge)
## Already done (guards against *new* contamination, not a fix for existing data)
- Gender gate in `worker_identity.match()` candidate filter + `_pending_match`
(opposite decided genders never match).
- Reconcile gender gate in `service.run_stage_reconcile` (skip pair if decided genders differ).
- `db.create_character` name-dedup: skip fold if decided genders conflict
(vision-hallucinated name on a wrong-gender crop).
- **NOTE:** existing roster is already polluted; guards only stop NEW contamination.
Needs `/stage/clear identity reconcile scene script` + re-run to benefit.
## Target architecture — two-tier (from multi-object tracking)
**Tier 1 — tracklets (local, greedy, high-precision).**
Link per-panel detections into per-person tracklets across a 5–10 panel window. Local
association is the reliable regime (same page, adjacent shot, stable appearance, often the
same speaker). Don't touch global identity yet. A tracklet = ordered crops for one person →
carries an embedding gallery + merged attributes + associated speaker/dialogue.
**Tier 2 — resolve tracklet → global character (deliberate, max evidence).**
Once a tracklet is stable, resolve identity ONCE per person per scene, using a whole gallery
of views + gemma's opinion — not greedily per-crop. Multi-view (front/side/angry/crying)
falls out for free (a tracklet accumulates poses as it spans panels).
## Signal fusion — keep ORDINAL, not a 9-weight learned sum
No labeled data to tune weights → a hand-weighted 9-signal cost is a worse treadmill.
Make it ordinal:
- **HARD GATE:** gender, species → block impossible matches.
- **DECIDER:** gemma "same person?" vs a text character-sheet (reuse reconcile `/same`).
- **STRONG FEATURE:** hair color+style, name / honorific / alias from dialogue
(Korean honorifics = gold anchors).
- **TIEBREAK:** embedding gallery max-sim (K~5), dialogue-speaker continuity, scene co-presence.
Only build a learned/weighted scorer AFTER labeling a couple chapters.
## Attributes to add (cherry-picked)
- Yes / cheap / high-value: honorifics, aliases, hair color/style, gender, species.
- Skip for now (noisy, marginal): age, body-shape, eye color — add only when a specific
mislabel needs them.
## Reuses existing infra (rewiring, not greenfield)
- `/direct/window` = windowed multi-image gemma calls in reading order → tracklet window.
- reconcile `/same` = pairwise "same person?" gemma call → tier-2 adjudicator.
- appearance attrs, gender, dialogue+speaker already in the scene graph.
## Suggested sequencing
1. **Tier-2 first** (~1 day, most of the win): gemma resolver vs text character-sheet,
gender-gated, gallery max-sim as tiebreak. Swaps cosine-decider for gemma-decider.
2. **Tier-1 tracklet spine:** link detections → tracklets, carry gallery + merged attrs,
resolve once per tracklet. Robust across poses; contains errors to a tracklet not a crop.
3. Add dialogue-continuity / scene-co-presence tiebreaks.
4. (Later, if labeled data exists) learned association cost.
## Open question
Confirm "**gemma decides, tracklets give it evidence**" is the intended core
(vs the fusion score being the core).
manga-recap-pipeline-spec.md
+248
View File
@@ -0,0 +1,248 @@
# Manga/Manhwa Recap Pipeline — Technical Spec
**Status:** draft v1
**Target:** fully local, single-GPU, resumable batch pipeline. No publishing (copyright out of scope).
**Core constraint:** no new PyTorch / ROCm wheels. Existing llama.cpp (gemma) stays. Everything else runs on `onnxruntime + opencv + numpy + scikit-learn`.
---
## 1. Goal
Given raw manga pages or manhwa strips for a series, automatically produce a narrated recap video with burned-in subtitles, with minimal human involvement (one-time character-bank setup per series).
This spec replaces a dependency on the `magi`/`magiv2` model by **decomposing its subtasks** into swappable, low-dependency components.
---
## 2. Design principles
- **Dependency isolation.** gemma runs on llama.cpp (unchanged). All CV/ML runs via ONNX Runtime. Detection models are small enough to run on CPU EP; ROCm EP optional. No `transformers`, no second torch install.
- **Specialists over generalist.** Do not ask one VLM to detect + identify + attribute. Each subtask is a dumb specialized component; a clean data bus (JSON files) connects them.
- **Resumable.** Every stage reads its input artifact from disk and writes its output artifact to disk. A stage is skipped if its output exists and is newer than its input (unless `--force`).
- **Supervised shortcut for identity.** Replace magi's unsupervised character clustering with a per-series character bank (exemplar crops + names). Simpler, more robust, and how magiv2 gets names anyway.
- **Fail loud, fail per-panel.** Low-confidence panels are flagged, not silently guessed. Confidence thresholds are config, not magic numbers in code.
---
## 3. Architecture overview
```
ingest → [1] panels → [2] detect → [3] identity → [4] text
│
[5] speaker-bind ←──────────┤
│
[6] filter essential ←──────┘
│
[7] transcript (ordered)
│
┌─────────────────────┴───────────────────┐
[8] scene-action (gemma) (transcript.json)
└─────────────────────┬───────────────────┘
│
[9] script gen (LLM, chapter + rolling summary)
│
[10] TTS (dots.tts) ──► [11] visual assembly (ffmpeg) ──► [12] subs (faster-whisper) ──► [13] mux
```
Stages 2–7 are the magi replacement. Stages 8–13 are the existing recap backend.
---
## 4. Filesystem layout
```
work/
<series>/
source/ # input pages/strips
ch<NN>/
panels/ # panel crops, ordered
panels.json # [1]
detections.json # [2]
identities.json # [3]
texts.json # [4]
transcript.json # [5][6][7] merged
script.json # [9]
audio/ # [10] wav per segment
video.mp4 # [13] final
bank/
bank.json # character bank (per series, hand-built once)
crops/ # exemplar images
```
Artifact-per-stage = resumability. Delete an artifact to re-run that stage forward.
---
## 5. Stage specs
### [1] Panel extraction
- **Manga (page-based):** `kumiko` → ordered panel polygons. Pure OpenCV.
- **Manhwa (vertical strip):** slice at horizontal whitespace bands. OpenCV: row-wise background uniformity → cut points. Produces pseudo-panels.
- **In:** `source/*` · **Out:** `panels/`, `panels.json`
- **Dep:** kumiko, opencv. No torch.
```jsonc
// panels.json
{
"type": "manga", // or "manhwa"
"reading_order": "rtl", // rtl | ltr | ttb
"panels": [
{ "id": "p001", "page": 1, "bbox": [x,y,w,h], "file": "panels/p001.png", "order": 0 }
]
}
```
### [2] Detection (text + balloons + characters)
- **Text + balloons:** `comic-text-detector` (ships ONNX). Returns text regions + balloon masks. This is what kills the OCR problem — you never OCR a full page again.
- **Character boxes:** YOLOv8 anime face/person model exported to ONNX. CPU-fine.
- **In:** `panels/` · **Out:** `detections.json`
- **Dep:** onnxruntime, opencv, numpy.
```jsonc
// detections.json (per panel)
{
"p001": {
"balloons": [
{ "id": "b0", "mask_poly": [[x,y],...], "centroid": [x,y], "bbox": [x,y,w,h] }
],
"text_regions": [
{ "id": "t0", "bbox": [x,y,w,h], "in_balloon": "b0" } // null if floating (sfx/sign)
],
"chars": [
{ "id": "c0", "bbox": [x,y,w,h], "crop": "..." }
]
}
}
```
### [3] Character identity (bank match)
- Embed every `chars[*]` crop with SigLIP or an anime ArcFace model (ONNX).
- Cosine-match against `bank.json` entries. Above threshold → assign name; else `"unknown"`.
- **Bank is built once per series by hand** (10 min): a few exemplar crops + a name each. This is the only required human touch.
- **In:** `detections.json`, `bank/` · **Out:** `identities.json`
- **Dep:** onnxruntime, numpy, sklearn (cosine / nearest-neighbour).
- **Note:** this is the accuracy-critical stage. Generic embeddings + same-face syndrome = the weakest link vs magi. The bank is what rescues it. Do NOT attempt unsupervised clustering as primary — it merges lookalikes.
```jsonc
// bank.json
{ "characters": [
{ "name": "Aria", "exemplars": ["bank/crops/aria_0.png","bank/crops/aria_1.png"] }
]}
// identities.json (per panel)
{ "p001": { "c0": { "name": "Aria", "score": 0.82 }, "c1": { "name": "unknown", "score": 0.41 } } }
```
### [4] Text extraction
- For each `in_balloon` text region: crop the balloon, feed the clean crop to gemma (llama.cpp) → read text. Isolated crops read far better than full pages.
- **In:** `detections.json`, `panels/` · **Out:** `texts.json`
- **Dep:** llama.cpp (existing). No torch.
```jsonc
// texts.json
{ "p001": { "t0": "We can't stay here." } }
```
### [5] Speaker binding
Two paths — start geometry, fall back to gemma on low confidence.
- **A — tail geometry (default, pure CV):** from the balloon mask, find the tail = sharpest protrusion off the centroid. Vector centroid→tip; nearest `chars[*]` box along that ray = speaker. Hand-rolled magiv2 tail logic.
- **B — set-of-mark + gemma (fallback):** draw numbered boxes on chars + balloons on the panel image, ask gemma "balloon 2 → face #3 or #5?". Grounding via drawn marks beats free-form spatial reasoning. No new deps.
- **Trigger fallback when:** >2 candidate chars, ambiguous/absent tail, or geometry confidence < threshold.
- **In:** `detections.json`, `identities.json`, `texts.json` · **Out:** merged into `transcript.json`.
### [6] Essential vs non-essential filter
- Pure geometry: `text_region.in_balloon != null` → dialogue. Floating on raw art → sfx/sign → **drop**. Kills "THUD" and street-sign garbage with no classifier.
### [7] Transcript assembly (reading order)
- Panel order from `panels.json`. Within panel: manga = sort (top → right-to-left); manhwa = top-down.
- Emit ordered speaker+line list.
```jsonc
// transcript.json
{ "chapter": 12, "lines": [
{ "panel": "p001", "speaker": "Aria", "line": "We can't stay here.", "conf": 0.79, "flagged": false }
]}
```
### [8] Scene-action description
- gemma describes physical action per panel ("she draws a sword") — the one thing magi does NOT do and gemma is good at. Runs in parallel with 2–7.
- **Out:** `scene` field per panel, merged for the script stage.
### [9] Script generation
- LLM input = **one chapter of transcript + scene-actions + a ~500-token rolling summary** of prior chapters. Do NOT token-max the context; attention degrades mid-window and coherence/attribution drop.
- Prompt: compress and narrate, not transcribe. Output narration segments each mapped to source panel IDs (needed for visual timing).
- **Out:** `script.json`
```jsonc
// script.json
{ "segments": [
{ "id": "s0", "text": "Cornered in the ruins, Aria makes her choice...", "panels": ["p001","p002"] }
]}
```
### [10] TTS
- `dots.tts`, wav per segment. Keep durations — they drive visual timing.
### [11] Visual assembly
- ffmpeg ken-burns (pan/zoom) per panel. Panel display time = proportional to its segment's audio length. moviepy to orchestrate or raw filtergraphs for lean/fast.
### [12] Subtitles
- `faster-whisper` on generated audio → timestamped SRT → burn in with ffmpeg. Easier than aligning from the script side.
### [13] Mux
- Combine video + audio + burned subs → `video.mp4`.
---
## 6. Dependency matrix
| Stage | Tooling | New torch? |
|---|---|---|
| 1 panels | kumiko, opencv | no |
| 2 detect | comic-text-detector (onnx), yolo-anime (onnx) | no |
| 3 identity | siglip/arcface (onnx), sklearn | no |
| 4 text | llama.cpp gemma | no |
| 5 bind | numpy geometry + gemma fallback | no |
| 6 filter | numpy geometry | no |
| 7 order | numpy | no |
| 8 scene | llama.cpp gemma | no |
| 9 script | local LLM | no |
| 10 tts | dots.tts | (its own env) |
| 11 video | ffmpeg, moviepy | no |
| 12 subs | faster-whisper | (ctranslate2, not torch) |
| 13 mux | ffmpeg | no |
Net: the magi replacement (2–7) adds **only ONNX Runtime + a couple of onnx model files**. No ROCm wheel churn.
---
## 7. Orchestration
- Bash driver calls python stage scripts; artifacts handed off as files.
- Each stage: `stage_N.py --series X --chapter NN [--force]`.
- Skip logic: if output exists and `mtime(output) > mtime(input)` and not `--force`, skip.
- Model loading: sequence stages so you never hold two large models in VRAM at once (gemma vs detectors vs faster-whisper). One card handles all, serially.
- Do NOT wrap in a Spring service. This is a batch job, not request/response — a service is pure added attack surface and state for zero benefit.
---
## 8. Hardware notes
- gemma (mmproj) is the VRAM heavyweight; detectors are CPU-viable.
- Sequence model loads; never co-resident. Target: single decent GPU, staged.
- Detection/embedding on CPU EP is fine and frees VRAM for gemma.
---
## 9. Known risks / caveats
- **Re-ID accuracy** is the weak link vs magi's end-to-end association. Mitigation: character bank (mandatory), tune cosine threshold per series, flag `unknown` rather than guess.
- **Manhwa layout:** comic-text-detector and the anime detectors are manga/anime-trained. Vertical webtoon art + non-japanese layout = degraded results. Expect tuning on the whitespace slicer and lower binding confidence.
- **Tail geometry** fails on off-panel speakers and thought bubbles. Fallback to gemma SoM; if still ambiguous, flag the line.
- **Rolling summary drift:** long series accumulate summary error. Periodically re-anchor the summary from a canonical synopsis if available.
---
## 10. Pre-build check (do this first)
Before building any of this: magi's HF weights may load through your **existing** ROCm torch directly, ignoring its pinned `requirements.txt` (the version pain is usually the wrapper deps, not torch). 5-minute test. If it instantiates and runs, you skip this entire rebuild. If it OOMs or the arch won't load, decompose per this spec.
plan.md
+414
View File
@@ -0,0 +1,414 @@
# Pipeline correctness plan
## Goal
Make character identity, naming, dialogue transcription, speaker attribution, scene grouping, and
script generation fail loudly instead of silently turning uncertain model output into canonical story
facts.
The desired end state is an evidence-bearing pipeline in which every name, speaker, quote, action,
and scene boundary can be traced to its source panel and confidence. Operational batching must never
create narrative boundaries.
## Scope and repository boundary
This worker repository implements the model and media stages. Scheduling, database mutations,
tracklet reconciliation, backfills, beat aggregation, and review gates live in the separate homesrv
orchestrator repository. Worker contract changes below must therefore be implemented and tested in
both repositories.
The current intended flow is:
```text
crop/order panels
-> character detection
-> identity shortlist and tracklets
-> identity adjudication/reconciliation
-> grounded dialogue transcription
-> speaker resolution
-> scene graph assembly
-> scene/beat grouping
-> narration and verification
-> TTS/render
```
## Current strengths
- Vision, dialogue, identity, direction, and script generation are separate focused passes.
- Identity uses gender-gated SigLIP shortlists and has a Gemma Tier-2 resolver.
- Unnamed one-off characters are held until a second sighting instead of always polluting the
permanent registry.
- Dialogue has a short-window mode for cross-panel turn-taking.
- Set-of-mark text and face grounding exists.
- Character-detection parse failures are distinguished from editorial `skip` decisions.
- Panel ordering and suspicious overlap checks exist.
- Direction and script stages support multi-panel beats.
## P0: stop silent corruption
These changes should land before further prompt tuning or visual polish.
### 1. Introduce typed, canonical speaker references
Problem: dialogue is allowed to return either a panel-local ID or an off-panel character name, but
scene assembly only maps local IDs. A valid off-panel name can therefore become `None` and be narrated
as "Someone."
Use an explicit speaker representation at the dialogue boundary:
```json
{
"speaker": {
"kind": "local_id|character_id|name|unknown|narrator",
"value": "person_1"
}
}
```
Normalize this to `character_id` before building a scene graph. Name lookup must support canonical
names and aliases, reject ambiguous duplicate-name matches, and retain unresolved names as unresolved
rather than discarding them.
Acceptance criteria:
- An off-panel named speaker resolves to the correct canonical character.
- An unknown speaker remains unknown and is never assigned to the main character by default.
- Duplicate/ambiguous names produce a review flag, not an arbitrary selection.
- Scene graphs never receive raw `person_N` or free-form name values as canonical speakers.
### 2. Preserve confidence, method, and provenance end to end
Problem: dialogue confidence and identity ambiguity are currently discarded by scene assembly. A weak
turn-taking guess is narrated exactly like a clearly grounded bubble tail.
Every dialogue line should retain at least:
```json
{
"dialogue_id": "p012_r03",
"panel_id": "p012",
"region_id": "r03",
"text": "...",
"type": "speech",
"speaker_id": "character_x",
"speaker_confidence": 0.72,
"speaker_method": "tail|som_face|turn_taking|solo_prior|manual",
"identity_confidence": 0.91,
"flags": []
}
```
Acceptance criteria:
- Confidence and method survive dialogue -> scene -> script review artifacts.
- Low-confidence attribution can be rendered neutrally without asserting a name.
- Configurable thresholds decide accept/review/reject; prompts do not decide policy.
- Manual corrections are distinguishable from model output and are never overwritten on resume.
### 3. Make all technical failures fail loudly
Problem: dialogue JSON failure and missing window results currently become empty dialogue, which is
indistinguishable from a genuinely silent panel.
All model endpoints should return explicit status metadata:
```json
{
"status": "ok|partial|failed",
"parse_failed": false,
"expected_items": 3,
"returned_items": 3,
"warnings": []
}
```
Acceptance criteria:
- Parse failure cannot mark a stage complete.
- A window response missing one requested panel becomes `partial` and queues only the missing work.
- Empty dialogue is accepted only when the model completed successfully and grounded detection found
no relevant text.
- Script/TTS/render cannot run for panels or beats with unresolved upstream technical failures.
### 4. Remove the unconditional solo-character speaker assignment
Problem: one visible character does not prove they own a bubble; reaction panels often show the
listener while an off-panel character speaks.
Treat sole presence as a prior only. It may raise a score, but it must not overwrite `unknown` without
tail/region evidence or strong conversation continuity.
Acceptance criteria:
- A one-face reaction panel can retain an off-panel speaker.
- Solo attribution records its evidence and confidence.
- Tests cover visible speaker, visible listener/off-panel speaker, thought bubble, and narration box.
## P1: ground transcription and identity
### 5. Make region-grounded dialogue the primary path
Set-of-mark text/face grounding currently exists but is disabled by default. Calibrate it on real
titles, then make it the normal path with a clearly reported holistic fallback.
Required validation:
- One output record per detected dialogue/caption region.
- No region can be consumed twice.
- Every line remains attached to the panel containing its region.
- Unmarked text invented by the VLM is rejected or flagged.
- Tail/face evidence is recorded separately from conversation-flow inference.
- Detector failure and zero detected regions are distinct states.
Do not treat all non-Latin text as inherently decorative. Language filtering should be configurable
per title/source; otherwise legitimate original-language dialogue can be deleted.
### 6. Separate name claims from canonical names
Problem: detection can emit a name and identity immediately persists a named unmatched character.
This lets one visual-language-model guess become permanent identity state and bypass the two-sighting
guard.
Represent naming as evidence:
```json
{
"claim_id": "...",
"panel_id": "p012",
"name": "Seonho",
"target_local_id": "person_2",
"evidence_type": "address|self_intro|caption|name_tag|roster_hint",
"confidence": 0.84
}
```
Canonical naming should require corroboration or review. Roster names are hints, never identity
evidence. A name claim alone must not create or merge a canonical character.
Acceptance criteria:
- Addressed-person and speaking-person tests cannot swap the name.
- Nearby chart/UI labels cannot become canonical names.
- Conflicting name claims are surfaced.
- A later confirmed name safely backfills earlier appearances and narration previews.
### 7. Strengthen character identity evidence
Current risks include context-heavy body crops, outfit changes, partial occlusion, a global threshold,
and a Tier-2 resolver that compares the crop mainly against generated text descriptions.
Implement:
- Face-first embeddings, with body/outfit evidence as a secondary channel.
- Multiple reference exemplars per canonical character.
- Tracklet aggregation across adjacent/overlapping windows before committing identity.
- Per-title calibrated thresholds.
- A real open-set result: known match, new character, or unresolved.
- Candidate reference images in Tier-2 comparison, not only prose descriptions.
- A same-panel exclusivity constraint: two distinct visible people cannot resolve to one identity
unless reflection/duplicate-art evidence explicitly allows it.
- Alias and duplicate-name collision checks.
Acceptance criteria:
- Outfit changes do not create a new identity when face evidence is strong.
- Similar-looking simultaneous characters do not collapse into one identity.
- Weak crops remain unresolved rather than minting or merging characters.
- Identity decisions expose all contributing evidence and thresholds.
## P1: preserve panel semantics across solo and batch processing
### 8. Stop destructively merging faceless webtoon fragments
Problem: vertically stacking a text-only fragment with a nearby face changes the visual evidence and
can manufacture speaker proximity. It also collapses source gaps and weakens source-coordinate
traceability.
Keep fragments separate and add contextual links instead:
```json
{
"fragment_id": "f12",
"source_bbox": [0, 1200, 800, 300],
"context_fragments": ["f11", "f13"],
"link_reason": "adjacent_text_without_face"
}
```
The dialogue window can see linked fragments while retaining their boundaries and coordinates.
Acceptance criteria:
- Source pixels and coordinates remain reconstructable.
- A caption/monologue is not assigned merely because a face fragment is adjacent.
- Fragment linkage can be reviewed and changed without recropping the chapter.
### 9. Reconcile overlapping dialogue windows
Operational windows should overlap by at least one or two panels. Results in the overlap must be
reconciled using region IDs, transcription similarity, and attribution confidence.
Acceptance criteria:
- Changing the window size does not change the final transcript for stable fixtures.
- No duplicate lines appear from overlaps.
- Conflicting overlap predictions produce a flag and retain both candidates for review.
- Conversation history is scene-aware and resets only at a confirmed boundary.
### 10. Prevent batch boundaries from becoming scene boundaries
The first panel of a direction window currently becomes a new scene even when it continues the prior
window. Add `continues_previous` or an equivalent cross-window edge, then run a final boundary
reconciliation pass over compact ordered beat summaries.
Validate grouping as an exact partition:
- Every input panel appears exactly once.
- Groups are contiguous and ordered.
- There are no duplicate or unknown panel IDs.
- Missing/invalid grouping fails or falls back with a warning.
Acceptance criteria:
- The same chapter grouped with different operational window sizes produces equivalent scene
boundaries.
- A conversation crossing a window boundary remains one beat.
- Non-contiguous model groups are rejected rather than silently applied.
## P2: make narration auditable
### 11. Use an ordered, evidence-bearing beat schema
A multi-panel script request should contain ordered member panels rather than a flattened scene graph
plus `panel_count`:
```json
{
"beat_id": "b07",
"panels": [
{"panel_id": "p20", "dialogue_ids": ["p20_r1"], "actions": [...]},
{"panel_id": "p21", "dialogue_ids": [], "actions": [...]}
]
}
```
This preserves action and dialogue chronology and gives render/subtitle stages exact mappings.
### 12. Add a deterministic script verifier before TTS
Verify generated narration against the beat artifact:
- Every used character name exists in the canonical cast map.
- Direct quotes fuzzy-match source dialogue and retain the correct speaker.
- No panel, dialogue line, or action is moved out of chronological order.
- Unsupported proper nouns and character names are rejected.
- Low-confidence facts are not asserted as certain.
- Each sentence/cue maps to one or more source panel/dialogue IDs.
Failed verification should regenerate once with structured feedback, then enter script review. It
must not proceed automatically to TTS.
### 13. Make review gates evidence-focused
Prioritize review items by impact:
1. Duplicate/conflicting canonical names.
2. Identity collisions or unresolved recurring tracklets.
3. Low-confidence or contradictory speaker assignments.
4. Missing/partial dialogue regions.
5. Cross-window scene-boundary conflicts.
6. Script verifier failures.
The reviewer should see the source panel, marked text region, candidate speaker/identity evidence,
confidence, and downstream narration affected by the decision.
## Test plan
Create a small checked-in fixture corpus with expected structured artifacts. Avoid testing only prompt
strings; test complete contracts and invariants.
Required fixture cases:
- Two similar characters speaking back and forth.
- One visible listener with an off-panel speaker.
- Bubble-only panel continuing a prior speaker.
- Name used to address someone other than the speaker.
- Self-introduction and caption-based naming.
- Same character in different clothing and lighting.
- Two simultaneous similar-looking characters.
- Thought, shout, narration, sign, UI text, and SFX classification.
- Sentence split across bubbles without crossing panel ownership.
- Webtoon text fragment between two face fragments.
- Scene and conversation crossing an operational window boundary.
- Malformed/truncated JSON and a model response omitting one panel.
- Multi-panel beat with interleaved action and dialogue chronology.
Contract/invariant tests should assert:
- Stable panel and region IDs across resume.
- No raw local ID or unresolved free-form name reaches the script as a canonical identity.
- No successful empty transcript when text regions were detected but not accounted for.
- No permanent character mutation from a single unsupported name claim.
- No duplicated panel/dialogue membership across beats.
- No TTS request before all blocking correctness gates pass.
Add end-to-end golden tests for:
```text
panels -> detections -> tracklets -> transcript -> beats -> script claims
```
The golden artifact should compare structured facts, not exact prose wording.
## Observability and evaluation
Track correctness metrics per title and pipeline version:
- Character-ID pairwise precision/recall over recurring characters.
- Canonical-name accuracy and unresolved-name rate.
- Dialogue region recall and hallucinated-line rate.
- Speaker-attribution accuracy, split by tail-grounded, solo prior, and off-panel inference.
- Scene-boundary precision/recall.
- Script unsupported-claim and wrong-name rates.
- Percentage of panels/beats sent to review.
- Result stability when operational batch/window sizes change.
Log model/prompt version, detector version, thresholds, window membership, and evidence method with
each decision so regressions are reproducible.
## Suggested implementation sequence
1. Define versioned shared schemas for dialogue lines, speaker references, name claims, identities,
panels, and beats.
2. Update worker and orchestrator contracts to preserve confidence/provenance and explicit failure
states.
3. Fix speaker normalization and remove the unconditional solo assignment.
4. Add fail-loud dialogue completeness checks.
5. Enable/calibrate region-grounded transcription and create dialogue fixtures.
6. Separate name claims from persistence; add canonical-name conflict handling.
7. Implement stronger tracklet evidence and image-based Tier-2 identity comparison.
8. Replace destructive webtoon fragment merging with contextual links.
9. Add overlapping-window reconciliation and cross-window scene edges.
10. Introduce ordered beat artifacts and the script verifier.
11. Build evidence-focused review screens and quality dashboards.
12. Re-run the golden corpus after every prompt/model/threshold change.
## Immediate repository hygiene
- Fix the stale `worker_script.py` self-check: implementation expects a ~25-word solo budget while
the assertion still expects 35 words.
- Add tests proving dialogue parse failure is not treated as successful empty dialogue.
- Add tests for named off-panel speaker normalization and the visible-listener/off-panel-speaker case.
- Document the configured dialogue and direction window sizes in the orchestrator.
- Confirm that crop warnings, identity ambiguity, backfills, and reconciliation confidence are
persisted and honored by review gates rather than merely logged.
## Definition of done
This correctness pass is complete when:
- Every narrated name, quote, speaker, action, and scene boundary is traceable to source evidence.
- Low-confidence and failed work cannot silently reach TTS/render.
- Identity and scene results are stable across resume and operational window-size changes.
- The fixture corpus passes end to end in both worker and orchestrator repositories.
- Remaining uncertainty is explicitly reviewable rather than hidden behind polished narration.
requirements.txt
+15
View File
@@ -0,0 +1,15 @@
fastapi
uvicorn[standard]
python-multipart
requests
minio
numpy
opencv-python
kumiko
onnxruntime # set-of-mark: comic-text-detector text regions + anime-face boxes (CPU EP); bubble_detect.py, face_detect.py
# GPU / in-process models (rocm torch build already on workpc):
torch
transformers
accelerate
soundfile
# system deps (not pip): ffmpeg, comfyui (external server)
scripts/analyze_video_frames.sh
Executable
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env bash
# Turn a video into timestamped contact sheets and scene-change frames for visual analysis.
# Optional --window START:DURATION arguments create dense 10-fps sheets around transitions.
set -euo pipefail
usage() {
cat <<'EOF'
Usage:
scripts/analyze_video_frames.sh VIDEO [OUTPUT_DIR] [--window START:DURATION ...]
Examples:
scripts/analyze_video_frames.sh chapter.mp4
scripts/analyze_video_frames.sh chapter.mp4 /tmp/chapter-analysis \
--window 5.8:2.2 --window 15.8:2.5
Outputs:
overview-001.jpg ... 2-fps timestamped contact sheets
scenes/scene-001.jpg ... frames selected by scene-change score
scene-timestamps.txt scene-change timestamps from FFmpeg showinfo
window-START.jpg 10-fps contact sheet for each requested window
Environment overrides:
OVERVIEW_FPS=2 SCENE_THRESHOLD=0.18 WINDOW_FPS=10
EOF
}
if [ "$#" -lt 1 ] || [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then
usage
exit $([ "$#" -lt 1 ] && echo 1 || echo 0)
fi
video=$1
shift
if [ ! -f "$video" ]; then
echo "video not found: $video" >&2
exit 1
fi
if [ "$#" -gt 0 ] && [[ $1 != --* ]]; then
output_dir=$1
shift
else
base=$(basename "$video")
output_dir="/tmp/${base%.*}-frames"
fi
overview_fps=${OVERVIEW_FPS:-2}
scene_threshold=${SCENE_THRESHOLD:-0.18}
window_fps=${WINDOW_FPS:-10}
windows=()
while [ "$#" -gt 0 ]; do
case "$1" in
--window)
[ "$#" -ge 2 ] || { echo "--window needs START:DURATION" >&2; exit 2; }
windows+=("$2")
shift 2
;;
*)
echo "unknown argument: $1" >&2
usage >&2
exit 2
;;
esac
done
mkdir -p "$output_dir/scenes"
# Six-by-six sheets keep individual frames readable while allowing videos of arbitrary length:
# tile emits additional overview-NNN images after every 36 sampled frames.
ffmpeg -hide_banner -loglevel error -y -i "$video" \
-vf "fps=${overview_fps},scale=320:-1,drawtext=text='%{pts\\:hms}':x=8:y=8:fontsize=20:fontcolor=yellow:borderw=2,tile=6x6:padding=4:margin=4" \
-fps_mode vfr "$output_dir/overview-%03d.jpg"
# Keep stderr because showinfo reports the original timestamps there.
ffmpeg -hide_banner -y -i "$video" \
-vf "select='gt(scene,${scene_threshold})',showinfo" -fps_mode vfr \
"$output_dir/scenes/scene-%03d.jpg" 2>"$output_dir/scene-showinfo.log" || true
sed -n 's/.*pts_time:\([^ ]*\).*/\1/p' "$output_dir/scene-showinfo.log" \
>"$output_dir/scene-timestamps.txt"
for window in "${windows[@]}"; do
start=${window%%:*}
duration=${window#*:}
if [ "$start" = "$window" ] || [ -z "$start" ] || [ -z "$duration" ]; then
echo "invalid window '$window'; expected START:DURATION" >&2
exit 2
fi
safe_start=${start//./_}
# A 5x5 sheet covers 2.5 seconds at the default 10 fps. Longer windows naturally emit more sheets.
ffmpeg -hide_banner -loglevel error -y -ss "$start" -t "$duration" -i "$video" \
-vf "fps=${window_fps},scale=320:-1,drawtext=text='%{pts\\:hms}':x=6:y=6:fontsize=18:fontcolor=yellow:borderw=2,tile=5x5:padding=3:margin=3" \
-fps_mode vfr "$output_dir/window-${safe_start}-%03d.jpg"
done
echo "video analysis frames: $output_dir"
scripts/pick_tts_voice.py
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""Interactively generate and choose a dots.tts narrator reference."""
import argparse
import os
from pathlib import Path
import shutil
import subprocess
import sys
import time
DEFAULT_TEXT = "The story continues as our hero steps forward into the unknown."
DEFAULT_MODEL = "rednote-hilab/dots.tts-base"
DEFAULT_VOICE_DIR = Path("~/.cache/manga-tts").expanduser()
def load_model(model_name: str):
# Match worker_tts.py's workaround for the tokenizer bundled with dots.tts.
import transformers
original = transformers.AutoTokenizer.from_pretrained.__func__
transformers.AutoTokenizer.from_pretrained = classmethod(
lambda cls, *args, **kwargs: original(
cls, *args, **{"fix_mistral_regex": True, **kwargs}
)
)
from dots_tts.runtime import DotsTtsRuntime
return DotsTtsRuntime.from_pretrained(model_name, precision="bfloat16")
def write_wav(result, path: Path) -> None:
import numpy as np
import soundfile as sf
audio = result["audio"]
if hasattr(audio, "detach"):
audio = audio.detach().cpu().numpy()
sf.write(path, np.asarray(audio, dtype="float32").squeeze(),
result["sample_rate"], subtype="PCM_16")
def play(path: Path) -> None:
players = (
("ffplay", "-nodisp", "-autoexit", "-loglevel", "error"),
("aplay", "-q"),
("paplay",),
)
for command in players:
if shutil.which(command[0]):
subprocess.run([*command, str(path)], check=False)
return
print(f"No ffplay, aplay, or paplay found; listen manually: {path}")
def choice() -> str:
while True:
answer = input("[p]ick / p[a]rk / [s]kip: ").strip().lower()
aliases = {"p": "pick", "pick": "pick", "a": "park", "park": "park",
"s": "skip", "skip": "skip"}
if answer in aliases:
return aliases[answer]
print("Enter p, a, or s.")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--text", default=os.environ.get("VOICE_REF_TEXT", DEFAULT_TEXT),
help="sample text to speak")
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--voice-dir", type=Path, default=DEFAULT_VOICE_DIR)
args = parser.parse_args()
voice_dir = args.voice_dir.expanduser()
parked_dir = voice_dir / "parked"
voice_dir.mkdir(parents=True, exist_ok=True)
parked_dir.mkdir(parents=True, exist_ok=True)
candidate = voice_dir / "voice_candidate.wav"
selected = voice_dir / "narrator_ref.wav"
print(f"Loading {args.model} ...")
model = load_model(args.model)
print(f"Reference text: {args.text!r}")
try:
while True:
print("\nGenerating a new voice ...")
write_wav(model.generate(text=args.text), candidate)
play(candidate)
action = choice()
if action == "pick":
os.replace(candidate, selected)
(voice_dir / "narrator_ref.txt").write_text(args.text + "\n")
print(f"Selected: {selected}")
return 0
if action == "park":
stamp = time.strftime("%Y%m%d-%H%M%S")
parked = parked_dir / f"voice-{stamp}.wav"
suffix = 2
while parked.exists():
parked = parked_dir / f"voice-{stamp}-{suffix}.wav"
suffix += 1
os.replace(candidate, parked)
parked.with_suffix(".txt").write_text(args.text + "\n")
print(f"Parked: {parked}")
else:
candidate.unlink(missing_ok=True)
print("Skipped.")
except (KeyboardInterrupt, EOFError):
candidate.unlink(missing_ok=True)
print("\nStopped without changing the selected voice.")
return 130
if __name__ == "__main__":
sys.exit(main())
session_manager.py
+238
View File
@@ -0,0 +1,238 @@
# session_manager.py — the GPU mutex. one warm model at a time on workpc.
# owns: model registry, llama-server subprocess lifecycle, session leases + reaper.
# workpc holds no durable state; this only guards the single local GPU.
#
# subprocess models (gemma4): this process starts/stops llama-server and health-waits.
# in-process models (siglip2, dots): no server — the lease just reserves the GPU; the
# worker loads the transformers model itself after /session/open returns (port=None).
# the guarantee that matters is the mutex: a second open() gets 409 until the first closes.
import os, time, uuid, threading, subprocess
import requests
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
MODELS = {
"gemma4": {
"binary": "/mnt/D/AI/llama-cpp/llama.cpp/build/bin/llama-server",
"model": "/mnt/D/AI/gemma4/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf",
"mmproj": "/mnt/D/AI/gemma4/mmproj-F16.gguf",
"port": 8090, "ctx": 32768, # 12B fits 32k KV unquantized on 16GB ROCm; do NOT -ctk/-ctv quant (leaks on ROCm)
# MTP speculative decoding: the ~430M gemma-4 draft head drafts up to 4 tokens the 12B verifies.
# needs `--spec-type draft-mtp` to actually engage (without it llama-server loads the draft but
# silently runs no speculation). ~42% accept on our JSON stages -> faster decode; ~0.9GB VRAM.
"args_extra": ["--jinja", "--reasoning-budget", "0",
"-md", "/mnt/D/AI/gemma4/mtp-gemma-4-12b-it-BF16.gguf", "-ngld", "99",
"--spec-type", "draft-mtp", "--spec-draft-n-max", "4"],
},
# in-process models: the worker loads them and holds ~5GB resident. `worker` is the port whose
# /unload frees that GPU memory on close -- the mutex alone can't reclaim it (leaked across runs,
# starving the next model on open). subprocess models (gemma4) free via terminate instead.
"siglip2": {"binary": "transformers", "model": "google/siglip2-so400m-patch16-384", "port": None, "worker": 8003},
"dots": {"binary": "transformers", "model": "rednote-hilab/dots.tts-base", "port": None, "worker": 8006},
}
SKIP_HEALTH = False # self-check flips this; real runs health-wait the llama-server
_lock = threading.Lock()
# the single resident session, or None. {model, session_id, opened_at, port, ttl, last_beat, proc}
_active = None
class OpenReq(BaseModel):
model: str
ttl: int = 3600
class SessionReq(BaseModel):
session_id: str
def _health_wait(port: int, timeout: int = 300):
deadline = time.time() + timeout
while time.time() < deadline:
try:
if requests.get(f"http://127.0.0.1:{port}/health", timeout=2).status_code == 200:
return
except requests.RequestException:
pass
time.sleep(1)
raise TimeoutError(f"llama-server on :{port} never became healthy")
def _start_subprocess(cfg):
"""spawn llama-server for a subprocess-type model, return the Popen once healthy."""
cmd = [cfg["binary"], "-m", cfg["model"], "--port", str(cfg["port"]),
"-c", str(cfg.get("ctx", 4096)), "-ngl", "99"]
if cfg.get("mmproj"):
cmd += ["--mmproj", cfg["mmproj"]]
cmd += cfg.get("args_extra", [])
# keep llama-server's stderr — it prints the assert / CUDA error / context-overflow that kills it.
# DEVNULL here is why crashes showed up as "no error in the logs". append so a respawn keeps history.
log = open(f"/tmp/llama-server-{cfg['port']}.log", "a")
proc = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT)
if not SKIP_HEALTH and cfg["port"]:
_health_wait(cfg["port"])
return proc
def _teardown(sess):
proc = sess.get("proc")
if proc:
if proc.poll() is None:
proc.terminate()
try:
proc.wait(timeout=15) # wait() even on an already-dead proc, else it stays a zombie
except subprocess.TimeoutExpired:
proc.kill()
# in-process models live in the worker, not here -- ask it to free the GPU (best-effort;
# if the worker is down the VRAM is already gone with it).
worker = sess.get("worker")
if worker:
try:
requests.post(f"http://127.0.0.1:{worker}/unload", timeout=30)
except requests.RequestException:
pass
@app.post("/session/open")
def open_session(req: OpenReq):
if req.model not in MODELS:
raise HTTPException(400, f"unknown model {req.model}")
global _active
cfg = MODELS[req.model]
session_id = uuid.uuid4().hex
with _lock:
if _active is not None:
raise HTTPException(409, f"{_active['model']} resident (session {_active['session_id']})")
# ponytail: claim the slot before the slow spawn/health-wait so /active and /close
# for other sessions aren't blocked behind it; only the actual subprocess start runs unlocked.
_active = {
"model": req.model, "session_id": session_id, "opened_at": time.time(),
"port": cfg.get("port"), "ttl": req.ttl, "last_beat": time.time(), "proc": None,
"worker": cfg.get("worker"),
}
proc = _start_subprocess(cfg) if cfg["binary"] != "transformers" else None
with _lock:
if _active is not None and _active["session_id"] == session_id:
_active["proc"] = proc
return {"session_id": session_id, "port": cfg.get("port")}
@app.post("/session/close")
def close_session(req: SessionReq):
with _lock:
global _active
if _active is None or _active["session_id"] != req.session_id:
raise HTTPException(404, "no such active session")
_teardown(_active)
_active = None
return {"ok": True}
@app.get("/session/active")
def active():
with _lock:
if _active is None:
return None
return {k: _active[k] for k in ("model", "session_id", "opened_at", "port")}
@app.post("/session/heartbeat")
def heartbeat(req: SessionReq):
with _lock:
if _active is None or _active["session_id"] != req.session_id:
raise HTTPException(404, "no such active session")
_active["last_beat"] = time.time()
return {"ok": True}
def _reap_once():
with _lock:
global _active
if _active and time.time() - _active["last_beat"] > _active["ttl"]:
_teardown(_active)
_active = None
return True
return False
def _supervise_once():
"""Respawn a crashed subprocess model in place. llama-server can OOM/die mid-session; the worker
talks to the port directly, so if we don't bring it back every remaining panel of the stage fails
with 'connection refused'. Restarting under the SAME session keeps the lease + port valid so the
orchestrator never has to re-open. Returns True if a respawn happened."""
with _lock:
global _active
sess = _active
if not sess:
return False
proc = sess.get("proc")
if proc is None or proc.poll() is None:
return False # in-process model, or subprocess still alive
try:
proc.wait(timeout=1) # reap the zombie
except subprocess.TimeoutExpired:
pass
print(f"[supervise] gemma subprocess died (rc={proc.returncode}) session {sess['session_id']}; "
f"respawning", flush=True)
try:
sess["proc"] = _start_subprocess(MODELS[sess["model"]])
sess["last_beat"] = time.time() # don't count the downtime against the TTL reaper
return True
except Exception as e:
print(f"[supervise] respawn failed: {e}; clearing session so the mutex frees", flush=True)
_active = None
return False
def _reaper():
while True:
time.sleep(1)
if _reap_once():
continue
_supervise_once()
@app.on_event("startup")
def _startup():
threading.Thread(target=_reaper, daemon=True).start()
if __name__ == "__main__":
# self-check: mutex (409 on second open), close releases, reaper force-closes a stale lease.
SKIP_HEALTH = True
MODELS["gemma4"] = {"binary": "/bin/sleep", "model": "5", "port": None, "args_extra": []}
s1 = open_session(OpenReq(model="gemma4", ttl=3600))
assert active()["model"] == "gemma4"
try:
open_session(OpenReq(model="siglip2"))
assert False, "second open must 409"
except HTTPException as e:
assert e.status_code == 409
close_session(SessionReq(session_id=s1["session_id"]))
assert active() is None
s2 = open_session(OpenReq(model="gemma4", ttl=2))
_active["last_beat"] -= 5 # simulate a lease that missed its heartbeat window
assert _reap_once() is True
assert active() is None
# supervisor: a crashed subprocess is respawned in place, keeping the same session_id + port.
s4 = open_session(OpenReq(model="gemma4", ttl=3600))
old_proc = _active["proc"]
old_proc.terminate(); old_proc.wait() # simulate the llama-server crashing mid-session
assert _supervise_once() is True
assert active()["session_id"] == s4["session_id"] # lease survived
assert _active["proc"] is not old_proc and _active["proc"].poll() is None # fresh live proc
close_session(SessionReq(session_id=s4["session_id"]))
assert active() is None
# in-process model: close must POST /unload to the worker so its resident VRAM is freed.
calls = []
requests.post = lambda url, **kw: calls.append(url) or type("R", (), {"status_code": 200})()
s3 = open_session(OpenReq(model="siglip2", ttl=3600))
close_session(SessionReq(session_id=s3["session_id"]))
assert any("/unload" in u and ":8003" in u for u in calls), calls
print("session_manager self-check ok")
spec-correctness.md
+25
View File
@@ -0,0 +1,25 @@
# Correctness status — worker side
Audit of `plan.md` against the actual code (2026-07-18). Most of the plan is **implemented**; this file
records only what's verified done and the remaining calibration work, so nobody re-does finished work. Orchestrator
counterpart: `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/spec-correctness.md`.
### #5 region-grounded dialogue — code done, calibration ongoing
Set-of-mark is now the **default path** (`SOM_ATTRIBUTION="1"`, `worker_vision.py:31`; the "off by
default" comment at `:22` is stale — fix it). Residual is per-title threshold calibration on real
chapters, not a code gap. Detector-failure vs zero-regions are already distinct (`_set_of_mark`).
## Verified done (was in plan/earlier drafts as TODO — do not redo)
- #10 cross-window direction continuity: the grouping pass emits `continues_previous`; the first row
carries it through `/direct/window`, and a true edge forces `new_scene=false`. The self-check covers
both a continuing window and the chapter-start guard.
- Script self-check (`python worker_script.py` passes). #2 confidence + `speaker_method` reach the scene
graph (`worker_scene.py:80-81`). H3/#3 fail-loud: `/dialogue` and `/dialogue/window` return a status
envelope (`_dialogue_envelope`) and skip dropped panels instead of manufacturing silent-empty.
- #4 solo assignment is a 0.7-confidence prior with `speaker_method=solo_prior`, not unconditional.
- #6 typed name claims emitted (`_normalize_claims`), roster is hints-only.
- #7 Tier-2 `/vision/resolve` sends candidate **reference images** (≤3 each), judges face-first, returns
open-set `state: known|new|unresolved`.
- #8 webtoon fragments remain independent and retain source bboxes; `context_fragment_links` emits
contextual links instead of the crop endpoints destructively stacking them.
spec-v3.md
+348
View File
@@ -0,0 +1,348 @@
this is a v3 delta spec: it lists what changes on top of the working v2 pipeline (see spec-v2.md).
v2 is validated end-to-end (a full chapter renders to one narrated mp4). v3 is about quality of the
narration and the look of the video, plus panel curation. structured for hand-off to an agent.
# manga recap v3 — quality + look
## 0. what v2 does today (baseline)
- one narrated `chapter.mp4` per chapter: fit-on-black panels, uniform ken-burns zoom, burned
subtitles, hard cuts, single default TTS voice.
- identity assigns a stable `character_id` per body (siglip2), but characters are unnamed, so the
script prints "Person A/B". the script worker sees one panel at a time (no memory).
- every panel is narrated (no curation).
---
## A. cross-panel identity & continuity [biggest quality lever] [DONE]
SHIPPED:
- descriptive frozen labels: worker_scene emits `label` per character (name -> frozen registry
appearance via `_describe`, "the one with black hair and glasses" -> Person X fallback);
worker_script `_name_map` consumes it. no "Person A" when appearance is known, char_id preserved.
- chapter_context populated: service.py `_chapter_context` feeds a cast+entities roster to every
script call so the narrator has memory (kills "Unknown voice", item 11).
- name acquisition + backfill (item 12): vision emits `name` only when a char is explicitly named;
service.py scene stage binds it via `set_character_name` (first name wins, only if unnamed);
script stage resolves display names from the FINAL registry (`names_by_id`) so a name learned in
ANY panel backfills to every panel's narration. ceiling: vision name-inference can misattribute;
first confident name sticks. upgrade path = roster seed for higher precision.
STILL TODO (small, deferred):
- true per-panel story-state (only what's been seen so far) for "establishing shot once / don't
re-narrate repeats" (the C-todo) — currently whole-manga roster.
--- original plan ---
one feature: carry a running **story-state** through a chapter so narration has memory. covers
former items 4, 11, 12.
- **stable anchor, mutable label.** the `character_id` is never lost (already stable via siglip2).
only its *display label* changes:
- name known -> the name.
- name unknown -> a **descriptive phrase** ("guy with black hair and glasses", "girl with short
hair holding a phone") derived ONCE from vision's appearance data and frozen per character_id,
reused across all panels. never regenerate per-panel (labels would drift).
- **name acquisition + backfill.** when dialogue addresses a character by name and speaker
attribution is confident, bind name -> character_id and switch the label to the name from then on
(optionally backfill earlier references).
- **story-state threaded per chapter.** accumulate {cast seen (id,label,name?), entities/props seen
(the cat)} and feed it into scene+script (populate the currently-empty `chapter_context`). this is
what lets the narrator say "he calls out to his cat" instead of leaking a bare name (item 12), and
removes "Unknown voice" (item 11).
- **optional roster seed (not required).** a per-manga character list (names, maybe ref art) makes
naming + cross-panel matching more reliable than dialogue inference. graceful-degrade without it;
add when available.
touches: worker_vision (appearance -> label), worker_scene / orchestrator (story-state store +
speaker continuity), worker_script (consume context), orchestrator characters table (label, name).
## B. narration voice & style (items 2, 3) [DONE]
- **less descriptive gemma** (item 2): DONE. prompt now demands TIGHT recap (1-2 sentences, ~35
words, "no purple prose"); max_tokens 512->160, temp 0.7->0.6 (worker_script).
- **one fixed voice** (item 3): DONE. dots.tts samples a RANDOM speaker per call with no reference,
so the voice drifted. worker_tts now bootstraps one seeded reference clip (`~/.cache/manga-tts/
narrator_ref.wav`) and clones from it every synth. delete the file to reroll; swap REF_WAV/REF_TEXT
for a hand-picked voice.
- **multi-voice per character: future**, not v3 (user: "later") — same mechanism, per-character ref clips.
## C. panel curation (item 10) [DONE — basic]
- skip panels with no narrative content (pure scenery / transition art); show an establishing shot
once, don't re-narrate repeats. vision flags "scenery-only / skip". big pacing + cost win; do early
because it shrinks every downstream stage.
- SHIPPED: vision emits `skip:true` for no-narrative panels (worker_vision prompt+schema); the
script stage honors it (service.py run_stage_script) -> no narration -> cascades to no audio, no
clip. Single chokepoint, no new db columns (skip rides in the vision result_json blob).
- STILL TODO (needs cross-panel memory, folds into A): "establishing shot once, don't re-narrate
repeats" — requires dedup of repeated scenery across panels. deferred to story-state.
## D. render / look overhaul (items 1, 5, 6, 7, 8)
all in worker_render.py (self-contained ffmpeg builder), except #6.
- **#1 dynamic clip duration** — DONE. `_audio_dur` probes narration length via ffprobe; scene_timing
kept only as fallback. no more 4.0s sawtooth / vanishing subs.
- **#5 blurred-panel background fill** — DONE. filter_complex: cover-scaled + boxblur copy behind the
fitted panel (replaces black bars).
- **#7 subtitles** — DONE. boxed caption: bold white on translucent black box (BorderStyle 3),
lower-center, DejaVu Sans 52. (user chose boxed over outline/minimal.)
- **#8 content-aware motion** — DONE. vision picks a camera.effect per panel from the fixed vocab
(+ to=[x,y] for dolly_to_subject); it rides the vision result_json -> scene_graph.camera ->
render. worker_render `_motion(camera, frames)` maps the vocab to a zoompan z/x/y expression
(static/zoom/pan/dolly/shake/orbit); default zoom_in = old ken burns. orchestrator render stage
reads scene_graph.camera. self-checked (ffmpeg renders pan + dolly clips).
- **#6 multi-panel composites + non-generic transitions** [the ambitious one, sequenced last]
- **non-generic transitions — DONE.** vision emits `transition` (cut|crossfade|dissolve|
fade_black|fade_white|wipe_left|wipe_right|push) per panel -> scene_graph.transition ->
orchestrator assemble collects a per-clip list -> worker_render `_xfade_chain` builds an
xfade+acrossfade graph (audio kept in sync, overlap clamped to clip length). all-cut/empty
keeps the fast stream-copy concat (no re-encode); any real transition re-encodes the chapter.
self-checked (two clips fade_white into one mp4). transition = OUT of that panel.
- **multi-panel composites — DONE (basic).** grouping source = new pass (`grouping.py`,
deterministic): consecutive SMALL panels (area < 0.7*page-max, same source page, capped at
MAX_GROUP=3) share ONE shot. render + assemble both call plan_groups so their grouping matches.
a composite = vertical stack of the members on a W×H still; each panel keeps its own script+tts
(per-panel stages unchanged); the audios play in sequence while a moving highlight marks the
active row ("swap the front panel while audio plays"). clip keyed by the group LEADER; assemble
takes one clip per group and the group's exit transition = last member's. solo groups render
exactly as before, so blast radius is only genuinely-small adjacent panels.
knobs: MAX_GROUP, SMALL_FRAC. ceilings: vertical-stack layout only (no grid/overlay/front-swap
animation beyond the highlight), size+adjacency grouping (no narrative-beat grouping), black
gutters (no blurred fill in composites). upgrade paths noted in code.
## D2. beat pacing, timed cues, subtitle redesign, collage (vikunja 185/187/188/183)
scene-level narration made the beat (not the panel) the render unit; these refine how a beat plays.
all in worker_render.py + collage.py, orchestrator run_stage_render.
- **185 content-aware panel timing — DONE.** a beat's one narration used to split EQUALLY across its
member panels (`D/n`). now the orchestrator scores each member from its scene graph (dialogue amount,
action density, camera emphasis: `_beat_weights`) and passes bounded weights to render; worker
`_beat_slices(D, n, weights)` distributes screen-time by weight with a per-panel floor (no flashing)
and a deterministic equal-split fallback (degenerate/short weights). self-checked.
- **187 timed subtitle cues — DONE.** the whole narration paragraph used to show from frame 1, leaking
later-panel lines early. `cue_plan` splits the flowing narration into sentence cues, times them on the
SAME [0,D] timeline as the panel slices (length-weighted + readable floor, reused `_beat_slices`),
maps each cue to the panel on screen at its start, and clamps a cue never to precede its panel. per-cue
ASS events (not one paragraph). cue plan returned + persisted to `clips.cues_json` for the review UI /
collage. no-TTS-alignment fallback = punctuation/length estimate. self-checked ("Hold up." can't
appear before its panel).
- **188 subtitle redesign — DONE.** `_ass_multi` now takes mode (off|minimal|boxed, env `SUB_MODE`,
default minimal) + orientation presets (portrait/landscape safe-areas). smaller type (40/32 vs 52),
minimal = thin outline + soft shadow (no box), boxed = restrained ~50% box. `_wrap2` caps every cue at
≤2 short lines (ellipsis truncation) so it never regrows into a paragraph. `_sub_align` dodges the
subject vertically using the director focus point (camera.to.y low -> caption to top). ceiling: no
real face/bubble CV yet — camera focus is the only subject signal wired.
- **183 animated manga collage — DONE (core + wired behind `COLLAGE` flag).** `collage.py plan_layout`
is a pure deterministic planner: ordered panel aspects + RTL + dominant + frame -> template
(centered_wide / vertical_pair / stacked_wides / strip_over_dominant / supporting_left_dominant_right
/ quad), aspect-correct resting rects (no stretch), entrance vectors (RTL slide / dominant scale),
z-order, hold/transition timing. `worker_render collage_cmd` renders it: blurred plate of the dominant
panel + sharp aspect-fit panels with restrained drop shadows, non-dominant panels sliding into place
over ~0.45s, crisp holds; reuses 185 slices + 187 cues + 188 subtitles. orchestrator run_stage_render
calls it when `COLLAGE=1` (else the ken-burns montage). ceilings (QA-tuned against the reference video,
no still to eyeball here): transition-only directional/radial MOTION BLUR not applied (clean
slide/scale); per-panel hold DRIFT omitted (static hold); beat-replace "streak" rides the assemble
xfade. planner + renderer both self-checked; visual acceptance is the QA run.
## D3. identity tracklets + review gates (vikunja 160/136)
- **160 tracklet spine — DONE (code, QA-pending).** the Tier-2 gemma decider (identity phase-2) used to
resolve identity once PER CROP, so one person got adjudicated repeatedly and could get two answers on
two panels. `tracklets.py link_tracklets` (pure, union-find, self-checked) first LINKS the per-panel
siglip shortlists into per-person tracklets over a locality window — hard gender gate + appearance-token
overlap + shared-candidate — then the resolver runs gemma ONCE per tracklet against the UNION of the
members' shortlists (the person's gallery, consensus-ordered) and applies that one character_id to every
crop in the tracklet. fewer gemma calls, one answer per person. ceiling: gemma NONE (cosine over-merged
onto an existing char) still isn't split to a fresh id — needs a 3rd siglip re-embed pass; add if the
NONE-rate is high.
- **136 review gates — DONE (code, QA-pending), off by default (`GATES=1`).** two human sign-off gates in
the pipeline loop: gate `script` pauses BEFORE tts/render, gate `audiovisual` pauses BEFORE assemble.
When enabled and unapproved the job goes `awaiting_review` and the loop returns; `/review/approve`
records the gate (`gates` table) and relaunches `_run_pipeline` (skips completed stages → resumes at the
gated stage). Gate-1: `/review/scripts` shows narration per panel grouped by beat + cheap textual flags
(`review_flags.py`: generic-handle / repeated / empty, self-checked; semantic flags — spoilers, altered
meaning, attribution — left to the human eye, ceiling noted). Gate-2: `/review/audio` serves the per-panel
wav so the reviewer actually listens; `/review/retts` re-synthesizes ONE panel's TTS + re-renders just its
beat clip (delete_clip forces it) and clears the AV gate. web UI (index.html review tab): gate-approve
buttons + status, per-panel audio player, flags badge, re-synthesize button. reuses existing
/review/panels + /review/script + /review/preview. ceiling: no panel-to-cue drag-edit UI (cues persisted
in clips.cues_json, edit endpoint not built); flags are textual only.
## E. reference
generic manga-recap youtube style (single narrator, static panels w/ slow pan/zoom, minimal text,
dramatic pacing) — but with **less generic transitions** (feeds #6). #6 goes beyond the reference.
---
## sequencing (proposed)
1. **C panel curation** — shrinks everything downstream first.
2. **A identity/continuity** — most changes how it *feels* (names + memory).
3. **B voice+style** — small, high impact.
4. **D basics (#1,#5,#7,#8)** — most changes how it *looks*, low risk.
5. **D#6 compositing** — last; highest risk/effort, isolated to render.
## direction schema (decided — enables #8 and #6)
narration (text->TTS) and direction (camera/transition/layers->render) are separate concerns from
the SAME scene graph. VISION picks the effect (it sees the panel + has char bboxes for focus points)
and emits a direction block into the scene_graph JSON; RENDER just executes it. fixed vocab:
- camera.effect: static | zoom_in | zoom_out | pan_left | pan_right | pan_up | pan_down |
dolly_to_subject | orbit | shake | hold (+ from/to [x,y] normalized, easing)
- transition.type: cut | crossfade | fade_black | fade_white | wipe_left | wipe_right | push | dissolve
(+ duration)
- layers: {artifact, motion} (parallax, when the depth renderer is wired)
not built yet; lands before #8 (needs camera) and #6 (needs transition).
## open design points
- roster seed format if/when available.
- B#3: which fixed dots.tts reference voice to pin for the whole video (needs a pick).
---
# v3.1 delta — identity is broken; add human-in-loop review
## evidence (last job, 116-panel Teto X Egen, from manga.db)
- **53 character rows for ~8 actual people.** "Seonho" stored 5× (ids 85ad36b0/811bf0d7/0033ff7f/
bbddf20c/90269899), Egen Girl 2×. ~40 rows have a single identity_assignment (one-time NPCs).
- **root cause: siglip2 over-splits.** same person across panels (pose/expression/B&W line art)
doesn't clear cosine 0.85, so nearly every appearance spawns a new character_id.
- **cascade:** names never propagate (panel 10 names id A, panel 51's Seonho is id C -> `names_by_id`
can't map it -> narration falls back to "the woman with straight dark hair", flips gender to "her").
re-introductions + DB/S3 bloat all follow from the split.
- **concrete bug:** `worker_identity.py` create call hardcodes `"name": None` — vision reads the name
but identity discards it at registration. `db.create_character` is a blind INSERT, zero dedup.
## E1. pipeline decomposition [DECIDED — the vision mega-call splits]
Reason: one gemma4 call doing 4 jobs (detect + dialogue + direction + scene) does each worse, and
there was nowhere to run a real cross-panel merge. Split the jobs; insert a merge stage where the
model actually compares two crops.
**Final stage order** (OCR removed — gemma4 transcribes bubble text image-natively; a `roster`
prepass now runs after crop, see below):
```
fetch → crop → roster → vision → identity → reconcile → dialogue → direct → scene → script → tts → layers → render → assemble
```
- **`vision`** (per panel, image call) — character DETECT only: `local_id`, `bbox`, `appearance`
(hair/clothing/features), `gender` (m|f|unknown), `emotion`, `action`. plus `skip` + `scene`
(location/time). the only stage that must precede identity (identity crops its bboxes).
- **`identity`** (per panel) — siglip assigns `local_id → character_id`. on a collision (siglip
ambiguous OR hair+clothing appearance overlap OR name collision) it records a **merge-candidate
pair** instead of silently minting a twin. still creates provisional ids; reconcile prunes them.
- **`reconcile`** (NEW, once per chapter, after all panels) — the second vision pass. for each
flagged candidate pair, `POST worker_vision /vision/same` with the two `ref_image_uris` crops ->
gemma4 "same character? {yes|no, confidence}". confirmed -> `db.merge_characters(loser->keeper)`:
repoint `identity_assignments`, union `name`/`aliases`/`ref_image_uris`, delete loser row. loser
ref crops are PRESERVED on the keeper as gallery views (not deleted from S3 — deleting them left the
keeper's own uris dangling). cast is CLEAN + named from here on. only flagged pairs get a call.
lives as `run_stage_reconcile` in service.py + `/vision/same` on worker_vision (reuses warm gemma4).
- **`dialogue`** (per panel, image call) — gemma reads bubble text straight from the panel image (no
external OCR): bubble classify (speech/thought/shout/narration/sfx), merge split bubbles, text
cleanup (sentence case, fix names/proper nouns, sfx literal), speaker attribution
(`speaker=local_id` + `confidence`), entities. prompt gets the CLEAN
named cast present in the panel as context -> better attribution ("Seonho (m) vs Haeseon (f)").
- **`direct`** (per panel, off scene understanding) — `camera.effect` (+`to`) and `transition` from
the fixed direction vocab (unchanged schema, section D/direction-schema). runs after identity so it
can weight who's present. may glance at the image or work from the scene graph — cheap.
new workers/endpoints: `worker_vision` becomes detect + `/vision/same`; new `worker_dialogue.py` and
`worker_direct.py` (or `/dialogue` + `/direct` endpoints — same warm gemma4, decide at build like
reconcile). db: `STAGES` gains `reconcile`,`dialogue`,`direct`; `run_stage_*` for each; session_proxy
call_* wrappers. the old omni-`vision` prompt/schema is partitioned across the four prompts.
## F. identity overhaul [rides on E1]
Goal: 53 rows -> ~12; named recurring cast stays one id; NPCs never persist.
- **F1 merge = the `reconcile` stage** (see E1). the fix for the hardcoded `"name": None` bug: vision/
identity now carry `name`+`gender`+`appearance` into `create_character` so provisional rows are
named/typed; reconcile does the actual dedup with the model comparing crops (not a blind string
match). `db.merge_characters` + candidate-flagging in identity.
- **F2 gender field.** add `gender TEXT` to characters schema. `vision` emits `gender`. worker_script
`_name_map` feeds it to the model ("Seonho (he)") for correct pronouns — kills "her bold
personality" flips even when the name resolves.
- **F3 confirm-before-persist.** hold unnamed provisional embeddings in a per-session cache; only
write a DB row + S3 crop/npy when an embedding is seen >=2x OR is named. the ~40 one-timers never
touch the bucket. session_id already flows into identity. (complements reconcile: reconcile merges
the twins that WERE created; F3 stops most from being created at all.)
- **F4 centroid + re-rank matching.** store multiple refs per char, match against the accumulated
centroid not one arbitrary embedding; break siglip ties by name/gender agreement. ceiling: perfect
auto-ID on B&W is not achievable — reconcile+F4 get ~80%, the UI (G) closes the rest. don't
over-invest.
- **DECIDED: start clean.** no migration script. wipe manga.db + the minio `manga` bucket before the
first v3.1 run (destructive ops step, run at build time). fixed logic applies to fresh data.
## G. review / edit UI (replaces the auto CoT-strip idea, item C-bleed)
Rationale: the human reviews the final video and knows exactly what's wrong and where. Give them a
synced editor instead of chasing every failure through the pipeline. ~80% of infra already exists in
the orchestrator: `static/index.html` dashboard, `/job/video` (streams mp4), `/db/tables` + `/db/table`,
`/stage/run` + `/stage/clear` (re-render), `set_character_name`.
- **timestamp -> panel is trivial:** clips concat in `panel_order`, so panel start offset = cumsum of
prior panels' `audio.duration`. store the offset at assemble time (or compute on the fly) as an
offset table the page binary-searches on `<video>` timeupdate.
- **view:** `<video>` + side panel that, for the current panel, shows editable: script text, assigned
characters (name / gender / appearance), camera.effect, transition.
- **edits -> small endpoints** (most exist): add `set_gender`, `merge_characters(ids[]->keep)`,
`set_script(panel_id,text)`, `set_camera(panel_id,effect)`. **merge_characters is where the Seonho
mess gets fixed in seconds** — and the merge is ground truth that feeds F.
- **re-render:** "re-render from stage X" button = existing `/stage/clear` + `/stage/run` on the
affected panels/clips. effect preview = dropdown of `_motion` vocab + `XFADE` map, re-render one clip.
- **lazy ceiling:** no DAW timeline, no non-destructive edit graph. "video + synced editable form +
re-render button" over existing data/endpoints. ~1 day. highest-leverage addition: turns every
future quality bug into a 30-second manual fix.
## v3.1 sequencing
1. **E1 decomposition** — split vision into `vision`/`dialogue`/`direct`, add `reconcile` +
`direct` + `dialogue` to STAGES with stage runners. Foundation; F1 lives inside it.
- 1a. `vision` -> detect-only (+ `gender`, F2). 1b. `dialogue` stage. 1c. `direct` stage.
- 1d. `identity` candidate-flagging + `db.merge_characters`. 1e. `reconcile` stage + `/vision/same`.
2. **F3** (confirm-before-persist) — kills NPC bloat.
3. **G** (review UI) — builds on the cleaner DB; closes the residual reconcile/F4 can't.
4. **F4** (centroid/re-rank) — only if G shows auto-ID still too noisy to be worth hand-fixing.
Build order note: each of 1a–1e is independently deployable/testable (workpc workers reload via
tmux; orchestrator needs a container rebuild). Do a full clean re-run after 1e before starting G.
## D#6 revisited — camera-traversal composite [DONE 2026-07-14]
the vstack composite was the wrong look; target = camera moving across panels in their REAL page
positions (motion-comic page traversal, partial-neighbor reveals). geometry already exists: panels
store `bbox` + `page_index` + `panel_order`, source page is retained. composite feeds a constant 9:16
crop-window that HOLDS on each member then quick-eases (0.5s) to the next (`traverse_cmd`,
`_pan_expr` in worker_render.py). page uri is reconstructed from `page_key(...)` (deterministic — no
schema change). vstack kept as the fallback when bbox/page_uri are missing.
orchestrator TODO: `run_stage_render` composite branch must pass each member's `bbox` + one
`page_uri` into `call_render_composite`.
## E2 — chapter roster / context pre-pass [SPEC — not built]
**problem.** every stage sees one panel at a time, so nobody has a chapter-level view. symptoms in
the wild: the MC is "the man in the yellow shirt" at panel 7 but named at 10 (name learned too late
to backfill cleanly); unnamed extras get re-described every panel; bubble-only panels (p005) have no
cast to attribute a line to; the narrator's tone/naming drifts. `_chapter_context` today is just the
DB registry accumulated *so far* — partial and late.
**idea.** one cheap gemma4 pass up front that reads the pages (text image-natively) and produces a
chapter view that grounds the per-panel stages. it does NOT replace vision/identity — it's coarse
context only, threaded back as NAME hints (never identity evidence).
**stage.** new `roster` stage, placed **after `crop`, before `vision`**. names surface in captions/
dialogue on the page images themselves (e.g. "Lim Seonho, Designer at Everyday"). one artifact per
chapter, stored on `chapters.roster_json`, resumable (skip if present), non-blocking on failure.
**method (implemented).** one multi-image `/roster` call over an even spread of ~6 chapter pages
(`ROSTER_SAMPLE_PAGES`) -> `{premise, characters:[{name,aliases,gender,species,description}]}`.
threaded into: vision detect (`known_characters` hints, registry wins), dialogue (off-panel cast for
bubble-only panels), script (`_chapter_context` premise). ceiling: sampled pages; widen the sample if
recall bites.
**output (stored per chapter, e.g. a `chapter_roster` blob):**
```
{"premise": "3-5 sentence chapter setup, present tense, no spoilers ahead",
"roster": [{"name":"Lim Seonho","aliases":["Seonho"],"gender":"m",
"appearance":"short brown hair, glasses, yellow shirt",
"role":"designer at Everyday; the MC"}]}
```
**consumers (all already have the hook, just feed them the roster instead of the thin registry):**
- `vision` (detect): pass roster as `known_characters` so detections carry a provisional name.
- `identity`/`reconcile`: seed name labels — a face gets its roster name on FIRST sighting, killing
"named at 10, described at 7". embeddings stay authoritative for who-is-who; roster only supplies
the label when appearance matches.
- `dialogue`: roster = the known cast for off-panel attribution (the p005 case now has names/handles
to point at instead of "".)
- `script`: seed `brief` from `premise` at panel 1 (no cold start); roster names -> `names_by_id` +
`introduced` from the start (stable handles, no re-description).
**caveats.** roster names are a COARSE read → HINTS, not ground truth: apply a roster name only when
a detected/identified character's appearance matches a roster entry; never override an embedding
match. one extra pass per chapter (latency/cost) — acceptable, it's once per chapter, warm session.
start_workers.sh
Executable
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# start_workers.sh — dev launcher. session manager first, then the stateless workers,
# each a uvicorn process in its own tmux window. production uses the systemd units in systemd/.
set -euo pipefail
cd "$(dirname "$0")"
APP_DIR="$PWD"
VENV="$APP_DIR/.venv/bin/activate"
# MIOpen (ROCm): persist tuned kernels + FAST find so identity/tts don't re-auto-tune the GPU
# on every start (that thrashed the CPU and stuttered Hyprland). FIND_ENFORCE forces exhaustive
# search even when cached -> keep it unset.
mkdir -p "$HOME/.config/miopen"
MIOPEN_ENV="export MIOPEN_USER_DB_PATH=$HOME/.config/miopen MIOPEN_SYSTEM_DB_PATH=$HOME/.config/miopen MIOPEN_FIND_MODE=2 && unset MIOPEN_FIND_ENFORCE"
SESSION="manga-workers"
# name:module:port (session_manager guards the GPU; workers wait for it to bind)
WORKERS=(
"session:session_manager:8095"
"crop:worker_crop:8000"
"vision:worker_vision:8002"
"identity:worker_identity:8003"
"scene:worker_scene:8004"
"script:worker_script:8005"
"tts:worker_tts:8006"
"layers:worker_layers:8007"
"render:worker_render:8008"
)
tmux kill-session -t "$SESSION" 2>/dev/null || true
tmux new-session -d -s "$SESSION" -n placeholder
for spec in "${WORKERS[@]}"; do
name="${spec%%:*}"; rest="${spec#*:}"; mod="${rest%%:*}"; port="${rest##*:}"
tmux new-window -t "$SESSION" -n "$name"
tmux send-keys -t "$SESSION:$name" \
"$MIOPEN_ENV && source $VENV && python -m uvicorn ${mod}:app --app-dir $APP_DIR --host 0.0.0.0 --port ${port}" C-m
if [ "$name" = "session" ]; then
# workers assume the session manager is up; wait for :8095 before launching the rest
until curl -sf http://127.0.0.1:8095/session/active >/dev/null 2>&1; do sleep 0.3; done
fi
done
tmux kill-window -t "$SESSION:placeholder" 2>/dev/null || true
echo "started ${#WORKERS[@]} processes in tmux session '$SESSION' (attach: tmux attach -t $SESSION)"
systemd/install.sh
Executable
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# systemd/install.sh — generate + install one system service per workpc process.
# run with sudo. units run as User=kami off the workpc venv. session manager is ordered first.
set -euo pipefail
APP_DIR="$(cd "$(dirname "$0")/.." && pwd)"
PY="$APP_DIR/.venv/bin/python"
DEST=/etc/systemd/system
# name module port (session_manager guards the GPU; workers want it up first)
WORKERS=(
"session session_manager 8095"
"crop worker_crop 8000"
"vision worker_vision 8002"
"identity worker_identity 8003"
"scene worker_scene 8004"
"script worker_script 8005"
"tts worker_tts 8006"
"layers worker_layers 8007"
"render worker_render 8008"
)
for spec in "${WORKERS[@]}"; do
read -r name mod port <<<"$spec"
wants=""
[ "$name" != "session" ] && wants="Wants=manga-session.service
After=manga-session.service"
unit="manga-${name}.service"
cat > "$DEST/$unit" <<EOF
[Unit]
Description=manga workpc — $name
$wants
[Service]
User=kami
WorkingDirectory=$APP_DIR
Environment=MIOPEN_USER_DB_PATH=/home/kami/.config/miopen MIOPEN_SYSTEM_DB_PATH=/home/kami/.config/miopen MIOPEN_FIND_MODE=2
ExecStart=$PY -m uvicorn ${mod}:app --host 0.0.0.0 --port ${port}
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
echo "wrote $DEST/$unit"
done
systemctl daemon-reload
echo "installed. enable all: systemctl enable --now manga-{session,crop,vision,identity,scene,script,tts,layers,render}"
test_vision_parse.py
+47
View File
@@ -0,0 +1,47 @@
"""Regression for task 182: a malformed/truncated/thought-wrapped model reply is a technical failure
that must trigger a repair retry and, if still unparseable, RAISE (so the orchestrator retries the
panel) — never a silent skip. Run: ./.venv/bin/python test_vision_parse.py"""
import worker_vision as wv
def test_extract_handles_thought_wrapped():
raw = "<|channel>thought I should answer<channel|>{\"skip\": false, \"characters\": []}"
assert wv._extract_json(raw) == {"skip": False, "characters": []}
def test_extract_rejects_malformed_and_truncated():
for bad in ("sorry, no json here", '{"skip": false, "characters": [', "", "```json\n{oops"):
try:
wv._extract_json(bad)
assert False, f"should have raised on: {bad!r}"
except (ValueError, ValueError):
pass
def test_repair_retry_recovers():
calls = []
good = '{"skip": false, "characters": [{"local_id": "person_1"}]}'
def fake(content, temperature=0.2, max_tokens=768):
calls.append(content)
return "garbled not-json" if len(calls) == 1 else good
wv.call_gemma4 = fake
result = wv.call_gemma4_json([{"type": "text", "text": "p"}])
assert result["characters"][0]["local_id"] == "person_1"
assert len(calls) == 2, "must make exactly one repair retry"
def test_raises_when_repair_also_fails():
wv.call_gemma4 = lambda content, temperature=0.2, max_tokens=768: "still broken {"
try:
wv.call_gemma4_json([{"type": "text", "text": "p"}])
assert False, "must raise so the orchestrator records a stage error and retries"
except ValueError:
pass
if __name__ == "__main__":
test_extract_handles_thought_wrapped()
test_extract_rejects_malformed_and_truncated()
test_repair_retry_recovers()
test_raises_when_repair_also_fails()
print("ok")
transport.py
+216
View File
@@ -0,0 +1,216 @@
# transport.py — minio artifact client, shared by all workers.
# workers are stateless: pull inputs by uri to local disk, push outputs back, return uris.
# no sqlite here (see plan phase 2b option A: homesrv orchestrator owns state).
# uri form: "s3://<bucket>/<key...>" or bare "<bucket>/<key...>". first path segment = bucket.
import os
import json
import time
import logging
from minio import Minio
from minio.error import S3Error
from starlette.responses import Response
_client = None
def _summarize(body: bytes, limit=6) -> str:
"""compact one-line view of a json body for observability: uri inputs/outputs (basename, or
key×N for lists) + short scalars; big lists/dicts (embeddings, panel arrays) shown as key[N].
this is what makes the log answer 'what did the stage get / return / where did it go'."""
try:
obj = json.loads(body)
except Exception:
return "-"
if not isinstance(obj, dict):
return f"[{len(obj)}]" if isinstance(obj, list) else "-"
parts = []
for k, v in obj.items():
if k == "panel_id":
continue
if isinstance(v, str) and (k.endswith("uri") or k.endswith("url")):
parts.append(f"{k}={v.rsplit('/', 1)[-1]}")
elif isinstance(v, list) and v and isinstance(v[0], str) and (k.endswith("uris") or k.endswith("urls")):
parts.append(f"{k}×{len(v)}")
elif isinstance(v, (int, float, bool)):
parts.append(f"{k}={v}")
elif isinstance(v, str):
parts.append(f"{k}={v[:24]!r}" if len(v) > 24 else f"{k}={v!r}")
elif isinstance(v, (list, dict)):
parts.append(f"{k}[{len(v)}]")
if len(parts) >= limit:
parts.append("…")
break
return " ".join(parts) or "{}"
def install_logging(app, name: str):
"""one log line per request for any worker:
`name METHOD /path panel=<id> in:{...} -> out:{...} 123ms 200` (or `... ERR: <exc>` on failure).
call once right after app = FastAPI(). in/out summarize the json bodies (uris + short scalars,
big arrays as key[N]); unparseable/empty bodies show `-`."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
force=False,
)
log = logging.getLogger(name)
@app.middleware("http")
async def _log(request, call_next):
body = await request.body() # buffered; downstream re-reads from the cached bytes
panel = "-"
if body:
try:
panel = str(json.loads(body).get("panel_id") or "-")
except Exception:
pass
req_in = _summarize(body) if body else "-"
t0 = time.perf_counter()
try:
resp = await call_next(request)
except Exception as e:
ms = int((time.perf_counter() - t0) * 1000)
log.error("%s %s panel=%s in:{%s} %dms ERR: %r",
request.method, request.url.path, panel, req_in, ms, e)
raise
# drain the response stream so we can summarize outputs, then hand back an identical response.
out = "-"
chunks = [c async for c in resp.body_iterator]
raw = b"".join(chunks)
if resp.headers.get("content-type", "").startswith("application/json"):
out = _summarize(raw)
resp = Response(content=raw, status_code=resp.status_code,
headers=dict(resp.headers), media_type=resp.media_type)
ms = int((time.perf_counter() - t0) * 1000)
lvl = log.error if resp.status_code >= 500 else log.info
lvl("%s %s panel=%s in:{%s} -> out:{%s} %dms %d",
request.method, request.url.path, panel, req_in, out, ms, resp.status_code)
return resp
def _mc():
global _client
if _client is None:
_client = Minio(
os.environ.get("MINIO_ENDPOINT", "192.168.1.104:9000"), # homesrv
access_key=os.environ.get("MINIO_ACCESS_KEY", "admin"),
secret_key=os.environ.get("MINIO_SECRET_KEY", "godforgiveus"),
secure=False,
)
return _client
def _split(uri: str):
"""(bucket, key) from an s3-style or bare uri."""
u = uri[5:] if uri.startswith("s3://") else uri
bucket, _, key = u.partition("/")
if not bucket or not key:
raise ValueError(f"bad uri: {uri!r}")
return bucket, key
_known_buckets = set()
def _ensure_bucket(c, bucket):
# ponytail: process-local cache, buckets are never deleted at runtime
if bucket in _known_buckets:
return
if not c.bucket_exists(bucket):
c.make_bucket(bucket)
_known_buckets.add(bucket)
def put(local_path: str, uri: str, client=None) -> str:
"""upload local file to minio at uri, return the s3:// uri. raises on failure."""
c = client or _mc()
bucket, key = _split(uri)
_ensure_bucket(c, bucket)
c.fput_object(bucket, key, local_path)
return f"s3://{bucket}/{key}"
def get(uri: str, local_path: str, client=None) -> str:
"""download uri to local_path, return local_path. workers pull inputs to /dev/shm."""
c = client or _mc()
bucket, key = _split(uri)
os.makedirs(os.path.dirname(local_path) or ".", exist_ok=True)
c.fget_object(bucket, key, local_path)
return local_path
def exists(uri: str, client=None) -> bool:
"""HEAD check for worker-level skip-if-exists (orchestrator is the real skip authority)."""
c = client or _mc()
bucket, key = _split(uri)
try:
c.stat_object(bucket, key)
return True
except S3Error:
return False
def put_bytes(data: bytes, uri: str, client=None) -> str:
"""for generated artifacts that never touch disk (embeddings, json)."""
import io
c = client or _mc()
bucket, key = _split(uri)
_ensure_bucket(c, bucket)
c.put_object(bucket, key, io.BytesIO(data), length=len(data))
return f"s3://{bucket}/{key}"
if __name__ == "__main__":
# self-check: fake in-memory minio, round-trip a file + bytes, exists true/false.
import tempfile
class _Fake:
def __init__(self):
self.store = {}
def bucket_exists(self, b):
return any(k[0] == b for k in self.store)
def make_bucket(self, b):
pass
def fput_object(self, b, k, path):
with open(path, "rb") as f:
self.store[(b, k)] = f.read()
def fget_object(self, b, k, path):
with open(path, "wb") as f:
f.write(self.store[(b, k)])
def put_object(self, b, k, stream, length):
self.store[(b, k)] = stream.read()
def stat_object(self, b, k):
if (b, k) not in self.store:
raise S3Error("NoSuchKey", "missing", "", "", "", None)
return True
fake = _Fake()
assert _split("s3://manga/a/b.png") == ("manga", "a/b.png")
assert _split("manga/a/b.png") == ("manga", "a/b.png")
src = tempfile.NamedTemporaryFile(delete=False)
src.write(b"hello panel"); src.close()
uri = put(src.name, "s3://manga/x/p001.png", client=fake)
assert uri == "s3://manga/x/p001.png", uri
assert exists(uri, client=fake)
assert not exists("s3://manga/x/nope.png", client=fake)
dst = src.name + ".out"
get(uri, dst, client=fake)
assert open(dst, "rb").read() == b"hello panel"
put_bytes(b"\x01\x02", "s3://manga/x/e.npy", client=fake)
assert exists("s3://manga/x/e.npy", client=fake)
os.remove(src.name); os.remove(dst)
# _summarize: uris -> basename, uri lists -> key×N, scalars kept, big arrays -> key[N], panel_id dropped
s = _summarize(json.dumps({
"panel_id": "p1", "audio_uri": "s3://manga/x/cl.wav",
"panel_uris": ["s3://a/1.png", "s3://a/2.png"], "active": 2, "rtl": True,
"weights": [0.1, 0.2, 0.7], "narration_text": "a very long line of narration here indeed",
}).encode())
assert "audio_uri=cl.wav" in s and "panel_uris×2" in s and "active=2" in s, s
assert "weights[3]" in s and "panel_id" not in s, s
assert _summarize(b"clip_uri", ) == "-" # non-json
assert _summarize(json.dumps({"clip_uri": "s3://m/c/clips/p1.mp4", "duration": 4.1}).encode()) \
== "clip_uri=p1.mp4 duration=4.1"
print("transport self-check ok")
worker_crop.py
+359
View File
@@ -0,0 +1,359 @@
# worker_crop.py — stage 2 panel detector. FastAPI :8000. cpu/opencv, no GPU, no session.
# one page in -> N panel crops (reading order) uploaded to minio, uris + bboxes returned.
# routes by aspect: framed pages -> kumiko; tall webtoon strips -> whitespace slicer.
import sys, os, uuid, logging
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import cv2
import numpy as np
import transport
try: # CPU-EP ONNX detectors; absent -> merge pass is a no-op
import face_detect, bubble_detect
except Exception:
face_detect = bubble_detect = None
log = logging.getLogger("crop")
sys.path.insert(0, os.path.expanduser("~/Programs/kumiko"))
app = FastAPI()
transport.install_logging(app, "crop")
SHM = "/dev/shm"
WEBTOON_RATIO = 2.0 # ponytail: manga ~1.4, webtoons 2.5-10+; tune if a tall page misroutes
class CropInput(BaseModel):
page_uri: str
manga_id: str
chapter_id: str
page_index: int = 0
job_id: str = ""
rtl: bool = True # manga reads right-to-left; set false for western comics/webtoons
class WebtoonInput(BaseModel):
# webtoon scrapers deliver one episode as arbitrary fixed-height tiles cut mid-panel.
# restitch the whole chapter into one strip, THEN slice — per-tile slicing severs panels.
page_uris: list # all tiles of the chapter, in order
manga_id: str
chapter_id: str
job_id: str = ""
def slice_webtoon(img, bg_thresh=235, min_gap=20, min_seg=64, max_seg=2500, blank_frac=0.995):
"""Cut a vertical strip on blank row-bands. Returns (crop, bbox[x,y,w,h]) top-to-bottom.
Defaults tuned on real webtoon strips: 235/20 finds true gutters over full-color art;
max_seg=2500 (~one phone screen) caps full-bleed art that has no internal gutter.
186: a row is 'blank' when >=blank_frac of its pixels are bright — NOT every pixel (min>=thresh),
because one dark speck/border pixel/stray letter in a true gutter used to defeat the whole cut."""
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) if img.ndim == 3 else img
row_blank = (gray >= bg_thresh).mean(axis=1) >= blank_frac
h, w = gray.shape[:2]
cuts, i = [0], 0
while i < h:
if row_blank[i]:
j = i
while j < h and row_blank[j]:
j += 1
if j - i >= min_gap:
cuts.append((i + j) // 2)
i = j
else:
i += 1
cuts.append(h)
out = []
for a, b in zip(cuts, cuts[1:]):
if b - a < min_seg:
continue
n = -(-(b - a) // max_seg)
step = (b - a) // n
for s in range(n):
y0 = a + s * step
y1 = b if s == n - 1 else a + (s + 1) * step
out.append((img[y0:y1], [0, y0, w, y1 - y0]))
return out or [(img, [0, 0, w, h])]
def _has_face(img):
try:
return bool(face_detect.detect_faces(img))
except Exception:
return True # detector broke -> assume a face so we do NOT merge (conservative)
def _has_text(img):
try:
return bool(bubble_detect.detect_text_regions(img))
except Exception:
return False
def _merge_plan(faces, texts):
"""Pure grouping decision (testable without a real face). faces/texts: per-fragment booleans,
top-to-bottom. A 'stranded caption' = has text but no face; its line has no face to attribute to.
Attach it to an adjacent face fragment — preferring the face ABOVE (the speaker usually appears,
then their caption follows), else the face below. Scenery (no text) and face fragments stay solo.
Returns a list of index-groups covering 0..n-1 in order; each group with >1 index gets vstacked."""
n = len(faces)
groups, i = [], 0
while i < n:
if faces[i]: # a face swallows the run of stranded captions below it
g = [i]
while i + 1 < n and texts[i + 1] and not faces[i + 1]:
i += 1
g.append(i)
groups.append(g)
elif texts[i] and not faces[i] and i + 1 < n and faces[i + 1]: # no face above -> ride the one below
groups.append([i, i + 1])
i += 1
else:
groups.append([i])
i += 1
return groups
def merge_faceless_captions(crops):
"""Post-slice pass (webtoon only): fold a stranded caption fragment into the vertically nearest
face-bearing fragment so the windowed vision/dialogue call sees the caption next to a real face,
instead of on a faceless panel where attribution has nothing to anchor to. This ONLY combines
images — it never assigns a speaker, so it can't misbind a monologue to the nearer face; gemma
still decides from the window's flow. Bonus: fewer panels -> fits gemma's window-count limit.
ponytail: runs face+text ONNX on every fragment (CPU); fine for batch, cache the models if a
huge chapter makes crop the bottleneck."""
if not (face_detect and bubble_detect) or len(crops) < 2:
return crops
faces = [_has_face(c) for c, _ in crops]
texts = [_has_text(c) for c, _ in crops]
plan = _merge_plan(faces, texts)
if len(plan) == len(crops):
return crops # nothing stranded -> untouched
out = []
for g in plan:
if len(g) == 1:
out.append(crops[g[0]])
continue
merged = np.vstack([crops[k][0] for k in g])
x, y, w, _ = crops[g[0]][1]
out.append((merged, [x, y, w, merged.shape[0]]))
return out
def context_fragment_links(crops):
"""Return non-destructive caption↔face links keyed by source-fragment index.
Both fragments remain independent panels with their original bboxes; the dialogue stage may use
the linked image as context, while the orchestrator remains free to accept/reject the linkage.
"""
links = {i: [] for i in range(len(crops))}
if not (face_detect and bubble_detect) or len(crops) < 2:
return links
faces = [_has_face(c) for c, _ in crops]
texts = [_has_text(c) for c, _ in crops]
for group in _merge_plan(faces, texts):
face_idxs = [i for i in group if faces[i]]
if not face_idxs:
continue
anchor = face_idxs[0]
for i in group:
if i == anchor or not (texts[i] and not faces[i]):
continue
links[i].append({"fragment_index": anchor, "bbox": crops[anchor][1],
"link_reason": "adjacent_face_context"})
links[anchor].append({"fragment_index": i, "bbox": crops[i][1],
"link_reason": "adjacent_text_context"})
return links
def reading_order(panels, rtl=True):
"""186: deterministic manga/comic reading order, replacing reliance on kumiko's neighbour-based
comparator (which emitted this RTL title left-to-right on irregular/overlapping layouts).
Group panels into horizontal row-bands by vertical overlap, order bands top-to-bottom, then within
a band order by x — right-to-left for rtl manga, left-to-right otherwise.
panels: [(crop, [x,y,w,h])]. ponytail: greedy vertical banding; a deeply interleaved splash
collage can still misband — upgrade to a full topological pass if a fixture proves it needed."""
if not panels:
return panels
bands = [] # each: {"y0","y1","items"}
for it in sorted(panels, key=lambda p: p[1][1]): # seed bands top-down
x, y, w, h = it[1]
for band in bands:
ov = max(0, min(y + h, band["y1"]) - max(y, band["y0"]))
if ov >= 0.5 * min(h, band["y1"] - band["y0"]): # substantial vertical overlap = same row
band["items"].append(it)
band["y0"], band["y1"] = min(band["y0"], y), max(band["y1"], y + h)
break
else:
bands.append({"y0": y, "y1": y + h, "items": [it]})
bands.sort(key=lambda b: b["y0"])
out = []
for band in bands:
band["items"].sort(key=lambda p: p[1][0], reverse=rtl) # x; rtl -> rightmost first
out.extend(band["items"])
return out
def kumiko_panels(path, rtl=True):
"""Framed-page panels via kumiko contour detection, then a DETERMINISTIC reading-order sort
(186 — do not trust kumiko's implicit comparator for RTL/irregular layouts)."""
from kumikolib import Kumiko
k = Kumiko({"rtl": rtl})
k.parse_image(path)
page = k.page_list[-1]
full = page.img
panels = [(full[p.y:p.b, p.x:p.r], [p.x, p.y, p.r - p.x, p.b - p.y]) for p in page.panels]
return reading_order(panels, rtl)
def flag_overlaps(panels, page_index=0):
"""Warn on panel pairs whose bboxes overlap by >50% of the smaller panel's area —
a sign of ambiguous splitting/ordering that silently corrupts narration sequence.
Returns the list of ambiguous (i, j) index pairs (also for the self-check)."""
def _ov(a, b):
ax, ay, aw, ah = a; bx, by, bw, bh = b
ix = max(0, min(ax + aw, bx + bw) - max(ax, bx))
iy = max(0, min(ay + ah, by + bh) - max(ay, by))
inter = ix * iy
return inter / max(1, min(aw * ah, bw * bh))
ambiguous = []
for i in range(len(panels)):
for j in range(i + 1, len(panels)):
if _ov(panels[i][1], panels[j][1]) > 0.5:
ambiguous.append((i, j))
if ambiguous:
log.warning("page=%d ambiguous panel overlap, order may be wrong: pairs=%s",
page_index, ambiguous)
return ambiguous
def restitch(paths):
"""Stack chapter tiles into one vertical strip. Tiles share width (scraper output)."""
imgs = [cv2.imread(p) for p in paths]
if any(i is None for i in imgs):
raise HTTPException(400, "a webtoon tile was not readable")
w = min(i.shape[1] for i in imgs)
imgs = [i if i.shape[1] == w else cv2.resize(i, (w, int(i.shape[0] * w / i.shape[1]))) for i in imgs]
return np.vstack(imgs)
@app.post("/crop/webtoon")
async def crop_webtoon(data: WebtoonInput):
tag = uuid.uuid4().hex[:8]
locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)]
strip = restitch(locals_)
crops = slice_webtoon(strip)
context_links = context_fragment_links(crops)
panels = []
for idx, (crop_img, bbox) in enumerate(crops):
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png"
# slicing is deterministic, so on a resume the same idx -> same key; skip re-upload.
if not transport.exists(uri):
out = f"{SHM}/wt_{tag}_p{idx:03d}.png"
cv2.imwrite(out, crop_img)
transport.put(out, uri)
os.remove(out)
panels.append({"panel_index": idx, "uri": uri, "bbox": bbox,
"context_fragments": context_links[idx]})
for p in locals_:
os.remove(p)
return {"page_index": 0, "panels": panels}
@app.post("/crop")
async def crop(data: CropInput):
local = transport.get(data.page_uri, f"{SHM}/crop_{uuid.uuid4().hex[:8]}.png")
img = cv2.imread(local)
if img is None:
raise HTTPException(400, f"page not readable: {data.page_uri}")
h, w = img.shape[:2]
webtoon = h / w >= WEBTOON_RATIO
crops = slice_webtoon(img) if webtoon else kumiko_panels(local, data.rtl)
context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))}
ambiguous = flag_overlaps(crops, data.page_index)
panels = []
for idx, (crop_img, bbox) in enumerate(crops):
out = f"{SHM}/pg{data.page_index:03d}_p{idx:02d}.png"
cv2.imwrite(out, crop_img)
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/pg{data.page_index:03d}_p{idx:02d}.png"
transport.put(out, uri)
os.remove(out)
panels.append({"panel_index": idx, "uri": uri, "bbox": bbox,
"context_fragments": context_links[idx]})
os.remove(local)
# 186: surface anomalies so review can see a suspect page instead of it silently completing.
warnings = [f"ambiguous overlap {p}" for p in ambiguous]
if not panels:
warnings.append("no panels detected")
return {"page_index": data.page_index, "panels": panels, "warnings": warnings}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: synthetic 2-band strip -> 2 panels top-to-bottom with sane bboxes.
strip = np.full((300, 100, 3), 255, np.uint8)
strip[50:100] = 0
strip[200:250] = 0
segs = slice_webtoon(strip)
assert len(segs) == 2, len(segs)
assert segs[0][1][1] < segs[1][1][1], "reading order top-to-bottom"
assert all(len(b) == 4 for _, b in segs)
# max_seg cap: a 6000px gutterless block must split into ceil(6000/2500)=3 beats
solid = np.zeros((6000, 100, 3), np.uint8)
capped = slice_webtoon(solid)
assert len(capped) == 3, len(capped)
# a true gutter with one dark speck must still cut (186: fraction, not min-pixel).
speckled = np.full((300, 100, 3), 255, np.uint8)
speckled[50:100] = 0; speckled[200:250] = 0
speckled[75, 40] = 0 # stray dark pixel inside the top band — used to defeat min()>=thresh
assert len(slice_webtoon(speckled)) == 2, "gutter detection must tolerate a speck"
# faceless-caption merge plan (pure grouping). F=face T=text per fragment, top-to-bottom.
assert _merge_plan([True, False], [False, True]) == [[0, 1]] # caption below a face -> absorbed up
assert _merge_plan([False, True], [True, False]) == [[0, 1]] # caption above a face -> absorbed down
assert _merge_plan([False], [True]) == [[0]] # lone caption, no face -> left alone
assert _merge_plan([False], [False]) == [[0]] # scenery (no text) -> left alone
assert _merge_plan([True, False, False], [False, True, True]) == [[0, 1, 2]] # face swallows caption run
assert _merge_plan([True, True], [False, False]) == [[0], [1]] # two faces -> untouched
# vstack path: a real merge stacks images and reports the combined height
a = (np.zeros((30, 10, 3), np.uint8), [0, 0, 10, 30])
b = (np.zeros((20, 10, 3), np.uint8), [0, 40, 10, 20])
import types
_fd, _bd = face_detect, bubble_detect
face_detect = types.SimpleNamespace(detect_faces=lambda im: [1] if im.shape[0] == 30 else [])
bubble_detect = types.SimpleNamespace(detect_text_regions=lambda im: [1] if im.shape[0] == 20 else [])
merged = merge_faceless_captions([a, b])
assert len(merged) == 1 and merged[0][0].shape[0] == 50, merged[0][0].shape
linked = context_fragment_links([a, b])
assert linked[0][0]["fragment_index"] == 1 and linked[1][0]["fragment_index"] == 0
assert linked[0][0]["bbox"] == b[1] and linked[1][0]["bbox"] == a[1]
face_detect, bubble_detect = _fd, _bd
# overlap flag: disjoint panels are clean; a >50% overlapping pair is flagged ambiguous.
clean = [(None, [0, 0, 10, 10]), (None, [20, 0, 10, 10])]
assert flag_overlaps(clean) == []
overlap = [(None, [0, 0, 10, 10]), (None, [2, 2, 10, 10])] # ~64% of the smaller area
assert flag_overlaps(overlap) == [(0, 1)]
# 186 reading order — bbox = [x,y,w,h]. ids track source panels so we can assert final order.
def ro_ids(boxes, rtl):
tagged = [(i, b) for i, b in enumerate(boxes)]
return [i for i, _ in reading_order(tagged, rtl)]
# two panels on one row: RTL reads the right one (x=60) first, LTR the left (x=0) first.
two = [[0, 0, 50, 100], [60, 0, 50, 100]]
assert ro_ids(two, rtl=True) == [1, 0], "RTL: rightmost first"
assert ro_ids(two, rtl=False) == [0, 1], "LTR: leftmost first"
# 2x2 grid: top row R->L then bottom row R->L. sources: 0=TL 1=TR 2=BL 3=BR.
grid = [[0, 0, 50, 50], [60, 0, 50, 50], [0, 60, 50, 50], [60, 60, 50, 50]]
assert ro_ids(grid, rtl=True) == [1, 0, 3, 2], ro_ids(grid, rtl=True)
# staggered rows (slightly offset y) still band by vertical overlap, not exact y.
stag = [[0, 0, 50, 100], [60, 5, 50, 100], [0, 200, 100, 80]]
assert ro_ids(stag, rtl=True) == [1, 0, 2], ro_ids(stag, rtl=True)
# splash (single full panel) -> unchanged.
assert ro_ids([[0, 0, 200, 300]], rtl=True) == [0]
print("worker_crop self-check ok")
worker_identity.py
+322
View File
@@ -0,0 +1,322 @@
# worker_identity.py — stage 5 character identity. FastAPI :8003.
# no model of its own conceptually, but siglip2 loads in-process here (transformers, rocm torch)
# after the orchestrator has opened a siglip2 session (the mutex guarantees it's the only GPU
# resident). known characters + their reference embeddings come from the homesrv orchestrator.
import os, uuid, json
from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np
import requests
import transport
app = FastAPI()
transport.install_logging(app, "identity")
SHM = "/dev/shm"
ORCH = os.environ.get("ORCHESTRATOR_URL", "http://192.168.1.104:9090")
DEFAULT_THRESHOLD = 0.85
AMBIGUOUS_MARGIN = 0.05 # two known chars within this of each other -> flag low-confidence
_siglip = None # (model, processor), lazy-loaded
# F3 confirm-before-persist: unnamed provisional characters are held here (in memory, keyed by
# session_id = one chapter) and only written to DB/S3 once seen >=2x. seen-once NPCs are never
# persisted -> no DB row, no bucket crop. named chars skip the cache and persist immediately.
# entry: {emb, crop(np), count, name, gender, appearance, occ:[(panel_id,local_id,conf)], cid}
_pending: dict[str, list[dict]] = {}
PENDING_PROMOTE_AT = 2
def _load_siglip():
global _siglip
if _siglip is None:
import torch
from transformers import AutoModel, AutoProcessor
name = "google/siglip2-so400m-patch16-384"
model = AutoModel.from_pretrained(name).to("cuda").eval()
_siglip = (model, AutoProcessor.from_pretrained(name), torch)
return _siglip
def embed_crop(img) -> np.ndarray:
"""siglip2 image embedding of a BGR/RGB crop (numpy HxWx3), L2-normalized."""
model, proc, torch = _load_siglip()
inputs = proc(images=img, return_tensors="pt").to("cuda")
with torch.no_grad():
out = model.get_image_features(**inputs)
# transformers returns a ModelOutput here (not a bare tensor): use the attention-pooled
# image embedding. fall back to mean-pooling patch tokens if a build lacks a pooler head.
feat = getattr(out, "pooler_output", None)
if feat is None:
feat = getattr(out, "last_hidden_state", out)
if hasattr(feat, "dim") and feat.dim() == 3:
feat = feat.mean(dim=1)
feat = feat.detach().cpu().numpy().reshape(-1)
return feat / (np.linalg.norm(feat) + 1e-8)
def cosine(a: np.ndarray, b: np.ndarray) -> float:
return float(np.dot(a, b) / ((np.linalg.norm(a) * np.linalg.norm(b)) + 1e-8))
def gender_ok(a: str, b: str) -> bool:
"""two people can be the same only if their DECIDED genders agree. unknown on either side is a
pass (don't over-block). the single biggest siglip cross-match error is a male crop scoring >0.85
against a female character (shared art style + panel context); this hard-blocks it."""
a, b = (a or "").strip().lower(), (b or "").strip().lower()
return not (a in ("m", "f") and b in ("m", "f") and a != b)
def match(emb: np.ndarray, known: list, threshold: float):
"""known: [{"character_id","embedding"(np)}]. returns (character_id|None, confidence, ambiguous)."""
if not known:
return None, 0.0, False
scored = sorted(((cosine(emb, k["embedding"]), k["character_id"]) for k in known), reverse=True)
best_conf, best_id = scored[0]
ambiguous = len(scored) > 1 and (best_conf - scored[1][0]) < AMBIGUOUS_MARGIN
if best_conf >= threshold:
return best_id, best_conf, ambiguous
return None, best_conf, ambiguous
def shortlist(emb: np.ndarray, known: list, k: int = 5, gender: str = None) -> list:
"""TIER-2 evidence: the top-k gender-gated known characters by cosine, best first. Cosine is now a
SHORTLISTER, not the decider — gemma /vision/resolve picks from this list. Each item carries the
full row (name/gender/description) so the resolver can build a text character-sheet. pure."""
cands = [(cosine(emb, c["embedding"]), c) for c in known if gender_ok(gender, c.get("gender"))]
cands.sort(key=lambda t: t[0], reverse=True)
return [{**c, "cosine": round(s, 3)} for s, c in cands[:k]]
def _crop_bbox(img, bbox):
# vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention).
x1, y1, x2, y2 = bbox
return img[y1:y2, x1:x2]
def _save_npy(emb: np.ndarray, uri: str):
"""upload an embedding in .npy format (np.load reads it back on the known-char side)."""
tmp = f"{SHM}/emb_{uuid.uuid4().hex[:8]}.npy"
np.save(tmp, emb.astype(np.float32))
transport.put(tmp, uri)
os.remove(tmp)
def _pending_match(pend: list, emb, threshold: float, gender: str = None):
"""index of the pending entry this embedding belongs to, or None (a new provisional). candidates
of a conflicting decided gender are excluded. character_id=i keeps the returned index original. pure."""
cands = [{"character_id": i, "embedding": e["emb"]}
for i, e in enumerate(pend) if gender_ok(gender, e.get("gender"))]
idx, conf, _ = match(emb, cands, threshold)
return idx, conf
def _persist_char(manga_id, panel_id, local_id, crop, emb, name, gender, appearance) -> str:
"""upload crop + embedding to S3 and register the row via the orchestrator; return its id."""
import cv2
key = f"{manga_id}/characters/_new/{panel_id}_{local_id}"
ref_img_uri, emb_uri = f"s3://manga/{key}.png", f"s3://manga/{key}.npy"
ref_png = f"{SHM}/crop_{uuid.uuid4().hex[:8]}.png"
cv2.imwrite(ref_png, crop)
transport.put(ref_png, ref_img_uri)
os.remove(ref_png)
_save_npy(emb, emb_uri)
resp = requests.post(f"{ORCH}/characters/create", json={
"manga_id": manga_id, "name": (name or "").strip() or None,
"gender": gender or "unknown", "description": appearance or {},
"ref_image_uri": ref_img_uri, "embedding_uri": emb_uri,
}, timeout=30).json()
_known_cache.pop(manga_id, None)
return resp["character_id"]
_known_cache: dict = {} # manga_id -> known list; invalidated in _persist_char
def _load_known(manga_id: str) -> list:
"""orchestrator /characters/known -> roster rows with their embeddings loaded. A dangling
embedding_uri (row persisted but .npy never landed / bucket wiped) is skipped, not fatal —
every panel loads the full roster up front and one bad ref must not 500 the panel.
Cached per manga_id for the run; _persist_char drops the entry when the roster changes."""
if manga_id in _known_cache:
return _known_cache[manga_id]
rows = requests.get(f"{ORCH}/characters/known", params={"manga_id": manga_id}, timeout=30).json()
known = []
for c in rows:
if not c.get("embedding_uri"):
continue
try:
ref = transport.get(c["embedding_uri"], f"{SHM}/ref_{uuid.uuid4().hex[:8]}.npy")
except Exception as e:
print(f"[identity] skip {c['character_id']}: bad embedding_uri {c['embedding_uri']}: {e}", flush=True)
continue
refs = c.get("ref_image_uris") or []
if isinstance(refs, str):
try:
refs = json.loads(refs)
except (ValueError, TypeError):
refs = []
known.append({"character_id": c["character_id"], "embedding": np.load(ref),
"gender": c.get("gender"), "name": c.get("name"),
"species": c.get("species"), "description": c.get("description"),
"reference_image_uris": refs})
os.remove(ref)
_known_cache[manga_id] = known
return known
class IdentityInput(BaseModel):
panel_uri: str
panel_id: str = ""
vision_characters: list = []
manga_id: str = ""
session_id: str = ""
k: int = 5 # shortlist width for gemma's tracklet decider (204: merged from /identity/candidates)
@app.post("/identity/resolve")
async def resolve(data: IdentityInput):
"""204: does the cosine assignment/persist pass AND builds gemma's shortlist in one crop/embed pass
(was two separate endpoints, each re-cropping + re-embedding + reloading the roster per character).
Cosine still makes the provisional assignment; the orchestrator's gemma tracklet phase (/vision/resolve)
can still override it — shortlists are returned for every crop regardless of assignment outcome."""
import cv2
local = transport.get(data.panel_uri, f"{SHM}/ident_{uuid.uuid4().hex[:8]}.png")
img = cv2.imread(local)
# orchestrator /characters/known returns a plain list of character rows (embedding_uri per row).
# ponytail: per-manga threshold isn't exposed by the orchestrator yet -> default; wire a
# manga_config lookup here if tuning per title ever matters.
threshold = DEFAULT_THRESHOLD
known = _load_known(data.manga_id)
# F3: only THIS session's pending cache is relevant (session == chapter); evict any others so a
# long-lived worker doesn't leak past chapters' provisionals.
for sid in [s for s in _pending if s != data.session_id]:
_pending.pop(sid, None)
pend = _pending.setdefault(data.session_id, [])
assignments, backfill, new_chars, shortlists = [], [], [], []
for ch in data.vision_characters:
crop = _crop_bbox(img, ch["bbox"])
if crop.size == 0: # degenerate/out-of-bounds bbox -> nothing to embed, skip
continue
emb = embed_crop(crop)
# shortlist for gemma's decider, from the roster as it stood before this crop's own outcome.
sl = shortlist(emb, known, data.k, ch.get("gender"))
crop_uri = f"s3://manga/{data.manga_id}/characters/_crops/{data.panel_id}_{ch['local_id']}.png"
cp = f"{SHM}/cc_{uuid.uuid4().hex[:8]}.png"; cv2.imwrite(cp, crop)
transport.put(cp, crop_uri); os.remove(cp)
shortlists.append({"local_id": ch["local_id"], "crop_uri": crop_uri,
"candidates": [{"character_id": c["character_id"], "name": c.get("name"),
"gender": c.get("gender"), "species": c.get("species"),
"appearance": c.get("description"), "cosine": c["cosine"],
"reference_image_uris": c.get("reference_image_uris", [])}
for c in sl]})
# gender gate: never match this crop to a known character of the opposite decided gender.
g = ch.get("gender")
cands = [k for k in known if gender_ok(g, k.get("gender"))]
cid, conf, ambiguous = match(emb, cands, threshold)
name = (ch.get("name") or "").strip()
if cid is not None: # matched an already-persisted character
assignments.append({"local_id": ch["local_id"], "character_id": cid,
"confidence": round(conf, 3), "ambiguous": ambiguous})
continue
if name: # named -> persist immediately (not an NPC)
cid = _persist_char(data.manga_id, data.panel_id, ch["local_id"], crop, emb,
name, ch.get("gender"), ch.get("appearance"))
known.append({"character_id": cid, "embedding": emb, "gender": ch.get("gender")})
new_chars.append(cid)
assignments.append({"local_id": ch["local_id"], "character_id": cid,
"confidence": round(conf, 3), "ambiguous": ambiguous})
continue
# unnamed + unknown -> confirm-before-persist. match against this chapter's pending cache.
pcid, pconf = _pending_match(pend, emb, threshold, ch.get("gender"))
if pcid is None: # first sighting: hold, do not persist or assign yet
pend.append({"emb": emb, "crop": crop, "count": 1, "name": name,
"gender": ch.get("gender"), "appearance": ch.get("appearance"),
"occ": [(data.panel_id, ch["local_id"], round(conf, 3))], "cid": None})
continue
e = pend[pcid] # seen before this chapter
e["count"] += 1
e["occ"].append((data.panel_id, ch["local_id"], round(pconf, 3)))
if e["cid"] is None and e["count"] >= PENDING_PROMOTE_AT: # promote -> persist + backfill
e["cid"] = _persist_char(data.manga_id, e["occ"][0][0], e["occ"][0][1], e["crop"],
e["emb"], e["name"], e["gender"], e["appearance"])
known.append({"character_id": e["cid"], "embedding": e["emb"], "gender": e.get("gender")})
new_chars.append(e["cid"])
for (bp, bl, bc) in e["occ"][:-1]: # earlier sightings that were deferred
backfill.append({"panel_id": bp, "local_id": bl,
"character_id": e["cid"], "confidence": bc})
if e["cid"]:
assignments.append({"local_id": ch["local_id"], "character_id": e["cid"],
"confidence": round(pconf, 3), "ambiguous": False})
os.remove(local)
return {"panel_id": data.panel_id, "assignments": assignments,
"backfill": backfill, "new_characters": new_chars, "shortlists": shortlists}
@app.post("/unload")
async def unload():
"""free the resident siglip2 so the session manager can hand the GPU to the next model.
the mutex can't reclaim in-process VRAM -- only the worker holding the model can."""
global _siglip
was = _siglip is not None
if was:
torch = _siglip[2]
_siglip = None
import gc; gc.collect()
torch.cuda.empty_cache()
_pending.clear() # F3: drop any held provisionals with the model
return {"ok": True, "unloaded": was}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: cosine + match decision. deterministic vectors, no real model.
v = lambda *xs: np.array(xs, dtype=np.float32)
a, b = v(1, 0, 0), v(1, 0, 0)
assert abs(cosine(a, b) - 1.0) < 1e-6
known = [{"character_id": "c1", "embedding": v(1, 0, 0)},
{"character_id": "c2", "embedding": v(0, 1, 0)}]
cid, conf, amb = match(v(0.99, 0.01, 0), known, 0.85)
assert cid == "c1" and conf > 0.85 and not amb, (cid, conf, amb)
cid, conf, amb = match(v(0, 0, 1), known, 0.85) # nothing close -> new
assert cid is None
# ambiguous: equidistant-ish between c1 and c2
_, _, amb = match(v(0.71, 0.70, 0), known, 0.5)
assert amb is True
# gender gate: opposite decided genders never match; unknown on either side passes.
assert gender_ok("m", "m") and gender_ok("m", "unknown") and gender_ok("", "f")
assert not gender_ok("m", "f") and not gender_ok("f", "m")
# a male crop must NOT match a female known char even at high cosine (the Choi Haeseon bug).
kn = [{"character_id": "female_char", "embedding": v(1, 0, 0), "gender": "f"}]
cands = [k for k in kn if gender_ok("m", k.get("gender"))]
assert match(v(1, 0, 0), cands, 0.85)[0] is None # gated out -> new male char, not the female id
# pending gate: a female provisional is skipped for a male crop even if embeddings are identical.
pend_g = [{"emb": v(1, 0, 0), "count": 1, "gender": "f"}]
assert _pending_match(pend_g, v(1, 0, 0), 0.85, "m")[0] is None
assert _pending_match(pend_g, v(1, 0, 0), 0.85, "f")[0] == 0
# F3 confirm-before-persist: first sighting is new (held, not persisted); a matching second
# sighting hits the same pending entry -> promotes.
pend = []
i, _ = _pending_match(pend, v(1, 0, 0), 0.85)
assert i is None # nothing pending yet -> new provisional
pend.append({"emb": v(1, 0, 0), "count": 1})
i, _ = _pending_match(pend, v(0.99, 0.02, 0), 0.85)
assert i == 0 # second sighting matches the held entry
i, _ = _pending_match(pend, v(0, 0, 1), 0.85)
assert i is None # unrelated crop -> its own new provisional
# tier-2 shortlist: top-k by cosine, gender-gated, best first; carries the row for the sheet.
kn = [{"character_id": "c1", "embedding": v(1, 0, 0), "gender": "m", "name": "Gojo"},
{"character_id": "c2", "embedding": v(0, 1, 0), "gender": "f", "name": "Choi"},
{"character_id": "c3", "embedding": v(0.9, 0.1, 0), "gender": "m", "name": "Nanami"}]
sl = shortlist(v(1, 0, 0), kn, k=2, gender="m")
assert [c["character_id"] for c in sl] == ["c1", "c3"] # female c2 gated out; c1 beats c3
assert sl[0]["cosine"] >= sl[1]["cosine"] and sl[0]["name"] == "Gojo"
print("worker_identity self-check ok")
worker_layers.py
+99
View File
@@ -0,0 +1,99 @@
# worker_layers.py — stage 9 part 1 depth-layer decomposition. FastAPI :8007.
# wraps the comfyui layered workflow (comfyui runs as its own external process, not session-mgr).
# panel in -> N layer pngs uploaded to minio, layer uris out.
import os, json, time, uuid
from fastapi import FastAPI
from pydantic import BaseModel
import requests
import transport
app = FastAPI()
transport.install_logging(app, "layers")
SHM = "/dev/shm"
COMFYUI_URL = os.environ.get("COMFYUI_URL", "http://127.0.0.1:8188")
LAYERED_WORKFLOW = os.path.join(os.path.dirname(__file__), "legacy", "qwen_layered_workflow.json")
class LayerInput(BaseModel):
panel_uri: str
panel_id: str = ""
num_layers: int = 4
prompt: str = ""
manga_id: str = ""
chapter_id: str = ""
def _comfy_upload(path: str) -> str:
with open(path, "rb") as f:
r = requests.post(f"{COMFYUI_URL}/upload/image",
files={"image": (os.path.basename(path), f)},
data={"overwrite": "true"}, timeout=30)
r.raise_for_status()
return r.json()["name"]
def _comfy_run(panel_path: str, num_layers: int, prompt_text: str) -> list[str]:
"""returns comfyui /view urls for the produced layers."""
workflow = json.load(open(LAYERED_WORKFLOW))
workflow["10"]["inputs"]["image"] = _comfy_upload(panel_path)
workflow["6"]["inputs"]["text"] = prompt_text
workflow["83"]["inputs"]["layers"] = num_layers - 1 # node returns incl. background
pid = requests.post(f"{COMFYUI_URL}/prompt",
json={"prompt": workflow, "client_id": str(uuid.uuid4())},
timeout=30).json()["prompt_id"]
deadline = time.time() + 600
while time.time() < deadline:
h = requests.get(f"{COMFYUI_URL}/history/{pid}", timeout=30).json()
if pid in h:
imgs = h[pid]["outputs"]["9"]["images"]
return [f"{COMFYUI_URL}/view?filename={i['filename']}&subfolder={i['subfolder']}&type={i['type']}"
for i in imgs]
time.sleep(2)
raise TimeoutError(f"comfyui prompt {pid} timed out")
def _comfy_up() -> bool:
try:
return requests.get(f"{COMFYUI_URL}/system_stats", timeout=2).ok
except requests.RequestException:
return False
@app.post("/layers")
async def layers(data: LayerInput):
if not _comfy_up():
return {"layer_uris": [], "skipped": "comfyui down"} # ponytail: skip stage if ComfyUI not running
local = transport.get(data.panel_uri, f"{SHM}/layer_{uuid.uuid4().hex[:8]}.png")
# orchestrator doesn't pass manga/chapter to layers; derive from the panel uri prefix.
parts = data.panel_uri.replace("s3://", "").split("/")
manga_id, chapter_id = parts[1], parts[2]
view_urls = _comfy_run(local, data.num_layers, data.prompt)
layer_uris = []
for idx, url in enumerate(view_urls):
png = f"{SHM}/layer_{uuid.uuid4().hex[:6]}.png"
with open(png, "wb") as f:
f.write(requests.get(url, timeout=60).content)
uri = f"s3://manga/{manga_id}/{chapter_id}/layers/{data.panel_id or 'p'}/{idx}.png"
transport.put(png, uri)
os.remove(png)
layer_uris.append(uri)
os.remove(local)
return {"layer_uris": layer_uris}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: mock comfyui history -> view-url extraction picks the SaveImage node images.
fake_history = {"pid": {"outputs": {"9": {"images": [
{"filename": "a.png", "subfolder": "", "type": "output"},
{"filename": "b.png", "subfolder": "sub", "type": "output"},
]}}}}
imgs = fake_history["pid"]["outputs"]["9"]["images"]
urls = [f"x?filename={i['filename']}&subfolder={i['subfolder']}" for i in imgs]
assert len(urls) == 2 and "a.png" in urls[0]
assert os.path.exists(LAYERED_WORKFLOW), LAYERED_WORKFLOW
print("worker_layers self-check ok")
worker_render.py
+996
View File
@@ -0,0 +1,996 @@
# worker_render.py — stage 9 part 2 rendering. FastAPI :8008.
# per-scene: motion (ken burns, or parallax when layers present) + burned subtitles -> mp4 clip.
# chapter assembly: concat clips with crossfades -> chapter.mp4. all outputs to minio.
# ponytail: ken burns for both cases for now; layer-parallax is the upgrade path when the
# depth renderer is dialed in — the layer_uris are already threaded through the input.
import os, re, uuid, subprocess
from concurrent.futures import ThreadPoolExecutor
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import transport
from collage import plan_layout
app = FastAPI()
transport.install_logging(app, "render")
SHM = "/dev/shm"
W, H = 1080, 1920 # vertical 9:16
PAD_S = float(os.environ.get("PANEL_PAD_S", "0.4")) # #13 trailing silence per panel for pacing
MUSIC_BED = os.environ.get("MUSIC_BED", "") # #13 path/uri of a music track; empty -> no bed
MUSIC_GAIN = os.environ.get("MUSIC_GAIN", "0.18") # bed level before ducking
def _mc_from_uri(uri: str):
"""(manga_id, chapter_id) from s3://<bucket>/<manga_id>/<chapter_id>/...
the orchestrator doesn't pass ids to render/layers, but every input uri encodes them."""
parts = uri.replace("s3://", "").split("/")
return parts[1], parts[2]
def _ts(s):
h = int(s // 3600); m = int(s % 3600 // 60); sec = s % 60
return f"{h}:{m:02d}:{sec:05.2f}"
# 188: readable narration subtitles. Modes: off (burn nothing), minimal (small text, thin outline +
# soft shadow, NO box — least art occlusion), boxed (small text on a restrained ~50% box). Presets carry
# portrait/landscape safe-areas. ASS colour is &HAABBGGRR (alpha 00=opaque..FF=clear).
SUB_MODE = os.environ.get("SUB_MODE", "minimal").lower() # off | minimal | boxed
SUB_PRESETS = {
# (mode, orientation): fontsize, borderstyle(1=outline,3=box), outline/pad, shadow, marginV, side
("minimal", "portrait"): dict(fs=40, bs=1, outline=3, shadow=2, mv=0.055, side=110),
("minimal", "landscape"): dict(fs=32, bs=1, outline=3, shadow=2, mv=0.09, side=260),
("boxed", "portrait"): dict(fs=40, bs=3, outline=6, shadow=0, mv=0.055, side=110),
("boxed", "landscape"): dict(fs=32, bs=3, outline=6, shadow=0, mv=0.09, side=260),
}
def _wrap2(text: str, width: int) -> str:
"""188: keep a cue to at most 2 short lines. Greedy word-wrap to `width`; if it still needs a 3rd
line, truncate the 2nd with an ellipsis so a long cue never grows back into a paragraph block."""
words, lines, cur = (text or "").split(), [], ""
for w in words:
if cur and len(cur) + 1 + len(w) > width:
lines.append(cur); cur = w
if len(lines) == 2:
break
else:
cur = f"{cur} {w}".strip()
if len(lines) < 2:
lines.append(cur)
elif cur or len(words) > sum(len(l.split()) for l in lines):
lines[1] = lines[1].rstrip(".,") + "…" # more words remained than fit two lines
return "\\N".join(l for l in lines if l)
def _ass_multi(events, path, mode=None, orientation="portrait"):
"""Burn caption events. events: [(text, start, end)] or [(text, start, end, align)] where align is an
ASS numpad alignment (2=bottom-center default, 8=top-center to dodge a low subject). mode off -> no
Dialogue lines (nothing burned). Style comes from SUB_PRESETS (188)."""
mode = (mode or SUB_MODE)
if mode == "off":
events = []
p = SUB_PRESETS.get((mode, orientation), SUB_PRESETS[("minimal", "portrait")])
fill = "&H80000000" if p["bs"] == 3 else "&H00000000" # boxed: ~50% box; minimal: opaque outline
fmt = ("Name, Fontname, Fontsize, PrimaryColour, OutlineColour, BackColour, "
"Bold, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV")
mv = int(p["mv"] * (W if orientation == "landscape" else H))
style = ("Def,DejaVu Sans,%d,&H00FFFFFF,%s,&H80000000,0,%d,%d,%d,2,%d,%d,%d"
% (p["fs"], fill, p["bs"], p["outline"], p["shadow"], p["side"], p["side"], mv))
width = 34 if orientation == "portrait" else 46
lines = ""
for ev in events:
t, s, e = ev[0], ev[1], ev[2]
align = ev[3] if len(ev) > 3 else 2
# {\anN} overrides alignment per line so a cue can jump to the top over a low subject.
lines += "Dialogue: 0,%s,%s,Def,{\\an%d}%s\n" % (_ts(s), _ts(e), align, _wrap2(t, width))
with open(path, "w") as f:
f.write(
"[Script Info]\nScriptType: v4.00+\nPlayResX: %d\nPlayResY: %d\nWrapStyle: 0\n\n"
"[V4+ Styles]\nFormat: %s\nStyle: %s\n\n"
"[Events]\nFormat: Layer, Start, End, Style, Text\n%s" % (W, H, fmt, style, lines)
)
def _sub_align(camera: dict) -> int:
"""188: dodge the subject where we can. The director's focus point (camera.to=[x,y], y normalized
top->bottom) is the one subject location we already know — if it sits low in frame, put the caption
at the TOP (an8) so it doesn't cover the face; otherwise keep it bottom (an2).
ponytail: no face/bubble detector yet — upgrade to real bbox avoidance when one is wired in."""
to = (camera or {}).get("to")
if to and len(to) >= 2 and float(to[1]) > 0.6:
return 8
return 2
def _ass(text: str, dur: float, path: str):
_ass_multi([(text, 0.0, dur)], path)
def _audio_dur(path: str) -> float:
"""clip length = narration length. scene_timing arrives empty, so probe the audio."""
r = subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=nk=1:nw=1", path], capture_output=True, text=True)
try:
return float(r.stdout.strip())
except ValueError:
return 0.0
ZMAX, ZPAN = 1.15, 1.18 # ken-burns zoom ceiling; constant zoom that gives pans room to travel
def _motion(camera: dict, frames: int) -> str:
"""#8 content-aware motion: map the vision `camera` block to a zoompan z/x/y expression.
vocab (spec-v3 direction schema): static|hold, zoom_in|zoom_out, pan_left/right/up/down,
dolly_to_subject (uses camera.to=[x,y] normalized), orbit, shake. default = gentle zoom_in
(the old ken-burns look) so panels without direction are unchanged.
x/y reference `zoom` for the live crop-window size; z is driven linearly by `on` (frame index)
over T frames so the move completes across the whole clip regardless of length."""
T = max(1, frames - 1)
eff = (camera or {}).get("effect", "zoom_in")
xc, yc = "iw/2-(iw/zoom/2)", "ih/2-(ih/zoom/2)" # centered crop
mx, my = "(iw-iw/zoom)", "(ih-ih/zoom)" # pan travel margin
if eff in ("static", "hold"):
z, x, y = "1.0", xc, yc
elif eff == "zoom_out":
z, x, y = f"{ZMAX}-{ZMAX-1:.3f}*on/{T}", xc, yc
elif eff == "pan_left":
z, x, y = f"{ZPAN}", f"{mx}*(1-on/{T})", yc
elif eff == "pan_right":
z, x, y = f"{ZPAN}", f"{mx}*on/{T}", yc
elif eff == "pan_up":
z, x, y = f"{ZPAN}", xc, f"{my}*(1-on/{T})"
elif eff == "pan_down":
z, x, y = f"{ZPAN}", xc, f"{my}*on/{T}"
elif eff == "dolly_to_subject":
to = (camera or {}).get("to") or [0.5, 0.35]
tx, ty = min(max(float(to[0]), 0.0), 1.0), min(max(float(to[1]), 0.0), 1.0)
z = f"1+{ZMAX-1:.3f}*on/{T}"
x = f"(iw/2+({tx}*iw-iw/2)*on/{T})-(iw/zoom/2)" # crop center eases toward subject
y = f"(ih/2+({ty}*ih-ih/2)*on/{T})-(ih/zoom/2)"
elif eff == "shake":
z, x, y = "1.06", f"{xc}+8*sin(on*1.5)", f"{yc}+8*cos(on*1.3)"
elif eff == "orbit":
z = "1.12"
x, y = f"{xc}+(iw*0.03)*sin(6.283*on/{T})", f"{yc}+(ih*0.03)*cos(6.283*on/{T})"
else: # zoom_in (default ken burns)
z, x, y = f"1+{ZMAX-1:.3f}*on/{T}", xc, yc
return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps=25"
def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict = None,
pad: float = PAD_S) -> list:
"""ffmpeg: still panel over a blurred fill of itself + content-aware motion, burned subs, 9:16.
#5 blurred bg replaces black bars: one copy scaled to COVER + blurred, the fitted panel on top.
#13 pad seconds of trailing silence (last frame held) give the panel a beat before the next."""
fps = 25
frames = max(1, int((dur + pad) * fps)) # hold the last frame through the pad
# overlay's W/H/w/h are ffmpeg's main/overlay dims -- kept literal (no f-string braces).
fc = (
f"[0:v]split=2[bg][fg];"
f"[bg]scale={W}:{H}:force_original_aspect_ratio=increase,crop={W}:{H},boxblur=20:2[bgb];"
f"[fg]scale={W}:{H}:force_original_aspect_ratio=decrease[fgs];"
f"[bgb][fgs]overlay=(W-w)/2:(H-h)/2,"
f"{_motion(camera, frames)},"
f"ass={ass}[v];"
f"[1:a]apad=pad_dur={pad:.3f}[a]" # trailing silence to match the held frames
)
return ["ffmpeg", "-y", "-loop", "1", "-i", img, "-i", audio,
"-filter_complex", fc, "-map", "[v]", "-map", "[a]",
"-c:v", "libx264", "-tune", "stillimage", "-pix_fmt", "yuv420p",
"-c:a", "aac", "-b:a", "192k", "-shortest", out]
class SceneInput(BaseModel):
panel_uri: str
layer_uris: list = []
audio_uri: str = ""
narration_text: str = ""
panel_id: str = ""
scene_timing: dict = {}
camera: dict = {} # #8 direction block from vision: {"effect": "...", "to": [x,y]}
manga_id: str = ""
chapter_id: str = ""
@app.post("/render/scene")
async def render_scene(data: SceneInput):
tag = uuid.uuid4().hex[:8]
img = transport.get(data.panel_uri, f"{SHM}/rnd_{tag}.png")
audio = transport.get(data.audio_uri, f"{SHM}/rnd_{tag}.wav")
# #1: real duration = narration length; scene_timing is empty in practice, 4.0 last-resort.
dur = (_audio_dur(audio)
or float(data.scene_timing.get("end", 0)) - float(data.scene_timing.get("start", 0))
or 4.0)
ass = f"{SHM}/rnd_{tag}.ass"
_ass(data.narration_text, dur, ass)
out = f"{SHM}/rnd_{tag}.mp4"
subprocess.run(scene_cmd(img, audio, ass, out, dur, data.camera), check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panel_uri)
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
for p in (img, audio, ass, out):
os.remove(p)
return {"clip_uri": uri, "duration": dur}
# #6 multi-panel composite: a group of small panels shares ONE shot (vertical stack). each panel
# keeps its own narration + audio (script/tts stay per-panel); they play in sequence while a moving
# highlight marks the active panel -- that's the "swap the front panel while audio plays" beat.
def _stack_still_cmd(images: list, out: str, rowh: int) -> list:
"""compose N panels fitted into equal vertical rows on one W×H still (black gutters)."""
n = len(images)
cmd = ["ffmpeg", "-y"]
for im in images:
cmd += ["-i", im]
parts = [f"[{i}:v]scale={W}:{rowh}:force_original_aspect_ratio=decrease,"
f"pad={W}:{rowh}:(ow-iw)/2:(oh-ih)/2:color=black[p{i}]" for i in range(n)]
stacked = "".join(f"[p{i}]" for i in range(n))
fc = ";".join(parts) + f";{stacked}vstack=inputs={n},pad={W}:{H}:0:0:color=black[v]"
return cmd + ["-filter_complex", fc, "-map", "[v]", "-frames:v", "1", out]
# D#6 revisited: camera-traversal composite. instead of vstacking cropped panels (black gutters,
# "wrong look"), pan across the REAL page — hold on each member's panel, quick-ease to the next,
# partial neighbours stay visible. groups are always single-page (grouping.py), so one page image.
TRAVERSE_EASE = 0.5 # seconds of quick slide between held panels
WIN_MARGIN = 1.6 # crop window height = tallest member bbox * this (room for neighbour reveal)
def _framed_window(pw: int, ph: int, bboxes: list) -> tuple:
"""one constant 9:16 crop window (px) sized to hold the tallest member with margin, aspect
preserved, clamped to the page. constant size => only x,y animate (crop supports per-frame x/y,
not per-frame w/h)."""
ar = W / H
ch = max((b[3] for b in bboxes), default=ph) * WIN_MARGIN
cw = ch * ar
if cw > pw:
cw, ch = pw, pw / ar
if ch > ph:
ch, cw = ph, ph * ar
return int(cw), int(ch)
def _win_tl(bbox: list, cw: int, ch: int, pw: int, ph: int) -> tuple:
"""top-left of the window centered on a bbox, clamped so it stays inside the page."""
cx, cy = bbox[0] + bbox[2] / 2, bbox[1] + bbox[3] / 2
return (min(max(cx - cw / 2, 0), pw - cw), min(max(cy - ch / 2, 0), ph - ch))
def _pan_expr(vals: list, segs: list, ease: float) -> str:
"""piecewise ffmpeg expr in t: hold vals[i] through beat i, linear-ease to vals[i+1] over the
beat's last `ease`s; final beat just holds. segs=[(start,end)] per beat."""
n = len(vals)
expr = f"{vals[-1]:.1f}"
for i in range(n - 2, -1, -1):
s, e = segs[i]
he = e - ease
a, b = vals[i], vals[i + 1]
beat = f"if(lt(t,{he:.3f}),{a:.1f},({a:.1f}+({b - a:.1f})*(t-{he:.3f})/{ease:.3f}))"
expr = f"if(lt(t,{e:.3f}),{beat},{expr})"
return expr
def traverse_cmd(page: str, audios: list, ass: str, segs: list, bboxes: list,
pw: int, ph: int, out: str, dur: float) -> list:
"""pan a constant 9:16 window across the page: hold on each member, quick-ease to the next."""
ease = min(TRAVERSE_EASE, min((e - s for s, e in segs), default=1.0) / 2)
cw, ch = _framed_window(pw, ph, bboxes)
tls = [_win_tl(b, cw, ch, pw, ph) for b in bboxes]
xexpr = _pan_expr([t[0] for t in tls], segs, ease)
yexpr = _pan_expr([t[1] for t in tls], segs, ease)
n = len(audios)
aconcat = "".join(f"[{i + 1}:a]" for i in range(n)) + f"concat=n={n}:v=0:a=1[a]"
fc = (f"[0:v]crop={cw}:{ch}:x='{xexpr}':y='{yexpr}',scale={W}:{H},setsar=1,"
f"ass={ass}[v];{aconcat}")
cmd = ["ffmpeg", "-y", "-loop", "1", "-i", page]
for a in audios:
cmd += ["-i", a]
return cmd + ["-filter_complex", fc, "-map", "[v]", "-map", "[a]", "-t", f"{dur:.3f}",
"-c:v", "libx264", "-tune", "stillimage", "-pix_fmt", "yuv420p",
"-c:a", "aac", "-b:a", "192k", out]
def composite_cmd(still: str, audios: list, ass: str, segs: list, rowh: int, out: str, dur: float) -> list:
"""loop the stacked still for the whole group, concat each panel's audio in order, burn the
per-segment subtitles, and outline the active row during its narration. segs=[(start,end)]."""
n = len(audios)
hl = "".join(
f"drawbox=x=0:y={i*rowh}:w={W}:h={rowh}:color=yellow@0.85:t=6:enable='between(t,{s:.2f},{e:.2f})',"
for i, (s, e) in enumerate(segs))
cmd = ["ffmpeg", "-y", "-loop", "1", "-i", still]
for a in audios:
cmd += ["-i", a]
aconcat = "".join(f"[{i+1}:a]" for i in range(n)) + f"concat=n={n}:v=0:a=1[a]"
fc = f"[0:v]{hl}ass={ass}[v];{aconcat}"
return cmd + ["-filter_complex", fc, "-map", "[v]", "-map", "[a]", "-t", f"{dur:.3f}",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", out]
class CompositeInput(BaseModel):
# ordered members: [{panel_uri, audio_uri, narration_text, bbox?, page_uri?}]. bbox=[x,y,w,h] on
# the source page + page_uri enable the camera-traversal look; absent -> vstack fallback.
panels: list = []
panel_id: str = "" # leader panel id -> clip key (assemble picks up one clip per group)
def _img_size(path: str) -> tuple:
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
"stream=width,height", "-of", "csv=p=0:s=x", path], capture_output=True, text=True)
w, h = r.stdout.strip().split("x")
return int(w), int(h)
@app.post("/render/composite")
async def render_composite(data: CompositeInput):
if len(data.panels) < 2:
raise HTTPException(400, "composite needs >=2 panels")
tag = uuid.uuid4().hex[:8]
auds = [transport.get(p["audio_uri"], f"{SHM}/cmp_{tag}_{i}.wav") for i, p in enumerate(data.panels)]
durs = [_audio_dur(a) or 3.0 for a in auds]
segs, t = [], 0.0
for d in durs:
segs.append((t, t + d)); t += d
n = len(data.panels)
ass, out = f"{SHM}/cmp_{tag}.ass", f"{SHM}/cmp_{tag}.mp4"
_ass_multi([(data.panels[i].get("narration_text", ""), segs[i][0], segs[i][1]) for i in range(n)], ass)
# camera-traversal path: needs a shared page + a bbox per member (grouping keeps groups single-page)
bboxes = [p.get("bbox") for p in data.panels]
page_uri = data.panels[0].get("page_uri")
if page_uri and all(b and len(b) == 4 for b in bboxes):
page = transport.get(page_uri, f"{SHM}/cmp_{tag}_pg.png")
pw, ph = _img_size(page)
subprocess.run(traverse_cmd(page, auds, ass, segs, bboxes, pw, ph, out, t),
check=True, capture_output=True)
os.remove(page)
else: # fallback: legacy vstack (black gutters) when geometry is unavailable
imgs = [transport.get(p["panel_uri"], f"{SHM}/cmp_{tag}_{i}.png") for i, p in enumerate(data.panels)]
rowh = H // n
still = f"{SHM}/cmp_{tag}_s.png"
subprocess.run(_stack_still_cmd(imgs, still, rowh), check=True, capture_output=True)
subprocess.run(composite_cmd(still, auds, ass, segs, rowh, out, t), check=True, capture_output=True)
for p in imgs + [still]:
os.remove(p)
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
for p in auds + [ass, out]:
os.remove(p)
return {"clip_uri": uri, "duration": t}
# director groups: render a beat's member panels as sequential FULL-FRAME shots (each its own
# scene_cmd clip: blurred bg + ken-burns + its narration), then xfade-chain them into one clip with
# the per-member transition. replaces the vstack/traverse composite look with real transitioned crops.
class GroupInput(BaseModel):
# ordered members: [{panel_uri, audio_uri, narration_text, camera, transition}]
panels: list = []
panel_id: str = "" # leader id -> clip key
@app.post("/render/group")
async def render_group(data: GroupInput):
if len(data.panels) < 2:
raise HTTPException(400, "group needs >=2 panels")
tag = uuid.uuid4().hex[:8]
clips, durs, cleanup = [], [], []
for i, p in enumerate(data.panels):
img = transport.get(p["panel_uri"], f"{SHM}/grp_{tag}_{i}.png")
aud = transport.get(p["audio_uri"], f"{SHM}/grp_{tag}_{i}.wav")
dur = _audio_dur(aud) or 3.0
ass = f"{SHM}/grp_{tag}_{i}.ass"
_ass(p.get("narration_text", ""), dur, ass)
out = f"{SHM}/grp_{tag}_{i}.mp4"
subprocess.run(scene_cmd(img, aud, ass, out, dur, p.get("camera") or {}),
check=True, capture_output=True)
clips.append(out)
durs.append(_audio_dur(out) or (dur + PAD_S)) # full clip length incl. trailing pad
cleanup += [img, aud, ass, out]
# transition INTO member i+1 = member i's transition (transition out of the leaving shot).
trans = [(p.get("transition") or "cut") for p in data.panels[:-1]]
fg, vmap, amap = _xfade_chain(durs, trans)
final = f"{SHM}/grp_{tag}.mp4"
cmd = ["ffmpeg", "-y"]
for c in clips:
cmd += ["-i", c]
cmd += ["-filter_complex", fg, "-map", vmap, "-map", amap,
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", final]
subprocess.run(cmd, check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(final, uri)
total = _audio_dur(final) or sum(durs)
for f in cleanup + [final]:
if os.path.exists(f):
os.remove(f)
return {"clip_uri": uri, "duration": total}
# scene-level narration: ONE narration+audio spans a whole beat, shown over its member panels as a
# ken-burns montage. the beat's single subtitle holds across every image; images split the narration
# duration equally. (per-panel narration is gone at this granularity — the beat is the story unit.)
class BeatInput(BaseModel):
panel_uris: list = [] # ordered member images (>=1)
audio_uri: str = "" # the beat's single narration audio
narration_text: str = ""
cameras: list = [] # per-image camera block; short/empty -> default zoom_in
weights: list = [] # 185: per-image relative screen-time; short/empty/degenerate -> equal split
panel_id: str = "" # leader id -> clip key
BEAT_MIN_PANEL_S = 1.0 # 185: no member panel flashes by faster than this
def _beat_slices(D: float, n: int, weights: list = None) -> list:
"""185: split beat duration D across n member panels by content weight, not evenly.
Each panel gets >= BEAT_MIN_PANEL_S so a low-weight panel never flashes. Degenerate input
(no/short weights, non-positive sum, or D too small for the floors) -> deterministic equal split.
Pure + total: sum(result) == D always."""
if n <= 0:
return []
if not weights or len(weights) < n or D <= n * BEAT_MIN_PANEL_S:
return [D / n] * n
w = [max(0.0, float(x)) for x in weights[:n]]
s = sum(w)
if s <= 0:
return [D / n] * n
floor = n * BEAT_MIN_PANEL_S
free = D - floor # distribute only the time above the floors by weight
return [BEAT_MIN_PANEL_S + free * (wi / s) for wi in w]
def _split_cues(text: str) -> list:
"""187: break the beat's one flowing narration into sentence/phrase cues. Split on sentence-ending
punctuation (kept) and newlines so short exclamations like 'Hold up.' stay their own cue."""
text = (text or "").strip()
if not text:
return []
parts = re.split(r"(?<=[.!?…。！？])\s+|\n+", text)
return [p.strip() for p in parts if p and p.strip()]
def cue_plan(text: str, D: float, slices: list) -> list:
"""187: a timed subtitle plan for one beat. Each cue = one sentence, timed along the SAME [0,D]
timeline as the per-panel slices, so a cue never appears before the panel on screen when it starts
(no future-panel facts leak early). Cue durations are length-weighted with a readable floor (reusing
_beat_slices); each cue is mapped to the member panel whose window contains its start. Pure/testable.
Returns [{text, start, end, panel_index}] in order."""
cues = _split_cues(text)
if not cues or D <= 0:
return []
durs = _beat_slices(D, len(cues), [len(c) for c in cues]) # length-weighted, min-hold, equal fallback
bounds, t = [], 0.0
for s in slices: # panel on-screen windows
bounds.append((t, t + s)); t += s
events, t = [], 0.0
for c, d in zip(cues, durs):
start, end = t, min(t + d, D)
pi = next((k for k, (a, b) in enumerate(bounds) if a <= start < b), max(0, len(bounds) - 1))
# clamp start to the panel's reveal so the caption is never ahead of its image
events.append({"text": c, "start": max(start, bounds[pi][0]) if bounds else start,
"end": end, "panel_index": pi})
t = end
return events
def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list = None,
weights: list = None) -> list:
"""ffmpeg: N images shown as a ken-burns montage under ONE narration audio; the last image holds
through the trailing pad. each image = blurred-fill bg + fitted panel + its camera move; the single
burned subtitle (in `ass`) spans the whole beat. 185: per-image screen-time is content-weighted
(see _beat_slices), equal split when weights are absent. pure -> testable without S3."""
cameras = cameras or []
n, fps = len(imgs), 25
slices = _beat_slices(D, n, weights) # 185: content-weighted, equal-split fallback
# one frame per image (no -loop): zoompan d=frames expands that single frame to exactly `frames`
# output frames = seg seconds. looping instead would feed many frames and zoompan multiplies each.
cmd = ["ffmpeg", "-y"]
for img in imgs:
cmd += ["-i", img]
cmd += ["-i", audio]
parts = []
for i in range(n):
cam = cameras[i] if i < len(cameras) else {}
seg = slices[i] + (PAD_S if i == n - 1 else 0.0)
frames = max(1, int(seg * fps))
parts.append(
f"[{i}:v]split=2[bg{i}][fg{i}];"
f"[bg{i}]scale={W}:{H}:force_original_aspect_ratio=increase,crop={W}:{H},boxblur=20:2[bgb{i}];"
f"[fg{i}]scale={W}:{H}:force_original_aspect_ratio=decrease[fgs{i}];"
f"[bgb{i}][fgs{i}]overlay=(W-w)/2:(H-h)/2,{_motion(cam, frames)}[v{i}]"
)
concat_in = "".join(f"[v{i}]" for i in range(n))
fc = (";".join(parts) + f";{concat_in}concat=n={n}:v=1:a=0[vc];"
f"[vc]ass={ass}[v];[{n}:a]apad=pad_dur={PAD_S:.3f}[a]")
cmd += ["-filter_complex", fc, "-map", "[v]", "-map", "[a]",
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", out]
return cmd
@app.post("/render/beat")
async def render_beat(data: BeatInput):
if not data.panel_uris:
raise HTTPException(400, "beat needs >=1 panel")
tag = uuid.uuid4().hex[:8]
audio = transport.get(data.audio_uri, f"{SHM}/bt_{tag}.wav")
D = _audio_dur(audio) or 4.0
imgs = [transport.get(u, f"{SHM}/bt_{tag}_{i}.png") for i, u in enumerate(data.panel_uris)]
ass = f"{SHM}/bt_{tag}.ass"
# 187: timed sentence cues instead of one beat-long paragraph; same slice timeline as the montage,
# so each line surfaces with its panel. Empty/unsplittable text -> single caption (old behaviour).
slices = _beat_slices(D, len(imgs), data.weights)
cues = cue_plan(data.narration_text, D, slices)
cams = data.cameras or []
if cues:
cues[-1]["end"] = D + PAD_S # hold the closing line through the trailing pad
# 188: each cue dodges its panel's subject (top vs bottom) via the director's focus point.
_ass_multi([(c["text"], c["start"], c["end"],
_sub_align(cams[c["panel_index"]] if c["panel_index"] < len(cams) else {}))
for c in cues], ass)
else:
_ass_multi([(data.narration_text, 0.0, D + PAD_S)], ass)
out = f"{SHM}/bt_{tag}.mp4"
subprocess.run(beat_cmd(imgs, audio, ass, out, D, data.cameras, data.weights),
check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panel_uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
total = _audio_dur(out) or (D + PAD_S)
for f in imgs + [audio, ass, out]:
if os.path.exists(f):
os.remove(f)
# 187: cue plan (text/timing/panel_index per line) returned so the orchestrator can persist it for
# the review UI to edit mapping+timing, and so #183's collage can key reveals off the same cues.
return {"clip_uri": uri, "duration": total, "cues": cues}
# --- 183: animated manga collage ------------------------------------------------------------------
# A beat's member panels laid out as a motion-comic collage instead of a full-frame ken-burns montage:
# a blurred plate of the dominant panel fills the frame, sharp aspect-fit panels rest in a planned
# template (collage.plan_layout), and non-dominant panels slide into place over a brief entrance while
# the dominant one resolves by scale. Holds stay crisp. Behind the pipeline's COLLAGE flag.
# ponytail ceilings (visual polish, QA-tuned against the reference video, no still to check here):
# - transition-only directional/radial MOTION BLUR is not applied (clean slide/scale entrance); the
# beat-to-beat "streak" transition still rides the assemble-stage xfade. Add tblend accumulation
# when a fixture shows the clean slide reads too flat.
# - per-panel slow "drift" during the hold is omitted (static hold); add a gentle zoompan when needed.
class CollageInput(BaseModel):
panel_uris: list = [] # ordered member images (reading order), 1..4
audio_uri: str = ""
narration_text: str = ""
weights: list = [] # 185 timing hints (also picks the dominant panel = argmax)
cameras: list = [] # for 188 subtitle subject-dodge
rtl: bool = True
active: int = -1 # dominant panel index; <0 -> argmax(weights) or 0
panel_id: str = ""
def collage_cmd(imgs, plate_i, rects, entrances, z_order, audio, ass, out, D, trans):
"""ffmpeg graph: blurred plate of imgs[plate_i] + each sharp panel scaled to its resting rect with a
restrained drop shadow, composited back-to-front (z_order), non-dominant panels sliding in from their
entrance offset over `trans` seconds. One narration audio; burned cues in `ass`. Pure -> testable."""
n, T = len(imgs), D + PAD_S
cmd = ["ffmpeg", "-y"]
for img in imgs:
cmd += ["-loop", "1", "-i", img] # still held for the whole clip; -t caps output
cmd += ["-i", audio]
parts = [f"[{plate_i}:v]scale={W}:{H}:force_original_aspect_ratio=increase,"
f"crop={W}:{H},boxblur=24:2,setsar=1[bg]"]
base = "bg"
for k, i in enumerate(z_order): # shadows first, at rest positions (static)
x, y, w, h = (int(round(v)) for v in rects[i])
parts.append(f"[{base}]drawbox=x={x + 7}:y={y + 7}:w={w}:h={h}:color=black@0.35:t=fill[sh{k}]")
base = f"sh{k}"
for i in range(n):
w, h = int(round(rects[i][2])), int(round(rects[i][3]))
parts.append(f"[{i}:v]scale={w}:{h},setsar=1[p{i}]")
cur = base
for k, i in enumerate(z_order):
x, y = int(round(rects[i][0])), int(round(rects[i][1]))
dx, dy = entrances[i]
# ease from (x+dx, y+dy) to (x, y) over `trans`s, then hold. commas safe inside the '...' quotes.
ease = f"max(0,1-t/{trans:.3f})"
parts.append(f"[{cur}][p{i}]overlay=x='{x}+({dx})*{ease}':y='{y}+({dy})*{ease}'[o{k}]")
cur = f"o{k}"
fc = ";".join(parts) + f";[{cur}]ass={ass}[v];[{n}:a]apad=pad_dur={PAD_S:.3f}[a]"
cmd += ["-filter_complex", fc, "-map", "[v]", "-map", "[a]", "-t", f"{T:.3f}",
"-r", "30", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", out]
return cmd
@app.post("/render/collage")
async def render_collage(data: CollageInput):
if not data.panel_uris:
raise HTTPException(400, "collage needs >=1 panel")
uris = data.panel_uris[:4] # planner templates cap at 4 readable panels
n = len(uris)
tag = uuid.uuid4().hex[:8]
audio = transport.get(data.audio_uri, f"{SHM}/cl_{tag}.wav")
D = _audio_dur(audio) or 4.0
imgs = [transport.get(u, f"{SHM}/cl_{tag}_{i}.png") for i, u in enumerate(uris)]
aspects = []
for p in imgs:
w, h = _img_size(p)
aspects.append(w / h if h else 1.0)
active = data.active if 0 <= data.active < n else (
max(range(n), key=lambda i: data.weights[i]) if len(data.weights) >= n else 0)
lay = plan_layout(aspects, data.rtl, active, (W, H))
# 185/187: content-weighted slices + timed cues, same as the montage path.
slices = _beat_slices(D, n, data.weights)
cues = cue_plan(data.narration_text, D, slices)
ass = f"{SHM}/cl_{tag}.ass"
if cues:
cues[-1]["end"] = D + PAD_S
cams = data.cameras or []
_ass_multi([(c["text"], c["start"], c["end"],
_sub_align(cams[c["panel_index"]] if c["panel_index"] < len(cams) else {}))
for c in cues], ass)
else:
_ass_multi([(data.narration_text, 0.0, D + PAD_S)], ass)
out = f"{SHM}/cl_{tag}.mp4"
subprocess.run(collage_cmd(imgs, active, lay["rects"], lay["entrances"], lay["z_order"],
audio, ass, out, D, lay["transition_s"]), check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
total = _audio_dur(out) or (D + PAD_S)
for f in imgs + [audio, ass, out]:
if os.path.exists(f):
os.remove(f)
return {"clip_uri": uri, "duration": total, "cues": cues, "template": lay["template"]}
# #6 non-generic transitions: map the direction schema's transition.type -> (xfade name, seconds).
# "cut" stays a hard cut via the concat fast-path; the rest re-encode through an xfade chain.
XFADE = {
"cut": ("fade", 0.0),
"crossfade": ("fade", 0.5),
"dissolve": ("dissolve", 0.5),
"fade_black": ("fadeblack", 0.6),
"fade_white": ("fadewhite", 0.6),
"wipe_left": ("wipeleft", 0.4),
"wipe_right": ("wiperight", 0.4),
"push": ("slideleft", 0.4),
}
# Chapter assembly used to open every clip in one ffmpeg process. A long chapter therefore created
# one enormous xfade graph: N decoders + N-1 full-frame filter stages, enough to exhaust RAM/VRAM and
# get ffmpeg SIGKILLed by the OOM killer. Assemble a bounded tree instead. Eight inputs keeps enough
# work in each encode to be efficient without letting decoder/filter threads grow with chapter size.
ASSEMBLE_BATCH = max(2, int(os.environ.get("ASSEMBLE_BATCH", "8")))
FFMPEG_THREADS = max(1, int(os.environ.get("FFMPEG_THREADS", "2")))
def _xfade_chain(durs: list, trans: list):
"""build a filter_complex that xfades N clips with per-boundary transitions, keeping audio in
sync via matching acrossfade. trans[i] is the transition OUT of clip i (boundary i->i+1).
returns (filtergraph, video_label, audio_label). offsets accumulate as clips overlap."""
# A clip whose duration probed as 0/unreadable must not poison the chain: with dur=0 the offset
# accumulator would run BACKWARDS (cum += dur - td), swallowing every later clip into a frozen
# overlap near the middle. Floor to a small positive length so the timeline stays monotonic.
durs = [d if (d and d > 0.1) else 0.1 for d in durs]
parts, vlast, alast, cum = [], "[0:v]", "[0:a]", durs[0]
for i in range(1, len(durs)):
name, td = XFADE.get(trans[i - 1] if i - 1 < len(trans) else "cut", XFADE["cut"])
td = max(0.05, min(td, durs[i - 1] - 0.05, durs[i] - 0.05)) # overlap fits in both clips
off = max(cum - td, 0)
parts.append(f"{vlast}[{i}:v]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]")
parts.append(f"{alast}[{i}:a]acrossfade=d={td:.3f}[a{i}]")
vlast, alast, cum = f"[v{i}]", f"[a{i}]", cum + durs[i] - td
return ";".join(parts), vlast, alast
def _assemble_once(inputs: list[str], trans: list[str], out: str):
"""Assemble one bounded batch. `trans[i]` is the transition out of inputs[i]."""
fancy = len(inputs) >= 2 and any(t not in ("", "cut") for t in trans[:len(inputs) - 1])
if fancy:
durs = [_audio_dur(p) for p in inputs]
fg, vmap, amap = _xfade_chain(durs, trans)
cmd = ["ffmpeg", "-y", "-filter_complex_threads", str(FFMPEG_THREADS)]
# Input-side -threads limits each decoder; otherwise ffmpeg may create a decoder thread pool
# for every input in the batch in addition to the filter and libx264 pools.
for p in inputs:
cmd += ["-threads", "1", "-i", p]
cmd += ["-filter_complex", fg, "-map", vmap, "-map", amap,
"-c:v", "libx264", "-threads", str(FFMPEG_THREADS), "-pix_fmt", "yuv420p",
"-c:a", "aac", "-b:a", "192k", out]
subprocess.run(cmd, check=True, capture_output=True)
return
# A cut-only batch is concatenated with the concat FILTER, not -c copy or the concat demuxer.
# The tree feeds xfade intermediates (irregular video timestamps) back in as inputs; the concat
# demuxer + -vsync cfr DROPS video frames to force CFR while audio survives -> the video ends up
# minutes short and freezes on a frame with narration playing on (the "one image, narration behind
# it" bug). The concat filter decodes and re-times every segment, so no frames are dropped. It needs
# N decoders, but the tree already bounds a batch to ASSEMBLE_BATCH inputs, so memory stays capped.
n = len(inputs)
cmd = ["ffmpeg", "-y"]
for p in inputs:
cmd += ["-i", p]
pre = "".join(f"[{i}:v]setsar=1,fps=30[v{i}];" for i in range(n))
fg = pre + "".join(f"[v{i}][{i}:a]" for i in range(n)) + f"concat=n={n}:v=1:a=1[v][a]"
cmd += ["-filter_complex", fg, "-map", "[v]", "-map", "[a]",
"-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p",
"-threads", str(FFMPEG_THREADS), "-c:a", "aac", "-b:a", "192k", out]
subprocess.run(cmd, check=True, capture_output=True)
def _assemble_batched(inputs: list[str], transitions: list[str], out: str, tag: str,
cleanup: list[str]):
"""Bounded tree assembly preserving every original transition.
Each intermediate carries the transition OUT of its final source clip. That becomes the boundary
transition between intermediates in the next round, so batching does not change direction. A
chapter of 100 clips with batch=8 uses at most eight simultaneous inputs instead of 100.
"""
items = [{"path": p,
"out_transition": transitions[i] if i < len(transitions) else "cut",
"temporary": False}
for i, p in enumerate(inputs)]
round_no = 0
while len(items) > 1:
final_round = len(items) <= ASSEMBLE_BATCH
next_items = []
for start in range(0, len(items), ASSEMBLE_BATCH):
group = items[start:start + ASSEMBLE_BATCH]
if len(group) == 1 and not final_round:
next_items.append(group[0])
continue
target = out if final_round else f"{SHM}/asm_{tag}_r{round_no}_{start // ASSEMBLE_BATCH}.mp4"
_assemble_once([x["path"] for x in group], [x["out_transition"] for x in group], target)
if not final_round:
cleanup.append(target)
next_items.append({"path": target, "out_transition": group[-1]["out_transition"],
"temporary": not final_round})
# Intermediates from the previous round are no longer needed. Keep cleanup idempotent: remove
# them here for low /dev/shm usage and from the cleanup list so endpoint cleanup won't retry.
for item in items:
p = item["path"]
if item["temporary"] and os.path.exists(p):
os.remove(p)
if p in cleanup:
cleanup.remove(p)
items = next_items
round_no += 1
class AssembleInput(BaseModel):
clip_uris: list
transitions: list = [] # #6 per-clip transition-out type; empty/all-"cut" -> fast concat
chapter_id: str = ""
manga_id: str = ""
def _music_filter(gain: str) -> str:
"""#13 loop a music bed under the narration, ducked by sidechain compression keyed on the
narration itself, then mix. duration=first ends the mix with the video's audio."""
return (
f"[1:a]volume={gain}[bed];"
"[0:a]asplit=2[nar][key];"
"[bed][key]sidechaincompress=threshold=0.03:ratio=8:attack=20:release=400[duck];"
"[nar][duck]amix=inputs=2:duration=first:dropout_transition=0[a]"
)
def _add_music_bed(video: str, tag: str, cleanup: list) -> str:
"""mix MUSIC_BED under the assembled chapter. no-op (returns input) when unset or on failure."""
if not MUSIC_BED:
return video
music = MUSIC_BED
if MUSIC_BED.startswith("s3://"):
music = transport.get(MUSIC_BED, f"{SHM}/bed_{tag}{os.path.splitext(MUSIC_BED)[1] or '.mp3'}")
cleanup.append(music)
mixed = f"{SHM}/chapter_{tag}_mus.mp4"
try:
subprocess.run(
["ffmpeg", "-y", "-i", video, "-stream_loop", "-1", "-i", music,
"-filter_complex", _music_filter(MUSIC_GAIN), "-map", "0:v", "-map", "[a]",
"-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest", mixed],
check=True, capture_output=True)
except (subprocess.CalledProcessError, FileNotFoundError) as e:
print(f"[render] music bed skipped: {e}", flush=True)
if os.path.exists(mixed):
os.remove(mixed)
return video
cleanup.append(mixed)
return mixed
@app.post("/render/assemble")
async def assemble(data: AssembleInput):
if not data.clip_uris:
raise HTTPException(400, "no clips to assemble")
tag = uuid.uuid4().hex[:8]
with ThreadPoolExecutor(max_workers=8) as pool:
locals_ = list(pool.map(
lambda iu: transport.get(iu[1], f"{SHM}/asm_{tag}_{iu[0]}.mp4"),
enumerate(data.clip_uris),
))
out = f"{SHM}/chapter_{tag}.mp4"
cleanup = list(locals_) + [out]
fancy = len(locals_) >= 2 and any(t not in ("", "cut") for t in data.transitions)
if fancy:
_assemble_batched(locals_, data.transitions, out, tag, cleanup)
else:
# all hard cuts: stream-copy concat (no re-encode) -- unchanged fast path.
listfile = f"{SHM}/asm_{tag}.txt"; cleanup.append(listfile)
with open(listfile, "w") as f:
f.write("".join(f"file '{p}'\n" for p in locals_))
subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", listfile, "-c", "copy", out],
check=True, capture_output=True)
out = _add_music_bed(out, tag, cleanup)
manga_id, chapter_id = _mc_from_uri(data.clip_uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/chapter.mp4"
transport.put(out, uri)
for p in cleanup:
os.remove(p)
return {"video_uri": uri} # orchestrator save_video reads video_uri
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: ass file well-formed; ffmpeg produces a clip from a synthetic image + silence.
import shutil
_ass("Hello\nworld", 2.0, f"{SHM}/t.ass")
txt = open(f"{SHM}/t.ass").read()
assert "Hello world" in txt and "Dialogue:" in txt # short cue -> one line
# 188: subtitle redesign. wrap to <=2 lines, off-mode burns nothing, subject-dodge picks top/bottom.
assert _wrap2("one two three four five six seven eight", 12).count("\\N") == 1 # exactly two lines
long = _wrap2(" ".join(["word"] * 40), 12)
assert long.count("\\N") == 1 and long.endswith("…") # 3rd line truncated, never a paragraph
_ass_multi([("hidden", 0.0, 2.0)], f"{SHM}/off.ass", mode="off")
assert "Dialogue:" not in open(f"{SHM}/off.ass").read() # off -> nothing burned
_ass_multi([("boxed", 0.0, 2.0)], f"{SHM}/bx.ass", mode="boxed")
assert ",3," in open(f"{SHM}/bx.ass").read() # BorderStyle 3 = box
assert _sub_align({"to": [0.5, 0.8]}) == 8 and _sub_align({"to": [0.5, 0.3]}) == 2 # dodge low subject
_ass_multi([("top", 0.0, 2.0, 8)], f"{SHM}/an.ass")
assert "{\\an8}top" in open(f"{SHM}/an.ass").read() # per-cue alignment override
for f in (f"{SHM}/off.ass", f"{SHM}/bx.ass", f"{SHM}/an.ass"):
os.remove(f)
# #8 motion: each effect yields a distinct, well-formed zoompan expr; dolly aims at its target.
assert "z='1.0'" in _motion({"effect": "static"}, 50) # truly still
assert "on/49" in _motion({"effect": "pan_left"}, 50) # travels over the clip
assert "0.8*iw" in _motion({"effect": "dolly_to_subject", "to": [0.8, 0.2]}, 50)
assert _motion({}, 50) == _motion({"effect": "zoom_in"}, 50) # default == ken burns
# 185: content-weighted beat slices. always sum to D; floor honored; degenerate -> equal split.
eq = _beat_slices(12.0, 3)
assert eq == [4.0, 4.0, 4.0], eq # no weights -> equal
w = _beat_slices(12.0, 3, [3, 1, 1]) # heavier panel gets more time
assert abs(sum(w) - 12.0) < 1e-6 and w[0] > w[1] and min(w) >= BEAT_MIN_PANEL_S, w
assert all(abs(s - 0.8) < 1e-9 for s in _beat_slices(2.4, 3, [3, 1, 1])) # below floors -> equal
assert _beat_slices(12.0, 3, [0, 0, 0]) == [4.0, 4.0, 4.0] # zero-sum -> equal split
assert _beat_slices(12.0, 3, [5]) == [4.0, 4.0, 4.0] # short weights -> equal split
# 187: timed cue plan. "Hold up." must be its own cue and must not start before its panel's window.
sl = _beat_slices(9.0, 3, [1, 1, 1]) # three 3.0s panel windows
cues = cue_plan("She sees the reflection. Those are psycho eyes. Hold up.", 9.0, sl)
assert [c["text"] for c in cues] == ["She sees the reflection.", "Those are psycho eyes.", "Hold up."]
assert cues[0]["start"] == 0.0 # first cue opens the beat
for c in cues: # every cue starts within/after its panel
assert c["start"] >= c["panel_index"] * 3.0 - 1e-6, c
assert cues[-1]["panel_index"] == 2, cues[-1] # last line maps to the last panel
assert cues == cue_plan("She sees the reflection. Those are psycho eyes. Hold up.", 9.0, sl) # deterministic
assert cue_plan("", 9.0, sl) == [] # empty narration -> no cues (paragraph fallback)
# Long chapter assembly is bounded, and the transition out of a batch's last source clip is used
# to join that batch to the next one. Mock the encoder so this remains a cheap pure orchestration test.
_real_once = _assemble_once
_calls = []
try:
globals()["_assemble_once"] = lambda ins, trs, out: _calls.append((list(ins), list(trs), out))
_n = ASSEMBLE_BATCH * 2 + 1
_batch_trans = [f"t{i}" for i in range(_n)]
_assemble_batched([f"in{i}.mp4" for i in range(_n)], _batch_trans,
"/tmp/final.mp4", "batchck", [])
assert all(len(ins) <= ASSEMBLE_BATCH for ins, _, _ in _calls), _calls
assert _calls[-1][1][0] == _batch_trans[ASSEMBLE_BATCH - 1], _calls[-1]
assert _calls[-1][1][1] == _batch_trans[ASSEMBLE_BATCH * 2 - 1], _calls[-1]
finally:
globals()["_assemble_once"] = _real_once
if shutil.which("ffmpeg"):
aud = f"{SHM}/t.wav"; out = f"{SHM}/t.mp4"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "anullsrc=r=16000:cl=mono", "-t", "1", aud],
check=True, capture_output=True)
assert abs(_audio_dur(aud) - 1.0) < 0.1, _audio_dur(aud) # #1: duration from audio
# both a wide-short and a tall webtoon-style panel must render (tall used to break pad)
cams = ({}, {"effect": "pan_right"}, {"effect": "dolly_to_subject", "to": [0.7, 0.3]})
for size, cam in zip(("200x60", "900x2200", "800x1200"), cams):
img = f"{SHM}/t.png"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", f"color=c=black:s={size}",
"-frames:v", "1", img], check=True, capture_output=True)
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", out, 1.0, cam), check=True, capture_output=True)
assert os.path.getsize(out) > 0, size
os.remove(img)
# #13 padding: a 1.0s narration clip runs ~1.0+PAD_S with the trailing silence held.
img = f"{SHM}/tp.png"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "color=c=black:s=400x600",
"-frames:v", "1", img], check=True, capture_output=True)
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", out, 1.0, pad=0.4), check=True, capture_output=True)
assert abs(_audio_dur(out) - 1.4) < 0.15, _audio_dur(out)
# #13 music bed: mixing a generated tone under the clip keeps duration and produces output.
bed = f"{SHM}/bed.wav"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "sine=frequency=220:duration=0.5", bed],
check=True, capture_output=True)
_saved = globals()["MUSIC_BED"]; globals()["MUSIC_BED"] = bed
cl = []
mixed = _add_music_bed(out, "selfck", cl)
assert mixed != out and os.path.exists(mixed) and abs(_audio_dur(mixed) - 1.4) < 0.2, _audio_dur(mixed)
globals()["MUSIC_BED"] = _saved
for p in (img, bed, *cl):
if os.path.exists(p): os.remove(p)
# #6 transitions: two real clips xfade into one chapter; graph offsets/labels well-formed.
fg, vmap, amap = _xfade_chain([1.0, 1.0], ["fade_white"])
assert "xfade=transition=fadewhite" in fg and vmap == "[v1]" and amap == "[a1]"
img = f"{SHM}/t.png"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "color=c=black:s=400x600",
"-frames:v", "1", img], check=True, capture_output=True)
c0, c1 = f"{SHM}/c0.mp4", f"{SHM}/c1.mp4"
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c0, 1.0), check=True, capture_output=True)
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c1, 1.0), check=True, capture_output=True)
subprocess.run(["ffmpeg", "-y", "-i", c0, "-i", c1, "-filter_complex", fg,
"-map", vmap, "-map", amap, "-c:v", "libx264", "-pix_fmt", "yuv420p",
"-c:a", "aac", out], check=True, capture_output=True)
assert os.path.getsize(out) > 0
for p in (img, c0, c1):
os.remove(p)
# #6 composite: 2 panels + 2 audios -> one stacked clip; duration = sum, subs timed per row.
a2 = f"{SHM}/a2.wav"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "anullsrc=r=16000:cl=mono", "-t", "1.5", a2],
check=True, capture_output=True)
i0, i1 = f"{SHM}/i0.png", f"{SHM}/i1.png"
for im, sz in ((i0, "300x200"), (i1, "500x300")):
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", f"color=c=gray:s={sz}",
"-frames:v", "1", im], check=True, capture_output=True)
segs = [(0.0, 1.0), (1.0, 2.5)]
still = f"{SHM}/stk.png"
subprocess.run(_stack_still_cmd([i0, i1], still, H // 2), check=True, capture_output=True)
_ass_multi([("first beat", *segs[0]), ("second beat", *segs[1])], f"{SHM}/t.ass")
subprocess.run(composite_cmd(still, [aud, a2], f"{SHM}/t.ass", segs, H // 2, out, 2.5),
check=True, capture_output=True)
assert abs(_audio_dur(out) - 2.5) < 0.2, _audio_dur(out) # duration = sum of both narrations
for p in (a2, i0, i1, still):
os.remove(p)
# scene-level narration: 3 images under ONE 1.5s narration -> clip = D + PAD_S, single subtitle.
b0, b1, b2 = f"{SHM}/b0.png", f"{SHM}/b1.png", f"{SHM}/b2.png"
for im, sz in ((b0, "300x200"), (b1, "800x1200"), (b2, "500x900")):
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", f"color=c=navy:s={sz}",
"-frames:v", "1", im], check=True, capture_output=True)
bnar = f"{SHM}/bnar.wav"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "anullsrc=r=16000:cl=mono", "-t", "1.5", bnar],
check=True, capture_output=True)
_ass("one flowing beat narration", 1.5 + PAD_S, f"{SHM}/t.ass")
cams = ({}, {"effect": "pan_right"}, {"effect": "dolly_to_subject", "to": [0.6, 0.4]})
subprocess.run(beat_cmd([b0, b1, b2], bnar, f"{SHM}/t.ass", out, 1.5, list(cams)),
check=True, capture_output=True)
assert abs(_audio_dur(out) - (1.5 + PAD_S)) < 0.2, _audio_dur(out) # one narration spans N images
# 183: collage of the same 3 panels -> one clip of D+PAD, laid out by plan_layout (dominant=2).
aspects = [_img_size(p)[0] / _img_size(p)[1] for p in (b0, b1, b2)]
lay = plan_layout(aspects, rtl=True, active=2, frame=(W, H))
_ass_multi([("collage cue", 0.0, 1.5 + PAD_S)], f"{SHM}/t.ass")
subprocess.run(collage_cmd([b0, b1, b2], 2, lay["rects"], lay["entrances"], lay["z_order"],
bnar, f"{SHM}/t.ass", out, 1.5, lay["transition_s"]),
check=True, capture_output=True)
assert abs(_audio_dur(out) - (1.5 + PAD_S)) < 0.2, _audio_dur(out) # collage clip = beat length
assert os.path.getsize(out) > 0
for p in (b0, b1, b2, bnar):
os.remove(p)
os.remove(aud); os.remove(out)
print("worker_render self-check ok (ffmpeg ran)")
else:
print("worker_render self-check ok (ffmpeg absent, ass-only)")
os.remove(f"{SHM}/t.ass")
worker_scene.py
+170
View File
@@ -0,0 +1,170 @@
# worker_scene.py — stage 6 scene graph. FastAPI :8004. no GPU, no session (cheap join).
# joins vision + identity into a named-character scene graph. speaker attribution comes straight from
# the vision/dialogue model (tail direction + turn-taking); the old OCR nearest-bbox heuristic is gone.
import json, logging
from fastapi import FastAPI
from pydantic import BaseModel
import transport
log = logging.getLogger("scene")
def _describe(appearance) -> str:
"""A: a stable descriptive phrase for an unnamed character, from its FIRST-seen appearance
(frozen in the registry). "the one with black hair and glasses" — reused across panels so the
label never drifts; empty string if appearance is bare (script then falls back to Person X)."""
if isinstance(appearance, str):
try:
appearance = json.loads(appearance or "{}")
except ValueError:
appearance = {}
if not isinstance(appearance, dict):
return ""
# animals/creatures get named as their species, not by clothes ("the black cat", not "the one
# with a gold bell"). spec A: story-state tracks "props/entities seen (the cat)".
species = (appearance.get("species") or "human").strip().lower()
if species not in ("", "human", "person", "man", "woman"):
color = (appearance.get("hair") or "").strip()
feats = [str(f).strip() for f in (appearance.get("features") or []) if str(f).strip()]
color = color or (feats[0] if feats else "")
return f"the {color} {species}".replace(" ", " ").strip()
hair = (appearance.get("hair") or "").strip()
clothing = (appearance.get("clothing") or "").strip()
feats = [str(f).strip() for f in (appearance.get("features") or []) if str(f).strip()]
bits = []
if hair:
# vision sometimes already includes the word "hair" ("brown hair") -> don't double it.
bits.append(hair if hair.lower().endswith("hair") else f"{hair} hair")
if feats:
bits.append(feats[0])
elif clothing:
# vision may still return a compound garment ("yellow shirt and yellow jacket");
# keep only the first clause so the label stays "the one with a yellow shirt".
bits.append(clothing.split(" and ")[0].strip())
return "the one with " + " and ".join(bits) if bits else ""
class SceneInput(BaseModel):
panel_id: str = ""
panel_uri: str = ""
vision_result: dict = {}
identity_assignments: list = []
characters_registry: list = [] # [{"character_id","name"}]
def build_scene(data: SceneInput):
name_by_id = {c["character_id"]: c.get("name", "") for c in data.characters_registry}
desc_by_id = {c["character_id"]: c.get("description") for c in data.characters_registry}
id_by_local = {a["local_id"]: a["character_id"] for a in data.identity_assignments}
characters, present = [], []
for vc in data.vision_result.get("characters", []):
cid = id_by_local.get(vc.get("local_id"))
if not cid:
continue
name = name_by_id.get(cid, "")
# frozen registry appearance first (stable across panels); fall back to this panel's vision.
label = name or _describe(desc_by_id.get(cid)) or _describe(vc.get("appearance"))
characters.append({"id": cid, "name": name, "label": label})
present.append({"character_id": cid, "name": name, "bbox": vc.get("bbox", [0, 0, 0, 0])})
# Speaker attribution is the vision/dialogue model's job now (it reads bubble tails + turn-taking).
# speaker is a local_id -> map to character_id via identity; None for narration/sfx/off-panel.
dialogue = []
for d in data.vision_result.get("dialogue", []):
raw = d.get("speaker")
# "unknown" is a DELIBERATE off-panel/indeterminate speaker (narrated as "someone"); it maps
# to None just like narration. id_by_local.get already yields None for it.
dialogue.append({"speaker": id_by_local.get(raw), "text": d.get("text", ""),
"type": d.get("type", "speech"),
"confidence": d.get("confidence"),
"speaker_method": d.get("speaker_method", "unknown")})
vchars = data.vision_result.get("characters", [])
action = "; ".join(c.get("action", "") for c in vchars if c.get("action")) or ""
return {"panel_id": data.panel_id, "characters": characters, "dialogue": dialogue,
"action": action, "entities": data.vision_result.get("entities", []),
"camera": data.vision_result.get("camera", {}), # #8 direction passthrough -> render
"transition": data.vision_result.get("transition", "cut")} # #6 transition out
app = FastAPI()
transport.install_logging(app, "scene")
@app.post("/scene/build")
async def scene(data: SceneInput):
return build_scene(data)
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
out = build_scene(SceneInput(
panel_id="p001",
vision_result={"characters": [{"local_id": "person_1", "bbox": [0, 0, 10, 10], "action": "waving"}]},
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
characters_registry=[{"character_id": "c1", "name": "Teto"}],
))
assert out["characters"] == [{"id": "c1", "name": "Teto", "label": "Teto"}]
# A: unnamed character -> stable descriptive label from frozen registry appearance
outu = build_scene(SceneInput(
panel_id="p003",
vision_result={"characters": [{"local_id": "person_1", "bbox": [0, 0, 10, 10]}]},
identity_assignments=[{"local_id": "person_1", "character_id": "c9"}],
characters_registry=[{"character_id": "c9", "name": "",
"description": '{"hair":"black","features":["glasses"]}'}],
))
assert outu["characters"][0]["name"] == "" and outu["characters"][0]["label"] == "the one with black hair and glasses"
assert _describe({}) == "" # bare appearance -> empty, script falls back to Person X
# animals are named as their species (color + species), never by clothing/"the one with..."
assert _describe({"species": "cat", "hair": "black"}) == "the black cat"
assert _describe({"species": "cat", "features": ["gold bell"]}) == "the gold bell cat"
assert _describe({"species": "cat"}) == "the cat"
assert _describe({"species": "human", "hair": "black"}) == "the one with black hair"
# compound garment collapses to the first clause (no "shirt and jacket" redundancy)
assert _describe({"clothing": "yellow shirt and yellow jacket"}) == "the one with yellow shirt"
assert out["action"] == "waving"
# vision-owned dialogue: typed, proper-cased, speaker mapped local_id -> character_id
out2 = build_scene(SceneInput(
panel_id="p002",
vision_result={
"characters": [{"local_id": "person_1", "bbox": [0, 0, 10, 10]}],
"dialogue": [{"speaker": "person_1", "type": "thought", "text": "So this is it."},
{"speaker": "", "type": "narration", "text": "Three years later."}],
"entities": [{"name": "Everyday", "kind": "shop"}],
},
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
characters_registry=[{"character_id": "c1", "name": "Teto"}],
))
assert out2["dialogue"][0]["speaker"] == "c1" and out2["dialogue"][0]["type"] == "thought"
assert out2["dialogue"][1]["speaker"] is None and out2["dialogue"][1]["type"] == "narration"
assert out2["entities"][0]["name"] == "Everyday"
# explicit "unknown" (off-panel/indeterminate) maps to None -> narrated as "someone", never
# pinned to a visible character. (the "line defaults to the MC" bug.)
out4 = build_scene(SceneInput(
panel_id="p005b",
vision_result={
"characters": [{"local_id": "person_1", "bbox": [0, 0, 10, 10]}],
"dialogue": [{"speaker": "unknown", "type": "speech", "text": "A Teto and Egen test?"}],
},
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
characters_registry=[{"character_id": "c1", "name": "MC"}],
))
assert out4["dialogue"][0]["speaker"] is None, out4["dialogue"][0]
# Attribution provenance is load-bearing: a weak guess must remain distinguishable downstream.
out5 = build_scene(SceneInput(
panel_id="p006",
vision_result={"dialogue": [{"speaker": "person_1", "type": "speech", "text": "Maybe.",
"confidence": 0.3, "speaker_method": "turn_taking"}]},
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
))
assert out5["dialogue"][0]["confidence"] == 0.3
assert out5["dialogue"][0]["speaker_method"] == "turn_taking"
print("worker_scene self-check ok")
worker_script.py
+300
View File
@@ -0,0 +1,300 @@
# worker_script.py — stage 7 narration. FastAPI :8005. calls the warm gemma4 server (:8090).
import os, re, json
from fastapi import FastAPI
from pydantic import BaseModel
import requests
import transport
app = FastAPI()
transport.install_logging(app, "script")
GEMMA4_URL = os.environ.get("GEMMA4_URL", "http://127.0.0.1:8090")
class ScriptInput(BaseModel):
scene_graph: dict
chapter_context: str = ""
panel_id: str = ""
session_id: str = ""
names_by_id: dict = {} # character_id -> name, resolved from the final registry (A backfill)
genders_by_id: dict = {} # character_id -> m|f|unknown, from the registry (F2, for pronouns)
brief: str = "" # #3: static whole-chapter synopsis (background for consistency/tone)
recent: list = [] # #1: last few panels' narration text (local flow, no repeats)
introduced: list = [] # #2: names already narrated -> refer by name/pronoun, don't re-describe
panel_count: int = 1 # scene-level narration: #panels in this beat -> scales the length budget
class SummaryInput(BaseModel):
prior: str = "" # the running "story so far" before this batch
recent: list = [] # narration of the panels since the last refresh
session_id: str = ""
def _strip_thought(text: str) -> str:
parts = re.split(r"<\|?channel\|?>", text)
return parts[-1].strip() if len(parts) > 1 else text
# how each bubble type reads in the narration. narration/sfx have no speaker.
_VERB = {"speech": "says", "thought": "thinks", "shout": "shouts"}
def _render_line(d, name_by_id):
typ = d.get("type", "speech")
txt = d.get("text", "")
if typ == "narration":
return f'Caption (scene fact, weave in — do not announce it): "{txt}"'
if typ == "sfx":
return f"Sound effect: {txt}"
who = name_by_id.get(d.get("speaker"), "Someone") if d.get("speaker") else "Someone"
return f'{who} {_VERB.get(typ, "says")} "{txt}"'
def _name_map(characters, names_by_id=None):
"""id -> display name. Final registry name first (names_by_id, so a name learned in a LATER
panel backfills to this one); else the name baked at scene time; else the frozen descriptive
label ("the one with black hair and glasses"); else a stable positional label ("Person A").
The model never sees raw DB ids like character_11004ac9."""
names_by_id = names_by_id or {}
name_by_id, n = {}, 0
for c in characters:
if names_by_id.get(c["id"]) or c.get("name"):
name_by_id[c["id"]] = names_by_id.get(c["id"]) or c["name"]
elif c.get("label"):
name_by_id[c["id"]] = c["label"]
else:
name_by_id[c["id"]] = f"Person {chr(65 + n)}"
n += 1
return name_by_id
_PRONOUN = {"m": "he", "f": "she"}
def _chars_line(name_by_id, genders_by_id):
"""render 'Name (he), Other (she)' so the narrator gets pronouns right (F2)."""
genders_by_id = genders_by_id or {}
out = []
for cid, disp in name_by_id.items():
p = _PRONOUN.get((genders_by_id.get(cid) or "").lower())
out.append(f"{disp} ({p})" if p else disp)
return ", ".join(out) or "none"
def build_prompt(sg: dict, chapter_context: str, names_by_id=None,
brief: str = "", recent=None, introduced=None, genders_by_id=None,
panel_count: int = 1) -> str:
name_by_id = _name_map(sg.get("characters", []), names_by_id)
chars = _chars_line(name_by_id, genders_by_id)
dialogue = "\n".join(_render_line(d, name_by_id) for d in sg.get("dialogue", [])) or "none"
ents = ", ".join(e.get("name", "") for e in sg.get("entities", []) if e.get("name"))
# cast+entities roster (whole-manga memory), kept distinct from the running plot summary below —
# they used to both render as "Story so far", so the static roster masqueraded as the story.
ctx = f"Cast & places: {chapter_context}\n\n" if chapter_context else ""
# #3 running "story so far": only what has happened up to now, so it grounds tone/naming
# without any risk of narrating ahead (future panels aren't in it yet).
ov = (f"Story so far (background — what has happened up to now; stay consistent with it, do "
f"NOT restate it verbatim): {brief}\n\n") if brief else ""
# #1 local flow: keep continuity with what was just said, don't restate it — and don't reuse its
# sentence scaffolding (LLMs happily repeat openers/structure even when the facts differ).
rec = ("Just narrated (continue smoothly from this; do NOT repeat these facts, and do NOT reuse "
"their sentence structure or opening words — vary the rhythm):\n"
+ "\n".join(f"- {r}" for r in recent) + "\n\n") if recent else ""
# camera->pacing: let the director's shot drive sentence rhythm (the biggest "feels AI" tell is
# narration pacing that ignores the cut). fast/tight shot + hard cut -> punchy; held/drifting shot
# or soft dissolve -> can breathe.
cam = (sg.get("camera") or {}).get("effect") or ""
trans = sg.get("transition") or ""
pace = (f"Shot pacing — camera: {cam or 'static'}, transition into this panel: {trans or 'cut'}. "
"Match your line's rhythm to the shot: a hard cut or tight/fast camera wants a short, "
"punchy sentence; a held or slowly drifting shot or a soft dissolve can run a little "
"longer and calmer.\n") if (cam or trans) else ""
# #2 no re-introductions once a character has appeared.
intro = ("Already introduced — refer to them by name or pronoun, do NOT re-describe their "
"appearance: " + ", ".join(introduced) + "\n") if introduced else ""
# scene-level narration: one beat can span several panels, so scale the budget with the beat size
# (one flowing recap of the whole moment, not a line per panel). solo panel keeps a tight cap.
if panel_count > 1:
unit = f"this {panel_count}-panel manga beat (one continuous moment)"
budget = f"{min(1 + panel_count, 4)} short sentences, max ~{min(20 * panel_count, 60)} words"
thin = "beat"
else:
unit, budget, thin = "this manga panel", "1-2 short sentences, max ~25 words", "panel"
return (
ov + rec + ctx + intro +
f"Write TIGHT recap narration for {unit}: {budget}, "
"present tense. Tell it like you're recapping to a friend — natural, with momentum. Output "
"only the narration — no preamble, no notes, no alternatives, never ask for more info. If the "
f"{thin} has little dialogue or action, one brief scene-setting sentence from what is given.\n"
"Voice rules (avoid machinery):\n"
"- Lead with what happens or what's said. When a character speaks, use their ACTUAL words in "
"a short quote — don't paraphrase into 'he remarks that…' / 'she asks if…'.\n"
"- Do NOT open with a physical-action gerund ('Looking up,', 'Adjusting his glasses,', "
"'Leaning forward,'). Vary sentence openings.\n"
"- Do NOT invent intent or filler ('prepares to…', 'ready to assert…', 'seeking "
"confirmation'). Only what is actually shown.\n"
"- Refer to characters by the names below. For anyone WITHOUT a real name, use a SHORT handle "
"(e.g. 'the guy in glasses') at most ONCE, then 'he'/'she' — never restate their full "
"appearance from panel to panel.\n"
"- A thought reads as internal, a shout as raised. Caption/box text is a scene fact — weave "
"it in; never say 'a narration box' or 'the screen displays'.\n"
+ (f"Spell these proper nouns exactly: {ents}\n" if ents else "") + "\n"
+ pace +
f"Characters present: {chars}\n"
f"Dialogue:\n{dialogue}\n"
f"Action: {sg.get('action','')}\n\n"
"Narration:"
)
_SYSTEM = ("You are a manga recap narrator. You output only the finished narration paragraph — "
"never your reasoning, drafts, options, or requests for more information.")
def call_gemma4(prompt: str, system: str = _SYSTEM, max_tokens: int = 160,
temperature: float = 0.6) -> str:
payload = {"messages": [{"role": "system", "content": system},
{"role": "user", "content": prompt}],
"temperature": temperature, "max_tokens": max_tokens}
r = requests.post(f"{GEMMA4_URL}/v1/chat/completions", json=payload, timeout=300)
r.raise_for_status()
return _strip_thought(r.json()["choices"][0]["message"]["content"])
def build_summary_prompt(prior: str, recent) -> str:
prior_s = prior.strip() or "(nothing yet — this is the start of the chapter)"
new = "\n".join(f"- {r}" for r in recent if r) or "- (none)"
return (
"You maintain a running 'story so far' summary for a manga recap. Given the previous "
"summary and the narration of the panels since, produce an UPDATED summary in 3-6 "
"sentences: compact and factual, consistent names/places, fold in the new events and drop "
"stale detail. Only what has happened so far — never speculate ahead. Output only the "
"summary.\n\n"
f"Previous summary:\n{prior_s}\n\nNewly narrated:\n{new}\n\nUpdated summary:"
)
@app.post("/script/summary")
async def summary(data: SummaryInput):
return {"brief": call_gemma4(build_summary_prompt(data.prior, data.recent))}
@app.post("/script")
async def script(data: ScriptInput):
text = call_gemma4(build_prompt(data.scene_graph, data.chapter_context, data.names_by_id,
data.brief, data.recent, data.introduced, data.genders_by_id,
data.panel_count))
return {"panel_id": data.panel_id, "text": text}
# --- roster normalization (spec E2-lite): dedup the accumulated cast+entities in one pass ---
_NORM_SYSTEM = ("You clean up a manga's cast and entity lists. You output ONLY one JSON object, no "
"prose, no reasoning.")
class NormalizeInput(BaseModel):
names: list = [] # canonical character names already in the registry (distinct people)
entities: list = [] # [{"name","kind"}] accumulated over the chapter, full of near-dups
session_id: str = ""
def _extract_json(raw: str) -> dict:
text = _strip_thought(raw)
m = re.search(r"\{.*\}", text, re.DOTALL)
if not m:
raise ValueError(f"no json: {text[:200]}")
return json.loads(m.group(0))
def build_normalize_prompt(names, entities) -> str:
ents = "\n".join(f"- {e.get('name','')} ({e.get('kind','')})" for e in entities) or "- (none)"
nms = ", ".join(n for n in names if n) or "(none)"
return (
"Clean up these places/organizations/brands extracted from a manga chapter (may be noisy, "
"duplicated). Do THREE things:\n"
"1. MERGE entries that are the same thing into ONE canonical spelling — casing/spacing "
"variants and truncations (e.g. 'Product Planning' and 'Product Planning Team' -> one).\n"
"2. KEEP genuinely different things separate — do NOT merge distinct brands (e.g. EGENPICK "
"and TETOPICK stay two).\n"
"3. DROP non-entities: comparison phrases ('Teto Pick vs Ege'), fragments, and any entry that "
"is actually one of the CHARACTERS listed below (a person's name is not a place/org).\n"
"Also propose casing fixes for the character names (Title Case proper nouns; NEVER merge two "
"different people — only fix the spelling of the SAME name).\n\n"
f"Characters (people — for reference; drop entities that are actually these): {nms}\n"
f"Entities:\n{ents}\n\n"
'Respond with ONLY this JSON:\n'
'{"entities":[{"name":"Everyday","kind":"shop"}],"name_fixes":{"CHOI HAESEON":"Choi Haeseon"}}'
)
@app.post("/normalize")
async def normalize(data: NormalizeInput):
prompt = build_normalize_prompt(data.names, data.entities)
try:
res = _extract_json(call_gemma4(prompt, system=_NORM_SYSTEM, max_tokens=768, temperature=0.1))
except (ValueError, json.JSONDecodeError):
return {"entities": data.entities, "name_fixes": {}} # fail safe: leave the roster unchanged
res.setdefault("entities", data.entities)
res.setdefault("name_fixes", {})
return res
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: prompt includes character names + quoted dialogue.
sg = {"characters": [{"id": "c1", "name": "Gojo Satoru"},
{"id": "character_deadbeef", "label": "the one with white hair"},
{"id": "character_nolabel"}],
"dialogue": [{"speaker": "c1", "type": "shout", "text": "stand proud"},
{"speaker": "character_deadbeef", "type": "speech", "text": "no"},
{"speaker": "character_nolabel", "type": "speech", "text": "hm"},
{"speaker": "", "type": "narration", "text": "Years passed."}],
"entities": [{"name": "Jujutsu High", "kind": "org"}], "action": "approaches"}
p = build_prompt(sg, "prior scene")
assert "Gojo Satoru shouts \"stand proud\"" in p
assert 'the one with white hair says "no"' in p # unnamed + label -> descriptive phrase
assert 'Person A says "hm"' in p # unnamed, no label -> positional fallback
assert "character_deadbeef" not in p # raw db id must never reach the model
# A backfill: a name learned later (registry) overrides the scene-time label for THIS panel
pb = build_prompt(sg, "prior scene", {"character_deadbeef": "Nanami"})
assert 'Nanami says "no"' in pb and "the one with white hair" not in pb
assert 'Caption (scene fact' in p and '"Years passed."' in p
assert "Jujutsu High" in p and "prior scene" in p
# scene-level narration: a multi-panel beat scales the budget and talks about a "beat", not a panel.
assert "this manga panel" in p and "25 words" in p # solo default
pbeat = build_prompt(sg, "prior scene", panel_count=3)
assert "3-panel manga beat" in pbeat and "60 words" in pbeat and "beat has little" in pbeat
# camera->pacing renders only when the graph carries a shot; absent by default.
assert "Shot pacing" not in p
pcam = build_prompt({**sg, "camera": {"effect": "dolly_to_subject"}, "transition": "dissolve"}, "s")
assert "dolly_to_subject" in pcam and "dissolve" in pcam and "Shot pacing" in pcam
# anti-repetition names structure, not just facts
pr = build_prompt(sg, "s", recent=["He drew his blade."])
assert "sentence structure" in pr
# F2 gender -> pronoun annotation on the characters-present line
pg = build_prompt(sg, "prior scene", {"c1": "Gojo Satoru"}, genders_by_id={"c1": "m"})
assert "Gojo Satoru (he)" in pg
# #1/#2/#3: brief (background), sliding window, introduced-set all render into the prompt
pc = build_prompt(sg, "prior scene", brief="A duel unfolds.",
recent=["He drew his blade.", "The crowd fell silent."],
introduced=["Gojo Satoru"])
assert "A duel unfolds" in pc and "Story so far" in pc
assert "- He drew his blade." in pc and "The crowd fell silent." in pc
assert "Already introduced" in pc and "Gojo Satoru" in pc
# #3 incremental: prior summary + new narration -> updated-summary prompt
sp = build_summary_prompt("Hero left the village.", ["He reached the city gate."])
assert "Hero left the village." in sp and "He reached the city gate." in sp
assert "Updated summary:" in sp
sp0 = build_summary_prompt("", []) # cold start, no narration yet
assert "nothing yet" in sp0
# normalization: prompt lists the entities + names; parser reads back a canonical roster
np = build_normalize_prompt(["Lim Seonho"], [{"name": "Product Planning", "kind": "team"},
{"name": "Product Planning Team", "kind": "org"}])
assert "Product Planning Team" in np and "Lim Seonho" in np and "MERGE" in np
nr = _extract_json('{"entities":[{"name":"Everyday","kind":"shop"}],'
'"name_fixes":{"CHOI HAESEON":"Choi Haeseon"}}')
assert nr["entities"][0]["name"] == "Everyday" and nr["name_fixes"]["CHOI HAESEON"] == "Choi Haeseon"
print("worker_script self-check ok")
worker_tts.py
+231
View File
@@ -0,0 +1,231 @@
# worker_tts.py — stage 8 TTS. FastAPI :8006. dots.tts loads in-process (session-guarded).
# generates audio locally, uploads to minio, returns uri + duration. no local persistence.
import os, re, json, uuid, wave, contextlib, subprocess, logging, functools
from fastapi import FastAPI
from pydantic import BaseModel
import transport
log = logging.getLogger("tts")
app = FastAPI()
transport.install_logging(app, "tts")
SHM = "/dev/shm"
DOTS_MODEL = "rednote-hilab/dots.tts-base"
_tts = None
# B#3 one fixed voice: dots.tts samples a RANDOM speaker each call unless given a reference clip,
# so the narrator's timbre drifts panel-to-panel. clone from ONE reference for every synth.
# to pin YOUR OWN voice: set VOICE_REF=/path/to/clip.wav + VOICE_REF_TEXT="its exact transcript"
# (5-15s of clean speech works best). no edit/rebuild needed -- just the env vars on the worker.
# with no VOICE_REF, we bootstrap a seeded reference once and persist it; delete it to reroll.
VOICE_DIR = os.path.expanduser("~/.cache/manga-tts")
REF_WAV = os.environ.get("VOICE_REF") or os.path.join(VOICE_DIR, "narrator_ref.wav")
REF_TEXT = os.environ.get("VOICE_REF_TEXT", "The story continues as our hero steps forward into the unknown.")
REF_SEED = 20260713
def _load_tts():
global _tts
if _tts is None:
# dots.tts's vendored loader (models/dots_tts/model.py) calls AutoTokenizer.from_pretrained
# with no kwargs, so its Mistral-derived tokenizer loads with the buggy split regex. Default
# fix_mistral_regex=True at the transformers layer to get canonical tokenization + kill the warning.
# ponytail: monkeypatch because the loader exposes no passthrough; drop if dots_tts adds one.
import transformers
_orig = transformers.AutoTokenizer.from_pretrained.__func__
transformers.AutoTokenizer.from_pretrained = classmethod(
lambda cls, *a, **kw: _orig(cls, *a, **{"fix_mistral_regex": True, **kw})
)
from dots_tts.runtime import DotsTtsRuntime
_tts = DotsTtsRuntime.from_pretrained(DOTS_MODEL, precision="bfloat16")
return _tts
def _wav_duration(path: str) -> float:
with contextlib.closing(wave.open(path, "rb")) as w:
return round(w.getnframes() / float(w.getframerate()), 3)
class TTSInput(BaseModel):
text: str
speaker: str = "narrator" # multi-voice is v3
panel_id: str = ""
session_id: str = ""
panel_uri: str = "" # optional: if passed, audio is stored beside its panel
def _audio_uri(data: "TTSInput") -> str:
# prefer the panel's own manga/chapter prefix; the orchestrator currently doesn't pass it,
# so fall back to a flat panel_id-keyed key (matches homesrv's panel_id-keyed audio table).
# ponytail: flat key collides across chapters (as does the homesrv audio table); pass
# panel_uri from run_stage_tts to make it per-chapter unique.
if data.panel_uri:
parts = data.panel_uri.replace("s3://", "").split("/")
return f"s3://manga/{parts[1]}/{parts[2]}/audio/{data.panel_id or 'p'}.wav"
return f"s3://manga/_audio/{data.panel_id or 'p'}.wav"
def _ensure_ref() -> str:
"""the fixed narrator reference clip; generate it once (seeded) and persist across restarts."""
if os.path.exists(REF_WAV):
return REF_WAV
os.makedirs(VOICE_DIR, exist_ok=True)
try:
import torch
torch.manual_seed(REF_SEED) # reproducible speaker for the bootstrap sample
except Exception:
pass
out = _load_tts().generate(text=REF_TEXT)
_write_wav(out["audio"], out["sample_rate"], REF_WAV)
return REF_WAV
def _calm(text: str) -> str:
"""dots.tts over-emotes on '!' (shouty prosody). soften exclamations to periods so the narrator
stays even. only the spoken text is calmed -- the burned subtitles keep the original '!'."""
return re.sub(r"\s*!+", ".", text)
# 158: dots.tts has no SSML/phoneme input, so proper nouns it mangles are fixed by respelling the SPOKEN
# text only (burned subtitles keep the original spelling — they're built elsewhere from the untouched
# script). Lazy v1: one global JSON map {term: phonetic}, ~8 lines, loaded once.
LEXICON_PATH = os.path.expanduser(os.environ.get("TTS_LEXICON", "~/.cache/manga-tts/lexicon.json"))
@functools.lru_cache(maxsize=1)
def _lexicon():
"""(compiled word-boundary pattern, {lower_term: phonetic}) or None. Cached; delete the file and
call _lexicon.cache_clear() to reload. Longest terms first so multi-word names match whole."""
try:
with open(LEXICON_PATH) as f:
m = json.load(f)
except (OSError, ValueError):
return None
m = {k: v for k, v in (m or {}).items() if k and v}
if not m:
return None
pat = re.compile(r"\b(" + "|".join(re.escape(k) for k in sorted(m, key=len, reverse=True)) + r")\b",
re.IGNORECASE)
return pat, {k.lower(): v for k, v in m.items()}
def _respell(text: str) -> str:
"""Substitute known proper nouns with their phonetic respelling (whole word, case-insensitive)."""
lex = _lexicon()
if not lex:
return text
pat, lookup = lex
return pat.sub(lambda mo: lookup[mo.group(0).lower()], text)
def _generate(text: str) -> str:
"""returns a local wav path. dots runtime returns {"audio": samples, "sample_rate": sr};
clone the fixed reference voice so every panel narrates in the same timbre."""
ref = _ensure_ref()
out = _load_tts().generate(text=_respell(_calm(text)), prompt_audio_path=ref, prompt_text=REF_TEXT)
return _write_wav(out["audio"], out["sample_rate"])
def _write_wav(audio, sample_rate: int, path: str | None = None) -> str:
import numpy as np, soundfile as sf
if hasattr(audio, "detach"): # torch tensor (possibly on GPU)
audio = audio.detach().cpu().numpy()
a = np.asarray(audio, dtype="float32").squeeze() # (samples,) mono
if path is None:
path = f"{SHM}/tts_{uuid.uuid4().hex[:8]}.wav"
sf.write(path, a, sample_rate, subtype="PCM_16")
return path
def _loudnorm(path: str) -> str:
"""EBU R128 loudness-normalize so narration volume is even panel-to-panel (#14).
Returns a normalized path; on any ffmpeg failure returns the original (never lose audio)."""
sr = wave.open(path, "rb").getframerate()
out = f"{SHM}/ln_{uuid.uuid4().hex[:8]}.wav"
try:
subprocess.run(
["ffmpeg", "-y", "-i", path, "-af", "loudnorm=I=-16:TP=-1.5:LRA=11",
"-ar", str(sr), out],
check=True, capture_output=True,
)
os.replace(out, path)
except (subprocess.CalledProcessError, FileNotFoundError) as e:
log.warning("loudnorm skipped for %s: %r", path, e)
if os.path.exists(out):
os.remove(out)
return path
@app.post("/tts")
async def tts(data: TTSInput):
local = _loudnorm(_generate(data.text))
uri = _audio_uri(data)
transport.put(local, uri)
dur = _wav_duration(local)
os.remove(local)
return {"audio_uri": uri, "duration": dur}
@app.post("/unload")
async def unload():
"""free the resident dots.tts so the session manager can hand the GPU to the next model."""
global _tts
was = _tts is not None
_tts = None
import gc; gc.collect()
try:
import torch; torch.cuda.empty_cache()
except Exception:
pass
return {"ok": True, "unloaded": was}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: _write_wav renders a sample array (1.0s @16k) to a PCM-16 wav that
# wave.open reads back at the right duration; then exercise upload + uri.
import math
class _FakeMC:
def __init__(self): self.store = {}
def bucket_exists(self, b): return True
def make_bucket(self, b): pass
def fput_object(self, b, k, path): self.store[(b, k)] = open(path, "rb").read()
samples = [0.3 * math.sin(i / 8) for i in range(16000)] # bare python list -> np.asarray
local = _write_wav(samples, 16000)
dur = _wav_duration(local)
uri = transport.put(local, "s3://manga/m/c/audio/p001.wav", client=_FakeMC())
assert uri.endswith("audio/p001.wav") and abs(dur - 1.0) < 0.01, (uri, dur)
os.remove(local)
explicit = f"{SHM}/tts_selfcheck_ref.wav" # _write_wav honors an explicit path (ref clip)
assert _write_wav(samples, 16000, explicit) == explicit and os.path.exists(explicit)
os.remove(explicit)
assert _calm("Stop!! Now!") == "Stop. Now." and _calm("no bangs") == "no bangs" # even prosody
# 158: pronunciation lexicon respells whole words only, case-insensitive, spoken text only.
import tempfile, json as _json
globals()["LEXICON_PATH"] = os.path.join(tempfile.mkdtemp(), "lex.json")
with open(LEXICON_PATH, "w") as f:
_json.dump({"Najimi": "nah-jee-mee", "Choi Haeseon": "chwe hae-son"}, f)
_lexicon.cache_clear()
assert _respell("Then Najimi ran.") == "Then nah-jee-mee ran." # single name
assert _respell("with najimi today") == "with nah-jee-mee today" # case-insensitive
assert _respell("Najimist stays") == "Najimist stays" # word boundary (no substring)
assert _respell("Choi Haeseon smiled") == "chwe hae-son smiled" # multi-word term
os.remove(LEXICON_PATH); _lexicon.cache_clear()
assert _respell("Najimi ran.") == "Najimi ran." # no file -> passthrough
# loudnorm: with ffmpeg present the wav is normalized in place and stays readable at its sr;
# without ffmpeg it's a safe no-op returning the same path (audio never lost).
ln = _write_wav(samples, 16000)
have_ffmpeg = subprocess.run(["ffmpeg", "-version"], capture_output=True).returncode == 0 \
if __import__("shutil").which("ffmpeg") else False
assert _loudnorm(ln) == ln and os.path.exists(ln)
assert wave.open(ln, "rb").getframerate() == 16000
os.remove(ln)
print("worker_tts self-check ok" + ("" if have_ffmpeg else " (ffmpeg absent, loudnorm no-op tested)"))
worker_vision.py
+1008
View File
File diff suppressed because it is too large Load Diff