kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'Restore the runtime and bring master up to the seventh session' (#1) from restore-runtime into master

Browse Source
This commit was merged in pull request #1.
This commit is contained in:
claude
2026-08-13 21:07:13 +02:00
parent ff6a512630 0d281016e7
commit a67f1c2501
40 changed files with 4710 additions and 471 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -4,3 +4,4 @@ __pycache__/
dots.tts/ dots.tts/
/dev/shm/ /dev/shm/
*.gguf *.gguf
models/
AGENTS.md
+21
View File
@@ -1,5 +1,26 @@
# Repository guidance # Repository guidance
## Restarting one worker
Python binds a module at process start, so a worker keeps serving the code it loaded. Run
`./check_stale.sh` before any stage run meant to prove a change. It exits non-zero and names the tmux
window for every worker older than its file (`decisions/identity-bbox.md#stale-worker-invalidates`).
**`tmux respawn-window -k` does not re-run the window's command.** It leaves a bare shell and the worker
stays down, silently. Restart by re-sending the `send-keys` line from `start_workers.sh` for that one
worker, then confirm on `/health` and with `check_stale.sh`:
```bash
M="export MIOPEN_USER_DB_PATH=$HOME/.config/miopen MIOPEN_SYSTEM_DB_PATH=$HOME/.config/miopen MIOPEN_FIND_MODE=2 && unset MIOPEN_FIND_ENFORCE"
tmux send-keys -t manga-workers:vision C-c
tmux send-keys -t manga-workers:vision "$M && source $PWD/.venv/bin/activate && python -m uvicorn worker_vision:app --app-dir $PWD --host 0.0.0.0 --port 8002" C-m
curl -s http://127.0.0.1:8002/health && ./check_stale.sh
```
The orchestrator's equivalent is `docker compose up -d --build orchestrator` on homesrv, because the image
bakes its source. That recreates the container, which drops any `docker cp`'d file such as
`audit_registry.py`.
## Video analysis ## Video analysis
When diagnosing render motion, transitions, timing, or visual artifacts, use When diagnosing render motion, transitions, timing, or visual artifacts, use
ARCHITECTURE.md
+353
View File
@@ -0,0 +1,353 @@
# ARCHITECTURE
The target shape of the pipeline. Written 2026-08-13 from the user's design, rewritten the same day under
the no-Magi constraint. This is **not** what the code does. `NEXT.md` holds the live state and `AUDIT.md`
holds the current pipeline. Every section ends with what exists today and what would make it done. The
gap stays legible and testable without reading both.
The governing principle:
> Do not make the next panel understand the previous panel. Make it understand the current world state
> produced by all previous panels.
Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below
follows from that split.
## Ruled out
Read this before proposing any of it again.
| ruled out | why | who decided |
| --- | --- | --- |
| Magi, Magiv2, Magiv3 as a detector or as pair heads | project constraint, not a measurement | user, 2026-08-13 |
| A trained `t2c` pair model, or any learned head | no labelled pages exist and no labeller is planned | follows from the above |
| Crop-to-crop cosine as a link signal | measured: two men reach 0.93, one man reaches 0.96, no threshold exists (`caveats/audit-open.md#cosine-not-identity`) | 2026-08-12 |
The consequence runs through the whole document. **Every structure below comes out of a gemma prompt
field, or out of plain Python over gemma's output.** Nothing below is trained. A learned score
in the original design becomes a hand-weighted sum. The weights are read off the labelled chapter, and
the DoD is the accuracy number rather than the mechanism.
## The measurement spine
Nothing below can be called done without this, and it does not exist yet.
The only ground truth in the project is the eyeball pass over the 19:44 run of 2026-08-12. It lives in
prose in `NEXT.md`. Write it to `eval/chapter-truth.json` against chapter
`7c944dd4-e972-42c7-ba60-9f6939548e80`, scoped to what was already checked by eye rather than to all 119
detections:
- the three characters walked crop by crop, each assignment marked as the real person or not
- 30 dialogue lines with their true speaker, typed as `visible | offscreen | narrator | unknown`
The baseline it records, from that run:
| character | assignments | correct | purity | note |
| --- | --- | --- | --- | --- |
| the lead | 16 | 14 | 0.88 | plus a photograph at `order 17` and a chibi at `order 20` |
| `character_2b1b12a1` | 13 | 13 | 1.00 | a main character the registry never named |
| `character_f0d4e901` | 9 | 7 | 0.78 | the other 2 are `2b1b12a1` |
Woman A is `2b1b12a1`. She has 15 occurrences split across 2 ids, so her fragmentation is 2.
`audit_registry.py` already walks panels, reads `identity_assignments` and counts per character. Extend
it to print purity and fragmentation against the truth file. Do not write an eval harness.
**Done when:** `audit_registry.py <chapter>` prints purity per labelled character and fragmentation per
labelled person, and reproduces the table above on the 19:44 run.
## 1. The page is a region graph, not a list of panels
```
page
├─ regions
│ ├─ panel
│ ├─ inset_panel
│ ├─ embedded_art
│ ├─ text
│ ├─ tail
│ └─ character_occurrence
│
└─ edges
├─ contains(region, region)
├─ reads_before(text, text)
├─ tail_of(tail, text)
├─ points_to(tail, character)
├─ spoken_by(text, character)
└─ same_identity(character, character)
```
A flat set of panels cannot express a television inside a room. That is the defect the current pipeline
shows most often.
With no detector to train, the graph comes out of the detection prompt. Gemma already returns a box per
character and per text. Two fields per detection buy most of the graph with no new model:
```
plane = story | screen | photo | poster | drawing | flashback | dream
species = human | animal | object
```
`plane` is the containment edge in disguise. A detection whose `plane` is not `story` sits inside embedded
art, and that is the fact every stage below needs. `species` is a separate axis and exists because vision
boxes cats as people and dresses them.
Tail regions stay unbuilt. The `det`/`seg` heads exist and are unused
(`caveats/speaker-attribution.md#tail-is-not-geometry`), and section 3 says why they are not the first
thing to spend on.
**Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a
non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no
containment edge, no region type, no `plane` and no `species`.
**Done when:** every detection carries `plane` and `species`. On the labelled chapter, `order 17` and
`order 20` of the lead are not `story`, none of his 14 correct crops is demoted, and `p081` and `p108` are
`animal`. Measured by `audit_registry.py`, which already reads the vision blob per panel.
## 2. Identity exists independently of names
```
occurrence c42
-> identity char_07
name = null
aliases = []
```
`char_07.name` may be filled later, or stay null forever and display as `unknown character #7`. The
occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster.
Three levels, never collapsed into one.
**Today:** already true. `identity_assignments` is the occurrence, `characters` owns the identity, `name`
is nullable and downstream already falls back to an anonymous display.
**Done when:** already done. No work item. The clustering is section 4 and the naming is
`decisions/identity-naming.md`.
## 3. Speaker attribution is a scored edge, not a procedure
Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge:
```
score(text, character) =
w1 * gemma_answer
+ w2 * spatial_evidence
+ w3 * same_panel
+ w4 * same_plane
+ w5 * conversation_continuity
+ w6 * character_activity_prior
```
The original design put a learned `t2c` head in the first term and called it load-bearing. No labelled
pages exist, so that term does not. **Gemma's answer becomes one term of six rather than the whole
procedure.** The geometry terms overrule it when they agree against it. The weights are constants read off
the 30 labelled lines. Six numbers in a module, not a training run.
Geometry carries normalized relative position, distance, overlap, same-panel and containment depth. Tail
direction is absent until a tail region exists, and it is not the first thing to build. `conversation
continuity` is free, and turn-taking is the strongest prior for a tail-less bubble.
Then the cases fall out of one mechanism instead of four:
| case | what carries it |
| --- | --- |
| bubble with a tail | gemma plus spatial, usually decisive |
| bubble with no tail | conversation continuity plus spatial |
| speaker outside the panel | recent identities plus an offscreen candidate |
| narration | the narrator candidate |
| nothing resolves | unknown speaker |
**A dialogue line must not be required to resolve to a visible character.** That is a failure mode, not a
safeguard. The speaker type is a union:
```
speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown
```
**Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator`
(`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a
prompt to gemma over a window of panels, with no geometry term at all.
**Done when:** `audit_speakers.py` reports accuracy over the 30 labelled lines, split by true type, and
the scored version beats the recorded gemma-window baseline. Two numbers must move the right way, and both
are reported. Correct assignments go up. **Forced** errors go down, where forced means a line given a
visible character while the truth is `offscreen`, `narrator` or `unknown`. Record the baseline before
touching the code.
## 4. Character recognition is occurrence, then identity, then name
```
character detection
↓
occurrence embeddings
↓
pairwise same_identity scores
↓
chapter-wide constrained clustering
↓
char_001, char_002, ...
↓
optional name claim
↓
name or unknown
```
Three rules the current code gets wrong.
**The embedding is not the character crop alone.** The crop embedding measures scene, not person, which is
why two men reach 0.93. Combine the face or head crop with the person crop instead of replacing one with
the other. `face_detect` already finds the face and pairs it for `has_face`, so the face box is free. This
is the queued experiment in `NEXT.md` item 1: re-embed the same 22 detections and recompute the matrix.
**Cluster chapter-wide, not page by page.** `tracklets.link_tracklets` groups within an 8-panel window.
**Two characters in the same panel may be one person.** Seven things break that rule. Mirrors,
photographs, flashbacks, insets, screens, imagined scenes, repeated action drawings. Make it a weak
cannot-link, and only between detections on the same `plane`.
That last rule has an ordering trap. Same-panel co-presence is currently a **hard** constraint and it is
load-bearing precisely because cosine cannot separate people. Weakening it before the embedding improves
will regress purity. The dependency is the embedding fix, not the `plane` field alone.
**Today:** the embedding is the person box only (`caveats/audit-open.md#cosine-not-identity`). Clustering
is greedy and local. `tracklets.cannot_link` treats same-panel co-presence as hard. Naming is
`db.add_name_claim`, corroboration over `name_claims`.
**Done when:** no labelled character holds more than one wrong assignment, and woman A's fragmentation is
1. Baseline is 2 wrong, 0 wrong, 2 wrong, and fragmentation 2. The bar is stated in errors rather than in
a purity ratio on purpose. The three characters hold 16, 13 and 9 assignments. At those counts any ratio
above 0.94 means zero tolerated errors, and the ratio hides that.
The face-plus-person embedding lands first and carries its own smaller check. On the 22 measured
detections, the highest different-person pair must fall below the lowest same-person pair.
## 5. The narrative plane is what stops art-in-art
The page is a hierarchical scene graph:
```
page
└── panel A plane=story
├── character c1
├── text t1
└── television/poster plane=screen
├── character c2
└── text t2
```
Perfect classification is not the point. The output that matters is one predicate:
```
same_narrative_plane(a, b)
```
It has exactly two consumers, and they are the reason the field is worth adding at all:
- section 3, as the `same_plane` term. A real character beside a poster does not get the poster person's
line.
- section 4, as the guard that makes the weak cannot-link safe.
**Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the
19:44 run his 16 assignments were 14 correct plus a photograph of another man and a chibi drawing. Both
are art inside a panel.
**Done when:** section 1's DoD, plus both consumers wired, plus section 4's purity DoD holds with the
cannot-link demoted to weak. If purity regresses when the constraint is weakened, the embedding is not
ready and the demotion reverts.
## 6. Narrative understanding is carried state, not a per-panel description
Today each stage reads its predecessor's blob for one panel or one beat. `recent`, a rolling list of the
last few dialogue lines, is the only carried state. That is the root of the invented narration.
The version worth building is one record per scene, inherited forward:
```
scene_31:
location: school_rooftop
time: evening
participants: {char_03: present, char_07: present, char_11: offscreen}
narrative_mode: present
last_speaker: char_07
addressee: char_02
```
A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character
absent for one panel has not left. A panel with no background has not changed location. A tail-less line
keeps the offscreen participant as a candidate. A close-up still belongs to the scene. `last_speaker` and
`participants` are what section 3's continuity term reads.
A panel produces a **delta** against that record, not another standalone prose interpretation:
```
panel 142:
- character_07 enters room_03
- character_02 is already present
- character_07 says "..."
```
Classify the transition, not just the panel:
```
CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START
FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE
```
`EMBEDDED_SCENE` is `plane != story` at scene granularity, and is what stops a television's contents
mutating the room around it.
**Scene state is written by a resolver, never by the vision model.** The path is
`observation -> resolver -> state transition`. The resolver may reject an impossible update. It is plain
Python over gemma's per-panel delta, and it is where the constraint lives.
**Today:** none of it exists. Chapter boundaries are a reset. Narration asserts things no panel shows
(`NEXT.md` item 6).
**Done when:** a scene record carries location, participants and `narrative_mode` across panels. A
character absent from one panel stays a participant. On the next full run the four invented-fact
timestamps do not recur. Those are 0:43, 2:03, 2:05 and 2:15, and they are the regression list. The
correctness verifier passed 116/116 over them because it checks quotes and names, never invented claims.
So the check is a re-watch of those four points, not a stage counter.
### Not building yet
Each of these was in the original design. Each is deferred with a trigger, not dropped.
| deferred | trigger to revisit |
| --- | --- |
| Facts, hypotheses and unknowns as separate records with confidences | when scene state exists and narration still asserts unshown claims |
| The seven-check consistency checker | when a scene record exists for it to check against |
| Unresolved references that survive and back-propagate | when a second chapter of the same manga runs |
| Chapter checkpoints and the two-memory split | when a second chapter of the same manga runs |
One reason covers all four. They sit on an identity layer still wrong on 2 of the lead's 16 crops. State
machinery over wrong identity produces confidently wrong state.
## 7. Build order
Detection, vision and character embeddings already exist. The order below is chosen so each step is
falsifiable by the step's own DoD before the next one starts.
```
0. eval/chapter-truth.json + purity and fragmentation in audit_registry.py
1. plane + species per detection -> section 1 DoD
2. face-plus-person embedding -> section 4 embedding check
3. chapter-wide clustering, weak cannot-link on plane -> section 4 purity DoD
4. scene record carried forward -> section 6 DoD
5. scored speaker edge, offscreen arm -> section 3 DoD
6. tail regions from the unused det/seg heads -> only if 5 misses its DoD
```
Steps 1 and 2 are independent and can land together. Step 3 depends on 2, which is the ordering trap in
section 4. Step 5 depends on 4, because the continuity term reads the scene record. Step 6 is
conditional on purpose: build a tail detector only after the cheap terms have been measured and found
insufficient.
The VLM keeps judging ambiguous edges. What changes is that it stops rediscovering every character and
dialogue relationship from raw pixels on every panel. The chapter graph carries the answer forward.
## Sources
Three ideas come from published comics-transcription work. Detection and association as graph generation,
the text-to-character pair head, and the character bank of exemplar images plus names. The
formulation is kept. The models are ruled out, see **Ruled out** above. No source in this document is a
runtime dependency.
AUDIT.md
+745
View File
@@ -0,0 +1,745 @@
# Manga recap pipeline audit
Date: 2026-08-11
Scope: workpc workers plus the homesrv orchestrator. This audit excludes FFmpeg changes and does not
recommend Magi or its embeddings. No GPU workloads were run during the audit.
## Status
Phase 1 landed on 2026-08-11. This file is the reasoning, not the state.
| where the finding went | file |
| --- | --- |
| implemented, with evidence and a runnable check | `decisions/audit-phase1.md` |
| deliberately not done, with a revisit trigger | `caveats/audit-open.md` |
| still ahead, in order | `ROADMAP.md` |
| what to do next | `NEXT.md` |
Every P0 below is closed. The sections that name one now point at its decision instead of repeating it.
## Outcome
The strongest performance gains are available without replacing the current models:
1. Keep Gemma resident across compatible stages, reducing roughly seven chapter-level Gemma loads to two.
2. Batch SigLIP crop embeddings instead of running one forward pass per detected character.
3. Add bounded parallelism to CPU/network work such as page uploads, framed-page cropping, prefetching,
and scene construction.
4. Stop repeatedly downloading and base64-encoding the same panels and character references.
5. Generate schema-constrained JSON and batch text-only script work by scene.
Correctness fixes should land before throughput work because the current tracklet and naming paths can
contaminate the permanent character registry.
## Highest-priority findings
### P0: tracklet gender gate uses the wrong enum — CLOSED, decisions/audit-phase1.md#tracklet-hard-constraints
`orchestrator/tracklets.py` checks `male|female`, while vision emits `m|f|unknown`. The tracklet hard
gender gate therefore never activates on normal pipeline data.
### P0: tracklets lack co-presence constraints — CLOSED, decisions/audit-phase1.md#tracklet-hard-constraints
The linker has no same-panel exclusion and uses transitive union-find. Two similar-looking people in the
same panel can be joined, and weak links such as A~B and B~C can merge A and C even when the endpoints are
incompatible.
### P0: face-to-character pairing is unconditional — CLOSED, decisions/audit-phase1.md#gated-face-pairing
`worker_vision._pair_faces_to_present()` assigns every detected face to the nearest vision character as
long as an unused character exists. Despite the docstring, there is no distance or overlap threshold.
A distant or unrelated face can therefore receive another character's identity and speaker label.
### P0: roster hints can become permanent identity evidence
The roster is described as hints-only, but those names are fed into detection. Any name emitted on a crop
causes `worker_identity` to persist it immediately. A coarse appearance-to-roster guess can therefore
contaminate a permanent character gallery.
### P0: script repair feedback is discarded — CLOSED, decisions/audit-phase1.md#verifier-false-positives
The orchestrator sends `beat` and `verifier_feedback` on a failed-script retry, but `worker_script.ScriptInput`
does not define either field and the prompt builder does not consume them. The second call is another
stochastic attempt rather than a targeted correction.
### P0: action evidence is missing from the verifier — CLOSED, decisions/audit-phase1.md#verifier-false-positives
The beat builder reads plural `actions`, while the scene worker emits singular `action`. Script validation
therefore receives little or no action evidence and cannot reliably detect invented or omitted actions.
### P0: GPU leases are unsafe for concurrent jobs — CLOSED, decisions/audit-phase1.md#no-lease-stealing
When `/session/open` returns 409, the homesrv proxy assumes the active lease is stale and closes it. A second
legitimate job can terminate the first job's model. `heartbeat_session()` exists but is not used by the
pipeline.
### P1: repeated model cold starts
A normal chapter can load Gemma separately for:
1. roster;
2. vision;
3. identity adjudication;
4. reconcile;
5. dialogue;
6. direction;
7. script.
The source itself notes that each load takes tens of seconds to minutes. Removing approximately five of
these loads is likely the largest safe wall-time improvement.
### P1: repeated transfers and encoding
- `/direct/window` downloads every image once for grouping and again for shot design.
- Identity resolver calls repeatedly download the same candidate reference images.
- Each local image is base64-expanded into the JSON sent to the local llama-server.
- Dialogue overlap causes repeated download and CPU set-of-mark work for overlap panels.
### P1: per-item GPU and database work
- SigLIP runs one forward pass per character crop rather than a bounded batch.
- Only one embedding URI is loaded per known character even when its reference gallery has several views.
- Scene construction makes one homesrv-to-workpc HTTP call per panel for a cheap JSON join.
- Many stage loops repeatedly open SQLite connections and commit one row at a time.
## Performance and scheduling plan
### 1. Keep Gemma resident across model phases
Use this chapter schedule:
```text
CPU: fetch -> crop
Gemma: roster -> vision
SigLIP: batched identity embeddings and shortlists
Gemma: identity adjudication -> reconcile -> dialogue -> direct -> CPU scene join -> script
Dots: TTS
Comfy: optional layers
```
This needs pipeline-owned model leases instead of every stage opening and closing its own lease. Scene
construction should run locally on homesrv or as a batch while the second Gemma lease remains open.
ComfyUI must be represented as a GPU resource too. It currently bypasses the session manager, so another
job could load Gemma, SigLIP, or Dots while ComfyUI is using the same GPU.
### 2. Batch SigLIP inference
For each request or chapter batch:
1. download a panel once;
2. clamp and crop all detected characters;
3. preprocess a bounded image batch;
4. run one model forward pass;
5. compare the resulting matrix with the known gallery in one vectorized operation.
Batch size should be tuned against VRAM rather than hard-coded optimistically.
### 3. Add safe bounded CPU parallelism
- Fetch/upload pages with concurrency 4-8.
- Crop framed pages with concurrency 2-4, buffer results, then insert them in page order.
- Keep whole-strip webtoon restitching sequential.
- Pre-download the next vision/dialogue window while Gemma processes the current window.
- Precompute face/text detections with a small CPU worker pool while avoiding ONNX thread oversubscription.
- Batch scene construction and SQLite reads/writes.
GPU calls should remain serialized initially. Multiple llama-server slots divide the configured context
among slots and need an explicit VRAM/context benchmark before enabling concurrent multimodal requests.
### 4. Eliminate repeated image movement
Add an ephemeral `/dev/shm` LRU keyed by S3 URI plus object version/etag. It is a cache, not durable worker
state, and may be dropped at any time.
Reuse downloaded image paths between the two direction passes and cache candidate reference images during
the identity phase. Configure llama-server with `--media-path /dev/shm` and send `file://` image paths
instead of base64 data URIs. The installed llama.cpp version supports local media paths and schema-constrained
responses.
Reference: [llama-server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
### 5. Use JSON-schema-constrained generation
Provide a JSON schema for detect, dialogue, roster, direction, same-person, and identity-resolution calls.
This should eliminate most malformed-output repair calls and make missing fields explicit. Keep semantic
validation after parsing because a structurally valid answer can still be wrong.
### 6. Reduce model call count
- Dialogue context should contain the current scene's recent 12-20 turns, not up to 400 lines from the
whole chapter.
- Reuse direction images first, then A/B test combining grouping and shot selection into one structured call.
- Replace the per-window LLM direction summary with deterministic structured continuity where possible.
- Generate 6-12 consecutive script beats per text-only call, grouped by scene, with stable beat IDs and
targeted per-beat repair.
## Character-recognition plan
The critical path keeps the current detector, SigLIP shortlisting, tracklets, and Gemma resolver.
### 1. Correct the tracklet spine
- Canonicalize gender as `m|f|unknown` and species as a validated enum everywhere.
- Add hard same-panel/co-presence cannot-links.
- Treat manual identity assignments as immutable constraints.
- Prevent transitive bridges from joining tracks whose hard constraints disagree.
- Give every detection a chapter-local provisional track ID, even if it never becomes a global character.
- Move provisional state out of the workpc process and into a resumable homesrv chapter artifact.
### 2. Improve visual evidence
- Validate and clamp all VLM bounding boxes before cropping.
- Match detected faces to person/body boxes with gated IoU and normalized distance assignment rather than
unconditional nearest-neighbour matching.
- Store both face and body crops.
- Score crop quality using face size, detector confidence, occlusion, and sharpness.
- Send Gemma the best 2-4 query views for a tracklet. The current resolver sends only one representative
query crop; its candidate union is not a real multi-view query gallery.
- Store multiple embedding vectors per global character and use gallery maximum or centroid similarity
only for shortlisting.
Face/body evidence plus local temporal constraints are supported by dedicated comic and manga re-ID work:
[Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification](https://arxiv.org/abs/2308.09096)
and [Unsupervised Manga Character Re-identification via Face-body and Spatial-temporal Associated Clustering](https://arxiv.org/abs/2204.04621).
### 3. Separate naming from identity
- Keep roster names as candidate vocabulary, never identity evidence.
- Detection may emit a typed name claim only when the panel visibly binds the name to that character.
- Require self-introduction, name-tag, or corroborated address evidence before promotion.
- Preserve aliases and honorific forms instead of stripping and discarding them.
- Never merge galleries on name equality alone.
- Resolve the tracklet first, then attach corroborated names to its global identity.
### 4. Add global constraints
Identity resolution should enforce:
- must-link membership within a validated tracklet;
- cannot-link for co-present characters;
- gender and species hard gates;
- manual locks;
- at most one appearance of a global character per panel;
- open-set outcomes `known|new|unresolved` without deleting the chapter-local track ID.
Gemma should receive a compact labelled identity board containing several query views and selected candidate
gallery views, rather than up to fifteen separately downloaded image parts for every representative crop.
### 5. Improve identity evaluation
Extend the existing labels and evaluator to report:
- false merges versus false splits;
- tracklet purity and fragmentation;
- global identity accuracy;
- name-binding accuracy;
- unknown/NONE rate;
- speaker accuracy conditioned on identity correctness.
A replacement embedding backbone should only be considered if the structural fixes plateau, using the
same labeled chapters for a controlled A/B comparison.
## Story-coherence plan
### 1. Build an evidence ledger before prose
Create a deterministic artifact for every beat:
```text
beat
-> ordered panels
-> ordered actions and chronology
-> dialogue/region IDs and exact text
-> typed speaker references, confidence, and method
-> characters present
-> location and time
-> named entities
-> uncertainty flags
```
This becomes the only factual input to script generation and validation.
### 2. Make context scene-scoped and evidence-bearing
- Reset speaker-turn context on strong location/time/scene changes.
- Keep a short structured story state rather than repeatedly summarizing generated prose.
- Give a script chunk only facts that occur before or inside that chunk.
- Do not pass a whole-chapter premise as if it were already-known story state; it can leak future events.
- Keep stable chapter-local handles for unresolved recurring characters so they do not disappear from the
scene graph while awaiting global promotion.
### 3. Generate narration in ordered scene chunks
Generate several consecutive beats in one text-only call and return:
```json
{"beats":[{"beat_id":"b01","text":"...","evidence_ids":["p001_r1"]}]}
```
This provides more continuity and fewer calls than one independent request per beat while keeping output
small enough for targeted validation and repair.
### 4. Respect uncertainty
Low-confidence or weak-method speaker assignments should become `someone`, an unattributed quote, or a
review flag instead of a confident named assertion. Identity and speaker uncertainty must survive into
the script prompt.
### 5. Strengthen verification
Validate:
- exact/fuzzy quote grounding;
- permitted character and entity names;
- action coverage and unsupported actions;
- event chronology;
- future-fact leakage;
- repeated facts and repeated sentence openings;
- evidence IDs returned by the model.
Feed the exact failures into the repair prompt. A final text-only consistency audit may report affected
beat IDs, but should not rewrite the whole chapter automatically.
## Stage clearing and resumability
Before parallel or identity work, stage ownership of shared data needs to be explicit:
- dialogue and direction currently mutate the shared vision JSON;
- clearing dialogue does not remove its keys, so a rerun can treat old dialogue as completed;
- reconcile merges are destructive and are not reversed by clearing the reconcile stage;
- clearing identity preserves the per-manga registry, so it is not a clean identity rerun;
- name claims attached to a merged-away character are not repointed with the identity assignments.
Use stage-specific artifacts or explicit key deletion, and preserve provenance through merges. A stage must
be genuinely idempotent before it is scheduled concurrently or resumed automatically.
## Further design ideas
### 1. Make identity merges reversible
Reconciliation currently deletes the losing character. Replace destructive merging with a versioned
cluster or redirect model:
```text
character_cluster
canonical_id
member_ids[]
merge_evidence[]
cannot_link[]
version
```
Assignments resolve through the canonical cluster while the original identities remain recoverable. A
reviewer can remove one bad merge edge without clearing and rebuilding the entire identity stage.
Preserve negative evidence as a first-class artifact. Co-presence is particularly valuable: if A and B
appear together as distinct detections, they are a permanent cannot-link unless a reviewer overrides it.
### 2. Separate detection, identity, and naming confidence
These are different questions and should never share one confidence value:
- detection confidence: is this actually a story character?
- identity confidence: which recurring visual identity is it?
- naming confidence: what is that identity's canonical name?
A person may be confidently detected and tracked as `chapter_person_3` while remaining unnamed. That is
enough for coherent narration such as "the colleague" without contaminating the global registry.
### 3. Use an uncertainty-driven identity cascade
Spend computation according to ambiguity and downstream impact:
```text
hard constraints resolve the case
-> accept cheaply
strong tracklet plus clear gallery margin
-> accept without Gemma
ambiguous candidates
-> multi-view Gemma resolution
high-impact unresolved identity
-> human review
one-off background person
-> chapter-local handle only
```
High-impact detections are recurring, speaking, named, or referenced by several future beats. A one-panel
extra should not consume the same identity budget as a protagonist.
### 4. Add model-aware scheduling across jobs
Strict chapter-by-chapter execution causes extra model swaps when several jobs are queued. A resource-aware
scheduler can group ready work by resident model:
```text
ready Gemma detect phases
-> ready SigLIP phases
-> ready Gemma post-identity phases
-> ready Dots phases
```
This improves total throughput at the cost of some per-job latency. Add fairness and maximum-wait limits so
one long title cannot starve other jobs. GPU conflicts must wait in a queue; they must never terminate the
active owner's lease.
### 5. Cache artifacts by dependency fingerprint
Replace broad "clear stage X and everything downstream" behavior with content-addressed artifacts keyed by:
```text
hash(input URIs
+ relevant upstream JSON
+ model version
+ prompt/schema version
+ stage settings)
```
Consequences:
- changing narration style does not rerun vision;
- correcting one identity invalidates only affected scenes and beats;
- changing direction does not invalidate dialogue;
- a resume reuses every artifact whose dependencies still match;
- prompt experiments can run side-by-side instead of overwriting the baseline.
Store dependency and version metadata beside every stage artifact so invalidation is explainable in the
review UI.
### 6. Detect repeated artwork before expensive stages
Use CPU-side perceptual hashing after masking detected text regions:
- same artwork and same text: reuse visual and dialogue artifacts;
- same artwork but different text: reuse detection/identity, rerun dialogue;
- near-duplicate establishing shot: mark `visual_repeat_of` so narration does not describe it again;
- exact duplicate with no new information: curate it out early.
Text masking is required because two visually identical panels may contain different dialogue. Combine the
art hash with text-region geometry and a text-content hash when available.
### 7. Make the manga registry temporal
Names, aliases, relationships, and facts need an evidence position:
```text
known_from: chapter 12, panel 37
source: self_intro
confidence: 0.97
```
The script can then use only knowledge available at that point. This prevents later chapters or out-of-order
processing from leaking names and revelations into earlier recaps. A configurable `spoiler_safe` versus
`omniscient_recap` policy can decide whether later knowledge may backfill earlier narration.
The sampled chapter premise needs the same guard because a synopsis built from the whole chapter can reveal
its payoff in the opening beat.
### 8. Represent dialogue as a relation graph
A spoken name often identifies the addressee or a third party rather than the speaker. Store:
```text
speaker_ref
addressed_character_ref
mentioned_character_refs[]
spoken_text
name_surface_form
honorific
region_id
speaker_evidence
```
For example, "Seonho-oppa!" is evidence about the addressee and their relationship, not the speaker's name.
Honorifics should be normalized but preserved as evidence instead of being stripped and discarded.
### 9. Treat speaker attribution as constrained assignment
For every speech region, build candidates from:
- bubble-tail or geometric evidence;
- visible faces and bodies;
- previous and following speaker turns;
- chapter-local identities;
- the off-panel cast;
- same-panel cannot-links.
Assign speakers jointly across a conversation window rather than deciding every line independently. Use
Gemma only to adjudicate ambiguous regions. Require `region_id` in the model output so numbered set-of-mark
regions are structurally connected to returned dialogue instead of remaining visual suggestions.
### 10. Separate stable identity traits from scene appearance
Store durable and temporary appearance separately:
```text
identity_appearance:
face, hair, species, stable marks
scene_appearance:
outfit, accessories, injuries, disguise, chibi/age form
```
Clothing can be strong evidence inside one local scene but should be down-weighted across scene or time
changes. This also lets narration mention an outfit change without creating a new global character.
### 11. Add metamorphic pipeline tests
Test invariants rather than only fixed outputs:
- changing dialogue window size must not move text to another panel;
- inserting a scenery panel must not change neighboring identities;
- resuming halfway through identity must match a clean run;
- renumbering panel-local IDs must not change global identity;
- distinct co-present characters must never merge;
- changing narration style must not change facts or quotes;
- processing chapters in a different order must not silently rewrite prior identities.
These tests exercise cross-stage contracts and resumability failures that per-file self-checks cannot catch.
### 12. Roll out identity changes in shadow mode
Write new identity results to a versioned shadow artifact without affecting scripts or the canonical registry.
The review UI can compare:
```text
current: Choi Haeseon
candidate: unresolved
reason: co-presence conflict
```
Promote a new resolver only after it improves the labeled-chapter metrics. Shadow mode prevents tuning work
from polluting the production gallery.
The highest-value additions beyond the core plan are reversible identity clusters with cannot-links,
dependency-fingerprinted artifacts, and uncertainty-driven review for recurring/speaking/named characters.
## Implementation sequence for approval
### Phase 1: correctness and scheduling safety — DONE 2026-08-11
Implemented and verified by CPU-only self-checks. Nothing ran on the GPU. What landed, with evidence
and the check that covers it, is in `decisions/audit-phase1.md`.
Two items from this list were NOT implemented and are recorded with a revisit trigger instead:
honest stage clearing (`caveats/audit-open.md#dishonest-clearing`) and ComfyUI under the GPU scheduler
(`caveats/audit-open.md#comfyui-unscheduled`). Both need a design decision, not a patch.
### Phase 2: safe throughput improvements
Now tracked in `ROADMAP.md`, with the done-when condition for each phase.
Verification: unit checks first, followed by one labeled chapter after GPU availability is approved.
### Phases 3 and 4
Now tracked in `ROADMAP.md`.
## Acceptance criteria
Now tracked in `ROADMAP.md`, together with the approval boundary: measure Phase 2 before deciding
whether the larger identity and narration changes are worth their size.
## Second-pass findings (2026-08-11)
Added after reading the workers and the orchestrator against each other. Every item was read in source and
is cited by file and line. Nothing was executed and no GPU work was run.
The first pass audited each side on its own terms. Most of what follows lives in the seam between them:
one repo changed a field's meaning and the other still reads the old one. The existing per-file
`__main__` self-checks cannot catch any of it, because each one asserts its own side of the contract.
### P0: the scene stage discards every speaker — CLOSED, decisions/audit-phase1.md#speaker-ref-is-canonical
`normalize_dialogue` rewrites `row["speaker"]` to a `character_id` or `None`
(`orchestrator/correctness.py:47`). `run_stage_dialogue` saves that shape into the vision blob
(`orchestrator/service.py:1159-1160`). `worker_scene.build_scene` still reads that field as a panel-local id
and maps it through `id_by_local` (`worker_scene.py:78`). A `character_id` is never a key in that map, so
every lookup returns `None`.
Every scene graph therefore reaches the script worker with `speaker: None`, and `_render_line` writes
`Someone says "..."` for all of it (`worker_script.py:48`). Set-of-mark face attribution, the solo-speaker
backstop, and window turn-taking all compute the right answer and then lose it one stage later.
The same field breaks the dialogue resume path: `service.py:1180` pushes the stored value into `recent` as a
speaker NAME, so a resumed run feeds raw strings like `character_ab12cd34` to the transcription prompt as if
they were people.
`worker_scene`'s self-check still passes because it feeds the pre-change contract
(`worker_scene.py:138`). Fix: prefer `speaker_ref` when its kind is `character_id`, keep the local-id path as
the fallback. This is the strongest argument for the cross-stage metamorphic tests proposed above.
### P0: the script verifier rejects correct narration, and its retry is a no-op — CLOSED, decisions/audit-phase1.md#verifier-false-positives
`verify_script` fails a beat on two rules that fire on valid output (`orchestrator/correctness.py:100-116`).
1. `unsupported-proper-noun` flags any capitalized token outside the cast, the entities, the beat's source
words, and a 20-word stop list. Ordinary sentence-initial words are not in that list: `Suddenly`,
`Behind`, `Inside`, `Both`, `Everyone`, `After`, `Two`. So is `Someone`, which the previous finding
guarantees the narrator will emit constantly.
2. `misquote` compares each quoted span against a WHOLE source line with `SequenceMatcher` at 0.82. The
narration prompt explicitly asks for a short quote of the actual words (`worker_script.py:130-132`). A
15-character quote taken from a 40-character line scores about 0.55 and fails. The prompt and the
verifier ask for opposite things.
The failure is not soft. `run_stage_script` retries once, then raises, so the beat produces no narration
(`service.py:1384-1386`). `done` never reaches `total`, `_finalize_stage` marks the stage failed, and
`_run_pipeline` stops the whole chapter. One false positive halts a run.
The retry is also inert for the reason already recorded above: `ScriptInput` defines neither `beat` nor
`verifier_feedback` (`worker_script.py:13-23`), and pydantic v2 ignores unknown fields by default, so both
are dropped without an error.
Fix order: correct the two rules first, then wire the feedback fields. Fixing the plumbing alone makes the
model retry against a broken oracle.
### P0: correctness flags block TTS permanently in the default configuration — CLOSED, decisions/audit-phase1.md#flag-resolution
`run_stage_tts` refuses to start while any unresolved flag of kind `script-verifier`, `ambiguous-speaker`,
`conflicting-name-claims`, or `partial-dialogue` exists (`service.py:1419-1423`). The only code that ever
resolves a flag is `resolve_correctness_flags`, called from `/review/approve` for the `script` gate
(`service.py:1955-1956`).
`GATES` defaults to off (`service.py:312`). The autonomous pipeline therefore has no path that clears a
flag. The first flag of those kinds wedges the chapter until a human calls an endpoint that the autonomous
mode never mentions. These flags are not rare: `partial-dialogue` is raised whenever a window response omits
one panel, and `dialogue_envelope` treats a legitimately silent panel inside a partial window as unresolved
(`correctness.py:62`), so one missing panel flags every wordless panel beside it.
There is a second, worse variant. Flag rows survive their panels. `get_correctness_flags` LEFT JOINs panels
and accepts rows where `p.panel_id IS NULL` (`db.py:573`), while `resolve_correctness_flags` can only resolve
flags whose panel still exists (`db.py:579`). Clearing the `crop` stage deletes panel rows
(`db.py:798-799`). Any flag raised against a deleted panel becomes visible to every chapter and can never be
resolved, blocking TTS for all future jobs.
Fix: resolve by flag identity rather than by surviving panel, and give the autonomous path an explicit
policy (auto-resolve below a rank, or fail the stage loudly) instead of an unreachable gate.
### P0: `awaiting_review` is immediately overwritten by `failed` — CLOSED, decisions/audit-phase1.md#flag-resolution
`run_stage_tts` sets `awaiting_review` and then raises (`service.py:1422-1423`). The raise unwinds into
`_run_pipeline`'s catch-all, which sets the job to `failed` (`service.py:391-393`). The review state the
stage just recorded is gone before anyone can read it, and the operator sees a generic failure rather than a
queue of flags. The gate path at `service.py:337-341` returns instead of raising and does not have this
problem.
### P1: set-of-mark face pairing is enabled by default, contrary to its own documentation — CLOSED, decisions/audit-phase1.md#gated-face-pairing
`worker_vision.py:22` states the feature is off by default and should be enabled per title once tuned.
`worker_vision.py:31` reads `SOM_ATTRIBUTION` with a default of `"1"`. It is on.
That matters because of the unconditional pairing already recorded as a P0. The mispairing does not stay
local. `_set_of_mark` writes the wrong name into the legend the model reads (`worker_vision.py:99-104`), the
model attributes a line to that face label, and `_apply_speaker_labels` converts the label back into a
`local_id` and stamps `speaker_method = "som_face"` (`worker_vision.py:117`). A geometric guess is laundered
into the highest-trust provenance value the system has. Gate the pairing before trusting that label, or
default the flag to off as documented.
The pairing is also greedy in face order rather than a joint assignment (`worker_vision.py:42-52`), so the
first face processed can claim a character that fits a later face far better.
### P1: the JSON repair pass can fabricate content — OPEN, caveats/audit-open.md#repair-fabricates
When a response fails to parse, `call_gemma4_json` sends the model its own truncated text and asks for the
JSON it should have been (`worker_vision.py:174-178`). The repair call carries no image. On a response
truncated by `max_tokens`, the model is being asked to complete dialogue it can no longer see. Anything it
adds is invented and is indistinguishable downstream from transcribed text.
Schema-constrained generation, already recommended above, removes most of this path. Until then the repair
pass should re-send the image, or a truncated response should be retried rather than repaired.
### P1: `/review/preview` silently replaces a beat clip — OPEN, caveats/audit-open.md#preview-overwrites-clip
`review_preview` renders one panel alone and calls `save_clip(panel_id, ...)` (`service.py:1902-1904`). Its
docstring calls this harmless because assemble would regenerate it. It does not. `_render_one_beat` returns
early when a clip already exists for the leader (`service.py:1591-1592`). Previewing a beat leader therefore
pins the solo preview into the final video and drops the rest of the beat's panels. `review_retts` gets this
right by calling `delete_clip` first (`service.py:2008`). Preview should write to a scratch key or delete the
clip row afterwards.
### P1: the session manager can orphan a llama-server and hold the GPU — CLOSED, decisions/audit-phase1.md#unlocked-model-load
`open_session` claims the slot, releases the lock, then spawns and health-waits for up to 300 seconds
(`session_manager.py:108-119`). A `/session/close` arriving during that window finds `proc = None`, tears
down nothing, and clears `_active`. The process that finishes starting afterwards is unreferenced and keeps
its VRAM until someone kills it by hand. The next `open` spawns a second server on the same port, and
`_health_wait` cannot tell the two apart because it only probes the port
(`session_manager.py:51-60`).
Related asymmetry: `_supervise_once` calls `_start_subprocess` while holding `_lock`
(`session_manager.py:169-181`), so a respawn blocks `/session/active`, `/session/close`, and `/session/open`
for the full health wait. The open path was deliberately written to avoid exactly this.
### P1: nothing limits concurrent jobs — CLOSED, decisions/audit-phase1.md#no-lease-stealing
`_background_jobs` accepts any number of pipelines (`service.py:302-303`, `service.py:427-428`). Each opens
its own model sessions. Combined with the 409 lease stealing already recorded, two jobs terminate each
other's models rather than queueing. The queue has to live at the orchestrator as job admission control. A
fix inside `session_proxy` alone still lets two pipelines interleave stages against one GPU.
### P1: two cheap throughput wins the plan does not name
**Order the shared prefix first so llama-server reuses its KV cache.** `/direct/window` sends the same
panel images twice, once for grouping and once for shot design (`worker_vision.py:702-733`). The audit
above frames this as duplicate downloads. The larger cost is the second vision-encoder prefill on the GPU.
llama-server reuses the longest common prompt prefix across requests, and the two calls currently differ in
their first token because the instruction text precedes the images. Putting the images first and the
differing instruction last makes the second pass nearly free on prefill. The same reordering helps any
stage that issues several calls over one image set.
**Give stored embeddings a version tag.** `embed_crop` uses `pooler_output` when present and silently falls
back to mean-pooled patch tokens otherwise (`worker_identity.py:48-52`). Those two paths produce different
vector spaces, and the fixed 0.85 threshold is only valid for one of them. Nothing recorded beside a stored
`.npy` says which model, revision, or pooling produced it, so a transformers upgrade mixes incompatible
vectors into one gallery with no error. Write the model id and pooling mode next to the vector and refuse to
compare across versions.
### P2: smaller confirmed defects — partly closed, see `decisions/audit-phase1.md#related` and `caveats/audit-open.md`
- An out-of-range `choice` from the resolver is mapped to NONE and then reported as `state: "new"`
(`worker_vision.py:898-899`), so a hallucinated index mints a brand new character. It should be
`unresolved`, like a parse failure.
- `_extract_json` matches greedily from the first `{` to the last `}` (`worker_vision.py:130`). Two objects
or any trailing braced prose produce an unparseable span and burn a repair call. `json.JSONDecoder().raw_decode`
from the first brace is exact.
- `worker_identity._known_cache` is invalidated only by `_persist_char` (`worker_identity.py:128-132`).
The reconcile stage deletes losing characters directly in the orchestrator database
(`db.py:506`), so a long-lived worker keeps shortlisting and assigning ids that no longer exist.
- `_pending` holds full crop images in worker memory for a whole chapter and is keyed by session
(`worker_identity.py:25`). This is durable per-chapter state inside a worker documented as stateless, it
is lost on restart, and a character seen exactly once receives no assignment at all, not even a
chapter-local handle.
- `get_conn` opens a connection per call with no `busy_timeout` (`db.py:190-197`). WAL tolerates one writer.
`PIPELINE=1` already writes clips from concurrent tasks while TTS writes audio, so the planned CPU
parallelism will surface as `database is locked` before it surfaces as throughput.
- Worker endpoints declared `async def` run blocking MinIO, OpenCV, torch, and ffmpeg calls directly on the
event loop (`worker_identity.py:177`, `worker_tts.py:170`, `worker_vision.py:230`). A busy worker cannot
answer `/health` or `/unload`. That makes `/health/workers` report a working worker as unreachable
(`service.py:206-216`) and puts the session manager's 30-second `/unload` at risk exactly when VRAM needs
freeing (`session_manager.py:93`). `def` instead of `async def` moves each to the threadpool.
- `layers` runs after `tts` in `STAGES` (`db.py:330-333`), while `run_stage_tts` warns that eager rendering
under `PIPELINE=1` needs layers to run first (`service.py:1427-1429`). With the flag on, solo beats always
render without parallax.
- Stage failure policy is inconsistent. A dropped vision panel fails the stage and halts the pipeline. A
failed direction window counts its panels as done (`service.py:1247`), and layers always finishes
completed (`service.py:1503`). `completed` does not mean the same thing across stages, which makes the
acceptance metrics below hard to read.
- `run_stage_assemble` does not check that `clip_uris` is non-empty before assembling and then marks the job
completed (`service.py:1661-1677`).
- The `/review/panels` timeline sums per-panel audio durations (`service.py:1750`), but assemble crossfades
clips using the per-beat transitions (`service.py:1671-1674`). Every non-`cut` transition shortens the
real video, so reviewer timestamps drift further out of sync the further into the chapter they scrub.
- MinIO credentials are hardcoded as defaults in committed source (`transport.py:95-99`,
`service.py:74-76`).
### What this changes in the plan — DONE
All five additions landed in Phase 1. See `decisions/audit-phase1.md`.
The acceptance metric they suggested, the share of narrated lines whose speaker is a named character
rather than `Someone`, is now in `ROADMAP.md`. It has not been measured yet.
CLAUDE.md
+65 -52
View File
@@ -1,67 +1,80 @@
# CLAUDE.md # CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. Goal, invariants, and working rules. Read this first.
## What this is | file | holds |
| --- | --- |
| `NEXT.md` | the current state and the live plan |
| `ROADMAP.md` | the ordered outcomes past the current one |
| `JOURNAL.md` | what was run and when, append-only |
| `decisions/` | every settled question, indexed in `decisions/CLAUDE.md` |
| `caveats/` | every known limit and its revisit trigger, indexed in `caveats/CLAUDE.md` |
| `AGENTS.md` | commands, with the traps beside them |
| `AUDIT.md` | the 2026-08-11 pipeline audit, the source of the roadmap |
| `ARCHITECTURE.md` | the target shape of the pipeline, and what exists against it today |
| `spec-v3.md` | current quality and look work, marked DONE/TODO per item |
The **workpc compute half** of a manga→narrated-video pipeline. This repo holds stateless Do not restate a finding here. Point at the decision.
GPU/CPU workers only. State, job scheduling, and stage orchestration live in a **separate
homesrv orchestrator repo** (`/mnt/server/home/kami/apps/Maven/` region) — not here. The two
communicate over a fixed HTTP contract; never add sqlite or durable state to a worker.
Machine split (memorize): workers run on **workpc** (RX 7900 GRE, ROCm). MinIO + orchestrator ## The goal
run on **homesrv** (`192.168.1.104`, CPU-only). Data dir `/mnt/server/home/kami/` is an SSHFS
mount of homesrv.
## Run / test This is the **workpc compute half** of a manga to narrated-video pipeline. It holds stateless GPU and
CPU workers only. State, job scheduling, and stage orchestration live in a separate homesrv
orchestrator repo (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`). The two talk over a
fixed HTTP contract. The hard part is that one GPU serves every model, so the schedule, not the model,
sets the wall time.
Machine split: workers run on **workpc** (RX 7900 GRE, ROCm). MinIO and the orchestrator run on
**homesrv** (`192.168.1.104`, CPU-only). `/mnt/server/home/kami/` is an SSHFS mount of homesrv.
## Invariants
1. **No durable state in a worker.** No sqlite, no cross-request memory. A worker pulls inputs from
MinIO by URI, does one stage, pushes outputs back, returns URIs. A `/dev/shm` cache is allowed
because it may be dropped at any time.
2. **One warm model at a time.** Every GPU stage takes a lease from `session_manager.py` on 8095. A 409
is a queue signal, never a stale lease
(`decisions/audit-phase1.md#no-lease-stealing`).
3. **Never hold `_lock` across a model load or a health wait** in `session_manager.py`
(`decisions/audit-phase1.md#unlocked-model-load`).
4. **A dialogue row's speaker is `speaker_ref`.** The flat `speaker` field is a compatibility value and
holds a `character_id`, not a panel-local id
(`decisions/audit-phase1.md#speaker-ref-is-canonical`).
5. **A stage never raises after setting `awaiting_review`.** The pipeline's catch-all overwrites it
with `failed` (`decisions/audit-phase1.md#flag-resolution`).
6. **Never mint a character from an unparseable or out-of-range model answer.** That is `unresolved`
(`decisions/audit-phase1.md#hallucinated-index`).
7. **The HTTP contract with the orchestrator is load-bearing.** Changing a worker's request or response
shape means reconciling the orchestrator in the same session. Neither repo's self-checks can catch a
contract break, because each asserts its own side.
8. **`ponytail:` comments mark deliberate simplifications** and name the upgrade path. Respect them.
9. **Nothing that positions an ffmpeg filter may use `format=duration`.** It reports `max(video, audio)`.
It hides A/V drift, and it walks xfade offsets past the end of their input. ffmpeg then discards clips
and still exits 0 (`decisions/chapter-assembly.md#offsets-from-min-stream`).
## Working rules
```bash ```bash
./start_workers.sh # dev: session_manager + 9 workers, each a uvicorn in a tmux window ./start_workers.sh # dev: session_manager + 9 workers, each a uvicorn in a tmux window
tmux attach -t manga-workers # watch logs; per-worker window tmux attach -t manga-workers # per-worker logs
sudo systemd/install.sh # production: one systemd unit per process (User=kami) sudo systemd/install.sh # production: one systemd unit per process (User=kami)
python session_manager.py # each module has a __main__ self-check (assert-based, no framework) .venv/bin/python worker_scene.py # every module has an assert-based __main__ self-check
python transport.py # run these to verify a file after editing it .venv/bin/python test_vision_parse.py
python test_vision_parse.py # the one standalone pytest-free test ruff check . # must exit 0; every ignore in ruff.toml carries its reason
cd /mnt/server/home/kami/docker-apps/manga-infra/orchestrator && pytest -q --ignore=test_api.py
``` ```
There is no lint/build step. `.venv` is the ROCm torch env; workers import `transport` by module name. There is no build step. `.venv` is the ROCm torch env. Workers import `transport` by module
name. Ports: crop 8000, vision 8002, identity 8003, scene 8004, script 8005, tts 8006, layers 8007,
render 8008, session_manager 8095.
## Architecture Every output S3 URI is a template in `transport.py`, not an f-string in a worker. Add one there when a
new artifact class appears.
**Workers are stateless HTTP stages.** Each `worker_*.py` is a FastAPI app on a fixed port. It - Any non-trivial logic gets ONE runnable check in `__main__`, assert-based, no framework. Run the file
pulls inputs from MinIO by URI to local disk (`/dev/shm`), does one stage, pushes outputs back, to verify it.
returns URIs. No cross-request memory. Ports: crop 8000, vision 8002, identity 8003, scene 8004, - Editing a worker's request or response shape means editing the orchestrator too, in the same session.
script 8005, tts 8006, layers 8007, render 8008, **session_manager 8095**. - Update `NEXT.md` alongside any change that moves the plan, and append to `JOURNAL.md` after a run.
- Do not run GPU work or a full pipeline without asking.
**`transport.py`** — shared MinIO client (`get`/`put`/`put_bytes`/`exists`) + `install_logging(app, name)`
(one log line per request with panel id). URIs are `s3://bucket/key`. Import it in every worker.
**`session_manager.py`** — the **GPU mutex**. Only one warm model at a time on the single local GPU.
Two model kinds:
- *subprocess* (`gemma4`): it spawns/health-waits/terminates `llama-server`, and a supervisor
respawns it in-place if it crashes mid-session (keeps the same session_id + port).
- *in-process* (`siglip2`, `dots`): returns `port=None`; the **worker** loads the transformers model
itself and must expose `/unload` so `/session/close` can free the ~5GB VRAM (the mutex alone
can't reclaim it). Leases have a TTL + heartbeat; a reaper force-closes stale ones.
A GPU worker's flow: `/session/open {model}` → (409 if busy) → do work against the returned port or
its own resident model → `/session/close`. CPU workers (crop) take no session.
## Conventions
- **`ponytail:` comments** mark deliberate simplifications and name the upgrade path — respect them,
don't "fix" them without reason.
- Any non-trivial logic gets ONE runnable check in `__main__` (assert-based `demo`/self-check), not a
test suite. Follow that pattern; run the file to verify.
- The HTTP contract with the orchestrator is load-bearing and shared across repos — changing a
worker's request/response shape means reconciling the orchestrator too (see commit history:
"reconcile worker contracts").
## Specs & docs
- `spec-v3.md` — current quality/look work (narration voice, panel curation, render overhaul), marked
DONE/TODO per item. `legacy/` holds the old single-repo workers.
- `AGENTS.md` — use `scripts/analyze_video_frames.sh` (not manual seeking) to diagnose render output;
its frames are `/tmp` artifacts, never commit them.
- `collage.py` (behind `COLLAGE` flag) and the render worker carry the animated-layout planners.
HANDOFF.md
+95 -64
View File
@@ -1,75 +1,106 @@
# HANDOFF: repo reconstructed from agent transcripts (2026-08-11) # HANDOFF, 2026-08-13 (eighth session)
## What was asked Live state is in `NEXT.md`. This file is only what this session did. The previous handoff is in
The working copy at `/home/kami/Programs/n8n-worker` was deleted with `rm`, including `.git`. `JOURNAL.md`.
This repo is the workpc GPU-compute half of the manga pipeline. The ask was to rebuild it from
the transcripts on disk, at minimum the transport layer and the pipeline flow.
## Sources used ## Asked
No backup and no remote survived. `~/.local/share/Trash` is empty. No copy exists under
`/home/kami`, `/mnt/D`, or `/mnt/server`. Gitea
(`/mnt/server/mnt/hdd2/gitea/git/repositories/`) holds only correx, hexis, maven,
model-training, muzick, orchestra, and test-e2e. There is no `n8n-worker` mirror.
Reconstruction replays three histories into one timeline, ordered by timestamp: Push the repo to Gitea and open a PR to `master`. Then, from the PR review: fix the s3 URI constants,
run ruff and fix what it finds, push, run a ponytail audit, fix, push, merge. Then audit.
1. Claude Code transcripts, 25 sessions in ## Result
`~/.claude/projects/-home-kami-Programs-n8n-worker/*.jsonl`. Contributes Write and Edit
content, full-file Read results, and `@`-mention attachments.
2. Codex rollouts, 5 sessions with `cwd=/home/kami/Programs/n8n-worker` under
`~/.codex/sessions/2026/07/{13,15,16,18}/rollout-*.jsonl`. Contributes 26 `apply_patch`
blocks. The 4 that the log marks `Script failed` are skipped. These carry the 2026-07-18
correctness work (`_dialogue_envelope`, `_normalize_claims`, `_annotate_speaker_methods`,
`SOM_ATTRIBUTION=1`) that exists in no Claude transcript.
3. `~/.claude/file-history/<session>/<hash>@vN` pre-edit blobs, used as cross-checks only.
Scripts live in No GPU work. Nothing ran on a pipeline stage. PR #1 is open at
`/tmp/claude-1000/-home-kami-Programs-n8n-worker/6a35d5d8-9f44-4412-955c-9cf088737ad5/scratchpad/`. `https://gitea.kvmx.ru/kami/manga-recap-pipeline/pulls/1`, `master` <- `restore-runtime`.
`replay2.py` is the merged replay. `codex.py` extracts and applies codex patches. `backups.py`
compares against file-history. `recon_final/_log.json` holds the per-file event log.
## Verification `master` held only the reconstruction commit `ff6a512`. All 27 commits of real work sat unpushed on
- Replayed to 2026-07-18T13:13:44Z, line counts match the `wc -l` recorded in the transcript at `restore-runtime`.
that moment, for all 10 files it covered. crop 326, identity 314, layers 99, render 996,
scene 159, script 300, tts 231, vision 871, session_manager 238, transport 216.
- Byte sizes match the `ls -l` recorded by the same command. AGENTS.md 867, CLAUDE.md 3678,
manga-recap-pipeline-spec.md 11413, plan.md 16511, spec-v3.md 26170. `.gitignore` is 53.
- 18 files are byte-identical to their newest file-history blob. The 4 that differ (render,
script, tts, spec-v3) differ only by edits made after that blob was taken.
- `python3 -m py_compile *.py scripts/*.py attic/*.py` passes. `bash -n` passes on all 3 shell
scripts. Runtime self-checks such as `python worker_vision.py` were NOT run, because `.venv/`
is gone.
## Recovered (live tree) ## Landed
Workers `worker_{crop,vision,identity,scene,script,tts,layers,render}.py`, the GPU mutex
`session_manager.py`, `transport.py`, helpers `bubble_detect.py`, `face_detect.py`,
`collage.py`, `test_vision_parse.py`, `start_workers.sh`, `systemd/install.sh`,
`scripts/{analyze_video_frames.sh,pick_tts_voice.py}`, `requirements.txt`, `.gitignore`,
and the docs `AGENTS.md`, `CLAUDE.md`, `plan.md`, `spec-v3.md`, `spec-correctness.md`,
`manga-recap-pipeline-spec.md`, `identity-rework-task.md`.
`attic/` holds files that had been deleted from the tree before the `rm`. They are kept but not | commit | what |
live: `worker_ocr.py`, `worker_parse.py`, `plan-workpc.md`, `char-recognition.md`. | --- | --- |
| `12d4ed8` | the `ARCHITECTURE.md` rewrite, which was finished but uncommitted |
| `be63b22` | deleted a 0-byte file named `II, d[i+4:i+12]))\n"; done`, added by `8113bdf` |
| `bec9411` | s3 URI templates into `transport.py`, plus `ruff.toml` and the lint fixes |
## Still open ### S3 URIs
- `.venv/` is gone. Rebuild with
`python -m venv .venv && .venv/bin/pip install -r requirements.txt`. The ROCm torch wheel is Eight templates now live in `transport.py`: `PANEL_URI`, `PAGE_PANEL_URI`, `AUDIO_URI`,
not pinned in requirements.txt, so install it the way workpc had it. `AUDIO_FLAT_URI`, `LAYER_URI`, `CLIP_URI`, `CHAPTER_URI`, `CHAR_PNG_URI`, `CHAR_NPY_URI`. Five workers
- `dots.tts/` (gitignored external checkout) and `legacy/` (pre-rewrite workers, moved there formatted their own before.
2026-07-13) are not recoverable from transcripts. Re-clone dots.tts if TTS is needed.
- `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable. Neither agent `transport.ids_from_uri` replaces three separate copies of the same parse in `worker_tts`,
ever read them in full. All three were already deleted from the working tree as of 2026-07-17. `worker_layers` and `worker_render`. It raises on a uri too short to carry the ids instead of returning
- Git history is gone. A fresh `git init` plus one commit replaces it. Old commits are not a wrong pair. `worker_render._mc_from_uri` is gone, its 6 call sites repointed.
recoverable.
- Two replay gaps were left unpatched. One Edit MISS in `worker_render.py` on 2026-07-14, and one ### Lint
orphan Edit against `attic/plan-workpc.md` on 2026-07-13. Both files were later replaced by a
full-file snapshot, so the final content stays anchored. `ruff check .` exits 0. Ruff's defaults found 115. Fixed: implicit `Optional` in 8 signatures, an
- Edits made outside Claude and codex after 2026-07-18T14:35Z (the last recorded write) cannot be unparenthesized implicit concatenation inside the ASS filter list, 5 `subprocess.run` calls now saying
detected. `unknown:` whether any exist. `check=False` out loud, 1 unused import, 1 duplicate exception handler, 1 non-executable shebang,
4 `dict()` calls and 2 `startswith` chains.
12 rules are off in `ruff.toml`, each with its reason. The ASYNC ones matter: ffmpeg and ffprobe run
synchronously inside `async def` endpoints, so a busy worker cannot answer `/health`. That is already
`caveats/audit-open.md#blocking-event-loop`, tracked as [#199], and it needs a per-handler refactor
rather than a lint fix.
`CLAUDE.md` said "There is no lint or build step". It now names `ruff check .` and the rule that every
output URI is a template in `transport.py`.
## Audit findings
Applied:
- `delete:` `attic/worker_ocr.py` and `attic/worker_parse.py`, 224 lines, imported by nothing and named
in no doc. `attic/char-recognition.md` and `attic/plan-workpc.md` kept, they are design history.
- `delete:` `worker_vision._panel_size`, 8 lines, one reference and it is the definition.
Found and NOT applied, in order of size:
- `shrink:` `call_gemma4`, `_extract_json` and `_strip_thought` each exist twice, in `worker_vision.py`
and `worker_script.py`, and **have already diverged**. `worker_vision.call_gemma4` is 21 lines and
takes a content list, `worker_script`'s is 8 and takes a prompt string plus a system prompt. The two
`_extract_json` bodies carry the same comment about `raw_decode` but different error text. This is
the live risk: `caveats/audit-open.md#repair-fabricates` says the JSON repair pass can fabricate
dialogue, and a fix would land in one copy. A shared `gemma.py` costs one new file and removes about
25 duplicated lines. Not done because it touches the two largest workers and nothing has run on a GPU
since.
- `stdlib:` `worker_render._wrap2`, 16 lines of greedy word-wrap with a 2-line cap, is close to
`textwrap.wrap(text, width, max_lines=2, placeholder="…")`. Not identical: `_wrap2` does
`rstrip(".,")` before the ellipsis, and textwrap reserves width for the placeholder, so the break
points differ. It renders burned-in subtitles, so the diff is user-visible. Verify against the render
self-check before swapping.
- `shrink:` `_letterbox` and `_load` are duplicated between `bubble_detect.py` and `face_detect.py`,
two ONNX detectors with the same preprocessing. About 16 lines.
- `delete:` `worker_layers.py:14` points `LAYERED_WORKFLOW` at `legacy/qwen_layered_workflow.json`.
`legacy/` was **never tracked in git** and is absent from disk, so the module's self-check has never
been able to pass here. This is the other half of `caveats/audit-open.md#layers-writes-nothing`.
## Checks
```bash
ruff check . # All checks passed
.venv/bin/python worker_render.py # ok, ffmpeg ran, about 4 minutes
```
Self-checks pass: `transport`, `collage`, `bubble_detect`, `test_vision_parse`, `worker_crop`,
`worker_scene`, `worker_script`, `worker_identity`, `worker_tts`, `session_manager`, `worker_vision`,
`worker_render`, `face_detect`.
`worker_layers` fails, and did before this branch, on the missing `legacy/` file above.
## Next command ## Next command
```
cd /home/kami/Programs/n8n-worker The fifth GPU cycle is still the next pipeline work. It is blocked only on the GPU being free. The
python -m venv .venv && .venv/bin/pip install -r requirements.txt exact sequence is in `JOURNAL.md` under the seventh session, and `NEXT.md` item 1 holds the
.venv/bin/python worker_vision.py # per-file __main__ self-checks, then ./start_workers.sh expectations. **Clear the panels prefix first** or the wired caption merge
``` will not take effect.
Before that, or instead of it while the GPU is busy, `ARCHITECTURE.md` step 0 is the measurement spine:
`eval/chapter-truth.json` plus purity and fragmentation printed by `audit_registry.py`.
One ordering trap, unresolved: the truth file cannot be keyed on `panel_id` or `character_id`. The
fifth cycle re-crops and calls `/characters/reset`, which destroys both. Key it on page-space geometry,
or write it after the fifth cycle rather than before.
JOURNAL.md
+758
View File
@@ -0,0 +1,758 @@
# JOURNAL
Append-only, newest last. One block per session or run. Not a changelog: this records what happened on
the day a number was produced, so a later postmortem can find it.
## 2026-08-11 Audit second pass [no task]
Command: none. Source reading only.
Outcome: finished. `AUDIT.md` grew from 565 to 771 lines with a `## Second-pass findings` section:
4 new P0, 6 new P1, 13 P2, 5 additions to the Phase 1 list.
Produced: commit `6d9df5b`, `AUDIT.md:566`.
## 2026-08-11 Audit Phase 1 implemented [#203]
Command: `python worker_scene.py worker_script.py worker_vision.py session_manager.py`,
`pytest -q --ignore=test_api.py` in the orchestrator.
Outcome: finished. All self-checks pass, 108 orchestrator tests pass. No GPU work, no pipeline run.
`test_api.py` was skipped because fastapi is not installed in the workpc venv.
Produced: `decisions/audit-phase1.md`, `caveats/audit-open.md`, `ROADMAP.md`, and this scaffold.
One existing test asserted the bug: `test_name_binding.test_conflict_flags_and_stays_unnamed` relied on
orphan flags leaking into every chapter, because it never created panel rows. It now creates them.
## 2026-08-11 Orchestrator half committed and deployed [no task]
Command: `pytest -q --ignore=test_api.py`, then `docker compose up -d --build orchestrator` on homesrv.
Outcome: finished. 108 tests pass. Commits `1c60710` (orchestrator half) and `94bd4d8` (minio pin) in
`/mnt/server/home/kami/docker-apps`. Orchestrator and minio both answer health on homesrv.
The rebuild recreated `minio` as a side effect and it crash-looped with `exec format error`: the
compose pin was the arm64 manifest digest of `minio/minio:latest` and homesrv is amd64. Repinned to the
amd64 digest. Nothing about Phase 1 caused this, but any compose action that recreates minio would have
hit it, so it was latent, not new.
Still unrun against a real chapter.
## 2026-08-11 S3 viewer and storage swap, tasks #116/#117 [#116 #117]
Command: docker compose on homesrv, `dig`, `openssl s_client`. No pipeline, no GPU.
Outcome: partial. Viewer works, storage swap staged and unfinished.
#117 needed no new software. `stowage` at `~/docker-apps/stowage` was already configured against the
manga MinIO and had been dead since 2026-07-19 with `exec /sbin/tini: exec format error`: its digest
pin was the arm64 manifest. Repinned to amd64 `sha256:91be7f13`, chowned `data/` to uid 65532 for the
new image, and it serves. MinIO had the identical bug, repinned to `sha256:a1a8bd4a`. A sweep of all
470 local images on homesrv found exactly those two arm64; nothing else in the homelab is affected.
#116 is staged, not done. `rustfs` runs alongside MinIO on `127.0.0.1:9010/9011`, pinned
`sha256:19b105cc`, data at `/mnt/hdd2/rustfs`. Buckets are empty: the `mc` mirror of
`audio layers manga panels raw video` (350M, all in `manga`) has NOT run. `/mnt/hdd2/minio/data` is
untouched and is the rollback. RustFS is `1.0.0-beta.12`, labeled `build-type=prerelease`. Cutover
would give rustfs 9000/9001 and repoint `MINIO_ENDPOINT=minio:9000` in the orchestrator plus
`stowage/config.yaml`; `transport.py:95` needs no change if rustfs takes `192.168.1.104:9000`.
Side quest, unrelated to the pipeline: the shared 41-domain cert stopped renewing. Root cause was DNS,
not nginx. Every `*.kvmx.ru` name pointed at a hard A record for `109.229.102.117` while the line had
moved to `109.229.127.149`; the Mercusys DDNS at `kvmx-home.mercusysddns.com` was correct the whole
time but nothing in the zone referenced it. Fixed with `CNAME * -> kvmx-home.mercusysddns.com` at
reg.ru. Certificate now issues.
Two measurement traps worth remembering. The ISP transparently intercepts ports 80 and 443 by
Host/SNI, so `curl` from workpc to ANY address returns kvmx.ru content and proves nothing about
external reachability; bare TCP connects also succeed against arbitrary addresses and then hang. Three
wrong root causes came out of trusting those probes before checking them.
Also patched `~/scripts/migrate-kvmx-https.sh:54` on homesrv. `need_stream_module` used
`sudo -n nginx -V` and `sudo -n nginx -T`; the NOPASSWD rule covers only `nginx -t`, so it reported
"stream module is not loaded" whenever it meant "could not ask for a password". Both checks now run
without sudo. `bash -n` passes and both conditions evaluate true.
## 2026-08-11 Per-artifact buckets, rustfs buckets, baseline chapter run [#116]
Command: `mc mb` on rustfs, `docker compose up -d --build orchestrator`, `pytest -q --ignore=test_api.py`,
`./start_workers.sh`, then `/job/create` + `/stage/clear` + `/job/resume` for chapter
`7c944dd4-e972-42c7-ba60-9f6939548e80` of "Teto X Egen" as job `778297bc-e7ce-439d-91b5-8a027060d17f`.
Outcome: partial. Storage split landed and is proven by the run. The run itself was still in `direct`
when the session ended.
Produced: `decisions/storage-layout.md`, `caveats/speaker-attribution.md`, 109 orchestrator tests pass.
Artifacts now split one bucket per class instead of everything under `manga`
(`decisions/storage-layout.md#bucket-per-artifact`). Both MinIO and rustfs hold all six buckets. The
run put 79 pages in `raw` and 116 panel crops in `panels`, so the split works end to end.
Two containers on homesrv had been dead for two weeks and blocked the work. `manga-fetch` was exited,
so `/job/create` failed with `httpx.ConnectError`; `manga-web` was exited, so `manga.kvmx.ru` had
nothing behind it on port 8083. Both started with `docker compose up -d`. Neither is related to the
storage change. Neither was caught by any check, because nothing watches these containers.
Stage timings, 116 panels: crop 85s, vision ~4min, identity ~1min, reconcile ~7min for 35 pairs,
dialogue ~8min. Faster than the 2026-07-17 run at 75 panels. The webtoon crop that 500'd in July
succeeded this time.
Quality cross-check against the panel images, the point of the run. Dialogue text extraction is
accurate. Character detection is accurate. Speaker attribution is not: three of three sampled
two-character panels attribute both speakers to the wrong person, always swapped
(`caveats/speaker-attribution.md#tail-is-not-geometry`). 24 of 81 speech lines resolve to a named
character, which is the Phase 1 headline metric at 30%, and the sample says that 30% is not
trustworthy. 26 of 113 detected people got an identity, and 25 of those 26 went to one character that
turns out to cover two different women.
The run then reached `scene` 116/116 and failed in `script` at 87/116, not on OOM: 28 beats were
rejected by the script verifier as `unsupported-proper-noun: ['Choi', 'Haeseon']`
(`caveats/speaker-attribution.md#multiword-name-verifier`). No two-word cast name can pass that check.
## 2026-08-11 Speaker provenance and the multi-word cast name
Command: `.venv/bin/python worker_vision.py`, `pytest -q --ignore=test_api.py` in the orchestrator.
Outcome: both pass, 110 orchestrator tests. Nothing deployed, no GPU work, no pipeline run.
Produced: `decisions/speaker-attribution.md`, two caveats rewritten.
`_annotate_speaker_methods` stopped stamping `tail` on a model guess. With two or more characters
present the guess is dropped to `unknown` at confidence 0.0. With one present it is kept as
`model_solo` at 0.7, the same claim the solo backstop already makes
(`decisions/speaker-attribution.md#no-fake-tail`). Grounded `som_face` and `solo_prior` rows are
untouched. Nothing outside `worker_vision.py` reads the literal `tail`, checked across both repos.
`verify_script` now tokenizes each cast name into `allowed`, so `Choi Haeseon` passes as two tokens
(`decisions/speaker-attribution.md#multiword-cast-names`). That is the 28 beats job `778297bc` lost.
The remaining `['Blur']` beat is a true positive that still halts the whole chapter, now recorded as
`caveats/speaker-attribution.md#one-word-halts-chapter`.
Neither fix is live. The orchestrator container is not rebuilt and the workers are not restarted.
## 2026-08-11 Rerun from dialogue: the honest speaker number is 9%
Command: `/job/cancel`, `/stage/clear dialogue`, `./start_workers.sh`, `/job/resume` on job
`778297bc-e7ce-439d-91b5-8a027060d17f`, twice. `docker compose up -d --build orchestrator` three times.
Outcome: `dialogue` 116/116. 112 orchestrator tests pass, `worker_vision.py` self-check passes.
The named-speaker share is 9%, 9 of 95 speech lines, down from a reported 30% that counted fake tails.
Multi-character panels contribute 0 of 40 lines by design. Single-character panels give 9 of 55. All 9
binds are `Choi Haeseon`, the row that covers two different women.
Five defects, four of them found by measuring the run rather than by reading code.
1. The fake `tail` label, fixed before the run (`decisions/speaker-attribution.md#no-fake-tail`).
2. The multi-word cast name in the script verifier
(`decisions/speaker-attribution.md#multiword-cast-names`).
3. `/stage/clear dialogue` deleted nothing and reported success. dialogue and direct write onto the
per-panel vision blob and had no `_STAGE_TABLES` entry, so `run_stage_dialogue` saw
`"dialogue" in vision` and would have skipped all 116 panels. The proof is the second clear:
116 dialogue blobs and 75 direct blobs stripped that the first had left. This is
`caveats/audit-open.md#dishonest-clearing` firing exactly where it was filed.
4. gemma answers the speaker field with whatever the prompt showed, most often the character
description, and every such answer became a free-form name that no registry entry matched. 28 of 51
sampled lines (`decisions/speaker-attribution.md#prompt-label-answers`). After the fix, 3 of 95.
5. All 7 `som_face` lines pointed at a mark whose face paired to no present character, so the
highest-trust provenance sat on a line with no speaker. Same defect class as the fake tail.
Identity is now the binding constraint, not attribution. 26 of 113 detected people carry an identity,
23%, and 25 of the 26 are the one over-merged row. Even perfect balloon binding caps this chapter near
23% named. The person who does hold an identity is stored as `Lim Seonho` while a separate row is named
`Seonho` with alias `Lim Seonho`, so either name matches two rows, raises `ambiguous-speaker` and binds
nothing.
Two more defects surfaced after `dialogue` finished. All 7 `som_face` lines pointed at a mark whose face
paired to no present character (`decisions/speaker-attribution.md#unpaired-mark`). Then `script` halted
at 112/116 because the narrator wrote `"...Hm?"` for the source line `"Uh... hum...?"`, and both verifier
rules fired on that two-letter interjection
(`decisions/speaker-attribution.md#interjection-false-positive`). After the fix, `script` passed 116/116,
the first time this chapter has cleared the verifier. `tts` then ran for the first time.
The run then completed end to end for the first time: `tts` 116/116, `layers` 116/116, `render` 116/116,
`assemble` 1/1, finished 2026-08-11T20:08:16Z. `s3://video/` holds 49 clips and a 50MiB `chapter.mp4`,
`s3://audio/` 49 objects at 32MiB. The per-artifact bucket split is now proven for every class except
layers (`decisions/storage-layout.md#bucket-per-artifact`).
Two honesty defects surfaced at the finish, both recorded rather than fixed. `layers` reported
`completed 116/116` with an empty bucket, and the completed job still carries
`error: "partial: 112/116 completed"` from the failure three resumes earlier.
## 2026-08-12 — the video got watched
No pipeline ran. The user watched `chapter.mp4` for the first time and read out 19 timestamped defects.
That found more than the previous four sessions of measuring, because the recorded metrics were all
measuring whether code ran rather than whether the result was right.
Two measurements came out of it. First, `chapter.mp4` is video 436.39s over audio 363.67s, so the
narration finishes 72.7s before the picture and the gap accumulates. The 49 clips are clean: every one is
25fps exactly, video and audio agree to 0.03s, and they sum to 363.6s. Assembly adds 72.7s of video and
no audio. Second, this manga holds 19 character rows of which 3 carry a name, and `Choi Haeseon` holds 25
of the chapter's 26 identity assignments. That is why the video calls the colleague Choi, never names the
MC, and flips gender.
The A/V bug was narrowed with a per-round probe over the 49 real clips. Round 0 of `_assemble_batched` is
correct, losing only the xfade overlap per group. Round 1 turns 359s of video into 100s while the audio
survives at 358.79s. The round-1 filtergraph is arithmetically correct, and re-running the same chain by
hand over only the 6 encoded intermediates gives a correct 348.24s with no warnings. Round 1 differs by
holding a 7th input: the leftover 49th clip, which `_assemble_batched` passes through un-encoded. That
passthrough is a third path beside `concat` and `xfade` and is the prime suspect.
`worker_render.py` gained an `FPS = 25` constant, fps normalization in the xfade branch to match the
concat branch, a `_stream_dur` helper, and a self-check that compares video against audio rather than
asserting the file is non-empty. The old check only asserted `getsize(out) > 0`, which is how a 20% sync
failure shipped. Pinning `-r FPS` on the output encodes was tried and reverted: it collapsed the chapter
to exactly 100.00s by dropping frames to force CFR, which the comment at the concat branch already
warned about. None of it is committed and none of it fixes the chapter yet.
The fps inconsistency between the two branches is real but not proven to be the shipped cause. The scene
graphs hold 356 `cut` against 6 `fade_black`, so the real run's final round most likely stayed on the
concat branch where no mixing happens.
### Panel 7, checked against the art
The same day, the user pulled up panel 7 and checked every detection by eye. It overturned the framing
this file carried an hour earlier, and it overturned two theories I proposed before being corrected.
Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, a wide establishing shot of an office through a
window, crop 900x1650. Vision emitted 6 characters. Zero of the two identity bindings are correct and the
one character who matters is unbound. `person_5`, described as "yellow sweater", is Seonho in the
foreground and got no identity. `person_6` is the colleague, who has no name in the story, and was
assigned `Choi Haeseon` at 0.9. `person_2` is a background extra and was assigned `Lim Seonho` at 0.9.
`person_1` is a window frame with nobody in it. `person_3` and `person_4` are background extras.
Three defects stack, recorded as `caveats/speaker-attribution.md#bbox-wrong-space`,
`#no-anonymous-identity` and `#extras-as-cast`. The `bbox` values are consumed as absolute pixels, and on
this panel that puts all six boxes in the top third with two inside a speech balloon. Divided by 1000
four of the six fit tightly. Identity therefore embedded crops of balloon edges and window frames, which
is how a 0.9 confidence lands on the wrong person. Blank crops embed alike, a plausible mechanism for one
row absorbing 25 of 26 assignments.
Two claims I made and had to withdraw. First, that rescaling by 1000 makes the boxes correct: after
scaling, `person_1` still sits on an empty window frame and `person_6` clips its subject, and the
descriptions are unreliable anyway, since `person_6` reads "white shirt" for a green dress. Second, that
the constraint is 16 nameless rows needing names. The opposite is true. The pipeline mints names onto
people who have none, and at least one nameless row is a real recurring person who should stay nameless.
The "26 of 113 detected people carry an identity" figure that framed the roadmap counted mostly
background extras. It should not be quoted again.
## 2026-08-12, chapter assembly, root cause and fix
Reproduced the A/V collapse offline with 49 synthetic clips at `ASSEMBLE_BATCH=8` and six `fade_black`
boundaries. It came out worse than the shipped run: **two round-0 groups of 8 fresh clips collapsed on
their own**, so the single-item passthrough theory from yesterday is dead
(`decisions/chapter-assembly.md#passthrough-innocent`).
Bisected one collapsing group by truncating the chain stage by stage:
```
k=7 out= 52.52 correct
k=8 out= 52.52 the last xfade contributed nothing
[v6][n7]xfade=duration=0.050:offset=52.500 <- [v6] is 52.52s long, 0.02s of margin
```
`_xfade_chain` took its durations from `_audio_dur`, which is `format=duration`, which is
`max(video, audio)`. Each clip's audio outlasts its video by about a frame, so the offset accumulator
crept ahead of the picture. Once the creep passed the transition width, xfade emitted the transition and
threw away the second input and every clip after it, at `rc 0` with nothing on stderr.
Fix: offsets come from `min(_stream_dur(v), _stream_dur(a))`, every input is floored to a whole frame
count and `trim`/`atrim`ed on both streams, and `_check_assembled` now verifies each encode against the
predicted timeline instead of trusting the exit code
(`decisions/chapter-assembly.md#offsets-from-min-stream`, `#check-assembled`).
Verified on the 49 real clips of chapter `7c944dd4`, re-downloaded from MinIO:
```
before r1 n=7 XFADE in v=359.29 a=359.60 -> out v= 99.96 a=358.79
after r1 n=7 XFADE in v=359.61 a=359.62 -> out v=358.76 a=358.76
chapter v=358.76 a=358.76 gap=+0.00 (shipped: v=436.39 a=363.67 gap=+72.72)
```
`worker_render.py` `__main__` passes. Two checks were added there, because the existing 4-clip A/V assert
passed all the way through the broken build. One asserts the frame-exact `trim` on both streams, one
assembles three clips whose audio outlasts their video by 0.4s. Mutation-tested by putting `_audio_dur`
back: the new check fires with `video=1.80 audio=3.56 expected=3.56`.
Not done: `s3://video/.../chapter.mp4` is still the broken 436s file. Rebuilding it means clearing the
`assemble` stage and resuming, which is CPU-only and was not run.
## 2026-08-12, the chapter rebuilt, and the bbox space settled
**The rebuild came out byte-identical to the broken file.** Clearing `assemble` and resuming produced
video 436.392031s over audio 363.674667s and `nb_frames` 9902 again, which proved the xfade fix committed
earlier today never runs for this chapter. With all-`cut` transitions `assemble` takes the `else` branch,
a `concat` demuxer with `-c copy`.
Reproduced that path offline in seconds and got the shipped numbers exactly. The cause is mixed frame
rates: 14 of the 49 clips are `r_frame_rate=30/1` at `time_base=1/15360`, the other 35 are `25/1` at
`1/12800`. `-c copy` writes the output in the first input's timebase, so those 14 play `15360/12800 = 1.2`
too long with their audio untouched. `collage_cmd` hardcoded `-r 30`, which yesterday's `FPS` sweep
missed. `decisions/chapter-assembly.md#mixed-rate-stream-copy`.
Fixed `collage_cmd` to emit `-r FPS`, and made `assemble` probe `r_frame_rate` across the clips and route
mixed rates through the re-encoding tree. Rebuilt:
```
before v=436.392 a=363.675 nb_frames=9902 avg_frame_rate=22.69
after v=364.120 a=364.122 nb_frames=9101 r=25/1
```
The 14 clips in the bucket are still 30fps. Assembly normalizes them, so the chapter is correct without
re-rendering, but the fast stream-copy path stays disabled for this chapter until `render` re-runs.
**The `bbox` space is 0-1000, not pixels.** Pulled all 113 detections from `/review/identity` and
measured: 47 boxes have `x2` past the 900px panel width, none has `y2` past 1000 on panels 1257 to 2307px
tall, 21 clamp at exactly 1000 in x, and the whole range is `[0, 1000]`. `/vision` now converts to pixels
before returning, so identity crops, gated face pairing, the set-of-mark boxes and the review UI all read
pixels (`decisions/identity-bbox.md#bbox-is-normalized`).
Checked by eye the way the user did. Drew the converted boxes on panel 7: five of six land on their
subject, including `person_5`, who is Seonho in the foreground with headphones and carried no identity.
`person_1` still frames an empty window mullion, which is the extra-versus-cast caveat, not this one.
Not done: `vision` and `identity` have not re-run, so every box, embedding and `ref_image_uris` in the
registry is still from the wrong space. That rerun is GPU work and was not started.
## 2026-08-12, third session — the rerun, and the run that did not count
Asked: move on from the chapter rebuild. Scope chosen mid-session: reset the registry, rerun vision,
identity and reconcile only, and inspect before spending the ~80 minutes downstream.
**The registry had to be wiped first, and nothing could do it.** `clear_stage_data` maps `identity` to
`identity_assignments` only and spares `characters` on purpose (`db.py:790`), because the registry is
cross-run. So a rerun would have matched fresh crops against 19 stale grid-space embeddings and kept the
duplicate `Lim Seonho` / `Seonho` rows that raise `ambiguous-speaker`. Added `db.reset_registry` and
`POST /characters/reset` (`confirm=true` required), which also clears `identity_assignment_sources` —
untouched by any stage clear, and a leftover `manual` row there makes `assign_identity` refuse the next
model assignment (`db.py:663`). Covered by `test_db.py:TestResetRegistry`; 115 orchestrator tests pass.
The orchestrator image bakes its source, so it needed `docker compose up -d --build orchestrator`.
**The first full cycle was void: the worker was serving pre-fix code.** vision + identity + reconcile
completed 116/116/20 and reproduced the defect exactly — 46 of 110 boxes past the 900px panel width,
coordinates clamping at 1000. The vision worker started 12:00:09, `worker_vision.py` changed 12:11:35,
and the commit carrying `_bbox_to_pixels` landed 12:16:22. Python had already bound the old module.
Nothing in the stage output said so (`decisions/identity-bbox.md#stale-worker-invalidates`). Added
`check_stale.sh`, which compares every worker's process start against its module mtime and exits
non-zero; mutation-tested by touching `worker_tts.py`.
**The rerun against restarted workers.** 8 minutes for all three stages.
```
before stale run after
x2 past panel width 47/113 46/110 0/110
coords exactly on 1000 21 27 1
max y2 1000 1000 2307
characters 19 8 8
coverage 26/113 = 23% 77/110 = 70% 77/110 = 70%
top character's share 96% Haeseon 47% 47%
```
`Choi Haeseon`, which had absorbed 25 of 26 assignments, no longer exists in the registry. On panel 7
Seonho is bound for the first time, and the unnamed colleague took an anonymous id instead of being
called `Choi Haeseon` at 0.9.
**Fixing the boxes made the extras problem worse.** With crops finally landing on their subjects, a
background extra bound to `Seonho` at confidence 1.00, putting an extra into the lead's reference set.
`/vision` now stamps `has_face` per character via `face_detect` + `_pair_faces_to_present`, and identity
skips `has_face is False` before it crops or embeds
(`decisions/identity-bbox.md#face-gates-enrollment`). Fails open on a missing or raising detector.
Self-checked in both workers. **Not yet proven on a GPU run** — that is the next command.
Deleted along the way and not recoverable: the rebuilt `chapter.mp4` and all 49 clips, by the cascade
from `/stage/clear vision`. The user chose not to keep a copy.
## 2026-08-12, fourth session — identity 2b, the resolver NONE branch
Asked: "how much will `has_face` help with character and identity problems?", then "fix 2b first".
Answered the first honestly: `has_face` reaches 2 of 6 detections on panel 7 and nothing else. It does not
touch naming or merging, and it cannot touch the chibi at 1:35, because an anime face detector detects a
chibi face. Also flagged its real cost: the model is face-only by design, so back-turned cast lose
enrollment along with the extras, and coverage is where that shows up first.
Measured before writing anything, read-only, no GPU:
```
registry: 8 characters, 1 named -> ['Seonho']
detections: 110 assignments: 77 = 70% coverage
spread: Seonho 36, character_565c88 24, character_759e23 9, character_f7a4fd 3,
character_25f682 3, character_d72710 1, character_823aba 1
```
That killed the assumed cause. Anonymous ids already recur, so the identity worker's own
pending-promote path gives stable anonymous identities. The defect was elsewhere.
`/vision/resolve` can answer "none of these" and always could: `worker_vision.py:1071` returns
`state="new"` for `choice: 0` and `state="unresolved"` for an out-of-range index. `service.py` read only
`character_id` and unassigned every crop of the tracklet for either. The stale `ponytail:` comment above
that block named the real blocker and was right: minting needs an `embedding_uri` the orchestrator cannot
compute, since siglip and gemma cannot both be resident.
Fixed by carrying the embedding, not by adding a GPU pass. `/identity/resolve` writes each crop's
embedding to the crop's key with a `.npy` suffix and returns `emb_uri`. `tracklets.resolve_outcome` holds
the three-way decision as a pure function. `service.py` mints via the existing `create_character` and
falls into the existing assign loop. `decisions/identity-bbox.md#none-mints-an-anonymous-character`.
Checks: `worker_identity self-check ok`, `tracklets self-check ok`, 115 passed on homesrv.
Deployed: image rebuilt, `resolve_outcome` verified inside the running container, `audit_registry.py`
re-copied after the recreate. Vision and identity restarted, `./check_stale.sh` exits 0.
New trap: `tmux respawn-window -k` does not re-run the window command. It leaves a bare shell and the
worker down. Both workers were dead for two minutes before `/health` caught it.
Not run: the GPU cycle. `has_face` and the NONE mint are both unproven on real panels and now land in the
same run.
## 2026-08-12, fourth session, continued — identity 2c, extras versus cast
Asked: "fix extra-vs-cast?"
Traced the three places that read `vision["characters"]` raw before changing anything. That corrected an
inference made earlier in the session. `build_scene` already drops an unassigned detection from
`characters` and `present` (`worker_scene.py:63`), so extras never reached the cast list at all.
The leak was their ACTIONS. `actions` was built from every detection, and that list is what the script
prompt renders and what the correctness verifier uses as evidence. So a background extra's "standing at the
window" arrived as a fact about the panel with no character attached, and the verifier confirmed it,
because the action really was in the blob. That is a second mechanism behind the invented-narration
complaints in item 3, independent of the model inventing anything.
Gated two consumers on `has_face is False`, matching the enrollment gate's semantics exactly:
`worker_scene`'s `actions`/`action`, and `service._beat`, which picks the director's "who" from the first
three detections and falls back to an action when a detection has no name.
Left `service._present_characters` ungated on purpose. It builds the dialogue stage's candidate speakers
and the set-of-mark boxes. An extra picked as speaker already resolves to unknown, not to a wrong name, so
the failure is contained. And the gate's cost lands hardest there, since a character drawn from behind has
no face box and gating would delete a real speaker from the only list that can attribute their line.
`decisions/identity-bbox.md#extras-gate-consumers`.
Checks: `worker_scene self-check ok` with three cases (cast action survives, faceless dropped, missing key
survives), 115 passed on homesrv. Deployed: image rebuilt, `_beat` verified inside the container, scene
worker restarted, `./check_stale.sh` clean. Committed `ca46617` and `8b27aec`.
Still not run on a GPU. Three changes now ride the same cycle: `has_face`, the NONE mint, and this.
## 2026-08-12, fourth session, continued — identity 2d, merge and split
Asked: "2d? merge and split?"
Checked the registry before planning anything, and 2d as written is stale. The registry reset earlier today
deleted both worked examples. There is no `Lim Seonho` to merge into, `character_afa7623b` does not exist,
and the current registry is 8 rows with one named character (`Seonho`). Nothing to merge or split until the
rerun mints a new set.
So the useful work was the safety net for that rerun, since `reconcile` runs inside it. The caveat's cost
line was the reason: one bad merge was unrecoverable without rebaselining the whole manga, and the cycle
about to run includes a merge pass over embeddings nobody has seen yet.
`merge_characters` no longer deletes the loser. It sets `merged_into = keeper`, so the row keeps its
embedding, description and gender, and it stamps every repointed assignment with
`method = merged_from:<loser_id>` in `identity_assignment_sources`. That reuses a free-form column on a
table that already existed rather than adding a merge log. `source` is left alone, so a reviewer's `manual`
assignment keeps its veto in `assign_identity` after being repointed.
Roster readers filter `merged_into IS NULL`, including the name-dedup in `create_character`, which would
otherwise fold new crops back into a character reconcile had retired. Lookup by id does not filter, because
an assignment or a narration reference may still point at a merged id.
Two existing assertions asserted the old destructive behaviour (`test_db.py:235`, `test_merge_refs.py:37`)
and were rewritten, not deleted: the invariant changed on purpose.
Checks: 116 passed on homesrv, up from 115. Additive `ALTER TABLE` through the existing `init_db` migration
block, verified on the live database (`merged_into` present, 0 rows merged). Committed `00096cc`.
Deliberately not built: the unmerge path and the split. No wrong merge has been observed since the crops
were fixed, so the consumer of these records waits for one. The forward case is partly covered by 2b, since
a resolver NONE now mints instead of folding a stranger into the nearest match.
## 2026-08-12, fifth session — the GPU cycle all four changes were waiting for
Asked: "go ahead" on the rerun.
First correction: the vision/identity/reconcile timestamps in `/job/status` are UTC and the git log is
local (UTC+4), so the run that looked like a rerun was the pre-change baseline. `_mark_has_face` landed at
19:07 local, the run finished 13:17 local. The handoff was right that nothing had run.
Reset the registry for `ef105a86` (8 characters, 77 assignments, 101 sources, 142 S3 objects), cleared
`vision` and everything downstream (116 vision results), restarted the identity worker for its in-process
known-character cache, then ran vision, identity and reconcile. 6 minutes wall: vision 3m48s, identity
1m25s, reconcile 48s. All three `completed`, 116/116/16.
### What the four changes did
| change | verdict |
| --- | --- |
| `has_face` stamp | works, present on all 110 detections, 67 true / 43 gated (39%) |
| extras gate | works, panel 7's two wrong bindings are gone |
| NONE mints | works, 16 characters minted where 8 existed |
| non-destructive merge | works, 7 rows carry `merged_into`, 9 assignments stamped `merged_from:` |
Coverage went 70% -> 61% (77 -> 67 assignments over the same 110 detections). That is the gate's price and
it is mostly correct, see below.
### Panel 7, the worked example, before and after
```
person_1 Seonho 1.00 -> -- none -- has_face=False
person_2 character_f7a4fd 0.00 -> character_519d2b 0.00 has_face=True
person_4 character_d72710 0.94 -> -- none -- WRONG binding removed
person_5 Seonho 1.00 -> -- none -- WRONG binding removed
```
The two wrong bindings the gate existed to kill are dead. It also took `person_1`, and that is correct:
ran `face_detect.detect_faces` on p006.png directly and it finds exactly one face on the whole panel, conf
0.599, inside `person_2`. Swept the threshold to 0.04 and nothing else appears above 0.056. Looked at the
crop: `person_1` is the lead drawn from behind at his desk, back of the head and headphones, no face in
frame. The detector is right and the gate is right. The cost is that a back-turned character cannot be
identified from that panel at all.
Also worth recording: `person_1`'s new box frames the lead correctly. The old note that it framed an empty
window mullion was written against the pre-fix boxes.
### The new defect: a roster hint named the wrong man
The registry holds three real people and one wrong name.
- `character_92129ac7` "Lim Seonho", 22 assignments. Verified against the art: the p010 introduction panel
captioned `LIM SEONHO (29)`, yellow plaid shirt and headphones, which is the roster's description for
"Seonho". This is the lead and the name is right.
- `character_556aef60`, 25 assignments, unnamed. Verified: the woman with short black hair and pearls, the
roster's second character, whom the roster itself calls "Unknown".
- `character_dbadfff7` "Seonho", 15 assignments. Verified: a different man, glasses, dark clothing, in
profile. He carries the lead's roster name.
p020 assigns `Seonho` to `person_1` and `Lim Seonho` to `person_3`, so the pipeline holds them as two
people in one panel. inference: the roster hint injected at `service.py:882` puts a name in front of
detection, and detection attached it to the wrong face before the caption panel could mint the real one.
This is not the alias-merge case and merging the two rows would be wrong. It needs name binding to require
evidence, the way `name_claims` already does for captions and address.
### Two smaller things the run exposed
- **Merge chains.** `character_e1ab7776 -> character_521c301f -> character_556aef60`. `merged_into` points
at a row that is itself merged, so a single-hop resolve lands on a retired character. Roster readers are
fine because they filter `merged_into IS NULL`. Anything that follows one hop is not.
- **Orphan `name_claims`.** All 5 rows point at `character_afa7623b`, which the reset deleted. The reset
clears characters and assignments but not claims.
Not run: dialogue and everything downstream. The job sits at `dialogue waiting`.
### Same session — the three fixes the run's evidence asked for
All three are orchestrator-side, so no worker changed and `check_stale.sh` is not in play.
- **Roster hints no longer seed detection.** Deleted the two lines at `service.py:882` that appended
`_roster_char_hints` to `known_characters`. `build_detect_prompt` drops any hint without a name, so a
nameless hint would have contributed nothing anyway. Names now reach detection only from registry rows,
which are embedding-backed and were themselves named from an in-panel caption or address. The roster
still feeds `roster_cast` in `run_stage_dialogue`, where names are matched against speech.
- **`merge_characters` resolves the keeper to its chain root** before merging, so `merged_into` stays one
hop deep. Bounded 64-step walk, the cap only exists so a cycle cannot hang reconcile.
- **`reset_registry` deletes `name_claims`** for the manga's panels and reports the count.
Checks: 117 passed on homesrv, up from 116. The new chain test was confirmed to fail with the walk
disabled, then the walk was restored and the suite re-run. Deployed by rebuilding the image; verified
inside the running container by parsing the deployed source, not by grepping it, because the first check
matched the word `_roster_char_hints` inside the comment that explains its removal.
Also copied `audit_registry.py` into the repo before the rebuild. It had only ever been `docker cp`'d, so
every rebuild dropped it. `Dockerfile` has `COPY . .`, so it is baked now and the trap is gone.
Open risk on the next run: without the roster hint the lead is unnamed until the p010 caption, so panels 1
to 9 may mint him as an anonymous character that reconcile then has to merge. Watch whether "Lim Seonho"
comes back as one row or two.
### Same session — run B, 17:38-17:44
Reset (16 characters, 67 assignments, 67 sources, **5 name_claims**, 156 S3 objects), cleared `vision`,
restarted the identity worker, ran the three stages. The `name_claims: 5` line is the orphan fix confirmed
on live data.
117 detections, 59 assignments, 50% coverage, 20 characters, 72 with `has_face` (45 gated).
The registry split is fixed. "Lim Seonho" is one row with 25 assignments, "Seonho" fell from 15 to 1. So
dropping the roster hint did not split the lead across the panels before his caption, which was the risk
recorded before the run.
Two findings, both now filed:
- detection still names from in-panel text and can attach the name to the wrong body (p011, p026), which
is `decisions/identity-bbox.md#roster-does-not-name`'s "not covered" paragraph
- one degenerate bbox, `caveats/audit-open.md#degenerate-bbox`
And one fix of my own that was incomplete: the merge chain came back as
`477c1894 -> a92d9df4 -> 4fb94c15` with the keeper walk deployed. The walk only sees the pair being merged
now. Retiring a row that is already somebody's keeper needed the second half, a repoint of the loser's
dependents. Both halves and both tests are in `decisions/identity-bbox.md#merge-chains-flatten`.
Checks: 118 passed, was 116. Each new merge test was run with its fix disabled and confirmed to fail
first. Deployed by rebuild and verified by parsing the container's source.
Coverage is now 70 -> 61 -> 50 across three runs and is the open question, recorded as
`caveats/audit-open.md#coverage-trend`. `identity_labels` already holds 145 rows of ground truth, so the
next move is to score with `eval_identity.py` rather than to keep reading the coverage number.
## 2026-08-12, sixth session — the ground truth is for the wrong manga
No GPU work. Two things, both cheap.
**`_bbox_to_pixels` orders its corners.** Each coordinate pair is sorted after clamping. The self-check
feeds it run B's real `[226, 417, 130, 551]` and asserts `[117, 688, 203, 909]`. Unsorted the call returns
`[203, 688, 117, 909]`, so the assert fails without the fix. `worker_vision.py` self-check passes.
The vision worker is now STALE and needs a restart before the next cycle.
**Item 1 of the plan cannot run as written.** All 145 rows of `identity_labels` belong to chapter
`8ca8249b`, a different manga with cast "Rico" and "Ikekin", spread over 81 panels. Chapter `7c944dd4`
has none, so `/review/identity?job_id=778297bc...` returns `labeled: 0, correct: 0, accuracy: null`.
Scoring `8ca8249b` anyway gives 7/138, with 113 rows labelled as a real person and left unassigned. That
chapter's identity run is stale: 44 assignments over 246 panels, predating every fix. The number measures
old code on the wrong chapter.
So the eval path is proven end to end and the ground truth is absent. A precision number for the current
registry needs a hand-labelling pass over `7c944dd4` through `POST /review/identity/label`, keyed by bbox
rather than by `local_id`, because vision reassigns `local_id` every run.
## 2026-08-12 18:07-18:13 UTC, third GPU cycle
Restarted vision (window 3) and identity (`/characters/reset` returned `restart_identity_worker: true`),
both by `send-keys C-c` then re-sending the launch line, never by `pgrep`. `check_stale.sh` exit 0 before
the run. Reset dropped 20 characters, 59 assignments, 69 sources, 168 S3 objects.
vision 116/116 in 3m59s, identity 116/116 in 1m24s, reconcile 18/18 in 50s.
| metric | 17:38 run | 18:07 run |
| --- | --- | --- |
| detections | 117 | 119 |
| assignments | 59 | 68 |
| coverage | 50% | 57% |
| degenerate boxes | 1 | **0** |
| `has_face` true / gated | 72 / 45 | 71 / 48 |
| assigned among face-bearing | 59/72 = 82% | **68/71 = 96%** |
| assigned among gated | -- | 0 |
| registry rows / merged | 20 / 7 | 18 / 9 |
| `merged_from` stamps | 9 | 22 |
| chains deeper than one hop | 1 | **0** |
The corner-ordering fix works: 0 degenerate boxes. The merge-chain fix holds under a heavier merge load,
9 merges and no chain.
**Coverage is now the gate and nothing else.** Every assignment landed on a face-bearing detection and none
on a gated one. Recall among face-bearing detections is 96%. So the 43% with no assignment is the 40% the
gate drops plus 3 detections, and the resolver is not losing cast. That closes the part of
`#coverage-trend` that suspected the resolver.
**The open question moved to precision.** The lead holds 36 of 68 assignments, 53%, against 25 of 59
before, and `audit_registry.py` flags it as absorbing. The merge count rose from 7 to 9 over the same
interval. Whether 36 is a protagonist in half the panels or an over-merge cannot be told apart without
labels, which is item 1.
Panel 7 got worse, not better: 5 of its 6 detections carry `has_face = False`, and `person_6`, the one that
does, went unassigned. The previous run bound its colleague. That is the cost of requiring a face, item 4.
## 2026-08-12, sixth session, later — the lead was over-merged, and cosine cannot fix it
The user checked the lead's crops in the review UI. `LIM SEONHO`'s 36 assignments cover at least six
different men, a chibi, and a cat. So 36 of 68 is a bug, not a protagonist.
Split by mechanism: 22 native resolves, 14 from reconcile merges. Both are broken, for different reasons.
**The tracklet linker chained across the chapter.** `window=8` bounds each pair, transitivity was
unbounded. The 22 native assignments came from 3 tracklets spanning 0, 22 and 30 panels, visible as two
confidence blocks of eleven rows at 0.95 and ten at 1.0. Fixed by capping the merged group's span
(`decisions/identity-bbox.md#tracklet-span-cap`).
**Both tokenizers linked on words that are not identity.** Whole chains hung on `short`; one pair linked on
the word `hair` itself. There is now one tokenizer with a `GENERIC` stopword set, shared with reconcile's
pre-filter (`decisions/identity-bbox.md#generic-tokens`).
Measured on the same 22 real crops, candidate overlap forced to pass: 3 tracklets at worst span 30 becomes
9 at worst span 8.
**Then the obvious next fix was ruled out by measurement.** Pulled all 22 crop embeddings from MinIO and
computed the pairwise cosine. The cat scores up to 0.82 against men, two different men score 0.93, and the
highest pair in the matrix is 0.96. No threshold exists, so crop-to-crop cosine is not a link signal, and
the `candidates` overlap condition is close to noise. Filed as
`caveats/audit-open.md#cosine-not-identity`. Inference: identity embeds the whole person box, so cosine
measures scene and style, not face.
Also seen: vision boxes cats as people and dresses them. `p081` and `p108` are cats, described
`short brown / yellow plaid shirt` and `short brown / white t-shirt`.
Checks: `tracklets.py` self-check ok, orchestrator 118 passed. Both new asserts confirmed to fail with
their fix disabled. Nothing deployed, nothing re-run on a GPU.
Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad
only, not committed.
## 2026-08-12, 19:44-19:52 UTC — fourth GPU cycle: the cast is the gallery
Three fixes ran together for the first time: the tracklet span cap and the shared `GENERIC` tokenizer,
both written last session and never deployed, plus the new one
(`decisions/identity-bbox.md#cast-is-the-gallery`). Job `778297bc`, chapter `7c944dd4`, registry reset,
vision cleared, `vision -> identity -> reconcile`. Vision 116/116 in 3m55s, identity 116/116 in 2m44s,
reconcile 20/20 in 44s.
| metric | 18:07 run | 19:44 run |
| --- | --- | --- |
| detections | 119 | 119 |
| assignments | 68 | 60 |
| coverage | 57% | 50% |
| tracklets over crops | 12 / 64 | 33 / 72 |
| lead's assignments | 36 | 16 |
| top character's share | -- | 16/60 = 27% |
| characters after reconcile | 18 | 14 |
| minted / cleared | -- | 10 / 12 |
**Coverage went down and that is the fix working.** gemma cleared 12 crops it used to be forced to name
from a cosine top-k that did not contain the right person. 72 crops entered resolution against 64, because
a crop with an empty cosine shortlist is no longer dropped.
**Checked by eye, which is the only check that sees this.** Contact sheets of every assigned crop, per
character, confirmed by the user. The lead holds 16 crops. 14 are him and 2 are art inside a panel, the
photograph at `order 17` and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and
she is a main character the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2
are `2b1b12a1` (`order 31`, `order 33`). Against the 18:07 run, where the lead's 36 covered six different
men, a chibi and a cat.
Every wrong crop on the lead is one defect: vision treats art inside a panel as the scene. It is not a
linker failure and no identity change will fix it.
**The cap fired twice and dropped the wrong rows.** `GALLERY_CAP = 10` truncated a 16-row gender-compatible
cast at `p097` and an 11-row one at `p109`. `cast_gallery` orders named first, so what it drops is exactly
the recently minted anonymous rows, which is the population a later tracklet most needs to match against.
Filed as `caveats/audit-open.md#gallery-cap-drops-the-unnamed`.
Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestrator 118 passed,
`./check_stale.sh` exit 0 before the cycle. Vision and identity workers both restarted, the second because
`/characters/reset` asked for it.
Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed.
## 2026-08-13 — the dialogue stage names nobody, and why
Ran `dialogue` 116/116 in 5m57s on the fourth cycle's registry, to see whether the fixed identity lets the
existing `name_claims` path name the female lead. It does not, and the six claims it produced name three
separate defects.
```
p040 character_2b1b12a1 "Choi Haeseon" caption 1.00 -> NOT promoted
p010 character_b1dd5659 "Lim Seonho" caption 1.00 -> conflict flag
p047 character_b1dd5659 "Seonho" address 0.90 -> conflict flag
p011 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
p026 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
p110 character_028d4a49 "Haeseon" address 1.00 -> already named
```
All three are fixed in `db.add_name_claim` and recorded in `decisions/identity-naming.md`: alias grouping,
a confident caption as strong evidence, and one name per character. `test_name_binding.py` replays these
six claims, 121 tests pass, and each new assert was confirmed to fail with its fix disabled.
Also wired `merge_faceless_captions` into both crop endpoints. It had been written and never called; both
endpoints called the non-destructive `context_fragment_links` instead, and no decision recorded that
choice. It does not cover the head-in-one-shot, body-in-the-next split that prompted the question, because
a body fragment has no text and `_merge_plan` only folds a fragment that has text and no face.
Found while wiring it: `crop_webtoon` skips an upload when the key exists, which is right for a resume and
silently wrong after a slicing change. Documented at the line and in `NEXT.md`.
Wrote `ARCHITECTURE.md` from the user's design: region graph, occurrence/identity/name, speaker as a scored
graph edge with a typed union, narrative plane for art-in-art, and a persistent story state machine. Every
section carries what exists against it today. Nothing in it is built.
Nothing ran on a GPU after the dialogue stage.
NEXT.md
+244
View File
@@ -0,0 +1,244 @@
# NEXT
Updated 2026-08-12 (sixth session). What the fifth session did is in `HANDOFF.md`, the runs are in
`JOURNAL.md`.
## State
The chapter runs end to end. The A/V sync defect is fixed and `chapter.mp4` is rebuilt: video 364.120s
against audio 364.122s at `25/1`.
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels.
Four GPU cycles ran on 2026-08-12, the last 19:44-19:52 UTC on a reset registry. The job is `running` and
parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are
current.
Eleven changes are proven on real panels. This session added bbox corner ordering. It then deployed and ran
the tracklet span cap, the shared `GENERIC` tokenizer and the cast gallery together.
**The registry is now roughly right and is worth reading.** Checked by eye, crop by crop, confirmed by the
user. The lead holds 16 assignments. 14 are him and 2 are art inside a panel, the photograph at `order 17`
and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and she is a main character
the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2 are `2b1b12a1`. On the
18:07 run the lead alone held 36, covering six men, a chibi and a cat.
So every wrong crop on the lead is one defect, item 7, and not a linker failure. The registry's other
weakness is that its biggest character has no name.
| metric | 18:07 run | 19:44 run |
| --- | --- | --- |
| detections | 119 | 119 |
| assignments | 60 | 60 |
| coverage | 57% | 50% |
| tracklets over crops | 12 / 64 | 33 / 72 |
| lead's assignments | 36 | 16 |
| characters after reconcile | 18 | 14 |
Coverage fell because gemma now clears 12 crops instead of naming them from a shortlist that did not
contain the right person. Coverage is still the `has_face` gate plus those refusals, and nothing else.
## Next
1. **Re-crop the chapter and run the fifth cycle.** Four changes are written and tested since the fourth
cycle, and none has touched a GPU.
- `merge_faceless_captions` is wired into both crop endpoints. It was written, never called, and
`context_fragment_links` was called instead. A stranded caption fragment now vstacks into the
face-bearing fragment it belongs to, so panel count and every panel index change.
- three naming fixes in `db.add_name_claim` (`decisions/identity-naming.md`): alias grouping, a caption
as strong evidence, and a name held by another character refusing to promote onto a second one.
**Clear `s3://panels/<manga>/<chapter>/panels/` before re-cropping.** `crop_webtoon` skips the upload
when the key exists, so a re-crop after a slicing change silently keeps the previous run's images.
Wiring the merge is a slicing change. Everything downstream is invalidated by it, so this is a full
re-run and not a stage rerun.
Expected: fewer than 116 panels, `2b1b12a1` named `Choi Haeseon` from the p040 caption, the lead's
`conflicting-name-claims` flag gone, and the green-dress woman no longer named `Seonho` but carrying a
`name-already-taken` flag instead.
Not fixed by any of it. Bare hair colour still links different men. Do not add a crop-to-crop cosine to
close that. Measured on 22 embeddings, different people reach 0.93 and the same person reaches 0.96, so
no threshold exists (`caveats/audit-open.md#cosine-not-identity`).
Then, separately, test embedding the FACE box rather than the person box. `face_detect` already finds
the face and pairs it for `has_face`. That is the likely root cause of cosine measuring scene instead of
person. The test is to re-embed these same 22 detections and recompute the matrix.
1b. **The head/body split that started the crop question is NOT fixed.** The wired merge only folds a
fragment that has text and no face. A body fragment carries no dialogue, so `_merge_plan` leaves it
solo and it becomes its own panel and its own shot. Finding it needs a different signal, most likely a
face touching the bottom edge of one fragment with a textless fragment below. No evidence has been
gathered yet on how often this chapter does it.
1c. **`ARCHITECTURE.md` was rewritten on 2026-08-13 under the no-Magi constraint.** Every section now
carries a DoD. Its build order starts at step 0, `eval/chapter-truth.json` plus purity and
fragmentation printed by `audit_registry.py`. Nothing below it is measurable without that file.
Steps 1 and 2 are independent and cheap: `plane` and `species` per detection, and the
face-plus-person embedding already queued in item 1 above. The weak cannot-link is step 3 and depends
on step 2, not on `plane` alone. Co-presence as a hard constraint is load-bearing while cosine cannot
separate people, so demoting it early regresses purity. `offscreen` as a fourth `speaker_ref` kind is
still cheap and independent.
2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0
degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`).
3. **Vision boxes animals as people and dresses them.** `p081` and `p108` are cats, described
`short brown / yellow plaid shirt` and `short brown / white t-shirt`. A detection prompt problem, not a
linker one, and it feeds every stage below.
4. **Detection still names from in-panel text and can hit the wrong body.** p011 and p026 emit
`name: "Seonho"` on `person_2` with the roster hint gone. That is the residue of the naming defect and
the reason the glasses man is named at all
(`decisions/identity-bbox.md#roster-does-not-name`, "not covered").
4. **Decide what a back-turned character costs.** This is now the whole of the coverage number. `has_face`
gates 40% of detections. On the 18:07 run panel 7 lost 5 of its 6 detections to the gate, and its one
face-bearing detection went unassigned. Panel 7's lead is at his desk from behind. `face_detect`
finds one face on the whole panel at conf 0.599, and nothing else above 0.056 even at a 0.04
threshold. The gate is right and the detector is right. The lead is still unidentifiable there. Options are a body or head detector alongside the face one, or letting the
tracklet carry identity across a back-turned panel. Do not lower `FACE_CONF`, the sweep shows nothing
to find.
Smaller follow-on: nine `_audio_dur` calls in `worker_render.py` measure finished clips with
`format=duration`. So the durations reported to the orchestrator are blind to per-clip drift.
They position no filter, so invariant 9 does not cover them. Worth converting to `_stream_dur`.
5. **Fix identity, in this order.** All of 2a-2d below are done and now proven on a GPU. Kept for the
evidence trail.
a. ~~Settle the `bbox` coordinate space.~~ **Done 2026-08-12**, proven over all 113 detections and
checked by eye on panel 7, where five of six converted boxes land on their subject
(`decisions/identity-bbox.md#bbox-is-normalized`). The rerun settles the last doubt: `person_1` now
frames the lead at his desk, not the window mullion the pre-fix box caught.
b. ~~Let identity abstain and stay abstained.~~ **Done 2026-08-12, run and verified on a GPU**
(`decisions/identity-bbox.md#none-mints-an-anonymous-character`). The resolver could always answer
"none of these". The orchestrator was discarding the answer: it read only `character_id`, so a
deliberate NONE and a hallucinated index both unassigned every crop of the tracklet. A NONE now
mints an anonymous character from the crop, using the embedding `/identity/resolve` ships beside
it as `emb_uri`. Deployed: image rebuilt, `resolve_outcome` verified inside the container.
c. ~~Separate extra from cast.~~ **Done 2026-08-12, run and verified on a GPU**
(`decisions/identity-bbox.md#face-gates-enrollment`,
`decisions/identity-bbox.md#extras-gate-consumers`). `has_face` stops a faceless detection
enrolling, and two more consumers now skip it: `worker_scene`'s `actions`, which is the script
prompt's content and the verifier's evidence, and `service._beat`, the director's "who".
`_present_characters` stays ungated on purpose, reasoned out in the decision.
The remaining gap is that vision still emits extras into the blob, which is deliberate so the
audit can see what was gated.
d. ~~Merge and split.~~ **Safety net done and now exercised.** The 18:07 run merged 9 rows and stamped
22 assignments `merged_from:`, with no chain deeper than one hop. The non-destructive path and the
flattening fix are both proven on real data.
A merge no
longer deletes the losing row: it sets `merged_into`, and stamps every repointed assignment with
`method = merged_from:<loser_id>`. A wrong merge now costs a hand-written SQL walk, not a full
rebaseline (`caveats/audit-open.md#destructive-reconcile`).
Deliberately not built: the unmerge path and the split. No wrong merge has been observed since the
crops were fixed, so the consumer of those records waits for one. The forward case is partly covered
by 2b, because a resolver NONE now mints rather than folding a stranger into the nearest match.
Splitting a character that is ALREADY over-merged still needs a re-embed pass over its detections.
**Cast profiles already exist. Do not rebuild them.** The user asked whether the main cast could get a
profile built from reference frames and reused. `characters` already carries `ref_image_uris` and
`embedding_uri`, and every row has both populated (16 rows after the rerun, 53 before it). The
mechanism was never missing. It was enrolled from the wrong crops, so it stored balloon edges and
window frames. Step (a) fixed that, and the audit now reports 0 characters missing a ref crop. Three things are genuinely absent and are the smaller follow-on:
- no quality gate on enrollment, so nothing checks that a reference crop holds a face at all
- nothing re-enrolls a reference set once it is written, so the wrong crops persist
**The visual "is this them?" check is already built. Do not write it again.** `/vision/resolve` at
`worker_vision.py:963` sends the query crop plus up to 3 labelled reference images per candidate.
`build_resolve_prompt` tells the model to judge face shape first, to treat hair and outfit as
secondary, that two people sharing a hair colour are not the same, and to answer `0` for NONE when
unsure. `choice: 0` becomes a new character, an out-of-range index becomes `unresolved`, and
`ref_image_uris` is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`. The
mechanism, the prompt and the abstain path are all correct. They are fed crops of the wrong region,
which is step (a).
- no human gate to name, merge or split the clusters. The user wants this as a minor adjustment on
top, not as the mechanism. The `gates` table and the review gates from [#136] are the place to hang
it
The chibi at 1:35 will survive all of this. He genuinely is brown hair plus a yellow shirt, so a
profile match is correct on appearance and wrong on reality. That needs item 4 below, plus requiring
a real face before a crop can enroll.
6. **Stop the narration inventing facts.** 0:43, 2:03, 2:05 and 2:15 assert things no panel shows. The
correctness verifier passed 116/116 because it checks quotes and names, never invented claims.
7. **Teach vision that art inside a panel is not the scene.** A chibi on a monitor became "a man holding
a drink" at 1:35. A colleague pointing into the distance became "pointing towards the screen" at
1:59.
8. **`layers` writes nothing** and reports `completed 116/116`, so no clip has parallax and a still
holds for 28s from 2:24 (`caveats/audit-open.md#layers-writes-nothing`).
9. **Clear the stale job error.** The completed job still carries `error: "partial: 112/116 completed"`
(`caveats/audit-open.md#stale-job-error`).
10. Balloon-to-speaker geometry via the unused `det`/`seg` heads
(`caveats/speaker-attribution.md#tail-is-not-geometry`) is now behind item 5. With no name to attach,
geometry buys nothing.
11. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3
unresolved refs describe a neighbouring panel in the same 8-panel call.
12. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work
(`caveats/audit-open.md#sqlite-locking`).
## Lesson worth keeping
Every metric recorded before this session said the pipeline was fine or nearly fine. `script` 116/116,
"9 named speech lines", `layers` 116/116, `assemble` 1/1. Watching two and a half minutes of output
found a 20% sync failure, a cast that is 84% anonymous, invented narration, and a stage that writes
nothing while reporting success. Stage counters measure whether code ran. They say nothing about whether
the result is correct. Watch the output before trusting a number.
## Running the pieces
```bash
./start_workers.sh # session_manager + 9 workers, each a uvicorn in a tmux window
tmux attach -t manga-workers # per-worker logs
.venv/bin/python worker_render.py # self-check, runs real ffmpeg, about 4 minutes
```
Read the state, or clear a stage and resume:
```bash
/usr/bin/ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"<stage>\"}'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
```
Traps: `tmux respawn-window -k` does NOT re-run the window's command, it drops to a bare shell and the
worker stays down. Restart one worker by re-sending the `send-keys` line from `start_workers.sh`, and
confirm with `/health` plus `./check_stale.sh`. `docker compose up --build orchestrator` recreates the
container, which drops any `docker cp`'d script such as `audit_registry.py`.
Plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin, so use `/usr/bin/ssh`.
`mc` aliases on homesrv are `homesrv` and `mio`. `local` returns Access Denied and `rfs` is the empty
rustfs. `cp` is aliased to `cp -i` and hangs on overwrite, so use `/usr/bin/cp -f`.
Re-fixing assembly needs the real clips, which the session scratchpad no longer holds:
```bash
/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/
```
## Storage and viewer, tasks #116/#117
[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had been
dead since 2026-07-19 on an arm64 digest pin.
[#116] is closer but not cut over. Artifacts split one bucket per class
(`decisions/storage-layout.md#bucket-per-artifact`), and both MinIO and `rustfs` hold all six buckets.
`rustfs` on `127.0.0.1:9010/9011` is still empty and nothing is repointed, so MinIO serves every read
and write. Remaining: `mc mirror` the live buckets, verify counts and sizes, then decide on cutover
(`decisions/storage-layout.md#rustfs-staged`).
Two containers on homesrv had been dead for two weeks and now run. `manga-fetch` is the one
`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them, and
nothing watches the workers.
## Open questions
Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for: the speaker
contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only [#203]
existed and is closed by `decisions/audit-phase1.md#unlocked-model-load`.
Three audit items are deliberately not done and are recorded as caveats rather than silently dropped:
honest stage clearing, ComfyUI under the session mutex, and reversible identity merges. Each needs a
design decision, not a patch.
Carried over from the reconstruction: `.venv` needs the ROCm torch wheel reinstalled, and `dots.tts/`,
`legacy/`, `RESUME_SPEC.md`, `pipeline-design-notes.md`, `spec-v2.md` are unrecoverable.
ROADMAP.md
+66
View File
@@ -0,0 +1,66 @@
# ROADMAP
Ordered outcomes past the current one. Phase 1 landed on 2026-08-11 and moved to
`decisions/audit-phase1.md`. The reasoning behind every phase is in `AUDIT.md`.
Approval boundary from the audit: ship Phase 2 and measure it. Then decide whether Phase 3 and 4 are
worth the size of the change.
## Phase 2: safe throughput
Unblocks: a chapter that finishes in a fraction of the current wall time without changing any model.
Done when: the same labeled chapter runs with two gemma loads instead of seven. The acceptance numbers
below are recorded in a report.
- Two-phase gemma lease schedule, so the model stays resident across compatible stages.
- JSON-schema-constrained generation for detect, dialogue, roster, direction, same-person, and resolve.
- `--media-path /dev/shm` and `file://` image paths instead of base64 data URIs.
- An ephemeral `/dev/shm` cache keyed by S3 URI plus etag, and reused direction downloads.
- Batched siglip embeddings, one forward pass per bounded image batch.
- Batched scene construction and SQLite writes. Set `busy_timeout` first
(`caveats/audit-open.md#sqlite-locking`).
- Bounded fetch and framed-page crop concurrency.
- Images before the differing instruction in every multi-call prompt, so llama-server reuses its KV
cache prefix across the two direction passes.
## Phase 3: multi-view constrained identity
Unblocks: an identity that survives a reviewer disagreeing with it.
Done when: false merges and false splits are reported per labeled chapter and the resolver beats the
Phase 1 baseline on both.
- Chapter-local tracklets persisted as a resumable homesrv artifact, not worker memory.
- Face and body galleries with crop-quality selection, and 2-4 query views per tracklet.
- Global constraints at resolution: must-link inside a tracklet, cannot-link for co-presence, gender
and species gates, manual locks, one appearance per character per panel.
- Name claims separated from visual identity. Roster names stay candidate vocabulary.
- Reversible merges: a cluster with `canonical_id`, `member_ids`, and `cannot_link`, replacing the
destructive delete (`caveats/audit-open.md#destructive-reconcile`).
- Identity evaluation extended to purity, fragmentation, name binding, and speaker accuracy.
## Phase 4: evidence-ledger narration
Unblocks: narration that can be checked against the panel rather than trusted.
Done when: every beat carries an evidence artifact and the verifier reports quote grounding, action
coverage, and future-fact leakage against it.
- Ordered beat evidence: panels, actions, chronology, dialogue ids, typed speaker refs, uncertainty.
- Scene-scoped story and turn-taking state, reset on location and time changes.
- Script chunks of 6-12 consecutive beats per text-only call, returning evidence ids.
- Confidence-aware wording: a weak speaker becomes `someone`, not a confident name.
- Targeted per-beat repair driven by the exact failures.
## Acceptance criteria
Capture a baseline and compare the same labeled chapter after every phase.
- Stage wall time and model-load time.
- Gemma call count and repair call count.
- MinIO bytes moved, prompt-evaluation and generation timings.
- Tracklet false merges and splits, identity accuracy.
- Name and speaker accuracy, unresolved correctness flags.
- Script repetition, unsupported facts, verifier retry rate.
- The share of narrated lines whose speaker is a named character rather than `Someone`. That one
number would have caught `decisions/audit-phase1.md#speaker-ref-is-canonical` the day it landed.
Read `caveats/audit-open.md#inconsistent-stage-policy` before trusting any per-stage count.
attic/worker_ocr.py
-99
View File
@@ -1,99 +0,0 @@
# worker_ocr.py — stage 3 text extraction. FastAPI :8001. cpu (easyocr), no session.
# panel in -> text blocks with bboxes + confidence out. orchestrator persists to sqlite.
# easyocr (not tesseract): it reads stylized manga lettering far better -- recovers whole
# lines tesseract garbles or drops. runs on GPU (~0.4s/page warm) by default; the OCR stage
# runs before any LLM session opens so it doesn't contend with the resident model. set
# OCR_GPU=0 to force CPU (~3s/page). GPU needs MIOPEN_FIND_MODE=FAST in the env or the first
# ROCm run spends ~60s in MIOpen's exhaustive kernel search -- the launcher sets it.
import os, uuid
from fastapi import FastAPI
from pydantic import BaseModel
import transport
app = FastAPI()
SHM = "/dev/shm"
MIN_CONF = 0.3 # easyocr line confidence floor
OCR_GPU = os.environ.get("OCR_GPU", "1") == "1"
_reader = None # easyocr.Reader, lazy-loaded on first request
def _get_reader():
global _reader
if _reader is None:
import easyocr
_reader = easyocr.Reader(["en"], gpu=OCR_GPU, verbose=False)
return _reader
def _detections_to_texts(detections):
"""easyocr readtext output [(box_pts, text, conf)] -> our text blocks with xywh bboxes.
box_pts is 4 corner [x,y] points. drops low-confidence and art-noise (<2 letters).
casing is left as-is (mixed) -- the vision stage re-cases from the image anyway."""
texts = []
for i, (box, txt, conf) in enumerate(detections):
txt = txt.strip()
if conf < MIN_CONF or sum(c.isalpha() for c in txt) < 2:
continue
xs = [p[0] for p in box]; ys = [p[1] for p in box]
x, y = int(min(xs)), int(min(ys))
texts.append({
"id": f"t{i+1:03d}",
"content": txt,
"bbox": [x, y, int(max(xs)) - x, int(max(ys)) - y],
"confidence": round(float(conf), 3),
})
return texts
def ocr_image(path: str):
return _detections_to_texts(_get_reader().readtext(path, detail=1, paragraph=False))
class OCRInput(BaseModel):
panel_uri: str
job_id: str = ""
panel_id: str = ""
@app.post("/ocr")
async def ocr(data: OCRInput):
local = transport.get(data.panel_uri, f"{SHM}/ocr_{uuid.uuid4().hex[:8]}.png")
texts = ocr_image(local)
os.remove(local)
return {"panel_id": data.panel_id, "texts": texts}
@app.post("/unload")
async def unload():
"""free the resident easyocr reader (~1-2GB) once the OCR stage is done, before gemma4 loads.
ocr isn't session-managed, so the orchestrator calls this at stage end."""
global _reader
was = _reader is not None
_reader = None
import gc; gc.collect()
try:
import torch; torch.cuda.empty_cache()
except Exception:
pass
return {"ok": True, "unloaded": was}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: detection->text-block conversion (pure, no model needed).
dets = [
([[10, 10], [110, 10], [110, 40], [10, 40]], "HELLO", 0.9), # kept
([[10, 200], [70, 200], [70, 230], [10, 230]], "WORLD", 0.8), # kept
([[5, 5], [13, 5], [13, 13], [5, 13]], "=", 0.9), # art-noise: <2 letters
([[0, 0], [50, 0], [50, 20], [0, 20]], "REAL", 0.1), # below MIN_CONF
]
texts = _detections_to_texts(dets)
assert [t["content"] for t in texts] == ["HELLO", "WORLD"], texts
assert texts[0]["bbox"] == [10, 10, 100, 30], texts[0]["bbox"]
assert texts[1]["confidence"] == 0.8
print("worker_ocr self-check ok")
attic/worker_parse.py
-125
View File
@@ -1,125 +0,0 @@
# worker_parse.py — manga parse (paged manga only). FastAPI :8009. GPU, session-guarded ("magi").
# Magi v2 chapter-wide pass: panel detection + reading order + OCR in one shot. Replaces the
# crop+ocr stages for paged manga; downstream ocr stage no-ops because rows are pre-populated.
# webtoons do NOT come here — they use worker_crop /crop/webtoon. See manga-two-repo-split memory.
import os, uuid
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import cv2
import numpy as np
import transport
app = FastAPI()
SHM = "/dev/shm"
MAGI_MODEL = "ragavsachdeva/magiv2"
_model = None
def _load_magi():
global _model
if _model is None:
import torch
from transformers import AutoModel
_model = AutoModel.from_pretrained(MAGI_MODEL, trust_remote_code=True).cuda().eval()
_model._torch = torch
return _model
class ParseInput(BaseModel):
page_uris: list # all pages of the chapter, in order
manga_id: str
chapter_id: str
session_id: str = "" # magi GPU lease (opened by orchestrator)
job_id: str = ""
def _center_in(box, panel) -> bool:
x1, y1, x2, y2 = box
px1, py1, px2, py2 = panel
cx, cy = (x1 + x2) / 2, (y1 + y2) / 2
return px1 <= cx <= px2 and py1 <= cy <= py2
def assemble_panels(pages, results, manga_id, chapter_id, put):
"""Flatten Magi's per-page output into chapter-order panels with their OCR.
`pages`: RGB np arrays. `results`: per-page dicts (Magi keys). `put(np_crop, uri)` uploads.
Text is assigned to the panel whose box contains the text-box center; SFX (non-essential)
is dropped so narration isn't polluted. bbox converted [x1,y1,x2,y2] -> [x,y,w,h]."""
out, gidx = [], 0
for img, res in zip(pages, results):
panels = res.get("panels", [])
texts = res.get("texts", [])
ocr = res.get("ocr", [])
essential = res.get("is_essential_text", [True] * len(texts))
for p in panels:
x1, y1, x2, y2 = (int(v) for v in p)
uri = f"s3://manga/{manga_id}/{chapter_id}/panels/p{gidx:03d}.png"
if not transport.exists(uri): # deterministic per gidx -> resumable
put(img[y1:y2, x1:x2], uri)
ocr_texts = []
for ti, tb in enumerate(texts):
if ti < len(ocr) and essential[ti] and _center_in(tb, p):
tx1, ty1, tx2, ty2 = (int(v) for v in tb)
ocr_texts.append({"text_id": f"t{ti}", "content": ocr[ti],
"bbox": [tx1, ty1, tx2 - tx1, ty2 - ty1], "confidence": 1.0})
out.append({"panel_index": gidx, "uri": uri,
"bbox": [x1, y1, x2 - x1, y2 - y1], "ocr": ocr_texts})
gidx += 1
return out
def _put_crop(np_rgb, uri):
tmp = f"{SHM}/parse_{uuid.uuid4().hex[:8]}.png"
cv2.imwrite(tmp, cv2.cvtColor(np_rgb, cv2.COLOR_RGB2BGR))
transport.put(tmp, uri)
os.remove(tmp)
@app.post("/parse")
async def parse(data: ParseInput):
tag = uuid.uuid4().hex[:8]
pages = []
for i, u in enumerate(data.page_uris):
local = transport.get(u, f"{SHM}/pp_{tag}_{i:03d}.png")
img = cv2.imread(local)
if img is None:
raise HTTPException(400, f"page not readable: {u}")
pages.append(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
os.remove(local)
model = _load_magi()
# ponytail: empty character bank in v1 — identity stays with the downstream siglip stage;
# feed a real bank (known-char ref crops + names) here to get Magi speaker association.
bank = {"images": [], "names": []}
with model._torch.no_grad():
results = model.do_chapter_wide_prediction(pages, bank, use_tqdm=False, do_ocr=True)
panels = assemble_panels(pages, results, data.manga_id, data.chapter_id, _put_crop)
return {"panels": panels}
@app.get("/health")
async def health():
return {"status": "ok"}
if __name__ == "__main__":
# self-check: model-free. Fake a 2-page Magi result and assert panel flattening,
# chapter-order indexing, text->panel containment, SFX drop, and bbox conversion.
transport.exists = lambda *a, **k: False # no minio in self-check
stored = {}
pages = [np.zeros((100, 100, 3), np.uint8), np.zeros((100, 100, 3), np.uint8)]
results = [
{"panels": [[0, 0, 50, 100], [50, 0, 100, 100]], # page 0: two panels
"texts": [[10, 10, 20, 20], [60, 10, 70, 20]], # one text in each
"ocr": ["HELLO", "BOOM"], "is_essential_text": [True, False]}, # BOOM = SFX, dropped
{"panels": [[0, 0, 100, 100]], # page 1: one panel
"texts": [[5, 5, 15, 15]], "ocr": ["WORLD"], "is_essential_text": [True]},
]
panels = assemble_panels(pages, results, "m", "c", lambda img, uri: stored.__setitem__(uri, img.shape))
assert [p["panel_index"] for p in panels] == [0, 1, 2], "chapter-order index"
assert panels[0]["ocr"][0]["content"] == "HELLO"
assert panels[1]["ocr"] == [], "SFX text dropped from panel 1"
assert panels[2]["ocr"][0]["content"] == "WORLD"
assert panels[0]["bbox"] == [0, 0, 50, 100], "xyxy->xywh"
assert panels[0]["ocr"][0]["bbox"] == [10, 10, 10, 10]
assert stored, "crops uploaded via put"
print("worker_parse self-check ok")
audit_registry.py
+93
View File
@@ -0,0 +1,93 @@
"""Registry audit for one chapter, after vision + identity + reconcile and before anything downstream.
Runs inside manga-orchestrator (reads /data/manga.db). `audit_speakers.py` answers the attribution
questions and needs the dialogue stage; this one answers the questions that decide whether dialogue is
worth running at all:
1. did the bbox fix land — are stored boxes pixels, or still gemma's 0-1000 grid,
2. how many characters did the rebaseline mint, and did one of them absorb the chapter again,
3. what happened on panel 7, the worked example.
Usage: docker exec manga-orchestrator python3 /app/audit_registry.py [chapter_id] [panel_index]
"""
import collections
import json
import sqlite3
import sys
CHAPTER = sys.argv[1] if len(sys.argv) > 1 else "7c944dd4-e972-42c7-ba60-9f6939548e80"
WORKED_EXAMPLE = int(sys.argv[2]) if len(sys.argv) > 2 else 7
c = sqlite3.connect("/data/manga.db")
c.row_factory = sqlite3.Row
manga_id = c.execute("SELECT manga_id FROM chapters WHERE chapter_id=?", (CHAPTER,)).fetchone()[0]
reg = {r["character_id"]: dict(r) for r in c.execute(
"SELECT character_id, name, aliases, gender, ref_image_uris, embedding_uri "
"FROM characters WHERE manga_id=?", (manga_id,))}
panels = c.execute(
'SELECT panel_id, panel_index, page_index, bbox FROM panels WHERE chapter_id=? ORDER BY panel_order',
(CHAPTER,)).fetchall()
# 1. coordinate space. A 0-1000 grid box on a panel wider or taller than 1000px cannot exceed 1000,
# and clamps AT 1000. Real pixel boxes track the panel and scatter past it. The tell is the ratio of
# the largest coordinate to the panel dimension, plus how many boxes sit exactly on 1000.
detections = 0
past_1000 = at_1000 = 0
max_ratio = 0.0
assigned_total = 0
per_char = collections.Counter()
worked = None
for p in panels:
row = c.execute("SELECT result_json FROM vision_results WHERE panel_id=?", (p["panel_id"],)).fetchone()
if not row:
continue
v = json.loads(row["result_json"])
# the vision blob carries no panel size. panels.bbox is the panel's box on its page and is
# [x, y, w, h], not corners — panel 3 of this chapter is [0, 615, 900, 106].
pb = json.loads(p["bbox"] or "null")
pw, ph = (pb[2], pb[3]) if pb and len(pb) == 4 else (None, None)
assigns = {a["local_id"]: (a["character_id"], a["confidence"]) for a in c.execute(
"SELECT local_id, character_id, confidence FROM identity_assignments WHERE panel_id=?",
(p["panel_id"],))}
assigned_total += len(assigns)
for cid, _ in assigns.values():
per_char[reg.get(cid, {}).get("name") or cid[:16]] += 1
people = [ch for ch in (v.get("characters") or []) if ch.get("bbox")]
detections += len(people)
for ch in people:
x1, y1, x2, y2 = ch["bbox"]
past_1000 += 1 if max(x2, y2) > 1000 else 0
at_1000 += 1 if 1000 in (x2, y2) else 0
if pw and ph:
max_ratio = max(max_ratio, x2 / pw, y2 / ph)
if p["panel_index"] == WORKED_EXAMPLE:
worked = (p, v, people, assigns, pw, ph)
named = [r for r in reg.values() if (r["name"] or "").strip()]
print(f"registry: {len(reg)} characters, {len(named)} named -> {sorted((r['name'] or '') for r in named)}")
print(f"detections: {detections} assignments: {assigned_total} "
f"= {100*assigned_total/max(detections,1):.0f}% coverage")
if per_char:
top, n = per_char.most_common(1)[0]
print(f"assignment spread: {dict(per_char.most_common(8))}")
print(f" top character holds {n}/{assigned_total} = {100*n/max(assigned_total,1):.0f}% "
f"({'ABSORBING, same signature as before' if n > 0.5 * assigned_total else 'ok'})")
print(f"bbox space: {past_1000}/{detections} boxes exceed 1000, {at_1000} sit exactly on 1000, "
f"largest coord/panel-dimension = {max_ratio:.2f}")
print(f" verdict: {'PIXELS' if past_1000 or max_ratio > 0.02 and at_1000 == 0 else 'STILL 0-1000 GRID'}")
missing_refs = [k for k, r in reg.items() if not r["ref_image_uris"] or not r["embedding_uri"]]
print(f"characters missing a ref crop or embedding: {len(missing_refs)}")
if worked:
p, v, people, assigns, pw, ph = worked
print(f"\npanel_index {WORKED_EXAMPLE} ({p['panel_id']}), {pw}x{ph}:")
for ch in people:
cid, conf = assigns.get(ch["local_id"], (None, None))
name = reg.get(cid, {}).get("name") or (cid[:16] if cid else "-- none --")
print(f" {ch['local_id']:10} {ch['bbox']!s:28} {name:22} "
f"{'' if conf is None else f'{conf:.2f}'}")
else:
print(f"\npanel_index {WORKED_EXAMPLE} not found in this chapter")
audit_speakers.py
+105
View File
@@ -0,0 +1,105 @@
"""Speaker-attribution audit for one chapter. Runs inside manga-orchestrator (reads /data/manga.db).
Answers the three questions asked of the 2026-08-11 rerun:
1. what happens on a multi-character panel whose people have no identity,
2. whether a known character gets mis-named,
3. what the named-speaker share actually is now.
"""
import collections
import json
import sqlite3
import sys
CHAPTER = sys.argv[1] if len(sys.argv) > 1 else "7c944dd4-e972-42c7-ba60-9f6939548e80"
SPEECH = {"speech", "shout", "thought"}
c = sqlite3.connect("/data/manga.db")
c.row_factory = sqlite3.Row
manga_id = c.execute("SELECT manga_id FROM chapters WHERE chapter_id=?", (CHAPTER,)).fetchone()[0]
reg = {r["character_id"]: dict(r) for r in c.execute(
"SELECT character_id, name, aliases, description FROM characters WHERE manga_id=?", (manga_id,))}
# duplicate registry rows make a correct name unresolvable: an answer matching two rows is ambiguous.
by_name = collections.defaultdict(list)
for cid, r in reg.items():
for n in [r["name"], *json.loads(r["aliases"] or "[]")]:
if n:
by_name[str(n).strip().casefold()].append(cid)
dupes = {n: ids for n, ids in by_name.items() if len(ids) > 1}
panels = c.execute("SELECT panel_id, panel_index, page_index FROM panels WHERE chapter_id=? ORDER BY \"order\"",
(CHAPTER,)).fetchall()
methods, kinds, per_char = collections.Counter(), collections.Counter(), collections.Counter()
speech = named = 0
crowded_speech = crowded_named = 0
solo_speech = solo_named = 0
unresolved_names = collections.Counter()
ambiguous = []
no_identity_crowd = 0
# a line attributed to a real present local_id that simply has no identity row: attribution succeeded
# and the name is lost anyway. This is the identity-coverage wall, not an attribution failure.
attributed_but_unassigned = collections.Counter()
for p in panels:
pid = p["panel_id"]
row = c.execute("SELECT result_json FROM vision_results WHERE panel_id=?", (pid,)).fetchone()
if not row:
continue
v = json.loads(row["result_json"])
if "dialogue" not in v:
continue
people = [ch for ch in (v.get("characters") or []) if ch.get("local_id")]
assigned = {a["local_id"] for a in c.execute(
"SELECT local_id FROM identity_assignments WHERE panel_id=?", (pid,))}
crowd = len(people) > 1
if crowd and not assigned:
no_identity_crowd += 1
for d in v["dialogue"]:
if d.get("type", "speech") not in SPEECH:
continue
speech += 1
ref = d.get("speaker_ref") or {}
kind = ref.get("kind")
methods[d.get("speaker_method")] += 1
kinds[kind] += 1
if crowd:
crowded_speech += 1
else:
solo_speech += 1
if kind == "character_id":
named += 1
per_char[reg.get(ref["value"], {}).get("name") or ref["value"]] += 1
if crowd:
crowded_named += 1
else:
solo_named += 1
elif kind == "unknown":
# normalize_dialogue nulls the flat `speaker` unless it resolved, but speaker_ref keeps the
# unresolved local_id as its value.
raw = str(ref.get("value") or "").strip()
local_ids = {ch["local_id"] for ch in people}
if raw in local_ids and raw not in assigned:
attributed_but_unassigned[d.get("speaker_method")] += 1
elif kind == "name":
unresolved_names[ref.get("value")] += 1
if ref.get("candidates"):
ambiguous.append((pid, ref.get("value"), ref["candidates"]))
print(f"panels with dialogue: {sum(1 for p in panels if (lambda r: r and 'dialogue' in json.loads(r['result_json']))(c.execute('SELECT result_json FROM vision_results WHERE panel_id=?', (p['panel_id'],)).fetchone()))}/{len(panels)}")
print(f"speech lines: {speech} named (character_id): {named} = {100*named/max(speech,1):.0f}%")
print(f" multi-character panels: {crowded_named}/{crowded_speech} named")
print(f" single-character panels: {solo_named}/{solo_speech} named")
print(f"multi-character panels with NO identity at all: {no_identity_crowd}")
print(f"attributed to a present local_id with NO identity row: {dict(attributed_but_unassigned)}")
print(f"speaker_method: {dict(methods)}")
print(f"speaker_ref kind: {dict(kinds)}")
print(f"named per character: {dict(per_char)}")
print(f"unresolved name refs: {dict(unresolved_names)}")
print(f"registry duplicate names (block a correct bind): "
f"{ {n: [reg[i]['name'] for i in ids] for n, ids in dupes.items()} }")
if ambiguous:
print("ambiguous binds:")
for pid, val, cands in ambiguous[:20]:
print(f" {pid} {val!r} -> {[reg.get(x, {}).get('name') for x in cands]}")
bubble_detect.py
+1 -1
View File
@@ -38,7 +38,7 @@ def _letterbox(img, sz=1024):
return canvas, r, px, py return canvas, r, px, py
def detect_text_regions(img, conf: float = None) -> list: def detect_text_regions(img, conf: float | None = None) -> list:
"""img: BGR ndarray (one panel). -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels, """img: BGR ndarray (one panel). -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels,
reading order (top->bottom, left->right). Empty list on no detections (caller falls back to the reading order (top->bottom, left->right). Empty list on no detections (caller falls back to the
holistic prompt). Load + inference are lazy so importing this never touches the GPU or the model.""" holistic prompt). Load + inference are lazy so importing this never touches the GPU or the model."""
caveats/CLAUDE.md
+47
View File
@@ -0,0 +1,47 @@
# caveats/
One entry per known breakage, deferral, or trap. An entry names what fails, who pays for it, and the
concrete trigger that makes fixing it worth the time.
A caveat that gets resolved moves to `decisions/` or gets deleted. A caveat with no revisit trigger is
a complaint, so give it one or drop it.
## Rules for this directory
* One file per source, one `##` section per limit.
* State the limit as what fails, not as a topic. "Reconcile deletes the losing character" beats
"Reconcile".
* Delete an entry the moment its trigger fires and the work lands.
## Index
| caveat | source |
| --- | --- |
| [Reconcile deletes the losing character irreversibly](audit-open.md#destructive-reconcile) | AUDIT.md |
| [Clearing a stage does not undo what it wrote](audit-open.md#dishonest-clearing) | AUDIT.md |
| [ComfyUI uses the GPU outside the session mutex](audit-open.md#comfyui-unscheduled) | AUDIT.md |
| [The JSON repair pass can fabricate dialogue](audit-open.md#repair-fabricates) | AUDIT.md |
| [`/review/preview` pins a solo clip into the final video](audit-open.md#preview-overwrites-clip) | AUDIT.md |
| [Stored embeddings carry no model or pooling version](audit-open.md#untagged-embeddings) | AUDIT.md |
| [Worker endpoints block the event loop](audit-open.md#blocking-event-loop) | AUDIT.md |
| [SQLite has no busy timeout, so parallelism will surface as lock errors](audit-open.md#sqlite-locking) | AUDIT.md |
| [`layers` runs after `tts`, so pipelined solo beats lose parallax](audit-open.md#layers-after-tts) | AUDIT.md |
| [`completed` means something different in each stage](audit-open.md#inconsistent-stage-policy) | AUDIT.md |
| [Identity worker caches characters the orchestrator has deleted](audit-open.md#stale-known-cache) | AUDIT.md |
| [The gemma helpers exist twice and have diverged](audit-open.md#gemma-helpers-duplicated) | repo audit |
| [A character seen once gets no assignment at all](audit-open.md#pending-in-worker-memory) | AUDIT.md |
| [MinIO credentials are hardcoded in committed source](audit-open.md#hardcoded-credentials) | AUDIT.md |
| [Assemble marks a job completed with no clips](audit-open.md#empty-assemble) | AUDIT.md |
| [Reviewer timestamps drift against the crossfaded video](audit-open.md#timeline-drift) | AUDIT.md |
| [Nothing attributes a speaker in a multi-character panel](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run |
| [One character id covers two different women](speaker-attribution.md#identity-over-merge) | 2026-08-11 run |
| [The character registry carries five weeks of wrong names](speaker-attribution.md#registry-pollution) | 2026-08-11 run |
| [One invented word still halts the chapter](speaker-attribution.md#one-word-halts-chapter) | 2026-08-11 run |
| [A completed job keeps the error from an earlier failure](audit-open.md#stale-job-error) | 2026-08-11 run |
| [`layers` reports success on an empty bucket](audit-open.md#layers-writes-nothing) | 2026-08-11 run |
| [Every `bbox` is read in the wrong coordinate space](speaker-attribution.md#bbox-wrong-space) | resolved, rerun pending |
| [Identity cannot say "a person with no name"](speaker-attribution.md#no-anonymous-identity) | 2026-08-12 panel 7 |
| [Vision does not separate a background extra from cast](speaker-attribution.md#extras-as-cast) | 2026-08-12 panel 7 |
| [Cast reference profiles are enrolled from wrong crops](speaker-attribution.md#poisoned-reference-set) | 2026-08-12 panel 7 |
| [Identity coverage has fallen on every run since the gate landed](audit-open.md#coverage-trend) | before the next downstream run |
| [Embedding cosine on a person crop cannot tell two people apart](audit-open.md#cosine-not-identity) | before building on cosine |
caveats/audit-open.md
+273
View File
@@ -0,0 +1,273 @@
# Open limits from the 2026-08-11 audit
Everything here was read in source during the audit and deliberately left unfixed in Phase 1. The
fixed findings live in `decisions/audit-phase1.md`. Line numbers are from the audit and may drift.
## Reconcile deletes the losing character irreversibly {#destructive-reconcile}
Reconciliation used to delete the losing character row. Clearing the reconcile stage did not undo it, and
nothing recorded which detections had been the loser's.
**Half-closed 2026-08-12.** The loss is no longer unrecoverable. `merge_characters` marks the loser
`merged_into = keeper` instead of deleting it, so its embedding, description and gender survive, and every
repointed assignment is stamped `method = merged_from:<loser_id>` in `identity_assignment_sources`. Those
two records are enough to walk a merge backwards. Roster readers filter `merged_into IS NULL`. Lookup by
id does not, so an assignment still pointing at a merged id resolves.
What is still missing is the mechanism that consumes them: there is no unmerge, and no split. Undoing a
merge today means a manual SQL walk of the two records above.
Costs: a wrong merge needs hand-written SQL to undo, not a rebaseline.
Revisit when: a reviewer reports a wrong merge, or the review gates from [#136] get a place to hang
name/merge/split actions. No wrong merge has been seen since the crops were fixed, so building the unmerge
before either trigger would be speculative.
## Clearing a stage does not undo what it wrote {#dishonest-clearing}
The dialogue and direct half of this is fixed and proven
(`decisions/storage-layout.md#clear-vision-blob`). It cost a wasted rerun on 2026-08-11 first: the clear
returned `{"ok": true}`, deleted nothing, and the stage skipped all 116 panels.
What remains: clearing identity preserves the per-manga registry by design, so a rerun inherits every
character it ever minted (`caveats/speaker-attribution.md#registry-pollution`). Nothing verifies that a
clear emptied what it claimed.
Costs: a rerun silently reuses stale output, which reads as a reproducible result.
Revisit when: any stage is scheduled concurrently or resumed automatically. A stage must be idempotent
before either is safe.
## ComfyUI uses the GPU outside the session mutex {#comfyui-unscheduled}
The layers stage calls ComfyUI directly and takes no lease. Another job can load gemma, siglip2, or
dots while ComfyUI holds the same GPU.
Costs: out-of-memory failures that look random and land on an unrelated stage.
Revisit when: layers is enabled on a real run, or a second concurrent job is allowed.
Workaround: `MAX_CONCURRENT_JOBS=1` keeps one pipeline at a time, which is the current default.
## The JSON repair pass can fabricate dialogue {#repair-fabricates}
`call_gemma4_json` hands the model its own truncated text and asks for the JSON it should have been
(`worker_vision.py`). The repair call carries no image. On a response truncated by `max_tokens`, the
model completes dialogue it can no longer see. What it adds is indistinguishable downstream from
transcribed text.
Costs: invented lines enter the script with normal provenance.
Revisit when: schema-constrained generation lands, which removes most of this path.
Workaround: resend the image on repair, or retry a truncated response instead of repairing it.
## `/review/preview` pins a solo clip into the final video {#preview-overwrites-clip}
`review_preview` renders one panel and calls `save_clip(panel_id, ...)`. `_render_one_beat` returns
early when a clip already exists for the leader. Previewing a beat leader therefore drops the rest of
the beat's panels. `review_retts` gets this right by calling `delete_clip` first.
Costs: a reviewer silently corrupts the output by looking at it.
Revisit when: the review UI is used on a real chapter.
Workaround: never preview a beat leader, or delete the clip row afterwards.
## Stored embeddings carry no model or pooling version {#untagged-embeddings}
`embed_crop` uses `pooler_output` when present and falls back to mean-pooled patch tokens otherwise.
The two paths produce different vector spaces, and the fixed 0.85 threshold is valid for one of them.
Nothing beside a stored `.npy` records which model, revision, or pooling produced it.
Costs: a transformers upgrade mixes incompatible vectors into one gallery with no error.
Revisit when: transformers or the siglip2 revision is upgraded. Before, not after.
Workaround: none. Write the model id and pooling mode beside the vector and refuse cross-version
comparison.
## Worker endpoints block the event loop {#blocking-event-loop}
Endpoints declared `async def` run blocking MinIO, OpenCV, torch, and ffmpeg calls directly on the
event loop. A busy worker cannot answer `/health` or `/unload`.
Costs: `/health/workers` reports a working worker as unreachable, and the session manager's 30-second
`/unload` can time out exactly when VRAM needs freeing.
Revisit when: a stage stalls on `/unload`, or before any bounded parallelism lands.
Workaround: `def` instead of `async def` moves each handler to the threadpool. Tracked as [#199] for
the render worker.
## SQLite has no busy timeout {#sqlite-locking}
`get_conn` opens a connection per call with no `busy_timeout`. WAL tolerates one writer. `PIPELINE=1`
already writes clips from concurrent tasks while TTS writes audio.
Costs: planned CPU parallelism will surface as `database is locked` before it surfaces as throughput.
Revisit when: Phase 2 concurrency work starts. Set the timeout first.
Workaround: keep `PIPELINE` off.
## `layers` runs after `tts` {#layers-after-tts}
`STAGES` orders `layers` after `tts`, while `run_stage_tts` warns that eager rendering under
`PIPELINE=1` needs layers to run first.
Costs: with the flag on, solo beats always render without parallax.
Revisit when: a real run enables `PIPELINE=1`.
Workaround: keep `PIPELINE` off, or reorder `STAGES`.
## `completed` means something different in each stage {#inconsistent-stage-policy}
A dropped vision panel fails the stage and halts the pipeline. A failed direction window counts its
panels as done. Layers always finishes completed.
Costs: the acceptance metrics in `ROADMAP.md` cannot be read across stages.
Revisit when: a baseline measurement is taken. The numbers are meaningless until then.
Workaround: none.
## Identity worker caches characters the orchestrator has deleted {#stale-known-cache}
`worker_identity._known_cache` is invalidated only by `_persist_char`. The reconcile stage deletes
losing characters directly in the orchestrator database. A long-lived worker keeps shortlisting and
assigning ids that no longer exist.
Costs: assignments point at rows that are gone.
Revisit when: reconcile runs on a chapter without a worker restart between stages. Tracked as [#201],
which proposes caching per `manga_id`.
Workaround: restart the identity worker after reconcile.
## A character seen once gets no assignment at all {#pending-in-worker-memory}
`worker_identity._pending` holds full crop images in worker memory for a whole chapter, keyed by
session. That is durable per-chapter state inside a worker documented as stateless, and it is lost on
restart. A character seen exactly once receives no assignment, not even a chapter-local handle.
Costs: one-off characters vanish from the scene graph.
Revisit when: chapter-local tracklet persistence lands (`ROADMAP.md`, Phase 3).
Workaround: none.
## MinIO credentials are hardcoded in committed source {#hardcoded-credentials}
Defaults live in `transport.py` and `service.py`.
Costs: the credentials are in git history for anyone who gets the repo.
Revisit when: the repo leaves this machine, or MinIO is reachable outside the LAN.
Workaround: the environment variables already override them. Set them and remove the defaults.
## Assemble marks a job completed with no clips {#empty-assemble}
`run_stage_assemble` does not check that `clip_uris` is non-empty before assembling, then marks the job
completed.
Costs: a failed chapter reports success.
Revisit when: any run reports completed without a video. One `if not clip_uris` guard fixes it.
Workaround: none.
## Reviewer timestamps drift against the crossfaded video {#timeline-drift}
`/review/panels` sums per-panel audio durations. Assemble crossfades clips using the per-beat
transitions, so every non-`cut` transition shortens the real video.
Costs: reviewer timestamps drift further out of sync the further into the chapter they scrub.
Revisit when: the review UI is used for timing work.
Workaround: subtract the transition overlaps by hand.
## A completed job keeps the error from an earlier failure {#stale-job-error}
`/job/resume` does not clear `jobs.error`. Job `778297bc` finished every stage and still reports
`status: "completed"` beside `error: "partial: 112/116 completed"`, a message from three resumes earlier.
Costs: any reader of the error field sees a failure on a successful job. The review UI and any future
alerting both read it.
Revisit when: anything branches on `jobs.error`, or a run is judged by its status alone.
## `layers` reports success on an empty bucket {#layers-writes-nothing}
Measured on job `778297bc`: `layers` reported `completed 116/116` while `s3://layers/` held 0 objects.
Every clip in that run therefore has no parallax. The stage is a sibling of
[#inconsistent-stage-policy](#inconsistent-stage-policy), but this is the measured instance:
`worker_layers.py`'s own self-check already fails on the missing
`legacy/qwen_layered_workflow.json`, and the stage still reports done for every panel.
Costs: a silent quality regression that no status field reveals.
Revisit when: parallax matters for a deliverable, or before quoting this run as a full-pipeline pass.
## Identity coverage has fallen on every run since the gate landed {#coverage-trend}
70% -> 61% -> 50% -> 57% across the four 2026-08-12 runs. The gate is stable at 39 to 40% of detections.
**The 18:07 run resolves most of this.** Coverage is now the gate and nothing else. All 68 assignments went
to face-bearing detections and none to a gated one. Recall among face-bearing detections is 68 of 71, or
96%, up from 82%. The resolver is not losing cast. The 43% that carries no assignment is the 40% the gate
drops plus 3 detections.
So the coverage number no longer measures the resolver. What it measures is the cost of requiring a face,
which is item 4 in `NEXT.md`. Panel 7 is the case: 5 of its 6 detections carry `has_face = False`, and the
one that does is unassigned.
The open question moved. The lead now holds 36 of 68 assignments, or 53%, against 25 of 59 before, and
`audit_registry.py` flags that as absorbing. Whether 36 is the protagonist appearing in half the panels or
an over-merge cannot be told apart without labels.
Revisit trigger: before trusting the registry for a downstream run.
**The 145 ground-truth labels do not cover this chapter.** Measured 2026-08-12. All 145 rows in
`identity_labels` key to chapter `8ca8249b`, a different manga with cast "Rico" and "Ikekin", over 81
distinct panels. Chapter `7c944dd4` has zero labels, so `/review/identity?job_id=778297bc...` returns
`labeled: 0, accuracy: null`. Scoring 8ca instead gives 7/138, because its identity run is stale: 44
assignments over 246 panels, predating all seven fixes. That number measures the old code on the wrong
chapter and settles nothing.
The plumbing works, the labels are missing. Precision and recall on the current registry need a hand
pass over `7c944dd4` first, via `POST /review/identity/label` per detection. Label against the bbox, not
the `local_id`. Vision is non-deterministic and reassigns `local_id` top-to-bottom every run. A label
taken before a vision rerun then points at whoever now occupies that slot.
## Embedding cosine on a person crop cannot tell two people apart {#cosine-not-identity}
Measured 2026-08-12 on the 18:07 run's 22 crops for `character_2367d70c`, the embeddings pulled from
`manga/<manga_id>/characters/_crops/*.npy` and L2-normalised. 1152 dimensions.
| pair | cosine |
| --- | --- |
| the cat at `p098` against any man | up to 0.82 |
| the red-robe chibi at `p088` against any man | up to 0.75 |
| two visibly different men (`order 60~80`, `88~110`) | 0.93, 0.92 |
| the highest pair in the matrix (`47~51`, one man, one shirt) | 0.96 |
Different people reach 0.93 and the same person reaches 0.96. No threshold separates them. So a
crop-to-crop cosine link is not available, and the `candidates` overlap condition in `link_tracklets` is
close to noise for the same reason: the top-k is chosen by this metric, so every crop shortlists the same
few rows.
Inference, not yet tested: identity embeds the whole person box, which holds background, clothing and pose.
Those change between scenes while every crop shares one art style, so cosine measures "manga crop of a
person in an office". `face_detect` already finds the face and pairs it to the person box for `has_face`,
so embedding the face region instead is a small change.
Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22
detections, embed them, and recompute this matrix. If a threshold appears, embed faces.
## The gallery cap drops exactly the rows a tracklet needs {#gallery-cap-drops-the-unnamed}
**Open. Observed on the 19:44 run of 2026-08-12.**
`cast_gallery` orders the live cast named-first, then registry order, and `run_stage_identity` truncates
the tail at `GALLERY_CAP = 10`. Registry order is creation order, so the tail is the anonymous characters
this very run minted — and a later tracklet of a recurring unnamed person is precisely what needs to match
one of those. The chapter has two named rows, so on this run the cap dropped 6 unnamed rows at `p097` and
1 at `p109`, and a tracklet that should have joined one of them can only mint a duplicate instead.
It is capped because every candidate costs at least one reference image in the resolve prompt beside the
query crop. The cap is a VRAM and context budget, not a modelling choice.
**Revisit when** the cast on a chapter routinely exceeds 10 gender-compatible rows, which it already did
here. The fix is to order the gallery by how many assignments each character already holds in this chapter,
so the tail is the rows nobody has matched rather than the rows nobody has named yet. That needs one count
query per tracklet.
## The gemma helpers exist twice and have diverged {#gemma-helpers-duplicated}
**Open. Found by the repo audit of 2026-08-13, not yet by a failure.**
`call_gemma4`, `_extract_json` and `_strip_thought` are defined in both `worker_vision.py` and
`worker_script.py`. They are no longer the same code. `worker_vision.call_gemma4` is 21 lines and takes a
content list; `worker_script`'s is 8 and takes a prompt string plus a system prompt. Both `_extract_json`
bodies carry the same comment about `raw_decode` stopping at the first object, and raise different errors.
Who pays: whoever fixes the JSON path. `#repair-fabricates` above says the repair pass can fabricate
dialogue. A fix written against one copy leaves the other worker on the old behaviour, and no self-check
compares them.
**Revisit when** either JSON path is touched for any reason. The fix is a `gemma.py` holding the three
helpers, imported by both workers, which removes about 25 duplicated lines and costs one file.
caveats/speaker-attribution.md
+162
View File
@@ -0,0 +1,162 @@
# speaker-attribution
Limits found by cross-checking the 2026-08-11 chapter run against the panel images.
## Nothing attributes a speaker in a multi-character panel {#tail-is-not-geometry}
The false `tail` label is gone (`decisions/speaker-attribution.md#no-fake-tail`). What replaced it is a
refusal, not an answer: with two or more characters present, every speech line now returns `unknown`.
On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back
to a `generic-handle`. That is the honest floor, and it is not the fix.
The measurement that forced it, on job `778297bc`, chapter `7c944dd4`. Three two-character panels were
checked against the art. All three are wrong, each with the two speakers swapped:
| panel key | line | truth | pipeline |
| --- | --- | --- | --- |
| `p010.png` | "…definitely an Egen guy, Seonho!" | the woman | Seonho, the person addressed |
| `p010.png` | "Y-you think so?" | Seonho | Choi Haeseon |
| `p012.png` | "Want me to send you the link?" | the woman | the man |
| `p059.png` | "If team leader Choi says it, it must be true." | the man | Choi Haeseon |
The last row needs no image: the line refers to Choi in the third person and is attributed to Choi.
The grounded path exists and almost never fires. Only 2 of 81 speech lines got `som_face`, because
attribution marks need `face_detect` boxes that survive `_pair_faces_to_present`, and these webtoon
close-ups rarely produce them. Inference, not measured: the face detector was not instrumented.
`worker_vision.py:356` already carries the `ponytail:` note that multi-character attribution needs
per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads carry balloon fill and
tail tips and are unused.
**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric
(`ROADMAP.md`). The 30% read on this run counted attributions the sample says are wrong. The next run
will read lower and will be the first honest number. Raising it means binding a balloon to a speaker
by tail geometry, using the unused `det`/`seg` heads.
## One character id covers two different women {#identity-over-merge}
`character_afa7623b` is stored as "black bob, white sweater" and is assigned both to that person
(`p059.png`) and to the brown-bob green-top coworker (`p010.png`, `p012.png`). It took 25 of the 26
identity assignments in the chapter, against 113 detected people. Coverage is 23%.
**Revisit trigger:** any work on the identity Tier-2 decider. A single id absorbing a whole chapter is
the signature to watch for.
## The character registry carries five weeks of wrong names {#registry-pollution}
The registry holds 53 characters for manga `ef105a86`, 41 of them unnamed, with "Kei" three times and
"Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only
Haeseon and Seonho do. `/stage/clear` leaves the per-manga registry intact by design, so every rerun
inherits the whole pile.
Duplicate rows also make a correct name unresolvable. This manga holds `Choi Haeseon`, `Seonho` with
aliases `["Lim Seonho", "Seonho"]`, and a separate `Lim Seonho`. An answer of "Lim Seonho" matches two
rows, so `normalize_speaker` returns candidates and raises `ambiguous-speaker` instead of binding. The
pipeline read the name correctly and still cannot name the speaker.
**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the
registry to a chapter or add a reviewed reset. Merging the duplicate rows needs the reversible-merge
design first (`caveats/audit-open.md#destructive-reconcile`).
## One invented word still halts the chapter {#one-word-halts-chapter}
The multi-word name failure is fixed (`decisions/speaker-attribution.md#multiword-cast-names`). The
blast radius it exposed is not. `run_stage_script` retries a rejected beat once, then raises, so a
single unsupported token ends the run at that beat. On job `778297bc` one of the 29 lost beats cited
`['Blur']`, an onomatopoeia the model invented. The verifier was right, and the whole chapter still
stopped.
**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer
is to flag the beat for review and continue, which is `#136` gate work, not a verifier change.
## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space}
**Resolved 2026-08-12, `decisions/identity-bbox.md#bbox-is-normalized`.** The space is gemma's 0-1000
grid, proven over all 113 detections, and `/vision` now converts to pixels before returning. The face
pairing at `worker_vision.py:57` was reading the same numbers against real pixel face boxes, so it is
fixed by the same change. What is left of this entry is the consequence. Every stored assignment,
embedding and `ref_image_uris` came from a wrong crop. Identity has to re-run before any of it means
anything. The rest below is kept as the record of how it read before.
Vision's `bbox` values are stored and consumed as absolute pixels. On panel
`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of
the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own
dimensions, four of the six fit their subjects tightly.
Two places in the code assert pixels, and the art contradicts both:
- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces
against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9
from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed
alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at
`worker_vision.py:57` reads the same numbers and was not checked.
Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with
nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already
calls the box "coarse, imprecise".
Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can
be judged while the crops are wrong.
## Identity cannot say "a person with no name" {#no-anonymous-identity}
The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9.
Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline
stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which
is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing
line.
This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an
unparseable model answer. The missing rule: never attach a name to a detection that carries no name
evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the
colleague" every time.
`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from
cosine clearing the threshold on a wrong crop.
**Resolved 2026-08-12, `decisions/identity-bbox.md#none-mints-an-anonymous-character`.** The gemma resolver
can answer "none of these" and always could. The orchestrator was discarding the answer. It read only
`character_id`, so a deliberate NONE and a hallucinated index both unassigned every crop of the tracklet.
A NONE now mints an anonymous character from the crop, using the embedding identity ships beside it. Not
yet proven on a GPU run.
## Vision does not separate a background extra from cast {#extras-as-cast}
Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground
and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame
with nobody in it. All six reach identity as equal candidates.
Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly
extras, so it measured nothing useful and should not be quoted again.
Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return.
## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set}
`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the
cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are
crops of balloon edges, window frames and background extras rather than of faces.
The visual comparison people reach for as the fix is **already implemented**, so do not build it again.
`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images
per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair
and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for
NONE when unsure. `choice: 0` returns `state="new"` and an out-of-range index returns `unresolved`. What
the orchestrator does with each is
`decisions/identity-bbox.md#none-mints-an-anonymous-character`. The
`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`,
so the references reach the model.
That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon
edge against references enrolled from window frames and background extras, then sometimes answers "same".
Nothing gates enrollment on the crop holding a face.
Who pays: every later match, because the reference set defines what a character looks like. Fixing
`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place.
Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the
existing `ref_image_uris` and `embedding_uri` values as invalid.
check_stale.sh
Executable
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Which running workers are serving code older than the file on disk.
#
# Python loads a worker module once, at process start. Editing the file afterwards changes nothing
# until that process restarts, and nothing in the stage output says so: the stage completes, the
# counters read 116/116, and the result is what the OLD code produced. On 2026-08-12 this ate a full
# vision + identity + reconcile cycle — the bbox fix was committed 16 minutes after the worker that
# was supposed to apply it had already started.
#
# Run this before any stage run that is meant to prove a code change.
# Exits non-zero if anything is stale, so it can gate a script.
set -uo pipefail
cd "$(dirname "$0")"
stale=0
# `ps -o lstart` is always exactly five fields: "Wed Aug 12 13:20:00 2026". Read them positionally
# rather than trying to split the line with a regex.
while read -r pid _dow mon day time year cmd; do
mod=$(grep -oP '(?:worker_\w+|session_manager)(?=:app)' <<<"$cmd" | head -1)
[ -n "$mod" ] && [ -f "$mod.py" ] || continue
p_epoch=$(date -d "$mon $day $time $year" +%s 2>/dev/null) || continue
f_epoch=$(stat -c %Y "$mod.py")
if [ "$f_epoch" -gt "$p_epoch" ]; then
printf 'STALE %-22s pid %-7s edited %dm after it started\n' \
"$mod.py" "$pid" "$(( (f_epoch - p_epoch) / 60 ))"
stale=1
else
printf 'ok %-22s pid %s\n' "$mod.py" "$pid"
fi
done < <(ps -eo pid,lstart,cmd | grep -E 'uvicorn (worker_|session_manager)' | grep -v grep)
if [ "$stale" -ne 0 ]; then
echo
echo "Restart the stale ones before trusting a stage result. Window numbers:"
tmux list-windows -t manga-workers -F ' #{window_index} #{window_name}' 2>/dev/null \
|| echo " (tmux session manga-workers is not running; ./start_workers.sh)"
fi
exit "$stale"
decisions/CLAUDE.md
+56
View File
@@ -0,0 +1,56 @@
# decisions/
One entry per settled question about this pipeline. An entry names the claim, the evidence, the date,
and what it forbids or permits next.
No entry is a plan. `NEXT.md` holds the plan, `ROADMAP.md` holds the horizon, and a limitation that is
still live belongs in `caveats/`.
## Rules for this directory
* One file per topic, one `##` section per question, so the index can link an anchor.
* An entry cites a file and line, a commit, or a test. No citation means it is an opinion, and an
opinion gets deleted rather than defended.
* **Closed** means acting on it is safe. **Open** means investigated and undecided. **Void** means the
evidence turned out to be invalid and the claim must not be cited again.
* Rewrite an entry when something contradicts it, and say what changed.
## Index
| decision | state |
| --- | --- |
| [A dialogue row's speaker is `speaker_ref`, not `speaker`](audit-phase1.md#speaker-ref-is-canonical) | closed |
| [Script verification must not fail on sentence-initial capitals or short quotes](audit-phase1.md#verifier-false-positives) | closed |
| [Tracklet links are transitive over similarity, never across a hard constraint](audit-phase1.md#tracklet-hard-constraints) | closed |
| [A detected face takes an identity only when it falls inside that character's box](audit-phase1.md#gated-face-pairing) | closed |
| [A 409 from the GPU mutex is a queue signal, not a stale lease](audit-phase1.md#no-lease-stealing) | closed |
| [Correctness flags resolve by flag id, and never block autonomous TTS](audit-phase1.md#flag-resolution) | closed |
| [An out-of-range resolver index is `unresolved`, never a new character](audit-phase1.md#hallucinated-index) | closed |
| [The session manager holds no lock across a model load](audit-phase1.md#unlocked-model-load) | closed |
| [One bucket per artifact class, not everything under `manga`](storage-layout.md#bucket-per-artifact) | closed |
| [The orchestrator creates missing buckets at startup](storage-layout.md#ensure-buckets) | closed |
| [RustFS is staged, not adopted](storage-layout.md#rustfs-staged) | open |
| [A model guess is never labelled `tail`](speaker-attribution.md#no-fake-tail) | closed |
| [The model's speaker answer is resolved against what the prompt showed](speaker-attribution.md#prompt-label-answers) | closed |
| [An unpaired set-of-mark label grounds nothing](speaker-attribution.md#unpaired-mark) | closed |
| [An interjection is not a name and not a misquote](speaker-attribution.md#interjection-false-positive) | closed |
| [Cast names enter the verifier tokenized](speaker-attribution.md#multiword-cast-names) | closed |
| [Clearing a stage strips the vision blob it wrote](storage-layout.md#clear-vision-blob) | closed |
| [The shipped 72.7s gap was a stream copy across mixed frame rates](chapter-assembly.md#mixed-rate-stream-copy) | closed |
| [xfade offsets come from `min(video, audio)`, never `format=duration`](chapter-assembly.md#offsets-from-min-stream) | closed |
| [A vision `bbox` is gemma's 0-1000 grid, converted to pixels at `/vision`](identity-bbox.md#bbox-is-normalized) | closed |
| [Assembly verifies its own output instead of trusting ffmpeg's exit code](chapter-assembly.md#check-assembled) | closed |
| [The single-item passthrough is not the assembly bug](chapter-assembly.md#passthrough-innocent) | void |
| [A stage result proves nothing until the worker is newer than the edit](identity-bbox.md#stale-worker-invalidates) | closed |
| [A detection with no detected face never enrolls or binds](identity-bbox.md#face-gates-enrollment) | closed |
| [A resolver NONE mints an anonymous character, it does not clear the crop](identity-bbox.md#none-mints-an-anonymous-character) | closed |
| [The extras gate runs at enrollment and at narration, not at the speaker prompt](identity-bbox.md#extras-gate-consumers) | closed |
| [A roster name is a guess, so it never reaches detection](identity-bbox.md#roster-does-not-name) | closed |
| [`merged_into` is exactly one hop deep](identity-bbox.md#merge-chains-flatten) | closed |
| [`_bbox_to_pixels` orders the corners, because the model sometimes swaps them](identity-bbox.md#bbox-corners-ordered) | closed |
| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed |
| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed |
| [The gallery is the live cast, not cosine's top-k](identity-bbox.md#cast-is-the-gallery) | closed |
| [A name is a word set, not a string](identity-naming.md#alias-grouping) | closed, GPU pending |
| [A confident caption names a character on its own](identity-naming.md#caption-is-strong) | closed, GPU pending |
| [A name belongs to one character](identity-naming.md#one-name-one-character) | closed, GPU pending |
decisions/audit-phase1.md
+144
View File
@@ -0,0 +1,144 @@
# Audit Phase 1: correctness and scheduling safety
Settled 2026-08-11 from `AUDIT.md`. Every entry was verified by a runnable check in the same commit.
No GPU work ran and no pipeline run was executed. Each claim rests on source and on the CPU-only
self-checks named below.
Files: `worker_scene.py`, `worker_script.py`, `worker_vision.py`, `session_manager.py` on workpc, and
`tracklets.py`, `correctness.py`, `db.py`, `service.py`, `session_proxy.py` in the homesrv orchestrator
(`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`).
## Speaker ref is canonical {#speaker-ref-is-canonical}
**Closed.** A dialogue row's speaker is `speaker_ref` when its kind is `character_id`. The flat
`speaker` field is a compatibility value. The orchestrator already rewrote it from a panel-local id to
a `character_id`.
Evidence: `correctness.py:normalize_dialogue` rewrites `row["speaker"]`. `worker_scene.build_scene`
mapped that value through `id_by_local`, whose keys are panel-local ids. Every lookup returned `None`,
so `worker_script._render_line` narrated every line as `Someone`.
Forbids: reading `speaker` as a panel-local id anywhere downstream of the dialogue stage.
Check: `python worker_scene.py`, the `p007` case.
The same field leaked into the dialogue resume path, where a stored `character_id` was pushed into
`recent` as a speaker name. It now renders the registry name (`service.py`, `run_stage_dialogue`).
## Verifier false positives {#verifier-false-positives}
**Closed.** `verify_script` must stay quiet on valid narration. A failure is not soft. The script stage
retries once and then raises, so one false positive halts the whole chapter.
Two rules fired on correct output.
1. `unsupported-proper-noun` flagged any capitalized token outside a 20-word stop list. That caught
ordinary sentence openers (`Suddenly`, `Behind`, `Inside`, `Both`, `Everyone`, `After`, `Two`) and
`Someone`. A capital opening a sentence is now grammar unless the word falls outside a real
narration vocabulary. A capital in the middle of a sentence still counts as name evidence.
2. `misquote` compared each quoted span against a WHOLE source line at ratio 0.82. The narration prompt
asks for a short quote of the actual words. A 15-character quote from a 40-character line scored
about 0.55. Grounding now matches the longest contiguous span instead.
Forbids: adding a verifier rule without a test that a correct beat passes it.
Check: `pytest test_script_verify.py` in the orchestrator.
Verifier feedback now reaches the retry. `worker_script.ScriptInput` declares `beat` and
`verifier_feedback`. Pydantic v2 dropped both silently before, so the retry was another blind sample.
The repair prompt names each failure and lists the exact quotable lines from the beat.
## Tracklet hard constraints {#tracklet-hard-constraints}
**Closed.** Tracklet linking is transitive over similarity evidence and never across a hard constraint.
The hard constraints are gender and co-presence. Gender is canonicalized first. Vision emits
`m|f|unknown` while the registry says `male|female`, and the old gate tested only the second spelling,
so it never fired on real data. Co-presence means two detections in one panel are two people by
construction. A merge is rejected when any cross pair between the two groups violates either rule.
A weak chain can no longer bridge two people seen together.
Forbids: comparing a raw gender string against a literal enum anywhere in the identity path.
Check: `python tracklets.py` and `pytest test_tracklets.py`.
## Gated face pairing {#gated-face-pairing}
**Closed.** A detected face takes a character's identity only when its centre falls inside that
character's gemma bbox. The box is grown by 25% first. An unpaired face stays `unknown`.
This matters because the label becomes `speaker_method="som_face"`, the highest-trust provenance the
pipeline records. Unconditional nearest-neighbour pairing laundered a geometric guess into evidence.
Pairs are now taken globally shortest-first, so the first face processed cannot claim a character that
fits a later face better.
Set-of-mark attribution stays ON by default now that the pairing is gated. The docstring said "off by
default" while the flag read `SOM_ATTRIBUTION` with default `"1"`. The code and the comment now agree.
Forbids: trusting a `som_face` speaker without the containment gate.
Check: `python worker_vision.py`.
## No lease stealing {#no-lease-stealing}
**Closed.** A 409 from `/session/open` means another job legitimately holds the GPU. The proxy queues
on it. Only the session manager's TTL reaper clears a dead lease, because only it can tell a dead
lease from a busy one.
`session_proxy.open_session` previously closed the active lease and retried, so two jobs terminated
each other's models. Job admission control now bounds concurrent pipelines through
`MAX_CONCURRENT_JOBS`, default 1. A lease queue alone still lets two pipelines interleave stages
against one GPU.
`heartbeat_session` is finally called. `open_session` starts a beat task and `close_session` cancels
it, so a stage running past its TTL is no longer reaped mid-run. Beating from open and close keeps all
nine stage call sites unchanged.
Forbids: closing a session the caller did not open.
## Flag resolution {#flag-resolution}
**Closed.** Correctness flags resolve by flag id, and unresolved flags never wedge an autonomous run.
Two defects combined into a permanent block. `get_correctness_flags` accepted rows whose panel no
longer existed. A flag raised against a panel that a crop rerun deleted became visible to every
chapter and could never be resolved. `resolve_correctness_flags` could only touch flags whose
panel still existed. The view no longer admits orphans, and resolution takes explicit flag ids.
The TTS block also had no reachable exit. It refused to start on four flag kinds that only
`/review/approve` clears, and `GATES` defaults to off. With gates on, TTS now returns rather than
raises, so `awaiting_review` survives instead of being overwritten by `failed` in the pipeline's
catch-all. With gates off, the flags are logged and cleared, because no reviewer exists to clear them.
Forbids: raising out of a stage that has just set `awaiting_review`.
Check: `pytest test_name_binding.py` in the orchestrator.
## Hallucinated index {#hallucinated-index}
**Closed.** An out-of-range `choice` from `/vision/resolve` is `unresolved`, like a parse failure. Only
an explicit `0` means NONE and mints a new character. Mapping a bad index to NONE created a brand new
entry in the permanent registry from a hallucination.
Check: `python worker_vision.py`.
## Unlocked model load {#unlocked-model-load} [#203]
**Closed.** The session manager never holds `_lock` across a model load or a health wait.
`open_session` was already written that way, but it could orphan the server it spawned. A
`/session/close` arriving during the load found `proc = None`, tore down nothing, and cleared
`_active`. The finished server then held its VRAM unreferenced, and the next open spawned a second one
on the same port. Open now tears down its own process and reports 409 when its lease vanished mid-load.
`_supervise_once` had the opposite asymmetry. It called `_start_subprocess` while holding `_lock`,
which blocked `/session/active`, `/session/close`, and `/session/open` for the full health wait. It now
claims the respawn by clearing `proc`, spawns unlocked, and tears the new process down if the lease
disappeared meanwhile.
Check: `python session_manager.py`.
## Related
`_extract_json` in both `worker_vision.py` and `worker_script.py` now uses
`json.JSONDecoder().raw_decode` from the first brace. The greedy `\{.*\}` ran to the LAST brace in the
reply. A second object or trailing braced prose burned a repair call on a response that parsed fine.
`worker_scene.build_scene` emits `actions` as a list beside the joined `action` string, because
`correctness.build_beat_artifact` reads the plural key. The verifier was receiving no action evidence
at all. The beat builder also falls back to splitting the singular string.
decisions/chapter-assembly.md
+107
View File
@@ -0,0 +1,107 @@
# chapter-assembly
Settled questions about `_assemble_batched` / `_assemble_once` / `_xfade_chain` in `worker_render.py`.
## The shipped 72.7s gap was a stream copy across mixed frame rates {#mixed-rate-stream-copy}
**Closed, 2026-08-12.**
`assemble` routes an all-`cut` chapter to a `concat` demuxer with `-c copy`. That path writes the output
with the **first** input's `time_base` and reinterprets every later packet in it.
14 of this chapter's 49 clips came off `collage_cmd`, which hardcoded `-r 30`. They carry
`r_frame_rate=30/1` and `time_base=1/15360`. The other 35 are `25/1` at `1/12800`. Copied into the first
clip's timebase, those 14 play `15360/12800 = 1.2` times too long while their audio is untouched. That is
the 1.2001 ratio, and the whole of video 436.39s over audio 363.67s.
Reproduced offline by running the same `-c copy` concat over the 49 real clips. Duration 436.392031 and
`nb_frames` 9902, identical to the shipped file. Seconds to run, no GPU.
Two changes hold it closed:
* `collage_cmd` emits `-r FPS` like every other clip path, and the `__main__` self-check asserts
`_fps_of(clip) == "25/1"` on a real collage encode.
* `assemble` probes `r_frame_rate` across the clips and sends mixed rates through `_assemble_batched`,
whose branches both normalize with `fps={FPS}`. Only a single shared rate keeps the stream copy.
Verified end to end. The rebuilt `chapter.mp4` is video 364.120s against audio 364.122s at `25/1`.
**An earlier version of this file, and commit `1457556`, blamed `#offsets-from-min-stream` below for the
shipped gap. That was wrong.** The rebuild came out byte-identical to the broken file, which proved the
xfade tree never ran for this chapter. The entry below is a real defect and stays closed on its own
evidence. It was not this one.
## xfade offsets are computed from `min(video, audio)`, never `format=duration` {#offsets-from-min-stream}
**Closed, 2026-08-12.** A real latent defect on the transition path. Not the cause of the shipped gap,
see `#mixed-rate-stream-copy` above.
`_xfade_chain` accumulates `cum += dur[i] - td` and hands each boundary `offset=cum-td`. That offset is
an assertion about where input `i-1` still has frames. It fed on `_audio_dur`, which probes
`format=duration`, which is `max(video, audio)`. A rendered clip's audio outlasts its video by about a
frame. So every boundary pushed the accumulator further ahead of the picture.
Once the accumulated overshoot exceeds the transition width, the xfade window starts after the last frame
of input `i-1`. ffmpeg emits the transition and then **silently discards input `i` and every clip
downstream of it**. `rc 0`, no warning on stderr, output file present and playable. Measured on a group of
8 real clips, chain truncated at each stage:
```
k=7 out= 52.52 correct
k=8 out= 52.52 the last xfade contributed nothing
[v6][n7]xfade=duration=0.050:offset=52.500 <- [v6] is only 52.52s long, so 0.02s of margin
```
This is the whole cause of the shipped chapter being video 436.39s over audio 363.67s. It reproduces with
synthetic clips in about four minutes and needs no GPU.
Two changes hold it closed:
* `_assemble_once` probes `min(_stream_dur(p, "v"), _stream_dur(p, "a"))`. The minimum, because either
stream running long breaks a different half of the graph.
* `_xfade_chain` floors every input to a whole frame count. It applies `trim` and `atrim` to both streams,
so the accumulator tracks the real timeline instead of estimating it. Transition widths are quantized to
frames for the same reason.
Verified over the 49 real clips of chapter `7c944dd4`. The round that previously turned 359s of video into
100s now loses 0.85s. The chapter comes out video 358.76s against audio 358.76s, agreeing to the frame.
Forbidden from here: `_audio_dur` in anything that positions a filter. It is fine for "how long is this
clip roughly", nothing else.
## Assembly verifies its own output instead of trusting ffmpeg's exit code {#check-assembled}
**Closed, 2026-08-12.**
Both assembly branches drop stream time without failing. xfade discards inputs as above. The concat
demuxer with `-vsync cfr` drops video frames to force a constant rate. Neither is an error to ffmpeg.
So `_assemble_once` calls `_check_assembled(out, expect)` after every encode. It compares the output's
video stream against the predicted timeline and against its own audio stream. It raises when either is off
by more than `ASSEMBLE_TOL_S`, which is 0.5s. That tolerance covers frame boundaries and aac padding. A
dropped input is off by whole seconds.
Without this the failure stays invisible until somebody watches the video. That is how a 50MiB chapter
with 72.7s of silent picture reached the bucket while every stage counter read success.
## The single-item passthrough is not the bug {#passthrough-innocent}
**Void, 2026-08-12.** Cited in `HANDOFF.md` for 2026-08-11 as the prime suspect and must not be cited
again.
With 49 clips and `ASSEMBLE_BATCH=8`, round 0 makes six groups of 8 plus a leftover group of 1, which
`_assemble_batched` carries forward un-encoded. The theory was that mixing that raw clip with six encoded
intermediates broke round 1. The instrumented run disproves it: **two round-0 groups of 8 fresh clips
collapse on their own**, before any passthrough exists.
```
n=8 XFADE in v= 63.52 a= 63.60 -> out v= 52.52 a= 62.77 lost_v= +11.00
n=8 XFADE in v= 58.12 a= 58.20 -> out v= 12.04 a= 56.83 lost_v= +46.08
```
Round 1's 259s loss was the cascade. Its inputs already held 309s of video against 364s of audio, and
`durs` read the audio.
Consequence for the plan: collapsing `concat`, `xfade` and the passthrough into one path was the
recommended fix in `NEXT.md` and is **not needed**. Three paths are fine once each one positions filters
on a real timeline. The tree keeps the bounded memory it was built for.
decisions/identity-bbox.md
+335
View File
@@ -0,0 +1,335 @@
# identity-bbox
The coordinate space of a vision character box, and what reads it.
## A vision `bbox` arrives on gemma's 0-1000 grid, and `/vision` converts it to pixels {#bbox-is-normalized}
**Closed, 2026-08-12.**
`build_detect_prompt` asks for a "pixel bounding box". The model answers on its own normalized grid
regardless. Measured over all 113 detections of job `778297bc`, read straight from `/review/identity`:
| test | result |
| --- | --- |
| boxes with `x2` past the 900px panel width | **47 of 113** |
| boxes with `y2` past 1000, on panels 1257 to 2307px tall | **0 of 113** |
| boxes clamped at exactly 1000 | 21 in x, 5 in y |
| coordinate range over every box | `[0, 1000]` |
Pixels cannot behave that way. A person standing in the lower half of a 2307px panel needs `y2` near
2000, and it never once exceeds 1000.
Consumed as pixels the boxes collapse into the top-left corner of the panel. Four consumers were reading
them:
* `worker_identity._crop_bbox` at `worker_identity.py:200`, which embeds the crop. This is why a crop of
a speech balloon's edge matched `Choi Haeseon` at 0.9.
* `_pair_faces_to_present` in `worker_vision.py`, which compares real detector face boxes, in pixels,
against these. The gate could almost never pass, which is the mechanism behind the 7 `unknown` results
out of 7 `som_face` lines already recorded at `worker_vision.py:169`.
* the set-of-mark boxes drawn for attribution.
* the review UI, which crops client-side off the panel PNG.
`/vision` now calls `_bbox_to_pixels(characters, w, h)` before returning, so all four see pixels and no
consumer needs to know the grid existed. Verified by drawing the converted boxes on panel 7. Five of six
land on their subject, including `person_5`, who is Seonho in the foreground and had no identity.
`person_1` still frames a window mullion with nobody in it, which is `#extras-as-cast`, not this.
The prompt text still says "pixel bounding box". Rewording it changes what the model emits and needs a
GPU run to re-verify, so the boundary converts instead. The `ponytail:` note on `_bbox_to_pixels` records
that. It also records the trap: a model that really answered in pixels would be scaled down here.
**Consequence: every assignment in the registry came from a wrong crop.** The existing embeddings and
`ref_image_uris` are enrolled on balloons and window frames. Re-running identity
is what makes the registry mean anything. The anonymous-identity and extra-versus-cast work cannot be
judged until that rerun happens.
## A stage result proves nothing until the worker is newer than the edit {#stale-worker-invalidates}
**Closed, 2026-08-12.** The first rerun after the bbox fix reproduced the defect exactly: 46 of 110
boxes past the 900px panel width, coordinates clamping at 1000, `y2` never once past 1000 on panels up
to 2307px tall. The same fingerprint as `#bbox-is-normalized` measured before the fix.
The fix was not wrong. It was not loaded.
```
vision worker process started 12:00:09
worker_vision.py modified 12:11:35
8113bdf, which contains _bbox_to_pixels, committed 12:16:22
```
Python binds a module once, at process start. `./start_workers.sh` had launched the worker eleven
minutes before the file changed, so `/vision` served pre-fix code for the whole run and returned raw
grid boxes. Nothing in the result said so. The stage reported `completed 116/116`, the orchestrator
recorded no error, and identity and reconcile ran to completion on top of it. Cost: one full
vision + identity + reconcile cycle, plus a registry reset to undo the 8 characters it minted.
The rerun against a restarted worker gives the opposite reading over the same 110 detections: 0 boxes
past the width, 0 past the height, one coordinate on 1000 which is now a real pixel value, and a
deepest box reaching 100% down its panel with `max y2 = 2307`. Boxes track the panel, so they are
pixels.
`check_stale.sh` compares every running worker's process start against its module's mtime and exits
non-zero if any is stale. This failure mode was already known as advice — the render worker "must be
restarted by hand to pick up an edit" — and advice did not stop it happening. Run the check before any
stage run that is meant to prove a code change.
Forbids: citing a stage result as evidence about a code change without establishing that the worker
serving it postdates the change.
## A detection with no detected face never enrolls or binds {#face-gates-enrollment}
**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**
Fixing the coordinate space made the extras problem worse, not better. With the boxes finally landing
on their subjects, panel 7's four background extras became four good crops of four irrelevant people,
and one of them bound to `Seonho` at confidence 1.00. Before the fix the same detection was a crop of
scenery and matched nothing much. Correct geometry turned a harmless failure into a poisoned reference
set for the lead.
The measured panel 7 outcome, converted boxes, against the art:
| box | who | assigned |
| --- | --- | --- |
| `[457, 657, 642, 937]` | Seonho, foreground | `Seonho` |
| `[669, 591, 763, 822]` | the colleague, unnamed in the story | `character_f7a4fd`, anonymous |
| `[428, 386, 496, 526]` | background extra | none |
| `[34, 414, 122, 564]` | background extra | none |
| `[498, 386, 568, 533]` | background extra | `character_d72710` at 0.94 |
| `[31, 554, 94, 728]` | background extra | `Seonho` at 1.00 |
`/vision` now stamps `has_face` on every character by running `face_detect.detect_faces` on the panel
and reusing `_pair_faces_to_present` for containment, so the gate uses the same margin and the same
global shortest-first assignment as the speaker path. `worker_identity.py` skips a character with
`has_face is False` before it crops, embeds, matches or mints.
Two properties are deliberate. It **fails open**: a missing or raising detector marks every character
`True`, because dropping a whole panel's cast is worse than the over-detection the gate exists to trim.
And it gates on `is False`, not falsiness, so a vision blob written before this change (no key) behaves
as it did rather than silently dropping every character.
Cost: a cast member drawn from behind, or in a style the detector misses, now takes no identity on that
panel. That is the abstain this pipeline already prefers to a wrong bind
(`caveats/speaker-attribution.md#no-anonymous-identity`).
Forbids: enrolling a reference crop, or binding a character, from a region no face detector confirms.
## A resolver NONE mints an anonymous character, it does not clear the crop {#none-mints-an-anonymous-character}
**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**
`caveats/speaker-attribution.md#no-anonymous-identity` asked whether the Tier-2 gemma resolver can answer
"none of these". It can, and it always could. `/vision/resolve` at `worker_vision.py:1071` maps `choice: 0`
to `state="new"`, an out-of-range index to `state="unresolved"`, and a parse failure to `unresolved` as
well. The abstain path was never the defect.
The defect was one branch on the other side of the contract. The orchestrator read only
`v.get("character_id")` and treated every falsy value the same way: `unassign_identity` on every crop of
the tracklet. So a deliberate "this is a real person the roster does not hold" and a hallucinated index
both produced nothing, and the unnamed colleague was `unknown` on every panel she appeared on. The stale
`ponytail:` comment above that block named the reason nobody fixed it, and the reason was real: minting a
character needs an `embedding_uri`, and the orchestrator cannot compute one. siglip is resident in the
identity worker, gemma is resident in the vision worker, and `session_manager` forbids both at once.
What removes the blocker is carrying the embedding, not a third GPU pass. `/identity/resolve` already
computes an embedding per crop and already uploads the crop to
`s3://manga/{manga}/characters/_crops/{panel}_{local}.png`. It now writes the embedding to the same key
with a `.npy` suffix and returns `emb_uri` in each shortlist entry. The mint is then a local
`create_character(manga_id, None, appearance, [crop_uri], emb_uri, gender)`, and the existing per-tracklet
assign loop binds every member to it.
`tracklets.resolve_outcome` holds the three-way decision as a pure function, so the branch that runs is
the branch the self-check covers: `known` on a named answer, `mint` on `state="new"` with an `emb_uri`,
`clear` on `unresolved`, on a NONE with no embedding, and on an older worker that sends no `state` at all.
Two limits are deliberate. A tracklet's candidate gallery is built before the loop mints anything, so one
person split across two unlinked tracklets still gets two anonymous ids; `run_stage_reconcile` merges
unnamed twins on appearance overlap and is what folds them. And an anonymous character's text sheet
(`worker_vision._sheet`) carries no name, so gemma re-recognising it on a later panel leans on the
reference images rather than the description.
Contract: `shortlists[].emb_uri` is new in the `/identity/resolve` response. Invariant 7 — both repos
changed in the same session.
Forbids: treating an absent `character_id` as one outcome. A resolver that answered and a resolver that
failed are different facts.
## The extras gate runs at enrollment and at narration, not at the speaker prompt {#extras-gate-consumers}
**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**
`#face-gates-enrollment` stops a faceless detection taking an identity. It does not stop the detection
being narrated, because three places read the raw vision character list and only one of them consults an
assignment. Panel 7 is the worked example: six detections, four of them extras or scenery.
`worker_scene.build_scene` already drops an unassigned detection from `characters` and `present`
(`worker_scene.py:63`), so an extra never reached the cast list. Its **action** did. `actions` was built
from every detection, and that list is what the script prompt renders and what the verifier uses as
evidence. So "standing at the window" arrived as a fact about the panel with no character attached, and
the verifier confirmed it because the action really was in the blob. Both now skip `has_face is False`.
`service._beat` picks the cinematographer's "who" from the first three detections, falling back to a
detection's action when it has no name. An extra could take a slot and steer the camera. Also gated.
`service._present_characters` is deliberately **not** gated. It builds the dialogue stage's candidate
speaker list and the set-of-mark boxes. Two reasons. The failure is already contained: an extra chosen as
the speaker has no identity assignment, so `normalize_dialogue` resolves it to unknown rather than to a
wrong name. And the gate's own cost lands hardest here, because a character drawn from behind has no face
box, so gating would delete a real speaker from the only list that can attribute their line.
All three gates test `is False`, never falsiness. A vision blob written before the gate existed carries no
`has_face` key, and a panel whose detector failed is marked `True` by the fail-open path. Both keep their
previous behaviour.
Forbids: adding a fourth consumer of `vision["characters"]` without deciding which side of this line it is
on. The blob keeps every detection on purpose, so the audit can still see what was gated.
## A roster name is a guess, so it never reaches detection {#roster-does-not-name}
`service.py` seeded `known_characters` with `_roster_char_hints` before every detect call, so the chapter
roster's names were in front of gemma before any panel had identified anyone. On the 2026-08-12 16:39 run
that put "Seonho (short brown hair, yellow plaid shirt)" on a different man wearing glasses, who then held
15 assignments under the lead's name. The real lead was minted separately from the p010 caption as
"Lim Seonho", and p020 held both as two people in one panel.
The seeding is removed. A name now reaches detection only from a registry row, and a registry row is named
from an in-panel caption or address through `name_claims`. The distinction is evidence: a roster name is a
claim about art nobody has looked at yet, and a registry row carries an embedding plus the panel that
named it.
The roster still feeds `roster_cast` in `run_stage_dialogue`. There the names are matched against speech,
not against faces, so a wrong guess costs an unresolved speaker rather than a wrong face binding.
Confirmed on the 17:38 run: "Lim Seonho" came back as one row with 25 assignments, so dropping the hint
did not split the lead across the panels before his caption. "Seonho" fell to 1 assignment.
Not covered: detection still reads a name off the panel and can attach it to the wrong body. p011 and p026
emit `name: "Seonho"` on `person_2` with no roster hint present. That is a separate mechanism and it is
what keeps the glasses man named at all.
## `merged_into` is exactly one hop deep {#merge-chains-flatten}
A merge chain means `merged_into` points at a row that is itself merged, so anything resolving one hop
lands on a retired character. Roster readers filter `merged_into IS NULL` and were never affected.
Two rules, one per direction, and both are needed:
- `merge_characters` resolves the keeper to its chain root before merging. Bounded 64-step walk, the cap
only exists so a cycle cannot hang reconcile.
- retiring a loser repoints every row whose `merged_into` was that loser.
The keeper walk alone is not enough, which the 17:38 run proved by producing
`477c1894 -> a92d9df4 -> 4fb94c15` with the walk deployed. At merge time that pair was fine. The chain
formed later, when a row that was already somebody's keeper was itself retired.
## `_bbox_to_pixels` orders the corners, because the model sometimes swaps them {#bbox-corners-ordered}
**Closed, 2026-08-12.**
The 17:38 run returned `p007 person_1` as `[226, 417, 130, 551]`, x1 greater than x2. One detection in 117.
Clamping each coordinate into the panel kept the swap, so the box stayed a negative-width region. It
cropped to nothing, so that detection could not enroll, embed or match, and nothing reported the loss.
`_bbox_to_pixels` now sorts each pair after clamping:
```python
xs = sorted((clamped_x1, clamped_x2))
ys = sorted((clamped_y1, clamped_y2))
c["bbox"] = [xs[0], ys[0], xs[1], ys[1]]
```
Sorting is enough. A zero-area box still crops to nothing, and no consumer needs a minimum size that it
does not already enforce. The self-check feeds the real swapped box in and asserts `[117, 688, 203, 909]`,
which fails without the sort.
Proven on the 18:07 run: 0 degenerate boxes over 119 detections.
## A generic word is not identity evidence, and one tokenizer serves both consumers {#generic-tokens}
**Closed, 2026-08-12, not yet run on a GPU.**
The tracklet linker and reconcile's pair pre-filter each carried their own copy of the appearance
tokenizer, and both linked on words that describe a person without distinguishing them. Measured over the
18:07 run's 22 crops for the lead:
```
p056~p057 hair=['short'] cloth=- two different men
p047~p054 hair=['hair'] cloth=- "brown hair" vs "dark hair", the field name in its own value
p109~p110 hair=- cloth=['shirt','white']
```
`tracklets.appearance_tokens` is now the single implementation and subtracts a `GENERIC` set: length words
(`short`, `long`, `medium`, `shoulder`, `length`), `hair`/`haired`, garment words (`shirt`, `top`,
`jacket`, `coat`, `sleeve`), and filler (`plain`, `casual`, `none`, `unknown`). `service._appearance_tokens`
delegates to it, so reconcile's pre-filter is fixed by the same change. That pre-filter is what let the
pink tank top reach `/vision/same` at all.
Four asserts cover it and all four fail with `GENERIC` emptied.
Measured effect on those 22 crops, with candidate overlap forced to pass (the shortlists are not stored,
so this is the most permissive assumption and the real split can only be finer):
| | tracklets | worst span |
| --- | --- | --- |
| as it ran | 3 | 30 panels |
| span cap only | 8 | 8 |
| span cap + `GENERIC` | 9 | 8 |
The stopwords alone split out `p054`, the beige-jacket man, and `p089`, the red-robe chibi.
**Not fixed by this.** Bare hair colour still links different men: `order 56` through `62` stay in one
five-crop tracklet on `brown`, and the cat at `p098` still joins its neighbours the same way. The obvious
next lever, a crop-to-crop cosine, is ruled out by `caveats/audit-open.md#cosine-not-identity`.
## A tracklet is bounded by span, not only by pairwise distance {#tracklet-span-cap}
**Closed, 2026-08-12, not yet run on a GPU.**
`window=8` bounded each PAIR, and linking is transitive, so nothing bounded the group. The 18:07 run
resolved 12 tracklets over 64 crops, and the lead's 22 native assignments fell into two of them spanning
panels 47-69 and 80-110. One gemma answer then covered a grey blazer, a denim jacket, a red robe and a cat.
The two blocks are visible in the stored confidences: eleven rows at 0.95 and ten at 1.0.
`link_tracklets` now rejects a merge whose resulting group would span more than `window` panels:
```python
orders = [dets[m].get("panel_order", m) for m in gi + gj]
if max(orders) - min(orders) > window:
continue
```
This makes the module docstring's claim true. The self-check links three compatible detections at panels 1,
9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled.
Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.
## The gallery is the live cast, not cosine's top-k {#cast-is-the-gallery}
**Closed, 2026-08-12, not yet run on a GPU.**
`run_stage_identity` built each tracklet's gallery by unioning the members' cosine shortlists, so the
metric that `caveats/audit-open.md#cosine-not-identity` shows cannot separate people decided who gemma was
even allowed to consider. Two different men reach 0.93 on this chapter's crops where the same man reaches
0.96. When the right character fell outside every member's top-5, gemma could only pick a wrong one or
answer NONE, and NONE mints a duplicate.
The cast is small. `tracklets.cast_gallery` builds the gallery from `get_known_characters` instead:
gender-compatible rows only, named first, then registry order. Cosine still shortlists per crop, and those
shortlists are still the linker's candidate-overlap evidence, but they no longer bound the answer.
Three consequences, each deliberate:
- **Every crop now reaches the resolver.** A crop only entered `shortlists` when its cosine top-k was
non-empty, so an early crop seen while the roster was still empty, or one with no gender-compatible row,
was dropped from resolution and could never mint. The guard is gone.
- **The gallery is re-read per tracklet**, so a character an earlier tracklet minted is visible to the
later ones. That closes the `ponytail:` note beside the mint path.
- **Two caps, because the prompt carries images.** `GALLERY_CAP = 10` in `service.py` bounds the
candidates and logs the ids it drops. `worker_vision.REF_BUDGET = 12` spreads reference images across
them, `max(1, min(3, 12 // n))` apiece, so a 9-character cast sends 9 images plus the query instead of
the 27 plus query that 3-apiece would have sent.
Cost: more gemma calls, because the crops that used to be dropped now each get one.
**Not fixed by this.** The gallery being right does not make the crop legible. A back-turned or tiny crop
still has no face for gemma to judge, which is `NEXT.md` item 4.
decisions/identity-naming.md
+51
View File
@@ -0,0 +1,51 @@
# Naming a character
How a discovered name reaches `characters.name`. The mechanism is `db.add_name_claim`, fed by the dialogue
stage through `service._absorb`.
## A name is a word set, not a string {#alias-grouping}
**Closed, 2026-08-13, not yet run on a GPU.**
Claims grouped on the casefolded name, so `Lim Seonho` from a p010 caption and `Seonho` from a p047 address
counted as two names for one character. `len(grouped) > 1` fired, a `conflicting-name-claims` flag was
filed, and promotion was blocked permanently on evidence that in fact corroborated.
`alias_groups` groups two names when one's word set contains the other's, and keeps the longer as
canonical. `Seonho` and `Lim Seonho` are one name and the registry stores `Lim Seonho`. `Seonho` and
`Haeseon` are still two, so a real conflict still flags.
## A confident caption names a character on its own {#caption-is-strong}
**Closed, 2026-08-13, not yet run on a GPU.**
Promotion needed two distinct panels, or one `self_intro` or `name_tag` claim above 0.9. On the 2026-08-12
chapter the only claim naming a main character was `Choi Haeseon`, a caption at p040 at confidence 1.0, and
it was discarded. She held 13 correct crops and stayed anonymous through the whole pipeline.
A caption is the narration naming the person it is drawn beside, which is how a webtoon introduces its
cast, so it joins `STRONG_EVIDENCE`. This is the loosest of the three changes and is only safe because of
the next one.
## A name belongs to one character {#one-name-one-character}
**Closed, 2026-08-13, not yet run on a GPU.**
Being addressed by name identifies the addressee. Choosing which drawn body that is fails often. Two panels
addressed `Seonho`, the dialogue model pointed `target_local_id` at the woman standing beside him, and two
independent claims promoted her. The registry then held a female `Seonho` over 9 crops beside the lead's
`LIM SEONHO`.
A promotion now checks every other live character of the same manga first, by alias group. A collision
refuses the promotion and files a `name-already-taken` flag carrying both ids. The collision is itself
evidence that either the addressee or the identity cluster is wrong, so it is worth surfacing rather than
resolving silently.
Deliberately not built: no attempt to decide WHICH character deserves the name. That needs the addressee
fixed, which is `ARCHITECTURE.md` section 3.
## Checks
`test_name_binding.py` replays the six real claims from the 2026-08-12 chapter. Each new assert was
confirmed to fail with its fix disabled: the caption test with `STRONG_EVIDENCE` reverted, the alias test
with casefold grouping restored. The taken-name test asserts a flag kind that only the new branch emits.
decisions/speaker-attribution.md
+96
View File
@@ -0,0 +1,96 @@
# Speaker attribution and cast names
Settled 2026-08-11 from the quality cross-check of job `778297bc`
(`JOURNAL.md`, `caveats/speaker-attribution.md`). No GPU work ran and no pipeline run was executed
after the change. Both claims rest on source and on the CPU-only self-checks named below.
Files: `worker_vision.py` on workpc, `correctness.py` and `test_script_verify.py` in the homesrv
orchestrator (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`).
## A model guess is never labelled `tail` {#no-fake-tail}
**Closed.** `speaker_method` names how a speaker was established, and nothing may claim geometry it did
not read. `_annotate_speaker_methods` stamped `tail`, the highest-trust label, on any line whose
speaker matched a `local_id` present in the panel, keeping gemma's confidence of 1.0. No balloon was
ever consulted.
Evidence: three of three sampled two-character panels had both speakers swapped
(`caveats/speaker-attribution.md#tail-is-not-geometry`). 31 of 81 speech lines carried `tail` with two
or more characters present.
The label is gone. With two or more characters present the guess is dropped: `speaker` becomes
`unknown`, confidence 0.0, method `unknown`. With one character present the claim equals the solo
backstop, so it is kept as `model_solo` at confidence 0.7. Grounded `som_face` and `solo_prior` rows are
untouched, because the function still skips any row that already carries a method.
Forbids: minting a provenance label for evidence that was not read, and shipping a multi-character
attribution as truth before balloon geometry exists.
Check: `python worker_vision.py`, the `crowd`/`lone` cases.
Cost: the named-speaker share will fall. The 30% headline was measured on attributions the sample says
are wrong, so the lower number is the first honest one.
## The model's speaker answer is resolved against what the prompt showed {#prompt-label-answers}
**Closed.** `build_dialogue_prompt` renders a present character as
`- person_1: brown ponytail, green dress (f)`. gemma answers with any part of that line, so every part
of it must map back to the `local_id`. It did not, and `normalize_speaker` classified each unmatched
answer as a free-form name that no registry entry could match.
Evidence, measured on 36 panels of the cancelled first rerun. 15 lines carried a description and 9 a bare
`local_id` with no identity assignment. 2 carried a stale `P1` mark label, 2 a name with the gender
marker attached. That is 28 of 51 speech lines. Only 3 resolved to a `character_id`.
`_apply_speaker_labels` now takes `present` and resolves the id, the name, the description, and each of
those plus the gender marker. A key shared by two present characters is dropped, because it identifies
neither. An id-shaped answer naming nobody present becomes `unknown`. A trailing gender marker is
stripped, so an off-panel `Seonho (m)` still matches the registry name `Seonho`.
`normalize_speaker` refuses an id-shaped value independently, because the worker is a separate process
and the contract is load-bearing (invariant 7).
Forbids: showing the model a label the worker cannot resolve back, and treating an unmatched speaker
string as a name.
Check: `python worker_vision.py`, the `shown`/`twins` cases. `pytest test_correctness.py`,
`test_an_id_shaped_speaker_is_never_a_name`.
## An unpaired mark grounds nothing {#unpaired-mark}
**Closed.** `_set_of_mark` labels a detected face `unknown` when gated pairing
(`decisions/audit-phase1.md#gated-face-pairing`) matched it to no present character. An answer pointing
at such a mark identifies nobody, so it must not carry `som_face`, the highest-trust provenance.
Evidence: all 7 `som_face` lines in the first 36 panels of the 2026-08-11 rerun had `speaker_ref` kind
`unknown`. The label sat on lines with no speaker. Same defect class as the fake `tail`.
Forbids: deriving a provenance label from the label map without checking what the label resolved to.
Check: `python worker_vision.py`, the `unpaired` case.
## An interjection is not a name and not a misquote {#interjection-false-positive}
**Closed.** `verify_script` must stay quiet on valid narration, because `run_stage_script` retries once
and then raises (`decisions/audit-phase1.md#verifier-false-positives`).
Evidence: the narrator wrote `"...Hm?"` for the source line `"Uh... hum...?"`. Both rules fired at once.
`Hm` was absent from the source words, and a 6-character quote needs 5 matching characters to ground,
so it got 4. That halted the `script` stage at 112/116 on job `778297bc`.
Interjections join `_STOPWORDS`. A quote of three letters or fewer is no longer grounded-checked. That
holds the same line as the dialogue prompt's 1-3 character noise rule. The stage then passed 116/116.
Forbids: scoring a quote too short for the ratio to carry meaning.
Check: `pytest test_script_verify.py`, `test_an_interjection_is_not_a_name_or_a_misquote`.
## Cast names enter the verifier tokenized {#multiword-cast-names}
**Closed.** `verify_script` compares single capitalized tokens, so every allowed name must be present as
tokens. `allowed` was built from `cast_names` verbatim, which put `"choi haeseon"` in the set as one
string while the checker looked up `Choi` and `Haeseon` separately.
Evidence: the `script` stage failed at 87/116 on job `778297bc`. 28 of the 29 lost beats cite
`unsupported-proper-noun: ['Choi', 'Haeseon']`. A one-word name such as `Seonho` always passed, which
is why this survived the Phase 1 verifier work
(`decisions/audit-phase1.md#verifier-false-positives`).
Forbids: adding any future allow-list to `verify_script` as whole strings.
Check: `pytest test_script_verify.py`, `test_multiword_cast_name_is_supported`.
decisions/storage-layout.md
+74
View File
@@ -0,0 +1,74 @@
# storage-layout
Settled questions about which S3 bucket holds what, and about the MinIO replacement.
## One bucket per artifact class {#bucket-per-artifact}
**State: closed. 2026-08-11.**
Six buckets were created on 2026-07-04 (`manga raw panels audio layers video`). Only `manga` ever
received an object, because `minio_layout.py` hardcoded `BUCKET = "manga"` and four workers built
their own keys as literal `s3://manga/...`. The other five sat empty for five weeks.
Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved:
| artifact | bucket |
| --- | --- |
| fetched pages | `raw` |
| panel crops | `panels` |
| tts wavs | `audio` |
| layer pngs | `layers` |
| clips and `chapter.mp4` | `video` |
| vision, identity, scene, script json, character registry | `manga` |
Orchestrator: `minio_layout.py` gained `BUCKET_RAW`/`BUCKET_PANELS`/`BUCKET_AUDIO`/`BUCKET_LAYERS`/
`BUCKET_VIDEO` and a `BUCKETS` tuple. `parse_key` accepts any of them and rejects anything else.
`service.py:_s3_delete_prefix` takes a `<bucket>/<prefix>` pair instead of assuming one bucket, and
`_stage_s3_prefixes` stops slicing the bucket off. Workers: `worker_crop.py`, `worker_tts.py`,
`worker_layers.py`, `worker_render.py`.
Every S3 URI is `s3://<bucket>/<manga_id>/<chapter_id>/...` and every consumer already derives the
bucket from the first path segment, so no reader needed a change.
What this forbids: writing an artifact under a bucket that is not in `BUCKETS`. `parse_key` returns
`{}` for one, and stage clearing would then silently delete nothing.
Objects written before this date stay under `manga/` at their old keys. Nothing reads them any more:
they are the rollback for the 2026-07-17 run, not live data.
Evidence: `test_minio_layout.py` (31 tests), and the 2026-08-11 chapter run, which put pages in `raw`
and 116 panel crops in `panels`.
## The orchestrator creates missing buckets at startup {#ensure-buckets}
**State: closed. 2026-08-11.**
Workers create a bucket on first write (`transport.py:115`), but the orchestrator uploads pages before
any worker runs and boto3 will not auto-create. `service.py:_ensure_buckets` runs in the FastAPI
lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning
instead. The check is not worth a failed start.
## RustFS is staged, not adopted {#rustfs-staged}
**State: open. 2026-08-11.**
`rustfs` holds all six buckets on `127.0.0.1:9010/9011`, all empty. MinIO still serves every read and
write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is
`1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run
meant to produce the baseline. Task [#116].
## Clearing a stage strips the vision blob it wrote {#clear-vision-blob}
**Closed. 2026-08-11.** `dialogue` and `direct` have no output table. They write onto the per-panel
vision blob, and `_STAGE_TABLES` had no entry for either, so `/stage/clear dialogue` deleted nothing and
still returned `{"ok": true}`. `run_stage_dialogue` then saw `"dialogue" in vision` and skipped all 116
panels.
Evidence: the first clear reported `scripts: 46, scene_graphs: 116` and no vision counts. After the fix
the same call reported `vision_results.dialogue: 116` and `vision_results.direct: 75`, all of which the
first clear had left in place. A whole rerun was wasted on stale data before this was found.
`_STAGE_VISION_KEYS` in `db.py` names the keys each stage owns, and `clear_stage_data` strips them.
Forbids: adding a stage that writes onto a shared blob without listing its keys there.
Check: `pytest test_db.py`, `TestClearStageData`.
face_detect.py
+1 -1
View File
@@ -32,7 +32,7 @@ def _letterbox(img, sz=640):
return canvas, r, px, py return canvas, r, px, py
def detect_faces(img, conf: float = None) -> list: def detect_faces(img, conf: float | None = None) -> list:
"""img: BGR ndarray. -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels, sorted """img: BGR ndarray. -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels, sorted
reading order. Empty on no faces. Lazy load so import never touches the model.""" reading order. Empty on no faces. Lazy load so import never touches the model."""
conf = CONF if conf is None else conf conf = CONF if conf is None else conf
requirements.txt
+13
View File
@@ -13,3 +13,16 @@ transformers
accelerate accelerate
soundfile soundfile
# system deps (not pip): ffmpeg, comfyui (external server) # system deps (not pip): ffmpeg, comfyui (external server)
#
# dots.tts (gitignored checkout, needed by worker_tts + scripts/pick_tts_voice). venv is built
# --system-site-packages so it inherits Arch's rocm torch:
# git clone https://github.com/rednote-hilab/dots.tts.git dots.tts
# pip install -e ./dots.tts --no-deps --ignore-requires-python # py3.14 > its <3.13 pin, skips gradio
# pip install librosa loguru einops torchdiffeq 'langcodes[data]' lingua-language-detector WeTextProcessing
# pip install --no-deps --index-url https://download.pytorch.org/whl/rocm7.2 torchvision==0.28.0+rocm7.2
# that last one is not optional: Arch's torchvision 0.25 is too old for torch 2.13, and every
# transformers model import dies with "operator torchvision::nms does not exist" until it is shadowed.
#
# models/ (gitignored, CPU onnx for set-of-mark):
# comictextdetector.pt.onnx <- github.com/zyddnys/manga-image-translator releases/beta-0.3
# anime_face_v1.4_s.onnx <- huggingface.co/deepghs/anime_face_detection face_detect_v1.4_s/model.onnx
ruff.toml
+21
View File
@@ -0,0 +1,21 @@
# Lint gate. `ruff check .` must exit 0, so CI can gate on it and a new finding means a new defect.
#
# Ruff's defaults flag about 100 things in this repo. Most are deliberate style in workers that must
# survive one bad panel rather than fail clean. Every rule turned off below carries its reason, so an
# ignore stays a decision rather than a shrug. Delete an entry the moment its reason stops holding.
[lint]
ignore = [
"I001", # import order: 25 files of churn, no behaviour change
"BLE001", # a worker catches Exception on purpose, so one bad panel cannot kill the stage
"SIM115", # short-lived open().read(); the handle drops with the refcount
"S110", # try/except/pass in best-effort cleanup, where the no-op IS the handling
"ASYNC210", # ffmpeg, ffprobe and MinIO run synchronously inside async endpoints. Real, and
"ASYNC221", # already recorded at caveats/audit-open.md#blocking-event-loop with [#199]. The fix
"ASYNC230", # is `def` over `async def` per handler, which is a refactor and not a lint fix.
"RUF046", # int(round(v)) says "pixels" out loud in the render geometry
"UP031", # the ASS subtitle template is %-formatted; f-string braces collide with its {\an} tags
"RUF059", # unpacking a whole bbox and using half of it beats indexing into it
"RUF007", # zip(x, x[1:]) reads better here than itertools.pairwise
"PLC3002", # one immediately-called lambda, in an audit script
]
scripts/pick_tts_voice.py
Regular → Executable
+12 -1
View File
@@ -25,7 +25,18 @@ def load_model(model_name: str):
cls, *args, **{"fix_mistral_regex": True, **kwargs} cls, *args, **{"fix_mistral_regex": True, **kwargs}
) )
) )
from dots_tts.runtime import DotsTtsRuntime # Match worker_tts.py's torch/torchaudio minor-mismatch bypass too.
import torch
from importlib import metadata
real_version = metadata.version
metadata.version = (
lambda name: torch.__version__ if name == "torchaudio" else real_version(name)
)
try:
from dots_tts.runtime import DotsTtsRuntime
finally:
metadata.version = real_version
return DotsTtsRuntime.from_pretrained(model_name, precision="bfloat16") return DotsTtsRuntime.from_pretrained(model_name, precision="bfloat16")
session_manager.py
+48 -12
View File
@@ -6,7 +6,7 @@
# in-process models (siglip2, dots): no server — the lease just reserves the GPU; the # in-process models (siglip2, dots): no server — the lease just reserves the GPU; the
# worker loads the transformers model itself after /session/open returns (port=None). # worker loads the transformers model itself after /session/open returns (port=None).
# the guarantee that matters is the mutex: a second open() gets 409 until the first closes. # the guarantee that matters is the mutex: a second open() gets 409 until the first closes.
import os, time, uuid, threading, subprocess import time, uuid, threading, subprocess
import requests import requests
from fastapi import FastAPI, HTTPException from fastapi import FastAPI, HTTPException
from pydantic import BaseModel from pydantic import BaseModel
@@ -116,7 +116,12 @@ def open_session(req: OpenReq):
with _lock: with _lock:
if _active is not None and _active["session_id"] == session_id: if _active is not None and _active["session_id"] == session_id:
_active["proc"] = proc _active["proc"] = proc
return {"session_id": session_id, "port": cfg.get("port")} return {"session_id": session_id, "port": cfg.get("port")}
# The lease was closed or reaped while we were spawning (close saw proc=None and tore down
# nothing). The server we just started is unreferenced and would hold its VRAM until someone
# killed it by hand, and the next open would spawn a SECOND one on the same port.
_teardown({"proc": proc})
raise HTTPException(409, "session closed while the model was loading")
@app.post("/session/close") @app.post("/session/close")
@@ -162,8 +167,8 @@ def _supervise_once():
talks to the port directly, so if we don't bring it back every remaining panel of the stage fails talks to the port directly, so if we don't bring it back every remaining panel of the stage fails
with 'connection refused'. Restarting under the SAME session keeps the lease + port valid so the with 'connection refused'. Restarting under the SAME session keeps the lease + port valid so the
orchestrator never has to re-open. Returns True if a respawn happened.""" orchestrator never has to re-open. Returns True if a respawn happened."""
global _active
with _lock: with _lock:
global _active
sess = _active sess = _active
if not sess: if not sess:
return False return False
@@ -174,16 +179,29 @@ def _supervise_once():
proc.wait(timeout=1) # reap the zombie proc.wait(timeout=1) # reap the zombie
except subprocess.TimeoutExpired: except subprocess.TimeoutExpired:
pass pass
print(f"[supervise] gemma subprocess died (rc={proc.returncode}) session {sess['session_id']}; " # claim the respawn before releasing the lock: proc=None makes the next _supervise_once pass
f"respawning", flush=True) # return early, so the health wait can't be entered twice for one death.
try: sess["proc"] = None
sess["proc"] = _start_subprocess(MODELS[sess["model"]]) session_id, model = sess["session_id"], sess["model"]
sess["last_beat"] = time.time() # don't count the downtime against the TTL reaper # ponytail: spawn + health-wait UNLOCKED, like open_session. Holding _lock here blocked
# /session/active, /close and /open for the full health wait (up to 300s).
print(f"[supervise] gemma subprocess died (rc={proc.returncode}) session {session_id}; "
f"respawning", flush=True)
try:
new_proc = _start_subprocess(MODELS[model])
except Exception as e:
print(f"[supervise] respawn failed: {e}; clearing session so the mutex frees", flush=True)
with _lock:
if _active is not None and _active["session_id"] == session_id:
_active = None
return False
with _lock:
if _active is not None and _active["session_id"] == session_id:
_active["proc"] = new_proc
_active["last_beat"] = time.time() # don't count the downtime against the TTL reaper
return True return True
except Exception as e: _teardown({"proc": new_proc}) # lease went away mid-respawn; don't orphan the server
print(f"[supervise] respawn failed: {e}; clearing session so the mutex frees", flush=True) return False
_active = None
return False
def _reaper(): def _reaper():
@@ -229,6 +247,24 @@ if __name__ == "__main__":
close_session(SessionReq(session_id=s4["session_id"])) close_session(SessionReq(session_id=s4["session_id"]))
assert active() is None assert active() is None
# a close landing DURING the spawn must not orphan the server: open reports 409 and tears it down.
MODELS["gemma4"]["binary"] = "/bin/sleep"
MODELS["gemma4"]["port"] = None
real_start = _start_subprocess
def racing_start(cfg):
p = real_start(cfg)
close_session(SessionReq(session_id=_active["session_id"])) # close mid-load
return p
_start_subprocess = racing_start
try:
open_session(OpenReq(model="gemma4", ttl=3600))
assert False, "open must 409 when its lease vanished mid-load"
except HTTPException as e:
assert e.status_code == 409
_start_subprocess = real_start
assert active() is None
# in-process model: close must POST /unload to the worker so its resident VRAM is freed. # in-process model: close must POST /unload to the worker so its resident VRAM is freed.
calls = [] calls = []
requests.post = lambda url, **kw: calls.append(url) or type("R", (), {"status_code": 200})() requests.post = lambda url, **kw: calls.append(url) or type("R", (), {"status_code": 200})()
test_vision_parse.py
+1 -1
View File
@@ -14,7 +14,7 @@ def test_extract_rejects_malformed_and_truncated():
try: try:
wv._extract_json(bad) wv._extract_json(bad)
assert False, f"should have raised on: {bad!r}" assert False, f"should have raised on: {bad!r}"
except (ValueError, ValueError): except ValueError:
pass pass
transport.py
+44 -3
View File
@@ -12,6 +12,29 @@ from starlette.responses import Response
_client = None _client = None
# --- artifact layout ---------------------------------------------------------------------------
# one bucket per artifact class (`decisions/storage-layout.md#bucket-per-artifact`). Every worker
# formats its output uri from these, so moving a class between buckets is one edit here rather than
# a grep across five workers. `name` is the panel id, or 'p' when a worker has none.
PANEL_URI = "s3://panels/{manga_id}/{chapter_id}/panels/p{idx:03d}.png"
PAGE_PANEL_URI = "s3://panels/{manga_id}/{chapter_id}/panels/pg{page_index:03d}_p{idx:02d}.png"
AUDIO_URI = "s3://audio/{manga_id}/{chapter_id}/audio/{name}.wav"
AUDIO_FLAT_URI = "s3://audio/_audio/{name}.wav"
LAYER_URI = "s3://layers/{manga_id}/{chapter_id}/layers/{name}/{idx}.png"
CLIP_URI = "s3://video/{manga_id}/{chapter_id}/clips/{name}.mp4"
CHAPTER_URI = "s3://video/{manga_id}/{chapter_id}/chapter.mp4"
CHAR_PNG_URI = "s3://manga/{key}.png"
CHAR_NPY_URI = "s3://manga/{key}.npy"
def ids_from_uri(uri: str):
"""(manga_id, chapter_id) from any artifact uri: <bucket>/<manga_id>/<chapter_id>/...
the orchestrator passes no ids to tts, layers or render, but every input uri encodes them."""
parts = (uri.removeprefix("s3://")).split("/")
if len(parts) < 3:
raise ValueError(f"uri carries no manga/chapter: {uri!r}")
return parts[1], parts[2]
def _summarize(body: bytes, limit=6) -> str: def _summarize(body: bytes, limit=6) -> str:
"""compact one-line view of a json body for observability: uri inputs/outputs (basename, or """compact one-line view of a json body for observability: uri inputs/outputs (basename, or
@@ -27,9 +50,9 @@ def _summarize(body: bytes, limit=6) -> str:
for k, v in obj.items(): for k, v in obj.items():
if k == "panel_id": if k == "panel_id":
continue continue
if isinstance(v, str) and (k.endswith("uri") or k.endswith("url")): if isinstance(v, str) and (k.endswith(("uri", "url"))):
parts.append(f"{k}={v.rsplit('/', 1)[-1]}") parts.append(f"{k}={v.rsplit('/', 1)[-1]}")
elif isinstance(v, list) and v and isinstance(v[0], str) and (k.endswith("uris") or k.endswith("urls")): elif isinstance(v, list) and v and isinstance(v[0], str) and (k.endswith(("uris", "urls"))):
parts.append(f"{k}×{len(v)}") parts.append(f"{k}×{len(v)}")
elif isinstance(v, (int, float, bool)): elif isinstance(v, (int, float, bool)):
parts.append(f"{k}={v}") parts.append(f"{k}={v}")
@@ -102,7 +125,7 @@ def _mc():
def _split(uri: str): def _split(uri: str):
"""(bucket, key) from an s3-style or bare uri.""" """(bucket, key) from an s3-style or bare uri."""
u = uri[5:] if uri.startswith("s3://") else uri u = uri.removeprefix("s3://")
bucket, _, key = u.partition("/") bucket, _, key = u.partition("/")
if not bucket or not key: if not bucket or not key:
raise ValueError(f"bad uri: {uri!r}") raise ValueError(f"bad uri: {uri!r}")
@@ -213,4 +236,22 @@ if __name__ == "__main__":
assert _summarize(b"clip_uri", ) == "-" # non-json assert _summarize(b"clip_uri", ) == "-" # non-json
assert _summarize(json.dumps({"clip_uri": "s3://m/c/clips/p1.mp4", "duration": 4.1}).encode()) \ assert _summarize(json.dumps({"clip_uri": "s3://m/c/clips/p1.mp4", "duration": 4.1}).encode()) \
== "clip_uri=p1.mp4 duration=4.1" == "clip_uri=p1.mp4 duration=4.1"
# artifact layout: templates format to the keys the workers wrote by hand before, and
# ids_from_uri recovers the ids the orchestrator never sends.
panel = PANEL_URI.format(manga_id="m1", chapter_id="c1", idx=7)
assert panel == "s3://panels/m1/c1/panels/p007.png", panel
assert PAGE_PANEL_URI.format(manga_id="m1", chapter_id="c1", page_index=2, idx=3) \
== "s3://panels/m1/c1/panels/pg002_p03.png"
assert CLIP_URI.format(manga_id="m1", chapter_id="c1", name="p003") \
== "s3://video/m1/c1/clips/p003.mp4"
assert LAYER_URI.format(manga_id="m1", chapter_id="c1", name="p003", idx=0) \
== "s3://layers/m1/c1/layers/p003/0.png"
assert ids_from_uri(panel) == ("m1", "c1")
assert ids_from_uri("panels/m1/c1/panels/p007.png") == ("m1", "c1")
try:
ids_from_uri("s3://panels/p007.png")
raise AssertionError("a uri with no chapter segment must raise")
except ValueError:
pass
print("transport self-check ok") print("transport self-check ok")
worker_crop.py
+8 -4
View File
@@ -240,12 +240,15 @@ async def crop_webtoon(data: WebtoonInput):
tag = uuid.uuid4().hex[:8] tag = uuid.uuid4().hex[:8]
locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)] locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)]
strip = restitch(locals_) strip = restitch(locals_)
crops = slice_webtoon(strip) crops = merge_faceless_captions(slice_webtoon(strip))
context_links = context_fragment_links(crops) context_links = context_fragment_links(crops)
panels = [] panels = []
for idx, (crop_img, bbox) in enumerate(crops): for idx, (crop_img, bbox) in enumerate(crops):
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png" uri = transport.PANEL_URI.format(manga_id=data.manga_id, chapter_id=data.chapter_id, idx=idx)
# slicing is deterministic, so on a resume the same idx -> same key; skip re-upload. # slicing is deterministic, so on a resume the same idx -> same key; skip re-upload.
# TRAP: that holds only while the PLAN is unchanged. Edit slice_webtoon or the merge pass and a
# re-crop silently keeps the previous run's images under the same keys, because every one of them
# already exists. Clear the s3://panels/<manga>/<chapter>/panels/ prefix before re-cropping.
if not transport.exists(uri): if not transport.exists(uri):
out = f"{SHM}/wt_{tag}_p{idx:03d}.png" out = f"{SHM}/wt_{tag}_p{idx:03d}.png"
cv2.imwrite(out, crop_img) cv2.imwrite(out, crop_img)
@@ -266,7 +269,7 @@ async def crop(data: CropInput):
raise HTTPException(400, f"page not readable: {data.page_uri}") raise HTTPException(400, f"page not readable: {data.page_uri}")
h, w = img.shape[:2] h, w = img.shape[:2]
webtoon = h / w >= WEBTOON_RATIO webtoon = h / w >= WEBTOON_RATIO
crops = slice_webtoon(img) if webtoon else kumiko_panels(local, data.rtl) crops = merge_faceless_captions(slice_webtoon(img)) if webtoon else kumiko_panels(local, data.rtl)
context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))} context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))}
ambiguous = flag_overlaps(crops, data.page_index) ambiguous = flag_overlaps(crops, data.page_index)
@@ -274,7 +277,8 @@ async def crop(data: CropInput):
for idx, (crop_img, bbox) in enumerate(crops): for idx, (crop_img, bbox) in enumerate(crops):
out = f"{SHM}/pg{data.page_index:03d}_p{idx:02d}.png" out = f"{SHM}/pg{data.page_index:03d}_p{idx:02d}.png"
cv2.imwrite(out, crop_img) cv2.imwrite(out, crop_img)
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/pg{data.page_index:03d}_p{idx:02d}.png" uri = transport.PAGE_PANEL_URI.format(manga_id=data.manga_id, chapter_id=data.chapter_id,
page_index=data.page_index, idx=idx)
transport.put(out, uri) transport.put(out, uri)
os.remove(out) os.remove(out)
panels.append({"panel_index": idx, "uri": uri, "bbox": bbox, panels.append({"panel_index": idx, "uri": uri, "bbox": bbox,
worker_identity.py
+23 -6
View File
@@ -78,7 +78,7 @@ def match(emb: np.ndarray, known: list, threshold: float):
return None, best_conf, ambiguous return None, best_conf, ambiguous
def shortlist(emb: np.ndarray, known: list, k: int = 5, gender: str = None) -> list: def shortlist(emb: np.ndarray, known: list, k: int = 5, gender: str | None = None) -> list:
"""TIER-2 evidence: the top-k gender-gated known characters by cosine, best first. Cosine is now a """TIER-2 evidence: the top-k gender-gated known characters by cosine, best first. Cosine is now a
SHORTLISTER, not the decider — gemma /vision/resolve picks from this list. Each item carries the SHORTLISTER, not the decider — gemma /vision/resolve picks from this list. Each item carries the
full row (name/gender/description) so the resolver can build a text character-sheet. pure.""" full row (name/gender/description) so the resolver can build a text character-sheet. pure."""
@@ -88,7 +88,10 @@ def shortlist(emb: np.ndarray, known: list, k: int = 5, gender: str = None) -> l
def _crop_bbox(img, bbox): def _crop_bbox(img, bbox):
# vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention). # [x1, y1, x2, y2] pixel corners. gemma answers on a 0-1000 normalized grid and `/vision` converts
# to pixels before returning (`worker_vision._bbox_to_pixels`), so this reads real pixels. It did
# not before 2026-08-12, which is why crops landed on balloons
# (`decisions/identity-bbox.md#bbox-is-normalized`).
x1, y1, x2, y2 = bbox x1, y1, x2, y2 = bbox
return img[y1:y2, x1:x2] return img[y1:y2, x1:x2]
@@ -101,7 +104,7 @@ def _save_npy(emb: np.ndarray, uri: str):
os.remove(tmp) os.remove(tmp)
def _pending_match(pend: list, emb, threshold: float, gender: str = None): def _pending_match(pend: list, emb, threshold: float, gender: str | None = None):
"""index of the pending entry this embedding belongs to, or None (a new provisional). candidates """index of the pending entry this embedding belongs to, or None (a new provisional). candidates
of a conflicting decided gender are excluded. character_id=i keeps the returned index original. pure.""" of a conflicting decided gender are excluded. character_id=i keeps the returned index original. pure."""
cands = [{"character_id": i, "embedding": e["emb"]} cands = [{"character_id": i, "embedding": e["emb"]}
@@ -114,7 +117,7 @@ def _persist_char(manga_id, panel_id, local_id, crop, emb, name, gender, appeara
"""upload crop + embedding to S3 and register the row via the orchestrator; return its id.""" """upload crop + embedding to S3 and register the row via the orchestrator; return its id."""
import cv2 import cv2
key = f"{manga_id}/characters/_new/{panel_id}_{local_id}" key = f"{manga_id}/characters/_new/{panel_id}_{local_id}"
ref_img_uri, emb_uri = f"s3://manga/{key}.png", f"s3://manga/{key}.npy" ref_img_uri, emb_uri = transport.CHAR_PNG_URI.format(key=key), transport.CHAR_NPY_URI.format(key=key)
ref_png = f"{SHM}/crop_{uuid.uuid4().hex[:8]}.png" ref_png = f"{SHM}/crop_{uuid.uuid4().hex[:8]}.png"
cv2.imwrite(ref_png, crop) cv2.imwrite(ref_png, crop)
transport.put(ref_png, ref_img_uri) transport.put(ref_png, ref_img_uri)
@@ -197,6 +200,13 @@ async def resolve(data: IdentityInput):
assignments, backfill, new_chars, shortlists = [], [], [], [] assignments, backfill, new_chars, shortlists = [], [], [], []
for ch in data.vision_characters: for ch in data.vision_characters:
# no detected face inside the box -> a background extra, a figure on a poster, or scenery
# gemma called a person. Embedding it pollutes the registry and, once the boxes were pixels,
# bound an extra to the lead at confidence 1.00. Abstain instead. `/vision` stamps this and
# fails open, so a panel it could not gate arrives with has_face=True on every character
# (decisions/identity-bbox.md#face-gates-enrollment). Absent key = an older vision blob.
if ch.get("has_face") is False:
continue
crop = _crop_bbox(img, ch["bbox"]) crop = _crop_bbox(img, ch["bbox"])
if crop.size == 0: # degenerate/out-of-bounds bbox -> nothing to embed, skip if crop.size == 0: # degenerate/out-of-bounds bbox -> nothing to embed, skip
continue continue
@@ -204,10 +214,17 @@ async def resolve(data: IdentityInput):
# shortlist for gemma's decider, from the roster as it stood before this crop's own outcome. # shortlist for gemma's decider, from the roster as it stood before this crop's own outcome.
sl = shortlist(emb, known, data.k, ch.get("gender")) sl = shortlist(emb, known, data.k, ch.get("gender"))
crop_uri = f"s3://manga/{data.manga_id}/characters/_crops/{data.panel_id}_{ch['local_id']}.png" # the embedding ships with the crop. gemma's decider can answer "none of these", and the
# orchestrator has to mint a character from that crop — which needs an embedding it cannot
# compute (siglip is resident here, gemma is resident there, and the mutex forbids both).
# Uploading it now is what removes the third siglip pass
# (`decisions/identity-bbox.md#none-mints-an-anonymous-character`).
key = f"{data.manga_id}/characters/_crops/{data.panel_id}_{ch['local_id']}"
crop_uri, emb_uri = transport.CHAR_PNG_URI.format(key=key), transport.CHAR_NPY_URI.format(key=key)
cp = f"{SHM}/cc_{uuid.uuid4().hex[:8]}.png"; cv2.imwrite(cp, crop) cp = f"{SHM}/cc_{uuid.uuid4().hex[:8]}.png"; cv2.imwrite(cp, crop)
transport.put(cp, crop_uri); os.remove(cp) transport.put(cp, crop_uri); os.remove(cp)
shortlists.append({"local_id": ch["local_id"], "crop_uri": crop_uri, _save_npy(emb, emb_uri)
shortlists.append({"local_id": ch["local_id"], "crop_uri": crop_uri, "emb_uri": emb_uri,
"candidates": [{"character_id": c["character_id"], "name": c.get("name"), "candidates": [{"character_id": c["character_id"], "name": c.get("name"),
"gender": c.get("gender"), "species": c.get("species"), "gender": c.get("gender"), "species": c.get("species"),
"appearance": c.get("description"), "cosine": c["cosine"], "appearance": c.get("description"), "cosine": c["cosine"],
worker_layers.py
+3 -3
View File
@@ -65,15 +65,15 @@ async def layers(data: LayerInput):
return {"layer_uris": [], "skipped": "comfyui down"} # ponytail: skip stage if ComfyUI not running return {"layer_uris": [], "skipped": "comfyui down"} # ponytail: skip stage if ComfyUI not running
local = transport.get(data.panel_uri, f"{SHM}/layer_{uuid.uuid4().hex[:8]}.png") local = transport.get(data.panel_uri, f"{SHM}/layer_{uuid.uuid4().hex[:8]}.png")
# orchestrator doesn't pass manga/chapter to layers; derive from the panel uri prefix. # orchestrator doesn't pass manga/chapter to layers; derive from the panel uri prefix.
parts = data.panel_uri.replace("s3://", "").split("/") manga_id, chapter_id = transport.ids_from_uri(data.panel_uri)
manga_id, chapter_id = parts[1], parts[2]
view_urls = _comfy_run(local, data.num_layers, data.prompt) view_urls = _comfy_run(local, data.num_layers, data.prompt)
layer_uris = [] layer_uris = []
for idx, url in enumerate(view_urls): for idx, url in enumerate(view_urls):
png = f"{SHM}/layer_{uuid.uuid4().hex[:6]}.png" png = f"{SHM}/layer_{uuid.uuid4().hex[:6]}.png"
with open(png, "wb") as f: with open(png, "wb") as f:
f.write(requests.get(url, timeout=60).content) f.write(requests.get(url, timeout=60).content)
uri = f"s3://manga/{manga_id}/{chapter_id}/layers/{data.panel_id or 'p'}/{idx}.png" uri = transport.LAYER_URI.format(manga_id=manga_id, chapter_id=chapter_id,
name=data.panel_id or 'p', idx=idx)
transport.put(png, uri) transport.put(png, uri)
os.remove(png) os.remove(png)
layer_uris.append(uri) layer_uris.append(uri)
worker_render.py
+171 -48
View File
@@ -19,13 +19,6 @@ MUSIC_BED = os.environ.get("MUSIC_BED", "") # #13 path/uri of a music t
MUSIC_GAIN = os.environ.get("MUSIC_GAIN", "0.18") # bed level before ducking MUSIC_GAIN = os.environ.get("MUSIC_GAIN", "0.18") # bed level before ducking
def _mc_from_uri(uri: str):
"""(manga_id, chapter_id) from s3://<bucket>/<manga_id>/<chapter_id>/...
the orchestrator doesn't pass ids to render/layers, but every input uri encodes them."""
parts = uri.replace("s3://", "").split("/")
return parts[1], parts[2]
def _ts(s): def _ts(s):
h = int(s // 3600); m = int(s % 3600 // 60); sec = s % 60 h = int(s // 3600); m = int(s % 3600 // 60); sec = s % 60
return f"{h}:{m:02d}:{sec:05.2f}" return f"{h}:{m:02d}:{sec:05.2f}"
@@ -37,10 +30,10 @@ def _ts(s):
SUB_MODE = os.environ.get("SUB_MODE", "minimal").lower() # off | minimal | boxed SUB_MODE = os.environ.get("SUB_MODE", "minimal").lower() # off | minimal | boxed
SUB_PRESETS = { SUB_PRESETS = {
# (mode, orientation): fontsize, borderstyle(1=outline,3=box), outline/pad, shadow, marginV, side # (mode, orientation): fontsize, borderstyle(1=outline,3=box), outline/pad, shadow, marginV, side
("minimal", "portrait"): dict(fs=40, bs=1, outline=3, shadow=2, mv=0.055, side=110), ("minimal", "portrait"): {"fs": 40, "bs": 1, "outline": 3, "shadow": 2, "mv": 0.055, "side": 110},
("minimal", "landscape"): dict(fs=32, bs=1, outline=3, shadow=2, mv=0.09, side=260), ("minimal", "landscape"): {"fs": 32, "bs": 1, "outline": 3, "shadow": 2, "mv": 0.09, "side": 260},
("boxed", "portrait"): dict(fs=40, bs=3, outline=6, shadow=0, mv=0.055, side=110), ("boxed", "portrait"): {"fs": 40, "bs": 3, "outline": 6, "shadow": 0, "mv": 0.055, "side": 110},
("boxed", "landscape"): dict(fs=32, bs=3, outline=6, shadow=0, mv=0.09, side=260), ("boxed", "landscape"): {"fs": 32, "bs": 3, "outline": 6, "shadow": 0, "mv": 0.09, "side": 260},
} }
@@ -109,15 +102,41 @@ def _ass(text: str, dur: float, path: str):
def _audio_dur(path: str) -> float: def _audio_dur(path: str) -> float:
"""clip length = narration length. scene_timing arrives empty, so probe the audio.""" """clip length = narration length. scene_timing arrives empty, so probe the audio."""
r = subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration", r = subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
"-of", "default=nk=1:nw=1", path], capture_output=True, text=True) "-of", "default=nk=1:nw=1", path], capture_output=True, text=True, check=False)
try: try:
return float(r.stdout.strip()) return float(r.stdout.strip())
except ValueError: except ValueError:
return 0.0 return 0.0
def _stream_dur(path: str, kind: str) -> float:
"""duration of one stream. `format=duration` is max(video,audio) and so hides A/V drift."""
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", f"{kind}:0",
"-show_entries", "stream=duration", "-of", "default=nk=1:nw=1", path],
capture_output=True, text=True, check=False)
try:
return float(r.stdout.strip())
except ValueError:
return 0.0
def _fps_of(path: str) -> str:
"""`r_frame_rate` as ffprobe reports it. Compared as a string on purpose: two clips agree only when
their rate AND therefore their time_base agree, and the stream-copy concat path cares about that."""
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0",
"-show_entries", "stream=r_frame_rate", "-of", "default=nk=1:nw=1", path],
capture_output=True, text=True, check=False)
return r.stdout.strip()
ZMAX, ZPAN = 1.15, 1.18 # ken-burns zoom ceiling; constant zoom that gives pans room to travel ZMAX, ZPAN = 1.15, 1.18 # ken-burns zoom ceiling; constant zoom that gives pans room to travel
# Every clip and every assembly stage MUST agree on this. xfade does not resample: it reinterprets the
# second input's frames at the first input's rate, so a 30fps input joined onto a 25fps one plays 1.2x
# too slow with the audio untouched -- the video ends minutes long and the narration runs ahead of the
# picture. That is exactly what a 25fps clip pipeline plus a `fps=30` concat branch produced.
FPS = 25
def _motion(camera: dict, frames: int) -> str: def _motion(camera: dict, frames: int) -> str:
"""#8 content-aware motion: map the vision `camera` block to a zoompan z/x/y expression. """#8 content-aware motion: map the vision `camera` block to a zoompan z/x/y expression.
@@ -155,16 +174,15 @@ def _motion(camera: dict, frames: int) -> str:
x, y = f"{xc}+(iw*0.03)*sin(6.283*on/{T})", f"{yc}+(ih*0.03)*cos(6.283*on/{T})" x, y = f"{xc}+(iw*0.03)*sin(6.283*on/{T})", f"{yc}+(ih*0.03)*cos(6.283*on/{T})"
else: # zoom_in (default ken burns) else: # zoom_in (default ken burns)
z, x, y = f"1+{ZMAX-1:.3f}*on/{T}", xc, yc z, x, y = f"1+{ZMAX-1:.3f}*on/{T}", xc, yc
return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps=25" return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps={FPS}"
def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict = None, def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict | None = None,
pad: float = PAD_S) -> list: pad: float = PAD_S) -> list:
"""ffmpeg: still panel over a blurred fill of itself + content-aware motion, burned subs, 9:16. """ffmpeg: still panel over a blurred fill of itself + content-aware motion, burned subs, 9:16.
#5 blurred bg replaces black bars: one copy scaled to COVER + blurred, the fitted panel on top. #5 blurred bg replaces black bars: one copy scaled to COVER + blurred, the fitted panel on top.
#13 pad seconds of trailing silence (last frame held) give the panel a beat before the next.""" #13 pad seconds of trailing silence (last frame held) give the panel a beat before the next."""
fps = 25 frames = max(1, int((dur + pad) * FPS)) # hold the last frame through the pad
frames = max(1, int((dur + pad) * fps)) # hold the last frame through the pad
# overlay's W/H/w/h are ffmpeg's main/overlay dims -- kept literal (no f-string braces). # overlay's W/H/w/h are ffmpeg's main/overlay dims -- kept literal (no f-string braces).
fc = ( fc = (
f"[0:v]split=2[bg][fg];" f"[0:v]split=2[bg][fg];"
@@ -206,8 +224,9 @@ async def render_scene(data: SceneInput):
_ass(data.narration_text, dur, ass) _ass(data.narration_text, dur, ass)
out = f"{SHM}/rnd_{tag}.mp4" out = f"{SHM}/rnd_{tag}.mp4"
subprocess.run(scene_cmd(img, audio, ass, out, dur, data.camera), check=True, capture_output=True) subprocess.run(scene_cmd(img, audio, ass, out, dur, data.camera), check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panel_uri) manga_id, chapter_id = transport.ids_from_uri(data.panel_uri)
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
name=data.panel_id or 'p')
transport.put(out, uri) transport.put(out, uri)
for p in (img, audio, ass, out): for p in (img, audio, ass, out):
os.remove(p) os.remove(p)
@@ -316,7 +335,7 @@ class CompositeInput(BaseModel):
def _img_size(path: str) -> tuple: def _img_size(path: str) -> tuple:
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries", r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
"stream=width,height", "-of", "csv=p=0:s=x", path], capture_output=True, text=True) "stream=width,height", "-of", "csv=p=0:s=x", path], capture_output=True, text=True, check=False)
w, h = r.stdout.strip().split("x") w, h = r.stdout.strip().split("x")
return int(w), int(h) return int(w), int(h)
@@ -352,8 +371,9 @@ async def render_composite(data: CompositeInput):
subprocess.run(composite_cmd(still, auds, ass, segs, rowh, out, t), check=True, capture_output=True) subprocess.run(composite_cmd(still, auds, ass, segs, rowh, out, t), check=True, capture_output=True)
for p in imgs + [still]: for p in imgs + [still]:
os.remove(p) os.remove(p)
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"]) manga_id, chapter_id = transport.ids_from_uri(data.panels[0]["panel_uri"])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
name=data.panel_id or 'p')
transport.put(out, uri) transport.put(out, uri)
for p in auds + [ass, out]: for p in auds + [ass, out]:
os.remove(p) os.remove(p)
@@ -399,8 +419,9 @@ async def render_group(data: GroupInput):
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", final] "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", final]
subprocess.run(cmd, check=True, capture_output=True) subprocess.run(cmd, check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"]) manga_id, chapter_id = transport.ids_from_uri(data.panels[0]["panel_uri"])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
name=data.panel_id or 'p')
transport.put(final, uri) transport.put(final, uri)
total = _audio_dur(final) or sum(durs) total = _audio_dur(final) or sum(durs)
for f in cleanup + [final]: for f in cleanup + [final]:
@@ -424,7 +445,7 @@ class BeatInput(BaseModel):
BEAT_MIN_PANEL_S = 1.0 # 185: no member panel flashes by faster than this BEAT_MIN_PANEL_S = 1.0 # 185: no member panel flashes by faster than this
def _beat_slices(D: float, n: int, weights: list = None) -> list: def _beat_slices(D: float, n: int, weights: list | None = None) -> list:
"""185: split beat duration D across n member panels by content weight, not evenly. """185: split beat duration D across n member panels by content weight, not evenly.
Each panel gets >= BEAT_MIN_PANEL_S so a low-weight panel never flashes. Degenerate input Each panel gets >= BEAT_MIN_PANEL_S so a low-weight panel never flashes. Degenerate input
(no/short weights, non-positive sum, or D too small for the floors) -> deterministic equal split. (no/short weights, non-positive sum, or D too small for the floors) -> deterministic equal split.
@@ -476,14 +497,14 @@ def cue_plan(text: str, D: float, slices: list) -> list:
return events return events
def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list = None, def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list | None = None,
weights: list = None) -> list: weights: list | None = None) -> list:
"""ffmpeg: N images shown as a ken-burns montage under ONE narration audio; the last image holds """ffmpeg: N images shown as a ken-burns montage under ONE narration audio; the last image holds
through the trailing pad. each image = blurred-fill bg + fitted panel + its camera move; the single through the trailing pad. each image = blurred-fill bg + fitted panel + its camera move; the single
burned subtitle (in `ass`) spans the whole beat. 185: per-image screen-time is content-weighted burned subtitle (in `ass`) spans the whole beat. 185: per-image screen-time is content-weighted
(see _beat_slices), equal split when weights are absent. pure -> testable without S3.""" (see _beat_slices), equal split when weights are absent. pure -> testable without S3."""
cameras = cameras or [] cameras = cameras or []
n, fps = len(imgs), 25 n, fps = len(imgs), FPS
slices = _beat_slices(D, n, weights) # 185: content-weighted, equal-split fallback slices = _beat_slices(D, n, weights) # 185: content-weighted, equal-split fallback
# one frame per image (no -loop): zoompan d=frames expands that single frame to exactly `frames` # one frame per image (no -loop): zoompan d=frames expands that single frame to exactly `frames`
# output frames = seg seconds. looping instead would feed many frames and zoompan multiplies each. # output frames = seg seconds. looping instead would feed many frames and zoompan multiplies each.
@@ -536,8 +557,9 @@ async def render_beat(data: BeatInput):
subprocess.run(beat_cmd(imgs, audio, ass, out, D, data.cameras, data.weights), subprocess.run(beat_cmd(imgs, audio, ass, out, D, data.cameras, data.weights),
check=True, capture_output=True) check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panel_uris[0]) manga_id, chapter_id = transport.ids_from_uri(data.panel_uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
name=data.panel_id or 'p')
transport.put(out, uri) transport.put(out, uri)
total = _audio_dur(out) or (D + PAD_S) total = _audio_dur(out) or (D + PAD_S)
for f in imgs + [audio, ass, out]: for f in imgs + [audio, ass, out]:
@@ -578,8 +600,8 @@ def collage_cmd(imgs, plate_i, rects, entrances, z_order, audio, ass, out, D, tr
for img in imgs: for img in imgs:
cmd += ["-loop", "1", "-i", img] # still held for the whole clip; -t caps output cmd += ["-loop", "1", "-i", img] # still held for the whole clip; -t caps output
cmd += ["-i", audio] cmd += ["-i", audio]
parts = [f"[{plate_i}:v]scale={W}:{H}:force_original_aspect_ratio=increase," parts = [(f"[{plate_i}:v]scale={W}:{H}:force_original_aspect_ratio=increase,"
f"crop={W}:{H},boxblur=24:2,setsar=1[bg]"] f"crop={W}:{H},boxblur=24:2,setsar=1[bg]")]
base = "bg" base = "bg"
for k, i in enumerate(z_order): # shadows first, at rest positions (static) for k, i in enumerate(z_order): # shadows first, at rest positions (static)
x, y, w, h = (int(round(v)) for v in rects[i]) x, y, w, h = (int(round(v)) for v in rects[i])
@@ -598,7 +620,7 @@ def collage_cmd(imgs, plate_i, rects, entrances, z_order, audio, ass, out, D, tr
cur = f"o{k}" cur = f"o{k}"
fc = ";".join(parts) + f";[{cur}]ass={ass}[v];[{n}:a]apad=pad_dur={PAD_S:.3f}[a]" fc = ";".join(parts) + f";[{cur}]ass={ass}[v];[{n}:a]apad=pad_dur={PAD_S:.3f}[a]"
cmd += ["-filter_complex", fc, "-map", "[v]", "-map", "[a]", "-t", f"{T:.3f}", cmd += ["-filter_complex", fc, "-map", "[v]", "-map", "[a]", "-t", f"{T:.3f}",
"-r", "30", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", out] "-r", str(FPS), "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", out]
return cmd return cmd
@@ -636,8 +658,9 @@ async def render_collage(data: CollageInput):
out = f"{SHM}/cl_{tag}.mp4" out = f"{SHM}/cl_{tag}.mp4"
subprocess.run(collage_cmd(imgs, active, lay["rects"], lay["entrances"], lay["z_order"], subprocess.run(collage_cmd(imgs, active, lay["rects"], lay["entrances"], lay["z_order"],
audio, ass, out, D, lay["transition_s"]), check=True, capture_output=True) audio, ass, out, D, lay["transition_s"]), check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(uris[0]) manga_id, chapter_id = transport.ids_from_uri(uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
name=data.panel_id or 'p')
transport.put(out, uri) transport.put(out, uri)
total = _audio_dur(out) or (D + PAD_S) total = _audio_dur(out) or (D + PAD_S)
for f in imgs + [audio, ass, out]: for f in imgs + [audio, ass, out]:
@@ -670,28 +693,57 @@ FFMPEG_THREADS = max(1, int(os.environ.get("FFMPEG_THREADS", "2")))
def _xfade_chain(durs: list, trans: list): def _xfade_chain(durs: list, trans: list):
"""build a filter_complex that xfades N clips with per-boundary transitions, keeping audio in """build a filter_complex that xfades N clips with per-boundary transitions, keeping audio in
sync via matching acrossfade. trans[i] is the transition OUT of clip i (boundary i->i+1). sync via matching acrossfade. trans[i] is the transition OUT of clip i (boundary i->i+1).
returns (filtergraph, video_label, audio_label). offsets accumulate as clips overlap.""" returns (filtergraph, video_label, audio_label, expected_duration). offsets accumulate as clips
overlap. `durs[i]` MUST be min(video, audio) of input i, not `format=duration`."""
# A clip whose duration probed as 0/unreadable must not poison the chain: with dur=0 the offset # A clip whose duration probed as 0/unreadable must not poison the chain: with dur=0 the offset
# accumulator would run BACKWARDS (cum += dur - td), swallowing every later clip into a frozen # accumulator would run BACKWARDS (cum += dur - td), swallowing every later clip into a frozen
# overlap near the middle. Floor to a small positive length so the timeline stays monotonic. # overlap near the middle. Floor to a small positive length so the timeline stays monotonic.
durs = [d if (d and d > 0.1) else 0.1 for d in durs] durs = [d if (d and d > 0.1) else 0.1 for d in durs]
parts, vlast, alast, cum = [], "[0:v]", "[0:a]", durs[0] # The offset accumulator is only as good as `durs`: an offset that lands even one frame past the end
# of input i-1 makes xfade emit the transition and then SILENTLY DROP input i and the whole rest of
# the chain -- rc 0, no warning, a chapter minutes short with the audio intact. So truncate every
# input to a whole number of frames it certainly has (floor, and min(v,a) from the caller) and trim
# both streams to exactly that. Then cum is the real timeline, not an estimate of it.
durs = [max(1, int(d * FPS)) / FPS for d in durs]
# Normalize every input to FPS/SAR before it reaches xfade, exactly as the concat branch does. Both
# branches feed the same tree, so an un-normalized xfade input is what stretched the chapter 1.2x.
parts = []
for i, d in enumerate(durs):
parts.append(f"[{i}:v]setsar=1,fps={FPS},trim=end={d:.3f},setpts=PTS-STARTPTS[n{i}]")
parts.append(f"[{i}:a]atrim=end={d:.3f},asetpts=PTS-STARTPTS[m{i}]")
vlast, alast, cum = "[n0]", "[m0]", durs[0]
for i in range(1, len(durs)): for i in range(1, len(durs)):
name, td = XFADE.get(trans[i - 1] if i - 1 < len(trans) else "cut", XFADE["cut"]) name, td = XFADE.get(trans[i - 1] if i - 1 < len(trans) else "cut", XFADE["cut"])
td = max(0.05, min(td, durs[i - 1] - 0.05, durs[i] - 0.05)) # overlap fits in both clips td = max(0.05, min(td, durs[i - 1] - 0.05, durs[i] - 0.05)) # overlap fits in both clips
td = max(1, int(td * FPS)) / FPS # ...on a frame boundary
off = max(cum - td, 0) off = max(cum - td, 0)
parts.append(f"{vlast}[{i}:v]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]") parts.append(f"{vlast}[n{i}]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]")
parts.append(f"{alast}[{i}:a]acrossfade=d={td:.3f}[a{i}]") parts.append(f"{alast}[m{i}]acrossfade=d={td:.3f}[a{i}]")
vlast, alast, cum = f"[v{i}]", f"[a{i}]", cum + durs[i] - td vlast, alast, cum = f"[v{i}]", f"[a{i}]", cum + durs[i] - td
return ";".join(parts), vlast, alast return ";".join(parts), vlast, alast, cum
ASSEMBLE_TOL_S = 0.5 # frame-boundary + aac-padding slack; a dropped input is off by whole seconds
def _check_assembled(out: str, expect: float):
"""ffmpeg drops xfade inputs and re-times concat segments without ever failing, so verify the
result instead of trusting rc 0. Both streams, because a video-only loss is the failure mode that
shipped a 436s picture over 364s of narration."""
v, a = _stream_dur(out, "v"), _stream_dur(out, "a")
if abs(v - expect) > ASSEMBLE_TOL_S or abs(v - a) > ASSEMBLE_TOL_S:
raise RuntimeError(f"assembly lost stream time in {os.path.basename(out)}: "
f"video={v:.2f} audio={a:.2f} expected={expect:.2f}")
def _assemble_once(inputs: list[str], trans: list[str], out: str): def _assemble_once(inputs: list[str], trans: list[str], out: str):
"""Assemble one bounded batch. `trans[i]` is the transition out of inputs[i].""" """Assemble one bounded batch. `trans[i]` is the transition out of inputs[i]."""
# min(video, audio), never `format=duration`: that is max(video, audio), and feeding it to
# _xfade_chain puts the offset accumulator ahead of the real video timeline.
durs = [min(_stream_dur(p, "v"), _stream_dur(p, "a")) for p in inputs]
fancy = len(inputs) >= 2 and any(t not in ("", "cut") for t in trans[:len(inputs) - 1]) fancy = len(inputs) >= 2 and any(t not in ("", "cut") for t in trans[:len(inputs) - 1])
if fancy: if fancy:
durs = [_audio_dur(p) for p in inputs] fg, vmap, amap, expect = _xfade_chain(durs, trans)
fg, vmap, amap = _xfade_chain(durs, trans)
cmd = ["ffmpeg", "-y", "-filter_complex_threads", str(FFMPEG_THREADS)] cmd = ["ffmpeg", "-y", "-filter_complex_threads", str(FFMPEG_THREADS)]
# Input-side -threads limits each decoder; otherwise ffmpeg may create a decoder thread pool # Input-side -threads limits each decoder; otherwise ffmpeg may create a decoder thread pool
# for every input in the batch in addition to the filter and libx264 pools. # for every input in the batch in addition to the filter and libx264 pools.
@@ -701,6 +753,7 @@ def _assemble_once(inputs: list[str], trans: list[str], out: str):
"-c:v", "libx264", "-threads", str(FFMPEG_THREADS), "-pix_fmt", "yuv420p", "-c:v", "libx264", "-threads", str(FFMPEG_THREADS), "-pix_fmt", "yuv420p",
"-c:a", "aac", "-b:a", "192k", out] "-c:a", "aac", "-b:a", "192k", out]
subprocess.run(cmd, check=True, capture_output=True) subprocess.run(cmd, check=True, capture_output=True)
_check_assembled(out, expect)
return return
# A cut-only batch is concatenated with the concat FILTER, not -c copy or the concat demuxer. # A cut-only batch is concatenated with the concat FILTER, not -c copy or the concat demuxer.
@@ -713,12 +766,13 @@ def _assemble_once(inputs: list[str], trans: list[str], out: str):
cmd = ["ffmpeg", "-y"] cmd = ["ffmpeg", "-y"]
for p in inputs: for p in inputs:
cmd += ["-i", p] cmd += ["-i", p]
pre = "".join(f"[{i}:v]setsar=1,fps=30[v{i}];" for i in range(n)) pre = "".join(f"[{i}:v]setsar=1,fps={FPS}[v{i}];" for i in range(n))
fg = pre + "".join(f"[v{i}][{i}:a]" for i in range(n)) + f"concat=n={n}:v=1:a=1[v][a]" fg = pre + "".join(f"[v{i}][{i}:a]" for i in range(n)) + f"concat=n={n}:v=1:a=1[v][a]"
cmd += ["-filter_complex", fg, "-map", "[v]", "-map", "[a]", cmd += ["-filter_complex", fg, "-map", "[v]", "-map", "[a]",
"-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p", "-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p",
"-threads", str(FFMPEG_THREADS), "-c:a", "aac", "-b:a", "192k", out] "-threads", str(FFMPEG_THREADS), "-c:a", "aac", "-b:a", "192k", out]
subprocess.run(cmd, check=True, capture_output=True) subprocess.run(cmd, check=True, capture_output=True)
_check_assembled(out, sum(durs))
def _assemble_batched(inputs: list[str], transitions: list[str], out: str, tag: str, def _assemble_batched(inputs: list[str], transitions: list[str], out: str, tag: str,
@@ -816,20 +870,31 @@ async def assemble(data: AssembleInput):
cleanup = list(locals_) + [out] cleanup = list(locals_) + [out]
fancy = len(locals_) >= 2 and any(t not in ("", "cut") for t in data.transitions) fancy = len(locals_) >= 2 and any(t not in ("", "cut") for t in data.transitions)
if fancy: # The stream-copy path writes the output with the FIRST input's time_base and reinterprets every
# later packet in it. A clip encoded at 30fps (time_base 1/15360) copied into a 25fps container
# (1/12800) therefore plays 15360/12800 = 1.2x too long with its audio untouched. That is the whole
# of the shipped chapter's 436.39s of video over 363.67s of narration: 14 of 49 clips came off the
# collage path, which hardcoded `-r 30`. Mixed rates must re-encode, so they go through the tree,
# whose branches both normalize with `fps={FPS}`.
rates = {_fps_of(p) for p in locals_}
if fancy or len(rates) > 1:
if len(rates) > 1:
print(f"[render] mixed clip rates {sorted(rates)}, re-encoding instead of stream copy",
flush=True)
_assemble_batched(locals_, data.transitions, out, tag, cleanup) _assemble_batched(locals_, data.transitions, out, tag, cleanup)
else: else:
# all hard cuts: stream-copy concat (no re-encode) -- unchanged fast path. # all hard cuts at one shared rate: stream-copy concat, no re-encode.
listfile = f"{SHM}/asm_{tag}.txt"; cleanup.append(listfile) listfile = f"{SHM}/asm_{tag}.txt"; cleanup.append(listfile)
with open(listfile, "w") as f: with open(listfile, "w") as f:
f.write("".join(f"file '{p}'\n" for p in locals_)) f.write("".join(f"file '{p}'\n" for p in locals_))
subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", listfile, "-c", "copy", out], subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", listfile, "-c", "copy", out],
check=True, capture_output=True) check=True, capture_output=True)
_check_assembled(out, sum(min(_stream_dur(p, "v"), _stream_dur(p, "a")) for p in locals_))
out = _add_music_bed(out, tag, cleanup) out = _add_music_bed(out, tag, cleanup)
manga_id, chapter_id = _mc_from_uri(data.clip_uris[0]) manga_id, chapter_id = transport.ids_from_uri(data.clip_uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/chapter.mp4" uri = transport.CHAPTER_URI.format(manga_id=manga_id, chapter_id=chapter_id)
transport.put(out, uri) transport.put(out, uri)
for p in cleanup: for p in cleanup:
os.remove(p) os.remove(p)
@@ -933,8 +998,14 @@ if __name__ == "__main__":
for p in (img, bed, *cl): for p in (img, bed, *cl):
if os.path.exists(p): os.remove(p) if os.path.exists(p): os.remove(p)
# #6 transitions: two real clips xfade into one chapter; graph offsets/labels well-formed. # #6 transitions: two real clips xfade into one chapter; graph offsets/labels well-formed.
fg, vmap, amap = _xfade_chain([1.0, 1.0], ["fade_white"]) fg, vmap, amap, exp = _xfade_chain([1.0, 1.0], ["fade_white"])
assert "xfade=transition=fadewhite" in fg and vmap == "[v1]" and amap == "[a1]" assert "xfade=transition=fadewhite" in fg and vmap == "[v1]" and amap == "[a1]"
# Both streams of every input trimmed to a whole frame count. That is what keeps the offset
# accumulator ON the real timeline: an offset one frame past the end of input i-1 makes xfade
# emit the transition and then silently drop input i and everything after it, rc 0, no warning.
assert fg.count(",trim=end=") == 2 and fg.count("]atrim=end=") == 2, fg
_td = max(1, int(XFADE["fade_white"][1] * FPS)) / FPS
assert abs(exp - (2.0 - _td)) < 0.001, (exp, _td)
img = f"{SHM}/t.png" img = f"{SHM}/t.png"
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "color=c=black:s=400x600", subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "color=c=black:s=400x600",
"-frames:v", "1", img], check=True, capture_output=True) "-frames:v", "1", img], check=True, capture_output=True)
@@ -945,7 +1016,54 @@ if __name__ == "__main__":
"-map", vmap, "-map", amap, "-c:v", "libx264", "-pix_fmt", "yuv420p", "-map", vmap, "-map", amap, "-c:v", "libx264", "-pix_fmt", "yuv420p",
"-c:a", "aac", out], check=True, capture_output=True) "-c:a", "aac", out], check=True, capture_output=True)
assert os.path.getsize(out) > 0 assert os.path.getsize(out) > 0
for p in (img, c0, c1): # A/V drift: the tree must not stretch video. The concat branch normalizes fps and the xfade
# branch used not to, so a chapter mixing both played 1.2x slow with the audio untouched and the
# narration ran ahead of the picture. Batch=2 over 4 clips forces BOTH branches plus a second
# round -- the shipped bug's exact shape. Compare the streams, not the file size.
c2, c3 = f"{SHM}/c2.mp4", f"{SHM}/c3.mp4"
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c2, 1.0), check=True, capture_output=True)
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c3, 1.0), check=True, capture_output=True)
_saved_batch, cl = ASSEMBLE_BATCH, []
globals()["ASSEMBLE_BATCH"] = 2
try:
# ["crossfade","crossfade","cut","cut"] with batch=2 is the one shape that mixes branches:
# round 1 sends [c0,c1] through xfade and [c2,c3] through concat, then round 2 xfades those
# two intermediates together. A cut-only first group would keep both on the concat branch and
# the rates would agree by accident, which is why this case has to be spelled out.
_assemble_batched([c0, c1, c2, c3], ["crossfade", "crossfade", "cut", "cut"],
out, "selfck", cl)
finally:
globals()["ASSEMBLE_BATCH"] = _saved_batch
for p in cl:
if os.path.exists(p): os.remove(p)
vd, ad = _stream_dur(out, "v"), _stream_dur(out, "a")
assert abs(vd - ad) < 0.25, f"A/V drift: video {vd:.2f}s vs audio {ad:.2f}s"
# ...and the guard that catches it in production must actually fire. `assemble` reports success
# off ffmpeg's rc, which is 0 even when a whole batch of inputs is thrown away.
try:
_check_assembled(out, vd + 5.0)
raise AssertionError("_check_assembled did not fire on a 5s loss")
except RuntimeError:
pass
for p in (img, c0, c1, c2, c3):
os.remove(p)
# The offset creep itself, reproduced small. A real clip's audio outlasts its video slightly, and
# `format=duration` reports the audio, so the accumulator walked ahead of the picture until one
# xfade window ran past the end of its first input -- at which point ffmpeg emitted the
# transition, threw away the second input and everything downstream, and exited 0. Clips with
# audio 0.4s longer than the video exaggerate one clip's worth of that creep.
cs = []
for i in range(3):
c = f"{SHM}/creep{i}.mp4"; cs.append(c)
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "testsrc2=s=160x120:r=25:d=1.0",
"-f", "lavfi", "-i", f"sine=f={300 + i * 40}:d=1.4",
"-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",
"-c:a", "aac", c], check=True, capture_output=True)
_assemble_once(cs, ["fade_black", "cut", "cut"], out) # raises if an input was dropped
vd, ad = _stream_dur(out, "v"), _stream_dur(out, "a")
assert vd > 2.2, f"xfade dropped inputs: {vd:.2f}s from 3 clips of 1.0s"
assert abs(vd - ad) < 0.1, f"A/V drift: video {vd:.2f}s vs audio {ad:.2f}s"
for p in cs:
os.remove(p) os.remove(p)
# #6 composite: 2 panels + 2 audios -> one stacked clip; duration = sum, subs timed per row. # #6 composite: 2 panels + 2 audios -> one stacked clip; duration = sum, subs timed per row.
a2 = f"{SHM}/a2.wav" a2 = f"{SHM}/a2.wav"
@@ -987,6 +1105,11 @@ if __name__ == "__main__":
check=True, capture_output=True) check=True, capture_output=True)
assert abs(_audio_dur(out) - (1.5 + PAD_S)) < 0.2, _audio_dur(out) # collage clip = beat length assert abs(_audio_dur(out) - (1.5 + PAD_S)) < 0.2, _audio_dur(out) # collage clip = beat length
assert os.path.getsize(out) > 0 assert os.path.getsize(out) > 0
# ...and it must come out at FPS like every other clip path. This one hardcoded `-r 30`, so 14 of
# 49 clips in the shipped chapter were 30fps. `concat -c copy` writes the output with the FIRST
# clip's time_base and reinterprets later packets in it, so those 14 played 1.2x too long with
# their narration untouched. That, not the xfade tree, is where the 72.7s gap came from.
assert _fps_of(out) == f"{FPS}/1", f"collage clip is {_fps_of(out)}, not {FPS}/1"
for p in (b0, b1, b2, bnar): for p in (b0, b1, b2, bnar):
os.remove(p) os.remove(p)
os.remove(aud); os.remove(out) os.remove(aud); os.remove(out)
worker_scene.py
+52 -3
View File
@@ -70,20 +70,38 @@ def build_scene(data: SceneInput):
# Speaker attribution is the vision/dialogue model's job now (it reads bubble tails + turn-taking). # Speaker attribution is the vision/dialogue model's job now (it reads bubble tails + turn-taking).
# speaker is a local_id -> map to character_id via identity; None for narration/sfx/off-panel. # speaker is a local_id -> map to character_id via identity; None for narration/sfx/off-panel.
# The orchestrator's normalize_dialogue already rewrote `speaker` to a character_id and put the
# typed answer in `speaker_ref`, so the local-id map no longer matches it. Read speaker_ref FIRST;
# every lookup used to miss and every line narrated as "Someone".
dialogue = [] dialogue = []
for d in data.vision_result.get("dialogue", []): for d in data.vision_result.get("dialogue", []):
raw = d.get("speaker") raw = d.get("speaker")
ref = d.get("speaker_ref") or {}
# "unknown" is a DELIBERATE off-panel/indeterminate speaker (narrated as "someone"); it maps # "unknown" is a DELIBERATE off-panel/indeterminate speaker (narrated as "someone"); it maps
# to None just like narration. id_by_local.get already yields None for it. # to None just like narration. id_by_local.get already yields None for it.
dialogue.append({"speaker": id_by_local.get(raw), "text": d.get("text", ""), if ref.get("kind") == "character_id" and ref.get("value"):
cid = ref["value"]
else:
cid = id_by_local.get(raw) or (raw if raw in name_by_id else None)
dialogue.append({"speaker": cid, "text": d.get("text", ""),
"type": d.get("type", "speech"), "type": d.get("type", "speech"),
"confidence": d.get("confidence"), "confidence": d.get("confidence"),
"speaker_method": d.get("speaker_method", "unknown")}) "speaker_method": d.get("speaker_method", "unknown")})
vchars = data.vision_result.get("characters", []) vchars = data.vision_result.get("characters", [])
action = "; ".join(c.get("action", "") for c in vchars if c.get("action")) or "" # `actions` is the list the orchestrator's beat builder reads for verifier evidence; `action` is the
# joined string the script prompt renders. Emitting only the string left verification blind.
# `characters` above already drops an unassigned detection, but its ACTION used to survive, so a
# background extra standing at a window became a sentence the narrator read out. A detection with no
# detected face is an extra, a figure on a poster, or scenery gemma called a person
# (`decisions/identity-bbox.md#face-gates-enrollment`). Gate on `is False`, like the enrollment gate,
# so a blob written before the gate existed and a panel whose detector failed both behave as before.
actions = [c["action"].strip() for c in vchars
if str(c.get("action") or "").strip() and c.get("has_face") is not False]
action = "; ".join(actions)
return {"panel_id": data.panel_id, "characters": characters, "dialogue": dialogue, return {"panel_id": data.panel_id, "characters": characters, "dialogue": dialogue,
"action": action, "entities": data.vision_result.get("entities", []), "action": action, "actions": actions,
"entities": data.vision_result.get("entities", []),
"camera": data.vision_result.get("camera", {}), # #8 direction passthrough -> render "camera": data.vision_result.get("camera", {}), # #8 direction passthrough -> render
"transition": data.vision_result.get("transition", "cut")} # #6 transition out "transition": data.vision_result.get("transition", "cut")} # #6 transition out
@@ -120,6 +138,20 @@ if __name__ == "__main__":
"description": '{"hair":"black","features":["glasses"]}'}], "description": '{"hair":"black","features":["glasses"]}'}],
)) ))
assert outu["characters"][0]["name"] == "" and outu["characters"][0]["label"] == "the one with black hair and glasses" assert outu["characters"][0]["name"] == "" and outu["characters"][0]["label"] == "the one with black hair and glasses"
# a faceless detection's ACTION never reaches the script prompt or the verifier's evidence, but a
# real cast member's does, and so does one on a panel where the detector never ran (no key).
p7 = build_scene(SceneInput(
panel_id="p007",
vision_result={"characters": [
{"local_id": "person_1", "action": "reading a report", "has_face": True},
{"local_id": "person_2", "action": "standing at the window", "has_face": False},
{"local_id": "person_3", "action": "pointing"},
]},
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
characters_registry=[{"character_id": "c1", "name": "Seonho"}],
))
assert p7["actions"] == ["reading a report", "pointing"], p7["actions"]
assert "window" not in p7["action"], p7["action"]
assert _describe({}) == "" # bare appearance -> empty, script falls back to Person X assert _describe({}) == "" # bare appearance -> empty, script falls back to Person X
# animals are named as their species (color + species), never by clothing/"the one with..." # animals are named as their species (color + species), never by clothing/"the one with..."
assert _describe({"species": "cat", "hair": "black"}) == "the black cat" assert _describe({"species": "cat", "hair": "black"}) == "the black cat"
@@ -167,4 +199,21 @@ if __name__ == "__main__":
)) ))
assert out5["dialogue"][0]["confidence"] == 0.3 assert out5["dialogue"][0]["confidence"] == 0.3
assert out5["dialogue"][0]["speaker_method"] == "turn_taking" assert out5["dialogue"][0]["speaker_method"] == "turn_taking"
# orchestrator-normalized rows: `speaker` is already a character_id and `speaker_ref` is typed.
# the local-id map cannot resolve either — speaker_ref must win, or everything narrates as Someone.
out6 = build_scene(SceneInput(
panel_id="p007",
vision_result={"characters": [{"local_id": "person_1", "bbox": [0, 0, 10, 10]}],
"dialogue": [{"speaker": "c1", "speaker_ref": {"kind": "character_id", "value": "c1"},
"type": "speech", "text": "Hey."},
{"speaker": None, "speaker_ref": {"kind": "unknown", "value": None},
"type": "speech", "text": "..."}]},
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
characters_registry=[{"character_id": "c1", "name": "Teto"}],
))
assert out6["dialogue"][0]["speaker"] == "c1", out6["dialogue"][0]
assert out6["dialogue"][1]["speaker"] is None, out6["dialogue"][1]
# actions survive as a list for the verifier, not only as the joined prompt string
assert out["actions"] == ["waving"] and out["action"] == "waving"
print("worker_scene self-check ok") print("worker_scene self-check ok")
worker_script.py
+48 -6
View File
@@ -21,6 +21,10 @@ class ScriptInput(BaseModel):
recent: list = [] # #1: last few panels' narration text (local flow, no repeats) recent: list = [] # #1: last few panels' narration text (local flow, no repeats)
introduced: list = [] # #2: names already narrated -> refer by name/pronoun, don't re-describe introduced: list = [] # #2: names already narrated -> refer by name/pronoun, don't re-describe
panel_count: int = 1 # scene-level narration: #panels in this beat -> scales the length budget panel_count: int = 1 # scene-level narration: #panels in this beat -> scales the length budget
# the orchestrator has been sending both of these on a failed-script retry since the verifier
# landed; pydantic dropped them silently, so the retry was another blind roll of the dice.
beat: dict = {} # {beat_id, panels:[{panel_id, dialogue:[{text}], actions}]} — the evidence
verifier_feedback: list = [] # [{kind, values|quote}] from verify_script on the previous attempt
class SummaryInput(BaseModel): class SummaryInput(BaseModel):
@@ -80,9 +84,33 @@ def _chars_line(name_by_id, genders_by_id):
return ", ".join(out) or "none" return ", ".join(out) or "none"
def _feedback_block(feedback, beat) -> str:
"""Render the previous attempt's verifier failures as concrete corrections. Without this the
retry is just another sample at the same temperature — the model never learns what was wrong."""
if not feedback:
return ""
lines = []
for f in feedback:
kind = f.get("kind")
if kind == "unsupported-proper-noun":
lines.append("- You used name(s) that are not in this beat: "
+ ", ".join(str(v) for v in f.get("values", []))
+ ". Use only the names listed under 'Characters present', or a pronoun.")
elif kind == "misquote":
lines.append(f'- Your quote "{f.get("quote","")}" is not what the panel says. '
"Quote the exact words below, or drop the quotation marks.")
else:
lines.append(f"- {kind}: {json.dumps(f, ensure_ascii=False)[:200]}")
quotes = [d.get("text", "") for p in (beat or {}).get("panels", [])
for d in p.get("dialogue", []) if d.get("text")]
exact = ("Exact lines you may quote from:\n" + "\n".join(f'- "{q}"' for q in quotes) + "\n") if quotes else ""
return ("Your previous attempt was REJECTED. Fix exactly these problems and rewrite it:\n"
+ "\n".join(lines) + "\n" + exact + "\n")
def build_prompt(sg: dict, chapter_context: str, names_by_id=None, def build_prompt(sg: dict, chapter_context: str, names_by_id=None,
brief: str = "", recent=None, introduced=None, genders_by_id=None, brief: str = "", recent=None, introduced=None, genders_by_id=None,
panel_count: int = 1) -> str: panel_count: int = 1, beat=None, verifier_feedback=None) -> str:
name_by_id = _name_map(sg.get("characters", []), names_by_id) name_by_id = _name_map(sg.get("characters", []), names_by_id)
chars = _chars_line(name_by_id, genders_by_id) chars = _chars_line(name_by_id, genders_by_id)
dialogue = "\n".join(_render_line(d, name_by_id) for d in sg.get("dialogue", [])) or "none" dialogue = "\n".join(_render_line(d, name_by_id) for d in sg.get("dialogue", [])) or "none"
@@ -120,7 +148,7 @@ def build_prompt(sg: dict, chapter_context: str, names_by_id=None,
else: else:
unit, budget, thin = "this manga panel", "1-2 short sentences, max ~25 words", "panel" unit, budget, thin = "this manga panel", "1-2 short sentences, max ~25 words", "panel"
return ( return (
ov + rec + ctx + intro + _feedback_block(verifier_feedback, beat) + ov + rec + ctx + intro +
f"Write TIGHT recap narration for {unit}: {budget}, " f"Write TIGHT recap narration for {unit}: {budget}, "
"present tense. Tell it like you're recapping to a friend — natural, with momentum. Output " "present tense. Tell it like you're recapping to a friend — natural, with momentum. Output "
"only the narration — no preamble, no notes, no alternatives, never ask for more info. If the " "only the narration — no preamble, no notes, no alternatives, never ask for more info. If the "
@@ -182,7 +210,7 @@ async def summary(data: SummaryInput):
async def script(data: ScriptInput): async def script(data: ScriptInput):
text = call_gemma4(build_prompt(data.scene_graph, data.chapter_context, data.names_by_id, text = call_gemma4(build_prompt(data.scene_graph, data.chapter_context, data.names_by_id,
data.brief, data.recent, data.introduced, data.genders_by_id, data.brief, data.recent, data.introduced, data.genders_by_id,
data.panel_count)) data.panel_count, data.beat, data.verifier_feedback))
return {"panel_id": data.panel_id, "text": text} return {"panel_id": data.panel_id, "text": text}
@@ -199,10 +227,16 @@ class NormalizeInput(BaseModel):
def _extract_json(raw: str) -> dict: def _extract_json(raw: str) -> dict:
text = _strip_thought(raw) text = _strip_thought(raw)
m = re.search(r"\{.*\}", text, re.DOTALL) i = text.find("{")
if not m: if i < 0:
raise ValueError(f"no json: {text[:200]}") raise ValueError(f"no json: {text[:200]}")
return json.loads(m.group(0)) # raw_decode stops at the end of the FIRST object. The old greedy `\{.*\}` swallowed any trailing
# brace in prose after it and turned a parseable reply into a wasted repair call.
try:
obj, _ = json.JSONDecoder().raw_decode(text[i:])
except json.JSONDecodeError as e:
raise ValueError(f"bad json: {text[:200]}") from e
return obj
def build_normalize_prompt(names, entities) -> str: def build_normalize_prompt(names, entities) -> str:
@@ -297,4 +331,12 @@ if __name__ == "__main__":
nr = _extract_json('{"entities":[{"name":"Everyday","kind":"shop"}],' nr = _extract_json('{"entities":[{"name":"Everyday","kind":"shop"}],'
'"name_fixes":{"CHOI HAESEON":"Choi Haeseon"}}') '"name_fixes":{"CHOI HAESEON":"Choi Haeseon"}}')
assert nr["entities"][0]["name"] == "Everyday" and nr["name_fixes"]["CHOI HAESEON"] == "Choi Haeseon" assert nr["entities"][0]["name"] == "Everyday" and nr["name_fixes"]["CHOI HAESEON"] == "Choi Haeseon"
# trailing braced prose after the object must not break the parse (was a wasted repair call)
assert _extract_json('{"a":1}\nnote: {not json}')["a"] == 1
# verifier feedback reaches the retry prompt with the exact allowed quotes; absent by default
assert "REJECTED" not in p
beat = {"beat_id": "b1", "panels": [{"panel_id": "p1", "dialogue": [{"text": "stand proud"}]}]}
pf = build_prompt(sg, "s", beat=beat, verifier_feedback=[
{"kind": "unsupported-proper-noun", "values": ["Kyoto"]}, {"kind": "misquote", "quote": "be proud"}])
assert "REJECTED" in pf and "Kyoto" in pf and "be proud" in pf and '- "stand proud"' in pf
print("worker_script self-check ok") print("worker_script self-check ok")
worker_tts.py
+23 -5
View File
@@ -36,7 +36,17 @@ def _load_tts():
transformers.AutoTokenizer.from_pretrained = classmethod( transformers.AutoTokenizer.from_pretrained = classmethod(
lambda cls, *a, **kw: _orig(cls, *a, **{"fix_mistral_regex": True, **kw}) lambda cls, *a, **kw: _orig(cls, *a, **{"fix_mistral_regex": True, **kw})
) )
from dots_tts.runtime import DotsTtsRuntime # dots_tts/__init__.py refuses to import when torch and torchaudio minors differ. workpc runs
# Arch's torch 2.13 but pytorch.org ships no torchaudio past 2.11 for ROCm, so the pair can't
# be satisfied; 2.11 loads and resamples fine against 2.13. Lie to the guard for the import.
# ponytail: drop this once a torchaudio matching torch's minor exists for ROCm.
import torch, importlib.metadata as _md
_ver = _md.version
_md.version = lambda n: torch.__version__ if n == "torchaudio" else _ver(n)
try:
from dots_tts.runtime import DotsTtsRuntime
finally:
_md.version = _ver
_tts = DotsTtsRuntime.from_pretrained(DOTS_MODEL, precision="bfloat16") _tts = DotsTtsRuntime.from_pretrained(DOTS_MODEL, precision="bfloat16")
return _tts return _tts
@@ -60,9 +70,10 @@ def _audio_uri(data: "TTSInput") -> str:
# ponytail: flat key collides across chapters (as does the homesrv audio table); pass # ponytail: flat key collides across chapters (as does the homesrv audio table); pass
# panel_uri from run_stage_tts to make it per-chapter unique. # panel_uri from run_stage_tts to make it per-chapter unique.
if data.panel_uri: if data.panel_uri:
parts = data.panel_uri.replace("s3://", "").split("/") manga_id, chapter_id = transport.ids_from_uri(data.panel_uri)
return f"s3://manga/{parts[1]}/{parts[2]}/audio/{data.panel_id or 'p'}.wav" return transport.AUDIO_URI.format(manga_id=manga_id, chapter_id=chapter_id,
return f"s3://manga/_audio/{data.panel_id or 'p'}.wav" name=data.panel_id or 'p')
return transport.AUDIO_FLAT_URI.format(name=data.panel_id or 'p')
def _ensure_ref() -> str: def _ensure_ref() -> str:
@@ -207,6 +218,13 @@ if __name__ == "__main__":
os.remove(explicit) os.remove(explicit)
assert _calm("Stop!! Now!") == "Stop. Now." and _calm("no bangs") == "no bangs" # even prosody assert _calm("Stop!! Now!") == "Stop. Now." and _calm("no bangs") == "no bangs" # even prosody
# _audio_uri: a panel uri puts the wav beside its chapter, no panel uri falls back to the flat key.
assert _audio_uri(TTSInput(text="x", panel_id="p003",
panel_uri="s3://panels/m1/c1/panels/p003.png")) \
== "s3://audio/m1/c1/audio/p003.wav"
assert _audio_uri(TTSInput(text="x", panel_id="p003")) == "s3://audio/_audio/p003.wav"
assert _audio_uri(TTSInput(text="x")) == "s3://audio/_audio/p.wav"
# 158: pronunciation lexicon respells whole words only, case-insensitive, spoken text only. # 158: pronunciation lexicon respells whole words only, case-insensitive, spoken text only.
import tempfile, json as _json import tempfile, json as _json
globals()["LEXICON_PATH"] = os.path.join(tempfile.mkdtemp(), "lex.json") globals()["LEXICON_PATH"] = os.path.join(tempfile.mkdtemp(), "lex.json")
@@ -223,7 +241,7 @@ if __name__ == "__main__":
# loudnorm: with ffmpeg present the wav is normalized in place and stays readable at its sr; # loudnorm: with ffmpeg present the wav is normalized in place and stays readable at its sr;
# without ffmpeg it's a safe no-op returning the same path (audio never lost). # without ffmpeg it's a safe no-op returning the same path (audio never lost).
ln = _write_wav(samples, 16000) ln = _write_wav(samples, 16000)
have_ffmpeg = subprocess.run(["ffmpeg", "-version"], capture_output=True).returncode == 0 \ have_ffmpeg = subprocess.run(["ffmpeg", "-version"], capture_output=True, check=False).returncode == 0 \
if __import__("shutil").which("ffmpeg") else False if __import__("shutil").which("ffmpeg") else False
assert _loudnorm(ln) == ln and os.path.exists(ln) assert _loudnorm(ln) == ln and os.path.exists(ln)
assert wave.open(ln, "rb").getframerate() == 16000 assert wave.open(ln, "rb").getframerate() == 16000
worker_vision.py
+312 -37
View File
@@ -19,7 +19,9 @@ GEMMA4_URL = os.environ.get("GEMMA4_URL", "http://127.0.0.1:8090")
# set-of-mark speaker attribution: detect text regions with comic-text-detector (ONNX, CPU) and draw # set-of-mark speaker attribution: detect text regions with comic-text-detector (ONNX, CPU) and draw
# numbered boxes on the panel so gemma transcribes/attributes grounded regions instead of eyeballing # numbered boxes on the panel so gemma transcribes/attributes grounded regions instead of eyeballing
# the whole image. Off by default — flip SOM_ATTRIBUTION=1 once tuned per title (see bubble_detect.py). # the whole image. ON by default: face->identity pairing is now gated on containment, so an unmatched
# face is labelled "unknown" instead of borrowing the nearest name (see _pair_faces_to_present).
# SOM_ATTRIBUTION=0 falls back to the holistic path (see bubble_detect.py).
try: try:
import bubble_detect import bubble_detect
except Exception: # onnxruntime/model absent -> feature simply stays unavailable except Exception: # onnxruntime/model absent -> feature simply stays unavailable
@@ -31,29 +33,45 @@ except Exception:
SOM = os.environ.get("SOM_ATTRIBUTION", "1") == "1" SOM = os.environ.get("SOM_ATTRIBUTION", "1") == "1"
def _pair_faces_to_present(det_faces: list, present: list) -> list: def _pair_faces_to_present(det_faces: list, present: list, margin: float = 0.25) -> list:
"""det_faces = real detector boxes (grounded but identity-less); present = characters gemma placed """det_faces = real detector boxes (grounded but identity-less); present = characters gemma placed
in the panel (identity + a coarse, imprecise bbox). Assign each detected face the identity of the in the panel (identity + a coarse, imprecise bbox). Pair a face with a present character ONLY when
present character whose gemma-bbox centre falls closest to it, so a green box carries a real name. the face centre falls inside that character's bbox grown by `margin` of its size — gemma's boxes are
A face with no nearby present char stays unknown; a present char is used at most once.""" imprecise but not arbitrary. An unpaired face stays unknown instead of borrowing whoever happens to
be nearest; the label it carries becomes `speaker_method="som_face"`, the highest-trust provenance
the pipeline has, so an ungated guess used to launder itself into evidence.
Pairs are taken globally shortest-first, so the first face processed cannot claim a character that
fits a later face far better. Each face and each character is used at most once."""
def cx_cy(b): return ((b[0] + b[2]) / 2, (b[1] + b[3]) / 2) def cx_cy(b): return ((b[0] + b[2]) / 2, (b[1] + b[3]) / 2)
used = set()
out = [] def contains(face_box, char_box) -> bool:
for i, f in enumerate(det_faces, 1): w, h = char_box[2] - char_box[0], char_box[3] - char_box[1]
if w <= 0 or h <= 0:
return False
fx, fy = cx_cy(face_box)
return (char_box[0] - margin * w <= fx <= char_box[2] + margin * w
and char_box[1] - margin * h <= fy <= char_box[3] + margin * h)
pairs = []
for i, f in enumerate(det_faces):
fx, fy = cx_cy(f["bbox"]) fx, fy = cx_cy(f["bbox"])
best, bestd = None, 1e18
for j, c in enumerate(present): for j, c in enumerate(present):
b = c.get("bbox") b = c.get("bbox")
if not b or j in used: if not b or len(b) != 4 or not contains(f["bbox"], b):
continue continue
px, py = cx_cy(b) px, py = cx_cy(b)
d = (px - fx) ** 2 + (py - fy) ** 2 pairs.append(((px - fx) ** 2 + (py - fy) ** 2, i, j))
if d < bestd: taken_f, taken_c, match = set(), set(), {}
best, bestd = j, d for _, i, j in sorted(pairs):
c = present[best] if best is not None else {} if i in taken_f or j in taken_c:
if best is not None: continue
used.add(best) taken_f.add(i)
out.append({"label": f"P{i}", "bbox": f["bbox"], "local_id": c.get("local_id"), taken_c.add(j)
match[i] = j
out = []
for i, f in enumerate(det_faces):
c = present[match[i]] if i in match else {}
out.append({"label": f"P{i + 1}", "bbox": f["bbox"], "local_id": c.get("local_id"),
"who": c.get("name") or c.get("desc") or "unknown", "gender": c.get("gender")}) "who": c.get("name") or c.get("desc") or "unknown", "gender": c.get("gender")})
return out return out
@@ -105,16 +123,65 @@ def _set_of_mark(local_path: str, present: list):
return marked, "\n" + "\n".join(parts) + "\n", {f["label"]: f["local_id"] or "unknown" for f in faces} return marked, "\n" + "\n".join(parts) + "\n", {f["label"]: f["local_id"] or "unknown" for f in faces}
def _apply_speaker_labels(dialogue: list, label_map: dict) -> list: _GENDER_SUFFIX = re.compile(r"\s*\((?:m|f|male|female|man|woman|unknown)\)\s*$", re.IGNORECASE)
"""map a set-of-mark face label ("P1") in the speaker field back to its local_id. gemma may also _ID_SHAPED = re.compile(r"^(?:person|char|face|p)[\s_-]*\d+$", re.IGNORECASE)
answer with the local_id directly (legend shows both) — that already matches, so it's left as-is."""
if not label_map:
return dialogue def _present_keys(present: list) -> dict:
"""Every string the dialogue prompt shows for a present character, mapped to its local_id.
`build_dialogue_prompt` renders each one as `- person_1: brown ponytail, green dress (f)`, and gemma
answers with any part of that line, most often the description. Those answers used to fall through
`normalize_speaker` as free-form NAMES and never matched the registry, losing 28 of 51 speech lines
on job 778297bc. A key shared by two present characters is dropped: it cannot identify either."""
keys: dict = {}
for c in present:
lid = (c.get("local_id") or "").strip()
if not lid:
continue
name, desc, gender = c.get("name") or "", c.get("desc") or "", c.get("gender") or ""
shown = name or desc or "unknown"
forms = {lid, name, desc, shown}
if gender and gender != "unknown":
forms |= {f"{f} ({gender})" for f in (name, desc, shown) if f}
for f in forms:
k = f.strip().casefold()
if not k or k == "unknown": # the prompt prints "unknown" for a nameless character
continue
keys[k] = lid if keys.get(k, lid) == lid else None # ambiguous key -> unusable
return {k: v for k, v in keys.items() if v}
def _apply_speaker_labels(dialogue: list, label_map: dict, present: list | None = None) -> list:
"""Map gemma's speaker answer back to a panel-local id.
Three answer shapes reach here: a set-of-mark face label ("P1"), a local_id, and any string the
prompt showed for a present character. An id-shaped answer that names nobody present is junk and
becomes "unknown" rather than a name claim (invariant 6). A trailing gender marker is stripped, so
"Seonho (m)" can still match the registry name "Seonho" off-panel."""
keys = _present_keys(present or [])
for d in dialogue: for d in dialogue:
s = (d.get("speaker") or "").strip() s = (d.get("speaker") or "").strip()
if not s:
continue
if s in label_map: if s in label_map:
d["speaker"] = label_map[s] lid = label_map[s]
d["speaker_method"] = "som_face" # _set_of_mark labels a detected face `unknown` when gated pairing matched it to no present
# character. An answer pointing at such a mark grounds nothing, so it must not carry
# `som_face`, the highest-trust label. 7 of 7 som_face lines on the first 36 panels of the
# 2026-08-11 rerun were this case.
d["speaker"] = lid
if lid != "unknown":
d["speaker_method"] = "som_face"
continue
bare = _GENDER_SUFFIX.sub("", s).strip()
lid = keys.get(s.casefold()) or keys.get(bare.casefold())
if lid:
d["speaker"] = lid
elif _ID_SHAPED.match(bare):
d["speaker"] = "unknown" # an id for nobody present: never mint a name from it
elif bare != s:
d["speaker"] = bare # off-panel name, gender marker stripped
return dialogue return dialogue
@@ -127,10 +194,17 @@ def _strip_thought(text: str) -> str:
def _extract_json(raw: str) -> dict: def _extract_json(raw: str) -> dict:
"""strip gemma4 thought, pull the first JSON object, parse it.""" """strip gemma4 thought, pull the first JSON object, parse it."""
text = _strip_thought(raw) text = _strip_thought(raw)
m = re.search(r"\{.*\}", text, re.DOTALL) i = text.find("{")
if not m: if i < 0:
raise ValueError(f"no json in response: {text[:200]}") raise ValueError(f"no json in response: {text[:200]}")
return json.loads(m.group(0)) # raw_decode stops at the end of the FIRST object. The old greedy `\{.*\}` ran to the LAST brace in
# the reply, so a second object or any trailing braced prose produced an unparseable span and burned
# a repair call on a response that was already fine.
try:
obj, _ = json.JSONDecoder().raw_decode(text[i:])
except json.JSONDecodeError as e:
raise ValueError(f"bad json in response: {text[:200]}") from e
return obj
def _img_part(image_path: str) -> dict: def _img_part(image_path: str) -> dict:
@@ -226,6 +300,73 @@ class VisionInput(BaseModel):
session_id: str = "" session_id: str = ""
BBOX_GRID = 1000 # gemma's native normalized box grid
def _bbox_to_pixels(chars: list, w: int, h: int) -> list:
"""Rewrite every character box from gemma's 0-1000 grid to pixels on this panel.
The prompt asks for pixels. The model answers on its own normalized grid regardless. Measured over
the 113 detections of job 778297bc: 47 boxes had x2 beyond the 900px panel width, and not one had y2
beyond 1000 on panels 1257 to 2307px tall. Consumed as pixels the boxes collapse into the top-left
corner of the panel, which is how identity came to embed crops of speech balloons and window frames
and match them at 0.9, and why gated face pairing returned 7 unknowns out of 7 real faces.
Convert once here so every consumer sees pixels: `_crop_bbox` in identity, the face pairing below,
the set-of-mark boxes, and the review UI's client-side crop.
ponytail: the prompt still says "pixel bounding box". Rewording it would change what the model
emits and needs a GPU run to re-verify, so the boundary converts instead. If a future model really
does answer in pixels, this scales them down -- check the box range before swapping models.
"""
for c in chars:
b = c.get("bbox")
if not (isinstance(b, list) and len(b) == 4 and all(isinstance(v, (int, float)) for v in b)):
continue
xs = sorted((min(w, max(0, round(b[0] * w / BBOX_GRID))),
min(w, max(0, round(b[2] * w / BBOX_GRID)))))
ys = sorted((min(h, max(0, round(b[1] * h / BBOX_GRID))),
min(h, max(0, round(b[3] * h / BBOX_GRID)))))
# ponytail: the model swapped corners on 1 of 117 boxes (p007 person_1, x1 > x2).
# Ordering here is enough because a zero-area box still crops to nothing downstream.
c["bbox"] = [xs[0], ys[0], xs[1], ys[1]]
return chars
def _mark_has_face(img, chars: list) -> list:
"""Stamp `has_face` on every character a real detected face lands inside.
gemma calls every person-shaped region a character. On panel 7 of job 778297bc that was two
people and four background extras, one of which is an empty window mullion. All six reached
identity as equal candidates, and after the bbox fix an extra took `Seonho` at confidence 1.00,
which puts an extra's crop into the lead's reference set. A faceless crop is a bad embedding as
well as a bad reference: the back of a head or a patch of coat matches almost anything.
Reuses `_pair_faces_to_present`, so containment, its margin, and the global shortest-first
assignment are exactly the rules the speaker path already uses. Requires pixel boxes, so call it
after `_bbox_to_pixels`.
Fails open. A missing or broken detector marks everything `True`, because dropping every
character is worse than the over-detection this gate exists to trim.
"""
def _all(v):
for c in chars:
c["has_face"] = v
return chars
if face_detect is None or not chars:
return _all(True)
try:
faces = face_detect.detect_faces(img)
except Exception as e:
print(f"[vision/detect] face detect failed, has_face gate off for this panel: {e}", flush=True)
return _all(True)
paired = {p["local_id"] for p in _pair_faces_to_present(faces, chars) if p.get("local_id")}
for c in chars:
c["has_face"] = c.get("local_id") in paired
return chars
@app.post("/vision") @app.post("/vision")
async def vision(data: VisionInput): async def vision(data: VisionInput):
local = transport.get(data.panel_uri, f"{SHM}/vision_{uuid.uuid4().hex[:8]}.png") local = transport.get(data.panel_uri, f"{SHM}/vision_{uuid.uuid4().hex[:8]}.png")
@@ -241,8 +382,17 @@ async def vision(data: VisionInput):
print(f"[vision/detect] parse failed for {data.panel_id} after repair retry: {e}", flush=True) print(f"[vision/detect] parse failed for {data.panel_id} after repair retry: {e}", flush=True)
result = {"skip": False, "parse_failed": True, "characters": [], "scene": {}} result = {"skip": False, "parse_failed": True, "characters": [], "scene": {}}
finally: finally:
import cv2
img = cv2.imread(local) # read once: the size and the face gate both need it
os.remove(local) os.remove(local)
pw, ph = (img.shape[1], img.shape[0]) if img is not None else (0, 0)
result.setdefault("characters", []) result.setdefault("characters", [])
if pw and ph:
_bbox_to_pixels(result["characters"], pw, ph)
_mark_has_face(img, result["characters"])
else:
print(f"[vision/detect] panel size unreadable for {data.panel_id}, boxes left normalized",
flush=True)
result["panel_id"] = data.panel_id result["panel_id"] = data.panel_id
return result return result
@@ -349,16 +499,31 @@ def resolve_speakers(dialogue: list, present: list) -> list:
def _annotate_speaker_methods(dialogue: list, present: list) -> list: def _annotate_speaker_methods(dialogue: list, present: list) -> list:
"""Fill provenance for model-attributed lines without overwriting grounded/backstop methods.""" """Fill provenance for model-attributed lines without overwriting grounded/backstop methods.
A speaker matching a present local_id used to be stamped `tail`, the highest-trust label, at
gemma's own confidence of 1.0. No balloon geometry was ever read. On the 2026-08-11 chapter every
sampled two-character panel had the speakers swapped
(`caveats/speaker-attribution.md#tail-is-not-geometry`), so with 2+ present the guess is dropped
rather than shipped as truth. With one present it is the same claim as the solo backstop, so it is
kept and named for what it is.
ponytail: drop-on-crowd is the honest floor, not the fix. Bind by tail geometry when the balloon
detector lands, then this branch reads a tail for real."""
local_ids = {c.get("local_id") for c in present if c.get("local_id")} local_ids = {c.get("local_id") for c in present if c.get("local_id")}
crowded = len(present) > 1
for d in dialogue: for d in dialogue:
if d.get("speaker_method"): if d.get("speaker_method"):
continue continue
speaker = (d.get("speaker") or "").strip() speaker = (d.get("speaker") or "").strip()
if d.get("type", "speech") not in _SPEECH or not speaker or speaker == "unknown": if d.get("type", "speech") not in _SPEECH or not speaker or speaker == "unknown":
d["speaker_method"] = "unknown" d["speaker_method"] = "unknown"
elif speaker in local_ids and crowded:
d["speaker"] = "unknown"
d["confidence"] = 0.0
d["speaker_method"] = "unknown"
elif speaker in local_ids: elif speaker in local_ids:
d["speaker_method"] = "tail" d["speaker_method"] = "model_solo"
d["confidence"] = min(float(d.get("confidence") or 0.7), 0.7)
else: else:
d["speaker_method"] = "turn_taking" d["speaker_method"] = "turn_taking"
return dialogue return dialogue
@@ -425,7 +590,7 @@ async def dialogue(data: DialogueInput):
result.setdefault("entities", []) result.setdefault("entities", [])
result.setdefault("named", []) # [{local_id, name}] explicit namings -> bound to character_id by orchestrator result.setdefault("named", []) # [{local_id, name}] explicit namings -> bound to character_id by orchestrator
result["named"] = _normalize_claims(result["named"], data.panel_id) result["named"] = _normalize_claims(result["named"], data.panel_id)
_apply_speaker_labels(result["dialogue"], label_map) _apply_speaker_labels(result["dialogue"], label_map, data.present_characters)
resolve_speakers(result["dialogue"], data.present_characters) resolve_speakers(result["dialogue"], data.present_characters)
result["panel_id"] = data.panel_id result["panel_id"] = data.panel_id
result.update(_dialogue_envelope([data.panel_id], [] if parse_failed else [result], parse_failed)) result.update(_dialogue_envelope([data.panel_id], [] if parse_failed else [result], parse_failed))
@@ -540,7 +705,7 @@ async def dialogue_window(data: DialogueWindowInput):
d = by_id.get(pid) d = by_id.get(pid)
if d is None: # missing is unresolved, never manufactured as a silent success if d is None: # missing is unresolved, never manufactured as a silent success
continue continue
_apply_speaker_labels(d.get("dialogue", []), label_maps.get(pid, {})) _apply_speaker_labels(d.get("dialogue", []), label_maps.get(pid, {}), present_by_id.get(pid, []))
out.append({ out.append({
"panel_id": pid, "panel_id": pid,
"dialogue": resolve_speakers(d.get("dialogue", []), present_by_id.get(pid, [])), "dialogue": resolve_speakers(d.get("dialogue", []), present_by_id.get(pid, [])),
@@ -856,9 +1021,16 @@ def build_resolve_prompt(candidates: list) -> str:
) )
REF_BUDGET = 12 # total reference images one /vision/resolve prompt may carry, beside the query crop
def _refs_per_candidate(n: int) -> int:
return max(1, min(3, REF_BUDGET // max(1, n)))
class ResolveInput(BaseModel): class ResolveInput(BaseModel):
crop_uri: str crop_uri: str
candidates: list = [] # gender-gated cosine shortlist [{character_id, name, gender, ...}] candidates: list = [] # gender-gated gallery [{character_id, name, gender, ...}]
session_id: str = "" session_id: str = ""
@@ -877,9 +1049,13 @@ async def vision_resolve(data: ResolveInput):
return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"} return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"}
crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png") crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png")
refs = [] refs = []
# image budget, not a per-candidate rule. The gallery is the live cast now, not a cosine top-k, so
# 3 references each was 27 images plus the query on a 9-character cast. Spread REF_BUDGET across the
# candidates instead: 3 references while the cast is small, 1 apiece once it is not.
per = _refs_per_candidate(len(data.candidates))
try: try:
for i, candidate in enumerate(data.candidates, 1): for i, candidate in enumerate(data.candidates, 1):
for uri in (candidate.get("reference_image_uris") or [])[:3]: for uri in (candidate.get("reference_image_uris") or [])[:per]:
try: try:
refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png"))) refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png")))
except Exception as e: except Exception as e:
@@ -893,10 +1069,18 @@ async def vision_resolve(data: ResolveInput):
os.remove(crop) os.remove(crop)
for _, path in refs: for _, path in refs:
os.remove(path) os.remove(path)
# map gemma's 1-based choice back to a character_id; 0 / out-of-range -> NONE (new character). # map gemma's 1-based choice back to a character_id. An explicit 0 means NONE -> a new character.
# An OUT-OF-RANGE index is a hallucination, not an answer: it must be `unresolved` like a parse
# failure, or a bad index silently mints a brand new character in the permanent registry.
choice = result.get("choice", 0) choice = result.get("choice", 0)
cid = data.candidates[choice - 1]["character_id"] if isinstance(choice, int) and 1 <= choice <= len(data.candidates) else None in_range = isinstance(choice, int) and 1 <= choice <= len(data.candidates)
state = "known" if cid else ("unresolved" if result.get("reason") == "parse_failed" else "new") cid = data.candidates[choice - 1]["character_id"] if in_range else None
if in_range:
state = "known"
elif choice == 0 and result.get("reason") != "parse_failed":
state = "new"
else:
state = "unresolved"
return {"character_id": cid, "state": state, "confidence": float(result.get("confidence", 0.0)), return {"character_id": cid, "state": state, "confidence": float(result.get("confidence", 0.0)),
"reason": result.get("reason", "")} "reason": result.get("reason", "")}
@@ -943,11 +1127,34 @@ if __name__ == "__main__":
two = [{"local_id": "person_1"}, {"local_id": "person_2"}] two = [{"local_id": "person_1"}, {"local_id": "person_2"}]
assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], two)[0]["speaker"] == "unknown" assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], two)[0]["speaker"] == "unknown"
assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], [])[0]["speaker"] == "unknown" assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], [])[0]["speaker"] == "unknown"
assert resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x"}], two)[0]["speaker_method"] == "tail" # a present-local_id match is gemma's guess, not a read tail: dropped when 2+ are present,
# kept but capped at 0.7 when only one is (caveats/speaker-attribution.md#tail-is-not-geometry)
crowd = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], two)[0]
assert crowd["speaker"] == "unknown" and crowd["speaker_method"] == "unknown" and crowd["confidence"] == 0.0
lone = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], solo)[0]
assert lone["speaker"] == "person_1" and lone["speaker_method"] == "model_solo" and lone["confidence"] == 0.7
# set-of-mark: gemma answers a face label -> remapped to local_id; a name/unknown passes through # set-of-mark: gemma answers a face label -> remapped to local_id; a name/unknown passes through
lbl = _apply_speaker_labels([{"speaker": "P1"}, {"speaker": "Aria"}, {"speaker": "unknown"}], lbl = _apply_speaker_labels([{"speaker": "P1"}, {"speaker": "Aria"}, {"speaker": "unknown"}],
{"P1": "person_3"}) {"P1": "person_3"})
assert lbl[0]["speaker"] == "person_3" and lbl[1]["speaker"] == "Aria" and lbl[2]["speaker"] == "unknown" assert lbl[0]["speaker"] == "person_3" and lbl[1]["speaker"] == "Aria" and lbl[2]["speaker"] == "unknown"
assert lbl[0]["speaker_method"] == "som_face"
# a mark whose face paired to nobody present grounds nothing, so it gets no som_face label
unpaired = _apply_speaker_labels([{"speaker": "P2"}], {"P2": "unknown"})[0]
assert unpaired["speaker"] == "unknown" and "speaker_method" not in unpaired, unpaired
# gemma answers with what the prompt SHOWED, not the id: description, name+gender, bare id, junk id.
# Each shape cost real lines on job 778297bc by falling through as a free-form name.
shown = [{"local_id": "person_1", "desc": "brown ponytail, green dress", "gender": "f"},
{"local_id": "person_2", "name": "Seonho", "gender": "m"}]
got = _apply_speaker_labels([{"speaker": "brown ponytail, green dress (f)"}, {"speaker": "Seonho"},
{"speaker": "person_2"}, {"speaker": "person_9"},
{"speaker": "Haeseon (f)"}, {"speaker": "unknown"}], {}, shown)
assert [d["speaker"] for d in got] == ["person_1", "person_2", "person_2", "unknown",
"Haeseon", "unknown"], got
# a description shared by two present characters identifies neither
twins = [{"local_id": "person_1", "desc": "schoolgirl"}, {"local_id": "person_2", "desc": "schoolgirl"}]
assert _apply_speaker_labels([{"speaker": "schoolgirl"}], {}, twins)[0]["speaker"] == "schoolgirl"
# a nameless present character is shown as "unknown"; that must not become an id
assert _apply_speaker_labels([{"speaker": "unknown"}], {}, [{"local_id": "person_1"}])[0]["speaker"] == "unknown"
# real detector face -> nearest present char's identity; each present char claimed once; leftover unknown # real detector face -> nearest present char's identity; each present char claimed once; leftover unknown
pf = _pair_faces_to_present( pf = _pair_faces_to_present(
[{"bbox": [10, 10, 50, 50]}, {"bbox": [200, 10, 240, 50]}, {"bbox": [400, 10, 440, 50]}], [{"bbox": [10, 10, 50, 50]}, {"bbox": [200, 10, 240, 50]}, {"bbox": [400, 10, 440, 50]}],
@@ -983,6 +1190,13 @@ if __name__ == "__main__":
_map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None) _map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None)
assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None
assert "reference images" in rp and "face shape/features first" in rp assert "reference images" in rp and "face shape/features first" in rp
# reference budget: a small gallery keeps 3 refs each, a cast-sized one drops to 1 and stays under
# REF_BUDGET images however big the cast gets.
assert _refs_per_candidate(1) == 3 and _refs_per_candidate(4) == 3
assert _refs_per_candidate(9) == 1 and _refs_per_candidate(0) == 3
# the budget holds up to REF_BUDGET candidates; past that the floor of 1 apiece wins, which is why
# the orchestrator caps the gallery (GALLERY_CAP) rather than relying on this alone.
assert all(n * _refs_per_candidate(n) <= REF_BUDGET for n in range(1, REF_BUDGET + 1))
# dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty. # dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty.
bad = _dialogue_envelope(["p1"], [], parse_failed=True) bad = _dialogue_envelope(["p1"], [], parse_failed=True)
@@ -1005,4 +1219,65 @@ if __name__ == "__main__":
assert False assert False
except ValueError: except ValueError:
pass pass
# trailing braced prose after a complete object parses (used to burn a repair call)
assert _extract_json('{"skip":false}\nnote {see above}')["skip"] is False
# gemma's boxes arrive on a 0-1000 grid and leave /vision as pixels. The 900x1650 panel below is
# panel 7 of job 778297bc: person_5 is Seonho in the foreground, and read as pixels his box lands in
# the top sixth of the panel, inside a speech balloon, which is what identity embedded.
_ch = [{"local_id": "person_5", "bbox": [222, 405, 654, 1000]},
{"local_id": "edge", "bbox": [0, 0, 1000, 1000]},
{"local_id": "junk", "bbox": "nope"},
# p007 person_1 of run B: x1 > x2, so the crop came out empty and the detection was lost
{"local_id": "swapped", "bbox": [226, 417, 130, 551]}]
_bbox_to_pixels(_ch, 900, 1650)
assert _ch[3]["bbox"] == [117, 688, 203, 909], _ch[3]["bbox"] # corners ordered, non-empty crop
assert _ch[0]["bbox"] == [200, 668, 589, 1650], _ch[0]["bbox"]
assert _ch[1]["bbox"] == [0, 0, 900, 1650], _ch[1]["bbox"] # a clamped box spans the whole panel
assert _ch[2]["bbox"] == "nope", _ch[2]["bbox"] # unparseable is left alone, not crashed
# the box must now cover the lower half of a tall panel, which the raw grid value never can
assert _ch[0]["bbox"][3] > 1000 > _ch[0]["bbox"][1]
# the has_face gate, on panel 7's real converted boxes. Seonho and the colleague carry a detected
# face; the two background extras do not, and identity must skip them rather than embed scenery.
_p7 = [{"local_id": "person_1", "bbox": [457, 657, 642, 937]}, # Seonho, foreground
{"local_id": "person_2", "bbox": [669, 591, 763, 822]}, # the colleague
{"local_id": "person_5", "bbox": [31, 554, 94, 728]}, # extra that took Seonho at 1.00
{"local_id": "person_6", "bbox": [34, 414, 122, 564]}] # extra
_faces = [{"bbox": [520, 690, 590, 770]}, {"bbox": [695, 610, 740, 660]}]
class _FD:
@staticmethod
def detect_faces(_img):
return _faces
_real_fd, globals()["face_detect"] = face_detect, _FD
try:
_mark_has_face(None, _p7)
assert [c["has_face"] for c in _p7] == [True, True, False, False], _p7
# fail open: a detector that raises must not silently drop every character in the panel
class _Boom:
@staticmethod
def detect_faces(_img):
raise RuntimeError("onnx died")
globals()["face_detect"] = _Boom
_mark_has_face(None, _p7)
assert all(c["has_face"] for c in _p7), _p7
globals()["face_detect"] = None
_mark_has_face(None, _p7)
assert all(c["has_face"] for c in _p7), _p7
finally:
globals()["face_detect"] = _real_fd
# face->identity pairing is GATED on containment: a face outside every gemma bbox stays unknown.
faces = [{"bbox": [10, 10, 30, 30]}, {"bbox": [900, 900, 920, 920]}]
present = [{"local_id": "person_1", "name": "Teto", "bbox": [0, 0, 100, 200]}]
paired = _pair_faces_to_present(faces, present)
assert paired[0]["local_id"] == "person_1" and paired[0]["who"] == "Teto"
assert paired[1]["local_id"] is None and paired[1]["who"] == "unknown", paired[1]
# globally shortest-first: the first face must not claim a character that fits the second better.
faces2 = [{"bbox": [95, 95, 105, 105]}, {"bbox": [8, 8, 12, 12]}]
present2 = [{"local_id": "a", "bbox": [0, 0, 20, 20]}, {"local_id": "b", "bbox": [80, 80, 120, 120]}]
p2 = _pair_faces_to_present(faces2, present2)
assert [f["local_id"] for f in p2] == ["b", "a"], p2
# no present characters at all -> every face unknown, never a phantom identity
assert _pair_faces_to_present(faces, [])[0]["local_id"] is None
print("worker_vision self-check ok") print("worker_vision self-check ok")