Merge pull request 'Restore the runtime and bring master up to the seventh session' (#1) from restore-runtime into master
This commit was merged in pull request #1.
This commit is contained in:
@@ -4,3 +4,4 @@ __pycache__/
|
||||
dots.tts/
|
||||
/dev/shm/
|
||||
*.gguf
|
||||
models/
|
||||
|
||||
@@ -1,5 +1,26 @@
|
||||
# Repository guidance
|
||||
|
||||
## Restarting one worker
|
||||
|
||||
Python binds a module at process start, so a worker keeps serving the code it loaded. Run
|
||||
`./check_stale.sh` before any stage run meant to prove a change. It exits non-zero and names the tmux
|
||||
window for every worker older than its file (`decisions/identity-bbox.md#stale-worker-invalidates`).
|
||||
|
||||
**`tmux respawn-window -k` does not re-run the window's command.** It leaves a bare shell and the worker
|
||||
stays down, silently. Restart by re-sending the `send-keys` line from `start_workers.sh` for that one
|
||||
worker, then confirm on `/health` and with `check_stale.sh`:
|
||||
|
||||
```bash
|
||||
M="export MIOPEN_USER_DB_PATH=$HOME/.config/miopen MIOPEN_SYSTEM_DB_PATH=$HOME/.config/miopen MIOPEN_FIND_MODE=2 && unset MIOPEN_FIND_ENFORCE"
|
||||
tmux send-keys -t manga-workers:vision C-c
|
||||
tmux send-keys -t manga-workers:vision "$M && source $PWD/.venv/bin/activate && python -m uvicorn worker_vision:app --app-dir $PWD --host 0.0.0.0 --port 8002" C-m
|
||||
curl -s http://127.0.0.1:8002/health && ./check_stale.sh
|
||||
```
|
||||
|
||||
The orchestrator's equivalent is `docker compose up -d --build orchestrator` on homesrv, because the image
|
||||
bakes its source. That recreates the container, which drops any `docker cp`'d file such as
|
||||
`audit_registry.py`.
|
||||
|
||||
## Video analysis
|
||||
|
||||
When diagnosing render motion, transitions, timing, or visual artifacts, use
|
||||
|
||||
+353
@@ -0,0 +1,353 @@
|
||||
# ARCHITECTURE
|
||||
|
||||
The target shape of the pipeline. Written 2026-08-13 from the user's design, rewritten the same day under
|
||||
the no-Magi constraint. This is **not** what the code does. `NEXT.md` holds the live state and `AUDIT.md`
|
||||
holds the current pipeline. Every section ends with what exists today and what would make it done. The
|
||||
gap stays legible and testable without reading both.
|
||||
|
||||
The governing principle:
|
||||
|
||||
> Do not make the next panel understand the previous panel. Make it understand the current world state
|
||||
> produced by all previous panels.
|
||||
|
||||
Vision produces observations. A persistent chapter graph owns identity and relationships. Everything below
|
||||
follows from that split.
|
||||
|
||||
## Ruled out
|
||||
|
||||
Read this before proposing any of it again.
|
||||
|
||||
| ruled out | why | who decided |
|
||||
| --- | --- | --- |
|
||||
| Magi, Magiv2, Magiv3 as a detector or as pair heads | project constraint, not a measurement | user, 2026-08-13 |
|
||||
| A trained `t2c` pair model, or any learned head | no labelled pages exist and no labeller is planned | follows from the above |
|
||||
| Crop-to-crop cosine as a link signal | measured: two men reach 0.93, one man reaches 0.96, no threshold exists (`caveats/audit-open.md#cosine-not-identity`) | 2026-08-12 |
|
||||
|
||||
The consequence runs through the whole document. **Every structure below comes out of a gemma prompt
|
||||
field, or out of plain Python over gemma's output.** Nothing below is trained. A learned score
|
||||
in the original design becomes a hand-weighted sum. The weights are read off the labelled chapter, and
|
||||
the DoD is the accuracy number rather than the mechanism.
|
||||
|
||||
## The measurement spine
|
||||
|
||||
Nothing below can be called done without this, and it does not exist yet.
|
||||
|
||||
The only ground truth in the project is the eyeball pass over the 19:44 run of 2026-08-12. It lives in
|
||||
prose in `NEXT.md`. Write it to `eval/chapter-truth.json` against chapter
|
||||
`7c944dd4-e972-42c7-ba60-9f6939548e80`, scoped to what was already checked by eye rather than to all 119
|
||||
detections:
|
||||
|
||||
- the three characters walked crop by crop, each assignment marked as the real person or not
|
||||
- 30 dialogue lines with their true speaker, typed as `visible | offscreen | narrator | unknown`
|
||||
|
||||
The baseline it records, from that run:
|
||||
|
||||
| character | assignments | correct | purity | note |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| the lead | 16 | 14 | 0.88 | plus a photograph at `order 17` and a chibi at `order 20` |
|
||||
| `character_2b1b12a1` | 13 | 13 | 1.00 | a main character the registry never named |
|
||||
| `character_f0d4e901` | 9 | 7 | 0.78 | the other 2 are `2b1b12a1` |
|
||||
|
||||
Woman A is `2b1b12a1`. She has 15 occurrences split across 2 ids, so her fragmentation is 2.
|
||||
|
||||
`audit_registry.py` already walks panels, reads `identity_assignments` and counts per character. Extend
|
||||
it to print purity and fragmentation against the truth file. Do not write an eval harness.
|
||||
|
||||
**Done when:** `audit_registry.py <chapter>` prints purity per labelled character and fragmentation per
|
||||
labelled person, and reproduces the table above on the 19:44 run.
|
||||
|
||||
## 1. The page is a region graph, not a list of panels
|
||||
|
||||
```
|
||||
page
|
||||
├─ regions
|
||||
│ ├─ panel
|
||||
│ ├─ inset_panel
|
||||
│ ├─ embedded_art
|
||||
│ ├─ text
|
||||
│ ├─ tail
|
||||
│ └─ character_occurrence
|
||||
│
|
||||
└─ edges
|
||||
├─ contains(region, region)
|
||||
├─ reads_before(text, text)
|
||||
├─ tail_of(tail, text)
|
||||
├─ points_to(tail, character)
|
||||
├─ spoken_by(text, character)
|
||||
└─ same_identity(character, character)
|
||||
```
|
||||
|
||||
A flat set of panels cannot express a television inside a room. That is the defect the current pipeline
|
||||
shows most often.
|
||||
|
||||
With no detector to train, the graph comes out of the detection prompt. Gemma already returns a box per
|
||||
character and per text. Two fields per detection buy most of the graph with no new model:
|
||||
|
||||
```
|
||||
plane = story | screen | photo | poster | drawing | flashback | dream
|
||||
species = human | animal | object
|
||||
```
|
||||
|
||||
`plane` is the containment edge in disguise. A detection whose `plane` is not `story` sits inside embedded
|
||||
art, and that is the fact every stage below needs. `species` is a separate axis and exists because vision
|
||||
boxes cats as people and dresses them.
|
||||
|
||||
Tail regions stay unbuilt. The `det`/`seg` heads exist and are unused
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`), and section 3 says why they are not the first
|
||||
thing to spend on.
|
||||
|
||||
**Today:** the crop stage emits a flat panel list with a bbox each, plus `context_fragments`, a
|
||||
non-destructive caption-to-face link. Vision emits per-panel characters and dialogue. There is no
|
||||
containment edge, no region type, no `plane` and no `species`.
|
||||
|
||||
**Done when:** every detection carries `plane` and `species`. On the labelled chapter, `order 17` and
|
||||
`order 20` of the lead are not `story`, none of his 14 correct crops is demoted, and `p081` and `p108` are
|
||||
`animal`. Measured by `audit_registry.py`, which already reads the vision blob per panel.
|
||||
|
||||
## 2. Identity exists independently of names
|
||||
|
||||
```
|
||||
occurrence c42
|
||||
-> identity char_07
|
||||
name = null
|
||||
aliases = []
|
||||
```
|
||||
|
||||
`char_07.name` may be filled later, or stay null forever and display as `unknown character #7`. The
|
||||
occurrence is the observation, the identity is the cluster, the name is an optional label on the cluster.
|
||||
Three levels, never collapsed into one.
|
||||
|
||||
**Today:** already true. `identity_assignments` is the occurrence, `characters` owns the identity, `name`
|
||||
is nullable and downstream already falls back to an anonymous display.
|
||||
|
||||
**Done when:** already done. No work item. The clustering is section 4 and the naming is
|
||||
`decisions/identity-naming.md`.
|
||||
|
||||
## 3. Speaker attribution is a scored edge, not a procedure
|
||||
|
||||
Do not write `find bubble -> find tail -> nearest character`. Score every plausible edge:
|
||||
|
||||
```
|
||||
score(text, character) =
|
||||
w1 * gemma_answer
|
||||
+ w2 * spatial_evidence
|
||||
+ w3 * same_panel
|
||||
+ w4 * same_plane
|
||||
+ w5 * conversation_continuity
|
||||
+ w6 * character_activity_prior
|
||||
```
|
||||
|
||||
The original design put a learned `t2c` head in the first term and called it load-bearing. No labelled
|
||||
pages exist, so that term does not. **Gemma's answer becomes one term of six rather than the whole
|
||||
procedure.** The geometry terms overrule it when they agree against it. The weights are constants read off
|
||||
the 30 labelled lines. Six numbers in a module, not a training run.
|
||||
|
||||
Geometry carries normalized relative position, distance, overlap, same-panel and containment depth. Tail
|
||||
direction is absent until a tail region exists, and it is not the first thing to build. `conversation
|
||||
continuity` is free, and turn-taking is the strongest prior for a tail-less bubble.
|
||||
|
||||
Then the cases fall out of one mechanism instead of four:
|
||||
|
||||
| case | what carries it |
|
||||
| --- | --- |
|
||||
| bubble with a tail | gemma plus spatial, usually decisive |
|
||||
| bubble with no tail | conversation continuity plus spatial |
|
||||
| speaker outside the panel | recent identities plus an offscreen candidate |
|
||||
| narration | the narrator candidate |
|
||||
| nothing resolves | unknown speaker |
|
||||
|
||||
**A dialogue line must not be required to resolve to a visible character.** That is a failure mode, not a
|
||||
safeguard. The speaker type is a union:
|
||||
|
||||
```
|
||||
speaker = visible(character_id) | offscreen(character_id?) | narrator | unknown
|
||||
```
|
||||
|
||||
**Today:** `speaker_ref` is already a typed union of `character_id | name | unknown | narrator`
|
||||
(`decisions/audit-phase1.md#speaker-ref-is-canonical`). `offscreen` is the missing arm. Attribution is a
|
||||
prompt to gemma over a window of panels, with no geometry term at all.
|
||||
|
||||
**Done when:** `audit_speakers.py` reports accuracy over the 30 labelled lines, split by true type, and
|
||||
the scored version beats the recorded gemma-window baseline. Two numbers must move the right way, and both
|
||||
are reported. Correct assignments go up. **Forced** errors go down, where forced means a line given a
|
||||
visible character while the truth is `offscreen`, `narrator` or `unknown`. Record the baseline before
|
||||
touching the code.
|
||||
|
||||
## 4. Character recognition is occurrence, then identity, then name
|
||||
|
||||
```
|
||||
character detection
|
||||
↓
|
||||
occurrence embeddings
|
||||
↓
|
||||
pairwise same_identity scores
|
||||
↓
|
||||
chapter-wide constrained clustering
|
||||
↓
|
||||
char_001, char_002, ...
|
||||
↓
|
||||
optional name claim
|
||||
↓
|
||||
name or unknown
|
||||
```
|
||||
|
||||
Three rules the current code gets wrong.
|
||||
|
||||
**The embedding is not the character crop alone.** The crop embedding measures scene, not person, which is
|
||||
why two men reach 0.93. Combine the face or head crop with the person crop instead of replacing one with
|
||||
the other. `face_detect` already finds the face and pairs it for `has_face`, so the face box is free. This
|
||||
is the queued experiment in `NEXT.md` item 1: re-embed the same 22 detections and recompute the matrix.
|
||||
|
||||
**Cluster chapter-wide, not page by page.** `tracklets.link_tracklets` groups within an 8-panel window.
|
||||
|
||||
**Two characters in the same panel may be one person.** Seven things break that rule. Mirrors,
|
||||
photographs, flashbacks, insets, screens, imagined scenes, repeated action drawings. Make it a weak
|
||||
cannot-link, and only between detections on the same `plane`.
|
||||
|
||||
That last rule has an ordering trap. Same-panel co-presence is currently a **hard** constraint and it is
|
||||
load-bearing precisely because cosine cannot separate people. Weakening it before the embedding improves
|
||||
will regress purity. The dependency is the embedding fix, not the `plane` field alone.
|
||||
|
||||
**Today:** the embedding is the person box only (`caveats/audit-open.md#cosine-not-identity`). Clustering
|
||||
is greedy and local. `tracklets.cannot_link` treats same-panel co-presence as hard. Naming is
|
||||
`db.add_name_claim`, corroboration over `name_claims`.
|
||||
|
||||
**Done when:** no labelled character holds more than one wrong assignment, and woman A's fragmentation is
|
||||
1. Baseline is 2 wrong, 0 wrong, 2 wrong, and fragmentation 2. The bar is stated in errors rather than in
|
||||
a purity ratio on purpose. The three characters hold 16, 13 and 9 assignments. At those counts any ratio
|
||||
above 0.94 means zero tolerated errors, and the ratio hides that.
|
||||
|
||||
The face-plus-person embedding lands first and carries its own smaller check. On the 22 measured
|
||||
detections, the highest different-person pair must fall below the lowest same-person pair.
|
||||
|
||||
## 5. The narrative plane is what stops art-in-art
|
||||
|
||||
The page is a hierarchical scene graph:
|
||||
|
||||
```
|
||||
page
|
||||
└── panel A plane=story
|
||||
├── character c1
|
||||
├── text t1
|
||||
└── television/poster plane=screen
|
||||
├── character c2
|
||||
└── text t2
|
||||
```
|
||||
|
||||
Perfect classification is not the point. The output that matters is one predicate:
|
||||
|
||||
```
|
||||
same_narrative_plane(a, b)
|
||||
```
|
||||
|
||||
It has exactly two consumers, and they are the reason the field is worth adding at all:
|
||||
|
||||
- section 3, as the `same_plane` term. A real character beside a poster does not get the poster person's
|
||||
line.
|
||||
- section 4, as the guard that makes the weak cannot-link safe.
|
||||
|
||||
**Today:** nothing models this, and it is the whole of the remaining identity error on the lead. On the
|
||||
19:44 run his 16 assignments were 14 correct plus a photograph of another man and a chibi drawing. Both
|
||||
are art inside a panel.
|
||||
|
||||
**Done when:** section 1's DoD, plus both consumers wired, plus section 4's purity DoD holds with the
|
||||
cannot-link demoted to weak. If purity regresses when the constraint is weakened, the embedding is not
|
||||
ready and the demotion reverts.
|
||||
|
||||
## 6. Narrative understanding is carried state, not a per-panel description
|
||||
|
||||
Today each stage reads its predecessor's blob for one panel or one beat. `recent`, a rolling list of the
|
||||
last few dialogue lines, is the only carried state. That is the root of the invented narration.
|
||||
|
||||
The version worth building is one record per scene, inherited forward:
|
||||
|
||||
```
|
||||
scene_31:
|
||||
location: school_rooftop
|
||||
time: evening
|
||||
participants: {char_03: present, char_07: present, char_11: offscreen}
|
||||
narrative_mode: present
|
||||
last_speaker: char_07
|
||||
addressee: char_02
|
||||
```
|
||||
|
||||
A panel inherits this unless visual evidence overrides it. That alone kills a class of errors. A character
|
||||
absent for one panel has not left. A panel with no background has not changed location. A tail-less line
|
||||
keeps the offscreen participant as a candidate. A close-up still belongs to the scene. `last_speaker` and
|
||||
`participants` are what section 3's continuity term reads.
|
||||
|
||||
A panel produces a **delta** against that record, not another standalone prose interpretation:
|
||||
|
||||
```
|
||||
panel 142:
|
||||
- character_07 enters room_03
|
||||
- character_02 is already present
|
||||
- character_07 says "..."
|
||||
```
|
||||
|
||||
Classify the transition, not just the panel:
|
||||
|
||||
```
|
||||
CONTINUE_SCENE | NEW_SCENE | LOCATION_CHANGE | TIME_SKIP | FLASHBACK_START
|
||||
FLASHBACK_END | DREAM/IMAGINATION | POV_CHANGE | EMBEDDED_SCENE
|
||||
```
|
||||
|
||||
`EMBEDDED_SCENE` is `plane != story` at scene granularity, and is what stops a television's contents
|
||||
mutating the room around it.
|
||||
|
||||
**Scene state is written by a resolver, never by the vision model.** The path is
|
||||
`observation -> resolver -> state transition`. The resolver may reject an impossible update. It is plain
|
||||
Python over gemma's per-panel delta, and it is where the constraint lives.
|
||||
|
||||
**Today:** none of it exists. Chapter boundaries are a reset. Narration asserts things no panel shows
|
||||
(`NEXT.md` item 6).
|
||||
|
||||
**Done when:** a scene record carries location, participants and `narrative_mode` across panels. A
|
||||
character absent from one panel stays a participant. On the next full run the four invented-fact
|
||||
timestamps do not recur. Those are 0:43, 2:03, 2:05 and 2:15, and they are the regression list. The
|
||||
correctness verifier passed 116/116 over them because it checks quotes and names, never invented claims.
|
||||
So the check is a re-watch of those four points, not a stage counter.
|
||||
|
||||
### Not building yet
|
||||
|
||||
Each of these was in the original design. Each is deferred with a trigger, not dropped.
|
||||
|
||||
| deferred | trigger to revisit |
|
||||
| --- | --- |
|
||||
| Facts, hypotheses and unknowns as separate records with confidences | when scene state exists and narration still asserts unshown claims |
|
||||
| The seven-check consistency checker | when a scene record exists for it to check against |
|
||||
| Unresolved references that survive and back-propagate | when a second chapter of the same manga runs |
|
||||
| Chapter checkpoints and the two-memory split | when a second chapter of the same manga runs |
|
||||
|
||||
One reason covers all four. They sit on an identity layer still wrong on 2 of the lead's 16 crops. State
|
||||
machinery over wrong identity produces confidently wrong state.
|
||||
|
||||
## 7. Build order
|
||||
|
||||
Detection, vision and character embeddings already exist. The order below is chosen so each step is
|
||||
falsifiable by the step's own DoD before the next one starts.
|
||||
|
||||
```
|
||||
0. eval/chapter-truth.json + purity and fragmentation in audit_registry.py
|
||||
1. plane + species per detection -> section 1 DoD
|
||||
2. face-plus-person embedding -> section 4 embedding check
|
||||
3. chapter-wide clustering, weak cannot-link on plane -> section 4 purity DoD
|
||||
4. scene record carried forward -> section 6 DoD
|
||||
5. scored speaker edge, offscreen arm -> section 3 DoD
|
||||
6. tail regions from the unused det/seg heads -> only if 5 misses its DoD
|
||||
```
|
||||
|
||||
Steps 1 and 2 are independent and can land together. Step 3 depends on 2, which is the ordering trap in
|
||||
section 4. Step 5 depends on 4, because the continuity term reads the scene record. Step 6 is
|
||||
conditional on purpose: build a tail detector only after the cheap terms have been measured and found
|
||||
insufficient.
|
||||
|
||||
The VLM keeps judging ambiguous edges. What changes is that it stops rediscovering every character and
|
||||
dialogue relationship from raw pixels on every panel. The chapter graph carries the answer forward.
|
||||
|
||||
## Sources
|
||||
|
||||
Three ideas come from published comics-transcription work. Detection and association as graph generation,
|
||||
the text-to-character pair head, and the character bank of exemplar images plus names. The
|
||||
formulation is kept. The models are ruled out, see **Ruled out** above. No source in this document is a
|
||||
runtime dependency.
|
||||
@@ -0,0 +1,745 @@
|
||||
# Manga recap pipeline audit
|
||||
|
||||
Date: 2026-08-11
|
||||
|
||||
Scope: workpc workers plus the homesrv orchestrator. This audit excludes FFmpeg changes and does not
|
||||
recommend Magi or its embeddings. No GPU workloads were run during the audit.
|
||||
|
||||
## Status
|
||||
|
||||
Phase 1 landed on 2026-08-11. This file is the reasoning, not the state.
|
||||
|
||||
| where the finding went | file |
|
||||
| --- | --- |
|
||||
| implemented, with evidence and a runnable check | `decisions/audit-phase1.md` |
|
||||
| deliberately not done, with a revisit trigger | `caveats/audit-open.md` |
|
||||
| still ahead, in order | `ROADMAP.md` |
|
||||
| what to do next | `NEXT.md` |
|
||||
|
||||
Every P0 below is closed. The sections that name one now point at its decision instead of repeating it.
|
||||
|
||||
## Outcome
|
||||
|
||||
The strongest performance gains are available without replacing the current models:
|
||||
|
||||
1. Keep Gemma resident across compatible stages, reducing roughly seven chapter-level Gemma loads to two.
|
||||
2. Batch SigLIP crop embeddings instead of running one forward pass per detected character.
|
||||
3. Add bounded parallelism to CPU/network work such as page uploads, framed-page cropping, prefetching,
|
||||
and scene construction.
|
||||
4. Stop repeatedly downloading and base64-encoding the same panels and character references.
|
||||
5. Generate schema-constrained JSON and batch text-only script work by scene.
|
||||
|
||||
Correctness fixes should land before throughput work because the current tracklet and naming paths can
|
||||
contaminate the permanent character registry.
|
||||
|
||||
## Highest-priority findings
|
||||
|
||||
### P0: tracklet gender gate uses the wrong enum — CLOSED, decisions/audit-phase1.md#tracklet-hard-constraints
|
||||
|
||||
`orchestrator/tracklets.py` checks `male|female`, while vision emits `m|f|unknown`. The tracklet hard
|
||||
gender gate therefore never activates on normal pipeline data.
|
||||
|
||||
### P0: tracklets lack co-presence constraints — CLOSED, decisions/audit-phase1.md#tracklet-hard-constraints
|
||||
|
||||
The linker has no same-panel exclusion and uses transitive union-find. Two similar-looking people in the
|
||||
same panel can be joined, and weak links such as A~B and B~C can merge A and C even when the endpoints are
|
||||
incompatible.
|
||||
|
||||
### P0: face-to-character pairing is unconditional — CLOSED, decisions/audit-phase1.md#gated-face-pairing
|
||||
|
||||
`worker_vision._pair_faces_to_present()` assigns every detected face to the nearest vision character as
|
||||
long as an unused character exists. Despite the docstring, there is no distance or overlap threshold.
|
||||
A distant or unrelated face can therefore receive another character's identity and speaker label.
|
||||
|
||||
### P0: roster hints can become permanent identity evidence
|
||||
|
||||
The roster is described as hints-only, but those names are fed into detection. Any name emitted on a crop
|
||||
causes `worker_identity` to persist it immediately. A coarse appearance-to-roster guess can therefore
|
||||
contaminate a permanent character gallery.
|
||||
|
||||
### P0: script repair feedback is discarded — CLOSED, decisions/audit-phase1.md#verifier-false-positives
|
||||
|
||||
The orchestrator sends `beat` and `verifier_feedback` on a failed-script retry, but `worker_script.ScriptInput`
|
||||
does not define either field and the prompt builder does not consume them. The second call is another
|
||||
stochastic attempt rather than a targeted correction.
|
||||
|
||||
### P0: action evidence is missing from the verifier — CLOSED, decisions/audit-phase1.md#verifier-false-positives
|
||||
|
||||
The beat builder reads plural `actions`, while the scene worker emits singular `action`. Script validation
|
||||
therefore receives little or no action evidence and cannot reliably detect invented or omitted actions.
|
||||
|
||||
### P0: GPU leases are unsafe for concurrent jobs — CLOSED, decisions/audit-phase1.md#no-lease-stealing
|
||||
|
||||
When `/session/open` returns 409, the homesrv proxy assumes the active lease is stale and closes it. A second
|
||||
legitimate job can terminate the first job's model. `heartbeat_session()` exists but is not used by the
|
||||
pipeline.
|
||||
|
||||
### P1: repeated model cold starts
|
||||
|
||||
A normal chapter can load Gemma separately for:
|
||||
|
||||
1. roster;
|
||||
2. vision;
|
||||
3. identity adjudication;
|
||||
4. reconcile;
|
||||
5. dialogue;
|
||||
6. direction;
|
||||
7. script.
|
||||
|
||||
The source itself notes that each load takes tens of seconds to minutes. Removing approximately five of
|
||||
these loads is likely the largest safe wall-time improvement.
|
||||
|
||||
### P1: repeated transfers and encoding
|
||||
|
||||
- `/direct/window` downloads every image once for grouping and again for shot design.
|
||||
- Identity resolver calls repeatedly download the same candidate reference images.
|
||||
- Each local image is base64-expanded into the JSON sent to the local llama-server.
|
||||
- Dialogue overlap causes repeated download and CPU set-of-mark work for overlap panels.
|
||||
|
||||
### P1: per-item GPU and database work
|
||||
|
||||
- SigLIP runs one forward pass per character crop rather than a bounded batch.
|
||||
- Only one embedding URI is loaded per known character even when its reference gallery has several views.
|
||||
- Scene construction makes one homesrv-to-workpc HTTP call per panel for a cheap JSON join.
|
||||
- Many stage loops repeatedly open SQLite connections and commit one row at a time.
|
||||
|
||||
## Performance and scheduling plan
|
||||
|
||||
### 1. Keep Gemma resident across model phases
|
||||
|
||||
Use this chapter schedule:
|
||||
|
||||
```text
|
||||
CPU: fetch -> crop
|
||||
Gemma: roster -> vision
|
||||
SigLIP: batched identity embeddings and shortlists
|
||||
Gemma: identity adjudication -> reconcile -> dialogue -> direct -> CPU scene join -> script
|
||||
Dots: TTS
|
||||
Comfy: optional layers
|
||||
```
|
||||
|
||||
This needs pipeline-owned model leases instead of every stage opening and closing its own lease. Scene
|
||||
construction should run locally on homesrv or as a batch while the second Gemma lease remains open.
|
||||
|
||||
ComfyUI must be represented as a GPU resource too. It currently bypasses the session manager, so another
|
||||
job could load Gemma, SigLIP, or Dots while ComfyUI is using the same GPU.
|
||||
|
||||
### 2. Batch SigLIP inference
|
||||
|
||||
For each request or chapter batch:
|
||||
|
||||
1. download a panel once;
|
||||
2. clamp and crop all detected characters;
|
||||
3. preprocess a bounded image batch;
|
||||
4. run one model forward pass;
|
||||
5. compare the resulting matrix with the known gallery in one vectorized operation.
|
||||
|
||||
Batch size should be tuned against VRAM rather than hard-coded optimistically.
|
||||
|
||||
### 3. Add safe bounded CPU parallelism
|
||||
|
||||
- Fetch/upload pages with concurrency 4-8.
|
||||
- Crop framed pages with concurrency 2-4, buffer results, then insert them in page order.
|
||||
- Keep whole-strip webtoon restitching sequential.
|
||||
- Pre-download the next vision/dialogue window while Gemma processes the current window.
|
||||
- Precompute face/text detections with a small CPU worker pool while avoiding ONNX thread oversubscription.
|
||||
- Batch scene construction and SQLite reads/writes.
|
||||
|
||||
GPU calls should remain serialized initially. Multiple llama-server slots divide the configured context
|
||||
among slots and need an explicit VRAM/context benchmark before enabling concurrent multimodal requests.
|
||||
|
||||
### 4. Eliminate repeated image movement
|
||||
|
||||
Add an ephemeral `/dev/shm` LRU keyed by S3 URI plus object version/etag. It is a cache, not durable worker
|
||||
state, and may be dropped at any time.
|
||||
|
||||
Reuse downloaded image paths between the two direction passes and cache candidate reference images during
|
||||
the identity phase. Configure llama-server with `--media-path /dev/shm` and send `file://` image paths
|
||||
instead of base64 data URIs. The installed llama.cpp version supports local media paths and schema-constrained
|
||||
responses.
|
||||
|
||||
Reference: [llama-server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
|
||||
|
||||
### 5. Use JSON-schema-constrained generation
|
||||
|
||||
Provide a JSON schema for detect, dialogue, roster, direction, same-person, and identity-resolution calls.
|
||||
This should eliminate most malformed-output repair calls and make missing fields explicit. Keep semantic
|
||||
validation after parsing because a structurally valid answer can still be wrong.
|
||||
|
||||
### 6. Reduce model call count
|
||||
|
||||
- Dialogue context should contain the current scene's recent 12-20 turns, not up to 400 lines from the
|
||||
whole chapter.
|
||||
- Reuse direction images first, then A/B test combining grouping and shot selection into one structured call.
|
||||
- Replace the per-window LLM direction summary with deterministic structured continuity where possible.
|
||||
- Generate 6-12 consecutive script beats per text-only call, grouped by scene, with stable beat IDs and
|
||||
targeted per-beat repair.
|
||||
|
||||
## Character-recognition plan
|
||||
|
||||
The critical path keeps the current detector, SigLIP shortlisting, tracklets, and Gemma resolver.
|
||||
|
||||
### 1. Correct the tracklet spine
|
||||
|
||||
- Canonicalize gender as `m|f|unknown` and species as a validated enum everywhere.
|
||||
- Add hard same-panel/co-presence cannot-links.
|
||||
- Treat manual identity assignments as immutable constraints.
|
||||
- Prevent transitive bridges from joining tracks whose hard constraints disagree.
|
||||
- Give every detection a chapter-local provisional track ID, even if it never becomes a global character.
|
||||
- Move provisional state out of the workpc process and into a resumable homesrv chapter artifact.
|
||||
|
||||
### 2. Improve visual evidence
|
||||
|
||||
- Validate and clamp all VLM bounding boxes before cropping.
|
||||
- Match detected faces to person/body boxes with gated IoU and normalized distance assignment rather than
|
||||
unconditional nearest-neighbour matching.
|
||||
- Store both face and body crops.
|
||||
- Score crop quality using face size, detector confidence, occlusion, and sharpness.
|
||||
- Send Gemma the best 2-4 query views for a tracklet. The current resolver sends only one representative
|
||||
query crop; its candidate union is not a real multi-view query gallery.
|
||||
- Store multiple embedding vectors per global character and use gallery maximum or centroid similarity
|
||||
only for shortlisting.
|
||||
|
||||
Face/body evidence plus local temporal constraints are supported by dedicated comic and manga re-ID work:
|
||||
[Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification](https://arxiv.org/abs/2308.09096)
|
||||
and [Unsupervised Manga Character Re-identification via Face-body and Spatial-temporal Associated Clustering](https://arxiv.org/abs/2204.04621).
|
||||
|
||||
### 3. Separate naming from identity
|
||||
|
||||
- Keep roster names as candidate vocabulary, never identity evidence.
|
||||
- Detection may emit a typed name claim only when the panel visibly binds the name to that character.
|
||||
- Require self-introduction, name-tag, or corroborated address evidence before promotion.
|
||||
- Preserve aliases and honorific forms instead of stripping and discarding them.
|
||||
- Never merge galleries on name equality alone.
|
||||
- Resolve the tracklet first, then attach corroborated names to its global identity.
|
||||
|
||||
### 4. Add global constraints
|
||||
|
||||
Identity resolution should enforce:
|
||||
|
||||
- must-link membership within a validated tracklet;
|
||||
- cannot-link for co-present characters;
|
||||
- gender and species hard gates;
|
||||
- manual locks;
|
||||
- at most one appearance of a global character per panel;
|
||||
- open-set outcomes `known|new|unresolved` without deleting the chapter-local track ID.
|
||||
|
||||
Gemma should receive a compact labelled identity board containing several query views and selected candidate
|
||||
gallery views, rather than up to fifteen separately downloaded image parts for every representative crop.
|
||||
|
||||
### 5. Improve identity evaluation
|
||||
|
||||
Extend the existing labels and evaluator to report:
|
||||
|
||||
- false merges versus false splits;
|
||||
- tracklet purity and fragmentation;
|
||||
- global identity accuracy;
|
||||
- name-binding accuracy;
|
||||
- unknown/NONE rate;
|
||||
- speaker accuracy conditioned on identity correctness.
|
||||
|
||||
A replacement embedding backbone should only be considered if the structural fixes plateau, using the
|
||||
same labeled chapters for a controlled A/B comparison.
|
||||
|
||||
## Story-coherence plan
|
||||
|
||||
### 1. Build an evidence ledger before prose
|
||||
|
||||
Create a deterministic artifact for every beat:
|
||||
|
||||
```text
|
||||
beat
|
||||
-> ordered panels
|
||||
-> ordered actions and chronology
|
||||
-> dialogue/region IDs and exact text
|
||||
-> typed speaker references, confidence, and method
|
||||
-> characters present
|
||||
-> location and time
|
||||
-> named entities
|
||||
-> uncertainty flags
|
||||
```
|
||||
|
||||
This becomes the only factual input to script generation and validation.
|
||||
|
||||
### 2. Make context scene-scoped and evidence-bearing
|
||||
|
||||
- Reset speaker-turn context on strong location/time/scene changes.
|
||||
- Keep a short structured story state rather than repeatedly summarizing generated prose.
|
||||
- Give a script chunk only facts that occur before or inside that chunk.
|
||||
- Do not pass a whole-chapter premise as if it were already-known story state; it can leak future events.
|
||||
- Keep stable chapter-local handles for unresolved recurring characters so they do not disappear from the
|
||||
scene graph while awaiting global promotion.
|
||||
|
||||
### 3. Generate narration in ordered scene chunks
|
||||
|
||||
Generate several consecutive beats in one text-only call and return:
|
||||
|
||||
```json
|
||||
{"beats":[{"beat_id":"b01","text":"...","evidence_ids":["p001_r1"]}]}
|
||||
```
|
||||
|
||||
This provides more continuity and fewer calls than one independent request per beat while keeping output
|
||||
small enough for targeted validation and repair.
|
||||
|
||||
### 4. Respect uncertainty
|
||||
|
||||
Low-confidence or weak-method speaker assignments should become `someone`, an unattributed quote, or a
|
||||
review flag instead of a confident named assertion. Identity and speaker uncertainty must survive into
|
||||
the script prompt.
|
||||
|
||||
### 5. Strengthen verification
|
||||
|
||||
Validate:
|
||||
|
||||
- exact/fuzzy quote grounding;
|
||||
- permitted character and entity names;
|
||||
- action coverage and unsupported actions;
|
||||
- event chronology;
|
||||
- future-fact leakage;
|
||||
- repeated facts and repeated sentence openings;
|
||||
- evidence IDs returned by the model.
|
||||
|
||||
Feed the exact failures into the repair prompt. A final text-only consistency audit may report affected
|
||||
beat IDs, but should not rewrite the whole chapter automatically.
|
||||
|
||||
## Stage clearing and resumability
|
||||
|
||||
Before parallel or identity work, stage ownership of shared data needs to be explicit:
|
||||
|
||||
- dialogue and direction currently mutate the shared vision JSON;
|
||||
- clearing dialogue does not remove its keys, so a rerun can treat old dialogue as completed;
|
||||
- reconcile merges are destructive and are not reversed by clearing the reconcile stage;
|
||||
- clearing identity preserves the per-manga registry, so it is not a clean identity rerun;
|
||||
- name claims attached to a merged-away character are not repointed with the identity assignments.
|
||||
|
||||
Use stage-specific artifacts or explicit key deletion, and preserve provenance through merges. A stage must
|
||||
be genuinely idempotent before it is scheduled concurrently or resumed automatically.
|
||||
|
||||
## Further design ideas
|
||||
|
||||
### 1. Make identity merges reversible
|
||||
|
||||
Reconciliation currently deletes the losing character. Replace destructive merging with a versioned
|
||||
cluster or redirect model:
|
||||
|
||||
```text
|
||||
character_cluster
|
||||
canonical_id
|
||||
member_ids[]
|
||||
merge_evidence[]
|
||||
cannot_link[]
|
||||
version
|
||||
```
|
||||
|
||||
Assignments resolve through the canonical cluster while the original identities remain recoverable. A
|
||||
reviewer can remove one bad merge edge without clearing and rebuilding the entire identity stage.
|
||||
|
||||
Preserve negative evidence as a first-class artifact. Co-presence is particularly valuable: if A and B
|
||||
appear together as distinct detections, they are a permanent cannot-link unless a reviewer overrides it.
|
||||
|
||||
### 2. Separate detection, identity, and naming confidence
|
||||
|
||||
These are different questions and should never share one confidence value:
|
||||
|
||||
- detection confidence: is this actually a story character?
|
||||
- identity confidence: which recurring visual identity is it?
|
||||
- naming confidence: what is that identity's canonical name?
|
||||
|
||||
A person may be confidently detected and tracked as `chapter_person_3` while remaining unnamed. That is
|
||||
enough for coherent narration such as "the colleague" without contaminating the global registry.
|
||||
|
||||
### 3. Use an uncertainty-driven identity cascade
|
||||
|
||||
Spend computation according to ambiguity and downstream impact:
|
||||
|
||||
```text
|
||||
hard constraints resolve the case
|
||||
-> accept cheaply
|
||||
|
||||
strong tracklet plus clear gallery margin
|
||||
-> accept without Gemma
|
||||
|
||||
ambiguous candidates
|
||||
-> multi-view Gemma resolution
|
||||
|
||||
high-impact unresolved identity
|
||||
-> human review
|
||||
|
||||
one-off background person
|
||||
-> chapter-local handle only
|
||||
```
|
||||
|
||||
High-impact detections are recurring, speaking, named, or referenced by several future beats. A one-panel
|
||||
extra should not consume the same identity budget as a protagonist.
|
||||
|
||||
### 4. Add model-aware scheduling across jobs
|
||||
|
||||
Strict chapter-by-chapter execution causes extra model swaps when several jobs are queued. A resource-aware
|
||||
scheduler can group ready work by resident model:
|
||||
|
||||
```text
|
||||
ready Gemma detect phases
|
||||
-> ready SigLIP phases
|
||||
-> ready Gemma post-identity phases
|
||||
-> ready Dots phases
|
||||
```
|
||||
|
||||
This improves total throughput at the cost of some per-job latency. Add fairness and maximum-wait limits so
|
||||
one long title cannot starve other jobs. GPU conflicts must wait in a queue; they must never terminate the
|
||||
active owner's lease.
|
||||
|
||||
### 5. Cache artifacts by dependency fingerprint
|
||||
|
||||
Replace broad "clear stage X and everything downstream" behavior with content-addressed artifacts keyed by:
|
||||
|
||||
```text
|
||||
hash(input URIs
|
||||
+ relevant upstream JSON
|
||||
+ model version
|
||||
+ prompt/schema version
|
||||
+ stage settings)
|
||||
```
|
||||
|
||||
Consequences:
|
||||
|
||||
- changing narration style does not rerun vision;
|
||||
- correcting one identity invalidates only affected scenes and beats;
|
||||
- changing direction does not invalidate dialogue;
|
||||
- a resume reuses every artifact whose dependencies still match;
|
||||
- prompt experiments can run side-by-side instead of overwriting the baseline.
|
||||
|
||||
Store dependency and version metadata beside every stage artifact so invalidation is explainable in the
|
||||
review UI.
|
||||
|
||||
### 6. Detect repeated artwork before expensive stages
|
||||
|
||||
Use CPU-side perceptual hashing after masking detected text regions:
|
||||
|
||||
- same artwork and same text: reuse visual and dialogue artifacts;
|
||||
- same artwork but different text: reuse detection/identity, rerun dialogue;
|
||||
- near-duplicate establishing shot: mark `visual_repeat_of` so narration does not describe it again;
|
||||
- exact duplicate with no new information: curate it out early.
|
||||
|
||||
Text masking is required because two visually identical panels may contain different dialogue. Combine the
|
||||
art hash with text-region geometry and a text-content hash when available.
|
||||
|
||||
### 7. Make the manga registry temporal
|
||||
|
||||
Names, aliases, relationships, and facts need an evidence position:
|
||||
|
||||
```text
|
||||
known_from: chapter 12, panel 37
|
||||
source: self_intro
|
||||
confidence: 0.97
|
||||
```
|
||||
|
||||
The script can then use only knowledge available at that point. This prevents later chapters or out-of-order
|
||||
processing from leaking names and revelations into earlier recaps. A configurable `spoiler_safe` versus
|
||||
`omniscient_recap` policy can decide whether later knowledge may backfill earlier narration.
|
||||
|
||||
The sampled chapter premise needs the same guard because a synopsis built from the whole chapter can reveal
|
||||
its payoff in the opening beat.
|
||||
|
||||
### 8. Represent dialogue as a relation graph
|
||||
|
||||
A spoken name often identifies the addressee or a third party rather than the speaker. Store:
|
||||
|
||||
```text
|
||||
speaker_ref
|
||||
addressed_character_ref
|
||||
mentioned_character_refs[]
|
||||
spoken_text
|
||||
name_surface_form
|
||||
honorific
|
||||
region_id
|
||||
speaker_evidence
|
||||
```
|
||||
|
||||
For example, "Seonho-oppa!" is evidence about the addressee and their relationship, not the speaker's name.
|
||||
Honorifics should be normalized but preserved as evidence instead of being stripped and discarded.
|
||||
|
||||
### 9. Treat speaker attribution as constrained assignment
|
||||
|
||||
For every speech region, build candidates from:
|
||||
|
||||
- bubble-tail or geometric evidence;
|
||||
- visible faces and bodies;
|
||||
- previous and following speaker turns;
|
||||
- chapter-local identities;
|
||||
- the off-panel cast;
|
||||
- same-panel cannot-links.
|
||||
|
||||
Assign speakers jointly across a conversation window rather than deciding every line independently. Use
|
||||
Gemma only to adjudicate ambiguous regions. Require `region_id` in the model output so numbered set-of-mark
|
||||
regions are structurally connected to returned dialogue instead of remaining visual suggestions.
|
||||
|
||||
### 10. Separate stable identity traits from scene appearance
|
||||
|
||||
Store durable and temporary appearance separately:
|
||||
|
||||
```text
|
||||
identity_appearance:
|
||||
face, hair, species, stable marks
|
||||
|
||||
scene_appearance:
|
||||
outfit, accessories, injuries, disguise, chibi/age form
|
||||
```
|
||||
|
||||
Clothing can be strong evidence inside one local scene but should be down-weighted across scene or time
|
||||
changes. This also lets narration mention an outfit change without creating a new global character.
|
||||
|
||||
### 11. Add metamorphic pipeline tests
|
||||
|
||||
Test invariants rather than only fixed outputs:
|
||||
|
||||
- changing dialogue window size must not move text to another panel;
|
||||
- inserting a scenery panel must not change neighboring identities;
|
||||
- resuming halfway through identity must match a clean run;
|
||||
- renumbering panel-local IDs must not change global identity;
|
||||
- distinct co-present characters must never merge;
|
||||
- changing narration style must not change facts or quotes;
|
||||
- processing chapters in a different order must not silently rewrite prior identities.
|
||||
|
||||
These tests exercise cross-stage contracts and resumability failures that per-file self-checks cannot catch.
|
||||
|
||||
### 12. Roll out identity changes in shadow mode
|
||||
|
||||
Write new identity results to a versioned shadow artifact without affecting scripts or the canonical registry.
|
||||
The review UI can compare:
|
||||
|
||||
```text
|
||||
current: Choi Haeseon
|
||||
candidate: unresolved
|
||||
reason: co-presence conflict
|
||||
```
|
||||
|
||||
Promote a new resolver only after it improves the labeled-chapter metrics. Shadow mode prevents tuning work
|
||||
from polluting the production gallery.
|
||||
|
||||
The highest-value additions beyond the core plan are reversible identity clusters with cannot-links,
|
||||
dependency-fingerprinted artifacts, and uncertainty-driven review for recurring/speaking/named characters.
|
||||
|
||||
## Implementation sequence for approval
|
||||
|
||||
### Phase 1: correctness and scheduling safety — DONE 2026-08-11
|
||||
|
||||
Implemented and verified by CPU-only self-checks. Nothing ran on the GPU. What landed, with evidence
|
||||
and the check that covers it, is in `decisions/audit-phase1.md`.
|
||||
|
||||
Two items from this list were NOT implemented and are recorded with a revisit trigger instead:
|
||||
honest stage clearing (`caveats/audit-open.md#dishonest-clearing`) and ComfyUI under the GPU scheduler
|
||||
(`caveats/audit-open.md#comfyui-unscheduled`). Both need a design decision, not a patch.
|
||||
|
||||
### Phase 2: safe throughput improvements
|
||||
|
||||
Now tracked in `ROADMAP.md`, with the done-when condition for each phase.
|
||||
|
||||
Verification: unit checks first, followed by one labeled chapter after GPU availability is approved.
|
||||
|
||||
### Phases 3 and 4
|
||||
|
||||
Now tracked in `ROADMAP.md`.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
Now tracked in `ROADMAP.md`, together with the approval boundary: measure Phase 2 before deciding
|
||||
whether the larger identity and narration changes are worth their size.
|
||||
|
||||
## Second-pass findings (2026-08-11)
|
||||
|
||||
Added after reading the workers and the orchestrator against each other. Every item was read in source and
|
||||
is cited by file and line. Nothing was executed and no GPU work was run.
|
||||
|
||||
The first pass audited each side on its own terms. Most of what follows lives in the seam between them:
|
||||
one repo changed a field's meaning and the other still reads the old one. The existing per-file
|
||||
`__main__` self-checks cannot catch any of it, because each one asserts its own side of the contract.
|
||||
|
||||
### P0: the scene stage discards every speaker — CLOSED, decisions/audit-phase1.md#speaker-ref-is-canonical
|
||||
|
||||
`normalize_dialogue` rewrites `row["speaker"]` to a `character_id` or `None`
|
||||
(`orchestrator/correctness.py:47`). `run_stage_dialogue` saves that shape into the vision blob
|
||||
(`orchestrator/service.py:1159-1160`). `worker_scene.build_scene` still reads that field as a panel-local id
|
||||
and maps it through `id_by_local` (`worker_scene.py:78`). A `character_id` is never a key in that map, so
|
||||
every lookup returns `None`.
|
||||
|
||||
Every scene graph therefore reaches the script worker with `speaker: None`, and `_render_line` writes
|
||||
`Someone says "..."` for all of it (`worker_script.py:48`). Set-of-mark face attribution, the solo-speaker
|
||||
backstop, and window turn-taking all compute the right answer and then lose it one stage later.
|
||||
|
||||
The same field breaks the dialogue resume path: `service.py:1180` pushes the stored value into `recent` as a
|
||||
speaker NAME, so a resumed run feeds raw strings like `character_ab12cd34` to the transcription prompt as if
|
||||
they were people.
|
||||
|
||||
`worker_scene`'s self-check still passes because it feeds the pre-change contract
|
||||
(`worker_scene.py:138`). Fix: prefer `speaker_ref` when its kind is `character_id`, keep the local-id path as
|
||||
the fallback. This is the strongest argument for the cross-stage metamorphic tests proposed above.
|
||||
|
||||
### P0: the script verifier rejects correct narration, and its retry is a no-op — CLOSED, decisions/audit-phase1.md#verifier-false-positives
|
||||
|
||||
`verify_script` fails a beat on two rules that fire on valid output (`orchestrator/correctness.py:100-116`).
|
||||
|
||||
1. `unsupported-proper-noun` flags any capitalized token outside the cast, the entities, the beat's source
|
||||
words, and a 20-word stop list. Ordinary sentence-initial words are not in that list: `Suddenly`,
|
||||
`Behind`, `Inside`, `Both`, `Everyone`, `After`, `Two`. So is `Someone`, which the previous finding
|
||||
guarantees the narrator will emit constantly.
|
||||
2. `misquote` compares each quoted span against a WHOLE source line with `SequenceMatcher` at 0.82. The
|
||||
narration prompt explicitly asks for a short quote of the actual words (`worker_script.py:130-132`). A
|
||||
15-character quote taken from a 40-character line scores about 0.55 and fails. The prompt and the
|
||||
verifier ask for opposite things.
|
||||
|
||||
The failure is not soft. `run_stage_script` retries once, then raises, so the beat produces no narration
|
||||
(`service.py:1384-1386`). `done` never reaches `total`, `_finalize_stage` marks the stage failed, and
|
||||
`_run_pipeline` stops the whole chapter. One false positive halts a run.
|
||||
|
||||
The retry is also inert for the reason already recorded above: `ScriptInput` defines neither `beat` nor
|
||||
`verifier_feedback` (`worker_script.py:13-23`), and pydantic v2 ignores unknown fields by default, so both
|
||||
are dropped without an error.
|
||||
|
||||
Fix order: correct the two rules first, then wire the feedback fields. Fixing the plumbing alone makes the
|
||||
model retry against a broken oracle.
|
||||
|
||||
### P0: correctness flags block TTS permanently in the default configuration — CLOSED, decisions/audit-phase1.md#flag-resolution
|
||||
|
||||
`run_stage_tts` refuses to start while any unresolved flag of kind `script-verifier`, `ambiguous-speaker`,
|
||||
`conflicting-name-claims`, or `partial-dialogue` exists (`service.py:1419-1423`). The only code that ever
|
||||
resolves a flag is `resolve_correctness_flags`, called from `/review/approve` for the `script` gate
|
||||
(`service.py:1955-1956`).
|
||||
|
||||
`GATES` defaults to off (`service.py:312`). The autonomous pipeline therefore has no path that clears a
|
||||
flag. The first flag of those kinds wedges the chapter until a human calls an endpoint that the autonomous
|
||||
mode never mentions. These flags are not rare: `partial-dialogue` is raised whenever a window response omits
|
||||
one panel, and `dialogue_envelope` treats a legitimately silent panel inside a partial window as unresolved
|
||||
(`correctness.py:62`), so one missing panel flags every wordless panel beside it.
|
||||
|
||||
There is a second, worse variant. Flag rows survive their panels. `get_correctness_flags` LEFT JOINs panels
|
||||
and accepts rows where `p.panel_id IS NULL` (`db.py:573`), while `resolve_correctness_flags` can only resolve
|
||||
flags whose panel still exists (`db.py:579`). Clearing the `crop` stage deletes panel rows
|
||||
(`db.py:798-799`). Any flag raised against a deleted panel becomes visible to every chapter and can never be
|
||||
resolved, blocking TTS for all future jobs.
|
||||
|
||||
Fix: resolve by flag identity rather than by surviving panel, and give the autonomous path an explicit
|
||||
policy (auto-resolve below a rank, or fail the stage loudly) instead of an unreachable gate.
|
||||
|
||||
### P0: `awaiting_review` is immediately overwritten by `failed` — CLOSED, decisions/audit-phase1.md#flag-resolution
|
||||
|
||||
`run_stage_tts` sets `awaiting_review` and then raises (`service.py:1422-1423`). The raise unwinds into
|
||||
`_run_pipeline`'s catch-all, which sets the job to `failed` (`service.py:391-393`). The review state the
|
||||
stage just recorded is gone before anyone can read it, and the operator sees a generic failure rather than a
|
||||
queue of flags. The gate path at `service.py:337-341` returns instead of raising and does not have this
|
||||
problem.
|
||||
|
||||
### P1: set-of-mark face pairing is enabled by default, contrary to its own documentation — CLOSED, decisions/audit-phase1.md#gated-face-pairing
|
||||
|
||||
`worker_vision.py:22` states the feature is off by default and should be enabled per title once tuned.
|
||||
`worker_vision.py:31` reads `SOM_ATTRIBUTION` with a default of `"1"`. It is on.
|
||||
|
||||
That matters because of the unconditional pairing already recorded as a P0. The mispairing does not stay
|
||||
local. `_set_of_mark` writes the wrong name into the legend the model reads (`worker_vision.py:99-104`), the
|
||||
model attributes a line to that face label, and `_apply_speaker_labels` converts the label back into a
|
||||
`local_id` and stamps `speaker_method = "som_face"` (`worker_vision.py:117`). A geometric guess is laundered
|
||||
into the highest-trust provenance value the system has. Gate the pairing before trusting that label, or
|
||||
default the flag to off as documented.
|
||||
|
||||
The pairing is also greedy in face order rather than a joint assignment (`worker_vision.py:42-52`), so the
|
||||
first face processed can claim a character that fits a later face far better.
|
||||
|
||||
### P1: the JSON repair pass can fabricate content — OPEN, caveats/audit-open.md#repair-fabricates
|
||||
|
||||
When a response fails to parse, `call_gemma4_json` sends the model its own truncated text and asks for the
|
||||
JSON it should have been (`worker_vision.py:174-178`). The repair call carries no image. On a response
|
||||
truncated by `max_tokens`, the model is being asked to complete dialogue it can no longer see. Anything it
|
||||
adds is invented and is indistinguishable downstream from transcribed text.
|
||||
|
||||
Schema-constrained generation, already recommended above, removes most of this path. Until then the repair
|
||||
pass should re-send the image, or a truncated response should be retried rather than repaired.
|
||||
|
||||
### P1: `/review/preview` silently replaces a beat clip — OPEN, caveats/audit-open.md#preview-overwrites-clip
|
||||
|
||||
`review_preview` renders one panel alone and calls `save_clip(panel_id, ...)` (`service.py:1902-1904`). Its
|
||||
docstring calls this harmless because assemble would regenerate it. It does not. `_render_one_beat` returns
|
||||
early when a clip already exists for the leader (`service.py:1591-1592`). Previewing a beat leader therefore
|
||||
pins the solo preview into the final video and drops the rest of the beat's panels. `review_retts` gets this
|
||||
right by calling `delete_clip` first (`service.py:2008`). Preview should write to a scratch key or delete the
|
||||
clip row afterwards.
|
||||
|
||||
### P1: the session manager can orphan a llama-server and hold the GPU — CLOSED, decisions/audit-phase1.md#unlocked-model-load
|
||||
|
||||
`open_session` claims the slot, releases the lock, then spawns and health-waits for up to 300 seconds
|
||||
(`session_manager.py:108-119`). A `/session/close` arriving during that window finds `proc = None`, tears
|
||||
down nothing, and clears `_active`. The process that finishes starting afterwards is unreferenced and keeps
|
||||
its VRAM until someone kills it by hand. The next `open` spawns a second server on the same port, and
|
||||
`_health_wait` cannot tell the two apart because it only probes the port
|
||||
(`session_manager.py:51-60`).
|
||||
|
||||
Related asymmetry: `_supervise_once` calls `_start_subprocess` while holding `_lock`
|
||||
(`session_manager.py:169-181`), so a respawn blocks `/session/active`, `/session/close`, and `/session/open`
|
||||
for the full health wait. The open path was deliberately written to avoid exactly this.
|
||||
|
||||
### P1: nothing limits concurrent jobs — CLOSED, decisions/audit-phase1.md#no-lease-stealing
|
||||
|
||||
`_background_jobs` accepts any number of pipelines (`service.py:302-303`, `service.py:427-428`). Each opens
|
||||
its own model sessions. Combined with the 409 lease stealing already recorded, two jobs terminate each
|
||||
other's models rather than queueing. The queue has to live at the orchestrator as job admission control. A
|
||||
fix inside `session_proxy` alone still lets two pipelines interleave stages against one GPU.
|
||||
|
||||
### P1: two cheap throughput wins the plan does not name
|
||||
|
||||
**Order the shared prefix first so llama-server reuses its KV cache.** `/direct/window` sends the same
|
||||
panel images twice, once for grouping and once for shot design (`worker_vision.py:702-733`). The audit
|
||||
above frames this as duplicate downloads. The larger cost is the second vision-encoder prefill on the GPU.
|
||||
llama-server reuses the longest common prompt prefix across requests, and the two calls currently differ in
|
||||
their first token because the instruction text precedes the images. Putting the images first and the
|
||||
differing instruction last makes the second pass nearly free on prefill. The same reordering helps any
|
||||
stage that issues several calls over one image set.
|
||||
|
||||
**Give stored embeddings a version tag.** `embed_crop` uses `pooler_output` when present and silently falls
|
||||
back to mean-pooled patch tokens otherwise (`worker_identity.py:48-52`). Those two paths produce different
|
||||
vector spaces, and the fixed 0.85 threshold is only valid for one of them. Nothing recorded beside a stored
|
||||
`.npy` says which model, revision, or pooling produced it, so a transformers upgrade mixes incompatible
|
||||
vectors into one gallery with no error. Write the model id and pooling mode next to the vector and refuse to
|
||||
compare across versions.
|
||||
|
||||
### P2: smaller confirmed defects — partly closed, see `decisions/audit-phase1.md#related` and `caveats/audit-open.md`
|
||||
|
||||
- An out-of-range `choice` from the resolver is mapped to NONE and then reported as `state: "new"`
|
||||
(`worker_vision.py:898-899`), so a hallucinated index mints a brand new character. It should be
|
||||
`unresolved`, like a parse failure.
|
||||
- `_extract_json` matches greedily from the first `{` to the last `}` (`worker_vision.py:130`). Two objects
|
||||
or any trailing braced prose produce an unparseable span and burn a repair call. `json.JSONDecoder().raw_decode`
|
||||
from the first brace is exact.
|
||||
- `worker_identity._known_cache` is invalidated only by `_persist_char` (`worker_identity.py:128-132`).
|
||||
The reconcile stage deletes losing characters directly in the orchestrator database
|
||||
(`db.py:506`), so a long-lived worker keeps shortlisting and assigning ids that no longer exist.
|
||||
- `_pending` holds full crop images in worker memory for a whole chapter and is keyed by session
|
||||
(`worker_identity.py:25`). This is durable per-chapter state inside a worker documented as stateless, it
|
||||
is lost on restart, and a character seen exactly once receives no assignment at all, not even a
|
||||
chapter-local handle.
|
||||
- `get_conn` opens a connection per call with no `busy_timeout` (`db.py:190-197`). WAL tolerates one writer.
|
||||
`PIPELINE=1` already writes clips from concurrent tasks while TTS writes audio, so the planned CPU
|
||||
parallelism will surface as `database is locked` before it surfaces as throughput.
|
||||
- Worker endpoints declared `async def` run blocking MinIO, OpenCV, torch, and ffmpeg calls directly on the
|
||||
event loop (`worker_identity.py:177`, `worker_tts.py:170`, `worker_vision.py:230`). A busy worker cannot
|
||||
answer `/health` or `/unload`. That makes `/health/workers` report a working worker as unreachable
|
||||
(`service.py:206-216`) and puts the session manager's 30-second `/unload` at risk exactly when VRAM needs
|
||||
freeing (`session_manager.py:93`). `def` instead of `async def` moves each to the threadpool.
|
||||
- `layers` runs after `tts` in `STAGES` (`db.py:330-333`), while `run_stage_tts` warns that eager rendering
|
||||
under `PIPELINE=1` needs layers to run first (`service.py:1427-1429`). With the flag on, solo beats always
|
||||
render without parallax.
|
||||
- Stage failure policy is inconsistent. A dropped vision panel fails the stage and halts the pipeline. A
|
||||
failed direction window counts its panels as done (`service.py:1247`), and layers always finishes
|
||||
completed (`service.py:1503`). `completed` does not mean the same thing across stages, which makes the
|
||||
acceptance metrics below hard to read.
|
||||
- `run_stage_assemble` does not check that `clip_uris` is non-empty before assembling and then marks the job
|
||||
completed (`service.py:1661-1677`).
|
||||
- The `/review/panels` timeline sums per-panel audio durations (`service.py:1750`), but assemble crossfades
|
||||
clips using the per-beat transitions (`service.py:1671-1674`). Every non-`cut` transition shortens the
|
||||
real video, so reviewer timestamps drift further out of sync the further into the chapter they scrub.
|
||||
- MinIO credentials are hardcoded as defaults in committed source (`transport.py:95-99`,
|
||||
`service.py:74-76`).
|
||||
|
||||
### What this changes in the plan — DONE
|
||||
|
||||
All five additions landed in Phase 1. See `decisions/audit-phase1.md`.
|
||||
|
||||
The acceptance metric they suggested, the share of narrated lines whose speaker is a named character
|
||||
rather than `Someone`, is now in `ROADMAP.md`. It has not been measured yet.
|
||||
@@ -1,67 +1,80 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
Goal, invariants, and working rules. Read this first.
|
||||
|
||||
## What this is
|
||||
| file | holds |
|
||||
| --- | --- |
|
||||
| `NEXT.md` | the current state and the live plan |
|
||||
| `ROADMAP.md` | the ordered outcomes past the current one |
|
||||
| `JOURNAL.md` | what was run and when, append-only |
|
||||
| `decisions/` | every settled question, indexed in `decisions/CLAUDE.md` |
|
||||
| `caveats/` | every known limit and its revisit trigger, indexed in `caveats/CLAUDE.md` |
|
||||
| `AGENTS.md` | commands, with the traps beside them |
|
||||
| `AUDIT.md` | the 2026-08-11 pipeline audit, the source of the roadmap |
|
||||
| `ARCHITECTURE.md` | the target shape of the pipeline, and what exists against it today |
|
||||
| `spec-v3.md` | current quality and look work, marked DONE/TODO per item |
|
||||
|
||||
The **workpc compute half** of a manga→narrated-video pipeline. This repo holds stateless
|
||||
GPU/CPU workers only. State, job scheduling, and stage orchestration live in a **separate
|
||||
homesrv orchestrator repo** (`/mnt/server/home/kami/apps/Maven/` region) — not here. The two
|
||||
communicate over a fixed HTTP contract; never add sqlite or durable state to a worker.
|
||||
Do not restate a finding here. Point at the decision.
|
||||
|
||||
Machine split (memorize): workers run on **workpc** (RX 7900 GRE, ROCm). MinIO + orchestrator
|
||||
run on **homesrv** (`192.168.1.104`, CPU-only). Data dir `/mnt/server/home/kami/` is an SSHFS
|
||||
mount of homesrv.
|
||||
## The goal
|
||||
|
||||
## Run / test
|
||||
This is the **workpc compute half** of a manga to narrated-video pipeline. It holds stateless GPU and
|
||||
CPU workers only. State, job scheduling, and stage orchestration live in a separate homesrv
|
||||
orchestrator repo (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`). The two talk over a
|
||||
fixed HTTP contract. The hard part is that one GPU serves every model, so the schedule, not the model,
|
||||
sets the wall time.
|
||||
|
||||
Machine split: workers run on **workpc** (RX 7900 GRE, ROCm). MinIO and the orchestrator run on
|
||||
**homesrv** (`192.168.1.104`, CPU-only). `/mnt/server/home/kami/` is an SSHFS mount of homesrv.
|
||||
|
||||
## Invariants
|
||||
|
||||
1. **No durable state in a worker.** No sqlite, no cross-request memory. A worker pulls inputs from
|
||||
MinIO by URI, does one stage, pushes outputs back, returns URIs. A `/dev/shm` cache is allowed
|
||||
because it may be dropped at any time.
|
||||
2. **One warm model at a time.** Every GPU stage takes a lease from `session_manager.py` on 8095. A 409
|
||||
is a queue signal, never a stale lease
|
||||
(`decisions/audit-phase1.md#no-lease-stealing`).
|
||||
3. **Never hold `_lock` across a model load or a health wait** in `session_manager.py`
|
||||
(`decisions/audit-phase1.md#unlocked-model-load`).
|
||||
4. **A dialogue row's speaker is `speaker_ref`.** The flat `speaker` field is a compatibility value and
|
||||
holds a `character_id`, not a panel-local id
|
||||
(`decisions/audit-phase1.md#speaker-ref-is-canonical`).
|
||||
5. **A stage never raises after setting `awaiting_review`.** The pipeline's catch-all overwrites it
|
||||
with `failed` (`decisions/audit-phase1.md#flag-resolution`).
|
||||
6. **Never mint a character from an unparseable or out-of-range model answer.** That is `unresolved`
|
||||
(`decisions/audit-phase1.md#hallucinated-index`).
|
||||
7. **The HTTP contract with the orchestrator is load-bearing.** Changing a worker's request or response
|
||||
shape means reconciling the orchestrator in the same session. Neither repo's self-checks can catch a
|
||||
contract break, because each asserts its own side.
|
||||
8. **`ponytail:` comments mark deliberate simplifications** and name the upgrade path. Respect them.
|
||||
9. **Nothing that positions an ffmpeg filter may use `format=duration`.** It reports `max(video, audio)`.
|
||||
It hides A/V drift, and it walks xfade offsets past the end of their input. ffmpeg then discards clips
|
||||
and still exits 0 (`decisions/chapter-assembly.md#offsets-from-min-stream`).
|
||||
|
||||
## Working rules
|
||||
|
||||
```bash
|
||||
./start_workers.sh # dev: session_manager + 9 workers, each a uvicorn in a tmux window
|
||||
tmux attach -t manga-workers # watch logs; per-worker window
|
||||
tmux attach -t manga-workers # per-worker logs
|
||||
sudo systemd/install.sh # production: one systemd unit per process (User=kami)
|
||||
|
||||
python session_manager.py # each module has a __main__ self-check (assert-based, no framework)
|
||||
python transport.py # run these to verify a file after editing it
|
||||
python test_vision_parse.py # the one standalone pytest-free test
|
||||
.venv/bin/python worker_scene.py # every module has an assert-based __main__ self-check
|
||||
.venv/bin/python test_vision_parse.py
|
||||
ruff check . # must exit 0; every ignore in ruff.toml carries its reason
|
||||
|
||||
cd /mnt/server/home/kami/docker-apps/manga-infra/orchestrator && pytest -q --ignore=test_api.py
|
||||
```
|
||||
|
||||
There is no lint/build step. `.venv` is the ROCm torch env; workers import `transport` by module name.
|
||||
There is no build step. `.venv` is the ROCm torch env. Workers import `transport` by module
|
||||
name. Ports: crop 8000, vision 8002, identity 8003, scene 8004, script 8005, tts 8006, layers 8007,
|
||||
render 8008, session_manager 8095.
|
||||
|
||||
## Architecture
|
||||
Every output S3 URI is a template in `transport.py`, not an f-string in a worker. Add one there when a
|
||||
new artifact class appears.
|
||||
|
||||
**Workers are stateless HTTP stages.** Each `worker_*.py` is a FastAPI app on a fixed port. It
|
||||
pulls inputs from MinIO by URI to local disk (`/dev/shm`), does one stage, pushes outputs back,
|
||||
returns URIs. No cross-request memory. Ports: crop 8000, vision 8002, identity 8003, scene 8004,
|
||||
script 8005, tts 8006, layers 8007, render 8008, **session_manager 8095**.
|
||||
|
||||
**`transport.py`** — shared MinIO client (`get`/`put`/`put_bytes`/`exists`) + `install_logging(app, name)`
|
||||
(one log line per request with panel id). URIs are `s3://bucket/key`. Import it in every worker.
|
||||
|
||||
**`session_manager.py`** — the **GPU mutex**. Only one warm model at a time on the single local GPU.
|
||||
Two model kinds:
|
||||
- *subprocess* (`gemma4`): it spawns/health-waits/terminates `llama-server`, and a supervisor
|
||||
respawns it in-place if it crashes mid-session (keeps the same session_id + port).
|
||||
- *in-process* (`siglip2`, `dots`): returns `port=None`; the **worker** loads the transformers model
|
||||
itself and must expose `/unload` so `/session/close` can free the ~5GB VRAM (the mutex alone
|
||||
can't reclaim it). Leases have a TTL + heartbeat; a reaper force-closes stale ones.
|
||||
|
||||
A GPU worker's flow: `/session/open {model}` → (409 if busy) → do work against the returned port or
|
||||
its own resident model → `/session/close`. CPU workers (crop) take no session.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **`ponytail:` comments** mark deliberate simplifications and name the upgrade path — respect them,
|
||||
don't "fix" them without reason.
|
||||
- Any non-trivial logic gets ONE runnable check in `__main__` (assert-based `demo`/self-check), not a
|
||||
test suite. Follow that pattern; run the file to verify.
|
||||
- The HTTP contract with the orchestrator is load-bearing and shared across repos — changing a
|
||||
worker's request/response shape means reconciling the orchestrator too (see commit history:
|
||||
"reconcile worker contracts").
|
||||
|
||||
## Specs & docs
|
||||
|
||||
- `spec-v3.md` — current quality/look work (narration voice, panel curation, render overhaul), marked
|
||||
DONE/TODO per item. `legacy/` holds the old single-repo workers.
|
||||
- `AGENTS.md` — use `scripts/analyze_video_frames.sh` (not manual seeking) to diagnose render output;
|
||||
its frames are `/tmp` artifacts, never commit them.
|
||||
- `collage.py` (behind `COLLAGE` flag) and the render worker carry the animated-layout planners.
|
||||
- Any non-trivial logic gets ONE runnable check in `__main__`, assert-based, no framework. Run the file
|
||||
to verify it.
|
||||
- Editing a worker's request or response shape means editing the orchestrator too, in the same session.
|
||||
- Update `NEXT.md` alongside any change that moves the plan, and append to `JOURNAL.md` after a run.
|
||||
- Do not run GPU work or a full pipeline without asking.
|
||||
|
||||
+95
-64
@@ -1,75 +1,106 @@
|
||||
# HANDOFF: repo reconstructed from agent transcripts (2026-08-11)
|
||||
# HANDOFF, 2026-08-13 (eighth session)
|
||||
|
||||
## What was asked
|
||||
The working copy at `/home/kami/Programs/n8n-worker` was deleted with `rm`, including `.git`.
|
||||
This repo is the workpc GPU-compute half of the manga pipeline. The ask was to rebuild it from
|
||||
the transcripts on disk, at minimum the transport layer and the pipeline flow.
|
||||
Live state is in `NEXT.md`. This file is only what this session did. The previous handoff is in
|
||||
`JOURNAL.md`.
|
||||
|
||||
## Sources used
|
||||
No backup and no remote survived. `~/.local/share/Trash` is empty. No copy exists under
|
||||
`/home/kami`, `/mnt/D`, or `/mnt/server`. Gitea
|
||||
(`/mnt/server/mnt/hdd2/gitea/git/repositories/`) holds only correx, hexis, maven,
|
||||
model-training, muzick, orchestra, and test-e2e. There is no `n8n-worker` mirror.
|
||||
## Asked
|
||||
|
||||
Reconstruction replays three histories into one timeline, ordered by timestamp:
|
||||
Push the repo to Gitea and open a PR to `master`. Then, from the PR review: fix the s3 URI constants,
|
||||
run ruff and fix what it finds, push, run a ponytail audit, fix, push, merge. Then audit.
|
||||
|
||||
1. Claude Code transcripts, 25 sessions in
|
||||
`~/.claude/projects/-home-kami-Programs-n8n-worker/*.jsonl`. Contributes Write and Edit
|
||||
content, full-file Read results, and `@`-mention attachments.
|
||||
2. Codex rollouts, 5 sessions with `cwd=/home/kami/Programs/n8n-worker` under
|
||||
`~/.codex/sessions/2026/07/{13,15,16,18}/rollout-*.jsonl`. Contributes 26 `apply_patch`
|
||||
blocks. The 4 that the log marks `Script failed` are skipped. These carry the 2026-07-18
|
||||
correctness work (`_dialogue_envelope`, `_normalize_claims`, `_annotate_speaker_methods`,
|
||||
`SOM_ATTRIBUTION=1`) that exists in no Claude transcript.
|
||||
3. `~/.claude/file-history/<session>/<hash>@vN` pre-edit blobs, used as cross-checks only.
|
||||
## Result
|
||||
|
||||
Scripts live in
|
||||
`/tmp/claude-1000/-home-kami-Programs-n8n-worker/6a35d5d8-9f44-4412-955c-9cf088737ad5/scratchpad/`.
|
||||
`replay2.py` is the merged replay. `codex.py` extracts and applies codex patches. `backups.py`
|
||||
compares against file-history. `recon_final/_log.json` holds the per-file event log.
|
||||
No GPU work. Nothing ran on a pipeline stage. PR #1 is open at
|
||||
`https://gitea.kvmx.ru/kami/manga-recap-pipeline/pulls/1`, `master` <- `restore-runtime`.
|
||||
|
||||
## Verification
|
||||
- Replayed to 2026-07-18T13:13:44Z, line counts match the `wc -l` recorded in the transcript at
|
||||
that moment, for all 10 files it covered. crop 326, identity 314, layers 99, render 996,
|
||||
scene 159, script 300, tts 231, vision 871, session_manager 238, transport 216.
|
||||
- Byte sizes match the `ls -l` recorded by the same command. AGENTS.md 867, CLAUDE.md 3678,
|
||||
manga-recap-pipeline-spec.md 11413, plan.md 16511, spec-v3.md 26170. `.gitignore` is 53.
|
||||
- 18 files are byte-identical to their newest file-history blob. The 4 that differ (render,
|
||||
script, tts, spec-v3) differ only by edits made after that blob was taken.
|
||||
- `python3 -m py_compile *.py scripts/*.py attic/*.py` passes. `bash -n` passes on all 3 shell
|
||||
scripts. Runtime self-checks such as `python worker_vision.py` were NOT run, because `.venv/`
|
||||
is gone.
|
||||
`master` held only the reconstruction commit `ff6a512`. All 27 commits of real work sat unpushed on
|
||||
`restore-runtime`.
|
||||
|
||||
## Recovered (live tree)
|
||||
Workers `worker_{crop,vision,identity,scene,script,tts,layers,render}.py`, the GPU mutex
|
||||
`session_manager.py`, `transport.py`, helpers `bubble_detect.py`, `face_detect.py`,
|
||||
`collage.py`, `test_vision_parse.py`, `start_workers.sh`, `systemd/install.sh`,
|
||||
`scripts/{analyze_video_frames.sh,pick_tts_voice.py}`, `requirements.txt`, `.gitignore`,
|
||||
and the docs `AGENTS.md`, `CLAUDE.md`, `plan.md`, `spec-v3.md`, `spec-correctness.md`,
|
||||
`manga-recap-pipeline-spec.md`, `identity-rework-task.md`.
|
||||
## Landed
|
||||
|
||||
`attic/` holds files that had been deleted from the tree before the `rm`. They are kept but not
|
||||
live: `worker_ocr.py`, `worker_parse.py`, `plan-workpc.md`, `char-recognition.md`.
|
||||
| commit | what |
|
||||
| --- | --- |
|
||||
| `12d4ed8` | the `ARCHITECTURE.md` rewrite, which was finished but uncommitted |
|
||||
| `be63b22` | deleted a 0-byte file named `II, d[i+4:i+12]))\n"; done`, added by `8113bdf` |
|
||||
| `bec9411` | s3 URI templates into `transport.py`, plus `ruff.toml` and the lint fixes |
|
||||
|
||||
## Still open
|
||||
- `.venv/` is gone. Rebuild with
|
||||
`python -m venv .venv && .venv/bin/pip install -r requirements.txt`. The ROCm torch wheel is
|
||||
not pinned in requirements.txt, so install it the way workpc had it.
|
||||
- `dots.tts/` (gitignored external checkout) and `legacy/` (pre-rewrite workers, moved there
|
||||
2026-07-13) are not recoverable from transcripts. Re-clone dots.tts if TTS is needed.
|
||||
- `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable. Neither agent
|
||||
ever read them in full. All three were already deleted from the working tree as of 2026-07-17.
|
||||
- Git history is gone. A fresh `git init` plus one commit replaces it. Old commits are not
|
||||
recoverable.
|
||||
- Two replay gaps were left unpatched. One Edit MISS in `worker_render.py` on 2026-07-14, and one
|
||||
orphan Edit against `attic/plan-workpc.md` on 2026-07-13. Both files were later replaced by a
|
||||
full-file snapshot, so the final content stays anchored.
|
||||
- Edits made outside Claude and codex after 2026-07-18T14:35Z (the last recorded write) cannot be
|
||||
detected. `unknown:` whether any exist.
|
||||
### S3 URIs
|
||||
|
||||
Eight templates now live in `transport.py`: `PANEL_URI`, `PAGE_PANEL_URI`, `AUDIO_URI`,
|
||||
`AUDIO_FLAT_URI`, `LAYER_URI`, `CLIP_URI`, `CHAPTER_URI`, `CHAR_PNG_URI`, `CHAR_NPY_URI`. Five workers
|
||||
formatted their own before.
|
||||
|
||||
`transport.ids_from_uri` replaces three separate copies of the same parse in `worker_tts`,
|
||||
`worker_layers` and `worker_render`. It raises on a uri too short to carry the ids instead of returning
|
||||
a wrong pair. `worker_render._mc_from_uri` is gone, its 6 call sites repointed.
|
||||
|
||||
### Lint
|
||||
|
||||
`ruff check .` exits 0. Ruff's defaults found 115. Fixed: implicit `Optional` in 8 signatures, an
|
||||
unparenthesized implicit concatenation inside the ASS filter list, 5 `subprocess.run` calls now saying
|
||||
`check=False` out loud, 1 unused import, 1 duplicate exception handler, 1 non-executable shebang,
|
||||
4 `dict()` calls and 2 `startswith` chains.
|
||||
|
||||
12 rules are off in `ruff.toml`, each with its reason. The ASYNC ones matter: ffmpeg and ffprobe run
|
||||
synchronously inside `async def` endpoints, so a busy worker cannot answer `/health`. That is already
|
||||
`caveats/audit-open.md#blocking-event-loop`, tracked as [#199], and it needs a per-handler refactor
|
||||
rather than a lint fix.
|
||||
|
||||
`CLAUDE.md` said "There is no lint or build step". It now names `ruff check .` and the rule that every
|
||||
output URI is a template in `transport.py`.
|
||||
|
||||
## Audit findings
|
||||
|
||||
Applied:
|
||||
|
||||
- `delete:` `attic/worker_ocr.py` and `attic/worker_parse.py`, 224 lines, imported by nothing and named
|
||||
in no doc. `attic/char-recognition.md` and `attic/plan-workpc.md` kept, they are design history.
|
||||
- `delete:` `worker_vision._panel_size`, 8 lines, one reference and it is the definition.
|
||||
|
||||
Found and NOT applied, in order of size:
|
||||
|
||||
- `shrink:` `call_gemma4`, `_extract_json` and `_strip_thought` each exist twice, in `worker_vision.py`
|
||||
and `worker_script.py`, and **have already diverged**. `worker_vision.call_gemma4` is 21 lines and
|
||||
takes a content list, `worker_script`'s is 8 and takes a prompt string plus a system prompt. The two
|
||||
`_extract_json` bodies carry the same comment about `raw_decode` but different error text. This is
|
||||
the live risk: `caveats/audit-open.md#repair-fabricates` says the JSON repair pass can fabricate
|
||||
dialogue, and a fix would land in one copy. A shared `gemma.py` costs one new file and removes about
|
||||
25 duplicated lines. Not done because it touches the two largest workers and nothing has run on a GPU
|
||||
since.
|
||||
- `stdlib:` `worker_render._wrap2`, 16 lines of greedy word-wrap with a 2-line cap, is close to
|
||||
`textwrap.wrap(text, width, max_lines=2, placeholder="…")`. Not identical: `_wrap2` does
|
||||
`rstrip(".,")` before the ellipsis, and textwrap reserves width for the placeholder, so the break
|
||||
points differ. It renders burned-in subtitles, so the diff is user-visible. Verify against the render
|
||||
self-check before swapping.
|
||||
- `shrink:` `_letterbox` and `_load` are duplicated between `bubble_detect.py` and `face_detect.py`,
|
||||
two ONNX detectors with the same preprocessing. About 16 lines.
|
||||
- `delete:` `worker_layers.py:14` points `LAYERED_WORKFLOW` at `legacy/qwen_layered_workflow.json`.
|
||||
`legacy/` was **never tracked in git** and is absent from disk, so the module's self-check has never
|
||||
been able to pass here. This is the other half of `caveats/audit-open.md#layers-writes-nothing`.
|
||||
|
||||
## Checks
|
||||
|
||||
```bash
|
||||
ruff check . # All checks passed
|
||||
.venv/bin/python worker_render.py # ok, ffmpeg ran, about 4 minutes
|
||||
```
|
||||
|
||||
Self-checks pass: `transport`, `collage`, `bubble_detect`, `test_vision_parse`, `worker_crop`,
|
||||
`worker_scene`, `worker_script`, `worker_identity`, `worker_tts`, `session_manager`, `worker_vision`,
|
||||
`worker_render`, `face_detect`.
|
||||
|
||||
`worker_layers` fails, and did before this branch, on the missing `legacy/` file above.
|
||||
|
||||
## Next command
|
||||
```
|
||||
cd /home/kami/Programs/n8n-worker
|
||||
python -m venv .venv && .venv/bin/pip install -r requirements.txt
|
||||
.venv/bin/python worker_vision.py # per-file __main__ self-checks, then ./start_workers.sh
|
||||
```
|
||||
|
||||
The fifth GPU cycle is still the next pipeline work. It is blocked only on the GPU being free. The
|
||||
exact sequence is in `JOURNAL.md` under the seventh session, and `NEXT.md` item 1 holds the
|
||||
expectations. **Clear the panels prefix first** or the wired caption merge
|
||||
will not take effect.
|
||||
|
||||
Before that, or instead of it while the GPU is busy, `ARCHITECTURE.md` step 0 is the measurement spine:
|
||||
`eval/chapter-truth.json` plus purity and fragmentation printed by `audit_registry.py`.
|
||||
|
||||
One ordering trap, unresolved: the truth file cannot be keyed on `panel_id` or `character_id`. The
|
||||
fifth cycle re-crops and calls `/characters/reset`, which destroys both. Key it on page-space geometry,
|
||||
or write it after the fifth cycle rather than before.
|
||||
|
||||
+758
@@ -0,0 +1,758 @@
|
||||
# JOURNAL
|
||||
|
||||
Append-only, newest last. One block per session or run. Not a changelog: this records what happened on
|
||||
the day a number was produced, so a later postmortem can find it.
|
||||
|
||||
## 2026-08-11 Audit second pass [no task]
|
||||
|
||||
Command: none. Source reading only.
|
||||
Outcome: finished. `AUDIT.md` grew from 565 to 771 lines with a `## Second-pass findings` section:
|
||||
4 new P0, 6 new P1, 13 P2, 5 additions to the Phase 1 list.
|
||||
Produced: commit `6d9df5b`, `AUDIT.md:566`.
|
||||
|
||||
## 2026-08-11 Audit Phase 1 implemented [#203]
|
||||
|
||||
Command: `python worker_scene.py worker_script.py worker_vision.py session_manager.py`,
|
||||
`pytest -q --ignore=test_api.py` in the orchestrator.
|
||||
Outcome: finished. All self-checks pass, 108 orchestrator tests pass. No GPU work, no pipeline run.
|
||||
`test_api.py` was skipped because fastapi is not installed in the workpc venv.
|
||||
Produced: `decisions/audit-phase1.md`, `caveats/audit-open.md`, `ROADMAP.md`, and this scaffold.
|
||||
|
||||
One existing test asserted the bug: `test_name_binding.test_conflict_flags_and_stays_unnamed` relied on
|
||||
orphan flags leaking into every chapter, because it never created panel rows. It now creates them.
|
||||
|
||||
## 2026-08-11 Orchestrator half committed and deployed [no task]
|
||||
|
||||
Command: `pytest -q --ignore=test_api.py`, then `docker compose up -d --build orchestrator` on homesrv.
|
||||
Outcome: finished. 108 tests pass. Commits `1c60710` (orchestrator half) and `94bd4d8` (minio pin) in
|
||||
`/mnt/server/home/kami/docker-apps`. Orchestrator and minio both answer health on homesrv.
|
||||
|
||||
The rebuild recreated `minio` as a side effect and it crash-looped with `exec format error`: the
|
||||
compose pin was the arm64 manifest digest of `minio/minio:latest` and homesrv is amd64. Repinned to the
|
||||
amd64 digest. Nothing about Phase 1 caused this, but any compose action that recreates minio would have
|
||||
hit it, so it was latent, not new.
|
||||
|
||||
Still unrun against a real chapter.
|
||||
|
||||
## 2026-08-11 S3 viewer and storage swap, tasks #116/#117 [#116 #117]
|
||||
|
||||
Command: docker compose on homesrv, `dig`, `openssl s_client`. No pipeline, no GPU.
|
||||
Outcome: partial. Viewer works, storage swap staged and unfinished.
|
||||
|
||||
#117 needed no new software. `stowage` at `~/docker-apps/stowage` was already configured against the
|
||||
manga MinIO and had been dead since 2026-07-19 with `exec /sbin/tini: exec format error`: its digest
|
||||
pin was the arm64 manifest. Repinned to amd64 `sha256:91be7f13`, chowned `data/` to uid 65532 for the
|
||||
new image, and it serves. MinIO had the identical bug, repinned to `sha256:a1a8bd4a`. A sweep of all
|
||||
470 local images on homesrv found exactly those two arm64; nothing else in the homelab is affected.
|
||||
|
||||
#116 is staged, not done. `rustfs` runs alongside MinIO on `127.0.0.1:9010/9011`, pinned
|
||||
`sha256:19b105cc`, data at `/mnt/hdd2/rustfs`. Buckets are empty: the `mc` mirror of
|
||||
`audio layers manga panels raw video` (350M, all in `manga`) has NOT run. `/mnt/hdd2/minio/data` is
|
||||
untouched and is the rollback. RustFS is `1.0.0-beta.12`, labeled `build-type=prerelease`. Cutover
|
||||
would give rustfs 9000/9001 and repoint `MINIO_ENDPOINT=minio:9000` in the orchestrator plus
|
||||
`stowage/config.yaml`; `transport.py:95` needs no change if rustfs takes `192.168.1.104:9000`.
|
||||
|
||||
Side quest, unrelated to the pipeline: the shared 41-domain cert stopped renewing. Root cause was DNS,
|
||||
not nginx. Every `*.kvmx.ru` name pointed at a hard A record for `109.229.102.117` while the line had
|
||||
moved to `109.229.127.149`; the Mercusys DDNS at `kvmx-home.mercusysddns.com` was correct the whole
|
||||
time but nothing in the zone referenced it. Fixed with `CNAME * -> kvmx-home.mercusysddns.com` at
|
||||
reg.ru. Certificate now issues.
|
||||
|
||||
Two measurement traps worth remembering. The ISP transparently intercepts ports 80 and 443 by
|
||||
Host/SNI, so `curl` from workpc to ANY address returns kvmx.ru content and proves nothing about
|
||||
external reachability; bare TCP connects also succeed against arbitrary addresses and then hang. Three
|
||||
wrong root causes came out of trusting those probes before checking them.
|
||||
|
||||
Also patched `~/scripts/migrate-kvmx-https.sh:54` on homesrv. `need_stream_module` used
|
||||
`sudo -n nginx -V` and `sudo -n nginx -T`; the NOPASSWD rule covers only `nginx -t`, so it reported
|
||||
"stream module is not loaded" whenever it meant "could not ask for a password". Both checks now run
|
||||
without sudo. `bash -n` passes and both conditions evaluate true.
|
||||
|
||||
## 2026-08-11 Per-artifact buckets, rustfs buckets, baseline chapter run [#116]
|
||||
|
||||
Command: `mc mb` on rustfs, `docker compose up -d --build orchestrator`, `pytest -q --ignore=test_api.py`,
|
||||
`./start_workers.sh`, then `/job/create` + `/stage/clear` + `/job/resume` for chapter
|
||||
`7c944dd4-e972-42c7-ba60-9f6939548e80` of "Teto X Egen" as job `778297bc-e7ce-439d-91b5-8a027060d17f`.
|
||||
Outcome: partial. Storage split landed and is proven by the run. The run itself was still in `direct`
|
||||
when the session ended.
|
||||
Produced: `decisions/storage-layout.md`, `caveats/speaker-attribution.md`, 109 orchestrator tests pass.
|
||||
|
||||
Artifacts now split one bucket per class instead of everything under `manga`
|
||||
(`decisions/storage-layout.md#bucket-per-artifact`). Both MinIO and rustfs hold all six buckets. The
|
||||
run put 79 pages in `raw` and 116 panel crops in `panels`, so the split works end to end.
|
||||
|
||||
Two containers on homesrv had been dead for two weeks and blocked the work. `manga-fetch` was exited,
|
||||
so `/job/create` failed with `httpx.ConnectError`; `manga-web` was exited, so `manga.kvmx.ru` had
|
||||
nothing behind it on port 8083. Both started with `docker compose up -d`. Neither is related to the
|
||||
storage change. Neither was caught by any check, because nothing watches these containers.
|
||||
|
||||
Stage timings, 116 panels: crop 85s, vision ~4min, identity ~1min, reconcile ~7min for 35 pairs,
|
||||
dialogue ~8min. Faster than the 2026-07-17 run at 75 panels. The webtoon crop that 500'd in July
|
||||
succeeded this time.
|
||||
|
||||
Quality cross-check against the panel images, the point of the run. Dialogue text extraction is
|
||||
accurate. Character detection is accurate. Speaker attribution is not: three of three sampled
|
||||
two-character panels attribute both speakers to the wrong person, always swapped
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`). 24 of 81 speech lines resolve to a named
|
||||
character, which is the Phase 1 headline metric at 30%, and the sample says that 30% is not
|
||||
trustworthy. 26 of 113 detected people got an identity, and 25 of those 26 went to one character that
|
||||
turns out to cover two different women.
|
||||
|
||||
The run then reached `scene` 116/116 and failed in `script` at 87/116, not on OOM: 28 beats were
|
||||
rejected by the script verifier as `unsupported-proper-noun: ['Choi', 'Haeseon']`
|
||||
(`caveats/speaker-attribution.md#multiword-name-verifier`). No two-word cast name can pass that check.
|
||||
|
||||
## 2026-08-11 Speaker provenance and the multi-word cast name
|
||||
|
||||
Command: `.venv/bin/python worker_vision.py`, `pytest -q --ignore=test_api.py` in the orchestrator.
|
||||
Outcome: both pass, 110 orchestrator tests. Nothing deployed, no GPU work, no pipeline run.
|
||||
Produced: `decisions/speaker-attribution.md`, two caveats rewritten.
|
||||
|
||||
`_annotate_speaker_methods` stopped stamping `tail` on a model guess. With two or more characters
|
||||
present the guess is dropped to `unknown` at confidence 0.0. With one present it is kept as
|
||||
`model_solo` at 0.7, the same claim the solo backstop already makes
|
||||
(`decisions/speaker-attribution.md#no-fake-tail`). Grounded `som_face` and `solo_prior` rows are
|
||||
untouched. Nothing outside `worker_vision.py` reads the literal `tail`, checked across both repos.
|
||||
|
||||
`verify_script` now tokenizes each cast name into `allowed`, so `Choi Haeseon` passes as two tokens
|
||||
(`decisions/speaker-attribution.md#multiword-cast-names`). That is the 28 beats job `778297bc` lost.
|
||||
|
||||
The remaining `['Blur']` beat is a true positive that still halts the whole chapter, now recorded as
|
||||
`caveats/speaker-attribution.md#one-word-halts-chapter`.
|
||||
|
||||
Neither fix is live. The orchestrator container is not rebuilt and the workers are not restarted.
|
||||
|
||||
## 2026-08-11 Rerun from dialogue: the honest speaker number is 9%
|
||||
|
||||
Command: `/job/cancel`, `/stage/clear dialogue`, `./start_workers.sh`, `/job/resume` on job
|
||||
`778297bc-e7ce-439d-91b5-8a027060d17f`, twice. `docker compose up -d --build orchestrator` three times.
|
||||
Outcome: `dialogue` 116/116. 112 orchestrator tests pass, `worker_vision.py` self-check passes.
|
||||
|
||||
The named-speaker share is 9%, 9 of 95 speech lines, down from a reported 30% that counted fake tails.
|
||||
Multi-character panels contribute 0 of 40 lines by design. Single-character panels give 9 of 55. All 9
|
||||
binds are `Choi Haeseon`, the row that covers two different women.
|
||||
|
||||
Five defects, four of them found by measuring the run rather than by reading code.
|
||||
|
||||
1. The fake `tail` label, fixed before the run (`decisions/speaker-attribution.md#no-fake-tail`).
|
||||
2. The multi-word cast name in the script verifier
|
||||
(`decisions/speaker-attribution.md#multiword-cast-names`).
|
||||
3. `/stage/clear dialogue` deleted nothing and reported success. dialogue and direct write onto the
|
||||
per-panel vision blob and had no `_STAGE_TABLES` entry, so `run_stage_dialogue` saw
|
||||
`"dialogue" in vision` and would have skipped all 116 panels. The proof is the second clear:
|
||||
116 dialogue blobs and 75 direct blobs stripped that the first had left. This is
|
||||
`caveats/audit-open.md#dishonest-clearing` firing exactly where it was filed.
|
||||
4. gemma answers the speaker field with whatever the prompt showed, most often the character
|
||||
description, and every such answer became a free-form name that no registry entry matched. 28 of 51
|
||||
sampled lines (`decisions/speaker-attribution.md#prompt-label-answers`). After the fix, 3 of 95.
|
||||
5. All 7 `som_face` lines pointed at a mark whose face paired to no present character, so the
|
||||
highest-trust provenance sat on a line with no speaker. Same defect class as the fake tail.
|
||||
|
||||
Identity is now the binding constraint, not attribution. 26 of 113 detected people carry an identity,
|
||||
23%, and 25 of the 26 are the one over-merged row. Even perfect balloon binding caps this chapter near
|
||||
23% named. The person who does hold an identity is stored as `Lim Seonho` while a separate row is named
|
||||
`Seonho` with alias `Lim Seonho`, so either name matches two rows, raises `ambiguous-speaker` and binds
|
||||
nothing.
|
||||
|
||||
Two more defects surfaced after `dialogue` finished. All 7 `som_face` lines pointed at a mark whose face
|
||||
paired to no present character (`decisions/speaker-attribution.md#unpaired-mark`). Then `script` halted
|
||||
at 112/116 because the narrator wrote `"...Hm?"` for the source line `"Uh... hum...?"`, and both verifier
|
||||
rules fired on that two-letter interjection
|
||||
(`decisions/speaker-attribution.md#interjection-false-positive`). After the fix, `script` passed 116/116,
|
||||
the first time this chapter has cleared the verifier. `tts` then ran for the first time.
|
||||
|
||||
The run then completed end to end for the first time: `tts` 116/116, `layers` 116/116, `render` 116/116,
|
||||
`assemble` 1/1, finished 2026-08-11T20:08:16Z. `s3://video/` holds 49 clips and a 50MiB `chapter.mp4`,
|
||||
`s3://audio/` 49 objects at 32MiB. The per-artifact bucket split is now proven for every class except
|
||||
layers (`decisions/storage-layout.md#bucket-per-artifact`).
|
||||
|
||||
Two honesty defects surfaced at the finish, both recorded rather than fixed. `layers` reported
|
||||
`completed 116/116` with an empty bucket, and the completed job still carries
|
||||
`error: "partial: 112/116 completed"` from the failure three resumes earlier.
|
||||
|
||||
## 2026-08-12 — the video got watched
|
||||
|
||||
No pipeline ran. The user watched `chapter.mp4` for the first time and read out 19 timestamped defects.
|
||||
That found more than the previous four sessions of measuring, because the recorded metrics were all
|
||||
measuring whether code ran rather than whether the result was right.
|
||||
|
||||
Two measurements came out of it. First, `chapter.mp4` is video 436.39s over audio 363.67s, so the
|
||||
narration finishes 72.7s before the picture and the gap accumulates. The 49 clips are clean: every one is
|
||||
25fps exactly, video and audio agree to 0.03s, and they sum to 363.6s. Assembly adds 72.7s of video and
|
||||
no audio. Second, this manga holds 19 character rows of which 3 carry a name, and `Choi Haeseon` holds 25
|
||||
of the chapter's 26 identity assignments. That is why the video calls the colleague Choi, never names the
|
||||
MC, and flips gender.
|
||||
|
||||
The A/V bug was narrowed with a per-round probe over the 49 real clips. Round 0 of `_assemble_batched` is
|
||||
correct, losing only the xfade overlap per group. Round 1 turns 359s of video into 100s while the audio
|
||||
survives at 358.79s. The round-1 filtergraph is arithmetically correct, and re-running the same chain by
|
||||
hand over only the 6 encoded intermediates gives a correct 348.24s with no warnings. Round 1 differs by
|
||||
holding a 7th input: the leftover 49th clip, which `_assemble_batched` passes through un-encoded. That
|
||||
passthrough is a third path beside `concat` and `xfade` and is the prime suspect.
|
||||
|
||||
`worker_render.py` gained an `FPS = 25` constant, fps normalization in the xfade branch to match the
|
||||
concat branch, a `_stream_dur` helper, and a self-check that compares video against audio rather than
|
||||
asserting the file is non-empty. The old check only asserted `getsize(out) > 0`, which is how a 20% sync
|
||||
failure shipped. Pinning `-r FPS` on the output encodes was tried and reverted: it collapsed the chapter
|
||||
to exactly 100.00s by dropping frames to force CFR, which the comment at the concat branch already
|
||||
warned about. None of it is committed and none of it fixes the chapter yet.
|
||||
|
||||
The fps inconsistency between the two branches is real but not proven to be the shipped cause. The scene
|
||||
graphs hold 356 `cut` against 6 `fade_black`, so the real run's final round most likely stayed on the
|
||||
concat branch where no mixing happens.
|
||||
|
||||
### Panel 7, checked against the art
|
||||
|
||||
The same day, the user pulled up panel 7 and checked every detection by eye. It overturned the framing
|
||||
this file carried an hour earlier, and it overturned two theories I proposed before being corrected.
|
||||
|
||||
Panel `7c944dd4-e972-42c7-ba60-9f6939548e80_p007`, a wide establishing shot of an office through a
|
||||
window, crop 900x1650. Vision emitted 6 characters. Zero of the two identity bindings are correct and the
|
||||
one character who matters is unbound. `person_5`, described as "yellow sweater", is Seonho in the
|
||||
foreground and got no identity. `person_6` is the colleague, who has no name in the story, and was
|
||||
assigned `Choi Haeseon` at 0.9. `person_2` is a background extra and was assigned `Lim Seonho` at 0.9.
|
||||
`person_1` is a window frame with nobody in it. `person_3` and `person_4` are background extras.
|
||||
|
||||
Three defects stack, recorded as `caveats/speaker-attribution.md#bbox-wrong-space`,
|
||||
`#no-anonymous-identity` and `#extras-as-cast`. The `bbox` values are consumed as absolute pixels, and on
|
||||
this panel that puts all six boxes in the top third with two inside a speech balloon. Divided by 1000
|
||||
four of the six fit tightly. Identity therefore embedded crops of balloon edges and window frames, which
|
||||
is how a 0.9 confidence lands on the wrong person. Blank crops embed alike, a plausible mechanism for one
|
||||
row absorbing 25 of 26 assignments.
|
||||
|
||||
Two claims I made and had to withdraw. First, that rescaling by 1000 makes the boxes correct: after
|
||||
scaling, `person_1` still sits on an empty window frame and `person_6` clips its subject, and the
|
||||
descriptions are unreliable anyway, since `person_6` reads "white shirt" for a green dress. Second, that
|
||||
the constraint is 16 nameless rows needing names. The opposite is true. The pipeline mints names onto
|
||||
people who have none, and at least one nameless row is a real recurring person who should stay nameless.
|
||||
|
||||
The "26 of 113 detected people carry an identity" figure that framed the roadmap counted mostly
|
||||
background extras. It should not be quoted again.
|
||||
|
||||
## 2026-08-12, chapter assembly, root cause and fix
|
||||
|
||||
Reproduced the A/V collapse offline with 49 synthetic clips at `ASSEMBLE_BATCH=8` and six `fade_black`
|
||||
boundaries. It came out worse than the shipped run: **two round-0 groups of 8 fresh clips collapsed on
|
||||
their own**, so the single-item passthrough theory from yesterday is dead
|
||||
(`decisions/chapter-assembly.md#passthrough-innocent`).
|
||||
|
||||
Bisected one collapsing group by truncating the chain stage by stage:
|
||||
|
||||
```
|
||||
k=7 out= 52.52 correct
|
||||
k=8 out= 52.52 the last xfade contributed nothing
|
||||
[v6][n7]xfade=duration=0.050:offset=52.500 <- [v6] is 52.52s long, 0.02s of margin
|
||||
```
|
||||
|
||||
`_xfade_chain` took its durations from `_audio_dur`, which is `format=duration`, which is
|
||||
`max(video, audio)`. Each clip's audio outlasts its video by about a frame, so the offset accumulator
|
||||
crept ahead of the picture. Once the creep passed the transition width, xfade emitted the transition and
|
||||
threw away the second input and every clip after it, at `rc 0` with nothing on stderr.
|
||||
|
||||
Fix: offsets come from `min(_stream_dur(v), _stream_dur(a))`, every input is floored to a whole frame
|
||||
count and `trim`/`atrim`ed on both streams, and `_check_assembled` now verifies each encode against the
|
||||
predicted timeline instead of trusting the exit code
|
||||
(`decisions/chapter-assembly.md#offsets-from-min-stream`, `#check-assembled`).
|
||||
|
||||
Verified on the 49 real clips of chapter `7c944dd4`, re-downloaded from MinIO:
|
||||
|
||||
```
|
||||
before r1 n=7 XFADE in v=359.29 a=359.60 -> out v= 99.96 a=358.79
|
||||
after r1 n=7 XFADE in v=359.61 a=359.62 -> out v=358.76 a=358.76
|
||||
chapter v=358.76 a=358.76 gap=+0.00 (shipped: v=436.39 a=363.67 gap=+72.72)
|
||||
```
|
||||
|
||||
`worker_render.py` `__main__` passes. Two checks were added there, because the existing 4-clip A/V assert
|
||||
passed all the way through the broken build. One asserts the frame-exact `trim` on both streams, one
|
||||
assembles three clips whose audio outlasts their video by 0.4s. Mutation-tested by putting `_audio_dur`
|
||||
back: the new check fires with `video=1.80 audio=3.56 expected=3.56`.
|
||||
|
||||
Not done: `s3://video/.../chapter.mp4` is still the broken 436s file. Rebuilding it means clearing the
|
||||
`assemble` stage and resuming, which is CPU-only and was not run.
|
||||
|
||||
## 2026-08-12, the chapter rebuilt, and the bbox space settled
|
||||
|
||||
**The rebuild came out byte-identical to the broken file.** Clearing `assemble` and resuming produced
|
||||
video 436.392031s over audio 363.674667s and `nb_frames` 9902 again, which proved the xfade fix committed
|
||||
earlier today never runs for this chapter. With all-`cut` transitions `assemble` takes the `else` branch,
|
||||
a `concat` demuxer with `-c copy`.
|
||||
|
||||
Reproduced that path offline in seconds and got the shipped numbers exactly. The cause is mixed frame
|
||||
rates: 14 of the 49 clips are `r_frame_rate=30/1` at `time_base=1/15360`, the other 35 are `25/1` at
|
||||
`1/12800`. `-c copy` writes the output in the first input's timebase, so those 14 play `15360/12800 = 1.2`
|
||||
too long with their audio untouched. `collage_cmd` hardcoded `-r 30`, which yesterday's `FPS` sweep
|
||||
missed. `decisions/chapter-assembly.md#mixed-rate-stream-copy`.
|
||||
|
||||
Fixed `collage_cmd` to emit `-r FPS`, and made `assemble` probe `r_frame_rate` across the clips and route
|
||||
mixed rates through the re-encoding tree. Rebuilt:
|
||||
|
||||
```
|
||||
before v=436.392 a=363.675 nb_frames=9902 avg_frame_rate=22.69
|
||||
after v=364.120 a=364.122 nb_frames=9101 r=25/1
|
||||
```
|
||||
|
||||
The 14 clips in the bucket are still 30fps. Assembly normalizes them, so the chapter is correct without
|
||||
re-rendering, but the fast stream-copy path stays disabled for this chapter until `render` re-runs.
|
||||
|
||||
**The `bbox` space is 0-1000, not pixels.** Pulled all 113 detections from `/review/identity` and
|
||||
measured: 47 boxes have `x2` past the 900px panel width, none has `y2` past 1000 on panels 1257 to 2307px
|
||||
tall, 21 clamp at exactly 1000 in x, and the whole range is `[0, 1000]`. `/vision` now converts to pixels
|
||||
before returning, so identity crops, gated face pairing, the set-of-mark boxes and the review UI all read
|
||||
pixels (`decisions/identity-bbox.md#bbox-is-normalized`).
|
||||
|
||||
Checked by eye the way the user did. Drew the converted boxes on panel 7: five of six land on their
|
||||
subject, including `person_5`, who is Seonho in the foreground with headphones and carried no identity.
|
||||
`person_1` still frames an empty window mullion, which is the extra-versus-cast caveat, not this one.
|
||||
|
||||
Not done: `vision` and `identity` have not re-run, so every box, embedding and `ref_image_uris` in the
|
||||
registry is still from the wrong space. That rerun is GPU work and was not started.
|
||||
|
||||
## 2026-08-12, third session — the rerun, and the run that did not count
|
||||
|
||||
Asked: move on from the chapter rebuild. Scope chosen mid-session: reset the registry, rerun vision,
|
||||
identity and reconcile only, and inspect before spending the ~80 minutes downstream.
|
||||
|
||||
**The registry had to be wiped first, and nothing could do it.** `clear_stage_data` maps `identity` to
|
||||
`identity_assignments` only and spares `characters` on purpose (`db.py:790`), because the registry is
|
||||
cross-run. So a rerun would have matched fresh crops against 19 stale grid-space embeddings and kept the
|
||||
duplicate `Lim Seonho` / `Seonho` rows that raise `ambiguous-speaker`. Added `db.reset_registry` and
|
||||
`POST /characters/reset` (`confirm=true` required), which also clears `identity_assignment_sources` —
|
||||
untouched by any stage clear, and a leftover `manual` row there makes `assign_identity` refuse the next
|
||||
model assignment (`db.py:663`). Covered by `test_db.py:TestResetRegistry`; 115 orchestrator tests pass.
|
||||
The orchestrator image bakes its source, so it needed `docker compose up -d --build orchestrator`.
|
||||
|
||||
**The first full cycle was void: the worker was serving pre-fix code.** vision + identity + reconcile
|
||||
completed 116/116/20 and reproduced the defect exactly — 46 of 110 boxes past the 900px panel width,
|
||||
coordinates clamping at 1000. The vision worker started 12:00:09, `worker_vision.py` changed 12:11:35,
|
||||
and the commit carrying `_bbox_to_pixels` landed 12:16:22. Python had already bound the old module.
|
||||
Nothing in the stage output said so (`decisions/identity-bbox.md#stale-worker-invalidates`). Added
|
||||
`check_stale.sh`, which compares every worker's process start against its module mtime and exits
|
||||
non-zero; mutation-tested by touching `worker_tts.py`.
|
||||
|
||||
**The rerun against restarted workers.** 8 minutes for all three stages.
|
||||
|
||||
```
|
||||
before stale run after
|
||||
x2 past panel width 47/113 46/110 0/110
|
||||
coords exactly on 1000 21 27 1
|
||||
max y2 1000 1000 2307
|
||||
characters 19 8 8
|
||||
coverage 26/113 = 23% 77/110 = 70% 77/110 = 70%
|
||||
top character's share 96% Haeseon 47% 47%
|
||||
```
|
||||
|
||||
`Choi Haeseon`, which had absorbed 25 of 26 assignments, no longer exists in the registry. On panel 7
|
||||
Seonho is bound for the first time, and the unnamed colleague took an anonymous id instead of being
|
||||
called `Choi Haeseon` at 0.9.
|
||||
|
||||
**Fixing the boxes made the extras problem worse.** With crops finally landing on their subjects, a
|
||||
background extra bound to `Seonho` at confidence 1.00, putting an extra into the lead's reference set.
|
||||
`/vision` now stamps `has_face` per character via `face_detect` + `_pair_faces_to_present`, and identity
|
||||
skips `has_face is False` before it crops or embeds
|
||||
(`decisions/identity-bbox.md#face-gates-enrollment`). Fails open on a missing or raising detector.
|
||||
Self-checked in both workers. **Not yet proven on a GPU run** — that is the next command.
|
||||
|
||||
Deleted along the way and not recoverable: the rebuilt `chapter.mp4` and all 49 clips, by the cascade
|
||||
from `/stage/clear vision`. The user chose not to keep a copy.
|
||||
|
||||
## 2026-08-12, fourth session — identity 2b, the resolver NONE branch
|
||||
|
||||
Asked: "how much will `has_face` help with character and identity problems?", then "fix 2b first".
|
||||
|
||||
Answered the first honestly: `has_face` reaches 2 of 6 detections on panel 7 and nothing else. It does not
|
||||
touch naming or merging, and it cannot touch the chibi at 1:35, because an anime face detector detects a
|
||||
chibi face. Also flagged its real cost: the model is face-only by design, so back-turned cast lose
|
||||
enrollment along with the extras, and coverage is where that shows up first.
|
||||
|
||||
Measured before writing anything, read-only, no GPU:
|
||||
|
||||
```
|
||||
registry: 8 characters, 1 named -> ['Seonho']
|
||||
detections: 110 assignments: 77 = 70% coverage
|
||||
spread: Seonho 36, character_565c88 24, character_759e23 9, character_f7a4fd 3,
|
||||
character_25f682 3, character_d72710 1, character_823aba 1
|
||||
```
|
||||
|
||||
That killed the assumed cause. Anonymous ids already recur, so the identity worker's own
|
||||
pending-promote path gives stable anonymous identities. The defect was elsewhere.
|
||||
|
||||
`/vision/resolve` can answer "none of these" and always could: `worker_vision.py:1071` returns
|
||||
`state="new"` for `choice: 0` and `state="unresolved"` for an out-of-range index. `service.py` read only
|
||||
`character_id` and unassigned every crop of the tracklet for either. The stale `ponytail:` comment above
|
||||
that block named the real blocker and was right: minting needs an `embedding_uri` the orchestrator cannot
|
||||
compute, since siglip and gemma cannot both be resident.
|
||||
|
||||
Fixed by carrying the embedding, not by adding a GPU pass. `/identity/resolve` writes each crop's
|
||||
embedding to the crop's key with a `.npy` suffix and returns `emb_uri`. `tracklets.resolve_outcome` holds
|
||||
the three-way decision as a pure function. `service.py` mints via the existing `create_character` and
|
||||
falls into the existing assign loop. `decisions/identity-bbox.md#none-mints-an-anonymous-character`.
|
||||
|
||||
Checks: `worker_identity self-check ok`, `tracklets self-check ok`, 115 passed on homesrv.
|
||||
Deployed: image rebuilt, `resolve_outcome` verified inside the running container, `audit_registry.py`
|
||||
re-copied after the recreate. Vision and identity restarted, `./check_stale.sh` exits 0.
|
||||
|
||||
New trap: `tmux respawn-window -k` does not re-run the window command. It leaves a bare shell and the
|
||||
worker down. Both workers were dead for two minutes before `/health` caught it.
|
||||
|
||||
Not run: the GPU cycle. `has_face` and the NONE mint are both unproven on real panels and now land in the
|
||||
same run.
|
||||
|
||||
## 2026-08-12, fourth session, continued — identity 2c, extras versus cast
|
||||
|
||||
Asked: "fix extra-vs-cast?"
|
||||
|
||||
Traced the three places that read `vision["characters"]` raw before changing anything. That corrected an
|
||||
inference made earlier in the session. `build_scene` already drops an unassigned detection from
|
||||
`characters` and `present` (`worker_scene.py:63`), so extras never reached the cast list at all.
|
||||
|
||||
The leak was their ACTIONS. `actions` was built from every detection, and that list is what the script
|
||||
prompt renders and what the correctness verifier uses as evidence. So a background extra's "standing at the
|
||||
window" arrived as a fact about the panel with no character attached, and the verifier confirmed it,
|
||||
because the action really was in the blob. That is a second mechanism behind the invented-narration
|
||||
complaints in item 3, independent of the model inventing anything.
|
||||
|
||||
Gated two consumers on `has_face is False`, matching the enrollment gate's semantics exactly:
|
||||
`worker_scene`'s `actions`/`action`, and `service._beat`, which picks the director's "who" from the first
|
||||
three detections and falls back to an action when a detection has no name.
|
||||
|
||||
Left `service._present_characters` ungated on purpose. It builds the dialogue stage's candidate speakers
|
||||
and the set-of-mark boxes. An extra picked as speaker already resolves to unknown, not to a wrong name, so
|
||||
the failure is contained. And the gate's cost lands hardest there, since a character drawn from behind has
|
||||
no face box and gating would delete a real speaker from the only list that can attribute their line.
|
||||
`decisions/identity-bbox.md#extras-gate-consumers`.
|
||||
|
||||
Checks: `worker_scene self-check ok` with three cases (cast action survives, faceless dropped, missing key
|
||||
survives), 115 passed on homesrv. Deployed: image rebuilt, `_beat` verified inside the container, scene
|
||||
worker restarted, `./check_stale.sh` clean. Committed `ca46617` and `8b27aec`.
|
||||
|
||||
Still not run on a GPU. Three changes now ride the same cycle: `has_face`, the NONE mint, and this.
|
||||
|
||||
## 2026-08-12, fourth session, continued — identity 2d, merge and split
|
||||
|
||||
Asked: "2d? merge and split?"
|
||||
|
||||
Checked the registry before planning anything, and 2d as written is stale. The registry reset earlier today
|
||||
deleted both worked examples. There is no `Lim Seonho` to merge into, `character_afa7623b` does not exist,
|
||||
and the current registry is 8 rows with one named character (`Seonho`). Nothing to merge or split until the
|
||||
rerun mints a new set.
|
||||
|
||||
So the useful work was the safety net for that rerun, since `reconcile` runs inside it. The caveat's cost
|
||||
line was the reason: one bad merge was unrecoverable without rebaselining the whole manga, and the cycle
|
||||
about to run includes a merge pass over embeddings nobody has seen yet.
|
||||
|
||||
`merge_characters` no longer deletes the loser. It sets `merged_into = keeper`, so the row keeps its
|
||||
embedding, description and gender, and it stamps every repointed assignment with
|
||||
`method = merged_from:<loser_id>` in `identity_assignment_sources`. That reuses a free-form column on a
|
||||
table that already existed rather than adding a merge log. `source` is left alone, so a reviewer's `manual`
|
||||
assignment keeps its veto in `assign_identity` after being repointed.
|
||||
|
||||
Roster readers filter `merged_into IS NULL`, including the name-dedup in `create_character`, which would
|
||||
otherwise fold new crops back into a character reconcile had retired. Lookup by id does not filter, because
|
||||
an assignment or a narration reference may still point at a merged id.
|
||||
|
||||
Two existing assertions asserted the old destructive behaviour (`test_db.py:235`, `test_merge_refs.py:37`)
|
||||
and were rewritten, not deleted: the invariant changed on purpose.
|
||||
|
||||
Checks: 116 passed on homesrv, up from 115. Additive `ALTER TABLE` through the existing `init_db` migration
|
||||
block, verified on the live database (`merged_into` present, 0 rows merged). Committed `00096cc`.
|
||||
|
||||
Deliberately not built: the unmerge path and the split. No wrong merge has been observed since the crops
|
||||
were fixed, so the consumer of these records waits for one. The forward case is partly covered by 2b, since
|
||||
a resolver NONE now mints instead of folding a stranger into the nearest match.
|
||||
|
||||
## 2026-08-12, fifth session — the GPU cycle all four changes were waiting for
|
||||
|
||||
Asked: "go ahead" on the rerun.
|
||||
|
||||
First correction: the vision/identity/reconcile timestamps in `/job/status` are UTC and the git log is
|
||||
local (UTC+4), so the run that looked like a rerun was the pre-change baseline. `_mark_has_face` landed at
|
||||
19:07 local, the run finished 13:17 local. The handoff was right that nothing had run.
|
||||
|
||||
Reset the registry for `ef105a86` (8 characters, 77 assignments, 101 sources, 142 S3 objects), cleared
|
||||
`vision` and everything downstream (116 vision results), restarted the identity worker for its in-process
|
||||
known-character cache, then ran vision, identity and reconcile. 6 minutes wall: vision 3m48s, identity
|
||||
1m25s, reconcile 48s. All three `completed`, 116/116/16.
|
||||
|
||||
### What the four changes did
|
||||
|
||||
| change | verdict |
|
||||
| --- | --- |
|
||||
| `has_face` stamp | works, present on all 110 detections, 67 true / 43 gated (39%) |
|
||||
| extras gate | works, panel 7's two wrong bindings are gone |
|
||||
| NONE mints | works, 16 characters minted where 8 existed |
|
||||
| non-destructive merge | works, 7 rows carry `merged_into`, 9 assignments stamped `merged_from:` |
|
||||
|
||||
Coverage went 70% -> 61% (77 -> 67 assignments over the same 110 detections). That is the gate's price and
|
||||
it is mostly correct, see below.
|
||||
|
||||
### Panel 7, the worked example, before and after
|
||||
|
||||
```
|
||||
person_1 Seonho 1.00 -> -- none -- has_face=False
|
||||
person_2 character_f7a4fd 0.00 -> character_519d2b 0.00 has_face=True
|
||||
person_4 character_d72710 0.94 -> -- none -- WRONG binding removed
|
||||
person_5 Seonho 1.00 -> -- none -- WRONG binding removed
|
||||
```
|
||||
|
||||
The two wrong bindings the gate existed to kill are dead. It also took `person_1`, and that is correct:
|
||||
ran `face_detect.detect_faces` on p006.png directly and it finds exactly one face on the whole panel, conf
|
||||
0.599, inside `person_2`. Swept the threshold to 0.04 and nothing else appears above 0.056. Looked at the
|
||||
crop: `person_1` is the lead drawn from behind at his desk, back of the head and headphones, no face in
|
||||
frame. The detector is right and the gate is right. The cost is that a back-turned character cannot be
|
||||
identified from that panel at all.
|
||||
|
||||
Also worth recording: `person_1`'s new box frames the lead correctly. The old note that it framed an empty
|
||||
window mullion was written against the pre-fix boxes.
|
||||
|
||||
### The new defect: a roster hint named the wrong man
|
||||
|
||||
The registry holds three real people and one wrong name.
|
||||
|
||||
- `character_92129ac7` "Lim Seonho", 22 assignments. Verified against the art: the p010 introduction panel
|
||||
captioned `LIM SEONHO (29)`, yellow plaid shirt and headphones, which is the roster's description for
|
||||
"Seonho". This is the lead and the name is right.
|
||||
- `character_556aef60`, 25 assignments, unnamed. Verified: the woman with short black hair and pearls, the
|
||||
roster's second character, whom the roster itself calls "Unknown".
|
||||
- `character_dbadfff7` "Seonho", 15 assignments. Verified: a different man, glasses, dark clothing, in
|
||||
profile. He carries the lead's roster name.
|
||||
|
||||
p020 assigns `Seonho` to `person_1` and `Lim Seonho` to `person_3`, so the pipeline holds them as two
|
||||
people in one panel. inference: the roster hint injected at `service.py:882` puts a name in front of
|
||||
detection, and detection attached it to the wrong face before the caption panel could mint the real one.
|
||||
This is not the alias-merge case and merging the two rows would be wrong. It needs name binding to require
|
||||
evidence, the way `name_claims` already does for captions and address.
|
||||
|
||||
### Two smaller things the run exposed
|
||||
|
||||
- **Merge chains.** `character_e1ab7776 -> character_521c301f -> character_556aef60`. `merged_into` points
|
||||
at a row that is itself merged, so a single-hop resolve lands on a retired character. Roster readers are
|
||||
fine because they filter `merged_into IS NULL`. Anything that follows one hop is not.
|
||||
- **Orphan `name_claims`.** All 5 rows point at `character_afa7623b`, which the reset deleted. The reset
|
||||
clears characters and assignments but not claims.
|
||||
|
||||
Not run: dialogue and everything downstream. The job sits at `dialogue waiting`.
|
||||
|
||||
### Same session — the three fixes the run's evidence asked for
|
||||
|
||||
All three are orchestrator-side, so no worker changed and `check_stale.sh` is not in play.
|
||||
|
||||
- **Roster hints no longer seed detection.** Deleted the two lines at `service.py:882` that appended
|
||||
`_roster_char_hints` to `known_characters`. `build_detect_prompt` drops any hint without a name, so a
|
||||
nameless hint would have contributed nothing anyway. Names now reach detection only from registry rows,
|
||||
which are embedding-backed and were themselves named from an in-panel caption or address. The roster
|
||||
still feeds `roster_cast` in `run_stage_dialogue`, where names are matched against speech.
|
||||
- **`merge_characters` resolves the keeper to its chain root** before merging, so `merged_into` stays one
|
||||
hop deep. Bounded 64-step walk, the cap only exists so a cycle cannot hang reconcile.
|
||||
- **`reset_registry` deletes `name_claims`** for the manga's panels and reports the count.
|
||||
|
||||
Checks: 117 passed on homesrv, up from 116. The new chain test was confirmed to fail with the walk
|
||||
disabled, then the walk was restored and the suite re-run. Deployed by rebuilding the image; verified
|
||||
inside the running container by parsing the deployed source, not by grepping it, because the first check
|
||||
matched the word `_roster_char_hints` inside the comment that explains its removal.
|
||||
|
||||
Also copied `audit_registry.py` into the repo before the rebuild. It had only ever been `docker cp`'d, so
|
||||
every rebuild dropped it. `Dockerfile` has `COPY . .`, so it is baked now and the trap is gone.
|
||||
|
||||
Open risk on the next run: without the roster hint the lead is unnamed until the p010 caption, so panels 1
|
||||
to 9 may mint him as an anonymous character that reconcile then has to merge. Watch whether "Lim Seonho"
|
||||
comes back as one row or two.
|
||||
|
||||
### Same session — run B, 17:38-17:44
|
||||
|
||||
Reset (16 characters, 67 assignments, 67 sources, **5 name_claims**, 156 S3 objects), cleared `vision`,
|
||||
restarted the identity worker, ran the three stages. The `name_claims: 5` line is the orphan fix confirmed
|
||||
on live data.
|
||||
|
||||
117 detections, 59 assignments, 50% coverage, 20 characters, 72 with `has_face` (45 gated).
|
||||
|
||||
The registry split is fixed. "Lim Seonho" is one row with 25 assignments, "Seonho" fell from 15 to 1. So
|
||||
dropping the roster hint did not split the lead across the panels before his caption, which was the risk
|
||||
recorded before the run.
|
||||
|
||||
Two findings, both now filed:
|
||||
|
||||
- detection still names from in-panel text and can attach the name to the wrong body (p011, p026), which
|
||||
is `decisions/identity-bbox.md#roster-does-not-name`'s "not covered" paragraph
|
||||
- one degenerate bbox, `caveats/audit-open.md#degenerate-bbox`
|
||||
|
||||
And one fix of my own that was incomplete: the merge chain came back as
|
||||
`477c1894 -> a92d9df4 -> 4fb94c15` with the keeper walk deployed. The walk only sees the pair being merged
|
||||
now. Retiring a row that is already somebody's keeper needed the second half, a repoint of the loser's
|
||||
dependents. Both halves and both tests are in `decisions/identity-bbox.md#merge-chains-flatten`.
|
||||
|
||||
Checks: 118 passed, was 116. Each new merge test was run with its fix disabled and confirmed to fail
|
||||
first. Deployed by rebuild and verified by parsing the container's source.
|
||||
|
||||
Coverage is now 70 -> 61 -> 50 across three runs and is the open question, recorded as
|
||||
`caveats/audit-open.md#coverage-trend`. `identity_labels` already holds 145 rows of ground truth, so the
|
||||
next move is to score with `eval_identity.py` rather than to keep reading the coverage number.
|
||||
|
||||
## 2026-08-12, sixth session — the ground truth is for the wrong manga
|
||||
|
||||
No GPU work. Two things, both cheap.
|
||||
|
||||
**`_bbox_to_pixels` orders its corners.** Each coordinate pair is sorted after clamping. The self-check
|
||||
feeds it run B's real `[226, 417, 130, 551]` and asserts `[117, 688, 203, 909]`. Unsorted the call returns
|
||||
`[203, 688, 117, 909]`, so the assert fails without the fix. `worker_vision.py` self-check passes.
|
||||
The vision worker is now STALE and needs a restart before the next cycle.
|
||||
|
||||
**Item 1 of the plan cannot run as written.** All 145 rows of `identity_labels` belong to chapter
|
||||
`8ca8249b`, a different manga with cast "Rico" and "Ikekin", spread over 81 panels. Chapter `7c944dd4`
|
||||
has none, so `/review/identity?job_id=778297bc...` returns `labeled: 0, correct: 0, accuracy: null`.
|
||||
|
||||
Scoring `8ca8249b` anyway gives 7/138, with 113 rows labelled as a real person and left unassigned. That
|
||||
chapter's identity run is stale: 44 assignments over 246 panels, predating every fix. The number measures
|
||||
old code on the wrong chapter.
|
||||
|
||||
So the eval path is proven end to end and the ground truth is absent. A precision number for the current
|
||||
registry needs a hand-labelling pass over `7c944dd4` through `POST /review/identity/label`, keyed by bbox
|
||||
rather than by `local_id`, because vision reassigns `local_id` every run.
|
||||
|
||||
## 2026-08-12 18:07-18:13 UTC, third GPU cycle
|
||||
|
||||
Restarted vision (window 3) and identity (`/characters/reset` returned `restart_identity_worker: true`),
|
||||
both by `send-keys C-c` then re-sending the launch line, never by `pgrep`. `check_stale.sh` exit 0 before
|
||||
the run. Reset dropped 20 characters, 59 assignments, 69 sources, 168 S3 objects.
|
||||
|
||||
vision 116/116 in 3m59s, identity 116/116 in 1m24s, reconcile 18/18 in 50s.
|
||||
|
||||
| metric | 17:38 run | 18:07 run |
|
||||
| --- | --- | --- |
|
||||
| detections | 117 | 119 |
|
||||
| assignments | 59 | 68 |
|
||||
| coverage | 50% | 57% |
|
||||
| degenerate boxes | 1 | **0** |
|
||||
| `has_face` true / gated | 72 / 45 | 71 / 48 |
|
||||
| assigned among face-bearing | 59/72 = 82% | **68/71 = 96%** |
|
||||
| assigned among gated | -- | 0 |
|
||||
| registry rows / merged | 20 / 7 | 18 / 9 |
|
||||
| `merged_from` stamps | 9 | 22 |
|
||||
| chains deeper than one hop | 1 | **0** |
|
||||
|
||||
The corner-ordering fix works: 0 degenerate boxes. The merge-chain fix holds under a heavier merge load,
|
||||
9 merges and no chain.
|
||||
|
||||
**Coverage is now the gate and nothing else.** Every assignment landed on a face-bearing detection and none
|
||||
on a gated one. Recall among face-bearing detections is 96%. So the 43% with no assignment is the 40% the
|
||||
gate drops plus 3 detections, and the resolver is not losing cast. That closes the part of
|
||||
`#coverage-trend` that suspected the resolver.
|
||||
|
||||
**The open question moved to precision.** The lead holds 36 of 68 assignments, 53%, against 25 of 59
|
||||
before, and `audit_registry.py` flags it as absorbing. The merge count rose from 7 to 9 over the same
|
||||
interval. Whether 36 is a protagonist in half the panels or an over-merge cannot be told apart without
|
||||
labels, which is item 1.
|
||||
|
||||
Panel 7 got worse, not better: 5 of its 6 detections carry `has_face = False`, and `person_6`, the one that
|
||||
does, went unassigned. The previous run bound its colleague. That is the cost of requiring a face, item 4.
|
||||
|
||||
## 2026-08-12, sixth session, later — the lead was over-merged, and cosine cannot fix it
|
||||
|
||||
The user checked the lead's crops in the review UI. `LIM SEONHO`'s 36 assignments cover at least six
|
||||
different men, a chibi, and a cat. So 36 of 68 is a bug, not a protagonist.
|
||||
|
||||
Split by mechanism: 22 native resolves, 14 from reconcile merges. Both are broken, for different reasons.
|
||||
|
||||
**The tracklet linker chained across the chapter.** `window=8` bounds each pair, transitivity was
|
||||
unbounded. The 22 native assignments came from 3 tracklets spanning 0, 22 and 30 panels, visible as two
|
||||
confidence blocks of eleven rows at 0.95 and ten at 1.0. Fixed by capping the merged group's span
|
||||
(`decisions/identity-bbox.md#tracklet-span-cap`).
|
||||
|
||||
**Both tokenizers linked on words that are not identity.** Whole chains hung on `short`; one pair linked on
|
||||
the word `hair` itself. There is now one tokenizer with a `GENERIC` stopword set, shared with reconcile's
|
||||
pre-filter (`decisions/identity-bbox.md#generic-tokens`).
|
||||
|
||||
Measured on the same 22 real crops, candidate overlap forced to pass: 3 tracklets at worst span 30 becomes
|
||||
9 at worst span 8.
|
||||
|
||||
**Then the obvious next fix was ruled out by measurement.** Pulled all 22 crop embeddings from MinIO and
|
||||
computed the pairwise cosine. The cat scores up to 0.82 against men, two different men score 0.93, and the
|
||||
highest pair in the matrix is 0.96. No threshold exists, so crop-to-crop cosine is not a link signal, and
|
||||
the `candidates` overlap condition is close to noise. Filed as
|
||||
`caveats/audit-open.md#cosine-not-identity`. Inference: identity embeds the whole person box, so cosine
|
||||
measures scene and style, not face.
|
||||
|
||||
Also seen: vision boxes cats as people and dresses them. `p081` and `p108` are cats, described
|
||||
`short brown / yellow plaid shirt` and `short brown / white t-shirt`.
|
||||
|
||||
Checks: `tracklets.py` self-check ok, orchestrator 118 passed. Both new asserts confirmed to fail with
|
||||
their fix disabled. Nothing deployed, nothing re-run on a GPU.
|
||||
|
||||
Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad
|
||||
only, not committed.
|
||||
|
||||
## 2026-08-12, 19:44-19:52 UTC — fourth GPU cycle: the cast is the gallery
|
||||
|
||||
Three fixes ran together for the first time: the tracklet span cap and the shared `GENERIC` tokenizer,
|
||||
both written last session and never deployed, plus the new one
|
||||
(`decisions/identity-bbox.md#cast-is-the-gallery`). Job `778297bc`, chapter `7c944dd4`, registry reset,
|
||||
vision cleared, `vision -> identity -> reconcile`. Vision 116/116 in 3m55s, identity 116/116 in 2m44s,
|
||||
reconcile 20/20 in 44s.
|
||||
|
||||
| metric | 18:07 run | 19:44 run |
|
||||
| --- | --- | --- |
|
||||
| detections | 119 | 119 |
|
||||
| assignments | 68 | 60 |
|
||||
| coverage | 57% | 50% |
|
||||
| tracklets over crops | 12 / 64 | 33 / 72 |
|
||||
| lead's assignments | 36 | 16 |
|
||||
| top character's share | -- | 16/60 = 27% |
|
||||
| characters after reconcile | 18 | 14 |
|
||||
| minted / cleared | -- | 10 / 12 |
|
||||
|
||||
**Coverage went down and that is the fix working.** gemma cleared 12 crops it used to be forced to name
|
||||
from a cosine top-k that did not contain the right person. 72 crops entered resolution against 64, because
|
||||
a crop with an empty cosine shortlist is no longer dropped.
|
||||
|
||||
**Checked by eye, which is the only check that sees this.** Contact sheets of every assigned crop, per
|
||||
character, confirmed by the user. The lead holds 16 crops. 14 are him and 2 are art inside a panel, the
|
||||
photograph at `order 17` and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and
|
||||
she is a main character the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2
|
||||
are `2b1b12a1` (`order 31`, `order 33`). Against the 18:07 run, where the lead's 36 covered six different
|
||||
men, a chibi and a cat.
|
||||
|
||||
Every wrong crop on the lead is one defect: vision treats art inside a panel as the scene. It is not a
|
||||
linker failure and no identity change will fix it.
|
||||
|
||||
**The cap fired twice and dropped the wrong rows.** `GALLERY_CAP = 10` truncated a 16-row gender-compatible
|
||||
cast at `p097` and an 11-row one at `p109`. `cast_gallery` orders named first, so what it drops is exactly
|
||||
the recently minted anonymous rows, which is the population a later tracklet most needs to match against.
|
||||
Filed as `caveats/audit-open.md#gallery-cap-drops-the-unnamed`.
|
||||
|
||||
Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestrator 118 passed,
|
||||
`./check_stale.sh` exit 0 before the cycle. Vision and identity workers both restarted, the second because
|
||||
`/characters/reset` asked for it.
|
||||
|
||||
Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed.
|
||||
|
||||
## 2026-08-13 — the dialogue stage names nobody, and why
|
||||
|
||||
Ran `dialogue` 116/116 in 5m57s on the fourth cycle's registry, to see whether the fixed identity lets the
|
||||
existing `name_claims` path name the female lead. It does not, and the six claims it produced name three
|
||||
separate defects.
|
||||
|
||||
```
|
||||
p040 character_2b1b12a1 "Choi Haeseon" caption 1.00 -> NOT promoted
|
||||
p010 character_b1dd5659 "Lim Seonho" caption 1.00 -> conflict flag
|
||||
p047 character_b1dd5659 "Seonho" address 0.90 -> conflict flag
|
||||
p011 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
|
||||
p026 character_f0d4e901 "Seonho" address 1.00 -> PROMOTED
|
||||
p110 character_028d4a49 "Haeseon" address 1.00 -> already named
|
||||
```
|
||||
|
||||
All three are fixed in `db.add_name_claim` and recorded in `decisions/identity-naming.md`: alias grouping,
|
||||
a confident caption as strong evidence, and one name per character. `test_name_binding.py` replays these
|
||||
six claims, 121 tests pass, and each new assert was confirmed to fail with its fix disabled.
|
||||
|
||||
Also wired `merge_faceless_captions` into both crop endpoints. It had been written and never called; both
|
||||
endpoints called the non-destructive `context_fragment_links` instead, and no decision recorded that
|
||||
choice. It does not cover the head-in-one-shot, body-in-the-next split that prompted the question, because
|
||||
a body fragment has no text and `_merge_plan` only folds a fragment that has text and no face.
|
||||
|
||||
Found while wiring it: `crop_webtoon` skips an upload when the key exists, which is right for a resume and
|
||||
silently wrong after a slicing change. Documented at the line and in `NEXT.md`.
|
||||
|
||||
Wrote `ARCHITECTURE.md` from the user's design: region graph, occurrence/identity/name, speaker as a scored
|
||||
graph edge with a typed union, narrative plane for art-in-art, and a persistent story state machine. Every
|
||||
section carries what exists against it today. Nothing in it is built.
|
||||
|
||||
Nothing ran on a GPU after the dialogue stage.
|
||||
@@ -0,0 +1,244 @@
|
||||
# NEXT
|
||||
|
||||
Updated 2026-08-12 (sixth session). What the fifth session did is in `HANDOFF.md`, the runs are in
|
||||
`JOURNAL.md`.
|
||||
|
||||
## State
|
||||
|
||||
The chapter runs end to end. The A/V sync defect is fixed and `chapter.mp4` is rebuilt: video 364.120s
|
||||
against audio 364.122s at `25/1`.
|
||||
|
||||
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels.
|
||||
Four GPU cycles ran on 2026-08-12, the last 19:44-19:52 UTC on a reset registry. The job is `running` and
|
||||
parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are
|
||||
current.
|
||||
|
||||
Eleven changes are proven on real panels. This session added bbox corner ordering. It then deployed and ran
|
||||
the tracklet span cap, the shared `GENERIC` tokenizer and the cast gallery together.
|
||||
|
||||
**The registry is now roughly right and is worth reading.** Checked by eye, crop by crop, confirmed by the
|
||||
user. The lead holds 16 assignments. 14 are him and 2 are art inside a panel, the photograph at `order 17`
|
||||
and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and she is a main character
|
||||
the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2 are `2b1b12a1`. On the
|
||||
18:07 run the lead alone held 36, covering six men, a chibi and a cat.
|
||||
|
||||
So every wrong crop on the lead is one defect, item 7, and not a linker failure. The registry's other
|
||||
weakness is that its biggest character has no name.
|
||||
|
||||
| metric | 18:07 run | 19:44 run |
|
||||
| --- | --- | --- |
|
||||
| detections | 119 | 119 |
|
||||
| assignments | 60 | 60 |
|
||||
| coverage | 57% | 50% |
|
||||
| tracklets over crops | 12 / 64 | 33 / 72 |
|
||||
| lead's assignments | 36 | 16 |
|
||||
| characters after reconcile | 18 | 14 |
|
||||
|
||||
Coverage fell because gemma now clears 12 crops instead of naming them from a shortlist that did not
|
||||
contain the right person. Coverage is still the `has_face` gate plus those refusals, and nothing else.
|
||||
|
||||
## Next
|
||||
|
||||
1. **Re-crop the chapter and run the fifth cycle.** Four changes are written and tested since the fourth
|
||||
cycle, and none has touched a GPU.
|
||||
|
||||
- `merge_faceless_captions` is wired into both crop endpoints. It was written, never called, and
|
||||
`context_fragment_links` was called instead. A stranded caption fragment now vstacks into the
|
||||
face-bearing fragment it belongs to, so panel count and every panel index change.
|
||||
- three naming fixes in `db.add_name_claim` (`decisions/identity-naming.md`): alias grouping, a caption
|
||||
as strong evidence, and a name held by another character refusing to promote onto a second one.
|
||||
|
||||
**Clear `s3://panels/<manga>/<chapter>/panels/` before re-cropping.** `crop_webtoon` skips the upload
|
||||
when the key exists, so a re-crop after a slicing change silently keeps the previous run's images.
|
||||
Wiring the merge is a slicing change. Everything downstream is invalidated by it, so this is a full
|
||||
re-run and not a stage rerun.
|
||||
|
||||
Expected: fewer than 116 panels, `2b1b12a1` named `Choi Haeseon` from the p040 caption, the lead's
|
||||
`conflicting-name-claims` flag gone, and the green-dress woman no longer named `Seonho` but carrying a
|
||||
`name-already-taken` flag instead.
|
||||
|
||||
Not fixed by any of it. Bare hair colour still links different men. Do not add a crop-to-crop cosine to
|
||||
close that. Measured on 22 embeddings, different people reach 0.93 and the same person reaches 0.96, so
|
||||
no threshold exists (`caveats/audit-open.md#cosine-not-identity`).
|
||||
|
||||
Then, separately, test embedding the FACE box rather than the person box. `face_detect` already finds
|
||||
the face and pairs it for `has_face`. That is the likely root cause of cosine measuring scene instead of
|
||||
person. The test is to re-embed these same 22 detections and recompute the matrix.
|
||||
|
||||
1b. **The head/body split that started the crop question is NOT fixed.** The wired merge only folds a
|
||||
fragment that has text and no face. A body fragment carries no dialogue, so `_merge_plan` leaves it
|
||||
solo and it becomes its own panel and its own shot. Finding it needs a different signal, most likely a
|
||||
face touching the bottom edge of one fragment with a textless fragment below. No evidence has been
|
||||
gathered yet on how often this chapter does it.
|
||||
|
||||
1c. **`ARCHITECTURE.md` was rewritten on 2026-08-13 under the no-Magi constraint.** Every section now
|
||||
carries a DoD. Its build order starts at step 0, `eval/chapter-truth.json` plus purity and
|
||||
fragmentation printed by `audit_registry.py`. Nothing below it is measurable without that file.
|
||||
|
||||
Steps 1 and 2 are independent and cheap: `plane` and `species` per detection, and the
|
||||
face-plus-person embedding already queued in item 1 above. The weak cannot-link is step 3 and depends
|
||||
on step 2, not on `plane` alone. Co-presence as a hard constraint is load-bearing while cosine cannot
|
||||
separate people, so demoting it early regresses purity. `offscreen` as a fourth `speaker_ref` kind is
|
||||
still cheap and independent.
|
||||
|
||||
2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0
|
||||
degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`).
|
||||
3. **Vision boxes animals as people and dresses them.** `p081` and `p108` are cats, described
|
||||
`short brown / yellow plaid shirt` and `short brown / white t-shirt`. A detection prompt problem, not a
|
||||
linker one, and it feeds every stage below.
|
||||
4. **Detection still names from in-panel text and can hit the wrong body.** p011 and p026 emit
|
||||
`name: "Seonho"` on `person_2` with the roster hint gone. That is the residue of the naming defect and
|
||||
the reason the glasses man is named at all
|
||||
(`decisions/identity-bbox.md#roster-does-not-name`, "not covered").
|
||||
4. **Decide what a back-turned character costs.** This is now the whole of the coverage number. `has_face`
|
||||
gates 40% of detections. On the 18:07 run panel 7 lost 5 of its 6 detections to the gate, and its one
|
||||
face-bearing detection went unassigned. Panel 7's lead is at his desk from behind. `face_detect`
|
||||
finds one face on the whole panel at conf 0.599, and nothing else above 0.056 even at a 0.04
|
||||
threshold. The gate is right and the detector is right. The lead is still unidentifiable there. Options are a body or head detector alongside the face one, or letting the
|
||||
tracklet carry identity across a back-turned panel. Do not lower `FACE_CONF`, the sweep shows nothing
|
||||
to find.
|
||||
|
||||
Smaller follow-on: nine `_audio_dur` calls in `worker_render.py` measure finished clips with
|
||||
`format=duration`. So the durations reported to the orchestrator are blind to per-clip drift.
|
||||
They position no filter, so invariant 9 does not cover them. Worth converting to `_stream_dur`.
|
||||
5. **Fix identity, in this order.** All of 2a-2d below are done and now proven on a GPU. Kept for the
|
||||
evidence trail.
|
||||
|
||||
a. ~~Settle the `bbox` coordinate space.~~ **Done 2026-08-12**, proven over all 113 detections and
|
||||
checked by eye on panel 7, where five of six converted boxes land on their subject
|
||||
(`decisions/identity-bbox.md#bbox-is-normalized`). The rerun settles the last doubt: `person_1` now
|
||||
frames the lead at his desk, not the window mullion the pre-fix box caught.
|
||||
b. ~~Let identity abstain and stay abstained.~~ **Done 2026-08-12, run and verified on a GPU**
|
||||
(`decisions/identity-bbox.md#none-mints-an-anonymous-character`). The resolver could always answer
|
||||
"none of these". The orchestrator was discarding the answer: it read only `character_id`, so a
|
||||
deliberate NONE and a hallucinated index both unassigned every crop of the tracklet. A NONE now
|
||||
mints an anonymous character from the crop, using the embedding `/identity/resolve` ships beside
|
||||
it as `emb_uri`. Deployed: image rebuilt, `resolve_outcome` verified inside the container.
|
||||
c. ~~Separate extra from cast.~~ **Done 2026-08-12, run and verified on a GPU**
|
||||
(`decisions/identity-bbox.md#face-gates-enrollment`,
|
||||
`decisions/identity-bbox.md#extras-gate-consumers`). `has_face` stops a faceless detection
|
||||
enrolling, and two more consumers now skip it: `worker_scene`'s `actions`, which is the script
|
||||
prompt's content and the verifier's evidence, and `service._beat`, the director's "who".
|
||||
`_present_characters` stays ungated on purpose, reasoned out in the decision.
|
||||
The remaining gap is that vision still emits extras into the blob, which is deliberate so the
|
||||
audit can see what was gated.
|
||||
d. ~~Merge and split.~~ **Safety net done and now exercised.** The 18:07 run merged 9 rows and stamped
|
||||
22 assignments `merged_from:`, with no chain deeper than one hop. The non-destructive path and the
|
||||
flattening fix are both proven on real data.
|
||||
|
||||
A merge no
|
||||
longer deletes the losing row: it sets `merged_into`, and stamps every repointed assignment with
|
||||
`method = merged_from:<loser_id>`. A wrong merge now costs a hand-written SQL walk, not a full
|
||||
rebaseline (`caveats/audit-open.md#destructive-reconcile`).
|
||||
|
||||
Deliberately not built: the unmerge path and the split. No wrong merge has been observed since the
|
||||
crops were fixed, so the consumer of those records waits for one. The forward case is partly covered
|
||||
by 2b, because a resolver NONE now mints rather than folding a stranger into the nearest match.
|
||||
Splitting a character that is ALREADY over-merged still needs a re-embed pass over its detections.
|
||||
|
||||
**Cast profiles already exist. Do not rebuild them.** The user asked whether the main cast could get a
|
||||
profile built from reference frames and reused. `characters` already carries `ref_image_uris` and
|
||||
`embedding_uri`, and every row has both populated (16 rows after the rerun, 53 before it). The
|
||||
mechanism was never missing. It was enrolled from the wrong crops, so it stored balloon edges and
|
||||
window frames. Step (a) fixed that, and the audit now reports 0 characters missing a ref crop. Three things are genuinely absent and are the smaller follow-on:
|
||||
|
||||
- no quality gate on enrollment, so nothing checks that a reference crop holds a face at all
|
||||
- nothing re-enrolls a reference set once it is written, so the wrong crops persist
|
||||
|
||||
**The visual "is this them?" check is already built. Do not write it again.** `/vision/resolve` at
|
||||
`worker_vision.py:963` sends the query crop plus up to 3 labelled reference images per candidate.
|
||||
`build_resolve_prompt` tells the model to judge face shape first, to treat hair and outfit as
|
||||
secondary, that two people sharing a hair colour are not the same, and to answer `0` for NONE when
|
||||
unsure. `choice: 0` becomes a new character, an out-of-range index becomes `unresolved`, and
|
||||
`ref_image_uris` is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`. The
|
||||
mechanism, the prompt and the abstain path are all correct. They are fed crops of the wrong region,
|
||||
which is step (a).
|
||||
- no human gate to name, merge or split the clusters. The user wants this as a minor adjustment on
|
||||
top, not as the mechanism. The `gates` table and the review gates from [#136] are the place to hang
|
||||
it
|
||||
|
||||
The chibi at 1:35 will survive all of this. He genuinely is brown hair plus a yellow shirt, so a
|
||||
profile match is correct on appearance and wrong on reality. That needs item 4 below, plus requiring
|
||||
a real face before a crop can enroll.
|
||||
6. **Stop the narration inventing facts.** 0:43, 2:03, 2:05 and 2:15 assert things no panel shows. The
|
||||
correctness verifier passed 116/116 because it checks quotes and names, never invented claims.
|
||||
7. **Teach vision that art inside a panel is not the scene.** A chibi on a monitor became "a man holding
|
||||
a drink" at 1:35. A colleague pointing into the distance became "pointing towards the screen" at
|
||||
1:59.
|
||||
8. **`layers` writes nothing** and reports `completed 116/116`, so no clip has parallax and a still
|
||||
holds for 28s from 2:24 (`caveats/audit-open.md#layers-writes-nothing`).
|
||||
9. **Clear the stale job error.** The completed job still carries `error: "partial: 112/116 completed"`
|
||||
(`caveats/audit-open.md#stale-job-error`).
|
||||
10. Balloon-to-speaker geometry via the unused `det`/`seg` heads
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`) is now behind item 5. With no name to attach,
|
||||
geometry buys nothing.
|
||||
11. Resolve a speaker answer across the whole dialogue window, not just the answering panel. The last 3
|
||||
unresolved refs describe a neighbouring panel in the same 8-panel call.
|
||||
12. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work
|
||||
(`caveats/audit-open.md#sqlite-locking`).
|
||||
|
||||
## Lesson worth keeping
|
||||
|
||||
Every metric recorded before this session said the pipeline was fine or nearly fine. `script` 116/116,
|
||||
"9 named speech lines", `layers` 116/116, `assemble` 1/1. Watching two and a half minutes of output
|
||||
found a 20% sync failure, a cast that is 84% anonymous, invented narration, and a stage that writes
|
||||
nothing while reporting success. Stage counters measure whether code ran. They say nothing about whether
|
||||
the result is correct. Watch the output before trusting a number.
|
||||
|
||||
## Running the pieces
|
||||
|
||||
```bash
|
||||
./start_workers.sh # session_manager + 9 workers, each a uvicorn in a tmux window
|
||||
tmux attach -t manga-workers # per-worker logs
|
||||
.venv/bin/python worker_render.py # self-check, runs real ffmpeg, about 4 minutes
|
||||
```
|
||||
|
||||
Read the state, or clear a stage and resume:
|
||||
|
||||
```bash
|
||||
/usr/bin/ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
|
||||
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"<stage>\"}'"
|
||||
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
|
||||
```
|
||||
|
||||
Traps: `tmux respawn-window -k` does NOT re-run the window's command, it drops to a bare shell and the
|
||||
worker stays down. Restart one worker by re-sending the `send-keys` line from `start_workers.sh`, and
|
||||
confirm with `/health` plus `./check_stale.sh`. `docker compose up --build orchestrator` recreates the
|
||||
container, which drops any `docker cp`'d script such as `audit_registry.py`.
|
||||
Plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin, so use `/usr/bin/ssh`.
|
||||
`mc` aliases on homesrv are `homesrv` and `mio`. `local` returns Access Denied and `rfs` is the empty
|
||||
rustfs. `cp` is aliased to `cp -i` and hangs on overwrite, so use `/usr/bin/cp -f`.
|
||||
|
||||
Re-fixing assembly needs the real clips, which the session scratchpad no longer holds:
|
||||
|
||||
```bash
|
||||
/usr/bin/ssh kami@192.168.1.104 'P=homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80; mc cp -q -r $P/clips/ /tmp/rclips/; cd /tmp/rclips && tar cf - .' | tar xf - -C clips/
|
||||
```
|
||||
|
||||
## Storage and viewer, tasks #116/#117
|
||||
|
||||
[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had been
|
||||
dead since 2026-07-19 on an arm64 digest pin.
|
||||
|
||||
[#116] is closer but not cut over. Artifacts split one bucket per class
|
||||
(`decisions/storage-layout.md#bucket-per-artifact`), and both MinIO and `rustfs` hold all six buckets.
|
||||
`rustfs` on `127.0.0.1:9010/9011` is still empty and nothing is repointed, so MinIO serves every read
|
||||
and write. Remaining: `mc mirror` the live buckets, verify counts and sizes, then decide on cutover
|
||||
(`decisions/storage-layout.md#rustfs-staged`).
|
||||
|
||||
Two containers on homesrv had been dead for two weeks and now run. `manga-fetch` is the one
|
||||
`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them, and
|
||||
nothing watches the workers.
|
||||
|
||||
## Open questions
|
||||
|
||||
Four Phase 1 items have no Vikunja task, because writing to the tracker was not asked for: the speaker
|
||||
contract fix, the verifier rules, the tracklet constraints, and the flag resolution path. Only [#203]
|
||||
existed and is closed by `decisions/audit-phase1.md#unlocked-model-load`.
|
||||
|
||||
Three audit items are deliberately not done and are recorded as caveats rather than silently dropped:
|
||||
honest stage clearing, ComfyUI under the session mutex, and reversible identity merges. Each needs a
|
||||
design decision, not a patch.
|
||||
|
||||
Carried over from the reconstruction: `.venv` needs the ROCm torch wheel reinstalled, and `dots.tts/`,
|
||||
`legacy/`, `RESUME_SPEC.md`, `pipeline-design-notes.md`, `spec-v2.md` are unrecoverable.
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
# ROADMAP
|
||||
|
||||
Ordered outcomes past the current one. Phase 1 landed on 2026-08-11 and moved to
|
||||
`decisions/audit-phase1.md`. The reasoning behind every phase is in `AUDIT.md`.
|
||||
|
||||
Approval boundary from the audit: ship Phase 2 and measure it. Then decide whether Phase 3 and 4 are
|
||||
worth the size of the change.
|
||||
|
||||
## Phase 2: safe throughput
|
||||
|
||||
Unblocks: a chapter that finishes in a fraction of the current wall time without changing any model.
|
||||
Done when: the same labeled chapter runs with two gemma loads instead of seven. The acceptance numbers
|
||||
below are recorded in a report.
|
||||
|
||||
- Two-phase gemma lease schedule, so the model stays resident across compatible stages.
|
||||
- JSON-schema-constrained generation for detect, dialogue, roster, direction, same-person, and resolve.
|
||||
- `--media-path /dev/shm` and `file://` image paths instead of base64 data URIs.
|
||||
- An ephemeral `/dev/shm` cache keyed by S3 URI plus etag, and reused direction downloads.
|
||||
- Batched siglip embeddings, one forward pass per bounded image batch.
|
||||
- Batched scene construction and SQLite writes. Set `busy_timeout` first
|
||||
(`caveats/audit-open.md#sqlite-locking`).
|
||||
- Bounded fetch and framed-page crop concurrency.
|
||||
- Images before the differing instruction in every multi-call prompt, so llama-server reuses its KV
|
||||
cache prefix across the two direction passes.
|
||||
|
||||
## Phase 3: multi-view constrained identity
|
||||
|
||||
Unblocks: an identity that survives a reviewer disagreeing with it.
|
||||
Done when: false merges and false splits are reported per labeled chapter and the resolver beats the
|
||||
Phase 1 baseline on both.
|
||||
|
||||
- Chapter-local tracklets persisted as a resumable homesrv artifact, not worker memory.
|
||||
- Face and body galleries with crop-quality selection, and 2-4 query views per tracklet.
|
||||
- Global constraints at resolution: must-link inside a tracklet, cannot-link for co-presence, gender
|
||||
and species gates, manual locks, one appearance per character per panel.
|
||||
- Name claims separated from visual identity. Roster names stay candidate vocabulary.
|
||||
- Reversible merges: a cluster with `canonical_id`, `member_ids`, and `cannot_link`, replacing the
|
||||
destructive delete (`caveats/audit-open.md#destructive-reconcile`).
|
||||
- Identity evaluation extended to purity, fragmentation, name binding, and speaker accuracy.
|
||||
|
||||
## Phase 4: evidence-ledger narration
|
||||
|
||||
Unblocks: narration that can be checked against the panel rather than trusted.
|
||||
Done when: every beat carries an evidence artifact and the verifier reports quote grounding, action
|
||||
coverage, and future-fact leakage against it.
|
||||
|
||||
- Ordered beat evidence: panels, actions, chronology, dialogue ids, typed speaker refs, uncertainty.
|
||||
- Scene-scoped story and turn-taking state, reset on location and time changes.
|
||||
- Script chunks of 6-12 consecutive beats per text-only call, returning evidence ids.
|
||||
- Confidence-aware wording: a weak speaker becomes `someone`, not a confident name.
|
||||
- Targeted per-beat repair driven by the exact failures.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
Capture a baseline and compare the same labeled chapter after every phase.
|
||||
|
||||
- Stage wall time and model-load time.
|
||||
- Gemma call count and repair call count.
|
||||
- MinIO bytes moved, prompt-evaluation and generation timings.
|
||||
- Tracklet false merges and splits, identity accuracy.
|
||||
- Name and speaker accuracy, unresolved correctness flags.
|
||||
- Script repetition, unsupported facts, verifier retry rate.
|
||||
- The share of narrated lines whose speaker is a named character rather than `Someone`. That one
|
||||
number would have caught `decisions/audit-phase1.md#speaker-ref-is-canonical` the day it landed.
|
||||
|
||||
Read `caveats/audit-open.md#inconsistent-stage-policy` before trusting any per-stage count.
|
||||
@@ -1,99 +0,0 @@
|
||||
# worker_ocr.py — stage 3 text extraction. FastAPI :8001. cpu (easyocr), no session.
|
||||
# panel in -> text blocks with bboxes + confidence out. orchestrator persists to sqlite.
|
||||
# easyocr (not tesseract): it reads stylized manga lettering far better -- recovers whole
|
||||
# lines tesseract garbles or drops. runs on GPU (~0.4s/page warm) by default; the OCR stage
|
||||
# runs before any LLM session opens so it doesn't contend with the resident model. set
|
||||
# OCR_GPU=0 to force CPU (~3s/page). GPU needs MIOPEN_FIND_MODE=FAST in the env or the first
|
||||
# ROCm run spends ~60s in MIOpen's exhaustive kernel search -- the launcher sets it.
|
||||
import os, uuid
|
||||
from fastapi import FastAPI
|
||||
from pydantic import BaseModel
|
||||
import transport
|
||||
|
||||
app = FastAPI()
|
||||
SHM = "/dev/shm"
|
||||
MIN_CONF = 0.3 # easyocr line confidence floor
|
||||
OCR_GPU = os.environ.get("OCR_GPU", "1") == "1"
|
||||
|
||||
_reader = None # easyocr.Reader, lazy-loaded on first request
|
||||
|
||||
|
||||
def _get_reader():
|
||||
global _reader
|
||||
if _reader is None:
|
||||
import easyocr
|
||||
_reader = easyocr.Reader(["en"], gpu=OCR_GPU, verbose=False)
|
||||
return _reader
|
||||
|
||||
|
||||
def _detections_to_texts(detections):
|
||||
"""easyocr readtext output [(box_pts, text, conf)] -> our text blocks with xywh bboxes.
|
||||
box_pts is 4 corner [x,y] points. drops low-confidence and art-noise (<2 letters).
|
||||
casing is left as-is (mixed) -- the vision stage re-cases from the image anyway."""
|
||||
texts = []
|
||||
for i, (box, txt, conf) in enumerate(detections):
|
||||
txt = txt.strip()
|
||||
if conf < MIN_CONF or sum(c.isalpha() for c in txt) < 2:
|
||||
continue
|
||||
xs = [p[0] for p in box]; ys = [p[1] for p in box]
|
||||
x, y = int(min(xs)), int(min(ys))
|
||||
texts.append({
|
||||
"id": f"t{i+1:03d}",
|
||||
"content": txt,
|
||||
"bbox": [x, y, int(max(xs)) - x, int(max(ys)) - y],
|
||||
"confidence": round(float(conf), 3),
|
||||
})
|
||||
return texts
|
||||
|
||||
|
||||
def ocr_image(path: str):
|
||||
return _detections_to_texts(_get_reader().readtext(path, detail=1, paragraph=False))
|
||||
|
||||
|
||||
class OCRInput(BaseModel):
|
||||
panel_uri: str
|
||||
job_id: str = ""
|
||||
panel_id: str = ""
|
||||
|
||||
|
||||
@app.post("/ocr")
|
||||
async def ocr(data: OCRInput):
|
||||
local = transport.get(data.panel_uri, f"{SHM}/ocr_{uuid.uuid4().hex[:8]}.png")
|
||||
texts = ocr_image(local)
|
||||
os.remove(local)
|
||||
return {"panel_id": data.panel_id, "texts": texts}
|
||||
|
||||
|
||||
@app.post("/unload")
|
||||
async def unload():
|
||||
"""free the resident easyocr reader (~1-2GB) once the OCR stage is done, before gemma4 loads.
|
||||
ocr isn't session-managed, so the orchestrator calls this at stage end."""
|
||||
global _reader
|
||||
was = _reader is not None
|
||||
_reader = None
|
||||
import gc; gc.collect()
|
||||
try:
|
||||
import torch; torch.cuda.empty_cache()
|
||||
except Exception:
|
||||
pass
|
||||
return {"ok": True, "unloaded": was}
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# self-check: detection->text-block conversion (pure, no model needed).
|
||||
dets = [
|
||||
([[10, 10], [110, 10], [110, 40], [10, 40]], "HELLO", 0.9), # kept
|
||||
([[10, 200], [70, 200], [70, 230], [10, 230]], "WORLD", 0.8), # kept
|
||||
([[5, 5], [13, 5], [13, 13], [5, 13]], "=", 0.9), # art-noise: <2 letters
|
||||
([[0, 0], [50, 0], [50, 20], [0, 20]], "REAL", 0.1), # below MIN_CONF
|
||||
]
|
||||
texts = _detections_to_texts(dets)
|
||||
assert [t["content"] for t in texts] == ["HELLO", "WORLD"], texts
|
||||
assert texts[0]["bbox"] == [10, 10, 100, 30], texts[0]["bbox"]
|
||||
assert texts[1]["confidence"] == 0.8
|
||||
print("worker_ocr self-check ok")
|
||||
@@ -1,125 +0,0 @@
|
||||
# worker_parse.py — manga parse (paged manga only). FastAPI :8009. GPU, session-guarded ("magi").
|
||||
# Magi v2 chapter-wide pass: panel detection + reading order + OCR in one shot. Replaces the
|
||||
# crop+ocr stages for paged manga; downstream ocr stage no-ops because rows are pre-populated.
|
||||
# webtoons do NOT come here — they use worker_crop /crop/webtoon. See manga-two-repo-split memory.
|
||||
import os, uuid
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from pydantic import BaseModel
|
||||
import cv2
|
||||
import numpy as np
|
||||
import transport
|
||||
|
||||
app = FastAPI()
|
||||
SHM = "/dev/shm"
|
||||
MAGI_MODEL = "ragavsachdeva/magiv2"
|
||||
_model = None
|
||||
|
||||
|
||||
def _load_magi():
|
||||
global _model
|
||||
if _model is None:
|
||||
import torch
|
||||
from transformers import AutoModel
|
||||
_model = AutoModel.from_pretrained(MAGI_MODEL, trust_remote_code=True).cuda().eval()
|
||||
_model._torch = torch
|
||||
return _model
|
||||
|
||||
|
||||
class ParseInput(BaseModel):
|
||||
page_uris: list # all pages of the chapter, in order
|
||||
manga_id: str
|
||||
chapter_id: str
|
||||
session_id: str = "" # magi GPU lease (opened by orchestrator)
|
||||
job_id: str = ""
|
||||
|
||||
|
||||
def _center_in(box, panel) -> bool:
|
||||
x1, y1, x2, y2 = box
|
||||
px1, py1, px2, py2 = panel
|
||||
cx, cy = (x1 + x2) / 2, (y1 + y2) / 2
|
||||
return px1 <= cx <= px2 and py1 <= cy <= py2
|
||||
|
||||
|
||||
def assemble_panels(pages, results, manga_id, chapter_id, put):
|
||||
"""Flatten Magi's per-page output into chapter-order panels with their OCR.
|
||||
`pages`: RGB np arrays. `results`: per-page dicts (Magi keys). `put(np_crop, uri)` uploads.
|
||||
Text is assigned to the panel whose box contains the text-box center; SFX (non-essential)
|
||||
is dropped so narration isn't polluted. bbox converted [x1,y1,x2,y2] -> [x,y,w,h]."""
|
||||
out, gidx = [], 0
|
||||
for img, res in zip(pages, results):
|
||||
panels = res.get("panels", [])
|
||||
texts = res.get("texts", [])
|
||||
ocr = res.get("ocr", [])
|
||||
essential = res.get("is_essential_text", [True] * len(texts))
|
||||
for p in panels:
|
||||
x1, y1, x2, y2 = (int(v) for v in p)
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/panels/p{gidx:03d}.png"
|
||||
if not transport.exists(uri): # deterministic per gidx -> resumable
|
||||
put(img[y1:y2, x1:x2], uri)
|
||||
ocr_texts = []
|
||||
for ti, tb in enumerate(texts):
|
||||
if ti < len(ocr) and essential[ti] and _center_in(tb, p):
|
||||
tx1, ty1, tx2, ty2 = (int(v) for v in tb)
|
||||
ocr_texts.append({"text_id": f"t{ti}", "content": ocr[ti],
|
||||
"bbox": [tx1, ty1, tx2 - tx1, ty2 - ty1], "confidence": 1.0})
|
||||
out.append({"panel_index": gidx, "uri": uri,
|
||||
"bbox": [x1, y1, x2 - x1, y2 - y1], "ocr": ocr_texts})
|
||||
gidx += 1
|
||||
return out
|
||||
|
||||
|
||||
def _put_crop(np_rgb, uri):
|
||||
tmp = f"{SHM}/parse_{uuid.uuid4().hex[:8]}.png"
|
||||
cv2.imwrite(tmp, cv2.cvtColor(np_rgb, cv2.COLOR_RGB2BGR))
|
||||
transport.put(tmp, uri)
|
||||
os.remove(tmp)
|
||||
|
||||
|
||||
@app.post("/parse")
|
||||
async def parse(data: ParseInput):
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
pages = []
|
||||
for i, u in enumerate(data.page_uris):
|
||||
local = transport.get(u, f"{SHM}/pp_{tag}_{i:03d}.png")
|
||||
img = cv2.imread(local)
|
||||
if img is None:
|
||||
raise HTTPException(400, f"page not readable: {u}")
|
||||
pages.append(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
|
||||
os.remove(local)
|
||||
model = _load_magi()
|
||||
# ponytail: empty character bank in v1 — identity stays with the downstream siglip stage;
|
||||
# feed a real bank (known-char ref crops + names) here to get Magi speaker association.
|
||||
bank = {"images": [], "names": []}
|
||||
with model._torch.no_grad():
|
||||
results = model.do_chapter_wide_prediction(pages, bank, use_tqdm=False, do_ocr=True)
|
||||
panels = assemble_panels(pages, results, data.manga_id, data.chapter_id, _put_crop)
|
||||
return {"panels": panels}
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# self-check: model-free. Fake a 2-page Magi result and assert panel flattening,
|
||||
# chapter-order indexing, text->panel containment, SFX drop, and bbox conversion.
|
||||
transport.exists = lambda *a, **k: False # no minio in self-check
|
||||
stored = {}
|
||||
pages = [np.zeros((100, 100, 3), np.uint8), np.zeros((100, 100, 3), np.uint8)]
|
||||
results = [
|
||||
{"panels": [[0, 0, 50, 100], [50, 0, 100, 100]], # page 0: two panels
|
||||
"texts": [[10, 10, 20, 20], [60, 10, 70, 20]], # one text in each
|
||||
"ocr": ["HELLO", "BOOM"], "is_essential_text": [True, False]}, # BOOM = SFX, dropped
|
||||
{"panels": [[0, 0, 100, 100]], # page 1: one panel
|
||||
"texts": [[5, 5, 15, 15]], "ocr": ["WORLD"], "is_essential_text": [True]},
|
||||
]
|
||||
panels = assemble_panels(pages, results, "m", "c", lambda img, uri: stored.__setitem__(uri, img.shape))
|
||||
assert [p["panel_index"] for p in panels] == [0, 1, 2], "chapter-order index"
|
||||
assert panels[0]["ocr"][0]["content"] == "HELLO"
|
||||
assert panels[1]["ocr"] == [], "SFX text dropped from panel 1"
|
||||
assert panels[2]["ocr"][0]["content"] == "WORLD"
|
||||
assert panels[0]["bbox"] == [0, 0, 50, 100], "xyxy->xywh"
|
||||
assert panels[0]["ocr"][0]["bbox"] == [10, 10, 10, 10]
|
||||
assert stored, "crops uploaded via put"
|
||||
print("worker_parse self-check ok")
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Registry audit for one chapter, after vision + identity + reconcile and before anything downstream.
|
||||
|
||||
Runs inside manga-orchestrator (reads /data/manga.db). `audit_speakers.py` answers the attribution
|
||||
questions and needs the dialogue stage; this one answers the questions that decide whether dialogue is
|
||||
worth running at all:
|
||||
|
||||
1. did the bbox fix land — are stored boxes pixels, or still gemma's 0-1000 grid,
|
||||
2. how many characters did the rebaseline mint, and did one of them absorb the chapter again,
|
||||
3. what happened on panel 7, the worked example.
|
||||
|
||||
Usage: docker exec manga-orchestrator python3 /app/audit_registry.py [chapter_id] [panel_index]
|
||||
"""
|
||||
import collections
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
CHAPTER = sys.argv[1] if len(sys.argv) > 1 else "7c944dd4-e972-42c7-ba60-9f6939548e80"
|
||||
WORKED_EXAMPLE = int(sys.argv[2]) if len(sys.argv) > 2 else 7
|
||||
|
||||
c = sqlite3.connect("/data/manga.db")
|
||||
c.row_factory = sqlite3.Row
|
||||
manga_id = c.execute("SELECT manga_id FROM chapters WHERE chapter_id=?", (CHAPTER,)).fetchone()[0]
|
||||
|
||||
reg = {r["character_id"]: dict(r) for r in c.execute(
|
||||
"SELECT character_id, name, aliases, gender, ref_image_uris, embedding_uri "
|
||||
"FROM characters WHERE manga_id=?", (manga_id,))}
|
||||
|
||||
panels = c.execute(
|
||||
'SELECT panel_id, panel_index, page_index, bbox FROM panels WHERE chapter_id=? ORDER BY panel_order',
|
||||
(CHAPTER,)).fetchall()
|
||||
|
||||
# 1. coordinate space. A 0-1000 grid box on a panel wider or taller than 1000px cannot exceed 1000,
|
||||
# and clamps AT 1000. Real pixel boxes track the panel and scatter past it. The tell is the ratio of
|
||||
# the largest coordinate to the panel dimension, plus how many boxes sit exactly on 1000.
|
||||
detections = 0
|
||||
past_1000 = at_1000 = 0
|
||||
max_ratio = 0.0
|
||||
assigned_total = 0
|
||||
per_char = collections.Counter()
|
||||
worked = None
|
||||
|
||||
for p in panels:
|
||||
row = c.execute("SELECT result_json FROM vision_results WHERE panel_id=?", (p["panel_id"],)).fetchone()
|
||||
if not row:
|
||||
continue
|
||||
v = json.loads(row["result_json"])
|
||||
# the vision blob carries no panel size. panels.bbox is the panel's box on its page and is
|
||||
# [x, y, w, h], not corners — panel 3 of this chapter is [0, 615, 900, 106].
|
||||
pb = json.loads(p["bbox"] or "null")
|
||||
pw, ph = (pb[2], pb[3]) if pb and len(pb) == 4 else (None, None)
|
||||
assigns = {a["local_id"]: (a["character_id"], a["confidence"]) for a in c.execute(
|
||||
"SELECT local_id, character_id, confidence FROM identity_assignments WHERE panel_id=?",
|
||||
(p["panel_id"],))}
|
||||
assigned_total += len(assigns)
|
||||
for cid, _ in assigns.values():
|
||||
per_char[reg.get(cid, {}).get("name") or cid[:16]] += 1
|
||||
people = [ch for ch in (v.get("characters") or []) if ch.get("bbox")]
|
||||
detections += len(people)
|
||||
for ch in people:
|
||||
x1, y1, x2, y2 = ch["bbox"]
|
||||
past_1000 += 1 if max(x2, y2) > 1000 else 0
|
||||
at_1000 += 1 if 1000 in (x2, y2) else 0
|
||||
if pw and ph:
|
||||
max_ratio = max(max_ratio, x2 / pw, y2 / ph)
|
||||
if p["panel_index"] == WORKED_EXAMPLE:
|
||||
worked = (p, v, people, assigns, pw, ph)
|
||||
|
||||
named = [r for r in reg.values() if (r["name"] or "").strip()]
|
||||
print(f"registry: {len(reg)} characters, {len(named)} named -> {sorted((r['name'] or '') for r in named)}")
|
||||
print(f"detections: {detections} assignments: {assigned_total} "
|
||||
f"= {100*assigned_total/max(detections,1):.0f}% coverage")
|
||||
if per_char:
|
||||
top, n = per_char.most_common(1)[0]
|
||||
print(f"assignment spread: {dict(per_char.most_common(8))}")
|
||||
print(f" top character holds {n}/{assigned_total} = {100*n/max(assigned_total,1):.0f}% "
|
||||
f"({'ABSORBING, same signature as before' if n > 0.5 * assigned_total else 'ok'})")
|
||||
print(f"bbox space: {past_1000}/{detections} boxes exceed 1000, {at_1000} sit exactly on 1000, "
|
||||
f"largest coord/panel-dimension = {max_ratio:.2f}")
|
||||
print(f" verdict: {'PIXELS' if past_1000 or max_ratio > 0.02 and at_1000 == 0 else 'STILL 0-1000 GRID'}")
|
||||
missing_refs = [k for k, r in reg.items() if not r["ref_image_uris"] or not r["embedding_uri"]]
|
||||
print(f"characters missing a ref crop or embedding: {len(missing_refs)}")
|
||||
|
||||
if worked:
|
||||
p, v, people, assigns, pw, ph = worked
|
||||
print(f"\npanel_index {WORKED_EXAMPLE} ({p['panel_id']}), {pw}x{ph}:")
|
||||
for ch in people:
|
||||
cid, conf = assigns.get(ch["local_id"], (None, None))
|
||||
name = reg.get(cid, {}).get("name") or (cid[:16] if cid else "-- none --")
|
||||
print(f" {ch['local_id']:10} {ch['bbox']!s:28} {name:22} "
|
||||
f"{'' if conf is None else f'{conf:.2f}'}")
|
||||
else:
|
||||
print(f"\npanel_index {WORKED_EXAMPLE} not found in this chapter")
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Speaker-attribution audit for one chapter. Runs inside manga-orchestrator (reads /data/manga.db).
|
||||
|
||||
Answers the three questions asked of the 2026-08-11 rerun:
|
||||
1. what happens on a multi-character panel whose people have no identity,
|
||||
2. whether a known character gets mis-named,
|
||||
3. what the named-speaker share actually is now.
|
||||
"""
|
||||
import collections
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
|
||||
CHAPTER = sys.argv[1] if len(sys.argv) > 1 else "7c944dd4-e972-42c7-ba60-9f6939548e80"
|
||||
SPEECH = {"speech", "shout", "thought"}
|
||||
|
||||
c = sqlite3.connect("/data/manga.db")
|
||||
c.row_factory = sqlite3.Row
|
||||
manga_id = c.execute("SELECT manga_id FROM chapters WHERE chapter_id=?", (CHAPTER,)).fetchone()[0]
|
||||
|
||||
reg = {r["character_id"]: dict(r) for r in c.execute(
|
||||
"SELECT character_id, name, aliases, description FROM characters WHERE manga_id=?", (manga_id,))}
|
||||
|
||||
# duplicate registry rows make a correct name unresolvable: an answer matching two rows is ambiguous.
|
||||
by_name = collections.defaultdict(list)
|
||||
for cid, r in reg.items():
|
||||
for n in [r["name"], *json.loads(r["aliases"] or "[]")]:
|
||||
if n:
|
||||
by_name[str(n).strip().casefold()].append(cid)
|
||||
dupes = {n: ids for n, ids in by_name.items() if len(ids) > 1}
|
||||
|
||||
panels = c.execute("SELECT panel_id, panel_index, page_index FROM panels WHERE chapter_id=? ORDER BY \"order\"",
|
||||
(CHAPTER,)).fetchall()
|
||||
|
||||
methods, kinds, per_char = collections.Counter(), collections.Counter(), collections.Counter()
|
||||
speech = named = 0
|
||||
crowded_speech = crowded_named = 0
|
||||
solo_speech = solo_named = 0
|
||||
unresolved_names = collections.Counter()
|
||||
ambiguous = []
|
||||
no_identity_crowd = 0
|
||||
# a line attributed to a real present local_id that simply has no identity row: attribution succeeded
|
||||
# and the name is lost anyway. This is the identity-coverage wall, not an attribution failure.
|
||||
attributed_but_unassigned = collections.Counter()
|
||||
|
||||
for p in panels:
|
||||
pid = p["panel_id"]
|
||||
row = c.execute("SELECT result_json FROM vision_results WHERE panel_id=?", (pid,)).fetchone()
|
||||
if not row:
|
||||
continue
|
||||
v = json.loads(row["result_json"])
|
||||
if "dialogue" not in v:
|
||||
continue
|
||||
people = [ch for ch in (v.get("characters") or []) if ch.get("local_id")]
|
||||
assigned = {a["local_id"] for a in c.execute(
|
||||
"SELECT local_id FROM identity_assignments WHERE panel_id=?", (pid,))}
|
||||
crowd = len(people) > 1
|
||||
if crowd and not assigned:
|
||||
no_identity_crowd += 1
|
||||
for d in v["dialogue"]:
|
||||
if d.get("type", "speech") not in SPEECH:
|
||||
continue
|
||||
speech += 1
|
||||
ref = d.get("speaker_ref") or {}
|
||||
kind = ref.get("kind")
|
||||
methods[d.get("speaker_method")] += 1
|
||||
kinds[kind] += 1
|
||||
if crowd:
|
||||
crowded_speech += 1
|
||||
else:
|
||||
solo_speech += 1
|
||||
if kind == "character_id":
|
||||
named += 1
|
||||
per_char[reg.get(ref["value"], {}).get("name") or ref["value"]] += 1
|
||||
if crowd:
|
||||
crowded_named += 1
|
||||
else:
|
||||
solo_named += 1
|
||||
elif kind == "unknown":
|
||||
# normalize_dialogue nulls the flat `speaker` unless it resolved, but speaker_ref keeps the
|
||||
# unresolved local_id as its value.
|
||||
raw = str(ref.get("value") or "").strip()
|
||||
local_ids = {ch["local_id"] for ch in people}
|
||||
if raw in local_ids and raw not in assigned:
|
||||
attributed_but_unassigned[d.get("speaker_method")] += 1
|
||||
elif kind == "name":
|
||||
unresolved_names[ref.get("value")] += 1
|
||||
if ref.get("candidates"):
|
||||
ambiguous.append((pid, ref.get("value"), ref["candidates"]))
|
||||
|
||||
print(f"panels with dialogue: {sum(1 for p in panels if (lambda r: r and 'dialogue' in json.loads(r['result_json']))(c.execute('SELECT result_json FROM vision_results WHERE panel_id=?', (p['panel_id'],)).fetchone()))}/{len(panels)}")
|
||||
print(f"speech lines: {speech} named (character_id): {named} = {100*named/max(speech,1):.0f}%")
|
||||
print(f" multi-character panels: {crowded_named}/{crowded_speech} named")
|
||||
print(f" single-character panels: {solo_named}/{solo_speech} named")
|
||||
print(f"multi-character panels with NO identity at all: {no_identity_crowd}")
|
||||
print(f"attributed to a present local_id with NO identity row: {dict(attributed_but_unassigned)}")
|
||||
print(f"speaker_method: {dict(methods)}")
|
||||
print(f"speaker_ref kind: {dict(kinds)}")
|
||||
print(f"named per character: {dict(per_char)}")
|
||||
print(f"unresolved name refs: {dict(unresolved_names)}")
|
||||
print(f"registry duplicate names (block a correct bind): "
|
||||
f"{ {n: [reg[i]['name'] for i in ids] for n, ids in dupes.items()} }")
|
||||
if ambiguous:
|
||||
print("ambiguous binds:")
|
||||
for pid, val, cands in ambiguous[:20]:
|
||||
print(f" {pid} {val!r} -> {[reg.get(x, {}).get('name') for x in cands]}")
|
||||
+1
-1
@@ -38,7 +38,7 @@ def _letterbox(img, sz=1024):
|
||||
return canvas, r, px, py
|
||||
|
||||
|
||||
def detect_text_regions(img, conf: float = None) -> list:
|
||||
def detect_text_regions(img, conf: float | None = None) -> list:
|
||||
"""img: BGR ndarray (one panel). -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels,
|
||||
reading order (top->bottom, left->right). Empty list on no detections (caller falls back to the
|
||||
holistic prompt). Load + inference are lazy so importing this never touches the GPU or the model."""
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# caveats/
|
||||
|
||||
One entry per known breakage, deferral, or trap. An entry names what fails, who pays for it, and the
|
||||
concrete trigger that makes fixing it worth the time.
|
||||
|
||||
A caveat that gets resolved moves to `decisions/` or gets deleted. A caveat with no revisit trigger is
|
||||
a complaint, so give it one or drop it.
|
||||
|
||||
## Rules for this directory
|
||||
|
||||
* One file per source, one `##` section per limit.
|
||||
* State the limit as what fails, not as a topic. "Reconcile deletes the losing character" beats
|
||||
"Reconcile".
|
||||
* Delete an entry the moment its trigger fires and the work lands.
|
||||
|
||||
## Index
|
||||
|
||||
| caveat | source |
|
||||
| --- | --- |
|
||||
| [Reconcile deletes the losing character irreversibly](audit-open.md#destructive-reconcile) | AUDIT.md |
|
||||
| [Clearing a stage does not undo what it wrote](audit-open.md#dishonest-clearing) | AUDIT.md |
|
||||
| [ComfyUI uses the GPU outside the session mutex](audit-open.md#comfyui-unscheduled) | AUDIT.md |
|
||||
| [The JSON repair pass can fabricate dialogue](audit-open.md#repair-fabricates) | AUDIT.md |
|
||||
| [`/review/preview` pins a solo clip into the final video](audit-open.md#preview-overwrites-clip) | AUDIT.md |
|
||||
| [Stored embeddings carry no model or pooling version](audit-open.md#untagged-embeddings) | AUDIT.md |
|
||||
| [Worker endpoints block the event loop](audit-open.md#blocking-event-loop) | AUDIT.md |
|
||||
| [SQLite has no busy timeout, so parallelism will surface as lock errors](audit-open.md#sqlite-locking) | AUDIT.md |
|
||||
| [`layers` runs after `tts`, so pipelined solo beats lose parallax](audit-open.md#layers-after-tts) | AUDIT.md |
|
||||
| [`completed` means something different in each stage](audit-open.md#inconsistent-stage-policy) | AUDIT.md |
|
||||
| [Identity worker caches characters the orchestrator has deleted](audit-open.md#stale-known-cache) | AUDIT.md |
|
||||
| [The gemma helpers exist twice and have diverged](audit-open.md#gemma-helpers-duplicated) | repo audit |
|
||||
| [A character seen once gets no assignment at all](audit-open.md#pending-in-worker-memory) | AUDIT.md |
|
||||
| [MinIO credentials are hardcoded in committed source](audit-open.md#hardcoded-credentials) | AUDIT.md |
|
||||
| [Assemble marks a job completed with no clips](audit-open.md#empty-assemble) | AUDIT.md |
|
||||
| [Reviewer timestamps drift against the crossfaded video](audit-open.md#timeline-drift) | AUDIT.md |
|
||||
| [Nothing attributes a speaker in a multi-character panel](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run |
|
||||
| [One character id covers two different women](speaker-attribution.md#identity-over-merge) | 2026-08-11 run |
|
||||
| [The character registry carries five weeks of wrong names](speaker-attribution.md#registry-pollution) | 2026-08-11 run |
|
||||
| [One invented word still halts the chapter](speaker-attribution.md#one-word-halts-chapter) | 2026-08-11 run |
|
||||
| [A completed job keeps the error from an earlier failure](audit-open.md#stale-job-error) | 2026-08-11 run |
|
||||
| [`layers` reports success on an empty bucket](audit-open.md#layers-writes-nothing) | 2026-08-11 run |
|
||||
| [Every `bbox` is read in the wrong coordinate space](speaker-attribution.md#bbox-wrong-space) | resolved, rerun pending |
|
||||
| [Identity cannot say "a person with no name"](speaker-attribution.md#no-anonymous-identity) | 2026-08-12 panel 7 |
|
||||
| [Vision does not separate a background extra from cast](speaker-attribution.md#extras-as-cast) | 2026-08-12 panel 7 |
|
||||
| [Cast reference profiles are enrolled from wrong crops](speaker-attribution.md#poisoned-reference-set) | 2026-08-12 panel 7 |
|
||||
| [Identity coverage has fallen on every run since the gate landed](audit-open.md#coverage-trend) | before the next downstream run |
|
||||
| [Embedding cosine on a person crop cannot tell two people apart](audit-open.md#cosine-not-identity) | before building on cosine |
|
||||
@@ -0,0 +1,273 @@
|
||||
# Open limits from the 2026-08-11 audit
|
||||
|
||||
Everything here was read in source during the audit and deliberately left unfixed in Phase 1. The
|
||||
fixed findings live in `decisions/audit-phase1.md`. Line numbers are from the audit and may drift.
|
||||
|
||||
## Reconcile deletes the losing character irreversibly {#destructive-reconcile}
|
||||
|
||||
Reconciliation used to delete the losing character row. Clearing the reconcile stage did not undo it, and
|
||||
nothing recorded which detections had been the loser's.
|
||||
|
||||
**Half-closed 2026-08-12.** The loss is no longer unrecoverable. `merge_characters` marks the loser
|
||||
`merged_into = keeper` instead of deleting it, so its embedding, description and gender survive, and every
|
||||
repointed assignment is stamped `method = merged_from:<loser_id>` in `identity_assignment_sources`. Those
|
||||
two records are enough to walk a merge backwards. Roster readers filter `merged_into IS NULL`. Lookup by
|
||||
id does not, so an assignment still pointing at a merged id resolves.
|
||||
|
||||
What is still missing is the mechanism that consumes them: there is no unmerge, and no split. Undoing a
|
||||
merge today means a manual SQL walk of the two records above.
|
||||
|
||||
Costs: a wrong merge needs hand-written SQL to undo, not a rebaseline.
|
||||
Revisit when: a reviewer reports a wrong merge, or the review gates from [#136] get a place to hang
|
||||
name/merge/split actions. No wrong merge has been seen since the crops were fixed, so building the unmerge
|
||||
before either trigger would be speculative.
|
||||
|
||||
## Clearing a stage does not undo what it wrote {#dishonest-clearing}
|
||||
|
||||
The dialogue and direct half of this is fixed and proven
|
||||
(`decisions/storage-layout.md#clear-vision-blob`). It cost a wasted rerun on 2026-08-11 first: the clear
|
||||
returned `{"ok": true}`, deleted nothing, and the stage skipped all 116 panels.
|
||||
|
||||
What remains: clearing identity preserves the per-manga registry by design, so a rerun inherits every
|
||||
character it ever minted (`caveats/speaker-attribution.md#registry-pollution`). Nothing verifies that a
|
||||
clear emptied what it claimed.
|
||||
|
||||
Costs: a rerun silently reuses stale output, which reads as a reproducible result.
|
||||
Revisit when: any stage is scheduled concurrently or resumed automatically. A stage must be idempotent
|
||||
before either is safe.
|
||||
|
||||
## ComfyUI uses the GPU outside the session mutex {#comfyui-unscheduled}
|
||||
|
||||
The layers stage calls ComfyUI directly and takes no lease. Another job can load gemma, siglip2, or
|
||||
dots while ComfyUI holds the same GPU.
|
||||
|
||||
Costs: out-of-memory failures that look random and land on an unrelated stage.
|
||||
Revisit when: layers is enabled on a real run, or a second concurrent job is allowed.
|
||||
Workaround: `MAX_CONCURRENT_JOBS=1` keeps one pipeline at a time, which is the current default.
|
||||
|
||||
## The JSON repair pass can fabricate dialogue {#repair-fabricates}
|
||||
|
||||
`call_gemma4_json` hands the model its own truncated text and asks for the JSON it should have been
|
||||
(`worker_vision.py`). The repair call carries no image. On a response truncated by `max_tokens`, the
|
||||
model completes dialogue it can no longer see. What it adds is indistinguishable downstream from
|
||||
transcribed text.
|
||||
|
||||
Costs: invented lines enter the script with normal provenance.
|
||||
Revisit when: schema-constrained generation lands, which removes most of this path.
|
||||
Workaround: resend the image on repair, or retry a truncated response instead of repairing it.
|
||||
|
||||
## `/review/preview` pins a solo clip into the final video {#preview-overwrites-clip}
|
||||
|
||||
`review_preview` renders one panel and calls `save_clip(panel_id, ...)`. `_render_one_beat` returns
|
||||
early when a clip already exists for the leader. Previewing a beat leader therefore drops the rest of
|
||||
the beat's panels. `review_retts` gets this right by calling `delete_clip` first.
|
||||
|
||||
Costs: a reviewer silently corrupts the output by looking at it.
|
||||
Revisit when: the review UI is used on a real chapter.
|
||||
Workaround: never preview a beat leader, or delete the clip row afterwards.
|
||||
|
||||
## Stored embeddings carry no model or pooling version {#untagged-embeddings}
|
||||
|
||||
`embed_crop` uses `pooler_output` when present and falls back to mean-pooled patch tokens otherwise.
|
||||
The two paths produce different vector spaces, and the fixed 0.85 threshold is valid for one of them.
|
||||
Nothing beside a stored `.npy` records which model, revision, or pooling produced it.
|
||||
|
||||
Costs: a transformers upgrade mixes incompatible vectors into one gallery with no error.
|
||||
Revisit when: transformers or the siglip2 revision is upgraded. Before, not after.
|
||||
Workaround: none. Write the model id and pooling mode beside the vector and refuse cross-version
|
||||
comparison.
|
||||
|
||||
## Worker endpoints block the event loop {#blocking-event-loop}
|
||||
|
||||
Endpoints declared `async def` run blocking MinIO, OpenCV, torch, and ffmpeg calls directly on the
|
||||
event loop. A busy worker cannot answer `/health` or `/unload`.
|
||||
|
||||
Costs: `/health/workers` reports a working worker as unreachable, and the session manager's 30-second
|
||||
`/unload` can time out exactly when VRAM needs freeing.
|
||||
Revisit when: a stage stalls on `/unload`, or before any bounded parallelism lands.
|
||||
Workaround: `def` instead of `async def` moves each handler to the threadpool. Tracked as [#199] for
|
||||
the render worker.
|
||||
|
||||
## SQLite has no busy timeout {#sqlite-locking}
|
||||
|
||||
`get_conn` opens a connection per call with no `busy_timeout`. WAL tolerates one writer. `PIPELINE=1`
|
||||
already writes clips from concurrent tasks while TTS writes audio.
|
||||
|
||||
Costs: planned CPU parallelism will surface as `database is locked` before it surfaces as throughput.
|
||||
Revisit when: Phase 2 concurrency work starts. Set the timeout first.
|
||||
Workaround: keep `PIPELINE` off.
|
||||
|
||||
## `layers` runs after `tts` {#layers-after-tts}
|
||||
|
||||
`STAGES` orders `layers` after `tts`, while `run_stage_tts` warns that eager rendering under
|
||||
`PIPELINE=1` needs layers to run first.
|
||||
|
||||
Costs: with the flag on, solo beats always render without parallax.
|
||||
Revisit when: a real run enables `PIPELINE=1`.
|
||||
Workaround: keep `PIPELINE` off, or reorder `STAGES`.
|
||||
|
||||
## `completed` means something different in each stage {#inconsistent-stage-policy}
|
||||
|
||||
A dropped vision panel fails the stage and halts the pipeline. A failed direction window counts its
|
||||
panels as done. Layers always finishes completed.
|
||||
|
||||
Costs: the acceptance metrics in `ROADMAP.md` cannot be read across stages.
|
||||
Revisit when: a baseline measurement is taken. The numbers are meaningless until then.
|
||||
Workaround: none.
|
||||
|
||||
## Identity worker caches characters the orchestrator has deleted {#stale-known-cache}
|
||||
|
||||
`worker_identity._known_cache` is invalidated only by `_persist_char`. The reconcile stage deletes
|
||||
losing characters directly in the orchestrator database. A long-lived worker keeps shortlisting and
|
||||
assigning ids that no longer exist.
|
||||
|
||||
Costs: assignments point at rows that are gone.
|
||||
Revisit when: reconcile runs on a chapter without a worker restart between stages. Tracked as [#201],
|
||||
which proposes caching per `manga_id`.
|
||||
Workaround: restart the identity worker after reconcile.
|
||||
|
||||
## A character seen once gets no assignment at all {#pending-in-worker-memory}
|
||||
|
||||
`worker_identity._pending` holds full crop images in worker memory for a whole chapter, keyed by
|
||||
session. That is durable per-chapter state inside a worker documented as stateless, and it is lost on
|
||||
restart. A character seen exactly once receives no assignment, not even a chapter-local handle.
|
||||
|
||||
Costs: one-off characters vanish from the scene graph.
|
||||
Revisit when: chapter-local tracklet persistence lands (`ROADMAP.md`, Phase 3).
|
||||
Workaround: none.
|
||||
|
||||
## MinIO credentials are hardcoded in committed source {#hardcoded-credentials}
|
||||
|
||||
Defaults live in `transport.py` and `service.py`.
|
||||
|
||||
Costs: the credentials are in git history for anyone who gets the repo.
|
||||
Revisit when: the repo leaves this machine, or MinIO is reachable outside the LAN.
|
||||
Workaround: the environment variables already override them. Set them and remove the defaults.
|
||||
|
||||
## Assemble marks a job completed with no clips {#empty-assemble}
|
||||
|
||||
`run_stage_assemble` does not check that `clip_uris` is non-empty before assembling, then marks the job
|
||||
completed.
|
||||
|
||||
Costs: a failed chapter reports success.
|
||||
Revisit when: any run reports completed without a video. One `if not clip_uris` guard fixes it.
|
||||
Workaround: none.
|
||||
|
||||
## Reviewer timestamps drift against the crossfaded video {#timeline-drift}
|
||||
|
||||
`/review/panels` sums per-panel audio durations. Assemble crossfades clips using the per-beat
|
||||
transitions, so every non-`cut` transition shortens the real video.
|
||||
|
||||
Costs: reviewer timestamps drift further out of sync the further into the chapter they scrub.
|
||||
Revisit when: the review UI is used for timing work.
|
||||
Workaround: subtract the transition overlaps by hand.
|
||||
|
||||
## A completed job keeps the error from an earlier failure {#stale-job-error}
|
||||
|
||||
`/job/resume` does not clear `jobs.error`. Job `778297bc` finished every stage and still reports
|
||||
`status: "completed"` beside `error: "partial: 112/116 completed"`, a message from three resumes earlier.
|
||||
|
||||
Costs: any reader of the error field sees a failure on a successful job. The review UI and any future
|
||||
alerting both read it.
|
||||
Revisit when: anything branches on `jobs.error`, or a run is judged by its status alone.
|
||||
|
||||
## `layers` reports success on an empty bucket {#layers-writes-nothing}
|
||||
|
||||
Measured on job `778297bc`: `layers` reported `completed 116/116` while `s3://layers/` held 0 objects.
|
||||
Every clip in that run therefore has no parallax. The stage is a sibling of
|
||||
[#inconsistent-stage-policy](#inconsistent-stage-policy), but this is the measured instance:
|
||||
`worker_layers.py`'s own self-check already fails on the missing
|
||||
`legacy/qwen_layered_workflow.json`, and the stage still reports done for every panel.
|
||||
|
||||
Costs: a silent quality regression that no status field reveals.
|
||||
Revisit when: parallax matters for a deliverable, or before quoting this run as a full-pipeline pass.
|
||||
|
||||
## Identity coverage has fallen on every run since the gate landed {#coverage-trend}
|
||||
|
||||
70% -> 61% -> 50% -> 57% across the four 2026-08-12 runs. The gate is stable at 39 to 40% of detections.
|
||||
|
||||
**The 18:07 run resolves most of this.** Coverage is now the gate and nothing else. All 68 assignments went
|
||||
to face-bearing detections and none to a gated one. Recall among face-bearing detections is 68 of 71, or
|
||||
96%, up from 82%. The resolver is not losing cast. The 43% that carries no assignment is the 40% the gate
|
||||
drops plus 3 detections.
|
||||
|
||||
So the coverage number no longer measures the resolver. What it measures is the cost of requiring a face,
|
||||
which is item 4 in `NEXT.md`. Panel 7 is the case: 5 of its 6 detections carry `has_face = False`, and the
|
||||
one that does is unassigned.
|
||||
|
||||
The open question moved. The lead now holds 36 of 68 assignments, or 53%, against 25 of 59 before, and
|
||||
`audit_registry.py` flags that as absorbing. Whether 36 is the protagonist appearing in half the panels or
|
||||
an over-merge cannot be told apart without labels.
|
||||
|
||||
Revisit trigger: before trusting the registry for a downstream run.
|
||||
|
||||
**The 145 ground-truth labels do not cover this chapter.** Measured 2026-08-12. All 145 rows in
|
||||
`identity_labels` key to chapter `8ca8249b`, a different manga with cast "Rico" and "Ikekin", over 81
|
||||
distinct panels. Chapter `7c944dd4` has zero labels, so `/review/identity?job_id=778297bc...` returns
|
||||
`labeled: 0, accuracy: null`. Scoring 8ca instead gives 7/138, because its identity run is stale: 44
|
||||
assignments over 246 panels, predating all seven fixes. That number measures the old code on the wrong
|
||||
chapter and settles nothing.
|
||||
|
||||
The plumbing works, the labels are missing. Precision and recall on the current registry need a hand
|
||||
pass over `7c944dd4` first, via `POST /review/identity/label` per detection. Label against the bbox, not
|
||||
the `local_id`. Vision is non-deterministic and reassigns `local_id` top-to-bottom every run. A label
|
||||
taken before a vision rerun then points at whoever now occupies that slot.
|
||||
|
||||
## Embedding cosine on a person crop cannot tell two people apart {#cosine-not-identity}
|
||||
|
||||
Measured 2026-08-12 on the 18:07 run's 22 crops for `character_2367d70c`, the embeddings pulled from
|
||||
`manga/<manga_id>/characters/_crops/*.npy` and L2-normalised. 1152 dimensions.
|
||||
|
||||
| pair | cosine |
|
||||
| --- | --- |
|
||||
| the cat at `p098` against any man | up to 0.82 |
|
||||
| the red-robe chibi at `p088` against any man | up to 0.75 |
|
||||
| two visibly different men (`order 60~80`, `88~110`) | 0.93, 0.92 |
|
||||
| the highest pair in the matrix (`47~51`, one man, one shirt) | 0.96 |
|
||||
|
||||
Different people reach 0.93 and the same person reaches 0.96. No threshold separates them. So a
|
||||
crop-to-crop cosine link is not available, and the `candidates` overlap condition in `link_tracklets` is
|
||||
close to noise for the same reason: the top-k is chosen by this metric, so every crop shortlists the same
|
||||
few rows.
|
||||
|
||||
Inference, not yet tested: identity embeds the whole person box, which holds background, clothing and pose.
|
||||
Those change between scenes while every crop shares one art style, so cosine measures "manga crop of a
|
||||
person in an office". `face_detect` already finds the face and pairs it to the person box for `has_face`,
|
||||
so embedding the face region instead is a small change.
|
||||
|
||||
Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22
|
||||
detections, embed them, and recompute this matrix. If a threshold appears, embed faces.
|
||||
|
||||
## The gallery cap drops exactly the rows a tracklet needs {#gallery-cap-drops-the-unnamed}
|
||||
|
||||
**Open. Observed on the 19:44 run of 2026-08-12.**
|
||||
|
||||
`cast_gallery` orders the live cast named-first, then registry order, and `run_stage_identity` truncates
|
||||
the tail at `GALLERY_CAP = 10`. Registry order is creation order, so the tail is the anonymous characters
|
||||
this very run minted — and a later tracklet of a recurring unnamed person is precisely what needs to match
|
||||
one of those. The chapter has two named rows, so on this run the cap dropped 6 unnamed rows at `p097` and
|
||||
1 at `p109`, and a tracklet that should have joined one of them can only mint a duplicate instead.
|
||||
|
||||
It is capped because every candidate costs at least one reference image in the resolve prompt beside the
|
||||
query crop. The cap is a VRAM and context budget, not a modelling choice.
|
||||
|
||||
**Revisit when** the cast on a chapter routinely exceeds 10 gender-compatible rows, which it already did
|
||||
here. The fix is to order the gallery by how many assignments each character already holds in this chapter,
|
||||
so the tail is the rows nobody has matched rather than the rows nobody has named yet. That needs one count
|
||||
query per tracklet.
|
||||
|
||||
## The gemma helpers exist twice and have diverged {#gemma-helpers-duplicated}
|
||||
|
||||
**Open. Found by the repo audit of 2026-08-13, not yet by a failure.**
|
||||
|
||||
`call_gemma4`, `_extract_json` and `_strip_thought` are defined in both `worker_vision.py` and
|
||||
`worker_script.py`. They are no longer the same code. `worker_vision.call_gemma4` is 21 lines and takes a
|
||||
content list; `worker_script`'s is 8 and takes a prompt string plus a system prompt. Both `_extract_json`
|
||||
bodies carry the same comment about `raw_decode` stopping at the first object, and raise different errors.
|
||||
|
||||
Who pays: whoever fixes the JSON path. `#repair-fabricates` above says the repair pass can fabricate
|
||||
dialogue. A fix written against one copy leaves the other worker on the old behaviour, and no self-check
|
||||
compares them.
|
||||
|
||||
**Revisit when** either JSON path is touched for any reason. The fix is a `gemma.py` holding the three
|
||||
helpers, imported by both workers, which removes about 25 duplicated lines and costs one file.
|
||||
@@ -0,0 +1,162 @@
|
||||
# speaker-attribution
|
||||
|
||||
Limits found by cross-checking the 2026-08-11 chapter run against the panel images.
|
||||
|
||||
## Nothing attributes a speaker in a multi-character panel {#tail-is-not-geometry}
|
||||
|
||||
The false `tail` label is gone (`decisions/speaker-attribution.md#no-fake-tail`). What replaced it is a
|
||||
refusal, not an answer: with two or more characters present, every speech line now returns `unknown`.
|
||||
On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back
|
||||
to a `generic-handle`. That is the honest floor, and it is not the fix.
|
||||
|
||||
The measurement that forced it, on job `778297bc`, chapter `7c944dd4`. Three two-character panels were
|
||||
checked against the art. All three are wrong, each with the two speakers swapped:
|
||||
|
||||
| panel key | line | truth | pipeline |
|
||||
| --- | --- | --- | --- |
|
||||
| `p010.png` | "…definitely an Egen guy, Seonho!" | the woman | Seonho, the person addressed |
|
||||
| `p010.png` | "Y-you think so?" | Seonho | Choi Haeseon |
|
||||
| `p012.png` | "Want me to send you the link?" | the woman | the man |
|
||||
| `p059.png` | "If team leader Choi says it, it must be true." | the man | Choi Haeseon |
|
||||
|
||||
The last row needs no image: the line refers to Choi in the third person and is attributed to Choi.
|
||||
|
||||
The grounded path exists and almost never fires. Only 2 of 81 speech lines got `som_face`, because
|
||||
attribution marks need `face_detect` boxes that survive `_pair_faces_to_present`, and these webtoon
|
||||
close-ups rarely produce them. Inference, not measured: the face detector was not instrumented.
|
||||
|
||||
`worker_vision.py:356` already carries the `ponytail:` note that multi-character attribution needs
|
||||
per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads carry balloon fill and
|
||||
tail tips and are unused.
|
||||
|
||||
**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric
|
||||
(`ROADMAP.md`). The 30% read on this run counted attributions the sample says are wrong. The next run
|
||||
will read lower and will be the first honest number. Raising it means binding a balloon to a speaker
|
||||
by tail geometry, using the unused `det`/`seg` heads.
|
||||
|
||||
## One character id covers two different women {#identity-over-merge}
|
||||
|
||||
`character_afa7623b` is stored as "black bob, white sweater" and is assigned both to that person
|
||||
(`p059.png`) and to the brown-bob green-top coworker (`p010.png`, `p012.png`). It took 25 of the 26
|
||||
identity assignments in the chapter, against 113 detected people. Coverage is 23%.
|
||||
|
||||
**Revisit trigger:** any work on the identity Tier-2 decider. A single id absorbing a whole chapter is
|
||||
the signature to watch for.
|
||||
|
||||
## The character registry carries five weeks of wrong names {#registry-pollution}
|
||||
|
||||
The registry holds 53 characters for manga `ef105a86`, 41 of them unnamed, with "Kei" three times and
|
||||
"Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only
|
||||
Haeseon and Seonho do. `/stage/clear` leaves the per-manga registry intact by design, so every rerun
|
||||
inherits the whole pile.
|
||||
|
||||
Duplicate rows also make a correct name unresolvable. This manga holds `Choi Haeseon`, `Seonho` with
|
||||
aliases `["Lim Seonho", "Seonho"]`, and a separate `Lim Seonho`. An answer of "Lim Seonho" matches two
|
||||
rows, so `normalize_speaker` returns candidates and raises `ambiguous-speaker` instead of binding. The
|
||||
pipeline read the name correctly and still cannot name the speaker.
|
||||
|
||||
**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the
|
||||
registry to a chapter or add a reviewed reset. Merging the duplicate rows needs the reversible-merge
|
||||
design first (`caveats/audit-open.md#destructive-reconcile`).
|
||||
|
||||
## One invented word still halts the chapter {#one-word-halts-chapter}
|
||||
|
||||
The multi-word name failure is fixed (`decisions/speaker-attribution.md#multiword-cast-names`). The
|
||||
blast radius it exposed is not. `run_stage_script` retries a rejected beat once, then raises, so a
|
||||
single unsupported token ends the run at that beat. On job `778297bc` one of the 29 lost beats cited
|
||||
`['Blur']`, an onomatopoeia the model invented. The verifier was right, and the whole chapter still
|
||||
stopped.
|
||||
|
||||
**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer
|
||||
is to flag the beat for review and continue, which is `#136` gate work, not a verifier change.
|
||||
|
||||
## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space}
|
||||
|
||||
**Resolved 2026-08-12, `decisions/identity-bbox.md#bbox-is-normalized`.** The space is gemma's 0-1000
|
||||
grid, proven over all 113 detections, and `/vision` now converts to pixels before returning. The face
|
||||
pairing at `worker_vision.py:57` was reading the same numbers against real pixel face boxes, so it is
|
||||
fixed by the same change. What is left of this entry is the consequence. Every stored assignment,
|
||||
embedding and `ref_image_uris` came from a wrong crop. Identity has to re-run before any of it means
|
||||
anything. The rest below is kept as the record of how it read before.
|
||||
|
||||
Vision's `bbox` values are stored and consumed as absolute pixels. On panel
|
||||
`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of
|
||||
the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own
|
||||
dimensions, four of the six fit their subjects tightly.
|
||||
|
||||
Two places in the code assert pixels, and the art contradicts both:
|
||||
|
||||
- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
|
||||
- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
|
||||
|
||||
Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces
|
||||
against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9
|
||||
from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed
|
||||
alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at
|
||||
`worker_vision.py:57` reads the same numbers and was not checked.
|
||||
|
||||
Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with
|
||||
nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already
|
||||
calls the box "coarse, imprecise".
|
||||
|
||||
Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can
|
||||
be judged while the crops are wrong.
|
||||
|
||||
## Identity cannot say "a person with no name" {#no-anonymous-identity}
|
||||
|
||||
The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9.
|
||||
Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline
|
||||
stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which
|
||||
is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing
|
||||
line.
|
||||
|
||||
This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an
|
||||
unparseable model answer. The missing rule: never attach a name to a detection that carries no name
|
||||
evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the
|
||||
colleague" every time.
|
||||
|
||||
`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from
|
||||
cosine clearing the threshold on a wrong crop.
|
||||
|
||||
**Resolved 2026-08-12, `decisions/identity-bbox.md#none-mints-an-anonymous-character`.** The gemma resolver
|
||||
can answer "none of these" and always could. The orchestrator was discarding the answer. It read only
|
||||
`character_id`, so a deliberate NONE and a hallucinated index both unassigned every crop of the tracklet.
|
||||
A NONE now mints an anonymous character from the crop, using the embedding identity ships beside it. Not
|
||||
yet proven on a GPU run.
|
||||
|
||||
## Vision does not separate a background extra from cast {#extras-as-cast}
|
||||
|
||||
Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground
|
||||
and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame
|
||||
with nobody in it. All six reach identity as equal candidates.
|
||||
|
||||
Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly
|
||||
extras, so it measured nothing useful and should not be quoted again.
|
||||
|
||||
Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return.
|
||||
|
||||
## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set}
|
||||
|
||||
`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the
|
||||
cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are
|
||||
crops of balloon edges, window frames and background extras rather than of faces.
|
||||
|
||||
The visual comparison people reach for as the fix is **already implemented**, so do not build it again.
|
||||
`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images
|
||||
per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair
|
||||
and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for
|
||||
NONE when unsure. `choice: 0` returns `state="new"` and an out-of-range index returns `unresolved`. What
|
||||
the orchestrator does with each is
|
||||
`decisions/identity-bbox.md#none-mints-an-anonymous-character`. The
|
||||
`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`,
|
||||
so the references reach the model.
|
||||
|
||||
That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon
|
||||
edge against references enrolled from window frames and background extras, then sometimes answers "same".
|
||||
Nothing gates enrollment on the crop holding a face.
|
||||
|
||||
Who pays: every later match, because the reference set defines what a character looks like. Fixing
|
||||
`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place.
|
||||
|
||||
Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the
|
||||
existing `ref_image_uris` and `embedding_uri` values as invalid.
|
||||
Executable
+38
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
# Which running workers are serving code older than the file on disk.
|
||||
#
|
||||
# Python loads a worker module once, at process start. Editing the file afterwards changes nothing
|
||||
# until that process restarts, and nothing in the stage output says so: the stage completes, the
|
||||
# counters read 116/116, and the result is what the OLD code produced. On 2026-08-12 this ate a full
|
||||
# vision + identity + reconcile cycle — the bbox fix was committed 16 minutes after the worker that
|
||||
# was supposed to apply it had already started.
|
||||
#
|
||||
# Run this before any stage run that is meant to prove a code change.
|
||||
# Exits non-zero if anything is stale, so it can gate a script.
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
stale=0
|
||||
# `ps -o lstart` is always exactly five fields: "Wed Aug 12 13:20:00 2026". Read them positionally
|
||||
# rather than trying to split the line with a regex.
|
||||
while read -r pid _dow mon day time year cmd; do
|
||||
mod=$(grep -oP '(?:worker_\w+|session_manager)(?=:app)' <<<"$cmd" | head -1)
|
||||
[ -n "$mod" ] && [ -f "$mod.py" ] || continue
|
||||
p_epoch=$(date -d "$mon $day $time $year" +%s 2>/dev/null) || continue
|
||||
f_epoch=$(stat -c %Y "$mod.py")
|
||||
if [ "$f_epoch" -gt "$p_epoch" ]; then
|
||||
printf 'STALE %-22s pid %-7s edited %dm after it started\n' \
|
||||
"$mod.py" "$pid" "$(( (f_epoch - p_epoch) / 60 ))"
|
||||
stale=1
|
||||
else
|
||||
printf 'ok %-22s pid %s\n' "$mod.py" "$pid"
|
||||
fi
|
||||
done < <(ps -eo pid,lstart,cmd | grep -E 'uvicorn (worker_|session_manager)' | grep -v grep)
|
||||
|
||||
if [ "$stale" -ne 0 ]; then
|
||||
echo
|
||||
echo "Restart the stale ones before trusting a stage result. Window numbers:"
|
||||
tmux list-windows -t manga-workers -F ' #{window_index} #{window_name}' 2>/dev/null \
|
||||
|| echo " (tmux session manga-workers is not running; ./start_workers.sh)"
|
||||
fi
|
||||
exit "$stale"
|
||||
@@ -0,0 +1,56 @@
|
||||
# decisions/
|
||||
|
||||
One entry per settled question about this pipeline. An entry names the claim, the evidence, the date,
|
||||
and what it forbids or permits next.
|
||||
|
||||
No entry is a plan. `NEXT.md` holds the plan, `ROADMAP.md` holds the horizon, and a limitation that is
|
||||
still live belongs in `caveats/`.
|
||||
|
||||
## Rules for this directory
|
||||
|
||||
* One file per topic, one `##` section per question, so the index can link an anchor.
|
||||
* An entry cites a file and line, a commit, or a test. No citation means it is an opinion, and an
|
||||
opinion gets deleted rather than defended.
|
||||
* **Closed** means acting on it is safe. **Open** means investigated and undecided. **Void** means the
|
||||
evidence turned out to be invalid and the claim must not be cited again.
|
||||
* Rewrite an entry when something contradicts it, and say what changed.
|
||||
|
||||
## Index
|
||||
|
||||
| decision | state |
|
||||
| --- | --- |
|
||||
| [A dialogue row's speaker is `speaker_ref`, not `speaker`](audit-phase1.md#speaker-ref-is-canonical) | closed |
|
||||
| [Script verification must not fail on sentence-initial capitals or short quotes](audit-phase1.md#verifier-false-positives) | closed |
|
||||
| [Tracklet links are transitive over similarity, never across a hard constraint](audit-phase1.md#tracklet-hard-constraints) | closed |
|
||||
| [A detected face takes an identity only when it falls inside that character's box](audit-phase1.md#gated-face-pairing) | closed |
|
||||
| [A 409 from the GPU mutex is a queue signal, not a stale lease](audit-phase1.md#no-lease-stealing) | closed |
|
||||
| [Correctness flags resolve by flag id, and never block autonomous TTS](audit-phase1.md#flag-resolution) | closed |
|
||||
| [An out-of-range resolver index is `unresolved`, never a new character](audit-phase1.md#hallucinated-index) | closed |
|
||||
| [The session manager holds no lock across a model load](audit-phase1.md#unlocked-model-load) | closed |
|
||||
| [One bucket per artifact class, not everything under `manga`](storage-layout.md#bucket-per-artifact) | closed |
|
||||
| [The orchestrator creates missing buckets at startup](storage-layout.md#ensure-buckets) | closed |
|
||||
| [RustFS is staged, not adopted](storage-layout.md#rustfs-staged) | open |
|
||||
| [A model guess is never labelled `tail`](speaker-attribution.md#no-fake-tail) | closed |
|
||||
| [The model's speaker answer is resolved against what the prompt showed](speaker-attribution.md#prompt-label-answers) | closed |
|
||||
| [An unpaired set-of-mark label grounds nothing](speaker-attribution.md#unpaired-mark) | closed |
|
||||
| [An interjection is not a name and not a misquote](speaker-attribution.md#interjection-false-positive) | closed |
|
||||
| [Cast names enter the verifier tokenized](speaker-attribution.md#multiword-cast-names) | closed |
|
||||
| [Clearing a stage strips the vision blob it wrote](storage-layout.md#clear-vision-blob) | closed |
|
||||
| [The shipped 72.7s gap was a stream copy across mixed frame rates](chapter-assembly.md#mixed-rate-stream-copy) | closed |
|
||||
| [xfade offsets come from `min(video, audio)`, never `format=duration`](chapter-assembly.md#offsets-from-min-stream) | closed |
|
||||
| [A vision `bbox` is gemma's 0-1000 grid, converted to pixels at `/vision`](identity-bbox.md#bbox-is-normalized) | closed |
|
||||
| [Assembly verifies its own output instead of trusting ffmpeg's exit code](chapter-assembly.md#check-assembled) | closed |
|
||||
| [The single-item passthrough is not the assembly bug](chapter-assembly.md#passthrough-innocent) | void |
|
||||
| [A stage result proves nothing until the worker is newer than the edit](identity-bbox.md#stale-worker-invalidates) | closed |
|
||||
| [A detection with no detected face never enrolls or binds](identity-bbox.md#face-gates-enrollment) | closed |
|
||||
| [A resolver NONE mints an anonymous character, it does not clear the crop](identity-bbox.md#none-mints-an-anonymous-character) | closed |
|
||||
| [The extras gate runs at enrollment and at narration, not at the speaker prompt](identity-bbox.md#extras-gate-consumers) | closed |
|
||||
| [A roster name is a guess, so it never reaches detection](identity-bbox.md#roster-does-not-name) | closed |
|
||||
| [`merged_into` is exactly one hop deep](identity-bbox.md#merge-chains-flatten) | closed |
|
||||
| [`_bbox_to_pixels` orders the corners, because the model sometimes swaps them](identity-bbox.md#bbox-corners-ordered) | closed |
|
||||
| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed |
|
||||
| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed |
|
||||
| [The gallery is the live cast, not cosine's top-k](identity-bbox.md#cast-is-the-gallery) | closed |
|
||||
| [A name is a word set, not a string](identity-naming.md#alias-grouping) | closed, GPU pending |
|
||||
| [A confident caption names a character on its own](identity-naming.md#caption-is-strong) | closed, GPU pending |
|
||||
| [A name belongs to one character](identity-naming.md#one-name-one-character) | closed, GPU pending |
|
||||
@@ -0,0 +1,144 @@
|
||||
# Audit Phase 1: correctness and scheduling safety
|
||||
|
||||
Settled 2026-08-11 from `AUDIT.md`. Every entry was verified by a runnable check in the same commit.
|
||||
No GPU work ran and no pipeline run was executed. Each claim rests on source and on the CPU-only
|
||||
self-checks named below.
|
||||
|
||||
Files: `worker_scene.py`, `worker_script.py`, `worker_vision.py`, `session_manager.py` on workpc, and
|
||||
`tracklets.py`, `correctness.py`, `db.py`, `service.py`, `session_proxy.py` in the homesrv orchestrator
|
||||
(`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`).
|
||||
|
||||
## Speaker ref is canonical {#speaker-ref-is-canonical}
|
||||
|
||||
**Closed.** A dialogue row's speaker is `speaker_ref` when its kind is `character_id`. The flat
|
||||
`speaker` field is a compatibility value. The orchestrator already rewrote it from a panel-local id to
|
||||
a `character_id`.
|
||||
|
||||
Evidence: `correctness.py:normalize_dialogue` rewrites `row["speaker"]`. `worker_scene.build_scene`
|
||||
mapped that value through `id_by_local`, whose keys are panel-local ids. Every lookup returned `None`,
|
||||
so `worker_script._render_line` narrated every line as `Someone`.
|
||||
|
||||
Forbids: reading `speaker` as a panel-local id anywhere downstream of the dialogue stage.
|
||||
Check: `python worker_scene.py`, the `p007` case.
|
||||
|
||||
The same field leaked into the dialogue resume path, where a stored `character_id` was pushed into
|
||||
`recent` as a speaker name. It now renders the registry name (`service.py`, `run_stage_dialogue`).
|
||||
|
||||
## Verifier false positives {#verifier-false-positives}
|
||||
|
||||
**Closed.** `verify_script` must stay quiet on valid narration. A failure is not soft. The script stage
|
||||
retries once and then raises, so one false positive halts the whole chapter.
|
||||
|
||||
Two rules fired on correct output.
|
||||
|
||||
1. `unsupported-proper-noun` flagged any capitalized token outside a 20-word stop list. That caught
|
||||
ordinary sentence openers (`Suddenly`, `Behind`, `Inside`, `Both`, `Everyone`, `After`, `Two`) and
|
||||
`Someone`. A capital opening a sentence is now grammar unless the word falls outside a real
|
||||
narration vocabulary. A capital in the middle of a sentence still counts as name evidence.
|
||||
2. `misquote` compared each quoted span against a WHOLE source line at ratio 0.82. The narration prompt
|
||||
asks for a short quote of the actual words. A 15-character quote from a 40-character line scored
|
||||
about 0.55. Grounding now matches the longest contiguous span instead.
|
||||
|
||||
Forbids: adding a verifier rule without a test that a correct beat passes it.
|
||||
Check: `pytest test_script_verify.py` in the orchestrator.
|
||||
|
||||
Verifier feedback now reaches the retry. `worker_script.ScriptInput` declares `beat` and
|
||||
`verifier_feedback`. Pydantic v2 dropped both silently before, so the retry was another blind sample.
|
||||
The repair prompt names each failure and lists the exact quotable lines from the beat.
|
||||
|
||||
## Tracklet hard constraints {#tracklet-hard-constraints}
|
||||
|
||||
**Closed.** Tracklet linking is transitive over similarity evidence and never across a hard constraint.
|
||||
|
||||
The hard constraints are gender and co-presence. Gender is canonicalized first. Vision emits
|
||||
`m|f|unknown` while the registry says `male|female`, and the old gate tested only the second spelling,
|
||||
so it never fired on real data. Co-presence means two detections in one panel are two people by
|
||||
construction. A merge is rejected when any cross pair between the two groups violates either rule.
|
||||
A weak chain can no longer bridge two people seen together.
|
||||
|
||||
Forbids: comparing a raw gender string against a literal enum anywhere in the identity path.
|
||||
Check: `python tracklets.py` and `pytest test_tracklets.py`.
|
||||
|
||||
## Gated face pairing {#gated-face-pairing}
|
||||
|
||||
**Closed.** A detected face takes a character's identity only when its centre falls inside that
|
||||
character's gemma bbox. The box is grown by 25% first. An unpaired face stays `unknown`.
|
||||
|
||||
This matters because the label becomes `speaker_method="som_face"`, the highest-trust provenance the
|
||||
pipeline records. Unconditional nearest-neighbour pairing laundered a geometric guess into evidence.
|
||||
Pairs are now taken globally shortest-first, so the first face processed cannot claim a character that
|
||||
fits a later face better.
|
||||
|
||||
Set-of-mark attribution stays ON by default now that the pairing is gated. The docstring said "off by
|
||||
default" while the flag read `SOM_ATTRIBUTION` with default `"1"`. The code and the comment now agree.
|
||||
|
||||
Forbids: trusting a `som_face` speaker without the containment gate.
|
||||
Check: `python worker_vision.py`.
|
||||
|
||||
## No lease stealing {#no-lease-stealing}
|
||||
|
||||
**Closed.** A 409 from `/session/open` means another job legitimately holds the GPU. The proxy queues
|
||||
on it. Only the session manager's TTL reaper clears a dead lease, because only it can tell a dead
|
||||
lease from a busy one.
|
||||
|
||||
`session_proxy.open_session` previously closed the active lease and retried, so two jobs terminated
|
||||
each other's models. Job admission control now bounds concurrent pipelines through
|
||||
`MAX_CONCURRENT_JOBS`, default 1. A lease queue alone still lets two pipelines interleave stages
|
||||
against one GPU.
|
||||
|
||||
`heartbeat_session` is finally called. `open_session` starts a beat task and `close_session` cancels
|
||||
it, so a stage running past its TTL is no longer reaped mid-run. Beating from open and close keeps all
|
||||
nine stage call sites unchanged.
|
||||
|
||||
Forbids: closing a session the caller did not open.
|
||||
|
||||
## Flag resolution {#flag-resolution}
|
||||
|
||||
**Closed.** Correctness flags resolve by flag id, and unresolved flags never wedge an autonomous run.
|
||||
|
||||
Two defects combined into a permanent block. `get_correctness_flags` accepted rows whose panel no
|
||||
longer existed. A flag raised against a panel that a crop rerun deleted became visible to every
|
||||
chapter and could never be resolved. `resolve_correctness_flags` could only touch flags whose
|
||||
panel still existed. The view no longer admits orphans, and resolution takes explicit flag ids.
|
||||
|
||||
The TTS block also had no reachable exit. It refused to start on four flag kinds that only
|
||||
`/review/approve` clears, and `GATES` defaults to off. With gates on, TTS now returns rather than
|
||||
raises, so `awaiting_review` survives instead of being overwritten by `failed` in the pipeline's
|
||||
catch-all. With gates off, the flags are logged and cleared, because no reviewer exists to clear them.
|
||||
|
||||
Forbids: raising out of a stage that has just set `awaiting_review`.
|
||||
Check: `pytest test_name_binding.py` in the orchestrator.
|
||||
|
||||
## Hallucinated index {#hallucinated-index}
|
||||
|
||||
**Closed.** An out-of-range `choice` from `/vision/resolve` is `unresolved`, like a parse failure. Only
|
||||
an explicit `0` means NONE and mints a new character. Mapping a bad index to NONE created a brand new
|
||||
entry in the permanent registry from a hallucination.
|
||||
|
||||
Check: `python worker_vision.py`.
|
||||
|
||||
## Unlocked model load {#unlocked-model-load} [#203]
|
||||
|
||||
**Closed.** The session manager never holds `_lock` across a model load or a health wait.
|
||||
|
||||
`open_session` was already written that way, but it could orphan the server it spawned. A
|
||||
`/session/close` arriving during the load found `proc = None`, tore down nothing, and cleared
|
||||
`_active`. The finished server then held its VRAM unreferenced, and the next open spawned a second one
|
||||
on the same port. Open now tears down its own process and reports 409 when its lease vanished mid-load.
|
||||
|
||||
`_supervise_once` had the opposite asymmetry. It called `_start_subprocess` while holding `_lock`,
|
||||
which blocked `/session/active`, `/session/close`, and `/session/open` for the full health wait. It now
|
||||
claims the respawn by clearing `proc`, spawns unlocked, and tears the new process down if the lease
|
||||
disappeared meanwhile.
|
||||
|
||||
Check: `python session_manager.py`.
|
||||
|
||||
## Related
|
||||
|
||||
`_extract_json` in both `worker_vision.py` and `worker_script.py` now uses
|
||||
`json.JSONDecoder().raw_decode` from the first brace. The greedy `\{.*\}` ran to the LAST brace in the
|
||||
reply. A second object or trailing braced prose burned a repair call on a response that parsed fine.
|
||||
|
||||
`worker_scene.build_scene` emits `actions` as a list beside the joined `action` string, because
|
||||
`correctness.build_beat_artifact` reads the plural key. The verifier was receiving no action evidence
|
||||
at all. The beat builder also falls back to splitting the singular string.
|
||||
@@ -0,0 +1,107 @@
|
||||
# chapter-assembly
|
||||
|
||||
Settled questions about `_assemble_batched` / `_assemble_once` / `_xfade_chain` in `worker_render.py`.
|
||||
|
||||
## The shipped 72.7s gap was a stream copy across mixed frame rates {#mixed-rate-stream-copy}
|
||||
|
||||
**Closed, 2026-08-12.**
|
||||
|
||||
`assemble` routes an all-`cut` chapter to a `concat` demuxer with `-c copy`. That path writes the output
|
||||
with the **first** input's `time_base` and reinterprets every later packet in it.
|
||||
|
||||
14 of this chapter's 49 clips came off `collage_cmd`, which hardcoded `-r 30`. They carry
|
||||
`r_frame_rate=30/1` and `time_base=1/15360`. The other 35 are `25/1` at `1/12800`. Copied into the first
|
||||
clip's timebase, those 14 play `15360/12800 = 1.2` times too long while their audio is untouched. That is
|
||||
the 1.2001 ratio, and the whole of video 436.39s over audio 363.67s.
|
||||
|
||||
Reproduced offline by running the same `-c copy` concat over the 49 real clips. Duration 436.392031 and
|
||||
`nb_frames` 9902, identical to the shipped file. Seconds to run, no GPU.
|
||||
|
||||
Two changes hold it closed:
|
||||
|
||||
* `collage_cmd` emits `-r FPS` like every other clip path, and the `__main__` self-check asserts
|
||||
`_fps_of(clip) == "25/1"` on a real collage encode.
|
||||
* `assemble` probes `r_frame_rate` across the clips and sends mixed rates through `_assemble_batched`,
|
||||
whose branches both normalize with `fps={FPS}`. Only a single shared rate keeps the stream copy.
|
||||
|
||||
Verified end to end. The rebuilt `chapter.mp4` is video 364.120s against audio 364.122s at `25/1`.
|
||||
|
||||
**An earlier version of this file, and commit `1457556`, blamed `#offsets-from-min-stream` below for the
|
||||
shipped gap. That was wrong.** The rebuild came out byte-identical to the broken file, which proved the
|
||||
xfade tree never ran for this chapter. The entry below is a real defect and stays closed on its own
|
||||
evidence. It was not this one.
|
||||
|
||||
## xfade offsets are computed from `min(video, audio)`, never `format=duration` {#offsets-from-min-stream}
|
||||
|
||||
**Closed, 2026-08-12.** A real latent defect on the transition path. Not the cause of the shipped gap,
|
||||
see `#mixed-rate-stream-copy` above.
|
||||
|
||||
`_xfade_chain` accumulates `cum += dur[i] - td` and hands each boundary `offset=cum-td`. That offset is
|
||||
an assertion about where input `i-1` still has frames. It fed on `_audio_dur`, which probes
|
||||
`format=duration`, which is `max(video, audio)`. A rendered clip's audio outlasts its video by about a
|
||||
frame. So every boundary pushed the accumulator further ahead of the picture.
|
||||
|
||||
Once the accumulated overshoot exceeds the transition width, the xfade window starts after the last frame
|
||||
of input `i-1`. ffmpeg emits the transition and then **silently discards input `i` and every clip
|
||||
downstream of it**. `rc 0`, no warning on stderr, output file present and playable. Measured on a group of
|
||||
8 real clips, chain truncated at each stage:
|
||||
|
||||
```
|
||||
k=7 out= 52.52 correct
|
||||
k=8 out= 52.52 the last xfade contributed nothing
|
||||
[v6][n7]xfade=duration=0.050:offset=52.500 <- [v6] is only 52.52s long, so 0.02s of margin
|
||||
```
|
||||
|
||||
This is the whole cause of the shipped chapter being video 436.39s over audio 363.67s. It reproduces with
|
||||
synthetic clips in about four minutes and needs no GPU.
|
||||
|
||||
Two changes hold it closed:
|
||||
|
||||
* `_assemble_once` probes `min(_stream_dur(p, "v"), _stream_dur(p, "a"))`. The minimum, because either
|
||||
stream running long breaks a different half of the graph.
|
||||
* `_xfade_chain` floors every input to a whole frame count. It applies `trim` and `atrim` to both streams,
|
||||
so the accumulator tracks the real timeline instead of estimating it. Transition widths are quantized to
|
||||
frames for the same reason.
|
||||
|
||||
Verified over the 49 real clips of chapter `7c944dd4`. The round that previously turned 359s of video into
|
||||
100s now loses 0.85s. The chapter comes out video 358.76s against audio 358.76s, agreeing to the frame.
|
||||
|
||||
Forbidden from here: `_audio_dur` in anything that positions a filter. It is fine for "how long is this
|
||||
clip roughly", nothing else.
|
||||
|
||||
## Assembly verifies its own output instead of trusting ffmpeg's exit code {#check-assembled}
|
||||
|
||||
**Closed, 2026-08-12.**
|
||||
|
||||
Both assembly branches drop stream time without failing. xfade discards inputs as above. The concat
|
||||
demuxer with `-vsync cfr` drops video frames to force a constant rate. Neither is an error to ffmpeg.
|
||||
|
||||
So `_assemble_once` calls `_check_assembled(out, expect)` after every encode. It compares the output's
|
||||
video stream against the predicted timeline and against its own audio stream. It raises when either is off
|
||||
by more than `ASSEMBLE_TOL_S`, which is 0.5s. That tolerance covers frame boundaries and aac padding. A
|
||||
dropped input is off by whole seconds.
|
||||
|
||||
Without this the failure stays invisible until somebody watches the video. That is how a 50MiB chapter
|
||||
with 72.7s of silent picture reached the bucket while every stage counter read success.
|
||||
|
||||
## The single-item passthrough is not the bug {#passthrough-innocent}
|
||||
|
||||
**Void, 2026-08-12.** Cited in `HANDOFF.md` for 2026-08-11 as the prime suspect and must not be cited
|
||||
again.
|
||||
|
||||
With 49 clips and `ASSEMBLE_BATCH=8`, round 0 makes six groups of 8 plus a leftover group of 1, which
|
||||
`_assemble_batched` carries forward un-encoded. The theory was that mixing that raw clip with six encoded
|
||||
intermediates broke round 1. The instrumented run disproves it: **two round-0 groups of 8 fresh clips
|
||||
collapse on their own**, before any passthrough exists.
|
||||
|
||||
```
|
||||
n=8 XFADE in v= 63.52 a= 63.60 -> out v= 52.52 a= 62.77 lost_v= +11.00
|
||||
n=8 XFADE in v= 58.12 a= 58.20 -> out v= 12.04 a= 56.83 lost_v= +46.08
|
||||
```
|
||||
|
||||
Round 1's 259s loss was the cascade. Its inputs already held 309s of video against 364s of audio, and
|
||||
`durs` read the audio.
|
||||
|
||||
Consequence for the plan: collapsing `concat`, `xfade` and the passthrough into one path was the
|
||||
recommended fix in `NEXT.md` and is **not needed**. Three paths are fine once each one positions filters
|
||||
on a real timeline. The tree keeps the bounded memory it was built for.
|
||||
@@ -0,0 +1,335 @@
|
||||
# identity-bbox
|
||||
|
||||
The coordinate space of a vision character box, and what reads it.
|
||||
|
||||
## A vision `bbox` arrives on gemma's 0-1000 grid, and `/vision` converts it to pixels {#bbox-is-normalized}
|
||||
|
||||
**Closed, 2026-08-12.**
|
||||
|
||||
`build_detect_prompt` asks for a "pixel bounding box". The model answers on its own normalized grid
|
||||
regardless. Measured over all 113 detections of job `778297bc`, read straight from `/review/identity`:
|
||||
|
||||
| test | result |
|
||||
| --- | --- |
|
||||
| boxes with `x2` past the 900px panel width | **47 of 113** |
|
||||
| boxes with `y2` past 1000, on panels 1257 to 2307px tall | **0 of 113** |
|
||||
| boxes clamped at exactly 1000 | 21 in x, 5 in y |
|
||||
| coordinate range over every box | `[0, 1000]` |
|
||||
|
||||
Pixels cannot behave that way. A person standing in the lower half of a 2307px panel needs `y2` near
|
||||
2000, and it never once exceeds 1000.
|
||||
|
||||
Consumed as pixels the boxes collapse into the top-left corner of the panel. Four consumers were reading
|
||||
them:
|
||||
|
||||
* `worker_identity._crop_bbox` at `worker_identity.py:200`, which embeds the crop. This is why a crop of
|
||||
a speech balloon's edge matched `Choi Haeseon` at 0.9.
|
||||
* `_pair_faces_to_present` in `worker_vision.py`, which compares real detector face boxes, in pixels,
|
||||
against these. The gate could almost never pass, which is the mechanism behind the 7 `unknown` results
|
||||
out of 7 `som_face` lines already recorded at `worker_vision.py:169`.
|
||||
* the set-of-mark boxes drawn for attribution.
|
||||
* the review UI, which crops client-side off the panel PNG.
|
||||
|
||||
`/vision` now calls `_bbox_to_pixels(characters, w, h)` before returning, so all four see pixels and no
|
||||
consumer needs to know the grid existed. Verified by drawing the converted boxes on panel 7. Five of six
|
||||
land on their subject, including `person_5`, who is Seonho in the foreground and had no identity.
|
||||
`person_1` still frames a window mullion with nobody in it, which is `#extras-as-cast`, not this.
|
||||
|
||||
The prompt text still says "pixel bounding box". Rewording it changes what the model emits and needs a
|
||||
GPU run to re-verify, so the boundary converts instead. The `ponytail:` note on `_bbox_to_pixels` records
|
||||
that. It also records the trap: a model that really answered in pixels would be scaled down here.
|
||||
|
||||
**Consequence: every assignment in the registry came from a wrong crop.** The existing embeddings and
|
||||
`ref_image_uris` are enrolled on balloons and window frames. Re-running identity
|
||||
is what makes the registry mean anything. The anonymous-identity and extra-versus-cast work cannot be
|
||||
judged until that rerun happens.
|
||||
|
||||
## A stage result proves nothing until the worker is newer than the edit {#stale-worker-invalidates}
|
||||
|
||||
**Closed, 2026-08-12.** The first rerun after the bbox fix reproduced the defect exactly: 46 of 110
|
||||
boxes past the 900px panel width, coordinates clamping at 1000, `y2` never once past 1000 on panels up
|
||||
to 2307px tall. The same fingerprint as `#bbox-is-normalized` measured before the fix.
|
||||
|
||||
The fix was not wrong. It was not loaded.
|
||||
|
||||
```
|
||||
vision worker process started 12:00:09
|
||||
worker_vision.py modified 12:11:35
|
||||
8113bdf, which contains _bbox_to_pixels, committed 12:16:22
|
||||
```
|
||||
|
||||
Python binds a module once, at process start. `./start_workers.sh` had launched the worker eleven
|
||||
minutes before the file changed, so `/vision` served pre-fix code for the whole run and returned raw
|
||||
grid boxes. Nothing in the result said so. The stage reported `completed 116/116`, the orchestrator
|
||||
recorded no error, and identity and reconcile ran to completion on top of it. Cost: one full
|
||||
vision + identity + reconcile cycle, plus a registry reset to undo the 8 characters it minted.
|
||||
|
||||
The rerun against a restarted worker gives the opposite reading over the same 110 detections: 0 boxes
|
||||
past the width, 0 past the height, one coordinate on 1000 which is now a real pixel value, and a
|
||||
deepest box reaching 100% down its panel with `max y2 = 2307`. Boxes track the panel, so they are
|
||||
pixels.
|
||||
|
||||
`check_stale.sh` compares every running worker's process start against its module's mtime and exits
|
||||
non-zero if any is stale. This failure mode was already known as advice — the render worker "must be
|
||||
restarted by hand to pick up an edit" — and advice did not stop it happening. Run the check before any
|
||||
stage run that is meant to prove a code change.
|
||||
|
||||
Forbids: citing a stage result as evidence about a code change without establishing that the worker
|
||||
serving it postdates the change.
|
||||
|
||||
## A detection with no detected face never enrolls or binds {#face-gates-enrollment}
|
||||
|
||||
**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**
|
||||
|
||||
Fixing the coordinate space made the extras problem worse, not better. With the boxes finally landing
|
||||
on their subjects, panel 7's four background extras became four good crops of four irrelevant people,
|
||||
and one of them bound to `Seonho` at confidence 1.00. Before the fix the same detection was a crop of
|
||||
scenery and matched nothing much. Correct geometry turned a harmless failure into a poisoned reference
|
||||
set for the lead.
|
||||
|
||||
The measured panel 7 outcome, converted boxes, against the art:
|
||||
|
||||
| box | who | assigned |
|
||||
| --- | --- | --- |
|
||||
| `[457, 657, 642, 937]` | Seonho, foreground | `Seonho` |
|
||||
| `[669, 591, 763, 822]` | the colleague, unnamed in the story | `character_f7a4fd`, anonymous |
|
||||
| `[428, 386, 496, 526]` | background extra | none |
|
||||
| `[34, 414, 122, 564]` | background extra | none |
|
||||
| `[498, 386, 568, 533]` | background extra | `character_d72710` at 0.94 |
|
||||
| `[31, 554, 94, 728]` | background extra | `Seonho` at 1.00 |
|
||||
|
||||
`/vision` now stamps `has_face` on every character by running `face_detect.detect_faces` on the panel
|
||||
and reusing `_pair_faces_to_present` for containment, so the gate uses the same margin and the same
|
||||
global shortest-first assignment as the speaker path. `worker_identity.py` skips a character with
|
||||
`has_face is False` before it crops, embeds, matches or mints.
|
||||
|
||||
Two properties are deliberate. It **fails open**: a missing or raising detector marks every character
|
||||
`True`, because dropping a whole panel's cast is worse than the over-detection the gate exists to trim.
|
||||
And it gates on `is False`, not falsiness, so a vision blob written before this change (no key) behaves
|
||||
as it did rather than silently dropping every character.
|
||||
|
||||
Cost: a cast member drawn from behind, or in a style the detector misses, now takes no identity on that
|
||||
panel. That is the abstain this pipeline already prefers to a wrong bind
|
||||
(`caveats/speaker-attribution.md#no-anonymous-identity`).
|
||||
|
||||
Forbids: enrolling a reference crop, or binding a character, from a region no face detector confirms.
|
||||
|
||||
## A resolver NONE mints an anonymous character, it does not clear the crop {#none-mints-an-anonymous-character}
|
||||
|
||||
**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**
|
||||
|
||||
`caveats/speaker-attribution.md#no-anonymous-identity` asked whether the Tier-2 gemma resolver can answer
|
||||
"none of these". It can, and it always could. `/vision/resolve` at `worker_vision.py:1071` maps `choice: 0`
|
||||
to `state="new"`, an out-of-range index to `state="unresolved"`, and a parse failure to `unresolved` as
|
||||
well. The abstain path was never the defect.
|
||||
|
||||
The defect was one branch on the other side of the contract. The orchestrator read only
|
||||
`v.get("character_id")` and treated every falsy value the same way: `unassign_identity` on every crop of
|
||||
the tracklet. So a deliberate "this is a real person the roster does not hold" and a hallucinated index
|
||||
both produced nothing, and the unnamed colleague was `unknown` on every panel she appeared on. The stale
|
||||
`ponytail:` comment above that block named the reason nobody fixed it, and the reason was real: minting a
|
||||
character needs an `embedding_uri`, and the orchestrator cannot compute one. siglip is resident in the
|
||||
identity worker, gemma is resident in the vision worker, and `session_manager` forbids both at once.
|
||||
|
||||
What removes the blocker is carrying the embedding, not a third GPU pass. `/identity/resolve` already
|
||||
computes an embedding per crop and already uploads the crop to
|
||||
`s3://manga/{manga}/characters/_crops/{panel}_{local}.png`. It now writes the embedding to the same key
|
||||
with a `.npy` suffix and returns `emb_uri` in each shortlist entry. The mint is then a local
|
||||
`create_character(manga_id, None, appearance, [crop_uri], emb_uri, gender)`, and the existing per-tracklet
|
||||
assign loop binds every member to it.
|
||||
|
||||
`tracklets.resolve_outcome` holds the three-way decision as a pure function, so the branch that runs is
|
||||
the branch the self-check covers: `known` on a named answer, `mint` on `state="new"` with an `emb_uri`,
|
||||
`clear` on `unresolved`, on a NONE with no embedding, and on an older worker that sends no `state` at all.
|
||||
|
||||
Two limits are deliberate. A tracklet's candidate gallery is built before the loop mints anything, so one
|
||||
person split across two unlinked tracklets still gets two anonymous ids; `run_stage_reconcile` merges
|
||||
unnamed twins on appearance overlap and is what folds them. And an anonymous character's text sheet
|
||||
(`worker_vision._sheet`) carries no name, so gemma re-recognising it on a later panel leans on the
|
||||
reference images rather than the description.
|
||||
|
||||
Contract: `shortlists[].emb_uri` is new in the `/identity/resolve` response. Invariant 7 — both repos
|
||||
changed in the same session.
|
||||
|
||||
Forbids: treating an absent `character_id` as one outcome. A resolver that answered and a resolver that
|
||||
failed are different facts.
|
||||
|
||||
## The extras gate runs at enrollment and at narration, not at the speaker prompt {#extras-gate-consumers}
|
||||
|
||||
**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**
|
||||
|
||||
`#face-gates-enrollment` stops a faceless detection taking an identity. It does not stop the detection
|
||||
being narrated, because three places read the raw vision character list and only one of them consults an
|
||||
assignment. Panel 7 is the worked example: six detections, four of them extras or scenery.
|
||||
|
||||
`worker_scene.build_scene` already drops an unassigned detection from `characters` and `present`
|
||||
(`worker_scene.py:63`), so an extra never reached the cast list. Its **action** did. `actions` was built
|
||||
from every detection, and that list is what the script prompt renders and what the verifier uses as
|
||||
evidence. So "standing at the window" arrived as a fact about the panel with no character attached, and
|
||||
the verifier confirmed it because the action really was in the blob. Both now skip `has_face is False`.
|
||||
|
||||
`service._beat` picks the cinematographer's "who" from the first three detections, falling back to a
|
||||
detection's action when it has no name. An extra could take a slot and steer the camera. Also gated.
|
||||
|
||||
`service._present_characters` is deliberately **not** gated. It builds the dialogue stage's candidate
|
||||
speaker list and the set-of-mark boxes. Two reasons. The failure is already contained: an extra chosen as
|
||||
the speaker has no identity assignment, so `normalize_dialogue` resolves it to unknown rather than to a
|
||||
wrong name. And the gate's own cost lands hardest here, because a character drawn from behind has no face
|
||||
box, so gating would delete a real speaker from the only list that can attribute their line.
|
||||
|
||||
All three gates test `is False`, never falsiness. A vision blob written before the gate existed carries no
|
||||
`has_face` key, and a panel whose detector failed is marked `True` by the fail-open path. Both keep their
|
||||
previous behaviour.
|
||||
|
||||
Forbids: adding a fourth consumer of `vision["characters"]` without deciding which side of this line it is
|
||||
on. The blob keeps every detection on purpose, so the audit can still see what was gated.
|
||||
|
||||
## A roster name is a guess, so it never reaches detection {#roster-does-not-name}
|
||||
|
||||
`service.py` seeded `known_characters` with `_roster_char_hints` before every detect call, so the chapter
|
||||
roster's names were in front of gemma before any panel had identified anyone. On the 2026-08-12 16:39 run
|
||||
that put "Seonho (short brown hair, yellow plaid shirt)" on a different man wearing glasses, who then held
|
||||
15 assignments under the lead's name. The real lead was minted separately from the p010 caption as
|
||||
"Lim Seonho", and p020 held both as two people in one panel.
|
||||
|
||||
The seeding is removed. A name now reaches detection only from a registry row, and a registry row is named
|
||||
from an in-panel caption or address through `name_claims`. The distinction is evidence: a roster name is a
|
||||
claim about art nobody has looked at yet, and a registry row carries an embedding plus the panel that
|
||||
named it.
|
||||
|
||||
The roster still feeds `roster_cast` in `run_stage_dialogue`. There the names are matched against speech,
|
||||
not against faces, so a wrong guess costs an unresolved speaker rather than a wrong face binding.
|
||||
|
||||
Confirmed on the 17:38 run: "Lim Seonho" came back as one row with 25 assignments, so dropping the hint
|
||||
did not split the lead across the panels before his caption. "Seonho" fell to 1 assignment.
|
||||
|
||||
Not covered: detection still reads a name off the panel and can attach it to the wrong body. p011 and p026
|
||||
emit `name: "Seonho"` on `person_2` with no roster hint present. That is a separate mechanism and it is
|
||||
what keeps the glasses man named at all.
|
||||
|
||||
## `merged_into` is exactly one hop deep {#merge-chains-flatten}
|
||||
|
||||
A merge chain means `merged_into` points at a row that is itself merged, so anything resolving one hop
|
||||
lands on a retired character. Roster readers filter `merged_into IS NULL` and were never affected.
|
||||
|
||||
Two rules, one per direction, and both are needed:
|
||||
|
||||
- `merge_characters` resolves the keeper to its chain root before merging. Bounded 64-step walk, the cap
|
||||
only exists so a cycle cannot hang reconcile.
|
||||
- retiring a loser repoints every row whose `merged_into` was that loser.
|
||||
|
||||
The keeper walk alone is not enough, which the 17:38 run proved by producing
|
||||
`477c1894 -> a92d9df4 -> 4fb94c15` with the walk deployed. At merge time that pair was fine. The chain
|
||||
formed later, when a row that was already somebody's keeper was itself retired.
|
||||
|
||||
## `_bbox_to_pixels` orders the corners, because the model sometimes swaps them {#bbox-corners-ordered}
|
||||
|
||||
**Closed, 2026-08-12.**
|
||||
|
||||
The 17:38 run returned `p007 person_1` as `[226, 417, 130, 551]`, x1 greater than x2. One detection in 117.
|
||||
Clamping each coordinate into the panel kept the swap, so the box stayed a negative-width region. It
|
||||
cropped to nothing, so that detection could not enroll, embed or match, and nothing reported the loss.
|
||||
|
||||
`_bbox_to_pixels` now sorts each pair after clamping:
|
||||
|
||||
```python
|
||||
xs = sorted((clamped_x1, clamped_x2))
|
||||
ys = sorted((clamped_y1, clamped_y2))
|
||||
c["bbox"] = [xs[0], ys[0], xs[1], ys[1]]
|
||||
```
|
||||
|
||||
Sorting is enough. A zero-area box still crops to nothing, and no consumer needs a minimum size that it
|
||||
does not already enforce. The self-check feeds the real swapped box in and asserts `[117, 688, 203, 909]`,
|
||||
which fails without the sort.
|
||||
|
||||
Proven on the 18:07 run: 0 degenerate boxes over 119 detections.
|
||||
|
||||
## A generic word is not identity evidence, and one tokenizer serves both consumers {#generic-tokens}
|
||||
|
||||
**Closed, 2026-08-12, not yet run on a GPU.**
|
||||
|
||||
The tracklet linker and reconcile's pair pre-filter each carried their own copy of the appearance
|
||||
tokenizer, and both linked on words that describe a person without distinguishing them. Measured over the
|
||||
18:07 run's 22 crops for the lead:
|
||||
|
||||
```
|
||||
p056~p057 hair=['short'] cloth=- two different men
|
||||
p047~p054 hair=['hair'] cloth=- "brown hair" vs "dark hair", the field name in its own value
|
||||
p109~p110 hair=- cloth=['shirt','white']
|
||||
```
|
||||
|
||||
`tracklets.appearance_tokens` is now the single implementation and subtracts a `GENERIC` set: length words
|
||||
(`short`, `long`, `medium`, `shoulder`, `length`), `hair`/`haired`, garment words (`shirt`, `top`,
|
||||
`jacket`, `coat`, `sleeve`), and filler (`plain`, `casual`, `none`, `unknown`). `service._appearance_tokens`
|
||||
delegates to it, so reconcile's pre-filter is fixed by the same change. That pre-filter is what let the
|
||||
pink tank top reach `/vision/same` at all.
|
||||
|
||||
Four asserts cover it and all four fail with `GENERIC` emptied.
|
||||
|
||||
Measured effect on those 22 crops, with candidate overlap forced to pass (the shortlists are not stored,
|
||||
so this is the most permissive assumption and the real split can only be finer):
|
||||
|
||||
| | tracklets | worst span |
|
||||
| --- | --- | --- |
|
||||
| as it ran | 3 | 30 panels |
|
||||
| span cap only | 8 | 8 |
|
||||
| span cap + `GENERIC` | 9 | 8 |
|
||||
|
||||
The stopwords alone split out `p054`, the beige-jacket man, and `p089`, the red-robe chibi.
|
||||
|
||||
**Not fixed by this.** Bare hair colour still links different men: `order 56` through `62` stay in one
|
||||
five-crop tracklet on `brown`, and the cat at `p098` still joins its neighbours the same way. The obvious
|
||||
next lever, a crop-to-crop cosine, is ruled out by `caveats/audit-open.md#cosine-not-identity`.
|
||||
|
||||
## A tracklet is bounded by span, not only by pairwise distance {#tracklet-span-cap}
|
||||
|
||||
**Closed, 2026-08-12, not yet run on a GPU.**
|
||||
|
||||
`window=8` bounded each PAIR, and linking is transitive, so nothing bounded the group. The 18:07 run
|
||||
resolved 12 tracklets over 64 crops, and the lead's 22 native assignments fell into two of them spanning
|
||||
panels 47-69 and 80-110. One gemma answer then covered a grey blazer, a denim jacket, a red robe and a cat.
|
||||
The two blocks are visible in the stored confidences: eleven rows at 0.95 and ten at 1.0.
|
||||
|
||||
`link_tracklets` now rejects a merge whose resulting group would span more than `window` panels:
|
||||
|
||||
```python
|
||||
orders = [dets[m].get("panel_order", m) for m in gi + gj]
|
||||
if max(orders) - min(orders) > window:
|
||||
continue
|
||||
```
|
||||
|
||||
This makes the module docstring's claim true. The self-check links three compatible detections at panels 1,
|
||||
9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled.
|
||||
|
||||
Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.
|
||||
|
||||
## The gallery is the live cast, not cosine's top-k {#cast-is-the-gallery}
|
||||
|
||||
**Closed, 2026-08-12, not yet run on a GPU.**
|
||||
|
||||
`run_stage_identity` built each tracklet's gallery by unioning the members' cosine shortlists, so the
|
||||
metric that `caveats/audit-open.md#cosine-not-identity` shows cannot separate people decided who gemma was
|
||||
even allowed to consider. Two different men reach 0.93 on this chapter's crops where the same man reaches
|
||||
0.96. When the right character fell outside every member's top-5, gemma could only pick a wrong one or
|
||||
answer NONE, and NONE mints a duplicate.
|
||||
|
||||
The cast is small. `tracklets.cast_gallery` builds the gallery from `get_known_characters` instead:
|
||||
gender-compatible rows only, named first, then registry order. Cosine still shortlists per crop, and those
|
||||
shortlists are still the linker's candidate-overlap evidence, but they no longer bound the answer.
|
||||
|
||||
Three consequences, each deliberate:
|
||||
|
||||
- **Every crop now reaches the resolver.** A crop only entered `shortlists` when its cosine top-k was
|
||||
non-empty, so an early crop seen while the roster was still empty, or one with no gender-compatible row,
|
||||
was dropped from resolution and could never mint. The guard is gone.
|
||||
- **The gallery is re-read per tracklet**, so a character an earlier tracklet minted is visible to the
|
||||
later ones. That closes the `ponytail:` note beside the mint path.
|
||||
- **Two caps, because the prompt carries images.** `GALLERY_CAP = 10` in `service.py` bounds the
|
||||
candidates and logs the ids it drops. `worker_vision.REF_BUDGET = 12` spreads reference images across
|
||||
them, `max(1, min(3, 12 // n))` apiece, so a 9-character cast sends 9 images plus the query instead of
|
||||
the 27 plus query that 3-apiece would have sent.
|
||||
|
||||
Cost: more gemma calls, because the crops that used to be dropped now each get one.
|
||||
|
||||
**Not fixed by this.** The gallery being right does not make the crop legible. A back-turned or tiny crop
|
||||
still has no face for gemma to judge, which is `NEXT.md` item 4.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Naming a character
|
||||
|
||||
How a discovered name reaches `characters.name`. The mechanism is `db.add_name_claim`, fed by the dialogue
|
||||
stage through `service._absorb`.
|
||||
|
||||
## A name is a word set, not a string {#alias-grouping}
|
||||
|
||||
**Closed, 2026-08-13, not yet run on a GPU.**
|
||||
|
||||
Claims grouped on the casefolded name, so `Lim Seonho` from a p010 caption and `Seonho` from a p047 address
|
||||
counted as two names for one character. `len(grouped) > 1` fired, a `conflicting-name-claims` flag was
|
||||
filed, and promotion was blocked permanently on evidence that in fact corroborated.
|
||||
|
||||
`alias_groups` groups two names when one's word set contains the other's, and keeps the longer as
|
||||
canonical. `Seonho` and `Lim Seonho` are one name and the registry stores `Lim Seonho`. `Seonho` and
|
||||
`Haeseon` are still two, so a real conflict still flags.
|
||||
|
||||
## A confident caption names a character on its own {#caption-is-strong}
|
||||
|
||||
**Closed, 2026-08-13, not yet run on a GPU.**
|
||||
|
||||
Promotion needed two distinct panels, or one `self_intro` or `name_tag` claim above 0.9. On the 2026-08-12
|
||||
chapter the only claim naming a main character was `Choi Haeseon`, a caption at p040 at confidence 1.0, and
|
||||
it was discarded. She held 13 correct crops and stayed anonymous through the whole pipeline.
|
||||
|
||||
A caption is the narration naming the person it is drawn beside, which is how a webtoon introduces its
|
||||
cast, so it joins `STRONG_EVIDENCE`. This is the loosest of the three changes and is only safe because of
|
||||
the next one.
|
||||
|
||||
## A name belongs to one character {#one-name-one-character}
|
||||
|
||||
**Closed, 2026-08-13, not yet run on a GPU.**
|
||||
|
||||
Being addressed by name identifies the addressee. Choosing which drawn body that is fails often. Two panels
|
||||
addressed `Seonho`, the dialogue model pointed `target_local_id` at the woman standing beside him, and two
|
||||
independent claims promoted her. The registry then held a female `Seonho` over 9 crops beside the lead's
|
||||
`LIM SEONHO`.
|
||||
|
||||
A promotion now checks every other live character of the same manga first, by alias group. A collision
|
||||
refuses the promotion and files a `name-already-taken` flag carrying both ids. The collision is itself
|
||||
evidence that either the addressee or the identity cluster is wrong, so it is worth surfacing rather than
|
||||
resolving silently.
|
||||
|
||||
Deliberately not built: no attempt to decide WHICH character deserves the name. That needs the addressee
|
||||
fixed, which is `ARCHITECTURE.md` section 3.
|
||||
|
||||
## Checks
|
||||
|
||||
`test_name_binding.py` replays the six real claims from the 2026-08-12 chapter. Each new assert was
|
||||
confirmed to fail with its fix disabled: the caption test with `STRONG_EVIDENCE` reverted, the alias test
|
||||
with casefold grouping restored. The taken-name test asserts a flag kind that only the new branch emits.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Speaker attribution and cast names
|
||||
|
||||
Settled 2026-08-11 from the quality cross-check of job `778297bc`
|
||||
(`JOURNAL.md`, `caveats/speaker-attribution.md`). No GPU work ran and no pipeline run was executed
|
||||
after the change. Both claims rest on source and on the CPU-only self-checks named below.
|
||||
|
||||
Files: `worker_vision.py` on workpc, `correctness.py` and `test_script_verify.py` in the homesrv
|
||||
orchestrator (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`).
|
||||
|
||||
## A model guess is never labelled `tail` {#no-fake-tail}
|
||||
|
||||
**Closed.** `speaker_method` names how a speaker was established, and nothing may claim geometry it did
|
||||
not read. `_annotate_speaker_methods` stamped `tail`, the highest-trust label, on any line whose
|
||||
speaker matched a `local_id` present in the panel, keeping gemma's confidence of 1.0. No balloon was
|
||||
ever consulted.
|
||||
|
||||
Evidence: three of three sampled two-character panels had both speakers swapped
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`). 31 of 81 speech lines carried `tail` with two
|
||||
or more characters present.
|
||||
|
||||
The label is gone. With two or more characters present the guess is dropped: `speaker` becomes
|
||||
`unknown`, confidence 0.0, method `unknown`. With one character present the claim equals the solo
|
||||
backstop, so it is kept as `model_solo` at confidence 0.7. Grounded `som_face` and `solo_prior` rows are
|
||||
untouched, because the function still skips any row that already carries a method.
|
||||
|
||||
Forbids: minting a provenance label for evidence that was not read, and shipping a multi-character
|
||||
attribution as truth before balloon geometry exists.
|
||||
Check: `python worker_vision.py`, the `crowd`/`lone` cases.
|
||||
|
||||
Cost: the named-speaker share will fall. The 30% headline was measured on attributions the sample says
|
||||
are wrong, so the lower number is the first honest one.
|
||||
|
||||
## The model's speaker answer is resolved against what the prompt showed {#prompt-label-answers}
|
||||
|
||||
**Closed.** `build_dialogue_prompt` renders a present character as
|
||||
`- person_1: brown ponytail, green dress (f)`. gemma answers with any part of that line, so every part
|
||||
of it must map back to the `local_id`. It did not, and `normalize_speaker` classified each unmatched
|
||||
answer as a free-form name that no registry entry could match.
|
||||
|
||||
Evidence, measured on 36 panels of the cancelled first rerun. 15 lines carried a description and 9 a bare
|
||||
`local_id` with no identity assignment. 2 carried a stale `P1` mark label, 2 a name with the gender
|
||||
marker attached. That is 28 of 51 speech lines. Only 3 resolved to a `character_id`.
|
||||
|
||||
`_apply_speaker_labels` now takes `present` and resolves the id, the name, the description, and each of
|
||||
those plus the gender marker. A key shared by two present characters is dropped, because it identifies
|
||||
neither. An id-shaped answer naming nobody present becomes `unknown`. A trailing gender marker is
|
||||
stripped, so an off-panel `Seonho (m)` still matches the registry name `Seonho`.
|
||||
|
||||
`normalize_speaker` refuses an id-shaped value independently, because the worker is a separate process
|
||||
and the contract is load-bearing (invariant 7).
|
||||
|
||||
Forbids: showing the model a label the worker cannot resolve back, and treating an unmatched speaker
|
||||
string as a name.
|
||||
Check: `python worker_vision.py`, the `shown`/`twins` cases. `pytest test_correctness.py`,
|
||||
`test_an_id_shaped_speaker_is_never_a_name`.
|
||||
|
||||
## An unpaired mark grounds nothing {#unpaired-mark}
|
||||
|
||||
**Closed.** `_set_of_mark` labels a detected face `unknown` when gated pairing
|
||||
(`decisions/audit-phase1.md#gated-face-pairing`) matched it to no present character. An answer pointing
|
||||
at such a mark identifies nobody, so it must not carry `som_face`, the highest-trust provenance.
|
||||
|
||||
Evidence: all 7 `som_face` lines in the first 36 panels of the 2026-08-11 rerun had `speaker_ref` kind
|
||||
`unknown`. The label sat on lines with no speaker. Same defect class as the fake `tail`.
|
||||
|
||||
Forbids: deriving a provenance label from the label map without checking what the label resolved to.
|
||||
Check: `python worker_vision.py`, the `unpaired` case.
|
||||
|
||||
## An interjection is not a name and not a misquote {#interjection-false-positive}
|
||||
|
||||
**Closed.** `verify_script` must stay quiet on valid narration, because `run_stage_script` retries once
|
||||
and then raises (`decisions/audit-phase1.md#verifier-false-positives`).
|
||||
|
||||
Evidence: the narrator wrote `"...Hm?"` for the source line `"Uh... hum...?"`. Both rules fired at once.
|
||||
`Hm` was absent from the source words, and a 6-character quote needs 5 matching characters to ground,
|
||||
so it got 4. That halted the `script` stage at 112/116 on job `778297bc`.
|
||||
|
||||
Interjections join `_STOPWORDS`. A quote of three letters or fewer is no longer grounded-checked. That
|
||||
holds the same line as the dialogue prompt's 1-3 character noise rule. The stage then passed 116/116.
|
||||
|
||||
Forbids: scoring a quote too short for the ratio to carry meaning.
|
||||
Check: `pytest test_script_verify.py`, `test_an_interjection_is_not_a_name_or_a_misquote`.
|
||||
|
||||
## Cast names enter the verifier tokenized {#multiword-cast-names}
|
||||
|
||||
**Closed.** `verify_script` compares single capitalized tokens, so every allowed name must be present as
|
||||
tokens. `allowed` was built from `cast_names` verbatim, which put `"choi haeseon"` in the set as one
|
||||
string while the checker looked up `Choi` and `Haeseon` separately.
|
||||
|
||||
Evidence: the `script` stage failed at 87/116 on job `778297bc`. 28 of the 29 lost beats cite
|
||||
`unsupported-proper-noun: ['Choi', 'Haeseon']`. A one-word name such as `Seonho` always passed, which
|
||||
is why this survived the Phase 1 verifier work
|
||||
(`decisions/audit-phase1.md#verifier-false-positives`).
|
||||
|
||||
Forbids: adding any future allow-list to `verify_script` as whole strings.
|
||||
Check: `pytest test_script_verify.py`, `test_multiword_cast_name_is_supported`.
|
||||
@@ -0,0 +1,74 @@
|
||||
# storage-layout
|
||||
|
||||
Settled questions about which S3 bucket holds what, and about the MinIO replacement.
|
||||
|
||||
## One bucket per artifact class {#bucket-per-artifact}
|
||||
|
||||
**State: closed. 2026-08-11.**
|
||||
|
||||
Six buckets were created on 2026-07-04 (`manga raw panels audio layers video`). Only `manga` ever
|
||||
received an object, because `minio_layout.py` hardcoded `BUCKET = "manga"` and four workers built
|
||||
their own keys as literal `s3://manga/...`. The other five sat empty for five weeks.
|
||||
|
||||
Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved:
|
||||
|
||||
| artifact | bucket |
|
||||
| --- | --- |
|
||||
| fetched pages | `raw` |
|
||||
| panel crops | `panels` |
|
||||
| tts wavs | `audio` |
|
||||
| layer pngs | `layers` |
|
||||
| clips and `chapter.mp4` | `video` |
|
||||
| vision, identity, scene, script json, character registry | `manga` |
|
||||
|
||||
Orchestrator: `minio_layout.py` gained `BUCKET_RAW`/`BUCKET_PANELS`/`BUCKET_AUDIO`/`BUCKET_LAYERS`/
|
||||
`BUCKET_VIDEO` and a `BUCKETS` tuple. `parse_key` accepts any of them and rejects anything else.
|
||||
`service.py:_s3_delete_prefix` takes a `<bucket>/<prefix>` pair instead of assuming one bucket, and
|
||||
`_stage_s3_prefixes` stops slicing the bucket off. Workers: `worker_crop.py`, `worker_tts.py`,
|
||||
`worker_layers.py`, `worker_render.py`.
|
||||
|
||||
Every S3 URI is `s3://<bucket>/<manga_id>/<chapter_id>/...` and every consumer already derives the
|
||||
bucket from the first path segment, so no reader needed a change.
|
||||
|
||||
What this forbids: writing an artifact under a bucket that is not in `BUCKETS`. `parse_key` returns
|
||||
`{}` for one, and stage clearing would then silently delete nothing.
|
||||
|
||||
Objects written before this date stay under `manga/` at their old keys. Nothing reads them any more:
|
||||
they are the rollback for the 2026-07-17 run, not live data.
|
||||
|
||||
Evidence: `test_minio_layout.py` (31 tests), and the 2026-08-11 chapter run, which put pages in `raw`
|
||||
and 116 panel crops in `panels`.
|
||||
|
||||
## The orchestrator creates missing buckets at startup {#ensure-buckets}
|
||||
|
||||
**State: closed. 2026-08-11.**
|
||||
|
||||
Workers create a bucket on first write (`transport.py:115`), but the orchestrator uploads pages before
|
||||
any worker runs and boto3 will not auto-create. `service.py:_ensure_buckets` runs in the FastAPI
|
||||
lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning
|
||||
instead. The check is not worth a failed start.
|
||||
|
||||
## RustFS is staged, not adopted {#rustfs-staged}
|
||||
|
||||
**State: open. 2026-08-11.**
|
||||
|
||||
`rustfs` holds all six buckets on `127.0.0.1:9010/9011`, all empty. MinIO still serves every read and
|
||||
write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is
|
||||
`1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run
|
||||
meant to produce the baseline. Task [#116].
|
||||
|
||||
## Clearing a stage strips the vision blob it wrote {#clear-vision-blob}
|
||||
|
||||
**Closed. 2026-08-11.** `dialogue` and `direct` have no output table. They write onto the per-panel
|
||||
vision blob, and `_STAGE_TABLES` had no entry for either, so `/stage/clear dialogue` deleted nothing and
|
||||
still returned `{"ok": true}`. `run_stage_dialogue` then saw `"dialogue" in vision` and skipped all 116
|
||||
panels.
|
||||
|
||||
Evidence: the first clear reported `scripts: 46, scene_graphs: 116` and no vision counts. After the fix
|
||||
the same call reported `vision_results.dialogue: 116` and `vision_results.direct: 75`, all of which the
|
||||
first clear had left in place. A whole rerun was wasted on stale data before this was found.
|
||||
|
||||
`_STAGE_VISION_KEYS` in `db.py` names the keys each stage owns, and `clear_stage_data` strips them.
|
||||
|
||||
Forbids: adding a stage that writes onto a shared blob without listing its keys there.
|
||||
Check: `pytest test_db.py`, `TestClearStageData`.
|
||||
+1
-1
@@ -32,7 +32,7 @@ def _letterbox(img, sz=640):
|
||||
return canvas, r, px, py
|
||||
|
||||
|
||||
def detect_faces(img, conf: float = None) -> list:
|
||||
def detect_faces(img, conf: float | None = None) -> list:
|
||||
"""img: BGR ndarray. -> [{"bbox":[x1,y1,x2,y2], "conf":float}] in ORIGINAL pixels, sorted
|
||||
reading order. Empty on no faces. Lazy load so import never touches the model."""
|
||||
conf = CONF if conf is None else conf
|
||||
|
||||
@@ -13,3 +13,16 @@ transformers
|
||||
accelerate
|
||||
soundfile
|
||||
# system deps (not pip): ffmpeg, comfyui (external server)
|
||||
#
|
||||
# dots.tts (gitignored checkout, needed by worker_tts + scripts/pick_tts_voice). venv is built
|
||||
# --system-site-packages so it inherits Arch's rocm torch:
|
||||
# git clone https://github.com/rednote-hilab/dots.tts.git dots.tts
|
||||
# pip install -e ./dots.tts --no-deps --ignore-requires-python # py3.14 > its <3.13 pin, skips gradio
|
||||
# pip install librosa loguru einops torchdiffeq 'langcodes[data]' lingua-language-detector WeTextProcessing
|
||||
# pip install --no-deps --index-url https://download.pytorch.org/whl/rocm7.2 torchvision==0.28.0+rocm7.2
|
||||
# that last one is not optional: Arch's torchvision 0.25 is too old for torch 2.13, and every
|
||||
# transformers model import dies with "operator torchvision::nms does not exist" until it is shadowed.
|
||||
#
|
||||
# models/ (gitignored, CPU onnx for set-of-mark):
|
||||
# comictextdetector.pt.onnx <- github.com/zyddnys/manga-image-translator releases/beta-0.3
|
||||
# anime_face_v1.4_s.onnx <- huggingface.co/deepghs/anime_face_detection face_detect_v1.4_s/model.onnx
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
# Lint gate. `ruff check .` must exit 0, so CI can gate on it and a new finding means a new defect.
|
||||
#
|
||||
# Ruff's defaults flag about 100 things in this repo. Most are deliberate style in workers that must
|
||||
# survive one bad panel rather than fail clean. Every rule turned off below carries its reason, so an
|
||||
# ignore stays a decision rather than a shrug. Delete an entry the moment its reason stops holding.
|
||||
|
||||
[lint]
|
||||
ignore = [
|
||||
"I001", # import order: 25 files of churn, no behaviour change
|
||||
"BLE001", # a worker catches Exception on purpose, so one bad panel cannot kill the stage
|
||||
"SIM115", # short-lived open().read(); the handle drops with the refcount
|
||||
"S110", # try/except/pass in best-effort cleanup, where the no-op IS the handling
|
||||
"ASYNC210", # ffmpeg, ffprobe and MinIO run synchronously inside async endpoints. Real, and
|
||||
"ASYNC221", # already recorded at caveats/audit-open.md#blocking-event-loop with [#199]. The fix
|
||||
"ASYNC230", # is `def` over `async def` per handler, which is a refactor and not a lint fix.
|
||||
"RUF046", # int(round(v)) says "pixels" out loud in the render geometry
|
||||
"UP031", # the ASS subtitle template is %-formatted; f-string braces collide with its {\an} tags
|
||||
"RUF059", # unpacking a whole bbox and using half of it beats indexing into it
|
||||
"RUF007", # zip(x, x[1:]) reads better here than itertools.pairwise
|
||||
"PLC3002", # one immediately-called lambda, in an audit script
|
||||
]
|
||||
Regular → Executable
+11
@@ -25,7 +25,18 @@ def load_model(model_name: str):
|
||||
cls, *args, **{"fix_mistral_regex": True, **kwargs}
|
||||
)
|
||||
)
|
||||
# Match worker_tts.py's torch/torchaudio minor-mismatch bypass too.
|
||||
import torch
|
||||
from importlib import metadata
|
||||
|
||||
real_version = metadata.version
|
||||
metadata.version = (
|
||||
lambda name: torch.__version__ if name == "torchaudio" else real_version(name)
|
||||
)
|
||||
try:
|
||||
from dots_tts.runtime import DotsTtsRuntime
|
||||
finally:
|
||||
metadata.version = real_version
|
||||
|
||||
return DotsTtsRuntime.from_pretrained(model_name, precision="bfloat16")
|
||||
|
||||
|
||||
+42
-6
@@ -6,7 +6,7 @@
|
||||
# in-process models (siglip2, dots): no server — the lease just reserves the GPU; the
|
||||
# worker loads the transformers model itself after /session/open returns (port=None).
|
||||
# the guarantee that matters is the mutex: a second open() gets 409 until the first closes.
|
||||
import os, time, uuid, threading, subprocess
|
||||
import time, uuid, threading, subprocess
|
||||
import requests
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from pydantic import BaseModel
|
||||
@@ -117,6 +117,11 @@ def open_session(req: OpenReq):
|
||||
if _active is not None and _active["session_id"] == session_id:
|
||||
_active["proc"] = proc
|
||||
return {"session_id": session_id, "port": cfg.get("port")}
|
||||
# The lease was closed or reaped while we were spawning (close saw proc=None and tore down
|
||||
# nothing). The server we just started is unreferenced and would hold its VRAM until someone
|
||||
# killed it by hand, and the next open would spawn a SECOND one on the same port.
|
||||
_teardown({"proc": proc})
|
||||
raise HTTPException(409, "session closed while the model was loading")
|
||||
|
||||
|
||||
@app.post("/session/close")
|
||||
@@ -162,8 +167,8 @@ def _supervise_once():
|
||||
talks to the port directly, so if we don't bring it back every remaining panel of the stage fails
|
||||
with 'connection refused'. Restarting under the SAME session keeps the lease + port valid so the
|
||||
orchestrator never has to re-open. Returns True if a respawn happened."""
|
||||
with _lock:
|
||||
global _active
|
||||
with _lock:
|
||||
sess = _active
|
||||
if not sess:
|
||||
return False
|
||||
@@ -174,16 +179,29 @@ def _supervise_once():
|
||||
proc.wait(timeout=1) # reap the zombie
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
print(f"[supervise] gemma subprocess died (rc={proc.returncode}) session {sess['session_id']}; "
|
||||
# claim the respawn before releasing the lock: proc=None makes the next _supervise_once pass
|
||||
# return early, so the health wait can't be entered twice for one death.
|
||||
sess["proc"] = None
|
||||
session_id, model = sess["session_id"], sess["model"]
|
||||
# ponytail: spawn + health-wait UNLOCKED, like open_session. Holding _lock here blocked
|
||||
# /session/active, /close and /open for the full health wait (up to 300s).
|
||||
print(f"[supervise] gemma subprocess died (rc={proc.returncode}) session {session_id}; "
|
||||
f"respawning", flush=True)
|
||||
try:
|
||||
sess["proc"] = _start_subprocess(MODELS[sess["model"]])
|
||||
sess["last_beat"] = time.time() # don't count the downtime against the TTL reaper
|
||||
return True
|
||||
new_proc = _start_subprocess(MODELS[model])
|
||||
except Exception as e:
|
||||
print(f"[supervise] respawn failed: {e}; clearing session so the mutex frees", flush=True)
|
||||
with _lock:
|
||||
if _active is not None and _active["session_id"] == session_id:
|
||||
_active = None
|
||||
return False
|
||||
with _lock:
|
||||
if _active is not None and _active["session_id"] == session_id:
|
||||
_active["proc"] = new_proc
|
||||
_active["last_beat"] = time.time() # don't count the downtime against the TTL reaper
|
||||
return True
|
||||
_teardown({"proc": new_proc}) # lease went away mid-respawn; don't orphan the server
|
||||
return False
|
||||
|
||||
|
||||
def _reaper():
|
||||
@@ -229,6 +247,24 @@ if __name__ == "__main__":
|
||||
close_session(SessionReq(session_id=s4["session_id"]))
|
||||
assert active() is None
|
||||
|
||||
# a close landing DURING the spawn must not orphan the server: open reports 409 and tears it down.
|
||||
MODELS["gemma4"]["binary"] = "/bin/sleep"
|
||||
MODELS["gemma4"]["port"] = None
|
||||
real_start = _start_subprocess
|
||||
|
||||
def racing_start(cfg):
|
||||
p = real_start(cfg)
|
||||
close_session(SessionReq(session_id=_active["session_id"])) # close mid-load
|
||||
return p
|
||||
_start_subprocess = racing_start
|
||||
try:
|
||||
open_session(OpenReq(model="gemma4", ttl=3600))
|
||||
assert False, "open must 409 when its lease vanished mid-load"
|
||||
except HTTPException as e:
|
||||
assert e.status_code == 409
|
||||
_start_subprocess = real_start
|
||||
assert active() is None
|
||||
|
||||
# in-process model: close must POST /unload to the worker so its resident VRAM is freed.
|
||||
calls = []
|
||||
requests.post = lambda url, **kw: calls.append(url) or type("R", (), {"status_code": 200})()
|
||||
|
||||
@@ -14,7 +14,7 @@ def test_extract_rejects_malformed_and_truncated():
|
||||
try:
|
||||
wv._extract_json(bad)
|
||||
assert False, f"should have raised on: {bad!r}"
|
||||
except (ValueError, ValueError):
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
|
||||
|
||||
+44
-3
@@ -12,6 +12,29 @@ from starlette.responses import Response
|
||||
|
||||
_client = None
|
||||
|
||||
# --- artifact layout ---------------------------------------------------------------------------
|
||||
# one bucket per artifact class (`decisions/storage-layout.md#bucket-per-artifact`). Every worker
|
||||
# formats its output uri from these, so moving a class between buckets is one edit here rather than
|
||||
# a grep across five workers. `name` is the panel id, or 'p' when a worker has none.
|
||||
PANEL_URI = "s3://panels/{manga_id}/{chapter_id}/panels/p{idx:03d}.png"
|
||||
PAGE_PANEL_URI = "s3://panels/{manga_id}/{chapter_id}/panels/pg{page_index:03d}_p{idx:02d}.png"
|
||||
AUDIO_URI = "s3://audio/{manga_id}/{chapter_id}/audio/{name}.wav"
|
||||
AUDIO_FLAT_URI = "s3://audio/_audio/{name}.wav"
|
||||
LAYER_URI = "s3://layers/{manga_id}/{chapter_id}/layers/{name}/{idx}.png"
|
||||
CLIP_URI = "s3://video/{manga_id}/{chapter_id}/clips/{name}.mp4"
|
||||
CHAPTER_URI = "s3://video/{manga_id}/{chapter_id}/chapter.mp4"
|
||||
CHAR_PNG_URI = "s3://manga/{key}.png"
|
||||
CHAR_NPY_URI = "s3://manga/{key}.npy"
|
||||
|
||||
|
||||
def ids_from_uri(uri: str):
|
||||
"""(manga_id, chapter_id) from any artifact uri: <bucket>/<manga_id>/<chapter_id>/...
|
||||
the orchestrator passes no ids to tts, layers or render, but every input uri encodes them."""
|
||||
parts = (uri.removeprefix("s3://")).split("/")
|
||||
if len(parts) < 3:
|
||||
raise ValueError(f"uri carries no manga/chapter: {uri!r}")
|
||||
return parts[1], parts[2]
|
||||
|
||||
|
||||
def _summarize(body: bytes, limit=6) -> str:
|
||||
"""compact one-line view of a json body for observability: uri inputs/outputs (basename, or
|
||||
@@ -27,9 +50,9 @@ def _summarize(body: bytes, limit=6) -> str:
|
||||
for k, v in obj.items():
|
||||
if k == "panel_id":
|
||||
continue
|
||||
if isinstance(v, str) and (k.endswith("uri") or k.endswith("url")):
|
||||
if isinstance(v, str) and (k.endswith(("uri", "url"))):
|
||||
parts.append(f"{k}={v.rsplit('/', 1)[-1]}")
|
||||
elif isinstance(v, list) and v and isinstance(v[0], str) and (k.endswith("uris") or k.endswith("urls")):
|
||||
elif isinstance(v, list) and v and isinstance(v[0], str) and (k.endswith(("uris", "urls"))):
|
||||
parts.append(f"{k}×{len(v)}")
|
||||
elif isinstance(v, (int, float, bool)):
|
||||
parts.append(f"{k}={v}")
|
||||
@@ -102,7 +125,7 @@ def _mc():
|
||||
|
||||
def _split(uri: str):
|
||||
"""(bucket, key) from an s3-style or bare uri."""
|
||||
u = uri[5:] if uri.startswith("s3://") else uri
|
||||
u = uri.removeprefix("s3://")
|
||||
bucket, _, key = u.partition("/")
|
||||
if not bucket or not key:
|
||||
raise ValueError(f"bad uri: {uri!r}")
|
||||
@@ -213,4 +236,22 @@ if __name__ == "__main__":
|
||||
assert _summarize(b"clip_uri", ) == "-" # non-json
|
||||
assert _summarize(json.dumps({"clip_uri": "s3://m/c/clips/p1.mp4", "duration": 4.1}).encode()) \
|
||||
== "clip_uri=p1.mp4 duration=4.1"
|
||||
|
||||
# artifact layout: templates format to the keys the workers wrote by hand before, and
|
||||
# ids_from_uri recovers the ids the orchestrator never sends.
|
||||
panel = PANEL_URI.format(manga_id="m1", chapter_id="c1", idx=7)
|
||||
assert panel == "s3://panels/m1/c1/panels/p007.png", panel
|
||||
assert PAGE_PANEL_URI.format(manga_id="m1", chapter_id="c1", page_index=2, idx=3) \
|
||||
== "s3://panels/m1/c1/panels/pg002_p03.png"
|
||||
assert CLIP_URI.format(manga_id="m1", chapter_id="c1", name="p003") \
|
||||
== "s3://video/m1/c1/clips/p003.mp4"
|
||||
assert LAYER_URI.format(manga_id="m1", chapter_id="c1", name="p003", idx=0) \
|
||||
== "s3://layers/m1/c1/layers/p003/0.png"
|
||||
assert ids_from_uri(panel) == ("m1", "c1")
|
||||
assert ids_from_uri("panels/m1/c1/panels/p007.png") == ("m1", "c1")
|
||||
try:
|
||||
ids_from_uri("s3://panels/p007.png")
|
||||
raise AssertionError("a uri with no chapter segment must raise")
|
||||
except ValueError:
|
||||
pass
|
||||
print("transport self-check ok")
|
||||
|
||||
+8
-4
@@ -240,12 +240,15 @@ async def crop_webtoon(data: WebtoonInput):
|
||||
tag = uuid.uuid4().hex[:8]
|
||||
locals_ = [transport.get(u, f"{SHM}/wt_{tag}_{i:04d}.png") for i, u in enumerate(data.page_uris)]
|
||||
strip = restitch(locals_)
|
||||
crops = slice_webtoon(strip)
|
||||
crops = merge_faceless_captions(slice_webtoon(strip))
|
||||
context_links = context_fragment_links(crops)
|
||||
panels = []
|
||||
for idx, (crop_img, bbox) in enumerate(crops):
|
||||
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png"
|
||||
uri = transport.PANEL_URI.format(manga_id=data.manga_id, chapter_id=data.chapter_id, idx=idx)
|
||||
# slicing is deterministic, so on a resume the same idx -> same key; skip re-upload.
|
||||
# TRAP: that holds only while the PLAN is unchanged. Edit slice_webtoon or the merge pass and a
|
||||
# re-crop silently keeps the previous run's images under the same keys, because every one of them
|
||||
# already exists. Clear the s3://panels/<manga>/<chapter>/panels/ prefix before re-cropping.
|
||||
if not transport.exists(uri):
|
||||
out = f"{SHM}/wt_{tag}_p{idx:03d}.png"
|
||||
cv2.imwrite(out, crop_img)
|
||||
@@ -266,7 +269,7 @@ async def crop(data: CropInput):
|
||||
raise HTTPException(400, f"page not readable: {data.page_uri}")
|
||||
h, w = img.shape[:2]
|
||||
webtoon = h / w >= WEBTOON_RATIO
|
||||
crops = slice_webtoon(img) if webtoon else kumiko_panels(local, data.rtl)
|
||||
crops = merge_faceless_captions(slice_webtoon(img)) if webtoon else kumiko_panels(local, data.rtl)
|
||||
context_links = context_fragment_links(crops) if webtoon else {i: [] for i in range(len(crops))}
|
||||
ambiguous = flag_overlaps(crops, data.page_index)
|
||||
|
||||
@@ -274,7 +277,8 @@ async def crop(data: CropInput):
|
||||
for idx, (crop_img, bbox) in enumerate(crops):
|
||||
out = f"{SHM}/pg{data.page_index:03d}_p{idx:02d}.png"
|
||||
cv2.imwrite(out, crop_img)
|
||||
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/pg{data.page_index:03d}_p{idx:02d}.png"
|
||||
uri = transport.PAGE_PANEL_URI.format(manga_id=data.manga_id, chapter_id=data.chapter_id,
|
||||
page_index=data.page_index, idx=idx)
|
||||
transport.put(out, uri)
|
||||
os.remove(out)
|
||||
panels.append({"panel_index": idx, "uri": uri, "bbox": bbox,
|
||||
|
||||
+23
-6
@@ -78,7 +78,7 @@ def match(emb: np.ndarray, known: list, threshold: float):
|
||||
return None, best_conf, ambiguous
|
||||
|
||||
|
||||
def shortlist(emb: np.ndarray, known: list, k: int = 5, gender: str = None) -> list:
|
||||
def shortlist(emb: np.ndarray, known: list, k: int = 5, gender: str | None = None) -> list:
|
||||
"""TIER-2 evidence: the top-k gender-gated known characters by cosine, best first. Cosine is now a
|
||||
SHORTLISTER, not the decider — gemma /vision/resolve picks from this list. Each item carries the
|
||||
full row (name/gender/description) so the resolver can build a text character-sheet. pure."""
|
||||
@@ -88,7 +88,10 @@ def shortlist(emb: np.ndarray, known: list, k: int = 5, gender: str = None) -> l
|
||||
|
||||
|
||||
def _crop_bbox(img, bbox):
|
||||
# vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention).
|
||||
# [x1, y1, x2, y2] pixel corners. gemma answers on a 0-1000 normalized grid and `/vision` converts
|
||||
# to pixels before returning (`worker_vision._bbox_to_pixels`), so this reads real pixels. It did
|
||||
# not before 2026-08-12, which is why crops landed on balloons
|
||||
# (`decisions/identity-bbox.md#bbox-is-normalized`).
|
||||
x1, y1, x2, y2 = bbox
|
||||
return img[y1:y2, x1:x2]
|
||||
|
||||
@@ -101,7 +104,7 @@ def _save_npy(emb: np.ndarray, uri: str):
|
||||
os.remove(tmp)
|
||||
|
||||
|
||||
def _pending_match(pend: list, emb, threshold: float, gender: str = None):
|
||||
def _pending_match(pend: list, emb, threshold: float, gender: str | None = None):
|
||||
"""index of the pending entry this embedding belongs to, or None (a new provisional). candidates
|
||||
of a conflicting decided gender are excluded. character_id=i keeps the returned index original. pure."""
|
||||
cands = [{"character_id": i, "embedding": e["emb"]}
|
||||
@@ -114,7 +117,7 @@ def _persist_char(manga_id, panel_id, local_id, crop, emb, name, gender, appeara
|
||||
"""upload crop + embedding to S3 and register the row via the orchestrator; return its id."""
|
||||
import cv2
|
||||
key = f"{manga_id}/characters/_new/{panel_id}_{local_id}"
|
||||
ref_img_uri, emb_uri = f"s3://manga/{key}.png", f"s3://manga/{key}.npy"
|
||||
ref_img_uri, emb_uri = transport.CHAR_PNG_URI.format(key=key), transport.CHAR_NPY_URI.format(key=key)
|
||||
ref_png = f"{SHM}/crop_{uuid.uuid4().hex[:8]}.png"
|
||||
cv2.imwrite(ref_png, crop)
|
||||
transport.put(ref_png, ref_img_uri)
|
||||
@@ -197,6 +200,13 @@ async def resolve(data: IdentityInput):
|
||||
|
||||
assignments, backfill, new_chars, shortlists = [], [], [], []
|
||||
for ch in data.vision_characters:
|
||||
# no detected face inside the box -> a background extra, a figure on a poster, or scenery
|
||||
# gemma called a person. Embedding it pollutes the registry and, once the boxes were pixels,
|
||||
# bound an extra to the lead at confidence 1.00. Abstain instead. `/vision` stamps this and
|
||||
# fails open, so a panel it could not gate arrives with has_face=True on every character
|
||||
# (decisions/identity-bbox.md#face-gates-enrollment). Absent key = an older vision blob.
|
||||
if ch.get("has_face") is False:
|
||||
continue
|
||||
crop = _crop_bbox(img, ch["bbox"])
|
||||
if crop.size == 0: # degenerate/out-of-bounds bbox -> nothing to embed, skip
|
||||
continue
|
||||
@@ -204,10 +214,17 @@ async def resolve(data: IdentityInput):
|
||||
|
||||
# shortlist for gemma's decider, from the roster as it stood before this crop's own outcome.
|
||||
sl = shortlist(emb, known, data.k, ch.get("gender"))
|
||||
crop_uri = f"s3://manga/{data.manga_id}/characters/_crops/{data.panel_id}_{ch['local_id']}.png"
|
||||
# the embedding ships with the crop. gemma's decider can answer "none of these", and the
|
||||
# orchestrator has to mint a character from that crop — which needs an embedding it cannot
|
||||
# compute (siglip is resident here, gemma is resident there, and the mutex forbids both).
|
||||
# Uploading it now is what removes the third siglip pass
|
||||
# (`decisions/identity-bbox.md#none-mints-an-anonymous-character`).
|
||||
key = f"{data.manga_id}/characters/_crops/{data.panel_id}_{ch['local_id']}"
|
||||
crop_uri, emb_uri = transport.CHAR_PNG_URI.format(key=key), transport.CHAR_NPY_URI.format(key=key)
|
||||
cp = f"{SHM}/cc_{uuid.uuid4().hex[:8]}.png"; cv2.imwrite(cp, crop)
|
||||
transport.put(cp, crop_uri); os.remove(cp)
|
||||
shortlists.append({"local_id": ch["local_id"], "crop_uri": crop_uri,
|
||||
_save_npy(emb, emb_uri)
|
||||
shortlists.append({"local_id": ch["local_id"], "crop_uri": crop_uri, "emb_uri": emb_uri,
|
||||
"candidates": [{"character_id": c["character_id"], "name": c.get("name"),
|
||||
"gender": c.get("gender"), "species": c.get("species"),
|
||||
"appearance": c.get("description"), "cosine": c["cosine"],
|
||||
|
||||
+3
-3
@@ -65,15 +65,15 @@ async def layers(data: LayerInput):
|
||||
return {"layer_uris": [], "skipped": "comfyui down"} # ponytail: skip stage if ComfyUI not running
|
||||
local = transport.get(data.panel_uri, f"{SHM}/layer_{uuid.uuid4().hex[:8]}.png")
|
||||
# orchestrator doesn't pass manga/chapter to layers; derive from the panel uri prefix.
|
||||
parts = data.panel_uri.replace("s3://", "").split("/")
|
||||
manga_id, chapter_id = parts[1], parts[2]
|
||||
manga_id, chapter_id = transport.ids_from_uri(data.panel_uri)
|
||||
view_urls = _comfy_run(local, data.num_layers, data.prompt)
|
||||
layer_uris = []
|
||||
for idx, url in enumerate(view_urls):
|
||||
png = f"{SHM}/layer_{uuid.uuid4().hex[:6]}.png"
|
||||
with open(png, "wb") as f:
|
||||
f.write(requests.get(url, timeout=60).content)
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/layers/{data.panel_id or 'p'}/{idx}.png"
|
||||
uri = transport.LAYER_URI.format(manga_id=manga_id, chapter_id=chapter_id,
|
||||
name=data.panel_id or 'p', idx=idx)
|
||||
transport.put(png, uri)
|
||||
os.remove(png)
|
||||
layer_uris.append(uri)
|
||||
|
||||
+171
-48
@@ -19,13 +19,6 @@ MUSIC_BED = os.environ.get("MUSIC_BED", "") # #13 path/uri of a music t
|
||||
MUSIC_GAIN = os.environ.get("MUSIC_GAIN", "0.18") # bed level before ducking
|
||||
|
||||
|
||||
def _mc_from_uri(uri: str):
|
||||
"""(manga_id, chapter_id) from s3://<bucket>/<manga_id>/<chapter_id>/...
|
||||
the orchestrator doesn't pass ids to render/layers, but every input uri encodes them."""
|
||||
parts = uri.replace("s3://", "").split("/")
|
||||
return parts[1], parts[2]
|
||||
|
||||
|
||||
def _ts(s):
|
||||
h = int(s // 3600); m = int(s % 3600 // 60); sec = s % 60
|
||||
return f"{h}:{m:02d}:{sec:05.2f}"
|
||||
@@ -37,10 +30,10 @@ def _ts(s):
|
||||
SUB_MODE = os.environ.get("SUB_MODE", "minimal").lower() # off | minimal | boxed
|
||||
SUB_PRESETS = {
|
||||
# (mode, orientation): fontsize, borderstyle(1=outline,3=box), outline/pad, shadow, marginV, side
|
||||
("minimal", "portrait"): dict(fs=40, bs=1, outline=3, shadow=2, mv=0.055, side=110),
|
||||
("minimal", "landscape"): dict(fs=32, bs=1, outline=3, shadow=2, mv=0.09, side=260),
|
||||
("boxed", "portrait"): dict(fs=40, bs=3, outline=6, shadow=0, mv=0.055, side=110),
|
||||
("boxed", "landscape"): dict(fs=32, bs=3, outline=6, shadow=0, mv=0.09, side=260),
|
||||
("minimal", "portrait"): {"fs": 40, "bs": 1, "outline": 3, "shadow": 2, "mv": 0.055, "side": 110},
|
||||
("minimal", "landscape"): {"fs": 32, "bs": 1, "outline": 3, "shadow": 2, "mv": 0.09, "side": 260},
|
||||
("boxed", "portrait"): {"fs": 40, "bs": 3, "outline": 6, "shadow": 0, "mv": 0.055, "side": 110},
|
||||
("boxed", "landscape"): {"fs": 32, "bs": 3, "outline": 6, "shadow": 0, "mv": 0.09, "side": 260},
|
||||
}
|
||||
|
||||
|
||||
@@ -109,15 +102,41 @@ def _ass(text: str, dur: float, path: str):
|
||||
def _audio_dur(path: str) -> float:
|
||||
"""clip length = narration length. scene_timing arrives empty, so probe the audio."""
|
||||
r = subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=nk=1:nw=1", path], capture_output=True, text=True)
|
||||
"-of", "default=nk=1:nw=1", path], capture_output=True, text=True, check=False)
|
||||
try:
|
||||
return float(r.stdout.strip())
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _stream_dur(path: str, kind: str) -> float:
|
||||
"""duration of one stream. `format=duration` is max(video,audio) and so hides A/V drift."""
|
||||
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", f"{kind}:0",
|
||||
"-show_entries", "stream=duration", "-of", "default=nk=1:nw=1", path],
|
||||
capture_output=True, text=True, check=False)
|
||||
try:
|
||||
return float(r.stdout.strip())
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _fps_of(path: str) -> str:
|
||||
"""`r_frame_rate` as ffprobe reports it. Compared as a string on purpose: two clips agree only when
|
||||
their rate AND therefore their time_base agree, and the stream-copy concat path cares about that."""
|
||||
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0",
|
||||
"-show_entries", "stream=r_frame_rate", "-of", "default=nk=1:nw=1", path],
|
||||
capture_output=True, text=True, check=False)
|
||||
return r.stdout.strip()
|
||||
|
||||
|
||||
ZMAX, ZPAN = 1.15, 1.18 # ken-burns zoom ceiling; constant zoom that gives pans room to travel
|
||||
|
||||
# Every clip and every assembly stage MUST agree on this. xfade does not resample: it reinterprets the
|
||||
# second input's frames at the first input's rate, so a 30fps input joined onto a 25fps one plays 1.2x
|
||||
# too slow with the audio untouched -- the video ends minutes long and the narration runs ahead of the
|
||||
# picture. That is exactly what a 25fps clip pipeline plus a `fps=30` concat branch produced.
|
||||
FPS = 25
|
||||
|
||||
|
||||
def _motion(camera: dict, frames: int) -> str:
|
||||
"""#8 content-aware motion: map the vision `camera` block to a zoompan z/x/y expression.
|
||||
@@ -155,16 +174,15 @@ def _motion(camera: dict, frames: int) -> str:
|
||||
x, y = f"{xc}+(iw*0.03)*sin(6.283*on/{T})", f"{yc}+(ih*0.03)*cos(6.283*on/{T})"
|
||||
else: # zoom_in (default ken burns)
|
||||
z, x, y = f"1+{ZMAX-1:.3f}*on/{T}", xc, yc
|
||||
return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps=25"
|
||||
return f"zoompan=z='{z}':x='{x}':y='{y}':d={frames}:s={W}x{H}:fps={FPS}"
|
||||
|
||||
|
||||
def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict = None,
|
||||
def scene_cmd(img: str, audio: str, ass: str, out: str, dur: float, camera: dict | None = None,
|
||||
pad: float = PAD_S) -> list:
|
||||
"""ffmpeg: still panel over a blurred fill of itself + content-aware motion, burned subs, 9:16.
|
||||
#5 blurred bg replaces black bars: one copy scaled to COVER + blurred, the fitted panel on top.
|
||||
#13 pad seconds of trailing silence (last frame held) give the panel a beat before the next."""
|
||||
fps = 25
|
||||
frames = max(1, int((dur + pad) * fps)) # hold the last frame through the pad
|
||||
frames = max(1, int((dur + pad) * FPS)) # hold the last frame through the pad
|
||||
# overlay's W/H/w/h are ffmpeg's main/overlay dims -- kept literal (no f-string braces).
|
||||
fc = (
|
||||
f"[0:v]split=2[bg][fg];"
|
||||
@@ -206,8 +224,9 @@ async def render_scene(data: SceneInput):
|
||||
_ass(data.narration_text, dur, ass)
|
||||
out = f"{SHM}/rnd_{tag}.mp4"
|
||||
subprocess.run(scene_cmd(img, audio, ass, out, dur, data.camera), check=True, capture_output=True)
|
||||
manga_id, chapter_id = _mc_from_uri(data.panel_uri)
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
|
||||
manga_id, chapter_id = transport.ids_from_uri(data.panel_uri)
|
||||
uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
|
||||
name=data.panel_id or 'p')
|
||||
transport.put(out, uri)
|
||||
for p in (img, audio, ass, out):
|
||||
os.remove(p)
|
||||
@@ -316,7 +335,7 @@ class CompositeInput(BaseModel):
|
||||
|
||||
def _img_size(path: str) -> tuple:
|
||||
r = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
|
||||
"stream=width,height", "-of", "csv=p=0:s=x", path], capture_output=True, text=True)
|
||||
"stream=width,height", "-of", "csv=p=0:s=x", path], capture_output=True, text=True, check=False)
|
||||
w, h = r.stdout.strip().split("x")
|
||||
return int(w), int(h)
|
||||
|
||||
@@ -352,8 +371,9 @@ async def render_composite(data: CompositeInput):
|
||||
subprocess.run(composite_cmd(still, auds, ass, segs, rowh, out, t), check=True, capture_output=True)
|
||||
for p in imgs + [still]:
|
||||
os.remove(p)
|
||||
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"])
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
|
||||
manga_id, chapter_id = transport.ids_from_uri(data.panels[0]["panel_uri"])
|
||||
uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
|
||||
name=data.panel_id or 'p')
|
||||
transport.put(out, uri)
|
||||
for p in auds + [ass, out]:
|
||||
os.remove(p)
|
||||
@@ -399,8 +419,9 @@ async def render_group(data: GroupInput):
|
||||
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", final]
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
|
||||
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"])
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
|
||||
manga_id, chapter_id = transport.ids_from_uri(data.panels[0]["panel_uri"])
|
||||
uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
|
||||
name=data.panel_id or 'p')
|
||||
transport.put(final, uri)
|
||||
total = _audio_dur(final) or sum(durs)
|
||||
for f in cleanup + [final]:
|
||||
@@ -424,7 +445,7 @@ class BeatInput(BaseModel):
|
||||
BEAT_MIN_PANEL_S = 1.0 # 185: no member panel flashes by faster than this
|
||||
|
||||
|
||||
def _beat_slices(D: float, n: int, weights: list = None) -> list:
|
||||
def _beat_slices(D: float, n: int, weights: list | None = None) -> list:
|
||||
"""185: split beat duration D across n member panels by content weight, not evenly.
|
||||
Each panel gets >= BEAT_MIN_PANEL_S so a low-weight panel never flashes. Degenerate input
|
||||
(no/short weights, non-positive sum, or D too small for the floors) -> deterministic equal split.
|
||||
@@ -476,14 +497,14 @@ def cue_plan(text: str, D: float, slices: list) -> list:
|
||||
return events
|
||||
|
||||
|
||||
def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list = None,
|
||||
weights: list = None) -> list:
|
||||
def beat_cmd(imgs: list, audio: str, ass: str, out: str, D: float, cameras: list | None = None,
|
||||
weights: list | None = None) -> list:
|
||||
"""ffmpeg: N images shown as a ken-burns montage under ONE narration audio; the last image holds
|
||||
through the trailing pad. each image = blurred-fill bg + fitted panel + its camera move; the single
|
||||
burned subtitle (in `ass`) spans the whole beat. 185: per-image screen-time is content-weighted
|
||||
(see _beat_slices), equal split when weights are absent. pure -> testable without S3."""
|
||||
cameras = cameras or []
|
||||
n, fps = len(imgs), 25
|
||||
n, fps = len(imgs), FPS
|
||||
slices = _beat_slices(D, n, weights) # 185: content-weighted, equal-split fallback
|
||||
# one frame per image (no -loop): zoompan d=frames expands that single frame to exactly `frames`
|
||||
# output frames = seg seconds. looping instead would feed many frames and zoompan multiplies each.
|
||||
@@ -536,8 +557,9 @@ async def render_beat(data: BeatInput):
|
||||
subprocess.run(beat_cmd(imgs, audio, ass, out, D, data.cameras, data.weights),
|
||||
check=True, capture_output=True)
|
||||
|
||||
manga_id, chapter_id = _mc_from_uri(data.panel_uris[0])
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
|
||||
manga_id, chapter_id = transport.ids_from_uri(data.panel_uris[0])
|
||||
uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
|
||||
name=data.panel_id or 'p')
|
||||
transport.put(out, uri)
|
||||
total = _audio_dur(out) or (D + PAD_S)
|
||||
for f in imgs + [audio, ass, out]:
|
||||
@@ -578,8 +600,8 @@ def collage_cmd(imgs, plate_i, rects, entrances, z_order, audio, ass, out, D, tr
|
||||
for img in imgs:
|
||||
cmd += ["-loop", "1", "-i", img] # still held for the whole clip; -t caps output
|
||||
cmd += ["-i", audio]
|
||||
parts = [f"[{plate_i}:v]scale={W}:{H}:force_original_aspect_ratio=increase,"
|
||||
f"crop={W}:{H},boxblur=24:2,setsar=1[bg]"]
|
||||
parts = [(f"[{plate_i}:v]scale={W}:{H}:force_original_aspect_ratio=increase,"
|
||||
f"crop={W}:{H},boxblur=24:2,setsar=1[bg]")]
|
||||
base = "bg"
|
||||
for k, i in enumerate(z_order): # shadows first, at rest positions (static)
|
||||
x, y, w, h = (int(round(v)) for v in rects[i])
|
||||
@@ -598,7 +620,7 @@ def collage_cmd(imgs, plate_i, rects, entrances, z_order, audio, ass, out, D, tr
|
||||
cur = f"o{k}"
|
||||
fc = ";".join(parts) + f";[{cur}]ass={ass}[v];[{n}:a]apad=pad_dur={PAD_S:.3f}[a]"
|
||||
cmd += ["-filter_complex", fc, "-map", "[v]", "-map", "[a]", "-t", f"{T:.3f}",
|
||||
"-r", "30", "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", out]
|
||||
"-r", str(FPS), "-c:v", "libx264", "-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", out]
|
||||
return cmd
|
||||
|
||||
|
||||
@@ -636,8 +658,9 @@ async def render_collage(data: CollageInput):
|
||||
out = f"{SHM}/cl_{tag}.mp4"
|
||||
subprocess.run(collage_cmd(imgs, active, lay["rects"], lay["entrances"], lay["z_order"],
|
||||
audio, ass, out, D, lay["transition_s"]), check=True, capture_output=True)
|
||||
manga_id, chapter_id = _mc_from_uri(uris[0])
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
|
||||
manga_id, chapter_id = transport.ids_from_uri(uris[0])
|
||||
uri = transport.CLIP_URI.format(manga_id=manga_id, chapter_id=chapter_id,
|
||||
name=data.panel_id or 'p')
|
||||
transport.put(out, uri)
|
||||
total = _audio_dur(out) or (D + PAD_S)
|
||||
for f in imgs + [audio, ass, out]:
|
||||
@@ -670,28 +693,57 @@ FFMPEG_THREADS = max(1, int(os.environ.get("FFMPEG_THREADS", "2")))
|
||||
def _xfade_chain(durs: list, trans: list):
|
||||
"""build a filter_complex that xfades N clips with per-boundary transitions, keeping audio in
|
||||
sync via matching acrossfade. trans[i] is the transition OUT of clip i (boundary i->i+1).
|
||||
returns (filtergraph, video_label, audio_label). offsets accumulate as clips overlap."""
|
||||
returns (filtergraph, video_label, audio_label, expected_duration). offsets accumulate as clips
|
||||
overlap. `durs[i]` MUST be min(video, audio) of input i, not `format=duration`."""
|
||||
# A clip whose duration probed as 0/unreadable must not poison the chain: with dur=0 the offset
|
||||
# accumulator would run BACKWARDS (cum += dur - td), swallowing every later clip into a frozen
|
||||
# overlap near the middle. Floor to a small positive length so the timeline stays monotonic.
|
||||
durs = [d if (d and d > 0.1) else 0.1 for d in durs]
|
||||
parts, vlast, alast, cum = [], "[0:v]", "[0:a]", durs[0]
|
||||
# The offset accumulator is only as good as `durs`: an offset that lands even one frame past the end
|
||||
# of input i-1 makes xfade emit the transition and then SILENTLY DROP input i and the whole rest of
|
||||
# the chain -- rc 0, no warning, a chapter minutes short with the audio intact. So truncate every
|
||||
# input to a whole number of frames it certainly has (floor, and min(v,a) from the caller) and trim
|
||||
# both streams to exactly that. Then cum is the real timeline, not an estimate of it.
|
||||
durs = [max(1, int(d * FPS)) / FPS for d in durs]
|
||||
# Normalize every input to FPS/SAR before it reaches xfade, exactly as the concat branch does. Both
|
||||
# branches feed the same tree, so an un-normalized xfade input is what stretched the chapter 1.2x.
|
||||
parts = []
|
||||
for i, d in enumerate(durs):
|
||||
parts.append(f"[{i}:v]setsar=1,fps={FPS},trim=end={d:.3f},setpts=PTS-STARTPTS[n{i}]")
|
||||
parts.append(f"[{i}:a]atrim=end={d:.3f},asetpts=PTS-STARTPTS[m{i}]")
|
||||
vlast, alast, cum = "[n0]", "[m0]", durs[0]
|
||||
for i in range(1, len(durs)):
|
||||
name, td = XFADE.get(trans[i - 1] if i - 1 < len(trans) else "cut", XFADE["cut"])
|
||||
td = max(0.05, min(td, durs[i - 1] - 0.05, durs[i] - 0.05)) # overlap fits in both clips
|
||||
td = max(1, int(td * FPS)) / FPS # ...on a frame boundary
|
||||
off = max(cum - td, 0)
|
||||
parts.append(f"{vlast}[{i}:v]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]")
|
||||
parts.append(f"{alast}[{i}:a]acrossfade=d={td:.3f}[a{i}]")
|
||||
parts.append(f"{vlast}[n{i}]xfade=transition={name}:duration={td:.3f}:offset={off:.3f}[v{i}]")
|
||||
parts.append(f"{alast}[m{i}]acrossfade=d={td:.3f}[a{i}]")
|
||||
vlast, alast, cum = f"[v{i}]", f"[a{i}]", cum + durs[i] - td
|
||||
return ";".join(parts), vlast, alast
|
||||
return ";".join(parts), vlast, alast, cum
|
||||
|
||||
|
||||
ASSEMBLE_TOL_S = 0.5 # frame-boundary + aac-padding slack; a dropped input is off by whole seconds
|
||||
|
||||
|
||||
def _check_assembled(out: str, expect: float):
|
||||
"""ffmpeg drops xfade inputs and re-times concat segments without ever failing, so verify the
|
||||
result instead of trusting rc 0. Both streams, because a video-only loss is the failure mode that
|
||||
shipped a 436s picture over 364s of narration."""
|
||||
v, a = _stream_dur(out, "v"), _stream_dur(out, "a")
|
||||
if abs(v - expect) > ASSEMBLE_TOL_S or abs(v - a) > ASSEMBLE_TOL_S:
|
||||
raise RuntimeError(f"assembly lost stream time in {os.path.basename(out)}: "
|
||||
f"video={v:.2f} audio={a:.2f} expected={expect:.2f}")
|
||||
|
||||
|
||||
def _assemble_once(inputs: list[str], trans: list[str], out: str):
|
||||
"""Assemble one bounded batch. `trans[i]` is the transition out of inputs[i]."""
|
||||
# min(video, audio), never `format=duration`: that is max(video, audio), and feeding it to
|
||||
# _xfade_chain puts the offset accumulator ahead of the real video timeline.
|
||||
durs = [min(_stream_dur(p, "v"), _stream_dur(p, "a")) for p in inputs]
|
||||
fancy = len(inputs) >= 2 and any(t not in ("", "cut") for t in trans[:len(inputs) - 1])
|
||||
if fancy:
|
||||
durs = [_audio_dur(p) for p in inputs]
|
||||
fg, vmap, amap = _xfade_chain(durs, trans)
|
||||
fg, vmap, amap, expect = _xfade_chain(durs, trans)
|
||||
cmd = ["ffmpeg", "-y", "-filter_complex_threads", str(FFMPEG_THREADS)]
|
||||
# Input-side -threads limits each decoder; otherwise ffmpeg may create a decoder thread pool
|
||||
# for every input in the batch in addition to the filter and libx264 pools.
|
||||
@@ -701,6 +753,7 @@ def _assemble_once(inputs: list[str], trans: list[str], out: str):
|
||||
"-c:v", "libx264", "-threads", str(FFMPEG_THREADS), "-pix_fmt", "yuv420p",
|
||||
"-c:a", "aac", "-b:a", "192k", out]
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
_check_assembled(out, expect)
|
||||
return
|
||||
|
||||
# A cut-only batch is concatenated with the concat FILTER, not -c copy or the concat demuxer.
|
||||
@@ -713,12 +766,13 @@ def _assemble_once(inputs: list[str], trans: list[str], out: str):
|
||||
cmd = ["ffmpeg", "-y"]
|
||||
for p in inputs:
|
||||
cmd += ["-i", p]
|
||||
pre = "".join(f"[{i}:v]setsar=1,fps=30[v{i}];" for i in range(n))
|
||||
pre = "".join(f"[{i}:v]setsar=1,fps={FPS}[v{i}];" for i in range(n))
|
||||
fg = pre + "".join(f"[v{i}][{i}:a]" for i in range(n)) + f"concat=n={n}:v=1:a=1[v][a]"
|
||||
cmd += ["-filter_complex", fg, "-map", "[v]", "-map", "[a]",
|
||||
"-c:v", "libx264", "-preset", "veryfast", "-crf", "20", "-pix_fmt", "yuv420p",
|
||||
"-threads", str(FFMPEG_THREADS), "-c:a", "aac", "-b:a", "192k", out]
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
_check_assembled(out, sum(durs))
|
||||
|
||||
|
||||
def _assemble_batched(inputs: list[str], transitions: list[str], out: str, tag: str,
|
||||
@@ -816,20 +870,31 @@ async def assemble(data: AssembleInput):
|
||||
cleanup = list(locals_) + [out]
|
||||
|
||||
fancy = len(locals_) >= 2 and any(t not in ("", "cut") for t in data.transitions)
|
||||
if fancy:
|
||||
# The stream-copy path writes the output with the FIRST input's time_base and reinterprets every
|
||||
# later packet in it. A clip encoded at 30fps (time_base 1/15360) copied into a 25fps container
|
||||
# (1/12800) therefore plays 15360/12800 = 1.2x too long with its audio untouched. That is the whole
|
||||
# of the shipped chapter's 436.39s of video over 363.67s of narration: 14 of 49 clips came off the
|
||||
# collage path, which hardcoded `-r 30`. Mixed rates must re-encode, so they go through the tree,
|
||||
# whose branches both normalize with `fps={FPS}`.
|
||||
rates = {_fps_of(p) for p in locals_}
|
||||
if fancy or len(rates) > 1:
|
||||
if len(rates) > 1:
|
||||
print(f"[render] mixed clip rates {sorted(rates)}, re-encoding instead of stream copy",
|
||||
flush=True)
|
||||
_assemble_batched(locals_, data.transitions, out, tag, cleanup)
|
||||
else:
|
||||
# all hard cuts: stream-copy concat (no re-encode) -- unchanged fast path.
|
||||
# all hard cuts at one shared rate: stream-copy concat, no re-encode.
|
||||
listfile = f"{SHM}/asm_{tag}.txt"; cleanup.append(listfile)
|
||||
with open(listfile, "w") as f:
|
||||
f.write("".join(f"file '{p}'\n" for p in locals_))
|
||||
subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", listfile, "-c", "copy", out],
|
||||
check=True, capture_output=True)
|
||||
_check_assembled(out, sum(min(_stream_dur(p, "v"), _stream_dur(p, "a")) for p in locals_))
|
||||
|
||||
out = _add_music_bed(out, tag, cleanup)
|
||||
|
||||
manga_id, chapter_id = _mc_from_uri(data.clip_uris[0])
|
||||
uri = f"s3://manga/{manga_id}/{chapter_id}/chapter.mp4"
|
||||
manga_id, chapter_id = transport.ids_from_uri(data.clip_uris[0])
|
||||
uri = transport.CHAPTER_URI.format(manga_id=manga_id, chapter_id=chapter_id)
|
||||
transport.put(out, uri)
|
||||
for p in cleanup:
|
||||
os.remove(p)
|
||||
@@ -933,8 +998,14 @@ if __name__ == "__main__":
|
||||
for p in (img, bed, *cl):
|
||||
if os.path.exists(p): os.remove(p)
|
||||
# #6 transitions: two real clips xfade into one chapter; graph offsets/labels well-formed.
|
||||
fg, vmap, amap = _xfade_chain([1.0, 1.0], ["fade_white"])
|
||||
fg, vmap, amap, exp = _xfade_chain([1.0, 1.0], ["fade_white"])
|
||||
assert "xfade=transition=fadewhite" in fg and vmap == "[v1]" and amap == "[a1]"
|
||||
# Both streams of every input trimmed to a whole frame count. That is what keeps the offset
|
||||
# accumulator ON the real timeline: an offset one frame past the end of input i-1 makes xfade
|
||||
# emit the transition and then silently drop input i and everything after it, rc 0, no warning.
|
||||
assert fg.count(",trim=end=") == 2 and fg.count("]atrim=end=") == 2, fg
|
||||
_td = max(1, int(XFADE["fade_white"][1] * FPS)) / FPS
|
||||
assert abs(exp - (2.0 - _td)) < 0.001, (exp, _td)
|
||||
img = f"{SHM}/t.png"
|
||||
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "color=c=black:s=400x600",
|
||||
"-frames:v", "1", img], check=True, capture_output=True)
|
||||
@@ -945,7 +1016,54 @@ if __name__ == "__main__":
|
||||
"-map", vmap, "-map", amap, "-c:v", "libx264", "-pix_fmt", "yuv420p",
|
||||
"-c:a", "aac", out], check=True, capture_output=True)
|
||||
assert os.path.getsize(out) > 0
|
||||
for p in (img, c0, c1):
|
||||
# A/V drift: the tree must not stretch video. The concat branch normalizes fps and the xfade
|
||||
# branch used not to, so a chapter mixing both played 1.2x slow with the audio untouched and the
|
||||
# narration ran ahead of the picture. Batch=2 over 4 clips forces BOTH branches plus a second
|
||||
# round -- the shipped bug's exact shape. Compare the streams, not the file size.
|
||||
c2, c3 = f"{SHM}/c2.mp4", f"{SHM}/c3.mp4"
|
||||
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c2, 1.0), check=True, capture_output=True)
|
||||
subprocess.run(scene_cmd(img, aud, f"{SHM}/t.ass", c3, 1.0), check=True, capture_output=True)
|
||||
_saved_batch, cl = ASSEMBLE_BATCH, []
|
||||
globals()["ASSEMBLE_BATCH"] = 2
|
||||
try:
|
||||
# ["crossfade","crossfade","cut","cut"] with batch=2 is the one shape that mixes branches:
|
||||
# round 1 sends [c0,c1] through xfade and [c2,c3] through concat, then round 2 xfades those
|
||||
# two intermediates together. A cut-only first group would keep both on the concat branch and
|
||||
# the rates would agree by accident, which is why this case has to be spelled out.
|
||||
_assemble_batched([c0, c1, c2, c3], ["crossfade", "crossfade", "cut", "cut"],
|
||||
out, "selfck", cl)
|
||||
finally:
|
||||
globals()["ASSEMBLE_BATCH"] = _saved_batch
|
||||
for p in cl:
|
||||
if os.path.exists(p): os.remove(p)
|
||||
vd, ad = _stream_dur(out, "v"), _stream_dur(out, "a")
|
||||
assert abs(vd - ad) < 0.25, f"A/V drift: video {vd:.2f}s vs audio {ad:.2f}s"
|
||||
# ...and the guard that catches it in production must actually fire. `assemble` reports success
|
||||
# off ffmpeg's rc, which is 0 even when a whole batch of inputs is thrown away.
|
||||
try:
|
||||
_check_assembled(out, vd + 5.0)
|
||||
raise AssertionError("_check_assembled did not fire on a 5s loss")
|
||||
except RuntimeError:
|
||||
pass
|
||||
for p in (img, c0, c1, c2, c3):
|
||||
os.remove(p)
|
||||
# The offset creep itself, reproduced small. A real clip's audio outlasts its video slightly, and
|
||||
# `format=duration` reports the audio, so the accumulator walked ahead of the picture until one
|
||||
# xfade window ran past the end of its first input -- at which point ffmpeg emitted the
|
||||
# transition, threw away the second input and everything downstream, and exited 0. Clips with
|
||||
# audio 0.4s longer than the video exaggerate one clip's worth of that creep.
|
||||
cs = []
|
||||
for i in range(3):
|
||||
c = f"{SHM}/creep{i}.mp4"; cs.append(c)
|
||||
subprocess.run(["ffmpeg", "-y", "-f", "lavfi", "-i", "testsrc2=s=160x120:r=25:d=1.0",
|
||||
"-f", "lavfi", "-i", f"sine=f={300 + i * 40}:d=1.4",
|
||||
"-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p",
|
||||
"-c:a", "aac", c], check=True, capture_output=True)
|
||||
_assemble_once(cs, ["fade_black", "cut", "cut"], out) # raises if an input was dropped
|
||||
vd, ad = _stream_dur(out, "v"), _stream_dur(out, "a")
|
||||
assert vd > 2.2, f"xfade dropped inputs: {vd:.2f}s from 3 clips of 1.0s"
|
||||
assert abs(vd - ad) < 0.1, f"A/V drift: video {vd:.2f}s vs audio {ad:.2f}s"
|
||||
for p in cs:
|
||||
os.remove(p)
|
||||
# #6 composite: 2 panels + 2 audios -> one stacked clip; duration = sum, subs timed per row.
|
||||
a2 = f"{SHM}/a2.wav"
|
||||
@@ -987,6 +1105,11 @@ if __name__ == "__main__":
|
||||
check=True, capture_output=True)
|
||||
assert abs(_audio_dur(out) - (1.5 + PAD_S)) < 0.2, _audio_dur(out) # collage clip = beat length
|
||||
assert os.path.getsize(out) > 0
|
||||
# ...and it must come out at FPS like every other clip path. This one hardcoded `-r 30`, so 14 of
|
||||
# 49 clips in the shipped chapter were 30fps. `concat -c copy` writes the output with the FIRST
|
||||
# clip's time_base and reinterprets later packets in it, so those 14 played 1.2x too long with
|
||||
# their narration untouched. That, not the xfade tree, is where the 72.7s gap came from.
|
||||
assert _fps_of(out) == f"{FPS}/1", f"collage clip is {_fps_of(out)}, not {FPS}/1"
|
||||
for p in (b0, b1, b2, bnar):
|
||||
os.remove(p)
|
||||
os.remove(aud); os.remove(out)
|
||||
|
||||
+52
-3
@@ -70,20 +70,38 @@ def build_scene(data: SceneInput):
|
||||
|
||||
# Speaker attribution is the vision/dialogue model's job now (it reads bubble tails + turn-taking).
|
||||
# speaker is a local_id -> map to character_id via identity; None for narration/sfx/off-panel.
|
||||
# The orchestrator's normalize_dialogue already rewrote `speaker` to a character_id and put the
|
||||
# typed answer in `speaker_ref`, so the local-id map no longer matches it. Read speaker_ref FIRST;
|
||||
# every lookup used to miss and every line narrated as "Someone".
|
||||
dialogue = []
|
||||
for d in data.vision_result.get("dialogue", []):
|
||||
raw = d.get("speaker")
|
||||
ref = d.get("speaker_ref") or {}
|
||||
# "unknown" is a DELIBERATE off-panel/indeterminate speaker (narrated as "someone"); it maps
|
||||
# to None just like narration. id_by_local.get already yields None for it.
|
||||
dialogue.append({"speaker": id_by_local.get(raw), "text": d.get("text", ""),
|
||||
if ref.get("kind") == "character_id" and ref.get("value"):
|
||||
cid = ref["value"]
|
||||
else:
|
||||
cid = id_by_local.get(raw) or (raw if raw in name_by_id else None)
|
||||
dialogue.append({"speaker": cid, "text": d.get("text", ""),
|
||||
"type": d.get("type", "speech"),
|
||||
"confidence": d.get("confidence"),
|
||||
"speaker_method": d.get("speaker_method", "unknown")})
|
||||
|
||||
vchars = data.vision_result.get("characters", [])
|
||||
action = "; ".join(c.get("action", "") for c in vchars if c.get("action")) or ""
|
||||
# `actions` is the list the orchestrator's beat builder reads for verifier evidence; `action` is the
|
||||
# joined string the script prompt renders. Emitting only the string left verification blind.
|
||||
# `characters` above already drops an unassigned detection, but its ACTION used to survive, so a
|
||||
# background extra standing at a window became a sentence the narrator read out. A detection with no
|
||||
# detected face is an extra, a figure on a poster, or scenery gemma called a person
|
||||
# (`decisions/identity-bbox.md#face-gates-enrollment`). Gate on `is False`, like the enrollment gate,
|
||||
# so a blob written before the gate existed and a panel whose detector failed both behave as before.
|
||||
actions = [c["action"].strip() for c in vchars
|
||||
if str(c.get("action") or "").strip() and c.get("has_face") is not False]
|
||||
action = "; ".join(actions)
|
||||
return {"panel_id": data.panel_id, "characters": characters, "dialogue": dialogue,
|
||||
"action": action, "entities": data.vision_result.get("entities", []),
|
||||
"action": action, "actions": actions,
|
||||
"entities": data.vision_result.get("entities", []),
|
||||
"camera": data.vision_result.get("camera", {}), # #8 direction passthrough -> render
|
||||
"transition": data.vision_result.get("transition", "cut")} # #6 transition out
|
||||
|
||||
@@ -120,6 +138,20 @@ if __name__ == "__main__":
|
||||
"description": '{"hair":"black","features":["glasses"]}'}],
|
||||
))
|
||||
assert outu["characters"][0]["name"] == "" and outu["characters"][0]["label"] == "the one with black hair and glasses"
|
||||
# a faceless detection's ACTION never reaches the script prompt or the verifier's evidence, but a
|
||||
# real cast member's does, and so does one on a panel where the detector never ran (no key).
|
||||
p7 = build_scene(SceneInput(
|
||||
panel_id="p007",
|
||||
vision_result={"characters": [
|
||||
{"local_id": "person_1", "action": "reading a report", "has_face": True},
|
||||
{"local_id": "person_2", "action": "standing at the window", "has_face": False},
|
||||
{"local_id": "person_3", "action": "pointing"},
|
||||
]},
|
||||
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
|
||||
characters_registry=[{"character_id": "c1", "name": "Seonho"}],
|
||||
))
|
||||
assert p7["actions"] == ["reading a report", "pointing"], p7["actions"]
|
||||
assert "window" not in p7["action"], p7["action"]
|
||||
assert _describe({}) == "" # bare appearance -> empty, script falls back to Person X
|
||||
# animals are named as their species (color + species), never by clothing/"the one with..."
|
||||
assert _describe({"species": "cat", "hair": "black"}) == "the black cat"
|
||||
@@ -167,4 +199,21 @@ if __name__ == "__main__":
|
||||
))
|
||||
assert out5["dialogue"][0]["confidence"] == 0.3
|
||||
assert out5["dialogue"][0]["speaker_method"] == "turn_taking"
|
||||
|
||||
# orchestrator-normalized rows: `speaker` is already a character_id and `speaker_ref` is typed.
|
||||
# the local-id map cannot resolve either — speaker_ref must win, or everything narrates as Someone.
|
||||
out6 = build_scene(SceneInput(
|
||||
panel_id="p007",
|
||||
vision_result={"characters": [{"local_id": "person_1", "bbox": [0, 0, 10, 10]}],
|
||||
"dialogue": [{"speaker": "c1", "speaker_ref": {"kind": "character_id", "value": "c1"},
|
||||
"type": "speech", "text": "Hey."},
|
||||
{"speaker": None, "speaker_ref": {"kind": "unknown", "value": None},
|
||||
"type": "speech", "text": "..."}]},
|
||||
identity_assignments=[{"local_id": "person_1", "character_id": "c1"}],
|
||||
characters_registry=[{"character_id": "c1", "name": "Teto"}],
|
||||
))
|
||||
assert out6["dialogue"][0]["speaker"] == "c1", out6["dialogue"][0]
|
||||
assert out6["dialogue"][1]["speaker"] is None, out6["dialogue"][1]
|
||||
# actions survive as a list for the verifier, not only as the joined prompt string
|
||||
assert out["actions"] == ["waving"] and out["action"] == "waving"
|
||||
print("worker_scene self-check ok")
|
||||
|
||||
+48
-6
@@ -21,6 +21,10 @@ class ScriptInput(BaseModel):
|
||||
recent: list = [] # #1: last few panels' narration text (local flow, no repeats)
|
||||
introduced: list = [] # #2: names already narrated -> refer by name/pronoun, don't re-describe
|
||||
panel_count: int = 1 # scene-level narration: #panels in this beat -> scales the length budget
|
||||
# the orchestrator has been sending both of these on a failed-script retry since the verifier
|
||||
# landed; pydantic dropped them silently, so the retry was another blind roll of the dice.
|
||||
beat: dict = {} # {beat_id, panels:[{panel_id, dialogue:[{text}], actions}]} — the evidence
|
||||
verifier_feedback: list = [] # [{kind, values|quote}] from verify_script on the previous attempt
|
||||
|
||||
|
||||
class SummaryInput(BaseModel):
|
||||
@@ -80,9 +84,33 @@ def _chars_line(name_by_id, genders_by_id):
|
||||
return ", ".join(out) or "none"
|
||||
|
||||
|
||||
def _feedback_block(feedback, beat) -> str:
|
||||
"""Render the previous attempt's verifier failures as concrete corrections. Without this the
|
||||
retry is just another sample at the same temperature — the model never learns what was wrong."""
|
||||
if not feedback:
|
||||
return ""
|
||||
lines = []
|
||||
for f in feedback:
|
||||
kind = f.get("kind")
|
||||
if kind == "unsupported-proper-noun":
|
||||
lines.append("- You used name(s) that are not in this beat: "
|
||||
+ ", ".join(str(v) for v in f.get("values", []))
|
||||
+ ". Use only the names listed under 'Characters present', or a pronoun.")
|
||||
elif kind == "misquote":
|
||||
lines.append(f'- Your quote "{f.get("quote","")}" is not what the panel says. '
|
||||
"Quote the exact words below, or drop the quotation marks.")
|
||||
else:
|
||||
lines.append(f"- {kind}: {json.dumps(f, ensure_ascii=False)[:200]}")
|
||||
quotes = [d.get("text", "") for p in (beat or {}).get("panels", [])
|
||||
for d in p.get("dialogue", []) if d.get("text")]
|
||||
exact = ("Exact lines you may quote from:\n" + "\n".join(f'- "{q}"' for q in quotes) + "\n") if quotes else ""
|
||||
return ("Your previous attempt was REJECTED. Fix exactly these problems and rewrite it:\n"
|
||||
+ "\n".join(lines) + "\n" + exact + "\n")
|
||||
|
||||
|
||||
def build_prompt(sg: dict, chapter_context: str, names_by_id=None,
|
||||
brief: str = "", recent=None, introduced=None, genders_by_id=None,
|
||||
panel_count: int = 1) -> str:
|
||||
panel_count: int = 1, beat=None, verifier_feedback=None) -> str:
|
||||
name_by_id = _name_map(sg.get("characters", []), names_by_id)
|
||||
chars = _chars_line(name_by_id, genders_by_id)
|
||||
dialogue = "\n".join(_render_line(d, name_by_id) for d in sg.get("dialogue", [])) or "none"
|
||||
@@ -120,7 +148,7 @@ def build_prompt(sg: dict, chapter_context: str, names_by_id=None,
|
||||
else:
|
||||
unit, budget, thin = "this manga panel", "1-2 short sentences, max ~25 words", "panel"
|
||||
return (
|
||||
ov + rec + ctx + intro +
|
||||
_feedback_block(verifier_feedback, beat) + ov + rec + ctx + intro +
|
||||
f"Write TIGHT recap narration for {unit}: {budget}, "
|
||||
"present tense. Tell it like you're recapping to a friend — natural, with momentum. Output "
|
||||
"only the narration — no preamble, no notes, no alternatives, never ask for more info. If the "
|
||||
@@ -182,7 +210,7 @@ async def summary(data: SummaryInput):
|
||||
async def script(data: ScriptInput):
|
||||
text = call_gemma4(build_prompt(data.scene_graph, data.chapter_context, data.names_by_id,
|
||||
data.brief, data.recent, data.introduced, data.genders_by_id,
|
||||
data.panel_count))
|
||||
data.panel_count, data.beat, data.verifier_feedback))
|
||||
return {"panel_id": data.panel_id, "text": text}
|
||||
|
||||
|
||||
@@ -199,10 +227,16 @@ class NormalizeInput(BaseModel):
|
||||
|
||||
def _extract_json(raw: str) -> dict:
|
||||
text = _strip_thought(raw)
|
||||
m = re.search(r"\{.*\}", text, re.DOTALL)
|
||||
if not m:
|
||||
i = text.find("{")
|
||||
if i < 0:
|
||||
raise ValueError(f"no json: {text[:200]}")
|
||||
return json.loads(m.group(0))
|
||||
# raw_decode stops at the end of the FIRST object. The old greedy `\{.*\}` swallowed any trailing
|
||||
# brace in prose after it and turned a parseable reply into a wasted repair call.
|
||||
try:
|
||||
obj, _ = json.JSONDecoder().raw_decode(text[i:])
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"bad json: {text[:200]}") from e
|
||||
return obj
|
||||
|
||||
|
||||
def build_normalize_prompt(names, entities) -> str:
|
||||
@@ -297,4 +331,12 @@ if __name__ == "__main__":
|
||||
nr = _extract_json('{"entities":[{"name":"Everyday","kind":"shop"}],'
|
||||
'"name_fixes":{"CHOI HAESEON":"Choi Haeseon"}}')
|
||||
assert nr["entities"][0]["name"] == "Everyday" and nr["name_fixes"]["CHOI HAESEON"] == "Choi Haeseon"
|
||||
# trailing braced prose after the object must not break the parse (was a wasted repair call)
|
||||
assert _extract_json('{"a":1}\nnote: {not json}')["a"] == 1
|
||||
# verifier feedback reaches the retry prompt with the exact allowed quotes; absent by default
|
||||
assert "REJECTED" not in p
|
||||
beat = {"beat_id": "b1", "panels": [{"panel_id": "p1", "dialogue": [{"text": "stand proud"}]}]}
|
||||
pf = build_prompt(sg, "s", beat=beat, verifier_feedback=[
|
||||
{"kind": "unsupported-proper-noun", "values": ["Kyoto"]}, {"kind": "misquote", "quote": "be proud"}])
|
||||
assert "REJECTED" in pf and "Kyoto" in pf and "be proud" in pf and '- "stand proud"' in pf
|
||||
print("worker_script self-check ok")
|
||||
|
||||
+22
-4
@@ -36,7 +36,17 @@ def _load_tts():
|
||||
transformers.AutoTokenizer.from_pretrained = classmethod(
|
||||
lambda cls, *a, **kw: _orig(cls, *a, **{"fix_mistral_regex": True, **kw})
|
||||
)
|
||||
# dots_tts/__init__.py refuses to import when torch and torchaudio minors differ. workpc runs
|
||||
# Arch's torch 2.13 but pytorch.org ships no torchaudio past 2.11 for ROCm, so the pair can't
|
||||
# be satisfied; 2.11 loads and resamples fine against 2.13. Lie to the guard for the import.
|
||||
# ponytail: drop this once a torchaudio matching torch's minor exists for ROCm.
|
||||
import torch, importlib.metadata as _md
|
||||
_ver = _md.version
|
||||
_md.version = lambda n: torch.__version__ if n == "torchaudio" else _ver(n)
|
||||
try:
|
||||
from dots_tts.runtime import DotsTtsRuntime
|
||||
finally:
|
||||
_md.version = _ver
|
||||
_tts = DotsTtsRuntime.from_pretrained(DOTS_MODEL, precision="bfloat16")
|
||||
return _tts
|
||||
|
||||
@@ -60,9 +70,10 @@ def _audio_uri(data: "TTSInput") -> str:
|
||||
# ponytail: flat key collides across chapters (as does the homesrv audio table); pass
|
||||
# panel_uri from run_stage_tts to make it per-chapter unique.
|
||||
if data.panel_uri:
|
||||
parts = data.panel_uri.replace("s3://", "").split("/")
|
||||
return f"s3://manga/{parts[1]}/{parts[2]}/audio/{data.panel_id or 'p'}.wav"
|
||||
return f"s3://manga/_audio/{data.panel_id or 'p'}.wav"
|
||||
manga_id, chapter_id = transport.ids_from_uri(data.panel_uri)
|
||||
return transport.AUDIO_URI.format(manga_id=manga_id, chapter_id=chapter_id,
|
||||
name=data.panel_id or 'p')
|
||||
return transport.AUDIO_FLAT_URI.format(name=data.panel_id or 'p')
|
||||
|
||||
|
||||
def _ensure_ref() -> str:
|
||||
@@ -207,6 +218,13 @@ if __name__ == "__main__":
|
||||
os.remove(explicit)
|
||||
assert _calm("Stop!! Now!") == "Stop. Now." and _calm("no bangs") == "no bangs" # even prosody
|
||||
|
||||
# _audio_uri: a panel uri puts the wav beside its chapter, no panel uri falls back to the flat key.
|
||||
assert _audio_uri(TTSInput(text="x", panel_id="p003",
|
||||
panel_uri="s3://panels/m1/c1/panels/p003.png")) \
|
||||
== "s3://audio/m1/c1/audio/p003.wav"
|
||||
assert _audio_uri(TTSInput(text="x", panel_id="p003")) == "s3://audio/_audio/p003.wav"
|
||||
assert _audio_uri(TTSInput(text="x")) == "s3://audio/_audio/p.wav"
|
||||
|
||||
# 158: pronunciation lexicon respells whole words only, case-insensitive, spoken text only.
|
||||
import tempfile, json as _json
|
||||
globals()["LEXICON_PATH"] = os.path.join(tempfile.mkdtemp(), "lex.json")
|
||||
@@ -223,7 +241,7 @@ if __name__ == "__main__":
|
||||
# loudnorm: with ffmpeg present the wav is normalized in place and stays readable at its sr;
|
||||
# without ffmpeg it's a safe no-op returning the same path (audio never lost).
|
||||
ln = _write_wav(samples, 16000)
|
||||
have_ffmpeg = subprocess.run(["ffmpeg", "-version"], capture_output=True).returncode == 0 \
|
||||
have_ffmpeg = subprocess.run(["ffmpeg", "-version"], capture_output=True, check=False).returncode == 0 \
|
||||
if __import__("shutil").which("ffmpeg") else False
|
||||
assert _loudnorm(ln) == ln and os.path.exists(ln)
|
||||
assert wave.open(ln, "rb").getframerate() == 16000
|
||||
|
||||
+311
-36
@@ -19,7 +19,9 @@ GEMMA4_URL = os.environ.get("GEMMA4_URL", "http://127.0.0.1:8090")
|
||||
|
||||
# set-of-mark speaker attribution: detect text regions with comic-text-detector (ONNX, CPU) and draw
|
||||
# numbered boxes on the panel so gemma transcribes/attributes grounded regions instead of eyeballing
|
||||
# the whole image. Off by default — flip SOM_ATTRIBUTION=1 once tuned per title (see bubble_detect.py).
|
||||
# the whole image. ON by default: face->identity pairing is now gated on containment, so an unmatched
|
||||
# face is labelled "unknown" instead of borrowing the nearest name (see _pair_faces_to_present).
|
||||
# SOM_ATTRIBUTION=0 falls back to the holistic path (see bubble_detect.py).
|
||||
try:
|
||||
import bubble_detect
|
||||
except Exception: # onnxruntime/model absent -> feature simply stays unavailable
|
||||
@@ -31,29 +33,45 @@ except Exception:
|
||||
SOM = os.environ.get("SOM_ATTRIBUTION", "1") == "1"
|
||||
|
||||
|
||||
def _pair_faces_to_present(det_faces: list, present: list) -> list:
|
||||
def _pair_faces_to_present(det_faces: list, present: list, margin: float = 0.25) -> list:
|
||||
"""det_faces = real detector boxes (grounded but identity-less); present = characters gemma placed
|
||||
in the panel (identity + a coarse, imprecise bbox). Assign each detected face the identity of the
|
||||
present character whose gemma-bbox centre falls closest to it, so a green box carries a real name.
|
||||
A face with no nearby present char stays unknown; a present char is used at most once."""
|
||||
in the panel (identity + a coarse, imprecise bbox). Pair a face with a present character ONLY when
|
||||
the face centre falls inside that character's bbox grown by `margin` of its size — gemma's boxes are
|
||||
imprecise but not arbitrary. An unpaired face stays unknown instead of borrowing whoever happens to
|
||||
be nearest; the label it carries becomes `speaker_method="som_face"`, the highest-trust provenance
|
||||
the pipeline has, so an ungated guess used to launder itself into evidence.
|
||||
Pairs are taken globally shortest-first, so the first face processed cannot claim a character that
|
||||
fits a later face far better. Each face and each character is used at most once."""
|
||||
def cx_cy(b): return ((b[0] + b[2]) / 2, (b[1] + b[3]) / 2)
|
||||
used = set()
|
||||
out = []
|
||||
for i, f in enumerate(det_faces, 1):
|
||||
|
||||
def contains(face_box, char_box) -> bool:
|
||||
w, h = char_box[2] - char_box[0], char_box[3] - char_box[1]
|
||||
if w <= 0 or h <= 0:
|
||||
return False
|
||||
fx, fy = cx_cy(face_box)
|
||||
return (char_box[0] - margin * w <= fx <= char_box[2] + margin * w
|
||||
and char_box[1] - margin * h <= fy <= char_box[3] + margin * h)
|
||||
|
||||
pairs = []
|
||||
for i, f in enumerate(det_faces):
|
||||
fx, fy = cx_cy(f["bbox"])
|
||||
best, bestd = None, 1e18
|
||||
for j, c in enumerate(present):
|
||||
b = c.get("bbox")
|
||||
if not b or j in used:
|
||||
if not b or len(b) != 4 or not contains(f["bbox"], b):
|
||||
continue
|
||||
px, py = cx_cy(b)
|
||||
d = (px - fx) ** 2 + (py - fy) ** 2
|
||||
if d < bestd:
|
||||
best, bestd = j, d
|
||||
c = present[best] if best is not None else {}
|
||||
if best is not None:
|
||||
used.add(best)
|
||||
out.append({"label": f"P{i}", "bbox": f["bbox"], "local_id": c.get("local_id"),
|
||||
pairs.append(((px - fx) ** 2 + (py - fy) ** 2, i, j))
|
||||
taken_f, taken_c, match = set(), set(), {}
|
||||
for _, i, j in sorted(pairs):
|
||||
if i in taken_f or j in taken_c:
|
||||
continue
|
||||
taken_f.add(i)
|
||||
taken_c.add(j)
|
||||
match[i] = j
|
||||
out = []
|
||||
for i, f in enumerate(det_faces):
|
||||
c = present[match[i]] if i in match else {}
|
||||
out.append({"label": f"P{i + 1}", "bbox": f["bbox"], "local_id": c.get("local_id"),
|
||||
"who": c.get("name") or c.get("desc") or "unknown", "gender": c.get("gender")})
|
||||
return out
|
||||
|
||||
@@ -105,16 +123,65 @@ def _set_of_mark(local_path: str, present: list):
|
||||
return marked, "\n" + "\n".join(parts) + "\n", {f["label"]: f["local_id"] or "unknown" for f in faces}
|
||||
|
||||
|
||||
def _apply_speaker_labels(dialogue: list, label_map: dict) -> list:
|
||||
"""map a set-of-mark face label ("P1") in the speaker field back to its local_id. gemma may also
|
||||
answer with the local_id directly (legend shows both) — that already matches, so it's left as-is."""
|
||||
if not label_map:
|
||||
return dialogue
|
||||
_GENDER_SUFFIX = re.compile(r"\s*\((?:m|f|male|female|man|woman|unknown)\)\s*$", re.IGNORECASE)
|
||||
_ID_SHAPED = re.compile(r"^(?:person|char|face|p)[\s_-]*\d+$", re.IGNORECASE)
|
||||
|
||||
|
||||
def _present_keys(present: list) -> dict:
|
||||
"""Every string the dialogue prompt shows for a present character, mapped to its local_id.
|
||||
|
||||
`build_dialogue_prompt` renders each one as `- person_1: brown ponytail, green dress (f)`, and gemma
|
||||
answers with any part of that line, most often the description. Those answers used to fall through
|
||||
`normalize_speaker` as free-form NAMES and never matched the registry, losing 28 of 51 speech lines
|
||||
on job 778297bc. A key shared by two present characters is dropped: it cannot identify either."""
|
||||
keys: dict = {}
|
||||
for c in present:
|
||||
lid = (c.get("local_id") or "").strip()
|
||||
if not lid:
|
||||
continue
|
||||
name, desc, gender = c.get("name") or "", c.get("desc") or "", c.get("gender") or ""
|
||||
shown = name or desc or "unknown"
|
||||
forms = {lid, name, desc, shown}
|
||||
if gender and gender != "unknown":
|
||||
forms |= {f"{f} ({gender})" for f in (name, desc, shown) if f}
|
||||
for f in forms:
|
||||
k = f.strip().casefold()
|
||||
if not k or k == "unknown": # the prompt prints "unknown" for a nameless character
|
||||
continue
|
||||
keys[k] = lid if keys.get(k, lid) == lid else None # ambiguous key -> unusable
|
||||
return {k: v for k, v in keys.items() if v}
|
||||
|
||||
|
||||
def _apply_speaker_labels(dialogue: list, label_map: dict, present: list | None = None) -> list:
|
||||
"""Map gemma's speaker answer back to a panel-local id.
|
||||
|
||||
Three answer shapes reach here: a set-of-mark face label ("P1"), a local_id, and any string the
|
||||
prompt showed for a present character. An id-shaped answer that names nobody present is junk and
|
||||
becomes "unknown" rather than a name claim (invariant 6). A trailing gender marker is stripped, so
|
||||
"Seonho (m)" can still match the registry name "Seonho" off-panel."""
|
||||
keys = _present_keys(present or [])
|
||||
for d in dialogue:
|
||||
s = (d.get("speaker") or "").strip()
|
||||
if not s:
|
||||
continue
|
||||
if s in label_map:
|
||||
d["speaker"] = label_map[s]
|
||||
lid = label_map[s]
|
||||
# _set_of_mark labels a detected face `unknown` when gated pairing matched it to no present
|
||||
# character. An answer pointing at such a mark grounds nothing, so it must not carry
|
||||
# `som_face`, the highest-trust label. 7 of 7 som_face lines on the first 36 panels of the
|
||||
# 2026-08-11 rerun were this case.
|
||||
d["speaker"] = lid
|
||||
if lid != "unknown":
|
||||
d["speaker_method"] = "som_face"
|
||||
continue
|
||||
bare = _GENDER_SUFFIX.sub("", s).strip()
|
||||
lid = keys.get(s.casefold()) or keys.get(bare.casefold())
|
||||
if lid:
|
||||
d["speaker"] = lid
|
||||
elif _ID_SHAPED.match(bare):
|
||||
d["speaker"] = "unknown" # an id for nobody present: never mint a name from it
|
||||
elif bare != s:
|
||||
d["speaker"] = bare # off-panel name, gender marker stripped
|
||||
return dialogue
|
||||
|
||||
|
||||
@@ -127,10 +194,17 @@ def _strip_thought(text: str) -> str:
|
||||
def _extract_json(raw: str) -> dict:
|
||||
"""strip gemma4 thought, pull the first JSON object, parse it."""
|
||||
text = _strip_thought(raw)
|
||||
m = re.search(r"\{.*\}", text, re.DOTALL)
|
||||
if not m:
|
||||
i = text.find("{")
|
||||
if i < 0:
|
||||
raise ValueError(f"no json in response: {text[:200]}")
|
||||
return json.loads(m.group(0))
|
||||
# raw_decode stops at the end of the FIRST object. The old greedy `\{.*\}` ran to the LAST brace in
|
||||
# the reply, so a second object or any trailing braced prose produced an unparseable span and burned
|
||||
# a repair call on a response that was already fine.
|
||||
try:
|
||||
obj, _ = json.JSONDecoder().raw_decode(text[i:])
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"bad json in response: {text[:200]}") from e
|
||||
return obj
|
||||
|
||||
|
||||
def _img_part(image_path: str) -> dict:
|
||||
@@ -226,6 +300,73 @@ class VisionInput(BaseModel):
|
||||
session_id: str = ""
|
||||
|
||||
|
||||
BBOX_GRID = 1000 # gemma's native normalized box grid
|
||||
|
||||
|
||||
def _bbox_to_pixels(chars: list, w: int, h: int) -> list:
|
||||
"""Rewrite every character box from gemma's 0-1000 grid to pixels on this panel.
|
||||
|
||||
The prompt asks for pixels. The model answers on its own normalized grid regardless. Measured over
|
||||
the 113 detections of job 778297bc: 47 boxes had x2 beyond the 900px panel width, and not one had y2
|
||||
beyond 1000 on panels 1257 to 2307px tall. Consumed as pixels the boxes collapse into the top-left
|
||||
corner of the panel, which is how identity came to embed crops of speech balloons and window frames
|
||||
and match them at 0.9, and why gated face pairing returned 7 unknowns out of 7 real faces.
|
||||
|
||||
Convert once here so every consumer sees pixels: `_crop_bbox` in identity, the face pairing below,
|
||||
the set-of-mark boxes, and the review UI's client-side crop.
|
||||
|
||||
ponytail: the prompt still says "pixel bounding box". Rewording it would change what the model
|
||||
emits and needs a GPU run to re-verify, so the boundary converts instead. If a future model really
|
||||
does answer in pixels, this scales them down -- check the box range before swapping models.
|
||||
"""
|
||||
for c in chars:
|
||||
b = c.get("bbox")
|
||||
if not (isinstance(b, list) and len(b) == 4 and all(isinstance(v, (int, float)) for v in b)):
|
||||
continue
|
||||
xs = sorted((min(w, max(0, round(b[0] * w / BBOX_GRID))),
|
||||
min(w, max(0, round(b[2] * w / BBOX_GRID)))))
|
||||
ys = sorted((min(h, max(0, round(b[1] * h / BBOX_GRID))),
|
||||
min(h, max(0, round(b[3] * h / BBOX_GRID)))))
|
||||
# ponytail: the model swapped corners on 1 of 117 boxes (p007 person_1, x1 > x2).
|
||||
# Ordering here is enough because a zero-area box still crops to nothing downstream.
|
||||
c["bbox"] = [xs[0], ys[0], xs[1], ys[1]]
|
||||
return chars
|
||||
|
||||
|
||||
def _mark_has_face(img, chars: list) -> list:
|
||||
"""Stamp `has_face` on every character a real detected face lands inside.
|
||||
|
||||
gemma calls every person-shaped region a character. On panel 7 of job 778297bc that was two
|
||||
people and four background extras, one of which is an empty window mullion. All six reached
|
||||
identity as equal candidates, and after the bbox fix an extra took `Seonho` at confidence 1.00,
|
||||
which puts an extra's crop into the lead's reference set. A faceless crop is a bad embedding as
|
||||
well as a bad reference: the back of a head or a patch of coat matches almost anything.
|
||||
|
||||
Reuses `_pair_faces_to_present`, so containment, its margin, and the global shortest-first
|
||||
assignment are exactly the rules the speaker path already uses. Requires pixel boxes, so call it
|
||||
after `_bbox_to_pixels`.
|
||||
|
||||
Fails open. A missing or broken detector marks everything `True`, because dropping every
|
||||
character is worse than the over-detection this gate exists to trim.
|
||||
"""
|
||||
def _all(v):
|
||||
for c in chars:
|
||||
c["has_face"] = v
|
||||
return chars
|
||||
|
||||
if face_detect is None or not chars:
|
||||
return _all(True)
|
||||
try:
|
||||
faces = face_detect.detect_faces(img)
|
||||
except Exception as e:
|
||||
print(f"[vision/detect] face detect failed, has_face gate off for this panel: {e}", flush=True)
|
||||
return _all(True)
|
||||
paired = {p["local_id"] for p in _pair_faces_to_present(faces, chars) if p.get("local_id")}
|
||||
for c in chars:
|
||||
c["has_face"] = c.get("local_id") in paired
|
||||
return chars
|
||||
|
||||
|
||||
@app.post("/vision")
|
||||
async def vision(data: VisionInput):
|
||||
local = transport.get(data.panel_uri, f"{SHM}/vision_{uuid.uuid4().hex[:8]}.png")
|
||||
@@ -241,8 +382,17 @@ async def vision(data: VisionInput):
|
||||
print(f"[vision/detect] parse failed for {data.panel_id} after repair retry: {e}", flush=True)
|
||||
result = {"skip": False, "parse_failed": True, "characters": [], "scene": {}}
|
||||
finally:
|
||||
import cv2
|
||||
img = cv2.imread(local) # read once: the size and the face gate both need it
|
||||
os.remove(local)
|
||||
pw, ph = (img.shape[1], img.shape[0]) if img is not None else (0, 0)
|
||||
result.setdefault("characters", [])
|
||||
if pw and ph:
|
||||
_bbox_to_pixels(result["characters"], pw, ph)
|
||||
_mark_has_face(img, result["characters"])
|
||||
else:
|
||||
print(f"[vision/detect] panel size unreadable for {data.panel_id}, boxes left normalized",
|
||||
flush=True)
|
||||
result["panel_id"] = data.panel_id
|
||||
return result
|
||||
|
||||
@@ -349,16 +499,31 @@ def resolve_speakers(dialogue: list, present: list) -> list:
|
||||
|
||||
|
||||
def _annotate_speaker_methods(dialogue: list, present: list) -> list:
|
||||
"""Fill provenance for model-attributed lines without overwriting grounded/backstop methods."""
|
||||
"""Fill provenance for model-attributed lines without overwriting grounded/backstop methods.
|
||||
|
||||
A speaker matching a present local_id used to be stamped `tail`, the highest-trust label, at
|
||||
gemma's own confidence of 1.0. No balloon geometry was ever read. On the 2026-08-11 chapter every
|
||||
sampled two-character panel had the speakers swapped
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`), so with 2+ present the guess is dropped
|
||||
rather than shipped as truth. With one present it is the same claim as the solo backstop, so it is
|
||||
kept and named for what it is.
|
||||
ponytail: drop-on-crowd is the honest floor, not the fix. Bind by tail geometry when the balloon
|
||||
detector lands, then this branch reads a tail for real."""
|
||||
local_ids = {c.get("local_id") for c in present if c.get("local_id")}
|
||||
crowded = len(present) > 1
|
||||
for d in dialogue:
|
||||
if d.get("speaker_method"):
|
||||
continue
|
||||
speaker = (d.get("speaker") or "").strip()
|
||||
if d.get("type", "speech") not in _SPEECH or not speaker or speaker == "unknown":
|
||||
d["speaker_method"] = "unknown"
|
||||
elif speaker in local_ids and crowded:
|
||||
d["speaker"] = "unknown"
|
||||
d["confidence"] = 0.0
|
||||
d["speaker_method"] = "unknown"
|
||||
elif speaker in local_ids:
|
||||
d["speaker_method"] = "tail"
|
||||
d["speaker_method"] = "model_solo"
|
||||
d["confidence"] = min(float(d.get("confidence") or 0.7), 0.7)
|
||||
else:
|
||||
d["speaker_method"] = "turn_taking"
|
||||
return dialogue
|
||||
@@ -425,7 +590,7 @@ async def dialogue(data: DialogueInput):
|
||||
result.setdefault("entities", [])
|
||||
result.setdefault("named", []) # [{local_id, name}] explicit namings -> bound to character_id by orchestrator
|
||||
result["named"] = _normalize_claims(result["named"], data.panel_id)
|
||||
_apply_speaker_labels(result["dialogue"], label_map)
|
||||
_apply_speaker_labels(result["dialogue"], label_map, data.present_characters)
|
||||
resolve_speakers(result["dialogue"], data.present_characters)
|
||||
result["panel_id"] = data.panel_id
|
||||
result.update(_dialogue_envelope([data.panel_id], [] if parse_failed else [result], parse_failed))
|
||||
@@ -540,7 +705,7 @@ async def dialogue_window(data: DialogueWindowInput):
|
||||
d = by_id.get(pid)
|
||||
if d is None: # missing is unresolved, never manufactured as a silent success
|
||||
continue
|
||||
_apply_speaker_labels(d.get("dialogue", []), label_maps.get(pid, {}))
|
||||
_apply_speaker_labels(d.get("dialogue", []), label_maps.get(pid, {}), present_by_id.get(pid, []))
|
||||
out.append({
|
||||
"panel_id": pid,
|
||||
"dialogue": resolve_speakers(d.get("dialogue", []), present_by_id.get(pid, [])),
|
||||
@@ -856,9 +1021,16 @@ def build_resolve_prompt(candidates: list) -> str:
|
||||
)
|
||||
|
||||
|
||||
REF_BUDGET = 12 # total reference images one /vision/resolve prompt may carry, beside the query crop
|
||||
|
||||
|
||||
def _refs_per_candidate(n: int) -> int:
|
||||
return max(1, min(3, REF_BUDGET // max(1, n)))
|
||||
|
||||
|
||||
class ResolveInput(BaseModel):
|
||||
crop_uri: str
|
||||
candidates: list = [] # gender-gated cosine shortlist [{character_id, name, gender, ...}]
|
||||
candidates: list = [] # gender-gated gallery [{character_id, name, gender, ...}]
|
||||
session_id: str = ""
|
||||
|
||||
|
||||
@@ -877,9 +1049,13 @@ async def vision_resolve(data: ResolveInput):
|
||||
return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"}
|
||||
crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png")
|
||||
refs = []
|
||||
# image budget, not a per-candidate rule. The gallery is the live cast now, not a cosine top-k, so
|
||||
# 3 references each was 27 images plus the query on a 9-character cast. Spread REF_BUDGET across the
|
||||
# candidates instead: 3 references while the cast is small, 1 apiece once it is not.
|
||||
per = _refs_per_candidate(len(data.candidates))
|
||||
try:
|
||||
for i, candidate in enumerate(data.candidates, 1):
|
||||
for uri in (candidate.get("reference_image_uris") or [])[:3]:
|
||||
for uri in (candidate.get("reference_image_uris") or [])[:per]:
|
||||
try:
|
||||
refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png")))
|
||||
except Exception as e:
|
||||
@@ -893,10 +1069,18 @@ async def vision_resolve(data: ResolveInput):
|
||||
os.remove(crop)
|
||||
for _, path in refs:
|
||||
os.remove(path)
|
||||
# map gemma's 1-based choice back to a character_id; 0 / out-of-range -> NONE (new character).
|
||||
# map gemma's 1-based choice back to a character_id. An explicit 0 means NONE -> a new character.
|
||||
# An OUT-OF-RANGE index is a hallucination, not an answer: it must be `unresolved` like a parse
|
||||
# failure, or a bad index silently mints a brand new character in the permanent registry.
|
||||
choice = result.get("choice", 0)
|
||||
cid = data.candidates[choice - 1]["character_id"] if isinstance(choice, int) and 1 <= choice <= len(data.candidates) else None
|
||||
state = "known" if cid else ("unresolved" if result.get("reason") == "parse_failed" else "new")
|
||||
in_range = isinstance(choice, int) and 1 <= choice <= len(data.candidates)
|
||||
cid = data.candidates[choice - 1]["character_id"] if in_range else None
|
||||
if in_range:
|
||||
state = "known"
|
||||
elif choice == 0 and result.get("reason") != "parse_failed":
|
||||
state = "new"
|
||||
else:
|
||||
state = "unresolved"
|
||||
return {"character_id": cid, "state": state, "confidence": float(result.get("confidence", 0.0)),
|
||||
"reason": result.get("reason", "")}
|
||||
|
||||
@@ -943,11 +1127,34 @@ if __name__ == "__main__":
|
||||
two = [{"local_id": "person_1"}, {"local_id": "person_2"}]
|
||||
assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], two)[0]["speaker"] == "unknown"
|
||||
assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], [])[0]["speaker"] == "unknown"
|
||||
assert resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x"}], two)[0]["speaker_method"] == "tail"
|
||||
# a present-local_id match is gemma's guess, not a read tail: dropped when 2+ are present,
|
||||
# kept but capped at 0.7 when only one is (caveats/speaker-attribution.md#tail-is-not-geometry)
|
||||
crowd = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], two)[0]
|
||||
assert crowd["speaker"] == "unknown" and crowd["speaker_method"] == "unknown" and crowd["confidence"] == 0.0
|
||||
lone = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], solo)[0]
|
||||
assert lone["speaker"] == "person_1" and lone["speaker_method"] == "model_solo" and lone["confidence"] == 0.7
|
||||
# set-of-mark: gemma answers a face label -> remapped to local_id; a name/unknown passes through
|
||||
lbl = _apply_speaker_labels([{"speaker": "P1"}, {"speaker": "Aria"}, {"speaker": "unknown"}],
|
||||
{"P1": "person_3"})
|
||||
assert lbl[0]["speaker"] == "person_3" and lbl[1]["speaker"] == "Aria" and lbl[2]["speaker"] == "unknown"
|
||||
assert lbl[0]["speaker_method"] == "som_face"
|
||||
# a mark whose face paired to nobody present grounds nothing, so it gets no som_face label
|
||||
unpaired = _apply_speaker_labels([{"speaker": "P2"}], {"P2": "unknown"})[0]
|
||||
assert unpaired["speaker"] == "unknown" and "speaker_method" not in unpaired, unpaired
|
||||
# gemma answers with what the prompt SHOWED, not the id: description, name+gender, bare id, junk id.
|
||||
# Each shape cost real lines on job 778297bc by falling through as a free-form name.
|
||||
shown = [{"local_id": "person_1", "desc": "brown ponytail, green dress", "gender": "f"},
|
||||
{"local_id": "person_2", "name": "Seonho", "gender": "m"}]
|
||||
got = _apply_speaker_labels([{"speaker": "brown ponytail, green dress (f)"}, {"speaker": "Seonho"},
|
||||
{"speaker": "person_2"}, {"speaker": "person_9"},
|
||||
{"speaker": "Haeseon (f)"}, {"speaker": "unknown"}], {}, shown)
|
||||
assert [d["speaker"] for d in got] == ["person_1", "person_2", "person_2", "unknown",
|
||||
"Haeseon", "unknown"], got
|
||||
# a description shared by two present characters identifies neither
|
||||
twins = [{"local_id": "person_1", "desc": "schoolgirl"}, {"local_id": "person_2", "desc": "schoolgirl"}]
|
||||
assert _apply_speaker_labels([{"speaker": "schoolgirl"}], {}, twins)[0]["speaker"] == "schoolgirl"
|
||||
# a nameless present character is shown as "unknown"; that must not become an id
|
||||
assert _apply_speaker_labels([{"speaker": "unknown"}], {}, [{"local_id": "person_1"}])[0]["speaker"] == "unknown"
|
||||
# real detector face -> nearest present char's identity; each present char claimed once; leftover unknown
|
||||
pf = _pair_faces_to_present(
|
||||
[{"bbox": [10, 10, 50, 50]}, {"bbox": [200, 10, 240, 50]}, {"bbox": [400, 10, 440, 50]}],
|
||||
@@ -983,6 +1190,13 @@ if __name__ == "__main__":
|
||||
_map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None)
|
||||
assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None
|
||||
assert "reference images" in rp and "face shape/features first" in rp
|
||||
# reference budget: a small gallery keeps 3 refs each, a cast-sized one drops to 1 and stays under
|
||||
# REF_BUDGET images however big the cast gets.
|
||||
assert _refs_per_candidate(1) == 3 and _refs_per_candidate(4) == 3
|
||||
assert _refs_per_candidate(9) == 1 and _refs_per_candidate(0) == 3
|
||||
# the budget holds up to REF_BUDGET candidates; past that the floor of 1 apiece wins, which is why
|
||||
# the orchestrator caps the gallery (GALLERY_CAP) rather than relying on this alone.
|
||||
assert all(n * _refs_per_candidate(n) <= REF_BUDGET for n in range(1, REF_BUDGET + 1))
|
||||
|
||||
# dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty.
|
||||
bad = _dialogue_envelope(["p1"], [], parse_failed=True)
|
||||
@@ -1005,4 +1219,65 @@ if __name__ == "__main__":
|
||||
assert False
|
||||
except ValueError:
|
||||
pass
|
||||
# trailing braced prose after a complete object parses (used to burn a repair call)
|
||||
assert _extract_json('{"skip":false}\nnote {see above}')["skip"] is False
|
||||
|
||||
# gemma's boxes arrive on a 0-1000 grid and leave /vision as pixels. The 900x1650 panel below is
|
||||
# panel 7 of job 778297bc: person_5 is Seonho in the foreground, and read as pixels his box lands in
|
||||
# the top sixth of the panel, inside a speech balloon, which is what identity embedded.
|
||||
_ch = [{"local_id": "person_5", "bbox": [222, 405, 654, 1000]},
|
||||
{"local_id": "edge", "bbox": [0, 0, 1000, 1000]},
|
||||
{"local_id": "junk", "bbox": "nope"},
|
||||
# p007 person_1 of run B: x1 > x2, so the crop came out empty and the detection was lost
|
||||
{"local_id": "swapped", "bbox": [226, 417, 130, 551]}]
|
||||
_bbox_to_pixels(_ch, 900, 1650)
|
||||
assert _ch[3]["bbox"] == [117, 688, 203, 909], _ch[3]["bbox"] # corners ordered, non-empty crop
|
||||
assert _ch[0]["bbox"] == [200, 668, 589, 1650], _ch[0]["bbox"]
|
||||
assert _ch[1]["bbox"] == [0, 0, 900, 1650], _ch[1]["bbox"] # a clamped box spans the whole panel
|
||||
assert _ch[2]["bbox"] == "nope", _ch[2]["bbox"] # unparseable is left alone, not crashed
|
||||
# the box must now cover the lower half of a tall panel, which the raw grid value never can
|
||||
assert _ch[0]["bbox"][3] > 1000 > _ch[0]["bbox"][1]
|
||||
|
||||
# the has_face gate, on panel 7's real converted boxes. Seonho and the colleague carry a detected
|
||||
# face; the two background extras do not, and identity must skip them rather than embed scenery.
|
||||
_p7 = [{"local_id": "person_1", "bbox": [457, 657, 642, 937]}, # Seonho, foreground
|
||||
{"local_id": "person_2", "bbox": [669, 591, 763, 822]}, # the colleague
|
||||
{"local_id": "person_5", "bbox": [31, 554, 94, 728]}, # extra that took Seonho at 1.00
|
||||
{"local_id": "person_6", "bbox": [34, 414, 122, 564]}] # extra
|
||||
_faces = [{"bbox": [520, 690, 590, 770]}, {"bbox": [695, 610, 740, 660]}]
|
||||
class _FD:
|
||||
@staticmethod
|
||||
def detect_faces(_img):
|
||||
return _faces
|
||||
_real_fd, globals()["face_detect"] = face_detect, _FD
|
||||
try:
|
||||
_mark_has_face(None, _p7)
|
||||
assert [c["has_face"] for c in _p7] == [True, True, False, False], _p7
|
||||
# fail open: a detector that raises must not silently drop every character in the panel
|
||||
class _Boom:
|
||||
@staticmethod
|
||||
def detect_faces(_img):
|
||||
raise RuntimeError("onnx died")
|
||||
globals()["face_detect"] = _Boom
|
||||
_mark_has_face(None, _p7)
|
||||
assert all(c["has_face"] for c in _p7), _p7
|
||||
globals()["face_detect"] = None
|
||||
_mark_has_face(None, _p7)
|
||||
assert all(c["has_face"] for c in _p7), _p7
|
||||
finally:
|
||||
globals()["face_detect"] = _real_fd
|
||||
|
||||
# face->identity pairing is GATED on containment: a face outside every gemma bbox stays unknown.
|
||||
faces = [{"bbox": [10, 10, 30, 30]}, {"bbox": [900, 900, 920, 920]}]
|
||||
present = [{"local_id": "person_1", "name": "Teto", "bbox": [0, 0, 100, 200]}]
|
||||
paired = _pair_faces_to_present(faces, present)
|
||||
assert paired[0]["local_id"] == "person_1" and paired[0]["who"] == "Teto"
|
||||
assert paired[1]["local_id"] is None and paired[1]["who"] == "unknown", paired[1]
|
||||
# globally shortest-first: the first face must not claim a character that fits the second better.
|
||||
faces2 = [{"bbox": [95, 95, 105, 105]}, {"bbox": [8, 8, 12, 12]}]
|
||||
present2 = [{"local_id": "a", "bbox": [0, 0, 20, 20]}, {"local_id": "b", "bbox": [80, 80, 120, 120]}]
|
||||
p2 = _pair_faces_to_present(faces2, present2)
|
||||
assert [f["local_id"] for f in p2] == ["b", "a"], p2
|
||||
# no present characters at all -> every face unknown, never a phantom identity
|
||||
assert _pair_faces_to_present(faces, [])[0]["local_id"] is None
|
||||
print("worker_vision self-check ok")
|
||||
|
||||
Reference in New Issue
Block a user