kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Audit second pass: cross-repo contract findings

Browse Source 
Read the workpc workers against the homesrv orchestrator and checked the
first-pass audit against source. Report only, no code changed.

Adds 4 P0, 6 P1, and 13 P2 findings to AUDIT.md, most of them in the seam
between the two repos:

- worker_scene reads dialogue `speaker` as a local id, but the orchestrator
  already rewrote it to a character_id, so all narration says "Someone"
- the script verifier fails valid narration on sentence-initial capitals and
  on short quotes, which halts the chapter
- correctness flags block TTS with no path to clear them when GATES is off
- session_manager can orphan a llama-server that keeps its VRAM

Confirms four first-pass claims in source: tracklet gender enum, missing
action evidence, dropped verifier feedback, 409 lease stealing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kami
2026-08-11 09:31:20 +04:00
parent ffeda47fd2
commit 6d9df5bf2f
2 changed files with 809 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AUDIT.md
+770
View File
@@ -0,0 +1,770 @@
# Manga recap pipeline audit
Date: 2026-08-11
Scope: workpc workers plus the homesrv orchestrator. This audit excludes FFmpeg changes and does not
recommend Magi or its embeddings. No GPU workloads were run during the audit.
## Outcome
The strongest performance gains are available without replacing the current models:
1. Keep Gemma resident across compatible stages, reducing roughly seven chapter-level Gemma loads to two.
2. Batch SigLIP crop embeddings instead of running one forward pass per detected character.
3. Add bounded parallelism to CPU/network work such as page uploads, framed-page cropping, prefetching,
and scene construction.
4. Stop repeatedly downloading and base64-encoding the same panels and character references.
5. Generate schema-constrained JSON and batch text-only script work by scene.
Correctness fixes should land before throughput work because the current tracklet and naming paths can
contaminate the permanent character registry.
## Highest-priority findings
### P0: tracklet gender gate uses the wrong enum
`orchestrator/tracklets.py` checks `male|female`, while vision emits `m|f|unknown`. The tracklet hard
gender gate therefore never activates on normal pipeline data.
### P0: tracklets lack co-presence constraints
The linker has no same-panel exclusion and uses transitive union-find. Two similar-looking people in the
same panel can be joined, and weak links such as A~B and B~C can merge A and C even when the endpoints are
incompatible.
### P0: face-to-character pairing is unconditional
`worker_vision._pair_faces_to_present()` assigns every detected face to the nearest vision character as
long as an unused character exists. Despite the docstring, there is no distance or overlap threshold.
A distant or unrelated face can therefore receive another character's identity and speaker label.
### P0: roster hints can become permanent identity evidence
The roster is described as hints-only, but those names are fed into detection. Any name emitted on a crop
causes `worker_identity` to persist it immediately. A coarse appearance-to-roster guess can therefore
contaminate a permanent character gallery.
### P0: script repair feedback is discarded
The orchestrator sends `beat` and `verifier_feedback` on a failed-script retry, but `worker_script.ScriptInput`
does not define either field and the prompt builder does not consume them. The second call is another
stochastic attempt rather than a targeted correction.
### P0: action evidence is missing from the verifier
The beat builder reads plural `actions`, while the scene worker emits singular `action`. Script validation
therefore receives little or no action evidence and cannot reliably detect invented or omitted actions.
### P0: GPU leases are unsafe for concurrent jobs
When `/session/open` returns 409, the homesrv proxy assumes the active lease is stale and closes it. A second
legitimate job can terminate the first job's model. `heartbeat_session()` exists but is not used by the
pipeline.
### P1: repeated model cold starts
A normal chapter can load Gemma separately for:
1. roster;
2. vision;
3. identity adjudication;
4. reconcile;
5. dialogue;
6. direction;
7. script.
The source itself notes that each load takes tens of seconds to minutes. Removing approximately five of
these loads is likely the largest safe wall-time improvement.
### P1: repeated transfers and encoding
- `/direct/window` downloads every image once for grouping and again for shot design.
- Identity resolver calls repeatedly download the same candidate reference images.
- Each local image is base64-expanded into the JSON sent to the local llama-server.
- Dialogue overlap causes repeated download and CPU set-of-mark work for overlap panels.
### P1: per-item GPU and database work
- SigLIP runs one forward pass per character crop rather than a bounded batch.
- Only one embedding URI is loaded per known character even when its reference gallery has several views.
- Scene construction makes one homesrv-to-workpc HTTP call per panel for a cheap JSON join.
- Many stage loops repeatedly open SQLite connections and commit one row at a time.
## Performance and scheduling plan
### 1. Keep Gemma resident across model phases
Use this chapter schedule:
```text
CPU: fetch -> crop
Gemma: roster -> vision
SigLIP: batched identity embeddings and shortlists
Gemma: identity adjudication -> reconcile -> dialogue -> direct -> CPU scene join -> script
Dots: TTS
Comfy: optional layers
```
This needs pipeline-owned model leases instead of every stage opening and closing its own lease. Scene
construction should run locally on homesrv or as a batch while the second Gemma lease remains open.
ComfyUI must be represented as a GPU resource too. It currently bypasses the session manager, so another
job could load Gemma, SigLIP, or Dots while ComfyUI is using the same GPU.
### 2. Batch SigLIP inference
For each request or chapter batch:
1. download a panel once;
2. clamp and crop all detected characters;
3. preprocess a bounded image batch;
4. run one model forward pass;
5. compare the resulting matrix with the known gallery in one vectorized operation.
Batch size should be tuned against VRAM rather than hard-coded optimistically.
### 3. Add safe bounded CPU parallelism
- Fetch/upload pages with concurrency 4-8.
- Crop framed pages with concurrency 2-4, buffer results, then insert them in page order.
- Keep whole-strip webtoon restitching sequential.
- Pre-download the next vision/dialogue window while Gemma processes the current window.
- Precompute face/text detections with a small CPU worker pool while avoiding ONNX thread oversubscription.
- Batch scene construction and SQLite reads/writes.
GPU calls should remain serialized initially. Multiple llama-server slots divide the configured context
among slots and need an explicit VRAM/context benchmark before enabling concurrent multimodal requests.
### 4. Eliminate repeated image movement
Add an ephemeral `/dev/shm` LRU keyed by S3 URI plus object version/etag. It is a cache, not durable worker
state, and may be dropped at any time.
Reuse downloaded image paths between the two direction passes and cache candidate reference images during
the identity phase. Configure llama-server with `--media-path /dev/shm` and send `file://` image paths
instead of base64 data URIs. The installed llama.cpp version supports local media paths and schema-constrained
responses.
Reference: [llama-server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
### 5. Use JSON-schema-constrained generation
Provide a JSON schema for detect, dialogue, roster, direction, same-person, and identity-resolution calls.
This should eliminate most malformed-output repair calls and make missing fields explicit. Keep semantic
validation after parsing because a structurally valid answer can still be wrong.
### 6. Reduce model call count
- Dialogue context should contain the current scene's recent 12-20 turns, not up to 400 lines from the
whole chapter.
- Reuse direction images first, then A/B test combining grouping and shot selection into one structured call.
- Replace the per-window LLM direction summary with deterministic structured continuity where possible.
- Generate 6-12 consecutive script beats per text-only call, grouped by scene, with stable beat IDs and
targeted per-beat repair.
## Character-recognition plan
The critical path keeps the current detector, SigLIP shortlisting, tracklets, and Gemma resolver.
### 1. Correct the tracklet spine
- Canonicalize gender as `m|f|unknown` and species as a validated enum everywhere.
- Add hard same-panel/co-presence cannot-links.
- Treat manual identity assignments as immutable constraints.
- Prevent transitive bridges from joining tracks whose hard constraints disagree.
- Give every detection a chapter-local provisional track ID, even if it never becomes a global character.
- Move provisional state out of the workpc process and into a resumable homesrv chapter artifact.
### 2. Improve visual evidence
- Validate and clamp all VLM bounding boxes before cropping.
- Match detected faces to person/body boxes with gated IoU and normalized distance assignment rather than
unconditional nearest-neighbour matching.
- Store both face and body crops.
- Score crop quality using face size, detector confidence, occlusion, and sharpness.
- Send Gemma the best 2-4 query views for a tracklet. The current resolver sends only one representative
query crop; its candidate union is not a real multi-view query gallery.
- Store multiple embedding vectors per global character and use gallery maximum or centroid similarity
only for shortlisting.
Face/body evidence plus local temporal constraints are supported by dedicated comic and manga re-ID work:
[Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification](https://arxiv.org/abs/2308.09096)
and [Unsupervised Manga Character Re-identification via Face-body and Spatial-temporal Associated Clustering](https://arxiv.org/abs/2204.04621).
### 3. Separate naming from identity
- Keep roster names as candidate vocabulary, never identity evidence.
- Detection may emit a typed name claim only when the panel visibly binds the name to that character.
- Require self-introduction, name-tag, or corroborated address evidence before promotion.
- Preserve aliases and honorific forms instead of stripping and discarding them.
- Never merge galleries on name equality alone.
- Resolve the tracklet first, then attach corroborated names to its global identity.
### 4. Add global constraints
Identity resolution should enforce:
- must-link membership within a validated tracklet;
- cannot-link for co-present characters;
- gender and species hard gates;
- manual locks;
- at most one appearance of a global character per panel;
- open-set outcomes `known|new|unresolved` without deleting the chapter-local track ID.
Gemma should receive a compact labelled identity board containing several query views and selected candidate
gallery views, rather than up to fifteen separately downloaded image parts for every representative crop.
### 5. Improve identity evaluation
Extend the existing labels and evaluator to report:
- false merges versus false splits;
- tracklet purity and fragmentation;
- global identity accuracy;
- name-binding accuracy;
- unknown/NONE rate;
- speaker accuracy conditioned on identity correctness.
A replacement embedding backbone should only be considered if the structural fixes plateau, using the
same labeled chapters for a controlled A/B comparison.
## Story-coherence plan
### 1. Build an evidence ledger before prose
Create a deterministic artifact for every beat:
```text
beat
-> ordered panels
-> ordered actions and chronology
-> dialogue/region IDs and exact text
-> typed speaker references, confidence, and method
-> characters present
-> location and time
-> named entities
-> uncertainty flags
```
This becomes the only factual input to script generation and validation.
### 2. Make context scene-scoped and evidence-bearing
- Reset speaker-turn context on strong location/time/scene changes.
- Keep a short structured story state rather than repeatedly summarizing generated prose.
- Give a script chunk only facts that occur before or inside that chunk.
- Do not pass a whole-chapter premise as if it were already-known story state; it can leak future events.
- Keep stable chapter-local handles for unresolved recurring characters so they do not disappear from the
scene graph while awaiting global promotion.
### 3. Generate narration in ordered scene chunks
Generate several consecutive beats in one text-only call and return:
```json
{"beats":[{"beat_id":"b01","text":"...","evidence_ids":["p001_r1"]}]}
```
This provides more continuity and fewer calls than one independent request per beat while keeping output
small enough for targeted validation and repair.
### 4. Respect uncertainty
Low-confidence or weak-method speaker assignments should become `someone`, an unattributed quote, or a
review flag instead of a confident named assertion. Identity and speaker uncertainty must survive into
the script prompt.
### 5. Strengthen verification
Validate:
- exact/fuzzy quote grounding;
- permitted character and entity names;
- action coverage and unsupported actions;
- event chronology;
- future-fact leakage;
- repeated facts and repeated sentence openings;
- evidence IDs returned by the model.
Feed the exact failures into the repair prompt. A final text-only consistency audit may report affected
beat IDs, but should not rewrite the whole chapter automatically.
## Stage clearing and resumability
Before parallel or identity work, stage ownership of shared data needs to be explicit:
- dialogue and direction currently mutate the shared vision JSON;
- clearing dialogue does not remove its keys, so a rerun can treat old dialogue as completed;
- reconcile merges are destructive and are not reversed by clearing the reconcile stage;
- clearing identity preserves the per-manga registry, so it is not a clean identity rerun;
- name claims attached to a merged-away character are not repointed with the identity assignments.
Use stage-specific artifacts or explicit key deletion, and preserve provenance through merges. A stage must
be genuinely idempotent before it is scheduled concurrently or resumed automatically.
## Further design ideas
### 1. Make identity merges reversible
Reconciliation currently deletes the losing character. Replace destructive merging with a versioned
cluster or redirect model:
```text
character_cluster
canonical_id
member_ids[]
merge_evidence[]
cannot_link[]
version
```
Assignments resolve through the canonical cluster while the original identities remain recoverable. A
reviewer can remove one bad merge edge without clearing and rebuilding the entire identity stage.
Preserve negative evidence as a first-class artifact. Co-presence is particularly valuable: if A and B
appear together as distinct detections, they are a permanent cannot-link unless a reviewer overrides it.
### 2. Separate detection, identity, and naming confidence
These are different questions and should never share one confidence value:
- detection confidence: is this actually a story character?
- identity confidence: which recurring visual identity is it?
- naming confidence: what is that identity's canonical name?
A person may be confidently detected and tracked as `chapter_person_3` while remaining unnamed. That is
enough for coherent narration such as "the colleague" without contaminating the global registry.
### 3. Use an uncertainty-driven identity cascade
Spend computation according to ambiguity and downstream impact:
```text
hard constraints resolve the case
-> accept cheaply
strong tracklet plus clear gallery margin
-> accept without Gemma
ambiguous candidates
-> multi-view Gemma resolution
high-impact unresolved identity
-> human review
one-off background person
-> chapter-local handle only
```
High-impact detections are recurring, speaking, named, or referenced by several future beats. A one-panel
extra should not consume the same identity budget as a protagonist.
### 4. Add model-aware scheduling across jobs
Strict chapter-by-chapter execution causes extra model swaps when several jobs are queued. A resource-aware
scheduler can group ready work by resident model:
```text
ready Gemma detect phases
-> ready SigLIP phases
-> ready Gemma post-identity phases
-> ready Dots phases
```
This improves total throughput at the cost of some per-job latency. Add fairness and maximum-wait limits so
one long title cannot starve other jobs. GPU conflicts must wait in a queue; they must never terminate the
active owner's lease.
### 5. Cache artifacts by dependency fingerprint
Replace broad "clear stage X and everything downstream" behavior with content-addressed artifacts keyed by:
```text
hash(input URIs
+ relevant upstream JSON
+ model version
+ prompt/schema version
+ stage settings)
```
Consequences:
- changing narration style does not rerun vision;
- correcting one identity invalidates only affected scenes and beats;
- changing direction does not invalidate dialogue;
- a resume reuses every artifact whose dependencies still match;
- prompt experiments can run side-by-side instead of overwriting the baseline.
Store dependency and version metadata beside every stage artifact so invalidation is explainable in the
review UI.
### 6. Detect repeated artwork before expensive stages
Use CPU-side perceptual hashing after masking detected text regions:
- same artwork and same text: reuse visual and dialogue artifacts;
- same artwork but different text: reuse detection/identity, rerun dialogue;
- near-duplicate establishing shot: mark `visual_repeat_of` so narration does not describe it again;
- exact duplicate with no new information: curate it out early.
Text masking is required because two visually identical panels may contain different dialogue. Combine the
art hash with text-region geometry and a text-content hash when available.
### 7. Make the manga registry temporal
Names, aliases, relationships, and facts need an evidence position:
```text
known_from: chapter 12, panel 37
source: self_intro
confidence: 0.97
```
The script can then use only knowledge available at that point. This prevents later chapters or out-of-order
processing from leaking names and revelations into earlier recaps. A configurable `spoiler_safe` versus
`omniscient_recap` policy can decide whether later knowledge may backfill earlier narration.
The sampled chapter premise needs the same guard because a synopsis built from the whole chapter can reveal
its payoff in the opening beat.
### 8. Represent dialogue as a relation graph
A spoken name often identifies the addressee or a third party rather than the speaker. Store:
```text
speaker_ref
addressed_character_ref
mentioned_character_refs[]
spoken_text
name_surface_form
honorific
region_id
speaker_evidence
```
For example, "Seonho-oppa!" is evidence about the addressee and their relationship, not the speaker's name.
Honorifics should be normalized but preserved as evidence instead of being stripped and discarded.
### 9. Treat speaker attribution as constrained assignment
For every speech region, build candidates from:
- bubble-tail or geometric evidence;
- visible faces and bodies;
- previous and following speaker turns;
- chapter-local identities;
- the off-panel cast;
- same-panel cannot-links.
Assign speakers jointly across a conversation window rather than deciding every line independently. Use
Gemma only to adjudicate ambiguous regions. Require `region_id` in the model output so numbered set-of-mark
regions are structurally connected to returned dialogue instead of remaining visual suggestions.
### 10. Separate stable identity traits from scene appearance
Store durable and temporary appearance separately:
```text
identity_appearance:
face, hair, species, stable marks
scene_appearance:
outfit, accessories, injuries, disguise, chibi/age form
```
Clothing can be strong evidence inside one local scene but should be down-weighted across scene or time
changes. This also lets narration mention an outfit change without creating a new global character.
### 11. Add metamorphic pipeline tests
Test invariants rather than only fixed outputs:
- changing dialogue window size must not move text to another panel;
- inserting a scenery panel must not change neighboring identities;
- resuming halfway through identity must match a clean run;
- renumbering panel-local IDs must not change global identity;
- distinct co-present characters must never merge;
- changing narration style must not change facts or quotes;
- processing chapters in a different order must not silently rewrite prior identities.
These tests exercise cross-stage contracts and resumability failures that per-file self-checks cannot catch.
### 12. Roll out identity changes in shadow mode
Write new identity results to a versioned shadow artifact without affecting scripts or the canonical registry.
The review UI can compare:
```text
current: Choi Haeseon
candidate: unresolved
reason: co-presence conflict
```
Promote a new resolver only after it improves the labeled-chapter metrics. Shadow mode prevents tuning work
from polluting the production gallery.
The highest-value additions beyond the core plan are reversible identity clusters with cannot-links,
dependency-fingerprinted artifacts, and uncertainty-driven review for recurring/speaking/named characters.
## Implementation sequence for approval
### Phase 1: correctness and scheduling safety
- Fix enum mismatches and add tracklet cannot-links.
- Gate face/body pairing.
- Wire action evidence and verifier feedback.
- Make stage clearing/resume behavior honest.
- Replace 409 lease stealing with a queue/wait policy.
- Run actual lease heartbeats.
- Register ComfyUI under the same GPU resource scheduler.
Verification: CPU-only unit/self-checks. No full pipeline or GPU run until explicitly scheduled.
### Phase 2: safe throughput improvements
- Introduce the two-phase Gemma lease schedule.
- Add JSON-schema outputs.
- Add local media paths and the ephemeral file/reference cache.
- Reuse direction downloads.
- Batch SigLIP embeddings.
- Batch scene/DB operations.
- Add bounded fetch and framed-page crop concurrency.
Verification: unit checks first, followed by one labeled chapter after GPU availability is approved.
### Phase 3: multi-view constrained identity
- Persist chapter-local tracklets.
- Add face/body galleries and crop-quality selection.
- Resolve with multi-view query evidence and global constraints.
- Separate name claims from visual identity.
- Expand identity evaluation.
### Phase 4: evidence-ledger narration
- Build ordered beat evidence artifacts.
- Add scene-scoped story and turn-taking state.
- Generate script chunks with evidence IDs.
- Add confidence-aware wording and targeted verification/repair.
## Acceptance criteria
Capture a baseline and compare the same labeled chapter after every phase:
- stage wall time and model-load time;
- number of Gemma calls and repair calls;
- MinIO bytes downloaded/uploaded;
- prompt-evaluation and generation timings;
- tracklet false merges/splits and identity accuracy;
- name and speaker accuracy;
- unresolved correctness flags;
- script repetition, unsupported facts, and verifier retry rate.
Recommended approval boundary: implement Phases 1 and 2 first, measure them, then decide whether to proceed
with the larger identity and narration changes.
## Second-pass findings (2026-08-11)
Added after reading the workers and the orchestrator against each other. Every item was read in source and
is cited by file and line. Nothing was executed and no GPU work was run.
The first pass audited each side on its own terms. Most of what follows lives in the seam between them:
one repo changed a field's meaning and the other still reads the old one. The existing per-file
`__main__` self-checks cannot catch any of it, because each one asserts its own side of the contract.
### P0: the scene stage discards every speaker
`normalize_dialogue` rewrites `row["speaker"]` to a `character_id` or `None`
(`orchestrator/correctness.py:47`). `run_stage_dialogue` saves that shape into the vision blob
(`orchestrator/service.py:1159-1160`). `worker_scene.build_scene` still reads that field as a panel-local id
and maps it through `id_by_local` (`worker_scene.py:78`). A `character_id` is never a key in that map, so
every lookup returns `None`.
Every scene graph therefore reaches the script worker with `speaker: None`, and `_render_line` writes
`Someone says "..."` for all of it (`worker_script.py:48`). Set-of-mark face attribution, the solo-speaker
backstop, and window turn-taking all compute the right answer and then lose it one stage later.
The same field breaks the dialogue resume path: `service.py:1180` pushes the stored value into `recent` as a
speaker NAME, so a resumed run feeds raw strings like `character_ab12cd34` to the transcription prompt as if
they were people.
`worker_scene`'s self-check still passes because it feeds the pre-change contract
(`worker_scene.py:138`). Fix: prefer `speaker_ref` when its kind is `character_id`, keep the local-id path as
the fallback. This is the strongest argument for the cross-stage metamorphic tests proposed above.
### P0: the script verifier rejects correct narration, and its retry is a no-op
`verify_script` fails a beat on two rules that fire on valid output (`orchestrator/correctness.py:100-116`).
1. `unsupported-proper-noun` flags any capitalized token outside the cast, the entities, the beat's source
words, and a 20-word stop list. Ordinary sentence-initial words are not in that list: `Suddenly`,
`Behind`, `Inside`, `Both`, `Everyone`, `After`, `Two`. So is `Someone`, which the previous finding
guarantees the narrator will emit constantly.
2. `misquote` compares each quoted span against a WHOLE source line with `SequenceMatcher` at 0.82. The
narration prompt explicitly asks for a short quote of the actual words (`worker_script.py:130-132`). A
15-character quote taken from a 40-character line scores about 0.55 and fails. The prompt and the
verifier ask for opposite things.
The failure is not soft. `run_stage_script` retries once, then raises, so the beat produces no narration
(`service.py:1384-1386`). `done` never reaches `total`, `_finalize_stage` marks the stage failed, and
`_run_pipeline` stops the whole chapter. One false positive halts a run.
The retry is also inert for the reason already recorded above: `ScriptInput` defines neither `beat` nor
`verifier_feedback` (`worker_script.py:13-23`), and pydantic v2 ignores unknown fields by default, so both
are dropped without an error.
Fix order: correct the two rules first, then wire the feedback fields. Fixing the plumbing alone makes the
model retry against a broken oracle.
### P0: correctness flags block TTS permanently in the default configuration
`run_stage_tts` refuses to start while any unresolved flag of kind `script-verifier`, `ambiguous-speaker`,
`conflicting-name-claims`, or `partial-dialogue` exists (`service.py:1419-1423`). The only code that ever
resolves a flag is `resolve_correctness_flags`, called from `/review/approve` for the `script` gate
(`service.py:1955-1956`).
`GATES` defaults to off (`service.py:312`). The autonomous pipeline therefore has no path that clears a
flag. The first flag of those kinds wedges the chapter until a human calls an endpoint that the autonomous
mode never mentions. These flags are not rare: `partial-dialogue` is raised whenever a window response omits
one panel, and `dialogue_envelope` treats a legitimately silent panel inside a partial window as unresolved
(`correctness.py:62`), so one missing panel flags every wordless panel beside it.
There is a second, worse variant. Flag rows survive their panels. `get_correctness_flags` LEFT JOINs panels
and accepts rows where `p.panel_id IS NULL` (`db.py:573`), while `resolve_correctness_flags` can only resolve
flags whose panel still exists (`db.py:579`). Clearing the `crop` stage deletes panel rows
(`db.py:798-799`). Any flag raised against a deleted panel becomes visible to every chapter and can never be
resolved, blocking TTS for all future jobs.
Fix: resolve by flag identity rather than by surviving panel, and give the autonomous path an explicit
policy (auto-resolve below a rank, or fail the stage loudly) instead of an unreachable gate.
### P0: `awaiting_review` is immediately overwritten by `failed`
`run_stage_tts` sets `awaiting_review` and then raises (`service.py:1422-1423`). The raise unwinds into
`_run_pipeline`'s catch-all, which sets the job to `failed` (`service.py:391-393`). The review state the
stage just recorded is gone before anyone can read it, and the operator sees a generic failure rather than a
queue of flags. The gate path at `service.py:337-341` returns instead of raising and does not have this
problem.
### P1: set-of-mark face pairing is enabled by default, contrary to its own documentation
`worker_vision.py:22` states the feature is off by default and should be enabled per title once tuned.
`worker_vision.py:31` reads `SOM_ATTRIBUTION` with a default of `"1"`. It is on.
That matters because of the unconditional pairing already recorded as a P0. The mispairing does not stay
local. `_set_of_mark` writes the wrong name into the legend the model reads (`worker_vision.py:99-104`), the
model attributes a line to that face label, and `_apply_speaker_labels` converts the label back into a
`local_id` and stamps `speaker_method = "som_face"` (`worker_vision.py:117`). A geometric guess is laundered
into the highest-trust provenance value the system has. Gate the pairing before trusting that label, or
default the flag to off as documented.
The pairing is also greedy in face order rather than a joint assignment (`worker_vision.py:42-52`), so the
first face processed can claim a character that fits a later face far better.
### P1: the JSON repair pass can fabricate content
When a response fails to parse, `call_gemma4_json` sends the model its own truncated text and asks for the
JSON it should have been (`worker_vision.py:174-178`). The repair call carries no image. On a response
truncated by `max_tokens`, the model is being asked to complete dialogue it can no longer see. Anything it
adds is invented and is indistinguishable downstream from transcribed text.
Schema-constrained generation, already recommended above, removes most of this path. Until then the repair
pass should re-send the image, or a truncated response should be retried rather than repaired.
### P1: `/review/preview` silently replaces a beat clip
`review_preview` renders one panel alone and calls `save_clip(panel_id, ...)` (`service.py:1902-1904`). Its
docstring calls this harmless because assemble would regenerate it. It does not. `_render_one_beat` returns
early when a clip already exists for the leader (`service.py:1591-1592`). Previewing a beat leader therefore
pins the solo preview into the final video and drops the rest of the beat's panels. `review_retts` gets this
right by calling `delete_clip` first (`service.py:2008`). Preview should write to a scratch key or delete the
clip row afterwards.
### P1: the session manager can orphan a llama-server and hold the GPU
`open_session` claims the slot, releases the lock, then spawns and health-waits for up to 300 seconds
(`session_manager.py:108-119`). A `/session/close` arriving during that window finds `proc = None`, tears
down nothing, and clears `_active`. The process that finishes starting afterwards is unreferenced and keeps
its VRAM until someone kills it by hand. The next `open` spawns a second server on the same port, and
`_health_wait` cannot tell the two apart because it only probes the port
(`session_manager.py:51-60`).
Related asymmetry: `_supervise_once` calls `_start_subprocess` while holding `_lock`
(`session_manager.py:169-181`), so a respawn blocks `/session/active`, `/session/close`, and `/session/open`
for the full health wait. The open path was deliberately written to avoid exactly this.
### P1: nothing limits concurrent jobs
`_background_jobs` accepts any number of pipelines (`service.py:302-303`, `service.py:427-428`). Each opens
its own model sessions. Combined with the 409 lease stealing already recorded, two jobs terminate each
other's models rather than queueing. The queue has to live at the orchestrator as job admission control. A
fix inside `session_proxy` alone still lets two pipelines interleave stages against one GPU.
### P1: two cheap throughput wins the plan does not name
**Order the shared prefix first so llama-server reuses its KV cache.** `/direct/window` sends the same
panel images twice, once for grouping and once for shot design (`worker_vision.py:702-733`). The audit
above frames this as duplicate downloads. The larger cost is the second vision-encoder prefill on the GPU.
llama-server reuses the longest common prompt prefix across requests, and the two calls currently differ in
their first token because the instruction text precedes the images. Putting the images first and the
differing instruction last makes the second pass nearly free on prefill. The same reordering helps any
stage that issues several calls over one image set.
**Give stored embeddings a version tag.** `embed_crop` uses `pooler_output` when present and silently falls
back to mean-pooled patch tokens otherwise (`worker_identity.py:48-52`). Those two paths produce different
vector spaces, and the fixed 0.85 threshold is only valid for one of them. Nothing recorded beside a stored
`.npy` says which model, revision, or pooling produced it, so a transformers upgrade mixes incompatible
vectors into one gallery with no error. Write the model id and pooling mode next to the vector and refuse to
compare across versions.
### P2: smaller confirmed defects
- An out-of-range `choice` from the resolver is mapped to NONE and then reported as `state: "new"`
(`worker_vision.py:898-899`), so a hallucinated index mints a brand new character. It should be
`unresolved`, like a parse failure.
- `_extract_json` matches greedily from the first `{` to the last `}` (`worker_vision.py:130`). Two objects
or any trailing braced prose produce an unparseable span and burn a repair call. `json.JSONDecoder().raw_decode`
from the first brace is exact.
- `worker_identity._known_cache` is invalidated only by `_persist_char` (`worker_identity.py:128-132`).
The reconcile stage deletes losing characters directly in the orchestrator database
(`db.py:506`), so a long-lived worker keeps shortlisting and assigning ids that no longer exist.
- `_pending` holds full crop images in worker memory for a whole chapter and is keyed by session
(`worker_identity.py:25`). This is durable per-chapter state inside a worker documented as stateless, it
is lost on restart, and a character seen exactly once receives no assignment at all, not even a
chapter-local handle.
- `get_conn` opens a connection per call with no `busy_timeout` (`db.py:190-197`). WAL tolerates one writer.
`PIPELINE=1` already writes clips from concurrent tasks while TTS writes audio, so the planned CPU
parallelism will surface as `database is locked` before it surfaces as throughput.
- Worker endpoints declared `async def` run blocking MinIO, OpenCV, torch, and ffmpeg calls directly on the
event loop (`worker_identity.py:177`, `worker_tts.py:170`, `worker_vision.py:230`). A busy worker cannot
answer `/health` or `/unload`. That makes `/health/workers` report a working worker as unreachable
(`service.py:206-216`) and puts the session manager's 30-second `/unload` at risk exactly when VRAM needs
freeing (`session_manager.py:93`). `def` instead of `async def` moves each to the threadpool.
- `layers` runs after `tts` in `STAGES` (`db.py:330-333`), while `run_stage_tts` warns that eager rendering
under `PIPELINE=1` needs layers to run first (`service.py:1427-1429`). With the flag on, solo beats always
render without parallax.
- Stage failure policy is inconsistent. A dropped vision panel fails the stage and halts the pipeline. A
failed direction window counts its panels as done (`service.py:1247`), and layers always finishes
completed (`service.py:1503`). `completed` does not mean the same thing across stages, which makes the
acceptance metrics below hard to read.
- `run_stage_assemble` does not check that `clip_uris` is non-empty before assembling and then marks the job
completed (`service.py:1661-1677`).
- The `/review/panels` timeline sums per-panel audio durations (`service.py:1750`), but assemble crossfades
clips using the per-beat transitions (`service.py:1671-1674`). Every non-`cut` transition shortens the
real video, so reviewer timestamps drift further out of sync the further into the chapter they scrub.
- MinIO credentials are hardcoded as defaults in committed source (`transport.py:95-99`,
`service.py:74-76`).
### What this changes in the plan
Add to Phase 1, before any throughput work:
1. Restore the speaker field across the dialogue, scene, and script boundary.
2. Correct the two verifier rules, then wire `beat` and `verifier_feedback`.
3. Give correctness flags a resolution path that does not require a disabled gate, and stop the TTS block
from erasing `awaiting_review`.
4. Decide the set-of-mark default deliberately, and gate face pairing before the label is trusted.
5. Add job admission control at the orchestrator, not only lease queueing in the proxy.
Add to the acceptance criteria: the share of narrated lines whose speaker is a named character rather than
`Someone`. That single number would have caught the first finding on the day it landed.
HANDOFF.md
+39 -60
View File
@@ -1,75 +1,54 @@
# HANDOFF: repo reconstructed from agent transcripts (2026-08-11) # HANDOFF: audit second pass (2026-08-11)
## What was asked ## What was asked
The working copy at `/home/kami/Programs/n8n-worker` was deleted with `rm`, including `.git`. Review `AUDIT.md` (untracked, first pass, 565 lines) and add anything missing.
This repo is the workpc GPU-compute half of the manga pipeline. The ask was to rebuild it from
the transcripts on disk, at minimum the transport layer and the pipeline flow.
## Sources used ## What was done
No backup and no remote survived. `~/.local/share/Trash` is empty. No copy exists under Read both halves of the pipeline and checked the first pass against source. No code changed.
`/home/kami`, `/mnt/D`, or `/mnt/server`. Gitea No GPU work, no pipeline run, no tests executed.
(`/mnt/server/mnt/hdd2/gitea/git/repositories/`) holds only correx, hexis, maven,
model-training, muzick, orchestra, and test-e2e. There is no `n8n-worker` mirror.
Reconstruction replays three histories into one timeline, ordered by timestamp: Files read in full:
- workpc: `worker_vision.py` (1008), `worker_identity.py` (322), `worker_script.py` (300),
`worker_scene.py` (170), `worker_tts.py` (241), `session_manager.py` (238), `transport.py` (216),
`worker_crop.py` (first 120 of 359).
- homesrv `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`: `service.py` (2048),
`db.py` (815), `session_proxy.py` (302), `correctness.py` (116), `tracklets.py` (121),
`minio_layout.py` (105), `grouping.py` (75).
1. Claude Code transcripts, 25 sessions in Result: `AUDIT.md` grew from 565 to 745 lines. New section `## Second-pass findings (2026-08-11)`
`~/.claude/projects/-home-kami-Programs-n8n-worker/*.jsonl`. Contributes Write and Edit at line 566. 4 new P0, 6 new P1, 13 P2, plus 5 additions to the Phase 1 list and 1 to the
content, full-file Read results, and `@`-mention attachments. acceptance criteria.
2. Codex rollouts, 5 sessions with `cwd=/home/kami/Programs/n8n-worker` under
`~/.codex/sessions/2026/07/{13,15,16,18}/rollout-*.jsonl`. Contributes 26 `apply_patch`
blocks. The 4 that the log marks `Script failed` are skipped. These carry the 2026-07-18
correctness work (`_dialogue_envelope`, `_normalize_claims`, `_annotate_speaker_methods`,
`SOM_ATTRIBUTION=1`) that exists in no Claude transcript.
3. `~/.claude/file-history/<session>/<hash>@vN` pre-edit blobs, used as cross-checks only.
Scripts live in Four first-pass claims were confirmed in source and are not restated in the new section:
`/tmp/claude-1000/-home-kami-Programs-n8n-worker/6a35d5d8-9f44-4412-955c-9cf088737ad5/scratchpad/`.
`replay2.py` is the merged replay. `codex.py` extracts and applies codex patches. `backups.py`
compares against file-history. `recon_final/_log.json` holds the per-file event log.
## Verification - tracklet gender enum: `tracklets.py:43` uses `male|female`, vision emits `m|f|unknown`.
- Replayed to 2026-07-18T13:13:44Z, line counts match the `wc -l` recorded in the transcript at - missing action evidence: `worker_scene.py:85` emits `action`, `correctness.py:91` reads `actions`.
that moment, for all 10 files it covered. crop 326, identity 314, layers 99, render 996, - dropped verifier feedback: `worker_script.py:13-23` defines neither field.
scene 159, script 300, tts 231, vision 871, session_manager 238, transport 216. - 409 lease stealing: `session_proxy.py:40-49`.
- Byte sizes match the `ls -l` recorded by the same command. AGENTS.md 867, CLAUDE.md 3678,
manga-recap-pipeline-spec.md 11413, plan.md 16511, spec-v3.md 26170. `.gitignore` is 53.
- 18 files are byte-identical to their newest file-history blob. The 4 that differ (render,
script, tts, spec-v3) differ only by edits made after that blob was taken.
- `python3 -m py_compile *.py scripts/*.py attic/*.py` passes. `bash -n` passes on all 3 shell
scripts. Runtime self-checks such as `python worker_vision.py` were NOT run, because `.venv/`
is gone.
## Recovered (live tree) ## Top 4 new findings
Workers `worker_{crop,vision,identity,scene,script,tts,layers,render}.py`, the GPU mutex 1. `worker_scene.py:78` reads `speaker` as a local id, but `correctness.py:47` already rewrote it
`session_manager.py`, `transport.py`, helpers `bubble_detect.py`, `face_detect.py`, to a character_id. Every lookup returns None, so all narration says `Someone`.
`collage.py`, `test_vision_parse.py`, `start_workers.sh`, `systemd/install.sh`, 2. `correctness.py:106-115` fails valid narration on sentence-initial capitals and on short
`scripts/{analyze_video_frames.sh,pick_tts_voice.py}`, `requirements.txt`, `.gitignore`, quotes, and `service.py:1384-1386` turns that into a halted chapter.
and the docs `AGENTS.md`, `CLAUDE.md`, `plan.md`, `spec-v3.md`, `spec-correctness.md`, 3. `service.py:1419-1423` blocks TTS on correctness flags that only `/review/approve` can clear,
`manga-recap-pipeline-spec.md`, `identity-rework-task.md`. while `GATES` defaults to off (`service.py:312`).
4. `session_manager.py:108-119` can orphan a llama-server that keeps its VRAM.
`attic/` holds files that had been deleted from the tree before the `rm`. They are kept but not
live: `worker_ocr.py`, `worker_parse.py`, `plan-workpc.md`, `char-recognition.md`.
## Still open ## Still open
- `.venv/` is gone. Rebuild with - Nothing from this session is half-finished. `AUDIT.md` is complete as written.
`python -m venv .venv && .venv/bin/pip install -r requirements.txt`. The ROCm torch wheel is - No finding has been fixed. All are report-only.
not pinned in requirements.txt, so install it the way workpc had it. - `AUDIT.md` is still untracked as of this session's start. Decide whether it belongs in git.
- `dots.tts/` (gitignored external checkout) and `legacy/` (pre-rewrite workers, moved there - Prose linter reports ~97 style hits in `AUDIT.md`. Almost all are in the first-pass sections
2026-07-13) are not recoverable from transcripts. Re-clone dots.tts if TTS is needed. (lines 1-565), which were left as the author wrote them.
- `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable. Neither agent - Carried over from the earlier reconstruction handoff of the same day:
ever read them in full. All three were already deleted from the working tree as of 2026-07-17. - `.venv/` is gone. Rebuild it, then install the ROCm torch wheel the way workpc had it.
- Git history is gone. A fresh `git init` plus one commit replaces it. Old commits are not - `dots.tts/` and `legacy/` are not recoverable from transcripts.
recoverable. - `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable.
- Two replay gaps were left unpatched. One Edit MISS in `worker_render.py` on 2026-07-14, and one
orphan Edit against `attic/plan-workpc.md` on 2026-07-13. Both files were later replaced by a
full-file snapshot, so the final content stays anchored.
- Edits made outside Claude and codex after 2026-07-18T14:35Z (the last recorded write) cannot be
detected. `unknown:` whether any exist.
## Next command ## Next command
``` ```
cd /home/kami/Programs/n8n-worker cd /home/kami/Programs/n8n-worker
python -m venv .venv && .venv/bin/pip install -r requirements.txt sed -n '566,745p' AUDIT.md # read the new section
.venv/bin/python worker_vision.py # per-file __main__ self-checks, then ./start_workers.sh
``` ```
Then pick the Phase 1 order at `AUDIT.md:511` as amended at the end of the new section.