diff --git a/AUDIT.md b/AUDIT.md new file mode 100644 index 0000000..e5a78b1 --- /dev/null +++ b/AUDIT.md @@ -0,0 +1,770 @@ +# Manga recap pipeline audit + +Date: 2026-08-11 + +Scope: workpc workers plus the homesrv orchestrator. This audit excludes FFmpeg changes and does not +recommend Magi or its embeddings. No GPU workloads were run during the audit. + +## Outcome + +The strongest performance gains are available without replacing the current models: + +1. Keep Gemma resident across compatible stages, reducing roughly seven chapter-level Gemma loads to two. +2. Batch SigLIP crop embeddings instead of running one forward pass per detected character. +3. Add bounded parallelism to CPU/network work such as page uploads, framed-page cropping, prefetching, + and scene construction. +4. Stop repeatedly downloading and base64-encoding the same panels and character references. +5. Generate schema-constrained JSON and batch text-only script work by scene. + +Correctness fixes should land before throughput work because the current tracklet and naming paths can +contaminate the permanent character registry. + +## Highest-priority findings + +### P0: tracklet gender gate uses the wrong enum + +`orchestrator/tracklets.py` checks `male|female`, while vision emits `m|f|unknown`. The tracklet hard +gender gate therefore never activates on normal pipeline data. + +### P0: tracklets lack co-presence constraints + +The linker has no same-panel exclusion and uses transitive union-find. Two similar-looking people in the +same panel can be joined, and weak links such as A~B and B~C can merge A and C even when the endpoints are +incompatible. + +### P0: face-to-character pairing is unconditional + +`worker_vision._pair_faces_to_present()` assigns every detected face to the nearest vision character as +long as an unused character exists. Despite the docstring, there is no distance or overlap threshold. +A distant or unrelated face can therefore receive another character's identity and speaker label. + +### P0: roster hints can become permanent identity evidence + +The roster is described as hints-only, but those names are fed into detection. Any name emitted on a crop +causes `worker_identity` to persist it immediately. A coarse appearance-to-roster guess can therefore +contaminate a permanent character gallery. + +### P0: script repair feedback is discarded + +The orchestrator sends `beat` and `verifier_feedback` on a failed-script retry, but `worker_script.ScriptInput` +does not define either field and the prompt builder does not consume them. The second call is another +stochastic attempt rather than a targeted correction. + +### P0: action evidence is missing from the verifier + +The beat builder reads plural `actions`, while the scene worker emits singular `action`. Script validation +therefore receives little or no action evidence and cannot reliably detect invented or omitted actions. + +### P0: GPU leases are unsafe for concurrent jobs + +When `/session/open` returns 409, the homesrv proxy assumes the active lease is stale and closes it. A second +legitimate job can terminate the first job's model. `heartbeat_session()` exists but is not used by the +pipeline. + +### P1: repeated model cold starts + +A normal chapter can load Gemma separately for: + +1. roster; +2. vision; +3. identity adjudication; +4. reconcile; +5. dialogue; +6. direction; +7. script. + +The source itself notes that each load takes tens of seconds to minutes. Removing approximately five of +these loads is likely the largest safe wall-time improvement. + +### P1: repeated transfers and encoding + +- `/direct/window` downloads every image once for grouping and again for shot design. +- Identity resolver calls repeatedly download the same candidate reference images. +- Each local image is base64-expanded into the JSON sent to the local llama-server. +- Dialogue overlap causes repeated download and CPU set-of-mark work for overlap panels. + +### P1: per-item GPU and database work + +- SigLIP runs one forward pass per character crop rather than a bounded batch. +- Only one embedding URI is loaded per known character even when its reference gallery has several views. +- Scene construction makes one homesrv-to-workpc HTTP call per panel for a cheap JSON join. +- Many stage loops repeatedly open SQLite connections and commit one row at a time. + +## Performance and scheduling plan + +### 1. Keep Gemma resident across model phases + +Use this chapter schedule: + +```text +CPU: fetch -> crop +Gemma: roster -> vision +SigLIP: batched identity embeddings and shortlists +Gemma: identity adjudication -> reconcile -> dialogue -> direct -> CPU scene join -> script +Dots: TTS +Comfy: optional layers +``` + +This needs pipeline-owned model leases instead of every stage opening and closing its own lease. Scene +construction should run locally on homesrv or as a batch while the second Gemma lease remains open. + +ComfyUI must be represented as a GPU resource too. It currently bypasses the session manager, so another +job could load Gemma, SigLIP, or Dots while ComfyUI is using the same GPU. + +### 2. Batch SigLIP inference + +For each request or chapter batch: + +1. download a panel once; +2. clamp and crop all detected characters; +3. preprocess a bounded image batch; +4. run one model forward pass; +5. compare the resulting matrix with the known gallery in one vectorized operation. + +Batch size should be tuned against VRAM rather than hard-coded optimistically. + +### 3. Add safe bounded CPU parallelism + +- Fetch/upload pages with concurrency 4-8. +- Crop framed pages with concurrency 2-4, buffer results, then insert them in page order. +- Keep whole-strip webtoon restitching sequential. +- Pre-download the next vision/dialogue window while Gemma processes the current window. +- Precompute face/text detections with a small CPU worker pool while avoiding ONNX thread oversubscription. +- Batch scene construction and SQLite reads/writes. + +GPU calls should remain serialized initially. Multiple llama-server slots divide the configured context +among slots and need an explicit VRAM/context benchmark before enabling concurrent multimodal requests. + +### 4. Eliminate repeated image movement + +Add an ephemeral `/dev/shm` LRU keyed by S3 URI plus object version/etag. It is a cache, not durable worker +state, and may be dropped at any time. + +Reuse downloaded image paths between the two direction passes and cache candidate reference images during +the identity phase. Configure llama-server with `--media-path /dev/shm` and send `file://` image paths +instead of base64 data URIs. The installed llama.cpp version supports local media paths and schema-constrained +responses. + +Reference: [llama-server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md). + +### 5. Use JSON-schema-constrained generation + +Provide a JSON schema for detect, dialogue, roster, direction, same-person, and identity-resolution calls. +This should eliminate most malformed-output repair calls and make missing fields explicit. Keep semantic +validation after parsing because a structurally valid answer can still be wrong. + +### 6. Reduce model call count + +- Dialogue context should contain the current scene's recent 12-20 turns, not up to 400 lines from the + whole chapter. +- Reuse direction images first, then A/B test combining grouping and shot selection into one structured call. +- Replace the per-window LLM direction summary with deterministic structured continuity where possible. +- Generate 6-12 consecutive script beats per text-only call, grouped by scene, with stable beat IDs and + targeted per-beat repair. + +## Character-recognition plan + +The critical path keeps the current detector, SigLIP shortlisting, tracklets, and Gemma resolver. + +### 1. Correct the tracklet spine + +- Canonicalize gender as `m|f|unknown` and species as a validated enum everywhere. +- Add hard same-panel/co-presence cannot-links. +- Treat manual identity assignments as immutable constraints. +- Prevent transitive bridges from joining tracks whose hard constraints disagree. +- Give every detection a chapter-local provisional track ID, even if it never becomes a global character. +- Move provisional state out of the workpc process and into a resumable homesrv chapter artifact. + +### 2. Improve visual evidence + +- Validate and clamp all VLM bounding boxes before cropping. +- Match detected faces to person/body boxes with gated IoU and normalized distance assignment rather than + unconditional nearest-neighbour matching. +- Store both face and body crops. +- Score crop quality using face size, detector confidence, occlusion, and sharpness. +- Send Gemma the best 2-4 query views for a tracklet. The current resolver sends only one representative + query crop; its candidate union is not a real multi-view query gallery. +- Store multiple embedding vectors per global character and use gallery maximum or centroid similarity + only for shortlisting. + +Face/body evidence plus local temporal constraints are supported by dedicated comic and manga re-ID work: +[Identity-Aware Semi-Supervised Learning for Comic Character Re-Identification](https://arxiv.org/abs/2308.09096) +and [Unsupervised Manga Character Re-identification via Face-body and Spatial-temporal Associated Clustering](https://arxiv.org/abs/2204.04621). + +### 3. Separate naming from identity + +- Keep roster names as candidate vocabulary, never identity evidence. +- Detection may emit a typed name claim only when the panel visibly binds the name to that character. +- Require self-introduction, name-tag, or corroborated address evidence before promotion. +- Preserve aliases and honorific forms instead of stripping and discarding them. +- Never merge galleries on name equality alone. +- Resolve the tracklet first, then attach corroborated names to its global identity. + +### 4. Add global constraints + +Identity resolution should enforce: + +- must-link membership within a validated tracklet; +- cannot-link for co-present characters; +- gender and species hard gates; +- manual locks; +- at most one appearance of a global character per panel; +- open-set outcomes `known|new|unresolved` without deleting the chapter-local track ID. + +Gemma should receive a compact labelled identity board containing several query views and selected candidate +gallery views, rather than up to fifteen separately downloaded image parts for every representative crop. + +### 5. Improve identity evaluation + +Extend the existing labels and evaluator to report: + +- false merges versus false splits; +- tracklet purity and fragmentation; +- global identity accuracy; +- name-binding accuracy; +- unknown/NONE rate; +- speaker accuracy conditioned on identity correctness. + +A replacement embedding backbone should only be considered if the structural fixes plateau, using the +same labeled chapters for a controlled A/B comparison. + +## Story-coherence plan + +### 1. Build an evidence ledger before prose + +Create a deterministic artifact for every beat: + +```text +beat + -> ordered panels + -> ordered actions and chronology + -> dialogue/region IDs and exact text + -> typed speaker references, confidence, and method + -> characters present + -> location and time + -> named entities + -> uncertainty flags +``` + +This becomes the only factual input to script generation and validation. + +### 2. Make context scene-scoped and evidence-bearing + +- Reset speaker-turn context on strong location/time/scene changes. +- Keep a short structured story state rather than repeatedly summarizing generated prose. +- Give a script chunk only facts that occur before or inside that chunk. +- Do not pass a whole-chapter premise as if it were already-known story state; it can leak future events. +- Keep stable chapter-local handles for unresolved recurring characters so they do not disappear from the + scene graph while awaiting global promotion. + +### 3. Generate narration in ordered scene chunks + +Generate several consecutive beats in one text-only call and return: + +```json +{"beats":[{"beat_id":"b01","text":"...","evidence_ids":["p001_r1"]}]} +``` + +This provides more continuity and fewer calls than one independent request per beat while keeping output +small enough for targeted validation and repair. + +### 4. Respect uncertainty + +Low-confidence or weak-method speaker assignments should become `someone`, an unattributed quote, or a +review flag instead of a confident named assertion. Identity and speaker uncertainty must survive into +the script prompt. + +### 5. Strengthen verification + +Validate: + +- exact/fuzzy quote grounding; +- permitted character and entity names; +- action coverage and unsupported actions; +- event chronology; +- future-fact leakage; +- repeated facts and repeated sentence openings; +- evidence IDs returned by the model. + +Feed the exact failures into the repair prompt. A final text-only consistency audit may report affected +beat IDs, but should not rewrite the whole chapter automatically. + +## Stage clearing and resumability + +Before parallel or identity work, stage ownership of shared data needs to be explicit: + +- dialogue and direction currently mutate the shared vision JSON; +- clearing dialogue does not remove its keys, so a rerun can treat old dialogue as completed; +- reconcile merges are destructive and are not reversed by clearing the reconcile stage; +- clearing identity preserves the per-manga registry, so it is not a clean identity rerun; +- name claims attached to a merged-away character are not repointed with the identity assignments. + +Use stage-specific artifacts or explicit key deletion, and preserve provenance through merges. A stage must +be genuinely idempotent before it is scheduled concurrently or resumed automatically. + +## Further design ideas + +### 1. Make identity merges reversible + +Reconciliation currently deletes the losing character. Replace destructive merging with a versioned +cluster or redirect model: + +```text +character_cluster + canonical_id + member_ids[] + merge_evidence[] + cannot_link[] + version +``` + +Assignments resolve through the canonical cluster while the original identities remain recoverable. A +reviewer can remove one bad merge edge without clearing and rebuilding the entire identity stage. + +Preserve negative evidence as a first-class artifact. Co-presence is particularly valuable: if A and B +appear together as distinct detections, they are a permanent cannot-link unless a reviewer overrides it. + +### 2. Separate detection, identity, and naming confidence + +These are different questions and should never share one confidence value: + +- detection confidence: is this actually a story character? +- identity confidence: which recurring visual identity is it? +- naming confidence: what is that identity's canonical name? + +A person may be confidently detected and tracked as `chapter_person_3` while remaining unnamed. That is +enough for coherent narration such as "the colleague" without contaminating the global registry. + +### 3. Use an uncertainty-driven identity cascade + +Spend computation according to ambiguity and downstream impact: + +```text +hard constraints resolve the case + -> accept cheaply + +strong tracklet plus clear gallery margin + -> accept without Gemma + +ambiguous candidates + -> multi-view Gemma resolution + +high-impact unresolved identity + -> human review + +one-off background person + -> chapter-local handle only +``` + +High-impact detections are recurring, speaking, named, or referenced by several future beats. A one-panel +extra should not consume the same identity budget as a protagonist. + +### 4. Add model-aware scheduling across jobs + +Strict chapter-by-chapter execution causes extra model swaps when several jobs are queued. A resource-aware +scheduler can group ready work by resident model: + +```text +ready Gemma detect phases +-> ready SigLIP phases +-> ready Gemma post-identity phases +-> ready Dots phases +``` + +This improves total throughput at the cost of some per-job latency. Add fairness and maximum-wait limits so +one long title cannot starve other jobs. GPU conflicts must wait in a queue; they must never terminate the +active owner's lease. + +### 5. Cache artifacts by dependency fingerprint + +Replace broad "clear stage X and everything downstream" behavior with content-addressed artifacts keyed by: + +```text +hash(input URIs + + relevant upstream JSON + + model version + + prompt/schema version + + stage settings) +``` + +Consequences: + +- changing narration style does not rerun vision; +- correcting one identity invalidates only affected scenes and beats; +- changing direction does not invalidate dialogue; +- a resume reuses every artifact whose dependencies still match; +- prompt experiments can run side-by-side instead of overwriting the baseline. + +Store dependency and version metadata beside every stage artifact so invalidation is explainable in the +review UI. + +### 6. Detect repeated artwork before expensive stages + +Use CPU-side perceptual hashing after masking detected text regions: + +- same artwork and same text: reuse visual and dialogue artifacts; +- same artwork but different text: reuse detection/identity, rerun dialogue; +- near-duplicate establishing shot: mark `visual_repeat_of` so narration does not describe it again; +- exact duplicate with no new information: curate it out early. + +Text masking is required because two visually identical panels may contain different dialogue. Combine the +art hash with text-region geometry and a text-content hash when available. + +### 7. Make the manga registry temporal + +Names, aliases, relationships, and facts need an evidence position: + +```text +known_from: chapter 12, panel 37 +source: self_intro +confidence: 0.97 +``` + +The script can then use only knowledge available at that point. This prevents later chapters or out-of-order +processing from leaking names and revelations into earlier recaps. A configurable `spoiler_safe` versus +`omniscient_recap` policy can decide whether later knowledge may backfill earlier narration. + +The sampled chapter premise needs the same guard because a synopsis built from the whole chapter can reveal +its payoff in the opening beat. + +### 8. Represent dialogue as a relation graph + +A spoken name often identifies the addressee or a third party rather than the speaker. Store: + +```text +speaker_ref +addressed_character_ref +mentioned_character_refs[] +spoken_text +name_surface_form +honorific +region_id +speaker_evidence +``` + +For example, "Seonho-oppa!" is evidence about the addressee and their relationship, not the speaker's name. +Honorifics should be normalized but preserved as evidence instead of being stripped and discarded. + +### 9. Treat speaker attribution as constrained assignment + +For every speech region, build candidates from: + +- bubble-tail or geometric evidence; +- visible faces and bodies; +- previous and following speaker turns; +- chapter-local identities; +- the off-panel cast; +- same-panel cannot-links. + +Assign speakers jointly across a conversation window rather than deciding every line independently. Use +Gemma only to adjudicate ambiguous regions. Require `region_id` in the model output so numbered set-of-mark +regions are structurally connected to returned dialogue instead of remaining visual suggestions. + +### 10. Separate stable identity traits from scene appearance + +Store durable and temporary appearance separately: + +```text +identity_appearance: + face, hair, species, stable marks + +scene_appearance: + outfit, accessories, injuries, disguise, chibi/age form +``` + +Clothing can be strong evidence inside one local scene but should be down-weighted across scene or time +changes. This also lets narration mention an outfit change without creating a new global character. + +### 11. Add metamorphic pipeline tests + +Test invariants rather than only fixed outputs: + +- changing dialogue window size must not move text to another panel; +- inserting a scenery panel must not change neighboring identities; +- resuming halfway through identity must match a clean run; +- renumbering panel-local IDs must not change global identity; +- distinct co-present characters must never merge; +- changing narration style must not change facts or quotes; +- processing chapters in a different order must not silently rewrite prior identities. + +These tests exercise cross-stage contracts and resumability failures that per-file self-checks cannot catch. + +### 12. Roll out identity changes in shadow mode + +Write new identity results to a versioned shadow artifact without affecting scripts or the canonical registry. +The review UI can compare: + +```text +current: Choi Haeseon +candidate: unresolved +reason: co-presence conflict +``` + +Promote a new resolver only after it improves the labeled-chapter metrics. Shadow mode prevents tuning work +from polluting the production gallery. + +The highest-value additions beyond the core plan are reversible identity clusters with cannot-links, +dependency-fingerprinted artifacts, and uncertainty-driven review for recurring/speaking/named characters. + +## Implementation sequence for approval + +### Phase 1: correctness and scheduling safety + +- Fix enum mismatches and add tracklet cannot-links. +- Gate face/body pairing. +- Wire action evidence and verifier feedback. +- Make stage clearing/resume behavior honest. +- Replace 409 lease stealing with a queue/wait policy. +- Run actual lease heartbeats. +- Register ComfyUI under the same GPU resource scheduler. + +Verification: CPU-only unit/self-checks. No full pipeline or GPU run until explicitly scheduled. + +### Phase 2: safe throughput improvements + +- Introduce the two-phase Gemma lease schedule. +- Add JSON-schema outputs. +- Add local media paths and the ephemeral file/reference cache. +- Reuse direction downloads. +- Batch SigLIP embeddings. +- Batch scene/DB operations. +- Add bounded fetch and framed-page crop concurrency. + +Verification: unit checks first, followed by one labeled chapter after GPU availability is approved. + +### Phase 3: multi-view constrained identity + +- Persist chapter-local tracklets. +- Add face/body galleries and crop-quality selection. +- Resolve with multi-view query evidence and global constraints. +- Separate name claims from visual identity. +- Expand identity evaluation. + +### Phase 4: evidence-ledger narration + +- Build ordered beat evidence artifacts. +- Add scene-scoped story and turn-taking state. +- Generate script chunks with evidence IDs. +- Add confidence-aware wording and targeted verification/repair. + +## Acceptance criteria + +Capture a baseline and compare the same labeled chapter after every phase: + +- stage wall time and model-load time; +- number of Gemma calls and repair calls; +- MinIO bytes downloaded/uploaded; +- prompt-evaluation and generation timings; +- tracklet false merges/splits and identity accuracy; +- name and speaker accuracy; +- unresolved correctness flags; +- script repetition, unsupported facts, and verifier retry rate. + +Recommended approval boundary: implement Phases 1 and 2 first, measure them, then decide whether to proceed +with the larger identity and narration changes. + +## Second-pass findings (2026-08-11) + +Added after reading the workers and the orchestrator against each other. Every item was read in source and +is cited by file and line. Nothing was executed and no GPU work was run. + +The first pass audited each side on its own terms. Most of what follows lives in the seam between them: +one repo changed a field's meaning and the other still reads the old one. The existing per-file +`__main__` self-checks cannot catch any of it, because each one asserts its own side of the contract. + +### P0: the scene stage discards every speaker + +`normalize_dialogue` rewrites `row["speaker"]` to a `character_id` or `None` +(`orchestrator/correctness.py:47`). `run_stage_dialogue` saves that shape into the vision blob +(`orchestrator/service.py:1159-1160`). `worker_scene.build_scene` still reads that field as a panel-local id +and maps it through `id_by_local` (`worker_scene.py:78`). A `character_id` is never a key in that map, so +every lookup returns `None`. + +Every scene graph therefore reaches the script worker with `speaker: None`, and `_render_line` writes +`Someone says "..."` for all of it (`worker_script.py:48`). Set-of-mark face attribution, the solo-speaker +backstop, and window turn-taking all compute the right answer and then lose it one stage later. + +The same field breaks the dialogue resume path: `service.py:1180` pushes the stored value into `recent` as a +speaker NAME, so a resumed run feeds raw strings like `character_ab12cd34` to the transcription prompt as if +they were people. + +`worker_scene`'s self-check still passes because it feeds the pre-change contract +(`worker_scene.py:138`). Fix: prefer `speaker_ref` when its kind is `character_id`, keep the local-id path as +the fallback. This is the strongest argument for the cross-stage metamorphic tests proposed above. + +### P0: the script verifier rejects correct narration, and its retry is a no-op + +`verify_script` fails a beat on two rules that fire on valid output (`orchestrator/correctness.py:100-116`). + +1. `unsupported-proper-noun` flags any capitalized token outside the cast, the entities, the beat's source + words, and a 20-word stop list. Ordinary sentence-initial words are not in that list: `Suddenly`, + `Behind`, `Inside`, `Both`, `Everyone`, `After`, `Two`. So is `Someone`, which the previous finding + guarantees the narrator will emit constantly. +2. `misquote` compares each quoted span against a WHOLE source line with `SequenceMatcher` at 0.82. The + narration prompt explicitly asks for a short quote of the actual words (`worker_script.py:130-132`). A + 15-character quote taken from a 40-character line scores about 0.55 and fails. The prompt and the + verifier ask for opposite things. + +The failure is not soft. `run_stage_script` retries once, then raises, so the beat produces no narration +(`service.py:1384-1386`). `done` never reaches `total`, `_finalize_stage` marks the stage failed, and +`_run_pipeline` stops the whole chapter. One false positive halts a run. + +The retry is also inert for the reason already recorded above: `ScriptInput` defines neither `beat` nor +`verifier_feedback` (`worker_script.py:13-23`), and pydantic v2 ignores unknown fields by default, so both +are dropped without an error. + +Fix order: correct the two rules first, then wire the feedback fields. Fixing the plumbing alone makes the +model retry against a broken oracle. + +### P0: correctness flags block TTS permanently in the default configuration + +`run_stage_tts` refuses to start while any unresolved flag of kind `script-verifier`, `ambiguous-speaker`, +`conflicting-name-claims`, or `partial-dialogue` exists (`service.py:1419-1423`). The only code that ever +resolves a flag is `resolve_correctness_flags`, called from `/review/approve` for the `script` gate +(`service.py:1955-1956`). + +`GATES` defaults to off (`service.py:312`). The autonomous pipeline therefore has no path that clears a +flag. The first flag of those kinds wedges the chapter until a human calls an endpoint that the autonomous +mode never mentions. These flags are not rare: `partial-dialogue` is raised whenever a window response omits +one panel, and `dialogue_envelope` treats a legitimately silent panel inside a partial window as unresolved +(`correctness.py:62`), so one missing panel flags every wordless panel beside it. + +There is a second, worse variant. Flag rows survive their panels. `get_correctness_flags` LEFT JOINs panels +and accepts rows where `p.panel_id IS NULL` (`db.py:573`), while `resolve_correctness_flags` can only resolve +flags whose panel still exists (`db.py:579`). Clearing the `crop` stage deletes panel rows +(`db.py:798-799`). Any flag raised against a deleted panel becomes visible to every chapter and can never be +resolved, blocking TTS for all future jobs. + +Fix: resolve by flag identity rather than by surviving panel, and give the autonomous path an explicit +policy (auto-resolve below a rank, or fail the stage loudly) instead of an unreachable gate. + +### P0: `awaiting_review` is immediately overwritten by `failed` + +`run_stage_tts` sets `awaiting_review` and then raises (`service.py:1422-1423`). The raise unwinds into +`_run_pipeline`'s catch-all, which sets the job to `failed` (`service.py:391-393`). The review state the +stage just recorded is gone before anyone can read it, and the operator sees a generic failure rather than a +queue of flags. The gate path at `service.py:337-341` returns instead of raising and does not have this +problem. + +### P1: set-of-mark face pairing is enabled by default, contrary to its own documentation + +`worker_vision.py:22` states the feature is off by default and should be enabled per title once tuned. +`worker_vision.py:31` reads `SOM_ATTRIBUTION` with a default of `"1"`. It is on. + +That matters because of the unconditional pairing already recorded as a P0. The mispairing does not stay +local. `_set_of_mark` writes the wrong name into the legend the model reads (`worker_vision.py:99-104`), the +model attributes a line to that face label, and `_apply_speaker_labels` converts the label back into a +`local_id` and stamps `speaker_method = "som_face"` (`worker_vision.py:117`). A geometric guess is laundered +into the highest-trust provenance value the system has. Gate the pairing before trusting that label, or +default the flag to off as documented. + +The pairing is also greedy in face order rather than a joint assignment (`worker_vision.py:42-52`), so the +first face processed can claim a character that fits a later face far better. + +### P1: the JSON repair pass can fabricate content + +When a response fails to parse, `call_gemma4_json` sends the model its own truncated text and asks for the +JSON it should have been (`worker_vision.py:174-178`). The repair call carries no image. On a response +truncated by `max_tokens`, the model is being asked to complete dialogue it can no longer see. Anything it +adds is invented and is indistinguishable downstream from transcribed text. + +Schema-constrained generation, already recommended above, removes most of this path. Until then the repair +pass should re-send the image, or a truncated response should be retried rather than repaired. + +### P1: `/review/preview` silently replaces a beat clip + +`review_preview` renders one panel alone and calls `save_clip(panel_id, ...)` (`service.py:1902-1904`). Its +docstring calls this harmless because assemble would regenerate it. It does not. `_render_one_beat` returns +early when a clip already exists for the leader (`service.py:1591-1592`). Previewing a beat leader therefore +pins the solo preview into the final video and drops the rest of the beat's panels. `review_retts` gets this +right by calling `delete_clip` first (`service.py:2008`). Preview should write to a scratch key or delete the +clip row afterwards. + +### P1: the session manager can orphan a llama-server and hold the GPU + +`open_session` claims the slot, releases the lock, then spawns and health-waits for up to 300 seconds +(`session_manager.py:108-119`). A `/session/close` arriving during that window finds `proc = None`, tears +down nothing, and clears `_active`. The process that finishes starting afterwards is unreferenced and keeps +its VRAM until someone kills it by hand. The next `open` spawns a second server on the same port, and +`_health_wait` cannot tell the two apart because it only probes the port +(`session_manager.py:51-60`). + +Related asymmetry: `_supervise_once` calls `_start_subprocess` while holding `_lock` +(`session_manager.py:169-181`), so a respawn blocks `/session/active`, `/session/close`, and `/session/open` +for the full health wait. The open path was deliberately written to avoid exactly this. + +### P1: nothing limits concurrent jobs + +`_background_jobs` accepts any number of pipelines (`service.py:302-303`, `service.py:427-428`). Each opens +its own model sessions. Combined with the 409 lease stealing already recorded, two jobs terminate each +other's models rather than queueing. The queue has to live at the orchestrator as job admission control. A +fix inside `session_proxy` alone still lets two pipelines interleave stages against one GPU. + +### P1: two cheap throughput wins the plan does not name + +**Order the shared prefix first so llama-server reuses its KV cache.** `/direct/window` sends the same +panel images twice, once for grouping and once for shot design (`worker_vision.py:702-733`). The audit +above frames this as duplicate downloads. The larger cost is the second vision-encoder prefill on the GPU. +llama-server reuses the longest common prompt prefix across requests, and the two calls currently differ in +their first token because the instruction text precedes the images. Putting the images first and the +differing instruction last makes the second pass nearly free on prefill. The same reordering helps any +stage that issues several calls over one image set. + +**Give stored embeddings a version tag.** `embed_crop` uses `pooler_output` when present and silently falls +back to mean-pooled patch tokens otherwise (`worker_identity.py:48-52`). Those two paths produce different +vector spaces, and the fixed 0.85 threshold is only valid for one of them. Nothing recorded beside a stored +`.npy` says which model, revision, or pooling produced it, so a transformers upgrade mixes incompatible +vectors into one gallery with no error. Write the model id and pooling mode next to the vector and refuse to +compare across versions. + +### P2: smaller confirmed defects + +- An out-of-range `choice` from the resolver is mapped to NONE and then reported as `state: "new"` + (`worker_vision.py:898-899`), so a hallucinated index mints a brand new character. It should be + `unresolved`, like a parse failure. +- `_extract_json` matches greedily from the first `{` to the last `}` (`worker_vision.py:130`). Two objects + or any trailing braced prose produce an unparseable span and burn a repair call. `json.JSONDecoder().raw_decode` + from the first brace is exact. +- `worker_identity._known_cache` is invalidated only by `_persist_char` (`worker_identity.py:128-132`). + The reconcile stage deletes losing characters directly in the orchestrator database + (`db.py:506`), so a long-lived worker keeps shortlisting and assigning ids that no longer exist. +- `_pending` holds full crop images in worker memory for a whole chapter and is keyed by session + (`worker_identity.py:25`). This is durable per-chapter state inside a worker documented as stateless, it + is lost on restart, and a character seen exactly once receives no assignment at all, not even a + chapter-local handle. +- `get_conn` opens a connection per call with no `busy_timeout` (`db.py:190-197`). WAL tolerates one writer. + `PIPELINE=1` already writes clips from concurrent tasks while TTS writes audio, so the planned CPU + parallelism will surface as `database is locked` before it surfaces as throughput. +- Worker endpoints declared `async def` run blocking MinIO, OpenCV, torch, and ffmpeg calls directly on the + event loop (`worker_identity.py:177`, `worker_tts.py:170`, `worker_vision.py:230`). A busy worker cannot + answer `/health` or `/unload`. That makes `/health/workers` report a working worker as unreachable + (`service.py:206-216`) and puts the session manager's 30-second `/unload` at risk exactly when VRAM needs + freeing (`session_manager.py:93`). `def` instead of `async def` moves each to the threadpool. +- `layers` runs after `tts` in `STAGES` (`db.py:330-333`), while `run_stage_tts` warns that eager rendering + under `PIPELINE=1` needs layers to run first (`service.py:1427-1429`). With the flag on, solo beats always + render without parallax. +- Stage failure policy is inconsistent. A dropped vision panel fails the stage and halts the pipeline. A + failed direction window counts its panels as done (`service.py:1247`), and layers always finishes + completed (`service.py:1503`). `completed` does not mean the same thing across stages, which makes the + acceptance metrics below hard to read. +- `run_stage_assemble` does not check that `clip_uris` is non-empty before assembling and then marks the job + completed (`service.py:1661-1677`). +- The `/review/panels` timeline sums per-panel audio durations (`service.py:1750`), but assemble crossfades + clips using the per-beat transitions (`service.py:1671-1674`). Every non-`cut` transition shortens the + real video, so reviewer timestamps drift further out of sync the further into the chapter they scrub. +- MinIO credentials are hardcoded as defaults in committed source (`transport.py:95-99`, + `service.py:74-76`). + +### What this changes in the plan + +Add to Phase 1, before any throughput work: + +1. Restore the speaker field across the dialogue, scene, and script boundary. +2. Correct the two verifier rules, then wire `beat` and `verifier_feedback`. +3. Give correctness flags a resolution path that does not require a disabled gate, and stop the TTS block + from erasing `awaiting_review`. +4. Decide the set-of-mark default deliberately, and gate face pairing before the label is trusted. +5. Add job admission control at the orchestrator, not only lease queueing in the proxy. + +Add to the acceptance criteria: the share of narrated lines whose speaker is a named character rather than +`Someone`. That single number would have caught the first finding on the day it landed. diff --git a/HANDOFF.md b/HANDOFF.md index f9ece08..e44c67f 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,75 +1,54 @@ -# HANDOFF: repo reconstructed from agent transcripts (2026-08-11) +# HANDOFF: audit second pass (2026-08-11) ## What was asked -The working copy at `/home/kami/Programs/n8n-worker` was deleted with `rm`, including `.git`. -This repo is the workpc GPU-compute half of the manga pipeline. The ask was to rebuild it from -the transcripts on disk, at minimum the transport layer and the pipeline flow. +Review `AUDIT.md` (untracked, first pass, 565 lines) and add anything missing. -## Sources used -No backup and no remote survived. `~/.local/share/Trash` is empty. No copy exists under -`/home/kami`, `/mnt/D`, or `/mnt/server`. Gitea -(`/mnt/server/mnt/hdd2/gitea/git/repositories/`) holds only correx, hexis, maven, -model-training, muzick, orchestra, and test-e2e. There is no `n8n-worker` mirror. +## What was done +Read both halves of the pipeline and checked the first pass against source. No code changed. +No GPU work, no pipeline run, no tests executed. -Reconstruction replays three histories into one timeline, ordered by timestamp: +Files read in full: +- workpc: `worker_vision.py` (1008), `worker_identity.py` (322), `worker_script.py` (300), + `worker_scene.py` (170), `worker_tts.py` (241), `session_manager.py` (238), `transport.py` (216), + `worker_crop.py` (first 120 of 359). +- homesrv `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`: `service.py` (2048), + `db.py` (815), `session_proxy.py` (302), `correctness.py` (116), `tracklets.py` (121), + `minio_layout.py` (105), `grouping.py` (75). -1. Claude Code transcripts, 25 sessions in - `~/.claude/projects/-home-kami-Programs-n8n-worker/*.jsonl`. Contributes Write and Edit - content, full-file Read results, and `@`-mention attachments. -2. Codex rollouts, 5 sessions with `cwd=/home/kami/Programs/n8n-worker` under - `~/.codex/sessions/2026/07/{13,15,16,18}/rollout-*.jsonl`. Contributes 26 `apply_patch` - blocks. The 4 that the log marks `Script failed` are skipped. These carry the 2026-07-18 - correctness work (`_dialogue_envelope`, `_normalize_claims`, `_annotate_speaker_methods`, - `SOM_ATTRIBUTION=1`) that exists in no Claude transcript. -3. `~/.claude/file-history//@vN` pre-edit blobs, used as cross-checks only. +Result: `AUDIT.md` grew from 565 to 745 lines. New section `## Second-pass findings (2026-08-11)` +at line 566. 4 new P0, 6 new P1, 13 P2, plus 5 additions to the Phase 1 list and 1 to the +acceptance criteria. -Scripts live in -`/tmp/claude-1000/-home-kami-Programs-n8n-worker/6a35d5d8-9f44-4412-955c-9cf088737ad5/scratchpad/`. -`replay2.py` is the merged replay. `codex.py` extracts and applies codex patches. `backups.py` -compares against file-history. `recon_final/_log.json` holds the per-file event log. +Four first-pass claims were confirmed in source and are not restated in the new section: -## Verification -- Replayed to 2026-07-18T13:13:44Z, line counts match the `wc -l` recorded in the transcript at - that moment, for all 10 files it covered. crop 326, identity 314, layers 99, render 996, - scene 159, script 300, tts 231, vision 871, session_manager 238, transport 216. -- Byte sizes match the `ls -l` recorded by the same command. AGENTS.md 867, CLAUDE.md 3678, - manga-recap-pipeline-spec.md 11413, plan.md 16511, spec-v3.md 26170. `.gitignore` is 53. -- 18 files are byte-identical to their newest file-history blob. The 4 that differ (render, - script, tts, spec-v3) differ only by edits made after that blob was taken. -- `python3 -m py_compile *.py scripts/*.py attic/*.py` passes. `bash -n` passes on all 3 shell - scripts. Runtime self-checks such as `python worker_vision.py` were NOT run, because `.venv/` - is gone. +- tracklet gender enum: `tracklets.py:43` uses `male|female`, vision emits `m|f|unknown`. +- missing action evidence: `worker_scene.py:85` emits `action`, `correctness.py:91` reads `actions`. +- dropped verifier feedback: `worker_script.py:13-23` defines neither field. +- 409 lease stealing: `session_proxy.py:40-49`. -## Recovered (live tree) -Workers `worker_{crop,vision,identity,scene,script,tts,layers,render}.py`, the GPU mutex -`session_manager.py`, `transport.py`, helpers `bubble_detect.py`, `face_detect.py`, -`collage.py`, `test_vision_parse.py`, `start_workers.sh`, `systemd/install.sh`, -`scripts/{analyze_video_frames.sh,pick_tts_voice.py}`, `requirements.txt`, `.gitignore`, -and the docs `AGENTS.md`, `CLAUDE.md`, `plan.md`, `spec-v3.md`, `spec-correctness.md`, -`manga-recap-pipeline-spec.md`, `identity-rework-task.md`. - -`attic/` holds files that had been deleted from the tree before the `rm`. They are kept but not -live: `worker_ocr.py`, `worker_parse.py`, `plan-workpc.md`, `char-recognition.md`. +## Top 4 new findings +1. `worker_scene.py:78` reads `speaker` as a local id, but `correctness.py:47` already rewrote it + to a character_id. Every lookup returns None, so all narration says `Someone`. +2. `correctness.py:106-115` fails valid narration on sentence-initial capitals and on short + quotes, and `service.py:1384-1386` turns that into a halted chapter. +3. `service.py:1419-1423` blocks TTS on correctness flags that only `/review/approve` can clear, + while `GATES` defaults to off (`service.py:312`). +4. `session_manager.py:108-119` can orphan a llama-server that keeps its VRAM. ## Still open -- `.venv/` is gone. Rebuild with - `python -m venv .venv && .venv/bin/pip install -r requirements.txt`. The ROCm torch wheel is - not pinned in requirements.txt, so install it the way workpc had it. -- `dots.tts/` (gitignored external checkout) and `legacy/` (pre-rewrite workers, moved there - 2026-07-13) are not recoverable from transcripts. Re-clone dots.tts if TTS is needed. -- `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable. Neither agent - ever read them in full. All three were already deleted from the working tree as of 2026-07-17. -- Git history is gone. A fresh `git init` plus one commit replaces it. Old commits are not - recoverable. -- Two replay gaps were left unpatched. One Edit MISS in `worker_render.py` on 2026-07-14, and one - orphan Edit against `attic/plan-workpc.md` on 2026-07-13. Both files were later replaced by a - full-file snapshot, so the final content stays anchored. -- Edits made outside Claude and codex after 2026-07-18T14:35Z (the last recorded write) cannot be - detected. `unknown:` whether any exist. +- Nothing from this session is half-finished. `AUDIT.md` is complete as written. +- No finding has been fixed. All are report-only. +- `AUDIT.md` is still untracked as of this session's start. Decide whether it belongs in git. +- Prose linter reports ~97 style hits in `AUDIT.md`. Almost all are in the first-pass sections + (lines 1-565), which were left as the author wrote them. +- Carried over from the earlier reconstruction handoff of the same day: + - `.venv/` is gone. Rebuild it, then install the ROCm torch wheel the way workpc had it. + - `dots.tts/` and `legacy/` are not recoverable from transcripts. + - `RESUME_SPEC.md`, `pipeline-design-notes.md`, and `spec-v2.md` are unrecoverable. ## Next command ``` cd /home/kami/Programs/n8n-worker -python -m venv .venv && .venv/bin/pip install -r requirements.txt -.venv/bin/python worker_vision.py # per-file __main__ self-checks, then ./start_workers.sh +sed -n '566,745p' AUDIT.md # read the new section ``` +Then pick the Phase 1 order at `AUDIT.md:511` as amended at the end of the new section.