manga-recap-pipeline/plan.md

Pipeline correctness plan

Goal

Make character identity, naming, dialogue transcription, speaker attribution, scene grouping, and script generation fail loudly instead of silently turning uncertain model output into canonical story facts.

The desired end state is an evidence-bearing pipeline in which every name, speaker, quote, action, and scene boundary can be traced to its source panel and confidence. Operational batching must never create narrative boundaries.

Scope and repository boundary

This worker repository implements the model and media stages. Scheduling, database mutations, tracklet reconciliation, backfills, beat aggregation, and review gates live in the separate homesrv orchestrator repository. Worker contract changes below must therefore be implemented and tested in both repositories.

The current intended flow is:

crop/order panels
  -> character detection
  -> identity shortlist and tracklets
  -> identity adjudication/reconciliation
  -> grounded dialogue transcription
  -> speaker resolution
  -> scene graph assembly
  -> scene/beat grouping
  -> narration and verification
  -> TTS/render

Current strengths

• Vision, dialogue, identity, direction, and script generation are separate focused passes.
• Identity uses gender-gated SigLIP shortlists and has a Gemma Tier-2 resolver.
• Unnamed one-off characters are held until a second sighting instead of always polluting the permanent registry.
• Dialogue has a short-window mode for cross-panel turn-taking.
• Set-of-mark text and face grounding exists.
• Character-detection parse failures are distinguished from editorial skip decisions.
• Panel ordering and suspicious overlap checks exist.
• Direction and script stages support multi-panel beats.

P0: stop silent corruption

These changes should land before further prompt tuning or visual polish.

1. Introduce typed, canonical speaker references

Problem: dialogue is allowed to return either a panel-local ID or an off-panel character name, but scene assembly only maps local IDs. A valid off-panel name can therefore become None and be narrated as "Someone."

Use an explicit speaker representation at the dialogue boundary:

{
  "speaker": {
    "kind": "local_id|character_id|name|unknown|narrator",
    "value": "person_1"
  }
}

Normalize this to character_id before building a scene graph. Name lookup must support canonical names and aliases, reject ambiguous duplicate-name matches, and retain unresolved names as unresolved rather than discarding them.

Acceptance criteria:

• An off-panel named speaker resolves to the correct canonical character.
• An unknown speaker remains unknown and is never assigned to the main character by default.
• Duplicate/ambiguous names produce a review flag, not an arbitrary selection.
• Scene graphs never receive raw person_N or free-form name values as canonical speakers.

2. Preserve confidence, method, and provenance end to end

Problem: dialogue confidence and identity ambiguity are currently discarded by scene assembly. A weak turn-taking guess is narrated exactly like a clearly grounded bubble tail.

Every dialogue line should retain at least:

{
  "dialogue_id": "p012_r03",
  "panel_id": "p012",
  "region_id": "r03",
  "text": "...",
  "type": "speech",
  "speaker_id": "character_x",
  "speaker_confidence": 0.72,
  "speaker_method": "tail|som_face|turn_taking|solo_prior|manual",
  "identity_confidence": 0.91,
  "flags": []
}

Acceptance criteria:

• Confidence and method survive dialogue -> scene -> script review artifacts.
• Low-confidence attribution can be rendered neutrally without asserting a name.
• Configurable thresholds decide accept/review/reject; prompts do not decide policy.
• Manual corrections are distinguishable from model output and are never overwritten on resume.

3. Make all technical failures fail loudly

Problem: dialogue JSON failure and missing window results currently become empty dialogue, which is indistinguishable from a genuinely silent panel.

All model endpoints should return explicit status metadata:

{
  "status": "ok|partial|failed",
  "parse_failed": false,
  "expected_items": 3,
  "returned_items": 3,
  "warnings": []
}

Acceptance criteria:

• Parse failure cannot mark a stage complete.
• A window response missing one requested panel becomes partial and queues only the missing work.
• Empty dialogue is accepted only when the model completed successfully and grounded detection found no relevant text.
• Script/TTS/render cannot run for panels or beats with unresolved upstream technical failures.

4. Remove the unconditional solo-character speaker assignment

Problem: one visible character does not prove they own a bubble; reaction panels often show the listener while an off-panel character speaks.

Treat sole presence as a prior only. It may raise a score, but it must not overwrite unknown without tail/region evidence or strong conversation continuity.

Acceptance criteria:

• A one-face reaction panel can retain an off-panel speaker.
• Solo attribution records its evidence and confidence.
• Tests cover visible speaker, visible listener/off-panel speaker, thought bubble, and narration box.

P1: ground transcription and identity

5. Make region-grounded dialogue the primary path

Set-of-mark text/face grounding currently exists but is disabled by default. Calibrate it on real titles, then make it the normal path with a clearly reported holistic fallback.

Required validation:

• One output record per detected dialogue/caption region.
• No region can be consumed twice.
• Every line remains attached to the panel containing its region.
• Unmarked text invented by the VLM is rejected or flagged.
• Tail/face evidence is recorded separately from conversation-flow inference.
• Detector failure and zero detected regions are distinct states.

Do not treat all non-Latin text as inherently decorative. Language filtering should be configurable per title/source; otherwise legitimate original-language dialogue can be deleted.

6. Separate name claims from canonical names

Problem: detection can emit a name and identity immediately persists a named unmatched character. This lets one visual-language-model guess become permanent identity state and bypass the two-sighting guard.

Represent naming as evidence:

{
  "claim_id": "...",
  "panel_id": "p012",
  "name": "Seonho",
  "target_local_id": "person_2",
  "evidence_type": "address|self_intro|caption|name_tag|roster_hint",
  "confidence": 0.84
}

Canonical naming should require corroboration or review. Roster names are hints, never identity evidence. A name claim alone must not create or merge a canonical character.

Acceptance criteria:

• Addressed-person and speaking-person tests cannot swap the name.
• Nearby chart/UI labels cannot become canonical names.
• Conflicting name claims are surfaced.
• A later confirmed name safely backfills earlier appearances and narration previews.

7. Strengthen character identity evidence

Current risks include context-heavy body crops, outfit changes, partial occlusion, a global threshold, and a Tier-2 resolver that compares the crop mainly against generated text descriptions.

Implement:

• Face-first embeddings, with body/outfit evidence as a secondary channel.
• Multiple reference exemplars per canonical character.
• Tracklet aggregation across adjacent/overlapping windows before committing identity.
• Per-title calibrated thresholds.
• A real open-set result: known match, new character, or unresolved.
• Candidate reference images in Tier-2 comparison, not only prose descriptions.
• A same-panel exclusivity constraint: two distinct visible people cannot resolve to one identity unless reflection/duplicate-art evidence explicitly allows it.
• Alias and duplicate-name collision checks.

Acceptance criteria:

• Outfit changes do not create a new identity when face evidence is strong.
• Similar-looking simultaneous characters do not collapse into one identity.
• Weak crops remain unresolved rather than minting or merging characters.
• Identity decisions expose all contributing evidence and thresholds.

P1: preserve panel semantics across solo and batch processing

8. Stop destructively merging faceless webtoon fragments

Problem: vertically stacking a text-only fragment with a nearby face changes the visual evidence and can manufacture speaker proximity. It also collapses source gaps and weakens source-coordinate traceability.

Keep fragments separate and add contextual links instead:

{
  "fragment_id": "f12",
  "source_bbox": [0, 1200, 800, 300],
  "context_fragments": ["f11", "f13"],
  "link_reason": "adjacent_text_without_face"
}

The dialogue window can see linked fragments while retaining their boundaries and coordinates.

Acceptance criteria:

• Source pixels and coordinates remain reconstructable.
• A caption/monologue is not assigned merely because a face fragment is adjacent.
• Fragment linkage can be reviewed and changed without recropping the chapter.

9. Reconcile overlapping dialogue windows

Operational windows should overlap by at least one or two panels. Results in the overlap must be reconciled using region IDs, transcription similarity, and attribution confidence.

Acceptance criteria:

• Changing the window size does not change the final transcript for stable fixtures.
• No duplicate lines appear from overlaps.
• Conflicting overlap predictions produce a flag and retain both candidates for review.
• Conversation history is scene-aware and resets only at a confirmed boundary.

10. Prevent batch boundaries from becoming scene boundaries

The first panel of a direction window currently becomes a new scene even when it continues the prior window. Add continues_previous or an equivalent cross-window edge, then run a final boundary reconciliation pass over compact ordered beat summaries.

Validate grouping as an exact partition:

• Every input panel appears exactly once.
• Groups are contiguous and ordered.
• There are no duplicate or unknown panel IDs.
• Missing/invalid grouping fails or falls back with a warning.

Acceptance criteria:

• The same chapter grouped with different operational window sizes produces equivalent scene boundaries.
• A conversation crossing a window boundary remains one beat.
• Non-contiguous model groups are rejected rather than silently applied.

P2: make narration auditable

11. Use an ordered, evidence-bearing beat schema

A multi-panel script request should contain ordered member panels rather than a flattened scene graph plus panel_count:

{
  "beat_id": "b07",
  "panels": [
    {"panel_id": "p20", "dialogue_ids": ["p20_r1"], "actions": [...]},
    {"panel_id": "p21", "dialogue_ids": [], "actions": [...]}
  ]
}

This preserves action and dialogue chronology and gives render/subtitle stages exact mappings.

12. Add a deterministic script verifier before TTS

Verify generated narration against the beat artifact:

• Every used character name exists in the canonical cast map.
• Direct quotes fuzzy-match source dialogue and retain the correct speaker.
• No panel, dialogue line, or action is moved out of chronological order.
• Unsupported proper nouns and character names are rejected.
• Low-confidence facts are not asserted as certain.
• Each sentence/cue maps to one or more source panel/dialogue IDs.

Failed verification should regenerate once with structured feedback, then enter script review. It must not proceed automatically to TTS.

13. Make review gates evidence-focused

Prioritize review items by impact:

1. Duplicate/conflicting canonical names.
2. Identity collisions or unresolved recurring tracklets.
3. Low-confidence or contradictory speaker assignments.
4. Missing/partial dialogue regions.
5. Cross-window scene-boundary conflicts.
6. Script verifier failures.

The reviewer should see the source panel, marked text region, candidate speaker/identity evidence, confidence, and downstream narration affected by the decision.

Test plan

Create a small checked-in fixture corpus with expected structured artifacts. Avoid testing only prompt strings; test complete contracts and invariants.

Required fixture cases:

• Two similar characters speaking back and forth.
• One visible listener with an off-panel speaker.
• Bubble-only panel continuing a prior speaker.
• Name used to address someone other than the speaker.
• Self-introduction and caption-based naming.
• Same character in different clothing and lighting.
• Two simultaneous similar-looking characters.
• Thought, shout, narration, sign, UI text, and SFX classification.
• Sentence split across bubbles without crossing panel ownership.
• Webtoon text fragment between two face fragments.
• Scene and conversation crossing an operational window boundary.
• Malformed/truncated JSON and a model response omitting one panel.
• Multi-panel beat with interleaved action and dialogue chronology.

Contract/invariant tests should assert:

• Stable panel and region IDs across resume.
• No raw local ID or unresolved free-form name reaches the script as a canonical identity.
• No successful empty transcript when text regions were detected but not accounted for.
• No permanent character mutation from a single unsupported name claim.
• No duplicated panel/dialogue membership across beats.
• No TTS request before all blocking correctness gates pass.

Add end-to-end golden tests for:

panels -> detections -> tracklets -> transcript -> beats -> script claims

The golden artifact should compare structured facts, not exact prose wording.

Observability and evaluation

Track correctness metrics per title and pipeline version:

• Character-ID pairwise precision/recall over recurring characters.
• Canonical-name accuracy and unresolved-name rate.
• Dialogue region recall and hallucinated-line rate.
• Speaker-attribution accuracy, split by tail-grounded, solo prior, and off-panel inference.
• Scene-boundary precision/recall.
• Script unsupported-claim and wrong-name rates.
• Percentage of panels/beats sent to review.
• Result stability when operational batch/window sizes change.

Log model/prompt version, detector version, thresholds, window membership, and evidence method with each decision so regressions are reproducible.

Suggested implementation sequence

1. Define versioned shared schemas for dialogue lines, speaker references, name claims, identities, panels, and beats.
2. Update worker and orchestrator contracts to preserve confidence/provenance and explicit failure states.
3. Fix speaker normalization and remove the unconditional solo assignment.
4. Add fail-loud dialogue completeness checks.
5. Enable/calibrate region-grounded transcription and create dialogue fixtures.
6. Separate name claims from persistence; add canonical-name conflict handling.
7. Implement stronger tracklet evidence and image-based Tier-2 identity comparison.
8. Replace destructive webtoon fragment merging with contextual links.
9. Add overlapping-window reconciliation and cross-window scene edges.
10. Introduce ordered beat artifacts and the script verifier.
11. Build evidence-focused review screens and quality dashboards.
12. Re-run the golden corpus after every prompt/model/threshold change.

Immediate repository hygiene

• Fix the stale worker_script.py self-check: implementation expects a ~25-word solo budget while the assertion still expects 35 words.
• Add tests proving dialogue parse failure is not treated as successful empty dialogue.
• Add tests for named off-panel speaker normalization and the visible-listener/off-panel-speaker case.
• Document the configured dialogue and direction window sizes in the orchestrator.
• Confirm that crop warnings, identity ambiguity, backfills, and reconciliation confidence are persisted and honored by review gates rather than merely logged.

Definition of done

This correctness pass is complete when:

• Every narrated name, quote, speaker, action, and scene boundary is traceable to source evidence.
• Low-confidence and failed work cannot silently reach TTS/render.
• Identity and scene results are stable across resume and operational window-size changes.
• The fixture corpus passes end to end in both worker and orchestrator repositories.
• Remaining uncertainty is explicitly reviewable rather than hidden behind polished narration.