manga-recap-pipeline/identity-rework-task.md

Rework manga character identity: tracklet spine + gemma resolver

Replace siglip-cosine as the primary identity signal. Priority: high.

Problem

Identity quality plateaus because siglip2 cosine is the wrong instrument: it's a semantic encoder ("man, office, manga panel"), not an instance re-id model. Different people in the same setting score high; the same person across scenes scores low. No threshold / gate / gallery fixes what the vector means — they only shave ~10% off the error. Symptoms on the test chapter:

• male-brown colleague labeled as female-black Choi Haeseon
• speech bubble mis-attributed (MC credited, colleague actually speaking)
• "MC becomes Choi Haeseon" at the end (reconcile over-merge)

Already done (guards against new contamination, not a fix for existing data)

• Gender gate in worker_identity.match() candidate filter + _pending_match (opposite decided genders never match).
• Reconcile gender gate in service.run_stage_reconcile (skip pair if decided genders differ).
• db.create_character name-dedup: skip fold if decided genders conflict (vision-hallucinated name on a wrong-gender crop).
• NOTE: existing roster is already polluted; guards only stop NEW contamination. Needs /stage/clear identity reconcile scene script + re-run to benefit.

Target architecture — two-tier (from multi-object tracking)

Tier 1 — tracklets (local, greedy, high-precision). Link per-panel detections into per-person tracklets across a 5–10 panel window. Local association is the reliable regime (same page, adjacent shot, stable appearance, often the same speaker). Don't touch global identity yet. A tracklet = ordered crops for one person → carries an embedding gallery + merged attributes + associated speaker/dialogue.

Tier 2 — resolve tracklet → global character (deliberate, max evidence). Once a tracklet is stable, resolve identity ONCE per person per scene, using a whole gallery of views + gemma's opinion — not greedily per-crop. Multi-view (front/side/angry/crying) falls out for free (a tracklet accumulates poses as it spans panels).

Signal fusion — keep ORDINAL, not a 9-weight learned sum

No labeled data to tune weights → a hand-weighted 9-signal cost is a worse treadmill. Make it ordinal:

• HARD GATE: gender, species → block impossible matches.
• DECIDER: gemma "same person?" vs a text character-sheet (reuse reconcile /same).
• STRONG FEATURE: hair color+style, name / honorific / alias from dialogue (Korean honorifics = gold anchors).
• TIEBREAK: embedding gallery max-sim (K~5), dialogue-speaker continuity, scene co-presence.

Only build a learned/weighted scorer AFTER labeling a couple chapters.

Attributes to add (cherry-picked)

• Yes / cheap / high-value: honorifics, aliases, hair color/style, gender, species.
• Skip for now (noisy, marginal): age, body-shape, eye color — add only when a specific mislabel needs them.

Reuses existing infra (rewiring, not greenfield)

• /direct/window = windowed multi-image gemma calls in reading order → tracklet window.
• reconcile /same = pairwise "same person?" gemma call → tier-2 adjudicator.
• appearance attrs, gender, dialogue+speaker already in the scene graph.

Suggested sequencing

1. Tier-2 first (~1 day, most of the win): gemma resolver vs text character-sheet, gender-gated, gallery max-sim as tiebreak. Swaps cosine-decider for gemma-decider.
2. Tier-1 tracklet spine: link detections → tracklets, carry gallery + merged attrs, resolve once per tracklet. Robust across poses; contains errors to a tracklet not a crop.
3. Add dialogue-continuity / scene-co-presence tiebreaks.
4. (Later, if labeled data exists) learned association cost.

Open question

Confirm "gemma decides, tracklets give it evidence" is the intended core (vs the fusion score being the core).