kami
ff6a512630
Working tree (including .git) was lost to an rm. Rebuilt by replaying Write/Edit/ Read/attachment events from 25 Claude sessions and 22 successful codex apply_patch blocks into one timestamp-ordered timeline. Verified against ground truth recorded in the transcripts: wc -l on 10 files and ls -l on 5 files at 2026-07-18T13:13:44Z both match exactly; 18 files are byte-identical to their newest ~/.claude/file-history blob. See HANDOFF.md for sources, gaps, and how to rebuild .venv. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
68 lines
3.8 KiB
Markdown
68 lines
3.8 KiB
Markdown
# Rework manga character identity: tracklet spine + gemma resolver
|
||
|
||
Replace siglip-cosine as the primary identity signal. Priority: high.
|
||
|
||
## Problem
|
||
Identity quality plateaus because **siglip2 cosine is the wrong instrument**: it's a
|
||
semantic encoder ("man, office, manga panel"), not an instance re-id model. Different
|
||
people in the same setting score high; the same person across scenes scores low. No
|
||
threshold / gate / gallery fixes what the vector *means* — they only shave ~10% off the
|
||
error. Symptoms on the test chapter:
|
||
- male-brown colleague labeled as female-black **Choi Haeseon**
|
||
- speech bubble mis-attributed (MC credited, colleague actually speaking)
|
||
- "MC becomes Choi Haeseon" at the end (reconcile over-merge)
|
||
|
||
## Already done (guards against *new* contamination, not a fix for existing data)
|
||
- Gender gate in `worker_identity.match()` candidate filter + `_pending_match`
|
||
(opposite decided genders never match).
|
||
- Reconcile gender gate in `service.run_stage_reconcile` (skip pair if decided genders differ).
|
||
- `db.create_character` name-dedup: skip fold if decided genders conflict
|
||
(vision-hallucinated name on a wrong-gender crop).
|
||
- **NOTE:** existing roster is already polluted; guards only stop NEW contamination.
|
||
Needs `/stage/clear identity reconcile scene script` + re-run to benefit.
|
||
|
||
## Target architecture — two-tier (from multi-object tracking)
|
||
**Tier 1 — tracklets (local, greedy, high-precision).**
|
||
Link per-panel detections into per-person tracklets across a 5–10 panel window. Local
|
||
association is the reliable regime (same page, adjacent shot, stable appearance, often the
|
||
same speaker). Don't touch global identity yet. A tracklet = ordered crops for one person →
|
||
carries an embedding gallery + merged attributes + associated speaker/dialogue.
|
||
|
||
**Tier 2 — resolve tracklet → global character (deliberate, max evidence).**
|
||
Once a tracklet is stable, resolve identity ONCE per person per scene, using a whole gallery
|
||
of views + gemma's opinion — not greedily per-crop. Multi-view (front/side/angry/crying)
|
||
falls out for free (a tracklet accumulates poses as it spans panels).
|
||
|
||
## Signal fusion — keep ORDINAL, not a 9-weight learned sum
|
||
No labeled data to tune weights → a hand-weighted 9-signal cost is a worse treadmill.
|
||
Make it ordinal:
|
||
- **HARD GATE:** gender, species → block impossible matches.
|
||
- **DECIDER:** gemma "same person?" vs a text character-sheet (reuse reconcile `/same`).
|
||
- **STRONG FEATURE:** hair color+style, name / honorific / alias from dialogue
|
||
(Korean honorifics = gold anchors).
|
||
- **TIEBREAK:** embedding gallery max-sim (K~5), dialogue-speaker continuity, scene co-presence.
|
||
|
||
Only build a learned/weighted scorer AFTER labeling a couple chapters.
|
||
|
||
## Attributes to add (cherry-picked)
|
||
- Yes / cheap / high-value: honorifics, aliases, hair color/style, gender, species.
|
||
- Skip for now (noisy, marginal): age, body-shape, eye color — add only when a specific
|
||
mislabel needs them.
|
||
|
||
## Reuses existing infra (rewiring, not greenfield)
|
||
- `/direct/window` = windowed multi-image gemma calls in reading order → tracklet window.
|
||
- reconcile `/same` = pairwise "same person?" gemma call → tier-2 adjudicator.
|
||
- appearance attrs, gender, dialogue+speaker already in the scene graph.
|
||
|
||
## Suggested sequencing
|
||
1. **Tier-2 first** (~1 day, most of the win): gemma resolver vs text character-sheet,
|
||
gender-gated, gallery max-sim as tiebreak. Swaps cosine-decider for gemma-decider.
|
||
2. **Tier-1 tracklet spine:** link detections → tracklets, carry gallery + merged attrs,
|
||
resolve once per tracklet. Robust across poses; contains errors to a tracklet not a crop.
|
||
3. Add dialogue-continuity / scene-co-presence tiebreaks.
|
||
4. (Later, if labeled data exists) learned association cost.
|
||
|
||
## Open question
|
||
Confirm "**gemma decides, tracklets give it evidence**" is the intended core
|
||
(vs the fusion score being the core).
|