kami
491dad1c67
Two identity fixes, neither yet run on a GPU. has_face, the extras gate: fixing the bbox coordinate space made extras worse, because crops finally landed on their subjects and a background extra bound to the lead at 1.00. /vision stamps has_face per character using face_detect and _pair_faces_to_present, so containment and its margin match the speaker path. Identity skips has_face is False before it crops or embeds. Fails open on a missing or raising detector, and gates on `is False` so an older vision blob behaves as before. The NONE mint: /identity/resolve now writes each crop's embedding to the crop's key with a .npy suffix and returns emb_uri. The orchestrator cannot compute an embedding of its own (siglip and gemma cannot both be resident), which is why it was clearing a deliberate NONE instead of minting an anonymous character. Also: audit_registry.py, and the tmux respawn-window trap, which leaves a bare shell rather than re-running the window command. decisions/identity-bbox.md#face-gates-enrollment decisions/identity-bbox.md#none-mints-an-anonymous-character Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
163 lines
10 KiB
Markdown
163 lines
10 KiB
Markdown
# speaker-attribution
|
|
|
|
Limits found by cross-checking the 2026-08-11 chapter run against the panel images.
|
|
|
|
## Nothing attributes a speaker in a multi-character panel {#tail-is-not-geometry}
|
|
|
|
The false `tail` label is gone (`decisions/speaker-attribution.md#no-fake-tail`). What replaced it is a
|
|
refusal, not an answer: with two or more characters present, every speech line now returns `unknown`.
|
|
On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back
|
|
to a `generic-handle`. That is the honest floor, and it is not the fix.
|
|
|
|
The measurement that forced it, on job `778297bc`, chapter `7c944dd4`. Three two-character panels were
|
|
checked against the art. All three are wrong, each with the two speakers swapped:
|
|
|
|
| panel key | line | truth | pipeline |
|
|
| --- | --- | --- | --- |
|
|
| `p010.png` | "…definitely an Egen guy, Seonho!" | the woman | Seonho, the person addressed |
|
|
| `p010.png` | "Y-you think so?" | Seonho | Choi Haeseon |
|
|
| `p012.png` | "Want me to send you the link?" | the woman | the man |
|
|
| `p059.png` | "If team leader Choi says it, it must be true." | the man | Choi Haeseon |
|
|
|
|
The last row needs no image: the line refers to Choi in the third person and is attributed to Choi.
|
|
|
|
The grounded path exists and almost never fires. Only 2 of 81 speech lines got `som_face`, because
|
|
attribution marks need `face_detect` boxes that survive `_pair_faces_to_present`, and these webtoon
|
|
close-ups rarely produce them. Inference, not measured: the face detector was not instrumented.
|
|
|
|
`worker_vision.py:356` already carries the `ponytail:` note that multi-character attribution needs
|
|
per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads carry balloon fill and
|
|
tail tips and are unused.
|
|
|
|
**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric
|
|
(`ROADMAP.md`). The 30% read on this run counted attributions the sample says are wrong. The next run
|
|
will read lower and will be the first honest number. Raising it means binding a balloon to a speaker
|
|
by tail geometry, using the unused `det`/`seg` heads.
|
|
|
|
## One character id covers two different women {#identity-over-merge}
|
|
|
|
`character_afa7623b` is stored as "black bob, white sweater" and is assigned both to that person
|
|
(`p059.png`) and to the brown-bob green-top coworker (`p010.png`, `p012.png`). It took 25 of the 26
|
|
identity assignments in the chapter, against 113 detected people. Coverage is 23%.
|
|
|
|
**Revisit trigger:** any work on the identity Tier-2 decider. A single id absorbing a whole chapter is
|
|
the signature to watch for.
|
|
|
|
## The character registry carries five weeks of wrong names {#registry-pollution}
|
|
|
|
The registry holds 53 characters for manga `ef105a86`, 41 of them unnamed, with "Kei" three times and
|
|
"Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only
|
|
Haeseon and Seonho do. `/stage/clear` leaves the per-manga registry intact by design, so every rerun
|
|
inherits the whole pile.
|
|
|
|
Duplicate rows also make a correct name unresolvable. This manga holds `Choi Haeseon`, `Seonho` with
|
|
aliases `["Lim Seonho", "Seonho"]`, and a separate `Lim Seonho`. An answer of "Lim Seonho" matches two
|
|
rows, so `normalize_speaker` returns candidates and raises `ambiguous-speaker` instead of binding. The
|
|
pipeline read the name correctly and still cannot name the speaker.
|
|
|
|
**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the
|
|
registry to a chapter or add a reviewed reset. Merging the duplicate rows needs the reversible-merge
|
|
design first (`caveats/audit-open.md#destructive-reconcile`).
|
|
|
|
## One invented word still halts the chapter {#one-word-halts-chapter}
|
|
|
|
The multi-word name failure is fixed (`decisions/speaker-attribution.md#multiword-cast-names`). The
|
|
blast radius it exposed is not. `run_stage_script` retries a rejected beat once, then raises, so a
|
|
single unsupported token ends the run at that beat. On job `778297bc` one of the 29 lost beats cited
|
|
`['Blur']`, an onomatopoeia the model invented. The verifier was right, and the whole chapter still
|
|
stopped.
|
|
|
|
**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer
|
|
is to flag the beat for review and continue, which is `#136` gate work, not a verifier change.
|
|
|
|
## Every `bbox` is read in the wrong coordinate space {#bbox-wrong-space}
|
|
|
|
**Resolved 2026-08-12, `decisions/identity-bbox.md#bbox-is-normalized`.** The space is gemma's 0-1000
|
|
grid, proven over all 113 detections, and `/vision` now converts to pixels before returning. The face
|
|
pairing at `worker_vision.py:57` was reading the same numbers against real pixel face boxes, so it is
|
|
fixed by the same change. What is left of this entry is the consequence. Every stored assignment,
|
|
embedding and `ref_image_uris` came from a wrong crop. Identity has to re-run before any of it means
|
|
anything. The rest below is kept as the record of how it read before.
|
|
|
|
Vision's `bbox` values are stored and consumed as absolute pixels. On panel
|
|
`7c944dd4-e972-42c7-ba60-9f6939548e80_p007` (crop 900x1650) all six boxes then land in the top third of
|
|
the panel, two of them inside the "YEAH!" speech balloon. Divided by 1000 against the panel's own
|
|
dimensions, four of the six fit their subjects tightly.
|
|
|
|
Two places in the code assert pixels, and the art contradicts both:
|
|
|
|
- `worker_vision.py:271`, prompt text: `pixel bounding box [x1,y1,x2,y2] (top-left, bottom-right corners)`
|
|
- `worker_identity.py:91`, comment: `vision emits [x1, y1, x2, y2] pixel corners (gemma4's native bbox convention)`
|
|
|
|
Who pays: identity embeds `_crop_bbox(img, ch["bbox"])` at `worker_identity.py:200`, so it matches faces
|
|
against crops of balloons and window frames. On panel 7 that produced `Choi Haeseon` at confidence 0.9
|
|
from a crop of a balloon edge and `Lim Seonho` at 0.9 from an empty window frame. Blank crops embed
|
|
alike, which is a plausible mechanism for one row absorbing 25 of 26 assignments. The face pairing at
|
|
`worker_vision.py:57` reads the same numbers and was not checked.
|
|
|
|
Rescaling is necessary and not sufficient. After scaling, `person_1` still sits on a window frame with
|
|
nobody in it, and `person_6` clips its subject and runs onto the frame. `worker_vision.py:38` already
|
|
calls the box "coarse, imprecise".
|
|
|
|
Revisit trigger: before any further identity or balloon-geometry work. Nothing downstream of `bbox` can
|
|
be judged while the crops are wrong.
|
|
|
|
## Identity cannot say "a person with no name" {#no-anonymous-identity}
|
|
|
|
The colleague on panel 7 has no name in the story. She was assigned `Choi Haeseon` at confidence 0.9.
|
|
Across the chapter that row holds 25 of 26 assignments, so in practice it is the label the pipeline
|
|
stamps on any unnamed woman. Narration then calls her Choi Haeseon and inherits that row's gender, which
|
|
is the direct cause of the user's 0:20 and 1:51 notes and of the gender flips at 1:45 and the closing
|
|
line.
|
|
|
|
This is the same shape as invariant 6 in `CLAUDE.md`, which forbids minting a character from an
|
|
unparseable model answer. The missing rule: never attach a name to a detection that carries no name
|
|
evidence. A recurring unnamed person needs a stable anonymous identity, so narration says "the
|
|
colleague" every time.
|
|
|
|
`match()` at `worker_identity.py:69` does abstain, returning `None` below threshold, so the 0.9 came from
|
|
cosine clearing the threshold on a wrong crop.
|
|
|
|
**Resolved 2026-08-12, `decisions/identity-bbox.md#none-mints-an-anonymous-character`.** The gemma resolver
|
|
can answer "none of these" and always could. The orchestrator was discarding the answer. It read only
|
|
`character_id`, so a deliberate NONE and a hallucinated index both unassigned every crop of the tracklet.
|
|
A NONE now mints an anonymous character from the crop, using the embedding identity ships beside it. Not
|
|
yet proven on a GPU run.
|
|
|
|
## Vision does not separate a background extra from cast {#extras-as-cast}
|
|
|
|
Panel 7 is a wide establishing shot. Vision emitted 6 characters. Two matter: Seonho in the foreground
|
|
and the unnamed colleague. Three are background office extras, and one (`person_1`) is a window frame
|
|
with nobody in it. All six reach identity as equal candidates.
|
|
|
|
Who pays: the roadmap's framing figure, "26 of 113 detected people carry an identity", counted mostly
|
|
extras, so it measured nothing useful and should not be quoted again.
|
|
|
|
Revisit trigger: with `#no-anonymous-identity`, since both change what identity is allowed to return.
|
|
|
|
## Cast reference profiles are enrolled from wrong crops {#poisoned-reference-set}
|
|
|
|
`characters` carries `ref_image_uris` and `embedding_uri`, and all 53 rows have both populated. So the
|
|
cast-profile mechanism exists. It is enrolled through `#bbox-wrong-space`, so the stored references are
|
|
crops of balloon edges, window frames and background extras rather than of faces.
|
|
|
|
The visual comparison people reach for as the fix is **already implemented**, so do not build it again.
|
|
`/vision/resolve` at `worker_vision.py:963` sends the query crop plus up to 3 labelled reference images
|
|
per candidate. `build_resolve_prompt` already tells the model to judge face shape first, to treat hair
|
|
and outfit as secondary, that two people sharing a hair colour are not the same, and to answer `0` for
|
|
NONE when unsure. `choice: 0` returns `state="new"` and an out-of-range index returns `unresolved`. What
|
|
the orchestrator does with each is
|
|
`decisions/identity-bbox.md#none-mints-an-anonymous-character`. The
|
|
`ref_image_uris` column is republished as `reference_image_uris` at `worker_identity.py:152` and `:161`,
|
|
so the references reach the model.
|
|
|
|
That is why this caveat is about the pixels and not the prompt. The resolver compares a crop of a balloon
|
|
edge against references enrolled from window frames and background extras, then sometimes answers "same".
|
|
Nothing gates enrollment on the crop holding a face.
|
|
|
|
Who pays: every later match, because the reference set defines what a character looks like. Fixing
|
|
`#bbox-wrong-space` without re-enrolling leaves the poisoned references in place.
|
|
|
|
Revisit trigger: as soon as `#bbox-wrong-space` lands, re-enroll from corrected crops and treat the
|
|
existing `ref_image_uris` and `embedding_uri` values as invalid.
|