manga-recap-pipeline/decisions/identity-bbox.md

# identity-bbox

The coordinate space of a vision character box, and what reads it.

## A vision `bbox` arrives on gemma's 0-1000 grid, and `/vision` converts it to pixels {#bbox-is-normalized}

**Closed, 2026-08-12.**

`build_detect_prompt` asks for a "pixel bounding box". The model answers on its own normalized grid
regardless. Measured over all 113 detections of job `778297bc`, read straight from `/review/identity`:

| test | result |
| --- | --- |
| boxes with `x2` past the 900px panel width | **47 of 113** |
| boxes with `y2` past 1000, on panels 1257 to 2307px tall | **0 of 113** |
| boxes clamped at exactly 1000 | 21 in x, 5 in y |
| coordinate range over every box | `[0, 1000]` |

Pixels cannot behave that way. A person standing in the lower half of a 2307px panel needs `y2` near
2000, and it never once exceeds 1000.

Consumed as pixels the boxes collapse into the top-left corner of the panel. Four consumers were reading
them:

* `worker_identity._crop_bbox` at `worker_identity.py:200`, which embeds the crop. This is why a crop of
  a speech balloon's edge matched `Choi Haeseon` at 0.9.
* `_pair_faces_to_present` in `worker_vision.py`, which compares real detector face boxes, in pixels,
  against these. The gate could almost never pass, which is the mechanism behind the 7 `unknown` results
  out of 7 `som_face` lines already recorded at `worker_vision.py:169`.
* the set-of-mark boxes drawn for attribution.
* the review UI, which crops client-side off the panel PNG.

`/vision` now calls `_bbox_to_pixels(characters, w, h)` before returning, so all four see pixels and no
consumer needs to know the grid existed. Verified by drawing the converted boxes on panel 7. Five of six
land on their subject, including `person_5`, who is Seonho in the foreground and had no identity.
`person_1` still frames a window mullion with nobody in it, which is `#extras-as-cast`, not this.

The prompt text still says "pixel bounding box". Rewording it changes what the model emits and needs a
GPU run to re-verify, so the boundary converts instead. The `ponytail:` note on `_bbox_to_pixels` records
that. It also records the trap: a model that really answered in pixels would be scaled down here.

**Consequence: every assignment in the registry came from a wrong crop.** The existing embeddings and
`ref_image_uris` are enrolled on balloons and window frames. Re-running identity
is what makes the registry mean anything. The anonymous-identity and extra-versus-cast work cannot be
judged until that rerun happens.

## A stage result proves nothing until the worker is newer than the edit {#stale-worker-invalidates}

**Closed, 2026-08-12.** The first rerun after the bbox fix reproduced the defect exactly: 46 of 110
boxes past the 900px panel width, coordinates clamping at 1000, `y2` never once past 1000 on panels up
to 2307px tall. The same fingerprint as `#bbox-is-normalized` measured before the fix.

The fix was not wrong. It was not loaded.

```
vision worker process started   12:00:09
worker_vision.py modified       12:11:35
8113bdf, which contains _bbox_to_pixels, committed  12:16:22
```

Python binds a module once, at process start. `./start_workers.sh` had launched the worker eleven
minutes before the file changed, so `/vision` served pre-fix code for the whole run and returned raw
grid boxes. Nothing in the result said so. The stage reported `completed 116/116`, the orchestrator
recorded no error, and identity and reconcile ran to completion on top of it. Cost: one full
vision + identity + reconcile cycle, plus a registry reset to undo the 8 characters it minted.

The rerun against a restarted worker gives the opposite reading over the same 110 detections: 0 boxes
past the width, 0 past the height, one coordinate on 1000 which is now a real pixel value, and a
deepest box reaching 100% down its panel with `max y2 = 2307`. Boxes track the panel, so they are
pixels.

`check_stale.sh` compares every running worker's process start against its module's mtime and exits
non-zero if any is stale. This failure mode was already known as advice — the render worker "must be
restarted by hand to pick up an edit" — and advice did not stop it happening. Run the check before any
stage run that is meant to prove a code change.

Forbids: citing a stage result as evidence about a code change without establishing that the worker
serving it postdates the change.

## A detection with no detected face never enrolls or binds {#face-gates-enrollment}

**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**

Fixing the coordinate space made the extras problem worse, not better. With the boxes finally landing
on their subjects, panel 7's four background extras became four good crops of four irrelevant people,
and one of them bound to `Seonho` at confidence 1.00. Before the fix the same detection was a crop of
scenery and matched nothing much. Correct geometry turned a harmless failure into a poisoned reference
set for the lead.

The measured panel 7 outcome, converted boxes, against the art:

| box | who | assigned |
| --- | --- | --- |
| `[457, 657, 642, 937]` | Seonho, foreground | `Seonho` |
| `[669, 591, 763, 822]` | the colleague, unnamed in the story | `character_f7a4fd`, anonymous |
| `[428, 386, 496, 526]` | background extra | none |
| `[34, 414, 122, 564]` | background extra | none |
| `[498, 386, 568, 533]` | background extra | `character_d72710` at 0.94 |
| `[31, 554, 94, 728]` | background extra | `Seonho` at 1.00 |

`/vision` now stamps `has_face` on every character by running `face_detect.detect_faces` on the panel
and reusing `_pair_faces_to_present` for containment, so the gate uses the same margin and the same
global shortest-first assignment as the speaker path. `worker_identity.py` skips a character with
`has_face is False` before it crops, embeds, matches or mints.

Two properties are deliberate. It **fails open**: a missing or raising detector marks every character
`True`, because dropping a whole panel's cast is worse than the over-detection the gate exists to trim.
And it gates on `is False`, not falsiness, so a vision blob written before this change (no key) behaves
as it did rather than silently dropping every character.

Cost: a cast member drawn from behind, or in a style the detector misses, now takes no identity on that
panel. That is the abstain this pipeline already prefers to a wrong bind
(`caveats/speaker-attribution.md#no-anonymous-identity`).

Forbids: enrolling a reference crop, or binding a character, from a region no face detector confirms.

## A resolver NONE mints an anonymous character, it does not clear the crop {#none-mints-an-anonymous-character}

**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**

`caveats/speaker-attribution.md#no-anonymous-identity` asked whether the Tier-2 gemma resolver can answer
"none of these". It can, and it always could. `/vision/resolve` at `worker_vision.py:1071` maps `choice: 0`
to `state="new"`, an out-of-range index to `state="unresolved"`, and a parse failure to `unresolved` as
well. The abstain path was never the defect.

The defect was one branch on the other side of the contract. The orchestrator read only
`v.get("character_id")` and treated every falsy value the same way: `unassign_identity` on every crop of
the tracklet. So a deliberate "this is a real person the roster does not hold" and a hallucinated index
both produced nothing, and the unnamed colleague was `unknown` on every panel she appeared on. The stale
`ponytail:` comment above that block named the reason nobody fixed it, and the reason was real: minting a
character needs an `embedding_uri`, and the orchestrator cannot compute one. siglip is resident in the
identity worker, gemma is resident in the vision worker, and `session_manager` forbids both at once.

What removes the blocker is carrying the embedding, not a third GPU pass. `/identity/resolve` already
computes an embedding per crop and already uploads the crop to
`s3://manga/{manga}/characters/_crops/{panel}_{local}.png`. It now writes the embedding to the same key
with a `.npy` suffix and returns `emb_uri` in each shortlist entry. The mint is then a local
`create_character(manga_id, None, appearance, [crop_uri], emb_uri, gender)`, and the existing per-tracklet
assign loop binds every member to it.

`tracklets.resolve_outcome` holds the three-way decision as a pure function, so the branch that runs is
the branch the self-check covers: `known` on a named answer, `mint` on `state="new"` with an `emb_uri`,
`clear` on `unresolved`, on a NONE with no embedding, and on an older worker that sends no `state` at all.

Two limits are deliberate. A tracklet's candidate gallery is built before the loop mints anything, so one
person split across two unlinked tracklets still gets two anonymous ids; `run_stage_reconcile` merges
unnamed twins on appearance overlap and is what folds them. And an anonymous character's text sheet
(`worker_vision._sheet`) carries no name, so gemma re-recognising it on a later panel leans on the
reference images rather than the description.

Contract: `shortlists[].emb_uri` is new in the `/identity/resolve` response. Invariant 7 — both repos
changed in the same session.

Forbids: treating an absent `character_id` as one outcome. A resolver that answered and a resolver that
failed are different facts.

## The extras gate runs at enrollment and at narration, not at the speaker prompt {#extras-gate-consumers}

**Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.**

`#face-gates-enrollment` stops a faceless detection taking an identity. It does not stop the detection
being narrated, because three places read the raw vision character list and only one of them consults an
assignment. Panel 7 is the worked example: six detections, four of them extras or scenery.

`worker_scene.build_scene` already drops an unassigned detection from `characters` and `present`
(`worker_scene.py:63`), so an extra never reached the cast list. Its **action** did. `actions` was built
from every detection, and that list is what the script prompt renders and what the verifier uses as
evidence. So "standing at the window" arrived as a fact about the panel with no character attached, and
the verifier confirmed it because the action really was in the blob. Both now skip `has_face is False`.

`service._beat` picks the cinematographer's "who" from the first three detections, falling back to a
detection's action when it has no name. An extra could take a slot and steer the camera. Also gated.

`service._present_characters` is deliberately **not** gated. It builds the dialogue stage's candidate
speaker list and the set-of-mark boxes. Two reasons. The failure is already contained: an extra chosen as
the speaker has no identity assignment, so `normalize_dialogue` resolves it to unknown rather than to a
wrong name. And the gate's own cost lands hardest here, because a character drawn from behind has no face
box, so gating would delete a real speaker from the only list that can attribute their line.

All three gates test `is False`, never falsiness. A vision blob written before the gate existed carries no
`has_face` key, and a panel whose detector failed is marked `True` by the fail-open path. Both keep their
previous behaviour.

Forbids: adding a fourth consumer of `vision["characters"]` without deciding which side of this line it is
on. The blob keeps every detection on purpose, so the audit can still see what was gated.

## A roster name is a guess, so it never reaches detection {#roster-does-not-name}

`service.py` seeded `known_characters` with `_roster_char_hints` before every detect call, so the chapter
roster's names were in front of gemma before any panel had identified anyone. On the 2026-08-12 16:39 run
that put "Seonho (short brown hair, yellow plaid shirt)" on a different man wearing glasses, who then held
15 assignments under the lead's name. The real lead was minted separately from the p010 caption as
"Lim Seonho", and p020 held both as two people in one panel.

The seeding is removed. A name now reaches detection only from a registry row, and a registry row is named
from an in-panel caption or address through `name_claims`. The distinction is evidence: a roster name is a
claim about art nobody has looked at yet, and a registry row carries an embedding plus the panel that
named it.

The roster still feeds `roster_cast` in `run_stage_dialogue`. There the names are matched against speech,
not against faces, so a wrong guess costs an unresolved speaker rather than a wrong face binding.

Confirmed on the 17:38 run: "Lim Seonho" came back as one row with 25 assignments, so dropping the hint
did not split the lead across the panels before his caption. "Seonho" fell to 1 assignment.

Not covered: detection still reads a name off the panel and can attach it to the wrong body. p011 and p026
emit `name: "Seonho"` on `person_2` with no roster hint present. That is a separate mechanism and it is
what keeps the glasses man named at all.

## `merged_into` is exactly one hop deep {#merge-chains-flatten}

A merge chain means `merged_into` points at a row that is itself merged, so anything resolving one hop
lands on a retired character. Roster readers filter `merged_into IS NULL` and were never affected.

Two rules, one per direction, and both are needed:

- `merge_characters` resolves the keeper to its chain root before merging. Bounded 64-step walk, the cap
  only exists so a cycle cannot hang reconcile.
- retiring a loser repoints every row whose `merged_into` was that loser.

The keeper walk alone is not enough, which the 17:38 run proved by producing
`477c1894 -> a92d9df4 -> 4fb94c15` with the walk deployed. At merge time that pair was fine. The chain
formed later, when a row that was already somebody's keeper was itself retired.

## `_bbox_to_pixels` orders the corners, because the model sometimes swaps them {#bbox-corners-ordered}

**Closed, 2026-08-12.**

The 17:38 run returned `p007 person_1` as `[226, 417, 130, 551]`, x1 greater than x2. One detection in 117.
Clamping each coordinate into the panel kept the swap, so the box stayed a negative-width region. It
cropped to nothing, so that detection could not enroll, embed or match, and nothing reported the loss.

`_bbox_to_pixels` now sorts each pair after clamping:

```python
xs = sorted((clamped_x1, clamped_x2))
ys = sorted((clamped_y1, clamped_y2))
c["bbox"] = [xs[0], ys[0], xs[1], ys[1]]
```

Sorting is enough. A zero-area box still crops to nothing, and no consumer needs a minimum size that it
does not already enforce. The self-check feeds the real swapped box in and asserts `[117, 688, 203, 909]`,
which fails without the sort.

Proven on the 18:07 run: 0 degenerate boxes over 119 detections.

## A generic word is not identity evidence, and one tokenizer serves both consumers {#generic-tokens}

**Closed, 2026-08-12, not yet run on a GPU.**

The tracklet linker and reconcile's pair pre-filter each carried their own copy of the appearance
tokenizer, and both linked on words that describe a person without distinguishing them. Measured over the
18:07 run's 22 crops for the lead:

```
p056~p057  hair=['short']   cloth=-        two different men
p047~p054  hair=['hair']    cloth=-        "brown hair" vs "dark hair", the field name in its own value
p109~p110  hair=-           cloth=['shirt','white']
```

`tracklets.appearance_tokens` is now the single implementation and subtracts a `GENERIC` set: length words
(`short`, `long`, `medium`, `shoulder`, `length`), `hair`/`haired`, garment words (`shirt`, `top`,
`jacket`, `coat`, `sleeve`), and filler (`plain`, `casual`, `none`, `unknown`). `service._appearance_tokens`
delegates to it, so reconcile's pre-filter is fixed by the same change. That pre-filter is what let the
pink tank top reach `/vision/same` at all.

Four asserts cover it and all four fail with `GENERIC` emptied.

Measured effect on those 22 crops, with candidate overlap forced to pass (the shortlists are not stored,
so this is the most permissive assumption and the real split can only be finer):

| | tracklets | worst span |
| --- | --- | --- |
| as it ran | 3 | 30 panels |
| span cap only | 8 | 8 |
| span cap + `GENERIC` | 9 | 8 |

The stopwords alone split out `p054`, the beige-jacket man, and `p089`, the red-robe chibi.

**Not fixed by this.** Bare hair colour still links different men: `order 56` through `62` stay in one
five-crop tracklet on `brown`, and the cat at `p098` still joins its neighbours the same way. The obvious
next lever, a crop-to-crop cosine, is ruled out by `caveats/audit-open.md#cosine-not-identity`.

## A tracklet is bounded by span, not only by pairwise distance {#tracklet-span-cap}

**Closed, 2026-08-12, not yet run on a GPU.**

`window=8` bounded each PAIR, and linking is transitive, so nothing bounded the group. The 18:07 run
resolved 12 tracklets over 64 crops, and the lead's 22 native assignments fell into two of them spanning
panels 47-69 and 80-110. One gemma answer then covered a grey blazer, a denim jacket, a red robe and a cat.
The two blocks are visible in the stored confidences: eleven rows at 0.95 and ten at 1.0.

`link_tracklets` now rejects a merge whose resulting group would span more than `window` panels:

```python
orders = [dets[m].get("panel_order", m) for m in gi + gj]
if max(orders) - min(orders) > window:
    continue
```

This makes the module docstring's claim true. The self-check links three compatible detections at panels 1,
9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled.

Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.

## The gallery is the live cast, not cosine's top-k {#cast-is-the-gallery}

**Closed, 2026-08-12, not yet run on a GPU.**

`run_stage_identity` built each tracklet's gallery by unioning the members' cosine shortlists, so the
metric that `caveats/audit-open.md#cosine-not-identity` shows cannot separate people decided who gemma was
even allowed to consider. Two different men reach 0.93 on this chapter's crops where the same man reaches
0.96. When the right character fell outside every member's top-5, gemma could only pick a wrong one or
answer NONE, and NONE mints a duplicate.

The cast is small. `tracklets.cast_gallery` builds the gallery from `get_known_characters` instead:
gender-compatible rows only, named first, then registry order. Cosine still shortlists per crop, and those
shortlists are still the linker's candidate-overlap evidence, but they no longer bound the answer.

Three consequences, each deliberate:

- **Every crop now reaches the resolver.** A crop only entered `shortlists` when its cosine top-k was
  non-empty, so an early crop seen while the roster was still empty, or one with no gender-compatible row,
  was dropped from resolution and could never mint. The guard is gone.
- **The gallery is re-read per tracklet**, so a character an earlier tracklet minted is visible to the
  later ones. That closes the `ponytail:` note beside the mint path.
- **Two caps, because the prompt carries images.** `GALLERY_CAP = 10` in `service.py` bounds the
  candidates and logs the ids it drops. `worker_vision.REF_BUDGET = 12` spreads reference images across
  them, `max(1, min(3, 12 // n))` apiece, so a 9-character cast sends 9 images plus the query instead of
  the 27 plus query that 3-apiece would have sent.

Cost: more gemma calls, because the crops that used to be dropped now each get one.

**Not fixed by this.** The gallery being right does not make the crop legible. A back-turned or tiny crop
still has no face for gemma to judge, which is `NEXT.md` item 4.