manga-recap-pipeline/decisions/identity-bbox.md

identity-bbox

The coordinate space of a vision character box, and what reads it.

A vision bbox arrives on gemma's 0-1000 grid, and /vision converts it to pixels

Closed, 2026-08-12.

build_detect_prompt asks for a "pixel bounding box". The model answers on its own normalized grid regardless. Measured over all 113 detections of job 778297bc, read straight from /review/identity:

test	result
boxes with x2 past the 900px panel width	47 of 113
boxes with y2 past 1000, on panels 1257 to 2307px tall	0 of 113
boxes clamped at exactly 1000	21 in x, 5 in y
coordinate range over every box	[0, 1000]

Pixels cannot behave that way. A person standing in the lower half of a 2307px panel needs y2 near 2000, and it never once exceeds 1000.

Consumed as pixels the boxes collapse into the top-left corner of the panel. Four consumers were reading them:

• worker_identity._crop_bbox at worker_identity.py:200, which embeds the crop. This is why a crop of a speech balloon's edge matched Choi Haeseon at 0.9.
• _pair_faces_to_present in worker_vision.py, which compares real detector face boxes, in pixels, against these. The gate could almost never pass, which is the mechanism behind the 7 unknown results out of 7 som_face lines already recorded at worker_vision.py:169.
• the set-of-mark boxes drawn for attribution.
• the review UI, which crops client-side off the panel PNG.

/vision now calls _bbox_to_pixels(characters, w, h) before returning, so all four see pixels and no consumer needs to know the grid existed. Verified by drawing the converted boxes on panel 7. Five of six land on their subject, including person_5, who is Seonho in the foreground and had no identity. person_1 still frames a window mullion with nobody in it, which is #extras-as-cast, not this.

The prompt text still says "pixel bounding box". Rewording it changes what the model emits and needs a GPU run to re-verify, so the boundary converts instead. The ponytail: note on _bbox_to_pixels records that. It also records the trap: a model that really answered in pixels would be scaled down here.

Consequence: every assignment in the registry came from a wrong crop. The existing embeddings and ref_image_uris are enrolled on balloons and window frames. Re-running identity is what makes the registry mean anything. The anonymous-identity and extra-versus-cast work cannot be judged until that rerun happens.

A stage result proves nothing until the worker is newer than the edit

Closed, 2026-08-12. The first rerun after the bbox fix reproduced the defect exactly: 46 of 110 boxes past the 900px panel width, coordinates clamping at 1000, y2 never once past 1000 on panels up to 2307px tall. The same fingerprint as #bbox-is-normalized measured before the fix.

The fix was not wrong. It was not loaded.

vision worker process started   12:00:09
worker_vision.py modified       12:11:35
8113bdf, which contains _bbox_to_pixels, committed  12:16:22

Python binds a module once, at process start. ./start_workers.sh had launched the worker eleven minutes before the file changed, so /vision served pre-fix code for the whole run and returned raw grid boxes. Nothing in the result said so. The stage reported completed 116/116, the orchestrator recorded no error, and identity and reconcile ran to completion on top of it. Cost: one full vision + identity + reconcile cycle, plus a registry reset to undo the 8 characters it minted.

The rerun against a restarted worker gives the opposite reading over the same 110 detections: 0 boxes past the width, 0 past the height, one coordinate on 1000 which is now a real pixel value, and a deepest box reaching 100% down its panel with max y2 = 2307. Boxes track the panel, so they are pixels.

check_stale.sh compares every running worker's process start against its module's mtime and exits non-zero if any is stale. This failure mode was already known as advice — the render worker "must be restarted by hand to pick up an edit" — and advice did not stop it happening. Run the check before any stage run that is meant to prove a code change.

Forbids: citing a stage result as evidence about a code change without establishing that the worker serving it postdates the change.

A detection with no detected face never enrolls or binds

Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.

Fixing the coordinate space made the extras problem worse, not better. With the boxes finally landing on their subjects, panel 7's four background extras became four good crops of four irrelevant people, and one of them bound to Seonho at confidence 1.00. Before the fix the same detection was a crop of scenery and matched nothing much. Correct geometry turned a harmless failure into a poisoned reference set for the lead.

The measured panel 7 outcome, converted boxes, against the art:

box	who	assigned
[457, 657, 642, 937]	Seonho, foreground	Seonho
[669, 591, 763, 822]	the colleague, unnamed in the story	character_f7a4fd, anonymous
[428, 386, 496, 526]	background extra	none
[34, 414, 122, 564]	background extra	none
[498, 386, 568, 533]	background extra	character_d72710 at 0.94
[31, 554, 94, 728]	background extra	Seonho at 1.00

/vision now stamps has_face on every character by running face_detect.detect_faces on the panel and reusing _pair_faces_to_present for containment, so the gate uses the same margin and the same global shortest-first assignment as the speaker path. worker_identity.py skips a character with has_face is False before it crops, embeds, matches or mints.

Two properties are deliberate. It fails open: a missing or raising detector marks every character True, because dropping a whole panel's cast is worse than the over-detection the gate exists to trim. And it gates on is False, not falsiness, so a vision blob written before this change (no key) behaves as it did rather than silently dropping every character.

Cost: a cast member drawn from behind, or in a style the detector misses, now takes no identity on that panel. That is the abstain this pipeline already prefers to a wrong bind (caveats/speaker-attribution.md#no-anonymous-identity).

Forbids: enrolling a reference crop, or binding a character, from a region no face detector confirms.

A resolver NONE mints an anonymous character, it does not clear the crop

Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.

caveats/speaker-attribution.md#no-anonymous-identity asked whether the Tier-2 gemma resolver can answer "none of these". It can, and it always could. /vision/resolve at worker_vision.py:1071 maps choice: 0 to state="new", an out-of-range index to state="unresolved", and a parse failure to unresolved as well. The abstain path was never the defect.

The defect was one branch on the other side of the contract. The orchestrator read only v.get("character_id") and treated every falsy value the same way: unassign_identity on every crop of the tracklet. So a deliberate "this is a real person the roster does not hold" and a hallucinated index both produced nothing, and the unnamed colleague was unknown on every panel she appeared on. The stale ponytail: comment above that block named the reason nobody fixed it, and the reason was real: minting a character needs an embedding_uri, and the orchestrator cannot compute one. siglip is resident in the identity worker, gemma is resident in the vision worker, and session_manager forbids both at once.

What removes the blocker is carrying the embedding, not a third GPU pass. /identity/resolve already computes an embedding per crop and already uploads the crop to s3://manga/{manga}/characters/_crops/{panel}_{local}.png. It now writes the embedding to the same key with a .npy suffix and returns emb_uri in each shortlist entry. The mint is then a local create_character(manga_id, None, appearance, [crop_uri], emb_uri, gender), and the existing per-tracklet assign loop binds every member to it.

tracklets.resolve_outcome holds the three-way decision as a pure function, so the branch that runs is the branch the self-check covers: known on a named answer, mint on state="new" with an emb_uri, clear on unresolved, on a NONE with no embedding, and on an older worker that sends no state at all.

Two limits are deliberate. A tracklet's candidate gallery is built before the loop mints anything, so one person split across two unlinked tracklets still gets two anonymous ids; run_stage_reconcile merges unnamed twins on appearance overlap and is what folds them. And an anonymous character's text sheet (worker_vision._sheet) carries no name, so gemma re-recognising it on a later panel leans on the reference images rather than the description.

Contract: shortlists[].emb_uri is new in the /identity/resolve response. Invariant 7 — both repos changed in the same session.

Forbids: treating an absent character_id as one outcome. A resolver that answered and a resolver that failed are different facts.

The extras gate runs at enrollment and at narration, not at the speaker prompt

Closed, 2026-08-12. Written and self-checked, not yet proven on a GPU run.

#face-gates-enrollment stops a faceless detection taking an identity. It does not stop the detection being narrated, because three places read the raw vision character list and only one of them consults an assignment. Panel 7 is the worked example: six detections, four of them extras or scenery.

worker_scene.build_scene already drops an unassigned detection from characters and present (worker_scene.py:63), so an extra never reached the cast list. Its action did. actions was built from every detection, and that list is what the script prompt renders and what the verifier uses as evidence. So "standing at the window" arrived as a fact about the panel with no character attached, and the verifier confirmed it because the action really was in the blob. Both now skip has_face is False.

service._beat picks the cinematographer's "who" from the first three detections, falling back to a detection's action when it has no name. An extra could take a slot and steer the camera. Also gated.

service._present_characters is deliberately not gated. It builds the dialogue stage's candidate speaker list and the set-of-mark boxes. Two reasons. The failure is already contained: an extra chosen as the speaker has no identity assignment, so normalize_dialogue resolves it to unknown rather than to a wrong name. And the gate's own cost lands hardest here, because a character drawn from behind has no face box, so gating would delete a real speaker from the only list that can attribute their line.

All three gates test is False, never falsiness. A vision blob written before the gate existed carries no has_face key, and a panel whose detector failed is marked True by the fail-open path. Both keep their previous behaviour.

Forbids: adding a fourth consumer of vision["characters"] without deciding which side of this line it is on. The blob keeps every detection on purpose, so the audit can still see what was gated.

A roster name is a guess, so it never reaches detection

service.py seeded known_characters with _roster_char_hints before every detect call, so the chapter roster's names were in front of gemma before any panel had identified anyone. On the 2026-08-12 16:39 run that put "Seonho (short brown hair, yellow plaid shirt)" on a different man wearing glasses, who then held 15 assignments under the lead's name. The real lead was minted separately from the p010 caption as "Lim Seonho", and p020 held both as two people in one panel.

The seeding is removed. A name now reaches detection only from a registry row, and a registry row is named from an in-panel caption or address through name_claims. The distinction is evidence: a roster name is a claim about art nobody has looked at yet, and a registry row carries an embedding plus the panel that named it.

The roster still feeds roster_cast in run_stage_dialogue. There the names are matched against speech, not against faces, so a wrong guess costs an unresolved speaker rather than a wrong face binding.

Confirmed on the 17:38 run: "Lim Seonho" came back as one row with 25 assignments, so dropping the hint did not split the lead across the panels before his caption. "Seonho" fell to 1 assignment.

Not covered: detection still reads a name off the panel and can attach it to the wrong body. p011 and p026 emit name: "Seonho" on person_2 with no roster hint present. That is a separate mechanism and it is what keeps the glasses man named at all.

merged_into is exactly one hop deep

A merge chain means merged_into points at a row that is itself merged, so anything resolving one hop lands on a retired character. Roster readers filter merged_into IS NULL and were never affected.

Two rules, one per direction, and both are needed:

• merge_characters resolves the keeper to its chain root before merging. Bounded 64-step walk, the cap only exists so a cycle cannot hang reconcile.
• retiring a loser repoints every row whose merged_into was that loser.

The keeper walk alone is not enough, which the 17:38 run proved by producing 477c1894 -> a92d9df4 -> 4fb94c15 with the walk deployed. At merge time that pair was fine. The chain formed later, when a row that was already somebody's keeper was itself retired.

_bbox_to_pixels orders the corners, because the model sometimes swaps them

Closed, 2026-08-12.

The 17:38 run returned p007 person_1 as [226, 417, 130, 551], x1 greater than x2. One detection in 117. Clamping each coordinate into the panel kept the swap, so the box stayed a negative-width region. It cropped to nothing, so that detection could not enroll, embed or match, and nothing reported the loss.

_bbox_to_pixels now sorts each pair after clamping:

xs = sorted((clamped_x1, clamped_x2))
ys = sorted((clamped_y1, clamped_y2))
c["bbox"] = [xs[0], ys[0], xs[1], ys[1]]

Sorting is enough. A zero-area box still crops to nothing, and no consumer needs a minimum size that it does not already enforce. The self-check feeds the real swapped box in and asserts [117, 688, 203, 909], which fails without the sort.

Proven on the 18:07 run: 0 degenerate boxes over 119 detections.

A generic word is not identity evidence, and one tokenizer serves both consumers

Closed, 2026-08-12, not yet run on a GPU.

The tracklet linker and reconcile's pair pre-filter each carried their own copy of the appearance tokenizer, and both linked on words that describe a person without distinguishing them. Measured over the 18:07 run's 22 crops for the lead:

p056~p057  hair=['short']   cloth=-        two different men
p047~p054  hair=['hair']    cloth=-        "brown hair" vs "dark hair", the field name in its own value
p109~p110  hair=-           cloth=['shirt','white']

tracklets.appearance_tokens is now the single implementation and subtracts a GENERIC set: length words (short, long, medium, shoulder, length), hair/haired, garment words (shirt, top, jacket, coat, sleeve), and filler (plain, casual, none, unknown). service._appearance_tokens delegates to it, so reconcile's pre-filter is fixed by the same change. That pre-filter is what let the pink tank top reach /vision/same at all.

Four asserts cover it and all four fail with GENERIC emptied.

Measured effect on those 22 crops, with candidate overlap forced to pass (the shortlists are not stored, so this is the most permissive assumption and the real split can only be finer):

	tracklets	worst span
as it ran	3	30 panels
span cap only	8	8
span cap + GENERIC	9	8

The stopwords alone split out p054, the beige-jacket man, and p089, the red-robe chibi.

Not fixed by this. Bare hair colour still links different men: order 56 through 62 stay in one five-crop tracklet on brown, and the cat at p098 still joins its neighbours the same way. The obvious next lever, a crop-to-crop cosine, is ruled out by caveats/audit-open.md#cosine-not-identity.

A tracklet is bounded by span, not only by pairwise distance

Closed, 2026-08-12, not yet run on a GPU.

window=8 bounded each PAIR, and linking is transitive, so nothing bounded the group. The 18:07 run resolved 12 tracklets over 64 crops, and the lead's 22 native assignments fell into two of them spanning panels 47-69 and 80-110. One gemma answer then covered a grey blazer, a denim jacket, a red robe and a cat. The two blocks are visible in the stored confidences: eleven rows at 0.95 and ten at 1.0.

link_tracklets now rejects a merge whose resulting group would span more than window panels:

orders = [dets[m].get("panel_order", m) for m in gi + gj]
if max(orders) - min(orders) > window:
    continue

This makes the module docstring's claim true. The self-check links three compatible detections at panels 1, 9 and 17 and asserts they do not land in one tracklet; it returns [[0, 1, 2]] with the cap disabled.

Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.

The gallery is the live cast, not cosine's top-k

Closed, 2026-08-12, not yet run on a GPU.

run_stage_identity built each tracklet's gallery by unioning the members' cosine shortlists, so the metric that caveats/audit-open.md#cosine-not-identity shows cannot separate people decided who gemma was even allowed to consider. Two different men reach 0.93 on this chapter's crops where the same man reaches 0.96. When the right character fell outside every member's top-5, gemma could only pick a wrong one or answer NONE, and NONE mints a duplicate.

The cast is small. tracklets.cast_gallery builds the gallery from get_known_characters instead: gender-compatible rows only, named first, then registry order. Cosine still shortlists per crop, and those shortlists are still the linker's candidate-overlap evidence, but they no longer bound the answer.

Three consequences, each deliberate:

• Every crop now reaches the resolver. A crop only entered shortlists when its cosine top-k was non-empty, so an early crop seen while the roster was still empty, or one with no gender-compatible row, was dropped from resolution and could never mint. The guard is gone.
• The gallery is re-read per tracklet, so a character an earlier tracklet minted is visible to the later ones. That closes the ponytail: note beside the mint path.
• Two caps, because the prompt carries images. GALLERY_CAP = 10 in service.py bounds the candidates and logs the ids it drops. worker_vision.REF_BUDGET = 12 spreads reference images across them, max(1, min(3, 12 // n)) apiece, so a 9-character cast sends 9 images plus the query instead of the 27 plus query that 3-apiece would have sent.

Cost: more gemma calls, because the crops that used to be dropped now each get one.

Not fixed by this. The gallery being right does not make the crop legible. A back-turned or tiny crop still has no face for gemma to judge, which is NEXT.md item 4.