Send the live cast to the resolver, not cosine's top-k

The gallery gemma chose from was the union of the tracklet members' cosine
shortlists. Cosine cannot separate people on this data, so a metric that
cannot tell two men apart decided who gemma was allowed to consider, and the
right character was often not on the list.

The gallery is now the live cast: gender-compatible rows from
get_known_characters, named first, capped, re-read per tracklet so a minted
character is visible to later ones. Every crop reaches the resolver now,
including one whose cosine top-k was empty; those used to be dropped.

worker_vision spreads reference images across a budget instead of sending 3
per candidate, so a 9-character cast costs 9 images and not 27.

Ran on a GPU, 19:44-19:52 UTC. The lead's assignments drop from 36 to 16 and
14 of the 16 are him; the other two are art inside a panel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-13 00:15:52 +04:00
parent de7e34bc0f
commit a386e9d910
5 changed files with 133 additions and 17 deletions
+44
View File
@@ -679,3 +679,47 @@ their fix disabled. Nothing deployed, nothing re-run on a GPU.
Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad
only, not committed. only, not committed.
## 2026-08-12, 19:44-19:52 UTC — fourth GPU cycle: the cast is the gallery
Three fixes ran together for the first time: the tracklet span cap and the shared `GENERIC` tokenizer,
both written last session and never deployed, plus the new one
(`decisions/identity-bbox.md#cast-is-the-gallery`). Job `778297bc`, chapter `7c944dd4`, registry reset,
vision cleared, `vision -> identity -> reconcile`. Vision 116/116 in 3m55s, identity 116/116 in 2m44s,
reconcile 20/20 in 44s.
| metric | 18:07 run | 19:44 run |
| --- | --- | --- |
| detections | 119 | 119 |
| assignments | 68 | 60 |
| coverage | 57% | 50% |
| tracklets over crops | 12 / 64 | 33 / 72 |
| lead's assignments | 36 | 16 |
| top character's share | -- | 16/60 = 27% |
| characters after reconcile | 18 | 14 |
| minted / cleared | -- | 10 / 12 |
**Coverage went down and that is the fix working.** gemma cleared 12 crops it used to be forced to name
from a cosine top-k that did not contain the right person. 72 crops entered resolution against 64, because
a crop with an empty cosine shortlist is no longer dropped.
**Checked by eye, which is the only check that sees this.** Contact sheets of every assigned crop, per
character, confirmed by the user. The lead holds 16 crops. 14 are him and 2 are art inside a panel, the
photograph at `order 17` and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and
she is a main character the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2
are `2b1b12a1` (`order 31`, `order 33`). Against the 18:07 run, where the lead's 36 covered six different
men, a chibi and a cat.
Every wrong crop on the lead is one defect: vision treats art inside a panel as the scene. It is not a
linker failure and no identity change will fix it.
**The cap fired twice and dropped the wrong rows.** `GALLERY_CAP = 10` truncated a 16-row gender-compatible
cast at `p097` and an 11-row one at `p109`. `cast_gallery` orders named first, so what it drops is exactly
the recently minted anonymous rows, which is the population a later tracklet most needs to match against.
Filed as `caveats/audit-open.md#gallery-cap-drops-the-unnamed`.
Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestrator 118 passed,
`./check_stale.sh` exit 0 before the cycle. Vision and identity workers both restarted, the second because
`/characters/reset` asked for it.
Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed.
+20 -15
View File
@@ -9,28 +9,33 @@ The chapter runs end to end. The A/V sync defect is fixed and `chapter.mp4` is r
against audio 364.122s at `25/1`. against audio 364.122s at `25/1`.
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels. Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels.
Three GPU cycles ran on 2026-08-12, the last 18:07-18:13 UTC on a reset registry. The job is `running` and Four GPU cycles ran on 2026-08-12, the last 19:44-19:52 UTC on a reset registry. The job is `running` and
parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are
current. current.
Eight changes are proven on real panels. The fourth session contributed four. The fifth added the Eleven changes are proven on real panels. This session added bbox corner ordering. It then deployed and ran
roster-hint removal, merge-chain flattening and `name_claims` reset. This session added bbox corner the tracklet span cap, the shared `GENERIC` tokenizer and the cast gallery together.
ordering.
**The registry is not trustworthy.** `LIM SEONHO`'s 36 assignments cover at least six different men, a **The registry is now roughly right and is worth reading.** Checked by eye, crop by crop, confirmed by the
chibi and a cat, confirmed by eye in the review UI. 22 of those are native resolves and 14 came from user. The lead holds 16 assignments. 14 are him and 2 are art inside a panel, the photograph at `order 17`
reconcile merges, so both mechanisms are wrong. and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and she is a main character
the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2 are `2b1b12a1`. On the
18:07 run the lead alone held 36, covering six men, a chibi and a cat.
Two fixes are written and tested but NOT deployed and NOT run on a GPU, both in the orchestrator repo: So every wrong crop on the lead is one defect, item 7, and not a linker failure. The registry's other
weakness is that its biggest character has no name.
- `link_tracklets` caps a tracklet's panel span (`decisions/identity-bbox.md#tracklet-span-cap`) | metric | 18:07 run | 19:44 run |
- one shared appearance tokenizer drops generic words (`decisions/identity-bbox.md#generic-tokens`) | --- | --- | --- |
| detections | 119 | 119 |
| assignments | 60 | 60 |
| coverage | 57% | 50% |
| tracklets over crops | 12 / 64 | 33 / 72 |
| lead's assignments | 36 | 16 |
| characters after reconcile | 18 | 14 |
On the same 22 crops, 3 tracklets at worst span 30 becomes 9 at worst span 8. Coverage fell because gemma now clears 12 crops instead of naming them from a shortlist that did not
contain the right person. Coverage is still the `has_face` gate plus those refusals, and nothing else.
Coverage was 57% on the 18:07 run, and it is the `has_face` gate and nothing else. All 68 assignments
landed on face-bearing detections, so recall among them is 68 of 71. Degenerate boxes are 0 in 119 and the
merge chain is flat with 22 `merged_from` stamps.
## Next ## Next
+18
View File
@@ -237,3 +237,21 @@ so embedding the face region instead is a small change.
Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22 Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22
detections, embed them, and recompute this matrix. If a threshold appears, embed faces. detections, embed them, and recompute this matrix. If a threshold appears, embed faces.
## The gallery cap drops exactly the rows a tracklet needs {#gallery-cap-drops-the-unnamed}
**Open. Observed on the 19:44 run of 2026-08-12.**
`cast_gallery` orders the live cast named-first, then registry order, and `run_stage_identity` truncates
the tail at `GALLERY_CAP = 10`. Registry order is creation order, so the tail is the anonymous characters
this very run minted — and a later tracklet of a recurring unnamed person is precisely what needs to match
one of those. The chapter has two named rows, so on this run the cap dropped 6 unnamed rows at `p097` and
1 at `p109`, and a tracklet that should have joined one of them can only mint a duplicate instead.
It is capped because every candidate costs at least one reference image in the resolve prompt beside the
query crop. The cap is a VRAM and context budget, not a modelling choice.
**Revisit when** the cast on a chapter routinely exceeds 10 gender-compatible rows, which it already did
here. The fix is to order the gallery by how many assignments each character already holds in this chapter,
so the tail is the rows nobody has matched rather than the rows nobody has named yet. That needs one count
query per tracklet.
+31
View File
@@ -302,3 +302,34 @@ This makes the module docstring's claim true. The self-check links three compati
9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled. 9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled.
Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes. Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.
## The gallery is the live cast, not cosine's top-k {#cast-is-the-gallery}
**Closed, 2026-08-12, not yet run on a GPU.**
`run_stage_identity` built each tracklet's gallery by unioning the members' cosine shortlists, so the
metric that `caveats/audit-open.md#cosine-not-identity` shows cannot separate people decided who gemma was
even allowed to consider. Two different men reach 0.93 on this chapter's crops where the same man reaches
0.96. When the right character fell outside every member's top-5, gemma could only pick a wrong one or
answer NONE, and NONE mints a duplicate.
The cast is small. `tracklets.cast_gallery` builds the gallery from `get_known_characters` instead:
gender-compatible rows only, named first, then registry order. Cosine still shortlists per crop, and those
shortlists are still the linker's candidate-overlap evidence, but they no longer bound the answer.
Three consequences, each deliberate:
- **Every crop now reaches the resolver.** A crop only entered `shortlists` when its cosine top-k was
non-empty, so an early crop seen while the roster was still empty, or one with no gender-compatible row,
was dropped from resolution and could never mint. The guard is gone.
- **The gallery is re-read per tracklet**, so a character an earlier tracklet minted is visible to the
later ones. That closes the `ponytail:` note beside the mint path.
- **Two caps, because the prompt carries images.** `GALLERY_CAP = 10` in `service.py` bounds the
candidates and logs the ids it drops. `worker_vision.REF_BUDGET = 12` spreads reference images across
them, `max(1, min(3, 12 // n))` apiece, so a 9-character cast sends 9 images plus the query instead of
the 27 plus query that 3-apiece would have sent.
Cost: more gemma calls, because the crops that used to be dropped now each get one.
**Not fixed by this.** The gallery being right does not make the crop legible. A back-turned or tiny crop
still has no face for gemma to judge, which is `NEXT.md` item 4.
+20 -2
View File
@@ -1031,9 +1031,16 @@ def build_resolve_prompt(candidates: list) -> str:
) )
REF_BUDGET = 12 # total reference images one /vision/resolve prompt may carry, beside the query crop
def _refs_per_candidate(n: int) -> int:
return max(1, min(3, REF_BUDGET // max(1, n)))
class ResolveInput(BaseModel): class ResolveInput(BaseModel):
crop_uri: str crop_uri: str
candidates: list = [] # gender-gated cosine shortlist [{character_id, name, gender, ...}] candidates: list = [] # gender-gated gallery [{character_id, name, gender, ...}]
session_id: str = "" session_id: str = ""
@@ -1052,9 +1059,13 @@ async def vision_resolve(data: ResolveInput):
return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"} return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"}
crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png") crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png")
refs = [] refs = []
# image budget, not a per-candidate rule. The gallery is the live cast now, not a cosine top-k, so
# 3 references each was 27 images plus the query on a 9-character cast. Spread REF_BUDGET across the
# candidates instead: 3 references while the cast is small, 1 apiece once it is not.
per = _refs_per_candidate(len(data.candidates))
try: try:
for i, candidate in enumerate(data.candidates, 1): for i, candidate in enumerate(data.candidates, 1):
for uri in (candidate.get("reference_image_uris") or [])[:3]: for uri in (candidate.get("reference_image_uris") or [])[:per]:
try: try:
refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png"))) refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png")))
except Exception as e: except Exception as e:
@@ -1189,6 +1200,13 @@ if __name__ == "__main__":
_map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None) _map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None)
assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None
assert "reference images" in rp and "face shape/features first" in rp assert "reference images" in rp and "face shape/features first" in rp
# reference budget: a small gallery keeps 3 refs each, a cast-sized one drops to 1 and stays under
# REF_BUDGET images however big the cast gets.
assert _refs_per_candidate(1) == 3 and _refs_per_candidate(4) == 3
assert _refs_per_candidate(9) == 1 and _refs_per_candidate(0) == 3
# the budget holds up to REF_BUDGET candidates; past that the floor of 1 apiece wins, which is why
# the orchestrator caps the gallery (GALLERY_CAP) rather than relying on this alone.
assert all(n * _refs_per_candidate(n) <= REF_BUDGET for n in range(1, REF_BUDGET + 1))
# dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty. # dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty.
bad = _dialogue_envelope(["p1"], [], parse_failed=True) bad = _dialogue_envelope(["p1"], [], parse_failed=True)