Send the live cast to the resolver, not cosine's top-k
The gallery gemma chose from was the union of the tracklet members' cosine shortlists. Cosine cannot separate people on this data, so a metric that cannot tell two men apart decided who gemma was allowed to consider, and the right character was often not on the list. The gallery is now the live cast: gender-compatible rows from get_known_characters, named first, capped, re-read per tracklet so a minted character is visible to later ones. Every crop reaches the resolver now, including one whose cosine top-k was empty; those used to be dropped. worker_vision spreads reference images across a budget instead of sending 3 per candidate, so a 9-character cast costs 9 images and not 27. Ran on a GPU, 19:44-19:52 UTC. The lead's assignments drop from 36 to 16 and 14 of the 16 are him; the other two are art inside a panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+44
@@ -679,3 +679,47 @@ their fix disabled. Nothing deployed, nothing re-run on a GPU.
|
|||||||
|
|
||||||
Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad
|
Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad
|
||||||
only, not committed.
|
only, not committed.
|
||||||
|
|
||||||
|
## 2026-08-12, 19:44-19:52 UTC — fourth GPU cycle: the cast is the gallery
|
||||||
|
|
||||||
|
Three fixes ran together for the first time: the tracklet span cap and the shared `GENERIC` tokenizer,
|
||||||
|
both written last session and never deployed, plus the new one
|
||||||
|
(`decisions/identity-bbox.md#cast-is-the-gallery`). Job `778297bc`, chapter `7c944dd4`, registry reset,
|
||||||
|
vision cleared, `vision -> identity -> reconcile`. Vision 116/116 in 3m55s, identity 116/116 in 2m44s,
|
||||||
|
reconcile 20/20 in 44s.
|
||||||
|
|
||||||
|
| metric | 18:07 run | 19:44 run |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| detections | 119 | 119 |
|
||||||
|
| assignments | 68 | 60 |
|
||||||
|
| coverage | 57% | 50% |
|
||||||
|
| tracklets over crops | 12 / 64 | 33 / 72 |
|
||||||
|
| lead's assignments | 36 | 16 |
|
||||||
|
| top character's share | -- | 16/60 = 27% |
|
||||||
|
| characters after reconcile | 18 | 14 |
|
||||||
|
| minted / cleared | -- | 10 / 12 |
|
||||||
|
|
||||||
|
**Coverage went down and that is the fix working.** gemma cleared 12 crops it used to be forced to name
|
||||||
|
from a cosine top-k that did not contain the right person. 72 crops entered resolution against 64, because
|
||||||
|
a crop with an empty cosine shortlist is no longer dropped.
|
||||||
|
|
||||||
|
**Checked by eye, which is the only check that sees this.** Contact sheets of every assigned crop, per
|
||||||
|
character, confirmed by the user. The lead holds 16 crops. 14 are him and 2 are art inside a panel, the
|
||||||
|
photograph at `order 17` and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and
|
||||||
|
she is a main character the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2
|
||||||
|
are `2b1b12a1` (`order 31`, `order 33`). Against the 18:07 run, where the lead's 36 covered six different
|
||||||
|
men, a chibi and a cat.
|
||||||
|
|
||||||
|
Every wrong crop on the lead is one defect: vision treats art inside a panel as the scene. It is not a
|
||||||
|
linker failure and no identity change will fix it.
|
||||||
|
|
||||||
|
**The cap fired twice and dropped the wrong rows.** `GALLERY_CAP = 10` truncated a 16-row gender-compatible
|
||||||
|
cast at `p097` and an 11-row one at `p109`. `cast_gallery` orders named first, so what it drops is exactly
|
||||||
|
the recently minted anonymous rows, which is the population a later tracklet most needs to match against.
|
||||||
|
Filed as `caveats/audit-open.md#gallery-cap-drops-the-unnamed`.
|
||||||
|
|
||||||
|
Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestrator 118 passed,
|
||||||
|
`./check_stale.sh` exit 0 before the cycle. Vision and identity workers both restarted, the second because
|
||||||
|
`/characters/reset` asked for it.
|
||||||
|
|
||||||
|
Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed.
|
||||||
|
|||||||
@@ -9,28 +9,33 @@ The chapter runs end to end. The A/V sync defect is fixed and `chapter.mp4` is r
|
|||||||
against audio 364.122s at `25/1`.
|
against audio 364.122s at `25/1`.
|
||||||
|
|
||||||
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels.
|
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels.
|
||||||
Three GPU cycles ran on 2026-08-12, the last 18:07-18:13 UTC on a reset registry. The job is `running` and
|
Four GPU cycles ran on 2026-08-12, the last 19:44-19:52 UTC on a reset registry. The job is `running` and
|
||||||
parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are
|
parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are
|
||||||
current.
|
current.
|
||||||
|
|
||||||
Eight changes are proven on real panels. The fourth session contributed four. The fifth added the
|
Eleven changes are proven on real panels. This session added bbox corner ordering. It then deployed and ran
|
||||||
roster-hint removal, merge-chain flattening and `name_claims` reset. This session added bbox corner
|
the tracklet span cap, the shared `GENERIC` tokenizer and the cast gallery together.
|
||||||
ordering.
|
|
||||||
|
|
||||||
**The registry is not trustworthy.** `LIM SEONHO`'s 36 assignments cover at least six different men, a
|
**The registry is now roughly right and is worth reading.** Checked by eye, crop by crop, confirmed by the
|
||||||
chibi and a cat, confirmed by eye in the review UI. 22 of those are native resolves and 14 came from
|
user. The lead holds 16 assignments. 14 are him and 2 are art inside a panel, the photograph at `order 17`
|
||||||
reconcile merges, so both mechanisms are wrong.
|
and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and she is a main character
|
||||||
|
the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2 are `2b1b12a1`. On the
|
||||||
|
18:07 run the lead alone held 36, covering six men, a chibi and a cat.
|
||||||
|
|
||||||
Two fixes are written and tested but NOT deployed and NOT run on a GPU, both in the orchestrator repo:
|
So every wrong crop on the lead is one defect, item 7, and not a linker failure. The registry's other
|
||||||
|
weakness is that its biggest character has no name.
|
||||||
|
|
||||||
- `link_tracklets` caps a tracklet's panel span (`decisions/identity-bbox.md#tracklet-span-cap`)
|
| metric | 18:07 run | 19:44 run |
|
||||||
- one shared appearance tokenizer drops generic words (`decisions/identity-bbox.md#generic-tokens`)
|
| --- | --- | --- |
|
||||||
|
| detections | 119 | 119 |
|
||||||
|
| assignments | 60 | 60 |
|
||||||
|
| coverage | 57% | 50% |
|
||||||
|
| tracklets over crops | 12 / 64 | 33 / 72 |
|
||||||
|
| lead's assignments | 36 | 16 |
|
||||||
|
| characters after reconcile | 18 | 14 |
|
||||||
|
|
||||||
On the same 22 crops, 3 tracklets at worst span 30 becomes 9 at worst span 8.
|
Coverage fell because gemma now clears 12 crops instead of naming them from a shortlist that did not
|
||||||
|
contain the right person. Coverage is still the `has_face` gate plus those refusals, and nothing else.
|
||||||
Coverage was 57% on the 18:07 run, and it is the `has_face` gate and nothing else. All 68 assignments
|
|
||||||
landed on face-bearing detections, so recall among them is 68 of 71. Degenerate boxes are 0 in 119 and the
|
|
||||||
merge chain is flat with 22 `merged_from` stamps.
|
|
||||||
|
|
||||||
## Next
|
## Next
|
||||||
|
|
||||||
|
|||||||
@@ -237,3 +237,21 @@ so embedding the face region instead is a small change.
|
|||||||
|
|
||||||
Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22
|
Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22
|
||||||
detections, embed them, and recompute this matrix. If a threshold appears, embed faces.
|
detections, embed them, and recompute this matrix. If a threshold appears, embed faces.
|
||||||
|
|
||||||
|
## The gallery cap drops exactly the rows a tracklet needs {#gallery-cap-drops-the-unnamed}
|
||||||
|
|
||||||
|
**Open. Observed on the 19:44 run of 2026-08-12.**
|
||||||
|
|
||||||
|
`cast_gallery` orders the live cast named-first, then registry order, and `run_stage_identity` truncates
|
||||||
|
the tail at `GALLERY_CAP = 10`. Registry order is creation order, so the tail is the anonymous characters
|
||||||
|
this very run minted — and a later tracklet of a recurring unnamed person is precisely what needs to match
|
||||||
|
one of those. The chapter has two named rows, so on this run the cap dropped 6 unnamed rows at `p097` and
|
||||||
|
1 at `p109`, and a tracklet that should have joined one of them can only mint a duplicate instead.
|
||||||
|
|
||||||
|
It is capped because every candidate costs at least one reference image in the resolve prompt beside the
|
||||||
|
query crop. The cap is a VRAM and context budget, not a modelling choice.
|
||||||
|
|
||||||
|
**Revisit when** the cast on a chapter routinely exceeds 10 gender-compatible rows, which it already did
|
||||||
|
here. The fix is to order the gallery by how many assignments each character already holds in this chapter,
|
||||||
|
so the tail is the rows nobody has matched rather than the rows nobody has named yet. That needs one count
|
||||||
|
query per tracklet.
|
||||||
|
|||||||
@@ -302,3 +302,34 @@ This makes the module docstring's claim true. The self-check links three compati
|
|||||||
9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled.
|
9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled.
|
||||||
|
|
||||||
Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.
|
Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.
|
||||||
|
|
||||||
|
## The gallery is the live cast, not cosine's top-k {#cast-is-the-gallery}
|
||||||
|
|
||||||
|
**Closed, 2026-08-12, not yet run on a GPU.**
|
||||||
|
|
||||||
|
`run_stage_identity` built each tracklet's gallery by unioning the members' cosine shortlists, so the
|
||||||
|
metric that `caveats/audit-open.md#cosine-not-identity` shows cannot separate people decided who gemma was
|
||||||
|
even allowed to consider. Two different men reach 0.93 on this chapter's crops where the same man reaches
|
||||||
|
0.96. When the right character fell outside every member's top-5, gemma could only pick a wrong one or
|
||||||
|
answer NONE, and NONE mints a duplicate.
|
||||||
|
|
||||||
|
The cast is small. `tracklets.cast_gallery` builds the gallery from `get_known_characters` instead:
|
||||||
|
gender-compatible rows only, named first, then registry order. Cosine still shortlists per crop, and those
|
||||||
|
shortlists are still the linker's candidate-overlap evidence, but they no longer bound the answer.
|
||||||
|
|
||||||
|
Three consequences, each deliberate:
|
||||||
|
|
||||||
|
- **Every crop now reaches the resolver.** A crop only entered `shortlists` when its cosine top-k was
|
||||||
|
non-empty, so an early crop seen while the roster was still empty, or one with no gender-compatible row,
|
||||||
|
was dropped from resolution and could never mint. The guard is gone.
|
||||||
|
- **The gallery is re-read per tracklet**, so a character an earlier tracklet minted is visible to the
|
||||||
|
later ones. That closes the `ponytail:` note beside the mint path.
|
||||||
|
- **Two caps, because the prompt carries images.** `GALLERY_CAP = 10` in `service.py` bounds the
|
||||||
|
candidates and logs the ids it drops. `worker_vision.REF_BUDGET = 12` spreads reference images across
|
||||||
|
them, `max(1, min(3, 12 // n))` apiece, so a 9-character cast sends 9 images plus the query instead of
|
||||||
|
the 27 plus query that 3-apiece would have sent.
|
||||||
|
|
||||||
|
Cost: more gemma calls, because the crops that used to be dropped now each get one.
|
||||||
|
|
||||||
|
**Not fixed by this.** The gallery being right does not make the crop legible. A back-turned or tiny crop
|
||||||
|
still has no face for gemma to judge, which is `NEXT.md` item 4.
|
||||||
|
|||||||
+20
-2
@@ -1031,9 +1031,16 @@ def build_resolve_prompt(candidates: list) -> str:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
REF_BUDGET = 12 # total reference images one /vision/resolve prompt may carry, beside the query crop
|
||||||
|
|
||||||
|
|
||||||
|
def _refs_per_candidate(n: int) -> int:
|
||||||
|
return max(1, min(3, REF_BUDGET // max(1, n)))
|
||||||
|
|
||||||
|
|
||||||
class ResolveInput(BaseModel):
|
class ResolveInput(BaseModel):
|
||||||
crop_uri: str
|
crop_uri: str
|
||||||
candidates: list = [] # gender-gated cosine shortlist [{character_id, name, gender, ...}]
|
candidates: list = [] # gender-gated gallery [{character_id, name, gender, ...}]
|
||||||
session_id: str = ""
|
session_id: str = ""
|
||||||
|
|
||||||
|
|
||||||
@@ -1052,9 +1059,13 @@ async def vision_resolve(data: ResolveInput):
|
|||||||
return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"}
|
return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"}
|
||||||
crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png")
|
crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png")
|
||||||
refs = []
|
refs = []
|
||||||
|
# image budget, not a per-candidate rule. The gallery is the live cast now, not a cosine top-k, so
|
||||||
|
# 3 references each was 27 images plus the query on a 9-character cast. Spread REF_BUDGET across the
|
||||||
|
# candidates instead: 3 references while the cast is small, 1 apiece once it is not.
|
||||||
|
per = _refs_per_candidate(len(data.candidates))
|
||||||
try:
|
try:
|
||||||
for i, candidate in enumerate(data.candidates, 1):
|
for i, candidate in enumerate(data.candidates, 1):
|
||||||
for uri in (candidate.get("reference_image_uris") or [])[:3]:
|
for uri in (candidate.get("reference_image_uris") or [])[:per]:
|
||||||
try:
|
try:
|
||||||
refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png")))
|
refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png")))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@@ -1189,6 +1200,13 @@ if __name__ == "__main__":
|
|||||||
_map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None)
|
_map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None)
|
||||||
assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None
|
assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None
|
||||||
assert "reference images" in rp and "face shape/features first" in rp
|
assert "reference images" in rp and "face shape/features first" in rp
|
||||||
|
# reference budget: a small gallery keeps 3 refs each, a cast-sized one drops to 1 and stays under
|
||||||
|
# REF_BUDGET images however big the cast gets.
|
||||||
|
assert _refs_per_candidate(1) == 3 and _refs_per_candidate(4) == 3
|
||||||
|
assert _refs_per_candidate(9) == 1 and _refs_per_candidate(0) == 3
|
||||||
|
# the budget holds up to REF_BUDGET candidates; past that the floor of 1 apiece wins, which is why
|
||||||
|
# the orchestrator caps the gallery (GALLERY_CAP) rather than relying on this alone.
|
||||||
|
assert all(n * _refs_per_candidate(n) <= REF_BUDGET for n in range(1, REF_BUDGET + 1))
|
||||||
|
|
||||||
# dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty.
|
# dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty.
|
||||||
bad = _dialogue_envelope(["p1"], [], parse_failed=True)
|
bad = _dialogue_envelope(["p1"], [], parse_failed=True)
|
||||||
|
|||||||
Reference in New Issue
Block a user