diff --git a/JOURNAL.md b/JOURNAL.md index 0b26de1..c23cf3a 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -679,3 +679,47 @@ their fix disabled. Nothing deployed, nothing re-run on a GPU. Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad only, not committed. + +## 2026-08-12, 19:44-19:52 UTC — fourth GPU cycle: the cast is the gallery + +Three fixes ran together for the first time: the tracklet span cap and the shared `GENERIC` tokenizer, +both written last session and never deployed, plus the new one +(`decisions/identity-bbox.md#cast-is-the-gallery`). Job `778297bc`, chapter `7c944dd4`, registry reset, +vision cleared, `vision -> identity -> reconcile`. Vision 116/116 in 3m55s, identity 116/116 in 2m44s, +reconcile 20/20 in 44s. + +| metric | 18:07 run | 19:44 run | +| --- | --- | --- | +| detections | 119 | 119 | +| assignments | 68 | 60 | +| coverage | 57% | 50% | +| tracklets over crops | 12 / 64 | 33 / 72 | +| lead's assignments | 36 | 16 | +| top character's share | -- | 16/60 = 27% | +| characters after reconcile | 18 | 14 | +| minted / cleared | -- | 10 / 12 | + +**Coverage went down and that is the fix working.** gemma cleared 12 crops it used to be forced to name +from a cosine top-k that did not contain the right person. 72 crops entered resolution against 64, because +a crop with an empty cosine shortlist is no longer dropped. + +**Checked by eye, which is the only check that sees this.** Contact sheets of every assigned crop, per +character, confirmed by the user. The lead holds 16 crops. 14 are him and 2 are art inside a panel, the +photograph at `order 17` and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and +she is a main character the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2 +are `2b1b12a1` (`order 31`, `order 33`). Against the 18:07 run, where the lead's 36 covered six different +men, a chibi and a cat. + +Every wrong crop on the lead is one defect: vision treats art inside a panel as the scene. It is not a +linker failure and no identity change will fix it. + +**The cap fired twice and dropped the wrong rows.** `GALLERY_CAP = 10` truncated a 16-row gender-compatible +cast at `p097` and an 11-row one at `p109`. `cast_gallery` orders named first, so what it drops is exactly +the recently minted anonymous rows, which is the population a later tracklet most needs to match against. +Filed as `caveats/audit-open.md#gallery-cap-drops-the-unnamed`. + +Checks: `worker_vision.py` self-check ok, `tracklets.py` self-check ok, orchestrator 118 passed, +`./check_stale.sh` exit 0 before the cycle. Vision and identity workers both restarted, the second because +`/characters/reset` asked for it. + +Artefacts: `sheet_*.png`, one contact sheet per character. Session scratchpad only, not committed. diff --git a/NEXT.md b/NEXT.md index fab707c..05c198d 100644 --- a/NEXT.md +++ b/NEXT.md @@ -9,28 +9,33 @@ The chapter runs end to end. The A/V sync defect is fixed and `chapter.mp4` is r against audio 364.122s at `25/1`. Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels. -Three GPU cycles ran on 2026-08-12, the last 18:07-18:13 UTC on a reset registry. The job is `running` and +Four GPU cycles ran on 2026-08-12, the last 19:44-19:52 UTC on a reset registry. The job is `running` and parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are current. -Eight changes are proven on real panels. The fourth session contributed four. The fifth added the -roster-hint removal, merge-chain flattening and `name_claims` reset. This session added bbox corner -ordering. +Eleven changes are proven on real panels. This session added bbox corner ordering. It then deployed and ran +the tracklet span cap, the shared `GENERIC` tokenizer and the cast gallery together. -**The registry is not trustworthy.** `LIM SEONHO`'s 36 assignments cover at least six different men, a -chibi and a cat, confirmed by eye in the review UI. 22 of those are native resolves and 14 came from -reconcile merges, so both mechanisms are wrong. +**The registry is now roughly right and is worth reading.** Checked by eye, crop by crop, confirmed by the +user. The lead holds 16 assignments. 14 are him and 2 are art inside a panel, the photograph at `order 17` +and the chibi at `order 20`. `character_2b1b12a1` holds 13, all of them her, and she is a main character +the registry never named. `character_f0d4e901` holds 9, of which 7 are her and 2 are `2b1b12a1`. On the +18:07 run the lead alone held 36, covering six men, a chibi and a cat. -Two fixes are written and tested but NOT deployed and NOT run on a GPU, both in the orchestrator repo: +So every wrong crop on the lead is one defect, item 7, and not a linker failure. The registry's other +weakness is that its biggest character has no name. -- `link_tracklets` caps a tracklet's panel span (`decisions/identity-bbox.md#tracklet-span-cap`) -- one shared appearance tokenizer drops generic words (`decisions/identity-bbox.md#generic-tokens`) +| metric | 18:07 run | 19:44 run | +| --- | --- | --- | +| detections | 119 | 119 | +| assignments | 60 | 60 | +| coverage | 57% | 50% | +| tracklets over crops | 12 / 64 | 33 / 72 | +| lead's assignments | 36 | 16 | +| characters after reconcile | 18 | 14 | -On the same 22 crops, 3 tracklets at worst span 30 becomes 9 at worst span 8. - -Coverage was 57% on the 18:07 run, and it is the `has_face` gate and nothing else. All 68 assignments -landed on face-bearing detections, so recall among them is 68 of 71. Degenerate boxes are 0 in 119 and the -merge chain is flat with 22 `merged_from` stamps. +Coverage fell because gemma now clears 12 crops instead of naming them from a shortlist that did not +contain the right person. Coverage is still the `has_face` gate plus those refusals, and nothing else. ## Next diff --git a/caveats/audit-open.md b/caveats/audit-open.md index 7777d8b..3fcc889 100644 --- a/caveats/audit-open.md +++ b/caveats/audit-open.md @@ -237,3 +237,21 @@ so embedding the face region instead is a small change. Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22 detections, embed them, and recompute this matrix. If a threshold appears, embed faces. + +## The gallery cap drops exactly the rows a tracklet needs {#gallery-cap-drops-the-unnamed} + +**Open. Observed on the 19:44 run of 2026-08-12.** + +`cast_gallery` orders the live cast named-first, then registry order, and `run_stage_identity` truncates +the tail at `GALLERY_CAP = 10`. Registry order is creation order, so the tail is the anonymous characters +this very run minted — and a later tracklet of a recurring unnamed person is precisely what needs to match +one of those. The chapter has two named rows, so on this run the cap dropped 6 unnamed rows at `p097` and +1 at `p109`, and a tracklet that should have joined one of them can only mint a duplicate instead. + +It is capped because every candidate costs at least one reference image in the resolve prompt beside the +query crop. The cap is a VRAM and context budget, not a modelling choice. + +**Revisit when** the cast on a chapter routinely exceeds 10 gender-compatible rows, which it already did +here. The fix is to order the gallery by how many assignments each character already holds in this chapter, +so the tail is the rows nobody has matched rather than the rows nobody has named yet. That needs one count +query per tracklet. diff --git a/decisions/identity-bbox.md b/decisions/identity-bbox.md index 77b4d73..4af2d4d 100644 --- a/decisions/identity-bbox.md +++ b/decisions/identity-bbox.md @@ -302,3 +302,34 @@ This makes the module docstring's claim true. The self-check links three compati 9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled. Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes. + +## The gallery is the live cast, not cosine's top-k {#cast-is-the-gallery} + +**Closed, 2026-08-12, not yet run on a GPU.** + +`run_stage_identity` built each tracklet's gallery by unioning the members' cosine shortlists, so the +metric that `caveats/audit-open.md#cosine-not-identity` shows cannot separate people decided who gemma was +even allowed to consider. Two different men reach 0.93 on this chapter's crops where the same man reaches +0.96. When the right character fell outside every member's top-5, gemma could only pick a wrong one or +answer NONE, and NONE mints a duplicate. + +The cast is small. `tracklets.cast_gallery` builds the gallery from `get_known_characters` instead: +gender-compatible rows only, named first, then registry order. Cosine still shortlists per crop, and those +shortlists are still the linker's candidate-overlap evidence, but they no longer bound the answer. + +Three consequences, each deliberate: + +- **Every crop now reaches the resolver.** A crop only entered `shortlists` when its cosine top-k was + non-empty, so an early crop seen while the roster was still empty, or one with no gender-compatible row, + was dropped from resolution and could never mint. The guard is gone. +- **The gallery is re-read per tracklet**, so a character an earlier tracklet minted is visible to the + later ones. That closes the `ponytail:` note beside the mint path. +- **Two caps, because the prompt carries images.** `GALLERY_CAP = 10` in `service.py` bounds the + candidates and logs the ids it drops. `worker_vision.REF_BUDGET = 12` spreads reference images across + them, `max(1, min(3, 12 // n))` apiece, so a 9-character cast sends 9 images plus the query instead of + the 27 plus query that 3-apiece would have sent. + +Cost: more gemma calls, because the crops that used to be dropped now each get one. + +**Not fixed by this.** The gallery being right does not make the crop legible. A back-turned or tiny crop +still has no face for gemma to judge, which is `NEXT.md` item 4. diff --git a/worker_vision.py b/worker_vision.py index 3bbfb4e..fb8efec 100644 --- a/worker_vision.py +++ b/worker_vision.py @@ -1031,9 +1031,16 @@ def build_resolve_prompt(candidates: list) -> str: ) +REF_BUDGET = 12 # total reference images one /vision/resolve prompt may carry, beside the query crop + + +def _refs_per_candidate(n: int) -> int: + return max(1, min(3, REF_BUDGET // max(1, n))) + + class ResolveInput(BaseModel): crop_uri: str - candidates: list = [] # gender-gated cosine shortlist [{character_id, name, gender, ...}] + candidates: list = [] # gender-gated gallery [{character_id, name, gender, ...}] session_id: str = "" @@ -1052,9 +1059,13 @@ async def vision_resolve(data: ResolveInput): return {"character_id": None, "confidence": 0.0, "reason": "no_candidates"} crop = transport.get(data.crop_uri, f"{SHM}/resolve_{uuid.uuid4().hex[:8]}.png") refs = [] + # image budget, not a per-candidate rule. The gallery is the live cast now, not a cosine top-k, so + # 3 references each was 27 images plus the query on a 9-character cast. Spread REF_BUDGET across the + # candidates instead: 3 references while the cast is small, 1 apiece once it is not. + per = _refs_per_candidate(len(data.candidates)) try: for i, candidate in enumerate(data.candidates, 1): - for uri in (candidate.get("reference_image_uris") or [])[:3]: + for uri in (candidate.get("reference_image_uris") or [])[:per]: try: refs.append((i, transport.get(uri, f"{SHM}/ref_{uuid.uuid4().hex[:8]}.png"))) except Exception as e: @@ -1189,6 +1200,13 @@ if __name__ == "__main__": _map = lambda ch: (cands[ch - 1]["character_id"] if isinstance(ch, int) and 1 <= ch <= len(cands) else None) assert _map(1) == "c1" and _map(2) == "c2" and _map(0) is None and _map(9) is None assert "reference images" in rp and "face shape/features first" in rp + # reference budget: a small gallery keeps 3 refs each, a cast-sized one drops to 1 and stays under + # REF_BUDGET images however big the cast gets. + assert _refs_per_candidate(1) == 3 and _refs_per_candidate(4) == 3 + assert _refs_per_candidate(9) == 1 and _refs_per_candidate(0) == 3 + # the budget holds up to REF_BUDGET candidates; past that the floor of 1 apiece wins, which is why + # the orchestrator caps the gallery (GALLERY_CAP) rather than relying on this alone. + assert all(n * _refs_per_candidate(n) <= REF_BUDGET for n in range(1, REF_BUDGET + 1)) # dialogue parsing is fail-loud; an omitted requested panel is partial, never silent-empty. bad = _dialogue_envelope(["p1"], [], parse_failed=True)