diff --git a/HANDOFF.md b/HANDOFF.md index 36be549..b6db079 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -5,13 +5,16 @@ Live state is in `NEXT.md`. This file is only what this session did. The previou ## Asked -The two files, then "go" on the third GPU cycle. +The two context files, then "go" on the third GPU cycle, then "commit", then "what's next". Then the user +checked the lead's crops in the review UI and said the assignments were wrong. Then "test on one of the +panels first". Then "mind presenting the panels with boxes". Then "so, we didn't fix it properly?". Then +"go with the 1". ## Result -One worker fix, one measurement that killed the plan's first item, one GPU cycle. The cycle proved the new -fix and the fifth session's merge fix. Coverage rose for the first time, 50% to 57%. It is now explained -entirely by the `has_face` gate. +One worker fix proven on a GPU. One measurement that killed the plan's item 1. One GPU cycle. Then the user +found the registry is over-merged, which invalidates the cycle's headline numbers, and two more fixes were +written for it. The third fix the user chose, item 1 of the three options, is NOT started. | metric | 17:38 run | 18:07 run | | --- | --- | --- | @@ -24,65 +27,98 @@ entirely by the `has_face` gate. | chains deeper than one hop | 1 | 0 | | `merged_from` stamps | 9 | 22 | -## The corner-ordering fix +Cycle timings, 18:07-18:13 UTC: vision 116/116 in 3m59s, identity 116/116 in 1m24s, reconcile 18/18 in 50s. -`_bbox_to_pixels` sorts each coordinate pair after clamping. Three lines -(`decisions/identity-bbox.md#bbox-corners-ordered`). The self-check feeds it run B's real -`[226, 417, 130, 551]` and asserts `[117, 688, 203, 909]`. Unsorted the call returns `[203, 688, 117, 909]`, -so the assert fails without the fix. +## Committed and proven on a GPU -The run returned 0 degenerate boxes over 119 detections. The caveat is deleted and the decision is indexed. +`_bbox_to_pixels` sorts each coordinate pair after clamping +(`decisions/identity-bbox.md#bbox-corners-ordered`). 0 degenerate boxes over 119 detections, against 1 in +117. Commit `54bd126` on `restore-runtime`. Its caveat is deleted. -## The measurement that killed item 1 +## Measured, no code -The plan said to score precision with `eval_identity.py` against the 145 rows of `identity_labels`. Every -one of those rows belongs to chapter `8ca8249b`, a different manga with cast "Rico" and "Ikekin", spread -over 81 panels. Chapter `7c944dd4` has zero labels, so -`/review/identity?job_id=778297bc...` returns `labeled: 0, correct: 0, accuracy: null`. +**The 145 ground-truth labels are for a different manga.** Every row in `identity_labels` keys to chapter +`8ca8249b`, cast "Rico" and "Ikekin", 81 panels. Chapter `7c944dd4` has zero, so +`/review/identity?job_id=778297bc...` returns `labeled: 0, accuracy: null`. Scoring `8ca8249b` gives 7/138 +on a run with 44 assignments over 246 panels, predating every fix. -Scoring `8ca8249b` anyway gives 7/138, with 113 rows labelled as a real person and left unassigned. That -chapter's identity run is stale: 44 assignments over 246 panels, predating every fix. The number measures -old code on the wrong chapter. +**Coverage is the `has_face` gate and nothing else.** All 68 assignments landed on face-bearing detections +and none on a gated one. Recall among face-bearing detections is 96%, up from 82%. -So the eval path is proven end to end and the ground truth is absent. A precision number needs a hand pass -over `7c944dd4` through `POST /review/identity/label`, keyed by bbox rather than `local_id`. +**Embedding cosine cannot separate people.** All 22 crop embeddings for the lead, pulled from +`manga//characters/_crops/*.npy`, 1152 dims, L2-normalised. The cat scores up to 0.82 against +men, two different men score 0.93, the highest pair in the matrix is 0.96 +(`caveats/audit-open.md#cosine-not-identity`). This rules out a crop-to-crop cosine link, which was the +fix proposed one message before the measurement. -## What the cycle settled +## Written, tested, NOT deployed and NOT run on a GPU -**Coverage is the gate and nothing else.** All 68 assignments landed on face-bearing detections and none on -a gated one. Recall among face-bearing detections is 96%, up from 82%. The resolver is not losing cast, so -the suspicion in `caveats/audit-open.md#coverage-trend` is closed. +Both in `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`, uncommitted there: -**The merge-chain fix holds under load.** 9 merges this run against 7 last run, and 22 `merged_from` -stamps. No chain is deeper than one hop. +- `tracklets.py`: `link_tracklets` rejects a merge whose group would span more than `window` panels + (`decisions/identity-bbox.md#tracklet-span-cap`). `window` bounded each pair, transitivity was unbounded, + and the lead's 22 native assignments came from 3 tracklets spanning 0, 22 and 30 panels. +- `tracklets.py` + `service.py`: one `appearance_tokens` with a `GENERIC` stopword set, and + `service._appearance_tokens` delegates to it, so reconcile's pre-filter is fixed too + (`decisions/identity-bbox.md#generic-tokens`). Whole chains hung on the word `short`, one pair on the + word `hair`. -## What the cycle opened +On the same 22 crops, candidate overlap forced to pass: 3 tracklets at worst span 30 becomes 9 at worst +span 8. -**The lead may be absorbing.** 36 of 68 assignments, 53%, against 25 of 59 before. `audit_registry.py` -flags it. A protagonist in half the panels looks identical to an over-merge without labels. +## Not started -**Panel 7 got worse.** 5 of its 6 detections carry `has_face = False`, and `person_6`, the one that does, -went unassigned. The previous run bound its colleague. That is the cost of requiring a face, `NEXT.md` -item 4. +**Item 1, which the user chose: stop letting cosine pick the gallery.** `run_stage_identity` builds +`union_cands` from the members' cosine top-k. So a metric that cannot separate people decides who gemma is +even allowed to consider. There are 9 live characters. Send the live cast instead, gender-gated, capped and +logged when truncated. Two traps found while reading it: -## Not done +- `/vision/resolve` sends up to 3 reference images per candidate (`worker_vision.py:1057`), so 9 candidates + is 27 images plus the query. It needs a cap. +- only a crop with a non-empty cosine shortlist enters `shortlists` at all, via the + `if s.get("candidates")` guard in `service.py`. An empty top-k drops the crop from resolution entirely. -- Nothing downstream re-ran. The job is still parked at `dialogue waiting` with every stage below cleared. -- Nothing is committed. All edits are in the working tree. +Also open: nothing downstream re-ran, the job is still parked at `dialogue waiting`, and vision boxes cats +as people and dresses them (`p081`, `p108`). ## Checks ```bash .venv/bin/python worker_vision.py # self-check ok, including the swapped-corner assert -./check_stale.sh # exit 0 before the run and after it +./check_stale.sh # exit 0 before the cycle and after it +/usr/bin/ssh kami@192.168.1.104 "cd /home/kami/docker-apps/manga-infra/orchestrator && python3 tracklets.py && python3 -m pytest -q --ignore=test_api.py" # self-check ok, 118 passed ``` -## Traps confirmed +Every new assert was confirmed to fail with its fix disabled: the span cap returns `[[0, 1, 2]]`, and the +generic-word pair links with `GENERIC` emptied. + +## Next command + +Deploy the two orchestrator fixes and run the fourth cycle. + +```bash +cd /home/kami/Programs/n8n-worker && ./check_stale.sh # must exit 0 +/usr/bin/ssh kami@192.168.1.104 "cd /home/kami/docker-apps/manga-infra/orchestrator && docker compose up -d --build orchestrator" +J=778297bc-e7ce-439d-91b5-8a027060d17f; M=ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e +/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/characters/reset -H 'Content-Type: application/json' -d '{\"manga_id\":\"$M\",\"confirm\":true}'" +# the reset returns restart_identity_worker: true -- honour it, see the traps +/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"$J\",\"stage\":\"vision\"}'" +for S in vision identity reconcile; do /usr/bin/ssh kami@192.168.1.104 "curl -s --max-time 5400 -X POST http://127.0.0.1:9090/stage/run -H 'Content-Type: application/json' -d '{\"job_id\":\"$J\",\"stage\":\"$S\"}'"; done +/usr/bin/ssh kami@192.168.1.104 "docker logs manga-orchestrator --since 1h 2>&1 | grep tracklet" # expect ~30 tracklets, was 12 +/usr/bin/ssh kami@192.168.1.104 "docker exec manga-orchestrator python3 /app/audit_registry.py 7c944dd4-e972-42c7-ba60-9f6939548e80 6" +``` + +Then re-check the lead's crops by eye. A stage counter will not show this defect. + +## Traps confirmed or found - Restart a worker with `tmux send-keys -t manga-workers: C-c`, then re-send the launch line from - `start_workers.sh` with the `MIOPEN_ENV` prefix. `pgrep`-based kills match the calling shell. + `start_workers.sh` with its `MIOPEN_ENV` prefix. A `pgrep` kill matches the calling shell. - `POST /characters/reset` returns `restart_identity_worker: true`. Honour it, the worker caches the registry. - `identity_assignments` has no `method` column. The `merged_from:` stamps live in - `identity_assignment_sources`. -- `chapters` has no `title` column. + `identity_assignment_sources`. `chapters` has no `title` column. The orchestrator image has no numpy. +- A heredoc piped into `docker exec` over `/usr/bin/ssh` silently produces no output. Write the script to a + file, `scp` it, `docker cp` it, then run it. +- Mixing `echo` with a `tar cf -` stream over ssh corrupts the archive. Separate the calls. +- `panel_order` and the panel filename differ by one: `panel_order` 10 is `p009.png`. diff --git a/JOURNAL.md b/JOURNAL.md index 857869e..0b26de1 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -644,3 +644,38 @@ labels, which is item 1. Panel 7 got worse, not better: 5 of its 6 detections carry `has_face = False`, and `person_6`, the one that does, went unassigned. The previous run bound its colleague. That is the cost of requiring a face, item 4. + +## 2026-08-12, sixth session, later — the lead was over-merged, and cosine cannot fix it + +The user checked the lead's crops in the review UI. `LIM SEONHO`'s 36 assignments cover at least six +different men, a chibi, and a cat. So 36 of 68 is a bug, not a protagonist. + +Split by mechanism: 22 native resolves, 14 from reconcile merges. Both are broken, for different reasons. + +**The tracklet linker chained across the chapter.** `window=8` bounds each pair, transitivity was +unbounded. The 22 native assignments came from 3 tracklets spanning 0, 22 and 30 panels, visible as two +confidence blocks of eleven rows at 0.95 and ten at 1.0. Fixed by capping the merged group's span +(`decisions/identity-bbox.md#tracklet-span-cap`). + +**Both tokenizers linked on words that are not identity.** Whole chains hung on `short`; one pair linked on +the word `hair` itself. There is now one tokenizer with a `GENERIC` stopword set, shared with reconcile's +pre-filter (`decisions/identity-bbox.md#generic-tokens`). + +Measured on the same 22 real crops, candidate overlap forced to pass: 3 tracklets at worst span 30 becomes +9 at worst span 8. + +**Then the obvious next fix was ruled out by measurement.** Pulled all 22 crop embeddings from MinIO and +computed the pairwise cosine. The cat scores up to 0.82 against men, two different men score 0.93, and the +highest pair in the matrix is 0.96. No threshold exists, so crop-to-crop cosine is not a link signal, and +the `candidates` overlap condition is close to noise. Filed as +`caveats/audit-open.md#cosine-not-identity`. Inference: identity embeds the whole person box, so cosine +measures scene and style, not face. + +Also seen: vision boxes cats as people and dresses them. `p081` and `p108` are cats, described +`short brown / yellow plaid shirt` and `short brown / white t-shirt`. + +Checks: `tracklets.py` self-check ok, orchestrator 118 passed. Both new asserts confirmed to fail with +their fix disabled. Nothing deployed, nothing re-run on a GPU. + +Artefact: `lead_tracklets.png`, the 22 panels with boxes, insets and tracklet grouping. Session scratchpad +only, not committed. diff --git a/NEXT.md b/NEXT.md index ccfb237..fab707c 100644 --- a/NEXT.md +++ b/NEXT.md @@ -13,38 +13,55 @@ Three GPU cycles ran on 2026-08-12, the last 18:07-18:13 UTC on a reset registry parked at `dialogue waiting`. Everything from `dialogue` down is cleared and stale. All 9 workers are current. -Eight changes are now proven on real panels. The fourth session contributed four. The fifth added the +Eight changes are proven on real panels. The fourth session contributed four. The fifth added the roster-hint removal, merge-chain flattening and `name_claims` reset. This session added bbox corner -ordering. The registry after the 18:07 run holds 18 rows, 9 live and 9 carrying `merged_into`: +ordering. -- `LIM SEONHO`, 36 assignments, the lead, one row, named from the p010 caption. -- `character_b112d4`, 23 assignments, the woman, correct and unnamed. -- `Seonho`, still the wrong man, still named off in-panel text. +**The registry is not trustworthy.** `LIM SEONHO`'s 36 assignments cover at least six different men, a +chibi and a cat, confirmed by eye in the review UI. 22 of those are native resolves and 14 came from +reconcile merges, so both mechanisms are wrong. -Coverage is 57%, from 50%, 61% and 70% before it, and it is now the `has_face` gate and nothing else. All 68 -assignments landed on face-bearing detections and none on a gated one. Recall among face-bearing detections -is 68 of 71, or 96%, up from 82%. Degenerate boxes went from 1 in 117 to 0 in 119, and the merge -chain is flat with 22 assignments stamped `merged_from`. +Two fixes are written and tested but NOT deployed and NOT run on a GPU, both in the orchestrator repo: -The open question moved. The lead holds 36 of 68 assignments, 53%, which `audit_registry.py` flags as -absorbing. Nothing separates a protagonist in half the panels from an over-merge -(`caveats/audit-open.md#coverage-trend`). +- `link_tracklets` caps a tracklet's panel span (`decisions/identity-bbox.md#tracklet-span-cap`) +- one shared appearance tokenizer drops generic words (`decisions/identity-bbox.md#generic-tokens`) + +On the same 22 crops, 3 tracklets at worst span 30 becomes 9 at worst span 8. + +Coverage was 57% on the 18:07 run, and it is the `has_face` gate and nothing else. All 68 assignments +landed on face-bearing detections, so recall among them is 68 of 71. Degenerate boxes are 0 in 119 and the +merge chain is flat with 22 `merged_from` stamps. ## Next -1. **Decide whether the lead absorbing 53% of assignments is real.** It needs labels this chapter lacks. - The resolver is cleared: recall among face-bearing detections is 96%. What is unmeasured is precision. - The lead went from 25 of 59 to 36 of 68 as the merge count rose from 7 to 9. +1. **Deploy the two tracklet fixes and run the fourth cycle.** Neither has touched a GPU. Rebuild the + orchestrator image, reset the registry, run vision/identity/reconcile, then re-check the lead's crops. + Expect roughly 30 tracklets over 64 crops instead of 12, so identity goes from about 1m25s to 3 or 4 + minutes. Watch the lead's assignment count against 36. - The previous plan pointed at `identity_labels` and its 145 rows. Measured this session: every one of - those rows belongs to chapter `8ca8249b`, a different manga. Chapter `7c944dd4` has zero labels, so - `/review/identity` reports `labeled: 0, accuracy: null`. Scoring `8ca8249b` gives 7/138 on an identity - run that predates all eight fixes. The eval plumbing works and the ground truth is missing. - Labelling `7c944dd4` by hand is the only path to a precision number - (`caveats/audit-open.md#coverage-trend`). + Neither fix is sufficient. Bare hair colour still links different men, and the cat still joins its + neighbours. Do not add a crop-to-crop cosine to close that: measured on this run's 22 embeddings, + different people reach 0.93 while the same person reaches 0.96, so no threshold exists + (`caveats/audit-open.md#cosine-not-identity`). + + Agreed next step after the cycle, chosen by the user and not started: **stop letting cosine pick the + gallery.** There are 9 live characters. `run_stage_identity` builds `union_cands` from the members' + cosine top-k shortlists, so a metric that cannot separate people decides who is even considered. Send + the live cast instead, gender-gated, capped and logged when truncated. Note two traps found while + reading it: `/vision/resolve` sends up to 3 reference images per candidate + (`worker_vision.py:1057`), so 9 candidates is 27 images plus the query and needs a cap; and only a crop + with a non-empty cosine shortlist enters `shortlists` at all, so an empty top-k currently drops the crop + from resolution entirely. + + Then, separately, test embedding the FACE box rather than the person box. `face_detect` already finds + the face and pairs it for `has_face`. That is the likely root cause of cosine measuring scene instead of + person, and the test is to re-embed these same 22 detections and recompute the matrix. 2. ~~**Order the corners in `_bbox_to_pixels`.**~~ **Done 2026-08-12, run and verified on a GPU.** 0 degenerate boxes over 119 detections (`decisions/identity-bbox.md#bbox-corners-ordered`). -3. **Detection still names from in-panel text and can hit the wrong body.** p011 and p026 emit +3. **Vision boxes animals as people and dresses them.** `p081` and `p108` are cats, described + `short brown / yellow plaid shirt` and `short brown / white t-shirt`. A detection prompt problem, not a + linker one, and it feeds every stage below. +4. **Detection still names from in-panel text and can hit the wrong body.** p011 and p026 emit `name: "Seonho"` on `person_2` with the roster hint gone. That is the residue of the naming defect and the reason the glasses man is named at all (`decisions/identity-bbox.md#roster-does-not-name`, "not covered"). diff --git a/caveats/CLAUDE.md b/caveats/CLAUDE.md index be09a22..d7fbb8e 100644 --- a/caveats/CLAUDE.md +++ b/caveats/CLAUDE.md @@ -43,3 +43,4 @@ a complaint, so give it one or drop it. | [Vision does not separate a background extra from cast](speaker-attribution.md#extras-as-cast) | 2026-08-12 panel 7 | | [Cast reference profiles are enrolled from wrong crops](speaker-attribution.md#poisoned-reference-set) | 2026-08-12 panel 7 | | [Identity coverage has fallen on every run since the gate landed](audit-open.md#coverage-trend) | before the next downstream run | +| [Embedding cosine on a person crop cannot tell two people apart](audit-open.md#cosine-not-identity) | before building on cosine | diff --git a/caveats/audit-open.md b/caveats/audit-open.md index 99c28c8..7777d8b 100644 --- a/caveats/audit-open.md +++ b/caveats/audit-open.md @@ -212,3 +212,28 @@ The plumbing works, the labels are missing. Precision and recall on the current pass over `7c944dd4` first, via `POST /review/identity/label` per detection. Label against the bbox, not the `local_id`. Vision is non-deterministic and reassigns `local_id` top-to-bottom every run. A label taken before a vision rerun then points at whoever now occupies that slot. + +## Embedding cosine on a person crop cannot tell two people apart {#cosine-not-identity} + +Measured 2026-08-12 on the 18:07 run's 22 crops for `character_2367d70c`, the embeddings pulled from +`manga//characters/_crops/*.npy` and L2-normalised. 1152 dimensions. + +| pair | cosine | +| --- | --- | +| the cat at `p098` against any man | up to 0.82 | +| the red-robe chibi at `p088` against any man | up to 0.75 | +| two visibly different men (`order 60~80`, `88~110`) | 0.93, 0.92 | +| the highest pair in the matrix (`47~51`, one man, one shirt) | 0.96 | + +Different people reach 0.93 and the same person reaches 0.96. No threshold separates them. So a +crop-to-crop cosine link is not available, and the `candidates` overlap condition in `link_tracklets` is +close to noise for the same reason: the top-k is chosen by this metric, so every crop shortlists the same +few rows. + +Inference, not yet tested: identity embeds the whole person box, which holds background, clothing and pose. +Those change between scenes while every crop shares one art style, so cosine measures "manga crop of a +person in an office". `face_detect` already finds the face and pairs it to the person box for `has_face`, +so embedding the face region instead is a small change. + +Revisit trigger: before building anything else on cosine. The test is to crop the faces of these same 22 +detections, embed them, and recompute this matrix. If a threshold appears, embed faces. diff --git a/decisions/CLAUDE.md b/decisions/CLAUDE.md index 98943df..73278ab 100644 --- a/decisions/CLAUDE.md +++ b/decisions/CLAUDE.md @@ -48,3 +48,5 @@ still live belongs in `caveats/`. | [A roster name is a guess, so it never reaches detection](identity-bbox.md#roster-does-not-name) | closed | | [`merged_into` is exactly one hop deep](identity-bbox.md#merge-chains-flatten) | closed | | [`_bbox_to_pixels` orders the corners, because the model sometimes swaps them](identity-bbox.md#bbox-corners-ordered) | closed | +| [A tracklet is bounded by span, not only by pairwise distance](identity-bbox.md#tracklet-span-cap) | closed, GPU pending | +| [A generic word is not identity evidence, and one tokenizer serves both consumers](identity-bbox.md#generic-tokens) | closed, GPU pending | diff --git a/decisions/identity-bbox.md b/decisions/identity-bbox.md index d5fc2f0..77b4d73 100644 --- a/decisions/identity-bbox.md +++ b/decisions/identity-bbox.md @@ -243,3 +243,62 @@ does not already enforce. The self-check feeds the real swapped box in and asser which fails without the sort. Proven on the 18:07 run: 0 degenerate boxes over 119 detections. + +## A generic word is not identity evidence, and one tokenizer serves both consumers {#generic-tokens} + +**Closed, 2026-08-12, not yet run on a GPU.** + +The tracklet linker and reconcile's pair pre-filter each carried their own copy of the appearance +tokenizer, and both linked on words that describe a person without distinguishing them. Measured over the +18:07 run's 22 crops for the lead: + +``` +p056~p057 hair=['short'] cloth=- two different men +p047~p054 hair=['hair'] cloth=- "brown hair" vs "dark hair", the field name in its own value +p109~p110 hair=- cloth=['shirt','white'] +``` + +`tracklets.appearance_tokens` is now the single implementation and subtracts a `GENERIC` set: length words +(`short`, `long`, `medium`, `shoulder`, `length`), `hair`/`haired`, garment words (`shirt`, `top`, +`jacket`, `coat`, `sleeve`), and filler (`plain`, `casual`, `none`, `unknown`). `service._appearance_tokens` +delegates to it, so reconcile's pre-filter is fixed by the same change. That pre-filter is what let the +pink tank top reach `/vision/same` at all. + +Four asserts cover it and all four fail with `GENERIC` emptied. + +Measured effect on those 22 crops, with candidate overlap forced to pass (the shortlists are not stored, +so this is the most permissive assumption and the real split can only be finer): + +| | tracklets | worst span | +| --- | --- | --- | +| as it ran | 3 | 30 panels | +| span cap only | 8 | 8 | +| span cap + `GENERIC` | 9 | 8 | + +The stopwords alone split out `p054`, the beige-jacket man, and `p089`, the red-robe chibi. + +**Not fixed by this.** Bare hair colour still links different men: `order 56` through `62` stay in one +five-crop tracklet on `brown`, and the cat at `p098` still joins its neighbours the same way. The obvious +next lever, a crop-to-crop cosine, is ruled out by `caveats/audit-open.md#cosine-not-identity`. + +## A tracklet is bounded by span, not only by pairwise distance {#tracklet-span-cap} + +**Closed, 2026-08-12, not yet run on a GPU.** + +`window=8` bounded each PAIR, and linking is transitive, so nothing bounded the group. The 18:07 run +resolved 12 tracklets over 64 crops, and the lead's 22 native assignments fell into two of them spanning +panels 47-69 and 80-110. One gemma answer then covered a grey blazer, a denim jacket, a red robe and a cat. +The two blocks are visible in the stored confidences: eleven rows at 0.95 and ten at 1.0. + +`link_tracklets` now rejects a merge whose resulting group would span more than `window` panels: + +```python +orders = [dets[m].get("panel_order", m) for m in gi + gj] +if max(orders) - min(orders) > window: + continue +``` + +This makes the module docstring's claim true. The self-check links three compatible detections at panels 1, +9 and 17 and asserts they do not land in one tracklet; it returns `[[0, 1, 2]]` with the cap disabled. + +Cost: 3 resolve calls become 8 for this character, so identity should go from about 1m25s to 3 or 4 minutes.