kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite the handoff for this session, document the restart trap

Browse Source 
HANDOFF.md now covers the fourth session: the four identity changes, the measured
numbers they were decided from, what was deliberately not built, and panel 7's
before-table so the next run has something to compare against.

AGENTS.md gains the worker-restart procedure, because tmux respawn-window -k
leaves a bare shell instead of re-running the command and took both vision and
identity down silently this session.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kami
2026-08-12 19:33:06 +04:00
parent d63a337f15
commit c51871348f
2 changed files with 109 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+21
View File
@@ -1,5 +1,26 @@
# Repository guidance
## Restarting one worker
Python binds a module at process start, so a worker keeps serving the code it loaded. Run
`./check_stale.sh` before any stage run meant to prove a change. It exits non-zero and names the tmux
window for every worker older than its file (`decisions/identity-bbox.md#stale-worker-invalidates`).
**`tmux respawn-window -k` does not re-run the window's command.** It leaves a bare shell and the worker
stays down, silently. Restart by re-sending the `send-keys` line from `start_workers.sh` for that one
worker, then confirm on `/health` and with `check_stale.sh`:
```bash
M="export MIOPEN_USER_DB_PATH=$HOME/.config/miopen MIOPEN_SYSTEM_DB_PATH=$HOME/.config/miopen MIOPEN_FIND_MODE=2 && unset MIOPEN_FIND_ENFORCE"
tmux send-keys -t manga-workers:vision C-c
tmux send-keys -t manga-workers:vision "$M && source $PWD/.venv/bin/activate && python -m uvicorn worker_vision:app --app-dir $PWD --host 0.0.0.0 --port 8002" C-m
curl -s http://127.0.0.1:8002/health && ./check_stale.sh
```
The orchestrator's equivalent is `docker compose up -d --build orchestrator` on homesrv, because the image
bakes its source. That recreates the container, which drops any `docker cp`'d file such as
`audit_registry.py`.
## Video analysis
When diagnosing render motion, transitions, timing, or visual artifacts, use
HANDOFF.md
+88 -74
View File
@@ -1,89 +1,96 @@
# HANDOFF, 2026-08-12 (third session of the day)
# HANDOFF, 2026-08-12 (fourth session of the day)
Live state is in `NEXT.md`. This file is only what this session did. The previous handoff is in
`JOURNAL.md`.
## Asked
Move on from the chapter rebuild. Scope chosen mid-session: reset the registry, rerun `vision`,
`identity` and `reconcile` only, inspect, then decide about the ~80 minutes downstream. After the
inspection the user chose to fix extras-versus-cast before running downstream.
"How much will `has_face` help with character and identity problems?", then in order: fix 2b, fix
extra-versus-cast, 2d.
## Result
The identity rebaseline ran and the numbers moved a long way. One code change on top of it is written
and self-checked but **not yet proven on a GPU run**.
Identity item 2 in `NEXT.md` is now done as far as it can go without a run. Four changes are written,
self-checked and deployed. **None has run on a GPU.** No pipeline stage was executed this session.
| item | was | now |
| --- | --- | --- |
| 2a bbox space | done last session | unchanged |
| 2b abstain and stay abstained | resolver blamed, unverified | fixed in the orchestrator, `f6804e7` + `ffda3df` |
| 2c extra versus cast | open | two consumers gated, `ca46617` + `8b27aec` |
| 2d merge and split | open | examples gone; merge made non-destructive, `00096cc` |
## What each change was, and where the cause turned out to be
**2b was never the resolver.** `/vision/resolve` at `worker_vision.py:1071` already returns
`state="new"` for `choice: 0` and `state="unresolved"` for an out-of-range index. `service.py` read only
`v.get("character_id")` and ran `unassign_identity` on every crop of the tracklet for either one. A
deliberate NONE produced nothing, so an unnamed recurring person was `unknown` on every panel.
The old `ponytail:` comment above that block named the real blocker and was right. Minting needs an
`embedding_uri` the orchestrator cannot compute. siglip is resident in the identity worker and gemma in
the vision worker. Fixed by carrying the embedding, not by adding a GPU pass.
`/identity/resolve` writes each crop's embedding to the crop's key with a `.npy` suffix and returns
`emb_uri`. `tracklets.resolve_outcome` holds the three-way decision as a pure function.
`decisions/identity-bbox.md#none-mints-an-anonymous-character`.
**2c: `build_scene` already dropped extras from the cast list.** `worker_scene.py:63` skips an unassigned
detection, so extras never reached `characters` or `present`. Their **actions** did. `actions` was built
from every detection. That list is the script prompt's content and the verifier's evidence. So a background
extra's "standing at the window" arrived as a fact with no character attached, and the verifier confirmed
it. Gated `worker_scene`'s `actions` and `service._beat` on `has_face is False`. Left
`service._present_characters` ungated on purpose, reasoned out in
`decisions/identity-bbox.md#extras-gate-consumers`.
**2d's worked examples no longer exist.** The registry reset deleted `Lim Seonho` and
`character_afa7623b`. The registry is 8 rows, one named. Built the safety net for the coming rerun
instead, since `reconcile` runs inside it: `merge_characters` sets `merged_into = keeper` rather than
deleting, and stamps every repointed assignment `method = merged_from:<loser_id>`. Roster readers filter
`merged_into IS NULL`, lookup by id does not.
`caveats/audit-open.md#destructive-reconcile` is half-closed.
## Measured, read-only, before writing anything
```
before stale run after
x2 past panel width 47/113 46/110 0/110
coords exactly on 1000 21 27 1
max y2 1000 1000 2307
characters 19 8 8
coverage 26/113 = 23% 77/110 = 70% 77/110 = 70%
top character's share 96% Haeseon 47% 47%
registry: 8 characters, 1 named -> ['Seonho']
detections: 110 assignments: 77 = 70% coverage
spread: Seonho 36, character_565c88 24, character_759e23 9, character_f7a4fd 3,
character_25f682 3, character_d72710 1, character_823aba 1
bbox space: 77/110 exceed 1000, 1 on 1000 -> PIXELS
```
`Choi Haeseon`, which held 25 of 26 assignments, is gone from the registry. Panel 7 against the art:
That killed the assumed cause of 2b. Anonymous ids already recur, so the identity worker's own
pending-promote path gives stable anonymous identities. Only the gemma NONE branch was discarding people.
| box | who | before | after |
| --- | --- | --- | --- |
| `[457, 657, 642, 937]` | Seonho, foreground | nothing | `Seonho` |
| `[669, 591, 763, 822]` | the colleague, unnamed in the story | `Choi Haeseon` 0.9 | `character_f7a4fd` |
| `[428, 386, 496, 526]` | background extra | extra as cast | none |
| `[34, 414, 122, 564]` | background extra | extra as cast | none |
| `[498, 386, 568, 533]` | background extra | extra as cast | `character_d72710` 0.94 |
| `[31, 554, 94, 728]` | background extra | `Lim Seonho` | `Seonho` **1.00** |
## Deliberately not built
## The run that did not count
The first full cycle completed 116/116/20 and reproduced the defect exactly. The fix was not wrong, it
was not loaded:
```
vision worker started 12:00:09
worker_vision.py modified 12:11:35
8113bdf, carrying _bbox_to_pixels 12:16:22
```
Python binds a module once, at process start. The stage reported success and the orchestrator recorded
no error. Cost: one vision + identity + reconcile cycle and a second registry reset to undo the 8
characters it minted. `decisions/identity-bbox.md#stale-worker-invalidates`.
`./check_stale.sh` now compares every running worker's start time against its module mtime and exits
non-zero. Mutation-tested. **Run it before any stage run meant to prove a code change.**
## Written this session
- `db.reset_registry` + `POST /characters/reset` (`confirm=true`). `/stage/clear` spares `characters`
by design (`db.py:790`), so nothing could rebaseline the registry. Also clears
`identity_assignment_sources`, which no stage clear touches and where a leftover `manual` row makes
`assign_identity` refuse the next model assignment (`db.py:663`).
Covered by `test_db.py:TestResetRegistry`. **Orchestrator, committed on homesrv, image rebuilt.**
- `check_stale.sh`, `audit_registry.py` (runs inside `manga-orchestrator`, already `docker cp`'d).
- **`has_face`, the extras gate. This is the part not yet proven.** Fixing the coordinate space made
extras worse. With crops finally on their subjects, an extra bound to the lead at 1.00. `/vision` stamps
`has_face` per character using `face_detect` + `_pair_faces_to_present`, so containment and its
margin are the rules the speaker path already uses. `worker_identity.py` skips `has_face is False`
before it crops or embeds. Fails open on a missing or raising detector, and gates on `is False` so an
older vision blob behaves as before. `decisions/identity-bbox.md#face-gates-enrollment`.
- **The unmerge path and the split.** No wrong merge has been seen since the crops were fixed. Undoing
one today is a hand-written SQL walk of the two records above. Revisit trigger is in the caveat.
- **`service._present_characters` gating.** An extra picked as speaker already resolves to unknown. A
character drawn from behind has no face box. Gating would delete a real speaker from the only list that
can attribute their line.
- **Vision still emits extras into the blob.** Deliberate, so the audit can see what was gated.
## Checks
```bash
.venv/bin/python worker_vision.py # includes the has_face gate + both fail-open paths
.venv/bin/python worker_identity.py
./check_stale.sh # exits non-zero if a worker predates its file
cd /home/kami/docker-apps/manga-infra/orchestrator && python3 -m pytest -q --ignore=test_api.py # 115 passed, on homesrv
.venv/bin/python worker_identity.py # ok
.venv/bin/python worker_scene.py # ok, 3 new cases on the actions gate
./check_stale.sh # exit 0, all 9 workers current
/usr/bin/ssh kami@192.168.1.104 "cd /home/kami/docker-apps/manga-infra/orchestrator && python3 -m pytest -q --ignore=test_api.py" # 116 passed
```
Deployed and verified inside the running container: `resolve_outcome` returns `mint`, `_beat` drops a
faceless detection, `merged_into` exists on the live database with 0 rows merged. Vision, identity and
scene workers restarted.
## Next command
The `has_face` gate has never run on a GPU. Restart vision and identity, reset, rerun, and check whether
the two wrong bindings on panel 7 disappear without taking Seonho with them.
Four changes ride one GPU cycle. Coverage is 70% and is the number to beat. A gate that abstains too hard
shows up there before it shows up on panel 7. Watch the identity log line for `minted N anonymous`.
```bash
cd /home/kami/Programs/n8n-worker && ./check_stale.sh # restart anything it flags
cd /home/kami/Programs/n8n-worker && ./check_stale.sh # must exit 0
J=778297bc-e7ce-439d-91b5-8a027060d17f; M=ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/characters/reset -H 'Content-Type: application/json' -d '{\"manga_id\":\"$M\",\"confirm\":true}'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"$J\",\"stage\":\"vision\"}'"
@@ -91,20 +98,27 @@ for S in vision identity reconcile; do /usr/bin/ssh kami@192.168.1.104 "curl -s
/usr/bin/ssh kami@192.168.1.104 "docker exec manga-orchestrator python3 /app/audit_registry.py 7c944dd4-e972-42c7-ba60-9f6939548e80 6"
```
Watch for coverage collapsing. 70% is the number to beat. A gate that abstains too hard shows up there
before it shows up on panel 7. Restart the identity worker after every reset: it caches the known list
in-process and only invalidates on enrollment.
Panel 7 (`panel_index 6`) is the worked example. Before this session's changes:
## Gone
```
person_1 [457, 657, 642, 937] Seonho 0.00 foreground, correct
person_2 [669, 591, 763, 822] character_f7a4fd 0.00 the unnamed colleague, correct
person_3 [428, 386, 496, 526] -- none -- extra
person_4 [498, 386, 568, 533] character_d72710 0.94 extra, WRONG
person_5 [31, 554, 94, 728] Seonho 1.00 extra bound to the lead, WRONG
person_6 [34, 414, 122, 564] -- none -- extra
```
The rebuilt `chapter.mp4` and all 49 clips were deleted by the cascade from `/stage/clear vision`. The
user chose not to keep a copy. Nothing downstream of `reconcile` exists for this job now.
`person_4` and `person_5` are what `has_face` must remove without taking `person_1` or `person_2` with
them. Restart the identity worker after every reset: it caches the known list in-process and only
invalidates on enrollment.
## Traps confirmed again
## Traps confirmed or found
- A stage reporting `completed 116/116` says the code ran, not that the current code ran.
- The SSHFS mount at `/mnt/server/home/kami` was absent and needed remounting by the user, since the
mountpoint needs root.
- The orchestrator image bakes its source. Editing the repo on homesrv does nothing until
`docker compose up -d --build orchestrator`.
- `rtk grep` searches files, not stdin. Piping into it silently searches the repo instead.
- **`tmux respawn-window -k` does not re-run the window command.** It leaves a bare shell. Both vision and
identity sat dead for two minutes before an empty `/health` caught it. Now in `AGENTS.md`.
- `docker compose up --build orchestrator` recreates the container and drops any `docker cp`'d file, so
`audit_registry.py` needs re-copying after every rebuild.
- Two test assertions asserted the old destructive merge (`test_db.py:235`, `test_merge_refs.py:37`). They
were rewritten, not deleted: the invariant changed on purpose.
- The orchestrator image bakes its source. Editing the repo on homesrv does nothing until the rebuild.