kami/manga-recap-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Ship each crop's embedding so a resolver NONE can mint

Browse Source 
Two identity fixes, neither yet run on a GPU.

has_face, the extras gate: fixing the bbox coordinate space made extras worse,
because crops finally landed on their subjects and a background extra bound to
the lead at 1.00. /vision stamps has_face per character using face_detect and
_pair_faces_to_present, so containment and its margin match the speaker path.
Identity skips has_face is False before it crops or embeds. Fails open on a
missing or raising detector, and gates on `is False` so an older vision blob
behaves as before.

The NONE mint: /identity/resolve now writes each crop's embedding to the crop's
key with a .npy suffix and returns emb_uri. The orchestrator cannot compute an
embedding of its own (siglip and gemma cannot both be resident), which is why it
was clearing a deliberate NONE instead of minting an anonymous character.

Also: audit_registry.py, and the tmux respawn-window trap, which
leaves a bare shell rather than re-running the window command.

decisions/identity-bbox.md#face-gates-enrollment
decisions/identity-bbox.md#none-mints-an-anonymous-character

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
kami
2026-08-12 19:07:56 +04:00
parent 7f25dc43e4
commit 491dad1c67
9 changed files with 476 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
HANDOFF.md
+77 -97
View File
@@ -1,130 +1,110 @@
# HANDOFF, 2026-08-12 (second session of the day)
# HANDOFF, 2026-08-12 (third session of the day)
Live state is in `NEXT.md`. This file is only what this session did. The previous handoff's content is
now in `JOURNAL.md` and in `decisions/`.
Live state is in `NEXT.md`. This file is only what this session did. The previous handoff is in
`JOURNAL.md`.
## Asked
Get up to speed from the previous handoff. Then: rebuild the chapter, and what about characters.
Move on from the chapter rebuild. Scope chosen mid-session: reset the registry, rerun `vision`,
`identity` and `reconcile` only, inspect, then decide about the ~80 minutes downstream. After the
inspection the user chose to fix extras-versus-cast before running downstream.
## Result
Two things landed. The A/V gap is fixed and the rebuilt `chapter.mp4` is in the bucket. The `bbox`
coordinate space is settled with measurements instead of comments, and converted at the boundary.
One correction to carry forward. **The first fix of the day named the wrong cause.** Commit `1457556`
claimed the xfade offset drift was the shipped 72.7s gap. The rebuild came out byte-identical to the
broken file, which disproved it. Both are real defects. Only the second one shipped.
## The chapter, rebuilt
`s3://video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/chapter.mp4`
The identity rebaseline ran and the numbers moved a long way. One code change on top of it is written
and self-checked but **not yet proven on a GPU run**.
```
before v=436.392 a=363.675 nb_frames=9902 avg_frame_rate=22.69 gap +72.72
after v=364.120 a=364.122 nb_frames=9101 r_frame_rate=25/1 gap -0.002
before stale run after
x2 past panel width 47/113 46/110 0/110
coords exactly on 1000 21 27 1
max y2 1000 1000 2307
characters 19 8 8
coverage 26/113 = 23% 77/110 = 70% 77/110 = 70%
top character's share 96% Haeseon 47% 47%
```
Cause: `assemble` sends an all-`cut` chapter down a `concat` demuxer with `-c copy`. That path writes
the output in the **first** input's `time_base` and reinterprets every later packet in it. 14 of the 49 clips
are `30/1` at `1/15360`, the other 35 are `25/1` at `1/12800`, so those 14 play `15360/12800 = 1.2` too
long with their audio untouched. `collage_cmd` hardcoded `-r 30`.
`decisions/chapter-assembly.md#mixed-rate-stream-copy`.
`Choi Haeseon`, which held 25 of 26 assignments, is gone from the registry. Panel 7 against the art:
Fixes: `collage_cmd` emits `-r FPS`. `assemble` probes `_fps_of` across the clips and routes mixed rates
through `_assemble_batched`, whose branches both normalize with `fps={FPS}`.
| box | who | before | after |
| --- | --- | --- | --- |
| `[457, 657, 642, 937]` | Seonho, foreground | nothing | `Seonho` |
| `[669, 591, 763, 822]` | the colleague, unnamed in the story | `Choi Haeseon` 0.9 | `character_f7a4fd` |
| `[428, 386, 496, 526]` | background extra | extra as cast | none |
| `[34, 414, 122, 564]` | background extra | extra as cast | none |
| `[498, 386, 568, 533]` | background extra | extra as cast | `character_d72710` 0.94 |
| `[31, 554, 94, 728]` | background extra | `Lim Seonho` | `Seonho` **1.00** |
A second, latent defect on the transition path was fixed and committed separately. `_xfade_chain` took
offsets from `format=duration`, which is `max(video, audio)`. The accumulator crept past the end of its
input, and ffmpeg discarded whole clips at `rc 0` with nothing on stderr.
`decisions/chapter-assembly.md#offsets-from-min-stream`.
## The run that did not count
The single-item passthrough theory from the previous handoff is dead, recorded void at
`decisions/chapter-assembly.md#passthrough-innocent`. The one-path rewrite it recommended is not needed.
**Still open here.** The 14 clips in the bucket are still 30fps. Assembly normalizes them, so the chapter
is correct, but the fast stream-copy path stays off for this chapter until `render` re-runs. Nobody has
watched the rebuilt video yet. The 2:52 slide transition and the 28s static hold from 2:24 were both
supposed to be re-judged after the sync fix.
## Characters: the `bbox` space, settled
All 113 detections, straight from `/review/identity`:
| test | result |
| --- | --- |
| boxes with `x2` past the 900px panel width | **47 of 113** |
| boxes with `y2` past 1000, on panels 1257 to 2307px tall | **0 of 113** |
| boxes clamped at exactly 1000 | 21 in x, 5 in y |
| coordinate range over every box | `[0, 1000]` |
Gemma's native 0-1000 grid. Not pixels. `worker_vision.py` prompt text and the old
`worker_identity.py:91` comment both claimed pixels and both were wrong.
`/vision` now calls `_bbox_to_pixels(characters, w, h)` before returning. Four consumers are fixed at
once: `_crop_bbox` in identity, `_pair_faces_to_present`, the set-of-mark boxes, and the review UI's
client-side crop. The pairing one was comparing real pixel face boxes against 0-1000 character boxes,
which is the likely mechanism behind 7 `unknown` out of 7 `som_face` lines.
`decisions/identity-bbox.md#bbox-is-normalized`.
Checked by eye on panel 7, not just asserted. Five of six converted boxes land on their subject. That
includes `person_5`, who is Seonho in the foreground with headphones and carried no identity. `person_1`
still frames an empty window mullion, which is `caveats/speaker-attribution.md#extras-as-cast`.
Converted boxes for panel 7, for whoever redraws the overlay:
The first full cycle completed 116/116/20 and reproduced the defect exactly. The fix was not wrong, it
was not loaded:
```
person_1 [226, 414, 286, 553] window mullion, nobody
person_2 [ 34, 558, 106, 749] background extra, was assigned Lim Seonho
person_3 [428, 384, 494, 533] background extra
person_4 [498, 389, 561, 549] background extra
person_5 [460, 657, 631, 939] Seonho, foreground. was assigned nothing
person_6 [646, 591, 767, 794] the colleague, no name in the story. was assigned Choi Haeseon at 0.9
vision worker started 12:00:09
worker_vision.py modified 12:11:35
8113bdf, carrying _bbox_to_pixels 12:16:22
```
**The registry is unchanged and still wrong.** Every stored box, embedding and `ref_image_uris` was
enrolled from the wrong space. `vision` and `identity` have to re-run before any of it means anything,
and that is GPU work nobody authorized. `caveats/speaker-attribution.md#bbox-wrong-space` is marked
resolved with the rerun pending.
Python binds a module once, at process start. The stage reported success and the orchestrator recorded
no error. Cost: one vision + identity + reconcile cycle and a second registry reset to undo the 8
characters it minted. `decisions/identity-bbox.md#stale-worker-invalidates`.
`./check_stale.sh` now compares every running worker's start time against its module mtime and exits
non-zero. Mutation-tested. **Run it before any stage run meant to prove a code change.**
## Written this session
- `db.reset_registry` + `POST /characters/reset` (`confirm=true`). `/stage/clear` spares `characters`
by design (`db.py:790`), so nothing could rebaseline the registry. Also clears
`identity_assignment_sources`, which no stage clear touches and where a leftover `manual` row makes
`assign_identity` refuse the next model assignment (`db.py:663`).
Covered by `test_db.py:TestResetRegistry`. **Orchestrator, committed on homesrv, image rebuilt.**
- `check_stale.sh`, `audit_registry.py` (runs inside `manga-orchestrator`, already `docker cp`'d).
- **`has_face`, the extras gate. This is the part not yet proven.** Fixing the coordinate space made
extras worse. With crops finally on their subjects, an extra bound to the lead at 1.00. `/vision` stamps
`has_face` per character using `face_detect` + `_pair_faces_to_present`, so containment and its
margin are the rules the speaker path already uses. `worker_identity.py` skips `has_face is False`
before it crops or embeds. Fails open on a missing or raising detector, and gates on `is False` so an
older vision blob behaves as before. `decisions/identity-bbox.md#face-gates-enrollment`.
## Checks
Every self-check runs from the repo root and passes:
```bash
.venv/bin/python worker_render.py # about 4 minutes, real ffmpeg
.venv/bin/python worker_vision.py
.venv/bin/python worker_vision.py # includes the has_face gate + both fail-open paths
.venv/bin/python worker_identity.py
./check_stale.sh # exits non-zero if a worker predates its file
cd /home/kami/docker-apps/manga-infra/orchestrator && python3 -m pytest -q --ignore=test_api.py # 115 passed, on homesrv
```
Three checks were added, because the existing ones passed all the way through both shipped defects:
- `_fps_of(collage clip) == "25/1"`, on a real collage encode. This is the one that would have caught the
mixed-rate bug at the source.
- three clips whose audio outlasts their video by 0.4s, assembled through the xfade branch. Mutation
tested by restoring `_audio_dur`: fires with `video=1.80 audio=3.56 expected=3.56`.
- `_bbox_to_pixels` against panel 7's real `person_5` box, asserting the result covers the lower half of a
1650px panel, which a raw grid value cannot.
`_check_assembled` now runs after every encode on both paths, because ffmpeg returns 0 while dropping
whole inputs.
## Next command
Watch the rebuilt chapter before anything else. That is what found every real defect so far.
The `has_face` gate has never run on a GPU. Restart vision and identity, reset, rerun, and check whether
the two wrong bindings on panel 7 disappear without taking Seonho with them.
```bash
/usr/bin/ssh kami@192.168.1.104 'mc cat homesrv/video/ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e/7c944dd4-e972-42c7-ba60-9f6939548e80/chapter.mp4' > chapter.mp4
cd /home/kami/Programs/n8n-worker && ./check_stale.sh # restart anything it flags
J=778297bc-e7ce-439d-91b5-8a027060d17f; M=ef105a86-4b7e-4ac4-b45c-b7d83b8f5b5e
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/characters/reset -H 'Content-Type: application/json' -d '{\"manga_id\":\"$M\",\"confirm\":true}'"
/usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"$J\",\"stage\":\"vision\"}'"
for S in vision identity reconcile; do /usr/bin/ssh kami@192.168.1.104 "curl -s --max-time 5400 -X POST http://127.0.0.1:9090/stage/run -H 'Content-Type: application/json' -d '{\"job_id\":\"$J\",\"stage\":\"$S\"}'"; done
/usr/bin/ssh kami@192.168.1.104 "docker exec manga-orchestrator python3 /app/audit_registry.py 7c944dd4-e972-42c7-ba60-9f6939548e80 6"
```
Then, with a go-ahead, the vision and identity rerun in `NEXT.md` item 1.
Watch for coverage collapsing. 70% is the number to beat. A gate that abstains too hard shows up there
before it shows up on panel 7. Restart the identity worker after every reset: it caches the known list
in-process and only invalidates on enrollment.
## Traps confirmed again this session
## Gone
- `tmux` session `manga-workers` was gone and both systemd units were inactive. `./start_workers.sh`
starts 9 windows. The render worker is window 9 and must be restarted by hand to pick up an edit.
- Plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`.
- `cd $dir && .venv/bin/python` fails, because the venv path is relative to the repo root. Use the
absolute interpreter path when the working directory is elsewhere.
- The whole assembly investigation ran offline on 49 downloaded clips with no GPU and no orchestrator.
Re-download with the command in `NEXT.md`.
The rebuilt `chapter.mp4` and all 49 clips were deleted by the cascade from `/stage/clear vision`. The
user chose not to keep a copy. Nothing downstream of `reconcile` exists for this job now.
## Traps confirmed again
- A stage reporting `completed 116/116` says the code ran, not that the current code ran.
- The SSHFS mount at `/mnt/server/home/kami` was absent and needed remounting by the user, since the
mountpoint needs root.
- The orchestrator image bakes its source. Editing the repo on homesrv does nothing until
`docker compose up -d --build orchestrator`.
- `rtk grep` searches files, not stdin. Piping into it silently searches the repo instead.