diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..01fb719 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,65 @@ +# HANDOFF, 2026-08-11 session + +Live state lives in `NEXT.md`. This file is only what this session did. + +## Asked + +1. Add the buckets to rustfs. +2. Fix the bucket saving on the orchestrator side. +3. Run the title recap. + +Mid-session: check `manga.kvmx.ru`, and cross-check whether the run got dialogue and identities right. + +## Changed + +Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`: + +- `worker_crop.py` panels -> `s3://panels/` +- `worker_tts.py` wavs -> `s3://audio/` +- `worker_layers.py` layers -> `s3://layers/` +- `worker_render.py` clips and `chapter.mp4` -> `s3://video/` +- `decisions/storage-layout.md`, `caveats/speaker-attribution.md`, indexes, `JOURNAL.md`, `NEXT.md` + +Orchestrator, `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator`: + +- `minio_layout.py` per-artifact bucket constants, `BUCKETS`, `parse_key` accepts any of them +- `service.py` `_s3_delete_prefix` takes `/`, `_stage_s3_prefixes` no longer slices, + new `_ensure_buckets` in the lifespan +- `test_minio_layout.py` updated, one test added + +Infrastructure, homesrv: + +- six buckets created on `rfs` (rustfs, `127.0.0.1:9010`) +- `manga-fetch` started, exited 2 weeks, `/job/create` fails without it +- `manga-web` started, exited 2 weeks, `manga.kvmx.ru` -> nginx -> `localhost:8083` +- orchestrator rebuilt and restarted +- `mc` aliases `mio` and `rfs` now exist on homesrv + +## Measured + +Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, +"Teto X Egen", 116 panels. + +- 109 orchestrator tests pass. `worker_crop.py`, `worker_tts.py`, `worker_render.py` self-checks pass. + `worker_layers.py` self-check fails on the missing `legacy/qwen_layered_workflow.json`, which + predates this session. +- buckets after the run: `raw` 64MiB/79, `panels` 101MiB/116, `manga` 366MiB/491, `audio` `layers` + `video` still empty because the run had not reached those stages. +- stage times: crop 85s, vision ~4min, identity ~1min, reconcile ~7min, dialogue ~8min. +- 24 of 81 speech lines resolve to a named character, 30%. +- 26 of 113 detected people got an identity, 23%, and 25 of the 26 went to one character. +- 3 of 3 sampled two-character panels attribute both speakers to the wrong person. + +## Open + +- The run stopped in `direct` at 113/116. `scene script tts layers render assemble` never ran. Resume: + + ```bash + ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" + ``` + +- Speaker attribution is wrong in multi-character panels and the fix is not written + (`caveats/speaker-attribution.md`). No code was changed for it. +- Objects from the 2026-07-17 run still sit under `manga///{pages,panels,audio, + layers,clips}`. Nothing reads them. They are the rollback, not live data. +- `rustfs` holds empty buckets. No mirror, no cutover. diff --git a/JOURNAL.md b/JOURNAL.md index fadcf44..1251b6b 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -67,3 +67,33 @@ Also patched `~/scripts/migrate-kvmx-https.sh:54` on homesrv. `need_stream_modul `sudo -n nginx -V` and `sudo -n nginx -T`; the NOPASSWD rule covers only `nginx -t`, so it reported "stream module is not loaded" whenever it meant "could not ask for a password". Both checks now run without sudo. `bash -n` passes and both conditions evaluate true. + +## 2026-08-11 Per-artifact buckets, rustfs buckets, baseline chapter run [#116] + +Command: `mc mb` on rustfs, `docker compose up -d --build orchestrator`, `pytest -q --ignore=test_api.py`, +`./start_workers.sh`, then `/job/create` + `/stage/clear` + `/job/resume` for chapter +`7c944dd4-e972-42c7-ba60-9f6939548e80` of "Teto X Egen" as job `778297bc-e7ce-439d-91b5-8a027060d17f`. +Outcome: partial. Storage split landed and is proven by the run. The run itself was still in `direct` +when the session ended. +Produced: `decisions/storage-layout.md`, `caveats/speaker-attribution.md`, 109 orchestrator tests pass. + +Artifacts now split one bucket per class instead of everything under `manga` +(`decisions/storage-layout.md#bucket-per-artifact`). Both MinIO and rustfs hold all six buckets. The +run put 79 pages in `raw` and 116 panel crops in `panels`, so the split works end to end. + +Two containers on homesrv had been dead for two weeks and blocked the work. `manga-fetch` was exited, +so `/job/create` failed with `httpx.ConnectError`; `manga-web` was exited, so `manga.kvmx.ru` had +nothing behind it on port 8083. Both started with `docker compose up -d`. Neither is related to the +storage change. Neither was caught by any check, because nothing watches these containers. + +Stage timings, 116 panels: crop 85s, vision ~4min, identity ~1min, reconcile ~7min for 35 pairs, +dialogue ~8min. Faster than the 2026-07-17 run at 75 panels. The webtoon crop that 500'd in July +succeeded this time. + +Quality cross-check against the panel images, the point of the run. Dialogue text extraction is +accurate. Character detection is accurate. Speaker attribution is not: three of three sampled +two-character panels attribute both speakers to the wrong person, always swapped +(`caveats/speaker-attribution.md#tail-is-not-geometry`). 24 of 81 speech lines resolve to a named +character, which is the Phase 1 headline metric at 30%, and the sample says that 30% is not +trustworthy. 26 of 113 detected people got an identity, and 25 of those 26 went to one character that +turns out to cover two different women. diff --git a/NEXT.md b/NEXT.md index b7aa0fd..7bdc421 100644 --- a/NEXT.md +++ b/NEXT.md @@ -23,23 +23,42 @@ the rebuild exposed. ## Next -1. Run one labeled chapter end to end. Record the baseline numbers from `ROADMAP.md`, above all the - share of narrated lines with a named speaker. That number is the check on the largest Phase 1 fix. -2. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work +1. Fix speaker attribution. The baseline run has been made and the metric it produced cannot be + trusted (`caveats/speaker-attribution.md#tail-is-not-geometry`). Three of three sampled + two-character panels swap the speakers, and the `tail` provenance label is stamped on guesses at + confidence 1.0. Smallest honest first step: stop labelling a guess `tail`, and return `unknown` + when two or more characters are present. +2. Rerun the chapter and re-read the named-speaker share. Only then is the Phase 1 headline number + real. +3. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work (`caveats/audit-open.md#sqlite-locking`). +## The 2026-08-11 chapter run + +Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 +panels. It was still in `direct` at 113/116 when the session ended, with `scene script tts layers +render assemble` unrun. Resume it, or read where it got to: + +```bash +ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +``` + +Numbers and the quality read are in `JOURNAL.md` and `caveats/speaker-attribution.md`. + ## Storage and viewer, tasks #116/#117 [#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had been dead since 2026-07-19 on an arm64 digest pin. Details in `JOURNAL.md`. -[#116] is staged and unfinished. `rustfs` runs on `127.0.0.1:9010/9011` with empty buckets. The next -step is the `mc` mirror of the six buckets, then verify object counts and sizes against MinIO, then -decide on cutover. Nothing is repointed yet and MinIO still serves every read and write. +[#116] is closer but not cut over. Artifacts now split one bucket per class +(`decisions/storage-layout.md#bucket-per-artifact`), and both MinIO and `rustfs` hold all six buckets. +`rustfs` on `127.0.0.1:9010/9011` is still empty and nothing is repointed, so MinIO serves every read +and write. Remaining: `mc mirror` the live buckets, verify counts and sizes, then decide on cutover +(`decisions/storage-layout.md#rustfs-staged`). -Two things to weigh before cutover, neither settled. RustFS is `1.0.0-beta.12`, labeled -`build-type=prerelease`. Swapping storage before the baseline chapter run also adds a variable to the -run that is meant to produce the baseline. +Two containers on homesrv had been dead for two weeks and are now running. `manga-fetch` is the one +`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them. ## Open questions diff --git a/caveats/CLAUDE.md b/caveats/CLAUDE.md index 459f244..eb56780 100644 --- a/caveats/CLAUDE.md +++ b/caveats/CLAUDE.md @@ -32,3 +32,6 @@ a complaint, so give it one or drop it. | [MinIO credentials are hardcoded in committed source](audit-open.md#hardcoded-credentials) | AUDIT.md | | [Assemble marks a job completed with no clips](audit-open.md#empty-assemble) | AUDIT.md | | [Reviewer timestamps drift against the crossfaded video](audit-open.md#timeline-drift) | AUDIT.md | +| [`speaker_method="tail"` never reads a tail](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run | +| [One character id covers two different women](speaker-attribution.md#identity-over-merge) | 2026-08-11 run | +| [The character registry carries five weeks of wrong names](speaker-attribution.md#registry-pollution) | 2026-08-11 run | diff --git a/caveats/speaker-attribution.md b/caveats/speaker-attribution.md new file mode 100644 index 0000000..72331c8 --- /dev/null +++ b/caveats/speaker-attribution.md @@ -0,0 +1,55 @@ +# speaker-attribution + +Limits found by cross-checking the 2026-08-11 chapter run against the panel images. + +## `speaker_method="tail"` never reads a tail {#tail-is-not-geometry} + +`worker_vision.py:376` (`_annotate_speaker_methods`) stamps `tail` on any line whose `speaker` matches +a `local_id` present in the panel. No balloon geometry is consulted. The caller keeps gemma's +confidence, usually 1.0. An unverified model guess thus carries the highest-trust provenance in the +pipeline. The review UI and the flag rules both believe it. + +Measured on job `778297bc`, chapter `7c944dd4`: 31 of 81 speech lines are `tail` with two or more +characters present. Three two-character panels were checked against the art, and all three are wrong, +each with the two speakers swapped: + +| panel key | line | truth | pipeline | +| --- | --- | --- | --- | +| `p010.png` | "…definitely an Egen guy, Seonho!" | the woman | Seonho, the person addressed | +| `p010.png` | "Y-you think so?" | Seonho | Choi Haeseon | +| `p012.png` | "Want me to send you the link?" | the woman | the man | +| `p059.png` | "If team leader Choi says it, it must be true." | the man | Choi Haeseon | + +The last row needs no image: the line refers to Choi in the third person and is attributed to Choi. + +The grounded path exists and almost never fires. Only 2 of 81 speech lines got `som_face`, because +attribution marks need `face_detect` boxes that survive `_pair_faces_to_present`, and these webtoon +close-ups rarely produce them. Inference, not measured: the face detector was not instrumented. + +`worker_vision.py:356` already carries the `ponytail:` note that multi-character attribution needs +per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads carry balloon fill and +tail tips and are unused. + +**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric +(`ROADMAP.md`). It reads 30% on this run, and the sample says that 30% is itself unreliable. The +metric cannot be trusted until this is fixed. Fix order: stop stamping a guess as `tail` at confidence +1.0. Prefer `unknown` when two or more characters are present. Then bind by tail geometry. + +## One character id covers two different women {#identity-over-merge} + +`character_afa7623b` is stored as "black bob, white sweater" and is assigned both to that person +(`p059.png`) and to the brown-bob green-top coworker (`p010.png`, `p012.png`). It took 25 of the 26 +identity assignments in the chapter, against 113 detected people. Coverage is 23%. + +**Revisit trigger:** any work on the identity Tier-2 decider. A single id absorbing a whole chapter is +the signature to watch for. + +## The character registry carries five weeks of wrong names {#registry-pollution} + +The registry holds 53 characters for manga `ef105a86`, 41 of them unnamed, with "Kei" three times and +"Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only +Haeseon and Seonho do. `/stage/clear` leaves the per-manga registry intact by design, so every rerun +inherits the whole pile. + +**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the +registry to a chapter or add a reviewed reset. diff --git a/decisions/CLAUDE.md b/decisions/CLAUDE.md index 0dabe16..319ef47 100644 --- a/decisions/CLAUDE.md +++ b/decisions/CLAUDE.md @@ -27,3 +27,6 @@ still live belongs in `caveats/`. | [Correctness flags resolve by flag id, and never block autonomous TTS](audit-phase1.md#flag-resolution) | closed | | [An out-of-range resolver index is `unresolved`, never a new character](audit-phase1.md#hallucinated-index) | closed | | [The session manager holds no lock across a model load](audit-phase1.md#unlocked-model-load) | closed | +| [One bucket per artifact class, not everything under `manga`](storage-layout.md#bucket-per-artifact) | closed | +| [The orchestrator creates missing buckets at startup](storage-layout.md#ensure-buckets) | closed | +| [RustFS is staged, not adopted](storage-layout.md#rustfs-staged) | open | diff --git a/decisions/storage-layout.md b/decisions/storage-layout.md new file mode 100644 index 0000000..0a97a88 --- /dev/null +++ b/decisions/storage-layout.md @@ -0,0 +1,58 @@ +# storage-layout + +Settled questions about which S3 bucket holds what, and about the MinIO replacement. + +## One bucket per artifact class {#bucket-per-artifact} + +**State: closed. 2026-08-11.** + +Six buckets were created on 2026-07-04 (`manga raw panels audio layers video`). Only `manga` ever +received an object, because `minio_layout.py` hardcoded `BUCKET = "manga"` and four workers built +their own keys as literal `s3://manga/...`. The other five sat empty for five weeks. + +Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved: + +| artifact | bucket | +| --- | --- | +| fetched pages | `raw` | +| panel crops | `panels` | +| tts wavs | `audio` | +| layer pngs | `layers` | +| clips and `chapter.mp4` | `video` | +| vision, identity, scene, script json, character registry | `manga` | + +Orchestrator: `minio_layout.py` gained `BUCKET_RAW`/`BUCKET_PANELS`/`BUCKET_AUDIO`/`BUCKET_LAYERS`/ +`BUCKET_VIDEO` and a `BUCKETS` tuple. `parse_key` accepts any of them and rejects anything else. +`service.py:_s3_delete_prefix` takes a `/` pair instead of assuming one bucket, and +`_stage_s3_prefixes` stops slicing the bucket off. Workers: `worker_crop.py`, `worker_tts.py`, +`worker_layers.py`, `worker_render.py`. + +Every S3 URI is `s3://///...` and every consumer already derives the +bucket from the first path segment, so no reader needed a change. + +What this forbids: writing an artifact under a bucket that is not in `BUCKETS`. `parse_key` returns +`{}` for one, and stage clearing would then silently delete nothing. + +Objects written before this date stay under `manga/` at their old keys. Nothing reads them any more: +they are the rollback for the 2026-07-17 run, not live data. + +Evidence: `test_minio_layout.py` (31 tests), and the 2026-08-11 chapter run, which put pages in `raw` +and 116 panel crops in `panels`. + +## The orchestrator creates missing buckets at startup {#ensure-buckets} + +**State: closed. 2026-08-11.** + +Workers create a bucket on first write (`transport.py:115`), but the orchestrator uploads pages before +any worker runs and boto3 will not auto-create. `service.py:_ensure_buckets` runs in the FastAPI +lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning +instead. The check is not worth a failed start. + +## RustFS is staged, not adopted {#rustfs-staged} + +**State: open. 2026-08-11.** + +`rustfs` holds all six buckets on `127.0.0.1:9010/9011`, all empty. MinIO still serves every read and +write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is +`1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run +meant to produce the baseline. Task [#116]. diff --git a/worker_crop.py b/worker_crop.py index db6b798..5c5fa48 100644 --- a/worker_crop.py +++ b/worker_crop.py @@ -244,7 +244,7 @@ async def crop_webtoon(data: WebtoonInput): context_links = context_fragment_links(crops) panels = [] for idx, (crop_img, bbox) in enumerate(crops): - uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png" + uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png" # slicing is deterministic, so on a resume the same idx -> same key; skip re-upload. if not transport.exists(uri): out = f"{SHM}/wt_{tag}_p{idx:03d}.png" @@ -274,7 +274,7 @@ async def crop(data: CropInput): for idx, (crop_img, bbox) in enumerate(crops): out = f"{SHM}/pg{data.page_index:03d}_p{idx:02d}.png" cv2.imwrite(out, crop_img) - uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/pg{data.page_index:03d}_p{idx:02d}.png" + uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/pg{data.page_index:03d}_p{idx:02d}.png" transport.put(out, uri) os.remove(out) panels.append({"panel_index": idx, "uri": uri, "bbox": bbox, diff --git a/worker_layers.py b/worker_layers.py index 3f9b74f..03cda30 100644 --- a/worker_layers.py +++ b/worker_layers.py @@ -73,7 +73,7 @@ async def layers(data: LayerInput): png = f"{SHM}/layer_{uuid.uuid4().hex[:6]}.png" with open(png, "wb") as f: f.write(requests.get(url, timeout=60).content) - uri = f"s3://manga/{manga_id}/{chapter_id}/layers/{data.panel_id or 'p'}/{idx}.png" + uri = f"s3://layers/{manga_id}/{chapter_id}/layers/{data.panel_id or 'p'}/{idx}.png" transport.put(png, uri) os.remove(png) layer_uris.append(uri) diff --git a/worker_render.py b/worker_render.py index e35e14b..6ddaf82 100644 --- a/worker_render.py +++ b/worker_render.py @@ -207,7 +207,7 @@ async def render_scene(data: SceneInput): out = f"{SHM}/rnd_{tag}.mp4" subprocess.run(scene_cmd(img, audio, ass, out, dur, data.camera), check=True, capture_output=True) manga_id, chapter_id = _mc_from_uri(data.panel_uri) - uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" + uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" transport.put(out, uri) for p in (img, audio, ass, out): os.remove(p) @@ -353,7 +353,7 @@ async def render_composite(data: CompositeInput): for p in imgs + [still]: os.remove(p) manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"]) - uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" + uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" transport.put(out, uri) for p in auds + [ass, out]: os.remove(p) @@ -400,7 +400,7 @@ async def render_group(data: GroupInput): subprocess.run(cmd, check=True, capture_output=True) manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"]) - uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" + uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" transport.put(final, uri) total = _audio_dur(final) or sum(durs) for f in cleanup + [final]: @@ -537,7 +537,7 @@ async def render_beat(data: BeatInput): check=True, capture_output=True) manga_id, chapter_id = _mc_from_uri(data.panel_uris[0]) - uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" + uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" transport.put(out, uri) total = _audio_dur(out) or (D + PAD_S) for f in imgs + [audio, ass, out]: @@ -637,7 +637,7 @@ async def render_collage(data: CollageInput): subprocess.run(collage_cmd(imgs, active, lay["rects"], lay["entrances"], lay["z_order"], audio, ass, out, D, lay["transition_s"]), check=True, capture_output=True) manga_id, chapter_id = _mc_from_uri(uris[0]) - uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" + uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4" transport.put(out, uri) total = _audio_dur(out) or (D + PAD_S) for f in imgs + [audio, ass, out]: @@ -829,7 +829,7 @@ async def assemble(data: AssembleInput): out = _add_music_bed(out, tag, cleanup) manga_id, chapter_id = _mc_from_uri(data.clip_uris[0]) - uri = f"s3://manga/{manga_id}/{chapter_id}/chapter.mp4" + uri = f"s3://video/{manga_id}/{chapter_id}/chapter.mp4" transport.put(out, uri) for p in cleanup: os.remove(p) diff --git a/worker_tts.py b/worker_tts.py index ad0ff32..1913029 100644 --- a/worker_tts.py +++ b/worker_tts.py @@ -71,8 +71,8 @@ def _audio_uri(data: "TTSInput") -> str: # panel_uri from run_stage_tts to make it per-chapter unique. if data.panel_uri: parts = data.panel_uri.replace("s3://", "").split("/") - return f"s3://manga/{parts[1]}/{parts[2]}/audio/{data.panel_id or 'p'}.wav" - return f"s3://manga/_audio/{data.panel_id or 'p'}.wav" + return f"s3://audio/{parts[1]}/{parts[2]}/audio/{data.panel_id or 'p'}.wav" + return f"s3://audio/_audio/{data.panel_id or 'p'}.wav" def _ensure_ref() -> str: