Split artifacts across per-class buckets, record the baseline run

Panels, wavs, layers, clips, and the chapter mp4 leave the `manga` bucket for
`panels`, `audio`, `layers`, and `video`. The key under the bucket is unchanged,
so every reader that derives the bucket from the first path segment keeps
working. The orchestrator half moves in the same commit, per invariant 7.

The 2026-08-11 chapter run proves the split for `raw` and `panels` and produced
the first quality read on speaker attribution, which is wrong in every sampled
multi-character panel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XD7cAy81MZrc7gCr6aZGWr
This commit is contained in:
2026-08-11 23:00:13 +04:00
parent b2cd11dd1c
commit 54c456801a
11 changed files with 253 additions and 20 deletions
+65
View File
@@ -0,0 +1,65 @@
# HANDOFF, 2026-08-11 session
Live state lives in `NEXT.md`. This file is only what this session did.
## Asked
1. Add the buckets to rustfs.
2. Fix the bucket saving on the orchestrator side.
3. Run the title recap.
Mid-session: check `manga.kvmx.ru`, and cross-check whether the run got dialogue and identities right.
## Changed
Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`:
- `worker_crop.py` panels -> `s3://panels/`
- `worker_tts.py` wavs -> `s3://audio/`
- `worker_layers.py` layers -> `s3://layers/`
- `worker_render.py` clips and `chapter.mp4` -> `s3://video/`
- `decisions/storage-layout.md`, `caveats/speaker-attribution.md`, indexes, `JOURNAL.md`, `NEXT.md`
Orchestrator, `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator`:
- `minio_layout.py` per-artifact bucket constants, `BUCKETS`, `parse_key` accepts any of them
- `service.py` `_s3_delete_prefix` takes `<bucket>/<prefix>`, `_stage_s3_prefixes` no longer slices,
new `_ensure_buckets` in the lifespan
- `test_minio_layout.py` updated, one test added
Infrastructure, homesrv:
- six buckets created on `rfs` (rustfs, `127.0.0.1:9010`)
- `manga-fetch` started, exited 2 weeks, `/job/create` fails without it
- `manga-web` started, exited 2 weeks, `manga.kvmx.ru` -> nginx -> `localhost:8083`
- orchestrator rebuilt and restarted
- `mc` aliases `mio` and `rfs` now exist on homesrv
## Measured
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`,
"Teto X Egen", 116 panels.
- 109 orchestrator tests pass. `worker_crop.py`, `worker_tts.py`, `worker_render.py` self-checks pass.
`worker_layers.py` self-check fails on the missing `legacy/qwen_layered_workflow.json`, which
predates this session.
- buckets after the run: `raw` 64MiB/79, `panels` 101MiB/116, `manga` 366MiB/491, `audio` `layers`
`video` still empty because the run had not reached those stages.
- stage times: crop 85s, vision ~4min, identity ~1min, reconcile ~7min, dialogue ~8min.
- 24 of 81 speech lines resolve to a named character, 30%.
- 26 of 113 detected people got an identity, 23%, and 25 of the 26 went to one character.
- 3 of 3 sampled two-character panels attribute both speakers to the wrong person.
## Open
- The run stopped in `direct` at 113/116. `scene script tts layers render assemble` never ran. Resume:
```bash
ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
```
- Speaker attribution is wrong in multi-character panels and the fix is not written
(`caveats/speaker-attribution.md`). No code was changed for it.
- Objects from the 2026-07-17 run still sit under `manga/<manga_id>/<chapter_id>/{pages,panels,audio,
layers,clips}`. Nothing reads them. They are the rollback, not live data.
- `rustfs` holds empty buckets. No mirror, no cutover.
+30
View File
@@ -67,3 +67,33 @@ Also patched `~/scripts/migrate-kvmx-https.sh:54` on homesrv. `need_stream_modul
`sudo -n nginx -V` and `sudo -n nginx -T`; the NOPASSWD rule covers only `nginx -t`, so it reported
"stream module is not loaded" whenever it meant "could not ask for a password". Both checks now run
without sudo. `bash -n` passes and both conditions evaluate true.
## 2026-08-11 Per-artifact buckets, rustfs buckets, baseline chapter run [#116]
Command: `mc mb` on rustfs, `docker compose up -d --build orchestrator`, `pytest -q --ignore=test_api.py`,
`./start_workers.sh`, then `/job/create` + `/stage/clear` + `/job/resume` for chapter
`7c944dd4-e972-42c7-ba60-9f6939548e80` of "Teto X Egen" as job `778297bc-e7ce-439d-91b5-8a027060d17f`.
Outcome: partial. Storage split landed and is proven by the run. The run itself was still in `direct`
when the session ended.
Produced: `decisions/storage-layout.md`, `caveats/speaker-attribution.md`, 109 orchestrator tests pass.
Artifacts now split one bucket per class instead of everything under `manga`
(`decisions/storage-layout.md#bucket-per-artifact`). Both MinIO and rustfs hold all six buckets. The
run put 79 pages in `raw` and 116 panel crops in `panels`, so the split works end to end.
Two containers on homesrv had been dead for two weeks and blocked the work. `manga-fetch` was exited,
so `/job/create` failed with `httpx.ConnectError`; `manga-web` was exited, so `manga.kvmx.ru` had
nothing behind it on port 8083. Both started with `docker compose up -d`. Neither is related to the
storage change. Neither was caught by any check, because nothing watches these containers.
Stage timings, 116 panels: crop 85s, vision ~4min, identity ~1min, reconcile ~7min for 35 pairs,
dialogue ~8min. Faster than the 2026-07-17 run at 75 panels. The webtoon crop that 500'd in July
succeeded this time.
Quality cross-check against the panel images, the point of the run. Dialogue text extraction is
accurate. Character detection is accurate. Speaker attribution is not: three of three sampled
two-character panels attribute both speakers to the wrong person, always swapped
(`caveats/speaker-attribution.md#tail-is-not-geometry`). 24 of 81 speech lines resolve to a named
character, which is the Phase 1 headline metric at 30%, and the sample says that 30% is not
trustworthy. 26 of 113 detected people got an identity, and 25 of those 26 went to one character that
turns out to cover two different women.
+28 -9
View File
@@ -23,23 +23,42 @@ the rebuild exposed.
## Next
1. Run one labeled chapter end to end. Record the baseline numbers from `ROADMAP.md`, above all the
share of narrated lines with a named speaker. That number is the check on the largest Phase 1 fix.
2. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work
1. Fix speaker attribution. The baseline run has been made and the metric it produced cannot be
trusted (`caveats/speaker-attribution.md#tail-is-not-geometry`). Three of three sampled
two-character panels swap the speakers, and the `tail` provenance label is stamped on guesses at
confidence 1.0. Smallest honest first step: stop labelling a guess `tail`, and return `unknown`
when two or more characters are present.
2. Rerun the chapter and re-read the named-speaker share. Only then is the Phase 1 headline number
real.
3. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work
(`caveats/audit-open.md#sqlite-locking`).
## The 2026-08-11 chapter run
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116
panels. It was still in `direct` at 113/116 when the session ended, with `scene script tts layers
render assemble` unrun. Resume it, or read where it got to:
```bash
ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
```
Numbers and the quality read are in `JOURNAL.md` and `caveats/speaker-attribution.md`.
## Storage and viewer, tasks #116/#117
[#117] is done. `stowage` serves the manga buckets. It was never a MinIO problem: the container had
been dead since 2026-07-19 on an arm64 digest pin. Details in `JOURNAL.md`.
[#116] is staged and unfinished. `rustfs` runs on `127.0.0.1:9010/9011` with empty buckets. The next
step is the `mc` mirror of the six buckets, then verify object counts and sizes against MinIO, then
decide on cutover. Nothing is repointed yet and MinIO still serves every read and write.
[#116] is closer but not cut over. Artifacts now split one bucket per class
(`decisions/storage-layout.md#bucket-per-artifact`), and both MinIO and `rustfs` hold all six buckets.
`rustfs` on `127.0.0.1:9010/9011` is still empty and nothing is repointed, so MinIO serves every read
and write. Remaining: `mc mirror` the live buckets, verify counts and sizes, then decide on cutover
(`decisions/storage-layout.md#rustfs-staged`).
Two things to weigh before cutover, neither settled. RustFS is `1.0.0-beta.12`, labeled
`build-type=prerelease`. Swapping storage before the baseline chapter run also adds a variable to the
run that is meant to produce the baseline.
Two containers on homesrv had been dead for two weeks and are now running. `manga-fetch` is the one
`/job/create` needs. `manga-web` is what `manga.kvmx.ru` proxies to on 8083. Nothing watches them.
## Open questions
+3
View File
@@ -32,3 +32,6 @@ a complaint, so give it one or drop it.
| [MinIO credentials are hardcoded in committed source](audit-open.md#hardcoded-credentials) | AUDIT.md |
| [Assemble marks a job completed with no clips](audit-open.md#empty-assemble) | AUDIT.md |
| [Reviewer timestamps drift against the crossfaded video](audit-open.md#timeline-drift) | AUDIT.md |
| [`speaker_method="tail"` never reads a tail](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run |
| [One character id covers two different women](speaker-attribution.md#identity-over-merge) | 2026-08-11 run |
| [The character registry carries five weeks of wrong names](speaker-attribution.md#registry-pollution) | 2026-08-11 run |
+55
View File
@@ -0,0 +1,55 @@
# speaker-attribution
Limits found by cross-checking the 2026-08-11 chapter run against the panel images.
## `speaker_method="tail"` never reads a tail {#tail-is-not-geometry}
`worker_vision.py:376` (`_annotate_speaker_methods`) stamps `tail` on any line whose `speaker` matches
a `local_id` present in the panel. No balloon geometry is consulted. The caller keeps gemma's
confidence, usually 1.0. An unverified model guess thus carries the highest-trust provenance in the
pipeline. The review UI and the flag rules both believe it.
Measured on job `778297bc`, chapter `7c944dd4`: 31 of 81 speech lines are `tail` with two or more
characters present. Three two-character panels were checked against the art, and all three are wrong,
each with the two speakers swapped:
| panel key | line | truth | pipeline |
| --- | --- | --- | --- |
| `p010.png` | "…definitely an Egen guy, Seonho!" | the woman | Seonho, the person addressed |
| `p010.png` | "Y-you think so?" | Seonho | Choi Haeseon |
| `p012.png` | "Want me to send you the link?" | the woman | the man |
| `p059.png` | "If team leader Choi says it, it must be true." | the man | Choi Haeseon |
The last row needs no image: the line refers to Choi in the third person and is attributed to Choi.
The grounded path exists and almost never fires. Only 2 of 81 speech lines got `som_face`, because
attribution marks need `face_detect` boxes that survive `_pair_faces_to_present`, and these webtoon
close-ups rarely produce them. Inference, not measured: the face detector was not instrumented.
`worker_vision.py:356` already carries the `ponytail:` note that multi-character attribution needs
per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads carry balloon fill and
tail tips and are unused.
**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric
(`ROADMAP.md`). It reads 30% on this run, and the sample says that 30% is itself unreliable. The
metric cannot be trusted until this is fixed. Fix order: stop stamping a guess as `tail` at confidence
1.0. Prefer `unknown` when two or more characters are present. Then bind by tail geometry.
## One character id covers two different women {#identity-over-merge}
`character_afa7623b` is stored as "black bob, white sweater" and is assigned both to that person
(`p059.png`) and to the brown-bob green-top coworker (`p010.png`, `p012.png`). It took 25 of the 26
identity assignments in the chapter, against 113 detected people. Coverage is 23%.
**Revisit trigger:** any work on the identity Tier-2 decider. A single id absorbing a whole chapter is
the signature to watch for.
## The character registry carries five weeks of wrong names {#registry-pollution}
The registry holds 53 characters for manga `ef105a86`, 41 of them unnamed, with "Kei" three times and
"Kanade" twice. Kei, Kanade, Zen, Rico, K3, and Watanabe occur zero times in this chapter's text. Only
Haeseon and Seonho do. `/stage/clear` leaves the per-manga registry intact by design, so every rerun
inherits the whole pile.
**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the
registry to a chapter or add a reviewed reset.
+3
View File
@@ -27,3 +27,6 @@ still live belongs in `caveats/`.
| [Correctness flags resolve by flag id, and never block autonomous TTS](audit-phase1.md#flag-resolution) | closed |
| [An out-of-range resolver index is `unresolved`, never a new character](audit-phase1.md#hallucinated-index) | closed |
| [The session manager holds no lock across a model load](audit-phase1.md#unlocked-model-load) | closed |
| [One bucket per artifact class, not everything under `manga`](storage-layout.md#bucket-per-artifact) | closed |
| [The orchestrator creates missing buckets at startup](storage-layout.md#ensure-buckets) | closed |
| [RustFS is staged, not adopted](storage-layout.md#rustfs-staged) | open |
+58
View File
@@ -0,0 +1,58 @@
# storage-layout
Settled questions about which S3 bucket holds what, and about the MinIO replacement.
## One bucket per artifact class {#bucket-per-artifact}
**State: closed. 2026-08-11.**
Six buckets were created on 2026-07-04 (`manga raw panels audio layers video`). Only `manga` ever
received an object, because `minio_layout.py` hardcoded `BUCKET = "manga"` and four workers built
their own keys as literal `s3://manga/...`. The other five sat empty for five weeks.
Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved:
| artifact | bucket |
| --- | --- |
| fetched pages | `raw` |
| panel crops | `panels` |
| tts wavs | `audio` |
| layer pngs | `layers` |
| clips and `chapter.mp4` | `video` |
| vision, identity, scene, script json, character registry | `manga` |
Orchestrator: `minio_layout.py` gained `BUCKET_RAW`/`BUCKET_PANELS`/`BUCKET_AUDIO`/`BUCKET_LAYERS`/
`BUCKET_VIDEO` and a `BUCKETS` tuple. `parse_key` accepts any of them and rejects anything else.
`service.py:_s3_delete_prefix` takes a `<bucket>/<prefix>` pair instead of assuming one bucket, and
`_stage_s3_prefixes` stops slicing the bucket off. Workers: `worker_crop.py`, `worker_tts.py`,
`worker_layers.py`, `worker_render.py`.
Every S3 URI is `s3://<bucket>/<manga_id>/<chapter_id>/...` and every consumer already derives the
bucket from the first path segment, so no reader needed a change.
What this forbids: writing an artifact under a bucket that is not in `BUCKETS`. `parse_key` returns
`{}` for one, and stage clearing would then silently delete nothing.
Objects written before this date stay under `manga/` at their old keys. Nothing reads them any more:
they are the rollback for the 2026-07-17 run, not live data.
Evidence: `test_minio_layout.py` (31 tests), and the 2026-08-11 chapter run, which put pages in `raw`
and 116 panel crops in `panels`.
## The orchestrator creates missing buckets at startup {#ensure-buckets}
**State: closed. 2026-08-11.**
Workers create a bucket on first write (`transport.py:115`), but the orchestrator uploads pages before
any worker runs and boto3 will not auto-create. `service.py:_ensure_buckets` runs in the FastAPI
lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning
instead. The check is not worth a failed start.
## RustFS is staged, not adopted {#rustfs-staged}
**State: open. 2026-08-11.**
`rustfs` holds all six buckets on `127.0.0.1:9010/9011`, all empty. MinIO still serves every read and
write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is
`1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run
meant to produce the baseline. Task [#116].
+2 -2
View File
@@ -244,7 +244,7 @@ async def crop_webtoon(data: WebtoonInput):
context_links = context_fragment_links(crops)
panels = []
for idx, (crop_img, bbox) in enumerate(crops):
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png"
uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/p{idx:03d}.png"
# slicing is deterministic, so on a resume the same idx -> same key; skip re-upload.
if not transport.exists(uri):
out = f"{SHM}/wt_{tag}_p{idx:03d}.png"
@@ -274,7 +274,7 @@ async def crop(data: CropInput):
for idx, (crop_img, bbox) in enumerate(crops):
out = f"{SHM}/pg{data.page_index:03d}_p{idx:02d}.png"
cv2.imwrite(out, crop_img)
uri = f"s3://manga/{data.manga_id}/{data.chapter_id}/panels/pg{data.page_index:03d}_p{idx:02d}.png"
uri = f"s3://panels/{data.manga_id}/{data.chapter_id}/panels/pg{data.page_index:03d}_p{idx:02d}.png"
transport.put(out, uri)
os.remove(out)
panels.append({"panel_index": idx, "uri": uri, "bbox": bbox,
+1 -1
View File
@@ -73,7 +73,7 @@ async def layers(data: LayerInput):
png = f"{SHM}/layer_{uuid.uuid4().hex[:6]}.png"
with open(png, "wb") as f:
f.write(requests.get(url, timeout=60).content)
uri = f"s3://manga/{manga_id}/{chapter_id}/layers/{data.panel_id or 'p'}/{idx}.png"
uri = f"s3://layers/{manga_id}/{chapter_id}/layers/{data.panel_id or 'p'}/{idx}.png"
transport.put(png, uri)
os.remove(png)
layer_uris.append(uri)
+6 -6
View File
@@ -207,7 +207,7 @@ async def render_scene(data: SceneInput):
out = f"{SHM}/rnd_{tag}.mp4"
subprocess.run(scene_cmd(img, audio, ass, out, dur, data.camera), check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panel_uri)
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
for p in (img, audio, ass, out):
os.remove(p)
@@ -353,7 +353,7 @@ async def render_composite(data: CompositeInput):
for p in imgs + [still]:
os.remove(p)
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
for p in auds + [ass, out]:
os.remove(p)
@@ -400,7 +400,7 @@ async def render_group(data: GroupInput):
subprocess.run(cmd, check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panels[0]["panel_uri"])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(final, uri)
total = _audio_dur(final) or sum(durs)
for f in cleanup + [final]:
@@ -537,7 +537,7 @@ async def render_beat(data: BeatInput):
check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(data.panel_uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
total = _audio_dur(out) or (D + PAD_S)
for f in imgs + [audio, ass, out]:
@@ -637,7 +637,7 @@ async def render_collage(data: CollageInput):
subprocess.run(collage_cmd(imgs, active, lay["rects"], lay["entrances"], lay["z_order"],
audio, ass, out, D, lay["transition_s"]), check=True, capture_output=True)
manga_id, chapter_id = _mc_from_uri(uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
uri = f"s3://video/{manga_id}/{chapter_id}/clips/{data.panel_id or 'p'}.mp4"
transport.put(out, uri)
total = _audio_dur(out) or (D + PAD_S)
for f in imgs + [audio, ass, out]:
@@ -829,7 +829,7 @@ async def assemble(data: AssembleInput):
out = _add_music_bed(out, tag, cleanup)
manga_id, chapter_id = _mc_from_uri(data.clip_uris[0])
uri = f"s3://manga/{manga_id}/{chapter_id}/chapter.mp4"
uri = f"s3://video/{manga_id}/{chapter_id}/chapter.mp4"
transport.put(out, uri)
for p in cleanup:
os.remove(p)
+2 -2
View File
@@ -71,8 +71,8 @@ def _audio_uri(data: "TTSInput") -> str:
# panel_uri from run_stage_tts to make it per-chapter unique.
if data.panel_uri:
parts = data.panel_uri.replace("s3://", "").split("/")
return f"s3://manga/{parts[1]}/{parts[2]}/audio/{data.panel_id or 'p'}.wav"
return f"s3://manga/_audio/{data.panel_id or 'p'}.wav"
return f"s3://audio/{parts[1]}/{parts[2]}/audio/{data.panel_id or 'p'}.wav"
return f"s3://audio/_audio/{data.panel_id or 'p'}.wav"
def _ensure_ref() -> str: