kami
54c456801a
Panels, wavs, layers, clips, and the chapter mp4 leave the `manga` bucket for `panels`, `audio`, `layers`, and `video`. The key under the bucket is unchanged, so every reader that derives the bucket from the first path segment keeps working. The orchestrator half moves in the same commit, per invariant 7. The 2026-08-11 chapter run proves the split for `raw` and `panels` and produced the first quality read on speaker attribution, which is wrong in every sampled multi-character panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XD7cAy81MZrc7gCr6aZGWr
2.7 KiB
storage-layout
Settled questions about which S3 bucket holds what, and about the MinIO replacement.
One bucket per artifact class
State: closed. 2026-08-11.
Six buckets were created on 2026-07-04 (manga raw panels audio layers video). Only manga ever
received an object, because minio_layout.py hardcoded BUCKET = "manga" and four workers built
their own keys as literal s3://manga/.... The other five sat empty for five weeks.
Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved:
| artifact | bucket |
|---|---|
| fetched pages | raw |
| panel crops | panels |
| tts wavs | audio |
| layer pngs | layers |
clips and chapter.mp4 |
video |
| vision, identity, scene, script json, character registry | manga |
Orchestrator: minio_layout.py gained BUCKET_RAW/BUCKET_PANELS/BUCKET_AUDIO/BUCKET_LAYERS/
BUCKET_VIDEO and a BUCKETS tuple. parse_key accepts any of them and rejects anything else.
service.py:_s3_delete_prefix takes a <bucket>/<prefix> pair instead of assuming one bucket, and
_stage_s3_prefixes stops slicing the bucket off. Workers: worker_crop.py, worker_tts.py,
worker_layers.py, worker_render.py.
Every S3 URI is s3://<bucket>/<manga_id>/<chapter_id>/... and every consumer already derives the
bucket from the first path segment, so no reader needed a change.
What this forbids: writing an artifact under a bucket that is not in BUCKETS. parse_key returns
{} for one, and stage clearing would then silently delete nothing.
Objects written before this date stay under manga/ at their old keys. Nothing reads them any more:
they are the rollback for the 2026-07-17 run, not live data.
Evidence: test_minio_layout.py (31 tests), and the 2026-08-11 chapter run, which put pages in raw
and 116 panel crops in panels.
The orchestrator creates missing buckets at startup
State: closed. 2026-08-11.
Workers create a bucket on first write (transport.py:115), but the orchestrator uploads pages before
any worker runs and boto3 will not auto-create. service.py:_ensure_buckets runs in the FastAPI
lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning
instead. The check is not worth a failed start.
RustFS is staged, not adopted
State: open. 2026-08-11.
rustfs holds all six buckets on 127.0.0.1:9010/9011, all empty. MinIO still serves every read and
write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is
1.0.0-beta.12, labeled build-type=prerelease. Swapping storage also adds a variable to the run
meant to produce the baseline. Task [#116].