kami
54c456801a
Panels, wavs, layers, clips, and the chapter mp4 leave the `manga` bucket for `panels`, `audio`, `layers`, and `video`. The key under the bucket is unchanged, so every reader that derives the bucket from the first path segment keeps working. The orchestrator half moves in the same commit, per invariant 7. The 2026-08-11 chapter run proves the split for `raw` and `panels` and produced the first quality read on speaker attribution, which is wrong in every sampled multi-character panel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XD7cAy81MZrc7gCr6aZGWr
59 lines
2.7 KiB
Markdown
59 lines
2.7 KiB
Markdown
# storage-layout
|
|
|
|
Settled questions about which S3 bucket holds what, and about the MinIO replacement.
|
|
|
|
## One bucket per artifact class {#bucket-per-artifact}
|
|
|
|
**State: closed. 2026-08-11.**
|
|
|
|
Six buckets were created on 2026-07-04 (`manga raw panels audio layers video`). Only `manga` ever
|
|
received an object, because `minio_layout.py` hardcoded `BUCKET = "manga"` and four workers built
|
|
their own keys as literal `s3://manga/...`. The other five sat empty for five weeks.
|
|
|
|
Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved:
|
|
|
|
| artifact | bucket |
|
|
| --- | --- |
|
|
| fetched pages | `raw` |
|
|
| panel crops | `panels` |
|
|
| tts wavs | `audio` |
|
|
| layer pngs | `layers` |
|
|
| clips and `chapter.mp4` | `video` |
|
|
| vision, identity, scene, script json, character registry | `manga` |
|
|
|
|
Orchestrator: `minio_layout.py` gained `BUCKET_RAW`/`BUCKET_PANELS`/`BUCKET_AUDIO`/`BUCKET_LAYERS`/
|
|
`BUCKET_VIDEO` and a `BUCKETS` tuple. `parse_key` accepts any of them and rejects anything else.
|
|
`service.py:_s3_delete_prefix` takes a `<bucket>/<prefix>` pair instead of assuming one bucket, and
|
|
`_stage_s3_prefixes` stops slicing the bucket off. Workers: `worker_crop.py`, `worker_tts.py`,
|
|
`worker_layers.py`, `worker_render.py`.
|
|
|
|
Every S3 URI is `s3://<bucket>/<manga_id>/<chapter_id>/...` and every consumer already derives the
|
|
bucket from the first path segment, so no reader needed a change.
|
|
|
|
What this forbids: writing an artifact under a bucket that is not in `BUCKETS`. `parse_key` returns
|
|
`{}` for one, and stage clearing would then silently delete nothing.
|
|
|
|
Objects written before this date stay under `manga/` at their old keys. Nothing reads them any more:
|
|
they are the rollback for the 2026-07-17 run, not live data.
|
|
|
|
Evidence: `test_minio_layout.py` (31 tests), and the 2026-08-11 chapter run, which put pages in `raw`
|
|
and 116 panel crops in `panels`.
|
|
|
|
## The orchestrator creates missing buckets at startup {#ensure-buckets}
|
|
|
|
**State: closed. 2026-08-11.**
|
|
|
|
Workers create a bucket on first write (`transport.py:115`), but the orchestrator uploads pages before
|
|
any worker runs and boto3 will not auto-create. `service.py:_ensure_buckets` runs in the FastAPI
|
|
lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning
|
|
instead. The check is not worth a failed start.
|
|
|
|
## RustFS is staged, not adopted {#rustfs-staged}
|
|
|
|
**State: open. 2026-08-11.**
|
|
|
|
`rustfs` holds all six buckets on `127.0.0.1:9010/9011`, all empty. MinIO still serves every read and
|
|
write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is
|
|
`1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run
|
|
meant to produce the baseline. Task [#116].
|