# storage-layout

Settled questions about which S3 bucket holds what, and about the MinIO replacement.

## One bucket per artifact class {#bucket-per-artifact}

**State: closed. 2026-08-11.**

Six buckets were created on 2026-07-04 (`manga raw panels audio layers video`). Only `manga` ever
received an object, because `minio_layout.py` hardcoded `BUCKET = "manga"` and four workers built
their own keys as literal `s3://manga/...`. The other five sat empty for five weeks.

Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved:

| artifact | bucket |
| --- | --- |
| fetched pages | `raw` |
| panel crops | `panels` |
| tts wavs | `audio` |
| layer pngs | `layers` |
| clips and `chapter.mp4` | `video` |
| vision, identity, scene, script json, character registry | `manga` |

Orchestrator: `minio_layout.py` gained `BUCKET_RAW`/`BUCKET_PANELS`/`BUCKET_AUDIO`/`BUCKET_LAYERS`/
`BUCKET_VIDEO` and a `BUCKETS` tuple. `parse_key` accepts any of them and rejects anything else.
`service.py:_s3_delete_prefix` takes a `<bucket>/<prefix>` pair instead of assuming one bucket, and
`_stage_s3_prefixes` stops slicing the bucket off. Workers: `worker_crop.py`, `worker_tts.py`,
`worker_layers.py`, `worker_render.py`.

Every S3 URI is `s3://<bucket>/<manga_id>/<chapter_id>/...` and every consumer already derives the
bucket from the first path segment, so no reader needed a change.

What this forbids: writing an artifact under a bucket that is not in `BUCKETS`. `parse_key` returns
`{}` for one, and stage clearing would then silently delete nothing.

Objects written before this date stay under `manga/` at their old keys. Nothing reads them any more:
they are the rollback for the 2026-07-17 run, not live data.

Evidence: `test_minio_layout.py` (31 tests), and the 2026-08-11 chapter run, which put pages in `raw`
and 116 panel crops in `panels`.

## The orchestrator creates missing buckets at startup {#ensure-buckets}

**State: closed. 2026-08-11.**

Workers create a bucket on first write (`transport.py:115`), but the orchestrator uploads pages before
any worker runs and boto3 will not auto-create. `service.py:_ensure_buckets` runs in the FastAPI
lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning
instead. The check is not worth a failed start.

## RustFS is staged, not adopted {#rustfs-staged}

**State: open. 2026-08-11.**

`rustfs` holds all six buckets on `127.0.0.1:9010/9011`, all empty. MinIO still serves every read and
write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is
`1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run
meant to produce the baseline. Task [#116].

## Clearing a stage strips the vision blob it wrote {#clear-vision-blob}

**Closed. 2026-08-11.** `dialogue` and `direct` have no output table. They write onto the per-panel
vision blob, and `_STAGE_TABLES` had no entry for either, so `/stage/clear dialogue` deleted nothing and
still returned `{"ok": true}`. `run_stage_dialogue` then saw `"dialogue" in vision` and skipped all 116
panels.

Evidence: the first clear reported `scripts: 46, scene_graphs: 116` and no vision counts. After the fix
the same call reported `vision_results.dialogue: 116` and `vision_results.direct: 75`, all of which the
first clear had left in place. A whole rerun was wasted on stale data before this was found.

`_STAGE_VISION_KEYS` in `db.py` names the keys each stage owns, and `clear_stage_data` strips them.

Forbids: adding a stage that writes onto a shared blob without listing its keys there.
Check: `pytest test_db.py`, `TestClearStageData`.