kami
63f7918a3e
Adds decision entries for the unpaired set-of-mark label, the interjection verifier false positive, and the vision-blob clearing bug, plus the per-run speaker audit script used to measure the chapter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
75 lines
3.6 KiB
Markdown
75 lines
3.6 KiB
Markdown
# storage-layout
|
|
|
|
Settled questions about which S3 bucket holds what, and about the MinIO replacement.
|
|
|
|
## One bucket per artifact class {#bucket-per-artifact}
|
|
|
|
**State: closed. 2026-08-11.**
|
|
|
|
Six buckets were created on 2026-07-04 (`manga raw panels audio layers video`). Only `manga` ever
|
|
received an object, because `minio_layout.py` hardcoded `BUCKET = "manga"` and four workers built
|
|
their own keys as literal `s3://manga/...`. The other five sat empty for five weeks.
|
|
|
|
Artifacts now split by class. The key under the bucket is unchanged, so only the leading segment moved:
|
|
|
|
| artifact | bucket |
|
|
| --- | --- |
|
|
| fetched pages | `raw` |
|
|
| panel crops | `panels` |
|
|
| tts wavs | `audio` |
|
|
| layer pngs | `layers` |
|
|
| clips and `chapter.mp4` | `video` |
|
|
| vision, identity, scene, script json, character registry | `manga` |
|
|
|
|
Orchestrator: `minio_layout.py` gained `BUCKET_RAW`/`BUCKET_PANELS`/`BUCKET_AUDIO`/`BUCKET_LAYERS`/
|
|
`BUCKET_VIDEO` and a `BUCKETS` tuple. `parse_key` accepts any of them and rejects anything else.
|
|
`service.py:_s3_delete_prefix` takes a `<bucket>/<prefix>` pair instead of assuming one bucket, and
|
|
`_stage_s3_prefixes` stops slicing the bucket off. Workers: `worker_crop.py`, `worker_tts.py`,
|
|
`worker_layers.py`, `worker_render.py`.
|
|
|
|
Every S3 URI is `s3://<bucket>/<manga_id>/<chapter_id>/...` and every consumer already derives the
|
|
bucket from the first path segment, so no reader needed a change.
|
|
|
|
What this forbids: writing an artifact under a bucket that is not in `BUCKETS`. `parse_key` returns
|
|
`{}` for one, and stage clearing would then silently delete nothing.
|
|
|
|
Objects written before this date stay under `manga/` at their old keys. Nothing reads them any more:
|
|
they are the rollback for the 2026-07-17 run, not live data.
|
|
|
|
Evidence: `test_minio_layout.py` (31 tests), and the 2026-08-11 chapter run, which put pages in `raw`
|
|
and 116 panel crops in `panels`.
|
|
|
|
## The orchestrator creates missing buckets at startup {#ensure-buckets}
|
|
|
|
**State: closed. 2026-08-11.**
|
|
|
|
Workers create a bucket on first write (`transport.py:115`), but the orchestrator uploads pages before
|
|
any worker runs and boto3 will not auto-create. `service.py:_ensure_buckets` runs in the FastAPI
|
|
lifespan and creates whatever is missing. A storage backend that is down at boot logs a warning
|
|
instead. The check is not worth a failed start.
|
|
|
|
## RustFS is staged, not adopted {#rustfs-staged}
|
|
|
|
**State: open. 2026-08-11.**
|
|
|
|
`rustfs` holds all six buckets on `127.0.0.1:9010/9011`, all empty. MinIO still serves every read and
|
|
write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is
|
|
`1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run
|
|
meant to produce the baseline. Task [#116].
|
|
|
|
## Clearing a stage strips the vision blob it wrote {#clear-vision-blob}
|
|
|
|
**Closed. 2026-08-11.** `dialogue` and `direct` have no output table. They write onto the per-panel
|
|
vision blob, and `_STAGE_TABLES` had no entry for either, so `/stage/clear dialogue` deleted nothing and
|
|
still returned `{"ok": true}`. `run_stage_dialogue` then saw `"dialogue" in vision` and skipped all 116
|
|
panels.
|
|
|
|
Evidence: the first clear reported `scripts: 46, scene_graphs: 116` and no vision counts. After the fix
|
|
the same call reported `vision_results.dialogue: 116` and `vision_results.direct: 75`, all of which the
|
|
first clear had left in place. A whole rerun was wasted on stale data before this was found.
|
|
|
|
`_STAGE_VISION_KEYS` in `db.py` names the keys each stage owns, and `clear_stage_data` strips them.
|
|
|
|
Forbids: adding a stage that writes onto a shared blob without listing its keys there.
|
|
Check: `pytest test_db.py`, `TestClearStageData`.
|