kami
bec9411af3
Five workers built output URIs with inline f-strings, so the bucket-per-artifact layout was spread across worker_tts, worker_identity, worker_crop, worker_layers and worker_render. Moving a class between buckets meant a grep. They are now templates in transport.py, formatted at each call site. Three of those workers also each reimplemented the same parse to recover manga_id and chapter_id from an input uri, because the orchestrator does not send them. That is transport.ids_from_uri now, and it raises on a uri too short to carry the ids rather than returning a wrong pair. ruff.toml makes `ruff check .` exit 0, so CI can gate on it and a new finding means a new defect. Fixed: an implicit Optional in 8 signatures, an unparenthesized implicit concatenation in the ASS filter list, 5 subprocess.run calls now saying check=False out loud, an unused import, a duplicate exception handler and a non-executable shebang. Every rule left off carries its reason in ruff.toml. The ASYNC rules are off because ffmpeg on the event loop is real and already recorded at caveats/audit-open.md#blocking-event-loop. It needs a refactor per handler, not a lint fix. Checked: transport, collage, bubble_detect, test_vision_parse, worker_crop, worker_scene, worker_script, worker_identity, worker_tts, session_manager, worker_vision and worker_render self-checks all pass. worker_layers still fails on a missing legacy/qwen_layered_workflow.json, which predates this branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
81 lines
4.4 KiB
Markdown
81 lines
4.4 KiB
Markdown
# CLAUDE.md
|
|
|
|
Goal, invariants, and working rules. Read this first.
|
|
|
|
| file | holds |
|
|
| --- | --- |
|
|
| `NEXT.md` | the current state and the live plan |
|
|
| `ROADMAP.md` | the ordered outcomes past the current one |
|
|
| `JOURNAL.md` | what was run and when, append-only |
|
|
| `decisions/` | every settled question, indexed in `decisions/CLAUDE.md` |
|
|
| `caveats/` | every known limit and its revisit trigger, indexed in `caveats/CLAUDE.md` |
|
|
| `AGENTS.md` | commands, with the traps beside them |
|
|
| `AUDIT.md` | the 2026-08-11 pipeline audit, the source of the roadmap |
|
|
| `ARCHITECTURE.md` | the target shape of the pipeline, and what exists against it today |
|
|
| `spec-v3.md` | current quality and look work, marked DONE/TODO per item |
|
|
|
|
Do not restate a finding here. Point at the decision.
|
|
|
|
## The goal
|
|
|
|
This is the **workpc compute half** of a manga to narrated-video pipeline. It holds stateless GPU and
|
|
CPU workers only. State, job scheduling, and stage orchestration live in a separate homesrv
|
|
orchestrator repo (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`). The two talk over a
|
|
fixed HTTP contract. The hard part is that one GPU serves every model, so the schedule, not the model,
|
|
sets the wall time.
|
|
|
|
Machine split: workers run on **workpc** (RX 7900 GRE, ROCm). MinIO and the orchestrator run on
|
|
**homesrv** (`192.168.1.104`, CPU-only). `/mnt/server/home/kami/` is an SSHFS mount of homesrv.
|
|
|
|
## Invariants
|
|
|
|
1. **No durable state in a worker.** No sqlite, no cross-request memory. A worker pulls inputs from
|
|
MinIO by URI, does one stage, pushes outputs back, returns URIs. A `/dev/shm` cache is allowed
|
|
because it may be dropped at any time.
|
|
2. **One warm model at a time.** Every GPU stage takes a lease from `session_manager.py` on 8095. A 409
|
|
is a queue signal, never a stale lease
|
|
(`decisions/audit-phase1.md#no-lease-stealing`).
|
|
3. **Never hold `_lock` across a model load or a health wait** in `session_manager.py`
|
|
(`decisions/audit-phase1.md#unlocked-model-load`).
|
|
4. **A dialogue row's speaker is `speaker_ref`.** The flat `speaker` field is a compatibility value and
|
|
holds a `character_id`, not a panel-local id
|
|
(`decisions/audit-phase1.md#speaker-ref-is-canonical`).
|
|
5. **A stage never raises after setting `awaiting_review`.** The pipeline's catch-all overwrites it
|
|
with `failed` (`decisions/audit-phase1.md#flag-resolution`).
|
|
6. **Never mint a character from an unparseable or out-of-range model answer.** That is `unresolved`
|
|
(`decisions/audit-phase1.md#hallucinated-index`).
|
|
7. **The HTTP contract with the orchestrator is load-bearing.** Changing a worker's request or response
|
|
shape means reconciling the orchestrator in the same session. Neither repo's self-checks can catch a
|
|
contract break, because each asserts its own side.
|
|
8. **`ponytail:` comments mark deliberate simplifications** and name the upgrade path. Respect them.
|
|
9. **Nothing that positions an ffmpeg filter may use `format=duration`.** It reports `max(video, audio)`.
|
|
It hides A/V drift, and it walks xfade offsets past the end of their input. ffmpeg then discards clips
|
|
and still exits 0 (`decisions/chapter-assembly.md#offsets-from-min-stream`).
|
|
|
|
## Working rules
|
|
|
|
```bash
|
|
./start_workers.sh # dev: session_manager + 9 workers, each a uvicorn in a tmux window
|
|
tmux attach -t manga-workers # per-worker logs
|
|
sudo systemd/install.sh # production: one systemd unit per process (User=kami)
|
|
|
|
.venv/bin/python worker_scene.py # every module has an assert-based __main__ self-check
|
|
.venv/bin/python test_vision_parse.py
|
|
ruff check . # must exit 0; every ignore in ruff.toml carries its reason
|
|
|
|
cd /mnt/server/home/kami/docker-apps/manga-infra/orchestrator && pytest -q --ignore=test_api.py
|
|
```
|
|
|
|
There is no build step. `.venv` is the ROCm torch env. Workers import `transport` by module
|
|
name. Ports: crop 8000, vision 8002, identity 8003, scene 8004, script 8005, tts 8006, layers 8007,
|
|
render 8008, session_manager 8095.
|
|
|
|
Every output S3 URI is a template in `transport.py`, not an f-string in a worker. Add one there when a
|
|
new artifact class appears.
|
|
|
|
- Any non-trivial logic gets ONE runnable check in `__main__`, assert-based, no framework. Run the file
|
|
to verify it.
|
|
- Editing a worker's request or response shape means editing the orchestrator too, in the same session.
|
|
- Update `NEXT.md` alongside any change that moves the plan, and append to `JOURNAL.md` after a run.
|
|
- Do not run GPU work or a full pipeline without asking.
|