Audit Phase 1: correctness and scheduling safety
Implements every P0 from AUDIT.md plus four P1s, across both halves of the pipeline. Verified by CPU-only self-checks and the orchestrator test suite. No GPU work ran and no pipeline ran. workpc: - worker_scene: read speaker_ref, not the rewritten speaker field. Every line narrated as "Someone" before this. Emit `actions` for the verifier. - worker_script: declare beat + verifier_feedback (pydantic dropped both, so the retry was blind) and render them as a repair prompt. - worker_vision: gate face->identity pairing on containment, assign globally shortest-first, map an out-of-range resolver index to `unresolved` instead of minting a character, parse JSON with raw_decode. - session_manager: tear down a server whose lease vanished mid-load, and spawn the supervisor respawn unlocked. orchestrator (edited in place, NOT committed there): - tracklets: canonicalize gender, add co-presence cannot-links, block transitive bridges across a hard constraint. - correctness: stop failing valid narration on sentence-initial capitals and short quotes; read action evidence from the singular key. - db: stop orphan flags leaking into every chapter; resolve by flag id. - service: TTS returns instead of raising under GATES, auto-resolves under autonomous mode; job admission control; registry names on dialogue resume. - session_proxy: queue on 409 instead of stealing the lease; run heartbeats. Docs restructured per the repo-structure layout: CLAUDE.md is a pointer table, NEXT.md replaces HANDOFF.md, plus ROADMAP.md, JOURNAL.md, decisions/ and caveats/. AUDIT.md now points at those instead of restating them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XD7cAy81MZrc7gCr6aZGWr
This commit is contained in:
@@ -0,0 +1,34 @@
|
||||
# caveats/
|
||||
|
||||
One entry per known breakage, deferral, or trap. An entry names what fails, who pays for it, and the
|
||||
concrete trigger that makes fixing it worth the time.
|
||||
|
||||
A caveat that gets resolved moves to `decisions/` or gets deleted. A caveat with no revisit trigger is
|
||||
a complaint, so give it one or drop it.
|
||||
|
||||
## Rules for this directory
|
||||
|
||||
* One file per source, one `##` section per limit.
|
||||
* State the limit as what fails, not as a topic. "Reconcile deletes the losing character" beats
|
||||
"Reconcile".
|
||||
* Delete an entry the moment its trigger fires and the work lands.
|
||||
|
||||
## Index
|
||||
|
||||
| caveat | source |
|
||||
| --- | --- |
|
||||
| [Reconcile deletes the losing character irreversibly](audit-open.md#destructive-reconcile) | AUDIT.md |
|
||||
| [Clearing a stage does not undo what it wrote](audit-open.md#dishonest-clearing) | AUDIT.md |
|
||||
| [ComfyUI uses the GPU outside the session mutex](audit-open.md#comfyui-unscheduled) | AUDIT.md |
|
||||
| [The JSON repair pass can fabricate dialogue](audit-open.md#repair-fabricates) | AUDIT.md |
|
||||
| [`/review/preview` pins a solo clip into the final video](audit-open.md#preview-overwrites-clip) | AUDIT.md |
|
||||
| [Stored embeddings carry no model or pooling version](audit-open.md#untagged-embeddings) | AUDIT.md |
|
||||
| [Worker endpoints block the event loop](audit-open.md#blocking-event-loop) | AUDIT.md |
|
||||
| [SQLite has no busy timeout, so parallelism will surface as lock errors](audit-open.md#sqlite-locking) | AUDIT.md |
|
||||
| [`layers` runs after `tts`, so pipelined solo beats lose parallax](audit-open.md#layers-after-tts) | AUDIT.md |
|
||||
| [`completed` means something different in each stage](audit-open.md#inconsistent-stage-policy) | AUDIT.md |
|
||||
| [Identity worker caches characters the orchestrator has deleted](audit-open.md#stale-known-cache) | AUDIT.md |
|
||||
| [A character seen once gets no assignment at all](audit-open.md#pending-in-worker-memory) | AUDIT.md |
|
||||
| [MinIO credentials are hardcoded in committed source](audit-open.md#hardcoded-credentials) | AUDIT.md |
|
||||
| [Assemble marks a job completed with no clips](audit-open.md#empty-assemble) | AUDIT.md |
|
||||
| [Reviewer timestamps drift against the crossfaded video](audit-open.md#timeline-drift) | AUDIT.md |
|
||||
@@ -0,0 +1,149 @@
|
||||
# Open limits from the 2026-08-11 audit
|
||||
|
||||
Everything here was read in source during the audit and deliberately left unfixed in Phase 1. The
|
||||
fixed findings live in `decisions/audit-phase1.md`. Line numbers are from the audit and may drift.
|
||||
|
||||
## Reconcile deletes the losing character irreversibly {#destructive-reconcile}
|
||||
|
||||
Reconciliation deletes the losing character row (`db.py:506`). Clearing the reconcile stage does not
|
||||
undo it, and name claims attached to the merged-away character are not repointed.
|
||||
|
||||
Costs: one bad merge is unrecoverable without rebuilding the identity stage for the whole manga.
|
||||
Revisit when: identity work resumes, or a reviewer reports a wrong merge on a real chapter.
|
||||
Workaround: none. Clear identity and rerun, which loses the good merges too.
|
||||
|
||||
## Clearing a stage does not undo what it wrote {#dishonest-clearing}
|
||||
|
||||
Dialogue and direction mutate the shared vision JSON. Clearing dialogue leaves its keys in place, so a
|
||||
rerun treats old dialogue as completed. Clearing identity preserves the per-manga registry.
|
||||
|
||||
Costs: a rerun silently reuses stale output, which reads as a reproducible result.
|
||||
Revisit when: any stage is scheduled concurrently or resumed automatically. A stage must be idempotent
|
||||
before either is safe.
|
||||
Workaround: delete the keys by hand, or clear from `crop` down.
|
||||
|
||||
## ComfyUI uses the GPU outside the session mutex {#comfyui-unscheduled}
|
||||
|
||||
The layers stage calls ComfyUI directly and takes no lease. Another job can load gemma, siglip2, or
|
||||
dots while ComfyUI holds the same GPU.
|
||||
|
||||
Costs: out-of-memory failures that look random and land on an unrelated stage.
|
||||
Revisit when: layers is enabled on a real run, or a second concurrent job is allowed.
|
||||
Workaround: `MAX_CONCURRENT_JOBS=1` keeps one pipeline at a time, which is the current default.
|
||||
|
||||
## The JSON repair pass can fabricate dialogue {#repair-fabricates}
|
||||
|
||||
`call_gemma4_json` hands the model its own truncated text and asks for the JSON it should have been
|
||||
(`worker_vision.py`). The repair call carries no image. On a response truncated by `max_tokens`, the
|
||||
model completes dialogue it can no longer see. What it adds is indistinguishable downstream from
|
||||
transcribed text.
|
||||
|
||||
Costs: invented lines enter the script with normal provenance.
|
||||
Revisit when: schema-constrained generation lands, which removes most of this path.
|
||||
Workaround: resend the image on repair, or retry a truncated response instead of repairing it.
|
||||
|
||||
## `/review/preview` pins a solo clip into the final video {#preview-overwrites-clip}
|
||||
|
||||
`review_preview` renders one panel and calls `save_clip(panel_id, ...)`. `_render_one_beat` returns
|
||||
early when a clip already exists for the leader. Previewing a beat leader therefore drops the rest of
|
||||
the beat's panels. `review_retts` gets this right by calling `delete_clip` first.
|
||||
|
||||
Costs: a reviewer silently corrupts the output by looking at it.
|
||||
Revisit when: the review UI is used on a real chapter.
|
||||
Workaround: never preview a beat leader, or delete the clip row afterwards.
|
||||
|
||||
## Stored embeddings carry no model or pooling version {#untagged-embeddings}
|
||||
|
||||
`embed_crop` uses `pooler_output` when present and falls back to mean-pooled patch tokens otherwise.
|
||||
The two paths produce different vector spaces, and the fixed 0.85 threshold is valid for one of them.
|
||||
Nothing beside a stored `.npy` records which model, revision, or pooling produced it.
|
||||
|
||||
Costs: a transformers upgrade mixes incompatible vectors into one gallery with no error.
|
||||
Revisit when: transformers or the siglip2 revision is upgraded. Before, not after.
|
||||
Workaround: none. Write the model id and pooling mode beside the vector and refuse cross-version
|
||||
comparison.
|
||||
|
||||
## Worker endpoints block the event loop {#blocking-event-loop}
|
||||
|
||||
Endpoints declared `async def` run blocking MinIO, OpenCV, torch, and ffmpeg calls directly on the
|
||||
event loop. A busy worker cannot answer `/health` or `/unload`.
|
||||
|
||||
Costs: `/health/workers` reports a working worker as unreachable, and the session manager's 30-second
|
||||
`/unload` can time out exactly when VRAM needs freeing.
|
||||
Revisit when: a stage stalls on `/unload`, or before any bounded parallelism lands.
|
||||
Workaround: `def` instead of `async def` moves each handler to the threadpool. Tracked as [#199] for
|
||||
the render worker.
|
||||
|
||||
## SQLite has no busy timeout {#sqlite-locking}
|
||||
|
||||
`get_conn` opens a connection per call with no `busy_timeout`. WAL tolerates one writer. `PIPELINE=1`
|
||||
already writes clips from concurrent tasks while TTS writes audio.
|
||||
|
||||
Costs: planned CPU parallelism will surface as `database is locked` before it surfaces as throughput.
|
||||
Revisit when: Phase 2 concurrency work starts. Set the timeout first.
|
||||
Workaround: keep `PIPELINE` off.
|
||||
|
||||
## `layers` runs after `tts` {#layers-after-tts}
|
||||
|
||||
`STAGES` orders `layers` after `tts`, while `run_stage_tts` warns that eager rendering under
|
||||
`PIPELINE=1` needs layers to run first.
|
||||
|
||||
Costs: with the flag on, solo beats always render without parallax.
|
||||
Revisit when: a real run enables `PIPELINE=1`.
|
||||
Workaround: keep `PIPELINE` off, or reorder `STAGES`.
|
||||
|
||||
## `completed` means something different in each stage {#inconsistent-stage-policy}
|
||||
|
||||
A dropped vision panel fails the stage and halts the pipeline. A failed direction window counts its
|
||||
panels as done. Layers always finishes completed.
|
||||
|
||||
Costs: the acceptance metrics in `ROADMAP.md` cannot be read across stages.
|
||||
Revisit when: a baseline measurement is taken. The numbers are meaningless until then.
|
||||
Workaround: none.
|
||||
|
||||
## Identity worker caches characters the orchestrator has deleted {#stale-known-cache}
|
||||
|
||||
`worker_identity._known_cache` is invalidated only by `_persist_char`. The reconcile stage deletes
|
||||
losing characters directly in the orchestrator database. A long-lived worker keeps shortlisting and
|
||||
assigning ids that no longer exist.
|
||||
|
||||
Costs: assignments point at rows that are gone.
|
||||
Revisit when: reconcile runs on a chapter without a worker restart between stages. Tracked as [#201],
|
||||
which proposes caching per `manga_id`.
|
||||
Workaround: restart the identity worker after reconcile.
|
||||
|
||||
## A character seen once gets no assignment at all {#pending-in-worker-memory}
|
||||
|
||||
`worker_identity._pending` holds full crop images in worker memory for a whole chapter, keyed by
|
||||
session. That is durable per-chapter state inside a worker documented as stateless, and it is lost on
|
||||
restart. A character seen exactly once receives no assignment, not even a chapter-local handle.
|
||||
|
||||
Costs: one-off characters vanish from the scene graph.
|
||||
Revisit when: chapter-local tracklet persistence lands (`ROADMAP.md`, Phase 3).
|
||||
Workaround: none.
|
||||
|
||||
## MinIO credentials are hardcoded in committed source {#hardcoded-credentials}
|
||||
|
||||
Defaults live in `transport.py` and `service.py`.
|
||||
|
||||
Costs: the credentials are in git history for anyone who gets the repo.
|
||||
Revisit when: the repo leaves this machine, or MinIO is reachable outside the LAN.
|
||||
Workaround: the environment variables already override them. Set them and remove the defaults.
|
||||
|
||||
## Assemble marks a job completed with no clips {#empty-assemble}
|
||||
|
||||
`run_stage_assemble` does not check that `clip_uris` is non-empty before assembling, then marks the job
|
||||
completed.
|
||||
|
||||
Costs: a failed chapter reports success.
|
||||
Revisit when: any run reports completed without a video. One `if not clip_uris` guard fixes it.
|
||||
Workaround: none.
|
||||
|
||||
## Reviewer timestamps drift against the crossfaded video {#timeline-drift}
|
||||
|
||||
`/review/panels` sums per-panel audio durations. Assemble crossfades clips using the per-beat
|
||||
transitions, so every non-`cut` transition shortens the real video.
|
||||
|
||||
Costs: reviewer timestamps drift further out of sync the further into the chapter they scrub.
|
||||
Revisit when: the review UI is used for timing work.
|
||||
Workaround: subtract the transition overlaps by hand.
|
||||
Reference in New Issue
Block a user