From 63f7918a3e951b60f4904e4f1f59564bf54ea377 Mon Sep 17 00:00:00 2001 From: kami Date: Tue, 11 Aug 2026 23:59:32 +0400 Subject: [PATCH] Record all six defects and the 9% baseline Adds decision entries for the unpaired set-of-mark label, the interjection verifier false positive, and the vision-blob clearing bug, plus the per-run speaker audit script used to measure the chapter. Co-Authored-By: Claude Opus 5 --- HANDOFF.md | 113 +++++++++++++++++-------------- JOURNAL.md | 7 ++ audit_speakers.py | 105 ++++++++++++++++++++++++++++ caveats/audit-open.md | 10 ++- decisions/CLAUDE.md | 3 + decisions/speaker-attribution.md | 27 ++++++++ decisions/storage-layout.md | 16 +++++ 7 files changed, 229 insertions(+), 52 deletions(-) create mode 100644 audit_speakers.py diff --git a/HANDOFF.md b/HANDOFF.md index aee2d4c..e531e1a 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,71 +1,86 @@ -# HANDOFF, 2026-08-11 session +# HANDOFF, 2026-08-11 late session -Live state lives in `NEXT.md`. This file is only what this session did. +Live state is in `NEXT.md`. This file is only what this session did. ## Asked -1. Add the buckets to rustfs. -2. Fix the bucket saving on the orchestrator side. -3. Run the title recap. +Continue from the previous handoff: fix speaker attribution, then rerun and re-read the named-speaker +share. Mid-session: what happens to unidentified people in a two-person panel, can a known character be +mis-named, and keep watch during the run. -Mid-session: check `manga.kvmx.ru`, and cross-check whether the run got dialogue and identities right. +## Result + +The named-speaker share is **9 of 95 speech lines, 9%**. The previous 30% counted fake tails. All 9 binds +are `Choi Haeseon`, the over-merged row. `script` passed 116/116 for the first time. + +**Identity, not attribution, is now the constraint.** 26 of 113 detected people carry an identity. 25 of +the 26 are that one row. This chapter caps near 23% named even with perfect balloon binding. + +Six defects, four found by measuring the run rather than by reading code. Each has a decision entry: + +| # | defect | entry | +| --- | --- | --- | +| 1 | `tail` stamped on a model guess at confidence 1.0 | `decisions/speaker-attribution.md#no-fake-tail` | +| 2 | a two-word cast name always failed the script verifier | `#multiword-cast-names` | +| 3 | `/stage/clear dialogue` deleted nothing and reported success | `decisions/storage-layout.md#clear-vision-blob` | +| 4 | gemma's speaker answer echoed the prompt label, became a name | `#prompt-label-answers` | +| 5 | `som_face` stamped on a mark that paired to nobody | `#unpaired-mark` | +| 6 | one two-letter interjection halted the chapter at 112/116 | `#interjection-false-positive` | ## Changed -Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`: +Workers, `/home/kami/Programs/n8n-worker`, branch `restore-runtime`, commits `e8941d8 8071137 a965077` +plus docs: -- `worker_crop.py` panels -> `s3://panels/` -- `worker_tts.py` wavs -> `s3://audio/` -- `worker_layers.py` layers -> `s3://layers/` -- `worker_render.py` clips and `chapter.mp4` -> `s3://video/` -- `decisions/storage-layout.md`, `caveats/speaker-attribution.md`, indexes, `JOURNAL.md`, `NEXT.md` +- `worker_vision.py`: `_annotate_speaker_methods`, `_apply_speaker_labels`, new `_present_keys` +- `decisions/speaker-attribution.md` (new, 5 sections), `decisions/storage-layout.md#clear-vision-blob`, + `caveats/speaker-attribution.md`, `caveats/audit-open.md#dishonest-clearing`, both indexes, + `JOURNAL.md`, `NEXT.md` -Orchestrator, `/mnt/server/home/kami/docker-apps/manga-infra/orchestrator`: +Orchestrator, `/mnt/server/home/kami/docker-apps`, commits `b18b6b4 603d388 db8d7c5 ccc3a8e` plus the +interjection commit: -- `minio_layout.py` per-artifact bucket constants, `BUCKETS`, `parse_key` accepts any of them -- `service.py` `_s3_delete_prefix` takes `/`, `_stage_s3_prefixes` no longer slices, - new `_ensure_buckets` in the lifespan -- `test_minio_layout.py` updated, one test added - -Infrastructure, homesrv: - -- six buckets created on `rfs` (rustfs, `127.0.0.1:9010`) -- `manga-fetch` started, exited 2 weeks, `/job/create` fails without it -- `manga-web` started, exited 2 weeks, `manga.kvmx.ru` -> nginx -> `localhost:8083` -- orchestrator rebuilt and restarted -- `mc` aliases `mio` and `rfs` now exist on homesrv +- `correctness.py`: tokenized `allowed`, `_ID_SHAPED` guard in `normalize_speaker`, interjection + stopwords, short-quote grounding skip +- `db.py`: `_STAGE_VISION_KEYS` and the strip pass in `clear_stage_data` +- `test_script_verify.py`, `test_correctness.py`, `test_db.py`: one case each ## Measured -Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, -"Teto X Egen", 116 panels. +Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 panels. -- 109 orchestrator tests pass. `worker_crop.py`, `worker_tts.py`, `worker_render.py` self-checks pass. - `worker_layers.py` self-check fails on the missing `legacy/qwen_layered_workflow.json`, which - predates this session. -- buckets after the run: `raw` 64MiB/79, `panels` 101MiB/116, `manga` 366MiB/491, `audio` `layers` - `video` still empty because the run had not reached those stages. -- stage times: crop 85s, vision ~4min, identity ~1min, reconcile ~7min, dialogue ~8min. -- 24 of 81 speech lines resolve to a named character, 30%. -- 26 of 113 detected people got an identity, 23%, and 25 of the 26 went to one character. -- 3 of 3 sampled two-character panels attribute both speakers to the wrong person. +- 113 orchestrator tests pass. `worker_vision.py` self-check passes. +- `dialogue` 116/116, `direct` 116/116, `scene` 116/116, `script` 116/116. +- speech lines 95, named 9. Multi-character panels 0 of 40 by design. Single-character 9 of 55. +- `speaker_method`: `unknown` 47, `model_solo` 28, `solo_prior` 8, `som_face` 7, `turn_taking` 5. +- unresolved name refs 3, all `brown ponytail, green dress`, from a neighbouring panel in the same + 8-panel window. Was 24 of 51 before the fix. +- identity assignments 26, of which `character_afa762` "Choi Haeseon" holds 25. +- registry duplicates that block a correct bind: `seonho` matches 2 rows, `lim seonho` matches 2 rows. +- buckets during `tts`: `panels` 116, `raw` 79, `manga` 491, `audio` 1, `layers` 0, `video` 0. ## Open -- The `script` stage failed at 87/116 with `unsupported-proper-noun: ['Choi', 'Haeseon']` on 28 beats. - Not OOM, the worker stayed healthy. One-line fix named in - `caveats/speaker-attribution.md#multiword-name-verifier`. Not applied. -- The run stopped in `direct` at 113/116. `scene script tts layers render assemble` never ran. Resume: - - Since then it reached `scene` 116/116 and failed in `script`. Fix the verifier, then: +- **The run is still going.** It was in `tts` at 26/116 when this was written, with `layers render + assemble` unrun. `tts` is the bottleneck. Read it: ```bash - ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"script\"}'" - ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" + /usr/bin/ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" ``` -- Speaker attribution is wrong in multi-character panels and the fix is not written - (`caveats/speaker-attribution.md`). No code was changed for it. -- Objects from the 2026-07-17 run still sit under `manga///{pages,panels,audio, - layers,clips}`. Nothing reads them. They are the rollback, not live data. -- `rustfs` holds empty buckets. No mirror, no cutover. + If it failed, clear the failed stage and resume: + + ```bash + /usr/bin/ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"\"}'" + /usr/bin/ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" + ``` + + Note: plain `ssh` is the kitty ssh kitten and refuses non-interactive stdin. Use `/usr/bin/ssh`. + +- Defects 5 and 6 landed after `dialogue` had already run, so this run's 7 `som_face` lines are still + labelled from the unpaired-mark path. The next dialogue pass fixes that. No name was affected. +- The audit script is `docker exec manga-orchestrator python3 /tmp/audit_speakers.py`, source in this + session's scratchpad. It is not in the repo. Copy it in if the metric is to be tracked per run. +- The 3 cross-panel unresolved refs need the window's whole present-list, not one panel's. +- Workers were restarted twice this session and are running in tmux `manga-workers`. Nothing watches + them, and nothing watches the homesrv containers. diff --git a/JOURNAL.md b/JOURNAL.md index 6425ff7..a5d813e 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -153,3 +153,10 @@ Identity is now the binding constraint, not attribution. 26 of 113 detected peop 23% named. The person who does hold an identity is stored as `Lim Seonho` while a separate row is named `Seonho` with alias `Lim Seonho`, so either name matches two rows, raises `ambiguous-speaker` and binds nothing. + +Two more defects surfaced after `dialogue` finished. All 7 `som_face` lines pointed at a mark whose face +paired to no present character (`decisions/speaker-attribution.md#unpaired-mark`). Then `script` halted +at 112/116 because the narrator wrote `"...Hm?"` for the source line `"Uh... hum...?"`, and both verifier +rules fired on that two-letter interjection +(`decisions/speaker-attribution.md#interjection-false-positive`). After the fix, `script` passed 116/116, +the first time this chapter has cleared the verifier. `tts` then ran for the first time. diff --git a/audit_speakers.py b/audit_speakers.py new file mode 100644 index 0000000..30791b0 --- /dev/null +++ b/audit_speakers.py @@ -0,0 +1,105 @@ +"""Speaker-attribution audit for one chapter. Runs inside manga-orchestrator (reads /data/manga.db). + +Answers the three questions asked of the 2026-08-11 rerun: + 1. what happens on a multi-character panel whose people have no identity, + 2. whether a known character gets mis-named, + 3. what the named-speaker share actually is now. +""" +import collections +import json +import sqlite3 +import sys + +CHAPTER = sys.argv[1] if len(sys.argv) > 1 else "7c944dd4-e972-42c7-ba60-9f6939548e80" +SPEECH = {"speech", "shout", "thought"} + +c = sqlite3.connect("/data/manga.db") +c.row_factory = sqlite3.Row +manga_id = c.execute("SELECT manga_id FROM chapters WHERE chapter_id=?", (CHAPTER,)).fetchone()[0] + +reg = {r["character_id"]: dict(r) for r in c.execute( + "SELECT character_id, name, aliases, description FROM characters WHERE manga_id=?", (manga_id,))} + +# duplicate registry rows make a correct name unresolvable: an answer matching two rows is ambiguous. +by_name = collections.defaultdict(list) +for cid, r in reg.items(): + for n in [r["name"], *json.loads(r["aliases"] or "[]")]: + if n: + by_name[str(n).strip().casefold()].append(cid) +dupes = {n: ids for n, ids in by_name.items() if len(ids) > 1} + +panels = c.execute("SELECT panel_id, panel_index, page_index FROM panels WHERE chapter_id=? ORDER BY \"order\"", + (CHAPTER,)).fetchall() + +methods, kinds, per_char = collections.Counter(), collections.Counter(), collections.Counter() +speech = named = 0 +crowded_speech = crowded_named = 0 +solo_speech = solo_named = 0 +unresolved_names = collections.Counter() +ambiguous = [] +no_identity_crowd = 0 +# a line attributed to a real present local_id that simply has no identity row: attribution succeeded +# and the name is lost anyway. This is the identity-coverage wall, not an attribution failure. +attributed_but_unassigned = collections.Counter() + +for p in panels: + pid = p["panel_id"] + row = c.execute("SELECT result_json FROM vision_results WHERE panel_id=?", (pid,)).fetchone() + if not row: + continue + v = json.loads(row["result_json"]) + if "dialogue" not in v: + continue + people = [ch for ch in (v.get("characters") or []) if ch.get("local_id")] + assigned = {a["local_id"] for a in c.execute( + "SELECT local_id FROM identity_assignments WHERE panel_id=?", (pid,))} + crowd = len(people) > 1 + if crowd and not assigned: + no_identity_crowd += 1 + for d in v["dialogue"]: + if d.get("type", "speech") not in SPEECH: + continue + speech += 1 + ref = d.get("speaker_ref") or {} + kind = ref.get("kind") + methods[d.get("speaker_method")] += 1 + kinds[kind] += 1 + if crowd: + crowded_speech += 1 + else: + solo_speech += 1 + if kind == "character_id": + named += 1 + per_char[reg.get(ref["value"], {}).get("name") or ref["value"]] += 1 + if crowd: + crowded_named += 1 + else: + solo_named += 1 + elif kind == "unknown": + # normalize_dialogue nulls the flat `speaker` unless it resolved, but speaker_ref keeps the + # unresolved local_id as its value. + raw = str(ref.get("value") or "").strip() + local_ids = {ch["local_id"] for ch in people} + if raw in local_ids and raw not in assigned: + attributed_but_unassigned[d.get("speaker_method")] += 1 + elif kind == "name": + unresolved_names[ref.get("value")] += 1 + if ref.get("candidates"): + ambiguous.append((pid, ref.get("value"), ref["candidates"])) + +print(f"panels with dialogue: {sum(1 for p in panels if (lambda r: r and 'dialogue' in json.loads(r['result_json']))(c.execute('SELECT result_json FROM vision_results WHERE panel_id=?', (p['panel_id'],)).fetchone()))}/{len(panels)}") +print(f"speech lines: {speech} named (character_id): {named} = {100*named/max(speech,1):.0f}%") +print(f" multi-character panels: {crowded_named}/{crowded_speech} named") +print(f" single-character panels: {solo_named}/{solo_speech} named") +print(f"multi-character panels with NO identity at all: {no_identity_crowd}") +print(f"attributed to a present local_id with NO identity row: {dict(attributed_but_unassigned)}") +print(f"speaker_method: {dict(methods)}") +print(f"speaker_ref kind: {dict(kinds)}") +print(f"named per character: {dict(per_char)}") +print(f"unresolved name refs: {dict(unresolved_names)}") +print(f"registry duplicate names (block a correct bind): " + f"{ {n: [reg[i]['name'] for i in ids] for n, ids in dupes.items()} }") +if ambiguous: + print("ambiguous binds:") + for pid, val, cands in ambiguous[:20]: + print(f" {pid} {val!r} -> {[reg.get(x, {}).get('name') for x in cands]}") diff --git a/caveats/audit-open.md b/caveats/audit-open.md index acd8a7c..afea734 100644 --- a/caveats/audit-open.md +++ b/caveats/audit-open.md @@ -14,13 +14,17 @@ Workaround: none. Clear identity and rerun, which loses the good merges too. ## Clearing a stage does not undo what it wrote {#dishonest-clearing} -Dialogue and direction mutate the shared vision JSON. Clearing dialogue leaves its keys in place, so a -rerun treats old dialogue as completed. Clearing identity preserves the per-manga registry. +The dialogue and direct half of this is fixed and proven +(`decisions/storage-layout.md#clear-vision-blob`). It cost a wasted rerun on 2026-08-11 first: the clear +returned `{"ok": true}`, deleted nothing, and the stage skipped all 116 panels. + +What remains: clearing identity preserves the per-manga registry by design, so a rerun inherits every +character it ever minted (`caveats/speaker-attribution.md#registry-pollution`). Nothing verifies that a +clear emptied what it claimed. Costs: a rerun silently reuses stale output, which reads as a reproducible result. Revisit when: any stage is scheduled concurrently or resumed automatically. A stage must be idempotent before either is safe. -Workaround: delete the keys by hand, or clear from `crop` down. ## ComfyUI uses the GPU outside the session mutex {#comfyui-unscheduled} diff --git a/decisions/CLAUDE.md b/decisions/CLAUDE.md index e8d5022..f695ef6 100644 --- a/decisions/CLAUDE.md +++ b/decisions/CLAUDE.md @@ -32,4 +32,7 @@ still live belongs in `caveats/`. | [RustFS is staged, not adopted](storage-layout.md#rustfs-staged) | open | | [A model guess is never labelled `tail`](speaker-attribution.md#no-fake-tail) | closed | | [The model's speaker answer is resolved against what the prompt showed](speaker-attribution.md#prompt-label-answers) | closed | +| [An unpaired set-of-mark label grounds nothing](speaker-attribution.md#unpaired-mark) | closed | +| [An interjection is not a name and not a misquote](speaker-attribution.md#interjection-false-positive) | closed | | [Cast names enter the verifier tokenized](speaker-attribution.md#multiword-cast-names) | closed | +| [Clearing a stage strips the vision blob it wrote](storage-layout.md#clear-vision-blob) | closed | diff --git a/decisions/speaker-attribution.md b/decisions/speaker-attribution.md index 4c867ac..0e7da37 100644 --- a/decisions/speaker-attribution.md +++ b/decisions/speaker-attribution.md @@ -54,6 +54,33 @@ string as a name. Check: `python worker_vision.py`, the `shown`/`twins` cases. `pytest test_correctness.py`, `test_an_id_shaped_speaker_is_never_a_name`. +## An unpaired mark grounds nothing {#unpaired-mark} + +**Closed.** `_set_of_mark` labels a detected face `unknown` when gated pairing +(`decisions/audit-phase1.md#gated-face-pairing`) matched it to no present character. An answer pointing +at such a mark identifies nobody, so it must not carry `som_face`, the highest-trust provenance. + +Evidence: all 7 `som_face` lines in the first 36 panels of the 2026-08-11 rerun had `speaker_ref` kind +`unknown`. The label sat on lines with no speaker. Same defect class as the fake `tail`. + +Forbids: deriving a provenance label from the label map without checking what the label resolved to. +Check: `python worker_vision.py`, the `unpaired` case. + +## An interjection is not a name and not a misquote {#interjection-false-positive} + +**Closed.** `verify_script` must stay quiet on valid narration, because `run_stage_script` retries once +and then raises (`decisions/audit-phase1.md#verifier-false-positives`). + +Evidence: the narrator wrote `"...Hm?"` for the source line `"Uh... hum...?"`. Both rules fired at once. +`Hm` was absent from the source words, and a 6-character quote needs 5 matching characters to ground, +so it got 4. That halted the `script` stage at 112/116 on job `778297bc`. + +Interjections join `_STOPWORDS`. A quote of three letters or fewer is no longer grounded-checked. That +holds the same line as the dialogue prompt's 1-3 character noise rule. The stage then passed 116/116. + +Forbids: scoring a quote too short for the ratio to carry meaning. +Check: `pytest test_script_verify.py`, `test_an_interjection_is_not_a_name_or_a_misquote`. + ## Cast names enter the verifier tokenized {#multiword-cast-names} **Closed.** `verify_script` compares single capitalized tokens, so every allowed name must be present as diff --git a/decisions/storage-layout.md b/decisions/storage-layout.md index 0a97a88..0eeaa46 100644 --- a/decisions/storage-layout.md +++ b/decisions/storage-layout.md @@ -56,3 +56,19 @@ instead. The check is not worth a failed start. write. Nothing is repointed. Two things still block a cutover, and neither is settled. RustFS is `1.0.0-beta.12`, labeled `build-type=prerelease`. Swapping storage also adds a variable to the run meant to produce the baseline. Task [#116]. + +## Clearing a stage strips the vision blob it wrote {#clear-vision-blob} + +**Closed. 2026-08-11.** `dialogue` and `direct` have no output table. They write onto the per-panel +vision blob, and `_STAGE_TABLES` had no entry for either, so `/stage/clear dialogue` deleted nothing and +still returned `{"ok": true}`. `run_stage_dialogue` then saw `"dialogue" in vision` and skipped all 116 +panels. + +Evidence: the first clear reported `scripts: 46, scene_graphs: 116` and no vision counts. After the fix +the same call reported `vision_results.dialogue: 116` and `vision_results.direct: 75`, all of which the +first clear had left in place. A whole rerun was wasted on stale data before this was found. + +`_STAGE_VISION_KEYS` in `db.py` names the keys each stage owns, and `clear_stage_data` strips them. + +Forbids: adding a stage that writes onto a shared blob without listing its keys there. +Check: `pytest test_db.py`, `TestClearStageData`.