From e8941d8ceb3fe7b9175c102f2e7d11ec716b35c8 Mon Sep 17 00:00:00 2001 From: kami Date: Tue, 11 Aug 2026 23:23:56 +0400 Subject: [PATCH] Stop labelling a model guess as a read tail _annotate_speaker_methods stamped `tail`, the highest-trust provenance, on any line whose speaker matched a present local_id, at gemma's confidence of 1.0. No balloon was read. Three of three sampled two-character panels had the speakers swapped, so a multi-character guess is now dropped to unknown, and a solo-panel guess is kept as model_solo at 0.7. Co-Authored-By: Claude Opus 5 --- JOURNAL.md | 20 ++++++++++++++ NEXT.md | 32 ++++++++++++++++------- caveats/CLAUDE.md | 4 +-- caveats/speaker-attribution.md | 43 +++++++++++++----------------- decisions/CLAUDE.md | 2 ++ decisions/speaker-attribution.md | 45 ++++++++++++++++++++++++++++++++ worker_vision.py | 26 +++++++++++++++--- 7 files changed, 133 insertions(+), 39 deletions(-) create mode 100644 decisions/speaker-attribution.md diff --git a/JOURNAL.md b/JOURNAL.md index 938e73a..fdd38d5 100644 --- a/JOURNAL.md +++ b/JOURNAL.md @@ -101,3 +101,23 @@ turns out to cover two different women. The run then reached `scene` 116/116 and failed in `script` at 87/116, not on OOM: 28 beats were rejected by the script verifier as `unsupported-proper-noun: ['Choi', 'Haeseon']` (`caveats/speaker-attribution.md#multiword-name-verifier`). No two-word cast name can pass that check. + +## 2026-08-11 Speaker provenance and the multi-word cast name + +Command: `.venv/bin/python worker_vision.py`, `pytest -q --ignore=test_api.py` in the orchestrator. +Outcome: both pass, 110 orchestrator tests. Nothing deployed, no GPU work, no pipeline run. +Produced: `decisions/speaker-attribution.md`, two caveats rewritten. + +`_annotate_speaker_methods` stopped stamping `tail` on a model guess. With two or more characters +present the guess is dropped to `unknown` at confidence 0.0. With one present it is kept as +`model_solo` at 0.7, the same claim the solo backstop already makes +(`decisions/speaker-attribution.md#no-fake-tail`). Grounded `som_face` and `solo_prior` rows are +untouched. Nothing outside `worker_vision.py` reads the literal `tail`, checked across both repos. + +`verify_script` now tokenizes each cast name into `allowed`, so `Choi Haeseon` passes as two tokens +(`decisions/speaker-attribution.md#multiword-cast-names`). That is the 28 beats job `778297bc` lost. + +The remaining `['Blur']` beat is a true positive that still halts the whole chapter, now recorded as +`caveats/speaker-attribution.md#one-word-halts-chapter`. + +Neither fix is live. The orchestrator container is not rebuilt and the workers are not restarted. diff --git a/NEXT.md b/NEXT.md index 7bdc421..0eedada 100644 --- a/NEXT.md +++ b/NEXT.md @@ -23,27 +23,41 @@ the rebuild exposed. ## Next -1. Fix speaker attribution. The baseline run has been made and the metric it produced cannot be - trusted (`caveats/speaker-attribution.md#tail-is-not-geometry`). Three of three sampled - two-character panels swap the speakers, and the `tail` provenance label is stamped on guesses at - confidence 1.0. Smallest honest first step: stop labelling a guess `tail`, and return `unknown` - when two or more characters are present. -2. Rerun the chapter and re-read the named-speaker share. Only then is the Phase 1 headline number - real. +Two fixes are written and checked, neither is deployed. + +- `worker_vision.py` no longer labels a model guess `tail`, and drops the guess entirely when two or + more characters are present (`decisions/speaker-attribution.md#no-fake-tail`). The workers need a + restart, and only a rerun from `dialogue` puts it on stored data. +- `correctness.py` tokenizes cast names for the script verifier + (`decisions/speaker-attribution.md#multiword-cast-names`). The orchestrator container needs a + rebuild. + +Then, in order: + +1. Rerun the chapter and re-read the named-speaker share. It will fall, and the lower number is the + first honest one. Deciding whether to resume `script` on the old attributions or clear back to + `dialogue` is open. +2. Bind a balloon to a speaker by tail geometry, using the unused `det`/`seg` heads + (`caveats/speaker-attribution.md#tail-is-not-geometry`). Until then multi-character panels have no + speaker at all. 3. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work (`caveats/audit-open.md#sqlite-locking`). ## The 2026-08-11 chapter run Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116 -panels. It was still in `direct` at 113/116 when the session ended, with `scene script tts layers -render assemble` unrun. Resume it, or read where it got to: +panels. It reached `scene` 116/116 and then failed in `script` at 87/116 on the verifier bug fixed +above. `tts layers render assemble` never ran. Read where it got to, clear the failed stage, resume: ```bash ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" +ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"script\"}'" ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'" ``` +That resume narrates the attributions the old `tail` label produced. Clearing back to `dialogue` +instead re-runs the GPU stages and produces the honest metric. + Numbers and the quality read are in `JOURNAL.md` and `caveats/speaker-attribution.md`. ## Storage and viewer, tasks #116/#117 diff --git a/caveats/CLAUDE.md b/caveats/CLAUDE.md index f81509c..b07997b 100644 --- a/caveats/CLAUDE.md +++ b/caveats/CLAUDE.md @@ -32,7 +32,7 @@ a complaint, so give it one or drop it. | [MinIO credentials are hardcoded in committed source](audit-open.md#hardcoded-credentials) | AUDIT.md | | [Assemble marks a job completed with no clips](audit-open.md#empty-assemble) | AUDIT.md | | [Reviewer timestamps drift against the crossfaded video](audit-open.md#timeline-drift) | AUDIT.md | -| [`speaker_method="tail"` never reads a tail](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run | +| [Nothing attributes a speaker in a multi-character panel](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run | | [One character id covers two different women](speaker-attribution.md#identity-over-merge) | 2026-08-11 run | | [The character registry carries five weeks of wrong names](speaker-attribution.md#registry-pollution) | 2026-08-11 run | -| [A multi-word character name always fails the script verifier](speaker-attribution.md#multiword-name-verifier) | 2026-08-11 run | +| [One invented word still halts the chapter](speaker-attribution.md#one-word-halts-chapter) | 2026-08-11 run | diff --git a/caveats/speaker-attribution.md b/caveats/speaker-attribution.md index 63835ae..3bb106f 100644 --- a/caveats/speaker-attribution.md +++ b/caveats/speaker-attribution.md @@ -2,16 +2,15 @@ Limits found by cross-checking the 2026-08-11 chapter run against the panel images. -## `speaker_method="tail"` never reads a tail {#tail-is-not-geometry} +## Nothing attributes a speaker in a multi-character panel {#tail-is-not-geometry} -`worker_vision.py:376` (`_annotate_speaker_methods`) stamps `tail` on any line whose `speaker` matches -a `local_id` present in the panel. No balloon geometry is consulted. The caller keeps gemma's -confidence, usually 1.0. An unverified model guess thus carries the highest-trust provenance in the -pipeline. The review UI and the flag rules both believe it. +The false `tail` label is gone (`decisions/speaker-attribution.md#no-fake-tail`). What replaced it is a +refusal, not an answer: with two or more characters present, every speech line now returns `unknown`. +On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back +to a `generic-handle`. That is the honest floor, and it is not the fix. -Measured on job `778297bc`, chapter `7c944dd4`: 31 of 81 speech lines are `tail` with two or more -characters present. Three two-character panels were checked against the art, and all three are wrong, -each with the two speakers swapped: +The measurement that forced it, on job `778297bc`, chapter `7c944dd4`. Three two-character panels were +checked against the art. All three are wrong, each with the two speakers swapped: | panel key | line | truth | pipeline | | --- | --- | --- | --- | @@ -31,9 +30,9 @@ per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads ca tail tips and are unused. **Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric -(`ROADMAP.md`). It reads 30% on this run, and the sample says that 30% is itself unreliable. The -metric cannot be trusted until this is fixed. Fix order: stop stamping a guess as `tail` at confidence -1.0. Prefer `unknown` when two or more characters are present. Then bind by tail geometry. +(`ROADMAP.md`). The 30% read on this run counted attributions the sample says are wrong. The next run +will read lower and will be the first honest number. Raising it means binding a balloon to a speaker +by tail geometry, using the unused `det`/`seg` heads. ## One character id covers two different women {#identity-over-merge} @@ -54,19 +53,13 @@ inherits the whole pile. **Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the registry to a chapter or add a reviewed reset. -## A multi-word character name always fails the script verifier {#multiword-name-verifier} +## One invented word still halts the chapter {#one-word-halts-chapter} -`correctness.py:174` builds `allowed` from `cast_names` verbatim, so a two-word name enters the set as -one string, `"choi haeseon"`. `_capitalized_tokens` then yields the tokens `Choi` and `Haeseon` -separately, neither of which is in `allowed`. Every beat whose narration uses a two-word name fails -with `unsupported-proper-noun`. A one-word name such as `Seonho` passes, which is why this went unseen. +The multi-word name failure is fixed (`decisions/speaker-attribution.md#multiword-cast-names`). The +blast radius it exposed is not. `run_stage_script` retries a rejected beat once, then raises, so a +single unsupported token ends the run at that beat. On job `778297bc` one of the 29 lost beats cited +`['Blur']`, an onomatopoeia the model invented. The verifier was right, and the whole chapter still +stopped. -Measured on job `778297bc`: the `script` stage failed at 87/116. 28 of the 29 lost beats cite -`['Choi', 'Haeseon']`, one cites `['Blur']`, an onomatopoeia the model invented. - -Fix: tokenize each cast name when building `allowed`, in `verify_script` -(`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/correctness.py:174`). One line. Existing -`test_script_verify.py` covers the function, so add the two-word case there. - -**Revisit trigger:** immediately. It costs a quarter of the chapter's narration on any title whose -cast has a surname. +**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer +is to flag the beat for review and continue, which is `#136` gate work, not a verifier change. diff --git a/decisions/CLAUDE.md b/decisions/CLAUDE.md index 319ef47..f54fa0a 100644 --- a/decisions/CLAUDE.md +++ b/decisions/CLAUDE.md @@ -30,3 +30,5 @@ still live belongs in `caveats/`. | [One bucket per artifact class, not everything under `manga`](storage-layout.md#bucket-per-artifact) | closed | | [The orchestrator creates missing buckets at startup](storage-layout.md#ensure-buckets) | closed | | [RustFS is staged, not adopted](storage-layout.md#rustfs-staged) | open | +| [A model guess is never labelled `tail`](speaker-attribution.md#no-fake-tail) | closed | +| [Cast names enter the verifier tokenized](speaker-attribution.md#multiword-cast-names) | closed | diff --git a/decisions/speaker-attribution.md b/decisions/speaker-attribution.md new file mode 100644 index 0000000..21c3124 --- /dev/null +++ b/decisions/speaker-attribution.md @@ -0,0 +1,45 @@ +# Speaker attribution and cast names + +Settled 2026-08-11 from the quality cross-check of job `778297bc` +(`JOURNAL.md`, `caveats/speaker-attribution.md`). No GPU work ran and no pipeline run was executed +after the change. Both claims rest on source and on the CPU-only self-checks named below. + +Files: `worker_vision.py` on workpc, `correctness.py` and `test_script_verify.py` in the homesrv +orchestrator (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`). + +## A model guess is never labelled `tail` {#no-fake-tail} + +**Closed.** `speaker_method` names how a speaker was established, and nothing may claim geometry it did +not read. `_annotate_speaker_methods` stamped `tail`, the highest-trust label, on any line whose +speaker matched a `local_id` present in the panel, keeping gemma's confidence of 1.0. No balloon was +ever consulted. + +Evidence: three of three sampled two-character panels had both speakers swapped +(`caveats/speaker-attribution.md#tail-is-not-geometry`). 31 of 81 speech lines carried `tail` with two +or more characters present. + +The label is gone. With two or more characters present the guess is dropped: `speaker` becomes +`unknown`, confidence 0.0, method `unknown`. With one character present the claim equals the solo +backstop, so it is kept as `model_solo` at confidence 0.7. Grounded `som_face` and `solo_prior` rows are +untouched, because the function still skips any row that already carries a method. + +Forbids: minting a provenance label for evidence that was not read, and shipping a multi-character +attribution as truth before balloon geometry exists. +Check: `python worker_vision.py`, the `crowd`/`lone` cases. + +Cost: the named-speaker share will fall. The 30% headline was measured on attributions the sample says +are wrong, so the lower number is the first honest one. + +## Cast names enter the verifier tokenized {#multiword-cast-names} + +**Closed.** `verify_script` compares single capitalized tokens, so every allowed name must be present as +tokens. `allowed` was built from `cast_names` verbatim, which put `"choi haeseon"` in the set as one +string while the checker looked up `Choi` and `Haeseon` separately. + +Evidence: the `script` stage failed at 87/116 on job `778297bc`. 28 of the 29 lost beats cite +`unsupported-proper-noun: ['Choi', 'Haeseon']`. A one-word name such as `Seonho` always passed, which +is why this survived the Phase 1 verifier work +(`decisions/audit-phase1.md#verifier-false-positives`). + +Forbids: adding any future allow-list to `verify_script` as whole strings. +Check: `pytest test_script_verify.py`, `test_multiword_cast_name_is_supported`. diff --git a/worker_vision.py b/worker_vision.py index 68ad8df..ae0de13 100644 --- a/worker_vision.py +++ b/worker_vision.py @@ -374,16 +374,31 @@ def resolve_speakers(dialogue: list, present: list) -> list: def _annotate_speaker_methods(dialogue: list, present: list) -> list: - """Fill provenance for model-attributed lines without overwriting grounded/backstop methods.""" + """Fill provenance for model-attributed lines without overwriting grounded/backstop methods. + + A speaker matching a present local_id used to be stamped `tail`, the highest-trust label, at + gemma's own confidence of 1.0. No balloon geometry was ever read. On the 2026-08-11 chapter every + sampled two-character panel had the speakers swapped + (`caveats/speaker-attribution.md#tail-is-not-geometry`), so with 2+ present the guess is dropped + rather than shipped as truth. With one present it is the same claim as the solo backstop, so it is + kept and named for what it is. + ponytail: drop-on-crowd is the honest floor, not the fix. Bind by tail geometry when the balloon + detector lands, then this branch reads a tail for real.""" local_ids = {c.get("local_id") for c in present if c.get("local_id")} + crowded = len(present) > 1 for d in dialogue: if d.get("speaker_method"): continue speaker = (d.get("speaker") or "").strip() if d.get("type", "speech") not in _SPEECH or not speaker or speaker == "unknown": d["speaker_method"] = "unknown" + elif speaker in local_ids and crowded: + d["speaker"] = "unknown" + d["confidence"] = 0.0 + d["speaker_method"] = "unknown" elif speaker in local_ids: - d["speaker_method"] = "tail" + d["speaker_method"] = "model_solo" + d["confidence"] = min(float(d.get("confidence") or 0.7), 0.7) else: d["speaker_method"] = "turn_taking" return dialogue @@ -976,7 +991,12 @@ if __name__ == "__main__": two = [{"local_id": "person_1"}, {"local_id": "person_2"}] assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], two)[0]["speaker"] == "unknown" assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], [])[0]["speaker"] == "unknown" - assert resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x"}], two)[0]["speaker_method"] == "tail" + # a present-local_id match is gemma's guess, not a read tail: dropped when 2+ are present, + # kept but capped at 0.7 when only one is (caveats/speaker-attribution.md#tail-is-not-geometry) + crowd = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], two)[0] + assert crowd["speaker"] == "unknown" and crowd["speaker_method"] == "unknown" and crowd["confidence"] == 0.0 + lone = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], solo)[0] + assert lone["speaker"] == "person_1" and lone["speaker_method"] == "model_solo" and lone["confidence"] == 0.7 # set-of-mark: gemma answers a face label -> remapped to local_id; a name/unknown passes through lbl = _apply_speaker_labels([{"speaker": "P1"}, {"speaker": "Aria"}, {"speaker": "unknown"}], {"P1": "person_3"})