Stop labelling a model guess as a read tail
_annotate_speaker_methods stamped `tail`, the highest-trust provenance, on any line whose speaker matched a present local_id, at gemma's confidence of 1.0. No balloon was read. Three of three sampled two-character panels had the speakers swapped, so a multi-character guess is now dropped to unknown, and a solo-panel guess is kept as model_solo at 0.7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+20
@@ -101,3 +101,23 @@ turns out to cover two different women.
|
||||
The run then reached `scene` 116/116 and failed in `script` at 87/116, not on OOM: 28 beats were
|
||||
rejected by the script verifier as `unsupported-proper-noun: ['Choi', 'Haeseon']`
|
||||
(`caveats/speaker-attribution.md#multiword-name-verifier`). No two-word cast name can pass that check.
|
||||
|
||||
## 2026-08-11 Speaker provenance and the multi-word cast name
|
||||
|
||||
Command: `.venv/bin/python worker_vision.py`, `pytest -q --ignore=test_api.py` in the orchestrator.
|
||||
Outcome: both pass, 110 orchestrator tests. Nothing deployed, no GPU work, no pipeline run.
|
||||
Produced: `decisions/speaker-attribution.md`, two caveats rewritten.
|
||||
|
||||
`_annotate_speaker_methods` stopped stamping `tail` on a model guess. With two or more characters
|
||||
present the guess is dropped to `unknown` at confidence 0.0. With one present it is kept as
|
||||
`model_solo` at 0.7, the same claim the solo backstop already makes
|
||||
(`decisions/speaker-attribution.md#no-fake-tail`). Grounded `som_face` and `solo_prior` rows are
|
||||
untouched. Nothing outside `worker_vision.py` reads the literal `tail`, checked across both repos.
|
||||
|
||||
`verify_script` now tokenizes each cast name into `allowed`, so `Choi Haeseon` passes as two tokens
|
||||
(`decisions/speaker-attribution.md#multiword-cast-names`). That is the 28 beats job `778297bc` lost.
|
||||
|
||||
The remaining `['Blur']` beat is a true positive that still halts the whole chapter, now recorded as
|
||||
`caveats/speaker-attribution.md#one-word-halts-chapter`.
|
||||
|
||||
Neither fix is live. The orchestrator container is not rebuilt and the workers are not restarted.
|
||||
|
||||
@@ -23,27 +23,41 @@ the rebuild exposed.
|
||||
|
||||
## Next
|
||||
|
||||
1. Fix speaker attribution. The baseline run has been made and the metric it produced cannot be
|
||||
trusted (`caveats/speaker-attribution.md#tail-is-not-geometry`). Three of three sampled
|
||||
two-character panels swap the speakers, and the `tail` provenance label is stamped on guesses at
|
||||
confidence 1.0. Smallest honest first step: stop labelling a guess `tail`, and return `unknown`
|
||||
when two or more characters are present.
|
||||
2. Rerun the chapter and re-read the named-speaker share. Only then is the Phase 1 headline number
|
||||
real.
|
||||
Two fixes are written and checked, neither is deployed.
|
||||
|
||||
- `worker_vision.py` no longer labels a model guess `tail`, and drops the guess entirely when two or
|
||||
more characters are present (`decisions/speaker-attribution.md#no-fake-tail`). The workers need a
|
||||
restart, and only a rerun from `dialogue` puts it on stored data.
|
||||
- `correctness.py` tokenizes cast names for the script verifier
|
||||
(`decisions/speaker-attribution.md#multiword-cast-names`). The orchestrator container needs a
|
||||
rebuild.
|
||||
|
||||
Then, in order:
|
||||
|
||||
1. Rerun the chapter and re-read the named-speaker share. It will fall, and the lower number is the
|
||||
first honest one. Deciding whether to resume `script` on the old attributions or clear back to
|
||||
`dialogue` is open.
|
||||
2. Bind a balloon to a speaker by tail geometry, using the unused `det`/`seg` heads
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`). Until then multi-character panels have no
|
||||
speaker at all.
|
||||
3. Start Phase 2 from `ROADMAP.md`. Set SQLite `busy_timeout` before any concurrency work
|
||||
(`caveats/audit-open.md#sqlite-locking`).
|
||||
|
||||
## The 2026-08-11 chapter run
|
||||
|
||||
Job `778297bc-e7ce-439d-91b5-8a027060d17f`, chapter `7c944dd4-e972-42c7-ba60-9f6939548e80`, 116
|
||||
panels. It was still in `direct` at 113/116 when the session ended, with `scene script tts layers
|
||||
render assemble` unrun. Resume it, or read where it got to:
|
||||
panels. It reached `scene` 116/116 and then failed in `script` at 87/116 on the verifier bug fixed
|
||||
above. `tts layers render assemble` never ran. Read where it got to, clear the failed stage, resume:
|
||||
|
||||
```bash
|
||||
ssh kami@192.168.1.104 "curl -s 'http://127.0.0.1:9090/job/status?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
|
||||
ssh kami@192.168.1.104 "curl -s -X POST http://127.0.0.1:9090/stage/clear -H 'Content-Type: application/json' -d '{\"job_id\":\"778297bc-e7ce-439d-91b5-8a027060d17f\",\"stage\":\"script\"}'"
|
||||
ssh kami@192.168.1.104 "curl -s -X POST 'http://127.0.0.1:9090/job/resume?job_id=778297bc-e7ce-439d-91b5-8a027060d17f'"
|
||||
```
|
||||
|
||||
That resume narrates the attributions the old `tail` label produced. Clearing back to `dialogue`
|
||||
instead re-runs the GPU stages and produces the honest metric.
|
||||
|
||||
Numbers and the quality read are in `JOURNAL.md` and `caveats/speaker-attribution.md`.
|
||||
|
||||
## Storage and viewer, tasks #116/#117
|
||||
|
||||
+2
-2
@@ -32,7 +32,7 @@ a complaint, so give it one or drop it.
|
||||
| [MinIO credentials are hardcoded in committed source](audit-open.md#hardcoded-credentials) | AUDIT.md |
|
||||
| [Assemble marks a job completed with no clips](audit-open.md#empty-assemble) | AUDIT.md |
|
||||
| [Reviewer timestamps drift against the crossfaded video](audit-open.md#timeline-drift) | AUDIT.md |
|
||||
| [`speaker_method="tail"` never reads a tail](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run |
|
||||
| [Nothing attributes a speaker in a multi-character panel](speaker-attribution.md#tail-is-not-geometry) | 2026-08-11 run |
|
||||
| [One character id covers two different women](speaker-attribution.md#identity-over-merge) | 2026-08-11 run |
|
||||
| [The character registry carries five weeks of wrong names](speaker-attribution.md#registry-pollution) | 2026-08-11 run |
|
||||
| [A multi-word character name always fails the script verifier](speaker-attribution.md#multiword-name-verifier) | 2026-08-11 run |
|
||||
| [One invented word still halts the chapter](speaker-attribution.md#one-word-halts-chapter) | 2026-08-11 run |
|
||||
|
||||
@@ -2,16 +2,15 @@
|
||||
|
||||
Limits found by cross-checking the 2026-08-11 chapter run against the panel images.
|
||||
|
||||
## `speaker_method="tail"` never reads a tail {#tail-is-not-geometry}
|
||||
## Nothing attributes a speaker in a multi-character panel {#tail-is-not-geometry}
|
||||
|
||||
`worker_vision.py:376` (`_annotate_speaker_methods`) stamps `tail` on any line whose `speaker` matches
|
||||
a `local_id` present in the panel. No balloon geometry is consulted. The caller keeps gemma's
|
||||
confidence, usually 1.0. An unverified model guess thus carries the highest-trust provenance in the
|
||||
pipeline. The review UI and the flag rules both believe it.
|
||||
The false `tail` label is gone (`decisions/speaker-attribution.md#no-fake-tail`). What replaced it is a
|
||||
refusal, not an answer: with two or more characters present, every speech line now returns `unknown`.
|
||||
On a chapter like this one that costs 31 of 81 speech lines their speaker. The narration then falls back
|
||||
to a `generic-handle`. That is the honest floor, and it is not the fix.
|
||||
|
||||
Measured on job `778297bc`, chapter `7c944dd4`: 31 of 81 speech lines are `tail` with two or more
|
||||
characters present. Three two-character panels were checked against the art, and all three are wrong,
|
||||
each with the two speakers swapped:
|
||||
The measurement that forced it, on job `778297bc`, chapter `7c944dd4`. Three two-character panels were
|
||||
checked against the art. All three are wrong, each with the two speakers swapped:
|
||||
|
||||
| panel key | line | truth | pipeline |
|
||||
| --- | --- | --- | --- |
|
||||
@@ -31,9 +30,9 @@ per-balloon geometry. `bubble_detect.py:9` records that the `det`/`seg` heads ca
|
||||
tail tips and are unused.
|
||||
|
||||
**Revisit trigger:** the share of narrated lines with a named speaker is the Phase 1 headline metric
|
||||
(`ROADMAP.md`). It reads 30% on this run, and the sample says that 30% is itself unreliable. The
|
||||
metric cannot be trusted until this is fixed. Fix order: stop stamping a guess as `tail` at confidence
|
||||
1.0. Prefer `unknown` when two or more characters are present. Then bind by tail geometry.
|
||||
(`ROADMAP.md`). The 30% read on this run counted attributions the sample says are wrong. The next run
|
||||
will read lower and will be the first honest number. Raising it means binding a balloon to a speaker
|
||||
by tail geometry, using the unused `det`/`seg` heads.
|
||||
|
||||
## One character id covers two different women {#identity-over-merge}
|
||||
|
||||
@@ -54,19 +53,13 @@ inherits the whole pile.
|
||||
**Revisit trigger:** before any run that is meant to produce a clean baseline. Either scope the
|
||||
registry to a chapter or add a reviewed reset.
|
||||
|
||||
## A multi-word character name always fails the script verifier {#multiword-name-verifier}
|
||||
## One invented word still halts the chapter {#one-word-halts-chapter}
|
||||
|
||||
`correctness.py:174` builds `allowed` from `cast_names` verbatim, so a two-word name enters the set as
|
||||
one string, `"choi haeseon"`. `_capitalized_tokens` then yields the tokens `Choi` and `Haeseon`
|
||||
separately, neither of which is in `allowed`. Every beat whose narration uses a two-word name fails
|
||||
with `unsupported-proper-noun`. A one-word name such as `Seonho` passes, which is why this went unseen.
|
||||
The multi-word name failure is fixed (`decisions/speaker-attribution.md#multiword-cast-names`). The
|
||||
blast radius it exposed is not. `run_stage_script` retries a rejected beat once, then raises, so a
|
||||
single unsupported token ends the run at that beat. On job `778297bc` one of the 29 lost beats cited
|
||||
`['Blur']`, an onomatopoeia the model invented. The verifier was right, and the whole chapter still
|
||||
stopped.
|
||||
|
||||
Measured on job `778297bc`: the `script` stage failed at 87/116. 28 of the 29 lost beats cite
|
||||
`['Choi', 'Haeseon']`, one cites `['Blur']`, an onomatopoeia the model invented.
|
||||
|
||||
Fix: tokenize each cast name when building `allowed`, in `verify_script`
|
||||
(`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/correctness.py:174`). One line. Existing
|
||||
`test_script_verify.py` covers the function, so add the two-word case there.
|
||||
|
||||
**Revisit trigger:** immediately. It costs a quarter of the chapter's narration on any title whose
|
||||
cast has a surname.
|
||||
**Revisit trigger:** the next `unsupported-proper-noun` halt that is a true positive. The likely answer
|
||||
is to flag the beat for review and continue, which is `#136` gate work, not a verifier change.
|
||||
|
||||
@@ -30,3 +30,5 @@ still live belongs in `caveats/`.
|
||||
| [One bucket per artifact class, not everything under `manga`](storage-layout.md#bucket-per-artifact) | closed |
|
||||
| [The orchestrator creates missing buckets at startup](storage-layout.md#ensure-buckets) | closed |
|
||||
| [RustFS is staged, not adopted](storage-layout.md#rustfs-staged) | open |
|
||||
| [A model guess is never labelled `tail`](speaker-attribution.md#no-fake-tail) | closed |
|
||||
| [Cast names enter the verifier tokenized](speaker-attribution.md#multiword-cast-names) | closed |
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Speaker attribution and cast names
|
||||
|
||||
Settled 2026-08-11 from the quality cross-check of job `778297bc`
|
||||
(`JOURNAL.md`, `caveats/speaker-attribution.md`). No GPU work ran and no pipeline run was executed
|
||||
after the change. Both claims rest on source and on the CPU-only self-checks named below.
|
||||
|
||||
Files: `worker_vision.py` on workpc, `correctness.py` and `test_script_verify.py` in the homesrv
|
||||
orchestrator (`/mnt/server/home/kami/docker-apps/manga-infra/orchestrator/`).
|
||||
|
||||
## A model guess is never labelled `tail` {#no-fake-tail}
|
||||
|
||||
**Closed.** `speaker_method` names how a speaker was established, and nothing may claim geometry it did
|
||||
not read. `_annotate_speaker_methods` stamped `tail`, the highest-trust label, on any line whose
|
||||
speaker matched a `local_id` present in the panel, keeping gemma's confidence of 1.0. No balloon was
|
||||
ever consulted.
|
||||
|
||||
Evidence: three of three sampled two-character panels had both speakers swapped
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`). 31 of 81 speech lines carried `tail` with two
|
||||
or more characters present.
|
||||
|
||||
The label is gone. With two or more characters present the guess is dropped: `speaker` becomes
|
||||
`unknown`, confidence 0.0, method `unknown`. With one character present the claim equals the solo
|
||||
backstop, so it is kept as `model_solo` at confidence 0.7. Grounded `som_face` and `solo_prior` rows are
|
||||
untouched, because the function still skips any row that already carries a method.
|
||||
|
||||
Forbids: minting a provenance label for evidence that was not read, and shipping a multi-character
|
||||
attribution as truth before balloon geometry exists.
|
||||
Check: `python worker_vision.py`, the `crowd`/`lone` cases.
|
||||
|
||||
Cost: the named-speaker share will fall. The 30% headline was measured on attributions the sample says
|
||||
are wrong, so the lower number is the first honest one.
|
||||
|
||||
## Cast names enter the verifier tokenized {#multiword-cast-names}
|
||||
|
||||
**Closed.** `verify_script` compares single capitalized tokens, so every allowed name must be present as
|
||||
tokens. `allowed` was built from `cast_names` verbatim, which put `"choi haeseon"` in the set as one
|
||||
string while the checker looked up `Choi` and `Haeseon` separately.
|
||||
|
||||
Evidence: the `script` stage failed at 87/116 on job `778297bc`. 28 of the 29 lost beats cite
|
||||
`unsupported-proper-noun: ['Choi', 'Haeseon']`. A one-word name such as `Seonho` always passed, which
|
||||
is why this survived the Phase 1 verifier work
|
||||
(`decisions/audit-phase1.md#verifier-false-positives`).
|
||||
|
||||
Forbids: adding any future allow-list to `verify_script` as whole strings.
|
||||
Check: `pytest test_script_verify.py`, `test_multiword_cast_name_is_supported`.
|
||||
+23
-3
@@ -374,16 +374,31 @@ def resolve_speakers(dialogue: list, present: list) -> list:
|
||||
|
||||
|
||||
def _annotate_speaker_methods(dialogue: list, present: list) -> list:
|
||||
"""Fill provenance for model-attributed lines without overwriting grounded/backstop methods."""
|
||||
"""Fill provenance for model-attributed lines without overwriting grounded/backstop methods.
|
||||
|
||||
A speaker matching a present local_id used to be stamped `tail`, the highest-trust label, at
|
||||
gemma's own confidence of 1.0. No balloon geometry was ever read. On the 2026-08-11 chapter every
|
||||
sampled two-character panel had the speakers swapped
|
||||
(`caveats/speaker-attribution.md#tail-is-not-geometry`), so with 2+ present the guess is dropped
|
||||
rather than shipped as truth. With one present it is the same claim as the solo backstop, so it is
|
||||
kept and named for what it is.
|
||||
ponytail: drop-on-crowd is the honest floor, not the fix. Bind by tail geometry when the balloon
|
||||
detector lands, then this branch reads a tail for real."""
|
||||
local_ids = {c.get("local_id") for c in present if c.get("local_id")}
|
||||
crowded = len(present) > 1
|
||||
for d in dialogue:
|
||||
if d.get("speaker_method"):
|
||||
continue
|
||||
speaker = (d.get("speaker") or "").strip()
|
||||
if d.get("type", "speech") not in _SPEECH or not speaker or speaker == "unknown":
|
||||
d["speaker_method"] = "unknown"
|
||||
elif speaker in local_ids and crowded:
|
||||
d["speaker"] = "unknown"
|
||||
d["confidence"] = 0.0
|
||||
d["speaker_method"] = "unknown"
|
||||
elif speaker in local_ids:
|
||||
d["speaker_method"] = "tail"
|
||||
d["speaker_method"] = "model_solo"
|
||||
d["confidence"] = min(float(d.get("confidence") or 0.7), 0.7)
|
||||
else:
|
||||
d["speaker_method"] = "turn_taking"
|
||||
return dialogue
|
||||
@@ -976,7 +991,12 @@ if __name__ == "__main__":
|
||||
two = [{"local_id": "person_1"}, {"local_id": "person_2"}]
|
||||
assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], two)[0]["speaker"] == "unknown"
|
||||
assert resolve_speakers([{"speaker": "unknown", "type": "speech", "text": "x"}], [])[0]["speaker"] == "unknown"
|
||||
assert resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x"}], two)[0]["speaker_method"] == "tail"
|
||||
# a present-local_id match is gemma's guess, not a read tail: dropped when 2+ are present,
|
||||
# kept but capped at 0.7 when only one is (caveats/speaker-attribution.md#tail-is-not-geometry)
|
||||
crowd = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], two)[0]
|
||||
assert crowd["speaker"] == "unknown" and crowd["speaker_method"] == "unknown" and crowd["confidence"] == 0.0
|
||||
lone = resolve_speakers([{"speaker": "person_1", "type": "speech", "text": "x", "confidence": 1.0}], solo)[0]
|
||||
assert lone["speaker"] == "person_1" and lone["speaker_method"] == "model_solo" and lone["confidence"] == 0.7
|
||||
# set-of-mark: gemma answers a face label -> remapped to local_id; a name/unknown passes through
|
||||
lbl = _apply_speaker_labels([{"speaker": "P1"}, {"speaker": "Aria"}, {"speaker": "unknown"}],
|
||||
{"P1": "person_3"})
|
||||
|
||||
Reference in New Issue
Block a user