Fix the real A/V gap: a stream copy across mixed frame rates
The rebuild after 1457556 came out byte-identical to the broken file,
which proved the xfade fix never runs for this chapter. An all-cut chapter
goes down the concat demuxer with -c copy, which writes the output in the
FIRST input's time_base and reinterprets every later packet in it. 14 of
49 clips are 30/1 at 1/15360 against 35 at 25/1 at 1/12800, so those 14
play 15360/12800 = 1.2 too long with their audio untouched. collage_cmd
hardcoded -r 30 and yesterday's FPS sweep missed it.
collage_cmd now emits -r FPS, and assemble probes r_frame_rate across the
clips and routes mixed rates through the re-encoding tree. Rebuilt
chapter.mp4 is 364.120s video against 364.122s audio at 25/1, from
436.392 over 363.675.
Also settle the bbox coordinate space, measured over all 113 detections:
47 boxes have x2 past the 900px panel width, none has y2 past 1000 on
panels up to 2307px tall, and the range is exactly [0, 1000]. It is
gemma's normalized grid, not pixels, whatever the prompt asks for.
/vision converts before returning, which fixes identity's crop, the gated
face pairing that was comparing pixel face boxes against grid boxes, the
set-of-mark boxes and the review UI at once. Checked by eye on panel 7:
five of six boxes now land on their subject, including the foreground
character who had no identity.
The registry still holds boxes and embeddings enrolled from the wrong
space. vision and identity have to re-run, which is GPU work and was not
started.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -300,6 +300,46 @@ class VisionInput(BaseModel):
|
||||
session_id: str = ""
|
||||
|
||||
|
||||
BBOX_GRID = 1000 # gemma's native normalized box grid
|
||||
|
||||
|
||||
def _bbox_to_pixels(chars: list, w: int, h: int) -> list:
|
||||
"""Rewrite every character box from gemma's 0-1000 grid to pixels on this panel.
|
||||
|
||||
The prompt asks for pixels. The model answers on its own normalized grid regardless. Measured over
|
||||
the 113 detections of job 778297bc: 47 boxes had x2 beyond the 900px panel width, and not one had y2
|
||||
beyond 1000 on panels 1257 to 2307px tall. Consumed as pixels the boxes collapse into the top-left
|
||||
corner of the panel, which is how identity came to embed crops of speech balloons and window frames
|
||||
and match them at 0.9, and why gated face pairing returned 7 unknowns out of 7 real faces.
|
||||
|
||||
Convert once here so every consumer sees pixels: `_crop_bbox` in identity, the face pairing below,
|
||||
the set-of-mark boxes, and the review UI's client-side crop.
|
||||
|
||||
ponytail: the prompt still says "pixel bounding box". Rewording it would change what the model
|
||||
emits and needs a GPU run to re-verify, so the boundary converts instead. If a future model really
|
||||
does answer in pixels, this scales them down -- check the box range before swapping models.
|
||||
"""
|
||||
for c in chars:
|
||||
b = c.get("bbox")
|
||||
if not (isinstance(b, list) and len(b) == 4 and all(isinstance(v, (int, float)) for v in b)):
|
||||
continue
|
||||
c["bbox"] = [min(w, max(0, round(b[0] * w / BBOX_GRID))),
|
||||
min(h, max(0, round(b[1] * h / BBOX_GRID))),
|
||||
min(w, max(0, round(b[2] * w / BBOX_GRID))),
|
||||
min(h, max(0, round(b[3] * h / BBOX_GRID)))]
|
||||
return chars
|
||||
|
||||
|
||||
def _panel_size(path: str) -> tuple:
|
||||
"""(width, height) of a panel image, or (0, 0) when it cannot be read."""
|
||||
import cv2
|
||||
img = cv2.imread(path)
|
||||
if img is None:
|
||||
return (0, 0)
|
||||
h, w = img.shape[:2]
|
||||
return (w, h)
|
||||
|
||||
|
||||
@app.post("/vision")
|
||||
async def vision(data: VisionInput):
|
||||
local = transport.get(data.panel_uri, f"{SHM}/vision_{uuid.uuid4().hex[:8]}.png")
|
||||
@@ -315,8 +355,14 @@ async def vision(data: VisionInput):
|
||||
print(f"[vision/detect] parse failed for {data.panel_id} after repair retry: {e}", flush=True)
|
||||
result = {"skip": False, "parse_failed": True, "characters": [], "scene": {}}
|
||||
finally:
|
||||
pw, ph = _panel_size(local)
|
||||
os.remove(local)
|
||||
result.setdefault("characters", [])
|
||||
if pw and ph:
|
||||
_bbox_to_pixels(result["characters"], pw, ph)
|
||||
else:
|
||||
print(f"[vision/detect] panel size unreadable for {data.panel_id}, boxes left normalized",
|
||||
flush=True)
|
||||
result["panel_id"] = data.panel_id
|
||||
return result
|
||||
|
||||
@@ -1128,6 +1174,19 @@ if __name__ == "__main__":
|
||||
# trailing braced prose after a complete object parses (used to burn a repair call)
|
||||
assert _extract_json('{"skip":false}\nnote {see above}')["skip"] is False
|
||||
|
||||
# gemma's boxes arrive on a 0-1000 grid and leave /vision as pixels. The 900x1650 panel below is
|
||||
# panel 7 of job 778297bc: person_5 is Seonho in the foreground, and read as pixels his box lands in
|
||||
# the top sixth of the panel, inside a speech balloon, which is what identity embedded.
|
||||
_ch = [{"local_id": "person_5", "bbox": [222, 405, 654, 1000]},
|
||||
{"local_id": "edge", "bbox": [0, 0, 1000, 1000]},
|
||||
{"local_id": "junk", "bbox": "nope"}]
|
||||
_bbox_to_pixels(_ch, 900, 1650)
|
||||
assert _ch[0]["bbox"] == [200, 668, 589, 1650], _ch[0]["bbox"]
|
||||
assert _ch[1]["bbox"] == [0, 0, 900, 1650], _ch[1]["bbox"] # a clamped box spans the whole panel
|
||||
assert _ch[2]["bbox"] == "nope", _ch[2]["bbox"] # unparseable is left alone, not crashed
|
||||
# the box must now cover the lower half of a tall panel, which the raw grid value never can
|
||||
assert _ch[0]["bbox"][3] > 1000 > _ch[0]["bbox"][1]
|
||||
|
||||
# face->identity pairing is GATED on containment: a face outside every gemma bbox stays unknown.
|
||||
faces = [{"bbox": [10, 10, 30, 30]}, {"bbox": [900, 900, 920, 920]}]
|
||||
present = [{"local_id": "person_1", "name": "Teto", "bbox": [0, 0, 100, 200]}]
|
||||
|
||||
Reference in New Issue
Block a user