Merge the audio-in routing measurement (#176)

2026-08-05 17:06:05 +04:00
parent b621c477a0 42a7bd88b2
commit 52d80394ec
2 changed files with 101 additions and 1 deletions
@@ -0,0 +1,93 @@
 # Routing from audio: four paths, one fixture
 **05-08-2026. Vikunja #486.** Workstation `gemma-4-12B-it-qat-UD-Q4_K_XL` with
 `mmproj-F16.gguf`, homesrv whisper `ggml-small`, piper `ru_RU-irina-medium`.
 **Verdict: transcribe, then route.** One call from audio straight to a route loses 36
 points, so it is not a candidate. Moving speech-to-text to the workstation buys 375ms and
 better transcripts at no measurable accuracy cost. So #486 proceeds on the two-call shape.
 ## The numbers
 72 Russian cases from `internal/router/eval/ru_routing_v1.json`, rendered by piper at
 16kHz mono, 153.9s of audio, mean 2.14s per clip. Every path used the daemon's own
 `routeSystem` prompt and `routeGrammar`, read out of `internal/router/llmrouter.go` at run
 time, at `temperature 0` and `enable_thinking:false`.
 | Path | Intent-only | Verbatim transcripts | p50 | p95 |
 |---|---|---|---|---|
 | text in, the ceiling | **90.3%** (65/72) | — | 361ms | 495ms |
 | whisper on homesrv, then route | **84.7%** (61/72) | 29/72 | 1372ms | 1546ms |
 | workstation transcribes, then routes | **83.3%** (60/72) | 48/72 | 997ms | 1177ms |
 | workstation, one call from audio | **54.2%** (39/72) | — | 425ms | 756ms |
 The 90.3% ceiling is the same model on the same 72 cases with the utterance as text. It is
 not the 93.5% in `docs/evals/2026-08-02-workstation-gemma4-12b.md`, which scored all 87
 cases including the English ones.
 The two speech-to-text paths differ by one case, which is noise on 72. So the choice
 between them is latency and transcript quality, and the workstation wins both.
 ## One call from audio is not a transcription failure
 The obvious reading of 54.2% is that the audio encoder cannot hear Russian. It can. Eight
 of the failing clips were sent back with a transcribe instruction instead of the router
 prompt:
 | Clip | Said | Heard, transcribing | Routing from audio |
 |---|---|---|---|
 | ru-sys-002 | какое число завтра | Какое число завтра? | `unknown` |
 | ru-sys-003 | переходи в тихий режим | Переходи в тихий режим. | `unknown` |
 | ru-query-001 | сколько воды я выпил с утра | Сколько воды я выпил с утра? | `fact`, value "выпил с утра" |
 | ru-act-002 | выключи свет в спальне | Выключи свет в спальне. | `fact`, value "включен" |
 Four clips it transcribes word for word, and routes wrong or refuses. The `ru-query-001`
 row shows the mechanism: the emitted slot holds the tail of the sentence and the
 interrogative head is gone. The model is not deaf, it stops attending to the audio once it
 is also holding a 3.5k-character classification prompt.
 That pattern decides the whole task. A long system prompt and an audio part compete, so the
 transcription has to be its own call with a short instruction. It also means the number
 would not be rescued by a better prompt, a longer clip, or a bigger `mmproj`.
 The failures cluster where the head of the sentence carries the intent: `ru-act` 1/6,
 `ru-sys` 2/5, `ru-query` 12/25. Reminders scored 10/10, because "напомни" is the first word
 and nothing after it changes the answer.
 ## Transcript quality and routing accuracy come apart
 The workstation transcribes 48 of 72 verbatim against whisper's 29, and routes one case
 worse. Both directions of that appear in the same run:
 - `ru-chat-002`: whisper heard "Кто думаешь про переезд", the workstation heard "Что ты
  думаешь про переезд". The correct transcript routed to `chat`, the broken one to `query`.
 - `ru-query-020`: whisper heard "Кто дальше?", the workstation heard the correct "Что
  дальше?". The **broken** transcript routed correctly and the correct one missed.
 A word error rate is not a proxy for routing accuracy here. Judge a speech-to-text change
 on the routing fixture, not on transcripts.
 Three cases only the text path gets right. No speech-to-text path recovers them, so they
 are lost in the rendering rather than in the model.
 ## Latency
 Whisper `ggml-small` on homesrv CPU costs p50 998ms for a 2.14s clip, which is nearly
 all of that path's 1372ms. The workstation does the same job inside its 997ms end-to-end
 total for two calls. So the transfer is worth about 375ms per turn at p50, and more at p95.
 Both are above the one-call 425ms, and that is the trade the table settles: 29 points of
 accuracy for 572ms.
 ## Notes for the next run
 - `--mmproj /mnt/D/AI/gemma4/mmproj-F16.gguf` has to be in `llama_args` in
  `~/.config/mavgpud.json`, or `/props` reports `modalities.audio: false` and every audio
  part is dropped silently. It was added for this measurement and removed afterwards, so
  the box is back to the text-only config.
 - `enable_thinking:false` is mandatory. It was set for all 224 calls here.
 - The degenerate `<|channel>thought` output recorded against #486 did not reproduce, in 80
  transcribe calls or in 144 routing calls.
 - Piper renders at 22050Hz mono. Every clip was resampled with
  `ffmpeg -ar 16000 -ac 1 -c:a pcm_s16le`, because 16kHz is what `audio.PCM16kMono`
  declares and what the earlier measurement used.
@@ -1,6 +1,6 @@
 # Offloading model work to the workstation
-*Last verified: 2026-08-03 @ 12530c8. Living doc: correct it in place, do not append.*
+*Last verified: 2026-08-05 @ b789676. Living doc: correct it in place, do not append.*
 Owner's call, 2026-08-02. Vikunja #483 is the umbrella. Tasks #484 to #487 are the
 work, and this file holds the shape and the rules all four must obey.
@@ -149,6 +149,13 @@ flips. It is wired anyway: `PhraseReminder` is on the same transport and is on.
 Then the embedder above, **whisper.cpp** in `mavsttd`, and **piper** in `mavttsd`.
 `mavwaked` uses no model at all: an energy-threshold VAD over 30ms frames.
 Speech-to-text stays two stages when it moves. One call carrying both a clip and the router
 prompt was measured on 05-08-2026. It scores 54.2% intent-only against 84.7% for whisper on
 homesrv, on the same 72 cases. The model transcribes clips it then routes wrong, so a long
 classification prompt and an audio part compete for attention. Transcribing on the
 workstation and routing the text scores 83.3% at p50 997ms. So the transfer buys 375ms and
 cleaner transcripts, not accuracy. See `docs/evals/2026-08-05-audio-in-routing.md`.
 ## Order
 1. **Transport** (#484). Nothing else is possible until a seam can cross a host.