diff --git a/docs/evals/2026-08-05-audio-in-routing.md b/docs/evals/2026-08-05-audio-in-routing.md new file mode 100644 index 0000000..d4286ec --- /dev/null +++ b/docs/evals/2026-08-05-audio-in-routing.md @@ -0,0 +1,93 @@ +# Routing from audio: four paths, one fixture + +**05-08-2026. Vikunja #486.** Workstation `gemma-4-12B-it-qat-UD-Q4_K_XL` with +`mmproj-F16.gguf`, homesrv whisper `ggml-small`, piper `ru_RU-irina-medium`. + +**Verdict: transcribe, then route.** One call from audio straight to a route loses 36 +points, so it is not a candidate. Moving speech-to-text to the workstation buys 375ms and +better transcripts at no measurable accuracy cost. So #486 proceeds on the two-call shape. + +## The numbers + +72 Russian cases from `internal/router/eval/ru_routing_v1.json`, rendered by piper at +16kHz mono, 153.9s of audio, mean 2.14s per clip. Every path used the daemon's own +`routeSystem` prompt and `routeGrammar`, read out of `internal/router/llmrouter.go` at run +time, at `temperature 0` and `enable_thinking:false`. + +| Path | Intent-only | Verbatim transcripts | p50 | p95 | +|---|---|---|---|---| +| text in, the ceiling | **90.3%** (65/72) | — | 361ms | 495ms | +| whisper on homesrv, then route | **84.7%** (61/72) | 29/72 | 1372ms | 1546ms | +| workstation transcribes, then routes | **83.3%** (60/72) | 48/72 | 997ms | 1177ms | +| workstation, one call from audio | **54.2%** (39/72) | — | 425ms | 756ms | + +The 90.3% ceiling is the same model on the same 72 cases with the utterance as text. It is +not the 93.5% in `docs/evals/2026-08-02-workstation-gemma4-12b.md`, which scored all 87 +cases including the English ones. + +The two speech-to-text paths differ by one case, which is noise on 72. So the choice +between them is latency and transcript quality, and the workstation wins both. + +## One call from audio is not a transcription failure + +The obvious reading of 54.2% is that the audio encoder cannot hear Russian. It can. Eight +of the failing clips were sent back with a transcribe instruction instead of the router +prompt: + +| Clip | Said | Heard, transcribing | Routing from audio | +|---|---|---|---| +| ru-sys-002 | какое число завтра | Какое число завтра? | `unknown` | +| ru-sys-003 | переходи в тихий режим | Переходи в тихий режим. | `unknown` | +| ru-query-001 | сколько воды я выпил с утра | Сколько воды я выпил с утра? | `fact`, value "выпил с утра" | +| ru-act-002 | выключи свет в спальне | Выключи свет в спальне. | `fact`, value "включен" | + +Four clips it transcribes word for word, and routes wrong or refuses. The `ru-query-001` +row shows the mechanism: the emitted slot holds the tail of the sentence and the +interrogative head is gone. The model is not deaf, it stops attending to the audio once it +is also holding a 3.5k-character classification prompt. + +That pattern decides the whole task. A long system prompt and an audio part compete, so the +transcription has to be its own call with a short instruction. It also means the number +would not be rescued by a better prompt, a longer clip, or a bigger `mmproj`. + +The failures cluster where the head of the sentence carries the intent: `ru-act` 1/6, +`ru-sys` 2/5, `ru-query` 12/25. Reminders scored 10/10, because "напомни" is the first word +and nothing after it changes the answer. + +## Transcript quality and routing accuracy come apart + +The workstation transcribes 48 of 72 verbatim against whisper's 29, and routes one case +worse. Both directions of that appear in the same run: + +- `ru-chat-002`: whisper heard "Кто думаешь про переезд", the workstation heard "Что ты + думаешь про переезд". The correct transcript routed to `chat`, the broken one to `query`. +- `ru-query-020`: whisper heard "Кто дальше?", the workstation heard the correct "Что + дальше?". The **broken** transcript routed correctly and the correct one missed. + +A word error rate is not a proxy for routing accuracy here. Judge a speech-to-text change +on the routing fixture, not on transcripts. + +Three cases only the text path gets right. No speech-to-text path recovers them, so they +are lost in the rendering rather than in the model. + +## Latency + +Whisper `ggml-small` on homesrv CPU costs p50 998ms for a 2.14s clip, which is nearly +all of that path's 1372ms. The workstation does the same job inside its 997ms end-to-end +total for two calls. So the transfer is worth about 375ms per turn at p50, and more at p95. + +Both are above the one-call 425ms, and that is the trade the table settles: 29 points of +accuracy for 572ms. + +## Notes for the next run + +- `--mmproj /mnt/D/AI/gemma4/mmproj-F16.gguf` has to be in `llama_args` in + `~/.config/mavgpud.json`, or `/props` reports `modalities.audio: false` and every audio + part is dropped silently. It was added for this measurement and removed afterwards, so + the box is back to the text-only config. +- `enable_thinking:false` is mandatory. It was set for all 224 calls here. +- The degenerate `<|channel>thought` output recorded against #486 did not reproduce, in 80 + transcribe calls or in 144 routing calls. +- Piper renders at 22050Hz mono. Every clip was resampled with + `ffmpeg -ar 16000 -ac 1 -c:a pcm_s16le`, because 16kHz is what `audio.PCM16kMono` + declares and what the earlier measurement used. diff --git a/docs/offload.md b/docs/offload.md index 122ad86..0d4868d 100644 --- a/docs/offload.md +++ b/docs/offload.md @@ -1,6 +1,6 @@ # Offloading model work to the workstation -*Last verified: 2026-08-03 @ 12530c8. Living doc: correct it in place, do not append.* +*Last verified: 2026-08-05 @ b789676. Living doc: correct it in place, do not append.* Owner's call, 2026-08-02. Vikunja #483 is the umbrella. Tasks #484 to #487 are the work, and this file holds the shape and the rules all four must obey. @@ -149,6 +149,13 @@ flips. It is wired anyway: `PhraseReminder` is on the same transport and is on. Then the embedder above, **whisper.cpp** in `mavsttd`, and **piper** in `mavttsd`. `mavwaked` uses no model at all: an energy-threshold VAD over 30ms frames. +Speech-to-text stays two stages when it moves. One call carrying both a clip and the router +prompt was measured on 05-08-2026. It scores 54.2% intent-only against 84.7% for whisper on +homesrv, on the same 72 cases. The model transcribes clips it then routes wrong, so a long +classification prompt and an audio part compete for attention. Transcribing on the +workstation and routing the text scores 83.3% at p50 997ms. So the transfer buys 375ms and +cleaner transcripts, not accuracy. See `docs/evals/2026-08-05-audio-in-routing.md`. + ## Order 1. **Transport** (#484). Nothing else is possible until a seam can cross a host.