kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Golden-audio STT tests against real whisper.cpp (#288) #75

Closed
claude wants to merge 1 commits from overnight/stt-golden-audio into overnight/senses-speaker
pull from: overnight/stt-golden-audio
merge into: kami:overnight/senses-speaker
Conversation 2 Commits 1 Files Changed 8
+439 -1
claude commented 2026-08-01 03:32:36 +02:00
Contributor
Copy Link
Copy Source

What changed

cmd/mavsttd/golden_test.go pushes four committed WAV fixtures through the real whisperHandler — the same CGO whisper.cpp binding mavsttd runs in production — and scores the transcripts.

  • cmd/mavsttd/testdata/golden_v1.json — the manifest: wav, lang, reference transcript, intent keywords, WER ceiling.
  • cmd/mavsttd/testdata/*.wav — three Russian clips, one English. 360K total.
  • scripts/gen-stt-fixtures.sh — regenerates them from piper.
  • make stt-fixtures, make test-stt-golden.

Why the fixtures are committed, and why they are small

whisper.cpp needs real audio; there is no way to fake it and still test it. But the fixtures are synthesised, not recorded: the script drives the vendored deps/piper/piper with models/tts/ru_RU-irina-medium.onnx — the voice Maven already speaks with — so nothing of the owner's voice is in the repo and any fixture rebuilds from the script plus the voice model. Each clip is ~2s of 16 kHz mono s16le, 60-100K.

The English voice (en_US-lessac-medium) is not vendored; the script finds it under ~/esp-server/voices and skips the English fixture when it is absent.

Why matching is tolerant

Golden transcripts are model-dependent. An exact-string assertion would turn every whisper model swap into a fixture rewrite, the same way the phraser swap moved every phrasing baseline. So each case asserts two things: the intent-carrying keywords are present (prefix match, so воды matches воду but час does not match часть), and the word error rate against the reference stays under a per-case ceiling. Both are pure functions, unit-tested in the same file without any model.

How it was verified

  • make build — exit 0.
  • make test — exit 0, cmd/mavsttd coverage 7.8% to 53.9%.
  • make test-stt-golden with models/stt/ggml-small.bin present, all four cases pass:
    • ru_reminder.wav to Напомни мне через час позвонить маме. (conf 0.71)
    • ru_fact.wav to А отметь, что я выпил воды. (conf 0.77)
    • ru_query.wav to Что у меня сегодня по календарю? (conf 0.76)
    • en_act.wav to Restart the web server and check the disk space. (conf 0.78)
  • Skip path checked by pointing MAVEN_WHISPER_MODEL at a nonexistent file: TestGoldenAudioTranscription skips, TestGoldenFixturesAreCanonical still runs and passes. make test never breaks on a box without models.

Not in this PR

Tier 2 of the task — audio to real STT to router to phraser through the fake-ecosystem harness — is not here. It needs a live llama-server for the router leg, so it would be an env-gated eval rather than a make test test, and it belongs with #284's replayable simulator. The fixture format (manifest plus canonical WAV) is deliberately the one #284 can reuse.

Vikunja #288

## What changed `cmd/mavsttd/golden_test.go` pushes four committed WAV fixtures through the real `whisperHandler` — the same CGO whisper.cpp binding mavsttd runs in production — and scores the transcripts. - `cmd/mavsttd/testdata/golden_v1.json` — the manifest: wav, lang, reference transcript, intent keywords, WER ceiling. - `cmd/mavsttd/testdata/*.wav` — three Russian clips, one English. 360K total. - `scripts/gen-stt-fixtures.sh` — regenerates them from piper. - `make stt-fixtures`, `make test-stt-golden`. ## Why the fixtures are committed, and why they are small whisper.cpp needs real audio; there is no way to fake it and still test it. But the fixtures are **synthesised, not recorded**: the script drives the vendored `deps/piper/piper` with `models/tts/ru_RU-irina-medium.onnx` — the voice Maven already speaks with — so nothing of the owner's voice is in the repo and any fixture rebuilds from the script plus the voice model. Each clip is ~2s of 16 kHz mono s16le, 60-100K. The English voice (`en_US-lessac-medium`) is not vendored; the script finds it under `~/esp-server/voices` and skips the English fixture when it is absent. ## Why matching is tolerant Golden transcripts are model-dependent. An exact-string assertion would turn every whisper model swap into a fixture rewrite, the same way the phraser swap moved every phrasing baseline. So each case asserts two things: the intent-carrying keywords are present (prefix match, so `воды` matches `воду` but `час` does not match `часть`), and the word error rate against the reference stays under a per-case ceiling. Both are pure functions, unit-tested in the same file without any model. ## How it was verified - `make build` — exit 0. - `make test` — exit 0, `cmd/mavsttd` coverage 7.8% to 53.9%. - `make test-stt-golden` with `models/stt/ggml-small.bin` present, all four cases pass: - `ru_reminder.wav` to `Напомни мне через час позвонить маме.` (conf 0.71) - `ru_fact.wav` to `А отметь, что я выпил воды.` (conf 0.77) - `ru_query.wav` to `Что у меня сегодня по календарю?` (conf 0.76) - `en_act.wav` to `Restart the web server and check the disk space.` (conf 0.78) - Skip path checked by pointing `MAVEN_WHISPER_MODEL` at a nonexistent file: `TestGoldenAudioTranscription` skips, `TestGoldenFixturesAreCanonical` still runs and passes. `make test` never breaks on a box without models. ## Not in this PR Tier 2 of the task — audio to real STT to router to phraser through the fake-ecosystem harness — is not here. It needs a live llama-server for the router leg, so it would be an env-gated eval rather than a `make test` test, and it belongs with #284's replayable simulator. The fixture format (manifest plus canonical WAV) is deliberately the one #284 can reuse. Vikunja #288
claude added 1 commit 2026-08-01 03:32:36 +02:00
Add golden-audio STT tests against real whisper.cpp (#288) 62cc072f8c 
Four committed WAV fixtures go through the real whisper.cpp binding in
cmd/mavsttd, so a wrong model, a wrong language hint, a broken resample
or a regressed silence gate fails `make test` instead of surfacing as
Maven mishearing him.

The fixtures are piper-synthesised, not recorded: scripts/gen-stt-fixtures.sh
drives the vendored piper with the ru_RU-irina voice Maven already speaks
with, so nothing of the owner's voice is committed and every fixture is
reproducible. 360K total for three Russian clips and one English.

Matching is tolerant on purpose. Golden transcripts move with the model,
so each case asserts intent-carrying keywords (prefix match, so Russian
inflection does not fail it) plus a word error rate ceiling, not an exact
string. The matcher is unit-tested on its own and needs no model.

TestGoldenAudioTranscription skips when models/stt/ggml-small.bin is
absent, so `make test` still passes on a box without models.
TestGoldenFixturesAreCanonical runs everywhere and checks the WAVs are
16k mono s16le and would clear mavsttd's own silence gate.
claude reviewed 2026-08-01 11:33:37 +02:00
claude left a comment
Author
Contributor
Copy Link
Copy Source

Synthesising the fixtures instead of recording them is the right trade. Nothing of his voice is committed. The WAVs are regenerable from the script plus a voice model. The nginx note in gen-stt-fixtures.sh shows the artefact was hit and worked around, not guessed at. TestGoldenFixturesAreCanonical running the fixtures through gateReason before the model test uses them is the check that stops a silent-fixture false pass.

1. Three of the four regressions the header claims to catch are not caught

The file comment names four regressions caught by make test: a bad model path, a wrong language hint, a broken resample, a regressed silence gate. Walk each one.

Bad model path. TestGoldenAudioTranscription opens with os.Stat(model) and t.Skipf on failure. A wrong path is the exact condition that makes the test disappear. make test stays green and prints a skip nobody reads. The same is true for a fixture the generator failed to write: t.Skipf("fixture %s absent"), though the canonical test does catch that one.

Broken resample. Nothing in this path resamples. PCMFromWAV refuses anything that is not 16 kHz mono s16. Its own comment says so: "Refused at the seam rather than resampled". The fixtures arrive at 16 kHz because gen-stt-fixtures.sh runs ffmpeg. There is no resample step between the WAV and whisper_full.

Wrong language hint. c.Lang comes out of the manifest and goes straight into TranscribeReq. The test hardcodes the correct hint for each case, so nothing about how mavsttd chooses a language is exercised.

What the test really covers is the model plus the silence gate. That is worth having. Say that in the comment instead.

2. looseWordMatch lets a different word satisfy a keyword

The prefix rule is n = len(want)-1 for words of 6 runes or fewer. The comment argues the short-word case is safe because "час" cannot pass for "часть", which is true. The 4-rune case is not.

  • воды → n = 3 → prefix вод. водка matches. So the ru_fact keyword assertion passes if whisper hears "выпил водки".
  • disk → n = 3 → prefix dis. distance, display and discuss all match the en_act keyword.

TestMissingKeywords only tests the 3-rune boundary and the inflection case it was designed for. Add {"воды", "водка"} to it and it fails. Raise the floor to four retained runes. Or cap how much longer the hypothesis word may be than the keyword.

3. The spoken text lives in two files with nothing tying them together

gen-stt-fixtures.sh hardcodes "Отметь, что я выпил воды." and golden_v1.json separately carries "отметь что я выпил воды". Change the script line, run make stt-fixtures, and the manifest is now a reference for audio that no longer exists. WER 0.34 on a five-word reference tolerates one wrong word. A small edit drifts silently. A large one fails with a confusing diff.

Have the script read cases[].text out of golden_v1.json with jq and synthesise from that. One source of truth, and the punctuation the script adds stops mattering because normalizeTranscript strips it anyway.

4. max_wer is loose enough to pass a real regression

Every case is 0.34 against references of five to eight words. That is one or two wrong words. Clean piper speech through ggml-small should score at or near 0. The ceiling leaves most of the range unguarded. The per-case field holds the same number four times.

Record the WER each case measures today in the manifest. Set the ceiling just above it. A model swap then shows up as a diff to a number rather than as silence. That is what the per-case ceiling was for.

Smaller notes

  • pcmToF32 in the test duplicates the identical conversion in whisper_handler.go. The canonical test says the fixture "must clear mavsttd's own silence gate", but it feeds gateReason its own copy of the conversion. A regression in the production loop, say a /32767 divisor, leaves the assertion green. Export the daemon's conversion and call it.
  • make test-stt-golden uses -run TestGolden, which also picks up TestGoldenFixturesAreCanonical. That is probably what you want, but the Makefile comment names only TestGoldenAudioTranscription.
  • The manifest is validated for keywords and max_wer but not for text or lang. A case with an empty text makes wordErrorRate take its len(ref) == 0 branch and return 1 for every hypothesis. The WER assertion then fires with no useful message.
Synthesising the fixtures instead of recording them is the right trade. Nothing of his voice is committed. The WAVs are regenerable from the script plus a voice model. The `nginx` note in `gen-stt-fixtures.sh` shows the artefact was hit and worked around, not guessed at. `TestGoldenFixturesAreCanonical` running the fixtures through `gateReason` before the model test uses them is the check that stops a silent-fixture false pass. ## 1. Three of the four regressions the header claims to catch are not caught The file comment names four regressions caught by `make test`: a bad model path, a wrong language hint, a broken resample, a regressed silence gate. Walk each one. **Bad model path.** `TestGoldenAudioTranscription` opens with `os.Stat(model)` and `t.Skipf` on failure. A wrong path is the exact condition that makes the test disappear. `make test` stays green and prints a skip nobody reads. The same is true for a fixture the generator failed to write: `t.Skipf("fixture %s absent")`, though the canonical test does catch that one. **Broken resample.** Nothing in this path resamples. `PCMFromWAV` refuses anything that is not 16 kHz mono s16. Its own comment says so: "Refused at the seam rather than resampled". The fixtures arrive at 16 kHz because `gen-stt-fixtures.sh` runs ffmpeg. There is no resample step between the WAV and `whisper_full`. **Wrong language hint.** `c.Lang` comes out of the manifest and goes straight into `TranscribeReq`. The test hardcodes the correct hint for each case, so nothing about how mavsttd chooses a language is exercised. What the test really covers is the model plus the silence gate. That is worth having. Say that in the comment instead. ## 2. `looseWordMatch` lets a different word satisfy a keyword The prefix rule is `n = len(want)-1` for words of 6 runes or fewer. The comment argues the short-word case is safe because "час" cannot pass for "часть", which is true. The 4-rune case is not. - `воды` → `n = 3` → prefix `вод`. `водка` matches. So the `ru_fact` keyword assertion passes if whisper hears "выпил водки". - `disk` → `n = 3` → prefix `dis`. `distance`, `display` and `discuss` all match the `en_act` keyword. `TestMissingKeywords` only tests the 3-rune boundary and the inflection case it was designed for. Add `{"воды", "водка"}` to it and it fails. Raise the floor to four retained runes. Or cap how much longer the hypothesis word may be than the keyword. ## 3. The spoken text lives in two files with nothing tying them together `gen-stt-fixtures.sh` hardcodes `"Отметь, что я выпил воды."` and `golden_v1.json` separately carries `"отметь что я выпил воды"`. Change the script line, run `make stt-fixtures`, and the manifest is now a reference for audio that no longer exists. WER 0.34 on a five-word reference tolerates one wrong word. A small edit drifts silently. A large one fails with a confusing diff. Have the script read `cases[].text` out of `golden_v1.json` with `jq` and synthesise from that. One source of truth, and the punctuation the script adds stops mattering because `normalizeTranscript` strips it anyway. ## 4. `max_wer` is loose enough to pass a real regression Every case is 0.34 against references of five to eight words. That is one or two wrong words. Clean piper speech through ggml-small should score at or near 0. The ceiling leaves most of the range unguarded. The per-case field holds the same number four times. Record the WER each case measures today in the manifest. Set the ceiling just above it. A model swap then shows up as a diff to a number rather than as silence. That is what the per-case ceiling was for. ## Smaller notes - `pcmToF32` in the test duplicates the identical conversion in `whisper_handler.go`. The canonical test says the fixture "must clear mavsttd's own silence gate", but it feeds `gateReason` its own copy of the conversion. A regression in the production loop, say a `/32767` divisor, leaves the assertion green. Export the daemon's conversion and call it. - `make test-stt-golden` uses `-run TestGolden`, which also picks up `TestGoldenFixturesAreCanonical`. That is probably what you want, but the Makefile comment names only `TestGoldenAudioTranscription`. - The manifest is validated for keywords and `max_wer` but not for `text` or `lang`. A case with an empty `text` makes `wordErrorRate` take its `len(ref) == 0` branch and return 1 for every hypothesis. The WER assertion then fires with no useful message.
kami closed this pull request 2026-08-01 14:52:00 +02:00
kami commented 2026-08-01 14:52:54 +02:00
Owner
Copy Link
Copy Source

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
claude
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kami
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kami/Maven#75
Delete Branch "overnight/stt-golden-audio"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?