stt: make the golden tests fail where they used to disappear

The file comment named four regressions caught here. Three were not.
Nothing on this path resamples, because PCMFromWAV refuses anything that
is not already 16 kHz mono s16. Nothing exercises language selection,
because the hint comes out of the manifest already correct. And a bad
model path was the one condition that made the whole test vanish behind
a skip nobody reads. The comment now claims the two things that are
real, an explicitly set MAVEN_WHISPER_MODEL that does not exist is a
failure, and a missing fixture is a failure rather than a skip.

looseWordMatch accepted a different word. Four retained runes of "воды"
is "вод", so whisper hearing "выпил водки" satisfied the ru_fact
keyword, and "dis" let display, distance and discuss all stand in for
"disk". A case ending adds a rune, not a syllable, so the hypothesis is
capped in length as well as matched on prefix.

The spoken text lived in the generator and in the manifest with nothing
tying them together. Editing one left the other describing audio that no
longer existed, and at a flat ceiling of 0.34 over a five-word reference
a one-word drift passed silently. The script reads text out of the
manifest now, and the ceilings are set just above what each case really
measures against ggml-small, with the measurement recorded beside them.

Also: the test carried its own copy of the PCM to float32 conversion, so
a regression in the daemon's copy left the silence-gate assertion green,
and the manifest was validated for keywords but not for text, where an
empty reference makes every hypothesis score a WER of 1.

Found in review of #75.
This commit is contained in:
kami
2026-08-01 14:16:02 +04:00
parent ec5167de3a
commit d68708b5e1
5 changed files with 152 additions and 39 deletions
+7 -2
View File
@@ -149,8 +149,13 @@ eval-models:
# stt-fixtures — regenerate the golden STT audio in cmd/mavsttd/testdata from
# the piper voices (#288). The committed WAVs are synthesised, never recorded,
# so this is the only way they should ever change. TestGoldenAudioTranscription
# then scores them against ggml-small; it self-skips when the model is absent.
# so this is the only way they should ever change. The spoken text is read out
# of testdata/golden_v1.json, so edit the transcript there and rerun this.
#
# test-stt-golden runs both golden tests: TestGoldenAudioTranscription, which
# scores the fixtures against ggml-small and self-skips when the model is
# absent, and TestGoldenFixturesAreCanonical, which checks the committed audio
# and the manifest with no model at all.
stt-fixtures:
./scripts/gen-stt-fixtures.sh