kami
d68708b5e1
The file comment named four regressions caught here. Three were not. Nothing on this path resamples, because PCMFromWAV refuses anything that is not already 16 kHz mono s16. Nothing exercises language selection, because the hint comes out of the manifest already correct. And a bad model path was the one condition that made the whole test vanish behind a skip nobody reads. The comment now claims the two things that are real, an explicitly set MAVEN_WHISPER_MODEL that does not exist is a failure, and a missing fixture is a failure rather than a skip. looseWordMatch accepted a different word. Four retained runes of "воды" is "вод", so whisper hearing "выпил водки" satisfied the ru_fact keyword, and "dis" let display, distance and discuss all stand in for "disk". A case ending adds a rune, not a syllable, so the hypothesis is capped in length as well as matched on prefix. The spoken text lived in the generator and in the manifest with nothing tying them together. Editing one left the other describing audio that no longer existed, and at a flat ceiling of 0.34 over a five-word reference a one-word drift passed silently. The script reads text out of the manifest now, and the ceilings are set just above what each case really measures against ggml-small, with the measurement recorded beside them. Also: the test carried its own copy of the PCM to float32 conversion, so a regression in the daemon's copy left the silence-gate assertion green, and the manifest was validated for keywords but not for text, where an empty reference makes every hypothesis score a WER of 1. Found in review of #75.
95 lines
3.9 KiB
Bash
Executable File
95 lines
3.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# gen-stt-fixtures.sh — regenerate the golden STT audio fixtures.
|
|
#
|
|
# The fixtures in cmd/mavsttd/testdata/*.wav are SYNTHESISED, not recorded.
|
|
# They come out of the same piper voices maven speaks with, so nothing of the
|
|
# owner's voice is committed and every fixture is reproducible from this
|
|
# script plus the voice model. They are also small: 16 kHz mono s16le, a
|
|
# couple of seconds each.
|
|
#
|
|
# Usage:
|
|
# scripts/gen-stt-fixtures.sh
|
|
#
|
|
# The spoken text is NOT written here. It is read out of
|
|
# cmd/mavsttd/testdata/golden_v1.json, which is the same file the test scores
|
|
# against. It used to live in both places, so editing this script and running
|
|
# make stt-fixtures left the manifest describing audio that no longer existed —
|
|
# and at a WER ceiling of 0.34 over a five-word reference, a one-word drift
|
|
# passed silently. Punctuation does not matter: normalizeTranscript strips it.
|
|
#
|
|
# Voices are picked up from, in order, $PIPER_VOICE_RU / $PIPER_VOICE_EN, then
|
|
# the repo's models/tts, then ~/esp-server/voices. The English voice is not
|
|
# vendored; if it is missing the English fixture is skipped and the existing
|
|
# one is left alone.
|
|
set -euo pipefail
|
|
|
|
root="$(cd "$(dirname "$0")/.." && pwd)"
|
|
out="$root/cmd/mavsttd/testdata"
|
|
piper="${PIPER_BIN:-$root/deps/piper/piper}"
|
|
espeak="${PIPER_ESPEAK:-$root/deps/piper/espeak-ng-data}"
|
|
|
|
pick_voice() {
|
|
for c in "$@"; do
|
|
[ -f "$c" ] && { echo "$c"; return 0; }
|
|
done
|
|
return 1
|
|
}
|
|
|
|
ru="$(pick_voice "${PIPER_VOICE_RU:-}" "$root/models/tts/ru_RU-irina-medium.onnx" "$HOME/esp-server/voices/ru_RU-irina-medium.onnx")" || {
|
|
echo "no russian piper voice found" >&2
|
|
exit 1
|
|
}
|
|
en="$(pick_voice "${PIPER_VOICE_EN:-}" "$root/models/tts/en_US-lessac-medium.onnx" "$HOME/esp-server/voices/en_US-lessac-medium.onnx")" || en=""
|
|
|
|
manifest="$root/cmd/mavsttd/testdata/golden_v1.json"
|
|
command -v jq >/dev/null || { echo "jq is required to read $manifest" >&2; exit 1; }
|
|
[ -f "$manifest" ] || { echo "missing $manifest" >&2; exit 1; }
|
|
|
|
# case_text <name> — the reference transcript for one manifest case.
|
|
case_text() {
|
|
local name="$1" text
|
|
text="$(jq -r --arg n "$name" '.cases[] | select(.name==$n) | .text' "$manifest")"
|
|
[ -n "$text" ] && [ "$text" != "null" ] || { echo "no case named $name in $manifest" >&2; exit 1; }
|
|
printf '%s' "$text"
|
|
}
|
|
|
|
# case_wav <name> — the file name the manifest expects for one case.
|
|
case_wav() {
|
|
local name="$1" wav
|
|
wav="$(jq -r --arg n "$name" '.cases[] | select(.name==$n) | .wav' "$manifest")"
|
|
[ -n "$wav" ] && [ "$wav" != "null" ] || { echo "no case named $name in $manifest" >&2; exit 1; }
|
|
printf '%s' "$wav"
|
|
}
|
|
|
|
# synth <voice> <out.wav> <text>
|
|
# piper emits raw 22050 Hz s16le on stdout; ffmpeg resamples to the canonical
|
|
# 16 kHz mono and writes a plain 44-byte-header WAV (-fflags bitexact keeps
|
|
# ffmpeg's encoder LIST chunk out, so the bytes are stable across ffmpeg
|
|
# builds and internal/audio.PCMFromWAV reads them without scanning).
|
|
synth() {
|
|
local voice="$1" dest="$2" text="$3"
|
|
printf '%s' "$text" | LD_LIBRARY_PATH="$(dirname "$piper")" "$piper" \
|
|
--model "$voice" --config "$voice.json" \
|
|
--espeak_data "$espeak" --output_raw --quiet |
|
|
ffmpeg -hide_banner -loglevel error -y \
|
|
-f s16le -ar 22050 -ac 1 -i - \
|
|
-af "adelay=200,apad=pad_dur=0.2" \
|
|
-ar 16000 -ac 1 -c:a pcm_s16le -fflags bitexact "$dest"
|
|
echo "wrote $dest ($(stat -c%s "$dest") bytes)"
|
|
}
|
|
|
|
for name in ru_reminder ru_fact ru_query; do
|
|
synth "$ru" "$out/$(case_wav "$name")" "$(case_text "$name")"
|
|
done
|
|
|
|
if [ -n "$en" ]; then
|
|
# Keep the English line in the manifest free of words piper spells out
|
|
# letter by letter — "nginx" comes out of lessac as "engine X", which is a
|
|
# TTS artefact and would make the fixture assert on the wrong thing.
|
|
synth "$en" "$out/$(case_wav en_act)" "$(case_text en_act)"
|
|
else
|
|
echo "no english piper voice found — skipping en_act.wav" >&2
|
|
fi
|
|
|
|
echo "fixtures regenerated; expected transcripts live in $out/golden_v1.json"
|