diff --git a/docs/evals/2026-08-09-crisperwhisper2-russian-wer.md b/docs/evals/2026-08-09-crisperwhisper2-russian-wer.md new file mode 100644 index 0000000..0a6603b --- /dev/null +++ b/docs/evals/2026-08-09-crisperwhisper2-russian-wer.md @@ -0,0 +1,110 @@ +# CrisperWhisper 2.0 in Russian, measured + +Date: 2026-08-09. Vikunja V-665. +Corpus: `bond005/sberdevices_golos_10h_crowd`, test split, first 200 clips. +Harness: `~/Programs/cw2-eval` on workpc, not in this repo. +Runner: `./.venv/bin/python run_asr.py ...` then `score.py`. + +The model card benchmarks disfluency F1 in German and English. It never names +Russian and publishes no per-language WER. So the measurement came before the +wiring. + +## The corpus + +200 clips, 13.7 minutes, 1001 reference words. Median clip 3.91s, range 1.04s +to 13.5s. Golos crowd is short crowd-sourced Russian spoken close to the +microphone, which is the nearest public thing to someone talking to Maven. The +alternatives are read speech, which flatters every model equally. + +Two rows carry a null transcription and are skipped. + +Scoring normalizes both sides: lowercase, `ё` to `е`, punctuation stripped, and +digits expanded to Russian words through num2words. Without that last step a +model is penalized for writing `60000` where the reference says +`шестьдесят тысяч`. Thousands separators are joined before expansion, or +`60 000` expands to `шестьдесят ноль`. + +## Headline + +| arm | WER | CER | exact | empty | RTF | +|---|---|---|---|---|---| +| cw2-turbo-intended | **10.4%** | 3.4% | 65.5% | 0 | 0.065 | +| cw2-turbo-verbatim | 10.8% | **3.1%** | **66.5%** | 0 | 0.065 | +| whisper-turbo | 11.8% | 4.1% | 64.0% | 0 | 0.031 | +| cw2-large-intended | 12.3% | 3.8% | 63.5% | 0 | 0.107 | +| whisper-small | 27.5% | 9.8% | 35.0% | 0 | 0.026 | + +`whisper-small` is the floor, because `ggml-small.bin` is what mavsttd loads on +homesrv today. CW2 turbo beats it by 17 points of WER and takes exact matches +from 35.0% to 65.5%. + +Two results are worth naming beyond the winner. CW2 turbo beats its own base +model, whisper-large-v3-turbo, by 1.4 points. And it beats CW2 large by 1.9 +points, which inverts what the card implies by calling turbo a degraded draft. +No arm returned an empty transcript. + +## Intended and verbatim are closer than the mode names suggest + +The two modes disagree on 70 of the 200 clips before normalization and on 29 +after it. So the raw difference is mostly casing and punctuation, which +normalization removes and which Maven does not read either. + +Verbatim scores worse on WER and better on CER and exact matches. The reason is +script, not disfluency: + +```text +ref: футбольный матч челси брайтон +int: Футбольный матч Chelsea-Брайтон. +ver: Футбольный матч Челси Брайтон. +``` + +Intended writes foreign entity names in Latin script and verbatim +transliterates them. Golos references are Cyrillic throughout, so verbatim +collects the exact matches. That is a property of this corpus rather than a +quality difference. + +**This corpus cannot settle the mode choice.** Golos crowd is clean short +commands with almost no disfluency. The two modes have nothing to disagree +about here. They separate on spontaneous speech with fillers, restarts and +repairs, which is what the owner speaks. Intended stays the choice for the +reason it was always the choice. Maven wants what was meant, not every stumble +on the way there. + +The Latin-script habit is the one finding here that touches routing. The +routing heads were trained on Cyrillic utterances, so an entity name arriving +in Latin script is out of distribution for them. Nothing measures that yet. + +## The runtime is workpc, because whisper.cpp cannot load CW2 + +`num_languages()` in `deps/whisper.cpp/src/whisper.cpp` derives the language +count from the vocabulary size: + +```cpp +return n_vocab - 51765 - (is_multilingual() ? 1 : 0); +``` + +CW2 carries 31 extra tokens, so `n_vocab` is 51897 and this yields 131 +languages. The derived `dt` offset becomes 33 and shifts seven special token +ids, including `token_beg` and `token_transcribe`. The architecture is +otherwise byte-identical to whisper-large-v3-turbo, and the new tokens sit +above every whisper special id. + +So loading CW2 in whisper.cpp is a patch to a vendored dependency, not a port. +It was not taken, because STT is moving to workpc anyway under V-486. CW2 turbo +becomes the preferred remote and `ggml-small.bin` on homesrv stays the floor, +which is the shape `modelSeam` already uses for routing and replies. The 27.5% +floor is what a turn falls back to when the workstation is down, and this table +is what that costs. + +## License + +Standard CW2 weights carry `nyra-health-non-commercial-research`. The Pro +variants are commercial-license only. Maven is personal and self-hosted, so the +standard weights are usable and the Pro ones are not free to take. + +## What is not measured + +Disfluent spontaneous speech, which is the whole reason to prefer Intended. +Long-form audio beyond 13.5s. Far-field or noisy microphones. English, which +Maven also speaks. The ONNX turbo export, which was never run, since the +transformers path already meets the latency budget at RTF 0.065.