diff --git a/cmd/mavwaked/wakeword.go b/cmd/mavwaked/wakeword.go index 5f83902..f97e3ee 100644 --- a/cmd/mavwaked/wakeword.go +++ b/cmd/mavwaked/wakeword.go @@ -21,11 +21,15 @@ import ( "sync" ) -// defaultWakeThreshold — score above which the keyword was said. Picked from -// the false-accept rate on held-out Russian speech, not from accuracy: a miss -// costs him a repeat, a false accept costs a turn nobody asked for. See -// docs/evals for the wakes-per-hour this buys. -const defaultWakeThreshold = 0.99 +// defaultWakeThreshold — score above which the keyword was said. +// +// Picked from the false-accept rate on held-out Russian speech, not from +// accuracy: a miss costs him a repeat, a false accept costs a turn nobody +// asked for. Over 65 minutes of Common Voice, 0.99 woke her three times and +// 0.999 once, and the difference in recall was one render out of 126. So the +// default is the strict one. `docs/evals/2026-08-09-wake-word.md` has both +// tables. +const defaultWakeThreshold = 0.999 // wakeWord is the streaming state around wakeModels. It is fed the same // capture frames the VAD sees and answers whether the keyword has just been diff --git a/docs/evals/2026-08-09-wake-word.md b/docs/evals/2026-08-09-wake-word.md new file mode 100644 index 0000000..5ee84c5 --- /dev/null +++ b/docs/evals/2026-08-09-wake-word.md @@ -0,0 +1,104 @@ +# The "Мэйвен" wake word: what it hears and what it invents + +*Measured 2026-08-09 on workpc and homesrv. V-487, stage two of two.* + +Stage one gave mavwaked silero-vad, which answers "is this frame speech". +Nothing answered "was this said to her", so every utterance near the +microphone became a turn. SurfaceVoice caps acts at L0, which made that safe +rather than expensive. L0 does not cap reading, so the room could still hear +his facts read back. + +The keyword is "Мэйвен". openWakeWord's two frozen feature models do the +hearing and a 100KB head trained here draws the boundary. It runs on CPU +beside silero and never touches the GPU. + +## Why a per-window accuracy is not a number anyone can act on + +The gate scores every 80ms. A 1.7% false-accept rate per window sounds small +and means a wake every few seconds. The useful question is how many times an hour +it wakes on speech that was not the keyword. So every table below counts +threshold crossings over whole clips and divides by the audio duration. + +A crossing, not a window above the threshold. A keyword held high for half a +second is one wake, not six. + +## The data + +Positives are 600 silero TTS renders of three stressings of the keyword, six +speakers, ten trailing phrases, augmented eight ways each. Hard negatives are +560 renders of confusable Russian words. Real speech is Common Voice ru and Golos. +The 74257 Common Voice clips were already on workpc from the CrisperWhisper +work. The 200 Golos clips came from the CW2 WER eval. + +Splits are by source file. Augmented copies of one render on both sides of a +split would measure memorisation. + +Golos was never trained on at any stage, so it answers the harder question: +does this survive a change of speakers and rooms. + +## Three heads + +Each row is a full retrain. The false-accept column is 8.89 hours of Common +Voice that no stage of training had seen. + +| trained on | recall (window) | false wakes/hour @0.99 | +|---|---|---| +| TTS + 13.7 min of Golos | 0.869 | not measurable | +| + 4000 Common Voice clips | 0.836 | 21.9 | +| + 3837 mined hard negatives | 0.784 | 4.2 | +| + 753 more mined | 0.810 | 3.4 | + +The first row is why the second exists. Thirteen minutes of held-out speech +cannot measure a rate for a gate that scores twelve times a second. A head +trained only against TTS learns to tell TTS from not-TTS. + +Mining is the whole story after that. Random negatives teach the head what +most speech sounds like. They do not teach it the few syllable sequences that +score high, because 4000 clips barely contain them. So the current head was +run over 20000 fresh clips, keeping every window it scored above 0.05. That +found 3837 windows in 855512. Repeating those ten times in the next training +run cut the rate five-fold. + +The second round found 753 in 852240, a fifth of the yield, and bought a +further 20%. It also recovered recall, which the first round had cost. Whether +a third round is worth 25 minutes of workpc is untested. + +## Where the threshold came from + +Both columns are held out. Positives are the 126 renders in the test split. +Speech is 65.1 minutes of Common Voice, disjoint from every training and +mining pool. Both were run through the built `mavwaked` binary reading PCM from a +file, not through the python that trained the head. + +| threshold | renders shipped | false wakes/hour | +|---|---|---| +| 0.99 | 116 / 126 | 2.8 | +| 0.999 | 115 / 126 | 0.9 | + +One render against a third of the false wakes. `defaultWakeThreshold` is +0.999. + +Golos disagrees. It gave 2 wakes in 14 minutes at every threshold, which is +8.7 per hour. Two events is not a rate. What it does say is that a handful of real utterances score above 0.999 +and no threshold will move them. + +## What it costs him + +Ten of the 126 held-out renders were heard and still dropped, and every one +was an utterance shorter than 1.32s. The head scores 16 embeddings, or 1.28s of +audio. The score therefore peaks up to a second after a short keyword ends. +By then the VAD has closed the utterance and dispatch has already asked. + +Real commands are "Мэйвен, " and run past two seconds, which gives +the head the whole request to peak during. A bare "Мэйвен" with nothing after +it is the case that fails. One fix would hold an ignored utterance for a grace +period and ship it if the keyword lands late. It is not built. + +## What was not measured + +No room recordings. Every negative above is a clean corpus clip. This gate +will live among a television, a fan and the far side of a kitchen. None of +those are in these numbers. + +No measurement of him. Training on his voice means copying his transcripts off +homesrv, which is his call and has not been asked.