Default the keyword threshold to 0.999, from the measurement (V-487)

Over 65.1 minutes of held-out Common Voice the built binary woke three times
at 0.99 and once at 0.999. The recall difference was one render out of 126.
One render is worth two thirds of the false wakes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

cmd/mavwaked/wakeword.go

@@ -21,11 +21,15 @@ import (
 	"sync"
 )
 
-// defaultWakeThreshold — score above which the keyword was said. Picked from
-// the false-accept rate on held-out Russian speech, not from accuracy: a miss
-// costs him a repeat, a false accept costs a turn nobody asked for. See
-// docs/evals for the wakes-per-hour this buys.
-const defaultWakeThreshold = 0.99
+// defaultWakeThreshold — score above which the keyword was said.
+//
+// Picked from the false-accept rate on held-out Russian speech, not from
+// accuracy: a miss costs him a repeat, a false accept costs a turn nobody
+// asked for. Over 65 minutes of Common Voice, 0.99 woke her three times and
+// 0.999 once, and the difference in recall was one render out of 126. So the
+// default is the strict one. `docs/evals/2026-08-09-wake-word.md` has both
+// tables.
+const defaultWakeThreshold = 0.999
 
 // wakeWord is the streaming state around wakeModels. It is fed the same
 // capture frames the VAD sees and answers whether the keyword has just been

docs/evals/2026-08-09-wake-word.md

@@ -0,0 +1,104 @@
+# The "Мэйвен" wake word: what it hears and what it invents
+
+*Measured 2026-08-09 on workpc and homesrv. V-487, stage two of two.*
+
+Stage one gave mavwaked silero-vad, which answers "is this frame speech".
+Nothing answered "was this said to her", so every utterance near the
+microphone became a turn. SurfaceVoice caps acts at L0, which made that safe
+rather than expensive. L0 does not cap reading, so the room could still hear
+his facts read back.
+
+The keyword is "Мэйвен". openWakeWord's two frozen feature models do the
+hearing and a 100KB head trained here draws the boundary. It runs on CPU
+beside silero and never touches the GPU.
+
+## Why a per-window accuracy is not a number anyone can act on
+
+The gate scores every 80ms. A 1.7% false-accept rate per window sounds small
+and means a wake every few seconds. The useful question is how many times an hour
+it wakes on speech that was not the keyword. So every table below counts
+threshold crossings over whole clips and divides by the audio duration.
+
+A crossing, not a window above the threshold. A keyword held high for half a
+second is one wake, not six.
+
+## The data
+
+Positives are 600 silero TTS renders of three stressings of the keyword, six
+speakers, ten trailing phrases, augmented eight ways each. Hard negatives are
+560 renders of confusable Russian words. Real speech is Common Voice ru and Golos.
+The 74257 Common Voice clips were already on workpc from the CrisperWhisper
+work. The 200 Golos clips came from the CW2 WER eval.
+
+Splits are by source file. Augmented copies of one render on both sides of a
+split would measure memorisation.
+
+Golos was never trained on at any stage, so it answers the harder question:
+does this survive a change of speakers and rooms.
+
+## Three heads
+
+Each row is a full retrain. The false-accept column is 8.89 hours of Common
+Voice that no stage of training had seen.
+
+| trained on | recall (window) | false wakes/hour @0.99 |
+|---|---|---|
+| TTS + 13.7 min of Golos | 0.869 | not measurable |
+| + 4000 Common Voice clips | 0.836 | 21.9 |
+| + 3837 mined hard negatives | 0.784 | 4.2 |
+| + 753 more mined | 0.810 | 3.4 |
+
+The first row is why the second exists. Thirteen minutes of held-out speech
+cannot measure a rate for a gate that scores twelve times a second. A head
+trained only against TTS learns to tell TTS from not-TTS.
+
+Mining is the whole story after that. Random negatives teach the head what
+most speech sounds like. They do not teach it the few syllable sequences that
+score high, because 4000 clips barely contain them. So the current head was
+run over 20000 fresh clips, keeping every window it scored above 0.05. That
+found 3837 windows in 855512. Repeating those ten times in the next training
+run cut the rate five-fold.
+
+The second round found 753 in 852240, a fifth of the yield, and bought a
+further 20%. It also recovered recall, which the first round had cost. Whether
+a third round is worth 25 minutes of workpc is untested.
+
+## Where the threshold came from
+
+Both columns are held out. Positives are the 126 renders in the test split.
+Speech is 65.1 minutes of Common Voice, disjoint from every training and
+mining pool. Both were run through the built `mavwaked` binary reading PCM from a
+file, not through the python that trained the head.
+
+| threshold | renders shipped | false wakes/hour |
+|---|---|---|
+| 0.99 | 116 / 126 | 2.8 |
+| 0.999 | 115 / 126 | 0.9 |
+
+One render against a third of the false wakes. `defaultWakeThreshold` is
+0.999.
+
+Golos disagrees. It gave 2 wakes in 14 minutes at every threshold, which is
+8.7 per hour. Two events is not a rate. What it does say is that a handful of real utterances score above 0.999
+and no threshold will move them.
+
+## What it costs him
+
+Ten of the 126 held-out renders were heard and still dropped, and every one
+was an utterance shorter than 1.32s. The head scores 16 embeddings, or 1.28s of
+audio. The score therefore peaks up to a second after a short keyword ends.
+By then the VAD has closed the utterance and dispatch has already asked.
+
+Real commands are "Мэйвен, <request>" and run past two seconds, which gives
+the head the whole request to peak during. A bare "Мэйвен" with nothing after
+it is the case that fails. One fix would hold an ignored utterance for a grace
+period and ship it if the keyword lands late. It is not built.
+
+## What was not measured
+
+No room recordings. Every negative above is a clean corpus clip. This gate
+will live among a television, a fan and the far side of a kitchen. None of
+those are in these numbers.
+
+No measurement of him. Training on his voice means copying his transcripts off
+homesrv, which is his call and has not been asked.