Half-duplex capture and barge-in in mavwaked (#287) #76

2026-08-01T03:36:34+02:00

claude commented

2026-08-01 03:36:34 +02:00

What changed

cmd/mavwaked — playback and the gate around it.

playback.go — player interface plus aplayPlayer, which holds the aplay process handle so playback can be cut mid-utterance. Previously go playAudio(...) with nothing to cancel.
session.go — the per-frame decision, lifted out of main.go's processFrame behind player and utteranceSender interfaces.
main.go — wires them, adds -barge-in, -barge-in-rms, -barge-in-frames.
session_test.go, playback_test.go — 11 tests, no mic, no speaker, no daemon.

Why

Two of the eight items on #287, and the two that are actually audible.

She answered herself. The capture loop kept feeding the VAD while the speaker was running, so Maven's own reply came back in through the mic, tripped the VAD, and went to the daemon as a fresh command. There is no acoustic echo canceller here, so the correct fix is half-duplex: while she speaks, capture is muted. This part is unconditional — it repairs a defect rather than adding a capability.

Talking over her did nothing. Now it can, but only when configured. -barge-in is off by default because the threshold is room-specific: without an echo canceller the only signal distinguishing "he is talking over her" from "the mic is hearing her" is that he is much louder, and how much louder depends on where the mic sits relative to the speaker. Defaults when enabled are 0.12 normalised RMS sustained over 5 frames (150 ms) — an order of magnitude above the VAD's own 0.01 floor, and long enough that a door closing does not cut her off.

Both are the same mechanism seen from opposite sides, which is why they are one PR.

How it was verified

make build and make test both exit 0. cmd/mavwaked coverage 32.4% to 44.4%. The tests assert the behaviour, not the tuning:

200 frames of loud audio fed while playing produce zero extra utterances (the self-hearing bug, now failing if reintroduced).
Barge-in off by default; 50 very loud frames never stop playback.
4 loud frames do not cut her off, the 5th does.
Two-frame bursts separated by quiet never accumulate into a false barge-in.
300 frames at speaker-leak level (0.05 RMS — above the VAD floor, below the barge bar) never trigger.
After a cut, the interrupting speech is captured as its own utterance with a clean VAD.
A failed round-trip surfaces the error and starts no playback; an empty reply leaves capture open.
aplayPlayer is safe to Stop() when idle and spawns nothing for empty audio — neither needs aplay(1) installed.

The acoustic half (does 0.12 actually clear this room's speaker leak) cannot be measured without the hardware; it is a QA step on the task.

Not in this PR

The other six open items on #287 — partial STT display, per-segment retry, per-room mic profiles, noise-floor calibration, short-response-while-speaking — are untouched and stay on the task. Partial STT in particular needs a streaming wire change, not a client change.

Vikunja #287

## What changed `cmd/mavwaked` — playback and the gate around it. - `playback.go` — `player` interface plus `aplayPlayer`, which holds the aplay process handle so playback can be cut mid-utterance. Previously `go playAudio(...)` with nothing to cancel. - `session.go` — the per-frame decision, lifted out of `main.go`'s `processFrame` behind `player` and `utteranceSender` interfaces. - `main.go` — wires them, adds `-barge-in`, `-barge-in-rms`, `-barge-in-frames`. - `session_test.go`, `playback_test.go` — 11 tests, no mic, no speaker, no daemon. ## Why Two of the eight items on #287, and the two that are actually audible. **She answered herself.** The capture loop kept feeding the VAD while the speaker was running, so Maven's own reply came back in through the mic, tripped the VAD, and went to the daemon as a fresh command. There is no acoustic echo canceller here, so the correct fix is half-duplex: while she speaks, capture is muted. This part is unconditional — it repairs a defect rather than adding a capability. **Talking over her did nothing.** Now it can, but only when configured. `-barge-in` is off by default because the threshold is room-specific: without an echo canceller the only signal distinguishing "he is talking over her" from "the mic is hearing her" is that he is much louder, and how much louder depends on where the mic sits relative to the speaker. Defaults when enabled are 0.12 normalised RMS sustained over 5 frames (150 ms) — an order of magnitude above the VAD's own 0.01 floor, and long enough that a door closing does not cut her off. Both are the same mechanism seen from opposite sides, which is why they are one PR. ## How it was verified `make build` and `make test` both exit 0. `cmd/mavwaked` coverage 32.4% to 44.4%. The tests assert the behaviour, not the tuning: - 200 frames of loud audio fed while playing produce zero extra utterances (the self-hearing bug, now failing if reintroduced). - Barge-in off by default; 50 very loud frames never stop playback. - 4 loud frames do not cut her off, the 5th does. - Two-frame bursts separated by quiet never accumulate into a false barge-in. - 300 frames at speaker-leak level (0.05 RMS — above the VAD floor, below the barge bar) never trigger. - After a cut, the interrupting speech is captured as its own utterance with a clean VAD. - A failed round-trip surfaces the error and starts no playback; an empty reply leaves capture open. - `aplayPlayer` is safe to `Stop()` when idle and spawns nothing for empty audio — neither needs aplay(1) installed. The acoustic half (does 0.12 actually clear this room's speaker leak) cannot be measured without the hardware; it is a QA step on the task. ## Not in this PR The other six open items on #287 — partial STT display, per-segment retry, per-room mic profiles, noise-floor calibration, short-response-while-speaking — are untouched and stay on the task. Partial STT in particular needs a streaming wire change, not a client change. Vikunja #287

claude added 1 commit 2026-08-01 03:36:34 +02:00

Stop mavwaked from hearing itself, and add barge-in (#287 ) fed33a4e16

Playback was `go playAudio(reply)` — fire and forget, nobody holding the
process handle. Two audible consequences fell out of that.

She answered herself. The capture loop kept feeding the VAD while the
speaker was running, so her own reply came back in through the mic,
tripped the VAD, and was shipped to the daemon as a fresh command. There
is no acoustic echo canceller in this pipeline, so the fix is
half-duplex: while she is speaking, the capture side is muted. That part
is unconditional — it repairs a defect, it is not a new capability.

And talking over her did nothing, because there was no handle to cancel.
-barge-in now cuts playback when sustained energy clears a room-tuned
threshold (-barge-in-rms, default 0.12 normalised, over -barge-in-frames
consecutive frames, default 5). It is off by default: without an echo
canceller the only way to tell "he is talking over her" from "the mic is
hearing her" is that he is much louder, and how much louder depends on
where the mic sits.

The frame decision moved out of main.go into session.feed, behind a
player and an utteranceSender interface, so all of it is testable with
no mic, no speaker and no daemon. Nine tests cover the self-hearing
case, the off-by-default case, the consecutive-frame requirement,
speaker-leak-level audio not triggering, capturing the interrupting
utterance after a cut, and failed round-trips not starting playback.

The other seven items on #287 (partial STT, per-segment retry, mic
profiles, noise-floor calibration, short-response-while-speaking) are
untouched and stay on the task.

claude reviewed 2026-08-01 11:35:51 +02:00

claude left a comment

Half-duplex is the right fix and the reasoning holds. With no echo canceller, muting capture is the only honest option. Barge-in has to be off by default because the threshold is a property of the room. Putting aplay behind a player interface makes both halves testable. The gen counter in aplayPlayer closes a real race, where a finishing playback clears the flag of the one that replaced it. TestSessionEchoLevelAudioNeverBargesIn is the test that matters most and it is there.

1. `dispatch` resets the VAD for a reason that is not the real one, and skips it on both paths that need it

The comment above the reset reads:

The VAD has been accumulating from the buffered mic stream while the round-trip blocked.

It has not. feed is called synchronously from captureLoop, and dispatch blocks inside it. While sender.Send is in flight, nothing reads from src and nothing is fed to the VAD. The audio piles up in arecord's pipe and the kernel buffer, not in v.utterance. The VAD state at that moment is exactly what Feed left it as when it emitted the utterance.

The backlog is real, but it arrives after dispatch returns. Which means the reset is placed correctly by luck on one path and is missing on the two that matter:

Empty reply (text-only turn). dispatch logs and returns before the reset and before Play. The player never starts, Playing() is false, and the whole pipe backlog feeds straight into the VAD at pipe speed. A round-trip is p50 2.7s through the LLM router. That is roughly 90 frames of room, of him finishing his sentence, of the TV. A spurious utterance goes to the daemon.
Send error. Same path, no reset, no playback. When the daemon is down, every failed turn seeds the next spurious trigger. A dead socket produces a retry loop driven by nothing but backlog.

TestSessionEmptyReplyLeavesCaptureOpen passes because fakeSender returns instantly. No test models a slow round-trip followed by a burst of buffered frames. That is the only condition under which any of this shows.

Reset unconditionally at the top of dispatch's return path, or better, drain the reader before resuming.

2. The same backlog is scored for barge-in, so she can be cut off before she says a word

With -barge-in on, the frames drained after Play starts are evaluated against barge.RMS with no notion of when they were captured. Walk it:

He starts a sentence longer than -max-ms. The VAD caps the utterance at 10s and emits it while he is still talking.
dispatch blocks for the round-trip. The rest of his sentence goes into the pipe.
Play starts. Playing() is true.
The loop resumes and drains the backlog as fast as the pipe delivers it. Those frames are him at the mic, so they clear 0.12 RMS easily.
Five in a row, which is 150ms of tape but microseconds of wall clock, and feed calls player.Stop().

She is cut off by audio recorded before she started speaking. The Frames: 5 guard is documented as "long enough that a door or a cough does not cut her off". That argument assumes the frames are real time. Draining a backlog breaks the assumption. Either discard whatever is already buffered at the moment Play starts, or make the loud-frame counter time-aware rather than count-aware.

3. Barge-in throws away the 150ms that proved he was talking

When loudFrames reaches barge.Frames, the handler stops playback, calls vad.Reset(), and returns. The five frames that triggered it are gone. The VAD begins from the next frame, so the first 150ms of his interruption is clipped before whisper ever sees it. On a short interrupt, that is the whole first word.

TestSessionCapturesTheInterruptingUtterance is named for this and does not test it. After the five loud frames it calls speakThenPause, which synthesises a fresh complete utterance. It shows that a later utterance is captured, which was never in doubt. It never checks what happened to the trigger frames. Keep them in a small ring and feed them to the VAD after the reset.

4. A stuck `aplay` now deafens her, where before it could not

Playing() gates all capture. aplayPlayer only clears the flag when the child exits or the 30s timeout fires. If the ALSA device is busy, aplay blocks on the write. The flag then stays set for the full 30s. During that window every frame is discarded and she answers nothing. Barge-in cannot rescue it, because barge-in is off by default.

Before this PR, go playAudio(reply) could not do this: a wedged player was inaudible, not deafening. The 30s ceiling is inherited from the old code, where it only bounded a leaked goroutine. Now it bounds how long she can be deaf. Cap the mute window by the reply's own duration plus a small margin, which you already have from reply.Duration().

Smaller notes

-barge-in -barge-in-rms 0 logs barge-in on (rms 0.0000 x 5) and then does nothing, because Enabled() requires RMS > 0. The log is the only feedback the operator gets, and it says the opposite of what happened.
s.sent++ in dispatch runs before the error check, so the counter reads "utterances shipped to the daemon" but counts failed round-trips too.
The doc comment in main.go says barge-in needs "sustained energy well above the speaker's leak level". bargeInConfig.RMS is compared against a raw frame RMS, with no reference to what the speaker leaks. So "well above" is a number the operator guesses, and guesses again when the mic moves. Log the observed RMS of suppressed frames at intervals. He can then tune it from data.

Half-duplex is the right fix and the reasoning holds. With no echo canceller, muting capture is the only honest option. Barge-in has to be off by default because the threshold is a property of the room. Putting `aplay` behind a `player` interface makes both halves testable. The `gen` counter in `aplayPlayer` closes a real race, where a finishing playback clears the flag of the one that replaced it. `TestSessionEchoLevelAudioNeverBargesIn` is the test that matters most and it is there. ## 1. `dispatch` resets the VAD for a reason that is not the real one, and skips it on both paths that need it The comment above the reset reads: > The VAD has been accumulating from the buffered mic stream while the round-trip blocked. It has not. `feed` is called synchronously from `captureLoop`, and `dispatch` blocks inside it. While `sender.Send` is in flight, nothing reads from `src` and nothing is fed to the VAD. The audio piles up in arecord's pipe and the kernel buffer, not in `v.utterance`. The VAD state at that moment is exactly what `Feed` left it as when it emitted the utterance. The backlog is real, but it arrives after `dispatch` returns. Which means the reset is placed correctly by luck on one path and is missing on the two that matter: - **Empty reply (text-only turn).** `dispatch` logs and returns before the reset and before `Play`. The player never starts, `Playing()` is false, and the whole pipe backlog feeds straight into the VAD at pipe speed. A round-trip is p50 2.7s through the LLM router. That is roughly 90 frames of room, of him finishing his sentence, of the TV. A spurious utterance goes to the daemon. - **Send error.** Same path, no reset, no playback. When the daemon is down, every failed turn seeds the next spurious trigger. A dead socket produces a retry loop driven by nothing but backlog. `TestSessionEmptyReplyLeavesCaptureOpen` passes because `fakeSender` returns instantly. No test models a slow round-trip followed by a burst of buffered frames. That is the only condition under which any of this shows. Reset unconditionally at the top of `dispatch`'s return path, or better, drain the reader before resuming. ## 2. The same backlog is scored for barge-in, so she can be cut off before she says a word With `-barge-in` on, the frames drained after `Play` starts are evaluated against `barge.RMS` with no notion of when they were captured. Walk it: 1. He starts a sentence longer than `-max-ms`. The VAD caps the utterance at 10s and emits it while he is still talking. 2. `dispatch` blocks for the round-trip. The rest of his sentence goes into the pipe. 3. `Play` starts. `Playing()` is true. 4. The loop resumes and drains the backlog as fast as the pipe delivers it. Those frames are him at the mic, so they clear 0.12 RMS easily. 5. Five in a row, which is 150ms of tape but microseconds of wall clock, and `feed` calls `player.Stop()`. She is cut off by audio recorded before she started speaking. The `Frames: 5` guard is documented as "long enough that a door or a cough does not cut her off". That argument assumes the frames are real time. Draining a backlog breaks the assumption. Either discard whatever is already buffered at the moment `Play` starts, or make the loud-frame counter time-aware rather than count-aware. ## 3. Barge-in throws away the 150ms that proved he was talking When `loudFrames` reaches `barge.Frames`, the handler stops playback, calls `vad.Reset()`, and returns. The five frames that triggered it are gone. The VAD begins from the next frame, so the first 150ms of his interruption is clipped before whisper ever sees it. On a short interrupt, that is the whole first word. `TestSessionCapturesTheInterruptingUtterance` is named for this and does not test it. After the five loud frames it calls `speakThenPause`, which synthesises a fresh complete utterance. It shows that a later utterance is captured, which was never in doubt. It never checks what happened to the trigger frames. Keep them in a small ring and feed them to the VAD after the reset. ## 4. A stuck `aplay` now deafens her, where before it could not `Playing()` gates all capture. `aplayPlayer` only clears the flag when the child exits or the 30s timeout fires. If the ALSA device is busy, `aplay` blocks on the write. The flag then stays set for the full 30s. During that window every frame is discarded and she answers nothing. Barge-in cannot rescue it, because barge-in is off by default. Before this PR, `go playAudio(reply)` could not do this: a wedged player was inaudible, not deafening. The 30s ceiling is inherited from the old code, where it only bounded a leaked goroutine. Now it bounds how long she can be deaf. Cap the mute window by the reply's own duration plus a small margin, which you already have from `reply.Duration()`. ## Smaller notes - `-barge-in -barge-in-rms 0` logs `barge-in on (rms 0.0000 x 5)` and then does nothing, because `Enabled()` requires `RMS > 0`. The log is the only feedback the operator gets, and it says the opposite of what happened. - `s.sent++` in `dispatch` runs before the error check, so the counter reads "utterances shipped to the daemon" but counts failed round-trips too. - The doc comment in `main.go` says barge-in needs "sustained energy well above the speaker's leak level". `bargeInConfig.RMS` is compared against a raw frame RMS, with no reference to what the speaker leaks. So "well above" is a number the operator guesses, and guesses again when the mic moves. Log the observed RMS of suppressed frames at intervals. He can then tune it from data.

kami referenced this issue from a commit

2026-08-01 14:43:54 +02:00

mavwaked: throw away the round-trip backlog before it becomes a turn

kami closed this pull request

2026-08-01 14:52:02 +02:00

kami commented

2026-08-01 14:52:55 +02:00

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#76