Maven/docs/plans/10-speaker-recognition.md

# Plan: Speaker Recognition

**Goal:** Maven can tell who is speaking on the voice channel, and tag what she writes with
who said it.

**Status (2026-08-01, Vikunja #255):** the enrolment half is shipped. The recognising half is
**BLOCKED on a model download** — there is no speaker-embedding model on this box, and one
was not invented to fill the gap. See "Blocked, and on what" below.

## What shipped

| Piece | Where | State |
|---|---|---|
| `Recognizer` — identify, list, get, forget | `internal/speaker/recognizer.go` | done; `Identify` answers `ErrDisabled` until a model exists |
| Enrolment — several samples, averaged, re-normalised | `internal/speaker/enroll.go` | done |
| Profile shape, id validation, cosine similarity | `internal/speaker/speaker.go` | done |
| Profile storage as `speaker:<id>` vectors | `internal/memory` `Catalog` + `internal/store/memory.go` | done, no schema migration |
| Config block, off by default | `internal/config` `SpeakerConfig` | done |
| `enroll_speaker` / `list_speakers` / `forget_speaker` | `internal/ipc` | done, absent unless configured |
| Authority rows | `internal/auth/policy.go` | done — enrol step-up, forget write, list read |
| Daemon wiring + honest startup log | `cmd/mavend/speaker.go` | done |
| Embedding backend | `newSpeakerEmbedder` | **BLOCKED** — returns nil, seam only |
| Tagging voice writes with the speaker | `cmd/mavend/voice.go` | not wired; nothing to tag with yet |

## Blocked, and on what

A voiceprint needs a speaker-embedding model. The box was searched: `/mnt/hdd1/llms` holds
sixteen ggufs across seven families and every one of them is a text model. There is no ECAPA,
no x-vector, no titanet, no wespeaker, and no `.onnx` under `/mnt/hdd1` at all. There are also
no enrolment samples, because nothing has ever recorded any.

To unblock, two things are needed and neither can be done from inside the repo:

1. **A model.** SpeechBrain ECAPA-TDNN exported to ONNX (`speechbrain/spkrec-ecapa-voxceleb`,
   192-dim) is the usual choice and runs on CPU in well under a second for a few seconds of
   audio. Download it per the recipe in `AGENTS.md`, put it beside the other models so the
   bind mount picks it up, and point `speaker.model_path` at it.
2. **An implementation of one function.** `newSpeakerEmbedder` in `cmd/mavend/speaker.go` is
   the entire seam: give it an ONNX session that turns `audio.Audio` into a `[]float32` and
   `Identify` starts working. Nothing else changes — not the store, not the protocol, not the
   authority table, not the handlers. `internal/onnx` already loads the e5 embedder, so the
   runtime wiring exists to copy.
3. **Enrolment samples**, three or more per person, recorded deliberately.

### Why there is no fallback

The original plan offered "a simple MFCC + GMM" as the floor. That is refused. MFCC cosine
distance is a channel and loudness detector as much as a voice detector: it will happily match
two different people who sit at the same distance from the same microphone, and it drifts when
the room changes. A general classifier that is sometimes wrong is a nuisance; a **biometric**
that is confidently wrong writes false claims about named people into his memory, and then
those claims get recalled as fact. For this capability a bad floor is worse than none, so the
shipped state is honest absence: `speaker.Disabled`, `ErrDisabled`, and a startup line saying
so.

## The refusals, and why

- **Unknown speakers are NOT enrolled on first interaction.** The plan's fourth "done when"
  bullet asked for exactly that, with a TTS "кто это?" prompt. It is refused in
  `enroll.go`'s doc comment and there is no request shape in the protocol that could express
  it. Enrolling a voice is taking a biometric of a person; doing it automatically to whoever
  walks past the microphone does it to guests who are not party to the exchange, and a
  synthesised question into a room is not consent from whoever happens to answer. Enrolment is
  an explicit act: an id, a name, and samples recorded for the purpose.
- **One sample is not enough.** Three separate utterances and nine seconds minimum. A profile
  built from one sentence encodes that sentence as much as the person, and the threshold then
  behaves unpredictably against everything else.
- **An unknown voice stays unknown.** Below threshold, `Identify` returns `ErrUnknown` naming
  the closest profile in the error text for diagnosis, never as an answer. Guessing who is in
  the room is how false memories about people get written.
- **Deletion is one authority rung below enrolment.** Everywhere else in `policy.go` the
  destructive direction is gated at least as hard as the constructive one. Here that would be
  backwards: getting rid of a biometric must never be the harder half.
- **The voiceprint never crosses the socket.** `ListSpeakersResp` carries ids, names, dates
  and sample counts. The vector stays in core.
- **Off unless configured.** No `speaker` block ⇒ the three methods answer
  `ErrUnknownMethod`. There is no wire path on a default box that takes a voiceprint.

## Storage

Profiles live in the existing `memory_vectors` table under the `speaker:` id prefix, as the
plan intended, so there is no migration. What that needed was a wider interface than
`memory.Store`: `memory.Catalog` adds `ByPrefix` and `Delete`. `Delete` is the load-bearing
one — a voiceprint someone asked to be rid of has to actually go, and a search-only store
cannot do that. `InMemoryStore.Insert` also became an upsert by id, matching what the
persistent store already did, so re-enrolling replaces a profile instead of stacking a second
one behind the first.

Profiles do not collide with note or fact vectors: they are only ever read through
`ByPrefix("speaker:")`, and a note search never returns one because the prefix is not in its
query path.

## Config

```json
"speaker": {
  "enabled": true,
  "model_path": "/opt/maven/models/spk/ecapa-voxceleb.onnx",
  "lib_path": "/opt/maven/lib",
  "threshold": 0.7,
  "min_seconds": 2.0
}
```

`Recognizes()` requires both `enabled` and a `model_path`, so a half-filled block reads as off
rather than as a capability that fails every turn. With `enabled` and no model the daemon still
attaches the three methods — profiles can be created, listed and deleted — and logs that
recognition is blocked.

## Still open

- The embedding backend (above). Everything below waits on it.
- **Tagging voice writes.** `Profile.Source("tap:voice")` already produces
  `tap:voice:speaker:kami`, which is the shape step 6 asked for, but nothing calls it yet:
  with no recogniser there is no id to tag with. When the model lands, the hook is in the
  voice path before STT.
- **Speaker as router/phraser context.** Same dependency. Note the persona constraint when it
  arrives: Maven addresses the owner informally and speaks to him, so "ok, <name>" needs care
  for anyone who is not him.
- **An enrolment surface.** The three IPC methods exist; no page drives them. Enrolment is
  step-up, so it belongs on `/dash` behind a passkey, with a per-profile forget button next to
  each row — that button is the reason `list_speakers` exists.
- **A speaker column on the meeting recorder** (#253). Attributing lines in a transcript is
  the obvious pairing, and it is the place where getting attribution wrong is most damaging,
  so it waits for a real model too.