# Note recall evaluation — 31-07-2026

The operator's goal is that Maven "memorize/note things … and know more about me/world". This
measures whether the note/recall path delivers that.

- Fixture + scorer: `internal/memory/recalleval/` (`ru_recall_v1.json`, 30 cases)
- Reproduce: `make eval-recall` — hash ratchet always, ONNX when `deps/` is present
- Commit: `43470ab` (harness)

Each case inserts its own 3 notes **plus 12 shared filler notes** into a fresh store, embeds the
query, takes the top 3 — the read path `cmd/mavend/voice.go` runs for `IntentQuery`. Filler is
load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable
cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5
that must recall **nothing**. `TestFixtureIsParaphrased` fails the build if a query shares over half
its words with its note; equal-score ties count as ties, not recall.

## Results

| | recall+hash (CI ratchet) | recall+onnx (deployed) |
|---|---|---|
| **recall@1** | 36.0% (9/25) | **60.0% (15/25)** |
| recall@3 | 76.0% (19/25) | 80.0% (20/25) |
| **answered after the 0.55 gate** | **0.0% (0/25)** | **48.0% (12/25)** |
| wrong note on top / tie on top | 9 / 7 | 10 / 0 |
| ranked first, then silenced by the gate | 9 | 3 |
| **false recall** | 0/5 | **1/5 (20%)** |
| top-1 score when right, min / median | n/a | 0.559 / 0.678 |
| top-1 when it must stay silent, median / max | 0.000 / 0.144 | 0.470 / **0.567** |
| RU / EN / `hard` cases passed | 4/24 / 1/6 / 0/11 | 13/24 / 3/6 / 2/11 |
| latency p50 / p95 / max | 49µs / 70µs | 59ms / 148ms / 194ms |

Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so
CI has a deterministic ratchet with no model files.

## Findings

### 1. Real recall is 48%, not 60%

The right note ranks first 60% of the time, but the daemon only *says* it 48% of the time — three
more cases rank first and are then silenced by `voice.go:776`'s `queryMinScore`. **Roughly one
useful question in two gets "не знаю".** This is not a working memory yet.

### 2. The gate cannot separate a real recall from a false one — the distributions overlap

Right-note top-1 scores start at **0.559**. Must-stay-silent top-1 scores reach **0.567**. No
threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25
answered, 1/5 false; **0.55 (default) → 12/25, 1/5**; **0.60 → 10/25, 0/5**; 0.70 → 5/25, 0/5. What
the data says about `DefaultQueryMinScore` (`internal/config/config.go:392`): **0.55 is
slightly too loose** — it admits one confident wrong answer ("как зовут сестру моего коллеги"
recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60
silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means
the threshold is the wrong dial anyway (finding 3).

### 3. Filler notes outrank the right answer — the model scores similarity, not relevance

`models/embedder/` is **paraphrase-multilingual-MiniLM-L12-v2** (`Makefile:119`), a *symmetric*
paraphrase model. It scores "do these sentences look alike", not "does this passage answer this
question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего
кончилось место" and "откуда берётся токен бота" both return `выучил пару аккордов на гитаре`
(0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note
not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning
problem — an asymmetric retrieval model (`multilingual-e5-small`, with `query:` / `passage:`
prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately:
`deploy/mavend.json:39` loads a 470MB fp32 `model.onnx` while `make download-embedder` fetches
`model_quantized.onnx` — not the same file.

`hard` cases score **2/11**: every one is a query where the operator did not reuse his own words.
That is the normal case weeks later, and exactly what docs/design.md's "recall when relevant" promises.

### 4. The memory-store recall branch is dead for notes

`voice.go:776` only reaches `h.memStore.Search` when the notes-RAG top score is already below
`queryMinScore`, and `bestRecall` (`cmd/mavend/recall.go:19`) then applies the **same** gate to the
same vector. A note is indexed in both places with the same embedding, so if it failed the gate in
`QueryNotes` it fails again here — the branch can only ever return a **fact**. Its comment calls it
"additive"; for notes it is not.

**Fixed (Vikunja #373).** The memory pass now runs *first*, as one search over notes and facts with
one gate, so whichever memory is clearly the best match answers — note or fact. The notes-only pass
stays behind it for notes the vector index does not hold. No threshold changed, so the set of
questions Maven answers is the same; only which memory answers them. The fixture gained two mixed
note+fact cases (`ru-mixed-031`, `ru-mixed-032`), which is why the counts below are out of 27
answerable cases and not 25: hash recall@1 36.0% (9/25) → 37.0% (10/27), e5 recall@1 72.0% (18/25) →
70.4% (19/27) with answered-after-gate 68.0% → 66.7% and false recall unchanged at 1/5.

### 5. Ranking has no recency or type signal, and the store is not the bottleneck

`internal/store/notes.go:67` sorts by cosine and uses `ts` only to break an exact float tie, which
never happens; `kind` never enters the ranking. Meanwhile `TestPersistentStoreScoresTheSame` scores
sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — both full-scan cosine
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
not the problem to solve.

## Re-measured after the embedder swap — 31-07-2026, later the same day

Changed: `models/embedder/` is now **multilingual-e5-small** (quantized, 118MB), with `query: ` in
front of a question and `passage: ` in front of a stored note (Vikunja #371). `deploy/mavend.json`
and `make download-embedder` now name the same file, and it is the quantized one — that is what the
column below measures (Vikunja #372). Everything else is unchanged: same fixture, same store, same
0.55 gate. The old column is the baseline and is left as it was.

| | recall+onnx, MiniLM (baseline) | recall+onnx, e5-small (new) |
|---|---|---|
| **recall@1** | 60.0% (15/25) | **72.0% (18/25)** |
| recall@3 | 80.0% (20/25) | 84.0% (21/25) |
| **answered after the 0.55 gate** | 48.0% (12/25) | **72.0% (18/25)** |
| wrong note on top / tie on top | 10 / 0 | 7 / 0 |
| ranked first, then silenced by the gate | 3 | 0 |
| **false recall** | 1/5 (20%) | **5/5 (100%)** |
| top-1 score when right, min / median | 0.559 / 0.678 | 0.791 / 0.857 |
| top-1 when it must stay silent, median / max | 0.470 / 0.567 | 0.815 / 0.835 |
| RU / EN / `hard` cases passed | 13/24 / 3/6 / 2/11 | 14/24 / 4/6 / 5/11 |
| latency p50 / p95 / max | 59ms / 148ms / 194ms | 18ms / 37ms / 49ms |

### What moved

Ranking got better and got faster. Half the previously-unwinnable `hard` cases now pass (2/11 →
5/11), the guitar note no longer beats the docker-logs note, and the gate stops silencing notes that
already ranked first. The quantized e5 is also ~3x quicker than the fp32 MiniLM it replaces.

### What got worse: the gate is now a no-op

e5 packs every cosine into a narrow high band. Right-note scores start at 0.791; must-stay-silent
scores reach 0.835. **The distributions still overlap, and now they overlap above the gate**, so
0.55 admits everything and false recall goes from 1/5 to 5/5. The sweep:

```
gate 0.50–0.70: answered 18/25 (72%)  false recall 5/5
gate 0.80:      answered 17/25 (68%)  false recall 4/5
gate 0.90:      answered  0/25 ( 0%)  false recall 0/5
```

There is no value that keeps real recall and rejects made-up questions — same conclusion as before,
now with a wider band and no room at all. `query_min_score` was left at 0.55 as instructed. **The
recommendation is to leave it there and stop tuning it**: any number under ~0.79 is a no-op and
anything above starts cutting real recall long before it stops the false ones. The fix is a margin
gate (`top1 − top2 > δ`), next-steps item 3, which is now the top item.

### The prefixes did not do the work

A control run with both prefixes set to the empty string scored the **same** recall@1 (72%), a
slightly better recall@3 (88%) and the same 5/5 false recall. So on this fixture the gain comes from
the model, not from the `query:` / `passage:` split. The prefixes are kept because they are how e5
was trained and the split is the right shape for the read path, but they are not worth defending on
this evidence — a bigger fixture may say otherwise.

### Stored vectors from the old model are now junk

Cosine between a MiniLM vector and an e5 vector means nothing. Every row already in `notes` and in
the vector memory table was written by the old model, so after this deploy they will score as noise
against a new query. A live database needs every note and fact re-embedded before recall works at
all. Filed as its own task.

## Margin gate — 31-07-2026, third run

Next-steps item 3, done. The absolute gate is replaced by a **margin gate**: answer only when the
top hit beats the runner-up by more than delta (`top1 − top2 > δ`). Same fixture, same e5 embedder,
same store as the run above. `internal/memory/gate.go` holds the check; both read paths call it
(`cmd/mavend/recall.go` and the notes-RAG branch in `voice.go`). New knob `voice.query_min_margin`
in `deploy/mavend.json`, default 0.008.

### Why the absolute gate could not work, in one line of data

The harness now prints the margin distributions, and they barely overlap where the raw scores
overlap completely:

| | top-1 score | margin (top1 − top2) |
|---|---|---|
| right note first (n=18) | min 0.810, median 0.862, max 0.890 | min 0.001, median 0.029, max 0.053 |
| must stay silent (n=5) | min 0.795, median 0.815, max 0.835 | min 0.000, median 0.002, **max 0.019** |

Four of the five must-be-silent cases have a margin at or under 0.002 — when there is nothing to
recall, e5 finds several notes equally close and no clear winner. That is the signal the absolute
score throws away.

### The delta sweep

Absolute gate held at 0.55 throughout.

```
delta 0.000: answered 18/25 (72%)  false recall 5/5
delta 0.002: answered 17/25 (68%)  false recall 3/5
delta 0.005: answered 17/25 (68%)  false recall 2/5
delta 0.008: answered 17/25 (68%)  false recall 1/5   <- chosen
delta 0.010: answered 15/25 (60%)  false recall 1/5
delta 0.012: answered 14/25 (56%)  false recall 1/5
delta 0.015: answered 12/25 (48%)  false recall 1/5
delta 0.020: answered 11/25 (44%)  false recall 0/5
delta 0.025: answered  9/25 (36%)  false recall 0/5
delta 0.030: answered  8/25 (32%)  false recall 0/5
delta 0.040: answered  4/25 (16%)  false recall 0/5
delta 0.050: answered  2/25 ( 8%)  false recall 0/5
delta 0.060: answered  0/25 ( 0%)  false recall 0/5
```

### Chosen: δ = 0.008

It is the best point on the frontier, not a taste call. **0.008 dominates 0.010, 0.012 and 0.015
outright** — same 1/5 false recall, 8 to 20 points more real recall. Everything below it buys recall
back only by admitting more false recalls (0.005 → 2/5, 0.002 → 3/5). The next real improvement is
0.020 at 0/5 false, and it costs 24 points of recall to get there.

The brief's bar was "recall above 60% with false recall at 1/5 or better". 0.008 clears it with room:
68% and 1/5.

### Before / after

| | absolute gate 0.55 (previous) | margin gate δ=0.008 |
|---|---|---|
| recall@1 (ranking, ungated) | 72.0% (18/25) | 72.0% (18/25) — unchanged, the gate does not rank |
| **answered after the gate** | 72.0% (18/25) | **68.0% (17/25)** |
| **false recall** | **5/5 (100%)** | **1/5 (20%)** |
| fixture cases passed | 18/30 | **21/30** |

Four false recalls removed for one real answer. That is the trade the spec asks for — she is not a
guesser-of-truth. The one survivor is `en-pref-025` ("should i be offered wine"), which recalls a
filler note at 0.796 with a 0.019 margin: the widest silent-case margin in the fixture, and it sits
inside the real-recall range, so no delta removes it without taking real answers with it.

### Does the absolute cutoff still earn its keep? Marginally — kept

On this fixture with e5 it is a **no-op**: the lowest right-note score is 0.791, so 0.55 rejects
nothing the margin does not already reject. It is kept for two reasons, neither glamorous. It still
does real work for the hash embedder (its own sweep shows answers dropping from 16% to 0% between
0.30 and 0.50), and it is the only thing standing between the user and a reply built from a store
where everything is far away but one row happens to be a little less far — a near-empty database, or
the stale-vector case below. Cheap insurance, no measured cost. If a later embedder makes it bite,
the sweep is one command.

### Caveat on the numbers

Five must-be-silent cases is a thin basis for a 4-point decision. 1/5 and 2/5 differ by one case.
The shape of the frontier is trustworthy — margins separate, absolute scores do not — but δ=0.008
itself should be re-read off a bigger fixture (next-steps item 6) before anyone defends the third
decimal.

## Next steps — ordered by value-to-risk; nothing here is a decision

1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
   change plus a prefix in `onnxembedder.go`, re-measurable in one command.
2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
   `query_min_score` picked against today's embedder describes a model on its way out.
3. ~~**Replace the absolute-score gate with a margin gate**~~ — done, see the section above.
   δ=0.008, false recall 5/5 → 1/5.
4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
   gate it separately, or restrict it to facts and say so.
5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
6. **Grow the fixture from real misses.** 30 cases can rank two embedders, not trust 4 points.
7. **Re-measure end to end.** Recall is gated twice — the utterance must first route to `query`,
   which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.
