Maven/RECALL-EVAL-31-07-2026.md

Note recall evaluation — 31-07-2026

The operator's goal is that Maven "memorize/note things … and know more about me/world". This measures whether the note/recall path delivers that.

• Fixture + scorer: internal/memory/recalleval/ (ru_recall_v1.json, 30 cases)
• Reproduce: make eval-recall — hash ratchet always, ONNX when deps/ is present
• Commit: 43470ab (harness)

Each case inserts its own 3 notes plus 12 shared filler notes into a fresh store, embeds the query, takes the top 3 — the read path cmd/mavend/voice.go runs for IntentQuery. Filler is load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5 that must recall nothing. TestFixtureIsParaphrased fails the build if a query shares over half its words with its note; equal-score ties count as ties, not recall.

Results

	recall+hash (CI ratchet)	recall+onnx (deployed)
recall@1	36.0% (9/25)	60.0% (15/25)
recall@3	76.0% (19/25)	80.0% (20/25)
answered after the 0.55 gate	0.0% (0/25)	48.0% (12/25)
wrong note on top / tie on top	9 / 7	10 / 0
ranked first, then silenced by the gate	9	3
false recall	0/5	1/5 (20%)
top-1 score when right, min / median	n/a	0.559 / 0.678
top-1 when it must stay silent, median / max	0.000 / 0.144	0.470 / 0.567
RU / EN / hard cases passed	4/24 / 1/6 / 0/11	13/24 / 3/6 / 2/11
latency p50 / p95 / max	49µs / 70µs	59ms / 148ms / 194ms

Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so CI has a deterministic ratchet with no model files.

Findings

1. Real recall is 48%, not 60%

The right note ranks first 60% of the time, but the daemon only says it 48% of the time — three more cases rank first and are then silenced by voice.go:776's queryMinScore. Roughly one useful question in two gets "не знаю". This is not a working memory yet.

2. The gate cannot separate a real recall from a false one — the distributions overlap

Right-note top-1 scores start at 0.559. Must-stay-silent top-1 scores reach 0.567. No threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25 answered, 1/5 false; 0.55 (default) → 12/25, 1/5; 0.60 → 10/25, 0/5; 0.70 → 5/25, 0/5. What the data says about DefaultQueryMinScore (internal/config/config.go:392): 0.55 is slightly too loose — it admits one confident wrong answer ("как зовут сестру моего коллеги" recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60 silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means the threshold is the wrong dial anyway (finding 3).

3. Filler notes outrank the right answer — the model scores similarity, not relevance

models/embedder/ is paraphrase-multilingual-MiniLM-L12-v2 (Makefile:119), a symmetric paraphrase model. It scores "do these sentences look alike", not "does this passage answer this question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего кончилось место" and "откуда берётся токен бота" both return выучил пару аккордов на гитаре (0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning problem — an asymmetric retrieval model (multilingual-e5-small, with query: / passage: prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately: deploy/mavend.json:39 loads a 470MB fp32 model.onnx while make download-embedder fetches model_quantized.onnx — not the same file.

hard cases score 2/11: every one is a query where the operator did not reuse his own words. That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises.

4. The memory-store recall branch is dead for notes

voice.go:776 only reaches h.memStore.Search when the notes-RAG top score is already below queryMinScore, and bestRecall (cmd/mavend/recall.go:19) then applies the same gate to the same vector. A note is indexed in both places with the same embedding, so if it failed the gate in QueryNotes it fails again here — the branch can only ever return a fact. Its comment calls it "additive"; for notes it is not.

5. Ranking has no recency or type signal, and the store is not the bottleneck

internal/store/notes.go:67 sorts by cosine and uses ts only to break an exact float tie, which never happens; kind never enters the ranking. Meanwhile TestPersistentStoreScoresTheSame scores sqlite-backed store.MemoryStore and memory.InMemoryStore identically — both full-scan cosine (internal/store/memory.go:64) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is not the problem to solve.

Re-measured after the embedder swap — 31-07-2026, later the same day

Changed: models/embedder/ is now multilingual-e5-small (quantized, 118MB), with query: in front of a question and passage: in front of a stored note (Vikunja #371). deploy/mavend.json and make download-embedder now name the same file, and it is the quantized one — that is what the column below measures (Vikunja #372). Everything else is unchanged: same fixture, same store, same 0.55 gate. The old column is the baseline and is left as it was.

	recall+onnx, MiniLM (baseline)	recall+onnx, e5-small (new)
recall@1	60.0% (15/25)	72.0% (18/25)
recall@3	80.0% (20/25)	84.0% (21/25)
answered after the 0.55 gate	48.0% (12/25)	72.0% (18/25)
wrong note on top / tie on top	10 / 0	7 / 0
ranked first, then silenced by the gate	3	0
false recall	1/5 (20%)	5/5 (100%)
top-1 score when right, min / median	0.559 / 0.678	0.791 / 0.857
top-1 when it must stay silent, median / max	0.470 / 0.567	0.815 / 0.835
RU / EN / hard cases passed	13/24 / 3/6 / 2/11	14/24 / 4/6 / 5/11
latency p50 / p95 / max	59ms / 148ms / 194ms	18ms / 37ms / 49ms

What moved

Ranking got better and got faster. Half the previously-unwinnable hard cases now pass (2/11 → 5/11), the guitar note no longer beats the docker-logs note, and the gate stops silencing notes that already ranked first. The quantized e5 is also ~3x quicker than the fp32 MiniLM it replaces.

What got worse: the gate is now a no-op

e5 packs every cosine into a narrow high band. Right-note scores start at 0.791; must-stay-silent scores reach 0.835. The distributions still overlap, and now they overlap above the gate, so 0.55 admits everything and false recall goes from 1/5 to 5/5. The sweep:

gate 0.50–0.70: answered 18/25 (72%)  false recall 5/5
gate 0.80:      answered 17/25 (68%)  false recall 4/5
gate 0.90:      answered  0/25 ( 0%)  false recall 0/5

There is no value that keeps real recall and rejects made-up questions — same conclusion as before, now with a wider band and no room at all. query_min_score was left at 0.55 as instructed. The recommendation is to leave it there and stop tuning it: any number under ~0.79 is a no-op and anything above starts cutting real recall long before it stops the false ones. The fix is a margin gate (top1 − top2 > δ), next-steps item 3, which is now the top item.

The prefixes did not do the work

A control run with both prefixes set to the empty string scored the same recall@1 (72%), a slightly better recall@3 (88%) and the same 5/5 false recall. So on this fixture the gain comes from the model, not from the query: / passage: split. The prefixes are kept because they are how e5 was trained and the split is the right shape for the read path, but they are not worth defending on this evidence — a bigger fixture may say otherwise.

Stored vectors from the old model are now junk

Cosine between a MiniLM vector and an e5 vector means nothing. Every row already in notes and in the vector memory table was written by the old model, so after this deploy they will score as noise against a new query. A live database needs every note and fact re-embedded before recall works at all. Filed as its own task.

Next steps — ordered by value-to-risk; nothing here is a decision

1. Swap the embedder to multilingual-e5-small with query:/passage: prefixes. One config change plus a prefix in onnxembedder.go, re-measurable in one command.
2. Re-run make eval-recall, then set the gate from the sweep — not before. Any query_min_score picked against today's embedder describes a model on its way out.
3. Replace the absolute-score gate with a margin gate (top1 − top2 > δ) — as the routing eval concluded, absolute cosine cannot see a flat distribution.
4. Delete or repair the dead memStore branch at voice.go:776 — search before the gate, gate it separately, or restrict it to facts and say so.
5. Add a mild time decay to ranking — the newest statement of a preference is the true one.
6. Grow the fixture from real misses. 30 cases can rank two embedders, not trust 4 points.
7. Re-measure end to end. Recall is gated twice — the utterance must first route to query, which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.