Maven/RECALL-EVAL-31-07-2026.md

Note recall evaluation — 31-07-2026

The operator's goal is that Maven "memorize/note things … and know more about me/world". This measures whether the note/recall path delivers that.

• Fixture + scorer: internal/memory/recalleval/ (ru_recall_v1.json, 30 cases)
• Reproduce: make eval-recall — hash ratchet always, ONNX when deps/ is present
• Commit: 43470ab (harness)

Each case inserts its own 3 notes plus 12 shared filler notes into a fresh store, embeds the query, takes the top 3 — the read path cmd/mavend/voice.go runs for IntentQuery. Filler is load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5 that must recall nothing. TestFixtureIsParaphrased fails the build if a query shares over half its words with its note; equal-score ties count as ties, not recall.

Results

	recall+hash (CI ratchet)	recall+onnx (deployed)
recall@1	36.0% (9/25)	60.0% (15/25)
recall@3	76.0% (19/25)	80.0% (20/25)
answered after the 0.55 gate	0.0% (0/25)	48.0% (12/25)
wrong note on top / tie on top	9 / 7	10 / 0
ranked first, then silenced by the gate	9	3
false recall	0/5	1/5 (20%)
top-1 score when right, min / median	n/a	0.559 / 0.678
top-1 when it must stay silent, median / max	0.000 / 0.144	0.470 / 0.567
RU / EN / hard cases passed	4/24 / 1/6 / 0/11	13/24 / 3/6 / 2/11
latency p50 / p95 / max	49µs / 70µs	59ms / 148ms / 194ms

Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so CI has a deterministic ratchet with no model files.

Findings

1. Real recall is 48%, not 60%

The right note ranks first 60% of the time, but the daemon only says it 48% of the time — three more cases rank first and are then silenced by voice.go:776's queryMinScore. Roughly one useful question in two gets "не знаю". This is not a working memory yet.

2. The gate cannot separate a real recall from a false one — the distributions overlap

Right-note top-1 scores start at 0.559. Must-stay-silent top-1 scores reach 0.567. No threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25 answered, 1/5 false; 0.55 (default) → 12/25, 1/5; 0.60 → 10/25, 0/5; 0.70 → 5/25, 0/5. What the data says about DefaultQueryMinScore (internal/config/config.go:392): 0.55 is slightly too loose — it admits one confident wrong answer ("как зовут сестру моего коллеги" recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60 silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means the threshold is the wrong dial anyway (finding 3).

3. Filler notes outrank the right answer — the model scores similarity, not relevance

models/embedder/ is paraphrase-multilingual-MiniLM-L12-v2 (Makefile:119), a symmetric paraphrase model. It scores "do these sentences look alike", not "does this passage answer this question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего кончилось место" and "откуда берётся токен бота" both return выучил пару аккордов на гитаре (0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning problem — an asymmetric retrieval model (multilingual-e5-small, with query: / passage: prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately: deploy/mavend.json:39 loads a 470MB fp32 model.onnx while make download-embedder fetches model_quantized.onnx — not the same file.

hard cases score 2/11: every one is a query where the operator did not reuse his own words. That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises.

4. The memory-store recall branch is dead for notes

voice.go:776 only reaches h.memStore.Search when the notes-RAG top score is already below queryMinScore, and bestRecall (cmd/mavend/recall.go:19) then applies the same gate to the same vector. A note is indexed in both places with the same embedding, so if it failed the gate in QueryNotes it fails again here — the branch can only ever return a fact. Its comment calls it "additive"; for notes it is not.

5. Ranking has no recency or type signal, and the store is not the bottleneck

internal/store/notes.go:67 sorts by cosine and uses ts only to break an exact float tie, which never happens; kind never enters the ranking. Meanwhile TestPersistentStoreScoresTheSame scores sqlite-backed store.MemoryStore and memory.InMemoryStore identically — both full-scan cosine (internal/store/memory.go:64) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is not the problem to solve.

Next steps — ordered by value-to-risk; nothing here is a decision

1. Swap the embedder to multilingual-e5-small with query:/passage: prefixes. One config change plus a prefix in onnxembedder.go, re-measurable in one command.
2. Re-run make eval-recall, then set the gate from the sweep — not before. Any query_min_score picked against today's embedder describes a model on its way out.
3. Replace the absolute-score gate with a margin gate (top1 − top2 > δ) — as the routing eval concluded, absolute cosine cannot see a flat distribution.
4. Delete or repair the dead memStore branch at voice.go:776 — search before the gate, gate it separately, or restrict it to facts and say so.
5. Add a mild time decay to ranking — the newest statement of a preference is the true one.
6. Grow the fixture from real misses. 30 cases can rank two embedders, not trust 4 points.
7. Re-measure end to end. Recall is gated twice — the utterance must first route to query, which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.