RECALL-EVAL-31-07-2026.md

@@ -0,0 +1,99 @@
+# Note recall evaluation — 31-07-2026
+
+The operator's goal is that Maven "memorize/note things … and know more about me/world". This
+measures whether the note/recall path delivers that.
+
+- Fixture + scorer: `internal/memory/recalleval/` (`ru_recall_v1.json`, 30 cases)
+- Reproduce: `make eval-recall` — hash ratchet always, ONNX when `deps/` is present
+- Commit: `43470ab` (harness)
+
+Each case inserts its own 3 notes **plus 12 shared filler notes** into a fresh store, embeds the
+query, takes the top 3 — the read path `cmd/mavend/voice.go` runs for `IntentQuery`. Filler is
+load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable
+cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5
+that must recall **nothing**. `TestFixtureIsParaphrased` fails the build if a query shares over half
+its words with its note; equal-score ties count as ties, not recall.
+
+## Results
+
+| | recall+hash (CI ratchet) | recall+onnx (deployed) |
+|---|---|---|
+| **recall@1** | 36.0% (9/25) | **60.0% (15/25)** |
+| recall@3 | 76.0% (19/25) | 80.0% (20/25) |
+| **answered after the 0.55 gate** | **0.0% (0/25)** | **48.0% (12/25)** |
+| wrong note on top / tie on top | 9 / 7 | 10 / 0 |
+| ranked first, then silenced by the gate | 9 | 3 |
+| **false recall** | 0/5 | **1/5 (20%)** |
+| top-1 score when right, min / median | n/a | 0.559 / 0.678 |
+| top-1 when it must stay silent, median / max | 0.000 / 0.144 | 0.470 / **0.567** |
+| RU / EN / `hard` cases passed | 4/24 / 1/6 / 0/11 | 13/24 / 3/6 / 2/11 |
+| latency p50 / p95 / max | 49µs / 70µs | 59ms / 148ms / 194ms |
+
+Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so
+CI has a deterministic ratchet with no model files.
+
+## Findings
+
+### 1. Real recall is 48%, not 60%
+
+The right note ranks first 60% of the time, but the daemon only *says* it 48% of the time — three
+more cases rank first and are then silenced by `voice.go:776`'s `queryMinScore`. **Roughly one
+useful question in two gets "не знаю".** This is not a working memory yet.
+
+### 2. The gate cannot separate a real recall from a false one — the distributions overlap
+
+Right-note top-1 scores start at **0.559**. Must-stay-silent top-1 scores reach **0.567**. No
+threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25
+answered, 1/5 false; **0.55 (default) → 12/25, 1/5**; **0.60 → 10/25, 0/5**; 0.70 → 5/25, 0/5. What
+the data says about `DefaultQueryMinScore` (`internal/config/config.go:392`): **0.55 is
+slightly too loose** — it admits one confident wrong answer ("как зовут сестру моего коллеги"
+recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60
+silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means
+the threshold is the wrong dial anyway (finding 3).
+
+### 3. Filler notes outrank the right answer — the model scores similarity, not relevance
+
+`models/embedder/` is **paraphrase-multilingual-MiniLM-L12-v2** (`Makefile:119`), a *symmetric*
+paraphrase model. It scores "do these sentences look alike", not "does this passage answer this
+question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего
+кончилось место" and "откуда берётся токен бота" both return `выучил пару аккордов на гитаре`
+(0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note
+not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning
+problem — an asymmetric retrieval model (`multilingual-e5-small`, with `query:` / `passage:`
+prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately:
+`deploy/mavend.json:39` loads a 470MB fp32 `model.onnx` while `make download-embedder` fetches
+`model_quantized.onnx` — not the same file.
+
+`hard` cases score **2/11**: every one is a query where the operator did not reuse his own words.
+That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises.
+
+### 4. The memory-store recall branch is dead for notes
+
+`voice.go:776` only reaches `h.memStore.Search` when the notes-RAG top score is already below
+`queryMinScore`, and `bestRecall` (`cmd/mavend/recall.go:19`) then applies the **same** gate to the
+same vector. A note is indexed in both places with the same embedding, so if it failed the gate in
+`QueryNotes` it fails again here — the branch can only ever return a **fact**. Its comment calls it
+"additive"; for notes it is not.
+
+### 5. Ranking has no recency or type signal, and the store is not the bottleneck
+
+`internal/store/notes.go:67` sorts by cosine and uses `ts` only to break an exact float tie, which
+never happens; `kind` never enters the ranking. Meanwhile `TestPersistentStoreScoresTheSame` scores
+sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — both full-scan cosine
+(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
+not the problem to solve.
+
+## Next steps — ordered by value-to-risk; nothing here is a decision
+
+1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
+   change plus a prefix in `onnxembedder.go`, re-measurable in one command.
+2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
+   `query_min_score` picked against today's embedder describes a model on its way out.
+3. **Replace the absolute-score gate with a margin gate** (`top1 − top2 > δ`) — as the routing eval
+   concluded, absolute cosine cannot see a flat distribution.
+4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
+   gate it separately, or restrict it to facts and say so.
+5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
+6. **Grow the fixture from real misses.** 30 cases can rank two embedders, not trust 4 points.
+7. **Re-measure end to end.** Recall is gated twice — the utterance must first route to `query`,
+   which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.

internal/memory/recalleval/recalleval.go

@@ -104,6 +104,34 @@ func InMemory() (memory.Store, func(), error) {
 	return memory.NewInMemoryStore(), func() {}, nil
 }
 
+// Cache wraps an embedder so repeated text is embedded once. The gate sweep
+// scores the same fixture at nine thresholds, and every case re-inserts the
+// filler notes — without this the ONNX run spends minutes re-embedding
+// identical strings. Latency numbers come from the uncached run.
+func Cache(inner router.Embedder) router.Embedder {
+	return &cachingEmbedder{inner: inner, seen: map[string][]float32{}}
+}
+
+type cachingEmbedder struct {
+	inner router.Embedder
+	seen  map[string][]float32
+}
+
+func (c *cachingEmbedder) Dim() int     { return c.inner.Dim() }
+func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner
+
+func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
+	if v, ok := c.seen[text]; ok {
+		return v, nil
+	}
+	v, err := c.inner.Embed(ctx, text)
+	if err != nil {
+		return nil, err
+	}
+	c.seen[text] = v
+	return v, nil
+}
+
 // Outcome — one scored case.
 type Outcome struct {
 	Case Case

internal/memory/recalleval/recalleval_test.go

@@ -268,7 +268,9 @@ func TestONNXRecall(t *testing.T) {
 		t.Fatalf("Score: %v", err)
 	}
 	t.Log("\n" + rep.String() + rep.Failures())
-	t.Log("\ngate sweep:\n" + sweep(t, emb, f))
+	// Cached for the sweep only: the headline run above must pay the real
+	// embedder cost so its latency numbers mean something.
+	t.Log("\ngate sweep:\n" + sweep(t, Cache(emb), f))
 }
 
 // sweep scores the fixture at a range of gates and renders one line each. Two