diff --git a/RECALL-EVAL-31-07-2026.md b/RECALL-EVAL-31-07-2026.md new file mode 100644 index 0000000..45852b9 --- /dev/null +++ b/RECALL-EVAL-31-07-2026.md @@ -0,0 +1,99 @@ +# Note recall evaluation — 31-07-2026 + +The operator's goal is that Maven "memorize/note things … and know more about me/world". This +measures whether the note/recall path delivers that. + +- Fixture + scorer: `internal/memory/recalleval/` (`ru_recall_v1.json`, 30 cases) +- Reproduce: `make eval-recall` — hash ratchet always, ONNX when `deps/` is present +- Commit: `43470ab` (harness) + +Each case inserts its own 3 notes **plus 12 shared filler notes** into a fresh store, embeds the +query, takes the top 3 — the read path `cmd/mavend/voice.go` runs for `IntentQuery`. Filler is +load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable +cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5 +that must recall **nothing**. `TestFixtureIsParaphrased` fails the build if a query shares over half +its words with its note; equal-score ties count as ties, not recall. + +## Results + +| | recall+hash (CI ratchet) | recall+onnx (deployed) | +|---|---|---| +| **recall@1** | 36.0% (9/25) | **60.0% (15/25)** | +| recall@3 | 76.0% (19/25) | 80.0% (20/25) | +| **answered after the 0.55 gate** | **0.0% (0/25)** | **48.0% (12/25)** | +| wrong note on top / tie on top | 9 / 7 | 10 / 0 | +| ranked first, then silenced by the gate | 9 | 3 | +| **false recall** | 0/5 | **1/5 (20%)** | +| top-1 score when right, min / median | n/a | 0.559 / 0.678 | +| top-1 when it must stay silent, median / max | 0.000 / 0.144 | 0.470 / **0.567** | +| RU / EN / `hard` cases passed | 4/24 / 1/6 / 0/11 | 13/24 / 3/6 / 2/11 | +| latency p50 / p95 / max | 49µs / 70µs | 59ms / 148ms / 194ms | + +Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so +CI has a deterministic ratchet with no model files. + +## Findings + +### 1. Real recall is 48%, not 60% + +The right note ranks first 60% of the time, but the daemon only *says* it 48% of the time — three +more cases rank first and are then silenced by `voice.go:776`'s `queryMinScore`. **Roughly one +useful question in two gets "не знаю".** This is not a working memory yet. + +### 2. The gate cannot separate a real recall from a false one — the distributions overlap + +Right-note top-1 scores start at **0.559**. Must-stay-silent top-1 scores reach **0.567**. No +threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25 +answered, 1/5 false; **0.55 (default) → 12/25, 1/5**; **0.60 → 10/25, 0/5**; 0.70 → 5/25, 0/5. What +the data says about `DefaultQueryMinScore` (`internal/config/config.go:392`): **0.55 is +slightly too loose** — it admits one confident wrong answer ("как зовут сестру моего коллеги" +recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60 +silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means +the threshold is the wrong dial anyway (finding 3). + +### 3. Filler notes outrank the right answer — the model scores similarity, not relevance + +`models/embedder/` is **paraphrase-multilingual-MiniLM-L12-v2** (`Makefile:119`), a *symmetric* +paraphrase model. It scores "do these sentences look alike", not "does this passage answer this +question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего +кончилось место" and "откуда берётся токен бота" both return `выучил пару аккордов на гитаре` +(0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note +not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning +problem — an asymmetric retrieval model (`multilingual-e5-small`, with `query:` / `passage:` +prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately: +`deploy/mavend.json:39` loads a 470MB fp32 `model.onnx` while `make download-embedder` fetches +`model_quantized.onnx` — not the same file. + +`hard` cases score **2/11**: every one is a query where the operator did not reuse his own words. +That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises. + +### 4. The memory-store recall branch is dead for notes + +`voice.go:776` only reaches `h.memStore.Search` when the notes-RAG top score is already below +`queryMinScore`, and `bestRecall` (`cmd/mavend/recall.go:19`) then applies the **same** gate to the +same vector. A note is indexed in both places with the same embedding, so if it failed the gate in +`QueryNotes` it fails again here — the branch can only ever return a **fact**. Its comment calls it +"additive"; for notes it is not. + +### 5. Ranking has no recency or type signal, and the store is not the bottleneck + +`internal/store/notes.go:67` sorts by cosine and uses `ts` only to break an exact float tie, which +never happens; `kind` never enters the ranking. Meanwhile `TestPersistentStoreScoresTheSame` scores +sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — both full-scan cosine +(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is +not the problem to solve. + +## Next steps — ordered by value-to-risk; nothing here is a decision + +1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config + change plus a prefix in `onnxembedder.go`, re-measurable in one command. +2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any + `query_min_score` picked against today's embedder describes a model on its way out. +3. **Replace the absolute-score gate with a margin gate** (`top1 − top2 > δ`) — as the routing eval + concluded, absolute cosine cannot see a flat distribution. +4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate, + gate it separately, or restrict it to facts and say so. +5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one. +6. **Grow the fixture from real misses.** 30 cases can rank two embedders, not trust 4 points. +7. **Re-measure end to end.** Recall is gated twice — the utterance must first route to `query`, + which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences. diff --git a/internal/memory/recalleval/recalleval.go b/internal/memory/recalleval/recalleval.go index 4763d19..1202683 100644 --- a/internal/memory/recalleval/recalleval.go +++ b/internal/memory/recalleval/recalleval.go @@ -104,6 +104,34 @@ func InMemory() (memory.Store, func(), error) { return memory.NewInMemoryStore(), func() {}, nil } +// Cache wraps an embedder so repeated text is embedded once. The gate sweep +// scores the same fixture at nine thresholds, and every case re-inserts the +// filler notes — without this the ONNX run spends minutes re-embedding +// identical strings. Latency numbers come from the uncached run. +func Cache(inner router.Embedder) router.Embedder { + return &cachingEmbedder{inner: inner, seen: map[string][]float32{}} +} + +type cachingEmbedder struct { + inner router.Embedder + seen map[string][]float32 +} + +func (c *cachingEmbedder) Dim() int { return c.inner.Dim() } +func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner + +func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) { + if v, ok := c.seen[text]; ok { + return v, nil + } + v, err := c.inner.Embed(ctx, text) + if err != nil { + return nil, err + } + c.seen[text] = v + return v, nil +} + // Outcome — one scored case. type Outcome struct { Case Case diff --git a/internal/memory/recalleval/recalleval_test.go b/internal/memory/recalleval/recalleval_test.go index 04dc361..9d80f2e 100644 --- a/internal/memory/recalleval/recalleval_test.go +++ b/internal/memory/recalleval/recalleval_test.go @@ -268,7 +268,9 @@ func TestONNXRecall(t *testing.T) { t.Fatalf("Score: %v", err) } t.Log("\n" + rep.String() + rep.Failures()) - t.Log("\ngate sweep:\n" + sweep(t, emb, f)) + // Cached for the sweep only: the headline run above must pay the real + // embedder cost so its latency numbers mean something. + t.Log("\ngate sweep:\n" + sweep(t, Cache(emb), f)) } // sweep scores the fixture at a range of gates and renders one line each. Two