kami
8a174c1c70
Real recall is 48% after the gate, and one must-be-silent query gets an answer anyway. Review finding 2 (the score distributions overlap, so no gate separates a real recall from a false one) and finding 4 (the memStore branch at voice.go:776 is unreachable for notes). Adds an embedder cache so the gate sweep does not re-embed the fixture nine times. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
100 lines
6.3 KiB
Markdown
100 lines
6.3 KiB
Markdown
# Note recall evaluation — 31-07-2026
|
||
|
||
The operator's goal is that Maven "memorize/note things … and know more about me/world". This
|
||
measures whether the note/recall path delivers that.
|
||
|
||
- Fixture + scorer: `internal/memory/recalleval/` (`ru_recall_v1.json`, 30 cases)
|
||
- Reproduce: `make eval-recall` — hash ratchet always, ONNX when `deps/` is present
|
||
- Commit: `43470ab` (harness)
|
||
|
||
Each case inserts its own 3 notes **plus 12 shared filler notes** into a fresh store, embeds the
|
||
query, takes the top 3 — the read path `cmd/mavend/voice.go` runs for `IntentQuery`. Filler is
|
||
load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable
|
||
cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5
|
||
that must recall **nothing**. `TestFixtureIsParaphrased` fails the build if a query shares over half
|
||
its words with its note; equal-score ties count as ties, not recall.
|
||
|
||
## Results
|
||
|
||
| | recall+hash (CI ratchet) | recall+onnx (deployed) |
|
||
|---|---|---|
|
||
| **recall@1** | 36.0% (9/25) | **60.0% (15/25)** |
|
||
| recall@3 | 76.0% (19/25) | 80.0% (20/25) |
|
||
| **answered after the 0.55 gate** | **0.0% (0/25)** | **48.0% (12/25)** |
|
||
| wrong note on top / tie on top | 9 / 7 | 10 / 0 |
|
||
| ranked first, then silenced by the gate | 9 | 3 |
|
||
| **false recall** | 0/5 | **1/5 (20%)** |
|
||
| top-1 score when right, min / median | n/a | 0.559 / 0.678 |
|
||
| top-1 when it must stay silent, median / max | 0.000 / 0.144 | 0.470 / **0.567** |
|
||
| RU / EN / `hard` cases passed | 4/24 / 1/6 / 0/11 | 13/24 / 3/6 / 2/11 |
|
||
| latency p50 / p95 / max | 49µs / 70µs | 59ms / 148ms / 194ms |
|
||
|
||
Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so
|
||
CI has a deterministic ratchet with no model files.
|
||
|
||
## Findings
|
||
|
||
### 1. Real recall is 48%, not 60%
|
||
|
||
The right note ranks first 60% of the time, but the daemon only *says* it 48% of the time — three
|
||
more cases rank first and are then silenced by `voice.go:776`'s `queryMinScore`. **Roughly one
|
||
useful question in two gets "не знаю".** This is not a working memory yet.
|
||
|
||
### 2. The gate cannot separate a real recall from a false one — the distributions overlap
|
||
|
||
Right-note top-1 scores start at **0.559**. Must-stay-silent top-1 scores reach **0.567**. No
|
||
threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25
|
||
answered, 1/5 false; **0.55 (default) → 12/25, 1/5**; **0.60 → 10/25, 0/5**; 0.70 → 5/25, 0/5. What
|
||
the data says about `DefaultQueryMinScore` (`internal/config/config.go:392`): **0.55 is
|
||
slightly too loose** — it admits one confident wrong answer ("как зовут сестру моего коллеги"
|
||
recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60
|
||
silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means
|
||
the threshold is the wrong dial anyway (finding 3).
|
||
|
||
### 3. Filler notes outrank the right answer — the model scores similarity, not relevance
|
||
|
||
`models/embedder/` is **paraphrase-multilingual-MiniLM-L12-v2** (`Makefile:119`), a *symmetric*
|
||
paraphrase model. It scores "do these sentences look alike", not "does this passage answer this
|
||
question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего
|
||
кончилось место" and "откуда берётся токен бота" both return `выучил пару аккордов на гитаре`
|
||
(0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note
|
||
not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning
|
||
problem — an asymmetric retrieval model (`multilingual-e5-small`, with `query:` / `passage:`
|
||
prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately:
|
||
`deploy/mavend.json:39` loads a 470MB fp32 `model.onnx` while `make download-embedder` fetches
|
||
`model_quantized.onnx` — not the same file.
|
||
|
||
`hard` cases score **2/11**: every one is a query where the operator did not reuse his own words.
|
||
That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises.
|
||
|
||
### 4. The memory-store recall branch is dead for notes
|
||
|
||
`voice.go:776` only reaches `h.memStore.Search` when the notes-RAG top score is already below
|
||
`queryMinScore`, and `bestRecall` (`cmd/mavend/recall.go:19`) then applies the **same** gate to the
|
||
same vector. A note is indexed in both places with the same embedding, so if it failed the gate in
|
||
`QueryNotes` it fails again here — the branch can only ever return a **fact**. Its comment calls it
|
||
"additive"; for notes it is not.
|
||
|
||
### 5. Ranking has no recency or type signal, and the store is not the bottleneck
|
||
|
||
`internal/store/notes.go:67` sorts by cosine and uses `ts` only to break an exact float tie, which
|
||
never happens; `kind` never enters the ranking. Meanwhile `TestPersistentStoreScoresTheSame` scores
|
||
sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — both full-scan cosine
|
||
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
|
||
not the problem to solve.
|
||
|
||
## Next steps — ordered by value-to-risk; nothing here is a decision
|
||
|
||
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
|
||
change plus a prefix in `onnxembedder.go`, re-measurable in one command.
|
||
2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
|
||
`query_min_score` picked against today's embedder describes a model on its way out.
|
||
3. **Replace the absolute-score gate with a margin gate** (`top1 − top2 > δ`) — as the routing eval
|
||
concluded, absolute cosine cannot see a flat distribution.
|
||
4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
|
||
gate it separately, or restrict it to facts and say so.
|
||
5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
|
||
6. **Grow the fixture from real misses.** 30 cases can rank two embedders, not trust 4 points.
|
||
7. **Re-measure end to end.** Recall is gated twice — the utterance must first route to `query`,
|
||
which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.
|