kami
1d48755d12
recall@1 60% to 72%, answered 48% to 72%, latency 3x better. But false recall went 1/5 to 5/5: e5 packs every score into a narrow high band, so the 0.55 gate now admits everything. Left the gate alone as instructed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
160 lines
9.6 KiB
Markdown
160 lines
9.6 KiB
Markdown
# Note recall evaluation — 31-07-2026
|
||
|
||
The operator's goal is that Maven "memorize/note things … and know more about me/world". This
|
||
measures whether the note/recall path delivers that.
|
||
|
||
- Fixture + scorer: `internal/memory/recalleval/` (`ru_recall_v1.json`, 30 cases)
|
||
- Reproduce: `make eval-recall` — hash ratchet always, ONNX when `deps/` is present
|
||
- Commit: `43470ab` (harness)
|
||
|
||
Each case inserts its own 3 notes **plus 12 shared filler notes** into a fresh store, embeds the
|
||
query, takes the top 3 — the read path `cmd/mavend/voice.go` runs for `IntentQuery`. Filler is
|
||
load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable
|
||
cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5
|
||
that must recall **nothing**. `TestFixtureIsParaphrased` fails the build if a query shares over half
|
||
its words with its note; equal-score ties count as ties, not recall.
|
||
|
||
## Results
|
||
|
||
| | recall+hash (CI ratchet) | recall+onnx (deployed) |
|
||
|---|---|---|
|
||
| **recall@1** | 36.0% (9/25) | **60.0% (15/25)** |
|
||
| recall@3 | 76.0% (19/25) | 80.0% (20/25) |
|
||
| **answered after the 0.55 gate** | **0.0% (0/25)** | **48.0% (12/25)** |
|
||
| wrong note on top / tie on top | 9 / 7 | 10 / 0 |
|
||
| ranked first, then silenced by the gate | 9 | 3 |
|
||
| **false recall** | 0/5 | **1/5 (20%)** |
|
||
| top-1 score when right, min / median | n/a | 0.559 / 0.678 |
|
||
| top-1 when it must stay silent, median / max | 0.000 / 0.144 | 0.470 / **0.567** |
|
||
| RU / EN / `hard` cases passed | 4/24 / 1/6 / 0/11 | 13/24 / 3/6 / 2/11 |
|
||
| latency p50 / p95 / max | 49µs / 70µs | 59ms / 148ms / 194ms |
|
||
|
||
Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so
|
||
CI has a deterministic ratchet with no model files.
|
||
|
||
## Findings
|
||
|
||
### 1. Real recall is 48%, not 60%
|
||
|
||
The right note ranks first 60% of the time, but the daemon only *says* it 48% of the time — three
|
||
more cases rank first and are then silenced by `voice.go:776`'s `queryMinScore`. **Roughly one
|
||
useful question in two gets "не знаю".** This is not a working memory yet.
|
||
|
||
### 2. The gate cannot separate a real recall from a false one — the distributions overlap
|
||
|
||
Right-note top-1 scores start at **0.559**. Must-stay-silent top-1 scores reach **0.567**. No
|
||
threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25
|
||
answered, 1/5 false; **0.55 (default) → 12/25, 1/5**; **0.60 → 10/25, 0/5**; 0.70 → 5/25, 0/5. What
|
||
the data says about `DefaultQueryMinScore` (`internal/config/config.go:392`): **0.55 is
|
||
slightly too loose** — it admits one confident wrong answer ("как зовут сестру моего коллеги"
|
||
recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60
|
||
silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means
|
||
the threshold is the wrong dial anyway (finding 3).
|
||
|
||
### 3. Filler notes outrank the right answer — the model scores similarity, not relevance
|
||
|
||
`models/embedder/` is **paraphrase-multilingual-MiniLM-L12-v2** (`Makefile:119`), a *symmetric*
|
||
paraphrase model. It scores "do these sentences look alike", not "does this passage answer this
|
||
question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего
|
||
кончилось место" and "откуда берётся токен бота" both return `выучил пару аккордов на гитаре`
|
||
(0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note
|
||
not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning
|
||
problem — an asymmetric retrieval model (`multilingual-e5-small`, with `query:` / `passage:`
|
||
prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately:
|
||
`deploy/mavend.json:39` loads a 470MB fp32 `model.onnx` while `make download-embedder` fetches
|
||
`model_quantized.onnx` — not the same file.
|
||
|
||
`hard` cases score **2/11**: every one is a query where the operator did not reuse his own words.
|
||
That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises.
|
||
|
||
### 4. The memory-store recall branch is dead for notes
|
||
|
||
`voice.go:776` only reaches `h.memStore.Search` when the notes-RAG top score is already below
|
||
`queryMinScore`, and `bestRecall` (`cmd/mavend/recall.go:19`) then applies the **same** gate to the
|
||
same vector. A note is indexed in both places with the same embedding, so if it failed the gate in
|
||
`QueryNotes` it fails again here — the branch can only ever return a **fact**. Its comment calls it
|
||
"additive"; for notes it is not.
|
||
|
||
### 5. Ranking has no recency or type signal, and the store is not the bottleneck
|
||
|
||
`internal/store/notes.go:67` sorts by cosine and uses `ts` only to break an exact float tie, which
|
||
never happens; `kind` never enters the ranking. Meanwhile `TestPersistentStoreScoresTheSame` scores
|
||
sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — both full-scan cosine
|
||
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
|
||
not the problem to solve.
|
||
|
||
## Re-measured after the embedder swap — 31-07-2026, later the same day
|
||
|
||
Changed: `models/embedder/` is now **multilingual-e5-small** (quantized, 118MB), with `query: ` in
|
||
front of a question and `passage: ` in front of a stored note (Vikunja #371). `deploy/mavend.json`
|
||
and `make download-embedder` now name the same file, and it is the quantized one — that is what the
|
||
column below measures (Vikunja #372). Everything else is unchanged: same fixture, same store, same
|
||
0.55 gate. The old column is the baseline and is left as it was.
|
||
|
||
| | recall+onnx, MiniLM (baseline) | recall+onnx, e5-small (new) |
|
||
|---|---|---|
|
||
| **recall@1** | 60.0% (15/25) | **72.0% (18/25)** |
|
||
| recall@3 | 80.0% (20/25) | 84.0% (21/25) |
|
||
| **answered after the 0.55 gate** | 48.0% (12/25) | **72.0% (18/25)** |
|
||
| wrong note on top / tie on top | 10 / 0 | 7 / 0 |
|
||
| ranked first, then silenced by the gate | 3 | 0 |
|
||
| **false recall** | 1/5 (20%) | **5/5 (100%)** |
|
||
| top-1 score when right, min / median | 0.559 / 0.678 | 0.791 / 0.857 |
|
||
| top-1 when it must stay silent, median / max | 0.470 / 0.567 | 0.815 / 0.835 |
|
||
| RU / EN / `hard` cases passed | 13/24 / 3/6 / 2/11 | 14/24 / 4/6 / 5/11 |
|
||
| latency p50 / p95 / max | 59ms / 148ms / 194ms | 18ms / 37ms / 49ms |
|
||
|
||
### What moved
|
||
|
||
Ranking got better and got faster. Half the previously-unwinnable `hard` cases now pass (2/11 →
|
||
5/11), the guitar note no longer beats the docker-logs note, and the gate stops silencing notes that
|
||
already ranked first. The quantized e5 is also ~3x quicker than the fp32 MiniLM it replaces.
|
||
|
||
### What got worse: the gate is now a no-op
|
||
|
||
e5 packs every cosine into a narrow high band. Right-note scores start at 0.791; must-stay-silent
|
||
scores reach 0.835. **The distributions still overlap, and now they overlap above the gate**, so
|
||
0.55 admits everything and false recall goes from 1/5 to 5/5. The sweep:
|
||
|
||
```
|
||
gate 0.50–0.70: answered 18/25 (72%) false recall 5/5
|
||
gate 0.80: answered 17/25 (68%) false recall 4/5
|
||
gate 0.90: answered 0/25 ( 0%) false recall 0/5
|
||
```
|
||
|
||
There is no value that keeps real recall and rejects made-up questions — same conclusion as before,
|
||
now with a wider band and no room at all. `query_min_score` was left at 0.55 as instructed. **The
|
||
recommendation is to leave it there and stop tuning it**: any number under ~0.79 is a no-op and
|
||
anything above starts cutting real recall long before it stops the false ones. The fix is a margin
|
||
gate (`top1 − top2 > δ`), next-steps item 3, which is now the top item.
|
||
|
||
### The prefixes did not do the work
|
||
|
||
A control run with both prefixes set to the empty string scored the **same** recall@1 (72%), a
|
||
slightly better recall@3 (88%) and the same 5/5 false recall. So on this fixture the gain comes from
|
||
the model, not from the `query:` / `passage:` split. The prefixes are kept because they are how e5
|
||
was trained and the split is the right shape for the read path, but they are not worth defending on
|
||
this evidence — a bigger fixture may say otherwise.
|
||
|
||
### Stored vectors from the old model are now junk
|
||
|
||
Cosine between a MiniLM vector and an e5 vector means nothing. Every row already in `notes` and in
|
||
the vector memory table was written by the old model, so after this deploy they will score as noise
|
||
against a new query. A live database needs every note and fact re-embedded before recall works at
|
||
all. Filed as its own task.
|
||
|
||
## Next steps — ordered by value-to-risk; nothing here is a decision
|
||
|
||
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
|
||
change plus a prefix in `onnxembedder.go`, re-measurable in one command.
|
||
2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
|
||
`query_min_score` picked against today's embedder describes a model on its way out.
|
||
3. **Replace the absolute-score gate with a margin gate** (`top1 − top2 > δ`) — as the routing eval
|
||
concluded, absolute cosine cannot see a flat distribution.
|
||
4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
|
||
gate it separately, or restrict it to facts and say so.
|
||
5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
|
||
6. **Grow the fixture from real misses.** 30 cases can rank two embedders, not trust 4 points.
|
||
7. **Re-measure end to end.** Recall is gated twice — the utterance must first route to `query`,
|
||
which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.
|