@@ -0,0 +1,88 @@
# Note recall after the e5-small swap — 04-08-2026
Closes Vikunja #371 , which asked for the swap and for this re-measurement. The embedder is no
longer paraphrase-multilingual-MiniLM-L12-v2. It is **multilingual-e5-small ** , quantized, with the
`query:` / `passage:` prefixes it was trained with (`internal/router/onnxembedder.go` ,
`EmbedQuery` / `EmbedPassage` ). `deploy/mavend.json` loads
`models/embedder/multilingual-e5-small/model_quantized.onnx` , which is the same file `make
download-embedder` fetches and the same file this run measured.
- Fixture + scorer: `internal/memory/recalleval/` — 32 cases now, not 30
- Reproduce: `make eval-recall`
- Commit: `b6abb19`
- Gate as deployed: `query_min_score` 0.55, `query_min_margin` 0.008
The fixture grew since 31-07, so the case counts are not comparable row for row. The percentages
are.
## Results
| | 31-07 MiniLM (onnx) | 04-08 e5-small (onnx) |
|---|---|---|
| **recall@1 ** | 60.0% (15/25) | **70.4% (19/27) ** |
| recall@3 | 80.0% (20/25) | **85.2% (23/27) ** |
| **answered after the gate ** | 48.0% (12/25) | **63.0% (17/27) ** |
| **false recall ** | 1/5 (20%) | **0/5 ** |
| wrong note on top / tie on top | 10 / 0 | 8 / 0 |
| ranked first, then silenced by the gate | 3 | 2 |
| `hard` cases passed | 2/11 | 5/12 |
| RU / EN passed | 13/24 / 3/6 | 18/26 / 4/6 |
| latency p50 / p95 / max | 59ms / 148ms / 194ms | **23ms / 41ms / 62ms ** |
The hash ratchet CI runs is unchanged in kind and still answers nothing after the gate: recall@1
37.0%, recall@3 74.1%, 0/27 answered, 0/5 false. It is lexical and exists so CI has a deterministic
floor. Never compare a hash number to an ONNX one.
## Findings
### 1. The swap paid on every axis at once, including latency
Ten points of recall@1 , fifteen points of * answered * , the one false recall gone, and it is 2.5×
faster because the quantized e5-small is 118MB against the 470MB fp32 file the old config loaded.
Finding 3 of the 31-07 eval predicted the recall half and said nothing about speed; the speed came
from fixing the second half of that finding, which was that the deployed path loaded a different
file than the download target.
The concrete case that eval named is fixed. "из-за чего кончилось место" no longer returns the
guitar-chords filler note. It now returns a homelab note, `n2` at 0.884, and the wanted note is
still not in the top 3 — so the query moved from absurd to merely wrong. That is the shape of what
is left.
### 2. The score distributions still overlap. The margin is what separates them
This is the part of #371 's premise that did not come true. Right-note-first top-1 scores run
0.791 / 0.857 / 0.890 (min / median / max). Must-stay-silent top-1 scores run 0.795 / 0.815 /
0.835. The silent cases sit * inside * the answering range, so no value of `query_min_score` keeps
every real recall and rejects every false one — the same verdict as 31-07, at a higher and tighter
band of scores.
What separates them is the second-place gap. Margin top1-top2 for a right first hit: median 0.024.
For a must-be-silent case: median 0.002, max 0.019. A false recall is a note that beats its
neighbours by nothing, because nothing in the store is about the question. The sweep:
| margin | answered | false recall |
|---|---|---|
| 0.000 | 18/27 (67%) | 3/5 |
| 0.005 | 17/27 (63%) | 1/5 |
| **0.008 (deployed) ** | **17/27 (63%) ** | **0/5 ** |
| 0.010 | 15/27 (56%) | 0/5 |
| 0.015 | 12/27 (44%) | 0/5 |
0.008 is the knee: it is the smallest margin that silences all five, and the next step up costs two
real answers for nothing. The score gate contributes almost nothing on its own — every value from
0.00 to 0.70 answers the same 18 and admits the same 3 — so `query_min_score` is now close to inert
and the margin is the live dial. Leave both where they are; #412 is where a further sweep belongs.
### 3. What is left is a retrieval problem, not a gate problem
Eight cases put the wrong note on top, and the failures cluster: `hard` 5/12, `preference` 5/9,
`homelab` 8/13. Four of the eight have the right note in the top 3, so a reranker would collect
them; the other four do not, so nothing downstream can. Two more rank first and are silenced by the
margin — `en-hard-024` at 0.826 with margin 0.023, and `ru-home-026` at 0.846 with margin 0.001,
which is a genuine near-tie against a second note that is also plausible.
Preference queries are the weakest class in a way that is not about the model. "когда запускать
резервное копирование" and "как мне присылать оповещения" both return a fact, not the note that
states the preference. Facts and notes are searched in one pass since #373 , so a confidently-scored
fact wins a question that a note answers better. That is a ranking policy question and it belongs
in its own task, not in a threshold.