claude
246db4e609
The swap itself already landed: deploy loads models/embedder/multilingual-e5-small/model_quantized.onnx, and onnxembedder.go grew EmbedQuery/EmbedPassage with the query:/passage: prefixes the model was trained with. What was missing is the half of #371 that says "re-run make eval-recall and compare against the recorded numbers", so nothing in the repo says whether it worked. It worked, on every axis at once. recall@1 60.0% → 70.4%, recall@3 80.0% → 85.2%, answered after the gate 48.0% → 63.0%, false recall 1/5 → 0/5, and latency p50 59ms → 23ms because the quantized file is 118MB against the 470MB fp32 one the old config loaded. The guitar-chords note no longer beats the docker-logs note. One premise of the task did not come true and the new doc says so. #371 expected a better retriever to separate the score distributions and make query_min_score tunable. It did not: right-first top-1 runs 0.791-0.890 and must-stay-silent runs 0.795-0.835, still overlapping, just higher and tighter. The margin separates them instead — 0.024 median against 0.002 — and 0.008 is the knee where all five silent cases are silenced at no cost. The score gate is close to inert now; the margin is the live dial. Neither is changed here, since #412 is where a sweep belongs. docs/evals/2026-08-04-recall-e5-small.md is the dated measurement. rearchitecture.md's "upgrade MiniLM → bge-m3 later" is now done and says so, CLAUDE.md names the retriever and the prefix rule where it already promises the embedder never leaves homesrv, and the Makefile comment points at this eval instead of the one that asked for the swap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
4.8 KiB
Note recall after the e5-small swap — 04-08-2026
Closes Vikunja #371, which asked for the swap and for this re-measurement. The embedder is no
longer paraphrase-multilingual-MiniLM-L12-v2. It is multilingual-e5-small, quantized, with the
query: / passage: prefixes it was trained with (internal/router/onnxembedder.go,
EmbedQuery / EmbedPassage). deploy/mavend.json loads
models/embedder/multilingual-e5-small/model_quantized.onnx, which is the same file make download-embedder fetches and the same file this run measured.
- Fixture + scorer:
internal/memory/recalleval/— 32 cases now, not 30 - Reproduce:
make eval-recall - Commit:
b6abb19 - Gate as deployed:
query_min_score0.55,query_min_margin0.008
The fixture grew since 31-07, so the case counts are not comparable row for row. The percentages are.
Results
| 31-07 MiniLM (onnx) | 04-08 e5-small (onnx) | |
|---|---|---|
| recall@1 | 60.0% (15/25) | 70.4% (19/27) |
| recall@3 | 80.0% (20/25) | 85.2% (23/27) |
| answered after the gate | 48.0% (12/25) | 63.0% (17/27) |
| false recall | 1/5 (20%) | 0/5 |
| wrong note on top / tie on top | 10 / 0 | 8 / 0 |
| ranked first, then silenced by the gate | 3 | 2 |
hard cases passed |
2/11 | 5/12 |
| RU / EN passed | 13/24 / 3/6 | 18/26 / 4/6 |
| latency p50 / p95 / max | 59ms / 148ms / 194ms | 23ms / 41ms / 62ms |
The hash ratchet CI runs is unchanged in kind and still answers nothing after the gate: recall@1 37.0%, recall@3 74.1%, 0/27 answered, 0/5 false. It is lexical and exists so CI has a deterministic floor. Never compare a hash number to an ONNX one.
Findings
1. The swap paid on every axis at once, including latency
Ten points of recall@1, fifteen points of answered, the one false recall gone, and it is 2.5× faster because the quantized e5-small is 118MB against the 470MB fp32 file the old config loaded. Finding 3 of the 31-07 eval predicted the recall half and said nothing about speed; the speed came from fixing the second half of that finding, which was that the deployed path loaded a different file than the download target.
The concrete case that eval named is fixed. "из-за чего кончилось место" no longer returns the
guitar-chords filler note. It now returns a homelab note, n2 at 0.884, and the wanted note is
still not in the top 3 — so the query moved from absurd to merely wrong. That is the shape of what
is left.
2. The score distributions still overlap. The margin is what separates them
This is the part of #371's premise that did not come true. Right-note-first top-1 scores run
0.791 / 0.857 / 0.890 (min / median / max). Must-stay-silent top-1 scores run 0.795 / 0.815 /
0.835. The silent cases sit inside the answering range, so no value of query_min_score keeps
every real recall and rejects every false one — the same verdict as 31-07, at a higher and tighter
band of scores.
What separates them is the second-place gap. Margin top1-top2 for a right first hit: median 0.024. For a must-be-silent case: median 0.002, max 0.019. A false recall is a note that beats its neighbours by nothing, because nothing in the store is about the question. The sweep:
| margin | answered | false recall |
|---|---|---|
| 0.000 | 18/27 (67%) | 3/5 |
| 0.005 | 17/27 (63%) | 1/5 |
| 0.008 (deployed) | 17/27 (63%) | 0/5 |
| 0.010 | 15/27 (56%) | 0/5 |
| 0.015 | 12/27 (44%) | 0/5 |
0.008 is the knee: it is the smallest margin that silences all five, and the next step up costs two
real answers for nothing. The score gate contributes almost nothing on its own — every value from
0.00 to 0.70 answers the same 18 and admits the same 3 — so query_min_score is now close to inert
and the margin is the live dial. Leave both where they are; #412 is where a further sweep belongs.
3. What is left is a retrieval problem, not a gate problem
Eight cases put the wrong note on top, and the failures cluster: hard 5/12, preference 5/9,
homelab 8/13. Four of the eight have the right note in the top 3, so a reranker would collect
them; the other four do not, so nothing downstream can. Two more rank first and are silenced by the
margin — en-hard-024 at 0.826 with margin 0.023, and ru-home-026 at 0.846 with margin 0.001,
which is a genuine near-tie against a second note that is also plausible.
Preference queries are the weakest class in a way that is not about the model. "когда запускать резервное копирование" and "как мне присылать оповещения" both return a fact, not the note that states the preference. Facts and notes are searched in one pass since #373, so a confidently-scored fact wins a question that a note answers better. That is a ranking policy question and it belongs in its own task, not in a threshold.