kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Swap the embedder for e5-small #16

Closed
claude wants to merge 4 commits from overnight/embedder-e5 into overnight/router-refusal
pull from: overnight/embedder-e5
merge into: kami:overnight/router-refusal
Conversation 2 Commits 4 Files Changed 11
+227 -23
claude commented 2026-07-31 09:40:34 +02:00
Contributor
Copy Link
Copy Source

Vikunja #371 and #372. Three commits, but they only make sense together.

She finds the right note far more often — and she also now answers when she should shut up. Both halves are real. Read the last row.

MiniLM (old) e5-small (new)
right note ranked 1st 60% 72%
actually answered after the gate 48% 72%
the hard cases 2/11 5/11
latency p50 59ms 18ms
false recall 1/5 5/5

The first four rows are the reason to do this. e5-small is trained for asymmetric retrieval — a short question against a longer stored note — which is exactly the shape of "what did I say about X". MiniLM was trained for sentence similarity, the wrong job. And the quantized e5 is 3x faster despite being the better model.

The last row is the cost. The 0.55 cosine gate used to catch 4 of the 5 cases where the right answer is silence. e5 packs every score into 0.79–0.89, so 0.55 now admits everything, including all five. A sweep found no better cutoff: 0.50–0.70 all give 5/5 false, 0.80 gives 4/5, 0.90 answers nothing at all. An absolute threshold cannot separate these two anymore. A margin gate (top1 minus top2) is the only dial left, and it is being built right now as a follow-up. I would land the two together.

Also in here (#372): the deployed embedder path and the one the Makefile downloads were different files. They all name the same one now.

An honest negative: a control run with the query: / passage: prefixes emptied scored the same recall@1 and a slightly better recall@3. On these 30 cases the win is the model, not the prefix split. The prefixes stayed because they are e5's trained contract, but they are not earning anything measurable yet.

Existing databases break. Old MiniLM vectors and new e5 vectors are different spaces — comparing them gives noise, not a low score. Every note and memory vector written before this deploy needs re-embedding or recall is broken for it. Filed as Vikunja #378, not done here.

Noticed and not touched: preTokenize lowercases before a cased sentencepiece tokenizer. Wrong for both models, pre-existing.

Vikunja #371 and #372. Three commits, but they only make sense together. **She finds the right note far more often — and she also now answers when she should shut up.** Both halves are real. Read the last row. | | MiniLM (old) | e5-small (new) | |---|---|---| | right note ranked 1st | 60% | **72%** | | actually answered after the gate | 48% | **72%** | | the hard cases | 2/11 | **5/11** | | latency p50 | 59ms | **18ms** | | **false recall** | **1/5** | **5/5** | The first four rows are the reason to do this. e5-small is trained for asymmetric retrieval — a short question against a longer stored note — which is exactly the shape of "what did I say about X". MiniLM was trained for sentence *similarity*, the wrong job. And the quantized e5 is 3x faster despite being the better model. **The last row is the cost.** The 0.55 cosine gate used to catch 4 of the 5 cases where the right answer is silence. e5 packs every score into 0.79–0.89, so 0.55 now admits everything, including all five. A sweep found no better cutoff: 0.50–0.70 all give 5/5 false, 0.80 gives 4/5, 0.90 answers nothing at all. **An absolute threshold cannot separate these two anymore.** A margin gate (top1 minus top2) is the only dial left, and it is being built right now as a follow-up. I would land the two together. Also in here (#372): the deployed embedder path and the one the Makefile downloads were different files. They all name the same one now. **An honest negative:** a control run with the `query:` / `passage:` prefixes emptied scored the *same* recall@1 and a slightly better recall@3. On these 30 cases the win is the model, not the prefix split. The prefixes stayed because they are e5's trained contract, but they are not earning anything measurable yet. **Existing databases break.** Old MiniLM vectors and new e5 vectors are different spaces — comparing them gives noise, not a low score. Every note and memory vector written before this deploy needs re-embedding or recall is broken for it. Filed as Vikunja #378, not done here. Noticed and not touched: `preTokenize` lowercases before a cased sentencepiece tokenizer. Wrong for both models, pre-existing.
claude changed target branch from master to overnight/router-refusal 2026-07-31 10:39:22 +02:00
claude added 4 commits 2026-07-31 10:39:22 +02:00
Embed a question and a stored note differently (Vikunja #371) 751c2a705f 
Note recall is asymmetric: a short question goes in, a longer note comes
out. Adds EmbedQuery/EmbedPassage helpers and the e5 prefixes, and points
the note/fact write path at the passage side and the query path at the
query side. Reviewers: the three call sites in voice.go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
Swap the embedder to multilingual-e5-small (Vikunja #371, #372) f6d5a2a7a4 
The old model was a symmetric paraphrase model, so it scored "do these
look alike" instead of "does this note answer this question". Also fixes
the file mismatch: the Makefile, the deploy config and both evals now all
name the same quantized file, and the quantized one is what gets measured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
Record the recall numbers after the embedder swap 1d48755d12 
recall@1 60% to 72%, answered 48% to 72%, latency 3x better. But false
recall went 1/5 to 5/5: e5 packs every score into a narrow high band, so
the 0.55 gate now admits everything. Left the gate alone as instructed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
Merge branch 'worktree-agent-ad5da57e47b822152' into overnight-jul31 34521c30b8
kami commented 2026-07-31 18:48:27 +02:00
Owner
Copy Link
Copy Source

is this one even worth reviewing?

is this one even worth reviewing?
kami commented 2026-07-31 20:21:54 +02:00
Owner
Copy Link
Copy Source

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.
kami closed this pull request 2026-07-31 20:21:56 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kami
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kami/Maven#16
Delete Branch "overnight/embedder-e5"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?