kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Measure how often Maven finds the right note #8

Closed
claude wants to merge 0 commits from overnight/recall-eval into master
pull from: overnight/recall-eval
merge into: kami:master
Conversation 2 Commits - Files Changed -
claude commented 2026-07-31 00:37:57 +02:00
Contributor
Copy Link
Copy Source

Measures something we had never measured: when you ask Maven a question, does she find the right note? Two commits — the harness, then the run and the write-up. No behaviour changes. Same shape as the routing eval: embedded fixture, Score(), a CI ratchet on the hash embedder, ONNX opt-in.

Read RECALL-EVAL-31-07-2026.md. The short version:

hash (CI) ONNX (deployed)
right note ranks 1st 36% 60%
actually answered 0% 48%
false recall 0/5 1/5

One useful question in two gets "не знаю" even though the right note was sitting at the top of the list. The 0.55 confidence gate throws it away.

And the gate cannot be tuned out of this. Correct recalls score from 0.559 up. False recalls reach 0.567. The two ranges overlap, so there is no threshold that keeps every real answer and rejects every wrong one. Raising it to 0.60 gets false recall to 0/5 and costs 8 points of real answers. I left the default alone — this is a number to change once, on evidence, not twice.

Why, named: models/embedder/ is a paraphrase model. It scores "do these two sentences look alike", not "does this note answer this question". So for "из-за чего кончилось место" a filler note about learning guitar chords beats the note about docker logs, 0.730. The fix is an asymmetric retriever (multilingual-e5-small, query:/passage: prefixes) — filed separately, and it should land before anyone touches the gate.

Worth flagging about the harness itself: the first version of this scorer reported 92% recall@1. It was wrong — it credited zero-score ties as hits, and with only 3 notes in the pool recall@3 was free. The fixture now has filler notes and ties do not count. Mentioning it because a good number from a new metric deserves suspicion, and this one earned it.

Measures something we had never measured: when you ask Maven a question, does she find the right note? Two commits — the harness, then the run and the write-up. **No behaviour changes.** Same shape as the routing eval: embedded fixture, `Score()`, a CI ratchet on the hash embedder, ONNX opt-in. Read `RECALL-EVAL-31-07-2026.md`. The short version: | | hash (CI) | ONNX (deployed) | |---|---|---| | right note ranks 1st | 36% | 60% | | **actually answered** | **0%** | **48%** | | false recall | 0/5 | 1/5 | **One useful question in two gets "не знаю"** even though the right note was sitting at the top of the list. The `0.55` confidence gate throws it away. **And the gate cannot be tuned out of this.** Correct recalls score from 0.559 up. False recalls reach 0.567. The two ranges overlap, so there is no threshold that keeps every real answer and rejects every wrong one. Raising it to 0.60 gets false recall to 0/5 and costs 8 points of real answers. I left the default alone — this is a number to change once, on evidence, not twice. **Why, named:** `models/embedder/` is a *paraphrase* model. It scores "do these two sentences look alike", not "does this note answer this question". So for "из-за чего кончилось место" a filler note about learning guitar chords beats the note about docker logs, 0.730. The fix is an asymmetric retriever (`multilingual-e5-small`, `query:`/`passage:` prefixes) — filed separately, and it should land *before* anyone touches the gate. **Worth flagging about the harness itself:** the first version of this scorer reported **92%** recall@1. It was wrong — it credited zero-score ties as hits, and with only 3 notes in the pool recall@3 was free. The fixture now has filler notes and ties do not count. Mentioning it because a good number from a new metric deserves suspicion, and this one earned it.
claude added 2 commits 2026-07-31 00:37:57 +02:00
Add a held-out note-recall harness (fixture + scorer) 43470abc57 
Measures whether Maven can find the right note again from a paraphrased
question. Review internal/memory/recalleval/recalleval.go's Score for how
rank, gate and false recall are kept as three separate numbers, and the
fixture's filler list for why recall@3 is not free.
Fixture JSON is generated data and does not count toward the diff limit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
Score the recall fixture and write up what it shows 8a174c1c70 
Real recall is 48% after the gate, and one must-be-silent query gets an
answer anyway. Review finding 2 (the score distributions overlap, so no
gate separates a real recall from a false one) and finding 4 (the memStore
branch at voice.go:776 is unreachable for notes). Adds an embedder cache
so the gate sweep does not re-embed the fixture nine times.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
kami commented 2026-07-31 18:45:32 +02:00
Owner
Copy Link
Copy Source

this one shouldn't go in master.

this one shouldn't go in master.
kami commented 2026-07-31 20:21:39 +02:00
Owner
Copy Link
Copy Source

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.
kami closed this pull request 2026-07-31 20:21:41 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kami
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kami/Maven#8
Delete Branch "overnight/recall-eval"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?