Gate note recall on the margin, not the raw score #19

2026-07-31T09:53:18+02:00

claude commented

2026-07-31 09:53:18 +02:00

The other half of PR 16. Merge them together — PR 16 alone lifts recall but makes her answer with the wrong note in all five cases where she should stay silent. This fixes that.

	absolute 0.55	margin δ=0.008
answered after the gate	72% (18/25)	68% (17/25)
false recall	5/5	1/5
fixture cases passed	18/30	21/30

Four wrong answers removed for the price of one right one. That is the trade Maven should take — a wrong note is worse than "I don't know".

Why the old gate stopped working. It compared the top score against a fixed number. e5 packs everything into 0.79–0.89, so any cutoff either admits all of it or none of it. The new gate compares the top hit against the runner-up instead. When there is genuinely nothing to recall, e5 finds several notes equally close and there is no clear winner — and that gap is the signal the raw score throws away. Right-note margins run 0.001–0.053; must-stay-silent margins run 0.000–0.019, four of the five at or under 0.002.

Why 0.008. The full sweep is in the doc. It beats 0.010, 0.012 and 0.015 outright — same 1/5 false recall, 8 to 20 points more real recall. Going lower only buys recall by letting false answers back in. The next real gain is 0.020 at 0/5 false, which costs 24 points of recall.

The old absolute cutoff stays in. On this fixture with e5 it is a no-op, but it still does real work for the hash embedder (which CI's ratchet depends on), and it is the only guard for a near-empty or stale-vector store where everything is far away but one row is marginally less far. It costs nothing measured.

Two caveats worth reading:

The last false recall cannot be tuned away. "should i be offered wine" pulls a filler note with a 0.019 margin — wider than some real hits. No delta removes it without taking real answers with it.
Five must-stay-silent cases is thin. 1/5 versus 2/5 is a single case. The shape of the trade-off is trustworthy; the third decimal of 0.008 is not. The fixture wants more silent cases before anyone tunes this further.

Both new settings are in deploy/mavend.json (query_min_margin), defaulted in code.

The other half of PR 16. **Merge them together** — PR 16 alone lifts recall but makes her answer with the wrong note in all five cases where she should stay silent. This fixes that. | | absolute 0.55 | margin δ=0.008 | |---|---|---| | answered after the gate | 72% (18/25) | **68% (17/25)** | | **false recall** | **5/5** | **1/5** | | fixture cases passed | 18/30 | **21/30** | **Four wrong answers removed for the price of one right one.** That is the trade Maven should take — a wrong note is worse than "I don't know". **Why the old gate stopped working.** It compared the top score against a fixed number. e5 packs everything into 0.79–0.89, so any cutoff either admits all of it or none of it. The new gate compares the top hit against the *runner-up* instead. When there is genuinely nothing to recall, e5 finds several notes equally close and there is no clear winner — and that gap is the signal the raw score throws away. Right-note margins run 0.001–0.053; must-stay-silent margins run 0.000–0.019, four of the five at or under 0.002. **Why 0.008.** The full sweep is in the doc. It beats 0.010, 0.012 and 0.015 outright — same 1/5 false recall, 8 to 20 points more real recall. Going lower only buys recall by letting false answers back in. The next real gain is 0.020 at 0/5 false, which costs 24 points of recall. The old absolute cutoff stays in. On this fixture with e5 it is a no-op, but it still does real work for the hash embedder (which CI's ratchet depends on), and it is the only guard for a near-empty or stale-vector store where everything is far away but one row is marginally less far. It costs nothing measured. **Two caveats worth reading:** - **The last false recall cannot be tuned away.** "should i be offered wine" pulls a filler note with a 0.019 margin — wider than some real hits. No delta removes it without taking real answers with it. - **Five must-stay-silent cases is thin.** 1/5 versus 2/5 is a single case. The shape of the trade-off is trustworthy; the third decimal of 0.008 is not. The fixture wants more silent cases before anyone tunes this further. Both new settings are in `deploy/mavend.json` (`query_min_margin`), defaulted in code.

claude changed target branch from master to overnight/router-on

2026-07-31 10:39:23 +02:00

claude added 3 commits 2026-07-31 10:39:23 +02:00

Gate recall on the margin over the runner-up, not just the score 11831c6ace

The e5 embedder puts every cosine in one narrow band (0.79-0.89), so the
absolute query_min_score gate cannot tell a real hit from a made-up
question: any value under the band answers everything, any value above it
answers nothing. False recall was 5/5.

New gate asks whether one note is clearly the best instead: top1 - top2 >
delta. New query_min_margin config knob, default 0.008, read off the sweep
in the recall harness. The absolute floor stays as a second check.

On the recall fixture with e5: answered 72% -> 68%, false recall 5/5 -> 1/5.

Vikunja #359

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Write up the margin gate results 43838445ab

Third section: why the absolute gate could not separate the two
distributions, the delta sweep, and the before/after. Marks next-steps
item 3 done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Merge commit '4383844' into overnight-jul31 5a1d465db5

# Conflicts:
#	deploy/mavend.json
#	internal/config/config.go

kami commented

2026-07-31 18:51:21 +02:00

there might be an even better embeder out there but we didn't eval it yet. worth being aware of it.

kami commented

2026-07-31 20:22:01 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:02 +02:00

kami referenced this issue from a commit

2026-08-04 01:29:59 +02:00

store: wake the routines accepted before the fire-forever fix (V-377)

kami referenced this issue from a commit

2026-08-04 02:45:50 +02:00

store: add list_items, the fourth append-only shape (V-453)

kami referenced this issue from a commit

2026-08-04 16:12:36 +02:00

Merge task/476 into the entity-reference branch (V-524)