diff --git a/docs/evals/2026-08-03-recall-topic-veto.md b/docs/evals/2026-08-03-recall-topic-veto.md new file mode 100644 index 0000000..80f7c92 --- /dev/null +++ b/docs/evals/2026-08-03-recall-topic-veto.md @@ -0,0 +1,65 @@ +# Recall topic veto, what it costs and what it buys, 2026-08-03 + +Vikunja #496. The task asked for a cross-language fix. Skip the topic veto in +`memory.RecallAllowed` when the question and the hit are in different scripts. +An English question would then stop losing a Russian note. + +No such case exists. No fixture case puts the question and its wanted note in +different scripts. The case the task named is not one either. + + en-hard-024 + query "what fixed the screen problem" + note "the flicker went away once i swapped the display cable" + +Both are English. It is a paraphrase failure, not a language failure. A script +test would not have changed a single case, and neither would a bilingual stem +map. + +## What the veto is worth today + +Measured with the real embedder, multilingual-e5-small int8, gate 0.55, margin +0.008. The first row is the veto as it ships. The second is `RecallAllowed` +forced to true. + +| | cases passing | answered | false recall | silenced by gate | +|---|---|---|---|---| +| veto on | 22/32 | 17/27 | 0/5 | 2 | +| veto off | 22/32 | 18/27 | 1/5 | 1 | + +The pass count does not move. The veto trades one true recall for one false one. +It costs `en-hard-024` and it buys `ru-silent-029`: + + ru-silent-029 + query "во сколько отходит поезд" + note "погулял вдоль реки" 0.835, margin 0.019 + +The second case counted as silenced by the gate is `ru-home-026` at margin +0.001, which the margin gate stops. The veto has nothing to do with it. + +## Why no lexical rule separates the two + +`en-hard-024` and `ru-silent-029` are in the same lexical class. Both questions +share zero content words with their hit, and neither carries a first-person +marker. The scores sit on top of each other, 0.826 against 0.835, and so do the +margins, 0.023 against 0.019. Only one thing separates them. A screen problem +and a swapped display cable are the same event. A train and a river walk are +not. The embedder scores that difference at nine thousandths. + +So the signal is semantic and the gate is lexical. Any rule cheap enough to sit +in `RecallAllowed` and strong enough to recover `en-hard-024` also re-admits +`ru-silent-029`, which puts false recall back to 1/5. + +One near-miss rule was tried on paper and rejected: let the veto pass when the +hit itself is first person. It works on these two, because the English note says +"i swapped" and the Russian note says only "погулял". It is backwards as a +principle. A first-person note is exactly the personal note the veto keeps away +from a world question. The rule would weaken the veto where it was designed to +bite. It survives here only because Russian drops the pronoun. + +## Decision + +Accept the loss. `en-hard-024` stays silenced and false recall stays 0/5. + +The way out is a reranker, not a longer word list. Recall@3 is 85.2% against +recall@1 at 70.4%, so the right note is usually in the returned set and ranked +wrong. That is where the remaining points are, and it is not this task. diff --git a/internal/memory/lexical.go b/internal/memory/lexical.go index 2dc3161..fa47e8d 100644 --- a/internal/memory/lexical.go +++ b/internal/memory/lexical.go @@ -60,6 +60,14 @@ var firstPerson = map[string]bool{ // kill one false one. A question about his own life keeps the embedder alone // as its judge. A question about the world has to name something the memory // actually mentions. +// +// The veto's price was re-measured on 2026-08-03 (#496, +// docs/evals/2026-08-03-recall-topic-veto.md). It costs one true recall and +// buys one false one, and the fixture pass count is the same either way. The +// lost case is an English paraphrase, not the cross-language loss it was +// reported as, and the fixture has no cross-language case at all. Do not add a +// script test or a bilingual stem map for it — both are no-ops here. The +// separating signal is semantic and belongs in a reranker, not in this file. func RecallAllowed(query, text string) bool { if mentionsHim(query) { return true diff --git a/internal/memory/lexical_test.go b/internal/memory/lexical_test.go index a4a6315..c6f805c 100644 --- a/internal/memory/lexical_test.go +++ b/internal/memory/lexical_test.go @@ -31,6 +31,22 @@ func TestRecallAllowed(t *testing.T) { } } +// The known cost of the veto and the thing that pays for it, both measured on +// the held-out fixture with the real embedder (#496, +// docs/evals/2026-08-03-recall-topic-veto.md). The two are one lexical class: +// zero shared content words, no first-person marker, scores 0.826 against 0.835 +// and margins 0.023 against 0.019. Recovering the first re-admits the second, +// which puts false recall back to 1/5. Anyone loosening the veto has to move +// the first line without moving the second. +func TestRecallVetoTradeIsPinned(t *testing.T) { + if RecallAllowed("what fixed the screen problem", "the flicker went away once i swapped the display cable") { + t.Error("en-hard-024 is expected to stay vetoed — if this passes now, re-measure false recall before celebrating") + } + if RecallAllowed("во сколько отходит поезд", "погулял вдоль реки") { + t.Error("ru-silent-029 must stay vetoed — this is the false recall the veto exists to stop") + } +} + // A question made only of filler has no topic word to match on, and the score // gate is then the only judge it can have. func TestRecallAllowedFallsBackWhenNothingToCompare(t *testing.T) {