claude
9e25f18a3e
#496 asked to skip the veto when the question and the hit are in different scripts, so an English question stops losing a Russian note. Measured first: the fixture has no cross-language case, and en-hard-024 is an English question against an English note. Both proposed fixes are no-ops. What the veto actually does on the fixture, with the real embedder: it costs en-hard-024 and buys ru-silent-029. Pass count is 22/32 either way; false recall is 0/5 with it and 1/5 without. The two cases are one lexical class, so no rule cheap enough for RecallAllowed separates them. Accepts the loss and pins both sides in a test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2.9 KiB
Recall topic veto, what it costs and what it buys, 2026-08-03
Vikunja #496. The task asked for a cross-language fix. Skip the topic veto in
memory.RecallAllowed when the question and the hit are in different scripts.
An English question would then stop losing a Russian note.
No such case exists. No fixture case puts the question and its wanted note in different scripts. The case the task named is not one either.
en-hard-024
query "what fixed the screen problem"
note "the flicker went away once i swapped the display cable"
Both are English. It is a paraphrase failure, not a language failure. A script test would not have changed a single case, and neither would a bilingual stem map.
What the veto is worth today
Measured with the real embedder, multilingual-e5-small int8, gate 0.55, margin
0.008. The first row is the veto as it ships. The second is RecallAllowed
forced to true.
| cases passing | answered | false recall | silenced by gate | |
|---|---|---|---|---|
| veto on | 22/32 | 17/27 | 0/5 | 2 |
| veto off | 22/32 | 18/27 | 1/5 | 1 |
The pass count does not move. The veto trades one true recall for one false one.
It costs en-hard-024 and it buys ru-silent-029:
ru-silent-029
query "во сколько отходит поезд"
note "погулял вдоль реки" 0.835, margin 0.019
The second case counted as silenced by the gate is ru-home-026 at margin
0.001, which the margin gate stops. The veto has nothing to do with it.
Why no lexical rule separates the two
en-hard-024 and ru-silent-029 are in the same lexical class. Both questions
share zero content words with their hit, and neither carries a first-person
marker. The scores sit on top of each other, 0.826 against 0.835, and so do the
margins, 0.023 against 0.019. Only one thing separates them. A screen problem
and a swapped display cable are the same event. A train and a river walk are
not. The embedder scores that difference at nine thousandths.
So the signal is semantic and the gate is lexical. Any rule cheap enough to sit
in RecallAllowed and strong enough to recover en-hard-024 also re-admits
ru-silent-029, which puts false recall back to 1/5.
One near-miss rule was tried on paper and rejected: let the veto pass when the hit itself is first person. It works on these two, because the English note says "i swapped" and the Russian note says only "погулял". It is backwards as a principle. A first-person note is exactly the personal note the veto keeps away from a world question. The rule would weaken the veto where it was designed to bite. It survives here only because Russian drops the pronoun.
Decision
Accept the loss. en-hard-024 stays silenced and false recall stays 0/5.
The way out is a reranker, not a longer word list. Recall@3 is 85.2% against recall@1 at 70.4%, so the right note is usually in the returned set and ranked wrong. That is where the remaining points are, and it is not this task.