claude
9e25f18a3e
#496 asked to skip the veto when the question and the hit are in different scripts, so an English question stops losing a Russian note. Measured first: the fixture has no cross-language case, and en-hard-024 is an English question against an English note. Both proposed fixes are no-ops. What the veto actually does on the fixture, with the real embedder: it costs en-hard-024 and buys ru-silent-029. Pass count is 22/32 either way; false recall is 0/5 with it and 1/5 without. The two cases are one lexical class, so no rule cheap enough for RecallAllowed separates them. Accepts the loss and pins both sides in a test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
66 lines
2.9 KiB
Markdown
66 lines
2.9 KiB
Markdown
# Recall topic veto, what it costs and what it buys, 2026-08-03
|
|
|
|
Vikunja #496. The task asked for a cross-language fix. Skip the topic veto in
|
|
`memory.RecallAllowed` when the question and the hit are in different scripts.
|
|
An English question would then stop losing a Russian note.
|
|
|
|
No such case exists. No fixture case puts the question and its wanted note in
|
|
different scripts. The case the task named is not one either.
|
|
|
|
en-hard-024
|
|
query "what fixed the screen problem"
|
|
note "the flicker went away once i swapped the display cable"
|
|
|
|
Both are English. It is a paraphrase failure, not a language failure. A script
|
|
test would not have changed a single case, and neither would a bilingual stem
|
|
map.
|
|
|
|
## What the veto is worth today
|
|
|
|
Measured with the real embedder, multilingual-e5-small int8, gate 0.55, margin
|
|
0.008. The first row is the veto as it ships. The second is `RecallAllowed`
|
|
forced to true.
|
|
|
|
| | cases passing | answered | false recall | silenced by gate |
|
|
|---|---|---|---|---|
|
|
| veto on | 22/32 | 17/27 | 0/5 | 2 |
|
|
| veto off | 22/32 | 18/27 | 1/5 | 1 |
|
|
|
|
The pass count does not move. The veto trades one true recall for one false one.
|
|
It costs `en-hard-024` and it buys `ru-silent-029`:
|
|
|
|
ru-silent-029
|
|
query "во сколько отходит поезд"
|
|
note "погулял вдоль реки" 0.835, margin 0.019
|
|
|
|
The second case counted as silenced by the gate is `ru-home-026` at margin
|
|
0.001, which the margin gate stops. The veto has nothing to do with it.
|
|
|
|
## Why no lexical rule separates the two
|
|
|
|
`en-hard-024` and `ru-silent-029` are in the same lexical class. Both questions
|
|
share zero content words with their hit, and neither carries a first-person
|
|
marker. The scores sit on top of each other, 0.826 against 0.835, and so do the
|
|
margins, 0.023 against 0.019. Only one thing separates them. A screen problem
|
|
and a swapped display cable are the same event. A train and a river walk are
|
|
not. The embedder scores that difference at nine thousandths.
|
|
|
|
So the signal is semantic and the gate is lexical. Any rule cheap enough to sit
|
|
in `RecallAllowed` and strong enough to recover `en-hard-024` also re-admits
|
|
`ru-silent-029`, which puts false recall back to 1/5.
|
|
|
|
One near-miss rule was tried on paper and rejected: let the veto pass when the
|
|
hit itself is first person. It works on these two, because the English note says
|
|
"i swapped" and the Russian note says only "погулял". It is backwards as a
|
|
principle. A first-person note is exactly the personal note the veto keeps away
|
|
from a world question. The rule would weaken the veto where it was designed to
|
|
bite. It survives here only because Russian drops the pronoun.
|
|
|
|
## Decision
|
|
|
|
Accept the loss. `en-hard-024` stays silenced and false recall stays 0/5.
|
|
|
|
The way out is a reranker, not a longer word list. Recall@3 is 85.2% against
|
|
recall@1 at 70.4%, so the right note is usually in the returned set and ranked
|
|
wrong. That is where the remaining points are, and it is not this task.
|