Record what Kiwix returns and why the gate is not one (V-668)
gofmt on cmd/mavwaked/silero.go came in with a99932b and blocked make test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
This commit is contained in:
@@ -0,0 +1,103 @@
|
||||
# Kiwix answered the wrong question, and the fix was not a relevance gate
|
||||
|
||||
Date: 2026-08-09. Task: V-668. Box: homesrv, workstation off.
|
||||
Book: `wikipedia_ru_all_maxi_2026-02` on `127.0.0.1:8034`.
|
||||
|
||||
## What started it
|
||||
|
||||
Two turns on 2026-08-09 came back wrong from the offline encyclopedia.
|
||||
"почему небо голубое" was answered off the song "Город золотой". "что такое
|
||||
TCP?" was answered off "Перехват TCP-соединения". Both were phrased
|
||||
confidently, because `queryKiwix` claims a turn whenever the search returns
|
||||
anything and `len(hits) == 0` is its only gate.
|
||||
|
||||
The plan was a relevance gate. multilingual-e5-small is asymmetric and trained
|
||||
for exactly this, `query:` against `passage:`, and the query vector is already
|
||||
held on the turn. The 2026-08-05 measurement that killed a search-quality gate
|
||||
killed three lexical signals. It says in its own words that it never probed
|
||||
Kiwix.
|
||||
|
||||
## The gate does not exist
|
||||
|
||||
Fourteen Russian questions, eight the encyclopedia can answer and six it
|
||||
cannot. Each question was searched, the top article read, and the cosine of
|
||||
`EmbedQuery(question)` against `EmbedPassage(article)` recorded.
|
||||
|
||||
| set | n | min | mean | max |
|
||||
|---|---|---|---|---|
|
||||
| answerable | 8 | 0.7934 | 0.8400 | 0.9087 |
|
||||
| not answerable | 6 | 0.7480 | 0.7852 | 0.8367 |
|
||||
|
||||
Two of the six unanswerable score above the weakest answerable one. That alone
|
||||
would be a poor threshold. The log killed it outright: seven of the eight
|
||||
answerable questions got a **wrong** article back, and those wrong articles
|
||||
scored high. The TCP hijacking article scored 0.8653, above five of the six
|
||||
unanswerable rows.
|
||||
|
||||
The finding is that this cosine measures topic and not answerhood. A page about
|
||||
hijacking TCP sessions is about TCP. No threshold separates it from a page that
|
||||
defines TCP, and one that tried would take the definition with it.
|
||||
|
||||
## The defect is retrieval
|
||||
|
||||
`internal/kiwix/client.go` has said it since it was written: ranking is keyword
|
||||
based, "why is the sky blue" finds a TV episode. `queryKiwix` sends the whole
|
||||
sentence. The English path has a rewriter that reduces a question to keywords
|
||||
with a model call. The Russian path reads the book verbatim (V-508) and had
|
||||
nothing. So the question words compete with the one word that names the article.
|
||||
|
||||
Dropping the question words changes the answer:
|
||||
|
||||
| sent | first hit |
|
||||
|---|---|
|
||||
| `кто написал Войну и мир` | Радуйся, мир (Доктор Кто) |
|
||||
| `Война и мир` | Война и мир |
|
||||
| `что такое TCP` | Перехват TCP-соединения |
|
||||
| `TCP` | TCP |
|
||||
|
||||
A ZIM is also addressable by title, which nothing here used. `/A/Франция`,
|
||||
`/A/TCP` and `/A/Небо` are 200. `/A/Трюмбальная_нидроскопия` is 404. So an
|
||||
exact title is safe to try first: it either answers or costs one request that
|
||||
says nothing.
|
||||
|
||||
## What shipped, measured
|
||||
|
||||
`kiwix.Topic` drops the narrative request, the interrogative and a verb sitting
|
||||
behind one. It keeps everything else, because a word it cannot classify is more
|
||||
likely the topic than noise. `kiwix.TitlePath` tries the exact article before
|
||||
any ranking runs. Both apply on the verbatim path only, since reducing twice
|
||||
would take the topic off the rewriter's input.
|
||||
|
||||
| question | before | after |
|
||||
|---|---|---|
|
||||
| что такое TCP? | Перехват TCP-соединения | **TCP** (by title) |
|
||||
| что такое фотосинтез | C4-фотосинтез | **Фотосинтез** |
|
||||
| кто такой Линус Торвальдс? | Tux | **Торвальдс, Линус** (by title) |
|
||||
| кто написал Войну и мир | Радуйся, мир (Доктор Кто) | **Война и мир** |
|
||||
| что такое чёрная дыра | Чёрная дыра | Чёрная дыра |
|
||||
| почему небо голубое | Город золотой | Под небом голубым… (фильм) |
|
||||
| почему трава зелёная | Сено | Зелень |
|
||||
| столица Франции | Список столиц Олимпийских игр | Список столиц Олимпийских игр |
|
||||
|
||||
Four questions reach the right article where they did not. Two were already
|
||||
right and stay right. Nothing regressed.
|
||||
|
||||
## What is still wrong
|
||||
|
||||
Two of the eight are still not answered, and both are the same shape. The
|
||||
question names no article. "почему небо голубое" is answered by Rayleigh
|
||||
scattering and "столица Франции" by the lead of Франция. Neither title is in
|
||||
the question. Keyword retrieval cannot bridge that and neither can a
|
||||
threshold. The candidates are a semantic index over titles, or asking the
|
||||
resident model for the article title rather than for keywords.
|
||||
|
||||
`Response.Empty()` is still the whole gate. A wrong article that the search
|
||||
does return is still spoken. What this change buys is that the article is
|
||||
usually right, not that a wrong one is caught.
|
||||
|
||||
## Not measured here
|
||||
|
||||
The English path, which still goes through the rewriter and was not touched.
|
||||
SearXNG, where the same question about answerhood is open and the 2026-08-05
|
||||
result stands. The cascade end to end, since the workstation is off and the
|
||||
phrasing arm is the resident model.
|
||||
Reference in New Issue
Block a user