Maven/docs/evals/2026-08-09-kiwix-topic-retrieval.md

Kiwix answered the wrong question, and the fix was not a relevance gate

Date: 2026-08-09. Task: V-668. Box: homesrv, workstation off. Book: wikipedia_ru_all_maxi_2026-02 on 127.0.0.1:8034.

What started it

Two turns on 2026-08-09 came back wrong from the offline encyclopedia. "почему небо голубое" was answered off the song "Город золотой". "что такое TCP?" was answered off "Перехват TCP-соединения". Both were phrased confidently, because queryKiwix claims a turn whenever the search returns anything and len(hits) == 0 is its only gate.

The plan was a relevance gate. multilingual-e5-small is asymmetric and trained for exactly this, query: against passage:, and the query vector is already held on the turn. The 2026-08-05 measurement that killed a search-quality gate killed three lexical signals. It says in its own words that it never probed Kiwix.

The gate does not exist

Fourteen Russian questions, eight the encyclopedia can answer and six it cannot. Each question was searched, the top article read, and the cosine of EmbedQuery(question) against EmbedPassage(article) recorded.

set	n	min	mean	max
answerable	8	0.7934	0.8400	0.9087
not answerable	6	0.7480	0.7852	0.8367

Two of the six unanswerable score above the weakest answerable one. That alone would be a poor threshold. The log killed it outright: seven of the eight answerable questions got a wrong article back, and those wrong articles scored high. The TCP hijacking article scored 0.8653, above five of the six unanswerable rows.

The finding is that this cosine measures topic and not answerhood. A page about hijacking TCP sessions is about TCP. No threshold separates it from a page that defines TCP, and one that tried would take the definition with it.

The defect is retrieval

internal/kiwix/client.go has said it since it was written: ranking is keyword based, "why is the sky blue" finds a TV episode. queryKiwix sends the whole sentence. The English path has a rewriter that reduces a question to keywords with a model call. The Russian path reads the book verbatim (V-508) and had nothing. So the question words compete with the one word that names the article.

Dropping the question words changes the answer:

sent	first hit
кто написал Войну и мир	Радуйся, мир (Доктор Кто)
Война и мир	Война и мир
что такое TCP	Перехват TCP-соединения
TCP	TCP

A ZIM is also addressable by title, which nothing here used. /A/Франция, /A/TCP and /A/Небо are 200. /A/Трюмбальная_нидроскопия is 404. So an exact title is safe to try first: it either answers or costs one request that says nothing.

What shipped, measured

kiwix.Topic drops the narrative request, the interrogative and a verb sitting behind one. It keeps everything else, because a word it cannot classify is more likely the topic than noise. kiwix.TitlePath tries the exact article before any ranking runs. Both apply on the verbatim path only, since reducing twice would take the topic off the rewriter's input.

question	before	after
что такое TCP?	Перехват TCP-соединения	TCP (by title)
что такое фотосинтез	C4-фотосинтез	Фотосинтез
кто такой Линус Торвальдс?	Tux	Торвальдс, Линус (by title)
кто написал Войну и мир	Радуйся, мир (Доктор Кто)	Война и мир
что такое чёрная дыра	Чёрная дыра	Чёрная дыра
почему небо голубое	Город золотой	Под небом голубым… (фильм)
почему трава зелёная	Сено	Зелень
столица Франции	Список столиц Олимпийских игр	Список столиц Олимпийских игр

Four questions reach the right article where they did not. Two were already right and stay right. Nothing regressed.

What is still wrong

Two of the eight are still not answered, and both are the same shape. The question names no article. "почему небо голубое" is answered by Rayleigh scattering and "столица Франции" by the lead of Франция. Neither title is in the question. Keyword retrieval cannot bridge that and neither can a threshold. The candidates are a semantic index over titles, or asking the resident model for the article title rather than for keywords.

Response.Empty() is still the whole gate. A wrong article that the search does return is still spoken. What this change buys is that the article is usually right, not that a wrong one is caught.

Not measured here

The English path, which still goes through the rewriter and was not touched. SearXNG, where the same question about answerhood is open and the 2026-08-05 result stands. The cascade end to end, since the workstation is off and the phrasing arm is the resident model.