From 999a5ad562f681e508287ffe5ee6448f4765798b Mon Sep 17 00:00:00 2001 From: claude Date: Sun, 9 Aug 2026 10:46:15 +0400 Subject: [PATCH] Record what Kiwix returns and why the gate is not one (V-668) gofmt on cmd/mavwaked/silero.go came in with a99932b and blocked make test. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN --- CLAUDE.md | 16 ++- cmd/mavwaked/silero.go | 8 +- .../evals/2026-08-09-kiwix-topic-retrieval.md | 103 ++++++++++++++++++ 3 files changed, 122 insertions(+), 5 deletions(-) create mode 100644 docs/evals/2026-08-09-kiwix-topic-retrieval.md diff --git a/CLAUDE.md b/CLAUDE.md index e4806f9..502639b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -675,11 +675,25 @@ world questions, so she needs to read external sources. What replaces it: `wikipedia_ru_all_maxi_2026-02` verbatim** through `kiwix.book_ru`. The RU→EN rewriter is the workaround for an English book and is skipped there. Kiwix catalog names come from the filename, not the `` field. + **That verbatim path sent the whole sentence to a keyword engine until 09-08-2026** + (V-668, `docs/evals/2026-08-09-kiwix-topic-retrieval.md`). Kiwix ranks by keyword + overlap, so the question words outrank the one word naming the article. "что такое TCP" + returned "Перехват TCP-соединения". "кто написал Войну и мир" returned an episode of + Doctor Who. `kiwix.Topic` drops the narrative request, the interrogative and a verb + behind one. `kiwix.TitlePath` tries the exact article first, since a ZIM is addressable + by title and a wrong title is a 404. Four of eight questions reach the right article + where they did not, two were already right, and nothing regressed. Both apply on the + verbatim path alone. The rewriter already reduces a question, and reducing twice takes + the topic off its input. `Response.Empty()` is the whole gate and there is no quality threshold in front of it: the three signals one could read were measured on 2026-08-05 and none of them separate a real question from an invented one. Token overlap would cost "столица Франции" its answer, because the answer is Париж and that word is not in the question. See - `docs/evals/2026-08-05-search-quality-signals.md` (V-539). **Which query source claimed + `docs/evals/2026-08-05-search-quality-signals.md` (V-539). **The embedder is not a + fourth signal**, measured 2026-08-09 (V-668). Query-to-passage cosine scores 0.79 to + 0.91 on answerable questions and 0.75 to 0.84 on unanswerable ones, and the sets + overlap. The wrong TCP article scored 0.8653, above five of six unanswerable rows. It + measures topic and not whether the passage answers, so no threshold splits them. **Which query source claimed a turn is readable on `/chat`** as a badge beside the reply, carried on `ipc.ChatReply.Source` and noted by `noteQuerySource` in `cmd/mavend/querysource.go`. It rides the context, so `handleText` keeps the one string signature the mic, telegram and diff --git a/cmd/mavwaked/silero.go b/cmd/mavwaked/silero.go index 6c015fb..ed8fa0c 100644 --- a/cmd/mavwaked/silero.go +++ b/cmd/mavwaked/silero.go @@ -52,10 +52,10 @@ type sileroVAD struct { mu sync.Mutex session *ort.DynamicAdvancedSession - pending []float32 // samples not yet part of a full window - context [sileroContext]float32 // tail of the previous window - state []float32 // [2][1][128], carried between windows - last float64 // most recent probability, held between windows + pending []float32 // samples not yet part of a full window + context [sileroContext]float32 // tail of the previous window + state []float32 // [2][1][128], carried between windows + last float64 // most recent probability, held between windows sr []int64 } diff --git a/docs/evals/2026-08-09-kiwix-topic-retrieval.md b/docs/evals/2026-08-09-kiwix-topic-retrieval.md new file mode 100644 index 0000000..115605d --- /dev/null +++ b/docs/evals/2026-08-09-kiwix-topic-retrieval.md @@ -0,0 +1,103 @@ +# Kiwix answered the wrong question, and the fix was not a relevance gate + +Date: 2026-08-09. Task: V-668. Box: homesrv, workstation off. +Book: `wikipedia_ru_all_maxi_2026-02` on `127.0.0.1:8034`. + +## What started it + +Two turns on 2026-08-09 came back wrong from the offline encyclopedia. +"почему небо голубое" was answered off the song "Город золотой". "что такое +TCP?" was answered off "Перехват TCP-соединения". Both were phrased +confidently, because `queryKiwix` claims a turn whenever the search returns +anything and `len(hits) == 0` is its only gate. + +The plan was a relevance gate. multilingual-e5-small is asymmetric and trained +for exactly this, `query:` against `passage:`, and the query vector is already +held on the turn. The 2026-08-05 measurement that killed a search-quality gate +killed three lexical signals. It says in its own words that it never probed +Kiwix. + +## The gate does not exist + +Fourteen Russian questions, eight the encyclopedia can answer and six it +cannot. Each question was searched, the top article read, and the cosine of +`EmbedQuery(question)` against `EmbedPassage(article)` recorded. + +| set | n | min | mean | max | +|---|---|---|---|---| +| answerable | 8 | 0.7934 | 0.8400 | 0.9087 | +| not answerable | 6 | 0.7480 | 0.7852 | 0.8367 | + +Two of the six unanswerable score above the weakest answerable one. That alone +would be a poor threshold. The log killed it outright: seven of the eight +answerable questions got a **wrong** article back, and those wrong articles +scored high. The TCP hijacking article scored 0.8653, above five of the six +unanswerable rows. + +The finding is that this cosine measures topic and not answerhood. A page about +hijacking TCP sessions is about TCP. No threshold separates it from a page that +defines TCP, and one that tried would take the definition with it. + +## The defect is retrieval + +`internal/kiwix/client.go` has said it since it was written: ranking is keyword +based, "why is the sky blue" finds a TV episode. `queryKiwix` sends the whole +sentence. The English path has a rewriter that reduces a question to keywords +with a model call. The Russian path reads the book verbatim (V-508) and had +nothing. So the question words compete with the one word that names the article. + +Dropping the question words changes the answer: + +| sent | first hit | +|---|---| +| `кто написал Войну и мир` | Радуйся, мир (Доктор Кто) | +| `Война и мир` | Война и мир | +| `что такое TCP` | Перехват TCP-соединения | +| `TCP` | TCP | + +A ZIM is also addressable by title, which nothing here used. `/A/Франция`, +`/A/TCP` and `/A/Небо` are 200. `/A/Трюмбальная_нидроскопия` is 404. So an +exact title is safe to try first: it either answers or costs one request that +says nothing. + +## What shipped, measured + +`kiwix.Topic` drops the narrative request, the interrogative and a verb sitting +behind one. It keeps everything else, because a word it cannot classify is more +likely the topic than noise. `kiwix.TitlePath` tries the exact article before +any ranking runs. Both apply on the verbatim path only, since reducing twice +would take the topic off the rewriter's input. + +| question | before | after | +|---|---|---| +| что такое TCP? | Перехват TCP-соединения | **TCP** (by title) | +| что такое фотосинтез | C4-фотосинтез | **Фотосинтез** | +| кто такой Линус Торвальдс? | Tux | **Торвальдс, Линус** (by title) | +| кто написал Войну и мир | Радуйся, мир (Доктор Кто) | **Война и мир** | +| что такое чёрная дыра | Чёрная дыра | Чёрная дыра | +| почему небо голубое | Город золотой | Под небом голубым… (фильм) | +| почему трава зелёная | Сено | Зелень | +| столица Франции | Список столиц Олимпийских игр | Список столиц Олимпийских игр | + +Four questions reach the right article where they did not. Two were already +right and stay right. Nothing regressed. + +## What is still wrong + +Two of the eight are still not answered, and both are the same shape. The +question names no article. "почему небо голубое" is answered by Rayleigh +scattering and "столица Франции" by the lead of Франция. Neither title is in +the question. Keyword retrieval cannot bridge that and neither can a +threshold. The candidates are a semantic index over titles, or asking the +resident model for the article title rather than for keywords. + +`Response.Empty()` is still the whole gate. A wrong article that the search +does return is still spoken. What this change buys is that the article is +usually right, not that a wrong one is caught. + +## Not measured here + +The English path, which still goes through the rewriter and was not touched. +SearXNG, where the same question about answerhood is open and the 2026-08-05 +result stands. The cascade end to end, since the workstation is off and the +phrasing arm is the resident model.