Rewrite Russian questions into Kiwix keywords — and the score that says it does not work yet (#403) #44

2026-07-31T16:24:36+02:00

claude commented

2026-07-31 16:24:36 +02:00

Kiwix search is keyword-based, not semantic. Asking it a Russian question
returns nothing useful, so this adds a step that turns the question into English
search keywords before the lookup, and a scorer that measures whether it helped.

It mostly does not. 3 of 8, three runs out of three. Temperature is 0 and the
result was identical case for case every time, so that is the model's ceiling on
this prompt, not sampling noise. Hand-written English keywords score 8 of 8 on
the same fixture, so the retrieval is fine — the rewrite is the weak part.

The failure is specific and worth knowing: the model translates the question,
it does not name the topic. For "почему небо синее" it wrote sky blue, which
finds the band Blue Sky and the 2020 Sky Blue FC season. It never once produced a
term the question did not already contain. Naming 'Rayleigh scattering' needs the
physics knowledge we were trying to avoid needing, so the approach is circular as
built.

Merging it anyway: the client, the grammar and the scorer are all reusable, and
the number is the point. Two untried paths that break the circularity are noted
in the issue — letting the model pick from the search results it gets back
('Rayleigh scattering' is on the first page of hits for sky blue), and
reranking those hits with the ONNX embedder we already ship.

Two commits: the rewriter, then the end-to-end score.

Vikunja #403.

Kiwix search is keyword-based, not semantic. Asking it a Russian question returns nothing useful, so this adds a step that turns the question into English search keywords before the lookup, and a scorer that measures whether it helped. **It mostly does not. 3 of 8, three runs out of three.** Temperature is 0 and the result was identical case for case every time, so that is the model's ceiling on this prompt, not sampling noise. Hand-written English keywords score 8 of 8 on the same fixture, so the retrieval is fine — the rewrite is the weak part. The failure is specific and worth knowing: **the model translates the question, it does not name the topic.** For "почему небо синее" it wrote `sky blue`, which finds the band Blue Sky and the 2020 Sky Blue FC season. It never once produced a term the question did not already contain. Naming \'Rayleigh scattering\' needs the physics knowledge we were trying to avoid needing, so the approach is circular as built. Merging it anyway: the client, the grammar and the scorer are all reusable, and the number is the point. Two untried paths that break the circularity are noted in the issue — letting the model pick from the search results it gets back (\'Rayleigh scattering\' is on the first page of hits for `sky blue`), and reranking those hits with the ONNX embedder we already ship. Two commits: the rewriter, then the end-to-end score. Vikunja #403.

claude added 2 commits 2026-07-31 16:24:37 +02:00

Rewrite a Russian question into English Kiwix keywords (#403 ) b300ac5c70

Kiwix ranks by keyword, not meaning, so a translated question finds song
and TV titles. This asks the resident model for the TOPIC instead: a short
English noun phrase, like a Wikipedia article title.

Locked down three ways, because a wrong query is silently wrong:
- A GBNF grammar, same idea as routeGrammar and responseGrammar. The
  reply must be {"query":"..."} with Latin words only. The JSON wrapper
  matters: this model always thinks out loud and this llama-server build
  ignores the thinking switch, so a bare word-list grammar just captured
  "Let me analyze this request carefully" for every question.
- max_tokens 32, since the answer is a few words.
- CleanQuery, which throws away empty, Russian and prose replies rather
  than passing them to Kiwix, and drops question words like "why" and
  "how much" that a keyword ranker cannot use anyway.

Client side only. Nothing is wired into the daemon or any config.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Score the Russian-to-keywords rewrite end to end (#403 ) 742b2ad1d7

Same 9 cases as the retrieval eval, so the numbers compare directly:
hand-written keywords hit 8 of 8, this is what the model reaches on its
own. Reports the hand-written query next to the model's for every case,
because where the phrasing differs is the useful part.

Opt-in on MAVEN_KIWIX_URL + MAVEN_LLM_URL, like the other evals.

Result on Qwen3.5-0.8B: 3 of 8, identical on all three runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

kami referenced this issue from a commit

2026-07-31 19:37:10 +02:00

gofmt the kiwix client and rewrite test

kami commented

2026-07-31 20:22:43 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:44 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#44