Add a Kiwix client and measure whether retrieval finds the right article #41

2026-07-31T15:49:27+02:00

claude commented

2026-07-31 15:49:27 +02:00

First step of #403. She makes facts up on general-knowledge questions because a
0.8B does not know them, so the plan is to let her read the local Wikipedia
mirror instead of recall. This commit is the client plus a harness that measures
whether retrieval works at all. No LLM, nothing wired into the daemon.

internal/kiwix/client.go searches a ZIM and returns title, path, snippet and
word count. It parses the RSS that kiwix-serve returns, strips the <b> match
highlighting out of the snippet and unescapes entities, so the snippet is plain
text ready to hand to a model.

The harness scores the nine general-knowledge questions from the talk fixture,
using hand-written English keywords. Writing the keywords by hand is the point:
it separates "retrieval is broken" from "the model writes bad queries".

Result: 8 of 8 answerable questions find a wanted article in the top 5, seven of
them at rank 1. All nine queries take 2.2s total. Retrieval is not the problem.

Three things the run showed that shape the next commit:

Kiwix never returns an empty list. The deliberately unanswerable question
("как зовут моего соседа снизу") came back with an awards ceremony and a
novel. Junk, correctly, but present. So a "did we find anything" gate cannot
count results — it needs a relevance threshold or the model declining.
Rank 1 is right, ranks 2-5 are noise. cat purr gives Purr, then a cartoon,
a musical and a mascot. Handing all five snippets to a 0.8B feeds it mostly
garbage. Top 1-2 is the cut.
Health and quantity questions retrieve worst — the water question only hit at
rank 5, behind two tangential articles. That is the kind of question he
actually asks.

Two things left out on purpose: no article fetch (the Rayleigh scattering page
is 98KB of HTML, useless at 4096 context, and the search snippet is ~500
readable characters), and Search takes the ZIM name from the caller with no
default, because nothing is wired up yet.

The whole approach now rests on one unmeasured step: can the 0.8B turn "почему
небо синее" into "Rayleigh scattering sky blue"? That needs its own scored
fixture against these same nine cases, with 8/8 from the Russian input as the
target.

Note for running it: kiwix-server had been stopped for 11 days, I started it.
Every ZIM on the box is English while she answers in Russian.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

First step of #403. She makes facts up on general-knowledge questions because a 0.8B does not know them, so the plan is to let her read the local Wikipedia mirror instead of recall. This commit is the client plus a harness that measures whether retrieval works at all. No LLM, nothing wired into the daemon. `internal/kiwix/client.go` searches a ZIM and returns title, path, snippet and word count. It parses the RSS that kiwix-serve returns, strips the `<b>` match highlighting out of the snippet and unescapes entities, so the snippet is plain text ready to hand to a model. The harness scores the nine general-knowledge questions from the talk fixture, using hand-written English keywords. Writing the keywords by hand is the point: it separates "retrieval is broken" from "the model writes bad queries". Result: 8 of 8 answerable questions find a wanted article in the top 5, seven of them at rank 1. All nine queries take 2.2s total. Retrieval is not the problem. Three things the run showed that shape the next commit: - Kiwix never returns an empty list. The deliberately unanswerable question ("как зовут моего соседа снизу") came back with an awards ceremony and a novel. Junk, correctly, but present. So a "did we find anything" gate cannot count results — it needs a relevance threshold or the model declining. - Rank 1 is right, ranks 2-5 are noise. `cat purr` gives `Purr`, then a cartoon, a musical and a mascot. Handing all five snippets to a 0.8B feeds it mostly garbage. Top 1-2 is the cut. - Health and quantity questions retrieve worst — the water question only hit at rank 5, behind two tangential articles. That is the kind of question he actually asks. Two things left out on purpose: no article fetch (the Rayleigh scattering page is 98KB of HTML, useless at 4096 context, and the search snippet is ~500 readable characters), and `Search` takes the ZIM name from the caller with no default, because nothing is wired up yet. The whole approach now rests on one unmeasured step: can the 0.8B turn "почему небо синее" into "Rayleigh scattering sky blue"? That needs its own scored fixture against these same nine cases, with 8/8 from the Russian input as the target. Note for running it: kiwix-server had been stopped for 11 days, I started it. Every ZIM on the box is English while she answers in Russian. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

claude added 1 commit 2026-07-31 15:49:27 +02:00

Add a Kiwix search client and score retrieval (Vikunja #403 ) ddb658ffbb

Step one of letting Maven read instead of recall. No LLM yet.

internal/kiwix/client.go: search a local Kiwix server, parse the RSS
reply, hand back title + path + plain-text snippet + word count. The
snippet is the unit of context; a full article is ~100KB of HTML and
will not fit a 4096 token window.

internal/kiwix/retrieval_eval.go plus knowledge_v1.json: the 9 knowledge
questions from the phrasing fixture, each with hand-written English
keywords, scored on whether a wanted article comes back in the top 5.
Opt-in via MAVEN_KIWIX_URL, since CI has no Kiwix. No pass bar, the
number is the finding.

Result on the live mirror: 8/8 answerable questions hit, 7 of them at
rank 1. Retrieval works. Keywords are written by hand on purpose, since
Kiwix ranks by keyword and not by meaning, so a natural question fails.
A query-rewrite step is the next piece of work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

kami referenced this issue from a commit

2026-07-31 19:37:10 +02:00

gofmt the kiwix client and rewrite test

kami commented

2026-07-31 20:22:38 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:40 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#41