Add a Kiwix search client and score retrieval (Vikunja #403)
Step one of letting Maven read instead of recall. No LLM yet. internal/kiwix/client.go: search a local Kiwix server, parse the RSS reply, hand back title + path + plain-text snippet + word count. The snippet is the unit of context; a full article is ~100KB of HTML and will not fit a 4096 token window. internal/kiwix/retrieval_eval.go plus knowledge_v1.json: the 9 knowledge questions from the phrasing fixture, each with hand-written English keywords, scored on whether a wanted article comes back in the top 5. Opt-in via MAVEN_KIWIX_URL, since CI has no Kiwix. No pass bar, the number is the finding. Result on the live mirror: 8/8 answerable questions hit, 7 of them at rank 1. Retrieval works. Keywords are written by hand on purpose, since Kiwix ranks by keyword and not by meaning, so a natural question fails. A query-rewrite step is the next piece of work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
{
|
||||
"name": "kiwix-knowledge-v1",
|
||||
"book": "wikipedia_en_all_maxi_2026-02",
|
||||
"note": "The 9 knowledge cases from internal/phraser/eval/talk_v1.json. Queries are hand-written English keywords on purpose: Kiwix ranks by keyword, not meaning, so a natural question fails. Writing them by hand separates 'retrieval is broken' from 'the model writes bad queries'.",
|
||||
"cases": [
|
||||
{
|
||||
"id": "know-sky-blue",
|
||||
"question": "почему небо синее?",
|
||||
"query": "Rayleigh scattering sky blue",
|
||||
"want_titles": ["Rayleigh scattering", "Diffuse sky radiation"]
|
||||
},
|
||||
{
|
||||
"id": "know-boil-egg",
|
||||
"question": "сколько варить яйцо вкрутую?",
|
||||
"query": "boiled egg cooking",
|
||||
"want_titles": ["Boiled egg", "Egg as food"]
|
||||
},
|
||||
{
|
||||
"id": "know-ssd-vs-hdd",
|
||||
"question": "чем ssd отличается от hdd?",
|
||||
"query": "solid-state drive",
|
||||
"want_titles": ["Solid-state drive", "Hard disk drive"]
|
||||
},
|
||||
{
|
||||
"id": "know-cat-purr",
|
||||
"question": "почему кошки мурчат?",
|
||||
"query": "cat purr",
|
||||
"want_titles": ["Purr", "Cat communication"]
|
||||
},
|
||||
{
|
||||
"id": "know-hiccups",
|
||||
"question": "как быстро избавиться от икоты?",
|
||||
"query": "hiccup",
|
||||
"want_titles": ["Hiccup"]
|
||||
},
|
||||
{
|
||||
"id": "know-polite-form",
|
||||
"question": "не могли бы вы объяснить, что такое vpn?",
|
||||
"query": "virtual private network",
|
||||
"want_titles": ["Virtual private network"]
|
||||
},
|
||||
{
|
||||
"id": "know-dont-know",
|
||||
"question": "как зовут моего соседа снизу?",
|
||||
"query": "name of my downstairs neighbour",
|
||||
"want_titles": [],
|
||||
"expect_miss": true,
|
||||
"note": "Unanswerable by design. Retrieval SHOULD find nothing useful. Counted as a hit only when nothing relevant comes back."
|
||||
},
|
||||
{
|
||||
"id": "know-water-per-day",
|
||||
"question": "сколько воды в день надо пить?",
|
||||
"query": "human daily water requirement drinking",
|
||||
"want_titles": ["Drinking water", "Water", "Dehydration", "Hydration"]
|
||||
},
|
||||
{
|
||||
"id": "know-thunder-delay",
|
||||
"question": "почему гром слышно позже молнии?",
|
||||
"query": "thunder speed of sound lightning",
|
||||
"want_titles": ["Thunder", "Lightning"]
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user