kami
ddb658ffbb
Step one of letting Maven read instead of recall. No LLM yet. internal/kiwix/client.go: search a local Kiwix server, parse the RSS reply, hand back title + path + plain-text snippet + word count. The snippet is the unit of context; a full article is ~100KB of HTML and will not fit a 4096 token window. internal/kiwix/retrieval_eval.go plus knowledge_v1.json: the 9 knowledge questions from the phrasing fixture, each with hand-written English keywords, scored on whether a wanted article comes back in the top 5. Opt-in via MAVEN_KIWIX_URL, since CI has no Kiwix. No pass bar, the number is the finding. Result on the live mirror: 8/8 answerable questions hit, 7 of them at rank 1. Retrieval works. Keywords are written by hand on purpose, since Kiwix ranks by keyword and not by meaning, so a natural question fails. A query-rewrite step is the next piece of work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
64 lines
2.4 KiB
JSON
64 lines
2.4 KiB
JSON
{
|
|
"name": "kiwix-knowledge-v1",
|
|
"book": "wikipedia_en_all_maxi_2026-02",
|
|
"note": "The 9 knowledge cases from internal/phraser/eval/talk_v1.json. Queries are hand-written English keywords on purpose: Kiwix ranks by keyword, not meaning, so a natural question fails. Writing them by hand separates 'retrieval is broken' from 'the model writes bad queries'.",
|
|
"cases": [
|
|
{
|
|
"id": "know-sky-blue",
|
|
"question": "почему небо синее?",
|
|
"query": "Rayleigh scattering sky blue",
|
|
"want_titles": ["Rayleigh scattering", "Diffuse sky radiation"]
|
|
},
|
|
{
|
|
"id": "know-boil-egg",
|
|
"question": "сколько варить яйцо вкрутую?",
|
|
"query": "boiled egg cooking",
|
|
"want_titles": ["Boiled egg", "Egg as food"]
|
|
},
|
|
{
|
|
"id": "know-ssd-vs-hdd",
|
|
"question": "чем ssd отличается от hdd?",
|
|
"query": "solid-state drive",
|
|
"want_titles": ["Solid-state drive", "Hard disk drive"]
|
|
},
|
|
{
|
|
"id": "know-cat-purr",
|
|
"question": "почему кошки мурчат?",
|
|
"query": "cat purr",
|
|
"want_titles": ["Purr", "Cat communication"]
|
|
},
|
|
{
|
|
"id": "know-hiccups",
|
|
"question": "как быстро избавиться от икоты?",
|
|
"query": "hiccup",
|
|
"want_titles": ["Hiccup"]
|
|
},
|
|
{
|
|
"id": "know-polite-form",
|
|
"question": "не могли бы вы объяснить, что такое vpn?",
|
|
"query": "virtual private network",
|
|
"want_titles": ["Virtual private network"]
|
|
},
|
|
{
|
|
"id": "know-dont-know",
|
|
"question": "как зовут моего соседа снизу?",
|
|
"query": "name of my downstairs neighbour",
|
|
"want_titles": [],
|
|
"expect_miss": true,
|
|
"note": "Unanswerable by design. Retrieval SHOULD find nothing useful. Counted as a hit only when nothing relevant comes back."
|
|
},
|
|
{
|
|
"id": "know-water-per-day",
|
|
"question": "сколько воды в день надо пить?",
|
|
"query": "human daily water requirement drinking",
|
|
"want_titles": ["Drinking water", "Water", "Dehydration", "Hydration"]
|
|
},
|
|
{
|
|
"id": "know-thunder-delay",
|
|
"question": "почему гром слышно позже молнии?",
|
|
"query": "thunder speed of sound lightning",
|
|
"want_titles": ["Thunder", "Lightning"]
|
|
}
|
|
]
|
|
}
|