kiwix: read the article, not the snippet, and state the persona as morphology

Two things that were each half-done.

The prompts handed the model copyable examples. chatSystemPrompt lost its
openers this morning and the "Я подумала, что" tic went with them, but "не
забыл ли я" appeared in its place: the removed example had been suppressing
the masculine self-reference by accident. Two predicatives are not enough
signal, so the rule is now stated as morphology (-ла) rather than as a pair of
words — a suffix rule generalises where an example only gets copied.
querySystemPrompt had the same defect and gets the same treatment; its "вот что
я нашла: " opener is deliberate and stays.

internal/kiwix had no caller. actions_query.go said "once internal/kiwix is
wired into this chain" and that never happened. It is wired now, between the
notes pass and the web source: everything of his answers first, and only what
is left over is looked up. Off unless a `kiwix` block names a server and a book.

Reading the search snippet does not work. Kiwix builds it from wherever the
keyword matched, which on Wikipedia is the navigation box at the foot of the
page — the first version of this answered "что такое фотосинтез?" by reciting
"Ecological economics Ecological footprint Ecological forecasting …". Client
grows an Article method; the head of the article is the lead paragraph, which
is the definition the snippet was meant to be. Verified on the box: the same
question now answers correctly off the ZIM.

Only the rewritten query leaves the process. A test asserts it: a turn carrying
a stored note must not put that note in the search string.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX

internal/config/config.go

@@ -199,6 +199,10 @@ type Config struct {
 	// fetches a page: not on request, not on a schedule. See CrawlConfig.
 	Crawl *CrawlConfig `json:"crawl,omitempty"`
 
+	// Kiwix — the offline ZIM reader (Vikunja #122 neighbourhood). nil / absent
+	// / url empty ⇒ the query chain has no ZIM source. See KiwixConfig.
+	Kiwix *KiwixConfig `json:"kiwix,omitempty"`
+
 	// Praxis — the ecosystem attention-state service. When configured, maven
 	// calls the Praxis HTTP tools API for attention listing and item lifecycle.
 	// Maven never touches Praxis's database directly (ecosystem invariant: no
@@ -1027,6 +1031,59 @@ type CrawlConfig struct {
 	MaxRunes int `json:"max_runes,omitempty"`
 }
 
+// KiwixConfig — the offline encyclopedia. A kiwix-serve instance holding ZIM
+// archives (Wikipedia, ifixit, devdocs) on the LAN, searched before anything
+// touches the network. Dark until configured, same as every other reach.
+//
+// This is the "local sources first" rule in CLAUDE.md made concrete: a 1.7B
+// does not know enough to answer a world question, but it can read. A local
+// read costs nothing and leaves the box only as far as the LAN.
+//
+// Only the rewritten search query leaves this process. His notes, facts,
+// persona block and history are never part of a request.
+type KiwixConfig struct {
+	// URL — base address of kiwix-serve, e.g. "http://kiwix:8080". Empty ⇒ the
+	// whole block is normalised to nil and the source stays off.
+	URL string `json:"url,omitempty"`
+
+	// Book — the ZIM to search, by its catalog name, e.g.
+	// "wikipedia_en_all_maxi_2026-02". Take it from the /content/… href in
+	// /catalog/v2/entries; the display title is not the name.
+	//
+	// Required. kiwix-serve answers 400 to a search with an empty books.name,
+	// so a block without one is normalised to nil rather than left to fail one
+	// query at a time.
+	Book string `json:"book,omitempty"`
+
+	// MaxResults — how many hits are asked for. 0 ⇒ DefaultKiwixResults.
+	// Only the top few reach the phraser regardless; the rest are context the
+	// snippet ranking throws away.
+	MaxResults int `json:"max_results,omitempty"`
+
+	// SnippetRunes — how much of the joined snippets is handed to the phraser.
+	// 0 ⇒ DefaultKiwixSnippetRunes. Sized against the 4096-token context, which
+	// also holds the persona block and the prompt.
+	SnippetRunes int `json:"snippet_runes,omitempty"`
+
+	// Rewrite — turn the Russian question into English keywords with the
+	// resident model before searching. The ZIMs are English and kiwix ranks by
+	// keyword, not meaning, so a Russian sentence matches nothing. Costs one
+	// short LLM call per query. Default true; set false only to measure the
+	// difference or when the books are Russian.
+	Rewrite *bool `json:"rewrite,omitempty"`
+}
+
+// RewriteEnabled — Rewrite with its default applied. Absent ⇒ on.
+func (k *KiwixConfig) RewriteEnabled() bool {
+	return k.Rewrite == nil || *k.Rewrite
+}
+
+// Kiwix defaults, applied in Normalise.
+const (
+	DefaultKiwixResults      = 5
+	DefaultKiwixSnippetRunes = 1500
+)
+
 // CrawlWatchConfig — one page kept an eye on.
 type CrawlWatchConfig struct {
 	Name     string   `json:"name"` // note source is "crawl:<name>"
@@ -1345,6 +1402,19 @@ func (c *Config) applyDefaults() {
 		c.Crawl = nil
 	}
 
+	// Same rule for the ZIM reader: no address or no book, nothing to search.
+	if c.Kiwix != nil && (strings.TrimSpace(c.Kiwix.URL) == "" || strings.TrimSpace(c.Kiwix.Book) == "") {
+		c.Kiwix = nil
+	}
+	if c.Kiwix != nil {
+		if c.Kiwix.MaxResults <= 0 {
+			c.Kiwix.MaxResults = DefaultKiwixResults
+		}
+		if c.Kiwix.SnippetRunes <= 0 {
+			c.Kiwix.SnippetRunes = DefaultKiwixSnippetRunes
+		}
+	}
+
 	if c.Voice != nil {
 		if c.Voice.RouterThreshold <= 0 {
 			c.Voice.RouterThreshold = DefaultRouterThreshold

internal/config/config_test.go

@@ -367,3 +367,49 @@ func TestUpdateBlockValidatedAtStartup(t *testing.T) {
 		t.Error("Load accepted an update block with no health_socket")
 	}
 }
+
+// A kiwix block with no address, or no book, has nothing to search.
+// kiwix-serve answers 400 to an empty books.name, so the block is dropped here
+// rather than left to fail one query at a time.
+func TestNormaliseDropsIncompleteKiwix(t *testing.T) {
+	for _, tc := range []struct {
+		name string
+		in   *KiwixConfig
+	}{
+		{"no url", &KiwixConfig{Book: "wikipedia_en_all_maxi"}},
+		{"no book", &KiwixConfig{URL: "http://kiwix:8080"}},
+		{"blank url", &KiwixConfig{URL: "   ", Book: "b"}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			c := &Config{Kiwix: tc.in}
+			c.applyDefaults()
+			if c.Kiwix != nil {
+				t.Errorf("kept an unusable kiwix block: %+v", c.Kiwix)
+			}
+		})
+	}
+}
+
+func TestNormaliseFillsKiwixDefaults(t *testing.T) {
+	c := &Config{Kiwix: &KiwixConfig{URL: "http://kiwix:8080", Book: "b"}}
+	c.applyDefaults()
+	if c.Kiwix == nil {
+		t.Fatal("dropped a complete kiwix block")
+	}
+	if c.Kiwix.MaxResults != DefaultKiwixResults {
+		t.Errorf("MaxResults = %d, want %d", c.Kiwix.MaxResults, DefaultKiwixResults)
+	}
+	if c.Kiwix.SnippetRunes != DefaultKiwixSnippetRunes {
+		t.Errorf("SnippetRunes = %d, want %d", c.Kiwix.SnippetRunes, DefaultKiwixSnippetRunes)
+	}
+	// Rewriting is on unless it is turned off: an English ZIM searched with a
+	// Russian sentence matches nothing, so the useful default is the on one.
+	if !c.Kiwix.RewriteEnabled() {
+		t.Error("rewriting defaulted to off")
+	}
+	off := false
+	c.Kiwix.Rewrite = &off
+	if c.Kiwix.RewriteEnabled() {
+		t.Error("rewrite: false was not honoured")
+	}
+}

internal/kiwix/client.go

@@ -4,8 +4,9 @@
 // article snippet beats letting her recall. Nothing here talks to the internet;
 // the Kiwix server is on the same box.
 //
-// This is search only. Full articles are ~100KB of HTML, far too big for a 4096
-// token context, so the unit of context is the search snippet (~500 chars).
+// Search finds the article; Article reads it. The snippet a search returns is
+// NOT usable context on its own — see the comment on Article — so the unit of
+// context is the head of the article, truncated to fit a 4096 token window.
 package kiwix
 
 import (
@@ -20,6 +21,8 @@ import (
 	"strconv"
 	"strings"
 	"time"
+
+	"github.com/kami/maven/internal/crawl"
 )
 
 // Result is one search hit.
@@ -73,6 +76,52 @@ func (c *Client) Search(ctx context.Context, pattern, book string, limit int) ([
 	return ParseSearchRSS(resp.Body)
 }
 
+// articleMaxBytes — how much of an article HTML document is read before the
+// rest is discarded. A maxi Wikipedia page is around 100KB; 512KB is slack for
+// the long ones and a hard stop against a ZIM entry that is really a binary.
+const articleMaxBytes = 512 << 10
+
+// Article fetches one article by the Path a search hit carries and returns it
+// as extracted plain text, capped at maxRunes (0 ⇒ crawl.DefaultMaxRunes).
+//
+// This exists because the search snippet is not usable context. Kiwix builds
+// the snippet from wherever the keyword matched, and on a Wikipedia ZIM that is
+// routinely the "see also" navigation box at the foot of the page: a search for
+// "photosynthesis" comes back with "Ecological economics Ecological footprint
+// Ecological forecasting …" and a model handed that writes nothing worth
+// hearing. The lead paragraphs are at the top of the document, so truncating an
+// article from the front gets the definition the snippet was supposed to be.
+//
+// Nothing here reaches the internet: the path is resolved against the same
+// server the search went to.
+func (c *Client) Article(ctx context.Context, path string, maxRunes int) (crawl.Page, error) {
+	path = strings.TrimSpace(path)
+	if path == "" {
+		return crawl.Page{}, fmt.Errorf("kiwix article: empty path")
+	}
+	if !strings.HasPrefix(path, "/") {
+		path = "/" + path
+	}
+	u := c.base + path
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+	if err != nil {
+		return crawl.Page{}, err
+	}
+	resp, err := c.http.Do(req)
+	if err != nil {
+		return crawl.Page{}, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode != http.StatusOK {
+		return crawl.Page{}, fmt.Errorf("kiwix article %s: http %d", path, resp.StatusCode)
+	}
+	body, err := io.ReadAll(io.LimitReader(resp.Body, articleMaxBytes))
+	if err != nil {
+		return crawl.Page{}, err
+	}
+	return crawl.Extract(u, body, maxRunes), nil
+}
+
 // rss mirrors just the bits of the RSS 2.0 reply we use.
 type rss struct {
 	Items []struct {

internal/phraser/llmphraser.go

@@ -458,9 +458,16 @@ func chatSystemPrompt(block func() string) string {
 	// "ты" instruction carries that on its own and those two produced the
 	// worst output. The last line says outright not to echo the instructions,
 	// because a small model will otherwise treat any quoted string as licence.
+	//
+	// Amended the same day: with the openers gone the tic went with them, but
+	// "не забыл ли я" appeared — masculine, about herself. The old "я подумала"
+	// had been suppressing that by accident, being a feminine past tense the
+	// model could copy. Two short predicatives are not enough signal on their
+	// own, so the rule is now stated as morphology (-ла) rather than as a pair
+	// of words. A suffix rule generalises where an example only gets copied.
 	base := `Ты разговариваешь с хозяином.
 
-О себе — в женском роде: "рада", не "рад"; "поняла", не "понял". Он мужчина: обращайся к нему на "ты", в мужском роде. Никогда не "вы"/"ваш" и никогда "он"/"его" — ты говоришь ему, а не о нём.
+О себе — в женском роде: "рада", не "рад"; "поняла", не "понял". Все свои глаголы в прошедшем времени оканчивай на -ла: сделала, забыла, записала, подумала. Он мужчина: обращайся к нему на "ты", в мужском роде. Никогда не "вы"/"ваш" и никогда "он"/"его" — ты говоришь ему, а не о нём.
 
 Отвечай по-русски, коротко: одна-три фразы, живым языком. Ты доброжелательная, тебе интересно, но чувства не изображай. Не повторяй формулировки из этой инструкции — отвечай своими словами.
 
@@ -718,7 +725,13 @@ func (p *LLMPhraser) systemPrompt() string {
 func (p *LLMPhraser) querySystemPrompt() string {
 	// No self-introduction here: the persona block prepended one line above
 	// already says who she is, same as router.KnowledgePrompt.
-	base := "Ты отвечаешь ему по своим заметкам. Отвечай по-русски, коротко и своими словами, начинай с \"вот что я нашла: \". О себе — в женском роде (\"нашла\", \"записала\"). Он мужчина, обращайся к нему на \"ты\". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}."
+	//
+	// The opener is deliberate and stays: this path answers from his own notes
+	// and the fixed prefix is what marks the answer as a lookup rather than as
+	// something she knows. The grammar examples are not deliberate — same
+	// defect chatSystemPrompt had, where a 1.7B copies a quoted word instead of
+	// generalising from it. Stated as morphology instead.
+	base := "Ты отвечаешь ему по своим заметкам. Отвечай по-русски, коротко и своими словами, начинай с \"вот что я нашла: \". О себе — в женском роде, глаголы в прошедшем времени с окончанием -ла. Он мужчина, обращайся к нему на \"ты\". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}."
 	return persona.Prepend(p.cfg.ContextBlock, base)
 }