kiwix: read the article, not the snippet, and state the persona as morphology
Two things that were each half-done. The prompts handed the model copyable examples. chatSystemPrompt lost its openers this morning and the "Я подумала, что" tic went with them, but "не забыл ли я" appeared in its place: the removed example had been suppressing the masculine self-reference by accident. Two predicatives are not enough signal, so the rule is now stated as morphology (-ла) rather than as a pair of words — a suffix rule generalises where an example only gets copied. querySystemPrompt had the same defect and gets the same treatment; its "вот что я нашла: " opener is deliberate and stays. internal/kiwix had no caller. actions_query.go said "once internal/kiwix is wired into this chain" and that never happened. It is wired now, between the notes pass and the web source: everything of his answers first, and only what is left over is looked up. Off unless a `kiwix` block names a server and a book. Reading the search snippet does not work. Kiwix builds it from wherever the keyword matched, which on Wikipedia is the navigation box at the foot of the page — the first version of this answered "что такое фотосинтез?" by reciting "Ecological economics Ecological footprint Ecological forecasting …". Client grows an Article method; the head of the article is the lead paragraph, which is the definition the snippet was meant to be. Verified on the box: the same question now answers correctly off the ZIM. Only the rewritten query leaves the process. A test asserts it: a turn carrying a stored note must not put that note in the search string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX
This commit is contained in:
@@ -4,8 +4,9 @@
|
||||
// article snippet beats letting her recall. Nothing here talks to the internet;
|
||||
// the Kiwix server is on the same box.
|
||||
//
|
||||
// This is search only. Full articles are ~100KB of HTML, far too big for a 4096
|
||||
// token context, so the unit of context is the search snippet (~500 chars).
|
||||
// Search finds the article; Article reads it. The snippet a search returns is
|
||||
// NOT usable context on its own — see the comment on Article — so the unit of
|
||||
// context is the head of the article, truncated to fit a 4096 token window.
|
||||
package kiwix
|
||||
|
||||
import (
|
||||
@@ -20,6 +21,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/crawl"
|
||||
)
|
||||
|
||||
// Result is one search hit.
|
||||
@@ -73,6 +76,52 @@ func (c *Client) Search(ctx context.Context, pattern, book string, limit int) ([
|
||||
return ParseSearchRSS(resp.Body)
|
||||
}
|
||||
|
||||
// articleMaxBytes — how much of an article HTML document is read before the
|
||||
// rest is discarded. A maxi Wikipedia page is around 100KB; 512KB is slack for
|
||||
// the long ones and a hard stop against a ZIM entry that is really a binary.
|
||||
const articleMaxBytes = 512 << 10
|
||||
|
||||
// Article fetches one article by the Path a search hit carries and returns it
|
||||
// as extracted plain text, capped at maxRunes (0 ⇒ crawl.DefaultMaxRunes).
|
||||
//
|
||||
// This exists because the search snippet is not usable context. Kiwix builds
|
||||
// the snippet from wherever the keyword matched, and on a Wikipedia ZIM that is
|
||||
// routinely the "see also" navigation box at the foot of the page: a search for
|
||||
// "photosynthesis" comes back with "Ecological economics Ecological footprint
|
||||
// Ecological forecasting …" and a model handed that writes nothing worth
|
||||
// hearing. The lead paragraphs are at the top of the document, so truncating an
|
||||
// article from the front gets the definition the snippet was supposed to be.
|
||||
//
|
||||
// Nothing here reaches the internet: the path is resolved against the same
|
||||
// server the search went to.
|
||||
func (c *Client) Article(ctx context.Context, path string, maxRunes int) (crawl.Page, error) {
|
||||
path = strings.TrimSpace(path)
|
||||
if path == "" {
|
||||
return crawl.Page{}, fmt.Errorf("kiwix article: empty path")
|
||||
}
|
||||
if !strings.HasPrefix(path, "/") {
|
||||
path = "/" + path
|
||||
}
|
||||
u := c.base + path
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||
if err != nil {
|
||||
return crawl.Page{}, err
|
||||
}
|
||||
resp, err := c.http.Do(req)
|
||||
if err != nil {
|
||||
return crawl.Page{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return crawl.Page{}, fmt.Errorf("kiwix article %s: http %d", path, resp.StatusCode)
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, articleMaxBytes))
|
||||
if err != nil {
|
||||
return crawl.Page{}, err
|
||||
}
|
||||
return crawl.Extract(u, body, maxRunes), nil
|
||||
}
|
||||
|
||||
// rss mirrors just the bits of the RSS 2.0 reply we use.
|
||||
type rss struct {
|
||||
Items []struct {
|
||||
|
||||
Reference in New Issue
Block a user