claude
2ea39a3d41
Kiwix ranks by keyword overlap, which the package doc has said since it was written: "why is the sky blue" finds a TV episode. queryKiwix sent the whole Russian sentence, because the verbatim path added by V-508 skips the rewriter that would have reduced it. Measured against the Russian ZIM on 2026-08-09, over eight questions. Four reach the right article where they did not: TCP was "Перехват TCP-соединения" and is TCP, фотосинтез was "C4-фотосинтез" and is Фотосинтез, Линус Торвальдс was "Tux", and "кто написал Войну и мир" was "Радуйся, мир (Доктор Кто)". Two were already right and stay right. Two are still wrong and were wrong before. Nothing regressed. kiwix.Topic drops the narrative request, the interrogative and a verb behind one, and keeps everything else. A word it cannot classify is more likely the topic than noise. TitlePath tries the exact article first, since a ZIM is addressable by title and a wrong title is a 404. The gate this task set out to build does not exist. Query-to-passage cosine scored 0.79-0.91 on answerable questions and 0.75-0.84 on unanswerable ones, and the sets overlap. The wrong TCP article scored 0.8653, above five of six unanswerable rows. e5 measures topic, not whether the passage answers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
99 lines
3.1 KiB
Go
99 lines
3.1 KiB
Go
package kiwix
|
||
|
||
import (
|
||
"strings"
|
||
"unicode"
|
||
|
||
"github.com/kami/maven/internal/lexicon"
|
||
"github.com/kami/maven/internal/morph"
|
||
)
|
||
|
||
// Topic reduces a question to the thing it is about, because Kiwix ranks by
|
||
// keyword overlap and a whole sentence buries the keyword that matters.
|
||
//
|
||
// This package's own doc says it: "why is the sky blue" finds a TV episode.
|
||
// Measured against the Russian ZIM on 2026-08-09, the sentence and the topic
|
||
// return different articles for the same question. "кто написал Войну и мир"
|
||
// returns "Радуйся, мир (Доктор Кто)"; "Войну и мир" returns the novel first.
|
||
// "что такое TCP" returns "Перехват TCP-соединения"; "TCP" returns TCP. The
|
||
// English path had a rewriter doing this with a model call. The Russian path
|
||
// reads the book verbatim (V-508) and had nothing.
|
||
//
|
||
// It drops three things off the front and stops: the narrative request, the
|
||
// interrogative, and a verb sitting between them and the noun. Everything else
|
||
// is kept, because a word this cannot classify is more likely the topic than
|
||
// noise. An empty return means the utterance was question words alone, and the
|
||
// caller searches the sentence as before.
|
||
func Topic(utterance string) string {
|
||
words := strings.Fields(strings.TrimSpace(utterance))
|
||
cut := 0
|
||
for cut < len(words) {
|
||
w := strings.Trim(strings.ToLower(words[cut]), ".,!?…:;\"'«»")
|
||
if w == "" {
|
||
cut++
|
||
continue
|
||
}
|
||
switch {
|
||
case inList(lexicon.NarrativeRequests(), w),
|
||
inList(lexicon.Interrogatives(), w),
|
||
inList(lexicon.FirstPerson(), w),
|
||
// "что ТАКОЕ x", "кто ТАКОЙ x" — the copula that only ever follows
|
||
// an interrogative, and never a topic on its own.
|
||
cut > 0 && isCopula(w),
|
||
// "расскажи ПРО x", "о x". One-letter and two-letter prepositions
|
||
// are not a closed class worth a lexicon set of their own.
|
||
cut > 0 && isLeadingPreposition(w),
|
||
// "кто НАПИСАЛ Войну и мир". A verb here is the question's own
|
||
// verb, not part of the title. Only after something was already
|
||
// dropped, so "написал отчёт" as a topic survives intact.
|
||
cut > 0 && morph.IsVerbForm(w):
|
||
cut++
|
||
default:
|
||
// The question mark is the sentence's, not the title's, and Kiwix
|
||
// carries it into the keyword match.
|
||
topic := strings.TrimRight(strings.Join(words[cut:], " "), " .,!?…:;\"'«»")
|
||
if !hasLetter(topic) {
|
||
return ""
|
||
}
|
||
return topic
|
||
}
|
||
}
|
||
return ""
|
||
}
|
||
|
||
func isCopula(w string) bool {
|
||
switch w {
|
||
case "такое", "такой", "такая", "такие", "is", "are", "was", "were":
|
||
return true
|
||
}
|
||
return false
|
||
}
|
||
|
||
func isLeadingPreposition(w string) bool {
|
||
switch w {
|
||
case "про", "о", "об", "обо", "по", "about", "of", "on":
|
||
return true
|
||
}
|
||
return false
|
||
}
|
||
|
||
func inList(list []string, w string) bool {
|
||
for _, x := range list {
|
||
if x == w {
|
||
return true
|
||
}
|
||
}
|
||
return false
|
||
}
|
||
|
||
// hasLetter is the guard against a topic that reduced to punctuation or digits
|
||
// alone, which no ZIM title matches.
|
||
func hasLetter(s string) bool {
|
||
for _, r := range s {
|
||
if unicode.IsLetter(r) {
|
||
return true
|
||
}
|
||
}
|
||
return false
|
||
}
|