kami
2c1b0eede0
The network fallback behind the local sources, off unless configured. internal/crawl is pure: a stdlib robots.txt parser (group specificity, wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a watcher that notes a watched page only when its text changed. It has no store access and no net/http; cmd/mavend/crawls.go is the impure half. Every limit is code and tested: the guarded fetcher from #258 enforces the host allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS rebinding and each redirect hop are covered), caps size and redirects, times out, and spaces requests per host. A robots.txt Disallow is refused with no override. On demand, reading is a query source placed last in the chain, after his memory, his notes, and the local Kiwix ZIMs once those are wired: no URL in the utterance means no fetch, and only the URL ever leaves the box. Scheduled watches write notes and announce nothing. The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers are stdlib. No new dependency.
40 lines
1.3 KiB
Go
40 lines
1.3 KiB
Go
package router
|
|
|
|
import (
|
|
"regexp"
|
|
"strings"
|
|
)
|
|
|
|
// Finding a URL in an utterance (Vikunja #259).
|
|
//
|
|
// This is deliberately strict: a scheme is required. "посмотри на example.org"
|
|
// is not treated as a fetch request, because a bare dotted word is also how
|
|
// people say file names, versions and Russian abbreviations, and the cost of a
|
|
// false positive here is an outbound request nobody asked for.
|
|
//
|
|
// Note where this runs: an utterance from STT. Whisper will mangle a spoken URL,
|
|
// which is fine — the URL that survives is one he pasted into the web chat, and
|
|
// a mangled one simply fails to match.
|
|
var urlRE = regexp.MustCompile(`(?i)\bhttps?://[^\s<>"']+`)
|
|
|
|
// FirstURL returns the first http(s) URL in text.
|
|
//
|
|
// Trailing punctuation is trimmed: he ends sentences, and "…/page." is not a
|
|
// path component. A closing bracket is only trimmed when it has no opener,
|
|
// because a wikipedia URL legitimately ends in one.
|
|
func FirstURL(text string) (string, bool) {
|
|
m := urlRE.FindString(text)
|
|
if m == "" {
|
|
return "", false
|
|
}
|
|
m = strings.TrimRight(m, ".,;:!?…")
|
|
if strings.HasSuffix(m, ")") && strings.Count(m, "(") == 0 {
|
|
m = strings.TrimSuffix(m, ")")
|
|
}
|
|
// A scheme with nothing after it is not a URL.
|
|
if rest := strings.SplitN(m, "//", 2); len(rest) < 2 || rest[1] == "" {
|
|
return "", false
|
|
}
|
|
return m, true
|
|
}
|