kami
2c1b0eede0
The network fallback behind the local sources, off unless configured. internal/crawl is pure: a stdlib robots.txt parser (group specificity, wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a watcher that notes a watched page only when its text changed. It has no store access and no net/http; cmd/mavend/crawls.go is the impure half. Every limit is code and tested: the guarded fetcher from #258 enforces the host allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS rebinding and each redirect hop are covered), caps size and redirects, times out, and spaces requests per host. A robots.txt Disallow is refused with no override. On demand, reading is a query source placed last in the chain, after his memory, his notes, and the local Kiwix ZIMs once those are wired: no URL in the utterance means no fetch, and only the URL ever leaves the box. Scheduled watches write notes and announce nothing. The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers are stdlib. No new dependency.
35 lines
1.0 KiB
Go
35 lines
1.0 KiB
Go
package router
|
|
|
|
import "testing"
|
|
|
|
func TestFirstURL(t *testing.T) {
|
|
cases := []struct {
|
|
text string
|
|
want string
|
|
}{
|
|
{"посмотри https://example.org/page — что там?", "https://example.org/page"},
|
|
{"почитай http://example.org/a/b?x=1 и скажи", "http://example.org/a/b?x=1"},
|
|
{"вот ссылка: https://example.org/page.", "https://example.org/page"},
|
|
{"https://ru.wikipedia.org/wiki/Небо_(значения)", "https://ru.wikipedia.org/wiki/Небо_(значения)"},
|
|
// No scheme ⇒ no fetch. A bare dotted word is not an instruction to
|
|
// reach out to the network.
|
|
{"посмотри на example.org", ""},
|
|
{"открой файл config.json", ""},
|
|
{"что нового?", ""},
|
|
{"https://", ""},
|
|
{"", ""},
|
|
}
|
|
for _, c := range cases {
|
|
got, ok := FirstURL(c.text)
|
|
if c.want == "" {
|
|
if ok {
|
|
t.Errorf("FirstURL(%q) = %q, want no match", c.text, got)
|
|
}
|
|
continue
|
|
}
|
|
if !ok || got != c.want {
|
|
t.Errorf("FirstURL(%q) = %q, %v; want %q", c.text, got, ok, c.want)
|
|
}
|
|
}
|
|
}
|