Read a web page when he names one, and watch a few on a timer (#259)

The network fallback behind the local sources, off unless configured.

internal/crawl is pure: a stdlib robots.txt parser (group specificity,
wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a
watcher that notes a watched page only when its text changed. It has no store
access and no net/http; cmd/mavend/crawls.go is the impure half.

Every limit is code and tested: the guarded fetcher from #258 enforces the host
allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS
rebinding and each redirect hop are covered), caps size and redirects, times out,
and spaces requests per host. A robots.txt Disallow is refused with no override.

On demand, reading is a query source placed last in the chain, after his memory,
his notes, and the local Kiwix ZIMs once those are wired: no URL in the
utterance means no fetch, and only the URL ever leaves the box. Scheduled
watches write notes and announce nothing.

The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers
are stdlib. No new dependency.

internal/router/url.go

@@ -0,0 +1,39 @@
+package router
+
+import (
+	"regexp"
+	"strings"
+)
+
+// Finding a URL in an utterance (Vikunja #259).
+//
+// This is deliberately strict: a scheme is required. "посмотри на example.org"
+// is not treated as a fetch request, because a bare dotted word is also how
+// people say file names, versions and Russian abbreviations, and the cost of a
+// false positive here is an outbound request nobody asked for.
+//
+// Note where this runs: an utterance from STT. Whisper will mangle a spoken URL,
+// which is fine — the URL that survives is one he pasted into the web chat, and
+// a mangled one simply fails to match.
+var urlRE = regexp.MustCompile(`(?i)\bhttps?://[^\s<>"']+`)
+
+// FirstURL returns the first http(s) URL in text.
+//
+// Trailing punctuation is trimmed: he ends sentences, and "…/page." is not a
+// path component. A closing bracket is only trimmed when it has no opener,
+// because a wikipedia URL legitimately ends in one.
+func FirstURL(text string) (string, bool) {
+	m := urlRE.FindString(text)
+	if m == "" {
+		return "", false
+	}
+	m = strings.TrimRight(m, ".,;:!?…")
+	if strings.HasSuffix(m, ")") && strings.Count(m, "(") == 0 {
+		m = strings.TrimSuffix(m, ")")
+	}
+	// A scheme with nothing after it is not a URL.
+	if rest := strings.SplitN(m, "//", 2); len(rest) < 2 || rest[1] == "" {
+		return "", false
+	}
+	return m, true
+}

internal/router/url_test.go

@@ -0,0 +1,34 @@
+package router
+
+import "testing"
+
+func TestFirstURL(t *testing.T) {
+	cases := []struct {
+		text string
+		want string
+	}{
+		{"посмотри https://example.org/page — что там?", "https://example.org/page"},
+		{"почитай http://example.org/a/b?x=1 и скажи", "http://example.org/a/b?x=1"},
+		{"вот ссылка: https://example.org/page.", "https://example.org/page"},
+		{"https://ru.wikipedia.org/wiki/Небо_(значения)", "https://ru.wikipedia.org/wiki/Небо_(значения)"},
+		// No scheme ⇒ no fetch. A bare dotted word is not an instruction to
+		// reach out to the network.
+		{"посмотри на example.org", ""},
+		{"открой файл config.json", ""},
+		{"что нового?", ""},
+		{"https://", ""},
+		{"", ""},
+	}
+	for _, c := range cases {
+		got, ok := FirstURL(c.text)
+		if c.want == "" {
+			if ok {
+				t.Errorf("FirstURL(%q) = %q, want no match", c.text, got)
+			}
+			continue
+		}
+		if !ok || got != c.want {
+			t.Errorf("FirstURL(%q) = %q, %v; want %q", c.text, got, ok, c.want)
+		}
+	}
+}