Read a web page when he names one, and watch a few on a timer (#259)

The network fallback behind the local sources, off unless configured.

internal/crawl is pure: a stdlib robots.txt parser (group specificity,
wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a
watcher that notes a watched page only when its text changed. It has no store
access and no net/http; cmd/mavend/crawls.go is the impure half.

Every limit is code and tested: the guarded fetcher from #258 enforces the host
allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS
rebinding and each redirect hop are covered), caps size and redirects, times out,
and spaces requests per host. A robots.txt Disallow is refused with no override.

On demand, reading is a query source placed last in the chain, after his memory,
his notes, and the local Kiwix ZIMs once those are wired: no URL in the
utterance means no fetch, and only the URL ever leaves the box. Scheduled
watches write notes and announce nothing.

The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers
are stdlib. No new dependency.

internal/config/config.go

@@ -161,6 +161,10 @@ type Config struct {
 	// the weather and telegram. See FeedsConfig.
 	Feeds *FeedsConfig `json:"feeds,omitempty"`
 
+	// Crawl — reading a web page (Vikunja #259). nil / absent ⇒ Maven never
+	// fetches a page: not on request, not on a schedule. See CrawlConfig.
+	Crawl *CrawlConfig `json:"crawl,omitempty"`
+
 	// Praxis — the ecosystem attention-state service. When configured, maven
 	// calls the Praxis HTTP tools API for attention listing and item lifecycle.
 	// Maven never touches Praxis's database directly (ecosystem invariant: no
@@ -454,6 +458,62 @@ type FeedSourceConfig struct {
 	Exclude  []string `json:"exclude,omitempty"`  // drop items containing any of these
 }
 
+// CrawlConfig — the web crawler (Vikunja #259, docs/plans/14-web-crawler.md).
+//
+// Absent ⇒ off, and off means no page is ever fetched. Present with neither
+// `on_demand` nor a `watches` entry is also off: there would be nothing to do.
+//
+// The crawler is the LAST place an answer is looked for, behind the model, his
+// own memory and the local Kiwix ZIMs. That ordering lives in the query-source
+// chain (cmd/mavend/actions_query.go), not here, but it is the reason this block
+// is small: it is a fallback, not a search engine.
+//
+// Only the URL leaves the box. His notes, facts, persona block and history are
+// never part of a request — the crawler package cannot even read the store.
+type CrawlConfig struct {
+	// OnDemand — may he ask her to read a page he names out loud
+	// ("посмотри https://… — что там пишут?"). false ⇒ the on-demand answer
+	// source stays off and only the watches below run.
+	OnDemand bool `json:"on_demand,omitempty"`
+
+	// Watches — pages re-read on a schedule. A page whose text changed is
+	// written as a note (source "crawl:<name>"); nothing is announced.
+	Watches []CrawlWatchConfig `json:"watches,omitempty"`
+
+	// Interval — default watch cadence. 0 ⇒ crawl.DefaultWatchInterval (6h).
+	Interval Duration `json:"interval,omitempty"`
+
+	// AllowHosts — when set, the ONLY hosts the crawler may reach (subdomains
+	// included). Watched pages' own hosts are added automatically. Setting this
+	// is how "she may read the arch wiki and nothing else" is expressed.
+	AllowHosts []string `json:"allow_hosts,omitempty"`
+
+	// DenyHosts — never reachable, checked first. Private addresses do not need
+	// to be listed: they are refused unconditionally (see internal/webfetch).
+	DenyHosts []string `json:"deny_hosts,omitempty"`
+
+	// UserAgent — sent on every request AND matched against robots.txt groups.
+	// Empty ⇒ webfetch.DefaultUserAgent.
+	UserAgent string `json:"user_agent,omitempty"`
+
+	// Timeout — per-request budget. 0 ⇒ webfetch.DefaultTimeout.
+	Timeout Duration `json:"timeout,omitempty"`
+
+	// MaxBytes — response size cap. 0 ⇒ webfetch.DefaultMaxBytes (2 MiB).
+	MaxBytes int64 `json:"max_bytes,omitempty"`
+
+	// MaxRunes — how much extracted text is kept. 0 ⇒ crawl.DefaultMaxRunes
+	// (4000), which is what fits a 4096-token context alongside a prompt.
+	MaxRunes int `json:"max_runes,omitempty"`
+}
+
+// CrawlWatchConfig — one page kept an eye on.
+type CrawlWatchConfig struct {
+	Name     string   `json:"name"` // note source is "crawl:<name>"
+	URL      string   `json:"url"`
+	Interval Duration `json:"interval,omitempty"` // 0 ⇒ CrawlConfig.Interval
+}
+
 // MemoryEvalConfig — the background memory-evaluation loop (Vikunja #248).
 // Absent ⇒ off, like every other capability that costs something the owner did
 // not ask for. Each evaluation is a full LLM round-trip on the one resident
@@ -697,6 +757,12 @@ func (c *Config) applyDefaults() {
 		c.Feeds = nil
 	}
 
+	// Same rule for the crawler: a block that neither answers on demand nor
+	// watches anything has nothing to do, so it is normalised to "off".
+	if c.Crawl != nil && !c.Crawl.OnDemand && len(c.Crawl.Watches) == 0 {
+		c.Crawl = nil
+	}
+
 	if c.Voice != nil {
 		if c.Voice.RouterThreshold <= 0 {
 			c.Voice.RouterThreshold = DefaultRouterThreshold