crawl: stop letting a watch widen on-demand reading, and honour Crawl-delay

The on-demand crawler was built over allow_hosts plus every watched host.
webfetch reads a non-empty allow list as these and nothing else, so a config
with one watch and no allow_hosts at all silently narrowed on-demand reading
to the watched site. Every other url he pasted came back as a flat refusal
with nothing in the log to explain it. The two crawlers now take two host
lists from one crawlHosts helper.

Crawl-delay was parsed into Rules and never read. The only pacing was the
fetcher's flat one request per host per second, which cannot express what a
site asked for, and deploy/README claimed the field was honoured. Page now
waits it out between the robots fetch and the page fetch, and a delay longer
than the turn fails the read instead of hanging it.

A robots.txt that failed was treated as no rules, so a site whose server was
having a bad minute became a site with no restrictions. A 5xx now refuses the
crawl. A 404 still means unrestricted, which is what the standard says.

The refusal check matched substrings of webfetch's message text from a package
that cannot import webfetch, so a reworded error would have silently turned
into a robots verdict. internal/crawl now exports ErrFetchRefused and
ErrFetchStatus and the adapter in cmd/mavend maps the webfetch sentinels onto
them. Robots group selection picks the longest matching agent prefix instead
of the first one in file order.

queryWeb passed a claim it could not serve when no crawler was configured, so
an unconfigured deployment answered a web question with an apology instead of
falling through to the model.

Found in review of #67.

internal/config/config.go

@@ -940,8 +940,13 @@ type CrawlConfig struct {
 	Interval Duration `json:"interval,omitempty"`
 
 	// AllowHosts — when set, the ONLY hosts the crawler may reach (subdomains
-	// included). Watched pages' own hosts are added automatically. Setting this
-	// is how "she may read the arch wiki and nothing else" is expressed.
+	// included). Setting this is how "she may read the arch wiki and nothing
+	// else" is expressed.
+	//
+	// A watched page's own host is reachable by the scheduled crawler whether
+	// or not it is listed here, because configuring a watch is already saying
+	// she may read it. That does NOT extend to on-demand reading: a watch is
+	// not an allowlist entry for pages he pastes.
 	AllowHosts []string `json:"allow_hosts,omitempty"`
 
 	// DenyHosts — never reachable, checked first. Private addresses do not need