crawl: stop letting a watch widen on-demand reading, and honour Crawl-delay

The on-demand crawler was built over allow_hosts plus every watched host.
webfetch reads a non-empty allow list as these and nothing else, so a config
with one watch and no allow_hosts at all silently narrowed on-demand reading
to the watched site. Every other url he pasted came back as a flat refusal
with nothing in the log to explain it. The two crawlers now take two host
lists from one crawlHosts helper.

Crawl-delay was parsed into Rules and never read. The only pacing was the
fetcher's flat one request per host per second, which cannot express what a
site asked for, and deploy/README claimed the field was honoured. Page now
waits it out between the robots fetch and the page fetch, and a delay longer
than the turn fails the read instead of hanging it.

A robots.txt that failed was treated as no rules, so a site whose server was
having a bad minute became a site with no restrictions. A 5xx now refuses the
crawl. A 404 still means unrestricted, which is what the standard says.

The refusal check matched substrings of webfetch's message text from a package
that cannot import webfetch, so a reworded error would have silently turned
into a robots verdict. internal/crawl now exports ErrFetchRefused and
ErrFetchStatus and the adapter in cmd/mavend maps the webfetch sentinels onto
them. Robots group selection picks the longest matching agent prefix instead
of the first one in file order.

queryWeb passed a claim it could not serve when no crawler was configured, so
an unconfigured deployment answered a web question with an apology instead of
falling through to the model.

Found in review of #67.

internal/crawl/robots.go

@@ -27,7 +27,8 @@ type Rules struct {
 	disallow []string
 	// Delay is Crawl-delay in seconds when the group named one, 0 otherwise.
 	// The fetcher's own per-host rate limit is the floor; this can only make
-	// Maven slower, never faster.
+	// Maven slower, never faster. Enforced in Crawler.waitCrawlDelay — the
+	// fetcher's limiter is flat and cannot express what a site asked for.
 	Delay time.Duration
 }
 
@@ -104,7 +105,11 @@ func ParseRobots(body string, agent string) Rules {
 		}
 	}
 
+	// Most specific wins, and specificity is the LENGTH of the matching agent
+	// string, not the file order. Two groups naming "mav" and "maven" used to be
+	// resolved by whichever came last in the file.
 	var star, exact *group
+	best := 0
 	for i := range groups {
 		for _, a := range groups[i].agents {
 			if a == "*" && star == nil {
@@ -112,8 +117,8 @@ func ParseRobots(body string, agent string) Rules {
 			}
 			// A robots.txt names "maven", we send "Maven/1.0 (…)": match on
 			// prefix, which is how every crawler reads this field.
-			if a != "*" && a != "" && strings.HasPrefix(agent, a) {
-				exact = &groups[i]
+			if a != "*" && a != "" && strings.HasPrefix(agent, a) && len(a) > best {
+				best, exact = len(a), &groups[i]
 			}
 		}
 	}