crawl: stop letting a watch widen on-demand reading, and honour Crawl-delay

The on-demand crawler was built over allow_hosts plus every watched host.
webfetch reads a non-empty allow list as these and nothing else, so a config
with one watch and no allow_hosts at all silently narrowed on-demand reading
to the watched site. Every other url he pasted came back as a flat refusal
with nothing in the log to explain it. The two crawlers now take two host
lists from one crawlHosts helper.

Crawl-delay was parsed into Rules and never read. The only pacing was the
fetcher's flat one request per host per second, which cannot express what a
site asked for, and deploy/README claimed the field was honoured. Page now
waits it out between the robots fetch and the page fetch, and a delay longer
than the turn fails the read instead of hanging it.

A robots.txt that failed was treated as no rules, so a site whose server was
having a bad minute became a site with no restrictions. A 5xx now refuses the
crawl. A 404 still means unrestricted, which is what the standard says.

The refusal check matched substrings of webfetch's message text from a package
that cannot import webfetch, so a reworded error would have silently turned
into a robots verdict. internal/crawl now exports ErrFetchRefused and
ErrFetchStatus and the adapter in cmd/mavend maps the webfetch sentinels onto
them. Robots group selection picks the longest matching agent prefix instead
of the first one in file order.

queryWeb passed a claim it could not serve when no crawler was configured, so
an unconfigured deployment answered a web question with an apology instead of
falling through to the model.

Found in review of #67.

deploy/README.md

@@ -95,10 +95,17 @@ switched:
   a fallback and not a habit;
 - `watches` re-reads a fixed list on its interval and writes a note when the
   text changed. Like the feeds, it announces nothing;
-- the answer path sits **last** in the query chain, behind his memory, his notes
-  and (once wired) the local Kiwix ZIMs. A local read costs nothing;
+- the answer path sits behind his memory and his notes, and ahead of the model
+  answering from what it remembers. Kiwix is not wired into the chain yet. A
+  local read costs nothing, so anything local goes first;
 - `robots.txt` is fetched first and obeyed with no override; a `Disallow` is a
-  refusal she says out loud. `Crawl-delay` is honoured;
+  refusal she says out loud. `Crawl-delay` is waited out before the page is
+  fetched, and a delay longer than the turn fails the read instead of hanging
+  it. A `robots.txt` that answers 5xx refuses the crawl — a broken server is
+  not permission;
+- `allow_hosts` limits on-demand reading to those hosts and nothing else.
+  Watched pages' hosts are reachable by the scheduled crawler whether listed or
+  not, but a watch does **not** widen what he may ask her to read;
 - same guarded fetcher as the feeds: allowlist/denylist, no private addresses,
   size cap, redirect cap, timeout, one request per host per second;
 - dedup state is the config fact `crawl:hash:<name>`;