A non-2xx reply was read in full first and only then rejected. Two costs
followed. A 500 with a large error page pulled up to MaxBytes off the wire for
nothing. An error page over the cap returned ErrTooLarge, which names the size
and hides the status the server actually sent.
The status is a typed error now. webfetch.StatusError carries the code and
unwraps to ErrStatus, so errors.Is keeps working and errors.As reads the number.
crawl.StatusError is the same shape on the other side of the seam, and
cmd/mavend/crawls.go carries the code across.
That removes the string grep in crawl.isServerError, which decided whether a
failed robots.txt blocks a crawl by looking for " 50" in an error message it did
not own. A reworded error would have turned a 503 robots.txt into permission to
crawl. It reads the code now.
Two comments corrected. webfetch.HostMatches said the crawler calls it and
nothing outside the package does. rss.PlainText said the crawler's extractor
goes through it and crawl/extract.go has its own pass.
The rss poller parses the feed straight off the byte slice instead of copying a
document that can run to a megabyte through a string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The on-demand crawler was built over allow_hosts plus every watched host.
webfetch reads a non-empty allow list as these and nothing else, so a config
with one watch and no allow_hosts at all silently narrowed on-demand reading
to the watched site. Every other url he pasted came back as a flat refusal
with nothing in the log to explain it. The two crawlers now take two host
lists from one crawlHosts helper.
Crawl-delay was parsed into Rules and never read. The only pacing was the
fetcher's flat one request per host per second, which cannot express what a
site asked for, and deploy/README claimed the field was honoured. Page now
waits it out between the robots fetch and the page fetch, and a delay longer
than the turn fails the read instead of hanging it.
A robots.txt that failed was treated as no rules, so a site whose server was
having a bad minute became a site with no restrictions. A 5xx now refuses the
crawl. A 404 still means unrestricted, which is what the standard says.
The refusal check matched substrings of webfetch's message text from a package
that cannot import webfetch, so a reworded error would have silently turned
into a robots verdict. internal/crawl now exports ErrFetchRefused and
ErrFetchStatus and the adapter in cmd/mavend maps the webfetch sentinels onto
them. Robots group selection picks the longest matching agent prefix instead
of the first one in file order.
queryWeb passed a claim it could not serve when no crawler was configured, so
an unconfigured deployment answered a web question with an apology instead of
falling through to the model.
Found in review of #67.
The network fallback behind the local sources, off unless configured.
internal/crawl is pure: a stdlib robots.txt parser (group specificity,
wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a
watcher that notes a watched page only when its text changed. It has no store
access and no net/http; cmd/mavend/crawls.go is the impure half.
Every limit is code and tested: the guarded fetcher from #258 enforces the host
allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS
rebinding and each redirect hop are covered), caps size and redirects, times out,
and spaces requests per host. A robots.txt Disallow is refused with no override.
On demand, reading is a query source placed last in the chain, after his memory,
his notes, and the local Kiwix ZIMs once those are wired: no URL in the
utterance means no fetch, and only the URL ever leaves the box. Scheduled
watches write notes and announce nothing.
The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers
are stdlib. No new dependency.
claude
kami