webfetch checks the status before it reads the body (V-581)

A non-2xx reply was read in full first and only then rejected. Two costs
followed. A 500 with a large error page pulled up to MaxBytes off the wire for
nothing. An error page over the cap returned ErrTooLarge, which names the size
and hides the status the server actually sent.

The status is a typed error now. webfetch.StatusError carries the code and
unwraps to ErrStatus, so errors.Is keeps working and errors.As reads the number.
crawl.StatusError is the same shape on the other side of the seam, and
cmd/mavend/crawls.go carries the code across.

That removes the string grep in crawl.isServerError, which decided whether a
failed robots.txt blocks a crawl by looking for " 50" in an error message it did
not own. A reworded error would have turned a 503 robots.txt into permission to
crawl. It reads the code now.

Two comments corrected. webfetch.HostMatches said the crawler calls it and
nothing outside the package does. rss.PlainText said the crawler's extractor
goes through it and crawl/extract.go has its own pass.

The rss poller parses the feed straight off the byte slice instead of copying a
document that can run to a megabyte through a string.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-06 03:12:59 +04:00
parent 316fb197a8
commit 04584fb2da
7 changed files with 71 additions and 17 deletions
+3 -2
View File
@@ -173,8 +173,9 @@ var (
)
// PlainText strips markup and decodes entities — feed summaries are HTML, and
// what reaches a note (and possibly the TTS) must be text. Exported because the
// crawler's extractor needs exactly this on a bigger input.
// what reaches a note (and possibly the TTS) must be text. Exported so a caller
// holding raw feed markup can reduce it the same way; crawl/extract.go does the
// bigger job on a whole document and does not go through here.
func PlainText(s string) string {
s = scriptRE.ReplaceAllString(s, " ")
s = tagRE.ReplaceAllString(s, " ")
+4 -1
View File
@@ -1,6 +1,7 @@
package rss
import (
"bytes"
"context"
"fmt"
"log"
@@ -164,7 +165,9 @@ func (p *Poller) PollFeed(ctx context.Context, f FeedConfig, now time.Time) (int
if err != nil {
return 0, err
}
feed, err := Parse(strings.NewReader(string(body.Bytes)))
// bytes.NewReader and not strings.NewReader(string(…)): the latter copied a
// feed document that can run to a megabyte, for nothing.
feed, err := Parse(bytes.NewReader(body.Bytes))
if err != nil {
return 0, err
}