webfetch checks the status before it reads the body (V-581)
A non-2xx reply was read in full first and only then rejected. Two costs followed. A 500 with a large error page pulled up to MaxBytes off the wire for nothing. An error page over the cap returned ErrTooLarge, which names the size and hides the status the server actually sent. The status is a typed error now. webfetch.StatusError carries the code and unwraps to ErrStatus, so errors.Is keeps working and errors.As reads the number. crawl.StatusError is the same shape on the other side of the seam, and cmd/mavend/crawls.go carries the code across. That removes the string grep in crawl.isServerError, which decided whether a failed robots.txt blocks a crawl by looking for " 50" in an error message it did not own. A reworded error would have turned a 503 robots.txt into permission to crawl. It reads the code now. Two comments corrected. webfetch.HostMatches said the crawler calls it and nothing outside the package does. rss.PlainText said the crawler's extractor goes through it and crawl/extract.go has its own pass. The rss poller parses the feed straight off the byte slice instead of copying a document that can run to a megabyte through a string. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -173,8 +173,9 @@ var (
|
||||
)
|
||||
|
||||
// PlainText strips markup and decodes entities — feed summaries are HTML, and
|
||||
// what reaches a note (and possibly the TTS) must be text. Exported because the
|
||||
// crawler's extractor needs exactly this on a bigger input.
|
||||
// what reaches a note (and possibly the TTS) must be text. Exported so a caller
|
||||
// holding raw feed markup can reduce it the same way; crawl/extract.go does the
|
||||
// bigger job on a whole document and does not go through here.
|
||||
func PlainText(s string) string {
|
||||
s = scriptRE.ReplaceAllString(s, " ")
|
||||
s = tagRE.ReplaceAllString(s, " ")
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package rss
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
@@ -164,7 +165,9 @@ func (p *Poller) PollFeed(ctx context.Context, f FeedConfig, now time.Time) (int
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
feed, err := Parse(strings.NewReader(string(body.Bytes)))
|
||||
// bytes.NewReader and not strings.NewReader(string(…)): the latter copied a
|
||||
// feed document that can run to a megabyte, for nothing.
|
||||
feed, err := Parse(bytes.NewReader(body.Bytes))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user