rss.Poller.seen held every undated item ever seen, one entry per id, for as
long as mavend ran. fresh() added and nothing removed. A feed that ships items
with no <pubDate> grew it forever.
seenIDs is the same set with a bound: the map answers the lookup, a slice
remembers insertion order, and the oldest id falls out past 512. The cap has
to stay above any one feed's front page or an item still listed there would be
written a second time, and a few hundred covers the largest page anyone
publishes. The set only ever had to span one poll window plus the resync
guard, not all of history.
Dedupe behaviour is unchanged. The comment at fresh() explains why the set
does not survive a restart; it never bounded it within one run.
A non-2xx reply was read in full first and only then rejected. Two costs
followed. A 500 with a large error page pulled up to MaxBytes off the wire for
nothing. An error page over the cap returned ErrTooLarge, which names the size
and hides the status the server actually sent.
The status is a typed error now. webfetch.StatusError carries the code and
unwraps to ErrStatus, so errors.Is keeps working and errors.As reads the number.
crawl.StatusError is the same shape on the other side of the seam, and
cmd/mavend/crawls.go carries the code across.
That removes the string grep in crawl.isServerError, which decided whether a
failed robots.txt blocks a crawl by looking for " 50" in an error message it did
not own. A reworded error would have turned a 503 robots.txt into permission to
crawl. It reads the code now.
Two comments corrected. webfetch.HostMatches said the crawler calls it and
nothing outside the package does. rss.PlainText said the crawler's extractor
goes through it and crawl/extract.go has its own pass.
The rss poller parses the feed straight off the byte slice instead of copying a
document that can run to a megabyte through a string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"что нового?" is a greeting, and the feed matcher claimed it: "нового" was a
feed noun and "что" an ask. With no feeds block, which is what ships, the answer
to hello was "я пока не читаю ленты — они не настроены". A newness word now
needs a named topic or a real feed noun beside it. The topic prepositions lose
"о" for the same class of reason: one rune of filler produced a category of
whatever followed it, and then "по этой теме в лентах пока ничего".
An undated feed was re-noted in full on every boot. Dated items are deduped
against the durable mark, undated ones against a map that dies with the process,
so five items became five more on the next start, stamped now, at the top of the
recent-notes window. A crash loop made that a flood. The mark is now set for an
undated feed too, and its existence marks the first poll after a restart as a
resync: those items are recorded as seen rather than written.
A burst larger than max_items lost its middle. The poll walked the feed
newest-first, stopped at the cap, and marked the newest item written, which put
everything below the cap behind the mark forever. The cap now applies to the
oldest candidates and the mark follows what was written, so max_items paces
instead of dropping.
The category tag was read out loud: "Заголовок [технологии]" went through piper
brackets and all, because the answer path took the whole first line. The tag is
parsed off for reading and is now the only thing a topic is matched against.
Matching the whole note meant "что нового про погоду" hit any tech headline
whose link contained "pogod".
Also: the charset comment on dec.Strict described something Strict does not do,
and a skipped feed is named in the log.
Found in review of #66.
internal/rss parses RSS 2.0 and Atom, and polls each configured feed on its own
interval; internal/webfetch is the one door either of them uses to touch the
network. The poller writes items as notes with source "rss:<feed>" and nothing
else: the answer path reads them back when he asks "что нового в лентах?", and
nothing is announced on arrival. A feed that dispatched would be a nag, which is
why the plan's breaking-news rule was left out rather than built.
webfetch is where the limits live, as code rather than a paragraph: http(s)
only, an allowlist (the configured feeds' hosts) and a denylist, a 2 MiB body
cap, a 3-redirect cap, one request per host per second, and a refusal to connect
to any private address — checked in the dialer's Control hook so it holds for
every resolved address and every redirect hop, not just for a literal IP.
Off unless configured: no "feeds" block, no poller, no outbound request. How far
a feed was read is a config fact (rss:latest:<name>), so a restart does not
re-note yesterday's headlines.
claude
kami