Read RSS and Atom feeds, and speak about them only when asked (#258)
internal/rss parses RSS 2.0 and Atom, and polls each configured feed on its own interval; internal/webfetch is the one door either of them uses to touch the network. The poller writes items as notes with source "rss:<feed>" and nothing else: the answer path reads them back when he asks "что нового в лентах?", and nothing is announced on arrival. A feed that dispatched would be a nag, which is why the plan's breaking-news rule was left out rather than built. webfetch is where the limits live, as code rather than a paragraph: http(s) only, an allowlist (the configured feeds' hosts) and a denylist, a 2 MiB body cap, a 3-redirect cap, one request per host per second, and a refusal to connect to any private address — checked in the dialer's Control hook so it holds for every resolved address and every redirect hop, not just for a literal IP. Off unless configured: no "feeds" block, no poller, no outbound request. How far a feed was read is a config fact (rss:latest:<name>), so a restart does not re-note yesterday's headlines.
This commit is contained in:
@@ -36,6 +36,37 @@ present (see `.dockerignore`).
|
||||
| `/var/lib/maven` (volume) | encrypted db at rest |
|
||||
| `/dev/shm` (tmpfs) | decrypted db working copy (RAM only) |
|
||||
|
||||
## Reading the outside world (off by default)
|
||||
|
||||
`mavend.json` ships without a `feeds` block, which means no RSS/Atom feed is
|
||||
fetched and no outbound request is made. Switching it on is adding the block:
|
||||
|
||||
```json
|
||||
"feeds": {
|
||||
"poll_interval": "30m",
|
||||
"max_items": 5,
|
||||
"max_age": "24h",
|
||||
"sources": [
|
||||
{ "name": "habr", "url": "https://habr.com/ru/rss/best/daily/",
|
||||
"category": "технологии", "exclude": ["реклама"] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
What it does and does not do:
|
||||
|
||||
- items are written as notes with source `rss:<name>`, visible on `/dash`;
|
||||
- **nothing is announced.** She reads them back when asked — "что нового в
|
||||
лентах?", "что нового по технологиям?" — and never on arrival. There is no
|
||||
severity or channel knob here on purpose;
|
||||
- the fetcher is allowlisted to the hosts of the configured feeds, plus any
|
||||
`allow_hosts`. It refuses non-http(s) schemes and every private address
|
||||
(loopback, the LAN, the `10.42.0.0/24` wg range, cloud metadata). It caps the
|
||||
response at 2 MiB and redirects at 3, and makes at most one request per host
|
||||
per second. See `internal/webfetch`;
|
||||
- how far each feed was read is stored as a config fact `rss:latest:<name>`, so
|
||||
a restart does not re-note yesterday's headlines.
|
||||
|
||||
## Not yet verified / host-dependent
|
||||
|
||||
This stack is correct-by-construction but has **not been build-tested here**
|
||||
|
||||
Reference in New Issue
Block a user