Read RSS and Atom feeds, and speak about them only when asked (#258) #66

2026-08-01T01:28:13+02:00

claude commented

2026-08-01 01:28:13 +02:00

Vikunja #258 — docs/plans/13-rss-news-feeds.md.

What changed

internal/webfetch — the one door Maven uses to read something off the network, and the reason a fetcher is allowed on this box at all. http(s) only; an allowlist and a denylist (the denylist wins); a 2 MiB body cap; a 3-redirect cap, re-checked against the host lists on every hop; one request per host per second; and a refusal to connect to any private address (loopback, RFC1918 — which covers the 10.42.0.0/24 wg tunnel and the 192.168.1.0/24 LAN — link-local incl. 169.254.169.254, CGNAT, v6 unique-local). The address check lives in the dialer's Control hook, so it sees every address the resolver returns and every redirect hop; a pre-flight lookup would leave a DNS-rebinding window open.
internal/rss — RSS 2.0 + Atom parser (stdlib encoding/xml, no new dependency) and a poller: per-feed interval, per-feed include/exclude keyword filter, MaxItems cap, cold-start MaxAge so switching a feed on imports today rather than the archive, and durable marks so a restart does not re-note yesterday's headlines.
cmd/mavend/feeds.go — ticker goroutine, the webfetch adapter, and marks stored as config facts (rss:latest:<feed>). In-core rather than its own daemon: mavpoll and mavmaild are separate processes because they hold a credential core must never see, and a feed URL is not a secret. The other half of that precedent, off unless configured, is kept.
feeds config block + the feeds query source (router.ParseFeedQuery), placed before the recall sources so "что нового?" is answered from the feeds instead of by the model inventing a bulletin.

Why it does not speak

Items are written as notes with source rss:<feed>, and that is the only output. Nothing dispatches, nudges or announces — she reads headlines back when asked ("что нового в лентах?", "что нового по технологиям?"). The plan's step 5 (breaking-news sev3 nudges) was deliberately not built: a feed that announces itself is a nag. Step 3 (embedder relevance) is a seam (rss.Ranker) wired nil, because scoring against an "interest profile" needs a profile that does not exist yet.

"ленты не настроены" and "в лентах пока ничего нового" are separate replies: they are different truths, and neither is delegated to the model.

Verification

make build and make test (go test -race) both green. New tests: every webfetch limit has one that fails loudly if the limit is removed (private literals, private DNS resolution, schemes, allow/deny, size, redirect re-check across hops, rate-limit spacing and its context handling); feed parsing for both dialects incl. undated items and garbage; poller dedup, mark-survives-restart, interval, cold start, caps, filter, ranker, one-dead-feed-does-not-abort; and the daemon side — query source by category, off-vs-empty replies, fact-mark round trip, newFeedWorker nil without config, and a poll of http://127.0.0.1:9100 through the wired fetcher failing.

Stock deploy/mavend.json has no feeds block, so nothing changes until Kami adds one. deploy/README.md documents it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX

Vikunja #258 — docs/plans/13-rss-news-feeds.md. ## What changed - **`internal/webfetch`** — the one door Maven uses to read something off the network, and the reason a fetcher is allowed on this box at all. http(s) only; an allowlist and a denylist (the denylist wins); a 2 MiB body cap; a 3-redirect cap, re-checked against the host lists on every hop; one request per host per second; and a refusal to connect to any private address (loopback, RFC1918 — which covers the `10.42.0.0/24` wg tunnel and the `192.168.1.0/24` LAN — link-local incl. `169.254.169.254`, CGNAT, v6 unique-local). The address check lives in the dialer's `Control` hook, so it sees every address the resolver returns and every redirect hop; a pre-flight lookup would leave a DNS-rebinding window open. - **`internal/rss`** — RSS 2.0 + Atom parser (stdlib `encoding/xml`, no new dependency) and a poller: per-feed interval, per-feed include/exclude keyword filter, `MaxItems` cap, cold-start `MaxAge` so switching a feed on imports today rather than the archive, and durable marks so a restart does not re-note yesterday's headlines. - **`cmd/mavend/feeds.go`** — ticker goroutine, the webfetch adapter, and marks stored as config facts (`rss:latest:<feed>`). In-core rather than its own daemon: mavpoll and mavmaild are separate processes because they hold a credential core must never see, and a feed URL is not a secret. The other half of that precedent, off unless configured, is kept. - **`feeds` config block** + the `feeds` query source (`router.ParseFeedQuery`), placed before the recall sources so "что нового?" is answered from the feeds instead of by the model inventing a bulletin. ## Why it does not speak Items are written as notes with source `rss:<feed>`, and that is the only output. Nothing dispatches, nudges or announces — she reads headlines back when asked ("что нового в лентах?", "что нового по технологиям?"). The plan's step 5 (breaking-news sev3 nudges) was deliberately **not** built: a feed that announces itself is a nag. Step 3 (embedder relevance) is a seam (`rss.Ranker`) wired nil, because scoring against an "interest profile" needs a profile that does not exist yet. "ленты не настроены" and "в лентах пока ничего нового" are separate replies: they are different truths, and neither is delegated to the model. ## Verification `make build` and `make test` (go test -race) both green. New tests: every webfetch limit has one that fails loudly if the limit is removed (private literals, private DNS resolution, schemes, allow/deny, size, redirect re-check across hops, rate-limit spacing and its context handling); feed parsing for both dialects incl. undated items and garbage; poller dedup, mark-survives-restart, interval, cold start, caps, filter, ranker, one-dead-feed-does-not-abort; and the daemon side — query source by category, off-vs-empty replies, fact-mark round trip, `newFeedWorker` nil without config, and a poll of `http://127.0.0.1:9100` through the wired fetcher failing. Stock `deploy/mavend.json` has no `feeds` block, so nothing changes until Kami adds one. `deploy/README.md` documents it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX

claude added 1 commit 2026-08-01 01:28:14 +02:00

Read RSS and Atom feeds, and speak about them only when asked (#258 ) cb3641e7bb

internal/rss parses RSS 2.0 and Atom, and polls each configured feed on its own
interval; internal/webfetch is the one door either of them uses to touch the
network. The poller writes items as notes with source "rss:<feed>" and nothing
else: the answer path reads them back when he asks "что нового в лентах?", and
nothing is announced on arrival. A feed that dispatched would be a nag, which is
why the plan's breaking-news rule was left out rather than built.

webfetch is where the limits live, as code rather than a paragraph: http(s)
only, an allowlist (the configured feeds' hosts) and a denylist, a 2 MiB body
cap, a 3-redirect cap, one request per host per second, and a refusal to connect
to any private address — checked in the dialer's Control hook so it holds for
every resolved address and every redirect hop, not just for a literal IP.

Off unless configured: no "feeds" block, no poller, no outbound request. How far
a feed was read is a config fact (rss:latest:<name>), so a restart does not
re-note yesterday's headlines.

claude reviewed 2026-08-01 11:30:06 +02:00

claude left a comment

The discipline is the good part. A feed reader is the most tempting way to break "not a nag". This one has no dispatch path at all: the poller writes notes, the answer path reads them, and the package doc says why. webfetch checks the private address in the dialer's Control hook rather than a pre-flight lookup. That is the version that survives DNS rebinding and every redirect hop. The Ranker seam is left nil, with a comment saying that a threshold over an embedder with no profile is a random filter with a confident name. That is the honest call. EmbedPassage rather than Embed for stored items is the detail that usually gets missed.

Four things.

1. "что нового?" is now answered by a capability that is off by default. ParseFeedQuery needs one feed noun and one ask marker. нового is a feed noun and что is an ask marker, so the bare two-word utterance matches. The feeds source sits above calendar, weather, embed and recall. Feeds are absent from deploy/mavend.json, so feedsOn is false. Today's shipping answer to "что нового?" is "я пока не читаю ленты — они не настроены." That is the most common opener in the language, and she replies with a configuration status. The source claims the turn deliberately. The reason given is sound for "что нового в лентах?" and does not extend to the bare form. Require the noun and the ask to be different tokens. Or require лента/новости/feed when nothing else in the utterance narrows it. The same shape shows up with feeds on and quiet: "в лентах пока ничего нового" as a reply to a greeting.

2. An undated feed is re-noted on every restart, and deploy/README.md says it is not. fresh has two rules. Dated items compare against the durable mark. Undated ones are deduped through p.seen, a map on the Poller that dies with the process. The comment names the trade ("IDs alone lose their memory on restart") and then takes the losing side with no backstop. Walked through: a feed with no pubDate on its items, max_items 5. Boot, five notes written, ts = now because Published is zero. Restart, PollDue runs immediately by design, the same five items are unseen again, five more notes with a fresh now. A crash loop turns that into a note flood. Every copy is timestamped later than the last, so the copies sit at the top of RecentNotes and crowd out real notes inside feedNoteWindow. The README states that the mark fact means "a restart does not re-note yesterday's headlines". True for dated feeds only. Either persist the seen IDs next to the mark, or advance the mark to now for a feed that produced only undated items.

3. A burst larger than max_items loses the middle permanently. PollFeed breaks at written >= p.cfg.MaxItems, then sets the mark to the newest item it wrote. Standard RSS is newest-first, so the five it writes are the five newest, and the mark jumps to the newest of all. Say twenty new items arrive between polls. Items 6 through 20 are then older than the mark, and they are dropped on the next poll and forever. max_items reads as a pacing knob in the config doc ("не завали мне /dash"). It is a drop. Advance the mark to the oldest item written instead of the newest and it becomes pacing. That costs a re-scan, which the filter and the dedup already make cheap.

4. The category tag is read aloud. NoteText writes Title [технологии] on the first line so CategoryMatches has something to match. queryFeeds takes everything up to the first newline as the headline. The spoken answer is "вот что нового: Заголовок [технологии]; ...", brackets and all, through piper. Match the category against a separate stored field, or strip a trailing [...] before reading out.

Smaller notes:

feedNoteWindow is 200 notes, not 200 feed notes. After a busy day of voice notes the newest feed item falls outside the window. She then says "в лентах пока ничего нового" while the poller is working fine. This is the same seam as the memeval dedupe window on PR 55. A source filter in the store query would settle both.
CategoryMatches substring-matches the 5-rune stem against the whole note text, which includes the summary and the link. "что нового про погоду" matches any tech headline whose URL contains pogod. Matching only the bracketed tag would make it exact.
feedCategory treats о as a topic preposition. Any utterance with a stray о yields a category of whatever token follows. Finding nothing then produces "по этой теме в лентах пока ничего" for a question that named no theme.
Parse sets dec.Strict = false under a comment about charsets. Strict does not change how charsets are read. A windows-1251 feed still fails on the nil CharsetReader, which is the behaviour the comment wants. What Strict = false buys is tolerance of unclosed tags. The outcome is right, the reason given is not.
NewPoller logs "configured but nothing pollable" when it returns nil. applyDefaults already normalises an empty sources list to a nil block. So that log fires only when every source lacks a name or a URL. Neither it nor the per-skip log says which source.

The discipline is the good part. A feed reader is the most tempting way to break "not a nag". This one has no dispatch path at all: the poller writes notes, the answer path reads them, and the package doc says why. `webfetch` checks the private address in the dialer's `Control` hook rather than a pre-flight lookup. That is the version that survives DNS rebinding and every redirect hop. The `Ranker` seam is left nil, with a comment saying that a threshold over an embedder with no profile is a random filter with a confident name. That is the honest call. `EmbedPassage` rather than `Embed` for stored items is the detail that usually gets missed. Four things. **1. "что нового?" is now answered by a capability that is off by default.** `ParseFeedQuery` needs one feed noun and one ask marker. `нового` is a feed noun and `что` is an ask marker, so the bare two-word utterance matches. The `feeds` source sits above `calendar`, `weather`, `embed` and recall. Feeds are absent from `deploy/mavend.json`, so `feedsOn` is false. Today's shipping answer to "что нового?" is "я пока не читаю ленты — они не настроены." That is the most common opener in the language, and she replies with a configuration status. The source claims the turn deliberately. The reason given is sound for "что нового в лентах?" and does not extend to the bare form. Require the noun and the ask to be different tokens. Or require `лента`/`новости`/`feed` when nothing else in the utterance narrows it. The same shape shows up with feeds on and quiet: "в лентах пока ничего нового" as a reply to a greeting. **2. An undated feed is re-noted on every restart, and `deploy/README.md` says it is not.** `fresh` has two rules. Dated items compare against the durable mark. Undated ones are deduped through `p.seen`, a map on the `Poller` that dies with the process. The comment names the trade ("IDs alone lose their memory on restart") and then takes the losing side with no backstop. Walked through: a feed with no `pubDate` on its items, `max_items` 5. Boot, five notes written, `ts = now` because `Published` is zero. Restart, `PollDue` runs immediately by design, the same five items are unseen again, five more notes with a fresh `now`. A crash loop turns that into a note flood. Every copy is timestamped later than the last, so the copies sit at the top of `RecentNotes` and crowd out real notes inside `feedNoteWindow`. The README states that the mark fact means "a restart does not re-note yesterday's headlines". True for dated feeds only. Either persist the seen IDs next to the mark, or advance the mark to `now` for a feed that produced only undated items. **3. A burst larger than `max_items` loses the middle permanently.** `PollFeed` breaks at `written >= p.cfg.MaxItems`, then sets the mark to the newest item it wrote. Standard RSS is newest-first, so the five it writes are the five newest, and the mark jumps to the newest of all. Say twenty new items arrive between polls. Items 6 through 20 are then older than the mark, and they are dropped on the next poll and forever. `max_items` reads as a pacing knob in the config doc ("не завали мне /dash"). It is a drop. Advance the mark to the oldest item written instead of the newest and it becomes pacing. That costs a re-scan, which the filter and the dedup already make cheap. **4. The category tag is read aloud.** `NoteText` writes `Title [технологии]` on the first line so `CategoryMatches` has something to match. `queryFeeds` takes everything up to the first newline as the headline. The spoken answer is "вот что нового: Заголовок [технологии]; ...", brackets and all, through piper. Match the category against a separate stored field, or strip a trailing `[...]` before reading out. Smaller notes: - `feedNoteWindow` is 200 *notes*, not 200 feed notes. After a busy day of voice notes the newest feed item falls outside the window. She then says "в лентах пока ничего нового" while the poller is working fine. This is the same seam as the memeval dedupe window on PR 55. A source filter in the store query would settle both. - `CategoryMatches` substring-matches the 5-rune stem against the whole note text, which includes the summary and the link. "что нового про погоду" matches any tech headline whose URL contains `pogod`. Matching only the bracketed tag would make it exact. - `feedCategory` treats `о` as a topic preposition. Any utterance with a stray `о` yields a category of whatever token follows. Finding nothing then produces "по этой теме в лентах пока ничего" for a question that named no theme. - `Parse` sets `dec.Strict = false` under a comment about charsets. `Strict` does not change how charsets are read. A `windows-1251` feed still fails on the nil `CharsetReader`, which is the behaviour the comment wants. What `Strict = false` buys is tolerance of unclosed tags. The outcome is right, the reason given is not. - `NewPoller` logs "configured but nothing pollable" when it returns nil. `applyDefaults` already normalises an empty `sources` list to a nil block. So that log fires only when every source lacks a name or a URL. Neither it nor the per-skip log says which source.

kami referenced this issue from a commit

2026-08-01 14:43:55 +02:00

rss: stop claiming "что нового" and stop re-noting the same items

kami referenced this issue from a commit

2026-08-01 14:43:55 +02:00

store: keep what she read out of what he said

kami closed this pull request

2026-08-01 14:51:51 +02:00

kami commented

2026-08-01 14:52:46 +02:00

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#66