Read a web page when he names one, and watch a few on a timer (#259) #67

2026-08-01T01:40:47+02:00

claude commented

2026-08-01 01:40:47 +02:00

What

Reading the outside world, part 2: a web crawler that is the network fallback behind the local sources, off unless configured.

internal/crawl (pure): stdlib robots.txt parser (group specificity, */$ wildcards, Crawl-delay, per-host cache with TTL), HTML-to-plaintext extraction, and a watcher that writes a note only when a watched page's text changed.
cmd/mavend/crawls.go (impure): the guarded fetcher, the 15m ticker, and the dedup hashes stored as config facts crawl:hash:<name>.
queryWeb in the existing query-source chain, placed last — after his memory, his notes, and the local Kiwix ZIMs once those are wired into the chain.
router.FirstURL — strict URL detection (scheme required), so a dotted word is never an outbound request.
Config: crawl block with two separately-switched halves, on_demand and watches. No block ⇒ nil crawler ⇒ nothing is ever fetched.

Why

CLAUDE.md: "never phones home" is deprecated, but local sources come first, external fetching is off unless configured, and his notes and facts are never sent outward. Only the URL leaves the box — no persona block, no history, no matched notes. Nothing is announced: a changed page is a note she reads when asked, not a nudge. Not a nag.

Limits on a homelab crawler are code, not documentation. The fetcher (internal/webfetch, added in #258) enforces a host allowlist/denylist, refuses private addresses in the dialer's Control hook so DNS rebinding and every redirect hop are covered (loopback, RFC1918 incl. the LAN and the 10.42.0.0/24 wg range, link-local incl. cloud metadata, CGNAT, v6 ULA), caps the response size and the redirect count, times out, and spaces requests per host. A robots.txt Disallow is refused with no override.

Deliberate deviations from docs/plans/14-web-crawler.md, documented in a Shipped section there: no CSS selectors or LLM structured extraction (plaintext as phraser context instead), no crawl act verb and no new IPC methods (reading is a query source, not a command), notes rather than facts, no dispatch, no /tools page (notes and hash facts are already on /dash). No new dependency — the vendored tree has no x/net/html, goquery or temoto/robotstxt.

How verified

make build and make test both exit 0.

New tests: robots group selection and Crawl-delay, wildcards, shared groups, cache TTL, missing robots allows, explicit Disallow refuses; extraction (title kept, script/style/nav/footer dropped, rune cap); fetcher refusal not swallowed; non-HTML refused; watcher notes a changed page, skips an unchanged one, respects the interval, survives a dead page, and its hash survives a restart; crawl.New(nil, …) == nil. Wiring tests: off by default, a watch list is not on-demand permission, the wired fetcher refuses a loopback URL, factHashes round trip, and every queryWeb reply including robots and not-configured.

Vikunja #259

## What Reading the outside world, part 2: a web crawler that is the network fallback **behind** the local sources, off unless configured. - `internal/crawl` (pure): stdlib `robots.txt` parser (group specificity, `*`/`$` wildcards, `Crawl-delay`, per-host cache with TTL), HTML-to-plaintext extraction, and a watcher that writes a note only when a watched page's text changed. - `cmd/mavend/crawls.go` (impure): the guarded fetcher, the 15m ticker, and the dedup hashes stored as config facts `crawl:hash:<name>`. - `queryWeb` in the existing query-source chain, placed **last** — after his memory, his notes, and the local Kiwix ZIMs once those are wired into the chain. - `router.FirstURL` — strict URL detection (scheme required), so a dotted word is never an outbound request. - Config: `crawl` block with two separately-switched halves, `on_demand` and `watches`. No block ⇒ nil crawler ⇒ nothing is ever fetched. ## Why `CLAUDE.md`: "never phones home" is deprecated, but local sources come first, external fetching is off unless configured, and his notes and facts are never sent outward. Only the URL leaves the box — no persona block, no history, no matched notes. Nothing is announced: a changed page is a note she reads when asked, not a nudge. Not a nag. Limits on a homelab crawler are code, not documentation. The fetcher (`internal/webfetch`, added in #258) enforces a host allowlist/denylist, refuses private addresses in the dialer's `Control` hook so DNS rebinding and every redirect hop are covered (loopback, RFC1918 incl. the LAN and the `10.42.0.0/24` wg range, link-local incl. cloud metadata, CGNAT, v6 ULA), caps the response size and the redirect count, times out, and spaces requests per host. A `robots.txt` `Disallow` is refused with no override. Deliberate deviations from `docs/plans/14-web-crawler.md`, documented in a Shipped section there: no CSS selectors or LLM structured extraction (plaintext as phraser context instead), no `crawl` act verb and no new IPC methods (reading is a query source, not a command), notes rather than facts, no dispatch, no `/tools` page (notes and hash facts are already on `/dash`). No new dependency — the vendored tree has no `x/net/html`, `goquery` or `temoto/robotstxt`. ## How verified `make build` and `make test` both exit 0. New tests: robots group selection and `Crawl-delay`, wildcards, shared groups, cache TTL, missing robots allows, explicit `Disallow` refuses; extraction (title kept, script/style/nav/footer dropped, rune cap); fetcher refusal not swallowed; non-HTML refused; watcher notes a changed page, skips an unchanged one, respects the interval, survives a dead page, and its hash survives a restart; `crawl.New(nil, …) == nil`. Wiring tests: off by default, a watch list is not on-demand permission, the wired fetcher refuses a loopback URL, `factHashes` round trip, and every `queryWeb` reply including robots and not-configured. Vikunja #259

claude added 1 commit 2026-08-01 01:40:48 +02:00

Read a web page when he names one, and watch a few on a timer (#259 ) 2c1b0eede0

The network fallback behind the local sources, off unless configured.

internal/crawl is pure: a stdlib robots.txt parser (group specificity,
wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a
watcher that notes a watched page only when its text changed. It has no store
access and no net/http; cmd/mavend/crawls.go is the impure half.

Every limit is code and tested: the guarded fetcher from #258 enforces the host
allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS
rebinding and each redirect hop are covered), caps size and redirects, times out,
and spaces requests per host. A robots.txt Disallow is refused with no override.

On demand, reading is a query source placed last in the chain, after his memory,
his notes, and the local Kiwix ZIMs once those are wired: no URL in the
utterance means no fetch, and only the URL ever leaves the box. Scheduled
watches write notes and announce nothing.

The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers
are stdlib. No new dependency.

claude reviewed 2026-08-01 11:32:11 +02:00

claude left a comment

The two halves being separately switched is the right split. Watching a fixed list on a timer and reading whatever URL turns up in an utterance are different permissions, and the config says so. FirstURL requiring a scheme is the correct strictness. A bare dotted word is a version string as often as a host, and a false positive here is an outbound request nobody asked for. Robots with no override, and Hash over extracted text rather than raw HTML so a rotating ad slot is not news. crawl cannot reach the store. That makes "only the URL leaves the box" a property of the package graph, not a promise in a comment.

Four things.

1. A watch silently becomes an allowlist for on-demand reading. newCrawler seeds hosts from cc.AllowHosts and then appends every watched page's host. webfetch treats a non-empty AllowHosts as "these and nothing else". So this config:

"crawl": { "on_demand": true, "watches": [ { "name": "changelog", "url": "https://example.org/changelog" } ] }

sets allow_hosts to ["example.org"] without him writing it. He then pastes a wikipedia link and gets "не получилось прочитать страницу." forever. The log that warns about the allowlist is guarded on len(cc.AllowHosts) > 0, which is exactly the case where he already knows. The two lists want to be separate fetchers, or the watch hosts should be appended only when cc.AllowHosts is already non-empty.

2. Crawl-delay is parsed, stored, and never read. ParseRobots fills Rules.Delay. Rules carries it under a comment about the fetcher rate limit being the floor. The only reader in the tree is robots_test.go. allowed returns a bool and throws the rules away. deploy/README.md states "Crawl-delay is honoured". A site asking for 30 seconds gets the fetcher's flat one second per host. Either wire it into waitTurn through a per-host override, or delete the field and the README line. Claiming politeness that is not implemented is worse than not claiming it.

3. Crawled text shares a recall space with his own notes. Nothing at read time tells them apart. Watcher.Check embeds NoteText and writes it through WriteNote. queryNotes does QueryNotes(ctx, t.vec, 5) over everything, checks ConfidentScores, and hands the raw texts to PhraseQuery. Nothing filters or labels by Source. So "что я говорил про переезд" can be answered from a stranger's page that lands near in the vector space. The fallback line is "вот что я нашла: " plus that text. PR 66 opened this with rss: notes, and this PR widens it to arbitrary page text he never chose word by word. The recall path already knows about provenance elsewhere. Either exclude rss: and crawl: from QueryNotes by default, or pass the source down so she says where it came from.

4. The package doc describes an ordering the chain does not have. crawl.go says the order is "the resident model, then his own memory, then the Kiwix ZIMs on the box, and only then the network". actions_query.go says the model "gets its turn before anything touches the network". In querySources, web sits immediately before general-knowledge. The model's own answer comes after the fetch, not before it. Kiwix is not in the chain at all yet, so the ZIM half of the claim describes nothing. The position may well be the one you want, since a named URL is an explicit instruction. The comment should then say "after his memory and his notes, before the model guesses", which is what the code does.

Smaller notes:

With no crawl block, the default, any query utterance holding an http:// gets "я не читаю страницы, это не настроено." Claiming the turn is right when the capability exists and failed. For one never enabled, falling through to the model is the older behaviour and says nothing about configuration.
isFatalFetchError matches three substrings of webfetch's error text from a package that deliberately does not import it. There is no test binding the two. If a message is reworded, a blocked host quietly stops being fatal at the robots step. The adapter in crawls.go imports both packages. It can map the sentinels to a crawl error and keep string matching out of the pure package.
The on-demand budget is 30 seconds and covers two sequential fetches, each with webfetch.DefaultTimeout at 20 seconds, plus the per-host one-second wait. A slow robots.txt eats the page's budget and he hears "не получилось прочитать страницу." Give the robots fetch its own shorter deadline.
A robots.txt that returns 500 is treated the same as a 404, so a site with a broken server is crawled. The standard asks for the opposite. webfetch collapses both into an error, so the crawler cannot currently tell them apart.
ParseRobots picks the last group whose agent is a prefix of ours, but the comment says the most specific match wins. Two candidate groups pick by file order, not by length.
Extracted page text goes into PhraseQuery beside his question. For watches it becomes a note, and later context for other questions. That text is written by whoever owns the page. It cannot reach a tool, but it can steer what she says. Worth a sentence in the package doc saying so, since every other trust boundary in this PR is documented.

The two halves being separately switched is the right split. Watching a fixed list on a timer and reading whatever URL turns up in an utterance are different permissions, and the config says so. `FirstURL` requiring a scheme is the correct strictness. A bare dotted word is a version string as often as a host, and a false positive here is an outbound request nobody asked for. Robots with no override, and `Hash` over extracted text rather than raw HTML so a rotating ad slot is not news. `crawl` cannot reach the store. That makes "only the URL leaves the box" a property of the package graph, not a promise in a comment. Four things. **1. A watch silently becomes an allowlist for on-demand reading.** `newCrawler` seeds `hosts` from `cc.AllowHosts` and then appends every watched page's host. `webfetch` treats a non-empty `AllowHosts` as "these and nothing else". So this config: ```json "crawl": { "on_demand": true, "watches": [ { "name": "changelog", "url": "https://example.org/changelog" } ] } ``` sets `allow_hosts` to `["example.org"]` without him writing it. He then pastes a wikipedia link and gets "не получилось прочитать страницу." forever. The log that warns about the allowlist is guarded on `len(cc.AllowHosts) > 0`, which is exactly the case where he already knows. The two lists want to be separate fetchers, or the watch hosts should be appended only when `cc.AllowHosts` is already non-empty. **2. `Crawl-delay` is parsed, stored, and never read.** `ParseRobots` fills `Rules.Delay`. `Rules` carries it under a comment about the fetcher rate limit being the floor. The only reader in the tree is `robots_test.go`. `allowed` returns a bool and throws the rules away. `deploy/README.md` states "`Crawl-delay` is honoured". A site asking for 30 seconds gets the fetcher's flat one second per host. Either wire it into `waitTurn` through a per-host override, or delete the field and the README line. Claiming politeness that is not implemented is worse than not claiming it. **3. Crawled text shares a recall space with his own notes.** Nothing at read time tells them apart. `Watcher.Check` embeds `NoteText` and writes it through `WriteNote`. `queryNotes` does `QueryNotes(ctx, t.vec, 5)` over everything, checks `ConfidentScores`, and hands the raw texts to `PhraseQuery`. Nothing filters or labels by `Source`. So "что я говорил про переезд" can be answered from a stranger's page that lands near in the vector space. The fallback line is "вот что я нашла: " plus that text. PR 66 opened this with `rss:` notes, and this PR widens it to arbitrary page text he never chose word by word. The recall path already knows about provenance elsewhere. Either exclude `rss:` and `crawl:` from `QueryNotes` by default, or pass the source down so she says where it came from. **4. The package doc describes an ordering the chain does not have.** `crawl.go` says the order is "the resident model, then his own memory, then the Kiwix ZIMs on the box, and only then the network". `actions_query.go` says the model "gets its turn before anything touches the network". In `querySources`, `web` sits immediately before `general-knowledge`. The model's own answer comes after the fetch, not before it. Kiwix is not in the chain at all yet, so the ZIM half of the claim describes nothing. The position may well be the one you want, since a named URL is an explicit instruction. The comment should then say "after his memory and his notes, before the model guesses", which is what the code does. Smaller notes: - With no `crawl` block, the default, any query utterance holding an `http://` gets "я не читаю страницы, это не настроено." Claiming the turn is right when the capability exists and failed. For one never enabled, falling through to the model is the older behaviour and says nothing about configuration. - `isFatalFetchError` matches three substrings of `webfetch`'s error text from a package that deliberately does not import it. There is no test binding the two. If a message is reworded, a blocked host quietly stops being fatal at the robots step. The adapter in `crawls.go` imports both packages. It can map the sentinels to a `crawl` error and keep string matching out of the pure package. - The on-demand budget is 30 seconds and covers two sequential fetches, each with `webfetch.DefaultTimeout` at 20 seconds, plus the per-host one-second wait. A slow robots.txt eats the page's budget and he hears "не получилось прочитать страницу." Give the robots fetch its own shorter deadline. - A robots.txt that returns 500 is treated the same as a 404, so a site with a broken server is crawled. The standard asks for the opposite. `webfetch` collapses both into an error, so the crawler cannot currently tell them apart. - `ParseRobots` picks the last group whose agent is a prefix of ours, but the comment says the most specific match wins. Two candidate groups pick by file order, not by length. - Extracted page text goes into `PhraseQuery` beside his question. For watches it becomes a note, and later context for other questions. That text is written by whoever owns the page. It cannot reach a tool, but it can steer what she says. Worth a sentence in the package doc saying so, since every other trust boundary in this PR is documented.

kami referenced this issue from a commit

2026-08-01 14:43:56 +02:00

store: keep what she read out of what he said

kami referenced this issue from a commit

2026-08-01 14:43:56 +02:00

crawl: stop letting a watch widen on-demand reading, and honour Crawl-delay

kami closed this pull request

2026-08-01 14:51:52 +02:00

kami commented

2026-08-01 14:52:46 +02:00

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#67