Maven/docs/plans/14-web-crawler.md

Plan: Web Crawler

Goal: Maven can crawl web pages on demand or on a schedule — fetch page content, extract structured data (via LLM or CSS selectors), and store results as facts, notes, or reminders. Used for: price monitoring, documentation updates, recipe extraction, content summarization.

Done when:

• internal/crawl/ package — HTTP fetcher with polite defaults (rate limiting, robots.txt respect, user-agent)
• Content extraction: HTML→plaintext (Go stdlib golang.org/x/net/html), or full-page LLM summarization
• Crawl scheduler in config — crawls: [{name, url, selector, schedule, store_as}]
• On-demand crawl via voice: "maven, посмотри страницу X и запиши цену"
• Results are stored as facts/notes through ipc.CoreAPI
• Crawl history visible on mavweb /tools page

Scope:

• New internal/crawl/ package — fetcher, parser, scheduler, extractor
• Config extension: crawls array in config.Config
• Reuses internal/llm.Client for intelligent extraction (e.g., "extract the price, description, and availability from this page")
• Reuses internal/ipc.CoreAPI for storing results
• Reuses internal/router.Embedder for deduplication (don't re-store identical content)
• Reuses internal/routine.Routine mechanics for scheduled crawls

Steps:

1. Create internal/crawl/fetcher.go — Fetch(url) ([]byte, error): HTTP GET with timeout (30s), rate limiting (1 req/sec), robots.txt check via github.com/temoto/robotstxt
2. Create internal/crawl/extractor.go — Extract(html []byte, extraction_type string) (map[string]string, error): for simple extraction use CSS selector (github.com/PuerkitoBio/goquery); for complex extraction use llm.Client with a prompt
3. Create internal/crawl/scheduler.go — Scheduler that reads crawls config, runs each on its cron schedule, tracks last-run via facts
4. Create internal/crawl/dedup.go — compute content hash, skip if identical to last fetched (stored as fact kind=config, key=crawl:hash:<name>)
5. Wire on-demand crawl into IntentAct — new tool verb crawl that accepts a URL argument
6. Wire scheduled crawls into cmd/mavend/main.go — separate goroutine manages the crawl scheduler
7. Add IPC methods MethodTriggerCrawl(name), MethodListCrawls, MethodGetCrawlResult(name)
8. Add crawls block to config.Config and deploy/mavend.json
9. Test with a static HTML page — verify extraction matches expected values, verify scheduling fires correctly

Shipped 2026-08-01 (#259)

Built as internal/crawl (pure: robots, extraction, watcher) plus cmd/mavend/crawls.go (fetcher, ticker, dedup facts), on top of the guarded internal/webfetch door added with the feed reader (#258). Off unless configured, in two separately-switched halves: crawl.on_demand for a URL he names, crawl.watches for a scheduled re-read.

Limits are code, not documentation (internal/webfetch, tested one test per limit): host allowlist/denylist, no private addresses (loopback, RFC1918 — hence the LAN and the 10.42.0.0/24 wg range —, link-local incl. cloud metadata, CGNAT, v6 ULA) enforced in the dialer's Control hook so DNS rebinding and every redirect hop are covered, response size cap, redirect cap, timeout, one request per host per second. robots.txt is fetched first, cached per host, and a Disallow is refused with no override.

Deliberate deviations from the plan above:

• No CSS selectors and no LLM structured extraction (steps 2). The output is plaintext handed to the phraser as context for the question he asked. A 1.7B extracting a JSON price table from 4000 runes is a worse bet than reading, and goquery is not vendored.
• No crawl act verb and no new IPC methods (steps 5, 7). Reading a page is a query source (queryWeb in actions_query.go, last in the chain, behind Kiwix once that is wired), not an action he commands. Nothing needs a new wire method to work.
• Notes, not facts. A page's text is not a fact about him. Only the dedup hash is a fact (crawl:hash:<name>, kind config, source poll:crawl).
• Nothing is dispatched. A changed page writes a note; it does not nudge. Not a nag.
• No /tools crawl history page. The notes and the hash facts are already visible on /dash.

No new dependency. The vendored tree has no x/net/html, no goquery and no temoto/robotstxt, so robots parsing and HTML-to-text are stdlib (regexp, html) — RE2 has no backreferences, hence the pairsRE builder in extract.go.