kami
2c1b0eede0
The network fallback behind the local sources, off unless configured. internal/crawl is pure: a stdlib robots.txt parser (group specificity, wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a watcher that notes a watched page only when its text changed. It has no store access and no net/http; cmd/mavend/crawls.go is the impure half. Every limit is code and tested: the guarded fetcher from #258 enforces the host allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS rebinding and each redirect hop are covered), caps size and redirects, times out, and spaces requests per host. A robots.txt Disallow is refused with no override. On demand, reading is a query source placed last in the chain, after his memory, his notes, and the local Kiwix ZIMs once those are wired: no URL in the utterance means no fetch, and only the URL ever leaves the box. Scheduled watches write notes and announce nothing. The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers are stdlib. No new dependency.
69 lines
4.5 KiB
Markdown
69 lines
4.5 KiB
Markdown
# Plan: Web Crawler
|
|
|
|
**Goal:** Maven can crawl web pages on demand or on a schedule — fetch page content, extract structured data (via LLM or CSS selectors), and store results as facts, notes, or reminders. Used for: price monitoring, documentation updates, recipe extraction, content summarization.
|
|
|
|
**Done when:**
|
|
- `internal/crawl/` package — HTTP fetcher with polite defaults (rate limiting, robots.txt respect, user-agent)
|
|
- Content extraction: HTML→plaintext (Go stdlib `golang.org/x/net/html`), or full-page LLM summarization
|
|
- Crawl scheduler in config — `crawls: [{name, url, selector, schedule, store_as}]`
|
|
- On-demand crawl via voice: "maven, посмотри страницу X и запиши цену"
|
|
- Results are stored as facts/notes through `ipc.CoreAPI`
|
|
- Crawl history visible on mavweb `/tools` page
|
|
|
|
**Scope:**
|
|
- New `internal/crawl/` package — fetcher, parser, scheduler, extractor
|
|
- Config extension: `crawls` array in `config.Config`
|
|
- Reuses `internal/llm.Client` for intelligent extraction (e.g., "extract the price, description, and availability from this page")
|
|
- Reuses `internal/ipc.CoreAPI` for storing results
|
|
- Reuses `internal/router.Embedder` for deduplication (don't re-store identical content)
|
|
- Reuses `internal/routine.Routine` mechanics for scheduled crawls
|
|
|
|
**Steps:**
|
|
1. Create `internal/crawl/fetcher.go` — `Fetch(url) ([]byte, error)`: HTTP GET with timeout (30s), rate limiting (1 req/sec), `robots.txt` check via `github.com/temoto/robotstxt`
|
|
2. Create `internal/crawl/extractor.go` — `Extract(html []byte, extraction_type string) (map[string]string, error)`: for simple extraction use CSS selector (`github.com/PuerkitoBio/goquery`); for complex extraction use `llm.Client` with a prompt
|
|
3. Create `internal/crawl/scheduler.go` — `Scheduler` that reads `crawls` config, runs each on its cron schedule, tracks last-run via facts
|
|
4. Create `internal/crawl/dedup.go` — compute content hash, skip if identical to last fetched (stored as fact `kind=config, key=crawl:hash:<name>`)
|
|
5. Wire on-demand crawl into `IntentAct` — new tool verb `crawl` that accepts a URL argument
|
|
6. Wire scheduled crawls into `cmd/mavend/main.go` — separate goroutine manages the crawl scheduler
|
|
7. Add IPC methods `MethodTriggerCrawl(name)`, `MethodListCrawls`, `MethodGetCrawlResult(name)`
|
|
8. Add `crawls` block to `config.Config` and `deploy/mavend.json`
|
|
9. Test with a static HTML page — verify extraction matches expected values, verify scheduling fires correctly
|
|
|
|
## Shipped 2026-08-01 (#259)
|
|
|
|
Built as `internal/crawl` (pure: robots, extraction, watcher) plus
|
|
`cmd/mavend/crawls.go` (fetcher, ticker, dedup facts), on top of the guarded
|
|
`internal/webfetch` door added with the feed reader (#258). Off unless
|
|
configured, in two separately-switched halves: `crawl.on_demand` for a URL he
|
|
names, `crawl.watches` for a scheduled re-read.
|
|
|
|
**Limits are code, not documentation** (`internal/webfetch`, tested one test per
|
|
limit): host allowlist/denylist, no private addresses (loopback, RFC1918 —
|
|
hence the LAN and the `10.42.0.0/24` wg range —, link-local incl. cloud
|
|
metadata, CGNAT, v6 ULA) enforced in the dialer's `Control` hook so DNS
|
|
rebinding and every redirect hop are covered, response size cap, redirect cap,
|
|
timeout, one request per host per second. `robots.txt` is fetched first, cached
|
|
per host, and a `Disallow` is refused with no override.
|
|
|
|
Deliberate deviations from the plan above:
|
|
|
|
- **No CSS selectors and no LLM structured extraction** (steps 2). The output is
|
|
plaintext handed to the phraser as context for the question he asked. A 1.7B
|
|
extracting a JSON price table from 4000 runes is a worse bet than reading, and
|
|
`goquery` is not vendored.
|
|
- **No `crawl` act verb and no new IPC methods** (steps 5, 7). Reading a page is
|
|
a query source (`queryWeb` in `actions_query.go`, last in the chain, behind
|
|
Kiwix once that is wired), not an action he commands. Nothing needs a new wire
|
|
method to work.
|
|
- **Notes, not facts.** A page's text is not a fact about him. Only the dedup
|
|
hash is a fact (`crawl:hash:<name>`, kind `config`, source `poll:crawl`).
|
|
- **Nothing is dispatched.** A changed page writes a note; it does not nudge.
|
|
Not a nag.
|
|
- **No `/tools` crawl history page.** The notes and the hash facts are already
|
|
visible on `/dash`.
|
|
|
|
**No new dependency.** The vendored tree has no `x/net/html`, no `goquery` and
|
|
no `temoto/robotstxt`, so robots parsing and HTML-to-text are stdlib
|
|
(`regexp`, `html`) — RE2 has no backreferences, hence the `pairsRE` builder in
|
|
`extract.go`.
|