kami
5fe8f228c1
Add a read-only /ecosystem page that consumes the sibling services' JSON APIs (Nexus entities, Praxis attention, Hexis capabilities), fetched concurrently with honest per-panel error states. Siblings stay headless — mavweb is their human surface (arch §16). Wired via mavweb -nexus/-praxis/-hexis flags; mavweb joins the ecosystem compose network. Fix mobile horizontal overflow across all pages: .content is a flex child with default min-width:auto, so it refused to shrink below the tables' intrinsic width. min-width:0 lets wide tables pan inside .scroll instead of dragging the page sideways. Verified via CDP geometry check (scrollWidth === clientWidth at 430px). Also includes in-progress Ethos UI redesign, ecosystem deploy compose, and planning docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
31 lines
2.5 KiB
Markdown
31 lines
2.5 KiB
Markdown
# Plan: Web Crawler
|
|
|
|
**Goal:** Maven can crawl web pages on demand or on a schedule — fetch page content, extract structured data (via LLM or CSS selectors), and store results as facts, notes, or reminders. Used for: price monitoring, documentation updates, recipe extraction, content summarization.
|
|
|
|
**Done when:**
|
|
- `internal/crawl/` package — HTTP fetcher with polite defaults (rate limiting, robots.txt respect, user-agent)
|
|
- Content extraction: HTML→plaintext (Go stdlib `golang.org/x/net/html`), or full-page LLM summarization
|
|
- Crawl scheduler in config — `crawls: [{name, url, selector, schedule, store_as}]`
|
|
- On-demand crawl via voice: "maven, посмотри страницу X и запиши цену"
|
|
- Results are stored as facts/notes through `ipc.CoreAPI`
|
|
- Crawl history visible on mavweb `/tools` page
|
|
|
|
**Scope:**
|
|
- New `internal/crawl/` package — fetcher, parser, scheduler, extractor
|
|
- Config extension: `crawls` array in `config.Config`
|
|
- Reuses `internal/llm.Client` for intelligent extraction (e.g., "extract the price, description, and availability from this page")
|
|
- Reuses `internal/ipc.CoreAPI` for storing results
|
|
- Reuses `internal/router.Embedder` for deduplication (don't re-store identical content)
|
|
- Reuses `internal/routine.Routine` mechanics for scheduled crawls
|
|
|
|
**Steps:**
|
|
1. Create `internal/crawl/fetcher.go` — `Fetch(url) ([]byte, error)`: HTTP GET with timeout (30s), rate limiting (1 req/sec), `robots.txt` check via `github.com/temoto/robotstxt`
|
|
2. Create `internal/crawl/extractor.go` — `Extract(html []byte, extraction_type string) (map[string]string, error)`: for simple extraction use CSS selector (`github.com/PuerkitoBio/goquery`); for complex extraction use `llm.Client` with a prompt
|
|
3. Create `internal/crawl/scheduler.go` — `Scheduler` that reads `crawls` config, runs each on its cron schedule, tracks last-run via facts
|
|
4. Create `internal/crawl/dedup.go` — compute content hash, skip if identical to last fetched (stored as fact `kind=config, key=crawl:hash:<name>`)
|
|
5. Wire on-demand crawl into `IntentAct` — new tool verb `crawl` that accepts a URL argument
|
|
6. Wire scheduled crawls into `cmd/mavend/main.go` — separate goroutine manages the crawl scheduler
|
|
7. Add IPC methods `MethodTriggerCrawl(name)`, `MethodListCrawls`, `MethodGetCrawlResult(name)`
|
|
8. Add `crawls` block to `config.Config` and `deploy/mavend.json`
|
|
9. Test with a static HTML page — verify extraction matches expected values, verify scheduling fires correctly
|