kami/Maven

Files

T

kami 5fe8f228c1 feat(mavweb): /ecosystem page consuming Nexus/Praxis/Hexis + shell fixes

Add a read-only /ecosystem page that consumes the sibling services'
JSON APIs (Nexus entities, Praxis attention, Hexis capabilities),
fetched concurrently with honest per-panel error states. Siblings stay
headless — mavweb is their human surface (arch §16). Wired via mavweb
-nexus/-praxis/-hexis flags; mavweb joins the ecosystem compose network.

Fix mobile horizontal overflow across all pages: .content is a flex
child with default min-width:auto, so it refused to shrink below the
tables' intrinsic width. min-width:0 lets wide tables pan inside .scroll
instead of dragging the page sideways. Verified via CDP geometry check
(scrollWidth === clientWidth at 430px).

Also includes in-progress Ethos UI redesign, ecosystem deploy compose,
and planning docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-19 22:04:23 +04:00

2.5 KiB

Raw Blame History

Plan: Web Crawler

Goal: Maven can crawl web pages on demand or on a schedule — fetch page content, extract structured data (via LLM or CSS selectors), and store results as facts, notes, or reminders. Used for: price monitoring, documentation updates, recipe extraction, content summarization.

Done when:

internal/crawl/ package — HTTP fetcher with polite defaults (rate limiting, robots.txt respect, user-agent)
Content extraction: HTML→plaintext (Go stdlib golang.org/x/net/html), or full-page LLM summarization
Crawl scheduler in config — crawls: [{name, url, selector, schedule, store_as}]
On-demand crawl via voice: "maven, посмотри страницу X и запиши цену"
Results are stored as facts/notes through ipc.CoreAPI
Crawl history visible on mavweb /tools page

Scope:

New internal/crawl/ package — fetcher, parser, scheduler, extractor
Config extension: crawls array in config.Config
Reuses internal/llm.Client for intelligent extraction (e.g., "extract the price, description, and availability from this page")
Reuses internal/ipc.CoreAPI for storing results
Reuses internal/router.Embedder for deduplication (don't re-store identical content)
Reuses internal/routine.Routine mechanics for scheduled crawls

Steps:

Create internal/crawl/fetcher.go — Fetch(url) ([]byte, error): HTTP GET with timeout (30s), rate limiting (1 req/sec), robots.txt check via github.com/temoto/robotstxt
Create internal/crawl/extractor.go — Extract(html []byte, extraction_type string) (map[string]string, error): for simple extraction use CSS selector (github.com/PuerkitoBio/goquery); for complex extraction use llm.Client with a prompt
Create internal/crawl/scheduler.go — Scheduler that reads crawls config, runs each on its cron schedule, tracks last-run via facts
Create internal/crawl/dedup.go — compute content hash, skip if identical to last fetched (stored as fact kind=config, key=crawl:hash:<name>)
Wire on-demand crawl into IntentAct — new tool verb crawl that accepts a URL argument
Wire scheduled crawls into cmd/mavend/main.go — separate goroutine manages the crawl scheduler
Add IPC methods MethodTriggerCrawl(name), MethodListCrawls, MethodGetCrawlResult(name)
Add crawls block to config.Config and deploy/mavend.json
Test with a static HTML page — verify extraction matches expected values, verify scheduling fires correctly

2.5 KiB Raw Blame History

Plan: Web Crawler

2.5 KiB

Raw Blame History