Files
kami 2c1b0eede0 Read a web page when he names one, and watch a few on a timer (#259)
The network fallback behind the local sources, off unless configured.

internal/crawl is pure: a stdlib robots.txt parser (group specificity,
wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a
watcher that notes a watched page only when its text changed. It has no store
access and no net/http; cmd/mavend/crawls.go is the impure half.

Every limit is code and tested: the guarded fetcher from #258 enforces the host
allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS
rebinding and each redirect hop are covered), caps size and redirects, times out,
and spaces requests per host. A robots.txt Disallow is refused with no override.

On demand, reading is a query source placed last in the chain, after his memory,
his notes, and the local Kiwix ZIMs once those are wired: no URL in the
utterance means no fetch, and only the URL ever leaves the box. Scheduled
watches write notes and announce nothing.

The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers
are stdlib. No new dependency.
2026-08-01 03:40:21 +04:00
..

Maven — Docker deployment

One image, one container per daemon (docker-compose.yml). Core (mavend) holds the encryption key and the db; the modules mount only the shared socket dir and read-only models.

First run

# 1. generate the at-rest db key (32 bytes, base64) — keep it safe, losing it loses the db
cp deploy/db_key.env.example deploy/db_key.env
printf 'MAVEN_DB_KEY=%s\n' "$(openssl rand 32 | base64 -w0)" > deploy/db_key.env

# 2. build + start
docker compose build
docker compose up -d

# 3. logs
docker compose logs -f mavend

models/ and deps/ are bind-mounted / baked from the host — they are NOT in git (fetched via make deps + downloaded models). The build context needs deps/lib, deps/piper, deps/include, and deps/whisper.cpp/ggml/include present (see .dockerignore).

Layout

Path (in container) What
/opt/maven/bin the six daemons
/opt/maven/lib native .so (whisper+vulkan, onnxruntime)
/opt/maven/piper piper binary + espeak data
/opt/maven/models (ro) bind-mount of ./models
/run/maven (volume) shared IPC sockets
/var/lib/maven (volume) encrypted db at rest
/dev/shm (tmpfs) decrypted db working copy (RAM only)

Reading the outside world (off by default)

mavend.json ships without a feeds block, which means no RSS/Atom feed is fetched and no outbound request is made. Switching it on is adding the block:

"feeds": {
  "poll_interval": "30m",
  "max_items": 5,
  "max_age": "24h",
  "sources": [
    { "name": "habr", "url": "https://habr.com/ru/rss/best/daily/",
      "category": "технологии", "exclude": ["реклама"] }
  ]
}

What it does and does not do:

  • items are written as notes with source rss:<name>, visible on /dash;
  • nothing is announced. She reads them back when asked — "что нового в лентах?", "что нового по технологиям?" — and never on arrival. There is no severity or channel knob here on purpose;
  • the fetcher is allowlisted to the hosts of the configured feeds, plus any allow_hosts. It refuses non-http(s) schemes and every private address (loopback, the LAN, the 10.42.0.0/24 wg range, cloud metadata). It caps the response at 2 MiB and redirects at 3, and makes at most one request per host per second. See internal/webfetch;
  • how far each feed was read is stored as a config fact rss:latest:<name>, so a restart does not re-note yesterday's headlines.

Reading a page (crawl, also off by default)

There is no crawl block either, so no page is fetched. Two halves, separately switched:

"crawl": {
  "on_demand": true,
  "interval": "6h",
  "max_runes": 4000,
  "watches": [
    { "name": "changelog", "url": "https://example.org/changelog", "interval": "12h" }
  ]
}
  • on_demand lets her read a page he names in the utterance: "посмотри https://example.org/x — что там?". The page becomes context for his question, and only the URL leaves the box. Without a URL nothing is fetched, so this is a fallback and not a habit;
  • watches re-reads a fixed list on its interval and writes a note when the text changed. Like the feeds, it announces nothing;
  • the answer path sits last in the query chain, behind his memory, his notes and (once wired) the local Kiwix ZIMs. A local read costs nothing;
  • robots.txt is fetched first and obeyed with no override; a Disallow is a refusal she says out loud. Crawl-delay is honoured;
  • same guarded fetcher as the feeds: allowlist/denylist, no private addresses, size cap, redirect cap, timeout, one request per host per second;
  • dedup state is the config fact crawl:hash:<name>.

Not yet verified / host-dependent

This stack is correct-by-construction but has not been build-tested here (no docker in the authoring env; ~1GB context; GPU). Expect a tweak on first build on the target host, most likely in one of these:

  • GPU passthroughmavsttd maps /dev/dri for Vulkan. On an NVIDIA host you'd swap to the nvidia container runtime instead of /dev/dri.
  • onnxruntime lib pathmavend's embedder needs libonnxruntime.so (on LD_LIBRARY_PATH=/opt/maven/lib). If the embedder wants an explicit path, set it in the config's embedder block.
  • cross-container voicemavweb -voice mavend:9100 only works once mavend binds its voice server on 0.0.0.0:9100 (Voice config, currently unset). Until then, voice-over-web is inert; /tools, passkey, and the dash work fine over the core socket.
  • netdatamavpoll reaches it via host.docker.internal; adjust if netdata runs elsewhere.