The network fallback behind the local sources, off unless configured.
internal/crawl is pure: a stdlib robots.txt parser (group specificity,
wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a
watcher that notes a watched page only when its text changed. It has no store
access and no net/http; cmd/mavend/crawls.go is the impure half.
Every limit is code and tested: the guarded fetcher from #258 enforces the host
allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS
rebinding and each redirect hop are covered), caps size and redirects, times out,
and spaces requests per host. A robots.txt Disallow is refused with no override.
On demand, reading is a query source placed last in the chain, after his memory,
his notes, and the local Kiwix ZIMs once those are wired: no URL in the
utterance means no fetch, and only the URL ever leaves the box. Scheduled
watches write notes and announce nothing.
The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers
are stdlib. No new dependency.
internal/rss parses RSS 2.0 and Atom, and polls each configured feed on its own
interval; internal/webfetch is the one door either of them uses to touch the
network. The poller writes items as notes with source "rss:<feed>" and nothing
else: the answer path reads them back when he asks "что нового в лентах?", and
nothing is announced on arrival. A feed that dispatched would be a nag, which is
why the plan's breaking-news rule was left out rather than built.
webfetch is where the limits live, as code rather than a paragraph: http(s)
only, an allowlist (the configured feeds' hosts) and a denylist, a 2 MiB body
cap, a 3-redirect cap, one request per host per second, and a refusal to connect
to any private address — checked in the dialer's Control hook so it holds for
every resolved address and every redirect hop, not just for a literal IP.
Off unless configured: no "feeds" block, no poller, no outbound request. How far
a feed was read is a config fact (rss:latest:<name>), so a restart does not
re-note yesterday's headlines.
Compose stack replacing start-maven.sh's bare `&`-backgrounded processes.
Single multi-stage image builds all six daemons (CGO + prebuilt native libs
from deps/); compose runs one container each with a different command. Only
mavend mounts the encryption key (env_file, gitignored) and the db volume; the
modules mount just the shared unix-socket dir and read-only models — so the
"key-free modules" boundary is OS-enforced (separate namespaces), not just a
code convention. IPC stays unix-domain over a shared volume: zero code change,
paths move to /run/maven. Encrypted db at rest on a named volume, decrypted
working copy in tmpfs (RAM) per the at-rest encryption landed earlier.
Validated: `docker compose config` clean, mavend.json parses, all daemon flags
confirmed. NOT build-tested (no docker/GPU in authoring env) — deploy/README.md
lists the host-dependent tweak points (GPU passthrough, onnxruntime path,
cross-container voice bind, netdata host).
Chosen Docker over interim systemd units per the "dockerize soon" call — no
throwaway supervisor built.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kami