kami
2c1b0eede0
The network fallback behind the local sources, off unless configured. internal/crawl is pure: a stdlib robots.txt parser (group specificity, wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a watcher that notes a watched page only when its text changed. It has no store access and no net/http; cmd/mavend/crawls.go is the impure half. Every limit is code and tested: the guarded fetcher from #258 enforces the host allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS rebinding and each redirect hop are covered), caps size and redirects, times out, and spaces requests per host. A robots.txt Disallow is refused with no override. On demand, reading is a query source placed last in the chain, after his memory, his notes, and the local Kiwix ZIMs once those are wired: no URL in the utterance means no fetch, and only the URL ever leaves the box. Scheduled watches write notes and announce nothing. The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers are stdlib. No new dependency.
117 lines
4.9 KiB
Markdown
117 lines
4.9 KiB
Markdown
# Maven — Docker deployment
|
|
|
|
One image, one container per daemon (`docker-compose.yml`). Core (`mavend`)
|
|
holds the encryption key and the db; the modules mount only the shared socket
|
|
dir and read-only models.
|
|
|
|
## First run
|
|
|
|
```sh
|
|
# 1. generate the at-rest db key (32 bytes, base64) — keep it safe, losing it loses the db
|
|
cp deploy/db_key.env.example deploy/db_key.env
|
|
printf 'MAVEN_DB_KEY=%s\n' "$(openssl rand 32 | base64 -w0)" > deploy/db_key.env
|
|
|
|
# 2. build + start
|
|
docker compose build
|
|
docker compose up -d
|
|
|
|
# 3. logs
|
|
docker compose logs -f mavend
|
|
```
|
|
|
|
`models/` and `deps/` are bind-mounted / baked from the host — they are NOT in
|
|
git (fetched via `make deps` + downloaded models). The build context needs
|
|
`deps/lib`, `deps/piper`, `deps/include`, and `deps/whisper.cpp/ggml/include`
|
|
present (see `.dockerignore`).
|
|
|
|
## Layout
|
|
|
|
| Path (in container) | What |
|
|
|----------------------------|-----------------------------------------|
|
|
| `/opt/maven/bin` | the six daemons |
|
|
| `/opt/maven/lib` | native .so (whisper+vulkan, onnxruntime)|
|
|
| `/opt/maven/piper` | piper binary + espeak data |
|
|
| `/opt/maven/models` (ro) | bind-mount of `./models` |
|
|
| `/run/maven` (volume) | shared IPC sockets |
|
|
| `/var/lib/maven` (volume) | encrypted db at rest |
|
|
| `/dev/shm` (tmpfs) | decrypted db working copy (RAM only) |
|
|
|
|
## Reading the outside world (off by default)
|
|
|
|
`mavend.json` ships without a `feeds` block, which means no RSS/Atom feed is
|
|
fetched and no outbound request is made. Switching it on is adding the block:
|
|
|
|
```json
|
|
"feeds": {
|
|
"poll_interval": "30m",
|
|
"max_items": 5,
|
|
"max_age": "24h",
|
|
"sources": [
|
|
{ "name": "habr", "url": "https://habr.com/ru/rss/best/daily/",
|
|
"category": "технологии", "exclude": ["реклама"] }
|
|
]
|
|
}
|
|
```
|
|
|
|
What it does and does not do:
|
|
|
|
- items are written as notes with source `rss:<name>`, visible on `/dash`;
|
|
- **nothing is announced.** She reads them back when asked — "что нового в
|
|
лентах?", "что нового по технологиям?" — and never on arrival. There is no
|
|
severity or channel knob here on purpose;
|
|
- the fetcher is allowlisted to the hosts of the configured feeds, plus any
|
|
`allow_hosts`. It refuses non-http(s) schemes and every private address
|
|
(loopback, the LAN, the `10.42.0.0/24` wg range, cloud metadata). It caps the
|
|
response at 2 MiB and redirects at 3, and makes at most one request per host
|
|
per second. See `internal/webfetch`;
|
|
- how far each feed was read is stored as a config fact `rss:latest:<name>`, so
|
|
a restart does not re-note yesterday's headlines.
|
|
|
|
### Reading a page (`crawl`, also off by default)
|
|
|
|
There is no `crawl` block either, so no page is fetched. Two halves, separately
|
|
switched:
|
|
|
|
```json
|
|
"crawl": {
|
|
"on_demand": true,
|
|
"interval": "6h",
|
|
"max_runes": 4000,
|
|
"watches": [
|
|
{ "name": "changelog", "url": "https://example.org/changelog", "interval": "12h" }
|
|
]
|
|
}
|
|
```
|
|
|
|
- `on_demand` lets her read a page he names in the utterance: "посмотри
|
|
https://example.org/x — что там?". The page becomes context for his question,
|
|
and only the URL leaves the box. Without a URL nothing is fetched, so this is
|
|
a fallback and not a habit;
|
|
- `watches` re-reads a fixed list on its interval and writes a note when the
|
|
text changed. Like the feeds, it announces nothing;
|
|
- the answer path sits **last** in the query chain, behind his memory, his notes
|
|
and (once wired) the local Kiwix ZIMs. A local read costs nothing;
|
|
- `robots.txt` is fetched first and obeyed with no override; a `Disallow` is a
|
|
refusal she says out loud. `Crawl-delay` is honoured;
|
|
- same guarded fetcher as the feeds: allowlist/denylist, no private addresses,
|
|
size cap, redirect cap, timeout, one request per host per second;
|
|
- dedup state is the config fact `crawl:hash:<name>`.
|
|
|
|
## Not yet verified / host-dependent
|
|
|
|
This stack is correct-by-construction but has **not been build-tested here**
|
|
(no docker in the authoring env; ~1GB context; GPU). Expect a tweak on first
|
|
build on the target host, most likely in one of these:
|
|
|
|
- **GPU passthrough** — `mavsttd` maps `/dev/dri` for Vulkan. On an NVIDIA host
|
|
you'd swap to the nvidia container runtime instead of `/dev/dri`.
|
|
- **onnxruntime lib path** — `mavend`'s embedder needs `libonnxruntime.so`
|
|
(on `LD_LIBRARY_PATH=/opt/maven/lib`). If the embedder wants an explicit
|
|
path, set it in the config's embedder block.
|
|
- **cross-container voice** — `mavweb -voice mavend:9100` only works once
|
|
`mavend` binds its voice server on `0.0.0.0:9100` (Voice config, currently
|
|
unset). Until then, voice-over-web is inert; `/tools`, passkey, and the dash
|
|
work fine over the core socket.
|
|
- **netdata** — `mavpoll` reaches it via `host.docker.internal`; adjust if
|
|
netdata runs elsewhere.
|