Maven/deploy/README.md

# Maven — Docker deployment

One image, one container per daemon (`docker-compose.yml`). Core (`mavend`)
holds the encryption key and the db; the modules mount only the shared socket
dir and read-only models.

## First run

```sh
# 1. generate the at-rest db key (32 bytes, base64) — keep it safe, losing it loses the db
cp deploy/db_key.env.example deploy/db_key.env
printf 'MAVEN_DB_KEY=%s\n' "$(openssl rand 32 | base64 -w0)" > deploy/db_key.env

# 1b. review the complete secret inventory. Copy homesrv values to
# deploy/telegram.env, ambient enable/token values to .env, and workpc values
# to the destinations named in the example. Live blocks reject empty secrets.
cp deploy/telegram.env.example deploy/telegram.env

# 2. build + start
docker compose build
docker compose up -d

# 3. logs
docker compose logs -f mavend
```

`models/` and `deps/` are bind-mounted / baked from the host — they are NOT in
git (fetched via `make deps` + downloaded models). The build context needs
`deps/lib`, `deps/piper`, `deps/include`, and `deps/whisper.cpp/ggml/include`
present (see `.dockerignore`).

## Layout

| Path (in container)        | What                                    |
|----------------------------|-----------------------------------------|
| `/opt/maven/bin`           | the six daemons                         |
| `/opt/maven/lib`           | native .so (whisper+vulkan, onnxruntime)|
| `/opt/maven/piper`         | piper binary + espeak data              |
| `/opt/maven/models` (ro)   | bind-mount of `./models`                |
| `/run/maven` (volume)      | shared IPC sockets                      |
| `/var/lib/maven` (volume)  | encrypted db at rest                    |
| `/dev/shm` (tmpfs)         | decrypted db working copy (RAM only)    |

## Reading the outside world (off by default)

`mavend.json` ships without a `feeds` block, which means no RSS/Atom feed is
fetched and no outbound request is made. Switching it on is adding the block:

```json
"feeds": {
  "poll_interval": "30m",
  "max_items": 5,
  "max_age": "24h",
  "sources": [
    { "name": "habr", "url": "https://habr.com/ru/rss/best/daily/",
      "category": "технологии", "exclude": ["реклама"] }
  ]
}
```

What it does and does not do:

- items are written as notes with source `rss:<name>`, visible on `/dash`;
- **nothing is announced.** She reads them back when asked — "что нового в
  лентах?", "что нового по технологиям?" — and never on arrival. There is no
  severity or channel knob here on purpose;
- the fetcher is allowlisted to the hosts of the configured feeds, plus any
  `allow_hosts`. It refuses non-http(s) schemes and every private address
  (loopback, the LAN, the `10.42.0.0/24` wg range, cloud metadata). It caps the
  response at 2 MiB and redirects at 3, and makes at most one request per host
  per second. See `internal/webfetch`;
- how far each feed was read is stored as a config fact `rss:latest:<name>`, so
  a restart does not re-note yesterday's headlines. A feed whose items carry no
  dates gets the same mark, and the first poll after a restart takes those items
  as already read rather than writing them all again;
- `max_items` paces, it does not drop: a burst larger than the cap arrives over
  the following polls, oldest first;
- feed notes are **not** part of recall. "что я говорил про X" searches what he
  said; headlines are read back only by asking about the feeds.

### Searching the web (`search`, on in this deploy)

`deploy/mavend.json` ships a `search` block, so a question that is not about him
reaches a self-hosted SearXNG before it reaches the ZIMs. Delete the block and
no query leaves the LAN again. The shipped shape:

```json
"search": {
  "url": "http://searxng:9563",
  "max_results": 4,
  "snippet_runes": 1500,
  "language": "auto",
  "timeout": "8s"
}
```

- the instance needs `json` in its `search.formats` (settings.yml). A stock
  SearXNG answers 403 to `format=json`, and then every search fails;
- the question goes out **verbatim**, in the language he asked it. There is no
  rewriter here, unlike Kiwix: SearXNG ranks through real engines;
- only the query string leaves the box. `internal/websearch` cannot read the
  store, so no note, fact, persona block or history can travel with a search;
- a question about him never becomes a query. The personal boundary in the
  query chain stops the walk above this source;
- this runs **before** Kiwix. A live search reads what is true today and the
  ZIMs read what was true when they were built, so the ZIMs are the fallback:
  an empty result, an unreachable instance or a dead line falls through to
  them and she never says the search failed;
- `engines` narrows the search to named engines, e.g. `"duckduckgo,wikipedia"`.
  Empty means whatever the instance has enabled.

### Reading a page (`crawl`, also off by default)

There is no `crawl` block either, so no page is fetched. Two halves, separately
switched:

```json
"crawl": {
  "on_demand": true,
  "interval": "6h",
  "max_runes": 4000,
  "watches": [
    { "name": "changelog", "url": "https://example.org/changelog", "interval": "12h" }
  ]
}
```

- `on_demand` lets her read a page he names in the utterance: "посмотри
  https://example.org/x — что там?". The page becomes context for his question,
  and only the URL leaves the box. Without a URL nothing is fetched, so this is
  a fallback and not a habit;
- `watches` re-reads a fixed list on its interval and writes a note when the
  text changed. Like the feeds, it announces nothing;
- the answer path sits behind his memory, his notes, the web search and the
  ZIMs, and ahead of the model answering from what it remembers. A page he
  named is an instruction, so it is read last and only when he named one;
- `robots.txt` is fetched first and obeyed with no override; a `Disallow` is a
  refusal she says out loud. `Crawl-delay` is waited out before the page is
  fetched, and a delay longer than the turn fails the read instead of hanging
  it. A `robots.txt` that answers 5xx refuses the crawl — a broken server is
  not permission;
- `allow_hosts` limits on-demand reading to those hosts and nothing else.
  Watched pages' hosts are reachable by the scheduled crawler whether listed or
  not, but a watch does **not** widen what he may ask her to read;
- same guarded fetcher as the feeds: allowlist/denylist, no private addresses,
  size cap, redirect cap, timeout, one request per host per second;
- dedup state is the config fact `crawl:hash:<name>`;
- like feed notes, watch notes are kept out of recall (`store.ReadSourcePrefixes`).
  Text from someone else's page is not something he said, so it must not come
  back as an answer to a question about him. Watch notes are visible on `/dash`.

## Not yet verified / host-dependent

This stack is correct-by-construction but has **not been build-tested here**
(no docker in the authoring env; ~1GB context; GPU). Expect a tweak on first
build on the target host, most likely in one of these:

- **GPU passthrough** — `mavsttd` maps `/dev/dri` for Vulkan. On an NVIDIA host
  you'd swap to the nvidia container runtime instead of `/dev/dri`.
- **onnxruntime lib path** — `mavend`'s embedder needs `libonnxruntime.so`
  (on `LD_LIBRARY_PATH=/opt/maven/lib`). If the embedder wants an explicit
  path, set it in the config's embedder block.
- **cross-container voice** — `mavweb -voice mavend:9100` only works once
  `mavend` binds its voice server on `0.0.0.0:9100` (Voice config, currently
  unset). Until then, voice-over-web is inert; `/tools`, passkey, and the dash
  work fine over the core socket.
- **netdata** — `mavpoll` reaches it via `host.docker.internal`; adjust if
  netdata runs elsewhere.

## Updating her (`mavupdate`, Vikunja #249)

Off unless configured, and there is deliberately no button for it. There is no
IPC method, no web route, no timer and no act that starts an update — the trigger
is a human running `mavupdate` on the host, which needs shell access, a strictly
higher bar than the step-up passkey gate that guards `/tools`. She cannot update
herself; she can be updated. Nothing here ever fetches code: the new version is
whatever you pulled into the working tree yourself.

Add an `update` block to `mavend.json` (mavend ignores it — only the CLI reads
it), with paths as they exist **on the host**, not inside a container:

```json
"update": {
  "source_dir": "/home/kami/apps/Maven",
  "install_dir": "/home/kami/apps/Maven",
  "snapshot_dir": "/var/lib/maven-snapshots",
  "source_rollback": "git",
  "binaries": ["mavend", "mavweb", "mavsttd", "mavttsd", "mavwaked",
               "mavenclient", "mavpoll", "mavcaldav", "mavmaild", "mavupdate"],
  "config_files": ["deploy/mavend.json"],
  "restart_cmd": ["docker", "compose", "up", "-d", "--build", "mavend"],
  "health_socket": "/run/maven-host/mavend.sock",
  "health_timeout_sec": 120
}
```

`snapshot_dir` must be outside both `install_dir` and `source_dir` (a restore
must not read from what the install writes, and a snapshot dir inside the tree
lands in the docker build context). `health_socket` is required: an update that
cannot check its own result cannot roll itself back, so the config is refused
without one.

**`source_rollback` is what makes a rollback real on this deployment.** Compose
builds the image from the tree — the Dockerfile copies `cmd/` and `internal/`
and runs the build in the builder stage, and `.dockerignore` keeps the host
binaries out — so `install_dir` is the tree, `install` is a no-op, and putting
the old binaries back puts back bytes nothing reads. A rollback that only did
that would rebuild the same bad image and burn a second health timeout proving
it. With `"source_rollback": "git"` the commit is recorded before the update and
checked back out before the restart, so the restore is of the thing that
actually gets deployed. It requires a clean tree: `apply` refuses to start with
uncommitted changes, because the recorded commit would not describe what is
deployed and the forced checkout on the way back would delete the work. It also
means a rollback moves every tracked file, `deploy/mavend.json` included, so on
this deployment a config edit belongs in a commit.

Leaving `source_rollback` out is only valid when `install_dir` holds what
actually runs. `Validate` refuses the combination of "same dir" and "no way to
put the source back" at startup rather than at the one rollback that mattered.

**The socket has to be one the account running `mavupdate` can open.** The
compose stack keeps IPC in a named volume, whose host path
(`/var/lib/docker/volumes/maven_sockets/_data`) is under a `drwx--x--- root
root` directory, and the socket itself is 0600 owned by the container's uid
10001. A non-root `mavupdate` gets EACCES on the dial, which reports as
`update: cannot open the health socket` rather than as a daemon that will not
answer. Bind-mount the socket dir to a host path he owns and run the daemon
under his uid instead:

```yaml
mavend:
  user: "1000:1000"
  volumes:
    - /run/maven-host:/run/maven
```

Do **not** work around it with `sudo mavupdate apply`. `verify` runs `make
build` and `make test` in `source_dir`, and as root that leaves root-owned
binaries, object files and a build cache in the working tree, so the next
ordinary `make` fails. `Verify` refuses to run as root over a tree owned by
someone else for exactly that reason.

Then:

```sh
mavupdate -config deploy/mavend.json verify        # make build + make test, deploys nothing
mavupdate -config deploy/mavend.json apply -yes    # snapshot, verify, install, restart, health-check
mavupdate -config deploy/mavend.json list          # what you can roll back to
mavupdate -config deploy/mavend.json rollback -yes # restore the previous artifacts and restart
```

`apply` refuses to start if she is not already answering — otherwise a failed
update and a box that was already broken are indistinguishable afterwards. On any
failure after the install it restores the snapshot, restarts, and checks again;
if that also fails it says so loudly and names the directory to copy back by hand.
The database is never snapshotted or rolled back (see the package comment in
`internal/update`); schema compatibility is `store.Migrate`'s job.