Maven/docs/offload.md

Offloading model work to the workstation

Last verified: 2026-08-02 @ 5c05163. Living doc: correct it in place, do not append.

Owner's call, 2026-08-02. Vikunja #483 is the umbrella. Tasks #484 to #487 are the work, and this file holds the shape and the rules all four must obey.

The goal

homesrv cannot grow a GPU. The workstation has 16GB of VRAM. Move the model work to the workstation and leave homesrv running the logic that must be always-on, deterministic and cheap.

Why this is tractable

The split already exists structurally. mavsttd and mavttsd are separate daemons that core reaches over a socket, not linked libraries. Moving them off-box is a transport change, not a redesign.

The microphone is at the workstation, because that is where the owner sits and homesrv is headless. So speech-to-text and the wake word are already on the workstation side by construction. Audio never has to cross the LAN. Only the core turn does.

The constraint that shapes everything

The workstation's GPU is often busy: CPT runs, experiments, Correx, the manga-recap pipeline. It also sleeps. homesrv does not.

So an offloaded model is never the model. It is the preferred one, with a floor on homesrv. That is the shape the cascade already has, where a router error falls through to the classifier.

The degradation rule

Two cases, and the line between them is sharp.

Fall back silently when the workstation model would only do the job better: routing, phrasing, a nudge. Falling back costs nothing that exists today, because the resident Qwen3-1.7B is today's production quality. The owner should not be told that his reply was phrased by the smaller model.

Name the gap when the resident model cannot do the job at all. A world question that a 1.7B answers by inventing is the case. A wrong answer is worse than "не могу сейчас". This is the rule CLAUDE.md already states for a sibling service being down.

Nothing in between. A turn never breaks on the workstation being asleep.

Admission control, not a scheduler

There is no GPU arbiter. That is a service with its own failure modes, and nothing here needs work distributed. It needs admission control. The workstation advertises free VRAM over a health endpoint, and Maven treats it as one more query source that claims a turn or passes. llama-server also refuses to load when VRAM is short, so the failure is detectable without cooperation from the owner's other jobs.

The caller must be able to ask "is this peer usable right now" without a turn hanging on a timeout. A dead remote is a normal state, not an error state. internal/llm.Pair is that check on the Maven side. A prober caches the answer, so Available() is an atomic read and no turn pays for a health check.

llama-server does not stay up on the workstation. It cannot: a resident 7-14B would hold 16GB against the owner's CPT runs. So a supervisor there owns its lifecycle, keeps it loaded while the card is free, and unloads it on idle or when another process needs the card (owner's call, 2026-08-02, Vikunja #488).

That supervisor is still not a scheduler, and the distinction is worth holding. It arbitrates nothing between callers. It reports whether it can take work and manages one process to back that answer. Maven never asks it to start anything and never learns that it did.

What stays on homesrv, permanently

The embedder (multilingual-e5-small, ONNX, CPU). It backs the classifier, which must answer while the GPU is saturated. It is also cheap enough on CPU that moving it buys nothing. Four callers:

Caller	What for
internal/router/classifier.go	the routing floor
cmd/mavend/actions_query.go (queryEmbed)	memory recall
cmd/mavend/feeds.go	ingest embedding for every RSS item
internal/crawl/watch.go	ingest embedding for every crawled page

internal/speaker becomes a fifth once it lands.

Inventory: what runs a model on homesrv today

The resident model is one llama-server with seven callers:

Caller	What for
cmd/mavend/voicewire.go	routing
cmd/mavend/replier_llm.go	replies
cmd/mavend/tick.go	digestion worker: PhraseNudge, PhraseReminder
cmd/mavend/capture.go	capture summarisation (unreachable, see #480)
cmd/mavend/mail.go	mail extraction (off, no IMAP)
cmd/mavend/kiwixwire.go	answering from a Kiwix, search or crawl passage
memoryeval.go, modelswap.go	admin and evals

Then the embedder above, whisper.cpp in mavsttd, and piper in mavttsd. mavwaked uses no model at all: an energy-threshold VAD over 30ms frames.

Order

1. Transport (#484). Nothing else is possible until a seam can cross a host. internal/netaddr landed in PR #92. A seam address now carries its own scheme, and a scheme-less one is still unix. A tcp seam requires a shared token, because the filesystem permission that authenticated the unix socket is gone.
2. The resident model (#485). Biggest quality delta. A 16GB card runs a 7-14B, which fixes what the 1.7B gets wrong: world knowledge, and the persona the CPT targets. The degradation path is already written and measured, since the classifier scores 68.8% full accuracy at p50 16.6µs on its own.
3. Speech-to-text and text-to-speech (#486). They gain a real margin, but on quality alone, and both already work.
4. The wake word (#487). Independent of all of the above.

Assumptions

• The LAN is trusted enough that wireguard is supported but not required (owner's call). What crosses the wire is still his utterances. That is why the tcp seam carries its own token instead of assuming a network boundary.
• The workstation is not expected to be up. Every child task must still serve a turn while it is down.