Maven/docs/offload.md

Offloading model work to the workstation

Last verified: 2026-08-09 @ 50c6637. Living doc: correct it in place, do not append.

Owner's call, 2026-08-02. Vikunja #483 is the umbrella. Tasks #484 to #487 are the work, and this file holds the shape and the rules all four must obey.

The goal

homesrv cannot grow a GPU. The workstation has 16GB of VRAM. Move the model work to the workstation and leave homesrv running the logic that must be always-on, deterministic and cheap.

Why this is tractable

The split already exists structurally. mavsttd and mavttsd are separate daemons that core reaches over a socket, not linked libraries. Moving them off-box is a transport change, not a redesign.

The microphone is at the workstation, because that is where the owner sits and homesrv is headless. So speech-to-text and the wake word are already on the workstation side by construction. Audio never has to cross the LAN. Only the core turn does.

The constraint that shapes everything

The workstation's GPU is often busy: CPT runs, experiments, Correx, the manga-recap pipeline. It also sleeps. homesrv does not.

So an offloaded model is never the model. It is the preferred one, with a floor on homesrv. That is the shape the cascade already has, where a router error falls through to the classifier.

The degradation rule

Two cases, and the line between them is sharp.

Fall back silently when the workstation model would only do the job better: routing, phrasing, a nudge. Falling back costs nothing that exists today, because the resident Qwen3-1.7B is today's production quality. The owner should not be told that his reply was phrased by the smaller model.

Name the gap when the resident model cannot do the job at all. A world question that a 1.7B answers by inventing is the case. A wrong answer is worse than "не могу сейчас". This is the rule CLAUDE.md already states for a sibling service being down.

Nothing in between. A turn never breaks on the workstation being asleep.

Both halves are wired, 03-08-2026. LLMPhraser.PhraseWorld (internal/phraser/world.go) is the naming half and has three outcomes, not two:

State	What he hears
no workstation block	the resident model answers, exactly as before the seam existed
configured, card free	the workstation answers
configured, asleep or busy	the gap, worldGap in cmd/mavend/worldmodel.go

The first row is the one worth stating. Naming a gap requires a gap. On a box with no second model the 1.7B is the whole product. Refusing every world question there would remove a capability the owner has today.

A source holding a passage is on the naming half too: a live search, a ZIM article, a page he named. None of them says "не могу сейчас". They read the passage back, which is what phraseSource returning "" selects. A real quote beats a gap, and neither path invents.

Admission control, not a scheduler

There is no GPU arbiter. That is a service with its own failure modes, and nothing here needs work distributed. It needs admission control. The workstation advertises free VRAM over a health endpoint, and Maven treats it as one more query source that claims a turn or passes. llama-server also refuses to load when VRAM is short, so the failure is detectable without cooperation from the owner's other jobs.

The caller must be able to ask "is this peer usable right now" without a turn hanging on a timeout. A dead remote is a normal state, not an error state. internal/llm.Pair is that check on the Maven side. A prober caches the answer, so Available() is an atomic read and no turn pays for a health check.

llama-server does not stay up on the workstation. It cannot: a resident 7-14B would hold 16GB against the owner's CPT runs. So a supervisor there owns its lifecycle, keeps it loaded while the card is free, and unloads it on idle or when another process needs the card (owner's call, 2026-08-02, Vikunja #488).

That supervisor is still not a scheduler, and the distinction is worth holding. It arbitrates nothing between callers. It reports whether it can take work and manages one process to back that answer. Maven never asks it to start anything and never learns that it did.

Contention is decided by presence under /sys/class/kfd/kfd/proc, not by a VRAM threshold. A ROCm process registers there when it initialises HIP, before it allocates anything. So the supervisor sees a contender during that job's startup, and yields before the job loses the memory it asked for. A threshold reads the card too late. By the time free VRAM has dropped, the other job has already lost the allocation race. Free VRAM is still read, but only as a precondition for loading, never as the eviction signal. One blind spot is known. A job can take the card without registering on the KFD, as a Vulkan or a video-decode job would. describe() logs every contender's comm, and that log is how we find out whether the blind spot is real.

mavgpud runs from a systemd unit on the workstation with deploy/mavgpud.json as its config, and llama_args is passed to llama-server untouched. The model, the context size, the layer count and the MTP flags are the owner's business and not this daemon's schema.

The port carries a bearer token and cannot be loopback (V-673). homesrv is the client, so this hop is on the LAN. Until 2026-08-11 anything on the network could spend the card, hold the model resident by touching the idle clock, and read /slots, which returns other callers' prompts. mavgpud now reads token_file and refuses to start when the listen address is reachable from the network without one. Downgrading to loopback instead would look safe and take the model arm down. Maven sends the same token from workstation.token, on the completion and on the /health probe alike. An unsigned probe answers 401, which Pair reads as a busy card, so a missing token degrades to the resident model rather than breaking a turn. The proxy also allowlists the five paths Maven calls, so a leaked token buys the model API and not llama-server's admin surface.

Every GPU service on that box belongs under this supervisor, added to cmd/mavgpud rather than to systemd beside it. The rule was learned on 2026-08-09. The CW2 transcriber ran as its own user unit and registered on the KFD like any ROCm job. So the supervisor read its own transcriber as a contender. It yielded the card every few seconds and the gemma-4-12b arm was down for eight minutes before anyone looked. So the supervisor takes a stt block and starts CW2 itself. Yielding is all or nothing, because a job that wants the card wants all of it. Idle unloading is not. It applies to llama-server, which holds 8GB. CW2 holds 1.6GB, and unloading it would cost the next voice turn its quality for nothing.

What stays on homesrv, permanently

The embedder (multilingual-e5-small, ONNX, CPU). It backs the classifier, which must answer while the GPU is saturated. It is also cheap enough on CPU that moving it buys nothing. Four callers:

Caller	What for
internal/router/classifier.go	the routing floor
cmd/mavend/actions_query.go (queryEmbed)	memory recall
cmd/mavend/feeds.go	ingest embedding for every RSS item
internal/crawl/watch.go	ingest embedding for every crawled page

internal/speaker becomes a fifth once it lands.

Inventory: what runs a model on homesrv today

The resident model is one llama-server with seven callers, and 03-08-2026 is the date each of them stopped or did not stop being resident-only:

Caller	What for	Offloaded
cmd/mavend/voicewire.go	routing	silently, through hot
cmd/mavend/replier_llm.go	replies	silently, through hot
cmd/mavend/tick.go	digestion worker: PhraseNudge, PhraseReminder	silently, inside the phraser
cmd/mavend/actions_query.go	world questions, and any fetched passage	names the gap
cmd/mavend/capture.go	capture summarisation (unreachable, see #480)	no, holds its own client
cmd/mavend/mail.go	mail extraction (off, no IMAP)	no, holds its own client
memoryeval.go, modelswap.go	admin and evals	no, and deliberately

The last three rows are resident-only on purpose. memoryeval.go and modelswap.go measure and swap the resident model, so sending their work elsewhere would measure the wrong thing. capture.go and mail.go are background jobs that hold a gated background client (llmBackgroundClientFor), and that priority has no equivalent on the remote yet. Both are also unreachable on this deploy, so wiring them would ship an untestable path.

The tick.go row needs one caveat. phraser.llm_nudges is false in deploy, so nudges come from templates and the seam under them changes nothing until that flips. It is wired anyway: PhraseReminder is on the same transport and is on.

Then the embedder above, whisper.cpp in mavsttd, and piper in mavttsd. mavwaked uses no model at all: an energy-threshold VAD over 30ms frames.

Speech-to-text is wired as of 09-08-2026, and it takes only the silent half of the rule. A worse transcript is still a turn, so there is nothing to name a gap about and stt.Pair has no TranscribeRemote. sttSeam in cmd/mavend/voicewire.go builds it, beside modelSeam and at the same place in wireVoice, so the voice path and the meeting recorder still share one transcriber.

The remote is not a second endpoint on mavgpud. whisper.cpp cannot load CrisperWhisper 2.0 at all. It reads its language count off the vocabulary size, and CW2's 51897 tokens shift seven special token ids. So CW2 runs under transformers as its own service on port 8081, and stt.HTTPTranscriber is the second transport for the same seam. It posts raw PCM with the format in headers. It carries a bearer token, because audio is the most sensitive thing that crosses here.

It is a second endpoint on nothing, but it is a second child of mavgpud, and that part is not optional. See the supervisor section above for why: a ROCm service the supervisor does not own is a contender it yields to.

The margin is the reason: CW2 turbo scores 10.4% WER in Russian against 27.5% for the ggml-small.bin mavsttd loads, over 200 Golos clips (docs/evals/2026-08-09-crisperwhisper2-russian-wer.md). Text-to-speech has not moved and piper on homesrv is still the only synthesizer.

Speech-to-text stays two stages when it moves. One call carrying both a clip and the router prompt was measured on 05-08-2026. It scores 54.2% intent-only against 84.7% for whisper on homesrv, on the same 72 cases. The model transcribes clips it then routes wrong, so a long classification prompt and an audio part compete for attention. Transcribing on the workstation and routing the text scores 83.3% at p50 997ms. So the transfer buys 375ms and cleaner transcripts, not accuracy. See docs/evals/2026-08-05-audio-in-routing.md.

Order

1. Transport (#484). Nothing else is possible until a seam can cross a host. internal/netaddr landed in PR #92. A seam address now carries its own scheme, and a scheme-less one is still unix. A tcp seam requires a shared token, because the filesystem permission that authenticated the unix socket is gone.
2. The resident model (#485, #490). Wired. A workstation block builds an llm.Pair in modelSeam (cmd/mavend/voicewire.go), routing and replies complete through it, and the phraser holds the same pair (UseRemote). Both halves of the rule are live: see the table above for which caller gets which. Measured, docs/evals/2026-08-02-workstation-gemma4-12b.md: gemma-4-12b through the cascade scores 84.4% full accuracy at p50 329ms. The resident model scores 72.7% at p50 0.80-1.04s. On the talk fixture it is 25/27 against 20/27. Biggest quality delta. A 16GB card runs a 7-14B, which fixes what the 1.7B gets wrong: world knowledge, and the persona the CPT targets. The degradation path is already written and measured, since the classifier scores 68.8% full accuracy at p50 16.6µs on its own.
3. Speech-to-text and text-to-speech (#486). Speech-to-text is wired, see above. Text-to-speech is not, and piper is good enough that nothing argues for moving it yet.
4. The wake word (#487). Independent of all of the above.

Assumptions

• The LAN is trusted enough that wireguard is supported but not required (owner's call). What crosses the wire is still his utterances. That is why the tcp seam carries its own token instead of assuming a network boundary.
• The workstation is not expected to be up. Every child task must still serve a turn while it is down.