# Offloading model work to the workstation *Last verified: 2026-08-09 @ 50c6637. Living doc: correct it in place, do not append.* Owner's call, 2026-08-02. Vikunja #483 is the umbrella. Tasks #484 to #487 are the work, and this file holds the shape and the rules all four must obey. ## The goal homesrv cannot grow a GPU. The workstation has 16GB of VRAM. Move the model work to the workstation and leave homesrv running the logic that must be always-on, deterministic and cheap. ## Why this is tractable The split already exists structurally. `mavsttd` and `mavttsd` are separate daemons that core reaches over a socket, not linked libraries. Moving them off-box is a transport change, not a redesign. The microphone is at the workstation, because that is where the owner sits and homesrv is headless. So speech-to-text and the wake word are already on the workstation side by construction. Audio never has to cross the LAN. Only the core turn does. ## The constraint that shapes everything The workstation's GPU is often busy: CPT runs, experiments, Correx, the manga-recap pipeline. It also sleeps. homesrv does not. So an offloaded model is never *the* model. It is the preferred one, with a floor on homesrv. That is the shape the cascade already has, where a router error falls through to the classifier. ## The degradation rule Two cases, and the line between them is sharp. **Fall back silently** when the workstation model would only do the job *better*: routing, phrasing, a nudge. Falling back costs nothing that exists today, because the resident Qwen3-1.7B is today's production quality. The owner should not be told that his reply was phrased by the smaller model. **Name the gap** when the resident model cannot do the job *at all*. A world question that a 1.7B answers by inventing is the case. A wrong answer is worse than "не могу сейчас". This is the rule CLAUDE.md already states for a sibling service being down. Nothing in between. A turn never breaks on the workstation being asleep. Both halves are wired, 03-08-2026. `LLMPhraser.PhraseWorld` (`internal/phraser/world.go`) is the naming half and has three outcomes, not two: | State | What he hears | |---|---| | no `workstation` block | the resident model answers, exactly as before the seam existed | | configured, card free | the workstation answers | | configured, asleep or busy | the gap, `worldGap` in `cmd/mavend/worldmodel.go` | The first row is the one worth stating. Naming a gap requires a gap. On a box with no second model the 1.7B is the whole product. Refusing every world question there would remove a capability the owner has today. A source holding a passage is on the naming half too: a live search, a ZIM article, a page he named. None of them says "не могу сейчас". They read the passage back, which is what `phraseSource` returning `""` selects. A real quote beats a gap, and neither path invents. ## Admission control, not a scheduler There is no GPU arbiter. That is a service with its own failure modes, and nothing here needs work *distributed*. It needs admission control. The workstation advertises free VRAM over a health endpoint, and Maven treats it as one more query source that claims a turn or passes. llama-server also refuses to load when VRAM is short, so the failure is detectable without cooperation from the owner's other jobs. The caller must be able to ask "is this peer usable right now" without a turn hanging on a timeout. A dead remote is a normal state, not an error state. `internal/llm.Pair` is that check on the Maven side. A prober caches the answer, so `Available()` is an atomic read and no turn pays for a health check. llama-server does not stay up on the workstation. It cannot: a resident 7-14B would hold 16GB against the owner's CPT runs. So a supervisor there owns its lifecycle, keeps it loaded while the card is free, and unloads it on idle or when another process needs the card (owner's call, 2026-08-02, Vikunja #488). That supervisor is still not a scheduler, and the distinction is worth holding. It arbitrates nothing between callers. It reports whether it can take work and manages one process to back that answer. Maven never asks it to start anything and never learns that it did. Contention is decided by presence under `/sys/class/kfd/kfd/proc`, not by a VRAM threshold. A ROCm process registers there when it initialises HIP, before it allocates anything. So the supervisor sees a contender during that job's startup, and yields before the job loses the memory it asked for. A threshold reads the card too late. By the time free VRAM has dropped, the other job has already lost the allocation race. Free VRAM is still read, but only as a precondition for loading, never as the eviction signal. One blind spot is known. A job can take the card without registering on the KFD, as a Vulkan or a video-decode job would. `describe()` logs every contender's comm, and that log is how we find out whether the blind spot is real. `mavgpud` runs from a systemd unit on the workstation with `deploy/mavgpud.json` as its config, and `llama_args` is passed to llama-server untouched. The model, the context size, the layer count and the MTP flags are the owner's business and not this daemon's schema. **The port carries a bearer token and cannot be loopback** (V-673). homesrv is the client, so this hop is on the LAN. Until 2026-08-11 anything on the network could spend the card, hold the model resident by touching the idle clock, and read `/slots`, which returns other callers' prompts. mavgpud now reads `token_file` and refuses to start when the listen address is reachable from the network without one. Downgrading to loopback instead would look safe and take the model arm down. Maven sends the same token from `workstation.token`, on the completion and on the `/health` probe alike. An unsigned probe answers 401, which Pair reads as a busy card, but Maven now rejects a live LAN workstation block with a missing token at startup: fallback must describe a temporarily unavailable machine, not hide a broken secret source. `model_disabled: true` is the explicit resident-only state and does not take CW2 down with it. The proxy also allowlists the five paths Maven calls, so a leaked token buys the model API and not llama-server's admin surface. **Every GPU service on that box belongs under this supervisor**, added to `cmd/mavgpud` rather than to systemd beside it. The rule was learned on 2026-08-09. The CW2 transcriber ran as its own user unit and registered on the KFD like any ROCm job. So the supervisor read its own transcriber as a contender. It yielded the card every few seconds and the gemma-4-12b arm was down for eight minutes before anyone looked. So the supervisor takes a `stt` block and starts CW2 itself. Yielding is all or nothing, because a job that wants the card wants all of it. Idle unloading is not. It applies to llama-server, which holds 8GB. CW2 holds 1.6GB, and unloading it would cost the next voice turn its quality for nothing. ## What stays on homesrv, permanently The **embedder** (multilingual-e5-small, ONNX, CPU). It backs the classifier, which must answer while the GPU is saturated. It is also cheap enough on CPU that moving it buys nothing. Four callers: | Caller | What for | |---|---| | `internal/router/classifier.go` | the routing floor | | `cmd/mavend/actions_query.go` (`queryEmbed`) | memory recall | | `cmd/mavend/feeds.go` | ingest embedding for every RSS item | | `internal/crawl/watch.go` | ingest embedding for every crawled page | `internal/speaker` becomes a fifth once it lands. ## Inventory: what runs a model on homesrv today The **resident model** is one llama-server with seven callers, and 03-08-2026 is the date each of them stopped or did not stop being resident-only: | Caller | What for | Offloaded | |---|---|---| | `cmd/mavend/voicewire.go` | routing | silently, through `hot` | | `cmd/mavend/replier_llm.go` | replies | silently, through `hot` | | `cmd/mavend/tick.go` | digestion worker: `PhraseNudge`, `PhraseReminder` | silently, inside the phraser | | `cmd/mavend/actions_query.go` | world questions, and any fetched passage | names the gap | | `cmd/mavend/capture.go` | capture summarisation (unreachable, see #480) | no, holds its own client | | `cmd/mavend/mail.go` | mail extraction (off, no IMAP) | no, holds its own client | | `memoryeval.go`, `modelswap.go` | admin and evals | no, and deliberately | The last three rows are resident-only on purpose. `memoryeval.go` and `modelswap.go` measure and swap the resident model, so sending their work elsewhere would measure the wrong thing. `capture.go` and `mail.go` are background jobs that hold a gated background client (`llmBackgroundClientFor`), and that priority has no equivalent on the remote yet. Both are also unreachable on this deploy, so wiring them would ship an untestable path. The `tick.go` row needs one caveat. `phraser.llm_nudges` is `false` in deploy, so nudges come from templates and the seam under them changes nothing until that flips. It is wired anyway: `PhraseReminder` is on the same transport and is on. Then the embedder above, **whisper.cpp** in `mavsttd`, and **piper** in `mavttsd`. `mavwaked` uses no model at all: an energy-threshold VAD over 30ms frames. Speech-to-text is wired as of 09-08-2026, and it takes only the silent half of the rule. A worse transcript is still a turn, so there is nothing to name a gap about and `stt.Pair` has no `TranscribeRemote`. `sttSeam` in `cmd/mavend/voicewire.go` builds it, beside `modelSeam` and at the same place in `wireVoice`, so the voice path and the meeting recorder still share one transcriber. The remote is not a second endpoint on mavgpud. whisper.cpp cannot load CrisperWhisper 2.0 at all. It reads its language count off the vocabulary size, and CW2's 51897 tokens shift seven special token ids. So CW2 runs under transformers as its own service on port 8081, and `stt.HTTPTranscriber` is the second transport for the same seam. It posts raw PCM with the format in headers. It carries a bearer token, because audio is the most sensitive thing that crosses here. It is a second endpoint on nothing, but it is a second **child** of mavgpud, and that part is not optional. See the supervisor section above for why: a ROCm service the supervisor does not own is a contender it yields to. The margin is the reason: CW2 turbo scores 10.4% WER in Russian against 27.5% for the `ggml-small.bin` mavsttd loads, over 200 Golos clips (`docs/evals/2026-08-09-crisperwhisper2-russian-wer.md`). Text-to-speech has not moved and piper on homesrv is still the only synthesizer. Speech-to-text stays two stages when it moves. One call carrying both a clip and the router prompt was measured on 05-08-2026. It scores 54.2% intent-only against 84.7% for whisper on homesrv, on the same 72 cases. The model transcribes clips it then routes wrong, so a long classification prompt and an audio part compete for attention. Transcribing on the workstation and routing the text scores 83.3% at p50 997ms. So the transfer buys 375ms and cleaner transcripts, not accuracy. See `docs/evals/2026-08-05-audio-in-routing.md`. ## Order 1. **Transport** (#484). Nothing else is possible until a seam can cross a host. `internal/netaddr` landed in PR #92. A seam address now carries its own scheme, and a scheme-less one is still unix. A tcp seam requires a shared token, because the filesystem permission that authenticated the unix socket is gone. 2. **The resident model** (#485, #490). Wired. A `workstation` block builds an `llm.Pair` in `modelSeam` (`cmd/mavend/voicewire.go`), routing and replies complete through it, and the phraser holds the same pair (`UseRemote`). Both halves of the rule are live: see the table above for which caller gets which. Measured, `docs/evals/2026-08-02-workstation-gemma4-12b.md`: gemma-4-12b through the cascade scores 84.4% full accuracy at p50 329ms. The resident model scores 72.7% at p50 0.80-1.04s. On the talk fixture it is 25/27 against 20/27. Biggest quality delta. A 16GB card runs a 7-14B, which fixes what the 1.7B gets wrong: world knowledge, and the persona the CPT targets. The degradation path is already written and measured, since the classifier scores 68.8% full accuracy at p50 16.6µs on its own. 3. **Speech-to-text and text-to-speech** (#486). Speech-to-text is wired, see above. Text-to-speech is not, and piper is good enough that nothing argues for moving it yet. 4. **The wake word** (#487). Independent of all of the above. ## Assumptions - The LAN is trusted enough that wireguard is supported but not required (owner's call). What crosses the wire is still his utterances. That is why the tcp seam carries its own token instead of assuming a network boundary. - The workstation is not expected to be up. Every child task must still serve a turn while it is down.