Maven/docs/offload.md

# Offloading model work to the workstation

*Last verified: 2026-08-09 @ 50c6637. Living doc: correct it in place, do not append.*

Owner's call, 2026-08-02. Vikunja #483 is the umbrella. Tasks #484 to #487 are the
work, and this file holds the shape and the rules all four must obey.

## The goal

homesrv cannot grow a GPU. The workstation has 16GB of VRAM. Move the model work
to the workstation and leave homesrv running the logic that must be always-on,
deterministic and cheap.

## Why this is tractable

The split already exists structurally. `mavsttd` and `mavttsd` are separate
daemons that core reaches over a socket, not linked libraries. Moving them off-box
is a transport change, not a redesign.

The microphone is at the workstation, because that is where the owner sits and
homesrv is headless. So speech-to-text and the wake word are already on the
workstation side by construction. Audio never has to cross the LAN. Only the core
turn does.

## The constraint that shapes everything

The workstation's GPU is often busy: CPT runs, experiments, Correx, the manga-recap
pipeline. It also sleeps. homesrv does not.

So an offloaded model is never *the* model. It is the preferred one, with a floor
on homesrv. That is the shape the cascade already has, where a router error falls
through to the classifier.

## The degradation rule

Two cases, and the line between them is sharp.

**Fall back silently** when the workstation model would only do the job *better*:
routing, phrasing, a nudge. Falling back costs nothing that exists today, because
the resident Qwen3-1.7B is today's production quality. The owner should not be told
that his reply was phrased by the smaller model.

**Name the gap** when the resident model cannot do the job *at all*. A world
question that a 1.7B answers by inventing is the case. A wrong answer is worse
than "не могу сейчас". This is the rule CLAUDE.md already states for a sibling
service being down.

Nothing in between. A turn never breaks on the workstation being asleep.

Both halves are wired, 03-08-2026. `LLMPhraser.PhraseWorld`
(`internal/phraser/world.go`) is the naming half and has three outcomes, not two:

| State | What he hears |
|---|---|
| no `workstation` block | the resident model answers, exactly as before the seam existed |
| configured, card free | the workstation answers |
| configured, asleep or busy | the gap, `worldGap` in `cmd/mavend/worldmodel.go` |

The first row is the one worth stating. Naming a gap requires a gap. On a box with
no second model the 1.7B is the whole product. Refusing every world question there
would remove a capability the owner has today.

A source holding a passage is on the naming half too: a live search, a ZIM
article, a page he named. None of them says "не могу сейчас". They read the
passage back, which is what `phraseSource` returning `""` selects. A real quote
beats a gap, and neither path invents.

## Admission control, not a scheduler

There is no GPU arbiter. That is a service with its own failure modes, and nothing
here needs work *distributed*. It needs admission control. The workstation
advertises free VRAM over a health endpoint, and Maven treats it as one more query
source that claims a turn or passes. llama-server also refuses to load when VRAM is
short, so the failure is detectable without cooperation from the owner's other
jobs.

The caller must be able to ask "is this peer usable right now" without a turn
hanging on a timeout. A dead remote is a normal state, not an error state.
`internal/llm.Pair` is that check on the Maven side. A prober caches the answer,
so `Available()` is an atomic read and no turn pays for a health check.

llama-server does not stay up on the workstation. It cannot: a resident 7-14B
would hold 16GB against the owner's CPT runs. So a supervisor there owns its
lifecycle, keeps it loaded while the card is free, and unloads it on idle or
when another process needs the card (owner's call, 2026-08-02, Vikunja #488).

That supervisor is still not a scheduler, and the distinction is worth holding.
It arbitrates nothing between callers. It reports whether it can take work and
manages one process to back that answer. Maven never asks it to start anything
and never learns that it did.

Contention is decided by presence under `/sys/class/kfd/kfd/proc`, not by a VRAM
threshold. A ROCm process registers there when it initialises HIP, before it
allocates anything. So the supervisor sees a contender during that job's startup,
and yields before the job loses the memory it asked for. A
threshold reads the card too late. By the time free VRAM has dropped, the other
job has already lost the allocation race. Free VRAM is still read, but only as a
precondition for loading, never as the eviction signal. One blind spot is known.
A job can take the card without registering on the KFD, as a Vulkan or a
video-decode job would. `describe()` logs every contender's comm, and that log is
how we find out whether the blind spot is real.

`mavgpud` runs from a systemd unit on the workstation with
`deploy/mavgpud.json` as its config, and `llama_args` is passed to llama-server
untouched. The model, the context size, the layer count and the MTP flags are the
owner's business and not this daemon's schema.

**The port carries a bearer token and cannot be loopback** (V-673). homesrv is
the client, so this hop is on the LAN. Until 2026-08-11 anything on the network
could spend the card, hold the model resident by touching the idle clock, and
read `/slots`, which returns other callers' prompts. mavgpud now reads
`token_file` and refuses to start when the listen address is reachable from the
network without one. Downgrading to loopback instead would look safe and take
the model arm down. Maven sends the same token from `workstation.token`, on the
completion and on the `/health` probe alike. An unsigned probe answers 401,
which Pair reads as a busy card, but Maven now rejects a live LAN workstation
block with a missing token at startup: fallback must describe a temporarily
unavailable machine, not hide a broken secret source. `model_disabled: true`
is the explicit resident-only state and does not take CW2 down with it. The
proxy also
allowlists the five paths Maven calls, so a leaked token buys the model API and
not llama-server's admin surface.

**Every GPU service on that box belongs under this supervisor**, added to
`cmd/mavgpud` rather than to systemd beside it. The rule was learned on
2026-08-09. The CW2 transcriber ran as its own user unit and registered on the
KFD like any ROCm job. So the supervisor read its own transcriber as a
contender. It yielded the card every few seconds and the gemma-4-12b arm was
down for eight minutes before anyone looked. So the supervisor takes a `stt`
block and starts CW2 itself. Yielding is all or nothing, because a job that
wants the card wants all of it. Idle unloading is not. It applies to
llama-server, which holds 8GB. CW2 holds 1.6GB, and unloading it would cost the
next voice turn its quality for nothing.

## What stays on homesrv, permanently

The **embedder** (multilingual-e5-small, ONNX, CPU). It backs the classifier, which
must answer while the GPU is saturated. It is also cheap enough on CPU that moving
it buys nothing. Four callers:

| Caller | What for |
|---|---|
| `internal/router/classifier.go` | the routing floor |
| `cmd/mavend/actions_query.go` (`queryEmbed`) | memory recall |
| `cmd/mavend/feeds.go` | ingest embedding for every RSS item |
| `internal/crawl/watch.go` | ingest embedding for every crawled page |

`internal/speaker` becomes a fifth once it lands.

## Inventory: what runs a model on homesrv today

The **resident model** is one llama-server with seven callers, and 03-08-2026 is
the date each of them stopped or did not stop being resident-only:

| Caller | What for | Offloaded |
|---|---|---|
| `cmd/mavend/voicewire.go` | routing | silently, through `hot` |
| `cmd/mavend/replier_llm.go` | replies | silently, through `hot` |
| `cmd/mavend/tick.go` | digestion worker: `PhraseNudge`, `PhraseReminder` | silently, inside the phraser |
| `cmd/mavend/actions_query.go` | world questions, and any fetched passage | names the gap |
| `cmd/mavend/capture.go` | capture summarisation (unreachable, see #480) | no, holds its own client |
| `cmd/mavend/mail.go` | mail extraction (off, no IMAP) | no, holds its own client |
| `memoryeval.go`, `modelswap.go` | admin and evals | no, and deliberately |

The last three rows are resident-only on purpose. `memoryeval.go` and
`modelswap.go` measure and swap the resident model, so sending their work
elsewhere would measure the wrong thing. `capture.go` and `mail.go` are
background jobs that hold a gated background client (`llmBackgroundClientFor`),
and that priority has no equivalent on the remote yet. Both are also unreachable
on this deploy, so wiring them would ship an untestable path.

The `tick.go` row needs one caveat. `phraser.llm_nudges` is `false` in deploy, so
nudges come from templates and the seam under them changes nothing until that
flips. It is wired anyway: `PhraseReminder` is on the same transport and is on.

Then the embedder above, **whisper.cpp** in `mavsttd`, and **piper** in `mavttsd`.
`mavwaked` uses no model at all: an energy-threshold VAD over 30ms frames.

Speech-to-text is wired as of 09-08-2026, and it takes only the silent half of the
rule. A worse transcript is still a turn, so there is nothing to name a gap about
and `stt.Pair` has no `TranscribeRemote`. `sttSeam` in `cmd/mavend/voicewire.go`
builds it, beside `modelSeam` and at the same place in `wireVoice`, so the voice
path and the meeting recorder still share one transcriber.

The remote is not a second endpoint on mavgpud. whisper.cpp cannot load
CrisperWhisper 2.0 at all. It reads its language count off the vocabulary
size, and CW2's 51897 tokens shift seven special token ids. So CW2 runs under
transformers as its own service on port 8081, and `stt.HTTPTranscriber` is the
second transport for the same seam. It posts raw PCM with the format in headers.
It carries a bearer token, because audio is the most sensitive thing that
crosses here.

It is a second endpoint on nothing, but it is a second **child** of mavgpud, and
that part is not optional. See the supervisor section above for why: a ROCm
service the supervisor does not own is a contender it yields to.

The margin is the reason: CW2 turbo scores 10.4% WER in Russian against 27.5% for
the `ggml-small.bin` mavsttd loads, over 200 Golos clips
(`docs/evals/2026-08-09-crisperwhisper2-russian-wer.md`). Text-to-speech has not
moved and piper on homesrv is still the only synthesizer.

Speech-to-text stays two stages when it moves. One call carrying both a clip and the router
prompt was measured on 05-08-2026. It scores 54.2% intent-only against 84.7% for whisper on
homesrv, on the same 72 cases. The model transcribes clips it then routes wrong, so a long
classification prompt and an audio part compete for attention. Transcribing on the
workstation and routing the text scores 83.3% at p50 997ms. So the transfer buys 375ms and
cleaner transcripts, not accuracy. See `docs/evals/2026-08-05-audio-in-routing.md`.

## Order

1. **Transport** (#484). Nothing else is possible until a seam can cross a host.
   `internal/netaddr` landed in PR #92. A seam address now carries its own scheme,
   and a scheme-less one is still unix. A tcp seam requires a shared token, because
   the filesystem permission that authenticated the unix socket is gone.
2. **The resident model** (#485, #490). Wired. A `workstation` block builds an
   `llm.Pair` in `modelSeam` (`cmd/mavend/voicewire.go`), routing and replies
   complete through it, and the phraser holds the same pair (`UseRemote`). Both
   halves of the rule are live: see the table above for which caller gets which.
   Measured, `docs/evals/2026-08-02-workstation-gemma4-12b.md`: gemma-4-12b
   through the cascade scores 84.4% full accuracy at p50 329ms. The resident
   model scores 72.7% at p50 0.80-1.04s. On the talk fixture it is 25/27
   against 20/27. Biggest quality delta. A 16GB card runs a 7-14B,
   which fixes what the 1.7B gets wrong: world knowledge, and the persona the CPT
   targets. The degradation path is already written and measured, since the
   classifier scores 68.8% full accuracy at p50 16.6µs on its own.
3. **Speech-to-text and text-to-speech** (#486). Speech-to-text is wired, see
   above. Text-to-speech is not, and piper is good enough that nothing argues
   for moving it yet.
4. **The wake word** (#487). Independent of all of the above.

## Assumptions

- The LAN is trusted enough that wireguard is supported but not required (owner's
  call). What crosses the wire is still his utterances. That is why the tcp seam
  carries its own token instead of assuming a network boundary.
- The workstation is not expected to be up. Every child task must still serve a
  turn while it is down.