Run the big model on the workstation, with admission control and the 1.7B as the floor #95

2026-08-02T16:52:47+02:00

claude commented

2026-08-02 16:52:47 +02:00

Closes Vikunja #488.

Acceptance criteria and quality gate are in TASK.md on this branch.
Review the review, not the diff — leave comments and the agent will apply them via task start 488.

Closes Vikunja #488. Acceptance criteria and quality gate are in `TASK.md` on this branch. Review the review, not the diff — leave comments and the agent will apply them via `task start 488`.

kami changed target branch from task/485-run-the-big-model-on-the-workstation-wit to master

2026-08-02 16:53:32 +02:00

kami added 3 commits 2026-08-02 17:03:59 +02:00

mavgpud: read the card from sysfs and own llama-server's lifecycle (V-488) ab42db2b87

The workstation cannot keep a 7-14B resident: it would hold 16GB against the
owner's CPT runs, Correx and the manga-recap pipeline. So the process that
stays up costs no VRAM and the model comes and goes under it.

Contention is detected by presence on the KFD, not by a VRAM threshold. A ROCm
process registers under /sys/class/kfd/kfd/proc when it initialises HIP, well
before it allocates, so we see a contender during its startup instead of after
it has already lost an allocation race. rocm-smi is not installed on that box
and a per-second subprocess would get tuned down until useless, so this reads
sysfs and forks nothing.

Free VRAM is read only to decide whether to start. It is never a reason to
stop: by the time free VRAM has dropped, the other job has already failed.

mavgpud: keep the model loaded while the card is free, yield when it is not (V-488) 2b97bac51e

The lifecycle rule from Vikunja #488. Not on demand, because a 7-14B takes tens
of seconds to load and a world question would meet a gap every time the card
had been quiet. Not always on, because that is what holds the card.

/health is answered locally and always, so Maven's prober costs nothing and
works while the model is down. Everything else is reverse-proxied to
llama-server, which is what makes the idle window measurable at all.

Yielding is checked before starting, and both transitions are damped by a poll
streak so a short-lived rocm process cannot evict the model.

mavgpud: test the probe against the sysfs the workstation actually has (V-488) e52c616592

The fixtures are the live numbers sampled from the box on 02-08-2026, where the
CPT run held 12.8GB of 16 as proc/478104/vram_35881.

The cases that matter are the ones where a mistake is silent: our own
llama-server counting as a contender, an unreadable card reading as free, and
/health hanging or proxying into a closed port instead of answering 503.

kami force-pushed task/488-workstation-a-supervisor-that-keeps-llam from 945ca6212b to e52c616592

2026-08-02 17:03:59 +02:00

Compare

kami merged commit 666b924d29 into master

2026-08-02 17:04:05 +02:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#95