Run the big model on the workstation, with admission control and the 1.7B as the floor #94

2026-08-02T15:20:43+02:00

claude commented

2026-08-02 15:20:43 +02:00

Closes Vikunja #485.

Acceptance criteria and quality gate are in TASK.md on this branch.
Review the review, not the diff — leave comments and the agent will apply them via task start 485.

Closes Vikunja #485. Acceptance criteria and quality gate are in `TASK.md` on this branch. Review the review, not the diff — leave comments and the agent will apply them via `task start 485`.

claude added 2 commits 2026-08-02 15:20:44 +02:00

llm: prefer the workstation model, floor on the resident one (V-485) 2076e4a788

Pair holds both models and decides which answers. A prober asks the
remote whether it will take work and caches the answer, so a request
reads an atomic bool rather than paying for a health check. Routing sits
at p50 825ms on the hot path and must never wait on a machine that may be
asleep.

The two methods are the two halves of the degradation rule in
docs/offload.md. Complete falls back silently, for routing, replies and
nudge phrasing, where the big model is only better. CompleteRemote
returns ErrRemoteUnavailable instead, for a world question, where the
1.7B does not answer worse but invents.

A nil remote is the unconfigured deploy: nothing probes, everything goes
to the floor, and the box behaves exactly as it does today.

llm: pin that a down workstation is invisible (V-485) fbcca449be

Seven cases. The load-bearing ones are the constraint from 483: an
unconfigured deploy never probes and always reaches the floor, a busy
card degrades silently with the remote untouched, and a remote that dies
between probes still completes the turn and corrects the cached answer on
its way out.

CompleteRemote is pinned not to fall back, because a named gap that
quietly became a 1.7B guess is the failure this whole split exists to
prevent. And 1000 Available calls are pinned to make zero probes.

kami added 1 commit 2026-08-02 16:14:37 +02:00

docs: the workstation supervisor owns llama-server's lifecycle (V-485) 2e97b905b4

The remote model cannot be a llama-server that is simply left running: a
resident 7-14B holds 16GB against the CPT runs the card is for. So what
is always up on the workstation is a supervisor, and llama-server is
loaded while the card is free.

Still not a scheduler. It arbitrates nothing between callers, and Maven
never asks it to start anything.

kami merged commit 94d553570d into master

2026-08-02 17:03:26 +02:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#94