llm: give voice turns priority on the single llama-server slot
llama-server is started without -np, so it serves one request at a time and everything else queues. Mail extraction is allowed two minutes on a Thinking 1.7B, and the reader hands core up to 25 messages back to back. A turn arriving mid-extraction therefore waited for whatever was left of that budget: the router timed out into the classifier cascade and its 36.8% floor, and the phraser, which has no floor, simply waited. Memory evaluation had the same shape with a five minute budget. llm.Gate is the bound. Foreground requests never wait. Background requests run one at a time and yield while a foreground request is in flight, plus a quiet window after it that covers the gap between the router call and the phraser call of one turn. Clients get their priority from llmClientFor or llmBackgroundClientFor, so which side a caller is on is decided at wiring time. It gates only what goes through those clients, which the comment on Gate says. mail intake: the extraction timeout no longer wraps the capture writes. A model answering at 119 seconds of a 120 second budget left the first CaptureTask one second and the third none, so candidates the model had already produced were dropped with a deadline error. The mailbox name is validated before it becomes provenance, since "email:" is not a source and neither is an arbitrary string posted at the socket. The enable log prints the normalised candidate bound rather than the configured one, which said "max 0" and then wrote three. Found in review of #64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX
This commit is contained in:
@@ -108,6 +108,34 @@ func wireModelSwap(srv *ipc.Server, phr phraser.Phraser, cfg *config.Config) {
|
||||
// rebuilt, so nothing that holds it has to know a swap happened.
|
||||
func llmClientFor(lp *phraser.LLMPhraser, timeout time.Duration) *llm.Client {
|
||||
c := llm.New(lp.BaseURL(), timeout)
|
||||
c.SetGate(residentGate, false)
|
||||
lp.OnSwap(func(base string) { c.SetBaseURL(base) })
|
||||
return c
|
||||
}
|
||||
|
||||
// backgroundQuiet — how long background work stays off the resident model after
|
||||
// a foreground request. Long enough to cover the gap between the router call and
|
||||
// the phraser call of one turn (router p50 is ~2.7s on this box), short enough
|
||||
// that a quiet mailbox is still read promptly.
|
||||
const backgroundQuiet = 10 * time.Second
|
||||
|
||||
// residentGate — the priority gate on the one llama-server slot, shared by every
|
||||
// client llmClientFor builds. Package level because the daemon owns exactly one
|
||||
// llama-server: two gates would be two opinions about one queue.
|
||||
//
|
||||
// The problem it solves: llama-server runs a single slot, so requests queue. Mail
|
||||
// extraction is allowed two minutes, and a first poll can hand core 25 messages
|
||||
// back to back. Without a gate a voice turn arriving mid-extraction waits for
|
||||
// whatever is left of that budget, the router times out into the classifier
|
||||
// cascade at its 36.8% floor, and the phraser just waits.
|
||||
var residentGate = llm.NewGate(backgroundQuiet)
|
||||
|
||||
// llmBackgroundClientFor is llmClientFor for work nobody is waiting on: mail
|
||||
// extraction and memory evaluation. Same swap-following client, but it yields
|
||||
// to voice turns and only one such request runs at a time.
|
||||
func llmBackgroundClientFor(lp *phraser.LLMPhraser, timeout time.Duration) *llm.Client {
|
||||
c := llm.New(lp.BaseURL(), timeout)
|
||||
c.SetGate(residentGate, true)
|
||||
lp.OnSwap(func(base string) { c.SetBaseURL(base) })
|
||||
return c
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user