llm: give voice turns priority on the single llama-server slot

llama-server is started without -np, so it serves one request at a time and
everything else queues. Mail extraction is allowed two minutes on a Thinking
1.7B, and the reader hands core up to 25 messages back to back. A turn arriving
mid-extraction therefore waited for whatever was left of that budget: the router
timed out into the classifier cascade and its 36.8% floor, and the phraser, which
has no floor, simply waited. Memory evaluation had the same shape with a five
minute budget.

llm.Gate is the bound. Foreground requests never wait. Background requests run
one at a time and yield while a foreground request is in flight, plus a quiet
window after it that covers the gap between the router call and the phraser call
of one turn. Clients get their priority from llmClientFor or
llmBackgroundClientFor, so which side a caller is on is decided at wiring time.
It gates only what goes through those clients, which the comment on Gate says.

mail intake: the extraction timeout no longer wraps the capture writes. A model
answering at 119 seconds of a 120 second budget left the first CaptureTask one
second and the third none, so candidates the model had already produced were
dropped with a deadline error. The mailbox name is validated before it becomes
provenance, since "email:" is not a source and neither is an arbitrary string
posted at the socket. The enable log prints the normalised candidate bound
rather than the configured one, which said "max 0" and then wrote three.
Found in review of #64.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX
This commit is contained in:
kami
2026-08-01 14:05:07 +04:00
parent 6c81df17ec
commit aee20a6abc
9 changed files with 443 additions and 11 deletions
+37
View File
@@ -23,6 +23,32 @@ type Client struct {
mu sync.RWMutex
base string
http *http.Client
// gate / background — priority on the single llama-server slot. Set once
// at wiring time (SetGate), read on every request. nil gate ⇒ no gating,
// which is what every test and every non-daemon caller gets.
gate *Gate
background bool
}
// SetGate gives this client a priority on the shared llama-server slot. Call it
// immediately after New, before the client is handed to anything: the fields are
// read under the same lock as base, but the intent is one-time wiring, not a
// knob to turn at runtime.
//
// background = false means "he is waiting for this" and never blocks.
// background = true means the request yields to voice turns and runs one at a
// time. See Gate.
func (c *Client) SetGate(g *Gate, background bool) {
c.mu.Lock()
c.gate, c.background = g, background
c.mu.Unlock()
}
func (c *Client) gateFor() (*Gate, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
return c.gate, c.background
}
func New(baseURL string, timeout time.Duration) *Client {
@@ -81,6 +107,17 @@ type resp struct {
}
func (c *Client) Complete(ctx context.Context, r Req) (string, error) {
if g, background := c.gateFor(); g != nil {
if background {
release, err := g.AcquireBackground(ctx)
if err != nil {
return "", err
}
defer release()
} else {
defer g.Foreground()()
}
}
b, _ := json.Marshal(body{
Messages: []msg{{Role: "system", Content: r.System}, {Role: "user", Content: r.User}},
MaxTokens: r.MaxTokens,