llm: give voice turns priority on the single llama-server slot

llama-server is started without -np, so it serves one request at a time and
everything else queues. Mail extraction is allowed two minutes on a Thinking
1.7B, and the reader hands core up to 25 messages back to back. A turn arriving
mid-extraction therefore waited for whatever was left of that budget: the router
timed out into the classifier cascade and its 36.8% floor, and the phraser, which
has no floor, simply waited. Memory evaluation had the same shape with a five
minute budget.

llm.Gate is the bound. Foreground requests never wait. Background requests run
one at a time and yield while a foreground request is in flight, plus a quiet
window after it that covers the gap between the router call and the phraser call
of one turn. Clients get their priority from llmClientFor or
llmBackgroundClientFor, so which side a caller is on is decided at wiring time.
It gates only what goes through those clients, which the comment on Gate says.

mail intake: the extraction timeout no longer wraps the capture writes. A model
answering at 119 seconds of a 120 second budget left the first CaptureTask one
second and the third none, so candidates the model had already produced were
dropped with a deadline error. The mailbox name is validated before it becomes
provenance, since "email:" is not a source and neither is an arbitrary string
posted at the socket. The enable log prints the normalised candidate bound
rather than the configured one, which said "max 0" and then wrote three.
Found in review of #64.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX
This commit is contained in:
kami
2026-08-01 14:05:07 +04:00
parent 6c81df17ec
commit aee20a6abc
9 changed files with 443 additions and 11 deletions
+21 -3
View File
@@ -35,6 +35,14 @@ import (
// search input" — mail is the same class), and Evidence keeps only the subject
// line, so the review page shows him where a candidate came from without the
// store growing a copy of his mailbox.
//
// One constraint for whoever adds task context to a prompt later: a candidate's
// text is a model paraphrase of the content of his mail, and it lives in
// tasks.text. "Maven never sends his mail anywhere" holds today because nothing
// assembles a context block out of live tasks. The moment something does, mail
// content reaches whatever that block is sent to, and an outbound search would
// be sending his mailbox out a paraphrase at a time. Tasks sourced "email:" have
// to be excluded there, not here.
// MaxCandidates — at most this many candidates per message, enforced by the
// grammar. A mail with four tasks in it is a mail he has to read himself; a
@@ -81,6 +89,12 @@ func NewExtractor(c Completer, max int, contextBlock func() string) *Extractor {
return &Extractor{llm: c, max: max, contextBlock: contextBlock}
}
// Max — the normalised candidate bound. Exported so the daemon logs what it will
// actually allow rather than what the config file said: 0 in the config means
// MaxCandidates here, and logging the raw value said "max 0" and then wrote
// three.
func (e *Extractor) Max() int { return e.max }
// extractGrammar — GBNF pinning the answer to a bounded array of fixed-shape
// candidates. Same reasoning as memeval's evalGrammar and the router's
// routeGrammar: the shape and the length bound are what keep a small model from
@@ -190,9 +204,13 @@ func renderForModel(msg Message) string {
return b.String()
}
// parseCandidates decodes the grammar-constrained reply, tolerating the
// wrappers a Thinking model sometimes leaves around it (a fenced block, or
// leading reasoning before the array).
// parseCandidates decodes the reply and trims a fenced block or stray prose
// around the array.
//
// Through Extract that tolerance is unreachable: extractGrammar pins the first
// token to "[", so the model cannot emit reasoning before it. It is kept for
// callers that pass a raw reply from an ungrammared path, and the note is here
// so the next reader does not conclude that thinking output is expected.
func parseCandidates(raw string) ([]Candidate, error) {
s := strings.TrimSpace(raw)
if i := strings.Index(s, "["); i > 0 {