phraser: gate every llm.Client call on the swap drain

The drain counted only the phrasing paths in internal/phraser. The router, the
replier, the mail extractor and the memory evaluator reach llama-server through
llm.Client, so quiesce could report zero requests in flight while the router was
mid-generation, and the old server was killed under it. The turn then finished
on the new model, which is the split turn the swap exists to prevent. llm.Client
now enters an optional Gate before every completion and LLMPhraser implements
it, so one counter covers every holder of the base URL.

A total failure also reported itself as a rollback. Swap set RolledBack on the
path where the rollback failed too, so the page rendered "rolled back to  — she
is still answering, with the old model" over an empty model name and a daemon
with no model at all. The total failure has its own flag now, LiveModel stops
naming a gguf that is not loaded, and the log says another attempt can recover
without a restart, which is true.

The swap also ran on the connection every other page shares. ipc.Client holds
its mutex for a whole roundtrip with no read deadline on either side, so a load
froze /dash, /history and /notifications for minutes. mavweb dials a second
connection for /models alone. POST /models joins the route table, and the load
settings no longer come off a form that renders no input for them.

Found in review of #68.
This commit is contained in:
kami
2026-08-01 14:15:58 +04:00
parent 810076451f
commit 3ff2a9340a
8 changed files with 312 additions and 17 deletions
+50 -5
View File
@@ -14,17 +14,57 @@ import (
"time"
)
// Gate — admission control for a completion. Enter blocks or refuses while the
// resident model is being swapped, and the returned release says the request is
// done. The phraser implements it: a swap kills the running llama-server, so
// every holder of a base URL has to be counted before the kill, not just the
// phrasing paths.
//
// Without this the drain saw only the phraser's own calls. The LLM router, the
// replier, the mail extractor and the memory evaluator all reach llama-server
// through this client, so a swap could report zero requests in flight and kill
// the server out from under a routing decision. The turn then finished on the
// new model, which is the "half of one model and half of another" the swap is
// supposed to make impossible.
type Gate interface {
Enter() (release func(), err error)
}
type Client struct {
// mu guards base only. The base URL changes when the daemon swaps the
// resident model (Vikunja #250): llama-server is relaunched on a fresh
// port, and every holder of this client — the LLM router, the replier, the
// mail extractor — must follow without being rebuilt. One mutexed field is
// the whole mechanism; a swap re-points the client, it does not replace it.
// mu guards base and gate. The base URL can change when the daemon swaps the
// resident model (Vikunja #250) and every holder of this client — the LLM
// router, the replier, the mail extractor — must follow without being
// rebuilt. A swap re-points the client, it does not replace it.
//
// On the deploy shape the new server binds the same fixed port the killed
// one released (startLlamaProc passes the port out of phraser.listen), so
// SetBaseURL is normally a no-op and the gate is the part doing the work.
// The re-pointing stays because nothing guarantees the port: a phraser
// listening on :0, or a future swap that moves the server, changes the base.
mu sync.RWMutex
base string
gate Gate
http *http.Client
}
// SetGate installs the admission gate. Nil (the default, and what the eval
// harness and the tests use) means no gating.
func (c *Client) SetGate(g Gate) {
c.mu.Lock()
c.gate = g
c.mu.Unlock()
}
func (c *Client) enter() (func(), error) {
c.mu.RLock()
g := c.gate
c.mu.RUnlock()
if g == nil {
return func() {}, nil
}
return g.Enter()
}
func New(baseURL string, timeout time.Duration) *Client {
return &Client{base: baseURL, http: &http.Client{Timeout: timeout}}
}
@@ -81,6 +121,11 @@ type resp struct {
}
func (c *Client) Complete(ctx context.Context, r Req) (string, error) {
release, err := c.enter()
if err != nil {
return "", err
}
defer release()
b, _ := json.Marshal(body{
Messages: []msg{{Role: "system", Content: r.System}, {Role: "user", Content: r.User}},
MaxTokens: r.MaxTokens,