phraser: gate every llm.Client call on the swap drain
The drain counted only the phrasing paths in internal/phraser. The router, the replier, the mail extractor and the memory evaluator reach llama-server through llm.Client, so quiesce could report zero requests in flight while the router was mid-generation, and the old server was killed under it. The turn then finished on the new model, which is the split turn the swap exists to prevent. llm.Client now enters an optional Gate before every completion and LLMPhraser implements it, so one counter covers every holder of the base URL. A total failure also reported itself as a rollback. Swap set RolledBack on the path where the rollback failed too, so the page rendered "rolled back to — she is still answering, with the old model" over an empty model name and a daemon with no model at all. The total failure has its own flag now, LiveModel stops naming a gguf that is not loaded, and the log says another attempt can recover without a restart, which is true. The swap also ran on the connection every other page shares. ipc.Client holds its mutex for a whole roundtrip with no read deadline on either side, so a load froze /dash, /history and /notifications for minutes. mavweb dials a second connection for /models alone. POST /models joins the route table, and the load settings no longer come off a form that renders no input for them. Found in review of #68.
This commit is contained in:
+13
-1
@@ -363,6 +363,11 @@ func main() {
|
||||
flag.Parse()
|
||||
|
||||
var core ipc.CoreAPI
|
||||
// swapConn — a second connection, for /models and nothing else. A model swap
|
||||
// is a multi-minute IPC call and ipc.Client serialises everything on one
|
||||
// mutex, so sharing the connection would freeze every other page for the
|
||||
// length of the load. See handleModels.
|
||||
var swapConn modelController
|
||||
if *coreSock != "" {
|
||||
c, err := ipc.DialWait(*coreSock, 60*time.Second)
|
||||
if err != nil {
|
||||
@@ -370,6 +375,12 @@ func main() {
|
||||
}
|
||||
defer c.Close()
|
||||
core = c
|
||||
if sc, err := ipc.Dial(*coreSock); err != nil {
|
||||
log.Printf("models: second core connection failed (%v) — /models will share the main one and a swap will block the other pages", err)
|
||||
} else {
|
||||
defer sc.Close()
|
||||
swapConn = sc
|
||||
}
|
||||
}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
@@ -499,13 +510,14 @@ func main() {
|
||||
// /tools, and for a comparable reason: which model is loaded decides how every
|
||||
// utterance is routed and how every reply is worded. GET is read-only.
|
||||
mux.HandleFunc("/models", func(w http.ResponseWriter, r *http.Request) {
|
||||
handleModels(w, r, core, stepUpSession, *requireStepUp)
|
||||
handleModels(w, r, core, swapConn, stepUpSession, *requireStepUp)
|
||||
})
|
||||
|
||||
// State-changing routes on this server, and their gate (Vikunja #317):
|
||||
//
|
||||
// POST /tools step-up — defines argv that internal/tool executes
|
||||
// POST /routines step-up — accepting schedules recurring firing
|
||||
// POST /models step-up — replaces the model that routes and phrases
|
||||
// POST /api/revert step-up — voids the latest fact for a key
|
||||
// POST /api/chat step-up — reaches the router, LLM and the act path
|
||||
// POST /api/signal none — appends a presence fact, no argv, no act
|
||||
|
||||
Reference in New Issue
Block a user