eval: score and time the resident model as router (V-320)
Item 2 was blocked because the resident llama-server binds --port 0 inside the container, so no host process can reach it. Cleared by taking the first of the three ways out the task listed: a second llama-server on the same gguf, on a fixed host port. Cascade + resident model scores 75.8% full and 80.2% intent-only at p50 1.19s and p95 1.65s, on the fixture as it now stands at 91 cases. That is a new baseline rather than a movement: 14 cases were added since the 77-case number in CLAUDE.md. The model alone scores 37.4% full against 61.5% intent-only. The gap is slots, not routing. Every reminder case leaves the time to the daemon, which is what the contract asks of it, and the cascade fills them. Item 3: the ~6s figure recorded in the task was one sample through the whole of POST /api/chat, not the router, and is not comparable. Item 4 is still not run. Killing the resident llama-server needs a permission this session does not have, and it now has a second half anyway, since with the workstation up only killing both proves the classifier answers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
This commit is contained in:
@@ -165,6 +165,15 @@ that stood here until 2026-08-02 was contention, not the model.** See `docs/eval
|
||||
p50 825ms / p95 1.2s / max 3.0s and the full cascade at p50 0.80-1.04s. Do not plan latency
|
||||
work off the bakeoff table.
|
||||
|
||||
**Re-measured 2026-08-05 on the fixture as it now stands, 91 cases** (V-320 item 2,
|
||||
`docs/evals/2026-08-05-routing-resident-model.md`): cascade + resident model scores
|
||||
**75.8% full / 80.2% intent-only at p50 1.19s / p95 1.65s**. That is a new baseline and not
|
||||
a movement, because 14 cases were added since the 77-case number above. The model alone
|
||||
scores 37.4% full against 61.5% intent-only, and the gap is slots rather than routing: it
|
||||
routes `reminder` and leaves the time to the daemon, which is what the contract asks. To
|
||||
re-run it, start a **second** llama-server on a fixed host port — the resident one binds
|
||||
`--port 0` inside the container and no host process can reach it.
|
||||
|
||||
**The numbers above are the homesrv floor, not the ceiling.** With the workstation up, routing
|
||||
completes through `llm.Pair` against gemma-4-12b and scores **84.4% full / 93.5% intent-only at
|
||||
p50 329ms** — better than the resident model and about 2.5× faster (`docs/evals/2026-08-02-workstation-gemma4-12b.md`,
|
||||
|
||||
Reference in New Issue
Block a user