From d49067f7dd48d839570d61578f4d8fe8b0bd7f48 Mon Sep 17 00:00:00 2001 From: claude Date: Wed, 5 Aug 2026 21:31:31 +0400 Subject: [PATCH] eval: score and time the resident model as router (V-320) Item 2 was blocked because the resident llama-server binds --port 0 inside the container, so no host process can reach it. Cleared by taking the first of the three ways out the task listed: a second llama-server on the same gguf, on a fixed host port. Cascade + resident model scores 75.8% full and 80.2% intent-only at p50 1.19s and p95 1.65s, on the fixture as it now stands at 91 cases. That is a new baseline rather than a movement: 14 cases were added since the 77-case number in CLAUDE.md. The model alone scores 37.4% full against 61.5% intent-only. The gap is slots, not routing. Every reminder case leaves the time to the daemon, which is what the contract asks of it, and the cascade fills them. Item 3: the ~6s figure recorded in the task was one sample through the whole of POST /api/chat, not the router, and is not comparable. Item 4 is still not run. Killing the resident llama-server needs a permission this session does not have, and it now has a second half anyway, since with the workstation up only killing both proves the classifier answers. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy --- CLAUDE.md | 9 +++ .../2026-08-05-routing-resident-model.md | 69 +++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 docs/evals/2026-08-05-routing-resident-model.md diff --git a/CLAUDE.md b/CLAUDE.md index eb43790..6bcbc5f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -165,6 +165,15 @@ that stood here until 2026-08-02 was contention, not the model.** See `docs/eval p50 825ms / p95 1.2s / max 3.0s and the full cascade at p50 0.80-1.04s. Do not plan latency work off the bakeoff table. +**Re-measured 2026-08-05 on the fixture as it now stands, 91 cases** (V-320 item 2, +`docs/evals/2026-08-05-routing-resident-model.md`): cascade + resident model scores +**75.8% full / 80.2% intent-only at p50 1.19s / p95 1.65s**. That is a new baseline and not +a movement, because 14 cases were added since the 77-case number above. The model alone +scores 37.4% full against 61.5% intent-only, and the gap is slots rather than routing: it +routes `reminder` and leaves the time to the daemon, which is what the contract asks. To +re-run it, start a **second** llama-server on a fixed host port — the resident one binds +`--port 0` inside the container and no host process can reach it. + **The numbers above are the homesrv floor, not the ceiling.** With the workstation up, routing completes through `llm.Pair` against gemma-4-12b and scores **84.4% full / 93.5% intent-only at p50 329ms** — better than the resident model and about 2.5× faster (`docs/evals/2026-08-02-workstation-gemma4-12b.md`, diff --git a/docs/evals/2026-08-05-routing-resident-model.md b/docs/evals/2026-08-05-routing-resident-model.md new file mode 100644 index 0000000..2a10c46 --- /dev/null +++ b/docs/evals/2026-08-05-routing-resident-model.md @@ -0,0 +1,69 @@ +# Routing with the resident model, re-measured + +Date: 2026-08-05. Vikunja #320 items 2 and 3. +Fixture: `internal/router/eval/ru_routing_v1.json`, now **91 cases** (76 ru, 15 en). +Model: Qwen3-1.7B-UD-Q4_K_XL, llama-server on the host at 127.0.0.1:8899. +Harness: `TestLLMRouterBaseline`, `make eval-models`. + +## How the block was cleared + +Item 2 was blocked because the resident llama-server binds `--host 127.0.0.1 +--port 0` inside `maven-mavend-1`. The port is kernel-assigned, scraped from +stderr and never published, so no `go test` on the host can reach it. The task +listed three ways out. This run took the first: a **second** llama-server on +the same gguf, on a fixed host port. The Vega takes the second copy of a 1.7B +without complaint. + +## The numbers + +| configuration | full | intent-only | p50 | p95 | +|---|---|---|---|---| +| llm-only | 34/91 (37.4%) | 61.5% | 1.24s | 1.65s | +| cascade + llm + hash fallback | 69/91 (75.8%) | 80.2% | 1.19s | 1.65s | + +By language, through the cascade: ru 57/76, en 12/15. +Clarify: 3 false, 1 missed. No errors. Six slots deferred to the daemon. + +For comparison, the figures that stood in CLAUDE.md were 72.7% full and 77.9% +intent-only, measured on 77 cases. The fixture has grown by 14 cases since, so +this is a new baseline rather than a movement. + +## llm-only is low for a reason that is not routing + +37.4% full against 61.5% intent-only is the gap, and it is almost entirely +slots. Every reminder case fails with "no time slot, want one". The model +routes `reminder` correctly and leaves the time to the daemon, which is what +the contract asks of it. The cascade fills those slots. That is why the same +model scores 38 points higher inside it. + +Three cases errored in the llm-only arm and none in the cascade, which is the +fallback working as designed. + +## Item 3: latency + +Router p50 1.19s, p95 1.65s, max 1.79s through the cascade. The one earlier +data point in the task, roughly 6s wall clock for `привет` through +`POST /api/chat`, was the whole path and not the router. It is not comparable +and should not be quoted as a routing number. + +These numbers are the homesrv floor. With the workstation up, routing completes +against gemma-4-12b at p50 329ms, measured separately in +`docs/evals/2026-08-02-workstation-gemma4-12b.md`. + +## What still misses + +The confusion is concentrated in one direction: `query→fact ×4`, +`query→note ×3`, `query→system ×3`. A question about his own rows that carries +no interrogative reads as a statement to the model. Ten of the twenty-two +failures are that shape, including "я сегодня вообще пил воду" and "чем я +занимался в среду". This is the case V-546's three-head classifier is aimed at. + +The two `разбуди меня` cases clarify at 0.300 instead of routing `reminder`. + +## Item 4 is still not run + +Killing the resident llama-server to confirm the classifier floor needs a +permission this session does not have. The test is otherwise ready. It now has +a second half. With the workstation up, killing the resident server should +still complete a turn through `modelSeam`. Only killing both proves the +classifier answers.