claude
d49067f7dd
Item 2 was blocked because the resident llama-server binds --port 0 inside the container, so no host process can reach it. Cleared by taking the first of the three ways out the task listed: a second llama-server on the same gguf, on a fixed host port. Cascade + resident model scores 75.8% full and 80.2% intent-only at p50 1.19s and p95 1.65s, on the fixture as it now stands at 91 cases. That is a new baseline rather than a movement: 14 cases were added since the 77-case number in CLAUDE.md. The model alone scores 37.4% full against 61.5% intent-only. The gap is slots, not routing. Every reminder case leaves the time to the daemon, which is what the contract asks of it, and the cascade fills them. Item 3: the ~6s figure recorded in the task was one sample through the whole of POST /api/chat, not the router, and is not comparable. Item 4 is still not run. Killing the resident llama-server needs a permission this session does not have, and it now has a second half anyway, since with the workstation up only killing both proves the classifier answers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
3.1 KiB
Routing with the resident model, re-measured
Date: 2026-08-05. Vikunja #320 items 2 and 3.
Fixture: internal/router/eval/ru_routing_v1.json, now 91 cases (76 ru, 15 en).
Model: Qwen3-1.7B-UD-Q4_K_XL, llama-server on the host at 127.0.0.1:8899.
Harness: TestLLMRouterBaseline, make eval-models.
How the block was cleared
Item 2 was blocked because the resident llama-server binds --host 127.0.0.1 --port 0 inside maven-mavend-1. The port is kernel-assigned, scraped from
stderr and never published, so no go test on the host can reach it. The task
listed three ways out. This run took the first: a second llama-server on
the same gguf, on a fixed host port. The Vega takes the second copy of a 1.7B
without complaint.
The numbers
| configuration | full | intent-only | p50 | p95 |
|---|---|---|---|---|
| llm-only | 34/91 (37.4%) | 61.5% | 1.24s | 1.65s |
| cascade + llm + hash fallback | 69/91 (75.8%) | 80.2% | 1.19s | 1.65s |
By language, through the cascade: ru 57/76, en 12/15. Clarify: 3 false, 1 missed. No errors. Six slots deferred to the daemon.
For comparison, the figures that stood in CLAUDE.md were 72.7% full and 77.9% intent-only, measured on 77 cases. The fixture has grown by 14 cases since, so this is a new baseline rather than a movement.
llm-only is low for a reason that is not routing
37.4% full against 61.5% intent-only is the gap, and it is almost entirely
slots. Every reminder case fails with "no time slot, want one". The model
routes reminder correctly and leaves the time to the daemon, which is what
the contract asks of it. The cascade fills those slots. That is why the same
model scores 38 points higher inside it.
Three cases errored in the llm-only arm and none in the cascade, which is the fallback working as designed.
Item 3: latency
Router p50 1.19s, p95 1.65s, max 1.79s through the cascade. The one earlier
data point in the task, roughly 6s wall clock for привет through
POST /api/chat, was the whole path and not the router. It is not comparable
and should not be quoted as a routing number.
These numbers are the homesrv floor. With the workstation up, routing completes
against gemma-4-12b at p50 329ms, measured separately in
docs/evals/2026-08-02-workstation-gemma4-12b.md.
What still misses
The confusion is concentrated in one direction: query→fact ×4,
query→note ×3, query→system ×3. A question about his own rows that carries
no interrogative reads as a statement to the model. Ten of the twenty-two
failures are that shape, including "я сегодня вообще пил воду" and "чем я
занимался в среду". This is the case V-546's three-head classifier is aimed at.
The two разбуди меня cases clarify at 0.300 instead of routing reminder.
Item 4 is still not run
Killing the resident llama-server to confirm the classifier floor needs a
permission this session does not have. The test is otherwise ready. It now has
a second half. With the workstation up, killing the resident server should
still complete a turn through modelSeam. Only killing both proves the
classifier answers.