Files
Maven/docs/evals
claude d49067f7dd eval: score and time the resident model as router (V-320)
Item 2 was blocked because the resident llama-server binds --port 0 inside the
container, so no host process can reach it. Cleared by taking the first of the
three ways out the task listed: a second llama-server on the same gguf, on a
fixed host port.

Cascade + resident model scores 75.8% full and 80.2% intent-only at p50 1.19s
and p95 1.65s, on the fixture as it now stands at 91 cases. That is a new
baseline rather than a movement: 14 cases were added since the 77-case number
in CLAUDE.md.

The model alone scores 37.4% full against 61.5% intent-only. The gap is slots,
not routing. Every reminder case leaves the time to the daemon, which is what
the contract asks of it, and the cascade fills them.

Item 3: the ~6s figure recorded in the task was one sample through the whole
of POST /api/chat, not the router, and is not comparable.

Item 4 is still not run. Killing the resident llama-server needs a permission
this session does not have, and it now has a second half anyway, since with the
workstation up only killing both proves the classifier answers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
2026-08-05 21:31:31 +04:00
..