claude
576dfd8b4c
V-405 measured reach with the classifier only, and the LLM router is the deployed default, so 16/30 was the floor rather than the shipped behaviour. TestReachWithLLMRouter scores the same 30 cases with the model, gated on MAVEN_LLM_URL like TestLLMRouterBaseline. The open question was whether the model writes a literal Praxis capability into the fn slot and reaches a service the classifier structurally cannot. It does not. Praxis is 0/12 with the model alone, exactly what the classifier alone scores, and all twelve fail the same way: local, empty fn. Nothing in the router prompt names a Praxis capability, so there is no string for it to write. So V-516's stage-0 grammars are the only path to Praxis, not a determinism argument. Through the cascade the model scores 28/30 with praxis 11/12, one point above the classifier baseline. Hexis is 10/10 either way. Overreach is 1 in both configurations, under the 4 the harness asserts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
3.3 KiB
Ecosystem reach with the resident model as router
Date: 2026-08-05. Vikunja #517, split out of #405.
Fixture: internal/router/eval/ru_ecosystem_v1.json, 30 held-out Russian cases.
Model: Qwen3-1.7B-UD-Q4_K_XL, llama-server on the host at 127.0.0.1:8899.
Harness: TestReachWithLLMRouter in internal/router/eval/llmrouter_test.go.
The numbers
| configuration | reached the right place | praxis | hexis | none |
|---|---|---|---|---|
| classifier + hash (V-405 floor) | 16/30 | 0/12 | — | — |
| classifier + ONNX, after the V-516 grammars | 27/30 | 11/12 | — | — |
| llm-only (resident model alone) | 17/30 (56.7%) | 0/12 | 10/10 | 7/8 |
| cascade + llm + hash fallback | 28/30 (93.3%) | 11/12 | 10/10 | 7/8 |
Latency: llm-only p50 1.29s, p95 1.65s. Cascade p50 1.11s, p95 1.64s. No case errored in either configuration.
The open question is answered: the model never reaches Praxis
The route grammar lets the model write any string into the fn slot. So it
could in principle emit a literal Praxis capability name, and reach a service
the classifier structurally cannot. It does not. Praxis is 0/12 with the
model alone. That is exactly what the classifier alone scores. Every one of
the twelve fails the same way: the utterance stays local with an empty fn.
So the stage-0 Praxis grammars from V-516 are not a determinism argument. They are the only path to Praxis that exists. Deleting them takes reach from 11/12 back to 0/12 whichever engine is answering.
The failure is not that the model routes these badly in its own terms. It
spreads them across query, fact, system and chat. Those are reasonable
readings of "что требует внимания" and "готово, закрывай" for a model that has
never been told Praxis exists. Nothing in the prompt names a Praxis capability,
so there is no string for it to write.
What the model does buy
Hexis is 10/10 with the model alone, and the mutating tag is 10/15 llm-only against 15/15 through the cascade. The model reaches everything Hexis owns without help, which is the half the act allowlist already names in the prompt.
Cascade + llm scores one point above the classifier baseline: 28/30 against 27/30, the difference being one attention case. That is the same shape as the routing fixture, where the router buys about 4 points rather than a doubling.
The two that still miss
eco-ru-021 "что там с нексусом". Routesquery, stays local, wants Praxis. Asking after a named service reads as a question about a thing, and no grammar claims a service name.eco-ru-029 "сделай это". Routesactand reaches Hexis. The fixture wants nothing reached, because "это" names no target. This is the overreach case and it is the one direction worth failing on. The confirmation binding downstream still resolves a canonical entity id before anything executes. The fixture is right that the turn should have asked.
Overreach is 1 in both configurations, under the 4 the harness asserts.
How to re-run
MAVEN_LLM_URL=http://127.0.0.1:8899 \
deps/go/go/bin/go test -v -count=1 -timeout 40m \
-run TestReachWithLLMRouter ./internal/router/eval/
The host http_proxy answers 503 for 127.0.0.1. noProxyLoopback in the test
excludes it. A run that scores every case as a route error measured the proxy.