claude
576dfd8b4c
V-405 measured reach with the classifier only, and the LLM router is the deployed default, so 16/30 was the floor rather than the shipped behaviour. TestReachWithLLMRouter scores the same 30 cases with the model, gated on MAVEN_LLM_URL like TestLLMRouterBaseline. The open question was whether the model writes a literal Praxis capability into the fn slot and reaches a service the classifier structurally cannot. It does not. Praxis is 0/12 with the model alone, exactly what the classifier alone scores, and all twelve fail the same way: local, empty fn. Nothing in the router prompt names a Praxis capability, so there is no string for it to write. So V-516's stage-0 grammars are the only path to Praxis, not a determinism argument. Through the cascade the model scores 28/30 with praxis 11/12, one point above the classifier baseline. Hexis is 10/10 either way. Overreach is 1 in both configurations, under the 4 the harness asserts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
71 lines
3.3 KiB
Markdown
71 lines
3.3 KiB
Markdown
# Ecosystem reach with the resident model as router
|
||
|
||
Date: 2026-08-05. Vikunja #517, split out of #405.
|
||
Fixture: `internal/router/eval/ru_ecosystem_v1.json`, 30 held-out Russian cases.
|
||
Model: Qwen3-1.7B-UD-Q4_K_XL, llama-server on the host at 127.0.0.1:8899.
|
||
Harness: `TestReachWithLLMRouter` in `internal/router/eval/llmrouter_test.go`.
|
||
|
||
## The numbers
|
||
|
||
| configuration | reached the right place | praxis | hexis | none |
|
||
|---|---|---|---|---|
|
||
| classifier + hash (V-405 floor) | 16/30 | 0/12 | — | — |
|
||
| classifier + ONNX, after the V-516 grammars | 27/30 | 11/12 | — | — |
|
||
| **llm-only** (resident model alone) | **17/30 (56.7%)** | **0/12** | 10/10 | 7/8 |
|
||
| **cascade + llm** + hash fallback | **28/30 (93.3%)** | **11/12** | 10/10 | 7/8 |
|
||
|
||
Latency: llm-only p50 1.29s, p95 1.65s. Cascade p50 1.11s, p95 1.64s.
|
||
No case errored in either configuration.
|
||
|
||
## The open question is answered: the model never reaches Praxis
|
||
|
||
The route grammar lets the model write any string into the `fn` slot. So it
|
||
could in principle emit a literal Praxis capability name, and reach a service
|
||
the classifier structurally cannot. It does not. **Praxis is 0/12 with the
|
||
model alone.** That is exactly what the classifier alone scores. Every one of
|
||
the twelve fails the same way: the utterance stays local with an empty `fn`.
|
||
|
||
So the stage-0 Praxis grammars from V-516 are not a determinism argument. They
|
||
are the only path to Praxis that exists. Deleting them takes reach from 11/12
|
||
back to 0/12 whichever engine is answering.
|
||
|
||
The failure is not that the model routes these badly in its own terms. It
|
||
spreads them across `query`, `fact`, `system` and `chat`. Those are reasonable
|
||
readings of "что требует внимания" and "готово, закрывай" for a model that has
|
||
never been told Praxis exists. Nothing in the prompt names a Praxis capability,
|
||
so there is no string for it to write.
|
||
|
||
## What the model does buy
|
||
|
||
Hexis is 10/10 with the model alone, and the mutating tag is 10/15 llm-only
|
||
against 15/15 through the cascade. The model reaches everything Hexis owns
|
||
without help, which is the half the act allowlist already names in the prompt.
|
||
|
||
Cascade + llm scores one point above the classifier baseline: 28/30 against
|
||
27/30, the difference being one attention case. That is the same shape as the
|
||
routing fixture, where the router buys about 4 points rather than a doubling.
|
||
|
||
## The two that still miss
|
||
|
||
- `eco-ru-021 "что там с нексусом"`. Routes `query`, stays local, wants Praxis.
|
||
Asking after a named service reads as a question about a thing, and no
|
||
grammar claims a service name.
|
||
- `eco-ru-029 "сделай это"`. Routes `act` and reaches Hexis. The fixture wants
|
||
nothing reached, because "это" names no target. This is the overreach case
|
||
and it is the one direction worth failing on. The confirmation binding
|
||
downstream still resolves a canonical entity id before anything executes.
|
||
The fixture is right that the turn should have asked.
|
||
|
||
Overreach is 1 in both configurations, under the 4 the harness asserts.
|
||
|
||
## How to re-run
|
||
|
||
```sh
|
||
MAVEN_LLM_URL=http://127.0.0.1:8899 \
|
||
deps/go/go/bin/go test -v -count=1 -timeout 40m \
|
||
-run TestReachWithLLMRouter ./internal/router/eval/
|
||
```
|
||
|
||
The host `http_proxy` answers 503 for 127.0.0.1. `noProxyLoopback` in the test
|
||
excludes it. A run that scores every case as a route error measured the proxy.
|