Score the LLM router on Qwen3.5-0.8B against the routing fixture (#319)

Completes #319's comparison. Three configurations, because "the LLM router"
was ambiguous: the model alone, the cascade #320 would actually ship (stage-0
grammar → model → classifier floor), and a thinking-off diagnostic.

                      intent-only  full    RU     missed-clarify  p50
  classifier+onnx     36.8%        36.8%   25/61  5/6             31ms
  llm-only (0.8B)     48.7%        23.7%   13/61  6/6             850ms
  cascade+llm (0.8B)  50.0%        32.9%   18/61  6/6             825ms

On the question asked — does the resident model route better? — yes, 50.0%
vs 36.8% intent accuracy. REARCH.md's premise holds. It costs 27x the
latency (p50 825ms vs 31ms, max 3.1s), on the same llama-server the phraser
needs, so it is a trade rather than a free win.

Three things the numbers surface that the headline hides:

query→fact x15 is the dominant failure, four times the classifier's x4 on
the same axis. routeSystem's decision order puts "сообщает или обновляет
состояние" (rule 3) above "хочет получить информацию" (rule 4), so any
utterance naming a fact key matches the earlier rule and a question about
past state reads as an assertion of it. A prompt fix, not a model limit.

The LLM router cannot clarify: llmrouter.go hardcodes Confidence 1.0, so
stage 3's gate can never fire on its decisions — 6/6 missed. With #359's
finding that the classifier's gate is miscalibrated under ONNX, neither path
currently refuses. Flipping #320 as-is removes the refusal lane.

The gap between 50.0% intent and 32.9% full accuracy is entirely slots: the
LLM path fills neither Fn nor Time (it returns Slots.Text for acts, and
Extract never runs on an LLM decision).

Also settles a hypothesis rather than leaving it in the air: thinking mode is
a non-issue under a grammar (identical score), and the grammar's unbounded
("," ws action)* repetition that ran away in an isolated smoke test does not
reproduce under the real prompt — 2 errors in 76, not 76. internal/llm
deliberately does not grow a chat_template_kwargs field.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
This commit is contained in:
kami
2026-07-31 00:50:20 +04:00
parent d34fdf40aa
commit 46259b4571
2 changed files with 211 additions and 3 deletions
+4 -3
View File
@@ -136,7 +136,7 @@ func TestClassifierBaseline(t *testing.T) {
}
// dim 1024: the hash embedder is bag-of-words, so a narrower space would
// collide tokens across intents and measure the hash, not the centroids.
rep, err := Score(context.Background(), "classifier+hash", newBaselineRouter(t, router.NewHashEmbedder(1024)), f)
rep, err := Score(context.Background(), "classifier+hash", newBaselineRouter(t, router.NewHashEmbedder(1024), nil), f)
if err != nil {
t.Fatalf("Score: %v", err)
}
@@ -202,7 +202,7 @@ func TestONNXBaseline(t *testing.T) {
if err != nil {
t.Fatalf("Load: %v", err)
}
rep, err := Score(context.Background(), "classifier+onnx", newBaselineRouter(t, emb), f)
rep, err := Score(context.Background(), "classifier+onnx", newBaselineRouter(t, emb, nil), f)
if err != nil {
t.Fatalf("Score: %v", err)
}
@@ -213,7 +213,7 @@ func TestONNXBaseline(t *testing.T) {
// set, same extractor floors, same seed corpus, same confidence gate — so the
// score reflects the deployed cascade and not a test-local approximation. Only
// the embedder varies, and that is the axis being measured.
func newBaselineRouter(t *testing.T, emb router.Embedder) *router.Router {
func newBaselineRouter(t *testing.T, emb router.Embedder, llmR *router.LLMRouter) *router.Router {
t.Helper()
acts := router.DefaultActMatcher{Fns: actFns}
cls := router.NewClassifier(emb)
@@ -245,6 +245,7 @@ func newBaselineRouter(t *testing.T, emb router.Embedder) *router.Router {
// The deployed gate, not a test-local one: a fixture scored at a looser
// threshold reports an accuracy no real turn would see.
Threshold: config.DefaultRouterThreshold,
LLM: llmR,
})
}