Let the router say it does not know #15

2026-07-31T09:38:47+02:00

claude commented

2026-07-31 09:38:47 +02:00

Builds on the router prompt fix (PR 10).

The eval fixture has six cases where the right answer is "ask a clarifying question". The router got 0 of 6 — it always picked something, because the intent enum gave it nothing else to pick. Vikunja #359.

The fix is one extra enum value: unknown. No confidence score, no threshold. A 0.8B's self-reported confidence is uncalibrated noise you would then have to tune a cutoff against — which is the exact trap #359 documents on the embedder side. One more enum token is free in the grammar and cannot come back malformed.

When the model says unknown, LLMRouter.Route returns ok=false, err=nil. That is the path that already existed for a parse failure: the cascade falls through to the classifier. No error is logged, because refusing is not a failure. Nothing new was invented.

The prompt gets one line naming the value, three worked refusals taken from the fixture's own ambiguous cases (сделай это, ну это, потом), and — as important — three counter-examples under "here you do NOT need unknown", so it does not start refusing anything short or unfamiliar.

Tests use the existing mockLLM, no model needed: the grammar accepts the value, the prompt teaches both sides of it, refusal returns ok=false with no error, and a real Router with a seeded classifier actually reaches stage 2 after a refusal.

One existing test changed rather than skipped: TestLLMRouterChatFallback used "unknown" as its example of an unrecognised intent, which is now a real one. It uses "banana" now and asserts the same behaviour.

Two things to know:

Vikunja #359 as written is about the classifier's 0.55 cosine threshold and a top1−top2 margin gate. That half is still untouched. This PR is the LLM-router half.
The remaining blocker for defaulting llm_router on is that slot extraction never runs on an LLM decision, so acts get no function and reminders get no time. That is in progress separately.

Builds on the router prompt fix (PR 10). The eval fixture has six cases where the right answer is "ask a clarifying question". The router got **0 of 6** — it always picked *something*, because the intent enum gave it nothing else to pick. Vikunja #359. **The fix is one extra enum value: `unknown`.** No confidence score, no threshold. A 0.8B's self-reported confidence is uncalibrated noise you would then have to tune a cutoff against — which is the exact trap #359 documents on the embedder side. One more enum token is free in the grammar and cannot come back malformed. When the model says `unknown`, `LLMRouter.Route` returns `ok=false, err=nil`. That is the path that already existed for a parse failure: the cascade falls through to the classifier. No error is logged, because refusing is not a failure. Nothing new was invented. The prompt gets one line naming the value, three worked refusals taken from the fixture's own ambiguous cases (`сделай это`, `ну это`, `потом`), and — as important — three counter-examples under "here you do NOT need unknown", so it does not start refusing anything short or unfamiliar. Tests use the existing `mockLLM`, no model needed: the grammar accepts the value, the prompt teaches both sides of it, refusal returns ok=false with no error, and a real Router with a seeded classifier actually reaches stage 2 after a refusal. One existing test changed rather than skipped: `TestLLMRouterChatFallback` used `"unknown"` as its example of an unrecognised intent, which is now a real one. It uses `"banana"` now and asserts the same behaviour. **Two things to know:** - Vikunja #359 as written is about the *classifier's* 0.55 cosine threshold and a top1−top2 margin gate. That half is still untouched. This PR is the LLM-router half. - The remaining blocker for defaulting `llm_router` on is that slot extraction never runs on an LLM decision, so acts get no function and reminders get no time. That is in progress separately.

claude changed target branch from master to overnight/eval-rerun

2026-07-31 10:39:22 +02:00

claude added 2 commits 2026-07-31 10:39:22 +02:00

Let the LLM router answer "unknown" when it cannot route bd16ca69e5

Chose an 8th enum value over a confidence number: the model already picks
one enum token, so it costs nothing in the grammar, while a score from a
0.8B model would be uncalibrated noise. A refusal returns "no decision"
with no error, which is the fall-through the caller already uses for a
bad parse, so the classifier and its clarify gate take the turn.

Reviewers: the prompt's counter-examples matter most — a small model will
over-use any easy escape hatch. The training workspace copy of the prompt
still needs the same edit (Vikunja #362).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Merge branch 'worktree-agent-a0ab2a9b7439296e3' into overnight-jul31 5d5b0cfd49

kami reviewed 2026-07-31 18:47:50 +02:00

internal/router/llmrouter.go

						
				@@ -79,0 +88,4 @@

				"потом" → {"intent":"unknown"}

				Но не путай — здесь unknown не нужен:

				"сделай кофе" → {"intent":"act","verb":"сделать кофе"}

				"что такое кватернион?" → {"intent":"query","text":"что такое кватернион"}

kami commented

2026-07-31 18:47:49 +02:00

реально - что такое кватернион?

kami reviewed 2026-07-31 20:18:37 +02:00

kami left a comment

Fair question — and it is deliberate, though the reason is not obvious from the line.

It is a few-shot example in the router prompt, not a fixture case (ru_routing_v1.json has no quaternions in it). It sits in the Но не путай — здесь unknown не нужен block, whose whole job is to stop the model reaching for unknown when the topic is unfamiliar. The intended lesson is that intent is decided by the form of the utterance, not by whether the model knows the subject: что такое X? is query for every X. An example with a familiar X can't teach that — the model would route it correctly by recognising the topic and learn nothing. The line right above it (что такое docker?) is the familiar case; this one is the same shape with the topic knocked out on purpose.

So no, nobody expects him to ask about quaternions. The word is a stand-in for "a subject the 1.7B has never heard of", which is a real and frequent case for a model this size — and the failure it guards against was measured: query→unknown/query→fact is Qwen's main error axis on the routing fixture.

One thing to know before touching it: llm/check_prompt_parity.py in the training workspace asserts the Go prompt and the relabelling prompt are byte-identical, so swapping the word means changing both, and it invalidates comparability with the routing scores already recorded. Not worth it for a word choice that is doing its job.

Fair question — and it is deliberate, though the reason is not obvious from the line. It is a **few-shot example in the router prompt**, not a fixture case (`ru_routing_v1.json` has no quaternions in it). It sits in the `Но не путай — здесь unknown не нужен` block, whose whole job is to stop the model reaching for `unknown` when the *topic* is unfamiliar. The intended lesson is that `intent` is decided by the **form** of the utterance, not by whether the model knows the subject: `что такое X?` is `query` for every X. An example with a familiar X can't teach that — the model would route it correctly by recognising the topic and learn nothing. The line right above it (`что такое docker?`) is the familiar case; this one is the same shape with the topic knocked out on purpose. So no, nobody expects him to ask about quaternions. The word is a stand-in for "a subject the 1.7B has never heard of", which is a real and frequent case for a model this size — and the failure it guards against was measured: `query→unknown`/`query→fact` is Qwen's main error axis on the routing fixture. One thing to know before touching it: `llm/check_prompt_parity.py` in the training workspace asserts the Go prompt and the relabelling prompt are byte-identical, so swapping the word means changing both, and it invalidates comparability with the routing scores already recorded. Not worth it for a word choice that is doing its job.

kami commented

2026-07-31 20:21:53 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:21:54 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#15