Fix the router prompt: questions were filed as facts (48.7% to 60.5%) #10

2026-07-31T00:43:23+02:00

claude commented

2026-07-31 00:43:23 +02:00

Two prompt-level commits. This is the biggest single accuracy win of the night, and it cost 81 lines.

routing fixture, 0.8B	intent before → after	intent+slots before → after	unparseable
llm-only (as deployed)	48.7% → 60.5%	23.7% → 36.8%	2 → 0
cascade + llm	50.0% → 63.2%	32.9% → 47.4%	0 → 0
llm-only, thinking off	48.7% → 67.1%	→ 42.1%	→ 0

Zero unparseable replies in every LLM configuration now. Classifier baselines unchanged, as expected.

The third row is not part of this PR but is the most interesting number in the table: turning thinking off scores higher than leaving it on, and it is also faster. Filed separately — it wants its own measurement, not a hasty flag flip.

Commit 1 — questions were being filed as facts

The query→fact confusion was the single biggest error class in ROUTING-EVAL-31-07-2026.md: 15 cases. Ask "сколько воды я выпил с утра" and Maven stored it as a fact instead of answering it. Now down to 7.

The cause was the order of the numbered decision list in the prompt. The fact test came first, so a 0.8B model matched it and stopped reading. The query tests now sit above it, with an explicit interrogative test (question word or "?"), a line saying query-vs-fact is decided by the form of the sentence rather than the topic, and two worked examples.

No change to the JSON contract or the grammar in this commit — only the wording the model reads.

Commit 2 — the repetition loop

repeat_penalty = 1.15, and the grammar's string rule is now bounded to 120 characters. Both of the unparseable replies were the model looping inside a string until it ran out of room. Zero now.

Two things you should know

The training workspace needs the same edit. llm/check_prompt_parity.py compares this Go prompt against the relabelling copy, and they now differ. Parity is broken until that copy is updated — and it matters, because the CPT'd 1.7B would otherwise be fine-tuned against the old ordering, the one measured at 48.7%. Filed as Vikunja #362; there is a comment above routeSystem saying so too. I cannot reach that repo from here.

Re-measured on a quiet box since — the fix holds. The first run shared llama-server with another eval. On an idle box: llm-only 61.8% / 38.2%, cascade 63.2% / 47.4%, thinking-off 67.1% / 42.1%, and p50 latency is ~1.0s rather than the 4-5s contention produced. Numbers are in ROUTING-EVAL-31-07-2026.md (separate PR).

Do not read 60.5% as a clean win yet. note→fact ×4 is the new dominant error and it is new since the doc — all five ru-note-* cases now go to fact. The agent that made this change flagged the honest possibility that its own edit caused it, by pulling assertive запиши что… phrasings toward fact. The quiet re-run shows the same thing, so it is real and not contention noise — that suspicion stands and should be chased. Filed as Vikunja #375.

Also stable across every run: system→query ×4 ("сколько сейчас времени в киеве"). The prompt has no rule saying clock and calendar questions are system, and the new interrogative test can only have reinforced that. Vikunja #374, and it looks cheap.

MissedClarify is still 6/6, untouched — that is the refusal signal, Vikunja #359, in flight separately.

Two prompt-level commits. **This is the biggest single accuracy win of the night, and it cost 81 lines.** | routing fixture, 0.8B | intent before → after | intent+slots before → after | unparseable | |---|---|---|---| | llm-only (as deployed) | 48.7% → **60.5%** | 23.7% → **36.8%** | 2 → **0** | | cascade + llm | 50.0% → **63.2%** | 32.9% → **47.4%** | 0 → 0 | | llm-only, thinking **off** | 48.7% → **67.1%** | → **42.1%** | → **0** | Zero unparseable replies in every LLM configuration now. Classifier baselines unchanged, as expected. The third row is not part of this PR but is the most interesting number in the table: **turning thinking off scores higher than leaving it on**, and it is also faster. Filed separately — it wants its own measurement, not a hasty flag flip. ## Commit 1 — questions were being filed as facts The `query→fact` confusion was the single biggest error class in `ROUTING-EVAL-31-07-2026.md`: **15 cases**. Ask "сколько воды я выпил с утра" and Maven stored it as a fact instead of answering it. Now down to 7. The cause was the *order* of the numbered decision list in the prompt. The fact test came first, so a 0.8B model matched it and stopped reading. The query tests now sit above it, with an explicit interrogative test (question word or "?"), a line saying query-vs-fact is decided by the **form** of the sentence rather than the topic, and two worked examples. No change to the JSON contract or the grammar in this commit — only the wording the model reads. ## Commit 2 — the repetition loop `repeat_penalty = 1.15`, and the grammar's `string` rule is now bounded to 120 characters. Both of the unparseable replies were the model looping inside a string until it ran out of room. Zero now. ## Two things you should know **The training workspace needs the same edit.** `llm/check_prompt_parity.py` compares this Go prompt against the relabelling copy, and they now differ. Parity is broken until that copy is updated — and it matters, because the CPT'd 1.7B would otherwise be fine-tuned against the old ordering, the one measured at 48.7%. Filed as Vikunja **#362**; there is a comment above `routeSystem` saying so too. I cannot reach that repo from here. **Re-measured on a quiet box since — the fix holds.** The first run shared llama-server with another eval. On an idle box: llm-only **61.8% / 38.2%**, cascade **63.2% / 47.4%**, thinking-off **67.1% / 42.1%**, and p50 latency is ~1.0s rather than the 4-5s contention produced. Numbers are in `ROUTING-EVAL-31-07-2026.md` (separate PR). **Do not read 60.5% as a clean win yet.** `note→fact ×4` is the new dominant error and it is *new since the doc* — all five `ru-note-*` cases now go to fact. The agent that made this change flagged the honest possibility that its own edit caused it, by pulling assertive `запиши что…` phrasings toward fact. **The quiet re-run shows the same thing, so it is real and not contention noise** — that suspicion stands and should be chased. Filed as Vikunja #375. Also stable across every run: `system→query ×4` ("сколько сейчас времени в киеве"). The prompt has no rule saying clock and calendar questions are `system`, and the new interrogative test can only have reinforced that. Vikunja #374, and it looks cheap. `MissedClarify` is still 6/6, untouched — that is the refusal signal, Vikunja #359, in flight separately.

claude referenced this pull request

2026-07-31 11:22:24 +02:00

Keep the conversation across a restart (#363) #25

kami commented

2026-07-31 18:45:43 +02:00

this one shouldn't go in master.

kami changed target branch from master to overnight/away-leak

2026-07-31 19:12:49 +02:00

kami added 3 commits 2026-07-31 19:12:49 +02:00

Route questions to query, not fact 17b47ce206

The router prompt tested "reports current state -> fact" before "wants
information -> query", so a question naming a fact key was written as a fact.
Query now comes first, plus an explicit question test.
Reviewers: the prompt block in llmrouter.go, and the note about the
training-side copy of the prompt that needs the same edit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Stop the router repetition loop d30618ecb7

Route now sets RepeatPenalty on the request, and the grammar's string rule is
capped at 120 characters. Two of 76 fixture cases looped one sentence inside
the text field until MaxTokens, which cut the JSON in half.
Reviewers: the new constant and the grammar string rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Merge branch 'worktree-agent-ab5b5c61a32cac4fe' into overnight-jul31 c47886c2bc

kami commented

2026-07-31 20:21:43 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:21:44 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#10