Correct the thinking-off routing numbers: it was a measurement bug (#376) #26

2026-07-31T11:33:06+02:00

claude commented

2026-07-31 11:33:06 +02:00

One commit, and it changes no behaviour. Vikunja #376.

A number I told you earlier was wrong, and this is the correction.

I reported that turning the model's thinking off scored 67.1% on routing against 63.2% with it on, and called it the obvious next win. Measured properly — same fixture (76 cases), same prompt, same grammar, same sampling, idle box, three configs strictly back to back, never concurrent:

	thinking on	thinking off	cascade
intent	59.2%	59.2%	61.8%
full decision	38.2%	38.2%	57.9%
route errors	3	3	0
p50	836ms	920ms	810ms

Not a tie on the headline — identical case for case. Same confusion matrix, same three unparseable replies. So no default changed.

Two independent reasons the old number was junk:

It was measured with a different HTTP client. The thinking-off column used a hand-rolled client inside the test file instead of the real one, and that client never sent repeat_penalty, which the router does send. Two variables moved, and the penalty was the one that mattered.
There was nothing to turn off. Probing the live server directly: enable_thinking=false, thinking=false and reasoning_budget=0 all return a byte-identical answer with the reasoning trace still present, and the server reports the prompt prefix as cached — the template never changed. llama.cpp b9351, Qwen3.5-0.8B.

What's in the diff: the corrected writeup, and the removal of the misleading third configuration and its ad-hoc client from the eval test, so that table can't be quoted again. Net −65 lines.

The full config change — pointer field, accessor, deploy/mavend.json, tests — was built first and then reverted when the measurement came back flat. Worth knowing so you don't go looking for a knob that isn't there.

On sample size, out loud: the fixture is 76 cases, so a 6-point gap is 4–5 cases. Thin enough that shipping a default on it would have been wrong even if it had reproduced. This came back at exactly zero cases plus a mechanical explanation, which makes it easy in the other direction.

Two things this exposed, worth knowing: two of the three route errors are the model writing Russian prose outside the grammar (one reply ends with Пояснение решения:), so the grammar is not fully constraining output on this build. And phrasing was never measured this way — whether thinking helps there is still open, and now blocked on the same "can it even be turned off" question.

One commit, and it changes no behaviour. Vikunja #376. **A number I told you earlier was wrong, and this is the correction.** I reported that turning the model's thinking off scored 67.1% on routing against 63.2% with it on, and called it the obvious next win. Measured properly — same fixture (76 cases), same prompt, same grammar, same sampling, idle box, three configs strictly back to back, never concurrent: | | thinking on | thinking off | cascade | |---|---|---|---| | intent | 59.2% | **59.2%** | 61.8% | | full decision | 38.2% | **38.2%** | 57.9% | | route errors | 3 | 3 | 0 | | p50 | 836ms | 920ms | 810ms | Not a tie on the headline — **identical case for case.** Same confusion matrix, same three unparseable replies. So no default changed. **Two independent reasons the old number was junk:** 1. **It was measured with a different HTTP client.** The thinking-off column used a hand-rolled client inside the test file instead of the real one, and that client never sent `repeat_penalty`, which the router does send. Two variables moved, and the penalty was the one that mattered. 2. **There was nothing to turn off.** Probing the live server directly: `enable_thinking=false`, `thinking=false` and `reasoning_budget=0` all return a **byte-identical** answer with the reasoning trace still present, and the server reports the prompt prefix as cached — the template never changed. llama.cpp b9351, Qwen3.5-0.8B. **What's in the diff:** the corrected writeup, and the removal of the misleading third configuration and its ad-hoc client from the eval test, so that table can't be quoted again. Net **−65 lines**. The full config change — pointer field, accessor, `deploy/mavend.json`, tests — was built first and then **reverted** when the measurement came back flat. Worth knowing so you don't go looking for a knob that isn't there. **On sample size, out loud:** the fixture is 76 cases, so a 6-point gap is 4–5 cases. Thin enough that shipping a default on it would have been wrong even if it *had* reproduced. This came back at exactly zero cases plus a mechanical explanation, which makes it easy in the other direction. **Two things this exposed, worth knowing:** two of the three route errors are the model writing Russian prose *outside* the grammar (one reply ends with `Пояснение решения:`), so the grammar is not fully constraining output on this build. And phrasing was never measured this way — whether thinking helps there is still open, and now blocked on the same "can it even be turned off" question.

claude added 2 commits 2026-07-31 11:33:06 +02:00

Measure thinking off on routing properly — it does not win (#376 ) 04c1088088

The 67.1% "thinking off" column in ROUTING-EVAL-31-07-2026.md was an
artefact. It came from a hand-rolled HTTP client in the eval test that
did not send repeat_penalty, so it differed from the reference run on two
axes and the penalty was the one that mattered.

Re-scored back to back on an idle box with everything else held equal:
thinking off is identical to thinking on, case for case, same confusion
matrix, same three unparseable replies. A direct probe of the running
llama-server shows enable_thinking, thinking and reasoning_budget are all
ignored for this model on this build, so there was nothing to turn off.

No defaults changed. The misleading third configuration is removed from
internal/router/eval so its table cannot be quoted again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Merge the thinking-off measurement 9a3bcd7c46

kami commented

2026-07-31 20:22:15 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:15 +02:00

kami referenced this issue from a commit

2026-08-13 10:11:40 +02:00

Check the digest before paying the phraser (V-687)

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#26