kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Correct the thinking-off routing numbers: it was a measurement bug (#376) #26

Closed
claude wants to merge 2 commits from overnight/thinking-off-pr into overnight/dialogue-persist-pr
pull from: overnight/thinking-off-pr
merge into: kami:overnight/dialogue-persist-pr
Conversation 1 Commits 2 Files Changed 2
+73 -106

2 Commits

Author SHA1 Message Date
kami 9a3bcd7c46 Merge the thinking-off measurement 2026-07-31 13:31:31 +04:00
kami 04c1088088 Measure thinking off on routing properly — it does not win (#376) 
The 67.1% "thinking off" column in ROUTING-EVAL-31-07-2026.md was an
artefact. It came from a hand-rolled HTTP client in the eval test that
did not send repeat_penalty, so it differed from the reference run on two
axes and the penalty was the one that mattered.

Re-scored back to back on an idle box with everything else held equal:
thinking off is identical to thinking on, case for case, same confusion
matrix, same three unparseable replies. A direct probe of the running
llama-server shows enable_thinking, thinking and reasoning_budget are all
ignored for this model on this build, so there was nothing to turn off.

No defaults changed. The misleading third configuration is removed from
internal/router/eval so its table cannot be quoted again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
 2026-07-31 13:30:22 +04:00