claude
4cfef41541
A spoken turn took 25-34 seconds and effectively all of it was one phrasing
call generating whitespace. Both interactive turns measured on 2026-08-04
decoded exactly 512 tokens, which is the phrasing MaxTokens, and both ran to
the cap. Background phrasing on the same server in the same window stopped at
32-36 tokens in 4.3s, so it was never the server and never contention.
`ws ::= [ \t\n]*` is a licence to emit whitespace until max_tokens. The model
opens the object, satisfies ws forever, and only the cap stops it. Bounding
the rule fixes it outright with no repeat penalty at all: three runs, three
clean stops at 33 tokens. routeGrammar carried the same rule and is bounded
too — it never ran away only because that path sends routeRepeatPenalty, which
is an accident rather than a defence.
chatReq had no repeat-penalty field at all, so every caller through
chatWithSystem ran at the server default of 1.0 while Replier.PhraseReply sent
1.3 through internal/llm and was protected by accident. Adding it is defence
in depth, not the fix. Two wire structs disagreeing about the sampler is not a
decision anybody made.
finish_reason is parsed on both transports now and a cap hit logs. Both replies
that ran away happened to parse — the grammar had already closed the JSON — so
a truncated generation was indistinguishable from a whole one at every layer
above the response struct.
The phraser test rejects unbounded repetition anywhere in responseGrammar
rather than checking ws by name. A grammar is a budget: every repetition in it
is something the model may do until the token cap, and the cap is not a design.
routeGrammar keeps one, `("," ws action)*`, because a compound utterance is any
number of actions and capping it would drop the last ask.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011x5DgnExQ5XZy8TZPs5bot
36 lines
1.5 KiB
Go
36 lines
1.5 KiB
Go
package router
|
|
|
|
import (
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// routeGrammar carried the same unbounded `ws ::= [ \t\n]*` the phrasing
|
|
// grammar did (Vikunja #531). It never ran away in practice because this path
|
|
// sends routeRepeatPenalty and the phrasing path sent nothing — an accident,
|
|
// not a defence. The bound is the defence.
|
|
//
|
|
// Unlike responseGrammar this grammar does have one legitimate unbounded
|
|
// repetition: `("," ws action)*` in root, because a compound utterance is any
|
|
// number of actions and capping it would silently drop the last ask. That one
|
|
// is bounded in practice by MaxTokens and by each action being fixed-shape.
|
|
// Whitespace has no such excuse.
|
|
func TestRouteGrammarWhitespaceIsBounded(t *testing.T) {
|
|
if !strings.Contains(routeGrammar, `ws ::= [ \t\n]{0,4}`) {
|
|
t.Errorf("the ws rule is not bounded — an unbounded one lets the model emit whitespace to the token cap:\n%s", routeGrammar)
|
|
}
|
|
if strings.Contains(routeGrammar, `[ \t\n]*`) {
|
|
t.Error("routeGrammar still contains an unbounded whitespace repetition")
|
|
}
|
|
}
|
|
|
|
// The penalty stays. It curbs the in-field repetition loop the bound cannot
|
|
// reach — a model repeating whole words inside a string rule is still inside
|
|
// the grammar — and removing it because the grammar is now bounded would be
|
|
// reading this fix as broader than it is.
|
|
func TestRouterStillSendsARepeatPenalty(t *testing.T) {
|
|
if routeRepeatPenalty <= 1.0 {
|
|
t.Errorf("routeRepeatPenalty is %v, which is no penalty at all", routeRepeatPenalty)
|
|
}
|
|
}
|