A spoken turn took 25-34 seconds and effectively all of it was one phrasing
call generating whitespace. Both interactive turns measured on 2026-08-04
decoded exactly 512 tokens, which is the phrasing MaxTokens, and both ran to
the cap. Background phrasing on the same server in the same window stopped at
32-36 tokens in 4.3s, so it was never the server and never contention.
`ws ::= [ \t\n]*` is a licence to emit whitespace until max_tokens. The model
opens the object, satisfies ws forever, and only the cap stops it. Bounding
the rule fixes it outright with no repeat penalty at all: three runs, three
clean stops at 33 tokens. routeGrammar carried the same rule and is bounded
too — it never ran away only because that path sends routeRepeatPenalty, which
is an accident rather than a defence.
chatReq had no repeat-penalty field at all, so every caller through
chatWithSystem ran at the server default of 1.0 while Replier.PhraseReply sent
1.3 through internal/llm and was protected by accident. Adding it is defence
in depth, not the fix. Two wire structs disagreeing about the sampler is not a
decision anybody made.
finish_reason is parsed on both transports now and a cap hit logs. Both replies
that ran away happened to parse — the grammar had already closed the JSON — so
a truncated generation was indistinguishable from a whole one at every layer
above the response struct.
The phraser test rejects unbounded repetition anywhere in responseGrammar
rather than checking ws by name. A grammar is a budget: every repetition in it
is something the model may do until the token cap, and the cap is not a design.
routeGrammar keeps one, `("," ws action)*`, because a compound utterance is any
number of actions and capping it would drop the last ask.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011x5DgnExQ5XZy8TZPs5bot
claude
4cfef41541