qa plan: the 30s turn is unbounded whitespace in the grammar, not reasoning (V-287)
Corrects the cause recorded an hour ago. responseGrammar ends with
ws ::= [ \t\n]*, and * is unbounded, so the model emits { and then satisfies
ws with whitespace until max_tokens stops it.
Reproduced on a second Qwen3-1.7B with the same grammar and system prompt:
repeat_penalty 1.0 runs to 512 and returns finish_reason=length, 1.3 stops at
24, and bounding the rule to {0,4} stops at 33 three times out of three with
no penalty at all.
internal/llm.Req sends repeat_penalty and the replier sets 1.3, so that path
is protected by accident. chatReq in the phraser sends none, so PhraseChat,
PhraseQuery, PhraseNudge and PhraseReminder run at the default 1.0.
Two wrong guesses recorded so nobody repeats them: not reasoning tokens, the
probe returned reasoning_content of length 0; and not --cache-ram 512, which
is MiB of prompt cache against a token count.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011x5DgnExQ5XZy8TZPs5bot
This commit is contained in:
+19
-6
@@ -161,12 +161,25 @@ turns look misaligned when they are not.
|
||||
**The slowness now has a cause and a number** (04-08-2026). A spoken turn
|
||||
takes 32 to 34 seconds. One phrasing call is 30.0s of that. STT is 1.0s
|
||||
and routing is under 10ms. Both interactive calls decoded exactly 512 tokens,
|
||||
which is the `chat` cap. So both ran to the cap and were truncated, to
|
||||
produce a 16-character reply. The resident model is a Thinking variant and nothing
|
||||
tells it to skip reasoning. `deploy/mavgpud.json` already passes
|
||||
`--chat-template-kwargs {"enable_thinking":false}` for the workstation and
|
||||
`llamaArgs` does not. Filed as **531**, which also asks whether that gap
|
||||
inflates the workstation comparison.
|
||||
which is the phrasing cap. Both were truncated, to produce a reply of
|
||||
under 25 characters.
|
||||
The cause is `responseGrammar`, not the model. Its last rule is
|
||||
`ws ::= [ \t\n]*`, and `*` is unbounded, so the model emits `{` and then
|
||||
satisfies `ws` with whitespace until `max_tokens` stops it. Reproduced on a
|
||||
second server: at `repeat_penalty` 1.0 it runs to 512 and returns
|
||||
`finish_reason=length`, at 1.3 it stops at 24. Bounding the rule to
|
||||
`[ \t\n]{0,4}` gives a clean stop at 33 tokens three times out of three with
|
||||
no penalty at all.
|
||||
Only some callers are exposed. `internal/llm.Req` sends `repeat_penalty` and
|
||||
the replier sets it to 1.3, so that path is protected by accident. `chatReq`
|
||||
in the phraser sends no penalty, so `PhraseChat`, `PhraseQuery`,
|
||||
`PhraseNudge` and `PhraseReminder` all run at the default 1.0. Filed as
|
||||
**531**.
|
||||
Two guesses were wrong on the way and are recorded so nobody repeats them.
|
||||
It is not reasoning tokens: the probe returned `reasoning_content` of length
|
||||
0, and the grammar constrains output from the first token. It is not the
|
||||
`--cache-ram 512` limit either: that is MiB of prompt cache and the 512 that
|
||||
was hit is a token count.
|
||||
The wandering is a second thing and stays on 287.
|
||||
|
||||
**The wake path cannot be checked here, and that is now the decision rather
|
||||
|
||||
Reference in New Issue
Block a user