grammar: bound ws, so phrasing stops when it is done (V-531)

A spoken turn took 25-34 seconds and effectively all of it was one phrasing
call generating whitespace. Both interactive turns measured on 2026-08-04
decoded exactly 512 tokens, which is the phrasing MaxTokens, and both ran to
the cap. Background phrasing on the same server in the same window stopped at
32-36 tokens in 4.3s, so it was never the server and never contention.

`ws ::= [ \t\n]*` is a licence to emit whitespace until max_tokens. The model
opens the object, satisfies ws forever, and only the cap stops it. Bounding
the rule fixes it outright with no repeat penalty at all: three runs, three
clean stops at 33 tokens. routeGrammar carried the same rule and is bounded
too — it never ran away only because that path sends routeRepeatPenalty, which
is an accident rather than a defence.

chatReq had no repeat-penalty field at all, so every caller through
chatWithSystem ran at the server default of 1.0 while Replier.PhraseReply sent
1.3 through internal/llm and was protected by accident. Adding it is defence
in depth, not the fix. Two wire structs disagreeing about the sampler is not a
decision anybody made.

finish_reason is parsed on both transports now and a cap hit logs. Both replies
that ran away happened to parse — the grammar had already closed the JSON — so
a truncated generation was indistinguishable from a whole one at every layer
above the response struct.

The phraser test rejects unbounded repetition anywhere in responseGrammar
rather than checking ws by name. A grammar is a budget: every repetition in it
is something the model may do until the token cap, and the cap is not a design.
routeGrammar keeps one, `("," ws action)*`, because a compound utterance is any
number of actions and capping it would drop the last ask.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011x5DgnExQ5XZy8TZPs5bot
This commit is contained in:
2026-08-05 01:25:42 +04:00
parent c586346a60
commit 4cfef41541
5 changed files with 186 additions and 9 deletions
+9
View File
@@ -9,6 +9,7 @@ import (
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"sync"
"time"
@@ -153,6 +154,11 @@ type body struct {
type resp struct {
Choices []struct {
Message msg `json:"message"`
// FinishReason — "length" means the token cap cut the generation off.
// Worth a log line on every path (Vikunja #531): a grammar-constrained
// generation that runs to the cap can still parse, so nothing above
// this struct can tell a truncated answer from a whole one.
FinishReason string `json:"finish_reason"`
} `json:"choices"`
}
@@ -205,6 +211,9 @@ func (c *Client) Complete(ctx context.Context, r Req) (string, error) {
if len(out.Choices) == 0 {
return "", fmt.Errorf("llm: no choices")
}
if out.Choices[0].FinishReason == "length" {
log.Printf("llm: generation hit the %d-token cap (finish_reason=length) — output truncated, or the model was looping", r.MaxTokens)
}
content := out.Choices[0].Message.Content
if content == "" {
content = out.Choices[0].Message.ReasoningContent