grammar: bound ws, so phrasing stops when it is done (V-531)
A spoken turn took 25-34 seconds and effectively all of it was one phrasing
call generating whitespace. Both interactive turns measured on 2026-08-04
decoded exactly 512 tokens, which is the phrasing MaxTokens, and both ran to
the cap. Background phrasing on the same server in the same window stopped at
32-36 tokens in 4.3s, so it was never the server and never contention.
`ws ::= [ \t\n]*` is a licence to emit whitespace until max_tokens. The model
opens the object, satisfies ws forever, and only the cap stops it. Bounding
the rule fixes it outright with no repeat penalty at all: three runs, three
clean stops at 33 tokens. routeGrammar carried the same rule and is bounded
too — it never ran away only because that path sends routeRepeatPenalty, which
is an accident rather than a defence.
chatReq had no repeat-penalty field at all, so every caller through
chatWithSystem ran at the server default of 1.0 while Replier.PhraseReply sent
1.3 through internal/llm and was protected by accident. Adding it is defence
in depth, not the fix. Two wire structs disagreeing about the sampler is not a
decision anybody made.
finish_reason is parsed on both transports now and a cap hit logs. Both replies
that ran away happened to parse — the grammar had already closed the JSON — so
a truncated generation was indistinguishable from a whole one at every layer
above the response struct.
The phraser test rejects unbounded repetition anywhere in responseGrammar
rather than checking ws by name. A grammar is a budget: every repetition in it
is something the model may do until the token cap, and the cap is not a design.
routeGrammar keeps one, `("," ws action)*`, because a compound utterance is any
number of actions and capping it would drop the last ask.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011x5DgnExQ5XZy8TZPs5bot
This commit is contained in:
@@ -9,6 +9,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -153,6 +154,11 @@ type body struct {
|
||||
type resp struct {
|
||||
Choices []struct {
|
||||
Message msg `json:"message"`
|
||||
// FinishReason — "length" means the token cap cut the generation off.
|
||||
// Worth a log line on every path (Vikunja #531): a grammar-constrained
|
||||
// generation that runs to the cap can still parse, so nothing above
|
||||
// this struct can tell a truncated answer from a whole one.
|
||||
FinishReason string `json:"finish_reason"`
|
||||
} `json:"choices"`
|
||||
}
|
||||
|
||||
@@ -205,6 +211,9 @@ func (c *Client) Complete(ctx context.Context, r Req) (string, error) {
|
||||
if len(out.Choices) == 0 {
|
||||
return "", fmt.Errorf("llm: no choices")
|
||||
}
|
||||
if out.Choices[0].FinishReason == "length" {
|
||||
log.Printf("llm: generation hit the %d-token cap (finish_reason=length) — output truncated, or the model was looping", r.MaxTokens)
|
||||
}
|
||||
content := out.Choices[0].Message.Content
|
||||
if content == "" {
|
||||
content = out.Choices[0].Message.ReasoningContent
|
||||
|
||||
Reference in New Issue
Block a user