kami
d7cdcb63bd
Two bugs, one symptom. A run of the talk eval produced replies that were
literally "{" and "{\n \"" — those strings went out as things Maven said.
First bug: the parser could not tell "the model answered in plain prose"
from "the model started a JSON object and got cut off". Both came back as
empty, and every caller then shipped the raw text. Now an unfinished object
returns an error and each caller uses its own fallback instead. Bare prose
with no JSON in it still passes through, because small models do sometimes
answer that way and the reply is fine.
Second bug, and the actual cause: the grammar capped the response field at
400 characters. I measured it against Qwen3.5-0.8B at three different token
caps — 256, 768 and 2048 — and the reply came back exactly 400 characters
every time, cut mid-word. So the token limit was never what stopped it.
The bound is 1000 now, about six Russian sentences, still low enough to cut
off a repetition loop.
Token caps go from 256 to 768 on the chat and query paths so 1000
characters of Russian actually fits. The nudge path keeps its own cap; a
nudge is meant to be one sentence.
Note: cmd/mavend/replier_llm.go has its own copy of this parser with the
same bug. Left alone here so this commit stays small — that duplicate is
Vikunja #396.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
55 lines
1.9 KiB
Go
55 lines
1.9 KiB
Go
package phraser
|
||
|
||
import (
|
||
"errors"
|
||
"strings"
|
||
"testing"
|
||
)
|
||
|
||
// A reply that starts a JSON object and never finishes it is a failed
|
||
// generation, not a reply. Before this, the parser returned ("", "") for these
|
||
// and every caller then shipped the raw fragment as the thing Maven said. A
|
||
// real run produced replies of literally "{" and "{\n \"".
|
||
func TestParseResponseMoodRejectsUnfinishedJSON(t *testing.T) {
|
||
for _, raw := range []string{
|
||
`{`,
|
||
"{\n \"",
|
||
`{"response": "неполн`,
|
||
`{"response": "текст", "mood":`,
|
||
} {
|
||
text, mood, err := parseResponseMood(raw)
|
||
if !errors.Is(err, errBrokenJSON) {
|
||
t.Errorf("parseResponseMood(%q) err = %v, want errBrokenJSON", raw, err)
|
||
}
|
||
if text != "" || mood != "" {
|
||
t.Errorf("parseResponseMood(%q) leaked %q/%q — a fragment must never come back as a reply", raw, text, mood)
|
||
}
|
||
}
|
||
}
|
||
|
||
// Bare prose is still fine. Small models sometimes answer without any JSON at
|
||
// all, and that reply is usable — so the new error must not swallow it.
|
||
func TestParseResponseMoodAllowsBareProse(t *testing.T) {
|
||
for _, raw := range []string{
|
||
"норм, а ты как?",
|
||
"вот что я нашла: ключ у соседа",
|
||
} {
|
||
text, mood, err := parseResponseMood(raw)
|
||
if err != nil {
|
||
t.Errorf("parseResponseMood(%q) err = %v, want nil", raw, err)
|
||
}
|
||
// No JSON means no fields; the caller ships raw as-is.
|
||
if text != "" || mood != "" {
|
||
t.Errorf("parseResponseMood(%q) = %q/%q, want empty", raw, text, mood)
|
||
}
|
||
}
|
||
}
|
||
|
||
// The measured failure: the model wants more than 400 characters and the old
|
||
// grammar cut it off mid-word. Guards the bound against being tightened back.
|
||
func TestGrammarStringBoundHasRoomForARealAnswer(t *testing.T) {
|
||
if !strings.Contains(responseGrammar, "{0,1000}") {
|
||
t.Error("grammar string bound is not 1000; 400 truncated real replies mid-word (see the comment on responseGrammar)")
|
||
}
|
||
}
|