Stop shipping half-written JSON as a reply #42

2026-07-31T16:18:47+02:00

claude commented

2026-07-31 16:18:47 +02:00

Two bugs, one symptom. A talk eval run produced replies that were literally {
and {\n ". Those strings went out as things Maven said, and the daemon would
have sent them to a text-to-speech voice.

The cause was the grammar, not the token cap. The grammar capped the response
field at 400 characters. Measured against Qwen3.5-0.8B at three different token
caps — 256, 768 and 2048 — the reply came back exactly 400 characters every time,
cut mid-word ("Нужно записать и,"). The token limit was never what stopped it.

Also: the parser could not tell "answered in plain prose" from "started a JSON
object and got cut off". Both returned empty, and every caller then shipped the
raw text. Now an unfinished object returns an error and each caller uses its own
fallback. Bare prose still passes through, because small models do sometimes
answer that way and the reply is fine.

A mistake worth reading, because it is easy to repeat: I first set the bound
to 1000 while the token cap was 768. Russian runs about 1.5 characters per token
here, so generation then died on the token cap instead, mid-object — which is
worse, because the reply becomes an unfinished fragment and the caller falls back
to "не знаю.". That showed up as 3, 3 and 6 canned fallbacks per run, from zero.
The two limits have to agree. It is 600 characters against a 1024-token cap now,
with the reasoning written into the comment.

Second commit fixes the eval check that let this through: nonempty scored 27/27
on the run where two replies were {. It only tested for a non-blank string, so
punctuation counted as content. It needs a letter now. Digits alone fail too —
the same run answered "сколько варить яйцо вкрутую?" with "15-16", and another
with "1".

Honest result: fixing the truncation did NOT improve the composite score (12, 13,
10 versus 11, 17, 15 before). A complete rambling wrong answer fails the same
checks a truncated one did. Worth doing anyway — she was emitting { out loud.

All the numbers are written down in TALK-EVAL-31-07-2026.md, which is a
separate PR stacked on this one.

Two things NOT fixed here:

cmd/mavend/replier_llm.go has its own copy of this parser with the same bug,
so the live daemon reply path is still unprotected. That is #396.
#397 is worse than filed. A merely busy llama-server produces a report reading
"0 errors" while a third of the fixture silently answers "не знаю.", because
PhraseChat and PhraseQuery swallow every failure. I voided one of my own runs
to this.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Two bugs, one symptom. A talk eval run produced replies that were literally `{` and `{\n "`. Those strings went out as things Maven said, and the daemon would have sent them to a text-to-speech voice. **The cause was the grammar, not the token cap.** The grammar capped the response field at 400 characters. Measured against Qwen3.5-0.8B at three different token caps — 256, 768 and 2048 — the reply came back exactly 400 characters every time, cut mid-word ("Нужно записать и,"). The token limit was never what stopped it. Also: the parser could not tell "answered in plain prose" from "started a JSON object and got cut off". Both returned empty, and every caller then shipped the raw text. Now an unfinished object returns an error and each caller uses its own fallback. Bare prose still passes through, because small models do sometimes answer that way and the reply is fine. **A mistake worth reading, because it is easy to repeat:** I first set the bound to 1000 while the token cap was 768. Russian runs about 1.5 characters per token here, so generation then died on the token cap instead, mid-object — which is worse, because the reply becomes an unfinished fragment and the caller falls back to "не знаю.". That showed up as 3, 3 and 6 canned fallbacks per run, from zero. The two limits have to agree. It is 600 characters against a 1024-token cap now, with the reasoning written into the comment. Second commit fixes the eval check that let this through: `nonempty` scored 27/27 on the run where two replies were `{`. It only tested for a non-blank string, so punctuation counted as content. It needs a letter now. Digits alone fail too — the same run answered "сколько варить яйцо вкрутую?" with "15-16", and another with "1". Honest result: fixing the truncation did NOT improve the composite score (12, 13, 10 versus 11, 17, 15 before). A complete rambling wrong answer fails the same checks a truncated one did. Worth doing anyway — she was emitting `{` out loud. All the numbers are written down in `TALK-EVAL-31-07-2026.md`, which is a separate PR stacked on this one. Two things NOT fixed here: - `cmd/mavend/replier_llm.go` has its own copy of this parser with the same bug, so **the live daemon reply path is still unprotected**. That is #396. - #397 is worse than filed. A merely busy llama-server produces a report reading "0 errors" while a third of the fixture silently answers "не знаю.", because PhraseChat and PhraseQuery swallow every failure. I voided one of my own runs to this. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

claude added 2 commits 2026-07-31 16:18:47 +02:00

Stop shipping half-written JSON as a reply d7cdcb63bd

Two bugs, one symptom. A run of the talk eval produced replies that were
literally "{" and "{\n  \"" — those strings went out as things Maven said.

First bug: the parser could not tell "the model answered in plain prose"
from "the model started a JSON object and got cut off". Both came back as
empty, and every caller then shipped the raw text. Now an unfinished object
returns an error and each caller uses its own fallback instead. Bare prose
with no JSON in it still passes through, because small models do sometimes
answer that way and the reply is fine.

Second bug, and the actual cause: the grammar capped the response field at
400 characters. I measured it against Qwen3.5-0.8B at three different token
caps — 256, 768 and 2048 — and the reply came back exactly 400 characters
every time, cut mid-word. So the token limit was never what stopped it.
The bound is 1000 now, about six Russian sentences, still low enough to cut
off a repetition loop.

Token caps go from 256 to 768 on the chat and query paths so 1000
characters of Russian actually fits. The nudge path keeps its own cap; a
nudge is meant to be one sentence.

Note: cmd/mavend/replier_llm.go has its own copy of this parser with the
same bug. Left alone here so this commit stays small — that duplicate is
Vikunja #396.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Make the nonempty check look for actual words aa8f5b2ee2

It scored 27/27 on a run where two replies were "{" and "{\n  \"". It only
tested that the string was not blank, so punctuation counted as content and
the worst replies of the run passed the first check.

Now a reply needs at least one letter, Cyrillic or Latin. Latin counts
because answers about ssd or vpn are legitimately part English.

Digits alone fail too. The same run answered "сколько варить яйцо
вкрутую?" with "15-16" — no unit, no words, and the wrong number as well.
That is not something she said.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

kami commented

2026-07-31 20:22:40 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:41 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#42