Constrain the phrasing output with a GBNF grammar #38

2026-07-31T15:35:40+02:00

claude commented

2026-07-31 15:35:40 +02:00

The router has been using a GBNF grammar to force well-formed JSON out of the
model for a while. The phraser was not — it asked nicely in the prompt and hoped.
It did not work: Qwen3.5-0.8B likes to write "Thinking Process:" as plain text
before the JSON, the JSON then never closes, the parse fails, and the plain-text
fallback ships that literal string to him.

So the phraser now sends the same kind of grammar. One const, one helper, wired
into the two places that build a request. Covers nudges, chat, both query
branches and reminders. NoGrammar in the config turns it off if it ever needs
turning off; default is on.

Measured on the talk fixture (27 cases, three runs each, Qwen3.5-0.8B at 4096):

check	before	after
overall	7, 12, 7	14, 15, 8
ellipsis	20, 19, 23	27, 27, 27
lang	13, 16, 15	23, 26, 26
knowledge path	4, 7, 5	8, 6, 5
query path	2, 4, 2	5, 6, 3

The nudge fixture is unchanged at 11-12 of 15.

What is left over is the model, not the plumbing: chat still writes things like
"Здорово, что ты устал от работы" and slips into вы. That is #400 and #402.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

The router has been using a GBNF grammar to force well-formed JSON out of the model for a while. The phraser was not — it asked nicely in the prompt and hoped. It did not work: Qwen3.5-0.8B likes to write "Thinking Process:" as plain text before the JSON, the JSON then never closes, the parse fails, and the plain-text fallback ships that literal string to him. So the phraser now sends the same kind of grammar. One const, one helper, wired into the two places that build a request. Covers nudges, chat, both query branches and reminders. `NoGrammar` in the config turns it off if it ever needs turning off; default is on. Measured on the talk fixture (27 cases, three runs each, Qwen3.5-0.8B at 4096): | check | before | after | |---|---|---| | overall | 7, 12, 7 | 14, 15, 8 | | ellipsis | 20, 19, 23 | 27, 27, 27 | | lang | 13, 16, 15 | 23, 26, 26 | | knowledge path | 4, 7, 5 | 8, 6, 5 | | query path | 2, 4, 2 | 5, 6, 3 | The nudge fixture is unchanged at 11-12 of 15. What is left over is the model, not the plumbing: chat still writes things like "Здорово, что ты устал от работы" and slips into вы. That is #400 and #402. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

claude added 1 commit 2026-07-31 15:35:41 +02:00

Constrain the phrasing output with a GBNF grammar 1890ff5d5d

The 0.8B answered about one chat turn in three with open reasoning as plain text, so no JSON ever closed and the fallback shipped "Thinking Process:" to the user. A grammar makes that output impossible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

kami commented

2026-07-31 20:22:33 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:34 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#38