kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Write down every conversational eval score from tonight #43

Closed
claude wants to merge 1 commits from overnight/eval-writeup into overnight/fix-truncation
pull from: overnight/eval-writeup
merge into: kami:overnight/fix-truncation
Conversation 1 Commits 1 Files Changed 1
+150
claude commented 2026-07-31 16:19:28 +02:00
Contributor
Copy Link
Copy Source

All four configurations on the 27-case talk fixture, three runs each: no grammar,
plus GBNF grammar, plus Russian prompts, plus the truncation fix. Composite,
per-path, per-check, and the command to reproduce.

The short version: the plumbing got fixed and the score barely moved. Grammar was
the real win (7,12,7 → 14,15,8). Russian prompts helped a little and cut p50
latency from ~11.5s to ~2.3s. The truncation fix was necessary and bought nothing.

Everything still failing is the model not knowing things or not holding a
constraint. address is stuck at 21-22/27 while the prompt explicitly forbids
exactly what she writes, and feminine still produces "я готов", "я понял".
Prompting is out of levers.

Three things written down because they are easy to lose:

  • The truncation cause was the grammar 400-character bound, not the token cap.
    Same 400 characters at every cap from 256 to 2048.
  • Then I set the bound to 1000 against a 768-token cap and made it worse.
  • One run is marked void: I ran an agent against the same llama-server and
    the report still claimed zero errors while a third of the fixture silently
    answered "не знаю.". That is #397, and it means any run made on a busy box
    cannot be trusted.

Also records that part of the ontopic column is measuring the fixture rather
than the model — chat-how-are-you fails a perfectly good reply because the
want_any list is too narrow. Left unfixed on purpose so the runs above stay
comparable.

Next steps listed at the bottom, including scoring Qwen3.5-2B-UD-Q4_K_XL, which
is already on the box and has never been measured. Not the 4B — too big for this
machine, owner call.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

All four configurations on the 27-case talk fixture, three runs each: no grammar, plus GBNF grammar, plus Russian prompts, plus the truncation fix. Composite, per-path, per-check, and the command to reproduce. The short version: the plumbing got fixed and the score barely moved. Grammar was the real win (7,12,7 → 14,15,8). Russian prompts helped a little and cut p50 latency from ~11.5s to ~2.3s. The truncation fix was necessary and bought nothing. Everything still failing is the model not knowing things or not holding a constraint. `address` is stuck at 21-22/27 while the prompt explicitly forbids exactly what she writes, and `feminine` still produces "я готов", "я понял". Prompting is out of levers. Three things written down because they are easy to lose: - The truncation cause was the grammar 400-character bound, not the token cap. Same 400 characters at every cap from 256 to 2048. - Then I set the bound to 1000 against a 768-token cap and made it worse. - One run is marked **void**: I ran an agent against the same llama-server and the report still claimed zero errors while a third of the fixture silently answered "не знаю.". That is #397, and it means any run made on a busy box cannot be trusted. Also records that part of the `ontopic` column is measuring the fixture rather than the model — `chat-how-are-you` fails a perfectly good reply because the `want_any` list is too narrow. Left unfixed on purpose so the runs above stay comparable. Next steps listed at the bottom, including scoring `Qwen3.5-2B-UD-Q4_K_XL`, which is already on the box and has never been measured. Not the 4B — too big for this machine, owner call. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
claude added 1 commit 2026-07-31 16:19:29 +02:00
Write down every conversational eval score from tonight 0b90952e55 
Records all four configurations on the 27-case talk fixture, three runs
each: no grammar, plus grammar, plus Russian prompts, plus the truncation
fix. Composite, per-path and per-check, with the reproduce command.

The short version is that the plumbing got fixed and the score barely
moved. Grammar was the real win. Russian prompts helped a little and cut
latency by 5x. The truncation fix was necessary and bought nothing.

Also writes down three things that are easy to lose:

- The truncation cause was the grammar's 400-character bound, not the
  token cap. Measured at three caps, same 400 characters every time.
- Then I set the bound to 1000 against a 768-token cap and made it worse.
  The two limits have to agree.
- One run is contaminated and marked void: I ran an agent against the same
  llama-server, and the report still claimed zero errors while a third of
  the fixture silently answered "не знаю.". That is #397 and it is worse
  than filed — a busy server is indistinguishable from bad phrasing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
kami commented 2026-07-31 20:22:42 +02:00
Owner
Copy Link
Copy Source

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.
kami closed this pull request 2026-07-31 20:22:43 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kami
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kami/Maven#43
Delete Branch "overnight/eval-writeup"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?