2.5 KiB
Talk fixture against the resident model, 2026-08-05
Vikunja #44 step 1. MAVEN_LLM_URL=http://127.0.0.1:8899 make eval-phrasing,
Qwen3-1.7B-UD-Q4_K_XL on the host, no workstation in the run. The fixture holds
36 cases now, against 27 when the bakeoff measured it. So the old score is not
a column in this table.
Result
| before the escape fix | after | |
|---|---|---|
| talk, passes every check | 2/36 (5.6%) | 25/36 (69.4%) |
| failed generations | 31 | 0 |
| by path: chat | 0/9 | 4/9 |
| by path: knowledge | 1/9 | 6/9 |
| by path: query | 0/9 | 9/9 |
| by path: reply | 1/9 | 6/9 |
| feminine | 5/36 | 36/36 |
| address | 5/36 | 33/36 |
| ontopic | 2/36 | 28/36 |
| p50 latency | 3.05s | 2.97s |
| nudges (15 cases) | 15/15 | 15/15 |
What the 31 errors were
Not the model. escapeRawControls in internal/phraser/llmphraser.go, added
for #537 to repair a raw newline written inside a string, escaped the whole
object. Qwen3-1.7B pretty-prints: it opens { and writes three newlines before
the first key. Those newlines became a literal backslash-n, which is legal
nowhere outside a string, so the object stopped parsing and parseResponseMood
reported errBrokenJSON.
The comment said escaping unconditionally could not turn valid JSON into anything else, because JSON permits no control character outside a string. It permits three. Newline, tab and return are whitespace between tokens, and that is what pretty-printing is made of.
Every chat reply and every knowledge answer the resident model wrote was being discarded for a stub line. The nudge path never showed it, because the nudge prompt gets compact JSON back.
The 11 that still fail
Eight are ontopic, three are address.
The address failures are all plural imperatives written to a formal listener:
держите, уточните, попробуйте. Feminine self-reference held in all 36,
which is the half #122 is training for. So the persona gap the CPT is aimed at
is now the address half, not the gender half.
The ontopic failures are the resident model answering next to the question
rather than in it. chat-joke describes crying dolls instead of telling one,
know-hiccups calls hiccups an icon, know-boil-egg answers about an omelette.
chat-about-me answers "Я - записка", which is the same confabulation the
bakeoff recorded.
Not measured here
The workstation. Every number above is the homesrv floor. make eval-phrasing
points at one URL, so a gemma-4-12b column needs its own run.