talk fixture on the resident model: 2/36 to 25/36 (V-44)

2026-08-05 14:03:45 +04:00
parent 4dbeca5a2e
commit b752ec037e
1 changed files with 60 additions and 0 deletions
@@ -0,0 +1,60 @@
+# Talk fixture against the resident model, 2026-08-05
+
+Vikunja #44 step 1. `MAVEN_LLM_URL=http://127.0.0.1:8899 make eval-phrasing`,
+Qwen3-1.7B-UD-Q4_K_XL on the host, no workstation in the run. The fixture holds
+36 cases now, against 27 when the bakeoff measured it. So the old score is not
+a column in this table.
+
+## Result
+
+| | before the escape fix | after |
+|---|---|---|
+| talk, passes every check | 2/36 (5.6%) | 25/36 (69.4%) |
+| failed generations | 31 | 0 |
+| by path: chat | 0/9 | 4/9 |
+| by path: knowledge | 1/9 | 6/9 |
+| by path: query | 0/9 | 9/9 |
+| by path: reply | 1/9 | 6/9 |
+| feminine | 5/36 | 36/36 |
+| address | 5/36 | 33/36 |
+| ontopic | 2/36 | 28/36 |
+| p50 latency | 3.05s | 2.97s |
+| nudges (15 cases) | 15/15 | 15/15 |
+
+## What the 31 errors were
+
+Not the model. `escapeRawControls` in `internal/phraser/llmphraser.go`, added
+for #537 to repair a raw newline written inside a string, escaped the whole
+object. Qwen3-1.7B pretty-prints: it opens `{` and writes three newlines before
+the first key. Those newlines became a literal backslash-n, which is legal
+nowhere outside a string, so the object stopped parsing and `parseResponseMood`
+reported `errBrokenJSON`.
+
+The comment said escaping unconditionally could not turn valid JSON into
+anything else, because JSON permits no control character outside a string. It
+permits three. Newline, tab and return are whitespace between tokens, and that
+is what pretty-printing is made of.
+
+Every chat reply and every knowledge answer the resident model wrote was being
+discarded for a stub line. The nudge path never showed it, because the nudge
+prompt gets compact JSON back.
+
+## The 11 that still fail
+
+Eight are `ontopic`, three are `address`.
+
+The address failures are all plural imperatives written to a formal listener:
+`держите`, `уточните`, `попробуйте`. Feminine self-reference held in all 36,
+which is the half #122 is training for. So the persona gap the CPT is aimed at
+is now the address half, not the gender half.
+
+The ontopic failures are the resident model answering next to the question
+rather than in it. `chat-joke` describes crying dolls instead of telling one,
+`know-hiccups` calls hiccups an icon, `know-boil-egg` answers about an omelette.
+`chat-about-me` answers "Я - записка", which is the same confabulation the
+bakeoff recorded.
+
+## Not measured here
+
+The workstation. Every number above is the homesrv floor. `make eval-phrasing`
+points at one URL, so a gemma-4-12b column needs its own run.