diff --git a/docs/evals/2026-08-05-talk-fixture-resident.md b/docs/evals/2026-08-05-talk-fixture-resident.md new file mode 100644 index 0000000..e32b0b6 --- /dev/null +++ b/docs/evals/2026-08-05-talk-fixture-resident.md @@ -0,0 +1,60 @@ +# Talk fixture against the resident model, 2026-08-05 + +Vikunja #44 step 1. `MAVEN_LLM_URL=http://127.0.0.1:8899 make eval-phrasing`, +Qwen3-1.7B-UD-Q4_K_XL on the host, no workstation in the run. The fixture holds +36 cases now, against 27 when the bakeoff measured it. So the old score is not +a column in this table. + +## Result + +| | before the escape fix | after | +|---|---|---| +| talk, passes every check | 2/36 (5.6%) | 25/36 (69.4%) | +| failed generations | 31 | 0 | +| by path: chat | 0/9 | 4/9 | +| by path: knowledge | 1/9 | 6/9 | +| by path: query | 0/9 | 9/9 | +| by path: reply | 1/9 | 6/9 | +| feminine | 5/36 | 36/36 | +| address | 5/36 | 33/36 | +| ontopic | 2/36 | 28/36 | +| p50 latency | 3.05s | 2.97s | +| nudges (15 cases) | 15/15 | 15/15 | + +## What the 31 errors were + +Not the model. `escapeRawControls` in `internal/phraser/llmphraser.go`, added +for #537 to repair a raw newline written inside a string, escaped the whole +object. Qwen3-1.7B pretty-prints: it opens `{` and writes three newlines before +the first key. Those newlines became a literal backslash-n, which is legal +nowhere outside a string, so the object stopped parsing and `parseResponseMood` +reported `errBrokenJSON`. + +The comment said escaping unconditionally could not turn valid JSON into +anything else, because JSON permits no control character outside a string. It +permits three. Newline, tab and return are whitespace between tokens, and that +is what pretty-printing is made of. + +Every chat reply and every knowledge answer the resident model wrote was being +discarded for a stub line. The nudge path never showed it, because the nudge +prompt gets compact JSON back. + +## The 11 that still fail + +Eight are `ontopic`, three are `address`. + +The address failures are all plural imperatives written to a formal listener: +`держите`, `уточните`, `попробуйте`. Feminine self-reference held in all 36, +which is the half #122 is training for. So the persona gap the CPT is aimed at +is now the address half, not the gender half. + +The ontopic failures are the resident model answering next to the question +rather than in it. `chat-joke` describes crying dolls instead of telling one, +`know-hiccups` calls hiccups an icon, `know-boil-egg` answers about an omelette. +`chat-about-me` answers "Я - записка", which is the same confabulation the +bakeoff recorded. + +## Not measured here + +The workstation. Every number above is the homesrv floor. `make eval-phrasing` +points at one URL, so a gemma-4-12b column needs its own run.