61 lines
2.5 KiB
Markdown
61 lines
2.5 KiB
Markdown
# Talk fixture against the resident model, 2026-08-05
|
|
|
|
Vikunja #44 step 1. `MAVEN_LLM_URL=http://127.0.0.1:8899 make eval-phrasing`,
|
|
Qwen3-1.7B-UD-Q4_K_XL on the host, no workstation in the run. The fixture holds
|
|
36 cases now, against 27 when the bakeoff measured it. So the old score is not
|
|
a column in this table.
|
|
|
|
## Result
|
|
|
|
| | before the escape fix | after |
|
|
|---|---|---|
|
|
| talk, passes every check | 2/36 (5.6%) | 25/36 (69.4%) |
|
|
| failed generations | 31 | 0 |
|
|
| by path: chat | 0/9 | 4/9 |
|
|
| by path: knowledge | 1/9 | 6/9 |
|
|
| by path: query | 0/9 | 9/9 |
|
|
| by path: reply | 1/9 | 6/9 |
|
|
| feminine | 5/36 | 36/36 |
|
|
| address | 5/36 | 33/36 |
|
|
| ontopic | 2/36 | 28/36 |
|
|
| p50 latency | 3.05s | 2.97s |
|
|
| nudges (15 cases) | 15/15 | 15/15 |
|
|
|
|
## What the 31 errors were
|
|
|
|
Not the model. `escapeRawControls` in `internal/phraser/llmphraser.go`, added
|
|
for #537 to repair a raw newline written inside a string, escaped the whole
|
|
object. Qwen3-1.7B pretty-prints: it opens `{` and writes three newlines before
|
|
the first key. Those newlines became a literal backslash-n, which is legal
|
|
nowhere outside a string, so the object stopped parsing and `parseResponseMood`
|
|
reported `errBrokenJSON`.
|
|
|
|
The comment said escaping unconditionally could not turn valid JSON into
|
|
anything else, because JSON permits no control character outside a string. It
|
|
permits three. Newline, tab and return are whitespace between tokens, and that
|
|
is what pretty-printing is made of.
|
|
|
|
Every chat reply and every knowledge answer the resident model wrote was being
|
|
discarded for a stub line. The nudge path never showed it, because the nudge
|
|
prompt gets compact JSON back.
|
|
|
|
## The 11 that still fail
|
|
|
|
Eight are `ontopic`, three are `address`.
|
|
|
|
The address failures are all plural imperatives written to a formal listener:
|
|
`держите`, `уточните`, `попробуйте`. Feminine self-reference held in all 36,
|
|
which is the half #122 is training for. So the persona gap the CPT is aimed at
|
|
is now the address half, not the gender half.
|
|
|
|
The ontopic failures are the resident model answering next to the question
|
|
rather than in it. `chat-joke` describes crying dolls instead of telling one,
|
|
`know-hiccups` calls hiccups an icon, `know-boil-egg` answers about an omelette.
|
|
`chat-about-me` answers "Я - записка", which is the same confabulation the
|
|
bakeoff recorded.
|
|
|
|
## Not measured here
|
|
|
|
The workstation. Every number above is the homesrv floor. `make eval-phrasing`
|
|
points at one URL, so a gemma-4-12b column needs its own run.
|