1 changed files with 138 additions and 0 deletions
@@ -0,0 +1,138 @@
+# Phrasing evaluation — 31-07-2026
+
+How Maven words a nudge, measured instead of argued. Counterpart to
+`ROUTING-EVAL-31-07-2026.md`.
+
+- Fixture + scorer: `internal/phraser/eval/` (`nudges_v1.json`, 15 cases; `eval.go`, `checks.go`)
+- Reproduce: `MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing`
+- Model: Qwen3.5-0.8B Q4_K_M, the resident model. Not swapped.
+- Commit: `a40bc55` (prompt fix)
+
+Every check is a string or length test a human can read and disagree with. No model
+grades another model here.
+
+## Result
+
+| | before | after |
+|---|---|---|
+| **cases passing every check** | **0/15** | **13/15** |
+| mood in enum | 6/15 | 15/15 |
+| Russian | 2/15 | 14/15 |
+| length (≤120 chars, ≤16 words) | 13/15 | 15/15 |
+| feminine self-reference | 15/15 | 15/15 |
+| no cringe | 13/15 | 15/15 |
+| on topic | 6/15 | 13/15 |
+| p50 latency | 11.4s | 11.4s |
+
+Latency did not move and is not good. 11s to word one nudge on this box.
+
+## The bug reproduced
+
+Yes, exactly as reported. 7 of 15 messages were the literal string `"..."`, and one was
+`"full voice message"`. Both are text copied straight out of the prompt.
+
+The system prompt said:
+
+```
+Respond ONLY with valid JSON: {"response": "full voice message", "mood": "neutral"}
+```
+
+and the user prompt said:
+
+```
+Respond as JSON: {"response": "...", "mood": "..."}
+```
+
+A 0.8B does not read `"..."` as "put your answer here". It reads it as the answer. The
+prompt was a worked example whose worked part was blank, so the model filled the slot by
+copying. This is the whole of finding 1.
+
+## What else was wrong
+
+Four separate faults, all prompt-side:
+
+1. **Placeholder echo** (7 cases) — above.
+2. **Wrong language** (13/15 failed the language check). The prompt was entirely English
+   and said "in the user's language (Russian or English)". The model picked English. It is
+   never English: the nudge is spoken by a Russian piper voice.
+3. **Rule names are English identifiers.** `netdata_critical`, `service_down`, `break` went
+   into the prompt raw. The model cannot nudge about a topic it has not been told in words,
+   so 9/15 were off topic. The daemon knows what its own rules mean; now it says so.
+4. **Mood invented** (`"warm"`, twice). The enum was listed in a parenthesis at the end of
+   an English sentence. Now it is its own line: "ровно одно из: neutral, happy, thinking,
+   tired, confused."
+
+Plus two non-prompt faults the run exposed:
+
+- **The no-parse fallback was English.** When the model returned nothing usable, the body
+  became `fmt.Sprintf("%s — %s", rule, sev)` — `"water — care"` — and that string went to
+  a Russian TTS. Now it falls back to plain Russian.
+- **Durations were English.** `humanDur` returns "3 hours"; it was landing verbatim inside
+  Russian sentences. Nudges now use a Russian formatter.
+
+## Three iterations, and what each taught
+
+| | score | change |
+|---|---|---|
+| baseline | 0/15 | — |
+| iter 1 | 2/15 | Russian prompt, filled-in examples, Russian durations |
+| iter 2 | 11/15 | required keyword per rule, one example instead of five, Russian fallback |
+| iter 3 | **13/15** | examples moved to topics that are not rules |
+
+The interesting step is 1 → 2. Fixing the placeholder did not fix the disease, it moved it:
+the model stopped copying `"..."` and started copying my first example instead. Five nudges
+in a row came back as `"Ты не пил воду три часа. Налей стакан."` regardless of the rule.
+
+**A small model copies the nearest concrete text in its prompt.** That is one failure mode
+with two symptoms. The fix that stuck was making the examples about laundry and a laptop
+battery — topics no rule ever produces, so copying them is visible in the score rather than
+invisibly passing the water cases.
+
+## Do not oversell 13/15
+
+Seven of the thirteen passes are the **deterministic fallback**, not the model:
+`"Напоминаю: таблетки."`, `"Сервис не отвечает."`, `"Критический алярм: проверь диск."`,
+`"Ты давно не пил воду."`. Those are strings this commit added to Go. The model returned
+nothing parseable and the fallback scored.
+
+So the honest reading is roughly **6/15 from the model, 7/15 from a fallback, 2/15 failing**.
+The prompt fix is real — `"..."` is nearly gone and the language and mood checks are clean —
+but a large part of the jump is that failure now degrades into Russian instead of into
+`"water — care"`. That is a genuine improvement for the operator and a weak one for the model.
+
+The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge
+that never says food.
+
+## Broken, found, not fixed
+
+1. **`checkFeminine` only catches half the constraint.** It scans for masculine
+   self-reference and passed 15/15 both runs — but three messages address the *owner* in
+   the feminine: "ты давно не отдыхал**а**", "он не ел". The owner is a man. The check has
+   no second-person gender test, so this scores clean while being exactly the persona
+   failure the constraint exists to prevent. This is the most important gap in the harness.
+2. **Grammar is not checked at all, and it is bad.** `"Он не ел 11 дней"` (it was 11 hours),
+   `"Сонуждились 7 дней"` (not a word), `"Они забыли воду"` (wrong person entirely). Every
+   one of these passes all six checks. The fixture measures properties, not fluency, and at
+   0.8B fluency is the binding constraint.
+3. **Unit confusion.** The model turns hours into days about a third of the time. The
+   prompt now says "11 ч"; it reads it as days.
+4. **11s p50.** Unchanged and untouched here. A nudge the model takes eleven seconds to
+   word has missed its moment. Worth its own task.
+5. **The keyword hint is close to teaching to the test.** `ruleKeywords` names the word the
+   on-topic check looks for. It is defensible — the daemon genuinely knows its rule topics
+   and the model genuinely cannot infer them from `netdata_critical` — but the on-topic
+   number is softer than the others because of it.
+
+## Next steps
+
+1. **Add a second-person gender check** to `checks.go`. Finding 1 above. Until it exists the
+   feminine column means less than it looks like.
+2. **Decide whether the fallback should count as a pass.** Right now `Score` cannot tell a
+   model answer from a fallback. Either mark fallback bodies in `PhrasedNudge` or count them
+   in their own column. Without that, any future prompt change can score well by failing
+   more.
+3. **Attack the 11s.** Nudge phrasing is short and non-interactive; thinking off is the first
+   thing to try, as it was for routing (#376).
+4. **Re-measure when #122 lands.** The CPT'd Qwen3-1.7B is the target. 13/15 with seven
+   fallbacks is the floor it has to beat, and the fluency problems above are the ones a
+   bigger, Russian-trained checkpoint should actually fix.