Maven/PHRASING-EVAL-31-07-2026.md

Phrasing evaluation — 31-07-2026

How Maven words a nudge, measured instead of argued. Counterpart to ROUTING-EVAL-31-07-2026.md.

• Fixture + scorer: internal/phraser/eval/ (nudges_v1.json, 15 cases; eval.go, checks.go)
• Reproduce: MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
• Model: Qwen3.5-0.8B Q4_K_M, the resident model. Not swapped.
• Commit: a40bc55 (prompt fix)

Every check is a string or length test a human can read and disagree with. No model grades another model here.

Result

	before	after
cases passing every check	0/15	13/15
mood in enum	6/15	15/15
Russian	2/15	14/15
length (≤120 chars, ≤16 words)	13/15	15/15
feminine self-reference	15/15	15/15
no cringe	13/15	15/15
on topic	6/15	13/15
p50 latency	11.4s	11.4s

Latency did not move and is not good. 11s to word one nudge on this box.

The bug reproduced

Yes, exactly as reported. 7 of 15 messages were the literal string "...", and one was "full voice message". Both are text copied straight out of the prompt.

The system prompt said:

Respond ONLY with valid JSON: {"response": "full voice message", "mood": "neutral"}

and the user prompt said:

Respond as JSON: {"response": "...", "mood": "..."}

A 0.8B does not read "..." as "put your answer here". It reads it as the answer. The prompt was a worked example whose worked part was blank, so the model filled the slot by copying. This is the whole of finding 1.

What else was wrong

Four separate faults, all prompt-side:

1. Placeholder echo (7 cases) — above.
2. Wrong language (13/15 failed the language check). The prompt was entirely English and said "in the user's language (Russian or English)". The model picked English. It is never English: the nudge is spoken by a Russian piper voice.
3. Rule names are English identifiers. netdata_critical, service_down, break went into the prompt raw. The model cannot nudge about a topic it has not been told in words, so 9/15 were off topic. The daemon knows what its own rules mean; now it says so.
4. Mood invented ("warm", twice). The enum was listed in a parenthesis at the end of an English sentence. Now it is its own line: "ровно одно из: neutral, happy, thinking, tired, confused."

Plus two non-prompt faults the run exposed:

• The no-parse fallback was English. When the model returned nothing usable, the body became fmt.Sprintf("%s — %s", rule, sev) — "water — care" — and that string went to a Russian TTS. Now it falls back to plain Russian.
• Durations were English. humanDur returns "3 hours"; it was landing verbatim inside Russian sentences. Nudges now use a Russian formatter.

Three iterations, and what each taught

	score	change
baseline	0/15	—
iter 1	2/15	Russian prompt, filled-in examples, Russian durations
iter 2	11/15	required keyword per rule, one example instead of five, Russian fallback
iter 3	13/15	examples moved to topics that are not rules

The interesting step is 1 → 2. Fixing the placeholder did not fix the disease, it moved it: the model stopped copying "..." and started copying my first example instead. Five nudges in a row came back as "Ты не пил воду три часа. Налей стакан." regardless of the rule.

A small model copies the nearest concrete text in its prompt. That is one failure mode with two symptoms. The fix that stuck was making the examples about laundry and a laptop battery — topics no rule ever produces, so copying them is visible in the score rather than invisibly passing the water cases.

Do not oversell 13/15

Seven of the thirteen passes are the deterministic fallback, not the model: "Напоминаю: таблетки.", "Сервис не отвечает.", "Критический алярм: проверь диск.", "Ты давно не пил воду.". Those are strings this commit added to Go. The model returned nothing parseable and the fallback scored.

So the honest reading is roughly 6/15 from the model, 7/15 from a fallback, 2/15 failing. The prompt fix is real — "..." is nearly gone and the language and mood checks are clean — but a large part of the jump is that failure now degrades into Russian instead of into "water — care". That is a genuine improvement for the operator and a weak one for the model.

The two remaining failures: one "..." recurrence (routine-stretch) and one meal nudge that never says food.

Broken, found, not fixed

1. checkFeminine only catches half the constraint. Fixed (#381). It scanned for masculine self-reference only, so three messages that addressed the owner in the feminine ("ты давно не отдыхала") scored clean. There is now a second check, hisgender: a feminine past-tense verb (-ла/-лась) in a sentence addressed to him ("ты", "тебе", "твой") fails, unless the verb is hers ("я заметила", "напомнила тебе"). It is a suffix rule, not a parser — see the comment in checks.go for what it misses. A fresh 15-case run after adding it scored 12/15 with hisgender 15/15; the model did not repeat the feminine address in that sample, and the check is pinned by unit tests on the recorded bad strings instead.
2. Grammar is not checked at all, and it is bad. "Он не ел 11 дней" (it was 11 hours), "Сонуждились 7 дней" (not a word), "Они забыли воду" (wrong person entirely). Every one of these passes all six checks. The fixture measures properties, not fluency, and at 0.8B fluency is the binding constraint.
3. Unit confusion. The model turns hours into days about a third of the time. The prompt now says "11 ч"; it reads it as days.
4. 11s p50. Unchanged and untouched here. A nudge the model takes eleven seconds to word has missed its moment. Worth its own task.
5. The keyword hint is close to teaching to the test. ruleKeywords names the word the on-topic check looks for. It is defensible — the daemon genuinely knows its rule topics and the model genuinely cannot infer them from netdata_critical — but the on-topic number is softer than the others because of it.

Next steps

1. Add a second-person gender check — done, hisgender in checks.go (#381).
2. Decide whether the fallback should count as a pass. Right now Score cannot tell a model answer from a fallback. Either mark fallback bodies in PhrasedNudge or count them in their own column. Without that, any future prompt change can score well by failing more.
3. Attack the 11s. Nudge phrasing is short and non-interactive; thinking off is the first thing to try, as it was for routing (#376).
4. Re-measure when #122 lands. The CPT'd Qwen3-1.7B is the target. 13/15 with seven fallbacks is the floor it has to beat, and the fluency problems above are the ones a bigger, Russian-trained checkpoint should actually fix.