diff --git a/PHRASING-EVAL-31-07-2026.md b/PHRASING-EVAL-31-07-2026.md new file mode 100644 index 0000000..cc971e5 --- /dev/null +++ b/PHRASING-EVAL-31-07-2026.md @@ -0,0 +1,138 @@ +# Phrasing evaluation — 31-07-2026 + +How Maven words a nudge, measured instead of argued. Counterpart to +`ROUTING-EVAL-31-07-2026.md`. + +- Fixture + scorer: `internal/phraser/eval/` (`nudges_v1.json`, 15 cases; `eval.go`, `checks.go`) +- Reproduce: `MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing` +- Model: Qwen3.5-0.8B Q4_K_M, the resident model. Not swapped. +- Commit: `a40bc55` (prompt fix) + +Every check is a string or length test a human can read and disagree with. No model +grades another model here. + +## Result + +| | before | after | +|---|---|---| +| **cases passing every check** | **0/15** | **13/15** | +| mood in enum | 6/15 | 15/15 | +| Russian | 2/15 | 14/15 | +| length (≤120 chars, ≤16 words) | 13/15 | 15/15 | +| feminine self-reference | 15/15 | 15/15 | +| no cringe | 13/15 | 15/15 | +| on topic | 6/15 | 13/15 | +| p50 latency | 11.4s | 11.4s | + +Latency did not move and is not good. 11s to word one nudge on this box. + +## The bug reproduced + +Yes, exactly as reported. 7 of 15 messages were the literal string `"..."`, and one was +`"full voice message"`. Both are text copied straight out of the prompt. + +The system prompt said: + +``` +Respond ONLY with valid JSON: {"response": "full voice message", "mood": "neutral"} +``` + +and the user prompt said: + +``` +Respond as JSON: {"response": "...", "mood": "..."} +``` + +A 0.8B does not read `"..."` as "put your answer here". It reads it as the answer. The +prompt was a worked example whose worked part was blank, so the model filled the slot by +copying. This is the whole of finding 1. + +## What else was wrong + +Four separate faults, all prompt-side: + +1. **Placeholder echo** (7 cases) — above. +2. **Wrong language** (13/15 failed the language check). The prompt was entirely English + and said "in the user's language (Russian or English)". The model picked English. It is + never English: the nudge is spoken by a Russian piper voice. +3. **Rule names are English identifiers.** `netdata_critical`, `service_down`, `break` went + into the prompt raw. The model cannot nudge about a topic it has not been told in words, + so 9/15 were off topic. The daemon knows what its own rules mean; now it says so. +4. **Mood invented** (`"warm"`, twice). The enum was listed in a parenthesis at the end of + an English sentence. Now it is its own line: "ровно одно из: neutral, happy, thinking, + tired, confused." + +Plus two non-prompt faults the run exposed: + +- **The no-parse fallback was English.** When the model returned nothing usable, the body + became `fmt.Sprintf("%s — %s", rule, sev)` — `"water — care"` — and that string went to + a Russian TTS. Now it falls back to plain Russian. +- **Durations were English.** `humanDur` returns "3 hours"; it was landing verbatim inside + Russian sentences. Nudges now use a Russian formatter. + +## Three iterations, and what each taught + +| | score | change | +|---|---|---| +| baseline | 0/15 | — | +| iter 1 | 2/15 | Russian prompt, filled-in examples, Russian durations | +| iter 2 | 11/15 | required keyword per rule, one example instead of five, Russian fallback | +| iter 3 | **13/15** | examples moved to topics that are not rules | + +The interesting step is 1 → 2. Fixing the placeholder did not fix the disease, it moved it: +the model stopped copying `"..."` and started copying my first example instead. Five nudges +in a row came back as `"Ты не пил воду три часа. Налей стакан."` regardless of the rule. + +**A small model copies the nearest concrete text in its prompt.** That is one failure mode +with two symptoms. The fix that stuck was making the examples about laundry and a laptop +battery — topics no rule ever produces, so copying them is visible in the score rather than +invisibly passing the water cases. + +## Do not oversell 13/15 + +Seven of the thirteen passes are the **deterministic fallback**, not the model: +`"Напоминаю: таблетки."`, `"Сервис не отвечает."`, `"Критический алярм: проверь диск."`, +`"Ты давно не пил воду."`. Those are strings this commit added to Go. The model returned +nothing parseable and the fallback scored. + +So the honest reading is roughly **6/15 from the model, 7/15 from a fallback, 2/15 failing**. +The prompt fix is real — `"..."` is nearly gone and the language and mood checks are clean — +but a large part of the jump is that failure now degrades into Russian instead of into +`"water — care"`. That is a genuine improvement for the operator and a weak one for the model. + +The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge +that never says food. + +## Broken, found, not fixed + +1. **`checkFeminine` only catches half the constraint.** It scans for masculine + self-reference and passed 15/15 both runs — but three messages address the *owner* in + the feminine: "ты давно не отдыхал**а**", "он не ел". The owner is a man. The check has + no second-person gender test, so this scores clean while being exactly the persona + failure the constraint exists to prevent. This is the most important gap in the harness. +2. **Grammar is not checked at all, and it is bad.** `"Он не ел 11 дней"` (it was 11 hours), + `"Сонуждились 7 дней"` (not a word), `"Они забыли воду"` (wrong person entirely). Every + one of these passes all six checks. The fixture measures properties, not fluency, and at + 0.8B fluency is the binding constraint. +3. **Unit confusion.** The model turns hours into days about a third of the time. The + prompt now says "11 ч"; it reads it as days. +4. **11s p50.** Unchanged and untouched here. A nudge the model takes eleven seconds to + word has missed its moment. Worth its own task. +5. **The keyword hint is close to teaching to the test.** `ruleKeywords` names the word the + on-topic check looks for. It is defensible — the daemon genuinely knows its rule topics + and the model genuinely cannot infer them from `netdata_critical` — but the on-topic + number is softer than the others because of it. + +## Next steps + +1. **Add a second-person gender check** to `checks.go`. Finding 1 above. Until it exists the + feminine column means less than it looks like. +2. **Decide whether the fallback should count as a pass.** Right now `Score` cannot tell a + model answer from a fallback. Either mark fallback bodies in `PhrasedNudge` or count them + in their own column. Without that, any future prompt change can score well by failing + more. +3. **Attack the 11s.** Nudge phrasing is short and non-interactive; thinking off is the first + thing to try, as it was for routing (#376). +4. **Re-measure when #122 lands.** The CPT'd Qwen3-1.7B is the target. 13/15 with seven + fallbacks is the floor it has to beat, and the fluency problems above are the ones a + bigger, Russian-trained checkpoint should actually fix. diff --git a/cmd/mavend/replier_llm.go b/cmd/mavend/replier_llm.go index bbba111..e215a1d 100644 --- a/cmd/mavend/replier_llm.go +++ b/cmd/mavend/replier_llm.go @@ -29,7 +29,9 @@ func newLLMReplier(c completer) *llmReplier { return &llmReplier{c: c, stub: voice.NewStubReplier()} } -const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}.` +const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused). +Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"} +Никогда не пиши "..." в поле response.` func (r *llmReplier) Reply(d router.Decision) string { if d.Clarify { diff --git a/internal/phraser/llmphraser.go b/internal/phraser/llmphraser.go index c61a3e1..05624a8 100644 --- a/internal/phraser/llmphraser.go +++ b/internal/phraser/llmphraser.go @@ -184,7 +184,10 @@ func (p *LLMPhraser) PhraseNudge(ctx context.Context, c loop.Candidate) (deliver body, _ = parsePhrase(resp) } if body == "" { - body = fmt.Sprintf("%s — %s", c.Rule.Name, sevLabel(c.Severity)) + // The model said nothing usable. Say it in Russian anyway — this text + // goes straight to a Russian piper voice, so the old "water — care" + // fallback was unspeakable. + body = fallbackNudge(c) } if mood == "" { mood = "neutral" @@ -428,8 +431,33 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma return stripThink(content), nil } +// nudgeSystem — the phrasing contract for nudges. +// +// Written as filled-in examples, not as a schema with "..." in it. A 0.8B +// copies whatever sits in the response slot, so a literal placeholder there +// teaches it to answer with the placeholder. Measured: 7/15 nudges came back +// as "..." before this. See PHRASING-EVAL-31-07-2026.md. +// +// Russian only, feminine self-reference, second person masculine (the owner is +// a man). One short sentence — the nudge is spoken aloud. +const nudgeSystem = `Ты — Maven, домашняя ассистентка. О себе говоришь в женском роде ("я проверила", "я записала"). Владелец — мужчина, обращайся к нему в мужском роде ("ты пил", "ты забыл"). + +Пиши ОДНО короткое напоминание по-русски: не больше 120 символов и не больше 16 слов. Только по делу. + +Запрещено: обращения ("дорогой", "милый"), эмодзи, извинения ("прости", "извини"), вопросы о самочувствии, похвала, больше одного восклицательного знака, английские слова кроме имён сервисов. + +Отвечай ТОЛЬКО одним объектом JSON с полями "response" и "mood". +"response" — сам текст напоминания. +"mood" — ровно одно из: neutral, happy, thinking, tired, confused. + +Так выглядит правильный ответ по форме. Темы здесь посторонние — их в запросе не будет: +{"response": "Стиральная машина закончила. Развесь бельё.", "mood": "neutral"} +{"response": "Ноутбук на трёх процентах. Я поставила его на зарядку.", "mood": "confused"} + +Это примеры ФОРМЫ, а не темы. Пиши только про ту ситуацию, которую тебе дали в запросе. Не копируй примеры и никогда не пиши "..." в поле response.` + func (p *LLMPhraser) systemPrompt() string { - base := `You are maven, a self-hosted personal assistant. Generate brief, natural nudge messages in the user's language (Russian or English). Respond ONLY with valid JSON: {"response": "full voice message", "mood": "neutral"}. "response" is what the user hears; "mood" reflects maven's tone (neutral/happy/thinking/tired/confused).` + base := nudgeSystem if p.cfg.Persona != "" { base = p.cfg.Persona + "\n\n" + base } @@ -446,22 +474,117 @@ func (p *LLMPhraser) querySystemPrompt() string { return base } +// ruleTopics — Russian gloss for each built-in rule name. The rule names are +// English identifiers; a 0.8B asked to nudge about "netdata_critical" writes +// about nothing. The daemon knows what its own rules mean, so it says so. +var ruleTopics = map[string]string{ + "water": "он давно не пил воду", + "meal": "он давно не ел", + "break": "он давно без перерыва, пора встать и размяться", + "service_down": "сервис не отвечает, лежит", + "netdata_critical": "критический алярм в netdata, проблема с диском или местом", +} + +// ruleKeywords — the word the message must contain. The 0.8B drifts to +// whatever topic it saw last unless the required word is named outright. +var ruleKeywords = map[string]string{ + "water": "воду", + "meal": "поешь", + "break": "перерыв", + "service_down": "сервис", + "netdata_critical": "диск", +} + +// ruleTopic turns a rule name into a Russian description of the situation. +// "routine:зарядка" and "morning:утро" carry their own Russian suffix. +func ruleTopic(rule string) string { + if t, ok := ruleTopics[rule]; ok { + return t + } + if i := strings.IndexByte(rule, ':'); i > 0 && i+1 < len(rule) { + switch rule[:i] { + case "morning": + return "утро, пора начать день: " + rule[i+1:] + default: + return "пора сделать по распорядку: " + rule[i+1:] + } + } + return rule +} + +// ruleKeyword — the word the nudge must contain, or "" when the rule name's +// own Russian suffix already is that word. +func ruleKeyword(rule string) string { + if k, ok := ruleKeywords[rule]; ok { + return k + } + if i := strings.IndexByte(rule, ':'); i > 0 && i+1 < len(rule) { + return rule[i+1:] + } + return "" +} + +// ruDur — duration in Russian. humanDur is English and its output was landing +// verbatim in the message. +func ruDur(d time.Duration) string { + if d < 0 { + d = 0 + } + h, m := int(d.Hours()), int(d.Minutes())%60 + switch { + case h >= 2: + return fmt.Sprintf("%d ч", h) + case h == 1 && m >= 30: + return "полтора часа" + case h == 1: + return "час" + default: + return fmt.Sprintf("%d мин", m) + } +} + +// fallbackNudge — plain Russian for when the model returns nothing parseable. +var fallbackNudges = map[string]string{ + "water": "Ты давно не пил воду.", + "meal": "Ты давно не ел, поешь.", + "break": "Пора сделать перерыв.", + "service_down": "Сервис не отвечает.", + "netdata_critical": "Критический алярм: проверь диск.", +} + +func fallbackNudge(c loop.Candidate) string { + if s, ok := fallbackNudges[c.Rule.Name]; ok { + return s + } + if kw := ruleKeyword(c.Rule.Name); kw != "" { + return "Напоминаю: " + kw + "." + } + return "Напоминаю о деле." +} + func buildNudgePrompt(c loop.Candidate) string { var ctxParts []string - ctxParts = append(ctxParts, fmt.Sprintf("Rule: %s", c.Rule.Name)) - ctxParts = append(ctxParts, fmt.Sprintf("Severity: %s", sevLabel(c.Severity))) - + ctxParts = append(ctxParts, "Ситуация: "+ruleTopic(c.Rule.Name)) + if f, ok := c.State.Facts[c.Rule.Name]; ok && f.Key != "" && f.Key != c.Rule.Name { + ctxParts = append(ctxParts, "Что именно: "+f.Key) + } if d, ok := c.State.Since(c.Rule.Name); ok { - ctxParts = append(ctxParts, fmt.Sprintf("Duration since last event: %s", humanDur(d))) + ctxParts = append(ctxParts, "Прошло: "+ruDur(d)) + } + switch sevLabel(c.Severity) { + case "alarm": + ctxParts = append(ctxParts, "Срочно, скажи прямо.") + case "ops": + ctxParts = append(ctxParts, "Это про сервер, не про здоровье.") + } + tail := "Напиши напоминание про эту ситуацию. Одно предложение, по-русски, в JSON." + if kw := ruleKeyword(c.Rule.Name); kw != "" { + // Last line on purpose: a 0.8B weights the end of the prompt hardest, + // and without the required word it drifts back to the examples. + tail += " Ответ ДОЛЖЕН содержать слово «" + kw + "»." } - return fmt.Sprintf( - `Generate a nudge message. Context: -%s - -Respond as JSON: {"response": "...", "mood": "..."}`, - strings.Join(ctxParts, "\n"), - ) + return strings.Join(ctxParts, "\n") + "\n\n" + tail } type responseMood struct {