kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Fix the phrasing prompt: she was reading the placeholder aloud (0/15 to 13/15) #21

Closed
claude wants to merge 3 commits from overnight/phrasing into overnight/bakeoff
pull from: overnight/phrasing
merge into: kami:overnight/bakeoff
Conversation 4 Commits 3 Files Changed 3
+277 -14
claude commented 2026-07-31 10:25:14 +02:00
Contributor
Copy Link
Copy Source

Two commits: the prompt fix, then the writeup.

0/15 → 13/15. But read the caveat before you believe that number.

The baseline was 0 out of 15, and 7 of the 15 messages were the literal string "...". One was "full voice message". Both were copied straight out of the prompt, which showed {"response": "...", "mood": "..."} as the shape to fill in. A 0.8B does not read "..." as "your answer goes here" — it reads it as the answer.

Four things were wrong with the prompt:

  • The placeholder echo above.
  • The whole prompt was in English and asked for a reply "in the user's language (Russian or English)". The model picked English — and piper then read it out in a Russian voice.
  • Rule names went in raw, as English identifiers like netdata_critical and service_down. The model had no way to name the topic it was talking about.
  • The mood enum was buried in a parenthesis, so it invented "warm" twice.

Two adjacent bugs the run exposed and this fixes: the no-parse fallback was sending "water — care" to Russian TTS, and humanDur returns English ("3 hours") which was landing inside Russian sentences.

before after
passing 0/15 13/15
right language 2 14
mood in the enum 6 15
on topic 6 13

Do not trust 13/15 as a model score. Seven of the thirteen passes are the deterministic Russian fallback that this PR adds, not the model. The honest reading is roughly 6/15 from the model, 7/15 from a fallback, 2/15 still failing. The prompt fix is real, but a good part of the jump is that failure now degrades into plain Russian instead of into "water — care".

The interesting bit was iteration 1. Fixing the placeholder took it 0→2/15 and moved the disease instead of curing it: the model stopped copying "..." and started copying my first worked example, emitting the water nudge for five unrelated rules. One failure mode, two symptoms — a small model copies the nearest concrete text. What actually worked was making the examples about laundry and a laptop battery, topics no rule produces, so any copying shows up in the score instead of hiding in it.

Found and not fixed — the first one matters most:

  1. checkFeminine tests the wrong half of the persona. It checks her self-reference and scores 15/15. But three messages addressed you in the feminine — "ты давно не отдыхала". There is no second-person gender check, so the exact failure the persona constraint exists to prevent scores clean. Vikunja #381.
  2. Fluency is unchecked and visibly bad. "Он не ел 11 дней" (it was 11 hours) and "Сонуждились 7 дней" (not a Russian word) pass all six checks.
  3. Hours get reported as days about a third of the time.
  4. 11s p50. That is far too slow to speak. Thinking-off is the obvious first thing to try, as it was for routing.
  5. The ruleKeywords hint I added is close to teaching to the test — the on-topic column is softer than the others because of it, and the doc says so.
Two commits: the prompt fix, then the writeup. **0/15 → 13/15.** But read the caveat before you believe that number. The baseline was 0 out of 15, and **7 of the 15 messages were the literal string `"..."`**. One was `"full voice message"`. Both were copied straight out of the prompt, which showed `{"response": "...", "mood": "..."}` as the shape to fill in. A 0.8B does not read `"..."` as "your answer goes here" — it reads it as the answer. Four things were wrong with the prompt: - **The placeholder echo** above. - **The whole prompt was in English** and asked for a reply "in the user's language (Russian or English)". The model picked English — and piper then read it out in a Russian voice. - **Rule names went in raw**, as English identifiers like `netdata_critical` and `service_down`. The model had no way to name the topic it was talking about. - **The mood enum was buried in a parenthesis**, so it invented `"warm"` twice. Two adjacent bugs the run exposed and this fixes: the no-parse fallback was sending `"water — care"` to Russian TTS, and `humanDur` returns English ("3 hours") which was landing inside Russian sentences. | | before | after | |---|---|---| | passing | 0/15 | **13/15** | | right language | 2 | **14** | | mood in the enum | 6 | **15** | | on topic | 6 | **13** | **Do not trust 13/15 as a model score.** Seven of the thirteen passes are the deterministic Russian fallback that this PR adds, not the model. The honest reading is roughly **6/15 from the model, 7/15 from a fallback, 2/15 still failing**. The prompt fix is real, but a good part of the jump is that failure now degrades into plain Russian instead of into `"water — care"`. **The interesting bit was iteration 1.** Fixing the placeholder took it 0→2/15 and *moved* the disease instead of curing it: the model stopped copying `"..."` and started copying my first worked example, emitting the water nudge for five unrelated rules. One failure mode, two symptoms — a small model copies the nearest concrete text. What actually worked was making the examples about laundry and a laptop battery, topics no rule produces, so any copying shows up in the score instead of hiding in it. **Found and not fixed — the first one matters most:** 1. **`checkFeminine` tests the wrong half of the persona.** It checks *her* self-reference and scores 15/15. But three messages addressed **you** in the feminine — "ты давно не отдыхал**а**". There is no second-person gender check, so the exact failure the persona constraint exists to prevent scores clean. Vikunja #381. 2. **Fluency is unchecked and visibly bad.** "Он не ел 11 дней" (it was 11 *hours*) and "Сонуждились 7 дней" (not a Russian word) pass all six checks. 3. Hours get reported as days about a third of the time. 4. **11s p50.** That is far too slow to speak. Thinking-off is the obvious first thing to try, as it was for routing. 5. The `ruleKeywords` hint I added is close to teaching to the test — the on-topic column is softer than the others because of it, and the doc says so.
claude changed target branch from master to overnight/bakeoff 2026-07-31 10:39:24 +02:00
claude added 3 commits 2026-07-31 10:39:24 +02:00
Fix the nudge phrasing prompt: stop teaching the model to echo the example a40bc559d5 
The system prompt showed the JSON contract as {"response": "..."} and the
user prompt repeated it. A 0.8B copies whatever sits in the response slot, so
7 of 15 nudges came back as literally "...".

Changes, all prompt-side — the {"response","mood"} contract is unchanged:
- nudge system prompt is Russian, feminine self-reference, with filled-in
  examples on topics that never appear as rules, so copying them is visible
- rule names get a Russian gloss and a required keyword, named last in the
  prompt where a small model weights it hardest
- durations render in Russian, not English
- the no-parse fallback says something Russian instead of "water — care",
  which was going straight to a Russian piper voice
- same "..." placeholder removed from replier_llm.go

Scored on internal/phraser/eval: 0/15 -> 13/15.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
Write up the phrasing eval: 0/15 to 13/15, and what the number hides 74a70880a8 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
Merge commit '74a7088' into overnight-jul31 796e6af3cf
kami reviewed 2026-07-31 18:52:58 +02:00
cmd/mavend/replier_llm.go
@@ -30,3 +30,3 @@
}
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}.`
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
kami commented 2026-07-31 18:52:58 +02:00
Owner
Copy Link
Copy Source

"тепло" might be confusing here.

"тепло" might be confusing here.
kami reviewed 2026-07-31 18:53:29 +02:00
internal/phraser/llmphraser.go
@@ -431,0 +444,4 @@
Пиши ОДНО короткое напоминание по-русски: не больше 120 символов и не больше 16 слов. Только по делу.
Запрещено: обращения ("дорогой", "милый"), эмодзи, извинения ("прости", "извини"), вопросы о самочувствии, похвала, больше одного восклицательного знака, английские слова кроме имён сервисов.
kami commented 2026-07-31 18:53:29 +02:00
Owner
Copy Link
Copy Source

there was something like this in tests?

there was something like this in tests?
kami commented 2026-07-31 18:54:11 +02:00
Owner
Copy Link
Copy Source

"Ками" might be used, btw.

"Ками" might be used, btw.
kami reviewed 2026-07-31 18:53:56 +02:00
internal/phraser/llmphraser.go
@@ -431,0 +452,4 @@
Так выглядит правильный ответ по форме. Темы здесь посторонние — их в запросе не будет:
{"response": "Стиральная машина закончила. Развесь бельё.", "mood": "neutral"}
{"response": "Ноутбук на трёх процентах. Я поставила его на зарядку.", "mood": "confused"}
kami commented 2026-07-31 18:53:56 +02:00
Owner
Copy Link
Copy Source

did she grow hands out of nowhere?

did she grow hands out of nowhere?
kami commented 2026-07-31 20:22:04 +02:00
Owner
Copy Link
Copy Source

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.
kami closed this pull request 2026-07-31 20:22:05 +02:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
kami
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kami
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kami/Maven#21
Delete Branch "overnight/phrasing"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?