Compare commits

...

5 Commits

Author SHA1 Message Date
kami ccc5cba2a3 Merge the example-led prompt finding 2026-07-31 15:41:01 +04:00
kami 89d83c0b11 Record the example-led nudge prompt experiment (#393) — it made things worse
Tried rewriting the nudge prompt to lead with five on-topic examples instead
of rules. Three eval runs each side: before 12/13/14 of 15, after 11/12/11.
The loss is all in the address check — formal "вы" and plural imperatives came
back once the "говоришь на ты" rule stopped being its own sentence, and the
on-topic examples leaked their wording into the wrong cases.

Prompt reverted. Only the finding is committed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 15:40:18 +04:00
kami a97f554802 Merge the informal address prompt rule 2026-07-31 14:54:20 +04:00
kami f4de2fc5e1 Don't let a verb count as the person being talked about
The third-person check asks whether anyone else was named before "он".
A nudge is mostly verbs, and they were counted as possible people, so
"попробуй встать и отдохнуть — у него есть перерыв" passed. Infinitives
and imperatives now join past tense as words that cannot be a person.

A plain noun before the pronoun still blinds it. That needs a parser,
and the comment says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:54:20 +04:00
kami eef5d4da4f Tell the phraser to speak to him informally, singular
The prompts stated the feminine self-reference rule but never said whom she is
speaking to, so the model produced formal plural ("Жду вас") and talked about
him in third person ("Он не ел 11 дней"). Adds the address rule right next to
the feminine one, in the nudge prompt and the confirmation prompt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:52:23 +04:00
5 changed files with 71 additions and 10 deletions
+29
View File
@@ -103,6 +103,35 @@ but a large part of the jump is that failure now degrades into Russian instead o
The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge
that never says food.
## Tried and reverted: an example-led nudge prompt (#393)
The idea was that a 0.8B copies examples better than it follows rules, so the nudge prompt
was rewritten to lead with five on-topic examples (water, break, pills, morning, service) and
the prose rules were compressed to pay for the tokens: 1190 chars down to 986.
It measured **worse**, three runs each side, same llama-server, same fixture:
| run | before | after |
|---|---|---|
| 1 | 12/15 (address 14) | 11/15 (address 13) |
| 2 | 13/15 (address 15) | 12/15 (address 15) |
| 3 | 14/15 (address 15) | 11/15 (address 12) |
`feminine` and `hisgender` were 15/15 on all six runs, so they measure nothing here. The
regression is all in `address`: 44/45 before, 40/45 after. Formal "вы"/"ваше" and plural
imperatives came back, and so did `"..."`.
Two likely causes, both about the same thing — **examples do not carry a prohibition**. The
old prompt spent a whole sentence on «говоришь на "ты", в единственном числе»; the new one
demoted that to one item in a long "никогда" list, and the model stopped obeying it. And
making the examples on-topic let their *wording* leak: a break case came back as
«Вы давно не пили воду. Выпей стакан.» — the water example, verbatim, in the wrong slot.
That is exactly the failure the laundry/laptop examples were chosen to avoid.
Change reverted. What survives is the measurement: a rule the model must obey needs its own
sentence, and examples must stay off-topic. Also note the before side alone spans 1214 of
15 — this fixture cannot resolve anything smaller than about three cases.
## Broken, found, not fixed
1. ~~**`checkFeminine` only catches half the constraint.**~~ **Fixed** (#381). It scanned for
+1 -1
View File
@@ -29,7 +29,7 @@ func newLLMReplier(c completer) *llmReplier {
return &llmReplier{c: c, stub: voice.NewStubReplier()}
}
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Владелец — мужчина, говоришь с ним на "ты", в единственном числе; никогда не "вы"/"ваш" и не "он"/"его". Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"}
Никогда не пиши "..." в поле response.`
@@ -24,3 +24,19 @@ func TestAddressTimeWordDoesNotBlind(t *testing.T) {
}
}
}
// TestAddressVerbIsNotAnAntecedent — a nudge is mostly verbs, and a verb is
// never who "он" refers to. This exact string passed the check before.
func TestAddressVerbIsNotAnAntecedent(t *testing.T) {
s := "попробуй встать и отдохнуть — у него есть перерыв"
if r := checkAddress(s); r.Pass {
t.Errorf("checkAddress(%q) passed, want a third-person failure", s)
}
// Still missed, and this is the documented hole: "выпей воды, он не пил" has
// a real noun ("воды") before the pronoun, so the scan believes somebody
// else was named. Telling that apart needs a parser, not a suffix rule.
// A named third party still wins over the verbs around it.
if r := checkAddress("сервис упал, он не отвечает"); !r.Pass {
t.Errorf("checkAddress on a real third party failed: %s", r.Detail)
}
}
+22 -8
View File
@@ -340,7 +340,9 @@ func prevWord(words []string, i int) string {
// - it only looks BACKWARD. "Он не отвечает, сервис упал" names the subject
// after the pronoun and is flagged wrongly.
// - any noun earlier in the message counts as an antecedent, even when it is
// not one ("после обеда он не ел" reads as legitimate and is missed). The
// not one ("после обеда он не ел", "выпей воды, он не пил" — both missed).
// Verbs and time words no longer count, which covers the usual nudge, but a
// plain noun before the pronoun still blinds it. The
// common time words are stoplisted so the usual nudge opening does not
// blind it, but a message with any other noun in front still slips through.
// This is the check's real hole; widening it further would start flagging
@@ -407,14 +409,26 @@ var notAnAntecedent = map[string]bool{
"твой": true, "твоя": true, "твоё": true, "твое": true, "твои": true, "твою": true,
}
// looksPastVerb — a past-tense verb needs a subject of its own, so it is not an
// antecedent either. Keeps "сервис упал, он не отвечает" working off "сервис".
func looksPastVerb(w string) bool {
if len([]rune(w)) < 3 {
// looksVerb — a verb is never the thing "он" refers to, so it must not count as
// an antecedent. Past tense keeps "сервис упал, он не отвечает" working off
// "сервис"; the infinitive and imperative endings are here because a nudge is
// mostly made of them ("попробуй встать и отдохнуть — у него есть перерыв"
// slipped through with "попробуй" taken for the person being talked about).
func looksVerb(w string) bool {
r := []rune(w)
if len(r) < 3 {
return false
}
return strings.HasSuffix(w, "л") || strings.HasSuffix(w, "ла") ||
strings.HasSuffix(w, о") || strings.HasSuffix(w, "ли")
for _, suf := range []string{
"л", а", "ло", "ли", // past tense
"ть", "ться", "ти", "чь", // infinitive
"й", "йся", "йте", // imperative
} {
if strings.HasSuffix(w, suf) {
return true
}
}
return false
}
func checkAddress(body string) Result {
@@ -441,7 +455,7 @@ func checkAddress(body string) Result {
if !unicode.Is(unicode.Cyrillic, []rune(p)[0]) && !isLatinWord(p) {
continue // punctuation
}
if notAnAntecedent[p] || prepositions[p] || thirdPersonHim[p] || looksPastVerb(p) {
if notAnAntecedent[p] || prepositions[p] || thirdPersonHim[p] || looksVerb(p) {
continue
}
named = true
+3 -1
View File
@@ -439,8 +439,10 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma
// as "..." before this. See PHRASING-EVAL-31-07-2026.md.
//
// Russian only, feminine self-reference, second person masculine (the owner is
// a man). One short sentence — the nudge is spoken aloud.
// a man). She talks TO him, informally, singular — never "вы", never "он".
// One short sentence — the nudge is spoken aloud.
const nudgeSystem = `Ты — Maven, домашняя ассистентка. О себе говоришь в женском роде ("я проверила", "я записала"). Владелец — мужчина, обращайся к нему в мужском роде ("ты пил", "ты забыл").
Говоришь с ним на "ты", в единственном числе ("выпей", "встань"). Никогда не "вы"/"вас"/"ваш" и никогда "он"/"его" — ты говоришь ему, а не о нём.
Пиши ОДНО короткое напоминание по-русски: не больше 120 символов и не больше 16 слов. Только по делу.