Record the example-led nudge prompt experiment (#393) — it made things worse

Tried rewriting the nudge prompt to lead with five on-topic examples instead
of rules. Three eval runs each side: before 12/13/14 of 15, after 11/12/11.
The loss is all in the address check — formal "вы" and plural imperatives came
back once the "говоришь на ты" rule stopped being its own sentence, and the
on-topic examples leaked their wording into the wrong cases.

Prompt reverted. Only the finding is committed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
This commit is contained in:
kami
2026-07-31 15:40:18 +04:00
parent a97f554802
commit 89d83c0b11
+29
View File
@@ -103,6 +103,35 @@ but a large part of the jump is that failure now degrades into Russian instead o
The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge
that never says food.
## Tried and reverted: an example-led nudge prompt (#393)
The idea was that a 0.8B copies examples better than it follows rules, so the nudge prompt
was rewritten to lead with five on-topic examples (water, break, pills, morning, service) and
the prose rules were compressed to pay for the tokens: 1190 chars down to 986.
It measured **worse**, three runs each side, same llama-server, same fixture:
| run | before | after |
|---|---|---|
| 1 | 12/15 (address 14) | 11/15 (address 13) |
| 2 | 13/15 (address 15) | 12/15 (address 15) |
| 3 | 14/15 (address 15) | 11/15 (address 12) |
`feminine` and `hisgender` were 15/15 on all six runs, so they measure nothing here. The
regression is all in `address`: 44/45 before, 40/45 after. Formal "вы"/"ваше" and plural
imperatives came back, and so did `"..."`.
Two likely causes, both about the same thing — **examples do not carry a prohibition**. The
old prompt spent a whole sentence on «говоришь на "ты", в единственном числе»; the new one
demoted that to one item in a long "никогда" list, and the model stopped obeying it. And
making the examples on-topic let their *wording* leak: a break case came back as
«Вы давно не пили воду. Выпей стакан.» — the water example, verbatim, in the wrong slot.
That is exactly the failure the laundry/laptop examples were chosen to avoid.
Change reverted. What survives is the measurement: a rule the model must obey needs its own
sentence, and examples must stay off-topic. Also note the before side alone spans 1214 of
15 — this fixture cannot resolve anything smaller than about three cases.
## Broken, found, not fixed
1. ~~**`checkFeminine` only catches half the constraint.**~~ **Fixed** (#381). It scanned for