From 89d83c0b11cda0790056880ac52b9bbad7328480 Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 31 Jul 2026 15:40:18 +0400 Subject: [PATCH] =?UTF-8?q?Record=20the=20example-led=20nudge=20prompt=20e?= =?UTF-8?q?xperiment=20(#393)=20=E2=80=94=20it=20made=20things=20worse?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tried rewriting the nudge prompt to lead with five on-topic examples instead of rules. Three eval runs each side: before 12/13/14 of 15, after 11/12/11. The loss is all in the address check — formal "вы" and plural imperatives came back once the "говоришь на ты" rule stopped being its own sentence, and the on-topic examples leaked their wording into the wrong cases. Prompt reverted. Only the finding is committed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ --- PHRASING-EVAL-31-07-2026.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/PHRASING-EVAL-31-07-2026.md b/PHRASING-EVAL-31-07-2026.md index b998b48..0573665 100644 --- a/PHRASING-EVAL-31-07-2026.md +++ b/PHRASING-EVAL-31-07-2026.md @@ -103,6 +103,35 @@ but a large part of the jump is that failure now degrades into Russian instead o The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge that never says food. +## Tried and reverted: an example-led nudge prompt (#393) + +The idea was that a 0.8B copies examples better than it follows rules, so the nudge prompt +was rewritten to lead with five on-topic examples (water, break, pills, morning, service) and +the prose rules were compressed to pay for the tokens: 1190 chars down to 986. + +It measured **worse**, three runs each side, same llama-server, same fixture: + +| run | before | after | +|---|---|---| +| 1 | 12/15 (address 14) | 11/15 (address 13) | +| 2 | 13/15 (address 15) | 12/15 (address 15) | +| 3 | 14/15 (address 15) | 11/15 (address 12) | + +`feminine` and `hisgender` were 15/15 on all six runs, so they measure nothing here. The +regression is all in `address`: 44/45 before, 40/45 after. Formal "вы"/"ваше" and plural +imperatives came back, and so did `"..."`. + +Two likely causes, both about the same thing — **examples do not carry a prohibition**. The +old prompt spent a whole sentence on «говоришь на "ты", в единственном числе»; the new one +demoted that to one item in a long "никогда" list, and the model stopped obeying it. And +making the examples on-topic let their *wording* leak: a break case came back as +«Вы давно не пили воду. Выпей стакан.» — the water example, verbatim, in the wrong slot. +That is exactly the failure the laundry/laptop examples were chosen to avoid. + +Change reverted. What survives is the measurement: a rule the model must obey needs its own +sentence, and examples must stay off-topic. Also note the before side alone spans 12–14 of +15 — this fixture cannot resolve anything smaller than about three cases. + ## Broken, found, not fixed 1. ~~**`checkFeminine` only catches half the constraint.**~~ **Fixed** (#381). It scanned for