diff --git a/PHRASING-EVAL-31-07-2026.md b/PHRASING-EVAL-31-07-2026.md index b998b48..0573665 100644 --- a/PHRASING-EVAL-31-07-2026.md +++ b/PHRASING-EVAL-31-07-2026.md @@ -103,6 +103,35 @@ but a large part of the jump is that failure now degrades into Russian instead o The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge that never says food. +## Tried and reverted: an example-led nudge prompt (#393) + +The idea was that a 0.8B copies examples better than it follows rules, so the nudge prompt +was rewritten to lead with five on-topic examples (water, break, pills, morning, service) and +the prose rules were compressed to pay for the tokens: 1190 chars down to 986. + +It measured **worse**, three runs each side, same llama-server, same fixture: + +| run | before | after | +|---|---|---| +| 1 | 12/15 (address 14) | 11/15 (address 13) | +| 2 | 13/15 (address 15) | 12/15 (address 15) | +| 3 | 14/15 (address 15) | 11/15 (address 12) | + +`feminine` and `hisgender` were 15/15 on all six runs, so they measure nothing here. The +regression is all in `address`: 44/45 before, 40/45 after. Formal "вы"/"ваше" and plural +imperatives came back, and so did `"..."`. + +Two likely causes, both about the same thing — **examples do not carry a prohibition**. The +old prompt spent a whole sentence on «говоришь на "ты", в единственном числе»; the new one +demoted that to one item in a long "никогда" list, and the model stopped obeying it. And +making the examples on-topic let their *wording* leak: a break case came back as +«Вы давно не пили воду. Выпей стакан.» — the water example, verbatim, in the wrong slot. +That is exactly the failure the laundry/laptop examples were chosen to avoid. + +Change reverted. What survives is the measurement: a rule the model must obey needs its own +sentence, and examples must stay off-topic. Also note the before side alone spans 12–14 of +15 — this fixture cannot resolve anything smaller than about three cases. + ## Broken, found, not fixed 1. ~~**`checkFeminine` only catches half the constraint.**~~ **Fixed** (#381). It scanned for