a conversation about one object has a fixture (V-542)
Five Russian turns, one monitor, four questions that say "он" and never name it again. Item 3 of the task: the shape had nowhere to fail, because the routing fixture scores one utterance at a time and a conversation that breaks on turn 2 cannot lose a point there. Routes are scripted exactly as the box produced them on 05-08-2026. Turn 1 files a fact despite "давай поболтаем", the questions go to query, turn 4 goes to chat, and none of the five replies names the monitor. Four steps assert the reply LACKS "монитор" and are marked WRONG in their notes with what each must become. The absence assertion is forced, not chosen. The simulator wires phraser.NewStub(), and PhraseChat posts raw HTTP to /v1/chat/completions rather than through the llm client the harness scripts, so a chat reply cannot be scripted at all. The wrong replies come from fallbacks_ru_v1.json, which picks between four variants per turn, so asserting a string would pin the picker. Missing referent holds whichever variant she reaches for. Items 1 and 2 stay open: they are owner decisions about which store a referent comes from and whether "давай поболтаем" claims a turn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,77 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"name": "conversation_anaphora",
|
||||
"description": "Five consecutive Russian turns about one object, replayed from the run that found V-542 on the box on 05-08-2026. He names a monitor, then asks four questions that all say \"он\" and never name it again.\n\nThis scenario exists because the shape had nowhere to fail. The routing fixture scores one utterance at a time, so a conversation that breaks on its second turn cannot lose a point there, and V-44 step 2 could only be verified by hand. That is item 3 of V-542.\n\nFour of the five replies below are WRONG, and the assertions pin them anyway. Read them as the recorded defect rather than the contract: she has the last four turns in front of her and never once names the thing he is asking about. Every wrong assertion is marked in its step note with what it must become. When V-542 lands, those flip and the ones marked correct do not move.\n\nWhat the four assert is that the reply LACKS \"монитор\", and that shape is forced by the harness rather than chosen. The simulator wires phraser.NewStub(), and LLMPhraser.PhraseChat posts raw HTTP to /v1/chat/completions rather than going through the llm client the harness scripts, so no scenario can script a reply on the chat path. Both wrong replies here therefore come from internal/phraser/fallbacks_ru_v1.json, which PICKS between four variants — \"тут я пас.\" one run and \"не знаю, честно.\" the next — so asserting a fallback string would pin the picker. Absence of the referent is the defect itself and holds whichever variant she reaches for. Giving the harness an httptest phraser is the prerequisite for asserting anything about what a conversation actually says; that is item 4 of V-542.\n\nThe routes are scripted exactly as the box produced them, because the failure is not the model's. Turn 1 went to fact despite \"давай поболтаем\", every question after it went to query, and turn 4 went to chat. A scripted route is what lets this scenario pin the daemon's half without a llama-server in the loop.",
|
||||
"start": "2026-08-05T14:00:00+03:00",
|
||||
"script": [
|
||||
{
|
||||
"match": "купил новый монитор",
|
||||
"route": "[{\"intent\":\"fact\",\"key\":\"purchase\",\"value\":\"новый монитор\"}]",
|
||||
"reply": "{\"response\":\"записала: новый монитор.\",\"mood\":\"neutral\"}"
|
||||
},
|
||||
{
|
||||
"match": "он большой",
|
||||
"route": "[{\"intent\":\"query\",\"text\":\"а он большой?\"}]"
|
||||
},
|
||||
{
|
||||
"match": "сколько он примерно стоит",
|
||||
"route": "[{\"intent\":\"query\",\"text\":\"сколько он примерно стоит по-твоему?\"}]"
|
||||
},
|
||||
{
|
||||
"match": "переплатил",
|
||||
"route": "[{\"intent\":\"chat\",\"text\":\"мне кажется я переплатил\"}]"
|
||||
},
|
||||
{
|
||||
"match": "стоит его вернуть",
|
||||
"route": "[{\"intent\":\"query\",\"text\":\"стоит его вернуть?\"}]"
|
||||
},
|
||||
{
|
||||
"match": "",
|
||||
"route": "[{\"intent\":\"chat\",\"text\":\"\"}]",
|
||||
"reply": "{\"response\":\"я рада тебя слышать.\",\"mood\":\"happy\"}"
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"at": "14:00",
|
||||
"note": "CORRECT, and it is the first half of the defect. \"давай поболтаем\" is an explicit request to converse and the turn is filed as a fact anyway. Storing what he said is not wrong on its own — he did buy a monitor — but the object then lives in the fact store and never enters the transcript PhraseChat reads. That is V-542 decision 2: either the marker claims the turn at stage 0, or it means nothing and comes out of the fixture.",
|
||||
"say": "давай поболтаем: я вчера купил новый монитор",
|
||||
"expect_events": ["purchase"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
"at": "14:01",
|
||||
"note": "WRONG. \"он\" is the monitor from one turn ago, and she says she has no record of it. followUpMerge inherits prev.Slots.Key, and a query turn asking about a pronoun has no key to merge, so the question reaches the query sources naked and the notes source answers the only way it can. Must become: an answer about the monitor, or a route to chat where the transcript is.",
|
||||
"say": "а он большой?",
|
||||
"expect_reply_lacks": ["монитор"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
"at": "14:02",
|
||||
"note": "WRONG, and it rules out one explanation. This is not the previous turn failing to stick — it is the same wall a second time, two turns from where the monitor was named. Nothing accumulates across query turns.",
|
||||
"say": "сколько он примерно стоит по-твоему?",
|
||||
"expect_reply_lacks": ["монитор"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
"at": "14:03",
|
||||
"note": "WRONG, and it is the same wall from the other side. This turn routed chat, so it HAD the history that Session.History holds, and it still cannot say what the device is — because turn 1's object went to the fact store rather than the transcript. So a source reading the conversation is not sufficient on its own; decision 1 has to say which store the referent comes from.",
|
||||
"say": "мне кажется я переплатил",
|
||||
"expect_reply_lacks": ["монитор"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
"at": "14:04",
|
||||
"note": "WRONG. The fifth turn is the one that shows the cost. A returns question about a purchase two minutes old is answered with \"не нашла у тебя такой записи\", which is wrong in kind rather than merely unhelpful: the record exists, she wrote it herself at 14:00 under the key purchase.",
|
||||
"say": "стоит его вернуть?",
|
||||
"expect_reply_lacks": ["монитор"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
"at": "14:05",
|
||||
"note": "CORRECT, and it is the control. Nothing in five conversational turns was sent at him unprompted, and a tick with him mid-conversation stays silent. Whatever V-542 changes must not change this.",
|
||||
"tick": true,
|
||||
"expect_no_send": true
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user