kami
d0e98a9419
expect_not_called could pass on a step that made the forbidden call. callPaths concatenates per server and callCount was a total, so slicing the concatenated list by the total examined the wrong window. With praxis on three requests and nexus on one, a fourth praxis call landed at index three and paths[4:] never saw it, while the stale nexus call was reported as new. The mark is now per server and the paths are taken per server from it. Neither scenario ever produced an act, so all three fakes saw zero requests and the fault lever changed no outcome. The two headline capabilities of the harness had no coverage. act_degraded scripts an act against an enabled allowlist row and runs it healthy, at 503 and healthy again, asserting the reply, the call, the absence of a call on a tick, and that nothing was pushed at him either way. That needed two seams the world did not have: allowlist rows from the scenario, and a matcher on the real store rather than a nil API, which would have panicked the moment any scenario produced an act. expect_no_events compared bus.Len(), which stops growing at the ring capacity, so a scenario long enough to fill the ring made every later expect_no_events pass unconditionally. It counts publishes through a subscriber now. A scenario could not express a fact below full confidence, because write hardcoded 1.0, and morning_missed annotated its ambient step as if it could. factPriority branches on exactly that, so no replay could reach the low branch. signalStep takes a confidence, the ambient step sets the 0.6 the ambient path writes, and the event line carries the priority so a scenario can assert it. TestSimulatorIsDeterministic compared the transcript against time.Now, which fails for the half hour a day the scenario itself covers. It checks that every stamped line falls inside the scenario span instead. TestSimulatorRefusesBackwardsSteps tested the forwards case, because reaching the backwards branch ended the test. A fatalf seam makes the refusal observable. Smaller notes: the step doc comment now states which assertions are run scoped and which are step scoped, audioText parses the golden manifest once per world rather than once per step, and the feminine checks list the masculine form with its following character, since the earlier check on a comma alone passed on "записал что ты выпил воды". Found in review of #79.
98 lines
4.4 KiB
JSON
98 lines
4.4 KiB
JSON
{
|
|
"schema_version": 1,
|
|
"name": "morning_missed",
|
|
"description": "The scenario from Vikunja #284's description, replayed. He appears at 08:30, things arrive through the morning while he is at the desk, and at 08:50 he asks what he missed. The assertions are as much about what did NOT happen — nothing was sent at him unprompted — as about what she said.",
|
|
"start": "2026-08-01T08:30:00+03:00",
|
|
"praxis_attention": "[{\"id\":\"item_1\",\"title\":\"medicine not taken\",\"importance\":3.0,\"rule\":\"morning_medicine\"}]",
|
|
"script": [
|
|
{
|
|
"match": "выпил воды",
|
|
"route": "[{\"intent\":\"fact\",\"key\":\"water\",\"value\":\"выпил\"}]"
|
|
},
|
|
{
|
|
"match": "записала факт: water",
|
|
"reply": "{\"response\":\"Записала, что ты выпил воды.\",\"mood\":\"neutral\"}"
|
|
},
|
|
{
|
|
"match": "что я пропустил",
|
|
"route": "[{\"intent\":\"query\",\"text\":\"что я пропустил\"}]"
|
|
},
|
|
{
|
|
"match": "",
|
|
"route": "[{\"intent\":\"chat\",\"text\":\"привет\"}]",
|
|
"reply": "{\"response\":\"Я рада тебя слышать.\",\"mood\":\"happy\"}"
|
|
}
|
|
],
|
|
"steps": [
|
|
{
|
|
"at": "08:30",
|
|
"note": "he appears at the desk",
|
|
"signal": { "key": "desk_active", "value": "true", "source": "infer:hyprland" },
|
|
"expect_events": ["infer:hyprland"],
|
|
"expect_no_send": true
|
|
},
|
|
{
|
|
"at": "08:32",
|
|
"note": "a feed item arrives, published half an hour ago",
|
|
"arrive": {
|
|
"source": "rss:tech",
|
|
"as_of": "08:02",
|
|
"note": { "text": "Вышло ядро 6.19 [tech]\nкраткое содержание\nhttps://example.org/a" }
|
|
},
|
|
"expect_events": ["rss:tech"],
|
|
"expect_no_send": true
|
|
},
|
|
{
|
|
"at": "08:35",
|
|
"note": "the mail reader extracts a candidate — a candidate is never spoken",
|
|
"arrive": {
|
|
"source": "email:inbox",
|
|
"task": { "text": "продлить домен", "evidence": "Домен истекает через 7 дней" }
|
|
},
|
|
"expect_events": ["email:inbox", "продлить домен"],
|
|
"expect_no_send": true
|
|
},
|
|
{
|
|
"at": "08:40",
|
|
"note": "the work calendar signal — a relayed notification, at the ambient path's own 0.6 rather than an observation she made herself. That is the branch factPriority takes, so the journal must file it low.",
|
|
"arrive": {
|
|
"source": "ambient:notif",
|
|
"fact": {
|
|
"key": "calendar_event_20260801_планёрка",
|
|
"value": "10:00-11:00 планёрка",
|
|
"confidence": 0.6
|
|
}
|
|
},
|
|
"expect_events": ["планёрка", "ambient:notif/fact pri=low"],
|
|
"expect_no_send": true
|
|
},
|
|
{
|
|
"at": "08:45",
|
|
"note": "a tick with him present and nothing wrong must stay silent",
|
|
"tick": true,
|
|
"expect_no_send": true
|
|
},
|
|
{
|
|
"at": "08:50",
|
|
"note": "he asks. The query path answers from local recall only: nothing stored clears the score gate, so she refuses rather than inventing a morning summary, and the replier is never reached. That refusal is the no-hallucination floor and this step pins it. Note what the persona check here is and is not: the reply is a constant in the Go source, so expect_reply_lacks pins that constant, not anything the model wrote. The step below is the one that reads model output.",
|
|
"say": "что я пропустил?",
|
|
"expect_reply_contains": ["не знаю"],
|
|
"expect_reply_lacks": ["рад ", "милый", "ваш"]
|
|
},
|
|
{
|
|
"at": "08:55",
|
|
"note": "stating a fact writes it and says so, in the feminine. This reply comes back through the replier from the scripted model, so the persona check is against generated text rather than a constant. The masculine forms are listed with their following character — \"записал \" and \"записал,\" — because \"записала\" contains \"записал\", and the earlier check on the comma alone passed on \"записал что ты выпил воды\".",
|
|
"say": "я выпил воды",
|
|
"expect_reply_contains": ["записала"],
|
|
"expect_reply_lacks": ["записал ", "записал,", "записал.", "милый"],
|
|
"expect_events": ["water"]
|
|
},
|
|
{
|
|
"at": "09:00",
|
|
"note": "a second tick, still nothing unprompted",
|
|
"tick": true,
|
|
"expect_no_send": true
|
|
}
|
|
]
|
|
}
|