simulator: make the negative assertions mean something, and give the ecosystem a scenario
expect_not_called could pass on a step that made the forbidden call. callPaths concatenates per server and callCount was a total, so slicing the concatenated list by the total examined the wrong window. With praxis on three requests and nexus on one, a fourth praxis call landed at index three and paths[4:] never saw it, while the stale nexus call was reported as new. The mark is now per server and the paths are taken per server from it. Neither scenario ever produced an act, so all three fakes saw zero requests and the fault lever changed no outcome. The two headline capabilities of the harness had no coverage. act_degraded scripts an act against an enabled allowlist row and runs it healthy, at 503 and healthy again, asserting the reply, the call, the absence of a call on a tick, and that nothing was pushed at him either way. That needed two seams the world did not have: allowlist rows from the scenario, and a matcher on the real store rather than a nil API, which would have panicked the moment any scenario produced an act. expect_no_events compared bus.Len(), which stops growing at the ring capacity, so a scenario long enough to fill the ring made every later expect_no_events pass unconditionally. It counts publishes through a subscriber now. A scenario could not express a fact below full confidence, because write hardcoded 1.0, and morning_missed annotated its ambient step as if it could. factPriority branches on exactly that, so no replay could reach the low branch. signalStep takes a confidence, the ambient step sets the 0.6 the ambient path writes, and the event line carries the priority so a scenario can assert it. TestSimulatorIsDeterministic compared the transcript against time.Now, which fails for the half hour a day the scenario itself covers. It checks that every stamped line falls inside the scenario span instead. TestSimulatorRefusesBackwardsSteps tested the forwards case, because reaching the backwards branch ended the test. A fatalf seam makes the refusal observable. Smaller notes: the step doc comment now states which assertions are run scoped and which are step scoped, audioText parses the golden manifest once per world rather than once per step, and the feminine checks list the masculine form with its following character, since the earlier check on a comma alone passed on "записал что ты выпил воды". Found in review of #79.
This commit is contained in:
kami
+58
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"name": "act_degraded",
|
||||
"description": "The act path against a Praxis that goes down and comes back. This is the case the harness promised and did not have: the other two scenarios never produce an act, so the ecosystem fakes saw zero requests and the fault lever was inert. Here a scripted act reaches an enabled allowlist row, the row is a Praxis verb, and the same utterance runs healthy, then at 503, then healthy again. The degraded turn must say she cannot reach it and must not send anything at him off the back of it.",
|
||||
"start": "2026-08-01T09:00:00+03:00",
|
||||
"praxis_attention": "[{\"id\":\"item_1\",\"title\":\"medicine not taken\",\"importance\":3.0,\"rule\":\"morning_medicine\"}]",
|
||||
"tools": [{ "name": "list_attention" }],
|
||||
"script": [
|
||||
{
|
||||
"match": "требует внимания",
|
||||
"route": "[{\"intent\":\"act\",\"verb\":\"list_attention\"}]"
|
||||
},
|
||||
{
|
||||
"match": "",
|
||||
"route": "[{\"intent\":\"chat\",\"text\":\"привет\"}]",
|
||||
"reply": "{\"response\":\"Я рада тебя слышать.\",\"mood\":\"happy\"}"
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"at": "09:00",
|
||||
"note": "a healthy act reaches Praxis and speaks what it found",
|
||||
"say": "что требует внимания?",
|
||||
"expect_reply_contains": ["medicine not taken"],
|
||||
"expect_called": ["/api/v1/tools/attention"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
"at": "09:05",
|
||||
"note": "the ecosystem goes down",
|
||||
"fault": 503
|
||||
},
|
||||
{
|
||||
"at": "09:10",
|
||||
"note": "the same act against a 503. She says she cannot reach it. She does not invent an answer and she does not push anything at him.",
|
||||
"say": "что требует внимания?",
|
||||
"expect_reply_contains": ["не могу сейчас узнать"],
|
||||
"expect_reply_lacks": ["medicine not taken"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
"at": "09:15",
|
||||
"note": "a tick while the ecosystem is down touches nothing out there — the proactive loop has no business calling Praxis",
|
||||
"tick": true,
|
||||
"expect_not_called": ["/api/v1"],
|
||||
"expect_no_send": true,
|
||||
"expect_no_events": true
|
||||
},
|
||||
{
|
||||
"at": "09:20",
|
||||
"note": "recovery: the same act works again, so the degraded turn left no sticky state",
|
||||
"clear_fault": true,
|
||||
"say": "что требует внимания?",
|
||||
"expect_reply_contains": ["medicine not taken"],
|
||||
"expect_no_send": true
|
||||
}
|
||||
]
|
||||
}
|
||||
+1
-1
@@ -25,7 +25,7 @@
|
||||
"note": "he speaks. The whole voice path runs: push-to-talk, the STT seam parked with the golden transcript, the real router, the real store write, the phrasing contract.",
|
||||
"audio": "ru_fact",
|
||||
"expect_reply_contains": ["записала"],
|
||||
"expect_reply_lacks": ["записал,", "милый", "ваш"],
|
||||
"expect_reply_lacks": ["записал ", "записал,", "записал.", "милый", "ваш"],
|
||||
"expect_events": ["water"]
|
||||
},
|
||||
{
|
||||
|
||||
+7
-6
@@ -54,15 +54,16 @@
|
||||
},
|
||||
{
|
||||
"at": "08:40",
|
||||
"note": "the work calendar signal — a relayed notification, below full confidence",
|
||||
"note": "the work calendar signal — a relayed notification, at the ambient path's own 0.6 rather than an observation she made herself. That is the branch factPriority takes, so the journal must file it low.",
|
||||
"arrive": {
|
||||
"source": "ambient:notif",
|
||||
"fact": {
|
||||
"key": "calendar_event_20260801_планёрка",
|
||||
"value": "10:00-11:00 планёрка"
|
||||
"value": "10:00-11:00 планёрка",
|
||||
"confidence": 0.6
|
||||
}
|
||||
},
|
||||
"expect_events": ["ambient:notif", "планёрка"],
|
||||
"expect_events": ["планёрка", "ambient:notif/fact pri=low"],
|
||||
"expect_no_send": true
|
||||
},
|
||||
{
|
||||
@@ -73,17 +74,17 @@
|
||||
},
|
||||
{
|
||||
"at": "08:50",
|
||||
"note": "he asks. The query path answers from local recall only: nothing stored clears the score gate, so she refuses rather than inventing a morning summary, and the replier is never reached. That refusal is the no-hallucination floor and this step pins it.",
|
||||
"note": "he asks. The query path answers from local recall only: nothing stored clears the score gate, so she refuses rather than inventing a morning summary, and the replier is never reached. That refusal is the no-hallucination floor and this step pins it. Note what the persona check here is and is not: the reply is a constant in the Go source, so expect_reply_lacks pins that constant, not anything the model wrote. The step below is the one that reads model output.",
|
||||
"say": "что я пропустил?",
|
||||
"expect_reply_contains": ["не знаю"],
|
||||
"expect_reply_lacks": ["рад ", "милый", "ваш"]
|
||||
},
|
||||
{
|
||||
"at": "08:55",
|
||||
"note": "stating a fact writes it and says so, in the feminine",
|
||||
"note": "stating a fact writes it and says so, in the feminine. This reply comes back through the replier from the scripted model, so the persona check is against generated text rather than a constant. The masculine forms are listed with their following character — \"записал \" and \"записал,\" — because \"записала\" contains \"записал\", and the earlier check on the comma alone passed on \"записал что ты выпил воды\".",
|
||||
"say": "я выпил воды",
|
||||
"expect_reply_contains": ["записала"],
|
||||
"expect_reply_lacks": ["записал,", "милый"],
|
||||
"expect_reply_lacks": ["записал ", "записал,", "записал.", "милый"],
|
||||
"expect_events": ["water"]
|
||||
},
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user