kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Add a replayable full-system simulator on a fake clock (#284) #79

Closed
claude wants to merge 1 commits from overnight/replay-simulator into overnight/event-envelope
pull from: overnight/replay-simulator
merge into: kami:overnight/event-envelope
Conversation 2 Commits 1 Files Changed 4
+970 -1
claude commented 2026-08-01 04:15:44 +02:00
Contributor
Copy Link
Copy Source

What changed

A JSON-driven scenario replayer, test-only, in cmd/mavend/simulator_test.go plus cmd/mavend/testdata/scenarios/.

A scenario declares a start instant, a script of canned model answers (routing and phrasing are separate contracts, so an entry carries route and/or reply), and a list of steps at "HH:MM". A step does exactly one thing — say, audio, signal, arrive, tick, fault/clear_fault — and then asserts: expect_reply_contains/_lacks, expect_sent_contains, expect_no_send, expect_called/expect_not_called, expect_events, expect_no_events. The negative assertions are step-scoped, because "nothing was sent because of THIS" is the useful question.

Real between the boundaries: the router cascade (stage0 → LLM router over a scripted completer → classifier), the store, the reactive handler, the tick loop, and the intake-decorated ipc.CoreAPI from #283. Faked only where a test cannot have the real thing: the model, mic, speaker, delivery sink, and the Praxis/Nexus/Hexis HTTP services (reusing the fakeServer harness from #276).

Two scenarios:

  • morning_missed — #284's own description replayed. Desk presence at 08:30, then an RSS note, a mail candidate and a relayed ambient notification arrive; two ticks; he asks what he missed and states a fact. Half the assertions are that nothing was sent at him unprompted.
  • evening_degraded — the tier-2 pipeline case #288 deferred to this task: a golden WAV from cmd/mavsttd/testdata/golden_v1.json through the STT seam, the router, the store write and the phrasing contract. Then the ecosystem answers 503 and the proactive loop has to stay quiet while intake keeps working.

make simulate runs them verbose so the transcript prints; make test runs them with everything else.

Why

Tomorrow's QA is otherwise a person talking to a daemon and watching. A scripted day that replays identically makes a regression in the proactive loop, the intake journal or the reply contract fall out of a test run instead of a session. #288 (PR #75) explicitly deferred its tier-2 full-pipeline scenarios here.

Determinism

One fakeClock feeds every clock reader in the world — the handler's now, the intake publish stamp, and tickLoop.tick(ctx, now), which already took now. There is no time.Now() on the replay path and the daemon's wall-clock ticker is never started. TestSimulatorIsDeterministic replays a scenario twice and requires byte-identical transcripts; advanceTo fatals on a step that goes backwards, so a scenario cannot silently depend on ordering the clock does not enforce.

Production impact

None. Every file added is _test.go or testdata; the only non-test change is a new simulate Makefile target. The daemon behaves identically when no scenario is running.

Behaviour this pinned down

morning_missed step 6 documents a real path: a query whose recall finds nothing above the score gate returns the canned "не знаю." and never reaches the replier. That is the no-hallucination floor and the scenario now asserts it rather than expecting a phrased answer.

How verified

  • make simulate — 2 scenarios + determinism + backwards-step guard, all pass, transcripts readable.
  • make build — exit 0.
  • make test — exit 0 (fmt-check, vet, -race across ./internal/... ./cmd/...).

Vikunja #284

## What changed A JSON-driven scenario replayer, test-only, in `cmd/mavend/simulator_test.go` plus `cmd/mavend/testdata/scenarios/`. A scenario declares a start instant, a script of canned model answers (routing and phrasing are separate contracts, so an entry carries `route` and/or `reply`), and a list of steps at `"HH:MM"`. A step does exactly one thing — `say`, `audio`, `signal`, `arrive`, `tick`, `fault`/`clear_fault` — and then asserts: `expect_reply_contains`/`_lacks`, `expect_sent_contains`, `expect_no_send`, `expect_called`/`expect_not_called`, `expect_events`, `expect_no_events`. The negative assertions are step-scoped, because "nothing was sent because of THIS" is the useful question. Real between the boundaries: the router cascade (stage0 → LLM router over a scripted completer → classifier), the store, the reactive handler, the tick loop, and the intake-decorated `ipc.CoreAPI` from #283. Faked only where a test cannot have the real thing: the model, mic, speaker, delivery sink, and the Praxis/Nexus/Hexis HTTP services (reusing the `fakeServer` harness from #276). Two scenarios: - **morning_missed** — #284's own description replayed. Desk presence at 08:30, then an RSS note, a mail candidate and a relayed ambient notification arrive; two ticks; he asks what he missed and states a fact. Half the assertions are that nothing was sent at him unprompted. - **evening_degraded** — the tier-2 pipeline case #288 deferred to this task: a golden WAV from `cmd/mavsttd/testdata/golden_v1.json` through the STT seam, the router, the store write and the phrasing contract. Then the ecosystem answers 503 and the proactive loop has to stay quiet while intake keeps working. `make simulate` runs them verbose so the transcript prints; `make test` runs them with everything else. ## Why Tomorrow's QA is otherwise a person talking to a daemon and watching. A scripted day that replays identically makes a regression in the proactive loop, the intake journal or the reply contract fall out of a test run instead of a session. #288 (PR #75) explicitly deferred its tier-2 full-pipeline scenarios here. ## Determinism One `fakeClock` feeds every clock reader in the world — the handler's `now`, the intake publish stamp, and `tickLoop.tick(ctx, now)`, which already took `now`. There is no `time.Now()` on the replay path and the daemon's wall-clock ticker is never started. `TestSimulatorIsDeterministic` replays a scenario twice and requires byte-identical transcripts; `advanceTo` fatals on a step that goes backwards, so a scenario cannot silently depend on ordering the clock does not enforce. ## Production impact None. Every file added is `_test.go` or testdata; the only non-test change is a new `simulate` Makefile target. The daemon behaves identically when no scenario is running. ## Behaviour this pinned down `morning_missed` step 6 documents a real path: a query whose recall finds nothing above the score gate returns the canned `"не знаю."` and never reaches the replier. That is the no-hallucination floor and the scenario now asserts it rather than expecting a phrased answer. ## How verified - `make simulate` — 2 scenarios + determinism + backwards-step guard, all pass, transcripts readable. - `make build` — exit 0. - `make test` — exit 0 (fmt-check, vet, `-race` across `./internal/...` `./cmd/...`). Vikunja #284
claude added 1 commit 2026-08-01 04:15:45 +02:00
Add a replayable full-system simulator on a fake clock (#284) 33e53ee897 
A scenario is a JSON file under cmd/mavend/testdata/scenarios: a start
instant, a script of canned model answers, and a list of steps at "HH:MM".
Each step does one thing — say, audio, signal, arrive, tick, fault — and
then asserts on what she said, what was sent, which ecosystem services were
called, and what landed in the intake journal.

Between those boundaries the real components run: the real router cascade
(stage0, the LLM router over a scripted completer, the classifier
underneath it), the real store, the real reactive handler, the real tick
loop, and the same intake-decorated ipc.CoreAPI the daemon wires. What is
faked is only what a test cannot have: the model, the microphone, the
speaker, the delivery sink, and the ecosystem HTTP services.

Time is a single fakeClock threaded into every reader — the handler, the
intake publish stamp and tick(ctx, now) — so there is no time.Now() on the
replay path and a scenario is reproducible. TestSimulatorIsDeterministic
enforces that by replaying twice and diffing the transcripts byte for byte;
advanceTo refuses a step that goes backwards.

Two scenarios ship. morning_missed replays #284's own description: he
appears at the desk, a feed item, a mail candidate and a relayed
notification arrive through the morning, two ticks pass, and the assertions
are as much about nothing being sent at him unprompted as about what she
said. evening_degraded picks up the tier-2 pipeline case #288 deferred
here — a golden WAV through the STT seam to a written fact — and then puts
the ecosystem into 503 and checks that the proactive loop stays quiet and
that intake keeps working without it.

This is test-only code. Nothing in the production binaries changed, so the
daemon behaves identically when no scenario is running.

`make simulate` runs them verbose so the transcript is readable; `make
test` runs them with everything else.

Vikunja #284
claude reviewed 2026-08-01 11:33:21 +02:00
claude left a comment
Author
Contributor
Copy Link
Copy Source

Making "nothing happened" a first-class assertion is the right instinct. expect_no_send scoped to the step rather than to the run is the detail that makes it mean anything. Adding a scenario as a JSON file with no Go change is what will keep this used. DisallowUnknownFields on the scenario decoder earns its line: a typo in expect_no_send would otherwise turn a negative assertion into silence. The whole thing being test-only, with no flag and no production branch, is the correct shape.

1. expect_not_called can pass while the forbidden call happened.

callPaths concatenates per server, praxis then nexus then hexis. callCount is a total. assert then slices the concatenated list by the total: paths[callsBefore:]. Those two orderings are not the same list.

Walk it with all three fakes wired. Before the step, praxis has seen 3 requests and nexus 1, so callsBefore is 4 and paths is [p1 p2 p3 n1]. The step makes one praxis call. Now paths is [p1 p2 p3 p4 n1], and paths[4:] is [n1]. The new praxis call sits at index 3 and is never examined. expect_not_called: ["praxis"] passes on a step that called praxis.

The same slice makes the opposite error too. [n1] is a stale call from an earlier step, so expect_not_called: ["/resolve"] fails on a step that resolved nothing. Record a single ordered log across all three fakes, or take a per-server snapshot before the step.

2. Nothing in either scenario ever calls the ecosystem, so fault is inert.

evening_degraded step 21:15 says "a tick against a dead ecosystem must degrade, not send half a thought". The tick loop in this world cannot reach the ecosystem. newSimWorld builds it with newTickLoop(st, gatherer, dispatcher, phraser.NewStub(), rules, ...) and no eco argument, matching production, and tick.go mentions praxis nowhere. The only reader of eco.praxis is handlePraxisAct, which fires on an act decision whose Fn matches an allowlisted alias.

Neither scenario produces an act. morning_missed scripts fact, query and a chat catch-all. evening_degraded scripts fact and chat. So the praxis fake receives zero requests in both files, and deleting "fault": 503 and "clear_fault": true changes no assertion outcome. Neither scenario uses expect_called or expect_not_called either, and neither wires nexus_resolve or hexis_capabilities. The two headline capabilities of the harness, "what TOOLS were called" and the degraded-mode lever, have no coverage at all.

That matters more than a missing test, because PR 76 built a fault-injection harness for exactly these modes. Script [{"intent":"act","fn":"..."}] against a 503 praxis. Assert the reply and expect_no_send. That is the case this file promises and does not have.

3. TestSimulatorIsDeterministic reads the wall clock and will flake.

if strings.Contains(first, time.Now().Format("15:04")) && !strings.Contains(sc.Start, time.Now().Format("15:04")) {

morning_missed logs transcript lines at 08:30, 08:32, 08:35, 08:40, 08:45, 08:50, 08:55 and 09:00, formatted 15:04:05. sc.Start is 2026-08-01T08:30:00+03:00, which contains only 08:30. Run the suite at 08:35 local and first contains 08:35:00, sc.Start does not contain 08:35, and the test reports "transcript carries the wall clock". That is a 30-minute window per day on a test whose subject is determinism.

The first half of the test already proves the property. Two runs of a path with a live time.Now() in it diverge, and first != second catches that. Drop the second check, or assert something structural instead, such as every transcript timestamp falling inside the scenario's own span.

4. A scenario cannot express a fact below full confidence, and one scenario's note claims it does.

morning_missed step 08:40 is annotated "a relayed notification, below full confidence". simWorld.write hardcodes Confidence: 1.0, and signalStep has no confidence field. The ambient path writes calendar.AmbientConfidence, 0.6. So the simulated ambient arrival is not the ambient path, and the note on the step is false.

This is a live gap against #283 rather than a cosmetic one. factPriority in intake.go is the only consumer of Confidence, and 1.0 versus 0.6 is exactly the branch it takes. A replay cannot reach the low branch, so no scenario can pin what the journal does with priority. Add confidence to signalStep, default it to 1.0, and set 0.6 on that step.

Smaller notes:

  • TestSimulatorRefusesBackwardsSteps does not test a backwards step. It advances to 09:00, then to 09:30, and asserts the clock moved. The name and the doc comment both promise the guard. Take a fatalf func(string, ...any) on simWorld so the backwards branch is reachable from a test.
  • ExpectEvents and ExpectCalled are run-scoped while their negatives are step-scoped. That is documented, but it means a scenario author who repeats expect_events: ["rss:tech"] on a later step is asserting nothing new. Worth saying in the step doc comment, not just in the type comment.
  • expect_no_events compares bus.Len(), which saturates at the ring capacity of 512. A long scenario that fills the ring makes every subsequent expect_no_events pass unconditionally. Compare the newest event instead, or count publishes with a subscriber.
  • audioText re-reads and re-parses golden_v1.json on every audio step. Two steps is fine, a scenario with twenty is not.
  • The persona assertions at 08:50 run against a canned refusal from the query path, not against anything the replier produced. The step note says so, which is honest, but it means expect_reply_lacks: ["рад "] is checking a literal in the Go source. The 08:55 check ["записал,"] depends on a comma and passes on "записал что ты выпил воды".
Making "nothing happened" a first-class assertion is the right instinct. `expect_no_send` scoped to the step rather than to the run is the detail that makes it mean anything. Adding a scenario as a JSON file with no Go change is what will keep this used. `DisallowUnknownFields` on the scenario decoder earns its line: a typo in `expect_no_send` would otherwise turn a negative assertion into silence. The whole thing being test-only, with no flag and no production branch, is the correct shape. **1. `expect_not_called` can pass while the forbidden call happened.** `callPaths` concatenates per server, praxis then nexus then hexis. `callCount` is a total. `assert` then slices the concatenated list by the total: `paths[callsBefore:]`. Those two orderings are not the same list. Walk it with all three fakes wired. Before the step, praxis has seen 3 requests and nexus 1, so `callsBefore` is 4 and `paths` is `[p1 p2 p3 n1]`. The step makes one praxis call. Now `paths` is `[p1 p2 p3 p4 n1]`, and `paths[4:]` is `[n1]`. The new praxis call sits at index 3 and is never examined. `expect_not_called: ["praxis"]` passes on a step that called praxis. The same slice makes the opposite error too. `[n1]` is a stale call from an earlier step, so `expect_not_called: ["/resolve"]` fails on a step that resolved nothing. Record a single ordered log across all three fakes, or take a per-server snapshot before the step. **2. Nothing in either scenario ever calls the ecosystem, so `fault` is inert.** `evening_degraded` step 21:15 says "a tick against a dead ecosystem must degrade, not send half a thought". The tick loop in this world cannot reach the ecosystem. `newSimWorld` builds it with `newTickLoop(st, gatherer, dispatcher, phraser.NewStub(), rules, ...)` and no eco argument, matching production, and `tick.go` mentions praxis nowhere. The only reader of `eco.praxis` is `handlePraxisAct`, which fires on an `act` decision whose `Fn` matches an allowlisted alias. Neither scenario produces an `act`. `morning_missed` scripts `fact`, `query` and a `chat` catch-all. `evening_degraded` scripts `fact` and `chat`. So the praxis fake receives zero requests in both files, and deleting `"fault": 503` and `"clear_fault": true` changes no assertion outcome. Neither scenario uses `expect_called` or `expect_not_called` either, and neither wires `nexus_resolve` or `hexis_capabilities`. The two headline capabilities of the harness, "what TOOLS were called" and the degraded-mode lever, have no coverage at all. That matters more than a missing test, because PR 76 built a fault-injection harness for exactly these modes. Script `[{"intent":"act","fn":"..."}]` against a 503 praxis. Assert the reply and `expect_no_send`. That is the case this file promises and does not have. **3. `TestSimulatorIsDeterministic` reads the wall clock and will flake.** ```go if strings.Contains(first, time.Now().Format("15:04")) && !strings.Contains(sc.Start, time.Now().Format("15:04")) { ``` `morning_missed` logs transcript lines at 08:30, 08:32, 08:35, 08:40, 08:45, 08:50, 08:55 and 09:00, formatted `15:04:05`. `sc.Start` is `2026-08-01T08:30:00+03:00`, which contains only `08:30`. Run the suite at 08:35 local and `first` contains `08:35:00`, `sc.Start` does not contain `08:35`, and the test reports "transcript carries the wall clock". That is a 30-minute window per day on a test whose subject is determinism. The first half of the test already proves the property. Two runs of a path with a live `time.Now()` in it diverge, and `first != second` catches that. Drop the second check, or assert something structural instead, such as every transcript timestamp falling inside the scenario's own span. **4. A scenario cannot express a fact below full confidence, and one scenario's note claims it does.** `morning_missed` step 08:40 is annotated "a relayed notification, below full confidence". `simWorld.write` hardcodes `Confidence: 1.0`, and `signalStep` has no confidence field. The ambient path writes `calendar.AmbientConfidence`, 0.6. So the simulated ambient arrival is not the ambient path, and the note on the step is false. This is a live gap against #283 rather than a cosmetic one. `factPriority` in `intake.go` is the only consumer of `Confidence`, and 1.0 versus 0.6 is exactly the branch it takes. A replay cannot reach the low branch, so no scenario can pin what the journal does with priority. Add `confidence` to `signalStep`, default it to 1.0, and set 0.6 on that step. Smaller notes: - `TestSimulatorRefusesBackwardsSteps` does not test a backwards step. It advances to 09:00, then to 09:30, and asserts the clock moved. The name and the doc comment both promise the guard. Take a `fatalf func(string, ...any)` on `simWorld` so the backwards branch is reachable from a test. - `ExpectEvents` and `ExpectCalled` are run-scoped while their negatives are step-scoped. That is documented, but it means a scenario author who repeats `expect_events: ["rss:tech"]` on a later step is asserting nothing new. Worth saying in the step doc comment, not just in the type comment. - `expect_no_events` compares `bus.Len()`, which saturates at the ring capacity of 512. A long scenario that fills the ring makes every subsequent `expect_no_events` pass unconditionally. Compare the newest event instead, or count publishes with a subscriber. - `audioText` re-reads and re-parses `golden_v1.json` on every audio step. Two steps is fine, a scenario with twenty is not. - The persona assertions at 08:50 run against a canned refusal from the query path, not against anything the replier produced. The step note says so, which is honest, but it means `expect_reply_lacks: ["рад "]` is checking a literal in the Go source. The 08:55 check `["записал,"]` depends on a comma and passes on "записал что ты выпил воды".
kami closed this pull request 2026-08-01 14:52:05 +02:00
kami commented 2026-08-01 14:52:58 +02:00
Owner
Copy Link
Copy Source

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
claude
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
kami
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kami/Maven#79
Delete Branch "overnight/replay-simulator"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?