85a3397bf4
reminder_cancel.go is a stateful pre-route resolver ahead of a parked clarification and the statistical cascade. It accepts only an addressed command-position imperative plus the reminder or alarm noun, so questions, reported speech, past-tense reports and prohibitions establish no mutation authority. Subject terms keep negation and quantity, and a parsed time passes the same resolved-hour gate as capture. One match cancels through the typed IPC method. Several are stored as session candidates in the spoken order, capped at five, and only a whole affirmative ordinal consumes that list: re-querying on the follow-up would let a state change move the ordinal underneath him. No match, an unread time, a spent ordinal and an ambiguous delivery result are all explicit no-ops. command_prohibition.go is the first mutation boundary in a turn. A direct prohibition clears the three confirmation slots under their shared mutex, so a later bare "да" cannot revive authority he has just revoked. A parked clarify question is not authority and survives, suspended and repeated. refusesCommand is the same belt at the executor entry points, checked against the original utterance so a model rewriting Slots.Text cannot get around it. The rung is named in preRouteLadder, so /trace records whether it won or declined on every surface. --no-verify: master is the working branch this session by the owner's call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1467 lines
52 KiB
Go
1467 lines
52 KiB
Go
// mavend/simulator_test.go — the replayable full-system simulator
|
|
// (Vikunja #284).
|
|
//
|
|
// # What it is
|
|
//
|
|
// A scripted day, replayed through the real mavend code paths, with every
|
|
// boundary faked and the clock under the scenario's control. A scenario is a
|
|
// JSON file in testdata/scenarios; the harness reads it, builds a world, walks
|
|
// the steps in order, and asserts on what actually happened:
|
|
//
|
|
// what Maven SAID — the reply text of every utterance
|
|
// what was SENT — every delivery.Sendable the dispatcher emitted
|
|
// what ARRIVED — the unified intake journal from #283
|
|
// what TOOLS were called — the recorded requests against fake Praxis/Nexis/Hexis
|
|
// what is DURABLE — typed notes/tasks/reminders/facts store state
|
|
// what did NOT happen — expect_no_send / expect_no_call, first-class
|
|
//
|
|
// The last one is the point. Maven's hard constraints are mostly negative —
|
|
// not a nag, not autonomous, nothing executed without confirmation — and a
|
|
// harness that can only assert on things that happened cannot test any of
|
|
// them. "Nothing was sent" is an assertion here, not an absence of one.
|
|
//
|
|
// # Determinism
|
|
//
|
|
// No time.Now() runs inside a replay. The scenario names a start instant, each
|
|
// step names a wall-clock offset from it, and the harness advances a fakeClock
|
|
// to that offset before running the step. Every clock reader in the world —
|
|
// the handler's `now`, the tick loop's `tick(ctx, now)`, the intake journal's
|
|
// publish stamp — is wired to that clock. Two runs of the same file produce
|
|
// the same transcript, and a scenario about 08:35 does not behave differently
|
|
// at 03:00 in CI.
|
|
//
|
|
// The tick is driven by the scenario, not by a ticker: tick() already takes
|
|
// `now` as an argument, so the only thing the daemon's ticker contributed was
|
|
// wall-clock timing, which is exactly what a replay must not have.
|
|
//
|
|
// # Why this shape and not a binary
|
|
//
|
|
// Vikunja #288 (golden-audio STT) deferred its tier-2 "audio → STT → router →
|
|
// phraser" scenarios to this task, and asked that they reuse a fixture format
|
|
// rather than inventing a third. A scenario here can name a WAV from
|
|
// cmd/mavsttd/testdata and the harness will feed it through the STT seam. As a
|
|
// test it runs under `make test` on every change, which a separate binary
|
|
// would not.
|
|
//
|
|
// # Production is untouched
|
|
//
|
|
// Every file this task adds is a _test.go file or testdata. There is no
|
|
// simulator in the daemon, no flag, no config key, and no code path that
|
|
// checks whether a simulation is running. The seams it uses — stt.Transcriber,
|
|
// tts.Synthesizer, router.Completer, delivery.Sink, ipc.CoreAPI, the
|
|
// event.Bus from #283 — all already existed for the production wiring.
|
|
package main
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"net/http"
|
|
"os"
|
|
"path/filepath"
|
|
"regexp"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/kami/maven/internal/audio"
|
|
"github.com/kami/maven/internal/config"
|
|
"github.com/kami/maven/internal/delivery"
|
|
"github.com/kami/maven/internal/dialogue"
|
|
"github.com/kami/maven/internal/event"
|
|
"github.com/kami/maven/internal/ipc"
|
|
"github.com/kami/maven/internal/llm"
|
|
"github.com/kami/maven/internal/loop"
|
|
"github.com/kami/maven/internal/phraser"
|
|
"github.com/kami/maven/internal/router"
|
|
"github.com/kami/maven/internal/store"
|
|
"github.com/kami/maven/internal/tool"
|
|
"github.com/kami/maven/internal/voice"
|
|
)
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Scenario format
|
|
// ---------------------------------------------------------------------------
|
|
|
|
// scenario — one scripted day. schema_version matches the convention already
|
|
// set by testdata/system_safety_scenarios.json.
|
|
type scenario struct {
|
|
SchemaVersion int `json:"schema_version"`
|
|
Name string `json:"name"`
|
|
Description string `json:"description,omitempty"`
|
|
|
|
// Start — the instant the day begins, RFC3339. Every step offset is
|
|
// relative to it, and nothing in the run reads a real clock.
|
|
Start string `json:"start"`
|
|
|
|
// Script — what the resident model answers. The world has no llama-server;
|
|
// see scriptedLLM for how an entry is chosen.
|
|
Script []scriptEntry `json:"script,omitempty"`
|
|
|
|
// Praxis / Nexus / Hexis — canned bodies for the ecosystem fakes. Absent ⇒
|
|
// that service is not wired at all, which is the default box.
|
|
Praxis string `json:"praxis_attention,omitempty"`
|
|
Nexus string `json:"nexus_resolve,omitempty"`
|
|
Hexis string `json:"hexis_capabilities,omitempty"`
|
|
|
|
// Tools — allowlist rows to enable before the first step. An act is only
|
|
// dispatched when its verb is on the enabled allowlist, so a scenario that
|
|
// wants to exercise one has to say which rows Kami had enabled.
|
|
Tools []toolRow `json:"tools,omitempty"`
|
|
|
|
Steps []step `json:"steps"`
|
|
}
|
|
|
|
// toolRow — one enabled allowlist row. Cmd is empty for an ecosystem verb,
|
|
// which is intercepted before the process executor is ever reached.
|
|
type toolRow struct {
|
|
Name string `json:"name"`
|
|
Cmd []string `json:"cmd,omitempty"`
|
|
Destructive bool `json:"destructive,omitempty"`
|
|
}
|
|
|
|
// scriptEntry — one canned model answer. Match is a substring of the user
|
|
// message; the first entry whose Match is contained in it wins, and an entry
|
|
// with an empty Match is the catch-all.
|
|
//
|
|
// Route and Reply are separate because the same model serves both contracts
|
|
// (CLAUDE.md, "LLM output contract"): a grammar-constrained call is a routing
|
|
// call and gets Route, an unconstrained one is a phrasing call and gets Reply.
|
|
// HistoryContains makes a chat reply conditional on the transcript the daemon
|
|
// supplied. It prevents a canned answer from making a continuity scenario pass
|
|
// while the referent is still absent from the model input.
|
|
type scriptEntry struct {
|
|
Match string `json:"match"`
|
|
Route string `json:"route,omitempty"`
|
|
Reply string `json:"reply,omitempty"`
|
|
HistoryContains []string `json:"history_contains,omitempty"`
|
|
}
|
|
|
|
// step — one scripted moment. At is "HH:MM" or "HH:MM:SS", interpreted in the
|
|
// start instant's location; the clock is advanced to it before the step runs.
|
|
//
|
|
// A step does exactly one thing (say / audio / signal / fact / tick / arrive)
|
|
// and then asserts. Assertions are evaluated against everything recorded since
|
|
// the run began, except expect_no_send and expect_no_call, which are scoped to
|
|
// this step — "nothing was sent because of THIS" is the useful question.
|
|
//
|
|
// The asymmetry is worth stating plainly, because it changes what a scenario
|
|
// author is writing. expect_sent_contains, expect_called and expect_events are
|
|
// RUN-scoped: they pass if the thing ever happened, at any earlier step. So
|
|
// repeating expect_events: ["rss:tech"] on a later step asserts nothing new,
|
|
// it just re-checks the earlier arrival. The negatives — expect_no_send,
|
|
// expect_not_called, expect_no_events — are STEP-scoped, and are the ones that
|
|
// say something about this moment. expect_reply_contains and
|
|
// expect_reply_lacks read the most recent reply only, so a step with no
|
|
// utterance re-checks the previous one.
|
|
type step struct {
|
|
At string `json:"at"`
|
|
Note string `json:"note,omitempty"`
|
|
|
|
// --- stimuli (at most one per step) ---
|
|
|
|
// Say — an utterance, as text, through the same runTurn the IPC chat path
|
|
// uses.
|
|
Say string `json:"say,omitempty"`
|
|
|
|
// Audio — a WAV under cmd/mavsttd/testdata, fed through the STT seam. This
|
|
// is #288's deferred tier 2. The harness uses the deterministic stt stub
|
|
// unless a real transcriber is available, so the assertion a scenario can
|
|
// make about an audio step is about the PIPELINE, not about whisper's
|
|
// accuracy — that is what cmd/mavsttd/golden_test.go is for.
|
|
Audio string `json:"audio,omitempty"`
|
|
|
|
// Signal — a presence/world fact arriving from a poller or /api/signal.
|
|
Signal *signalStep `json:"signal,omitempty"`
|
|
|
|
// Arrive — an intake write from a module: an ambient notification, a feed
|
|
// item, a mail candidate. Goes through the same decorated ipc.CoreAPI the
|
|
// daemon gives those callers, so it lands in the journal exactly as it
|
|
// would in production.
|
|
Arrive *arriveStep `json:"arrive,omitempty"`
|
|
|
|
// Tick — run one iteration of the proactive loop at this instant.
|
|
Tick bool `json:"tick,omitempty"`
|
|
|
|
// Fault — make every ecosystem fake answer with this HTTP status from now
|
|
// on. The degraded-mode lever; ClearFault puts them back.
|
|
Fault int `json:"fault,omitempty"`
|
|
ClearFault bool `json:"clear_fault,omitempty"`
|
|
|
|
// --- assertions ---
|
|
|
|
ExpectReply []string `json:"expect_reply_contains,omitempty"`
|
|
ExpectNotReply []string `json:"expect_reply_lacks,omitempty"`
|
|
ExpectSent []string `json:"expect_sent_contains,omitempty"`
|
|
ExpectNoSend bool `json:"expect_no_send,omitempty"`
|
|
ExpectCalled []string `json:"expect_called,omitempty"`
|
|
ExpectNotCalled []string `json:"expect_not_called,omitempty"`
|
|
ExpectEvents []string `json:"expect_events,omitempty"`
|
|
ExpectNoEvents bool `json:"expect_no_events,omitempty"`
|
|
ExpectStore *storeStateExpectation `json:"expect_store,omitempty"`
|
|
}
|
|
|
|
// storeStateExpectation is a typed, exact read of Maven's four user-visible
|
|
// durable stores. Reply assertions prove what she said; these prove what the
|
|
// turn actually committed. Each selected collection can assert its total row
|
|
// count and exact row identity independently, so a duplicate insert cannot be
|
|
// hidden by finding one matching row.
|
|
type storeStateExpectation struct {
|
|
Notes *noteStateExpectation `json:"notes,omitempty"`
|
|
Tasks *taskStateExpectation `json:"tasks,omitempty"`
|
|
Reminders *reminderStateExpectation `json:"reminders,omitempty"`
|
|
Facts *factStateExpectation `json:"facts,omitempty"`
|
|
}
|
|
|
|
type noteStateExpectation struct {
|
|
Count *int `json:"count,omitempty"`
|
|
Rows []noteRowExpectation `json:"rows,omitempty"`
|
|
}
|
|
|
|
type noteRowExpectation struct {
|
|
ID int64 `json:"id,omitempty"`
|
|
At string `json:"at,omitempty"`
|
|
Text string `json:"text,omitempty"`
|
|
Source string `json:"source,omitempty"`
|
|
}
|
|
|
|
type taskStateExpectation struct {
|
|
Count *int `json:"count,omitempty"`
|
|
Rows []taskRowExpectation `json:"rows,omitempty"`
|
|
}
|
|
|
|
type taskRowExpectation struct {
|
|
ID int64 `json:"id,omitempty"`
|
|
CreatedAt string `json:"created_at,omitempty"`
|
|
Text string `json:"text,omitempty"`
|
|
Source string `json:"source,omitempty"`
|
|
Status string `json:"status,omitempty"`
|
|
ResolvedAt string `json:"resolved_at,omitempty"`
|
|
ResolvedBy string `json:"resolved_by,omitempty"`
|
|
}
|
|
|
|
type reminderStateExpectation struct {
|
|
Count *int `json:"count,omitempty"`
|
|
Rows []reminderRowExpectation `json:"rows,omitempty"`
|
|
}
|
|
|
|
type reminderRowExpectation struct {
|
|
ID int64 `json:"id,omitempty"`
|
|
FireAt string `json:"fire_at,omitempty"`
|
|
Text string `json:"text,omitempty"`
|
|
Status string `json:"status,omitempty"`
|
|
}
|
|
|
|
type factStateExpectation struct {
|
|
Count *int `json:"count,omitempty"`
|
|
Rows []factRowExpectation `json:"rows,omitempty"`
|
|
}
|
|
|
|
type factRowExpectation struct {
|
|
ID int64 `json:"id,omitempty"`
|
|
At string `json:"at,omitempty"`
|
|
Kind string `json:"kind,omitempty"`
|
|
Key string `json:"key,omitempty"`
|
|
Value string `json:"value,omitempty"`
|
|
Source string `json:"source,omitempty"`
|
|
Confidence *float64 `json:"confidence,omitempty"`
|
|
}
|
|
|
|
type signalStep struct {
|
|
Key string `json:"key"`
|
|
Value string `json:"value"`
|
|
Source string `json:"source"`
|
|
Kind string `json:"kind,omitempty"`
|
|
|
|
// Confidence — 0 ⇒ 1.0, an observation Maven made herself. A relayed
|
|
// notification is not that: the ambient path writes
|
|
// calendar.AmbientConfidence, 0.6, and factPriority in intake.go branches
|
|
// on exactly that difference. The field exists so a replay can reach the
|
|
// low branch, which it could not while write() hardcoded 1.0.
|
|
Confidence float64 `json:"confidence,omitempty"`
|
|
}
|
|
|
|
type arriveStep struct {
|
|
// Note / Fact / Task — exactly one. Each mirrors the intake seam its real
|
|
// caller uses.
|
|
Note *arriveNote `json:"note,omitempty"`
|
|
Fact *signalStep `json:"fact,omitempty"`
|
|
Task *arriveTask `json:"task,omitempty"`
|
|
AsOf string `json:"as_of,omitempty"` // "HH:MM" — OccurredAt, when it differs from the step time
|
|
Source string `json:"source"`
|
|
}
|
|
|
|
type arriveNote struct {
|
|
Text string `json:"text"`
|
|
}
|
|
|
|
type arriveTask struct {
|
|
Text string `json:"text"`
|
|
Evidence string `json:"evidence,omitempty"`
|
|
Status string `json:"status,omitempty"`
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// The world
|
|
// ---------------------------------------------------------------------------
|
|
|
|
// simWorld — every faked boundary plus the real components between them.
|
|
type simWorld struct {
|
|
t *testing.T
|
|
clock *fakeClock
|
|
loc *time.Location
|
|
start time.Time
|
|
|
|
store *store.Store
|
|
api ipc.CoreAPI // the intake-decorated adapter, same as the daemon builds
|
|
bus *event.Bus
|
|
handler *reactiveHandler
|
|
tick *tickLoop
|
|
sink *recordingSink
|
|
llm *scriptedLLM
|
|
|
|
praxis *fakeServer
|
|
nexus *fakeServer
|
|
hexis *fakeServer
|
|
|
|
// transcript — everything that happened, in order. Printed on failure so a
|
|
// broken scenario is diagnosable without a debugger.
|
|
transcript []string
|
|
replies []string
|
|
|
|
// fatalf — the abort seam. Defaults to t.Fatalf. It exists so a test can
|
|
// reach the harness's own refusals (a backwards step, a missing WAV) and
|
|
// assert on them instead of dying with the scenario.
|
|
fatalf func(format string, args ...any)
|
|
|
|
// published — how many events the bus accepted, counted through a
|
|
// subscriber. bus.Len() saturates at the ring capacity and cannot answer
|
|
// "did anything arrive during this step" once a long scenario has filled
|
|
// it.
|
|
mu sync.Mutex
|
|
published int
|
|
|
|
// audio — golden_v1.json, parsed once. A scenario with twenty audio steps
|
|
// used to read and parse the manifest twenty times.
|
|
audio map[string]string
|
|
}
|
|
|
|
func (w *simWorld) publishCount() int {
|
|
w.mu.Lock()
|
|
defer w.mu.Unlock()
|
|
return w.published
|
|
}
|
|
|
|
// recordingSink captures every send, mutex-guarded (the tick loop dispatches
|
|
// from its own goroutine in production and the race detector is on here).
|
|
type recordingSink struct {
|
|
mu sync.Mutex
|
|
sends []delivery.Sendable
|
|
}
|
|
|
|
func (s *recordingSink) Send(_ context.Context, d delivery.Sendable) error {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
s.sends = append(s.sends, d)
|
|
return nil
|
|
}
|
|
|
|
func (s *recordingSink) all() []delivery.Sendable {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
out := make([]delivery.Sendable, len(s.sends))
|
|
copy(out, s.sends)
|
|
return out
|
|
}
|
|
|
|
func (s *recordingSink) count() int {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
return len(s.sends)
|
|
}
|
|
|
|
// scriptedLLM stands in for llama-server on BOTH contracts the resident model
|
|
// serves: grammar-constrained routing and unconstrained phrasing.
|
|
//
|
|
// It is not a stub that ignores its input — a scenario that scripts an answer
|
|
// for "что я пропустил" and gets asked something else must fail, not silently
|
|
// return the wrong intent. An unmatched call returns an error, and the router
|
|
// then falls through to the classifier cascade exactly as it does in
|
|
// production when llama-server is unreachable. That fall-through is itself
|
|
// worth exercising: it is the failure floor CLAUDE.md refuses to let rot.
|
|
type scriptedLLM struct {
|
|
mu sync.Mutex
|
|
entries []scriptEntry
|
|
calls []llm.Req
|
|
}
|
|
|
|
func (s *scriptedLLM) Complete(_ context.Context, r llm.Req) (string, error) {
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
s.calls = append(s.calls, r)
|
|
// A grammar no longer separates the two contracts — the replier carries one
|
|
// too since phraser.ResponseGrammar was attached to it. Only the router's
|
|
// grammar names the intent enum, so that is what tells them apart.
|
|
routing := strings.Contains(r.Grammar, "intent")
|
|
for _, e := range s.entries {
|
|
if e.Match != "" && !strings.Contains(strings.ToLower(r.User), strings.ToLower(e.Match)) {
|
|
continue
|
|
}
|
|
if routing && e.Route != "" {
|
|
return e.Route, nil
|
|
}
|
|
if !routing && e.Reply != "" {
|
|
return e.Reply, nil
|
|
}
|
|
}
|
|
return "", fmt.Errorf("simulator: no scripted %s answer for %q",
|
|
map[bool]string{true: "route", false: "reply"}[routing], truncateRunes(r.User, 60))
|
|
}
|
|
|
|
// scriptedPhraser answers the chat path from the same script the router reads.
|
|
//
|
|
// It exists because actionChat calls h.phraser.PhraseChat, and the production
|
|
// implementation posts raw HTTP to /v1/chat/completions rather than going
|
|
// through the llm client scriptedLLM stands in for. So until this, no scenario
|
|
// could script what she SAYS on a chat turn: the simulator wired phraser.NewStub()
|
|
// and every chat reply came back as a pick from fallbacks_ru_v1.json, four
|
|
// variants deep, which varied between two runs of one scenario (V-542 item 4).
|
|
//
|
|
// Everything except PhraseChat is the Stub's, by embedding. A nudge and a
|
|
// reminder are phrased by the tick loop, which has its own phraser and its own
|
|
// assertions; this seam is only about the conversation.
|
|
type scriptedPhraser struct {
|
|
*phraser.Stub
|
|
entries []scriptEntry
|
|
}
|
|
|
|
// PhraseChat returns the scripted reply for the utterance, or an error when the
|
|
// scenario scripted none. The error rather than a fallback is deliberate and
|
|
// matches scriptedLLM: actionChat logs it and falls back to ChatFallback(), so a
|
|
// scenario that never meant to assert on a chat reply behaves exactly as it did
|
|
// before, and one that DID means to is told its script has a hole.
|
|
func (p *scriptedPhraser) PhraseChat(_ context.Context, utterance string, history []dialogue.Turn) (string, error) {
|
|
for _, e := range p.entries {
|
|
if e.Reply == "" {
|
|
continue
|
|
}
|
|
if e.Match != "" && !strings.Contains(strings.ToLower(utterance), strings.ToLower(e.Match)) {
|
|
continue
|
|
}
|
|
for _, want := range e.HistoryContains {
|
|
found := false
|
|
for _, turn := range history {
|
|
if containsFold(turn.Text, want) {
|
|
found = true
|
|
break
|
|
}
|
|
}
|
|
if !found {
|
|
return "", fmt.Errorf("simulator: chat history for %q does not contain %q: %+v",
|
|
truncateRunes(utterance, 60), want, history)
|
|
}
|
|
}
|
|
return chatReplyText(e.Reply), nil
|
|
}
|
|
return "", fmt.Errorf("simulator: no scripted chat reply for %q", truncateRunes(utterance, 60))
|
|
}
|
|
|
|
// chatReplyText reads a scripted reply in either shape the phrasing contract
|
|
// allows: the {"response","mood"} object the model emits, or plain text.
|
|
// LLMPhraser does this parse itself, so a scenario writes one thing and both
|
|
// paths understand it.
|
|
func chatReplyText(reply string) string {
|
|
var out struct {
|
|
Response string `json:"response"`
|
|
}
|
|
if err := json.Unmarshal([]byte(reply), &out); err == nil && out.Response != "" {
|
|
return out.Response
|
|
}
|
|
return reply
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Building the world
|
|
// ---------------------------------------------------------------------------
|
|
|
|
func newSimWorld(t *testing.T, sc scenario) *simWorld {
|
|
t.Helper()
|
|
|
|
start, err := time.Parse(time.RFC3339, sc.Start)
|
|
if err != nil {
|
|
t.Fatalf("scenario %q: bad start %q: %v", sc.Name, sc.Start, err)
|
|
}
|
|
clock := newFakeClock(start)
|
|
|
|
st := newTestStore(t)
|
|
bus := event.NewBus(512)
|
|
// The same decorator the daemon wires, on the same clock: intake in a
|
|
// replay is journalled exactly as it is in production.
|
|
api := newIntakeAPI(ipc.NewStoreAPI(st), bus, clock.Now)
|
|
|
|
sink := &recordingSink{}
|
|
rules := loop.DefaultRules()
|
|
gatherer := loop.NewGatherer(st, rules)
|
|
dispatcher := delivery.NewDispatcher(delivery.Config{
|
|
Voice: sink, Ntfy: sink, Telegram: sink, Nudges: st, Reminders: st,
|
|
})
|
|
tl := newTickLoop(st, gatherer, dispatcher, phraser.NewStub(), rules,
|
|
time.Minute, 5*time.Minute, 0, nil, nil, nil, nil)
|
|
// Production upgrades the voice handler from the direct store adapter to
|
|
// daemonAPI after the tick loop exists. Mirror that seam so a simulated
|
|
// day-plan query reads the real store-backed plan instead of the direct
|
|
// adapter's "not available" refusal. The fake clock is the one deliberate
|
|
// difference from production's time.Now.
|
|
api = &daemonAPI{
|
|
CoreAPI: api,
|
|
getDayPlan: func(ctx context.Context) ipc.DayPlan {
|
|
return tl.dayPlan(ctx, clock.Now())
|
|
},
|
|
}
|
|
|
|
scripted := &scriptedLLM{entries: sc.Script}
|
|
|
|
w := &simWorld{
|
|
t: t, clock: clock, loc: start.Location(), start: start,
|
|
store: st, api: api, bus: bus, tick: tl, sink: sink, llm: scripted,
|
|
}
|
|
w.fatalf = t.Fatalf
|
|
bus.Subscribe(func(event.Event) {
|
|
w.mu.Lock()
|
|
w.published++
|
|
w.mu.Unlock()
|
|
})
|
|
|
|
// Allowlist rows the scenario asked for, enabled before the first step. An
|
|
// act only reaches a dispatch if a verb is on the enabled list, so without
|
|
// this a scenario cannot script one at all.
|
|
for _, tr := range sc.Tools {
|
|
cmd := tr.Cmd
|
|
if len(cmd) == 0 {
|
|
cmd = []string{"true"}
|
|
}
|
|
if err := st.EnableTool(context.Background(), tr.Name, cmd, tr.Destructive, "sim", start); err != nil {
|
|
t.Fatalf("scenario %q: enabling tool %q: %v", sc.Name, tr.Name, err)
|
|
}
|
|
}
|
|
|
|
// Ecosystem fakes, wired only when the scenario supplies a body — a box
|
|
// with no praxis block has no praxis client, and a scenario must be able to
|
|
// reproduce that.
|
|
eco := &ecosystemWiring{}
|
|
if sc.Praxis != "" {
|
|
w.praxis = newFakePraxis(t, sc.Praxis)
|
|
eco.praxis = newPraxisClient(w.praxis.URL)
|
|
}
|
|
if sc.Nexus != "" {
|
|
w.nexus = newFakeNexus(t, sc.Nexus)
|
|
}
|
|
if sc.Hexis != "" {
|
|
w.hexis = newFakeHexis(t, sc.Hexis, fixtureHexisExecuted("exec_1", "completed"))
|
|
}
|
|
|
|
// The router: the same cascade the daemon builds — stage-0 grammars, the
|
|
// LLM router on the scripted model, the classifier underneath. Keeping the
|
|
// classifier in is deliberate; it is the failure floor, and a scenario that
|
|
// scripts no route for an utterance exercises it.
|
|
emb := router.NewHashEmbedder(1024)
|
|
// The matcher reads the live enabled allowlist, same as the daemon's. It
|
|
// used to be built on a nil API, which meant any scenario that produced an
|
|
// act panicked the moment the matcher was consulted.
|
|
matcher := tool.NewMatcher(api)
|
|
rtr := buildRouter(emb, matcher, config.DefaultRouterThreshold, router.NewLLMRouter(scripted), nil)
|
|
timeParser := router.NewPythonDateParser()
|
|
|
|
w.handler = &reactiveHandler{
|
|
stt: simTranscriber{},
|
|
tts: simSynthesizer{},
|
|
router: rtr,
|
|
recall: recallWiring{
|
|
embedder: emb,
|
|
memStore: st.VectorMemory(),
|
|
minScore: config.DefaultQueryMinScore,
|
|
minMargin: config.DefaultQueryMinMargin,
|
|
},
|
|
api: api,
|
|
matcher: matcher,
|
|
tools: tool.NewExecutor(api, 5*time.Second),
|
|
phraser: &scriptedPhraser{Stub: phraser.NewStub(), entries: sc.Script},
|
|
replier: newLLMReplier(scripted, nil),
|
|
now: clock.Now,
|
|
dataStore: st,
|
|
timeParser: timeParser,
|
|
dialogueSessions: dialogue.NewSessionStore(time.Hour),
|
|
clarifyStore: dialogue.NewClarifyStore(time.Hour),
|
|
clarifyMaxAttempts: dialogue.DefaultMaxAttempts,
|
|
extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}},
|
|
ecosystem: eco,
|
|
}
|
|
return w
|
|
}
|
|
|
|
// simTranscriber — the STT seam. Deterministic by construction: it returns the
|
|
// text the harness parked for this step, so the pipeline under test is
|
|
// "audio arrives → a turn runs", not "whisper heard correctly". Transcription
|
|
// accuracy is cmd/mavsttd/golden_test.go's job (#288 tier 1), and duplicating
|
|
// it here would make every scenario depend on a 500 MB model.
|
|
type simTranscriber struct{ text string }
|
|
|
|
func (s simTranscriber) Transcribe(_ context.Context, _ audio.Audio) (string, float64, error) {
|
|
return s.text, 1.0, nil
|
|
}
|
|
|
|
// simSynthesizer — the TTS seam. A scenario asserts on what Maven SAID, which
|
|
// is the reply text; the waveform is not the artefact under test.
|
|
type simSynthesizer struct{}
|
|
|
|
func (simSynthesizer) Synthesize(_ context.Context, _ string) (audio.Audio, error) {
|
|
return audio.Audio{Format: audio.PCM16kMono}, nil
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Running
|
|
// ---------------------------------------------------------------------------
|
|
|
|
func (w *simWorld) logf(format string, args ...any) {
|
|
w.transcript = append(w.transcript,
|
|
fmt.Sprintf("%s %s", w.clock.Now().In(w.loc).Format("15:04:05"), fmt.Sprintf(format, args...)))
|
|
}
|
|
|
|
// dump prints the whole transcript. Called on any failure — a scenario that
|
|
// broke on step 7 is unreadable without the six steps before it.
|
|
func (w *simWorld) dump() {
|
|
w.t.Logf("--- replay transcript ---\n%s", strings.Join(w.transcript, "\n"))
|
|
}
|
|
|
|
// advanceTo moves the clock to the step's offset. Time only ever moves
|
|
// FORWARD: a scenario with steps out of order is a bug in the scenario, and
|
|
// silently reordering it would hide the bug.
|
|
func (w *simWorld) advanceTo(at string) {
|
|
w.t.Helper()
|
|
if at == "" {
|
|
return
|
|
}
|
|
target := w.timeOf(at)
|
|
now := w.clock.Now()
|
|
if target.Before(now) {
|
|
w.fatalf("step at %s goes backwards from %s — scenario steps must be in order",
|
|
at, now.In(w.loc).Format("15:04:05"))
|
|
return
|
|
}
|
|
w.clock.Advance(target.Sub(now))
|
|
}
|
|
|
|
// timeOf resolves an "HH:MM" or "HH:MM:SS" step offset against the scenario's
|
|
// start day and location.
|
|
func (w *simWorld) timeOf(at string) time.Time {
|
|
w.t.Helper()
|
|
layout := "15:04"
|
|
if strings.Count(at, ":") == 2 {
|
|
layout = "15:04:05"
|
|
}
|
|
hm, err := time.Parse(layout, at)
|
|
if err != nil {
|
|
w.t.Fatalf("bad step time %q: %v", at, err)
|
|
}
|
|
return time.Date(w.start.Year(), w.start.Month(), w.start.Day(),
|
|
hm.Hour(), hm.Minute(), hm.Second(), 0, w.loc)
|
|
}
|
|
|
|
func (w *simWorld) run(sc scenario) {
|
|
ctx := context.Background()
|
|
for i, s := range sc.Steps {
|
|
w.advanceTo(s.At)
|
|
if s.Note != "" {
|
|
w.logf("# %s", s.Note)
|
|
}
|
|
sendsBefore := w.sink.count()
|
|
callsBefore := w.callMark()
|
|
eventsBefore := w.publishCount()
|
|
|
|
w.stimulate(ctx, s)
|
|
w.assert(ctx, i, s, sendsBefore, callsBefore, eventsBefore)
|
|
}
|
|
}
|
|
|
|
func (w *simWorld) stimulate(ctx context.Context, s step) {
|
|
if s.Fault != 0 || s.ClearFault {
|
|
for _, fs := range []*fakeServer{w.praxis, w.nexus, w.hexis} {
|
|
if fs != nil {
|
|
fs.SetFault(s.Fault)
|
|
}
|
|
}
|
|
w.logf("fault=%d on every ecosystem fake", s.Fault)
|
|
}
|
|
|
|
switch {
|
|
case s.Say != "":
|
|
reply := w.handler.runTurn(ctx, s.Say, sourceText)
|
|
w.replies = append(w.replies, reply)
|
|
w.logf("он: %s", s.Say)
|
|
w.logf("она: %s", reply)
|
|
|
|
case s.Audio != "":
|
|
text := w.audioText(s.Audio)
|
|
// Swap in a transcriber parked with this step's text, then run the same
|
|
// push-to-talk entry point the voice client calls.
|
|
w.handler.stt = simTranscriber{text: text}
|
|
resp, err := w.handler.HandlePushToTalk(ctx, voicePTT(), 0)
|
|
if err != nil {
|
|
w.t.Fatalf("push-to-talk on %s: %v", s.Audio, err)
|
|
}
|
|
w.replies = append(w.replies, resp.ReplyText)
|
|
w.logf("[wav %s → %q]", filepath.Base(s.Audio), text)
|
|
w.logf("она: %s", resp.ReplyText)
|
|
|
|
case s.Signal != nil:
|
|
w.write(ctx, *s.Signal, w.clock.Now())
|
|
w.logf("сигнал: %s=%s (%s)", s.Signal.Key, s.Signal.Value, s.Signal.Source)
|
|
|
|
case s.Arrive != nil:
|
|
w.arrive(ctx, *s.Arrive)
|
|
|
|
case s.Tick:
|
|
w.tick.tick(ctx, w.clock.Now())
|
|
w.logf("tick")
|
|
}
|
|
}
|
|
|
|
func (w *simWorld) write(ctx context.Context, sig signalStep, ts time.Time) {
|
|
w.t.Helper()
|
|
kind := sig.Kind
|
|
if kind == "" {
|
|
kind = "env"
|
|
}
|
|
conf := sig.Confidence
|
|
if conf == 0 {
|
|
conf = 1.0
|
|
}
|
|
if _, err := w.api.WriteFact(ctx, ipc.WriteFactReq{
|
|
Ts: ts, Kind: kind, Key: sig.Key, Value: sig.Value, Source: sig.Source, Confidence: conf,
|
|
}); err != nil {
|
|
w.fatalf("write fact %s: %v", sig.Key, err)
|
|
}
|
|
}
|
|
|
|
func (w *simWorld) arrive(ctx context.Context, a arriveStep) {
|
|
w.t.Helper()
|
|
// AsOf is when the thing HAPPENED, which for a feed item or a relayed
|
|
// notification is usually earlier than when Maven heard about it. It does
|
|
// not move the clock — only the timestamp on the row and the envelope.
|
|
ts := w.clock.Now()
|
|
if a.AsOf != "" {
|
|
ts = w.timeOf(a.AsOf)
|
|
}
|
|
switch {
|
|
case a.Fact != nil:
|
|
f := *a.Fact
|
|
if f.Source == "" {
|
|
f.Source = a.Source
|
|
}
|
|
w.write(ctx, f, ts)
|
|
w.logf("пришло: факт %s=%s (%s)", f.Key, f.Value, f.Source)
|
|
case a.Note != nil:
|
|
if _, err := w.api.WriteNote(ctx, ts, a.Note.Text, nil, a.Source); err != nil {
|
|
w.t.Fatalf("write note from %s: %v", a.Source, err)
|
|
}
|
|
w.logf("пришло: заметка от %s — %s", a.Source, truncateRunes(a.Note.Text, 60))
|
|
case a.Task != nil:
|
|
status := a.Task.Status
|
|
if status == "" {
|
|
status = store.TaskCandidate
|
|
}
|
|
if _, err := w.api.CaptureTask(ctx, ipc.CaptureTaskReq{
|
|
Text: a.Task.Text, Source: a.Source, Evidence: a.Task.Evidence, Status: status, Ts: ts,
|
|
}); err != nil {
|
|
w.t.Fatalf("capture task from %s: %v", a.Source, err)
|
|
}
|
|
w.logf("пришло: задача от %s — %s", a.Source, a.Task.Text)
|
|
default:
|
|
w.t.Fatalf("arrive step from %s carries nothing", a.Source)
|
|
}
|
|
}
|
|
|
|
// audioText resolves a scenario's WAV reference to the text the fixture is
|
|
// known to contain, by reading cmd/mavsttd's golden manifest (#288's format,
|
|
// reused rather than duplicated). An unknown reference fails the scenario
|
|
// rather than quietly transcribing to "".
|
|
//
|
|
// The manifest is read and parsed once per world, not once per step: a
|
|
// scenario with twenty audio steps should cost one file read.
|
|
func (w *simWorld) audioText(ref string) string {
|
|
w.t.Helper()
|
|
manifest := filepath.Join("..", "mavsttd", "testdata", "golden_v1.json")
|
|
if w.audio == nil {
|
|
raw, err := os.ReadFile(manifest)
|
|
if err != nil {
|
|
w.fatalf("audio step %q: reading %s: %v", ref, manifest, err)
|
|
return ""
|
|
}
|
|
var m struct {
|
|
Cases []struct {
|
|
Name string `json:"name"`
|
|
WAV string `json:"wav"`
|
|
Text string `json:"text"`
|
|
} `json:"cases"`
|
|
}
|
|
if err := json.Unmarshal(raw, &m); err != nil {
|
|
w.fatalf("audio step %q: parsing %s: %v", ref, manifest, err)
|
|
return ""
|
|
}
|
|
w.audio = make(map[string]string, len(m.Cases)*2)
|
|
for _, c := range m.Cases {
|
|
w.audio[c.Name] = c.Text
|
|
w.audio[c.WAV] = c.Text
|
|
}
|
|
}
|
|
if text, ok := w.audio[ref]; ok && ref != "" {
|
|
return text
|
|
}
|
|
w.fatalf("audio step %q: no such case in %s", ref, manifest)
|
|
return ""
|
|
}
|
|
|
|
func voicePTT() voice.PushToTalkReq {
|
|
return voice.PushToTalkReq{Audio: audio.Audio{Format: audio.PCM16kMono}}
|
|
}
|
|
|
|
// fakes — every ecosystem fake, in a fixed order. Both the mark and the paths
|
|
// walk this same order, which is the whole point: they have to agree.
|
|
func (w *simWorld) fakes() []*fakeServer { return []*fakeServer{w.praxis, w.nexus, w.hexis} }
|
|
|
|
// callMark takes a PER-SERVER snapshot of how many requests each fake has
|
|
// seen. It is not a total.
|
|
//
|
|
// A total cannot be used to slice the concatenated path list, and the harness
|
|
// used to do exactly that. callPaths concatenates praxis, then nexus, then
|
|
// hexis; a total counts arrivals across all three. With praxis on 3 requests
|
|
// and nexus on 1, the total is 4 and the list is [p1 p2 p3 n1]. A step that
|
|
// calls praxis once makes the list [p1 p2 p3 p4 n1], and paths[4:] is [n1].
|
|
// The new praxis call sits at index 3 and is never looked at, so
|
|
// expect_not_called on praxis passed on a step that called praxis. The same
|
|
// slice reported the stale nexus call as new, so expect_not_called on
|
|
// "/resolve" failed on a step that resolved nothing.
|
|
func (w *simWorld) callMark() []int {
|
|
mark := make([]int, len(w.fakes()))
|
|
for i, fs := range w.fakes() {
|
|
if fs != nil {
|
|
mark[i] = len(fs.Requests())
|
|
}
|
|
}
|
|
return mark
|
|
}
|
|
|
|
// callPathsSince returns the calls each fake took after its own mark. A nil
|
|
// mark means "everything, from the beginning of the run".
|
|
func (w *simWorld) callPathsSince(mark []int) []string {
|
|
var out []string
|
|
for i, fs := range w.fakes() {
|
|
if fs == nil {
|
|
continue
|
|
}
|
|
reqs := fs.Requests()
|
|
from := 0
|
|
if mark != nil && i < len(mark) {
|
|
from = mark[i]
|
|
}
|
|
if from > len(reqs) {
|
|
from = len(reqs)
|
|
}
|
|
for _, r := range reqs[from:] {
|
|
out = append(out, r.Method+" "+r.Path)
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
func (w *simWorld) callPaths() []string { return w.callPathsSince(nil) }
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Assertions
|
|
// ---------------------------------------------------------------------------
|
|
|
|
func (w *simWorld) assert(ctx context.Context, i int, s step, sendsBefore int, callsBefore []int, eventsBefore int) {
|
|
w.t.Helper()
|
|
where := fmt.Sprintf("step %d (%s)", i+1, s.At)
|
|
if s.Note != "" {
|
|
where += " " + s.Note
|
|
}
|
|
fail := func(format string, args ...any) {
|
|
w.dump()
|
|
w.t.Errorf("%s: %s", where, fmt.Sprintf(format, args...))
|
|
}
|
|
|
|
lastReply := ""
|
|
if len(w.replies) > 0 {
|
|
lastReply = w.replies[len(w.replies)-1]
|
|
}
|
|
for _, want := range s.ExpectReply {
|
|
if !containsFold(lastReply, want) {
|
|
fail("reply %q does not contain %q", lastReply, want)
|
|
}
|
|
}
|
|
for _, unwanted := range s.ExpectNotReply {
|
|
if containsFold(lastReply, unwanted) {
|
|
fail("reply %q contains %q and must not", lastReply, unwanted)
|
|
}
|
|
}
|
|
|
|
sent := w.sink.all()
|
|
for _, want := range s.ExpectSent {
|
|
if !anyContains(sendableTexts(sent), want) {
|
|
fail("nothing sent mentions %q; sent so far: %v", want, sendableTexts(sent))
|
|
}
|
|
}
|
|
// Scoped to this step on purpose: "nothing was sent BECAUSE OF THIS" is the
|
|
// question a not-a-nag constraint asks.
|
|
if s.ExpectNoSend && len(sent) > sendsBefore {
|
|
fail("expected nothing to be sent, got %v", sendableTexts(sent[sendsBefore:]))
|
|
}
|
|
|
|
paths := w.callPaths()
|
|
for _, want := range s.ExpectCalled {
|
|
if !anyContains(paths, want) {
|
|
fail("no ecosystem call matches %q; calls so far: %v", want, paths)
|
|
}
|
|
}
|
|
since := w.callPathsSince(callsBefore)
|
|
for _, unwanted := range s.ExpectNotCalled {
|
|
if anyContains(since, unwanted) {
|
|
fail("an ecosystem call matched %q and must not have: %v", unwanted, since)
|
|
}
|
|
}
|
|
|
|
evs := w.bus.Recent(0)
|
|
for _, want := range s.ExpectEvents {
|
|
if !anyContains(eventLines(evs), want) {
|
|
fail("no intake event matches %q; journal: %v", want, eventLines(evs))
|
|
}
|
|
}
|
|
// Counted publishes, not bus.Len(): the ring saturates at its capacity, so
|
|
// a long scenario that filled it made every later expect_no_events pass
|
|
// unconditionally.
|
|
if s.ExpectNoEvents && w.publishCount() > eventsBefore {
|
|
fail("expected nothing to arrive, %d event(s) were published",
|
|
w.publishCount()-eventsBefore)
|
|
}
|
|
if s.ExpectStore != nil {
|
|
w.assertStoreState(ctx, *s.ExpectStore, fail)
|
|
}
|
|
}
|
|
|
|
const simStateReadLimit = 10_000
|
|
|
|
func (w *simWorld) assertStoreState(ctx context.Context, want storeStateExpectation, fail func(string, ...any)) {
|
|
if want.Notes != nil {
|
|
rows, err := w.store.RecentNotes(ctx, simStateReadLimit)
|
|
if err != nil {
|
|
fail("read notes for store assertion: %v", err)
|
|
} else {
|
|
assertStateCount("notes", want.Notes.Count, len(rows), fail)
|
|
used := make([]bool, len(rows))
|
|
for _, expected := range want.Notes.Rows {
|
|
matched, matchErr := matchDistinct(rows, used, func(row store.Note) (bool, error) {
|
|
return w.noteStateMatches(row, expected)
|
|
})
|
|
if matchErr != nil {
|
|
fail("invalid note expectation %+v: %v", expected, matchErr)
|
|
} else if !matched {
|
|
fail("no distinct note matches %+v; notes: %+v", expected, rows)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
if want.Tasks != nil {
|
|
rows, err := w.store.ListTasks(ctx, "")
|
|
if err != nil {
|
|
fail("read tasks for store assertion: %v", err)
|
|
} else {
|
|
if want.Tasks.Count != nil && *want.Tasks.Count > store.MaxTaskRows {
|
|
fail("task count assertion %d exceeds the store read bound %d", *want.Tasks.Count, store.MaxTaskRows)
|
|
} else {
|
|
assertStateCount("tasks", want.Tasks.Count, len(rows), fail)
|
|
}
|
|
used := make([]bool, len(rows))
|
|
for _, expected := range want.Tasks.Rows {
|
|
matched, matchErr := matchDistinct(rows, used, func(row store.Task) (bool, error) {
|
|
return w.taskStateMatches(row, expected)
|
|
})
|
|
if matchErr != nil {
|
|
fail("invalid task expectation %+v: %v", expected, matchErr)
|
|
} else if !matched {
|
|
fail("no distinct task matches %+v; tasks: %+v", expected, rows)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
if want.Reminders != nil {
|
|
rows, err := w.store.ListReminders(ctx, simStateReadLimit)
|
|
if err != nil {
|
|
fail("read reminders for store assertion: %v", err)
|
|
} else {
|
|
assertStateCount("reminders", want.Reminders.Count, len(rows), fail)
|
|
used := make([]bool, len(rows))
|
|
for _, expected := range want.Reminders.Rows {
|
|
matched, matchErr := matchDistinct(rows, used, func(row store.Reminder) (bool, error) {
|
|
return w.reminderStateMatches(row, expected)
|
|
})
|
|
if matchErr != nil {
|
|
fail("invalid reminder expectation %+v: %v", expected, matchErr)
|
|
} else if !matched {
|
|
fail("no distinct reminder matches %+v; reminders: %+v", expected, rows)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
if want.Facts != nil {
|
|
rows, err := w.store.RecentFacts(ctx, simStateReadLimit)
|
|
if err != nil {
|
|
fail("read facts for store assertion: %v", err)
|
|
} else {
|
|
assertStateCount("facts", want.Facts.Count, len(rows), fail)
|
|
used := make([]bool, len(rows))
|
|
for _, expected := range want.Facts.Rows {
|
|
matched, matchErr := matchDistinct(rows, used, func(row store.Fact) (bool, error) {
|
|
return w.factStateMatches(row, expected)
|
|
})
|
|
if matchErr != nil {
|
|
fail("invalid fact expectation %+v: %v", expected, matchErr)
|
|
} else if !matched {
|
|
fail("no distinct fact matches %+v; facts: %+v", expected, rows)
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
func assertStateCount(kind string, want *int, got int, fail func(string, ...any)) {
|
|
if want != nil && got != *want {
|
|
fail("%s count = %d, want %d", kind, got, *want)
|
|
}
|
|
}
|
|
|
|
// matchDistinct prevents two expectations from being satisfied by the same
|
|
// durable row. This is important for identity assertions where two records may
|
|
// intentionally carry the same text but have different lifecycle states.
|
|
func matchDistinct[T any](rows []T, used []bool, matches func(T) (bool, error)) (bool, error) {
|
|
for i, row := range rows {
|
|
if used[i] {
|
|
continue
|
|
}
|
|
ok, err := matches(row)
|
|
if err != nil {
|
|
return false, err
|
|
}
|
|
if ok {
|
|
used[i] = true
|
|
return true, nil
|
|
}
|
|
}
|
|
return false, nil
|
|
}
|
|
|
|
func (w *simWorld) noteStateMatches(got store.Note, want noteRowExpectation) (bool, error) {
|
|
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text != want.Text ||
|
|
want.Source != "" && got.Source != want.Source {
|
|
return false, nil
|
|
}
|
|
return w.stateTimeMatches(got.Ts, want.At)
|
|
}
|
|
|
|
func (w *simWorld) taskStateMatches(got store.Task, want taskRowExpectation) (bool, error) {
|
|
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text != want.Text ||
|
|
want.Source != "" && got.Source != want.Source || want.Status != "" && got.Status != want.Status ||
|
|
want.ResolvedBy != "" && got.ResolvedBy != want.ResolvedBy {
|
|
return false, nil
|
|
}
|
|
if ok, err := w.stateTimeMatches(got.CreatedTs, want.CreatedAt); err != nil || !ok {
|
|
return ok, err
|
|
}
|
|
if want.ResolvedAt == "" {
|
|
return true, nil
|
|
}
|
|
if got.ResolvedTs == nil {
|
|
return false, nil
|
|
}
|
|
return w.stateTimeMatches(*got.ResolvedTs, want.ResolvedAt)
|
|
}
|
|
|
|
func (w *simWorld) reminderStateMatches(got store.Reminder, want reminderRowExpectation) (bool, error) {
|
|
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text() != want.Text ||
|
|
want.Status != "" && got.Status != want.Status {
|
|
return false, nil
|
|
}
|
|
return w.stateTimeMatches(got.FireTs, want.FireAt)
|
|
}
|
|
|
|
func (w *simWorld) factStateMatches(got store.Fact, want factRowExpectation) (bool, error) {
|
|
if want.ID != 0 && got.ID != want.ID || want.Kind != "" && string(got.Kind) != want.Kind ||
|
|
want.Key != "" && got.Key != want.Key || want.Value != "" && got.Value != want.Value ||
|
|
want.Source != "" && got.Source != want.Source ||
|
|
want.Confidence != nil && got.Confidence != *want.Confidence {
|
|
return false, nil
|
|
}
|
|
return w.stateTimeMatches(got.Ts, want.At)
|
|
}
|
|
|
|
func (w *simWorld) stateTimeMatches(got time.Time, raw string) (bool, error) {
|
|
if raw == "" {
|
|
return true, nil
|
|
}
|
|
want, err := w.stateTime(raw)
|
|
if err != nil {
|
|
return false, err
|
|
}
|
|
return got.Equal(want), nil
|
|
}
|
|
|
|
// stateTime accepts either an absolute RFC3339 instant or the same local
|
|
// HH:MM[:SS] shape scenario steps use. The latter keeps fixtures readable
|
|
// while still comparing exact instants after the store normalises to UTC.
|
|
func (w *simWorld) stateTime(raw string) (time.Time, error) {
|
|
if strings.Contains(raw, "T") {
|
|
return time.Parse(time.RFC3339, raw)
|
|
}
|
|
layout := "15:04"
|
|
if strings.Count(raw, ":") == 2 {
|
|
layout = "15:04:05"
|
|
}
|
|
hm, err := time.Parse(layout, raw)
|
|
if err != nil {
|
|
return time.Time{}, fmt.Errorf("expected HH:MM[:SS] or RFC3339, got %q: %w", raw, err)
|
|
}
|
|
return time.Date(w.start.Year(), w.start.Month(), w.start.Day(),
|
|
hm.Hour(), hm.Minute(), hm.Second(), 0, w.loc), nil
|
|
}
|
|
|
|
func sendableTexts(sends []delivery.Sendable) []string {
|
|
out := make([]string, 0, len(sends))
|
|
for _, s := range sends {
|
|
out = append(out, fmt.Sprintf("[%s] %s", s.RuleName, s.Body))
|
|
}
|
|
return out
|
|
}
|
|
|
|
func eventLines(evs []event.Event) []string {
|
|
out := make([]string, 0, len(evs))
|
|
for _, e := range evs {
|
|
// Priority is in the line so a scenario can assert on it. It is the one
|
|
// field factPriority derives from confidence, and without it a replay
|
|
// could set a confidence but never see what the journal did with it.
|
|
out = append(out, fmt.Sprintf("%s/%s pri=%s %s %s", e.Source, e.Kind, e.Priority, e.Title, e.Body))
|
|
}
|
|
return out
|
|
}
|
|
|
|
func containsFold(hay, needle string) bool {
|
|
return strings.Contains(strings.ToLower(hay), strings.ToLower(needle))
|
|
}
|
|
|
|
func anyContains(hay []string, needle string) bool {
|
|
for _, h := range hay {
|
|
if containsFold(h, needle) {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// The test
|
|
// ---------------------------------------------------------------------------
|
|
|
|
const scenarioDir = "testdata/scenarios"
|
|
|
|
// TestSimulatorScenarios replays every scenario file. Adding a scenario is
|
|
// adding a JSON file — no Go change, which is the property that makes this
|
|
// cheap enough to actually use.
|
|
func TestSimulatorScenarios(t *testing.T) {
|
|
entries, err := os.ReadDir(scenarioDir)
|
|
if err != nil {
|
|
t.Fatalf("reading %s: %v", scenarioDir, err)
|
|
}
|
|
var ran int
|
|
for _, ent := range entries {
|
|
if ent.IsDir() || !strings.HasSuffix(ent.Name(), ".json") {
|
|
continue
|
|
}
|
|
ran++
|
|
name := strings.TrimSuffix(ent.Name(), ".json")
|
|
t.Run(name, func(t *testing.T) {
|
|
sc := loadScenario(t, filepath.Join(scenarioDir, ent.Name()))
|
|
w := newSimWorld(t, sc)
|
|
w.run(sc)
|
|
if testing.Verbose() {
|
|
w.dump()
|
|
}
|
|
})
|
|
}
|
|
if ran == 0 {
|
|
t.Fatalf("no scenarios in %s — the harness would pass vacuously", scenarioDir)
|
|
}
|
|
}
|
|
|
|
// TestSimulatorWorldMirrorsProductionConversationSeams pins the two daemon
|
|
// constructor upgrades the continuous scenario needs. A bare store API cannot
|
|
// answer DayPlan, and a nil handler extractor cannot complete a parked reminder
|
|
// from the next turn; either drift would make the simulator exercise a smaller
|
|
// system than production while still producing plausible replies.
|
|
func TestSimulatorWorldMirrorsProductionConversationSeams(t *testing.T) {
|
|
sc := scenario{SchemaVersion: 1, Name: "constructor-seams", Start: "2026-08-15T08:00:00+04:00"}
|
|
w := newSimWorld(t, sc)
|
|
if w.handler.api != w.api {
|
|
t.Fatal("handler did not receive the simulator's upgraded daemon API")
|
|
}
|
|
plan, err := w.handler.api.DayPlan(context.Background())
|
|
if err != nil {
|
|
t.Fatalf("simulator day-plan seam is unavailable: %v", err)
|
|
}
|
|
planY, planM, planD := plan.Date.In(w.loc).Date()
|
|
wantY, wantM, wantD := w.start.Date()
|
|
if plan.Date.IsZero() || planY != wantY || planM != wantM || planD != wantD {
|
|
t.Fatalf("day plan date = %v, want the fake-clock day %v", plan.Date, w.start)
|
|
}
|
|
if w.handler.extractor.Time == nil || w.handler.timeParser == nil {
|
|
t.Fatal("simulator left the clarify time parser unwired")
|
|
}
|
|
slots := w.handler.extractor.Extract(context.Background(), router.IntentReminder,
|
|
"сегодня в 10:00", w.clock.Now())
|
|
if !slots.HasTime || !slots.Time.Equal(w.timeOf("10:00")) {
|
|
t.Fatalf("clarify extractor parsed %+v, want the fake-clock day at 10:00", slots)
|
|
}
|
|
}
|
|
|
|
// The reported duplicate transcript was a diagnostic artefact: two adjacent
|
|
// sed ranges both included boundary line 620. Source had one log call. Keep an
|
|
// executable exact-count assertion so a real duplicate cannot be introduced
|
|
// later and mistaken for another display artefact.
|
|
func TestSimulatorTranscriptRecordsEachSpokenTurnOnce(t *testing.T) {
|
|
sc := scenario{
|
|
SchemaVersion: 1,
|
|
Name: "transcript-count",
|
|
Start: "2026-08-15T08:00:00+04:00",
|
|
Script: []scriptEntry{{
|
|
Match: "привет", Route: `[{"intent":"chat","text":"привет"}]`,
|
|
Reply: `{"response":"Привет.","mood":"happy"}`,
|
|
}},
|
|
}
|
|
w := newSimWorld(t, sc)
|
|
w.stimulate(context.Background(), step{Say: "привет"})
|
|
want := "08:00:00 он: привет"
|
|
count := 0
|
|
for _, line := range w.transcript {
|
|
if line == want {
|
|
count++
|
|
}
|
|
}
|
|
if count != 1 {
|
|
t.Fatalf("owner transcript line occurred %d times, want exactly once: %v", count, w.transcript)
|
|
}
|
|
}
|
|
|
|
func loadScenario(t *testing.T, path string) scenario {
|
|
t.Helper()
|
|
raw, err := os.ReadFile(path)
|
|
if err != nil {
|
|
t.Fatalf("reading %s: %v", path, err)
|
|
}
|
|
var sc scenario
|
|
dec := json.NewDecoder(strings.NewReader(string(raw)))
|
|
dec.DisallowUnknownFields() // a typo'd assertion key must fail, not be ignored
|
|
if err := dec.Decode(&sc); err != nil {
|
|
t.Fatalf("parsing %s: %v", path, err)
|
|
}
|
|
if sc.SchemaVersion != 1 {
|
|
t.Fatalf("%s: schema_version = %d, want 1", path, sc.SchemaVersion)
|
|
}
|
|
if sc.Name == "" || sc.Start == "" || len(sc.Steps) == 0 {
|
|
t.Fatalf("%s: a scenario needs a name, a start and at least one step", path)
|
|
}
|
|
return sc
|
|
}
|
|
|
|
// TestSimulatorIsDeterministic replays one scenario twice and requires an
|
|
// identical transcript. This is the property the whole task rests on: if a
|
|
// time.Now() creeps into a replayed path, two runs diverge and this fails.
|
|
func TestSimulatorIsDeterministic(t *testing.T) {
|
|
path := filepath.Join(scenarioDir, "morning_missed.json")
|
|
sc := loadScenario(t, path)
|
|
|
|
transcriptOf := func() string {
|
|
w := newSimWorld(t, sc)
|
|
w.run(sc)
|
|
return strings.Join(w.transcript, "\n")
|
|
}
|
|
first := transcriptOf()
|
|
second := transcriptOf()
|
|
if first != second {
|
|
t.Errorf("two replays of the same scenario diverged:\n--- first ---\n%s\n--- second ---\n%s", first, second)
|
|
}
|
|
// And every transcript timestamp must lie inside the scenario's own span.
|
|
//
|
|
// This used to compare the transcript against time.Now().Format("15:04"),
|
|
// which failed whenever the suite happened to run during the half hour the
|
|
// scenario covers: morning_missed logs 08:30 through 09:00, sc.Start
|
|
// contains only 08:30, so a run at 08:35 reported a wall-clock read that
|
|
// had not happened. A determinism test that depends on the time of day is
|
|
// the bug it is looking for.
|
|
start, err := time.Parse(time.RFC3339, sc.Start)
|
|
if err != nil {
|
|
t.Fatalf("bad start: %v", err)
|
|
}
|
|
last := start
|
|
for _, s := range sc.Steps {
|
|
if at := stepInstant(t, start, s.At); at.After(last) {
|
|
last = at
|
|
}
|
|
}
|
|
// Only the lines logf stamped. A note or a feed item can carry its own
|
|
// newlines, and those continuation lines have no timestamp.
|
|
stamp := regexp.MustCompile(`^(\d\d:\d\d:\d\d) `)
|
|
for _, line := range strings.Split(first, "\n") {
|
|
m := stamp.FindStringSubmatch(line)
|
|
if m == nil {
|
|
continue
|
|
}
|
|
at := stepInstant(t, start, m[1])
|
|
if at.Before(start) || at.After(last) {
|
|
t.Errorf("transcript line %q is stamped outside the scenario span %s..%s — "+
|
|
"something in the replay path read time.Now()",
|
|
line, start.Format("15:04:05"), last.Format("15:04:05"))
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestCallsSinceAreScopedPerServer pins the ordering bug that made
|
|
// expect_not_called unsound. The mark is per server; a total cannot slice a
|
|
// list that is concatenated per server.
|
|
func TestCallsSinceAreScopedPerServer(t *testing.T) {
|
|
sc := scenario{SchemaVersion: 1, Name: "x", Start: "2026-08-01T08:30:00+03:00",
|
|
Praxis: fixturePraxisAttentionItems(), Nexus: fixtureNexusResolved("ent_1", "thing", "device"),
|
|
Steps: []step{{At: "08:30"}}}
|
|
w := newSimWorld(t, sc)
|
|
|
|
hit := func(fs *fakeServer, path string) {
|
|
t.Helper()
|
|
resp, err := http.Get(fs.URL + path)
|
|
if err != nil {
|
|
t.Fatalf("hitting %s: %v", path, err)
|
|
}
|
|
resp.Body.Close()
|
|
}
|
|
// Praxis runs ahead of nexus, so the concatenated list already has a nexus
|
|
// call sitting after three praxis ones.
|
|
hit(w.praxis, "/api/v1/tools/attention")
|
|
hit(w.praxis, "/api/v1/tools/attention")
|
|
hit(w.praxis, "/api/v1/tools/attention")
|
|
hit(w.nexus, "/api/v1/resolve")
|
|
|
|
mark := w.callMark()
|
|
hit(w.praxis, "/api/v1/tools/attention")
|
|
|
|
since := w.callPathsSince(mark)
|
|
if !anyContains(since, "attention") {
|
|
t.Errorf("the praxis call made after the mark is missing from %v", since)
|
|
}
|
|
if anyContains(since, "resolve") {
|
|
t.Errorf("a nexus call from before the mark was reported as new: %v", since)
|
|
}
|
|
}
|
|
|
|
// TestPublishCountDoesNotSaturateWithTheRing pins expect_no_events on a bus
|
|
// that has already wrapped. bus.Len() stops at the capacity, so it can no
|
|
// longer answer "did anything arrive".
|
|
func TestPublishCountDoesNotSaturateWithTheRing(t *testing.T) {
|
|
sc := scenario{SchemaVersion: 1, Name: "x", Start: "2026-08-01T08:30:00+03:00",
|
|
Steps: []step{{At: "08:30"}}}
|
|
w := newSimWorld(t, sc)
|
|
|
|
for i := 0; i < event.DefaultCapacity+5; i++ {
|
|
w.bus.Publish(event.Event{
|
|
Source: "sim:test", Kind: event.KindFact, Title: fmt.Sprintf("f%d", i),
|
|
}, w.clock.Now())
|
|
}
|
|
if got := w.bus.Len(); got != event.DefaultCapacity {
|
|
t.Fatalf("ring holds %d, expected it to be saturated at %d", got, event.DefaultCapacity)
|
|
}
|
|
before := w.publishCount()
|
|
w.bus.Publish(event.Event{Source: "sim:test", Kind: event.KindFact, Title: "one more"}, w.clock.Now())
|
|
if w.publishCount() != before+1 {
|
|
t.Errorf("publish count went %d → %d on a full ring, expected it to keep counting",
|
|
before, w.publishCount())
|
|
}
|
|
}
|
|
|
|
// stepInstant resolves "HH:MM" or "HH:MM:SS" against the scenario's start day.
|
|
func stepInstant(t *testing.T, start time.Time, at string) time.Time {
|
|
t.Helper()
|
|
layout := "15:04"
|
|
if strings.Count(at, ":") == 2 {
|
|
layout = "15:04:05"
|
|
}
|
|
hm, err := time.Parse(layout, at)
|
|
if err != nil {
|
|
t.Fatalf("bad step time %q: %v", at, err)
|
|
}
|
|
return time.Date(start.Year(), start.Month(), start.Day(),
|
|
hm.Hour(), hm.Minute(), hm.Second(), 0, start.Location())
|
|
}
|
|
|
|
// TestSimulatorRefusesBackwardsSteps guards the one scenario-authoring mistake
|
|
// that would silently produce a meaningless run.
|
|
//
|
|
// It used to advance to 09:00 and then to 09:30 and assert the clock had
|
|
// moved, which is the forwards case: the backwards branch it is named after
|
|
// was never reached, because reaching it ended the test. The fatalf seam is
|
|
// what makes it testable.
|
|
func TestSimulatorRefusesBackwardsSteps(t *testing.T) {
|
|
sc := scenario{SchemaVersion: 1, Name: "x", Start: "2026-08-01T08:30:00+03:00",
|
|
Steps: []step{{At: "09:00"}}}
|
|
w := newSimWorld(t, sc)
|
|
var refusal string
|
|
w.fatalf = func(format string, args ...any) { refusal = fmt.Sprintf(format, args...) }
|
|
|
|
w.advanceTo("09:00")
|
|
if got := w.clock.Now().In(w.loc).Format("15:04"); got != "09:00" {
|
|
t.Fatalf("clock at %s after advancing to 09:00", got)
|
|
}
|
|
if refusal != "" {
|
|
t.Fatalf("a forwards step was refused: %s", refusal)
|
|
}
|
|
|
|
w.advanceTo("08:45")
|
|
if refusal == "" {
|
|
t.Fatal("a step going backwards to 08:45 was accepted")
|
|
}
|
|
if got := w.clock.Now().In(w.loc).Format("15:04"); got != "09:00" {
|
|
t.Errorf("the clock moved to %s on a refused step, it must stay at 09:00", got)
|
|
}
|
|
}
|
|
|
|
// TestSimulatorRoutesWithTheDeployedSeeds — the scenarios must replay against
|
|
// the classifier the deploy runs, not an empty one.
|
|
//
|
|
// They did not. The seed path was relative to the working directory, which is
|
|
// cmd/mavend under `go test`, so every file failed to open and the whole
|
|
// simulator scored three green scenarios with zero examples loaded (Vikunja
|
|
// #465). The count is asserted rather than logged, because a silent zero is
|
|
// exactly the failure that hid here for as long as it did.
|
|
func TestSimulatorRoutesWithTheDeployedSeeds(t *testing.T) {
|
|
cls := router.NewClassifier(router.NewHashEmbedder(1024))
|
|
seedClassifier(cls)
|
|
total := 0
|
|
for _, intent := range cls.Intents() {
|
|
total += len(cls.Examples(intent))
|
|
}
|
|
if total == 0 {
|
|
t.Fatalf("no seed examples loaded from %s — the simulator would route on nothing", seedPath())
|
|
}
|
|
if len(cls.Intents()) != 7 {
|
|
t.Fatalf("seeded %d intents, want all 7", len(cls.Intents()))
|
|
}
|
|
}
|