"я выпил воды" came back as "Проверила, что ты выпел стакан воды". The
verb is not a Russian word, the glass was never mentioned, and nothing
had been checked.
The store was right throughout: DefaultFactParser files this as
key=water value="drank", and no row anywhere held "стакан". Every
Russian word in that sentence was generated. replyContext hands the
model "записала факт: water \"drank\"", so the model had nothing to
phrase FROM and reached for the nearest plausible sentence — the example
in ReplySystemPrompt, which was literally "Записала, что ты выпил стакан
воды."
So the fact path stops generating, the way the note payload did in
V-576. The confirmation is a fixed deck frame with his own sentence in
it, in both repliers, and the prompt example is contentless now. The
stub also read the parser's KEY back at him, which is machine
vocabulary he never said.
The clarify half of this — a fact clarified out of "запиши" answers with
"запиши" and nothing else — lands with V-593, which touches the same
lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Item 4. actionChat calls h.phraser.PhraseChat, and LLMPhraser posts raw
HTTP to /v1/chat/completions rather than going through the llm client
scriptedLLM stands in for. The simulator wired phraser.NewStub() anyway,
so no scenario could assert what she says on a chat turn: every reply came
back as a pick from fallbacks_ru_v1.json, four variants deep, and the same
scenario returned "тут я пас." one run and "не знаю, честно." the next.
scriptedPhraser embeds the Stub and overrides PhraseChat only, reading the
same script entries the router reads. A reply is accepted in either shape
the phrasing contract allows, the {"response","mood"} object or plain text,
so a scenario writes one thing for both paths.
An unscripted chat turn returns an error rather than a fallback, matching
scriptedLLM: actionChat logs it and uses ChatFallback(), so scenarios that
never meant to assert a chat reply behave as before.
conversation_anaphora turn 4 now pins its text — the reply that asks which
device he means, which is the recorded defect in the box's own words.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five Russian turns, one monitor, four questions that say "он" and never
name it again. Item 3 of the task: the shape had nowhere to fail, because
the routing fixture scores one utterance at a time and a conversation that
breaks on turn 2 cannot lose a point there.
Routes are scripted exactly as the box produced them on 05-08-2026. Turn 1
files a fact despite "давай поболтаем", the questions go to query, turn 4
goes to chat, and none of the five replies names the monitor. Four steps
assert the reply LACKS "монитор" and are marked WRONG in their notes with
what each must become.
The absence assertion is forced, not chosen. The simulator wires
phraser.NewStub(), and PhraseChat posts raw HTTP to /v1/chat/completions
rather than through the llm client the harness scripts, so a chat reply
cannot be scripted at all. The wrong replies come from
fallbacks_ru_v1.json, which picks between four variants per turn, so
asserting a string would pin the picker. Missing referent holds whichever
variant she reaches for.
Items 1 and 2 stay open: they are owner decisions about which store a
referent comes from and whether "давай поболтаем" claims a turn.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
turnIsAbout scored t.vec, and t.vec was set in one place: queryEmbed, the
source at actions_query.go:125. Every topic source sits above it — attention,
list, feeds, home, network, weather. So best was handed an empty slice on every
deployed turn, returned ok=false, and all six recognisers ran on their keyword
floors. The seeds have decided nothing outside the tests since the mechanism
landed.
TestONNXTopics passes because it embeds each utterance itself and calls best
directly. That is the shape that hid this for a month: it measures the scorer
and never the wiring. Found on the box instead — "что мне нужно купить" was
answered from an old note about a monitor, and the seeds place it as the list by
0.0841.
turnVector computes the vector on first ask and caches it on the turn;
queryEmbed returns early when it is already set. Chosen over moving the embed
source up the list, because the cost is then paid only by turns that ask a
topic source, and the order of querySources keeps meaning what its comments
argue for.
One scenario assertion moved, and it is a behaviour change rather than a bent
test. morning_missed step 5 pinned "не знаю" for "что я пропустил?" with an
unresolved Praxis item on the board. isAttentionQuery does not match that
phrasing and the topicAttend seeds carry "что важное я пропустил" almost
verbatim, so she reads the item back now. Reading a surfaced item aloud is not
inventing a morning summary, so the floor that step exists for still holds;
what moved is which source answers.
expect_not_called could pass on a step that made the forbidden call. callPaths
concatenates per server and callCount was a total, so slicing the concatenated
list by the total examined the wrong window. With praxis on three requests and
nexus on one, a fourth praxis call landed at index three and paths[4:] never
saw it, while the stale nexus call was reported as new. The mark is now
per server and the paths are taken per server from it.
Neither scenario ever produced an act, so all three fakes saw zero requests and
the fault lever changed no outcome. The two headline capabilities of the
harness had no coverage. act_degraded scripts an act against an enabled
allowlist row and runs it healthy, at 503 and healthy again, asserting the
reply, the call, the absence of a call on a tick, and that nothing was pushed
at him either way. That needed two seams the world did not have: allowlist rows
from the scenario, and a matcher on the real store rather than a nil API, which
would have panicked the moment any scenario produced an act.
expect_no_events compared bus.Len(), which stops growing at the ring capacity,
so a scenario long enough to fill the ring made every later expect_no_events
pass unconditionally. It counts publishes through a subscriber now.
A scenario could not express a fact below full confidence, because write
hardcoded 1.0, and morning_missed annotated its ambient step as if it could.
factPriority branches on exactly that, so no replay could reach the low branch.
signalStep takes a confidence, the ambient step sets the 0.6 the ambient path
writes, and the event line carries the priority so a scenario can assert it.
TestSimulatorIsDeterministic compared the transcript against time.Now, which
fails for the half hour a day the scenario itself covers. It checks that every
stamped line falls inside the scenario span instead. TestSimulatorRefusesBackwardsSteps
tested the forwards case, because reaching the backwards branch ended the test.
A fatalf seam makes the refusal observable.
Smaller notes: the step doc comment now states which assertions are run scoped
and which are step scoped, audioText parses the golden manifest once per world
rather than once per step, and the feminine checks list the masculine form with
its following character, since the earlier check on a comma alone passed on
"записал что ты выпил воды".
Found in review of #79.
A scenario is a JSON file under cmd/mavend/testdata/scenarios: a start
instant, a script of canned model answers, and a list of steps at "HH:MM".
Each step does one thing — say, audio, signal, arrive, tick, fault — and
then asserts on what she said, what was sent, which ecosystem services were
called, and what landed in the intake journal.
Between those boundaries the real components run: the real router cascade
(stage0, the LLM router over a scripted completer, the classifier
underneath it), the real store, the real reactive handler, the real tick
loop, and the same intake-decorated ipc.CoreAPI the daemon wires. What is
faked is only what a test cannot have: the model, the microphone, the
speaker, the delivery sink, and the ecosystem HTTP services.
Time is a single fakeClock threaded into every reader — the handler, the
intake publish stamp and tick(ctx, now) — so there is no time.Now() on the
replay path and a scenario is reproducible. TestSimulatorIsDeterministic
enforces that by replaying twice and diffing the transcripts byte for byte;
advanceTo refuses a step that goes backwards.
Two scenarios ship. morning_missed replays #284's own description: he
appears at the desk, a feed item, a mail candidate and a relayed
notification arrive through the morning, two ticks pass, and the assertions
are as much about nothing being sent at him unprompted as about what she
said. evening_degraded picks up the tier-2 pipeline case #288 deferred
here — a golden WAV through the STT seam to a written fact — and then puts
the ecosystem into 503 and checks that the proactive loop stays quiet and
that intake keeps working without it.
This is test-only code. Nothing in the production binaries changed, so the
daemon behaves identically when no scenario is running.
`make simulate` runs them verbose so the transcript is readable; `make
test` runs them with everything else.
Vikunja #284
The review's verification plan needs an instrument to settle whether the
classifier or the LLM router handles RU queries better, but that comparison
only means something if the daemon's own safety invariants are pinned
independently first.
These scenarios are deliberately narrow. They consume already-normalized
router decisions and assert what the post-router daemon owns: that a decision
requiring confirmation cannot execute before it is confirmed, that an
unresolved entity is never guessed at, and that named capabilities stay
unexecuted. Model routing quality is a separate question, evaluated against a
held-out contract fixture — mixing the two would produce a suite that fails for
two unrelated reasons.
The fixture is versioned (schema_version) so scenarios can be added without
rewriting the loader.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
claude
kami