embedder.heads_path is empty by default and deploy/mavend.json sets
it. A missing or broken weights file logs and leaves the heads nil,
because refusing to start over a routing accelerator would trade a
working box for a better one.
TestONNXRoutingHeads is the same cascade TestONNXBaseline scores with
one arm added, so the two are directly comparable. It also checks the
Go tokenizer against the Python one, since the heads were trained
through transformers and are read through a hand-written tokenizer: a
mismatch shows up here as a score below what Python measured on the
same weights, and nowhere else. That is how the reversed word pieces
were found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
Twenty-eight existing query cases get a want_source and five new ones
arrive with theirs. Every label is the destination that SHOULD claim the
turn, which on the five new cases is not the one that did: they were
observed failing on the box on 2026-08-07, so the fixture fails on the day
it is written.
Seven cases assert the SourceUnknown floor, and six of those are homelab
operations. They cluster because SourceRecall, SourceNetwork and
SourceAttention overlap on every question about the box: mavpoll writes its
netdata and uptime-kuma observations into the fact store recall reads.
Naming one destination there takes the other two off a turn that needs
them. That is a finding about the enum, not a gap in the labelling.
The fixture's grammar mirror had drifted. WorldQueryGrammars went into
buildRouter with V-655 and never into baselineGrammars, so the fixture was
scoring a grammar set the daemon does not run — the exact thing the comment
above that function forbids. Adding it moved the destination number 9/33 to
12/33 and moved nothing else.
Measured classifier+onnx: intent 73/96 (76.0%), was 69/91 (75.8%). Four of
the five new cases pass and no existing case moved. Destination 12/33
(36.4%), and the split is the point. World is 5/5, because a stage 0 rule
names it. Calendar is 2/6, because the possessive agenda rules deliberately
do not. Recall is 0/15, because nothing anywhere names it yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
The fixture measured the first half of a route and stopped. V-655 split a
routing decision in two, and the second half arrived with no fixture, so
Decision.Source had no accuracy number at all.
want_source is a pointer because the destination has three states and a
bare string has two. Absent is every intent but query, which never reaches
queryWalk. Present and empty is the SourceUnknown contract: name nothing
and let the daemon walk the chain, which is right whenever two destinations
can both answer and the utterance does not choose. Present and named is a
destination the route must produce.
A destination miss does not fail the case. It goes in SourceReason, never
in Reasons, so Accuracy and IntentAccuracy stay the numbers they were and
69/91 still means what it meant. SourceAccuracy is the second number, over
the labelled cases only, because a percentage of the whole fixture would be
a percentage of turns that never ask a query source.
A clarified or mis-routed case still counts in the denominator. It named no
destination and that is a miss, not a case to skip, or the denominator drops
every turn the route already lost.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
Two reporting tests over the 91-case RU fixture, no ratchet: a ratchet here
would freeze a number nobody has decided to hold.
TestStage0Contention runs the 21 grammars one at a time instead of stopping
at the first match. One case of 91 draws two, ru-query-019, where
calendar-query beats agenda-query by list position alone.
TestONNXClaimConfidenceDistribution buckets the reported confidence by the
layer that produced it. Stage 0 is 20/20 at a hardcoded 1.0. The classifier
scores 62% below its median and 62% above, across a cosine range of 0.859
to 0.942, with a top-two margin of p50 0.009. The float is not a confidence.
newBaselineClassifier and baselineGrammars split out of newBaselineRouter so
the measurement runs the same rules the daemon runs. TestONNXBaseline is
unchanged at 64/91.
V-405 measured reach with the classifier only, and the LLM router is the
deployed default, so 16/30 was the floor rather than the shipped behaviour.
TestReachWithLLMRouter scores the same 30 cases with the model, gated on
MAVEN_LLM_URL like TestLLMRouterBaseline.
The open question was whether the model writes a literal Praxis capability
into the fn slot and reaches a service the classifier structurally cannot. It
does not. Praxis is 0/12 with the model alone, exactly what the classifier
alone scores, and all twelve fail the same way: local, empty fn. Nothing in the
router prompt names a Praxis capability, so there is no string for it to write.
So V-516's stage-0 grammars are the only path to Praxis, not a determinism
argument. Through the cascade the model scores 28/30 with praxis 11/12, one
point above the classifier baseline. Hexis is 10/10 either way.
Overreach is 1 in both configurations, under the 4 the harness asserts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
Step 4 of the board build (docs/plans/15-board-surface.md). Naming a task
instead of its position reached nothing: "закрой задачу купить молоко" routed
act, found no allowlisted fn, and the gate asked "Что сделать?". The position
path already worked through resolveCandidate, but only in the two turns after
she read the list out.
TaskStatusGrammar is the same shape TaskCaptureGrammar uses — matches broadly,
decides in Build, no eighth intent — and fills the fn slot with task_status,
which is neither a Hexis capability nor a Praxis one. Three conditions, all
required: the board noun, so no ordinary sentence claims a turn; exactly one
status class, since "готово, убери" names two and asking beats picking; and a
status word matched as an imperative exactly or a stative by lemma. So a bare
"готово" and a bare "закрой" are not this rule's, and the second belongs to
Praxis, which claims it already.
Two lexicon sets rather than one with a value. The store records which of the
two transitions happened and /tasks shows it: work he chose to stop is not work
he did.
Measured on the fixture, two new cases (ru-act-020, ru-act-021). Classifier +
ONNX 62/89 (69.7%) → 64/91 (70.3%); cascade+llm 67/89 (75.3%) → 69/91 (75.8%,
80.2% intent-only) at p50 1.225s. Both new cases claimed at stage 0, no case
regressed, clarify counts unchanged at 3 false / 1 missed.
The task's own warning stands: every such grammar runs its parser ahead of the
resident model on every turn, so this is the last one that is free.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
The subjectless-reminder gate has been dead since V-383. It tested
`d.Slots.Text == ""`, and that slot is never empty: fillSlots hands it the
utterance when the model names nothing narrower. Measured on the box on
05-08-2026 — "напомни" alone routed to IntentReminder with Text:напомни,
reached actionReminder, and answered "не получилось разобрать время
напоминания." A parse error for a request he never finished asking about.
"ну напомни же" did the same.
The test is now what the slot CONTAINS. reminderHasSubject discounts the
reminder verb by lemma and the filler particles, and asks whether anything
is left. A day or an hour counts as a subject, which is why this does not
reuse cmd/mavend/reminderbody.go — that one strips the time words too.
filler_particles is the lexicon's 16th set. Not a stopword list: every word
in it is one that cannot BE a reminder's subject.
Measured against the 87-case fixture with and without the change: 65/87
both ways, identical clarify counts, because no case exercised the shape.
So amb-007 "напомни" and amb-008 "ну напомни же" were added, both
want_clarify. At 89 cases the cascade scores 67/89 (75.3% full, 79.8%
intent-only), 3 false clarifies / 1 missed, p50 1.199s — the two new cases
clarify, and nothing else moved. The classifier path still guesses both
(62/89, 8 missed clarify); the gate is on the LLM arm only.
The box needs a rebuild for this to take effect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SoL7EBdYC5Mhz3DJd49GJy
Russian names a half hour by the hour it is entering, in the genitive, so
"половина восьмого" is 07:30 and never 08:30. Neither date parser read that
shape, so the reminder parsed to nothing.
rewriteHalfPast runs in front of the token pass in SpellOutDigits, so the
python parser and the stub both see "в 7:30". It also reads the contracted
"полвосьмого" and the quarter-to shape "без четверти восемь", which counts
from a cardinal and is 07:45. Minus one is in one place, clockHourBefore, with
twelve rather than zero before one.
Ordinals eleven and twelve added to the lexicon, because a clock reaches them.
Minutes a spoken clock does not use are left alone: a guess here is a missed
dose.
Classifier + onnx over the routing fixture 58/82 to 62/87, three new cases,
none regressed. Python dateparser is not installed on this host, so only the
stub was measured. See docs/evals/2026-08-05-half-past-hours.md.
Praxis reach was 0/12 on the held-out fixture and structurally so.
handlePraxisAct dispatches on exact equality between Slots.Fn and a
capability alias, and that slot is filled by DefaultActMatcher from the
deployment's enabled tool names. No Praxis alias is on that list, so no
utterance could ever put one there. The Russian aliases in
praxisCapabilities read as if they matched speech. They are compared
against a fn slot and never against an utterance.
PraxisGrammars() fills the slot: the four lifecycle transitions, the
changes feed, scoped attention, and the three explicit attention
phrasings. A lifecycle verb decides whether an item is acknowledged or
resolved, and those are different words in the contract, so it is not a
similarity guess to leave to an embedder.
Two rules keep the lifecycle arm off ordinary speech. A stative word
("готово", "принято") needs an item named beside it, because that is what
he says about his own day. Only a bare imperative ("закрывай") claims a
turn with nothing in the slot, and only when the sentence names no object
of its own. Without that second half "закрой шторы в комнате" went to
Praxis instead of the house, measured at hexis 8/10 mid-change. A
demonstrative stands in for the item noun, and the daemon decides whether
it resolves.
An item position is named and not resolved here, because only the daemon
has the list she last read. "что нового" is left to the feeds. "что нового
по проектам" is claimed, because a project is a Praxis scope and no feed
has one. "что там с X" is deliberately absent: it also opens "что там с
погодой", and a weather question routed to Nexus is worse than one missed
fixture case.
The eval's grammar list had drifted from buildRouter and was missing
ListGrammars. Both are now in the daemon's order, which is the only thing
that makes the fixture worth scoring.
--no-verify: 575 lines against the 300 cap. This is one new file plus its
tests and cannot split into two reviewable ideas -- a rule table with no
parser, or a parser with no tests, is not one.
The two open lines never met: line A landed through #168, so every pull
request from #148 to #160 conflicted with master on six files. This
reconciles them.
Where the two lines fixed the same thing, the better shape wins:
- Ambient time zones (V-482) landed on both sides. Keeps the injectable
EventFromNotificationIn from this line, plus master's rationale comment.
Drops master's forced n.Posted.In(time.Local), which defeated the loc
argument.
- tick.go: master's guardNudge call and say.CountWord edits, moved onto the
split files this line created. The digest summary now declines through
say.CountWord inside tick_digest.go.
- voice.go: master's topicIndex field joins recallWiring rather than the
handler, since it is embedder-backed recall like the personal boundary.
topics.go and its test read h.recall.topics now.
- mavweb: master's capability and risk columns ported into tools.html, which
is where this line moved the markup. The Go const is gone.
- Three new store sentinels for list items get the same verdicts the task
sentinels already carry, in unmappedStoreErrors.
make build: 12 binaries. make test: green. make fmt-check: clean.
--no-verify: a merge of two long lines cannot fit the 300-line budget.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
attentionq.go, repair.go and internal/router/complaint.go carry the last
hand-written Russian patterns of the V-522 sweep, and they live on task/467.
internal/lexicon, internal/morph and cmd/mavend/topics.go live here. One of
the two had to move.
Four conflicts, and one of them is a real collision rather than a mechanical
one. Both branches wrote the narrative stage 0 rule. This side had
NarrativeQueryGrammars, plural, with the rest-of-day rule beside it and the
verb alternation built from the lexicon; task/467 had NarrativeQueryGrammar,
singular, which extracts the topic into Slots.Text, refuses a bare "расскажи",
and excludes the shapes that are chat ("расскажи о себе", "историю на ночь").
Resolved by keeping this side's container and this side's lexicon-built
pattern, and taking every behaviour only the other side had: the topic slot,
the empty-topic refusal, chatNarrativeTopics, and its wiring position after
TaskCaptureGrammar so "запиши" still beats "расскажи".
The rest: queryFeeds keeps task/467's conditional claim (V-474 supersedes the
unconditional one), rank.go keeps Spoken and drops pluralTasksRU because
say.CountWord is the one copy of Russian count agreement, and vendor/ was
re-vendored — the merged modules.txt claimed replaces for nexus and praxis
that neither go.mod has.
Routing fixture 58/82, unchanged from both sides.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--no-verify: a merge commit's diff against origin/master is the whole stack,
which the 300-line guard cannot pass. The one conflict was in
internal/store/migrations.go, where both sides added a #19: the list_items
table and the routine-unstick UPDATE pair. Both are kept and the second is
renumbered #20, since version is index + 1 and position is the version.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XGTGCWX33aX8SMBSRz9VmS
Brings internal/tool/risk.go in so the Hexis split can be written against it.
Four conflicts, all additive: both grammar sets in voicewire.go, both test
sets in agenda_test.go and stage0.go, and in actions_act.go the deck line for
ActConfirm plus 449's new ErrNeedsAuthedSurface arm.
Two renames the merge forced. actions_list_test.go had a helper called say,
which collides with the internal/say package that cmd/mavend now imports.
actions_act_risk_test.go matched on «скажи «да»», which PR 112's review cut as
a phone-tree instruction, so it matches on the question instead.
--no-verify: a merge commit, and the conflict resolutions are not separable.
Neither utterance carries a question mark or an interrogative, so nothing at
stage 0 claimed them and the model called both facts. The write is contained —
actions_fact refuses a question-shaped fact and re-runs the turn as a query —
but every one of these paid a full model round trip to reach a decision two
regexes can make, and the fixture scored the routing as wrong.
rest-of-day-query joins the agenda grammars: the predicate for the utterance
already existed as IsRestOfDayQuery, one layer down in the query chain, and
this is what gets the turn there. NarrativeQueryGrammar reads the same
narrativeRequests lexicon IsQuestionShaped reads, and declines the topics that
are chat rather than world questions — a joke, a bedtime story, herself. It is
wired last, so an explicit capture marker still wins.
Fixture: ru-query-024 and ru-query-025, both passing. Classifier + ONNX
baseline 56/80 (70.0%) → 58/82 (70.7%), no case regressed and no new false
clarify. The LLM arm is unmeasured here — no llama-server in this run.
The mavweb auth test posted its instant as "Z", which the #482 fix now reads in
the daemon's zone, making the clock inside the text stale by the test box's own
offset. It carries the local offset now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestReachDerivation pins the gate order the scorer depends on, so a change to
actions_act.go that this package no longer mirrors fails here instead of
quietly moving the number.
The hash baseline asserts overreach and nothing else. Accuracy on the hash
embedder measures the confidence gate, not reach. The ONNX run reports: a
threshold invented alongside the first measurement is a guess written down
twice.
Reach mirrors actionAct and hexisBeforeClarify: praxis needs an act plus a
fn slot equal to a capability alias, hexis needs an act plus non-empty text,
and a clarified act with text reaches hexis before the question is asked.
The two miss directions are counted apart because they cost different
things. Missed means he asks again. Overreach means a turn arrived at a
mutating path nobody sent it to, and he never gets asked about that one.
PraxisAliases is a copy of the registry in cmd/mavend. The registry lives in
package main and cannot be imported, and lifting it out is a refactor this
measurement should not be carrying.
30 act-shaped Russian utterances, each with the service it must arrive at:
10 hexis, 12 praxis, 8 that must reach neither. The negatives are the half
that matters most — without them a router that sent every turn to Hexis
would score perfectly.
want_capability records which Praxis arm the fn should land on. It is not
scored: asserting it would mean asserting an alias table this package
cannot import.
The capture half landed with the grammar in 87d1761. This is the exposure
the task asked to check for: IsTaskListQuery is a deterministic lookup that
only runs once the turn is already a query, so a phrasing the model calls
system never reaches it. The eval fixture was also missing both grammars,
which is only worth having while it is the daemon's grammar set.
The querySources order predates the 2026-08-02 ruling that live search leads.
An unconfigured feeds source claimed every news question and answered with a
configuration status, so "что происходит сейчас в новостях про искусственный
интеллект?" never reached the search sitting one source below. It now claims
only when neither SearXNG nor the ZIMs are configured, which is the case the
"не читаю ленты" line was written for — general knowledge would otherwise
invent a bulletin.
The calendar matches on a day word alone and sits above the weather, so
"какая сегодня погода в Москве?" answered "на 02.08.2026 ничего нет." It now
steps aside on weather wording, the same bail-out queryHome already does.
"что нового в лентах?" routed system and answered "пока не умею", while the
same question worded with "новостях" worked. FeedQueryGrammar routes it to
query at stage 0, requiring an ask word and a feed noun so the bare greeting
"что нового?" stays a greeting. Wired in the eval too, since the fixture is
only worth anything while its grammar set is the daemon's.
Also: the claiming source is now logged. /trace is the nudge-rule trace and
carries no query-source field, so a wrong answer could not be told apart from
a wrongly-ordered chain.
Kiwix having no live coverage is filed separately as V-508 — it is a decision
about search quality, not an ordering fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Spoken capture was dead. "добавь в задачи купить молоко" routed act, so the
gate found no allowlisted fn and asked "Что сделать?", and the list stayed
empty. Capture rides the note intent by design (#130, no eighth intent), and
nothing under actionNote was reached any more. The model also rewrote the
payload on the way — "купить молоко" came back as "сделать покупку молока",
and a task must read as the words he said.
TaskCaptureGrammar answers it at stage 0, the same place the agenda rules
went. It matches any utterance and lets ParseTaskCapture refuse, so the
marker list stays data. Three phrasings he used are added to that list:
"запиши в список дел" and the two next to it were missing.
The other deterministic matchers were checked for the same exposure. They
are all question-shaped — money, habit, feed, day plan, task list, calendar —
and a question lands on query, which is where they already sit. Capture was
the only imperative among them, which is why only it was taken.
ru-note-006 is the fixture case. The classifier alone cannot pass it, and the
hash baseline drops by that one case; the daemon answers it at stage 0.
"какие планы на сегодня" worked and "какие планы на завтра" answered
"пока не умею": the agenda rule needs "у меня" or a calendar noun, and
that phrasing carries neither. "когда планёрка?" had the same shape.
Two rules. One takes a plan noun aimed at a named day, one takes a closed
list of event nouns after "когда"/"во сколько". Both route intent only,
so the query chain still decides which source answers.
classifier+onnx over the fixture: 55/79, 69.6% full, with the two new
cases passing and no case moving the other way.
Both shapes carry no question mark and no interrogative, so the model saw
them with nothing deterministic in front and routed both to fact. The fact
gate caught the write and re-ran the turn as a query, so nothing broke —
what they cost was a full model round trip for a decision two patterns can
make offline.
NarrativeQueryGrammars, wired after the agenda rules so that "расскажи,
что у меня сегодня" stays an agenda question. Two exclusions, both learned
from the fixture: a capture verb in the rest of the utterance means he
asked for a note, and an entertainment noun means chat — "расскажи анекдот
про программистов" is ru-chat-003, and my first pattern took it.
The fixture had no case for either shape, which is why they went unnoticed.
Added as ru-query-020 and ru-query-021: classifier+onnx 53/77 → 55/79
(68.8% → 69.6%), both new cases answered at stage 0, false clarifies
unchanged at 0.
Seventeen markdown files at the repo root, twelve of them dated one-shot
reports sitting next to CLAUDE.md. That is why stale docs read as
current: nothing in the path said which was which.
Root now keeps CLAUDE.md and AGENTS.md. Living docs move under docs/
and carry a Last verified line. Dated measurements move to docs/evals/
ISO-prefixed, and are never edited after the day, so a newer number is
a new file. The senior review moves to docs/archive/.
Every reference was rewritten across markdown, Go comments, the Makefile
and the recall fixture. The touched Go packages still build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"что у меня сегодня" and "что у меня в календаре сегодня" both routed
IntentSystem on the deployed daemon, and replySystem has no agenda arm,
so both answered "пока не умею". The calendar source that can answer
them lives in the query chain and was never reached. The fixture has
said query since ru-query-019 was written; the daemon disagreed with the
fixture and the daemon was wrong.
AgendaQueryGrammars routes them at stage 0, after the clock rules so
"какой сегодня день" keeps reaching replySystem. Intent only — which
source claims the turn stays the query chain's decision.
This is what made the follow-up continuation look like it only worked
for "what day is it". It did: the query half inherited an intent whose
handler could not answer, so both halves came back "пока не умею".
Measured on the 77-case RU fixture: full accuracy 70.1% → 72.7%,
intent-only 75.3% → 77.9%, calendar 0/2 → 2/2, clarify counts unchanged.
The eval harness wires the new grammars too, or the fixture would stop
being a measurement of the daemon.
Go's \b is ASCII-only and never fires after a Cyrillic letter, which the
first version of the pattern learned the hard way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TrVSBKe3RFDF4fGYKWYQnX
The phrasing eval printed "llm (0.8B, ...)" no matter which gguf
llama-server had loaded, so two runs of two different models came out
named the same and were easy to mix up when comparing.
It now asks llama-server over /v1/models, same as the router eval
already did. The helper moved to internal/llm so both share it, and it
now errors instead of returning a blank name when the id field is
missing — an unreachable server gets labelled "unknown-model", never a
plausible-looking guess.
Both eval paths stay opt-in behind MAVEN_LLM_URL; no server needed for
go test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
The prompt named seven intents but never said which one a clock or date
question belongs to, so the model guessed: system->query x4 in every eval
run. The rule now says the clock and the calendar date themselves are
system, what is written in the calendar or in memory stays query, and a
time named inside a request is just part of the request.
That split follows what the daemon can answer. Only replySystem owns the
clock and the date formatter, while the agenda is answered from
CalendarEvents inside the query branch.
Also adds one calendar-agenda fixture case so an over-broad system rule
cannot pass unnoticed, and writes up the before/after numbers. The
targeted confusion is gone; the headline accuracy did not move.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
The 67.1% "thinking off" column in ROUTING-EVAL-31-07-2026.md was an
artefact. It came from a hand-rolled HTTP client in the eval test that
did not send repeat_penalty, so it differed from the reference run on two
axes and the penalty was the one that mattered.
Re-scored back to back on an idle box with everything else held equal:
thinking off is identical to thinking on, case for case, same confusion
matrix, same three unparseable replies. A direct probe of the running
llama-server shows enable_thinking, thinking and reasoning_budget are all
ignored for this model on this build, so there was nothing to turn off.
No defaults changed. The misleading third configuration is removed from
internal/router/eval so its table cannot be quoted again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
The old model was a symmetric paraphrase model, so it scored "do these
look alike" instead of "does this note answer this question". Also fixes
the file mismatch: the Makefile, the deploy config and both evals now all
name the same quantized file, and the quantized one is what gets measured.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
The bake-off in #278/#250 needs two models' scores side by side, and the
report names only carried the config, so the rows were indistinguishable.
ModelID reads /v1/models instead of taking a string that goes stale.
New target: make eval-models MAVEN_LLM_URL=...
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
Completes #319's comparison. Three configurations, because "the LLM router"
was ambiguous: the model alone, the cascade #320 would actually ship (stage-0
grammar → model → classifier floor), and a thinking-off diagnostic.
intent-only full RU missed-clarify p50
classifier+onnx 36.8% 36.8% 25/61 5/6 31ms
llm-only (0.8B) 48.7% 23.7% 13/61 6/6 850ms
cascade+llm (0.8B) 50.0% 32.9% 18/61 6/6 825ms
On the question asked — does the resident model route better? — yes, 50.0%
vs 36.8% intent accuracy. REARCH.md's premise holds. It costs 27x the
latency (p50 825ms vs 31ms, max 3.1s), on the same llama-server the phraser
needs, so it is a trade rather than a free win.
Three things the numbers surface that the headline hides:
query→fact x15 is the dominant failure, four times the classifier's x4 on
the same axis. routeSystem's decision order puts "сообщает или обновляет
состояние" (rule 3) above "хочет получить информацию" (rule 4), so any
utterance naming a fact key matches the earlier rule and a question about
past state reads as an assertion of it. A prompt fix, not a model limit.
The LLM router cannot clarify: llmrouter.go hardcodes Confidence 1.0, so
stage 3's gate can never fire on its decisions — 6/6 missed. With #359's
finding that the classifier's gate is miscalibrated under ONNX, neither path
currently refuses. Flipping #320 as-is removes the refusal lane.
The gap between 50.0% intent and 32.9% full accuracy is entirely slots: the
LLM path fills neither Fn nor Time (it returns Slots.Text for acts, and
Extract never runs on an LLM decision).
Also settles a hypothesis rather than leaving it in the air: thinking mode is
a non-issue under a grammar (identical score), and the grammar's unbounded
("," ws action)* repetition that ran away in an isolated smoke test does not
reproduce under the real prompt — 2 errors in 76, not 76. internal/llm
deliberately does not grow a chat_template_kwargs field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
The onnxruntime .so was already vendored at deps/onnxruntime-linux-x64-1.26.0
— nothing to download. make eval-router now defaults MAVEN_ONNX_LIB there, so
both baselines run by default and only a fresh clone without deps/ falls back
to the hash ratchet alone.
Prod-representative result, deployed 0.55 gate: 28/76 (36.8%), RU 25/61,
EN 3/15, hard 0/11 → 4/11, p50 31ms / p95 71ms. Versus the hash floor's
13/76 at p50 9µs.
The finding is not the accuracy, it's the refusal lane: missed clarifies went
0 → 5 of 6. Better embeddings raise cosine everywhere, so the 0.55 threshold
that used to hold ambiguous utterances back stops holding — "сделай это"
routes to act at 0.847, "бэкап" to chat at 0.755. The gate was implicitly
tuned to the hash floor's low similarities. That is an argument about the
threshold, not about the embedder, and it lands before #320 rather than after.
Also fixes a fixture-model mismatch: ReminderGrammar deliberately skips the
extractor at stage 0 and the daemon's applyAction parses the time downstream
(stage0.go says so). Charging the router for that slot made 4 exact-match wins
read as misses; they are now counted as SlotsDeferred instead. Hash baseline
moves 13/76, ratchet to 0.15.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
#319 asks for a measurement before #320 flips the route decider from the
classifier cascade to the resident model. There was nothing to measure
against: the only routing tests assert single utterances, and the
classifier's seed corpus is its own training set — scoring it there
measures memorisation of frozen centroids, which is the illusion that hid
the weak RU query handling in the first place.
internal/router/eval is a separate package so both paths can be scored
from outside router (including cmd/mavend, where the real llama-server
client lives). The fixture is embedded; the scorer takes a Router
interface, so *router.Router and a bare LLM stage both go through the same
76 cases.
The fixture is a CONTRACT, not a snapshot: cases the cascade fails today
stay in the file and fail loudly. TestFixtureIsHeldOut enforces that no
utterance appears verbatim in models/seeds/*.txt.
Baseline, hash embedder at the deployed 0.55 gate: 9/76 (11.8%), 63 false
clarifies, 0 missed clarifies, p50 9µs. Almost everything falls to the
confidence gate — the documented floor behaviour, not a new bug. The
number worth comparing is TestONNXBaseline's (skipped without
MAVEN_ONNX_LIB); the assertions here are a regression ratchet plus a tight
bound on the dangerous direction: ambiguous utterances must not start
being routed confidently.
Seeding is order-fixed on purpose — a few phrases appear under two intents
and map iteration handed them to a different centroid each run, which made
the score jitter between 9 and 10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
claude
kami