TestLegacyBaseline builds a minimal but complete router (stage-0
grammars, hash-embedder classifier seeded from models/seeds, deployed
confidence threshold) and runs it against the full 136-example corpus.
Reports macro F1, false-action rate, fast-path/residual/router-residual
accuracy, per-route P/R/F1, confusion matrix, and contrast family
breakdown (negation, question, reported_speech, quotation, hypothetical,
capability_question). Pre-route consumed cases are tracked separately
from the fast-path bucket.
The hash embedder is deterministic, so this baseline is reproducible.
The ONNX embedder would score higher; measure both before drawing
conclusions.
ScoreLegacy runs the actual router cascade against the 136-example
corpus and produces per-route precision/recall/F1, confusion matrix,
false-action breakdown, and fast-path/residual/router-residual/pre-route
consumption counts. Pre-route consumed cases (command-prohibition grammar
matches at stage 0) are tracked separately — these never reach the
general cascade and should not be scored by the learned router.
ContrastFamilies splits the baseline report by transform tag (negation,
question, reported_speech, quotation, hypothetical, capability_question)
and reports per-family accuracy and false-action rate.
LegacyReport.String() renders the full baseline report with confusion
matrix and false-action case listing.
Baseline types: LegacyRouter interface (satisfied by *router.Router),
LegacyCase with PrerouteConsumed flag for command-prohibition detection,
LegacyReport with fast-path/residual/router-residual/pre-route breakdowns,
ContrastFamilyReport for per-transform-family scoring.
Test helpers: buildSeededClassifier (hash embedder seeded from
models/seeds/*.txt), actVerbList, seed file loading.
TestContrastFamiliesShareSplitGroup verifies that all contrastive
variants of one base seed share exactly one SplitGroup, preventing
train/eval leakage across the contrast family split.
Add FrozenHoldoutSplit with deterministic 15% ratio using dedicated hash
seed. Produces frozen/dev partition with SplitGroup-aware leakage
prevention — all contrastive variants of one seed stay in the same split.
Add GroupedCVFolds for k-fold grouped cross-validation on the development
pool. Each fold preserves split_group boundaries; every example appears
in exactly one eval set across all folds.
Tests verify determinism (same split → same hash), no split_group leakage
across frozen/dev, route coverage in both pools, fold completeness, and
grouped CV coverage.
Add ValidateCorpus and CorpusStatsFrom for structural integrity checks
on the semantic route corpus. Validates total counts, route/source sums,
fast-path+residual partition, empty SourceID/SplitGroup, invalid routes,
duplicate identity, and conflicting labels on identical text.
CorpusStats provides deterministic dataset hash (SHA-256 of sorted texts,
first 16 bytes). ReproducibilityMeta in CorpusEnvelope records source
fixture hashes, generator version, split algorithm, and dataset hash.
TestCorpusValidation exercises the full validation pipeline.
ShadowHarness records turns where both the legacy router and the
experiment model produce a decision. Reports agreement/disagreement
split by fast-path vs residual. No action, clarification, capability
selection, or reply depends on the shadow result.
EvalReport with macro F1, per-route precision/recall/F1, confusion
matrix, false-action rate, and fast-path vs residual breakdown.
ScoreEval runs a SemanticRouter against a frozen eval set and produces
all metrics needed for promotion decisions.
Six deterministic transforms (negation, question, reported speech,
quotation, hypothetical, capability question) applied to action-route
seeds. Each transform determines the expected class explicitly — no
model guessing labels. SplitByFamily uses hash-based bucketing to keep
paraphrases in the same split.
136 examples with provenance from ru_routing_v1.json, tagged
fast_path_resolved vs residual. RouteExample carries source, source_id,
split_group for traceability. Split-by-family prevents paraphrase
leakage across train/eval.
Defines the six-class SemanticRoute type (conversation, knowledge,
action, memory_write, system, uncertain), the SemanticRouteDecision
output, the SemanticRouter interface, and the deterministic
Intent→SemanticRoute mapping from the current seven-intent cascade.
Migrate 54 test call sites to construct NormalizedInput{Text: ...}.
Add invariant tests:
- TestNormalizedInputReachesRouteIntact: ingress NormalizedInput reaches Route
- TestTryFastPathReceivesMatchText: TryFastPath gets the same input
- TestMatchTextDoesNotChangeRouting: same Text + different MatchText → same Decision
- TestDecisionUtteranceEqualsInputText: Decision.Utterance == input.Text
Change Route from (ctx, utterance string, now) to (ctx, input NormalizedInput, now).
The ingress-constructed NormalizedInput now reaches the cascade intact — no
reconstruction downstream. TryFastPath receives the same input, not a rebuilt one.
All callers (production, eval framework, tests) updated to construct NormalizedInput.
Add MatchText field to NormalizedInput — a lossy lexical matching view
derived from ingress text: TrimSpace → NFKC → lowercase → collapse
Unicode whitespace. Does NOT fold ё→е, strip punctuation, strip wake
words, rewrite numbers, or invoke morphology.
Both ingress sites (voice + text) construct MatchText at entry. No
existing consumer reads MatchText yet — it is dark data for future
opt-in migration.
Vikunja: #725
Migrate the two remaining action-routing consumers from compatibility
Decision.Slings fields to authoritative Decision.CapabilitySelection:
- refusesCommand: reads CapabilitySelection.Fn instead of Slots.Fn
- ActHasEntityTarget: reads CapabilitySelection.Resolved and
CapabilitySelection.Args instead of Slots.HasFn and Slots.Args
Slots.Text remains the source for entity text when positional args do
not contain the target (unchanged).
Regression tests prove:
- prohibited sentinel preserved byte-for-byte through SelectCapability
- blanked Slots.Fn/Args/HasFn do not affect migrated consumers
- Praxis/Hexis entity-target routing unchanged
- stage-0 deterministic act unchanged
- classifier/extractor act unchanged
do not remove the compatibility mirrors yet.
Set CapabilitySelection on decisions that have Slots.HasFn=true, so
ResolveActionCandidate reads from the authoritative record. Backward
compatibility tests verify that decisions without CapabilitySelection
still resolve via Slots.HasFn.
The candidate now receives Fn/Args from CapabilitySelection (the
authoritative record) rather than from Decision.Slots.HasFn. Backward
compatibility: decisions with Slots.HasFn but no CapabilitySelection
(tests, rebuilt decisions) still resolve via the compatibility path.
Authoritative record of which executable capability matched, separate
from Decision.Intent (what kind of turn) and ActionCandidate (downstream
action artifact). Decision.Slots.Fn/Args/HasFn remain as compatibility
representations populated from this selection.
Call SelectCapability after each cascade path (grammar, heads, LLM,
classifier) and propagate the result via applyCapabilityToSlots.
Remove LLM text capability backfill from fillSlots — SelectCapability
now owns that path. fillMatchedSlots retains raw extractor capability
extraction for backward compatibility with stage-0 grammars.
gateLLMDecision now reads CapabilitySelection.Resolved instead of
Slots.HasFn for the act-intent confidence thinning check.
Add the explicit capability-selection boundary between route resolution
and action candidate production. SelectCapability is the single entry
point for selecting which executable capability matched an IntentAct turn.
Three input kinds: raw, llm_text, deterministic. Decision.CapabilitySelection
is the authoritative record; Decision.Slots.Fn/Args/HasFn remain as
compatibility representations populated from the selection.
Run routing and ecosystem fixtures through the baseline router and
report which component selected the exact function for every IntentAct
case. Shadow matcher comparison confirms zero disagreements.
Five disjoint values tracking which component selected the exact function:
grammar_fixed, grammar_matcher, extractor_raw, extractor_llm_text,
fallback_matcher. Slots.ResolvedBy carries provenance at the selection
point.
Introduce ActionValidationStatus enum (valid, unresolved, missing_argument,
invalid_argument, ambiguous_target) as the typed classification of validation
outcomes. ActionValidationResult now carries Status instead of boolean flags.
Backward-compatible: Unresolved() and Valid() methods preserved on the result.
Existing validation behavior unchanged: only blank Fn produces invalid_argument.
All downstream behavior (proposeGap, confirmation, task_status, praxis, hexis)
unchanged.
Tests added for all five status values, backward compatibility, and the full
validation → execution boundary.
Introduce the typed boundary between routing and action resolution:
- ActionCandidate: Fn, Args, Source (route|matcher), Producer, Confidence
- ResolveActionCandidate(dec, m): standalone function usable by both
the daemon and the eval harness
- Update eval harness Reach() to use ResolveActionCandidate instead of
duplicating the matcher fallback logic
This is the routing-side half of the action-resolution boundary.
The daemon integration follows in the next commit.
12 focused tests proving the first slice properties:
- text and voice enter equivalent typed turn input after stt
- stage-0 outputs remain identical with grammar producer
- classifier floor sets its producer
- clarification carries the classifier producer
- pre-route claims produce no route producer
- route producer appears on the decision record
- input source is preserved on the decision record
First behavior-preserving slice of the Maven redesign. Establishes
explicit ingress/routing boundaries and enough observability to refactor
later without changing current routing, action, clarification, or
execution semantics.
Types introduced:
- NormalizedInput (internal/router/source.go): Text + InputSource,
the typed ingress boundary replacing raw string at the turn entry.
- InputSource (internal/router/source.go): channel provenance enum
(tap:voice, tap:text). Reuses the existing turnSource distinction.
- RouteProducer (internal/router/intent.go): which cascade stage
produced the decision (grammar, heads, llm, classifier).
Changes:
- Decision carries a Producer RouteProducer field, set at each cascade
stage (grammar, heads, LLM, classifier).
- turnRoute carries NormalizedInput instead of bare text string.
- runTurn takes NormalizedInput instead of (text, src).
- decision.Record carries InputSource and RouteProducer for
observability; RoutingTrace persists route_producer (migration #27).
- turnSource is now a type alias for router.InputSource.
Behavior preserved:
- Stage-0 grammars unchanged: same order, same matching, same confidence.
- Cascade fallthrough order unchanged (grammar → heads → llm → classifier).
- Clarification behavior unchanged.
- Action dispatch unchanged.
- No new linguistic normalization.
MavenHelpGrammar keeps "как отменить напоминание" on SourceSelf, where the
answer names the command Maven accepts, instead of leaking to search.
PublicCurrentVersionGrammar anchors an explicitly current release on
SourceWorld and declines first-person ownership.
AmbiguousFragmentGrammar refuses filler plus an unresolved demonstrative
rather than letting a statistical head invent context.
ImplicitElapsedQueryGrammar reads Russian question word order in "давно я
не тренировался" as recall; the declarative order stays a statement.
ReminderCancellationReportGrammar keeps "я отменил напоминание" in the
non-mutating chat lane.
CommandProhibitionGrammar routes a direct negative command to a sentinel
fn that can never collide with an enabled tool. ActHasEntityTarget stops a
bare verb or a demonstrative-only tail from crossing into Nexus.
Praxis attention now accepts "что там с X" for the four service names only.
taskstatus separates command mood from result words so a first-person
report cannot mutate the board. question.go exports the open-question and
locative shapes the recall gate reads.
--no-verify: master is the working branch this session by the owner's call.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reference-count the process-global ONNX Runtime across embedder and routing-head sessions, make close idempotent, and require named proof that both aggregate routing gates executed rather than self-skipped (V-716). Owner explicitly requested direct commits to master.
Replace nearest-neighbour personal routing with a frozen class-balanced linear head measured on historical, stratified, cross-validation, holdout, and fresh challenge gates (V-702). Close the four repair handoff holes, preserve nested clarification flows, and route Russian possession statements through structural grammar rather than lexical exceptions (V-573). Owner explicitly requested direct commits to master.
buildRouter held the real set and baselineGrammars in eval_test.go restated it
by hand, in the daemon's order, with its own comment saying so. Three test files
score against the fixture and nothing compared the two lists. They had already
drifted: BareCaptureGrammar went into the daemon with V-557 and never into the
fixture, so every routing measurement since has scored a set nobody runs. That
is the failure CLAUDE.md warns about by name, and a diff test would have caught
it one grammar late.
The list moves to router.StageZeroGrammars in internal/router/stagezero.go, with
the ordering comments, which are the load-bearing part. buildRouter and the
fixture both call it. One list cannot drift from itself.
Measured before and after on the 96-case fixture: classifier+onnx 72/96, 75.0%
intent, 33.3% destination, identical either way, and the deterministic claim and
reach hash ratchets do not move. So the missing grammar cost no measurable
accuracy. That is the point rather than a reprieve: the fixture had been scoring
the wrong set for four days and nothing could say so.
The invariants caveat is deleted, both entries, since V-692 landed the other
guard in the previous commit. The reasoning for both now sits in docs/routing.md
beside the subsystem, which is where a fix's durable record belongs.
Unrelated and pre-existing: TestONNXPersonalBoundary fails on "я рассказывал
тебе про байкал?" (personal 0.9068, world 0.9413) at the merge base too.
Naming a destination takes the guessing query sources off a turn, and the
personal boundary is one of them. Every other guesser costs an answer when it
is wrongly dropped. This one costs the rule that a question about him never
reaches an upstream engine.
Three deciders name a destination now and two of them infer it: the routing
heads and the resident model. Decision.SourceAnchored says a stage 0 grammar
read the words instead. queryWalk honours it for the source marked
boundary: true and for no other, so the rest of the table is unchanged.
Owner's call of 2026-08-09.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
embedder.heads_path is empty by default and deploy/mavend.json sets
it. A missing or broken weights file logs and leaves the heads nil,
because refusing to start over a routing accelerator would trade a
working box for a better one.
TestONNXRoutingHeads is the same cascade TestONNXBaseline scores with
one arm added, so the two are directly comparable. It also checks the
Go tokenizer against the Python one, since the heads were trained
through transformers and are read through a hand-written tokenizer: a
mismatch shows up here as a score below what Python measured on the
same weights, and nowhere else. That is how the reversed word pieces
were found.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
They run before the model because they are two orders of magnitude
faster and score better on both halves of the route. They decline
rather than clarify, so a declined turn carries on to the model and
then the classifier, which is what a box with no weights file does on
every turn. Nil heads are byte-for-byte the cascade that shipped
before this.
Measured on the 96-case fixture, classifier+ONNX either way:
intent 76.0% -> 96.9%
destination 36.4% -> 75.8%
false clarify 0 -> 1
missed clarify 8 -> 1
p50 24.5ms -> 27.9ms
That beats the gemma-4-12b cascade on both halves, 84.4% and 72.7%, at
a twelfth of its 329ms. The four remaining destination misses are all
calendar, which is the stage 0 trade V-660 flagged and the owner has
not called yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
The heads trained in V-661 ran nowhere. This loads the exported graph
and reads intent, destination and clarify off one forward pass. It
declines below 0.6 max softmax rather than clarifying, so a declined
turn reaches whatever is behind it.
The slot head is exported and deliberately not read: slots already
come from the stage-2 extractor, and mapping BIO tags back to text
needs character offsets the tokenizer does not keep.
The clarify head decides on its own and decides first. It answers a
different question from the intent head, so a low intent confidence is
no reason to discard it. Reading it only above the intent threshold
cost 6 of the 8 ambiguous cases on the fixture: the word for water
reads as intent act at 0.23 and clarify at 0.98.
0.6 is the knee measured on the intent fixture: every higher value up
to 0.9 drops right answers and keeps the same two wrong ones.
The body is a fine-tuned COPY of the resident embedder and must never
replace it, because memory recall depends on that file scoring what it
scored.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
encodeWord backtracks the Viterbi path from the end of the word and
prepends each piece, which puts them back in reading order. A second
reverse after that loop undid it. So "query: вода" tokenized to
[0 12 1294 41 12489 2] where the reference tokenizer gives
[0 41 1294 12 12489 2], and every multi-piece Russian word reached the
model with its pieces in the wrong order.
Measured on the recall fixture, same 27 cases either way:
recall@1 70.4% -> 77.8%
recall@3 85.2% -> 96.3%
answered after gate 63.0% -> 66.7%
false recall 0/5 -> 1/5
The classifier barely moves, 76.0% to 75.0% on the routing fixture,
because seeds and queries were mangled the same way and cosine survived
it. Recall is where it cost, because a stored passage and a live query
are different lengths and break differently.
The embedder id now names a tokenizer revision. Stored vectors were
written under rev 1 and no longer sit in the same space as a query
embedded now, and the model file's name never moved, so nothing would
have triggered ReembedAll.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
The training workspace labels with routeSystem and routeGrammar, and it held
its own copies. V-660 changed both. A retyped prompt drifts silently, which is
the problem llm/check_prompt_parity.py exists for on the other side.
Inert unless MAVEN_DUMP_PROMPT names a directory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
V-659 measured the destination at 12/33 on the classifier cascade and named
the gap: recall 0/15, because nothing anywhere names it. The model could not
help, for a structural reason rather than a capability one. Nothing in
routeSystem mentioned a Source and routeGrammar could not emit one, so there
was no string for it to write. Same shape as the Praxis reach V-517
measured at 0/12.
routeGrammar grows a source rule, closed over router.Sources plus the empty
floor. A grammar cannot emit a destination that does not exist, which is the
guarantee V-546 wants from a softmax and gets here for free. The prompt
lists the twelve in Russian, one line each, and says plainly that "" is a
normal answer to give often: two sources that can both answer means the
chain walks, and guessing is the failure mode this whole field exists to
stop.
The read-back goes through ValidSource and runs on IntentQuery alone. The
grammar already bounds the enum, but it is a request to a server that may be
running another build, and only a query reaches queryWalk.
Measured against gemma-4-12b on the workstation, same fixture, cascade with
a hash fallback: destination 24/33 (72.7%) against the classifier's 12/33,
and intent 81/96 (84.4%) which is where it already was. Recall is the whole
move, 0/15 to 14/15. The model alone scores 26/33.
Four cases the cascade loses and llm-only wins are calendar. The possessive
agenda rules claim them at stage 0 and deliberately name nothing, because
"что у меня в списке покупок" matches the same rule and naming the calendar
would take the list source off the turn. So stage 0's caution now costs four
destination points it did not cost before. That is a real trade and it wants
its own argument, not a quiet edit here.
The resident Qwen3-1.7B is unmeasured: it binds --port 0 inside the
container and no host process can reach it.
llm/check_prompt_parity.py in the training workspace compares its copy of
routeSystem to this one and will fail until that copy gets the same edit.
V-362 covers the catch-up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
claude