TestLegacyBaseline builds a minimal but complete router (stage-0
grammars, hash-embedder classifier seeded from models/seeds, deployed
confidence threshold) and runs it against the full 136-example corpus.
Reports macro F1, false-action rate, fast-path/residual/router-residual
accuracy, per-route P/R/F1, confusion matrix, and contrast family
breakdown (negation, question, reported_speech, quotation, hypothetical,
capability_question). Pre-route consumed cases are tracked separately
from the fast-path bucket.
The hash embedder is deterministic, so this baseline is reproducible.
The ONNX embedder would score higher; measure both before drawing
conclusions.
ScoreLegacy runs the actual router cascade against the 136-example
corpus and produces per-route precision/recall/F1, confusion matrix,
false-action breakdown, and fast-path/residual/router-residual/pre-route
consumption counts. Pre-route consumed cases (command-prohibition grammar
matches at stage 0) are tracked separately — these never reach the
general cascade and should not be scored by the learned router.
ContrastFamilies splits the baseline report by transform tag (negation,
question, reported_speech, quotation, hypothetical, capability_question)
and reports per-family accuracy and false-action rate.
LegacyReport.String() renders the full baseline report with confusion
matrix and false-action case listing.
Baseline types: LegacyRouter interface (satisfied by *router.Router),
LegacyCase with PrerouteConsumed flag for command-prohibition detection,
LegacyReport with fast-path/residual/router-residual/pre-route breakdowns,
ContrastFamilyReport for per-transform-family scoring.
Test helpers: buildSeededClassifier (hash embedder seeded from
models/seeds/*.txt), actVerbList, seed file loading.
TestContrastFamiliesShareSplitGroup verifies that all contrastive
variants of one base seed share exactly one SplitGroup, preventing
train/eval leakage across the contrast family split.
Add FrozenHoldoutSplit with deterministic 15% ratio using dedicated hash
seed. Produces frozen/dev partition with SplitGroup-aware leakage
prevention — all contrastive variants of one seed stay in the same split.
Add GroupedCVFolds for k-fold grouped cross-validation on the development
pool. Each fold preserves split_group boundaries; every example appears
in exactly one eval set across all folds.
Tests verify determinism (same split → same hash), no split_group leakage
across frozen/dev, route coverage in both pools, fold completeness, and
grouped CV coverage.
Add ValidateCorpus and CorpusStatsFrom for structural integrity checks
on the semantic route corpus. Validates total counts, route/source sums,
fast-path+residual partition, empty SourceID/SplitGroup, invalid routes,
duplicate identity, and conflicting labels on identical text.
CorpusStats provides deterministic dataset hash (SHA-256 of sorted texts,
first 16 bytes). ReproducibilityMeta in CorpusEnvelope records source
fixture hashes, generator version, split algorithm, and dataset hash.
TestCorpusValidation exercises the full validation pipeline.
ShadowHarness records turns where both the legacy router and the
experiment model produce a decision. Reports agreement/disagreement
split by fast-path vs residual. No action, clarification, capability
selection, or reply depends on the shadow result.
EvalReport with macro F1, per-route precision/recall/F1, confusion
matrix, false-action rate, and fast-path vs residual breakdown.
ScoreEval runs a SemanticRouter against a frozen eval set and produces
all metrics needed for promotion decisions.
Six deterministic transforms (negation, question, reported speech,
quotation, hypothetical, capability question) applied to action-route
seeds. Each transform determines the expected class explicitly — no
model guessing labels. SplitByFamily uses hash-based bucketing to keep
paraphrases in the same split.
136 examples with provenance from ru_routing_v1.json, tagged
fast_path_resolved vs residual. RouteExample carries source, source_id,
split_group for traceability. Split-by-family prevents paraphrase
leakage across train/eval.
Defines the six-class SemanticRoute type (conversation, knowledge,
action, memory_write, system, uncertain), the SemanticRouteDecision
output, the SemanticRouter interface, and the deterministic
Intent→SemanticRoute mapping from the current seven-intent cascade.
Migrate 54 test call sites to construct NormalizedInput{Text: ...}.
Add invariant tests:
- TestNormalizedInputReachesRouteIntact: ingress NormalizedInput reaches Route
- TestTryFastPathReceivesMatchText: TryFastPath gets the same input
- TestMatchTextDoesNotChangeRouting: same Text + different MatchText → same Decision
- TestDecisionUtteranceEqualsInputText: Decision.Utterance == input.Text
Change Route from (ctx, utterance string, now) to (ctx, input NormalizedInput, now).
The ingress-constructed NormalizedInput now reaches the cascade intact — no
reconstruction downstream. TryFastPath receives the same input, not a rebuilt one.
All callers (production, eval framework, tests) updated to construct NormalizedInput.
Add MatchText field to NormalizedInput — a lossy lexical matching view
derived from ingress text: TrimSpace → NFKC → lowercase → collapse
Unicode whitespace. Does NOT fold ё→е, strip punctuation, strip wake
words, rewrite numbers, or invoke morphology.
Both ingress sites (voice + text) construct MatchText at entry. No
existing consumer reads MatchText yet — it is dark data for future
opt-in migration.
Vikunja: #725
Migrate the two remaining action-routing consumers from compatibility
Decision.Slings fields to authoritative Decision.CapabilitySelection:
- refusesCommand: reads CapabilitySelection.Fn instead of Slots.Fn
- ActHasEntityTarget: reads CapabilitySelection.Resolved and
CapabilitySelection.Args instead of Slots.HasFn and Slots.Args
Slots.Text remains the source for entity text when positional args do
not contain the target (unchanged).
Regression tests prove:
- prohibited sentinel preserved byte-for-byte through SelectCapability
- blanked Slots.Fn/Args/HasFn do not affect migrated consumers
- Praxis/Hexis entity-target routing unchanged
- stage-0 deterministic act unchanged
- classifier/extractor act unchanged
do not remove the compatibility mirrors yet.
Set CapabilitySelection on decisions that have Slots.HasFn=true, so
ResolveActionCandidate reads from the authoritative record. Backward
compatibility tests verify that decisions without CapabilitySelection
still resolve via Slots.HasFn.
The candidate now receives Fn/Args from CapabilitySelection (the
authoritative record) rather than from Decision.Slots.HasFn. Backward
compatibility: decisions with Slots.HasFn but no CapabilitySelection
(tests, rebuilt decisions) still resolve via the compatibility path.
Authoritative record of which executable capability matched, separate
from Decision.Intent (what kind of turn) and ActionCandidate (downstream
action artifact). Decision.Slots.Fn/Args/HasFn remain as compatibility
representations populated from this selection.
Call SelectCapability after each cascade path (grammar, heads, LLM,
classifier) and propagate the result via applyCapabilityToSlots.
Remove LLM text capability backfill from fillSlots — SelectCapability
now owns that path. fillMatchedSlots retains raw extractor capability
extraction for backward compatibility with stage-0 grammars.
gateLLMDecision now reads CapabilitySelection.Resolved instead of
Slots.HasFn for the act-intent confidence thinning check.
Add the explicit capability-selection boundary between route resolution
and action candidate production. SelectCapability is the single entry
point for selecting which executable capability matched an IntentAct turn.
Three input kinds: raw, llm_text, deterministic. Decision.CapabilitySelection
is the authoritative record; Decision.Slots.Fn/Args/HasFn remain as
compatibility representations populated from the selection.
Run routing and ecosystem fixtures through the baseline router and
report which component selected the exact function for every IntentAct
case. Shadow matcher comparison confirms zero disagreements.
Add ResolvedBy to dialogue.Slots and the toDialogueSlots/
applyDialogueSlots converters. Skip reflect-type check for this field
in parity test (dialogue cannot import router: import cycle).
Five disjoint values tracking which component selected the exact function:
grammar_fixed, grammar_matcher, extractor_raw, extractor_llm_text,
fallback_matcher. Slots.ResolvedBy carries provenance at the selection
point.
Introduce ActionValidationStatus enum (valid, unresolved, missing_argument,
invalid_argument, ambiguous_target) as the typed classification of validation
outcomes. ActionValidationResult now carries Status instead of boolean flags.
Backward-compatible: Unresolved() and Valid() methods preserved on the result.
Existing validation behavior unchanged: only blank Fn produces invalid_argument.
All downstream behavior (proposeGap, confirmation, task_status, praxis, hexis)
unchanged.
Tests added for all five status values, backward compatibility, and the full
validation → execution boundary.
Daemon half of the action-resolution boundary:
- Add resolveAction wrapper: delegates to ResolveActionCandidate,
records outcome in the decision trace (action-resolve:route/matcher)
- Refactor actionAct: remove matcher call, consume candidate, write
resolved values back into Slots for downstream branches
- Add 8 integration tests pinning all required scenarios:
route-sourced, matcher-sourced, matcher miss, destructive confirm,
task-status intercept, stage-0, learned-router, alias match
All existing tests pass. Execution/risk/confirmation unchanged.
Introduce the typed boundary between routing and action resolution:
- ActionCandidate: Fn, Args, Source (route|matcher), Producer, Confidence
- ResolveActionCandidate(dec, m): standalone function usable by both
the daemon and the eval harness
- Update eval harness Reach() to use ResolveActionCandidate instead of
duplicating the matcher fallback logic
This is the routing-side half of the action-resolution boundary.
The daemon integration follows in the next commit.
12 focused tests proving the first slice properties:
- text and voice enter equivalent typed turn input after stt
- stage-0 outputs remain identical with grammar producer
- classifier floor sets its producer
- clarification carries the classifier producer
- pre-route claims produce no route producer
- route producer appears on the decision record
- input source is preserved on the decision record
claude