TestLegacyBaseline builds a minimal but complete router (stage-0
grammars, hash-embedder classifier seeded from models/seeds, deployed
confidence threshold) and runs it against the full 136-example corpus.
Reports macro F1, false-action rate, fast-path/residual/router-residual
accuracy, per-route P/R/F1, confusion matrix, and contrast family
breakdown (negation, question, reported_speech, quotation, hypothetical,
capability_question). Pre-route consumed cases are tracked separately
from the fast-path bucket.
The hash embedder is deterministic, so this baseline is reproducible.
The ONNX embedder would score higher; measure both before drawing
conclusions.
ScoreLegacy runs the actual router cascade against the 136-example
corpus and produces per-route precision/recall/F1, confusion matrix,
false-action breakdown, and fast-path/residual/router-residual/pre-route
consumption counts. Pre-route consumed cases (command-prohibition grammar
matches at stage 0) are tracked separately — these never reach the
general cascade and should not be scored by the learned router.
ContrastFamilies splits the baseline report by transform tag (negation,
question, reported_speech, quotation, hypothetical, capability_question)
and reports per-family accuracy and false-action rate.
LegacyReport.String() renders the full baseline report with confusion
matrix and false-action case listing.
Baseline types: LegacyRouter interface (satisfied by *router.Router),
LegacyCase with PrerouteConsumed flag for command-prohibition detection,
LegacyReport with fast-path/residual/router-residual/pre-route breakdowns,
ContrastFamilyReport for per-transform-family scoring.
Test helpers: buildSeededClassifier (hash embedder seeded from
models/seeds/*.txt), actVerbList, seed file loading.
TestContrastFamiliesShareSplitGroup verifies that all contrastive
variants of one base seed share exactly one SplitGroup, preventing
train/eval leakage across the contrast family split.
Add FrozenHoldoutSplit with deterministic 15% ratio using dedicated hash
seed. Produces frozen/dev partition with SplitGroup-aware leakage
prevention — all contrastive variants of one seed stay in the same split.
Add GroupedCVFolds for k-fold grouped cross-validation on the development
pool. Each fold preserves split_group boundaries; every example appears
in exactly one eval set across all folds.
Tests verify determinism (same split → same hash), no split_group leakage
across frozen/dev, route coverage in both pools, fold completeness, and
grouped CV coverage.
Add ValidateCorpus and CorpusStatsFrom for structural integrity checks
on the semantic route corpus. Validates total counts, route/source sums,
fast-path+residual partition, empty SourceID/SplitGroup, invalid routes,
duplicate identity, and conflicting labels on identical text.
CorpusStats provides deterministic dataset hash (SHA-256 of sorted texts,
first 16 bytes). ReproducibilityMeta in CorpusEnvelope records source
fixture hashes, generator version, split algorithm, and dataset hash.
TestCorpusValidation exercises the full validation pipeline.
ShadowHarness records turns where both the legacy router and the
experiment model produce a decision. Reports agreement/disagreement
split by fast-path vs residual. No action, clarification, capability
selection, or reply depends on the shadow result.
EvalReport with macro F1, per-route precision/recall/F1, confusion
matrix, false-action rate, and fast-path vs residual breakdown.
ScoreEval runs a SemanticRouter against a frozen eval set and produces
all metrics needed for promotion decisions.
Six deterministic transforms (negation, question, reported speech,
quotation, hypothetical, capability question) applied to action-route
seeds. Each transform determines the expected class explicitly — no
model guessing labels. SplitByFamily uses hash-based bucketing to keep
paraphrases in the same split.
136 examples with provenance from ru_routing_v1.json, tagged
fast_path_resolved vs residual. RouteExample carries source, source_id,
split_group for traceability. Split-by-family prevents paraphrase
leakage across train/eval.
Defines the six-class SemanticRoute type (conversation, knowledge,
action, memory_write, system, uncertain), the SemanticRouteDecision
output, the SemanticRouter interface, and the deterministic
Intent→SemanticRoute mapping from the current seven-intent cascade.
claude