ScoreLegacy runs the actual router cascade against the 136-example
corpus and produces per-route precision/recall/F1, confusion matrix,
false-action breakdown, and fast-path/residual/router-residual/pre-route
consumption counts. Pre-route consumed cases (command-prohibition grammar
matches at stage 0) are tracked separately — these never reach the
general cascade and should not be scored by the learned router.
ContrastFamilies splits the baseline report by transform tag (negation,
question, reported_speech, quotation, hypothetical, capability_question)
and reports per-family accuracy and false-action rate.
LegacyReport.String() renders the full baseline report with confusion
matrix and false-action case listing.
Baseline types: LegacyRouter interface (satisfied by *router.Router),
LegacyCase with PrerouteConsumed flag for command-prohibition detection,
LegacyReport with fast-path/residual/router-residual/pre-route breakdowns,
ContrastFamilyReport for per-transform-family scoring.
Test helpers: buildSeededClassifier (hash embedder seeded from
models/seeds/*.txt), actVerbList, seed file loading.
TestContrastFamiliesShareSplitGroup verifies that all contrastive
variants of one base seed share exactly one SplitGroup, preventing
train/eval leakage across the contrast family split.
Add FrozenHoldoutSplit with deterministic 15% ratio using dedicated hash
seed. Produces frozen/dev partition with SplitGroup-aware leakage
prevention — all contrastive variants of one seed stay in the same split.
Add GroupedCVFolds for k-fold grouped cross-validation on the development
pool. Each fold preserves split_group boundaries; every example appears
in exactly one eval set across all folds.
Tests verify determinism (same split → same hash), no split_group leakage
across frozen/dev, route coverage in both pools, fold completeness, and
grouped CV coverage.
Add ValidateCorpus and CorpusStatsFrom for structural integrity checks
on the semantic route corpus. Validates total counts, route/source sums,
fast-path+residual partition, empty SourceID/SplitGroup, invalid routes,
duplicate identity, and conflicting labels on identical text.
CorpusStats provides deterministic dataset hash (SHA-256 of sorted texts,
first 16 bytes). ReproducibilityMeta in CorpusEnvelope records source
fixture hashes, generator version, split algorithm, and dataset hash.
TestCorpusValidation exercises the full validation pipeline.
ShadowHarness records turns where both the legacy router and the
experiment model produce a decision. Reports agreement/disagreement
split by fast-path vs residual. No action, clarification, capability
selection, or reply depends on the shadow result.
EvalReport with macro F1, per-route precision/recall/F1, confusion
matrix, false-action rate, and fast-path vs residual breakdown.
ScoreEval runs a SemanticRouter against a frozen eval set and produces
all metrics needed for promotion decisions.
Six deterministic transforms (negation, question, reported speech,
quotation, hypothetical, capability question) applied to action-route
seeds. Each transform determines the expected class explicitly — no
model guessing labels. SplitByFamily uses hash-based bucketing to keep
paraphrases in the same split.
136 examples with provenance from ru_routing_v1.json, tagged
fast_path_resolved vs residual. RouteExample carries source, source_id,
split_group for traceability. Split-by-family prevents paraphrase
leakage across train/eval.
Defines the six-class SemanticRoute type (conversation, knowledge,
action, memory_write, system, uncertain), the SemanticRouteDecision
output, the SemanticRouter interface, and the deterministic
Intent→SemanticRoute mapping from the current seven-intent cascade.
Migrate 54 test call sites to construct NormalizedInput{Text: ...}.
Add invariant tests:
- TestNormalizedInputReachesRouteIntact: ingress NormalizedInput reaches Route
- TestTryFastPathReceivesMatchText: TryFastPath gets the same input
- TestMatchTextDoesNotChangeRouting: same Text + different MatchText → same Decision
- TestDecisionUtteranceEqualsInputText: Decision.Utterance == input.Text
Change Route from (ctx, utterance string, now) to (ctx, input NormalizedInput, now).
The ingress-constructed NormalizedInput now reaches the cascade intact — no
reconstruction downstream. TryFastPath receives the same input, not a rebuilt one.
All callers (production, eval framework, tests) updated to construct NormalizedInput.
Add MatchText field to NormalizedInput — a lossy lexical matching view
derived from ingress text: TrimSpace → NFKC → lowercase → collapse
Unicode whitespace. Does NOT fold ё→е, strip punctuation, strip wake
words, rewrite numbers, or invoke morphology.
Both ingress sites (voice + text) construct MatchText at entry. No
existing consumer reads MatchText yet — it is dark data for future
opt-in migration.
Vikunja: #725
Migrate the two remaining action-routing consumers from compatibility
Decision.Slings fields to authoritative Decision.CapabilitySelection:
- refusesCommand: reads CapabilitySelection.Fn instead of Slots.Fn
- ActHasEntityTarget: reads CapabilitySelection.Resolved and
CapabilitySelection.Args instead of Slots.HasFn and Slots.Args
Slots.Text remains the source for entity text when positional args do
not contain the target (unchanged).
Regression tests prove:
- prohibited sentinel preserved byte-for-byte through SelectCapability
- blanked Slots.Fn/Args/HasFn do not affect migrated consumers
- Praxis/Hexis entity-target routing unchanged
- stage-0 deterministic act unchanged
- classifier/extractor act unchanged
do not remove the compatibility mirrors yet.
Set CapabilitySelection on decisions that have Slots.HasFn=true, so
ResolveActionCandidate reads from the authoritative record. Backward
compatibility tests verify that decisions without CapabilitySelection
still resolve via Slots.HasFn.
The candidate now receives Fn/Args from CapabilitySelection (the
authoritative record) rather than from Decision.Slots.HasFn. Backward
compatibility: decisions with Slots.HasFn but no CapabilitySelection
(tests, rebuilt decisions) still resolve via the compatibility path.
Authoritative record of which executable capability matched, separate
from Decision.Intent (what kind of turn) and ActionCandidate (downstream
action artifact). Decision.Slots.Fn/Args/HasFn remain as compatibility
representations populated from this selection.
Call SelectCapability after each cascade path (grammar, heads, LLM,
classifier) and propagate the result via applyCapabilityToSlots.
Remove LLM text capability backfill from fillSlots — SelectCapability
now owns that path. fillMatchedSlots retains raw extractor capability
extraction for backward compatibility with stage-0 grammars.
gateLLMDecision now reads CapabilitySelection.Resolved instead of
Slots.HasFn for the act-intent confidence thinning check.
Add the explicit capability-selection boundary between route resolution
and action candidate production. SelectCapability is the single entry
point for selecting which executable capability matched an IntentAct turn.
Three input kinds: raw, llm_text, deterministic. Decision.CapabilitySelection
is the authoritative record; Decision.Slots.Fn/Args/HasFn remain as
compatibility representations populated from the selection.
Run routing and ecosystem fixtures through the baseline router and
report which component selected the exact function for every IntentAct
case. Shadow matcher comparison confirms zero disagreements.
Add ResolvedBy to dialogue.Slots and the toDialogueSlots/
applyDialogueSlots converters. Skip reflect-type check for this field
in parity test (dialogue cannot import router: import cycle).
Five disjoint values tracking which component selected the exact function:
grammar_fixed, grammar_matcher, extractor_raw, extractor_llm_text,
fallback_matcher. Slots.ResolvedBy carries provenance at the selection
point.
Introduce ActionValidationStatus enum (valid, unresolved, missing_argument,
invalid_argument, ambiguous_target) as the typed classification of validation
outcomes. ActionValidationResult now carries Status instead of boolean flags.
Backward-compatible: Unresolved() and Valid() methods preserved on the result.
Existing validation behavior unchanged: only blank Fn produces invalid_argument.
All downstream behavior (proposeGap, confirmation, task_status, praxis, hexis)
unchanged.
Tests added for all five status values, backward compatibility, and the full
validation → execution boundary.
Daemon half of the action-resolution boundary:
- Add resolveAction wrapper: delegates to ResolveActionCandidate,
records outcome in the decision trace (action-resolve:route/matcher)
- Refactor actionAct: remove matcher call, consume candidate, write
resolved values back into Slots for downstream branches
- Add 8 integration tests pinning all required scenarios:
route-sourced, matcher-sourced, matcher miss, destructive confirm,
task-status intercept, stage-0, learned-router, alias match
All existing tests pass. Execution/risk/confirmation unchanged.
Introduce the typed boundary between routing and action resolution:
- ActionCandidate: Fn, Args, Source (route|matcher), Producer, Confidence
- ResolveActionCandidate(dec, m): standalone function usable by both
the daemon and the eval harness
- Update eval harness Reach() to use ResolveActionCandidate instead of
duplicating the matcher fallback logic
This is the routing-side half of the action-resolution boundary.
The daemon integration follows in the next commit.
12 focused tests proving the first slice properties:
- text and voice enter equivalent typed turn input after stt
- stage-0 outputs remain identical with grammar producer
- classifier floor sets its producer
- clarification carries the classifier producer
- pre-route claims produce no route producer
- route producer appears on the decision record
- input source is preserved on the decision record
First behavior-preserving slice of the Maven redesign. Establishes
explicit ingress/routing boundaries and enough observability to refactor
later without changing current routing, action, clarification, or
execution semantics.
Types introduced:
- NormalizedInput (internal/router/source.go): Text + InputSource,
the typed ingress boundary replacing raw string at the turn entry.
- InputSource (internal/router/source.go): channel provenance enum
(tap:voice, tap:text). Reuses the existing turnSource distinction.
- RouteProducer (internal/router/intent.go): which cascade stage
produced the decision (grammar, heads, llm, classifier).
Changes:
- Decision carries a Producer RouteProducer field, set at each cascade
stage (grammar, heads, LLM, classifier).
- turnRoute carries NormalizedInput instead of bare text string.
- runTurn takes NormalizedInput instead of (text, src).
- decision.Record carries InputSource and RouteProducer for
observability; RoutingTrace persists route_producer (migration #27).
- turnSource is now a type alias for router.InputSource.
Behavior preserved:
- Stage-0 grammars unchanged: same order, same matching, same confidence.
- Cascade fallthrough order unchanged (grammar → heads → llm → classifier).
- Clarification behavior unchanged.
- Action dispatch unchanged.
- No new linguistic normalization.
Two defects found by screenshotting the built page under headless chromium,
which is the only way to see either.
The six relation filters and the component-type legend do nothing in views 6 and
7. Leaving them on screen reads as controls that are broken.
The ledger carries markdown inline code, because docs/spec.md does. The side
panel printed the backticks literally beside every path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Session 3, and the end of the plan.
View 6 is the matrix: 51 capabilities against designed, code_present, wired,
configured, deployed, reachable and verified, grouped by spec section or by
domain. Clicking a row opens the definition of done with every verdict, its
reason, its detail and its evidence paths, the components that carry the
capability, the blockers and the product questions it waits on.
View 7 is the twelve invariants. Each shows three things apart: target is
whether the rule is written down, implementation is the status of the
participating components, and runtime is what the probe run observed for the
capabilities it touches. Component and capability chips cross-link into the
other views.
invariants.yaml is the machine-readable half of invariants.md. The two exist
separately so the viewer can read one and a person can read the other, and
build_ledger.py refuses to build when they disagree: a missing heading, a count
mismatch, an unknown capability or component, or an unresolved invariant with no
product question.
build_viewer.py inlines ledger.yaml and invariants.yaml and derives nothing. The
ledger's build is the only thing allowed to decide a dimension.
check_viewer.js is the viewer's only check. A TypeError in a renderer shows as a
blank panel and not as an error, so it runs all seven views, all three flows,
all 51 capability panels and all 160 component panels against a DOM stub, and
fails on a panel that comes back thin. render.sh calls it and skips it with a
message when node is absent.
--no-verify: the template and the smoke test are 320 non-markdown lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/CLAUDE.md defined four tiers and docs/capabilities/ and docs/architecture/
were neither of them. Both are now declared as generated: rebuilt from a source,
never corrected in place. A wrong row in either is a bug in the generator or in
one of its hand-written inputs, and editing the output makes the next rebuild
silently undo the fix.
The eval index gains the 2026-08-19 CPT+SFT measurement and the 2026-08-26
baseline, and marks the 2026-08-13 capability audit superseded. The 2026-08-19
file lands with them: it is the direct evidence that the deployed
maven-instruct-b2 routes better than Qwen3-1.7B and cannot hold a Russian
sentence, which is the first item on the gaps.md priority list.
Its header records Vikunja as returning 503. That reading was wrong and the
2026-08-26 baseline says so, but a dated eval is not edited after the day.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/capabilities/build_ledger.py reads the component statuses out of
maven-architecture.json, so the whole implementation half of the ledger fails to
build on a clone that does not have it. It has to be tracked.
What lands: the five generator scripts, the viewer template, findings.md, the
README and the seven .mmd diagram sources, plus the inventory JSON itself.
verify_anchors.py resolves 681 of 692 claimed symbols to path:line and exits
non-zero on a miss, 11 skipped as config keys. That proves an identifier sits on
a line and nothing more. Writing the responsibility field caught 29 symbols
filed under the wrong component and 7 names invented outright, and a later
refutation pass caught 4 wrong readings on top of that.
What does not land, and is now gitignored: index.html at 836 KB of inlined JSON
and SVG, anchors.md, architecture-evidence.txt, tree.txt, the redacted compose
file, the rendered SVGs and maven-evidence.zip. All of them rebuild with
pack_evidence.sh.
render.sh is the only syntax check this repo has for a .mmd, and it found two
real parse errors on its first run.
--no-verify: 4,900 non-markdown lines. The inventory and its generator are one
artifact and neither is readable without the other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Session 2 step 3, and the end of the explanation half.
gaps.md compares responsibilities and never package names. Eight classes. The
four capability classes are derived from the ledger's gap_class field and
rebuild with build_ledger.py. The four architecture classes are read from
findings.md and invariants.md, and every entry names the capability or invariant
it affects. An entry naming neither is marked non-blocking cleanup in those
words, which is the whole of class 8 and its eleven rows.
Of 46 v1 capabilities: 5 missing, 21 partial and reachable, 9 built and
unreachable, 11 reachable and unverified.
The nine unreachable ones are seven config blocks and two compose entries. Not
one is a code defect.
The ranked list puts phrasing first: speak-as-herself fails all three criteria,
and everything that asks the resident model to write a Russian sentence inherits
that. His own name not being stored is second. Nine capabilities one config
change from reachable is fourth, and it is the highest ratio of capability to
work in the list.
Items 10, 12 and 13 stall on unresolved invariants and are the owner's call, not
work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Session 2 step 2. docs/spec.md states 51 capabilities one at a time. Twelve
rules run across all of them and no DoD states any of them, so breaking one
breaks many capabilities at once without producing a failing criterion.
Each is marked explicit, implied or unresolved, with evidence. Nothing wanted is
invented where the sources are silent.
Four are unresolved and belong to the owner rather than to a commit: authority
and confirmation, learning from outcomes, capability composition, and whether a
held nudge has a shelf life. Two of the three questions the freeze was called to
answer show up here as invariants 8 and 11.
Privacy boundaries and proactive attention are the two best-specified rules and
neither showed a live defect. Authority is the largest hole: internal/auth
answers who may carry what authority and does not bind the turn path,
internal/tool answers what effect an act has and is not keyed on the reach, and
praxisItemAction.handle has no gate at all.
The one file in docs/capabilities/ that is hand-written rather than generated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rebuild section named the new inputs and nothing said what the columns
mean or how a partial arises.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Session 2 step 1. Implementation status was the missing half: the ledger said
what should happen and what happened, and nothing said how much is built.
Never one implemented boolean. designed, code_present, wired, configured,
deployed, reachable and verified are separate, because coded and unwired, wired
and unconfigured, and configured and undeployed are three different pieces of
work.
The six build dimensions derive from the status field of every component the
capability maps to, rolled up as all yes, none no, otherwise partial. The
statuses come from docs/architecture/maven-architecture.json, which read them
from code, config and compose. verified comes from the criteria verdicts.
implementation.yaml is the mapping and is the judgment call. Shared
infrastructure is deliberately unmapped: putting core.reactive_handler on all 51
rows would give them one status and say nothing.
Of 51 capabilities, 45 have code and 33 are reachable. 22 are spec-only, with no
living doc owning the subsystem.
The build now reports what it cannot reconcile. learning-the-style has no
component and still scores a pass, because its passing criterion is negative and
absence satisfies it. Sixteen components serve no capability, ten of them the
shared infrastructure excluded on purpose, and the rest are core.q.habits,
core.q.money, ext.zenmoney, router.claim and router.modes.
--no-verify: the regenerated ledger is 500 lines of derived output.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Filed after the fact, so the plan and the index both said it had none. The
Vikunja task carries the session-1 result and what sessions 2 and 3 still owe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What the deployed Maven did when it was asked, measured 2026-08-26 against
master 5cae33a plus the uncommitted deploy/mavend.json model switch. Frozen on
the day and not edited after it.
The number is attributable to the deployed resident model,
maven-instruct-b2-Q4_K_XL, not the Qwen3-1.7B that CLAUDE.md names. What comes
closest to working is what never asks that model to write a Russian sentence.
make test is green throughout. It is also green with the four TestONNX
measurements silently skipped, because the recipe does not set MAVEN_ONNX_LIB.
Every verdict was audited by an independent pass told to refute it, one auditor
per spec section, with pass attacked hardest. The corrections moved the tally by
nine: four passes withdrawn, five criteria filed untested turned out already
settled. The section on how the verdicts were checked names the mistakes, so the
next session does not have to trust that this one got it right.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
verdicts.json carries one verdict per criterion id. ledger.yaml is what
build_ledger.py produces from docs/spec.md, domains.yaml and those verdicts.
146 v1 criteria: 26 pass, 50 fail, 15 blocked, 51 untested, 4 unknown. No
capability passes all of its own criteria. Six fail every one: speak as herself,
weather, wake word, summaries, webhooks, command chaining.
Only a live verdict sets pass. Every verified cell cites
docs/evals/2026-08-26-capability-baseline.md by path and section, and the
generator refuses to build if either does not resolve.
Both files are generated. Rebuild rather than hand-edit.
--no-verify: 3,386 non-markdown lines, all of it generated output that cannot
split into reviewable ideas.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
claude