Compare commits

...

91 Commits

Author SHA1 Message Date
claude 1a309b283b router/semantic: slice 23 report — corrected population keeps every slice-22 conclusion, hash floor collapses to 0.066, pool 1509 non-action + 720 OOD 2026-09-08 04:02:03 +04:00
claude be71ac406b router/semantic: slice 23 corpus fast-path reconciliation — DeriveFastPath over the real router replaces the regex mirror, factory/merge validated, corpus rebuilt (dataset_hash unchanged), drift diagnostic and rerun tooling 2026-09-08 04:02:00 +04:00
claude bb6bd8efb9 router/semantic: slice 22 report — linear e5 at parity with the deployed cascade, cross-family transfer absent, 143 illegal actions, levers are deterministic 2026-09-08 01:27:21 +04:00
claude 29f74dd3cc router/semantic: slice 22 residual non-action router — emit step, Go harness (legacy + heads modes), Python experiment 2026-09-08 01:26:56 +04:00
claude bdced83b46 router/semantic: slice 21 execution-frame guard report — capability dangerous pass 0/126, guard→sparse P 96.7% FA 6
Deterministic structural guard blocks every capability-question row (0/126 on
all three stress views), never blocks a true action (0/796), declines 196
malformed action rows as ambiguous, and grows no safety leak under
compose: sparse alone FA 9 (capQ 1) → guard→sparse FA 6 (capQ 0).
Execution eligibility recorded as a deterministic policy boundary, separate
from semantic routing and capability selection; next slice returns to the
coarse non-action router instead of another pragmatics training run.

Task/725.
2026-09-07 23:45:33 +04:00
claude fa98e4722e router/semantic: slice 21 deterministic execution-frame guard engine, fixtures, runner and emit step
The guard answers one question — may this utterance become an executable
action — as a three-way policy gate (permissive / blocked / ambiguous) and
never decides what the utterance is. Rules are the encoding of the measured
slice-20 dev-pool discriminators: 126 capability-question rows are 42/42/42
addressed / bare-ability / bare-future; 127 bare можешь+пожалуйста rows are
100% action; can-you-please is 100% action. Reuses the shipped prohibition
parser, morph finiteness and lexicon fillers; reason vocabulary is closed.

Slice 21 (task/725, brief after the accepted slice 20).
2026-09-07 23:43:47 +04:00
claude de1cb40456 router/semantic: slice 20 pretrained fine-tune report — in-pool best, LOFO 1.000
cointegrated/rubert-tiny fine-tuned end-to-end: pair ordering 0.933-0.970
(+0.56-0.78 margin, in-pool) beats every from-scratch config, but the
held-out capability-question family fires 126/126 as action on all 18
config/seeds (FA_rate 1.000, mean proba 0.86-0.99, family σ 0.0045). tiny2
ceiling (newer, larger vocab) does not move LOFO (1.000) and is worse
in-pool. sparse 0.667 remains the safety floor; supervised encoder scaling
ruled out for the generator boundary.
2026-09-07 21:40:53 +04:00
claude dc73fc4e31 router/semantic: slice 20 pretrained-rubert-tiny pragmatics specialist tooling + tokenizer audit gate
cmd/semantic-router-experiment/slice20_pretrained.py (516 lines, over
300-line hook): tokenize/grouped/lfo/metrics/runtime/onnx/ceiling
subcommands, MAX_LEN=25, lr grid 1e-5/2e-5/5e-5 x seeds 42/17/7, 4-thread
fp32, early stop on val PR-AUC. slice20_audit.py: WordPiece audit gate
(UNK rate 0.0026, seq p99 19, no loss above 96). artifacts under /tmp/mvn-s20/.
2026-09-07 21:40:49 +04:00
claude cf2af6ce1e router/semantic: slice 19 report — correct claim head and prose to the seed-stable fail-rate cut 2026-09-07 17:59:10 +04:00
claude 5397f09785 router/semantic: slice 19 sequence pragmatics specialist report — sequence beats sparse on capability holdout, caps near chance on seed-stable seeds 2026-09-07 17:58:16 +04:00
claude 7d31de5a6b router/semantic: slice 19 from-scratch sequence pragmatics specialist tooling + eval index 2026-09-07 17:58:09 +04:00
claude 20215650f2 router/semantic: slice 18 sparse action-gate report — aggregate boundary solved, capability boundary collapsed 2026-09-07 15:35:48 +04:00
claude f2b65cd5d2 router/semantic: slice 18 sparse lexical action-gate probe tooling + eval index 2026-09-07 15:35:41 +04:00
claude 9a25945567 router/semantic: slice 17 nonlinear e5 MLP probe — action gate experiment report 2026-09-07 15:23:46 +04:00
claude e80d45f4b3 router/semantic: slice 17 nonlinear MLP probe tooling + eval index 2026-09-07 15:23:39 +04:00
claude 4ed0238a25 router/semantic: add slice 16 diagnostic to evals index 2026-09-07 14:02:59 +04:00
claude 774c0660b0 router/semantic: slice 16 diagnostic — action/non-action boundary analysis
Experiments:
- Residual-only OOF safety metrics
- False-action decomposition by semantic family (7 families, 24 split groups)
- Paired e5 geometry (gap=0.010 between action and capability question)
- Binary action probe (PR-AUC=0.707, ROC-AUC=0.863)
- Cost-sensitive linear classification
- Expanded action-threshold sweep (7 thresholds)
- Voice-like punctuation stress evaluation
- e5 + tiny structural features
- Four-hypothesis comparison table

Verdict: e5 representation is the primary bottleneck. Action seeds and
capability questions are nearly indistinguishable in embedding space.
2026-09-07 14:02:00 +04:00
claude cbac8b9c4f router/semantic: experiment reports (slice 15)
- 2026-09-07-linear-e5-router-experiment: 136-example baseline (superseded)
- 2026-09-07-expanded-corpus-linear-head-experiment: 3025-example expanded (live)
- CLAUDE.md: index updates for both evals
2026-09-07 13:52:54 +04:00
claude ad3f2d6e99 router/semantic: contract tests for expanded corpus (slice 15)
Minimum corpus size check (>=1000), source count validation,
split-group leakage tests for contrast families.
2026-09-07 13:52:30 +04:00
claude 87411e5fb1 router/semantic: expanded v2 corpus (3025 examples, 93 seed families)
Merged corpus: 535 frozen holdout + 2490 development pool.
Route distribution: action=990, knowledge=880, memory_write=705,
system=226, uncertain=121, conversation=103.
Dataset hash: b27fd48f478ca477cab1e59773bb353a
2026-09-07 13:52:29 +04:00
claude 4bb7555459 router/semantic: corpus-factory deterministic surface generation (slice 15)
93 semantic seed families across 6 routes. Surface generators produce:
direct, polite, modal, first-person, reordered, English, question forms.
Each seed specifies verb forms, subjects, objects, values, tags.
2026-09-07 13:52:12 +04:00
claude 9df223907b router/semantic: sklearn grouped CV linear head experiment (slice 15)
Python experiment: 6-class logistic regression over frozen e5 embeddings,
grouped cross-validation, calibration, abstention curves, contrast-family
analysis, action-threshold sweep.
2026-09-07 13:52:12 +04:00
claude f8ec77de8a router/semantic: e5 embedding cache for linear head experiment (slice 15)
Go tool that computes e5-small embeddings for every corpus row,
assigns frozen/dev split and grouped CV folds, outputs JSON.
2026-09-07 13:51:54 +04:00
claude 6397ea1cc9 router/semantic: merge-corpus tool for frozen holdout preservation (slice 15)
Merges frozen holdout from v1 with v2 development pool, deduplicates,
preserves frozen entries, validates, writes merged corpus.
2026-09-07 13:51:53 +04:00
claude 07bfcea8cc router/semantic: semantic seed type and surface generator (slice 15) 2026-09-07 13:51:33 +04:00
claude 59a0a08d32 router/semantic: legacy baseline eval test (slice 13)
TestLegacyBaseline builds a minimal but complete router (stage-0
grammars, hash-embedder classifier seeded from models/seeds, deployed
confidence threshold) and runs it against the full 136-example corpus.

Reports macro F1, false-action rate, fast-path/residual/router-residual
accuracy, per-route P/R/F1, confusion matrix, and contrast family
breakdown (negation, question, reported_speech, quotation, hypothetical,
capability_question). Pre-route consumed cases are tracked separately
from the fast-path bucket.

The hash embedder is deterministic, so this baseline is reproducible.
The ONNX embedder would score higher; measure both before drawing
conclusions.
2026-09-07 02:09:02 +04:00
claude 1b8ae3c3e5 router/semantic: baseline scoring and contrast family analysis (slice 13)
ScoreLegacy runs the actual router cascade against the 136-example
corpus and produces per-route precision/recall/F1, confusion matrix,
false-action breakdown, and fast-path/residual/router-residual/pre-route
consumption counts. Pre-route consumed cases (command-prohibition grammar
matches at stage 0) are tracked separately — these never reach the
general cascade and should not be scored by the learned router.

ContrastFamilies splits the baseline report by transform tag (negation,
question, reported_speech, quotation, hypothetical, capability_question)
and reports per-family accuracy and false-action rate.

LegacyReport.String() renders the full baseline report with confusion
matrix and false-action case listing.
2026-09-07 02:08:51 +04:00
claude 56051e58c0 router/semantic: baseline types, test helpers, leakage test (slice 13)
Baseline types: LegacyRouter interface (satisfied by *router.Router),
LegacyCase with PrerouteConsumed flag for command-prohibition detection,
LegacyReport with fast-path/residual/router-residual/pre-route breakdowns,
ContrastFamilyReport for per-transform-family scoring.

Test helpers: buildSeededClassifier (hash embedder seeded from
models/seeds/*.txt), actVerbList, seed file loading.

TestContrastFamiliesShareSplitGroup verifies that all contrastive
variants of one base seed share exactly one SplitGroup, preventing
train/eval leakage across the contrast family split.
2026-09-07 02:08:40 +04:00
claude d63619bd0e router/semantic: frozen holdout split and grouped CV (slice 13)
Add FrozenHoldoutSplit with deterministic 15% ratio using dedicated hash
seed. Produces frozen/dev partition with SplitGroup-aware leakage
prevention — all contrastive variants of one seed stay in the same split.

Add GroupedCVFolds for k-fold grouped cross-validation on the development
pool. Each fold preserves split_group boundaries; every example appears
in exactly one eval set across all folds.

Tests verify determinism (same split → same hash), no split_group leakage
across frozen/dev, route coverage in both pools, fold completeness, and
grouped CV coverage.
2026-09-07 02:08:30 +04:00
claude fb9b719f0b router/semantic: corpus validation infrastructure (slice 13)
Add ValidateCorpus and CorpusStatsFrom for structural integrity checks
on the semantic route corpus. Validates total counts, route/source sums,
fast-path+residual partition, empty SourceID/SplitGroup, invalid routes,
duplicate identity, and conflicting labels on identical text.

CorpusStats provides deterministic dataset hash (SHA-256 of sorted texts,
first 16 bytes). ReproducibilityMeta in CorpusEnvelope records source
fixture hashes, generator version, split algorithm, and dataset hash.

TestCorpusValidation exercises the full validation pipeline.
2026-09-07 02:06:58 +04:00
claude 2b38390c87 router/semantic: tests for eval scoring and shadow harness (slice 12) 2026-09-07 01:43:30 +04:00
claude 0ff4562e60 router/semantic: tests for contrastive safety transforms (slice 12) 2026-09-07 01:43:26 +04:00
claude bb7c460d12 router/semantic: tests for contract, mapping, corpus, and split (slice 12) 2026-09-07 01:43:23 +04:00
claude dd98da5c42 router/semantic: shadow harness for experiment observation (slice 12)
ShadowHarness records turns where both the legacy router and the
experiment model produce a decision. Reports agreement/disagreement
split by fast-path vs residual. No action, clarification, capability
selection, or reply depends on the shadow result.
2026-09-07 01:42:39 +04:00
claude 9e2abea16b router/semantic: eval metrics for frozen evaluation set (slice 12)
EvalReport with macro F1, per-route precision/recall/F1, confusion
matrix, false-action rate, and fast-path vs residual breakdown.
ScoreEval runs a SemanticRouter against a frozen eval set and produces
all metrics needed for promotion decisions.
2026-09-07 01:42:36 +04:00
claude b481792c02 router/semantic: contrastive safety transforms and split-by-family (slice 12)
Six deterministic transforms (negation, question, reported speech,
quotation, hypothetical, capability question) applied to action-route
seeds. Each transform determines the expected class explicitly — no
model guessing labels. SplitByFamily uses hash-based bucketing to keep
paraphrases in the same split.
2026-09-07 01:42:27 +04:00
claude 5865699a5a router/semantic: corpus format and v1 dataset (slice 12)
136 examples with provenance from ru_routing_v1.json, tagged
fast_path_resolved vs residual. RouteExample carries source, source_id,
split_group for traceability. Split-by-family prevents paraphrase
leakage across train/eval.
2026-09-07 01:42:22 +04:00
claude b6a21c17eb router/semantic: coarse-route contract, interface, and legacy mapping (slice 12)
Defines the six-class SemanticRoute type (conversation, knowledge,
action, memory_write, system, uncertain), the SemanticRouteDecision
output, the SemanticRouter interface, and the deterministic
Intent→SemanticRoute mapping from the current seven-intent cascade.
2026-09-07 01:42:17 +04:00
claude 494c719a5d router: update all Route callers for NormalizedInput + add invariant tests (slice 11)
Migrate 54 test call sites to construct NormalizedInput{Text: ...}.
Add invariant tests:
- TestNormalizedInputReachesRouteIntact: ingress NormalizedInput reaches Route
- TestTryFastPathReceivesMatchText: TryFastPath gets the same input
- TestMatchTextDoesNotChangeRouting: same Text + different MatchText → same Decision
- TestDecisionUtteranceEqualsInputText: Decision.Utterance == input.Text
2026-09-07 01:32:49 +04:00
claude dad3cd0738 router: Router.Route consumes NormalizedInput directly (slice 11)
Change Route from (ctx, utterance string, now) to (ctx, input NormalizedInput, now).
The ingress-constructed NormalizedInput now reaches the cascade intact — no
reconstruction downstream. TryFastPath receives the same input, not a rebuilt one.
All callers (production, eval framework, tests) updated to construct NormalizedInput.
2026-09-07 01:32:45 +04:00
claude a1e4743492 router: add TryFastPath tests and Route identity invariant (slice 10) 2026-09-07 01:24:28 +04:00
claude 2ef470c2ba router: add TryFastPath with stage-0 grammar evaluation (slice 10) 2026-09-07 01:24:25 +04:00
claude 80f807d2ad router: extract stage-0 grammars into TryFastPath (slice 10) 2026-09-07 01:24:19 +04:00
claude e9049ad27e router: add NormalizeMatchText tests and property invariants (slice 9b)
17 tests covering: whitespace collapse, case folding, NFKC normalization,
punctuation preservation, mixed-script identifiers, ё passthrough, and
property invariants (idempotent, deterministic, never removes punctuation/
wake words/numbers, never mutates original).

Test-only foldYo helper proves ё→е is lossy (всё → все) without
exposing an unused production function.

Vikunja: #725
2026-09-06 22:33:27 +04:00
claude 431e052e6f router: add MatchText to NormalizedInput and NormalizeMatchText function (slice 9a)
Add MatchText field to NormalizedInput — a lossy lexical matching view
derived from ingress text: TrimSpace → NFKC → lowercase → collapse
Unicode whitespace. Does NOT fold ё→е, strip punctuation, strip wake
words, rewrite numbers, or invoke morphology.

Both ingress sites (voice + text) construct MatchText at entry. No
existing consumer reads MatchText yet — it is dark data for future
opt-in migration.

Vikunja: #725
2026-09-06 22:32:51 +04:00
claude 0a2e194e76 router: migrate refusesCommand and ActHasEntityTarget to CapabilitySelection (slice 6c)
Migrate the two remaining action-routing consumers from compatibility
Decision.Slings fields to authoritative Decision.CapabilitySelection:

- refusesCommand: reads CapabilitySelection.Fn instead of Slots.Fn
- ActHasEntityTarget: reads CapabilitySelection.Resolved and
  CapabilitySelection.Args instead of Slots.HasFn and Slots.Args

Slots.Text remains the source for entity text when positional args do
not contain the target (unchanged).

Regression tests prove:
- prohibited sentinel preserved byte-for-byte through SelectCapability
- blanked Slots.Fn/Args/HasFn do not affect migrated consumers
- Praxis/Hexis entity-target routing unchanged
- stage-0 deterministic act unchanged
- classifier/extractor act unchanged

do not remove the compatibility mirrors yet.
2026-09-06 21:43:14 +04:00
claude 18900dd613 docs: add slice 6b capability selection boundary report 2026-09-06 21:02:21 +04:00
claude 2d42e98871 router: update ActionCandidate tests for CapabilitySelection (slice 6b)
Set CapabilitySelection on decisions that have Slots.HasFn=true, so
ResolveActionCandidate reads from the authoritative record. Backward
compatibility tests verify that decisions without CapabilitySelection
still resolve via Slots.HasFn.
2026-09-06 21:02:17 +04:00
claude 4a17821b06 router: ResolveActionCandidate reads CapabilitySelection first (slice 6b)
The candidate now receives Fn/Args from CapabilitySelection (the
authoritative record) rather than from Decision.Slots.HasFn. Backward
compatibility: decisions with Slots.HasFn but no CapabilitySelection
(tests, rebuilt decisions) still resolve via the compatibility path.
2026-09-06 21:02:13 +04:00
claude 4b24fbad98 router: add CapabilitySelection field to Decision (slice 6b)
Authoritative record of which executable capability matched, separate
from Decision.Intent (what kind of turn) and ActionCandidate (downstream
action artifact). Decision.Slots.Fn/Args/HasFn remain as compatibility
representations populated from this selection.
2026-09-06 21:02:09 +04:00
claude 8d83c66d11 router: wire SelectCapability into Router.Route pipeline (slice 6b)
Call SelectCapability after each cascade path (grammar, heads, LLM,
classifier) and propagate the result via applyCapabilityToSlots.

Remove LLM text capability backfill from fillSlots — SelectCapability
now owns that path. fillMatchedSlots retains raw extractor capability
extraction for backward compatibility with stage-0 grammars.

gateLLMDecision now reads CapabilitySelection.Resolved instead of
Slots.HasFn for the act-intent confidence thinning check.
2026-09-06 21:02:05 +04:00
claude a6a62a2d56 router: introduce CapabilitySelection type and SelectCapability stage (slice 6b)
Add the explicit capability-selection boundary between route resolution
and action candidate production. SelectCapability is the single entry
point for selecting which executable capability matched an IntentAct turn.

Three input kinds: raw, llm_text, deterministic. Decision.CapabilitySelection
is the authoritative record; Decision.Slots.Fn/Args/HasFn remain as
compatibility representations populated from the selection.
2026-09-06 21:01:59 +04:00
claude 05f791735d router: add diagnostic resolution method matrix tests
Run routing and ecosystem fixtures through the baseline router and
report which component selected the exact function for every IntentAct
case. Shadow matcher comparison confirms zero disagreements.
2026-09-06 13:51:37 +04:00
claude bdd79ad585 mavend: carry ResolvedBy through dialogue Slots bridge
Add ResolvedBy to dialogue.Slots and the toDialogueSlots/
applyDialogueSlots converters. Skip reflect-type check for this field
in parity test (dialogue cannot import router: import cycle).
2026-09-06 13:50:30 +04:00
claude 66f06796cb router: add provenance tests for ActionResolutionMethod
Pin each path: grammar_fixed, grammar_matcher, extractor_raw,
extractor_llm_text, fallback_matcher. Verify unresolved has empty
ResolvedBy. Verify fn/args remain byte-for-byte identical.
2026-09-06 13:50:22 +04:00
claude 06adc4702d router: set ResolvedBy at each function selection point
Assign provenance where the exact fn is produced:
- grammar_fixed: praxis/task-status grammars hardcode fn
- grammar_matcher: wakeword-act grammar invokes ActMatcher
- extractor_raw: Extractor.Extract matches over raw utterance
- extractor_llm_text: fillSlots LLM backfill matches cleaned text
- fallback_matcher: ResolveActionCandidate runs the fallback matcher

ResolveActionCandidate propagates Slots.ResolvedBy into
ActionCandidate.ResolvedBy. No selection behavior changes.
2026-09-06 13:50:03 +04:00
claude 1d02ba8936 router: add ActionResolutionMethod type and ResolvedBy field to Slots
Five disjoint values tracking which component selected the exact function:
grammar_fixed, grammar_matcher, extractor_raw, extractor_llm_text,
fallback_matcher. Slots.ResolvedBy carries provenance at the selection
point.
2026-09-06 13:49:50 +04:00
claude 66c578a6f4 router: introduce typed ActionValidationStatus boundary (slice 5)
Introduce ActionValidationStatus enum (valid, unresolved, missing_argument,
invalid_argument, ambiguous_target) as the typed classification of validation
outcomes. ActionValidationResult now carries Status instead of boolean flags.

Backward-compatible: Unresolved() and Valid() methods preserved on the result.
Existing validation behavior unchanged: only blank Fn produces invalid_argument.
All downstream behavior (proposeGap, confirmation, task_status, praxis, hexis)
unchanged.

Tests added for all five status values, backward compatibility, and the full
validation → execution boundary.
2026-09-06 13:07:08 +04:00
claude 356766bce1 mavend: centralize action validation boundary (slice 4) 2026-09-06 12:53:54 +04:00
claude 6a402bf556 docs: add action-resolution boundary slice report 2026-09-05 21:51:43 +04:00
claude f6d7b05161 mavend: add action-resolution regression tests
Eight integration tests pinning the action-resolution boundary:

1. TestActRouteSource_NoMatcherInvoke — HasFn=true, route-sourced
2. TestActMatcherSource_FallbackMatch — no Fn, matcher resolves
3. TestActMatcherMiss_ProposeGap — matcher miss → propose-gap
4. TestActDestructive_ConfirmationUnchanged — destructive → confirm
5. TestActTaskStatus_InterceptUnchanged — task-status intercepted
6. TestActStage0_SameResult — stage-0 act executes same tool
7. TestActLearnedRouter_NoFn_FallbackMatch — LLM no Fn → matcher
8. TestResolveAction_CandidateSource_Verified — verifies all paths

All 8 pass. All existing tests pass.
2026-09-05 21:50:50 +04:00
claude 025f81e961 mavend: wire resolveAction into actionAct
Daemon half of the action-resolution boundary:

- Add resolveAction wrapper: delegates to ResolveActionCandidate,
  records outcome in the decision trace (action-resolve:route/matcher)
- Refactor actionAct: remove matcher call, consume candidate, write
  resolved values back into Slots for downstream branches
- Add 8 integration tests pinning all required scenarios:
  route-sourced, matcher-sourced, matcher miss, destructive confirm,
  task-status intercept, stage-0, learned-router, alias match

All existing tests pass. Execution/risk/confirmation unchanged.
2026-09-05 21:50:33 +04:00
claude 064747f192 router: add ActionCandidate type and ResolveActionCandidate
Introduce the typed boundary between routing and action resolution:

- ActionCandidate: Fn, Args, Source (route|matcher), Producer, Confidence
- ResolveActionCandidate(dec, m): standalone function usable by both
  the daemon and the eval harness
- Update eval harness Reach() to use ResolveActionCandidate instead of
  duplicating the matcher fallback logic

This is the routing-side half of the action-resolution boundary.
The daemon integration follows in the next commit.
2026-09-05 21:49:25 +04:00
claude a55d90954a router: add boundary tests for typed ingress and route producer
12 focused tests proving the first slice properties:
- text and voice enter equivalent typed turn input after stt
- stage-0 outputs remain identical with grammar producer
- classifier floor sets its producer
- clarification carries the classifier producer
- pre-route claims produce no route producer
- route producer appears on the decision record
- input source is preserved on the decision record
2026-09-05 20:16:57 +04:00
claude 87a3b163e7 router: introduce typed ingress boundary and route producer observability
First behavior-preserving slice of the Maven redesign. Establishes
explicit ingress/routing boundaries and enough observability to refactor
later without changing current routing, action, clarification, or
execution semantics.

Types introduced:
- NormalizedInput (internal/router/source.go): Text + InputSource,
  the typed ingress boundary replacing raw string at the turn entry.
- InputSource (internal/router/source.go): channel provenance enum
  (tap:voice, tap:text). Reuses the existing turnSource distinction.
- RouteProducer (internal/router/intent.go): which cascade stage
  produced the decision (grammar, heads, llm, classifier).

Changes:
- Decision carries a Producer RouteProducer field, set at each cascade
  stage (grammar, heads, LLM, classifier).
- turnRoute carries NormalizedInput instead of bare text string.
- runTurn takes NormalizedInput instead of (text, src).
- decision.Record carries InputSource and RouteProducer for
  observability; RoutingTrace persists route_producer (migration #27).
- turnSource is now a type alias for router.InputSource.

Behavior preserved:
- Stage-0 grammars unchanged: same order, same matching, same confidence.
- Cascade fallthrough order unchanged (grammar → heads → llm → classifier).
- Clarification behavior unchanged.
- Action dispatch unchanged.
- No new linguistic normalization.
2026-09-05 20:16:40 +04:00
claude 2f338a1ab6 Hide the relation filters in the capability views and render inline code (V-725)
Two defects found by screenshotting the built page under headless chromium,
which is the only way to see either.

The six relation filters and the component-type legend do nothing in views 6 and
7. Leaving them on screen reads as controls that are broken.

The ledger carries markdown inline code, because docs/spec.md does. The side
panel printed the backticks literally beside every path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 13:21:21 +04:00
claude be062b2d48 Add the capabilities and invariants views to the viewer (V-725)
Session 3, and the end of the plan.

View 6 is the matrix: 51 capabilities against designed, code_present, wired,
configured, deployed, reachable and verified, grouped by spec section or by
domain. Clicking a row opens the definition of done with every verdict, its
reason, its detail and its evidence paths, the components that carry the
capability, the blockers and the product questions it waits on.

View 7 is the twelve invariants. Each shows three things apart: target is
whether the rule is written down, implementation is the status of the
participating components, and runtime is what the probe run observed for the
capabilities it touches. Component and capability chips cross-link into the
other views.

invariants.yaml is the machine-readable half of invariants.md. The two exist
separately so the viewer can read one and a person can read the other, and
build_ledger.py refuses to build when they disagree: a missing heading, a count
mismatch, an unknown capability or component, or an unresolved invariant with no
product question.

build_viewer.py inlines ledger.yaml and invariants.yaml and derives nothing. The
ledger's build is the only thing allowed to decide a dimension.

check_viewer.js is the viewer's only check. A TypeError in a renderer shows as a
blank panel and not as an error, so it runs all seven views, all three flows,
all 51 capability panels and all 160 component panels against a DOM stub, and
fails on a panel that comes back thin. render.sh calls it and skips it with a
message when node is absent.

--no-verify: the template and the smoke test are 320 non-markdown lines.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 13:15:24 +04:00
claude 7f804b84e7 Declare the two generated doc tiers and index their evals (V-725)
docs/CLAUDE.md defined four tiers and docs/capabilities/ and docs/architecture/
were neither of them. Both are now declared as generated: rebuilt from a source,
never corrected in place. A wrong row in either is a bug in the generator or in
one of its hand-written inputs, and editing the output makes the next rebuild
silently undo the fix.

The eval index gains the 2026-08-19 CPT+SFT measurement and the 2026-08-26
baseline, and marks the 2026-08-13 capability audit superseded. The 2026-08-19
file lands with them: it is the direct evidence that the deployed
maven-instruct-b2 routes better than Qwen3-1.7B and cannot hold a Russian
sentence, which is the first item on the gaps.md priority list.

Its header records Vikunja as returning 503. That reading was wrong and the
2026-08-26 baseline says so, but a dated eval is not edited after the day.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:45:54 +04:00
claude bae81b66c8 Track the architecture observation and its inventory (V-725)
docs/capabilities/build_ledger.py reads the component statuses out of
maven-architecture.json, so the whole implementation half of the ledger fails to
build on a clone that does not have it. It has to be tracked.

What lands: the five generator scripts, the viewer template, findings.md, the
README and the seven .mmd diagram sources, plus the inventory JSON itself.
verify_anchors.py resolves 681 of 692 claimed symbols to path:line and exits
non-zero on a miss, 11 skipped as config keys. That proves an identifier sits on
a line and nothing more. Writing the responsibility field caught 29 symbols
filed under the wrong component and 7 names invented outright, and a later
refutation pass caught 4 wrong readings on top of that.

What does not land, and is now gitignored: index.html at 836 KB of inlined JSON
and SVG, anchors.md, architecture-evidence.txt, tree.txt, the redacted compose
file, the rendered SVGs and maven-evidence.zip. All of them rebuild with
pack_evidence.sh.

render.sh is the only syntax check this repo has for a .mmd, and it found two
real parse errors on its first run.

--no-verify: 4,900 non-markdown lines. The inventory and its generator are one
artifact and neither is readable without the other.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:45:20 +04:00
claude 8153e5eaa5 Classify every gap and rank the work (V-725)
Session 2 step 3, and the end of the explanation half.

gaps.md compares responsibilities and never package names. Eight classes. The
four capability classes are derived from the ledger's gap_class field and
rebuild with build_ledger.py. The four architecture classes are read from
findings.md and invariants.md, and every entry names the capability or invariant
it affects. An entry naming neither is marked non-blocking cleanup in those
words, which is the whole of class 8 and its eleven rows.

Of 46 v1 capabilities: 5 missing, 21 partial and reachable, 9 built and
unreachable, 11 reachable and unverified.

The nine unreachable ones are seven config blocks and two compose entries. Not
one is a code defect.

The ranked list puts phrasing first: speak-as-herself fails all three criteria,
and everything that asks the resident model to write a Russian sentence inherits
that. His own name not being stored is second. Nine capabilities one config
change from reachable is fourth, and it is the highest ratio of capability to
work in the list.

Items 10, 12 and 13 stall on unresolved invariants and are the owner's call, not
work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:44:15 +04:00
claude 41c97bba8d Write the twelve cross-cutting invariants (V-725)
Session 2 step 2. docs/spec.md states 51 capabilities one at a time. Twelve
rules run across all of them and no DoD states any of them, so breaking one
breaks many capabilities at once without producing a failing criterion.

Each is marked explicit, implied or unresolved, with evidence. Nothing wanted is
invented where the sources are silent.

Four are unresolved and belong to the owner rather than to a commit: authority
and confirmation, learning from outcomes, capability composition, and whether a
held nudge has a shelf life. Two of the three questions the freeze was called to
answer show up here as invariants 8 and 11.

Privacy boundaries and proactive attention are the two best-specified rules and
neither showed a live defect. Authority is the largest hole: internal/auth
answers who may carry what authority and does not bind the turn path,
internal/tool answers what effect an act has and is not keyed on the reach, and
praxisItemAction.handle has no gate at all.

The one file in docs/capabilities/ that is hand-written rather than generated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:40:52 +04:00
claude f9b0a96d9d Document the seven dimensions in the directory README (V-725)
The rebuild section named the new inputs and nothing said what the columns
mean or how a partial arises.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:38:18 +04:00
claude 40ec0c0d4b Map every capability to its components in seven dimensions (V-725)
Session 2 step 1. Implementation status was the missing half: the ledger said
what should happen and what happened, and nothing said how much is built.

Never one implemented boolean. designed, code_present, wired, configured,
deployed, reachable and verified are separate, because coded and unwired, wired
and unconfigured, and configured and undeployed are three different pieces of
work.

The six build dimensions derive from the status field of every component the
capability maps to, rolled up as all yes, none no, otherwise partial. The
statuses come from docs/architecture/maven-architecture.json, which read them
from code, config and compose. verified comes from the criteria verdicts.

implementation.yaml is the mapping and is the judgment call. Shared
infrastructure is deliberately unmapped: putting core.reactive_handler on all 51
rows would give them one status and say nothing.

Of 51 capabilities, 45 have code and 33 are reachable. 22 are spec-only, with no
living doc owning the subsystem.

The build now reports what it cannot reconcile. learning-the-style has no
component and still scores a pass, because its passing criterion is negative and
absence satisfies it. Sixteen components serve no capability, ten of them the
shared infrastructure excluded on purpose, and the rest are core.q.habits,
core.q.money, ext.zenmoney, router.claim and router.modes.

--no-verify: the regenerated ledger is 500 lines of derived output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:37:37 +04:00
claude af6e6c9979 Give the plan its task id (V-725)
Filed after the fact, so the plan and the index both said it had none. The
Vikunja task carries the session-1 result and what sessions 2 and 3 still owe.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:27:54 +04:00
claude 3cced9a2e9 Freeze the empirical baseline as a dated eval (V-725)
What the deployed Maven did when it was asked, measured 2026-08-26 against
master 5cae33a plus the uncommitted deploy/mavend.json model switch. Frozen on
the day and not edited after it.

The number is attributable to the deployed resident model,
maven-instruct-b2-Q4_K_XL, not the Qwen3-1.7B that CLAUDE.md names. What comes
closest to working is what never asks that model to write a Russian sentence.

make test is green throughout. It is also green with the four TestONNX
measurements silently skipped, because the recipe does not set MAVEN_ONNX_LIB.

Every verdict was audited by an independent pass told to refute it, one auditor
per spec section, with pass attacked hardest. The corrections moved the tally by
nine: four passes withdrawn, five criteria filed untested turned out already
settled. The section on how the verdicts were checked names the mistakes, so the
next session does not have to trust that this one got it right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:26:34 +04:00
claude bc1ef0f57f Score every criterion and build the ledger (V-725)
verdicts.json carries one verdict per criterion id. ledger.yaml is what
build_ledger.py produces from docs/spec.md, domains.yaml and those verdicts.

146 v1 criteria: 26 pass, 50 fail, 15 blocked, 51 untested, 4 unknown. No
capability passes all of its own criteria. Six fail every one: speak as herself,
weather, wake word, summaries, webhooks, command chaining.

Only a live verdict sets pass. Every verified cell cites
docs/evals/2026-08-26-capability-baseline.md by path and section, and the
generator refuses to build if either does not resolve.

Both files are generated. Rebuild rather than hand-edit.

--no-verify: 3,386 non-markdown lines, all of it generated output that cannot
split into reviewable ideas.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:26:33 +04:00
claude f002ce0e9c Add the probe harness and the raw output of the field run (V-725)
probes_field.json is 25 multi-turn probes drawn from the owner's real week.
run_probes.py drives them through the deployed stack: POST /api/chat on
127.0.0.1:9201, which runs a real turn through the pre-route ladder, the stage 0
grammars, the routing heads, the resident model, the query walk, the act path
and the phraser. Readback is cmd/e2eprobe over the mavend IPC socket, never the
plaintext sqlite copy in /dev/shm and never the mavweb HTML pages.

store_counts.py reads row counts per store over IPC, before and after.

out/ holds what the run produced. field.contaminated.jsonl is the discarded
first run: mavweb hardcodes one conversation id for the whole web reach, so a
clarify parked by one probe was still parked for the next.

field.transcript.tsv is the evidence for the baseline and is not summarised
anywhere else. The store it came from was wiped afterwards.

--no-verify: 326 non-markdown lines of new harness plus its captured output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:26:32 +04:00
claude 3adfc3e0f9 Add the ledger generator, its domain axis and the directory README (V-725)
build_ledger.py extracts 51 capabilities and 156 DoD criteria from docs/spec.md
and joins them with domains.yaml and verdicts.json. The generator is also the
checker: it exits non-zero on a capability with no DoD criteria, no State line,
no domain or more than two, an unknown domain, a criterion id collision, a
domains.yaml or verdicts.json row naming something that does not exist, a
verdict word outside the five, and a reason outside the plan's list. It caught
the domain reconciler silently dropping recall from its 51.

It also refuses an evidence path that does not resolve, a section heading absent
from the file it names, a pass whose reason is not passes, and a fail resting on
no runtime proof. Sixteen verdicts had cited a section of the eval that did not
exist.

domains.yaml is the one judgment call in the extraction and is hand-edited.

--no-verify: 584 non-markdown lines, all of them new files. The generator and
the domain table it reads are one reviewable idea and splitting them leaves
neither readable alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:26:30 +04:00
claude b6666196c1 Plan the capability ledger and the empirical baseline (V-725)
Three sessions on one causal order: the spec says what should happen, the
empirical run says what actually happens, code and architecture explain why,
priority says what to fix. The predecessor audit had a green suite while 22 of
39 capabilities were not live, which is the failure mode this order exists to
stop.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:26:28 +04:00
claude 5cae33a517 Split honesty into three milestones, defer five capabilities (V-719)
Owner's call. M1 is the turn path, M2 is memory he cannot correct, M3 is
step-up. They were one milestone and are three jobs: M1 and M2 touch
different code and owe different docs, and step-up is configuration, not
honesty. Nine milestones now.

Speaker recognition, smart home, bluetooth, model swap and self-update
move past v1. Bluetooth was on the v1 list and comes off it: no bluez on
the box. Their spec entries keep their DoD.

--no-verify: committing on master by the owner's call this session.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 18:40:45 +04:00
claude 02e3d27aa9 Order the spec into seven milestones (V-719)
Ordered by what makes her untrustworthy if it ships late, not by code
work: the audit ruled that out, since none of the four broken
capabilities is a code defect. Honesty, then the config-and-data four,
then voice, then proactive delivery, then breadth, then email and
calendar behind their product decision, then the undesigned seven.

Doc gaps and missing scenarios bind every milestone rather than forming
one, so they cannot collect at the end.

--no-verify: committing on master by the owner's call this session.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 18:21:36 +04:00
claude 78a9c61acb Write the capability spec with a DoD for each (V-719)
51 capabilities: the 39 rows from the 2026-08-13 audit plus 12 v1 items
that had no audit row. Each entry carries a state reference to the living
doc that owns it, a plain DoD list observable on the running box, and the
scenario file that scopes it.

Applying "state is a reference" found 17 capabilities with no living doc.
Only 5 of 51 entries cite a scenario that exists.

--no-verify: committing on master by the owner's call this session.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 18:21:36 +04:00
claude 231248a990 Freeze the capability audit as a dated baseline (V-719)
39 capabilities read off the running five-container stack on 2026-08-13,
not off the code. 17 live, 9 partial, 4 broken, 9 off. The owner's
corrections are applied: speech in, speech out and wake word are live, and
he proved all three by speaking to her. The voice reach stays broken,
because reaching her by speaking is a pull and a proactive message needs a
session to push into.

The claim the spec has to be written against: none of the four broken
capabilities is a code defect. Weather has no config block, Nexus has no
data, the voice reach has no listener, step-up has no WebAuthn credential.
The race suite was green during a run where 22 of 39 capabilities were not
live, so no definition of done that a test suite can score is worth
writing. Every criterion has to be observable on the box.

The task id is unfiled: Vikunja answered 503 for the whole session.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 18:03:50 +04:00
claude db50c34c23 Record the recall measurement and the two subsystem contracts (V-719)
docs/evals/2026-08-15-locative-answerability-verifier.md rules the
resident model out as a recall answerability verifier. Its constrained
output was syntactically reliable and neither semantically reliable nor
isolated from instructions inside stored memory: five false accepts out of
32 held-out cases, two of them prompt injections carried in the memory
text, all five identical across three fixed-seed repeats.

design.md carries the reminder row as it now is, one-shot or recurring,
with the outbox and the cancellation invariants. routing.md carries the
new stage 0 frames and the cancellation rung. deployment.md carries the
/reminders contract. The assistant_workday scenario exercises the turn
sequence end to end.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 17:19:38 +04:00
claude 1b5d35ad37 Make /reminders the complete cancellation surface (V-719)
GET merges every pending reminder, ordered by next fire, with the latest
50 rows and no duplicates, so old pending work cannot fall off a history
window. Recurring rows show their next fire and cron expression.

A pending row carries an inline cancel POST. Success answers 303 so a
refresh cannot repeat the mutation. A missing id is 404, a terminal or
in-flight row is 409, a malformed id or action is 400, and a transport
failure keeps the sanitized 502 problem response.

The page calls the same CoreAPI methods the voice path uses rather than
opening a second route into the store.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 17:19:38 +04:00
claude 85a3397bf4 Cancel a reminder by voice, and honour a refusal (V-719)
reminder_cancel.go is a stateful pre-route resolver ahead of a parked
clarification and the statistical cascade. It accepts only an addressed
command-position imperative plus the reminder or alarm noun, so questions,
reported speech, past-tense reports and prohibitions establish no mutation
authority. Subject terms keep negation and quantity, and a parsed time
passes the same resolved-hour gate as capture.

One match cancels through the typed IPC method. Several are stored as
session candidates in the spoken order, capped at five, and only a whole
affirmative ordinal consumes that list: re-querying on the follow-up would
let a state change move the ordinal underneath him. No match, an unread
time, a spent ordinal and an ambiguous delivery result are all explicit
no-ops.

command_prohibition.go is the first mutation boundary in a turn. A direct
prohibition clears the three confirmation slots under their shared mutex,
so a later bare "да" cannot revive authority he has just revoked. A parked
clarify question is not authority and survives, suspended and repeated.
refusesCommand is the same belt at the executor entry points, checked
against the original utterance so a model rewriting Slots.Text cannot get
around it.

The rung is named in preRouteLadder, so /trace records whether it won or
declined on every surface.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 17:19:25 +04:00
claude 0b057df2a3 Give reminder cancellation its own store and IPC path (V-719)
CancelReminder replaces the cancelled half of MarkReminder, which stays
delivery-only. Cancellation has to win against the start of an external
send, so it refuses when the occurrence has a pending, sent or unknown
outbox row, and clears the delivery group inside the same transaction.
BeginDeliveryAttempt takes the mirror lock for reminder sends, so no
interleaving lets both operations report success.

Cancelling one member of a collapsed catch-up bundle invalidates the
cached phrase on every pending sibling; a later retry would otherwise keep
saying "three reminders" after one was removed.

Legacy rows carry the empty delivery group from migration 25, so they only
count as this occurrence when they began at or after its next-fire
boundary. Without that bound one old success would make a recurring series
permanently uncancellable.

ListPendingReminders returns cancellable rows in firing order, with no
limit by default, because spoken resolution must not miss an old reminder
that newer fired history pushed out of ListReminders' window.

Cancellation is ordinary authenticated write authority: it prevents a
future send and cannot create one. cmd/e2eprobe drives both from outside.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 17:19:13 +04:00
claude 5b0b29dfad Make locative recall prove identity, not overlap (V-719)
The spare-key note scored 0.832 to 0.867 against a spare passport, a blue
shirt, a blue document box and a car key. Score and margin cannot separate
those: the right note runs 0.817 to 0.892 and the silent cases 0.787 to
0.874, so the ranges overlap and structure has to decide.

RecallAllowed now takes two structural facts from the router. A locative
question must corroborate every identity term against the candidate's
subject, read up to its first dictionary-proven verb, so a location object
in the note cannot answer for the thing being located. A turn that is not
question-shaped needs a named shared topic even when it ends in '?', which
is what "я отменил напоминание про молоко" lacked when it recalled an
unrelated note at 0.825 with no runner-up to fail the margin.

query_min_score moves 0.55 to 0.80 for tokenizer rev 2. The held-out
fixture answers 14/27 real recalls and 0/14 false ones.

LocativeAnswerVerifier is the resident-model second opinion, kept behind
the deterministic gate and wired into nothing. The measurement that says
why is docs/evals/2026-08-15-locative-answerability-verifier.md.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 17:19:01 +04:00
claude a97764c5f7 Add seven stage 0 frames and tighten three more (V-720)
MavenHelpGrammar keeps "как отменить напоминание" on SourceSelf, where the
answer names the command Maven accepts, instead of leaking to search.
PublicCurrentVersionGrammar anchors an explicitly current release on
SourceWorld and declines first-person ownership.

AmbiguousFragmentGrammar refuses filler plus an unresolved demonstrative
rather than letting a statistical head invent context.
ImplicitElapsedQueryGrammar reads Russian question word order in "давно я
не тренировался" as recall; the declarative order stays a statement.
ReminderCancellationReportGrammar keeps "я отменил напоминание" in the
non-mutating chat lane.

CommandProhibitionGrammar routes a direct negative command to a sentinel
fn that can never collide with an enabled tool. ActHasEntityTarget stops a
bare verb or a demonstrative-only tail from crossing into Nexus.

Praxis attention now accepts "что там с X" for the four service names only.
taskstatus separates command mood from result words so a first-person
report cannot mutate the board. question.go exports the open-question and
locative shapes the recall gate reads.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 17:18:48 +04:00
claude 9944ec8c58 Add the closed classes the new stage 0 frames need (V-720)
Reminder cancel verbs, cancel reports, reminder nouns and their frame,
unresolved references, current-version nouns and markers, personal
possessives, ecosystem service names and task done/drop command and state
splits. Each set carries the note that says how a caller must match it.

self_state_verbs is the head list a prohibition may not take: "ну не знаю"
answers a parked question and must not be consumed as "do not do that".

TaskDoneWords goes: TaskDoneCommands and TaskDoneStates replaced it, and
the deadcode gate fails on an accessor nobody calls.

--no-verify: master is the working branch this session by the owner's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-15 17:18:37 +04:00
claude 98ab646206 Make the hash-floor gate deterministic (V-718)
Owner explicitly requested direct commits to master. Keep startup cost benchmarked without turning ambient race/coverage load into a correctness failure; record live reminder proof, stale-task reconciliation, and the temporary delegation quota caveat.
2026-08-15 02:15:51 +04:00
263 changed files with 91442 additions and 621 deletions
+20 -12
View File
@@ -11,14 +11,18 @@ started with an agent that inferred the goal instead of stating it back.
## 0. Get on the branch
```sh
task start <vikunja-id>
task start <vikunja-id> # with an id
git checkout -b task/<slug> # without one
```
`~/.local/bin/task` owns the branch, the identity and the PR. It cuts
`task/<id>-<slug>` off `origin/master` and sets the commit author to the `claude`
gitea user. It writes `TASK.md` from the Vikunja task, and pulls any waiting
An id is optional (owner's call, 2026-08-25). With one, `~/.local/bin/task` owns
the branch, the identity and the PR. It cuts `task/<id>-<slug>` off
`origin/master` and sets the commit author to the `claude` gitea user. It writes `TASK.md` from the Vikunja task, and pulls any waiting
review comments into `.task/review-comments.md`. Do not hand-roll any of that.
Without an id, branch by hand and skip `TASK.md`. The user's own brief is then
the goal, and step 4 restates it back to him instead.
`TASK.md` is the brief and it is immutable. If it says a PR already exists, this
is a review-fix session and not new work. Read the comments first.
@@ -34,9 +38,12 @@ If there is no handoff, that is normal. It means the last session closed clean.
In this order, and stop as soon as you have enough:
- The Vikunja task, by id. Project Maven is ID 2, MCP at `http://localhost:9100/mcp`.
The task description and its comments hold the goal, the constraints, and the
assumption ledger. This outranks the handoff on every conflict.
- The Vikunja task, if there is one. Project Maven is ID 2, MCP at
`http://localhost:9100/mcp`, reachable from workpc only through
`ssh -N -f -L 9100:127.0.0.1:9100 kami@192.168.1.104`. A refused connection is
the missing tunnel, not an outage. The task description and its comments hold
the goal, the constraints, and the assumption ledger. This outranks the handoff
on every conflict.
- `CLAUDE.md`, the section that covers the area you are about to touch.
- The one file under `docs/` that owns the area. Check its `Last verified` line.
If the sha is behind the code you are reading, say so in step 4 and trust the code.
@@ -44,8 +51,8 @@ In this order, and stop as soon as you have enough:
Do not read the dated files under `docs/evals/`. They are measurements from one day,
never updated. Read one only when you need the number it recorded.
If no task id is known, ask for one before doing anything else. Work without a task
is work nobody can resume.
With no task id, do not ask for one and do not stall. State it in step 4 as
`Task: unfiled` and carry on.
## 3. Look at the ground
@@ -58,7 +65,7 @@ Write at most five bullets and stop. Do not write code, do not open files to "ch
one thing first", do not start with a small safe change.
```
Task: V-359, one line.
Task: V-359, one line. `unfiled` when there is no id.
Done: what is already on the branch.
Next: the one thing this session does.
Constraints: what would make this wrong.
@@ -67,8 +74,9 @@ Assuming: the beliefs that, if false, waste the session.
Then ask: is this right? Wait for the answer.
A corrected assumption goes into the Vikunja task as a comment, not into the handoff.
The handoff dies tonight. The task does not.
A corrected assumption goes into the Vikunja task as a comment where there is a
task, because the handoff dies tonight and the task does not. Unfiled, it goes
into the handoff and nowhere else.
## 5. Then begin
+10 -4
View File
@@ -36,11 +36,13 @@ a commit message, not into a comment in the code.
Under 300 changed lines per commit in non-markdown files, enforced by `.githooks/pre-commit`.
Markdown is exempt and may land as one batch.
Each commit is one idea, subject in the repo's voice, lowercase area prefix, and it
ends with the Vikunja ref:
Each commit is one idea, subject in the repo's voice, lowercase area prefix. A
Vikunja ref is welcome where a task exists and is required nowhere: the
`commit-msg` hook that demanded it was deleted on 2026-08-25.
```
router: narrow the single-token rule (V-359)
router: narrow the single-token rule
```
If a change genuinely cannot split under 300 lines, say why in the commit body before
@@ -56,13 +58,17 @@ It refuses a dirty tree, pushes, opens or refreshes the PR against the repo defa
branch, labels the Vikunja task in-review, comments the PR url on it, and pushes an
ntfy. Do not push by hand and do not call `tea` yourself.
`task pr` needs an id. On a hand-cut branch with no task, push the branch and open
the PR by hand, and skip step 5.
## 5. Record what `task pr` cannot know
Comment on the Vikunja task: what you measured, what is still open. List every
assumption that turned out to be wrong. If the session found new work, create a task
for it now rather than describing it in prose.
This step is what makes the handoff disposable.
This step is what makes the handoff disposable. With no task, it cannot run, so the
handoff carries that content instead and stops being disposable. Say so in it.
## 6. Leave the handoff, or leave none
@@ -75,7 +81,7 @@ resume, and no history:
```markdown
# Handoff — <date>
Task: V-359 <one line>
Task: V-359 <one line>, or `unfiled`
Branch: task/359-<slug>, cut from master
## Where I stopped
-30
View File
@@ -1,30 +0,0 @@
#!/bin/sh
# Every commit names the Vikunja task it belongs to.
#
# router: narrow the single-token rule (V-359)
#
# V- and not #, because Gitea autolinks #359 to a Gitea issue, which is a
# different tracker and a wrong link.
#
# Exempt: merges, reverts, fixup/squash, and the initial commit.
msg_file=$1
subject=$(sed -n '1p' "$msg_file")
case "$subject" in
Merge\ *|Revert\ *|fixup!\ *|squash!\ *|amend!\ *) exit 0 ;;
esac
if [ -f "$(git rev-parse --git-dir)/MERGE_HEAD" ]; then
exit 0
fi
if printf '%s' "$subject" | grep -qE '\(V-[0-9]+\)$'; then
exit 0
fi
echo "commit-msg: subject must end with a Vikunja task ref." >&2
echo " got: $subject" >&2
echo " want: router: narrow the single-token rule (V-359)" >&2
echo " No task yet? Create one. Work without a task is work nobody can resume." >&2
exit 1
+15
View File
@@ -76,3 +76,18 @@ __pycache__/
.env
# silero-vad, downloaded (see AGENTS.md)
/models/vad/
# Go build cache and GOPATH from the containerised e2eprobe build. Created by
# the command in docs/capabilities/README.md, which runs as root in a container
# and so cannot share the host cache. Multi-GB, entirely reproducible.
/.cache/
# docs/architecture/ derived output. The sources, findings.md and the inventory
# JSON are tracked; these rebuild from them with pack_evidence.sh and are large.
/docs/architecture/index.html
/docs/architecture/anchors.md
/docs/architecture/architecture-evidence.txt
/docs/architecture/tree.txt
/docs/architecture/docker-compose.redacted.yml
/docs/architecture/diagrams/*.svg
/maven-evidence.zip
+7 -5
View File
@@ -15,6 +15,8 @@ under `docs/`. Read that doc before changing the subsystem.
| `docs/world.md` | touching search, Kiwix or the world chain |
| `docs/language.md` | changing a prompt contract or a Russian word list |
| `docs/ecosystem.md` | touching Nexus, Praxis or Hexis |
| `docs/spec.md` | asking what a capability is for, or whether it is done |
| `docs/roadmap.md` | picking what to work on next |
| `docs/rearchitecture.md`, `docs/design.md` | changing the shape of anything |
| `docs/workflow.md` | the five stores, the doc tiers, the guards |
| `docs/caveats/` | a known limit, its task id and its revisit trigger |
@@ -199,14 +201,14 @@ ToolSearch("select:mcp__vikunja__list_tasks,mcp__vikunja__get_task_details,mcp__
```
- This repo is Vikunja project **Maven** (ID 2), MCP at
`http://localhost:9100/mcp`, or `http://192.168.1.104:9100/mcp` from workpc.
- **A session with no task id asks for one before it starts**, because work
without one is work nobody can resume.
`http://localhost:9100/mcp` on homesrv. **`vikunja-mcp` publishes to
`127.0.0.1:9100` only, so the LAN address never answers from workpc.** A
refused connection is that, not an outage: three sessions read it as "Vikunja
is down" and filed nothing. Tunnel first, then use `localhost`:
`ssh -N -f -L 9100:127.0.0.1:9100 kami@192.168.1.104`.
- **Close a finished task with `done: true` and nothing else** (owner's call,
2026-08-07). `update_task` carrying a `description` resets `done` to false.
- **`pre-commit` refuses master** and more than 300 changed lines in
non-markdown files. Markdown is exempt and may land as one batch.
- **`commit-msg` requires the subject to end with `(V-<id>)`.** `V-` and not
`#`, because Gitea autolinks `#123` to the wrong tracker.
- **`diff-budget.sh` blocks edits past 600 changed lines** on a `task/` branch.
- **`--no-verify` exists.** Using it means saying why in the commit body.
+93
View File
@@ -0,0 +1,93 @@
# Handoff
Master is at `5cae33a`, pushed, tree clean apart from this file. Working on master
raw by the owner's call: no branch, `--no-verify` on every commit with the reason
in the body.
## Landed this session
Twelve commits pushed. Nine were the previous session's tree, already described in
the commit log. Three are new:
- `78a9c61` `docs/spec.md`, 51 capabilities with a DoD each.
- `02e3d27` `docs/roadmap.md`, seven milestones, plus both pointer rows in `CLAUDE.md`.
- `5cae33a` honesty split into three milestones, five capabilities deferred past v1.
**Read `docs/spec.md` and `docs/roadmap.md` before anything else.** Every decision
from this session is in them. This file holds only what they do not.
## The two documents
`docs/spec.md` is the union of the 39 audited rows
(`docs/evals/2026-08-13-capability-audit.md`) and the owner's 18-item v1 list.
Twelve of his items had no audit row, so the file has 51 entries. Each entry
carries three parts. State is a reference to the living doc that owns it. DoD is
a plain list observable on the running box. Scenario names a file in
`cmd/mavend/testdata/scenarios/`.
`docs/roadmap.md` orders them into nine milestones. Honesty, then reach, then
breadth. Not ordered by code work, because none of the four broken capabilities is
a code defect.
## Decided, do not re-ask
- **v1 is a voice assistant, minimum viable.** Each DoD is written at
"voice-reachable and honest", not "feature-complete".
- **Honesty splits into three.** M1 the turn path, M2 memory he cannot correct,
M3 step-up. M1 and M2 touch different code and owe different docs. Step-up is
configuration, not honesty, and sits before M4 because M4 is what first makes
acts real.
- **Five capabilities deferred past v1** (owner's call, 2026-08-15): speaker
recognition, smart home, bluetooth control, model swap, self-update. Bluetooth
was on the v1 list and came off it. Their spec entries keep their DoD.
- **A milestone closes its own doc gaps and writes its own scenarios.** Neither
becomes a milestone of its own. Otherwise the 17 missing docs and 46 missing
scenario files collect at the end.
- **Learning means behavioral, not weights.** Stored outcomes only. No adapter, no
training set.
- **Email and calendar need the product decision before deploying.** Both are
built and neither is in `docker-compose.yml`. That is M8.
## Findings the spec pass produced
- **Recurring reminders do not exist on the spoken path.** `store.Reminder` carries
`Cron` and `ipc.CreateReminder` takes one. `grep "Cron:" --include='*.go'`
outside tests returns only `internal/ipc/client.go`, `internal/ipc/storeapi.go`
and `cmd/mavend/tick_routines.go`, and the last is routines, a separate
mechanism. Pills, the dog, the vet and the kibble are unbuilt on top of finished
storage and delivery. This is M6.
- **Seventeen capabilities have no living doc.** Memory is the worst cluster:
facts, notes and the digestion worker have no owning document at all.
- **Webhooks barely exist.** The only one in the tree is
`internal/delivery/telegramsink/intake.go`, Telegram's own inbound hook.
- **Command chaining does not exist.** The `chain` in `internal/router` is the
world chain and the source chain.
- **Desk notifications are inbound only.** `ambient:notif` reads his desktop.
There is no outbound desk reach, and which direction he meant is undecided.
- **Only 5 of 51 spec entries cite a scenario that exists.**
## Not filed, and this is the risk
Vikunja returned 503 across this session and the last, so **none of this has a
task id**. The three commits above are tagged `V-719`, which is the
reminder-cancellation task, not this work. Retag or file when Vikunja is back.
Unfiled, listed again at the end of `docs/roadmap.md`:
1. The capability audit itself, headed "unfiled".
2. Remember-versus-query misroute, two of seven audit probes.
3. The masculine reply on the wire, caught live while `CheckFeminine` passed.
4. Recurring reminders having no caller.
5. The seventeen capabilities with no living doc.
6. The 46 scenario files the spec names and does not have.
## Next
Open a session on M1, which is three gate items on the turn path and owes no new
doc. If Vikunja is up, file the six above first and give M1 a real id.
One command still outstanding from the last session, cheap and unrelated:
```sh
docker compose up -d --force-recreate mavsttd mavttsd mavpoll
```
+58
View File
@@ -281,3 +281,61 @@ expires, and when it has been drained. `./cmd/mavend/ -run TestSuppressedCareDig
passes under race in 4.626s, the digest store and loop cases in 4.123s and
1.046s, and the three full packages in 264.076s, 64.496s and 4.280s. The caveat
`docs/caveats/workers.md#nudges` and the `SA4006` baseline entry are deleted.
## 2026-08-15 — authoritative-state continuation
The continuation began by treating the checked-out tree and external task state
as authoritative. `master` was clean, identical to `origin/master`, and already
contained the V-717 and V-687 branch merges. Their interrupted worker messages
were therefore not used as evidence of missing work.
### Live reminder catch-up closes V-570 and V-715
The rebuilt stack loaded the resident Qwen model, multilingual ONNX embedder,
and routing heads. On its first eligible tick it phrased the three overdue
reminders once as one catch-up message. Disabled ntfy left one durable failed
attempt for delivery group `reminder #83`; ordered fallback then left exactly
one successful Telegram attempt for the same group. All originals became
`fired`. Four later ticks produced no second phrase, send, pending attempt, or
unknown attempt. `/`, `/reminders`, and `/notifications` each returned HTTP 200
with a server-generated request ID. This supplies the live evidence that was
still missing in the 2026-08-13 entry; V-715 and the stale-open V-570 are closed.
The first HTTP repro command also exposed a zsh test-harness trap: `path` is a
special array tied to `PATH`, so using it as a loop variable removed command
lookup inside the loop. The corrected probe used `probe_path` and `wget
--no-proxy`; the deployed web process had not crashed.
### Stale-open task reconciliation
The open Vikunja list was read with `done: false` and compared with the current
tree before choosing work. V-397 already described itself as done and merged;
V-557 is present as `ea0eb16` plus the forced missing-slot dialogue contract;
V-570 is covered by the live catch-up proof above. Those three records were
closed rather than reimplemented. V-651 remains open: its nil-sink and reminder
spin halves landed with V-715, but the first failed severity-4 Telegram send is
still not represented in the repeat-until-ack stream.
### Stable hash-floor performance evidence [V-718]
`TestPersonalBoundaryHashFloorLatency` coupled correctness to ambient machine
load while running a numeric training loop under race and coverage
instrumentation. It is now a deterministic fit-and-score test that also proves
the 1024-dimensional head was built. Elapsed time remains observable through
`BenchmarkPersonalBoundaryHashFloorFitAndScore`, where three one-iteration runs
on this host measured 75.1 ms, 76.8 ms and 81.4 ms without making those host
figures a CI pass condition.
The exact race-plus-coverage focused test passed in 9.445s. `make test` then
passed on its first run: formatting, vet, CW2 configuration tests, and every
internal and command package under race plus coverage; `cmd/mavend` completed
in 205.712s. This closes V-718 without raising a brittle timeout.
### Delegation availability
All three available subagent slots were filled: clarification exhaustion,
durable nudge identity, and a read-only live deployment probe. The first two
left complete merges on `master`; all three later reported the same shared
Codex usage limit, with capacity unavailable until 2026-08-20. Work continues
serially. The temporary constraint and revisit trigger are recorded at
`docs/caveats/workers.md#agent-quota` under the V-714 completion umbrella.
File diff suppressed because it is too large Load Diff
+284
View File
@@ -0,0 +1,284 @@
// e2eprobe is a temporary typed IPC driver used by the 2026-08-15 isolated
// whole-Maven acceptance session. It is removed after the session; keeping the
// driver inside the module lets it import Maven's internal IPC contract rather
// than peeking into sqlite.
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"math"
"os"
"strconv"
"strings"
"time"
"github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
)
func main() {
if err := run(os.Args[1:]); err != nil {
fmt.Fprintln(os.Stderr, "e2eprobe:", err)
os.Exit(1)
}
}
func run(args []string) error {
fs := flag.NewFlagSet("e2eprobe", flag.ContinueOnError)
sock := fs.String("sock", "", "mavend unix socket")
if err := fs.Parse(args); err != nil {
return err
}
argv := fs.Args()
if len(argv) == 0 {
return errors.New("usage: e2eprobe -sock PATH COMMAND [ARGS]")
}
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
if argv[0] == "score-pair" {
out, err := scorePair(ctx, argv)
if err != nil {
return err
}
return encode(out)
}
if argv[0] == "parse-task-status" {
if len(argv) != 2 {
return errors.New("parse-task-status needs TEXT")
}
parsed, ok := router.ParseTaskStatus(argv[1])
return encode(map[string]any{"accepted": ok, "parsed": parsed})
}
if *sock == "" {
return errors.New("usage: e2eprobe -sock PATH COMMAND [ARGS]")
}
cli, err := ipc.DialWait(*sock, 15*time.Second)
if err != nil {
return err
}
defer cli.Close()
var out any
switch argv[0] {
case "ping":
out, err = cli.Ping(ctx)
case "chat":
if len(argv) < 3 {
return errors.New("chat needs CONVERSATION TEXT")
}
out, err = cli.Chat(ctx, argv[1], strings.Join(argv[2:], " "))
case "create-reminder":
if len(argv) < 3 || len(argv) > 4 {
return errors.New("create-reminder needs RFC3339 TEXT [CRON]")
}
fire, parseErr := time.Parse(time.RFC3339, argv[1])
if parseErr != nil {
return parseErr
}
cron := ""
if len(argv) == 4 {
cron = argv[3]
}
id, createErr := cli.CreateReminder(ctx, fire, `{"text":`+quote(argv[2])+`}`, cron)
out, err = map[string]any{"id": id}, createErr
case "cancel-reminder":
id, parseErr := oneID(argv)
if parseErr != nil {
return parseErr
}
err = cli.CancelReminder(ctx, id)
out = map[string]any{"cancelled": id}
case "mark-reminder":
if len(argv) != 3 {
return errors.New("mark-reminder needs ID STATUS")
}
id, parseErr := strconv.ParseInt(argv[1], 10, 64)
if parseErr != nil {
return parseErr
}
err = cli.MarkReminder(ctx, id, argv[2])
out = map[string]any{"marked": id, "status": argv[2]}
case "reminders":
n, parseErr := optionalN(argv, 200)
if parseErr != nil {
return parseErr
}
out, err = cli.ListReminders(ctx, n)
case "pending-reminders":
n, parseErr := optionalN(argv, 0)
if parseErr != nil {
return parseErr
}
out, err = cli.ListPendingReminders(ctx, n)
case "create-task":
if len(argv) != 2 {
return errors.New("create-task needs TEXT")
}
out, err = cli.CaptureTask(ctx, ipc.CaptureTaskReq{
Text: argv[1], Source: "tap:web", Status: store.TaskOpen, Ts: time.Now(),
})
case "tasks":
status := "live"
if len(argv) == 2 {
status = argv[1]
} else if len(argv) != 1 {
return errors.New("tasks takes optional STATUS")
}
out, err = cli.ListTasks(ctx, status)
case "notes":
n, parseErr := optionalN(argv, 50)
if parseErr != nil {
return parseErr
}
out, err = cli.RecentNotes(ctx, n)
case "query-notes":
if len(argv) != 2 {
return errors.New("query-notes needs TEXT")
}
embedder, embedErr := router.NewONNXEmbedder(
"models/embedder/multilingual-e5-small/model_quantized.onnx",
"models/embedder/multilingual-e5-small/tokenizer.json",
"deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so",
)
if embedErr != nil {
return embedErr
}
defer embedder.Close()
vec, embedErr := router.EmbedQuery(ctx, embedder, argv[1])
if embedErr != nil {
return embedErr
}
out, err = cli.QueryNotes(ctx, vec, 10)
case "score-pair":
out, err = scorePair(ctx, argv)
case "facts":
n, parseErr := optionalN(argv, 50)
if parseErr != nil {
return parseErr
}
out, err = cli.RecentFacts(ctx, n)
case "decisions":
n, parseErr := optionalN(argv, 50)
if parseErr != nil {
return parseErr
}
out, err = cli.TurnDecisions(ctx, n)
case "events":
n, parseErr := optionalN(argv, 50)
if parseErr != nil {
return parseErr
}
out, err = cli.RecentEvents(ctx, n)
case "eco-traces":
n, parseErr := optionalN(argv, 50)
if parseErr != nil {
return parseErr
}
out, err = cli.RecentEcosystemTraces(ctx, n)
case "delivery-attempts":
status := ""
if len(argv) == 2 {
status = argv[1]
} else if len(argv) != 1 {
return errors.New("delivery-attempts takes optional STATUS")
}
out, err = cli.DeliveryAttempts(ctx, status, 200)
case "nudges":
n, parseErr := optionalN(argv, 50)
if parseErr != nil {
return parseErr
}
out, err = cli.RecentNudges(ctx, n)
case "tools":
status := ""
if len(argv) == 2 {
status = argv[1]
} else if len(argv) != 1 {
return errors.New("tools takes optional STATUS")
}
out, err = cli.ListTools(ctx, status)
case "plan":
out, err = cli.DayPlan(ctx)
case "correct":
if len(argv) != 3 {
return errors.New("correct needs TRACE_ID SHOULD_BE")
}
id, parseErr := strconv.ParseInt(argv[1], 10, 64)
if parseErr != nil {
return parseErr
}
err = cli.CorrectTurn(ctx, id, argv[2])
out = map[string]any{"corrected": id, "should_be": argv[2]}
default:
return fmt.Errorf("unknown command %q", argv[0])
}
if err != nil {
return err
}
return encode(out)
}
func encode(out any) error {
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
return enc.Encode(out)
}
func scorePair(ctx context.Context, argv []string) (any, error) {
if len(argv) != 3 {
return nil, errors.New("score-pair needs QUERY PASSAGE")
}
embedder, err := router.NewONNXEmbedder(
"models/embedder/multilingual-e5-small/model_quantized.onnx",
"models/embedder/multilingual-e5-small/tokenizer.json",
"deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so",
)
if err != nil {
return nil, err
}
defer embedder.Close()
qvec, err := router.EmbedQuery(ctx, embedder, argv[1])
if err != nil {
return nil, err
}
pvec, err := router.EmbedPassage(ctx, embedder, argv[2])
if err != nil {
return nil, err
}
if len(qvec) != len(pvec) {
return nil, fmt.Errorf("embedding widths differ: %d != %d", len(qvec), len(pvec))
}
var dot float64
for i := range qvec {
dot += float64(qvec[i]) * float64(pvec[i])
}
return map[string]any{"score": math.Round(dot*1e9) / 1e9}, nil
}
func quote(s string) string {
b, _ := json.Marshal(s)
return string(b)
}
func oneID(argv []string) (int64, error) {
if len(argv) != 2 {
return 0, errors.New("command needs ID")
}
return strconv.ParseInt(argv[1], 10, 64)
}
func optionalN(argv []string, fallback int) (int, error) {
if len(argv) == 1 {
return fallback, nil
}
if len(argv) != 2 {
return 0, errors.New("command takes optional N")
}
return strconv.Atoi(argv[1])
}
+92
View File
@@ -0,0 +1,92 @@
package main
import (
"context"
"github.com/kami/maven/internal/decision"
"github.com/kami/maven/internal/router"
)
// resolveAction produces an ActionCandidate from a routing decision. It is the
// single boundary between routing and action execution: everything downstream
// (refusesCommand, task-status, Praxis, Hexis, proposeGap, tool.Executor.Exec)
// consumes the candidate rather than re-resolving the function.
//
// Delegates to router.ResolveActionCandidate for the resolution logic, then
// records the outcome in the decision trace.
func (h *reactiveHandler) resolveAction(ctx context.Context, dec router.Decision) router.ActionCandidate {
candidate := router.ResolveActionCandidate(dec, h.matcher)
// Record the resolution outcome in the decision trace.
if dec.Intent == router.IntentAct {
if candidate.ActionResolved() {
noteActionResolution(ctx, string(candidate.Source), candidate.Fn, true)
} else {
noteActionResolution(ctx, "matcher", "", false)
}
}
return candidate
}
// noteActionResolution records the action resolution outcome in the decision
// trace. A nil recorder is the normal case in tests.
func noteActionResolution(ctx context.Context, source, fn string, resolved bool) {
rec := decision.From(ctx)
if rec == nil {
return
}
outcome := decision.Declined
reason := "no match"
if resolved {
outcome = decision.Won
reason = "resolved via " + source
if fn != "" {
reason += ": " + fn
}
}
rec.Note(decision.Claim{
Stage: decision.StageAction,
Claimant: "action-resolve",
Outcome: outcome,
Reason: reason,
})
}
// noteActionValidation records the structural validation outcome in the
// decision trace. Five outcomes: unresolved (matcher miss), valid
// (structurally admissible), invalid_argument, missing_argument, or
// ambiguous_target (structurally malformed).
func noteActionValidation(ctx context.Context, v router.ActionValidationResult) {
rec := decision.From(ctx)
if rec == nil {
return
}
switch v.Status {
case router.ActionUnresolved:
rec.Note(decision.Claim{
Stage: decision.StageAction,
Claimant: "action-validation",
Outcome: decision.Declined,
Reason: "unresolved",
})
case router.ActionValid:
rec.Note(decision.Claim{
Stage: decision.StageAction,
Claimant: "action-validation",
Outcome: decision.Won,
Reason: "valid",
})
default:
reason := string(v.Status)
if len(v.Issues) > 0 {
reason = string(v.Status) + ":" + v.Issues[0].Reason
}
rec.Note(decision.Claim{
Stage: decision.StageAction,
Claimant: "action-validation",
Outcome: decision.Declined,
Reason: reason,
})
}
}
+556
View File
@@ -0,0 +1,556 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/decision"
"github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
"github.com/kami/maven/internal/tool"
)
// newActHandler builds a handler with the act path wired: a matcher over
// whatever tools the test enabled, no model, no ecosystem.
func newActHandler(t *testing.T) (*reactiveHandler, *store.Store) {
t.Helper()
st := newTestStore(t)
api := ipc.NewStoreAPI(st)
matcher := tool.NewMatcher(api)
h := &reactiveHandler{
api: api,
tools: tool.NewExecutor(api, 2*time.Second),
matcher: matcher,
now: func() time.Time { return time.Date(2026, 8, 1, 9, 0, 0, 0, time.UTC) },
}
return h, st
}
// TestActRouteSource_NoMatcherInvoke pins that an act with HasFn=true
// produces a candidate from the route and does not invoke the matcher.
func TestActRouteSource_NoMatcherInvoke(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
// Enable a tool so the matcher has something to match against.
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
// Act with HasFn=true: the candidate must come from the route.
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "status",
Slots: router.Slots{Fn: "status", HasFn: true},
})
if !strings.Contains(reply, "готово") {
t.Errorf("route-sourced act replied %q; want it to have run", reply)
}
}
// TestActMatcherSource_FallbackMatch pins that an act without Fn invokes
// the matcher and produces a matcher-sourced candidate.
func TestActMatcherSource_FallbackMatch(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
// Enable a tool so the matcher can find it.
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
// Act without HasFn: the matcher must resolve "status" from the text.
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "check status",
Slots: router.Slots{Text: "status"},
})
if !strings.Contains(reply, "готово") {
t.Errorf("matcher-sourced act replied %q; want it to have run", reply)
}
}
// TestActMatcherMiss_ProposeGap pins that a matcher miss produces the
// same propose-gap behavior as before.
func TestActMatcherMiss_ProposeGap(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
// Enable one tool so the matcher has an allowlist, but not the one asked for.
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
// Act without HasFn and text that doesn't match any tool.
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "deploy the thing",
Slots: router.Slots{Text: "deploy the thing"},
})
if !strings.Contains(strings.ToLower(reply), "предлож") {
t.Errorf("matcher miss replied %q; want propose-gap behavior", reply)
}
}
// TestActDestructive_ConfirmationUnchanged pins that a destructive tool
// still triggers the confirmation flow.
func TestActDestructive_ConfirmationUnchanged(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "restart",
Slots: router.Slots{Fn: "restart", HasFn: true},
})
if !strings.Contains(reply, "да или нет") {
t.Errorf("destructive act replied %q; want a confirm turn", reply)
}
}
// TestActTaskStatus_InterceptUnchanged pins that task_status is intercepted
// before reaching the tool executor.
func TestActTaskStatus_InterceptUnchanged(t *testing.T) {
h, _ := newActHandler(t)
ctx := context.Background()
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "task status",
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true,
Text: "task status"},
})
// task_status is intercepted by resolveTaskStatus, which returns a
// status phrase. The exact reply depends on the store state, but it
// must not be a tool execution result.
if strings.Contains(reply, "готово") {
t.Errorf("task_status was not intercepted, got %q", reply)
}
}
// TestActStage0_SameResult pins that a stage-0 act (grammar match with
// HasFn=true) produces the same tool execution as before.
func TestActStage0_SameResult(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, false, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Stage: 0,
Confidence: 1.0,
Utterance: "maven, restart nginx",
Slots: router.Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
Producer: router.RouteProducerGrammar,
})
if !strings.Contains(reply, "сделала") && !strings.Contains(reply, "готово") {
t.Errorf("stage-0 act replied %q; want it to have run", reply)
}
}
// TestActLearnedRouter_NoFn_FallbackMatch pins that a learned-router act
// without Fn falls through to the matcher and produces the same result.
func TestActLearnedRouter_NoFn_FallbackMatch(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, false, "test", now); err != nil {
t.Fatal(err)
}
// LLM routed the act but did not fill Fn (common when the model returns
// the verb in Text but not in Fn).
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Stage: 1,
Confidence: 0.85,
Utterance: "could you restart nginx",
Slots: router.Slots{Text: "restart nginx"},
Producer: router.RouteProducerLLM,
})
if !strings.Contains(reply, "сделала") && !strings.Contains(reply, "готово") {
t.Errorf("learned-router act replied %q; want it to have run", reply)
}
}
// TestResolveAction_CandidateSource_Verified pins the candidate source
// for both route-resolved and matcher-resolved actions.
func TestResolveAction_CandidateSource_Verified(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
// Route-resolved: HasFn=true.
c1 := h.resolveAction(ctx, router.Decision{
Intent: router.IntentAct,
Slots: router.Slots{Fn: "status", HasFn: true},
})
if c1.Source != router.ActionSourceRoute {
t.Errorf("route candidate source = %q, want route", c1.Source)
}
if c1.Fn != "status" {
t.Errorf("route candidate Fn = %q, want status", c1.Fn)
}
// Matcher-resolved: no Fn, text matches.
c2 := h.resolveAction(ctx, router.Decision{
Intent: router.IntentAct,
Slots: router.Slots{Text: "status"},
})
if c2.Source != router.ActionSourceMatcher {
t.Errorf("matcher candidate source = %q, want matcher", c2.Source)
}
if c2.Fn != "status" {
t.Errorf("matcher candidate Fn = %q, want status", c2.Fn)
}
// Matcher miss: no Fn, text doesn't match.
c3 := h.resolveAction(ctx, router.Decision{
Intent: router.IntentAct,
Slots: router.Slots{Text: "deploy everything"},
})
if c3.ActionResolved() {
t.Errorf("miss candidate resolved = true, want false")
}
}
// --- structural validation integration tests ---
// TestActValidation_MalformedCandidate_BlankFn pins that a resolved
// candidate with a blank (whitespace-only) Fn does not execute and
// produces a failure response.
func TestActValidation_MalformedCandidate_BlankFn(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
// Simulate a malformed candidate by writing a blank Fn into Slots
// after resolution. This tests that the validation layer catches
// structurally invalid candidates.
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "status",
Slots: router.Slots{Fn: " ", HasFn: true},
})
// The blank Fn should not reach tool execution. It either hits
// the validation gate (ActFail) or the existing error paths.
if reply == "" {
t.Error("expected a response, got empty string")
}
}
// TestActValidation_UnresolvedCandidate_ProposeGap pins that an unresolved
// candidate (matcher miss) still flows to proposeGap, unchanged.
func TestActValidation_UnresolvedCandidate_ProposeGap(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "deploy everything",
Slots: router.Slots{Text: "deploy everything"},
})
if !strings.Contains(strings.ToLower(reply), "предлож") {
t.Errorf("unresolved candidate replied %q; want propose-gap behavior", reply)
}
}
// TestActValidation_DestructiveValid_StillConfirms pins that a destructive
// valid action still reaches the confirmation path through validation.
func TestActValidation_DestructiveValid_StillConfirms(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "restart",
Slots: router.Slots{Fn: "restart", HasFn: true},
})
if !strings.Contains(reply, "да или нет") {
t.Errorf("destructive valid act replied %q; want confirm turn", reply)
}
}
// TestActValidation_IrreversibleValid_NeedsAuthedSurface pins that an
// irreversible valid action still reaches ErrNeedsAuthedSurface.
func TestActValidation_IrreversibleValid_NeedsAuthedSurface(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
// Register an irreversible tool: cmd containing "drop" triggers the
// irreversible tier via RiskOf → isIrreversible.
if err := st.EnableTool(ctx, "drop_table", []string{"drop"}, true, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "drop_table",
Slots: router.Slots{Fn: "drop_table", HasFn: true},
})
// Irreversible tools return ErrNeedsAuthedSurface, which produces
// a specific phraser response.
if !strings.Contains(reply, "выполню") && !strings.Contains(reply, "запусти") {
t.Errorf("irreversible valid act replied %q; want authed-surface response", reply)
}
}
// TestActValidation_ValidationTracing pins that validation outcomes are
// recorded in the decision trace.
func TestActValidation_ValidationTracing(t *testing.T) {
h, st := newActHandler(t)
now := h.now()
// Valid candidate: trace should show action-validation:won.
ctx, rec := decision.With(context.Background(), "status", "tap:text")
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "status",
Slots: router.Slots{Fn: "status", HasFn: true},
})
records := rec.Claims
found := false
for _, c := range records {
if c.Claimant == "action-validation" && c.Outcome == decision.Won {
found = true
break
}
}
if !found {
t.Errorf("expected action-validation:won in trace, got %v", records)
}
}
// TestActExecutionFromCandidateNotSlots pins that downstream execution reads
// resolved action data from ActionCandidate, not from Decision.Slots. The
// decision has empty Fn/Args/HasFn — the bridge used to copy candidate values
// back into these fields. After the bridge removal, execution must still
// succeed because the candidate carries the resolved function.
func TestActExecutionFromCandidateNotSlots(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
// Act without any Fn/Args/HasFn in Slots — the matcher resolves from Text.
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "check status",
Slots: router.Slots{Text: "status"},
})
if !strings.Contains(reply, "готово") {
t.Errorf("execution from candidate replied %q; want tool success", reply)
}
}
// --- validation status boundary tests ---
// TestActValidation_StatusValidRoute pins that a route-resolved valid action
// produces ActionValid status and reaches execution.
func TestActValidation_StatusValidRoute(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "status",
Slots: router.Slots{Fn: "status", HasFn: true},
})
if !strings.Contains(reply, "готово") {
t.Errorf("valid route act replied %q; want tool success", reply)
}
}
// TestActValidation_StatusValidMatcher pins that a matcher-resolved valid
// action produces ActionValid status and reaches execution.
func TestActValidation_StatusValidMatcher(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "check status",
Slots: router.Slots{Text: "status"},
})
if !strings.Contains(reply, "готово") {
t.Errorf("valid matcher act replied %q; want tool success", reply)
}
}
// TestActValidation_StatusUnresolved pins that an unresolved candidate
// produces ActionUnresolved status and flows to proposeGap.
func TestActValidation_StatusUnresolved(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "deploy everything",
Slots: router.Slots{Text: "deploy everything"},
})
if !strings.Contains(strings.ToLower(reply), "предлож") {
t.Errorf("unresolved act replied %q; want propose-gap", reply)
}
}
// TestActValidation_StatusInvalid pins that a structurally invalid candidate
// produces ActionInvalidArgument status and refuses execution.
func TestActValidation_StatusInvalid(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "status",
Slots: router.Slots{Fn: " ", HasFn: true},
})
if reply == "" {
t.Error("expected a response for invalid candidate")
}
if strings.Contains(reply, "готово") {
t.Error("invalid candidate should not reach tool execution")
}
}
// TestActValidation_DestructiveValidStatus pins that a destructive but
// structurally valid action still produces ActionValid status and reaches
// the confirmation path (not validation failure).
func TestActValidation_DestructiveValidStatus(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "restart",
Slots: router.Slots{Fn: "restart", HasFn: true},
})
if !strings.Contains(reply, "да или нет") {
t.Errorf("destructive valid act replied %q; want confirm turn", reply)
}
}
// TestActValidation_ConfirmationUnchanged pins that the confirmation flow
// is unchanged by validation.
func TestActValidation_ConfirmationUnchanged(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, true, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "restart nginx",
Slots: router.Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
})
if !strings.Contains(reply, "да или нет") {
t.Errorf("confirmation act replied %q; want confirm turn", reply)
}
}
// TestActValidation_TaskStatusInterceptUnchanged pins that task_status
// interception is unchanged by validation.
func TestActValidation_TaskStatusInterceptUnchanged(t *testing.T) {
h, _ := newActHandler(t)
ctx := context.Background()
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "task status",
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Text: "task status"},
})
if strings.Contains(reply, "готово") {
t.Errorf("task_status was not intercepted, got %q", reply)
}
}
// TestActValidation_NoExecutionOnFailure pins that validation failure
// prevents downstream execution.
func TestActValidation_NoExecutionOnFailure(t *testing.T) {
h, st := newActHandler(t)
ctx := context.Background()
now := h.now()
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
t.Fatal(err)
}
reply := h.actionAct(ctx, router.Decision{
Intent: router.IntentAct,
Utterance: "status",
Slots: router.Slots{Fn: " ", HasFn: true},
})
if strings.Contains(reply, "готово") {
t.Error("validation failure should not reach tool execution")
}
}
+42 -22
View File
@@ -11,54 +11,74 @@ import (
"github.com/kami/maven/internal/tool"
)
// actionAct handles router.IntentAct: match a verb to an enabled tool, offer
// it to the ecosystems first, and run it behind the confirm gate and the
// allowlist. proposeGap and the confirm gate itself live in confirm.go.
// actionAct handles router.IntentAct: resolve the action, offer it to the
// ecosystems first, and run it behind the confirm gate and the allowlist.
// proposeGap and the confirm gate itself live in confirm.go.
//
// Action resolution happens in resolveAction (actionresolve.go) — a single
// boundary that produces an ActionCandidate before execution. This function
// consumes the candidate; it no longer decides which function/tool the user
// meant.
func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) string {
// tool executor: run the matched fn against the enabled allowlist.
// HasFn=false ⇒ try the matcher (for LLM-routed acts where the verb
// didn't go through the stage-0 act grammar).
if !dec.Slots.HasFn && dec.Slots.Text != "" && h.matcher != nil {
if fn, args, ok := h.matcher.Match(dec.Slots.Text); ok {
dec.Slots.Fn, dec.Slots.Args, dec.Slots.HasFn = fn, args, true
}
// An allowlist or a model route is evidence about WHAT could run, never
// authority to run it. Keep the user's negative command at the execution
// boundary too: actionAct is also reached by rebuilt decisions outside the
// ordinary pre-route ladder.
if refusesCommand(dec) {
return commandProhibitionReply
}
// Resolve the action: produce an ActionCandidate from the routing
// decision. The candidate carries the resolved function, its arguments,
// and where the resolution came from (route or matcher).
candidate := h.resolveAction(ctx, dec)
// Structural validation: is this candidate complete enough to proceed?
// Unresolved (Fn empty) flows to proposeGap; invalid (Fn present but
// malformed) is refused; valid proceeds to execution.
validation := router.ValidateActionCandidate(candidate)
noteActionValidation(ctx, validation)
if !validation.Unresolved() && !validation.Valid() {
// Resolved but structurally malformed: refuse execution.
return phraser.A(phraser.ActFail, nil)
}
// The board is Maven's own store, so a spoken status change is answered here
// and never offered to an ecosystem client (Vikunja #512). First, because
// task_status is on no allowlist and no capability registry: reaching either
// of them would answer a turn about his own task list with a gap.
if dec.Slots.Fn == router.TaskStatusFn {
return h.resolveTaskStatus(ctx, dec)
if candidate.Fn == router.TaskStatusFn {
return h.resolveTaskStatus(ctx, dec, candidate)
}
// Praxis ecosystem tools: intercept before the system command executor.
if h.ecosystem != nil && h.ecosystem.praxis != nil && dec.Slots.HasFn {
if reply := h.handlePraxisAct(ctx, dec); reply != "" {
if h.ecosystem != nil && h.ecosystem.praxis != nil && candidate.ActionResolved() {
if reply := h.handlePraxisAct(ctx, dec, candidate); reply != "" {
return reply
}
}
// Hexis ecosystem action: if ecosystem is configured and we have a verb
// + entity text, try to resolve the entity and execute via Hexis.
if h.ecosystem != nil && h.ecosystem.hexis != nil && dec.Slots.Text != "" {
if reply := h.handleHexisAct(ctx, dec); reply != "" {
if h.ecosystem != nil && h.ecosystem.hexis != nil && router.ActHasEntityTarget(dec) {
if reply := h.handleHexisAct(ctx, dec, candidate); reply != "" {
return reply
}
}
// HasFn still false ⇒ no allowlist match: scaffold a 'proposed' tool
// Unresolved candidate ⇒ no allowlist match: scaffold a 'proposed' tool
// the user can enable on the authed surface ("earn the right to ask").
if !dec.Slots.HasFn {
if !candidate.ActionResolved() {
return h.proposeGap(ctx, dec)
}
out, err := h.tools.Exec(ctx, dec.Slots.Fn, dec.Slots.Args, false)
out, err := h.tools.Exec(ctx, candidate.Fn, candidate.Args, false)
if err != nil {
switch {
case errors.Is(err, tool.ErrNeedsConfirm):
// destructive: park it and ask. The next utterance answers.
phrase := actPhrase(dec.Slots.Fn, dec.Slots.Args)
h.park(dec.Slots.Fn, dec.Slots.Args, phrase)
phrase := actPhrase(candidate.Fn, candidate.Args)
h.park(candidate.Fn, candidate.Args, phrase)
return phraser.A(phraser.ActConfirm, map[string]string{"name": phrase})
case errors.Is(err, tool.ErrUnknownTarget):
// The verb reached a tool and the tail did not reach a target, so
@@ -93,7 +113,7 @@ func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) st
// where a human types them.
return phraser.A(phraser.ActNeedsArgs, nil)
}
log.Printf("voice: tool %s: %v", dec.Slots.Fn, err)
log.Printf("voice: tool %s: %v", candidate.Fn, err)
if out != "" {
return phraser.A(phraser.ActFailOut, map[string]string{"out": firstLine(out)})
}
+9
View File
@@ -38,6 +38,15 @@ func (h *reactiveHandler) actionFact(ctx context.Context, dec router.Decision) s
// claim the turn before any real source ran.
q.Slots.Key, q.Slots.HasKey = "", false
q.Slots.Value = ""
// Defensive reconstruction must preserve the same literal destination
// the stage-0 router would have named. A learned fact decision has no
// source, and without restoring this anchored world frame the personal
// boundary can claim "latest Go version" by similarity and prevent the
// live source from ever being asked.
if world, ok := router.WorldQueryDecision(dec.Utterance); ok {
q.Source = world.Source
q.SourceAnchored = world.SourceAnchored
}
return h.actionQuery(ctx, q)
}
// A complaint is not a fact either (#481). "сеть какая-то медленная" and
+12 -7
View File
@@ -16,10 +16,11 @@ const nothingToCorrectReply = "не поняла, что поправить. с
// actionNote handles router.IntentNote: embed the note, persist it, and
// index it for recall.
//
// The stored body is dec.Utterance and nothing else (V-576). It is not
// Slots.Text, not phraser output and not any other model string: a note is
// durable, the embedder indexes it, and it comes back later as recall in his
// own words. Phrasing belongs in the spoken confirmation.
// The stored body comes only from dec.Utterance (V-576/V-721). An explicit
// leading capture frame is structurally removed; an unmarked note is otherwise
// byte-for-byte his utterance. It is never Slots.Text, phraser output or any
// other model string: a note is durable, the embedder indexes it, and it comes
// back later as recall in his own words. Phrasing belongs in the confirmation.
func (h *reactiveHandler) actionNote(ctx context.Context, dec router.Decision) string {
// A correction with no referent. Everything that could own one has already
// run by here: clarify, confirm and repair are all resolved before routing,
@@ -39,16 +40,20 @@ func (h *reactiveHandler) actionNote(ctx context.Context, dec router.Decision) s
if reply, ok := h.captureListFromNote(ctx, dec); ok {
return reply
}
noteText := dec.Utterance
if body, explicit := router.ParseNoteCapture(dec.Utterance); explicit {
noteText = body
}
// embed the note text with the same model the classifier uses, persist
// via CoreAPI (source=tap:voice). Semantic recall lives in `notes`, not
// facts — no predicate reads it (spec's two-memory split).
vec, err := router.EmbedPassage(ctx, h.recall.embedder, dec.Utterance)
vec, err := router.EmbedPassage(ctx, h.recall.embedder, noteText)
if err != nil {
log.Printf("voice: embed note: %v", err)
return phraser.Ack(phraser.FailNote, nil)
}
noteTs := h.now()
noteID, err := h.api.WriteNote(ctx, noteTs, dec.Utterance, vec, "tap:voice")
noteID, err := h.api.WriteNote(ctx, noteTs, noteText, vec, "tap:voice")
if err != nil {
log.Printf("voice: write note: %v", err)
return phraser.Ack(phraser.FailNote, nil)
@@ -59,7 +64,7 @@ func (h *reactiveHandler) actionNote(ctx context.Context, dec router.Decision) s
if err := h.recall.memStore.Insert(ctx, "note:"+strconv.FormatInt(noteID, 10), vec, map[string]string{
"source": "voice",
"type": "note",
"text": dec.Utterance,
"text": noteText,
"ts": strconv.FormatInt(noteTs.Unix(), 10),
}); err != nil {
log.Printf("voice: memory insert: %v", err)
+3 -2
View File
@@ -641,10 +641,11 @@ const (
// небо синее?", because the right-note and must-be-silent score ranges overlap
// and no threshold sits between them.
func recallOnTopic(utterance, text string) bool {
if memory.RecallAllowed(utterance, text) {
if memory.RecallAllowed(utterance, text,
router.IsOpenQuestionShaped(utterance), router.IsLocativeQuestionShaped(utterance)) {
return true
}
log.Printf("voice: recall %q rejected for %q: a world question and no shared topic word", text, utterance)
log.Printf("voice: recall %q rejected for %q: no structural ask with a shared topic, or a world/locative question with no shared topic", text, utterance)
return false
}
+6
View File
@@ -14,6 +14,12 @@ import (
// actionReminder handles router.IntentReminder: parse the time when stage-0
// skipped the extractor, then create the reminder.
func (h *reactiveHandler) actionReminder(ctx context.Context, dec router.Decision) string {
// The pre-route belt normally answers this before routing. Keep the write
// boundary guarded as well: a model calling the sentence a reminder does
// not turn "don't ..." into permission to create a row.
if refusesCommand(dec) {
return commandProhibitionReply
}
if !dec.Slots.HasTime {
// Stage-0 (reminder-wakeword grammar) skips the extractor, so the
// time wasn't parsed. Run the parser as a fallback.
+1 -1
View File
@@ -106,7 +106,7 @@ func (h *reactiveHandler) queryTasks(ctx context.Context, t *queryTurn) (string,
// match on more than one asks which, because closing the wrong task is work he
// never finished being marked done. No task named asks which too, since the
// router claims the turn without the referent and the list lives here.
func (h *reactiveHandler) resolveTaskStatus(ctx context.Context, dec router.Decision) string {
func (h *reactiveHandler) resolveTaskStatus(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
live, err := h.api.ListTasks(ctx, "live")
if err != nil {
log.Printf("voice: task status: list: %v", err)
+33 -3
View File
@@ -279,7 +279,7 @@ func TestResolveTaskStatusMovesTheNamedTask(t *testing.T) {
reply := h.resolveTaskStatus(context.Background(), router.Decision{
Intent: router.IntentAct,
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: "молоко"},
})
}, routeCandidate(router.TaskStatusFn))
if api.listArg != "live" {
t.Errorf("listed %q, want live — a resolved task cannot be resolved again", api.listArg)
}
@@ -294,6 +294,36 @@ func TestResolveTaskStatusMovesTheNamedTask(t *testing.T) {
}
}
// The regression crosses the grammar/action seam instead of handing the
// action a repaired Decision. The stored title is a normal imperative title,
// while the spoken marker names only its topic; framing words must not become
// identity and the unrelated live row must remain untouched.
func TestActionActMarkerReferentMovesOnlyTheNamedStoredTask(t *testing.T) {
api := &taskAPI{tasks: []ipc.Task{
{ID: 17, Text: "настроить бэкапы", Status: "open"},
{ID: 18, Text: "обновить сертификаты", Status: "open"},
}}
h := taskHandler(api)
dec, matched, accepted := router.TaskStatusGrammar().Evaluate("отметь задачу про бэкапы как сделанную")
if !matched || !accepted {
t.Fatalf("task-status grammar matched=%v accepted=%v", matched, accepted)
}
reply := h.actionAct(context.Background(), dec)
if api.listArg != "live" {
t.Errorf("listed %q, want live", api.listArg)
}
if len(api.moved) != 1 {
t.Fatalf("moved %+v, want exactly the named stored task", api.moved)
}
if got := api.moved[0]; got.id != 17 || got.status != "done" || got.by != "tap:voice" {
t.Errorf("moved %+v, want task 17 → done by tap:voice", got)
}
if !strings.Contains(reply, "настроить бэкапы") {
t.Errorf("reply = %q, want the transitioned stored title", reply)
}
}
func TestResolveTaskStatusRefusesToGuess(t *testing.T) {
cases := []struct {
name string
@@ -314,7 +344,7 @@ func TestResolveTaskStatusRefusesToGuess(t *testing.T) {
h := taskHandler(api)
reply := h.resolveTaskStatus(context.Background(), router.Decision{
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: c.named},
})
}, routeCandidate(router.TaskStatusFn))
if len(api.moved) != 0 {
t.Errorf("moved %+v — closing the wrong task is the failure this arm exists to avoid", api.moved)
}
@@ -332,7 +362,7 @@ func TestResolveTaskStatusOpensACandidateFirst(t *testing.T) {
h := taskHandler(api)
h.resolveTaskStatus(context.Background(), router.Decision{
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: "продлить домен"},
})
}, routeCandidate(router.TaskStatusFn))
if len(api.moved) != 2 {
t.Fatalf("moved %+v, want open then done", api.moved)
}
+5 -5
View File
@@ -14,7 +14,7 @@ func TestAttentionEmptyWithHealthySourcesIsAllClear(t *testing.T) {
praxis := newFakePraxisWithSources(t, `[]`, `[{"source_id":"src_ntfy","health":"ok"}]`)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
if !strings.Contains(reply, "ничего не требует внимания") {
t.Fatalf("healthy and quiet should be an all-clear, got %q", reply)
}
@@ -28,7 +28,7 @@ func TestAttentionEmptyWithAFailedSourceHedges(t *testing.T) {
]`)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
if strings.Contains(reply, "ничего не требует внимания") {
t.Fatalf("a failed source must not read as all-clear, got %q", reply)
}
@@ -47,7 +47,7 @@ func TestAttentionEmptyWithNoSourcesHedges(t *testing.T) {
praxis := newFakePraxisWithSources(t, `[]`, `[]`)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
if strings.Contains(reply, "ничего не требует внимания") {
t.Fatalf("a Praxis with no sources must not answer all-clear, got %q", reply)
}
@@ -64,7 +64,7 @@ func TestAttentionDegradedEnvelopeIsReadWithoutASourcesCall(t *testing.T) {
`[{"source_id":"src_ntfy","health":"ok"}]`)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
if !strings.Contains(reply, "src_metrics") {
t.Fatalf("the envelope's degraded source is not named: %q", reply)
}
@@ -82,7 +82,7 @@ func TestAttentionKeepsAllClearWhenSourcesCannotBeRead(t *testing.T) {
praxis.SetRouteFault("/api/v1/sources", 500)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
if !strings.Contains(reply, "ничего не требует внимания") {
t.Fatalf("an unreadable sources list should leave the answer alone, got %q", reply)
}
+3
View File
@@ -63,6 +63,9 @@ func (h *reactiveHandler) queryAttention(ctx context.Context, t *queryTurn) (str
Utterance: t.dec.Utterance,
Intent: router.IntentAct,
Slots: router.Slots{Fn: "list_attention", HasFn: true},
}, router.ActionCandidate{
Fn: "list_attention",
Source: router.ActionSourceRoute,
})
if reply == "" {
return "", false
+136
View File
@@ -0,0 +1,136 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/decision"
"github.com/kami/maven/internal/router"
)
// TestTextAndVoiceConvergeOnNormalizedInput — both entry points construct a
// NormalizedInput and pass it to runTurn. The same utterance produces the same
// route intent regardless of whether it arrived as text or voice.
func TestTextAndVoiceConvergeOnNormalizedInput(t *testing.T) {
h, _ := newRoutingClarifyHandler(t)
h.decisions = decision.NewRing()
ctx := context.Background()
utterance := "который час"
voiceCtx := withDialogueID(ctx, dialogueIDFor(sourceVoice, ""))
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
voiceReply := h.runTurn(voiceCtx, router.NormalizedInput{Text: utterance, Source: sourceVoice})
textReply := h.runTurn(textCtx, router.NormalizedInput{Text: utterance, Source: sourceText})
// Both paths should produce the same kind of reply (time answer).
for _, pair := range []struct {
label, reply string
}{
{"voice", voiceReply},
{"text", textReply},
} {
if !strings.Contains(pair.reply, "час") && !strings.Contains(pair.reply, "время") {
t.Errorf("%s reply %q does not look like a time answer", pair.label, pair.reply)
}
}
}
// TestNormalizedInputSourcePreserved — the source survives into the decision
// record so a trace can tell voice from text.
func TestNormalizedInputSourcePreserved(t *testing.T) {
h, _ := newRoutingClarifyHandler(t)
h.decisions = decision.NewRing()
ctx := context.Background()
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
h.runTurn(textCtx, router.NormalizedInput{Text: "привет", Source: sourceText})
recs := h.decisions.Recent(1)
if len(recs) == 0 {
t.Fatal("no decision record")
}
if recs[0].InputSource != string(sourceText) {
t.Errorf("InputSource = %q, want %q", recs[0].InputSource, sourceText)
}
}
// TestRouteProducerOnDecisionRecord — the producer is carried from the router
// decision into the decision record for observability.
func TestRouteProducerOnDecisionRecord(t *testing.T) {
h, _ := newRoutingClarifyHandler(t)
h.decisions = decision.NewRing()
ctx := context.Background()
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
h.runTurn(textCtx, router.NormalizedInput{Text: "который час", Source: sourceText})
recs := h.decisions.Recent(1)
if len(recs) == 0 {
t.Fatal("no decision record")
}
// A time query is a stage-0 grammar match.
if recs[0].RouteProducer != string(router.RouteProducerGrammar) {
t.Errorf("RouteProducer = %q, want %q", recs[0].RouteProducer, router.RouteProducerGrammar)
}
}
// TestPreRouteClaimHasNoRouteProducer — a turn claimed by a pre-route resolver
// never reaches the router, so the record's RouteProducer must be empty.
func TestPreRouteClaimHasNoRouteProducer(t *testing.T) {
h, _ := newRoutingClarifyHandler(t)
h.decisions = decision.NewRing()
// Park a confirm so the next "да" is consumed before routing.
// newRoutingClarifyHandler uses a fixed clock at 2026-07-31 09:00 UTC.
h.pending = &pendingAct{
fn: "test",
phrase: "delete everything",
expiry: time.Date(2026, 7, 31, 9, 1, 0, 0, time.UTC),
}
ctx := context.Background()
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
h.runTurn(textCtx, router.NormalizedInput{Text: "да", Source: sourceText})
recs := h.decisions.Recent(1)
if len(recs) == 0 {
t.Fatal("no decision record")
}
if recs[0].RouteProducer != "" {
t.Errorf("RouteProducer = %q, want empty (pre-route claimed the turn)", recs[0].RouteProducer)
}
}
// TestStage0ProducerUnchanged — grammars still produce the exact same intents
// at confidence 1.0. This pins stage-0 behavior through the new boundary.
func TestStage0ProducerUnchanged(t *testing.T) {
h, _ := newRoutingClarifyHandler(t)
h.decisions = decision.NewRing()
ctx := context.Background()
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
cases := []struct {
utterance string
intent router.Intent
}{
{"напомни позвонить маме завтра", router.IntentReminder},
{"который час", router.IntentSystem},
}
for _, c := range cases {
reply := h.runTurn(textCtx, router.NormalizedInput{Text: c.utterance, Source: sourceText})
_ = reply // behavior unchanged; we test the record, not the reply text.
recs := h.decisions.Recent(1)
if len(recs) == 0 {
t.Errorf("%s: no decision record", c.utterance)
continue
}
rec := recs[0]
if rec.RouteProducer != string(router.RouteProducerGrammar) {
t.Errorf("%s: RouteProducer = %q, want %q", c.utterance, rec.RouteProducer, router.RouteProducerGrammar)
}
// Clear the ring for the next case.
h.decisions = decision.NewRing()
}
}
+1 -1
View File
@@ -439,7 +439,7 @@ func TestClarifySecondGapExhaustionResumesLowerFlow(t *testing.T) {
h.clarifyStore.Push(voiceDialogueID, older)
h.clarifyStore.Push(voiceDialogueID, top)
reply := h.runTurn(ctx, "купить хлеб", sourceText)
reply := h.runTurn(ctx, router.NormalizedInput{Text: "купить хлеб", Source: sourceText})
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
want := withResumed(clarifyGaveUp, resumed)
if reply != want {
+49
View File
@@ -0,0 +1,49 @@
package main
import (
"context"
"github.com/kami/maven/internal/router"
)
// commandProhibitionReply is deliberately operation-neutral. At this boundary
// Maven may know only that the user denied authority, not whether the model
// would have called it a reminder, board transition, local tool or Hexis act.
const commandProhibitionReply = "хорошо, не буду."
// resolveCommandProhibition is the first mutation boundary in a turn. It runs
// before a parked clarify answer or candidate selection can consume the words,
// and before any route/model is consulted. A direct prohibition is complete in
// itself: it needs no target lookup and makes no external call.
//
// A parked clarify request is unrelated state. Preserve it and say the pending
// question again, using the same bounded suspend policy as every other side
// request. Candidate lists likewise remain untouched; no ordinal was selected.
func (h *reactiveHandler) resolveCommandProhibition(ctx context.Context, text string) (string, bool) {
if !router.IsCommandProhibition(text) {
return "", false
}
// A later bare "да" must not revive authority the user has just revoked.
// Confirmation slots are all mutation authority and are process-local, so
// clearing the three under their shared mutex is both conservative and
// atomic. Clarify questions and candidate lists are not authority and stay.
h.mu.Lock()
h.pending = nil
h.pendingHexis = nil
h.pendingRoutine = nil
h.mu.Unlock()
if h.clarifyStore != nil {
if q := h.clarifyStore.Get(dialogueIDOf(ctx), h.now()); q != nil {
h.noteSuspended(ctx, q)
}
}
return commandProhibitionReply, true
}
// refusesCommand is the defense-in-depth form for execution entry points which
// can also be called with a reconstructed or test decision outside runTurn.
// The sentinel cannot be renamed into an enabled function, and the original
// utterance remains the authority even when a model rewrites Slots.Text.
func refusesCommand(dec router.Decision) bool {
return dec.CapabilitySelection.Fn == router.ProhibitedActFn || router.IsCommandProhibition(dec.Utterance)
}
+79
View File
@@ -0,0 +1,79 @@
package main
import (
"testing"
"github.com/kami/maven/internal/router"
)
// TestRefusesCommandUsesCapabilitySelection proves that refusesCommand reads
// CapabilitySelection.Fn rather than the compatibility Slots.Fn. When
// CapabilitySelection is populated with the prohibited sentinel but Slots.Fn
// is blank, the refusal must still fire.
func TestRefusesCommandUsesCapabilitySelection(t *testing.T) {
dec := router.Decision{
Intent: router.IntentAct,
CapabilitySelection: router.CapabilitySelection{
Fn: router.ProhibitedActFn,
Resolved: true,
},
// Slots compatibility fields deliberately blank.
}
if !refusesCommand(dec) {
t.Fatal("refusesCommand should return true when CapabilitySelection.Fn == ProhibitedActFn")
}
}
// TestRefusesCommandUtteranceFallback proves that the utterance-based
// prohibition check still works as a defense-in-depth belt when
// CapabilitySelection does not carry the sentinel.
func TestRefusesCommandUtteranceFallback(t *testing.T) {
dec := router.Decision{
Utterance: "don't restart nginx",
}
if !refusesCommand(dec) {
t.Fatal("refusesCommand should return true for a prohibited utterance")
}
}
// TestRefusesCommandNonProhibitedCapability proves that an act with a
// non-prohibited capability is NOT refused even when Slots.Fn happens to
// carry the prohibited sentinel (cross-contamination).
func TestRefusesCommandNonProhibitedCapability(t *testing.T) {
dec := router.Decision{
Intent: router.IntentAct,
Utterance: "restart nginx",
CapabilitySelection: router.CapabilitySelection{
Fn: "restart",
Resolved: true,
},
}
if refusesCommand(dec) {
t.Fatal("refusesCommand should return false for a non-prohibited capability")
}
}
// TestRefusesCommandProhibitedSentinelPreservedThroughPipeline proves that the
// command-prohibition grammar sentinel survives through SelectCapability into
// CapabilitySelection byte-for-byte.
func TestRefusesCommandProhibitedSentinelPreservedThroughPipeline(t *testing.T) {
dec := router.Decision{
Intent: router.IntentAct,
Slots: router.Slots{
Fn: router.ProhibitedActFn,
HasFn: true,
},
}
sel := router.SelectCapability(dec, nil)
dec.CapabilitySelection = sel
if dec.CapabilitySelection.Fn != router.ProhibitedActFn {
t.Errorf("CapabilitySelection.Fn = %q, want %q", dec.CapabilitySelection.Fn, router.ProhibitedActFn)
}
if !dec.CapabilitySelection.Resolved {
t.Error("CapabilitySelection.Resolved should be true")
}
if !refusesCommand(dec) {
t.Fatal("refusesCommand should return true after pipeline preserves the sentinel")
}
}
+2 -2
View File
@@ -31,8 +31,8 @@ import (
// and nothing should: a missing name costs one line of the record, while a
// check that walks the ladder would have to run the ladder.
var preRouteLadder = []string{
"confirm", "repair", "repair-negative", "clarify-answer", "quiet-toggle",
"snooze", "ack", "ordinal",
"confirm", "repair", "repair-negative", "command-prohibition", "clarify-answer", "quiet-toggle",
"snooze", "ack", "reminder-cancel", "ordinal",
}
// notePreRoute records one rung of that ladder and passes its verdict through
+27 -7
View File
@@ -107,7 +107,7 @@ var praxisCapabilities = []praxisCapability{
// handlePraxisAct — dispatches ecosystem tool acts through the Praxis tools API.
// Returns "" when the act is not a Praxis verb (the caller falls through to the
// system command executor). Returns a reply string otherwise.
func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decision) string {
func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
if h.ecosystem == nil || h.ecosystem.praxis == nil {
return ""
}
@@ -127,7 +127,7 @@ func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decisi
}
for _, capability := range praxisCapabilities {
for _, alias := range capability.aliases() {
if alias == dec.Slots.Fn {
if alias == candidate.Fn {
return capability.handle(ctx, h, px, dec)
}
}
@@ -657,7 +657,13 @@ func (h *reactiveHandler) resolveEntityCandidates(ctx context.Context, refs []st
// handleHexisAct — resolves entity references through Nexus and executes
// matching capabilities through Hexis. Returns a reply string when handled,
// or "" to fall through to the system command executor.
func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decision) string {
func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
// This method is intentionally callable outside runTurn by ecosystem
// harnesses. Refuse before correlation ids, Nexus resolution or capability
// discovery so the no-op sentinel can never leak into Hexis as a verb.
if refusesCommand(dec) {
return commandProhibitionReply
}
if h.ecosystem == nil {
return ""
}
@@ -715,7 +721,7 @@ func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decisio
// Match the user's verb to a capability by name/description. Collect all
// matches: more than one is itself ambiguous, so we ask rather than pick
// the first (ecosystem invariant: no arbitrary target for mutation).
verb := dec.Slots.Fn
verb := candidate.Fn
if verb == "" {
verb = dec.Slots.Text
}
@@ -726,7 +732,7 @@ func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decisio
// round then: the phrase is the haystack and the capability name is what we
// look for in it (Vikunja #476). Only when the fn slot is empty — a matched
// fn is a single verb and containment already means what it says.
loose := !dec.Slots.HasFn
loose := !candidate.ActionResolved()
var matches []*hexisclient.Capability
for i, c := range caps {
name := strings.ToLower(c.Name)
@@ -841,13 +847,27 @@ func (h *reactiveHandler) execHexis(ctx context.Context, capID, capName, entityI
// resolution stops on ambiguity and a mutating capability still goes through
// the spoken confirm in handleHexisAct.
func (h *reactiveHandler) hexisBeforeClarify(ctx context.Context, dec router.Decision) string {
// A thinned model act reaches this hook before actionAct. Negative authority
// must therefore stop here as well, before even a read to Nexus/Hexis.
if refusesCommand(dec) {
return commandProhibitionReply
}
if h.ecosystem == nil || h.ecosystem.hexis == nil {
return ""
}
if dec.Intent != router.IntentAct || dec.Slots.HasFn || dec.Slots.Text == "" {
if dec.Intent != router.IntentAct || dec.Slots.HasFn || !router.ActHasEntityTarget(dec) {
return ""
}
return h.handleHexisAct(ctx, dec)
// Resolve the action candidate. Use the matcher when available; when the
// handler has no matcher (ecosystem-only test harnesses), build an
// unresolved candidate directly — the matcher would not have matched either.
var candidate router.ActionCandidate
if h.matcher != nil {
candidate = h.resolveAction(ctx, dec)
} else {
candidate = router.ResolveActionCandidate(dec, nil)
}
return h.handleHexisAct(ctx, dec, candidate)
}
// attentionCannotTell returns the hedge to say instead of an all-clear, or ""
+24 -24
View File
@@ -96,7 +96,7 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
// A Nexus outage during a Hexis act writes a failure trace, and a shared
// store is the one thing the Praxis path could inherit it through.
nexus.SetFault(503)
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); actRan(reply) {
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); actRan(reply) {
t.Fatalf("nexus outage must not report success, got %q", reply)
}
if len(tracesFor(t, h, "nexus", "resolve")) == 0 {
@@ -104,7 +104,7 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
}
nexus.SetFault(0)
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
if !strings.Contains(reply, "disk almost full") {
t.Fatalf("a recorded nexus failure must not degrade the praxis digest, got %q", reply)
}
@@ -114,10 +114,10 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
// And the reverse: a Praxis outage mid-session leaves the Hexis path whole.
praxis.SetFault(503)
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); strings.Contains(reply, "disk") {
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); strings.Contains(reply, "disk") {
t.Fatalf("praxis outage must not serve content, got %q", reply)
}
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
t.Fatalf("a praxis outage must not block the hexis path, got %q", reply)
}
}
@@ -132,7 +132,7 @@ func TestEcosystem_OneEndpointDownDoesNotMuteTheService(t *testing.T) {
h := ecoHandler(t, nil, praxis, nil)
praxis.SetRouteFault("/api/v1/tools/surface", 503)
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
if !strings.Contains(reply, "disk almost full") {
t.Fatalf("a downed surface endpoint must not mute the digest, got %q", reply)
}
@@ -150,7 +150,7 @@ func TestEcosystem_ResolvedWithoutEntityFailsClosed(t *testing.T) {
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if reply == "" {
t.Fatal("a resolve with no entity must degrade, not fall through to local execution")
}
@@ -172,7 +172,7 @@ func TestEcosystem_RejectedCredentialSaysSo(t *testing.T) {
h := ecoHandler(t, nexus, nil, hexis)
nexus.SetFault(status)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !strings.Contains(reply, "токен") {
t.Fatalf("http %d must read as a credential problem, got %q", status, reply)
}
@@ -193,7 +193,7 @@ func TestEcosystem_MalformedPraxisBodyDegrades(t *testing.T) {
h := ecoHandler(t, nil, praxis, nil)
praxis.SetBody(`[{"title":`)
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
if reply == "" {
t.Fatal("a malformed praxis body must not answer with silence")
}
@@ -211,7 +211,7 @@ func TestEcosystem_MalformedNexusResponseFailsClosed(t *testing.T) {
h := ecoHandler(t, nexus, nil, hexis)
nexus.SetBody(`{"status":"resolved","entity":`)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if reply == "" || actRan(reply) {
t.Fatalf("malformed nexus body must degrade, got %q", reply)
}
@@ -232,7 +232,7 @@ func TestEcosystem_UnknownContractFieldsTolerated(t *testing.T) {
nexus := newFakeNexus(t, body)
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
t.Fatalf("%s contract shape must still resolve and execute, got %q", name, reply)
}
})
@@ -249,7 +249,7 @@ func TestEcosystem_CancelledContextDegrades(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
defer cancel()
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if reply == "" || actRan(reply) {
t.Fatalf("cancelled resolve must degrade, got %q", reply)
}
@@ -267,7 +267,7 @@ func TestEcosystem_ExecutionFailureIsNotSuccess(t *testing.T) {
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecutionFailed("exec_1", "unit not found"))
h := ecoHandler(t, nexus, nil, hexis)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if actRan(reply) {
t.Fatalf("failed execution must not read as success, got %q", reply)
}
@@ -291,7 +291,7 @@ func TestEcosystem_SuccessfulActionWritesATrace(t *testing.T) {
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
t.Fatalf("setup: expected success, got %q", reply)
}
exec := tracesFor(t, h, "hexis", "execute")
@@ -313,7 +313,7 @@ func TestEcosystem_TracesStayOutOfFacts(t *testing.T) {
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
t.Fatalf("setup: expected success, got %q", reply)
}
if len(traces(t, h)) == 0 {
@@ -339,7 +339,7 @@ func TestEcosystem_AmbiguousTargetBlocksExecution(t *testing.T) {
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
reply := h.handleHexisAct(ctx, actDec("muzick"))
reply := h.handleHexisAct(ctx, actDec("muzick"), routeCandidate("restart"))
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Muzick web") {
t.Fatalf("ambiguous resolve must list candidates, got %q", reply)
}
@@ -360,7 +360,7 @@ func TestEcosystem_NoAutonomousPraxisToHexis(t *testing.T) {
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, praxis, hexis)
_ = h.handlePraxisAct(ctx, praxisActDec("list_attention"))
_ = h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
if hexis.Count("", "/api/v1") != 0 {
t.Fatal("attention digest must not contact hexis on its own")
}
@@ -378,7 +378,7 @@ func TestEcosystem_MutatingCapabilityWaitsForConfirmation(t *testing.T) {
hexis := newFakeHexis(t, caps, fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
reply := h.handleHexisAct(ctx, actDec("restart"))
reply := h.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart"))
if !strings.Contains(reply, "restart") || !strings.Contains(reply, "да") {
t.Fatalf("mutating capability must ask for confirmation, got %q", reply)
}
@@ -404,7 +404,7 @@ func TestEcosystem_SurfaceFailureStillDelivers(t *testing.T) {
praxis.SetRouteFault("/api/v1/tools/surface", 500)
h := ecoHandler(t, nil, praxis, nil)
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
if !strings.Contains(reply, "disk almost full") {
t.Fatalf("failed surface must not swallow the digest, got %q", reply)
}
@@ -426,10 +426,10 @@ func TestEcosystem_TotalOutageSaysSoForEveryPath(t *testing.T) {
h := ecoHandler(t, nexus, praxis, hexis)
for name, reply := range map[string]string{
"hexis act": h.handleHexisAct(ctx, actDec("muzick indexer")),
"attention": h.handlePraxisAct(ctx, praxisActDec("list_attention")),
"changes": h.handlePraxisAct(ctx, praxisActDec("list_changes")),
"acknowledge": h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1")),
"hexis act": h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")),
"attention": h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")),
"changes": h.handlePraxisAct(ctx, praxisActDec("list_changes"), routeCandidate("list_changes")),
"acknowledge": h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item")),
} {
if reply == "" {
t.Errorf("%s: total outage must not answer with silence", name)
@@ -458,11 +458,11 @@ func TestEcosystem_RecoveryAfterOutageNeedsNoRestart(t *testing.T) {
h := ecoHandler(t, nil, praxis, nil)
praxis.SetFault(503)
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); strings.Contains(reply, "disk") {
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); strings.Contains(reply, "disk") {
t.Fatalf("outage must not serve content, got %q", reply)
}
praxis.SetFault(0)
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); !strings.Contains(reply, "disk almost full") {
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); !strings.Contains(reply, "disk almost full") {
t.Fatalf("recovery must work on the next turn, got %q", reply)
}
}
+5 -5
View File
@@ -59,7 +59,7 @@ func TestHexisDiscovery401IsDeniedNotDown(t *testing.T) {
h := hexisGapHandler(t, nexus.URL, hexis.URL)
hexis.SetFault(401)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !denied(serviceHexis, reply) {
t.Fatalf("401 from hexis discovery: got %q, want the denied line naming Hexis", reply)
}
@@ -75,7 +75,7 @@ func TestHexisDiscoveryOutageIsDownNotDenied(t *testing.T) {
nexus := newFakeNexus(t, fixtureNexusResolved("ent_muzick", muzickIndexer, "service"))
h := hexisGapHandler(t, nexus.URL, unreachableURL)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !down(serviceHexis, reply) {
t.Fatalf("connection refused from hexis: got %q, want the outage line naming Hexis", reply)
}
@@ -96,7 +96,7 @@ func TestHexisExecute401IsDeniedNotCommandFailure(t *testing.T) {
// Discovery stays healthy; only the execute endpoint refuses. A blanket
// fault would never reach the site under test.
hexis.SetRouteFault("/api/v1/execute", 401)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !denied(serviceHexis, reply) {
t.Fatalf("401 from hexis execute: got %q, want the denied line naming Hexis", reply)
}
@@ -129,7 +129,7 @@ func TestHexisExecuteOutageIsDown(t *testing.T) {
t.Cleanup(hexis.Close)
h := hexisGapHandler(t, nexus.URL, hexis.URL)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !down(serviceHexis, reply) {
t.Fatalf("dropped connection on hexis execute: got %q, want the outage line", reply)
}
@@ -149,7 +149,7 @@ func TestHexisExecutionFailedStaysCommandFailure(t *testing.T) {
hexis := newFakeHexis(t, caps, fixtureHexisExecutionFailed("exec_1", "unit refused to start"))
h := hexisGapHandler(t, nexus.URL, hexis.URL)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if down(serviceHexis, reply) || denied(serviceHexis, reply) {
t.Fatalf("a failed execution must not be reported as an ecosystem gap, got %q", reply)
}
+12 -7
View File
@@ -19,6 +19,11 @@ func praxisActDec(fn string) router.Decision {
return router.Decision{Intent: router.IntentAct, Slots: router.Slots{Fn: fn, HasFn: true}}
}
// routeCandidate builds an ActionCandidate matching a route-resolved Decision.
func routeCandidate(fn string) router.ActionCandidate {
return router.ActionCandidate{Fn: fn, Source: router.ActionSourceRoute}
}
// praxisItemDec is praxisActDec for the lifecycle verbs, which need an item id
// in the value slot. Without one they answer "which item?" and never reach
// Praxis at all, which makes them useless for testing a Praxis outage.
@@ -46,7 +51,7 @@ func TestPraxisAttention_HappyPathSurfacesItems(t *testing.T) {
praxis := newFakePraxis(t, items)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
if !strings.Contains(reply, "disk almost full") {
t.Fatalf("expected attention digest to mention the item, got %q", reply)
}
@@ -77,7 +82,7 @@ func TestPraxisAttention_DegradedFailsClosedNotEmpty(t *testing.T) {
praxis.SetFault(500)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
if reply == "" {
t.Fatal("praxis outage must not produce an empty reply")
}
@@ -104,13 +109,13 @@ func TestFakeNexus_FaultInjectionThenRecovery(t *testing.T) {
}
nexus.SetFault(503)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if actRan(reply) {
t.Fatalf("nexus outage must not report success, got %q", reply)
}
nexus.SetFault(0)
reply = h.handleHexisAct(ctx, actDec("muzick indexer"))
reply = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !actRan(reply) {
t.Fatalf("expected success once nexus recovers, got %q", reply)
}
@@ -134,7 +139,7 @@ func TestPraxisEntityAttention_RemembersWhatItReadOut(t *testing.T) {
reply := h.handlePraxisAct(ctx, router.Decision{
Intent: router.IntentAct,
Slots: router.Slots{Fn: "entity_attention", HasFn: true, Value: "muzick indexer"},
})
}, routeCandidate("entity_attention"))
if !strings.Contains(reply, "indexer wedged") {
t.Fatalf("expected the scoped item to be read out, got %q", reply)
}
@@ -147,7 +152,7 @@ func TestPraxisEntityAttention_RemembersWhatItReadOut(t *testing.T) {
}
// The follow-up resolves against what he just heard, not the stale list.
if reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "last")); reply == "" {
if reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "last"), routeCandidate("resolve_item")); reply == "" {
t.Fatal("positional follow-up should have been claimed by praxis")
}
var body string
@@ -175,7 +180,7 @@ func TestHexisConfirm_KeepsOneCorrelationIDPerAction(t *testing.T) {
hexis := newFakeHexis(t, caps, fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
if reply := h.handleHexisAct(ctx, actDec("restart")); !strings.Contains(reply, "да") {
if reply := h.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart")); !strings.Contains(reply, "да") {
t.Fatalf("mutating capability must ask for confirmation, got %q", reply)
}
resolve := findTrace(t, h, "nexus", "resolve")
+11 -11
View File
@@ -73,7 +73,7 @@ func TestHexisMutatingRequiresConfirm(t *testing.T) {
caps := `[{"id":"cap_restart","name":"restart","read_only":false,"risk":"high"}]`
h, executed := newHexisTestHandler(t, resolved, caps)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !strings.Contains(reply, "да") {
t.Fatalf("mutating cap should ask to confirm, got %q", reply)
}
@@ -103,7 +103,7 @@ func TestHexisConfirmNoDoesNotExecute(t *testing.T) {
caps := `[{"id":"cap_restart","name":"restart","read_only":false}]`
h, executed := newHexisTestHandler(t, resolved, caps)
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
reply, handled := h.resolveConfirm(ctx, "нет")
if !handled || !strings.Contains(reply, "отменила") {
t.Fatalf("no should cancel, got handled=%v reply=%q", handled, reply)
@@ -119,7 +119,7 @@ func TestHexisReadOnlyExecutesImmediately(t *testing.T) {
caps := `[{"id":"cap_status","name":"restart","read_only":true}]`
h, executed := newHexisTestHandler(t, resolved, caps)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !*executed {
t.Fatal("read-only cap should execute without confirmation")
}
@@ -136,7 +136,7 @@ func TestHexisAmbiguousAsksClarification(t *testing.T) {
ambiguous := `{"status":"ambiguous","candidates":[{"entity_id":"ent_muzick","display_name":"Muzick indexer"},{"entity_id":"ent_manga","display_name":"Manga indexer"}]}`
h, executed := newHexisTestHandler(t, ambiguous, `[]`)
reply := h.handleHexisAct(ctx, actDec("the indexer"))
reply := h.handleHexisAct(ctx, actDec("the indexer"), routeCandidate("restart"))
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Manga indexer") {
t.Fatalf("ambiguous should list candidates, got %q", reply)
}
@@ -154,7 +154,7 @@ func TestHexisResolveFlatShapeAccepted(t *testing.T) {
caps := `[{"id":"cap_status","name":"restart","read_only":true}]`
h, executed := newHexisTestHandler(t, flat, caps)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !*executed {
t.Fatalf("flat-shaped resolved entity should still execute, got reply %q", reply)
}
@@ -183,7 +183,7 @@ func TestHexisNexusErrorFailsClosed(t *testing.T) {
ecosystem: stubEcosystem(nexus.URL, hexis.URL),
}
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if reply == "" {
t.Fatal("nexus dependency failure must not fall through with an empty reply")
}
@@ -216,7 +216,7 @@ func TestHexisUnavailableFailsClosed(t *testing.T) {
ecosystem: stubEcosystem(nexus.URL, hexis.URL),
}
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if reply == "" {
t.Fatal("hexis dependency failure must not fall through with an empty reply")
}
@@ -234,7 +234,7 @@ func TestHexisNotFoundStillFallsThrough(t *testing.T) {
notFound := `{"status":"not_found"}`
h, executed := newHexisTestHandler(t, notFound, `[]`)
reply := h.handleHexisAct(ctx, actDec("turn off the lights"))
reply := h.handleHexisAct(ctx, actDec("turn off the lights"), routeCandidate("restart"))
if reply != "" {
t.Fatalf("not_found resolution should fall through with empty reply, got %q", reply)
}
@@ -264,7 +264,7 @@ func TestHexisIrreversibleCapabilityIsNotRunFromVoice(t *testing.T) {
caps := `[{"id":"cap_wipe","name":"restart","read_only":false,"risk":"irreversible","requires_confirmation":true}]`
h, executed := newHexisTestHandler(t, resolved, caps)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if *executed {
t.Fatal("an irreversible capability ran from the voice path")
}
@@ -284,7 +284,7 @@ func TestHexisSafeCapabilityRunsOnItsDeclaredTier(t *testing.T) {
caps := `[{"id":"cap_status","name":"restart","read_only":true,"risk":"safe"}]`
h, executed := newHexisTestHandler(t, resolved, caps)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if !*executed {
t.Fatal("a capability Hexis calls safe should run")
}
@@ -301,7 +301,7 @@ func TestHexisUndeclaredTierStillConfirms(t *testing.T) {
caps := `[{"id":"cap_restart","name":"restart","read_only":false}]`
h, executed := newHexisTestHandler(t, resolved, caps)
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
if *executed {
t.Fatal("a mutating capability ran without a confirm")
}
+7 -7
View File
@@ -142,7 +142,7 @@ func TestEcosystemTrace_SuccessfulActionTracesEveryHop(t *testing.T) {
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
t.Fatalf("setup: expected success, got %q", reply)
}
@@ -186,7 +186,7 @@ func TestEcosystemTrace_OneCorrelationIDPerPraxisAction(t *testing.T) {
))
h := ecoHandler(t, nil, praxis, nil)
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); !strings.Contains(reply, "disk almost full") {
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); !strings.Contains(reply, "disk almost full") {
t.Fatalf("setup: expected the digest, got %q", reply)
}
@@ -218,7 +218,7 @@ func TestEcosystemTrace_FailuresAreTracedToo(t *testing.T) {
h := ecoHandler(t, nexus, nil, hexis)
nexus.SetFault(401)
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
d := findTrace(t, h, "nexus", "resolve")
if d == nil {
@@ -242,7 +242,7 @@ func TestEcosystemTrace_UnreachableIsNotRefused(t *testing.T) {
h := ecoHandler(t, nil, nil, nil)
h.ecosystem.nexus = newNexusClient("http://127.0.0.1:1")
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
d := findTrace(t, h, "nexus", "resolve")
if d == nil {
@@ -263,7 +263,7 @@ func TestEcosystemTrace_RedactsTheUtterance(t *testing.T) {
nexus := newFakeNexus(t, fixtureNexusNotFound())
h := ecoHandler(t, nexus, nil, nil)
_ = h.handleHexisAct(ctx, actDec("перезапусти кофемашину"))
_ = h.handleHexisAct(ctx, actDec("перезапусти кофемашину"), routeCandidate("restart"))
recorded := traces(t, h)
if len(recorded) == 0 {
@@ -295,7 +295,7 @@ func TestEcosystemTrace_AmbiguityAndConfirmationAreRecorded(t *testing.T) {
))
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, ambig, nil, hexis)
_ = h.handleHexisAct(ctx, actDec("muzick"))
_ = h.handleHexisAct(ctx, actDec("muzick"), routeCandidate("restart"))
if d := findTrace(t, h, "nexus", "resolve"); d == nil || d.Status != traceAmbig {
t.Fatalf("ambiguous resolve must be traced as such, got %+v", d)
}
@@ -303,7 +303,7 @@ func TestEcosystemTrace_AmbiguityAndConfirmationAreRecorded(t *testing.T) {
nexus := newFakeNexus(t, fixtureNexusResolved("ent_muzick", "Muzick indexer", "service"))
mutating := fixtureHexisCapabilities(map[string]any{"id": "cap_restart", "name": "restart", "read_only": false})
h2 := ecoHandler(t, nexus, nil, newFakeHexis(t, mutating, fixtureHexisExecuted("exec_1", "succeeded")))
_ = h2.handleHexisAct(ctx, actDec("restart"))
_ = h2.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart"))
d := findTrace(t, h2, "hexis", "confirmation")
if d == nil || d.Status != tracePending {
t.Fatalf("a parked confirmation must be traced, got %+v", d)
+45 -3
View File
@@ -91,7 +91,7 @@ func TestNexusIsAskedForTheNameHeSaid(t *testing.T) {
Intent: router.IntentAct,
Slots: router.Slots{Text: "перезагрузить музик индексер", Fn: "restart", HasFn: true},
}
h.handleHexisAct(ctx, dec)
h.handleHexisAct(ctx, dec, routeCandidate("restart"))
reqs := nexus.Requests()
if len(reqs) == 0 {
@@ -147,6 +147,48 @@ func TestClarifyStillAsksWithoutHexis(t *testing.T) {
}
}
// A verb is not an entity. Before the reach gate, an exact local matcher hit
// with no arguments still sent the raw verb to Nexus and could discover a
// similarly named entity through Hexis. The local tool lane may handle or
// reject it, but the ecosystem must not be consulted without a target.
func TestBareMatchedActNeverReachesNexus(t *testing.T) {
nexus := newFakeNexus(t, fixtureNexusResolved("ent_power", "Power", "service"))
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h, _, _ := newClarifyHandler(t)
h.ecosystem = ecoHandler(t, nexus, nil, hexis).ecosystem
reply := h.actionAct(context.Background(), router.Decision{
Utterance: "выключи",
Intent: router.IntentAct,
Slots: router.Slots{Fn: "выключи", HasFn: true, Text: "выключи"},
})
if len(nexus.Requests()) != 0 {
t.Fatalf("bare verb reached Nexus: %+v", nexus.Requests())
}
if reply == "" {
t.Fatal("bare act disappeared instead of staying in Maven's local lane")
}
}
func TestUnresolvedActNeverReachesNexusBeforeClarify(t *testing.T) {
nexus := newFakeNexus(t, fixtureNexusResolved("ent_it", "It", "service"))
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
h := ecoHandler(t, nexus, nil, hexis)
dec := router.Decision{
Utterance: "сделай это",
Intent: router.IntentAct,
Stage: 3,
Clarify: true,
Slots: router.Slots{Text: "сделай это"},
}
if reply := h.hexisBeforeClarify(context.Background(), dec); reply != "" {
t.Fatalf("unresolved act was answered by Hexis: %q", reply)
}
if len(nexus.Requests()) != 0 {
t.Fatalf("unresolved act reached Nexus: %+v", nexus.Requests())
}
}
// nexusInOrder serves one resolve answer per call, in order, so a test can say
// what Nexus knows about the first name and what it knows about the second. The
// last body repeats once the list runs out.
@@ -184,7 +226,7 @@ func TestTwoResolvedNamesAsk(t *testing.T) {
Intent: router.IntentAct,
Slots: router.Slots{Text: "перезагрузить нгинкс", Fn: "restart", HasFn: true},
}
reply := h.handleHexisAct(ctx, dec)
reply := h.handleHexisAct(ctx, dec, routeCandidate("restart"))
if !strings.Contains(reply, "nginx") || !strings.Contains(reply, "Muzick indexer") {
t.Fatalf("reply = %q, want both names she found", reply)
}
@@ -209,7 +251,7 @@ func TestTheNameNexusKnowsWins(t *testing.T) {
Intent: router.IntentAct,
Slots: router.Slots{Text: "перезагрузить нгинкс", Fn: "restart", HasFn: true},
}
reply := h.handleHexisAct(ctx, dec)
reply := h.handleHexisAct(ctx, dec, routeCandidate("restart"))
if reply == "" {
t.Fatal("the resolvable name must carry the act")
}
+10 -10
View File
@@ -33,7 +33,7 @@ func TestEntityAttention_ScopesPraxisByCanonicalID(t *testing.T) {
))
h := ecoHandler(t, nexus, praxis, nil)
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
if !strings.Contains(reply, "indexer queue is backing up") {
t.Fatalf("expected the scoped item in the reply, got %q", reply)
}
@@ -70,7 +70,7 @@ func TestEntityAttention_FoldsInLocalFactsForSameEntity(t *testing.T) {
t.Fatalf("ResolveFactEntity: %v", err)
}
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"), routeCandidate("entity_attention"))
if !strings.Contains(reply, "descaled in june") {
t.Fatalf("expected entity-scoped local facts in the reply, got %q", reply)
}
@@ -88,7 +88,7 @@ func TestEntityAttention_UnscopedPraxisResponseIsRefused(t *testing.T) {
))
h := ecoHandler(t, nexus, praxis, nil)
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
if strings.Contains(reply, "disk almost full") {
t.Fatalf("an unscoped response must not be read back as entity-scoped, got %q", reply)
}
@@ -112,7 +112,7 @@ func TestEntityAttention_ForeignItemsAreDropped(t *testing.T) {
praxis := newFakePraxis(t, mustJSON(mixed))
h := ecoHandler(t, nexus, praxis, nil)
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
if !strings.Contains(reply, "indexer queue is backing up") {
t.Fatalf("the matching item must be spoken, got %q", reply)
}
@@ -140,7 +140,7 @@ func TestEntityAttention_TruncationIsNamed(t *testing.T) {
}
}
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"), routeCandidate("entity_attention"))
if !strings.Contains(reply, "и это не всё") {
t.Fatalf("a truncated recall must say it is truncated, got %q", reply)
}
@@ -156,7 +156,7 @@ func TestEntityAttention_AmbiguousAsksInsteadOfGuessing(t *testing.T) {
praxis := newFakePraxis(t, fixturePraxisAttentionItems())
h := ecoHandler(t, nexus, praxis, nil)
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick"), routeCandidate("entity_attention"))
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Muzick web") {
t.Fatalf("ambiguous subject must ask, got %q", reply)
}
@@ -173,13 +173,13 @@ func TestEntityAttention_MissingAndDegradedAreDistinct(t *testing.T) {
praxis := newFakePraxis(t, fixturePraxisAttentionItems())
h := ecoHandler(t, nexus, praxis, nil)
missing := h.handlePraxisAct(ctx, entityAttentionDec("нечто"))
missing := h.handlePraxisAct(ctx, entityAttentionDec("нечто"), routeCandidate("entity_attention"))
if missing == "" {
t.Fatal("an unknown entity must still get an answer")
}
nexus.SetFault(503)
degraded := h.handlePraxisAct(ctx, entityAttentionDec("нечто"))
degraded := h.handlePraxisAct(ctx, entityAttentionDec("нечто"), routeCandidate("entity_attention"))
if degraded == missing {
t.Fatalf("outage and unknown-entity must not read the same: %q", degraded)
}
@@ -195,7 +195,7 @@ func TestEntityAttention_DelayedNexusDegradesNotHangs(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
defer cancel()
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
if reply == "" {
t.Fatal("a delayed resolve must still answer")
}
@@ -213,7 +213,7 @@ func TestEntityAttention_WithoutNexusSaysSo(t *testing.T) {
))
h := ecoHandler(t, nil, praxis, nil)
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
if strings.Contains(reply, "disk almost full") {
t.Fatalf("without nexus, items must not be passed off as entity-scoped, got %q", reply)
}
+6 -1
View File
@@ -106,7 +106,12 @@ func TestSystemSafetyScenarios(t *testing.T) {
hexis := newFakeHexis(t, fixtureHexisCapabilities(map[string]any{"id": "restart", "name": "restart", "read_only": false}), fixtureHexisExecuted("exec_1", "succeeded"))
h, _ := newSafetyHandler(t)
h.ecosystem = stubEcosystem(nexus.URL, hexis.URL)
reply := h.applyAction(ctx, router.Decision{Intent: router.IntentAct, Slots: router.Slots{Fn: "restart", HasFn: true, Text: "indexer"}})
// A matched function carries its entity target in Args. Text may be
// model phrasing, but Args is the production matcher contract and the
// ecosystem reach gate deliberately requires that evidence.
reply := h.applyAction(ctx, router.Decision{Intent: router.IntentAct, Slots: router.Slots{
Fn: "restart", HasFn: true, Args: []string{"indexer"}, Text: "indexer",
}})
if !strings.Contains(reply, "Indexer A") || !strings.Contains(reply, "Indexer B") {
t.Fatalf("ambiguous entity must prompt for clarification, got %q", reply)
}
+15 -5
View File
@@ -2,6 +2,9 @@ package main
import (
"context"
"net/http"
"net/url"
"strings"
"testing"
"time"
@@ -33,6 +36,10 @@ func newFactGateHandler(t *testing.T, now time.Time) (*reactiveHandler, ipc.Core
func TestActionFact_QuestionIsNotWritten(t *testing.T) {
ctx := context.Background()
h, api := newFactGateHandler(t, time.Now())
searchH, seen := searchHandler(t,
`{"answers":["Актуальная версия Go — 1.25."],"results":[]}`,
http.StatusOK)
h.search = searchH.search
reply := h.actionFact(ctx, router.Decision{
Intent: router.IntentFact,
@@ -50,11 +57,14 @@ func TestActionFact_QuestionIsNotWritten(t *testing.T) {
if len(hits) != 0 {
t.Fatalf("the question was indexed for recall: %+v", hits)
}
// It went down the query chain instead. Nothing is configured to answer a
// world question in this harness, so "не знаю." is the honest outcome —
// what matters is that the turn was answered, not stored.
if reply == "" {
t.Fatal("the turn was neither stored nor answered")
// It went down the world query chain instead. This asserts the actual
// destination, not merely that the write was refused: the regression was
// the personal boundary claiming this question before search.
if !strings.Contains(reply, "1.25") {
t.Fatalf("reply = %q, want live world evidence", reply)
}
if !strings.Contains(*seen, "q="+url.QueryEscape("какая последняя версия языка Go?")) {
t.Fatalf("search query = %q; world source was not reached verbatim", *seen)
}
}
+11 -9
View File
@@ -72,15 +72,16 @@ func dialogueIDOf(ctx context.Context) string {
// toDialogueSlots projects the router's slots onto the dialogue layer's copy.
func toDialogueSlots(s router.Slots) dialogue.Slots {
return dialogue.Slots{
Time: s.Time,
HasTime: s.HasTime,
Key: s.Key,
Value: s.Value,
HasKey: s.HasKey,
Text: s.Text,
Fn: s.Fn,
Args: s.Args,
HasFn: s.HasFn,
Time: s.Time,
HasTime: s.HasTime,
Key: s.Key,
Value: s.Value,
HasKey: s.HasKey,
Text: s.Text,
Fn: s.Fn,
Args: s.Args,
HasFn: s.HasFn,
ResolvedBy: string(s.ResolvedBy),
}
}
@@ -90,6 +91,7 @@ func applyDialogueSlots(base router.Slots, d dialogue.Slots) router.Slots {
base.Key, base.Value, base.HasKey = d.Key, d.Value, d.HasKey
base.Text = d.Text
base.Fn, base.Args, base.HasFn = d.Fn, d.Args, d.HasFn
base.ResolvedBy = router.ActionResolutionMethod(d.ResolvedBy)
return base
}
+59
View File
@@ -9,6 +9,7 @@ import (
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/lexicon"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
)
@@ -67,6 +68,44 @@ func parseOrdinal(text string) (int, bool) {
return 0, false
}
// parseReminderCancelChoice is intentionally narrower than parseOrdinal. A
// task ordinal may appear inside a sentence carrying its transition verb; the
// reminder list was already offered specifically for cancellation, so the next
// mutation requires the whole turn to be one affirmative position answer.
// Questions, negation, two positions and new requests all decline and route as
// fresh turns instead of cancelling whichever ordinal happened to appear.
func parseReminderCancelChoice(text string) (int, bool) {
if router.IsQuestionShaped(text) {
return 0, false
}
tokens := turnTokens(text)
nth, positions := 0, 0
for _, tok := range tokens {
if reminderCancelNegation(tok) {
return 0, false
}
if n, ok := candidateDigits[tok]; ok {
nth, positions = n, positions+1
continue
}
if n, ok := lexicon.Ordinal(tok); ok {
nth, positions = n, positions+1
continue
}
if lexicon.IsFillerParticle(tok) || reminderCancelVerbs[tok] ||
isReminderCancelTarget(tok) || reminderCancelFrame[tok] {
continue
}
switch tok {
case "номер", "вариант", "number", "option", "one":
continue
default:
return 0, false
}
}
return nth, positions == 1
}
// candidateVerbs — what he wants done with the one he picked. Nothing here is
// destructive: a task moves forward or is dropped, and both are recorded with a
// provenance the /tasks page shows.
@@ -112,7 +151,21 @@ func (h *reactiveHandler) resolveCandidate(ctx context.Context, text string, src
if sess == nil || len(sess.Candidates) == 0 {
return "", false
}
reminderList := true
for _, candidate := range sess.Candidates {
if candidate.Kind != "reminder-cancel" {
reminderList = false
break
}
}
if reminderList && classifyConfirm(text) == confirmNo {
h.dialogueSessions.SetCandidates(dialogueIDOf(ctx), h.now(), nil)
return "хорошо, ничего не отменяю.", true
}
nth, ok := parseOrdinal(text)
if reminderList {
nth, ok = parseReminderCancelChoice(text)
}
if !ok {
return "", false
}
@@ -125,6 +178,12 @@ func (h *reactiveHandler) resolveCandidate(ctx context.Context, text string, src
return fmt.Sprintf("я назвала только %d.", len(sess.Candidates)), true
}
pick := sess.Candidates[nth-1]
if pick.Kind == "reminder-cancel" {
// Unlike a task list, this list was offered in answer to the explicit
// question "which reminder should I cancel?" A bare ordinal is the
// answer to that question and therefore completes the cancellation.
return h.cancelReminderChoice(ctx, pick.Ref, pick.Label), true
}
status, say, hasVerb := parseCandidateVerb(text)
if !hasVerb || pick.Kind != "task" {
// Read it back and keep the list: naming one is often the first half of
+23 -6
View File
@@ -186,22 +186,39 @@ func TestPersonalBoundaryFrozenHeadDecodes(t *testing.T) {
}
}
func TestPersonalBoundaryHashFloorLatency(t *testing.T) {
func TestPersonalBoundaryHashFloorFitsAndScores(t *testing.T) {
b := &personalBoundary{}
embedder := router.NewHashEmbedder(1024)
query, err := router.EmbedQuery(context.Background(), embedder, "когда моя встреча")
if err != nil {
t.Fatal(err)
}
started := time.Now()
b.load(context.Background(), embedder)
if _, _, ok := b.score(query); !ok {
t.Fatal("hash-floor boundary declined to score")
}
elapsed := time.Since(started)
t.Logf("hash-floor corpus fit+score: %s", elapsed)
if elapsed > 2*time.Second {
t.Errorf("hash-floor boundary took %s, exceeds 2s local floor ceiling", elapsed)
if len(b.head.weights) != 1024 {
t.Fatalf("hash-floor boundary has %d weights, want 1024", len(b.head.weights))
}
}
// BenchmarkPersonalBoundaryHashFloorFitAndScore keeps startup cost measurable
// without making ambient CI load a correctness condition. In particular,
// -race and coverage instrumentation both multiply the cost of this numeric
// training loop; the functional test above is the deterministic gate.
func BenchmarkPersonalBoundaryHashFloorFitAndScore(b *testing.B) {
embedder := router.NewHashEmbedder(1024)
query, err := router.EmbedQuery(context.Background(), embedder, "когда моя встреча")
if err != nil {
b.Fatal(err)
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
boundary := &personalBoundary{}
boundary.load(context.Background(), embedder)
if _, _, ok := boundary.score(query); !ok {
b.Fatal("hash-floor boundary declined to score")
}
}
}
+3 -3
View File
@@ -19,7 +19,7 @@ func TestPraxisLifecycle401NamesPraxis(t *testing.T) {
h := newPraxisTestHandler(t, praxis)
praxis.SetFault(401)
reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "item_1"))
reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "item_1"), routeCandidate("resolve_item"))
if !strings.Contains(reply, servicePraxis) {
t.Fatalf("praxis failure does not name Praxis: %q", reply)
}
@@ -40,10 +40,10 @@ func TestPraxisLifecycleOutageDiffersFrom401(t *testing.T) {
h := newPraxisTestHandler(t, praxis)
praxis.SetFault(401)
refused := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"))
refused := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item"))
h.ecosystem = &ecosystemWiring{praxis: newPraxisClient(unreachableURL)}
outage := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"))
outage := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item"))
if refused == outage {
t.Fatalf("a refused token and an outage still say the same thing: %q", refused)
+14 -14
View File
@@ -17,7 +17,7 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
]`)
h := newPraxisTestHandler(t, praxis)
if reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention")); reply == "" {
if reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention")); reply == "" {
t.Fatal("attention returned nothing")
}
@@ -28,7 +28,7 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
}
for _, c := range cases {
praxis.ResetRequests()
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", c.ref))
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", c.ref), routeCandidate("acknowledge_item"))
if !strings.Contains(reply, "принято") {
t.Errorf("ref %q: reply %q", c.ref, reply)
}
@@ -42,10 +42,10 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
func TestPositionPastTheEndAsksInsteadOfGuessing(t *testing.T) {
praxis := newFakePraxis(t, `[{"id":"item_a","title":"диск заканчивается"}]`)
h := newPraxisTestHandler(t, praxis)
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
praxis.ResetRequests()
reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "4"))
reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "4"), routeCandidate("resolve_item"))
if !strings.Contains(reply, "какой пункт") {
t.Errorf("a position with no item should ask, got %q", reply)
}
@@ -59,7 +59,7 @@ func TestPositionWithNoSpokenListAsks(t *testing.T) {
praxis := newFakePraxis(t, `[]`)
h := newPraxisTestHandler(t, praxis)
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"))
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"), routeCandidate("acknowledge_item"))
if !strings.Contains(reply, "какой пункт") {
t.Errorf("want the ask, got %q", reply)
}
@@ -69,10 +69,10 @@ func TestPositionWithNoSpokenListAsks(t *testing.T) {
func TestExplicitItemIDIsNotRewritten(t *testing.T) {
praxis := newFakePraxis(t, `[{"id":"item_a","title":"диск"}]`)
h := newPraxisTestHandler(t, praxis)
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
praxis.ResetRequests()
h.handlePraxisAct(context.Background(), praxisItemDec("pin_item", "item_zz"))
h.handlePraxisAct(context.Background(), praxisItemDec("pin_item", "item_zz"), routeCandidate("pin_item"))
if !requestedPathContaining(praxis, "item_zz") {
t.Errorf("the id he gave was not the one called; paths %v", paths(praxis))
}
@@ -85,10 +85,10 @@ func TestUnspokenItemsHoldNoPosition(t *testing.T) {
{"id":"item_said","title":"бэкап не прошёл"}
]`)
h := newPraxisTestHandler(t, praxis)
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
praxis.ResetRequests()
h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"))
h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"), routeCandidate("acknowledge_item"))
if !requestedPathContaining(praxis, "item_said") {
t.Errorf("position 1 is the first item she SAID; paths %v", paths(praxis))
}
@@ -116,10 +116,10 @@ func requestedPathContaining(f *fakeServer, want string) bool {
func TestDemonstrativeResolvesWhenOneItemWasSpoken(t *testing.T) {
praxis := newFakePraxis(t, `[{"id":"item_only","title":"бэкап не прошёл"}]`)
h := newPraxisTestHandler(t, praxis)
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
praxis.ResetRequests()
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "this"))
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "this"), routeCandidate("acknowledge_item"))
if !strings.Contains(reply, "принято") {
t.Errorf("reply %q", reply)
}
@@ -136,10 +136,10 @@ func TestDemonstrativeWithSeveralItemsGivesTheTurnBack(t *testing.T) {
{"id":"item_b","title":"бэкап"}
]`)
h := newPraxisTestHandler(t, praxis)
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
praxis.ResetRequests()
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this")); reply != "" {
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this"), routeCandidate("resolve_item")); reply != "" {
t.Errorf("want a fall-through, got %q", reply)
}
for _, p := range paths(praxis) {
@@ -154,7 +154,7 @@ func TestDemonstrativeWithNoDigestGivesTheTurnBack(t *testing.T) {
praxis := newFakePraxis(t, `[]`)
h := newPraxisTestHandler(t, praxis)
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this")); reply != "" {
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this"), routeCandidate("resolve_item")); reply != "" {
t.Errorf("want a fall-through, got %q", reply)
}
}
+62 -2
View File
@@ -193,9 +193,9 @@ func TestQueryRecallNoteCanWin(t *testing.T) {
t.Run("the better-matching fact answers", func(t *testing.T) {
h, _ := buildRecallHandler(t, q, []recallCase{
{text: "молоко стоит в холодильнике", score: 0.80, kind: "note"},
{text: "купил молоко в среду", score: 0.95, kind: "fact"},
{text: "молоко было в холодильнике в среду", score: 0.95, kind: "fact"},
})
if reply := askQuery(t, h, q); reply != "купил молоко в среду" {
if reply := askQuery(t, h, q); reply != "молоко было в холодильнике в среду" {
t.Errorf("reply %q, want the fact read back", reply)
}
})
@@ -212,3 +212,63 @@ func TestQueryRecallNoteCanWin(t *testing.T) {
}
})
}
// TestQueryRecallRequiresStructuralOrTopicEvidence — the whole-assistant
// cold-start regression. The routing heads called an ordinary past-tense
// report a query; with one note in the store the margin gate has no runner-up,
// and cosine 0.825 was enough to speak a completely unrelated spare-key note.
// A bare question mark does not turn the proposition into an open information
// question, negation must not weaken the refusal, and a locative question must
// corroborate the target it asks Maven to locate (V-719).
func TestQueryRecallRequiresStructuralOrTopicEvidence(t *testing.T) {
const unrelated = "запомни: запасной ключ лежит в синей коробке"
for _, tc := range []struct {
query string
score float64
}{
{"я отменил напоминание про молоко", 0.825031306},
{"я отменил напоминание про молоко?", 0.825031306},
{"я не отменил напоминание про молоко", 0.825031306},
{"я не отменил напоминание про молоко?", 0.825031306},
{"где мой паспорт?", 0.817210},
{"где я отменил напоминание про молоко?", 0.805800},
{"где лежит синяя рубашка?", 0.837694},
{"где лежит синяя папка?", 0.837472},
{"где мой запасной паспорт?", 0.831662},
{"где лежит запасная флешка?", 0.838980},
{"где находится синяя коробка с документами?", 0.866553},
{"где лежит ключ от машины?", 0.843853},
{"где синяя коробка?", 0.90},
} {
t.Run(tc.query, func(t *testing.T) {
h, phr := buildRecallHandler(t, tc.query, []recallCase{
{text: unrelated, score: tc.score, kind: "note"},
})
reply := askQuery(t, h, tc.query)
if strings.Contains(reply, "запасной ключ") {
t.Fatalf("unrelated note escaped into reply %q", reply)
}
if len(phr.notes) != 0 {
t.Fatalf("unrelated note reached the phraser: %q", phr.notes)
}
})
}
// Voice punctuation is optional. A nominal request with no interrogative
// still works when the candidate itself corroborates the named topic.
const nominal = "адрес домашнего сервера"
h, _ := buildRecallHandler(t, nominal, []recallCase{
{text: "домашний сервер на 192.168.1.104", score: 0.90, kind: "note"},
})
if reply := askQuery(t, h, nominal); !strings.Contains(reply, "домашний сервер") {
t.Fatalf("nominal recall lost its shared-topic answer: %q", reply)
}
const locative = "где лежит запасной ключ?"
h, _ = buildRecallHandler(t, locative, []recallCase{
{text: "запасной ключ лежит в синей коробке", score: 0.90, kind: "note"},
})
if reply := askQuery(t, h, locative); !strings.Contains(reply, "запасной ключ") {
t.Fatalf("locative recall lost its corroborated target: %q", reply)
}
}
+67 -2
View File
@@ -8,6 +8,7 @@ import (
"github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/memory"
"github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
"github.com/kami/maven/internal/tool"
@@ -84,12 +85,76 @@ func TestReactiveNotesReminders(t *testing.T) {
t.Fatal("expected at least one note, got none")
}
last := notes[0]
if last.Text != "запомни что кофе закончился" {
t.Errorf("note text = %q, want %q", last.Text, "запомни что кофе закончился")
if last.Text != "кофе закончился" {
t.Errorf("note text = %q, want %q", last.Text, "кофе закончился")
}
})
}
// TestRunTurnExplicitNoteStoresOnlyTheBody pins the live failure end to end:
// a routed text turn reaches actionNote, stores only the dictated body in both
// durable and vector memory, and cannot ask the resident model to choose the
// acknowledgement's grammatical gender (V-721).
func TestRunTurnExplicitNoteStoresOnlyTheBody(t *testing.T) {
ctx := context.Background()
st := newTestStore(t)
api := ipc.NewStoreAPI(st)
now := time.Date(2026, 8, 15, 8, 0, 0, 0, time.FixedZone("+04", 4*60*60))
emb := router.NewHashEmbedder(1024)
mem := memory.NewInMemoryStore()
rtr := router.New(router.Config{
Grammars: []router.Grammar{{
Name: "explicit-note-test",
Decide: func(string) (router.Decision, bool) {
return router.Decision{
Stage: 0, Intent: router.IntentNote, Confidence: 1,
// Deliberately hostile model slot: neither persistence nor
// acknowledgement may use it.
Slots: router.Slots{Text: "ты поедешь на дачу"},
}, true
},
}},
Threshold: 0.55,
})
model := &countingCompleter{out: `{"response":"Хорошо, сохранил.","mood":"neutral"}`}
h := &reactiveHandler{
api: api, router: rtr,
recall: recallWiring{embedder: emb, memStore: mem},
replier: newLLMReplier(model, nil),
now: func() time.Time { return now },
dataStore: st,
}
const utterance = "запомни: запасной ключ лежит в синей коробке"
if reply := h.runTurn(ctx, router.NormalizedInput{Text: utterance, Source: sourceText}); reply != "сохранила заметку." {
t.Fatalf("reply = %q, want the fixed feminine acknowledgement", reply)
}
if model.calls != 0 {
t.Fatalf("resident model was called %d time(s) for a note acknowledgement", model.calls)
}
notes, err := st.RecentNotes(ctx, 10)
if err != nil {
t.Fatalf("RecentNotes: %v", err)
}
const body = "запасной ключ лежит в синей коробке"
if len(notes) != 1 || notes[0].Text != body || notes[0].Source != "tap:voice" || !notes[0].Ts.Equal(now) {
t.Fatalf("stored notes = %+v, want one exact body at the turn time", notes)
}
records, err := mem.ByPrefix(ctx, "note:")
if err != nil {
t.Fatalf("vector catalog: %v", err)
}
if len(records) != 1 || records[0].Meta["text"] != body {
t.Fatalf("vector records = %+v, want the same extracted body", records)
}
if records[0].Meta["text"] == utterance || records[0].Meta["text"] == "ты поедешь на дачу" {
t.Fatalf("vector metadata used a command or model rewrite: %+v", records[0].Meta)
}
if !phraser.IsAck(phraser.AckNote, nil, "сохранила заметку.") {
t.Fatal("fixed acknowledgement is not registered as the note acknowledgement")
}
}
// TestSpokenTaskCaptureFilesATask — the whole path, from the utterance to the
// task table. It went dead when the router started claiming the marker as an
// act: capture rides the note intent, so nothing below actionNote was ever
+1 -1
View File
@@ -56,7 +56,7 @@ type recallWiring struct {
// minScore — the note-recall confidence gate. Top cosine below this ⇒
// "I don't know" instead of a guess. Tuned for the ONNX embedder; a knob,
// not load-bearing math (same posture as the presence thresholds). Set by
// wireVoice from VoiceConfig; default 0.55.
// wireVoice from VoiceConfig; default 0.80.
minScore float64
// minMargin — the second half of that gate: how far the top hit must beat
+383
View File
@@ -0,0 +1,383 @@
package main
import (
"context"
"errors"
"fmt"
"log"
"strconv"
"strings"
"time"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/lexicon"
"github.com/kami/maven/internal/morph"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
)
// reminderCancelRequest exists to make the parser's contract explicit: a hit
// proves only that the turn is an addressed imperative naming the reminder
// store. Subject and time are resolved separately after that safety boundary.
type reminderCancelRequest struct{}
var reminderCancelVerbs = func() map[string]bool {
out := make(map[string]bool)
for _, word := range lexicon.ReminderCancelVerbs() {
out[strings.ToLower(word)] = true
}
return out
}()
var reminderCancelFrame = func() map[string]bool {
out := make(map[string]bool)
for _, word := range lexicon.ReminderCancelFrame() {
out[strings.ToLower(word)] = true
}
return out
}()
// isReminderCancelTarget is deliberately a noun test, not a substring test.
// A committed reminder must be named, otherwise "убери со стола" would reach
// the reminder store. Russian cases are grammar and go through morph; the
// English singular/plural forms are closed command vocabulary.
func isReminderCancelTarget(tok string) bool {
if morph.SameWord(tok, "напоминание") || morph.SameWord(tok, "будильник") {
return true
}
switch tok {
case "reminder", "reminders", "alarm", "alarms":
return true
default:
return false
}
}
// reminderCancelLead reports which words may precede the imperative without
// becoming a subject of their own. Filler/politeness vocabulary already has
// one home in the lexicon; Maven's name is an address, not a Russian class.
func reminderCancelLead(tok string) bool {
return lexicon.IsFillerParticle(tok) || tok == "мавен" || tok == "maven"
}
// parseReminderCancelRequest recognizes an exact cancel imperative at the
// start of the addressed command plus an explicit reminder noun. Both are
// whole tokens. Requiring command position is the safety boundary: infinitive
// questions ("как отменить ..."), reported speech ("он сказал: отмени ...")
// and past-tense remarks never reach the reminder store. A relative clause
// after a real command remains valid even though it may contain a question
// pronoun, so this is stronger and more precise than a punctuation test.
func parseReminderCancelRequest(text string) (reminderCancelRequest, bool) {
tokens := turnTokens(text)
verbAt := -1
for i, tok := range tokens {
if reminderCancelVerbs[tok] {
verbAt = i
break
}
}
if verbAt < 0 {
return reminderCancelRequest{}, false
}
for _, tok := range tokens[:verbAt] {
if !reminderCancelLead(tok) {
return reminderCancelRequest{}, false
}
}
for _, tok := range tokens[verbAt+1:] {
if isReminderCancelTarget(tok) {
return reminderCancelRequest{}, true
}
}
return reminderCancelRequest{}, false
}
func reminderCancelNegation(tok string) bool {
switch tok {
case "не", "ни", "not", "no", "don't", "dont":
return true
default:
return false
}
}
func reminderCancelTimeLead(tok string) bool {
switch tok {
case "в", "во", "на", "к", "ко", "через", "спустя",
"at", "in", "by", "until", "after", "before":
return true
default:
return false
}
}
func reminderCancelTimeUnit(tok string) bool {
if lexicon.IsHourUnit(tok) || lexicon.IsMinuteUnit(tok) {
return true
}
for _, part := range lexicon.PartsOfDay() {
if tok == part {
return true
}
}
return tok == "утра" || tok == "дня" || tok == "вечера" || tok == "ночи" ||
tok == "am" || tok == "pm" || tok == "noon" || tok == "midnight"
}
func reminderCancelNumeral(tok string) (int, bool) {
if n, ok := lexicon.Cardinal(tok); ok {
return n, true
}
if n, ok := lexicon.Ordinal(tok); ok && n > 0 {
return n, true
}
n, err := strconv.Atoi(tok)
return n, err == nil
}
// reminderClockTokenBudget records the numeric pieces that came from a written
// clock. turnTokens deliberately splits 21:30 into 21 and 30, so a small
// multiset lets subject extraction ignore exactly those occurrences without
// discarding the same number when it also belongs to the reminder text.
func reminderClockTokenBudget(text string) map[string]int {
out := make(map[string]int)
for _, field := range strings.Fields(strings.ToLower(text)) {
field = strings.Trim(field, ".,!?;()[]{}«»\"'")
hour, minute, ok := strings.Cut(field, ":")
if !ok || len(minute) != 2 {
continue
}
h, herr := strconv.Atoi(hour)
m, merr := strconv.Atoi(minute)
if herr != nil || merr != nil || h < 0 || h > 23 || m < 0 || m > 59 {
continue
}
out[hour]++
out[minute]++
}
return out
}
// reminderCancellationTerms keeps identity-bearing words, including negation
// and quantities. The old ownContent shortcut erased both, so "не звонить" and
// "звонить", or "одну таблетку" and "две таблетки", could select the same
// row. Time framing is removed only after the shared parser proved that this
// turn actually carries a readable time; numerals are removed only in a clock
// position, never merely because they are numbers.
func reminderCancellationTerms(text string, hasTime bool) []string {
tokens := turnTokens(text)
clockBudget := reminderClockTokenBudget(text)
out := make([]string, 0, len(tokens))
for i, tok := range tokens {
if reminderCancelVerbs[tok] || isReminderCancelTarget(tok) ||
reminderCancelFrame[tok] || lexicon.IsFillerParticle(tok) {
continue
}
if !hasTime || reminderCancelNegation(tok) {
out = append(out, tok)
continue
}
if clockBudget[tok] > 0 {
clockBudget[tok]--
continue
}
if _, numeric := reminderCancelNumeral(tok); numeric {
prevTime := i > 0 && reminderCancelTimeLead(tokens[i-1])
nextTime := i+1 < len(tokens) && reminderCancelTimeUnit(tokens[i+1])
if prevTime || nextTime {
continue
}
}
// frameWords is assembled exclusively from the closed time/grammar
// lexicons. At this point a time was parsed, and negation has already
// been preserved above, so these words identify the time rather than
// the stored reminder body.
if frameWords[tok] {
continue
}
out = append(out, tok)
}
return out
}
// reminderCancellationTime applies the same parse and resolved-hour gate as a
// newly created reminder. A time expression that is present but unread is not
// silently discarded: the caller asks for a clearer time instead of cancelling
// whichever row happens to match the remaining words.
func (h *reactiveHandler) reminderCancellationTime(ctx context.Context, text string) (time.Time, bool) {
if slots := h.extractor.Extract(ctx, router.IntentReminder, text, h.now()); slots.HasTime {
return slots.Time, true
}
if h.timeParser == nil {
return time.Time{}, false
}
parsed, ok, err := h.timeParser.Parse(ctx, text, h.now())
if err != nil || !ok || !router.ResolvedTheHour(text, parsed) {
return time.Time{}, false
}
return parsed, true
}
func reminderNextFire(r ipc.Reminder) time.Time {
if !r.NextFireTs.IsZero() {
return r.NextFireTs
}
return r.FireTs
}
// reminderTimeMatches lets state disambiguate a clock when the day was not
// named. "На девять" can therefore select the sole 09:00/21:00 reminder, but
// if both exist they both remain candidates and Maven asks. A named day or an
// interval denotes an absolute minute and must match that minute exactly.
func reminderTimeMatches(text string, parsed, fire time.Time) bool {
local := fire.In(parsed.Location())
if router.NamesADay(text) || router.NamesAnInterval(text) {
return local.Truncate(time.Minute).Equal(parsed.Truncate(time.Minute))
}
if router.HourIsAmbiguous(text) {
return local.Minute() == parsed.Minute() && local.Hour()%12 == parsed.Hour()%12
}
return local.Hour() == parsed.Hour() && local.Minute() == parsed.Minute()
}
func reminderTextMatchesTerms(r ipc.Reminder, terms []string) bool {
if len(terms) == 0 {
return true
}
words := turnTokens(store.ReminderText(r.Payload))
used := make([]bool, len(words))
for _, term := range terms {
found := false
for i, word := range words {
if used[i] {
continue
}
tn, tok := reminderCancelNumeral(term)
wn, wok := reminderCancelNumeral(word)
if term == word || morph.SameWord(term, word) || (tok && wok && tn == wn) {
used[i] = true
found = true
break
}
}
if !found {
return false
}
}
return true
}
func reminderCancellationLabel(r ipc.Reminder, now time.Time) string {
fire := reminderNextFire(r).In(now.Location())
when := dayPrefix(now, fire)
if when == "это" {
when = fmt.Sprintf("%d %s", fire.Day(), lexicon.MonthGenitive(int(fire.Month())))
}
return fmt.Sprintf("%s в %s — %s", when, fire.Format("15:04"), store.ReminderText(r.Payload))
}
// offerReminderCancellations binds exactly the rows Maven names, in that order.
// An ordinal on the next turn therefore points at the spoken list, never at a
// fresh query whose order may have changed in between.
func (h *reactiveHandler) offerReminderCancellations(ctx context.Context, text string, matches []ipc.Reminder) string {
const maxSpoken = 5
truncated := len(matches) > maxSpoken
if len(matches) > maxSpoken {
matches = matches[:maxSpoken]
}
candidates := make([]dialogue.Candidate, 0, len(matches))
parts := make([]string, 0, len(matches))
for i, r := range matches {
label := reminderCancellationLabel(r, h.now())
candidates = append(candidates, dialogue.Candidate{Kind: "reminder-cancel", Ref: r.ID, Label: label})
parts = append(parts, fmt.Sprintf("%d: %s", i+1, label))
}
if h.dialogueSessions == nil {
return "нашла несколько подходящих напоминаний — уточни текст или время."
}
id, now := dialogueIDOf(ctx), h.now()
// This command is its own turn. Reusing an older session would keep stale
// intent/slots alive after the choice and let the next utterance inherit
// unrelated state, so the offered list gets a fresh system session.
h.dialogueSessions.Put(id, &dialogue.Session{
Intent: dialogue.IntentSystem, Utterance: text, Timestamp: now,
Candidates: candidates,
})
prefix := "нашла несколько подходящих. какое отменить? "
if truncated {
prefix = "нашла больше пяти подходящих; называю первые пять. если нужного здесь нет, уточни текст или время. какое отменить? "
}
return prefix + strings.Join(parts, "; ") + ". ответь одним порядковым словом, например «второе»."
}
func (h *reactiveHandler) clearReminderCandidates(ctx context.Context) {
if h.dialogueSessions != nil {
h.dialogueSessions.SetCandidates(dialogueIDOf(ctx), h.now(), nil)
}
}
func (h *reactiveHandler) cancelReminderChoice(ctx context.Context, id int64, label string) string {
if err := h.api.CancelReminder(ctx, id); err != nil {
switch {
case errors.Is(err, ipc.ErrReminderNotFound), errors.Is(err, ipc.ErrReminderState):
h.clearReminderCandidates(ctx)
return "это напоминание уже не ожидает отправки."
case errors.Is(err, ipc.ErrReminderInFlight):
h.clearReminderCandidates(ctx)
return "я уже начала отправлять это напоминание — надёжно отменить его уже нельзя."
default:
log.Printf("voice: cancel reminder %d: %v", id, err)
return "не получилось отменить напоминание."
}
}
h.clearReminderCandidates(ctx)
log.Printf("voice: cancelled reminder %d (%q)", id, label)
return "отменила напоминание: " + label + "."
}
// resolveReminderCancellation is the stateful pre-route resolver for a
// committed reminder. It claims only the explicit structural command above,
// resolves against every pending row, and never ranks an ambiguous set down to
// one. One match cancels; more than one is an offered, ordinal-bound question.
func (h *reactiveHandler) resolveReminderCancellation(ctx context.Context, text string) (string, bool) {
_, ok := parseReminderCancelRequest(text)
if !ok {
return "", false
}
rows, err := h.api.ListPendingReminders(ctx, 0)
if err != nil {
log.Printf("voice: list reminders for cancellation: %v", err)
return "не получилось посмотреть напоминания.", true
}
if len(rows) == 0 {
return "ожидающих напоминаний нет.", true
}
parsed, hasTime := h.reminderCancellationTime(ctx, text)
if router.MentionsTime(text) && !hasTime {
return "не смогла разобрать время напоминания — уточни его.", true
}
terms := reminderCancellationTerms(text, hasTime)
matches := make([]ipc.Reminder, 0, len(rows))
for _, r := range rows {
if !reminderTextMatchesTerms(r, terms) {
continue
}
if hasTime && !reminderTimeMatches(text, parsed, reminderNextFire(r)) {
continue
}
matches = append(matches, r)
}
switch len(matches) {
case 0:
return "не нашла такого ожидающего напоминания.", true
case 1:
label := reminderCancellationLabel(matches[0], h.now())
return h.cancelReminderChoice(ctx, matches[0].ID, label), true
default:
return h.offerReminderCancellations(ctx, text, matches), true
}
}
+425
View File
@@ -0,0 +1,425 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/decision"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
"github.com/kami/maven/internal/tts"
)
func TestParseReminderCancelRequest(t *testing.T) {
for _, tc := range []struct {
text string
ok bool
}{
{"отмени напоминание про врача", true},
{"убери моё напоминание о визите", true},
{"удали будильник на девять", true},
{"пожалуйста, Maven, cancel the reminder about doctor", true},
{"отмени напоминание, которое стоит на завтра", true},
{"напоминание про врача", false},
{"отмени задачу про врача", false},
{"я отменил напоминание про врача", false},
{"как отменить напоминание про врача?", false},
{"можно отменить напоминание про врача?", false},
{"он сказал: отмени напоминание про врача", false},
{"how to cancel the reminder about doctor?", false},
{"can you cancel the reminder about doctor?", false},
{"убери со стола", false},
{"отмена", false},
} {
_, ok := parseReminderCancelRequest(tc.text)
if ok != tc.ok {
t.Errorf("parseReminderCancelRequest(%q) ok = %v, want %v", tc.text, ok, tc.ok)
}
}
}
func TestReminderCancellationTermsPreserveIdentity(t *testing.T) {
for _, tc := range []struct {
text string
hasTime bool
want []string
}{
{"отмени напоминание про врача", false, []string{"врача"}},
{"отмени напоминание не звонить врачу", false, []string{"не", "звонить", "врачу"}},
{"отмени напоминание принять две таблетки", false, []string{"принять", "две", "таблетки"}},
{"отмени напоминание принять две таблетки на девять", true, []string{"принять", "две", "таблетки"}},
{"cancel the reminder to take 2 pills at 21:30", true, []string{"take", "2", "pills"}},
} {
got := reminderCancellationTerms(tc.text, tc.hasTime)
if strings.Join(got, "|") != strings.Join(tc.want, "|") {
t.Errorf("reminderCancellationTerms(%q) = %v, want %v", tc.text, got, tc.want)
}
}
}
func seedVoiceReminder(t *testing.T, st *store.Store, fire time.Time, text string) int64 {
t.Helper()
id, err := st.CreateReminder(context.Background(), fire, `{"text":"`+text+`"}`, "")
if err != nil {
t.Fatalf("create reminder: %v", err)
}
return id
}
func reminderStatuses(t *testing.T, st *store.Store) map[int64]string {
t.Helper()
rows, err := st.ListReminders(context.Background(), 100)
if err != nil {
t.Fatalf("list reminders: %v", err)
}
out := make(map[int64]string, len(rows))
for _, row := range rows {
out[row.ID] = row.Status
}
return out
}
func TestReminderCancellationResolvesSubjectByMorphology(t *testing.T) {
h, st, now := newClarifyHandler(t)
h.timeParser = router.StubDateTimeParser{}
doctor := seedVoiceReminder(t, st, now.Add(3*time.Hour), "позвонить врачу")
bread := seedVoiceReminder(t, st, now.Add(4*time.Hour), "купить хлеб")
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание про врача")
if !handled || !strings.Contains(reply, "отменила") || !strings.Contains(reply, "позвонить врачу") {
t.Fatalf("reply = %q, handled=%v", reply, handled)
}
statuses := reminderStatuses(t, st)
if statuses[doctor] != store.ReminderCancelled || statuses[bread] != store.ReminderPending {
t.Fatalf("statuses = %+v, want doctor cancelled and bread pending", statuses)
}
}
func TestReminderCancellationKeepsNegationAndQuantityDistinct(t *testing.T) {
t.Run("negation", func(t *testing.T) {
h, st, now := newClarifyHandler(t)
positive := seedVoiceReminder(t, st, now.Add(time.Hour), "звонить врачу")
negative := seedVoiceReminder(t, st, now.Add(2*time.Hour), "не звонить врачу")
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание не звонить врачу")
if !handled || !strings.Contains(reply, "не звонить врачу") {
t.Fatalf("reply = %q, handled=%v", reply, handled)
}
statuses := reminderStatuses(t, st)
if statuses[positive] != store.ReminderPending || statuses[negative] != store.ReminderCancelled {
t.Fatalf("negation selected the wrong row: %+v", statuses)
}
})
t.Run("quantity", func(t *testing.T) {
h, st, now := newClarifyHandler(t)
one := seedVoiceReminder(t, st, now.Add(time.Hour), "принять одну таблетку")
two := seedVoiceReminder(t, st, now.Add(2*time.Hour), "принять две таблетки")
reply, handled := h.resolveReminderCancellation(context.Background(), "удали напоминание принять две таблетки")
if !handled || !strings.Contains(reply, "две таблетки") {
t.Fatalf("reply = %q, handled=%v", reply, handled)
}
statuses := reminderStatuses(t, st)
if statuses[one] != store.ReminderPending || statuses[two] != store.ReminderCancelled {
t.Fatalf("quantity selected the wrong row: %+v", statuses)
}
})
}
func TestReminderCancellationQuestionNeverMutates(t *testing.T) {
h, st, now := newClarifyHandler(t)
id := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
for _, text := range []string{
"как отменить напоминание про врача?",
"можно отменить напоминание про врача?",
"он сказал: отмени напоминание про врача",
} {
if reply, handled := h.resolveReminderCancellation(context.Background(), text); handled || reply != "" {
t.Fatalf("non-command %q was claimed: reply=%q handled=%v", text, reply, handled)
}
if got := reminderStatuses(t, st)[id]; got != store.ReminderPending {
t.Fatalf("non-command %q changed reminder to %q", text, got)
}
}
}
func TestReminderCancellationUsesClockAndAsksWhenStateIsAmbiguous(t *testing.T) {
t.Run("one matching half of day is enough", func(t *testing.T) {
h, st, now := newClarifyHandler(t)
h.timeParser = router.StubDateTimeParser{}
evening := seedVoiceReminder(t, st, time.Date(now.Year(), now.Month(), now.Day(), 21, 0, 0, 0, now.Location()), "вечернее лекарство")
seedVoiceReminder(t, st, now.Add(2*time.Hour), "купить хлеб")
reply, handled := h.resolveReminderCancellation(context.Background(), "убери напоминание на девять")
if !handled || !strings.Contains(reply, "отменила") {
t.Fatalf("reply = %q, handled=%v", reply, handled)
}
if got := reminderStatuses(t, st)[evening]; got != store.ReminderCancelled {
t.Fatalf("21:00 status = %q, want cancelled", got)
}
})
t.Run("two matching halves are offered and ordinal is bound", func(t *testing.T) {
h, st, now := newClarifyHandler(t)
h.timeParser = router.StubDateTimeParser{}
evening := seedVoiceReminder(t, st, time.Date(now.Year(), now.Month(), now.Day(), 21, 0, 0, 0, now.Location()), "вечернее лекарство")
morning := seedVoiceReminder(t, st, time.Date(now.Year(), now.Month(), now.Day()+1, 9, 0, 0, 0, now.Location()), "утреннее лекарство")
reply, handled := h.resolveReminderCancellation(context.Background(), "убери напоминание на девять")
if !handled || !strings.Contains(reply, "порядковым словом") {
t.Fatalf("ambiguous reply = %q, handled=%v", reply, handled)
}
statuses := reminderStatuses(t, st)
if statuses[evening] != store.ReminderPending || statuses[morning] != store.ReminderPending {
t.Fatalf("ambiguous command mutated rows: %+v", statuses)
}
sess := h.dialogueSessions.Get(dialogueIDOf(context.Background()), h.now())
if sess == nil || len(sess.Candidates) != 2 || sess.Candidates[1].Ref != morning {
t.Fatalf("bound candidates = %+v", sess)
}
reply, handled = h.resolveCandidate(context.Background(), "второе", sourceVoice)
if !handled || !strings.Contains(reply, "утреннее лекарство") {
t.Fatalf("ordinal reply = %q, handled=%v", reply, handled)
}
statuses = reminderStatuses(t, st)
if statuses[evening] != store.ReminderPending || statuses[morning] != store.ReminderCancelled {
t.Fatalf("ordinal cancelled the wrong row: %+v", statuses)
}
if sess := h.dialogueSessions.Get(dialogueIDOf(context.Background()), h.now()); sess == nil || len(sess.Candidates) != 0 {
t.Fatalf("spent candidates survived: %+v", sess)
}
})
}
func TestReminderCancellationChoiceRequiresAWholeAffirmativeOrdinal(t *testing.T) {
unsafe := []string{
"почему второе?",
"не второе",
"второе не отменяй",
"первое и второе",
"напомни мне первого сентября оплатить счёт",
}
for _, answer := range unsafe {
t.Run(answer, func(t *testing.T) {
h, st, now := newClarifyHandler(t)
h.timeParser = router.StubDateTimeParser{}
first := seedVoiceReminder(t, st, now.Add(time.Hour), "первое лекарство")
second := seedVoiceReminder(t, st, now.Add(2*time.Hour), "второе лекарство")
if _, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание"); !handled {
t.Fatal("ambiguous cancellation was not offered")
}
if reply, handled := h.resolveCandidate(context.Background(), answer, sourceVoice); handled || reply != "" {
t.Fatalf("unsafe answer was claimed: reply=%q handled=%v", reply, handled)
}
statuses := reminderStatuses(t, st)
if statuses[first] != store.ReminderPending || statuses[second] != store.ReminderPending {
t.Fatalf("unsafe answer mutated rows: %+v", statuses)
}
})
}
}
func TestReminderCancellationChoiceCanBeAbandoned(t *testing.T) {
for _, answer := range []string{"отмена", "не надо", "no"} {
t.Run(answer, func(t *testing.T) {
h, st, now := newClarifyHandler(t)
first := seedVoiceReminder(t, st, now.Add(time.Hour), "первое")
second := seedVoiceReminder(t, st, now.Add(2*time.Hour), "второе")
if _, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание"); !handled {
t.Fatal("ambiguous cancellation was not offered")
}
reply, handled := h.resolveCandidate(context.Background(), answer, sourceVoice)
if !handled || !strings.Contains(reply, "ничего не отменяю") {
t.Fatalf("cancel answer = %q handled=%v", reply, handled)
}
statuses := reminderStatuses(t, st)
if statuses[first] != store.ReminderPending || statuses[second] != store.ReminderPending {
t.Fatalf("abandoning the choice mutated rows: %+v", statuses)
}
if sess := h.dialogueSessions.Get(dialogueIDOf(context.Background()), h.now()); sess == nil || len(sess.Candidates) != 0 {
t.Fatalf("abandoned candidates survived: %+v", sess)
}
})
}
}
func TestReminderCancellationOfferStartsFreshAndNamesTruncation(t *testing.T) {
h, st, now := newClarifyHandler(t)
id := dialogueIDOf(context.Background())
h.dialogueSessions.Put(id, &dialogue.Session{
Intent: dialogue.IntentReminder,
Slots: dialogue.Slots{Text: "stale subject", HasTime: true, Time: now.Add(time.Hour)},
Timestamp: now.Add(-time.Minute),
})
for i := 0; i < 6; i++ {
seedVoiceReminder(t, st, now.Add(time.Duration(i+1)*time.Hour), fmt.Sprintf("row %d", i+1))
}
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание")
if !handled || !strings.Contains(reply, "первые пять") || !strings.Contains(reply, "уточни текст или время") {
t.Fatalf("truncated offer = %q handled=%v", reply, handled)
}
sess := h.dialogueSessions.Get(id, h.now())
if sess == nil || sess.Intent != dialogue.IntentSystem || sess.Slots.Text != "" ||
len(sess.Candidates) != 5 || sess.Utterance != "отмени напоминание" {
t.Fatalf("offer reused stale dialogue state: %+v", sess)
}
}
func TestReminderCancellationNeverGuesses(t *testing.T) {
t.Run("bare command over several rows", func(t *testing.T) {
h, st, now := newClarifyHandler(t)
first := seedVoiceReminder(t, st, now.Add(time.Hour), "первое")
second := seedVoiceReminder(t, st, now.Add(2*time.Hour), "второе")
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание")
if !handled || !strings.Contains(reply, "порядковым словом") {
t.Fatalf("reply = %q, handled=%v", reply, handled)
}
statuses := reminderStatuses(t, st)
if statuses[first] != store.ReminderPending || statuses[second] != store.ReminderPending {
t.Fatalf("bare ambiguous command mutated rows: %+v", statuses)
}
})
t.Run("unread time", func(t *testing.T) {
h, st, now := newClarifyHandler(t)
h.timeParser = router.StubDateTimeParser{}
id := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание через вечность")
if !handled || !strings.Contains(reply, "не смогла разобрать время") {
t.Fatalf("reply = %q, handled=%v", reply, handled)
}
if got := reminderStatuses(t, st)[id]; got != store.ReminderPending {
t.Fatalf("unread time cancelled reminder: %q", got)
}
})
}
type cancelReminderAPI struct {
ipc.UnimplementedCoreAPI
rows []ipc.Reminder
listErr error
cancelErr error
calls []int64
}
func (a *cancelReminderAPI) ListPendingReminders(context.Context, int) ([]ipc.Reminder, error) {
return a.rows, a.listErr
}
func (a *cancelReminderAPI) CancelReminder(_ context.Context, id int64) error {
a.calls = append(a.calls, id)
return a.cancelErr
}
func cancelHandler(api ipc.CoreAPI) *reactiveHandler {
now := time.Date(2026, 8, 15, 9, 0, 0, 0, time.UTC)
parser := router.StubDateTimeParser{}
return &reactiveHandler{
api: api, now: func() time.Time { return now }, timeParser: parser,
extractor: router.Extractor{Time: parser},
dialogueSessions: dialogue.NewSessionStore(2 * time.Minute),
}
}
func TestReminderCancellationReportsStoreOutcomes(t *testing.T) {
row := ipc.Reminder{
ID: 7, FireTs: time.Date(2026, 8, 15, 12, 0, 0, 0, time.UTC),
NextFireTs: time.Date(2026, 8, 15, 12, 0, 0, 0, time.UTC),
Payload: `{"text":"позвонить врачу"}`, Status: store.ReminderPending,
}
for _, tc := range []struct {
name string
err error
want string
}{
{"already terminal", ipc.ErrReminderState, "уже не ожидает"},
{"delivery in flight", ipc.ErrReminderInFlight, "уже начала отправлять"},
{"transport", errors.New("socket closed"), "не получилось отменить"},
} {
t.Run(tc.name, func(t *testing.T) {
api := &cancelReminderAPI{rows: []ipc.Reminder{row}, cancelErr: tc.err}
reply, handled := cancelHandler(api).resolveReminderCancellation(context.Background(), "отмени напоминание про врача")
if !handled || !strings.Contains(reply, tc.want) || len(api.calls) != 1 || api.calls[0] != 7 {
t.Fatalf("reply=%q handled=%v calls=%v", reply, handled, api.calls)
}
})
}
t.Run("list failure", func(t *testing.T) {
api := &cancelReminderAPI{listErr: errors.New("offline")}
reply, handled := cancelHandler(api).resolveReminderCancellation(context.Background(), "отмени напоминание")
if !handled || !strings.Contains(reply, "не получилось посмотреть") || len(api.calls) != 0 {
t.Fatalf("reply=%q handled=%v calls=%v", reply, handled, api.calls)
}
})
t.Run("nothing pending", func(t *testing.T) {
api := &cancelReminderAPI{}
reply, handled := cancelHandler(api).resolveReminderCancellation(context.Background(), "отмени напоминание")
if !handled || !strings.Contains(reply, "ожидающих напоминаний нет") || len(api.calls) != 0 {
t.Fatalf("reply=%q handled=%v calls=%v", reply, handled, api.calls)
}
})
}
func TestReminderCancellationIsAPreRouteTurnAndDoesNotGetSwallowedByClarify(t *testing.T) {
h, st, now := newClarifyHandler(t)
h.timeParser = router.StubDateTimeParser{}
h.decisions = decision.NewRing()
id := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
ctx := withDialogueID(context.Background(), dialogueIDFor(sourceText, "web"))
h.clarifyStore.Put(dialogueIDOf(ctx), &dialogue.PendingQuestion{
Intent: dialogue.IntentReminder, Missing: []dialogue.Slot{dialogue.SlotTime},
Utterance: "напомни позвонить маме", Asked: h.now(), TTL: clarifyTTL,
})
reply := h.runTurn(ctx, router.NormalizedInput{Text: "отмени напоминание про врача", Source: sourceText})
if !strings.Contains(reply, clarifyDropped) || !strings.Contains(reply, "отменила напоминание") {
t.Fatalf("reply = %q, want dropped clarify notice and cancellation", reply)
}
if h.clarifyStore.Get(dialogueIDOf(ctx), h.now()) != nil {
t.Fatal("the superseded clarify question survived the cancellation request")
}
if got := reminderStatuses(t, st)[id]; got != store.ReminderCancelled {
t.Fatalf("status = %q, want cancelled", got)
}
recs := h.decisions.Recent(1)
if len(recs) != 1 {
t.Fatalf("decision records = %d, want 1", len(recs))
}
claim := findClaim(recs[0], "reminder-cancel")
if claim == nil || claim.Outcome != decision.Won {
t.Fatalf("reminder-cancel claim = %+v, want pre-route winner", claim)
}
}
func TestReminderCancellationThroughPushToTalk(t *testing.T) {
h, st, now := newClarifyHandler(t)
h.stt = simTranscriber{text: "отмени напоминание про врача"}
h.tts = tts.NewStub()
h.timeParser = router.StubDateTimeParser{}
h.router = buildRouter(router.NewHashEmbedder(64), h.matcher, 0.55, nil, nil)
doctor := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
bread := seedVoiceReminder(t, st, now.Add(2*time.Hour), "купить хлеб")
resp, err := h.HandlePushToTalk(context.Background(), voicePTT(), 0)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(resp.ReplyText, "отменила напоминание") || len(resp.ReplyAudio.Bytes) == 0 {
t.Fatalf("PTT response = text %q audio=%d bytes", resp.ReplyText, len(resp.ReplyAudio.Bytes))
}
statuses := reminderStatuses(t, st)
if statuses[doctor] != store.ReminderCancelled || statuses[bread] != store.ReminderPending {
t.Fatalf("PTT cancellation changed the wrong rows: %+v", statuses)
}
}
+3 -3
View File
@@ -149,7 +149,7 @@ func TestRepairResumesQuestionParkedAfterTheCorrectedTurn(t *testing.T) {
t.Fatal("expected a parked reminder question")
}
reply := h.runTurn(ctx, "нет, это был вопрос", sourceText)
reply := h.runTurn(ctx, router.NormalizedInput{Text: "нет, это был вопрос", Source: sourceText})
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
if !strings.HasSuffix(reply, resumed) {
t.Fatalf("the correction hid the still-live question: reply=%q want suffix=%q", reply, resumed)
@@ -176,14 +176,14 @@ func TestRepairedClarifyCompletesWithoutDroppingTheOlderQuestion(t *testing.T) {
t.Fatal("expected the older reminder question")
}
if reply := h.runTurn(ctx, "нет, это было напоминание", sourceText); !strings.Contains(reply, "Когда") {
if reply := h.runTurn(ctx, router.NormalizedInput{Text: "нет, это было напоминание", Source: sourceText}); !strings.Contains(reply, "Когда") {
t.Fatalf("the repaired reminder did not ask for its missing time: %q", reply)
}
if depth := h.clarifyStore.Depth(voiceDialogueID); depth != 2 {
t.Fatalf("the repaired question overwrote the older one: depth=%d want=2", depth)
}
reply := h.runTurn(ctx, "сегодня в 15:00", sourceText)
reply := h.runTurn(ctx, router.NormalizedInput{Text: "сегодня в 15:00", Source: sourceText})
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
if !strings.HasSuffix(reply, resumed) {
t.Fatalf("completing the repaired request did not resume the older one: reply=%q", reply)
+7
View File
@@ -39,6 +39,13 @@ func (r *llmReplier) Reply(ctx context.Context, d router.Decision) string {
// что ты выпел стакан воды" for "я выпил воды".
return phraser.FactAck(d.Utterance)
}
if d.Intent == router.IntentNote {
// A successful durable write needs no generation. The resident model
// answered one live capture with masculine self-reference ("сохранил")
// despite the prompt; the hand-written line is both faster and a hard
// persona guarantee on the daemon's reply path (V-721).
return phraser.Ack(phraser.AckNote, nil)
}
out, err := r.p.PhraseReply(ctx, d)
if err != nil || out == "" {
return r.stub.Reply(ctx, d)
+28 -4
View File
@@ -20,22 +20,46 @@ type stubCompleter struct {
func (s stubCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return s.out, s.err }
func TestLLMReplierPassesTheModelReplyThrough(t *testing.T) {
func TestLLMReplierPassesTheModelReplyThroughForOtherIntents(t *testing.T) {
r := newLLMReplier(stubCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}, nil)
got := r.Reply(context.Background(), router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
got := r.Reply(context.Background(), router.Decision{Intent: router.IntentReminder, Slots: router.Slots{Text: "кофе закончился"}})
if got != "записала, кофе закончился" {
t.Errorf("got %q, want %q", got, "записала, кофе закончился")
}
}
type countingCompleter struct {
out string
calls int
}
func (c *countingCompleter) Complete(_ context.Context, _ llm.Req) (string, error) {
c.calls++
return c.out, nil
}
func TestLLMReplierNoteUsesFixedFeminineAcknowledgement(t *testing.T) {
c := &countingCompleter{out: `{"response":"Хорошо, сохранил.","mood":"neutral"}`}
r := newLLMReplier(c, nil)
got := r.Reply(context.Background(), router.Decision{
Intent: router.IntentNote, Slots: router.Slots{Text: "запасной ключ лежит в синей коробке"},
})
if c.calls != 0 {
t.Fatalf("note acknowledgement called the resident model %d time(s), want none", c.calls)
}
if got != "сохранила заметку." {
t.Fatalf("note acknowledgement = %q, want the fixed feminine line", got)
}
}
func TestLLMReplierFallsBackToStubOnError(t *testing.T) {
r := newLLMReplier(stubCompleter{err: errReplierTest}, nil)
assertAck(t, r, router.Decision{Intent: router.IntentNote}, phraser.AckNote, "llm error")
assertAck(t, r, router.Decision{Intent: router.IntentReminder}, phraser.AckReminder, "llm error")
}
func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) {
r := newLLMReplier(stubCompleter{out: ""}, nil)
assertAck(t, r, router.Decision{Intent: router.IntentNote}, phraser.AckNote, "empty llm")
assertAck(t, r, router.Decision{Intent: router.IntentReminder}, phraser.AckReminder, "empty llm")
}
// A clarify never reaches the model, and since Vikunja #457 it is answered from
+1
View File
@@ -120,6 +120,7 @@ func (h *reactiveHandler) persistDecision(turnCtx context.Context, rec *decision
Source: string(src),
Winner: rec.Winner,
Intent: wonIntent(rec),
RouteProducer: rec.RouteProducer,
ClaimedBeforeHead: claimedBeforeHead(rec),
EncoderID: h.encoderID,
Outcome: wonAt(rec, decision.StageAction),
+11
View File
@@ -6,6 +6,7 @@ import (
"regexp"
"github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/router"
)
// A question about her — "что ты умеешь", "кто ты" — used to have no answer at
@@ -98,6 +99,16 @@ func selfFloor(utterance string) bool {
// asked — "что ты умеешь" and "кто ты" want different halves of it — and falls
// back to the text itself, which is already readable, if the model is down.
func (h *reactiveHandler) querySelf(ctx context.Context, t *queryTurn) (string, bool) {
// Product help is self knowledge too (Vikunja V-720), but unlike the prose description it
// must be exact: these examples name the grammar Maven actually accepts.
// Answer them before topic scoring so a phrasing such as "как отменить
// задачу" cannot leak to SearXNG as generic third-party instructions.
switch router.LocalHelpTopic(t.dec.Utterance) {
case router.HelpReminderCancel:
return "Скажи, например: «отмени напоминание про молоко». Если совпадений несколько, я попрошу выбрать одно.", true
case router.HelpTaskDrop:
return "Скажи, например: «убери из задач настроить бэкапы». Я уберу задачу из активного списка, не отмечая её выполненной.", true
}
if !h.turnIsAbout(ctx, t, topicSelf, selfFloor) {
return "", false
}
+30
View File
@@ -65,6 +65,36 @@ func TestSelfSourceAnswersFromTheDescription(t *testing.T) {
}
}
// V-720: asking how to operate Maven is never a third-party web-search query.
func TestMavenHowToAnswersLocallyWithoutSearch(t *testing.T) {
for _, testCase := range []struct {
utterance string
want string
}{
{"как отменить напоминание про молоко?", "отмени напоминание"},
{"как отменить задачу настроить бэкапы?", "убери из задач"},
{"можно ли отменить напоминание?", "отмени напоминание"},
{"can I cancel a reminder?", "отмени напоминание"},
{"could I cancel a task?", "убери из задач"},
} {
h, seen := searchHandler(t,
`{"answers":["Инструкция стороннего приложения"],"results":[]}`,
200)
reply := h.actionQuery(context.Background(), router.Decision{
Intent: router.IntentQuery,
Utterance: testCase.utterance,
Source: router.SourceSelf,
SourceAnchored: true,
})
if !strings.Contains(reply, testCase.want) {
t.Errorf("%q reply = %q, want local usage example containing %q", testCase.utterance, reply, testCase.want)
}
if *seen != "" {
t.Errorf("%q leaked to search as %q", testCase.utterance, *seen)
}
}
}
// TestSelfDescriptionHoldsThePersona — it is her own text and she reads it out,
// so the same rules the phrasing eval enforces apply to it. Feminine
// self-reference, informal address, no pet names.
+342 -12
View File
@@ -12,6 +12,7 @@
// what was SENT — every delivery.Sendable the dispatcher emitted
// what ARRIVED — the unified intake journal from #283
// what TOOLS were called — the recorded requests against fake Praxis/Nexis/Hexis
// what is DURABLE — typed notes/tasks/reminders/facts store state
// what did NOT happen — expect_no_send / expect_no_call, first-class
//
// The last one is the point. Maven's hard constraints are mostly negative —
@@ -190,14 +191,81 @@ type step struct {
// --- assertions ---
ExpectReply []string `json:"expect_reply_contains,omitempty"`
ExpectNotReply []string `json:"expect_reply_lacks,omitempty"`
ExpectSent []string `json:"expect_sent_contains,omitempty"`
ExpectNoSend bool `json:"expect_no_send,omitempty"`
ExpectCalled []string `json:"expect_called,omitempty"`
ExpectNotCalled []string `json:"expect_not_called,omitempty"`
ExpectEvents []string `json:"expect_events,omitempty"`
ExpectNoEvents bool `json:"expect_no_events,omitempty"`
ExpectReply []string `json:"expect_reply_contains,omitempty"`
ExpectNotReply []string `json:"expect_reply_lacks,omitempty"`
ExpectSent []string `json:"expect_sent_contains,omitempty"`
ExpectNoSend bool `json:"expect_no_send,omitempty"`
ExpectCalled []string `json:"expect_called,omitempty"`
ExpectNotCalled []string `json:"expect_not_called,omitempty"`
ExpectEvents []string `json:"expect_events,omitempty"`
ExpectNoEvents bool `json:"expect_no_events,omitempty"`
ExpectStore *storeStateExpectation `json:"expect_store,omitempty"`
}
// storeStateExpectation is a typed, exact read of Maven's four user-visible
// durable stores. Reply assertions prove what she said; these prove what the
// turn actually committed. Each selected collection can assert its total row
// count and exact row identity independently, so a duplicate insert cannot be
// hidden by finding one matching row.
type storeStateExpectation struct {
Notes *noteStateExpectation `json:"notes,omitempty"`
Tasks *taskStateExpectation `json:"tasks,omitempty"`
Reminders *reminderStateExpectation `json:"reminders,omitempty"`
Facts *factStateExpectation `json:"facts,omitempty"`
}
type noteStateExpectation struct {
Count *int `json:"count,omitempty"`
Rows []noteRowExpectation `json:"rows,omitempty"`
}
type noteRowExpectation struct {
ID int64 `json:"id,omitempty"`
At string `json:"at,omitempty"`
Text string `json:"text,omitempty"`
Source string `json:"source,omitempty"`
}
type taskStateExpectation struct {
Count *int `json:"count,omitempty"`
Rows []taskRowExpectation `json:"rows,omitempty"`
}
type taskRowExpectation struct {
ID int64 `json:"id,omitempty"`
CreatedAt string `json:"created_at,omitempty"`
Text string `json:"text,omitempty"`
Source string `json:"source,omitempty"`
Status string `json:"status,omitempty"`
ResolvedAt string `json:"resolved_at,omitempty"`
ResolvedBy string `json:"resolved_by,omitempty"`
}
type reminderStateExpectation struct {
Count *int `json:"count,omitempty"`
Rows []reminderRowExpectation `json:"rows,omitempty"`
}
type reminderRowExpectation struct {
ID int64 `json:"id,omitempty"`
FireAt string `json:"fire_at,omitempty"`
Text string `json:"text,omitempty"`
Status string `json:"status,omitempty"`
}
type factStateExpectation struct {
Count *int `json:"count,omitempty"`
Rows []factRowExpectation `json:"rows,omitempty"`
}
type factRowExpectation struct {
ID int64 `json:"id,omitempty"`
At string `json:"at,omitempty"`
Kind string `json:"kind,omitempty"`
Key string `json:"key,omitempty"`
Value string `json:"value,omitempty"`
Source string `json:"source,omitempty"`
Confidence *float64 `json:"confidence,omitempty"`
}
type signalStep struct {
@@ -440,6 +508,17 @@ func newSimWorld(t *testing.T, sc scenario) *simWorld {
})
tl := newTickLoop(st, gatherer, dispatcher, phraser.NewStub(), rules,
time.Minute, 5*time.Minute, 0, nil, nil, nil, nil)
// Production upgrades the voice handler from the direct store adapter to
// daemonAPI after the tick loop exists. Mirror that seam so a simulated
// day-plan query reads the real store-backed plan instead of the direct
// adapter's "not available" refusal. The fake clock is the one deliberate
// difference from production's time.Now.
api = &daemonAPI{
CoreAPI: api,
getDayPlan: func(ctx context.Context) ipc.DayPlan {
return tl.dayPlan(ctx, clock.Now())
},
}
scripted := &scriptedLLM{entries: sc.Script}
@@ -492,6 +571,7 @@ func newSimWorld(t *testing.T, sc scenario) *simWorld {
// act panicked the moment the matcher was consulted.
matcher := tool.NewMatcher(api)
rtr := buildRouter(emb, matcher, config.DefaultRouterThreshold, router.NewLLMRouter(scripted), nil)
timeParser := router.NewPythonDateParser()
w.handler = &reactiveHandler{
stt: simTranscriber{},
@@ -510,10 +590,11 @@ func newSimWorld(t *testing.T, sc scenario) *simWorld {
replier: newLLMReplier(scripted, nil),
now: clock.Now,
dataStore: st,
timeParser: router.StubDateTimeParser{},
timeParser: timeParser,
dialogueSessions: dialogue.NewSessionStore(time.Hour),
clarifyStore: dialogue.NewClarifyStore(time.Hour),
clarifyMaxAttempts: dialogue.DefaultMaxAttempts,
extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}},
ecosystem: eco,
}
return w
@@ -599,7 +680,7 @@ func (w *simWorld) run(sc scenario) {
eventsBefore := w.publishCount()
w.stimulate(ctx, s)
w.assert(i, s, sendsBefore, callsBefore, eventsBefore)
w.assert(ctx, i, s, sendsBefore, callsBefore, eventsBefore)
}
}
@@ -615,7 +696,7 @@ func (w *simWorld) stimulate(ctx context.Context, s step) {
switch {
case s.Say != "":
reply := w.handler.runTurn(ctx, s.Say, sourceText)
reply := w.handler.runTurn(ctx, router.NormalizedInput{Text: s.Say, Source: sourceText})
w.replies = append(w.replies, reply)
w.logf("он: %s", s.Say)
w.logf("она: %s", reply)
@@ -800,7 +881,7 @@ func (w *simWorld) callPaths() []string { return w.callPathsSince(nil) }
// Assertions
// ---------------------------------------------------------------------------
func (w *simWorld) assert(i int, s step, sendsBefore int, callsBefore []int, eventsBefore int) {
func (w *simWorld) assert(ctx context.Context, i int, s step, sendsBefore int, callsBefore []int, eventsBefore int) {
w.t.Helper()
where := fmt.Sprintf("step %d (%s)", i+1, s.At)
if s.Note != "" {
@@ -864,6 +945,197 @@ func (w *simWorld) assert(i int, s step, sendsBefore int, callsBefore []int, eve
fail("expected nothing to arrive, %d event(s) were published",
w.publishCount()-eventsBefore)
}
if s.ExpectStore != nil {
w.assertStoreState(ctx, *s.ExpectStore, fail)
}
}
const simStateReadLimit = 10_000
func (w *simWorld) assertStoreState(ctx context.Context, want storeStateExpectation, fail func(string, ...any)) {
if want.Notes != nil {
rows, err := w.store.RecentNotes(ctx, simStateReadLimit)
if err != nil {
fail("read notes for store assertion: %v", err)
} else {
assertStateCount("notes", want.Notes.Count, len(rows), fail)
used := make([]bool, len(rows))
for _, expected := range want.Notes.Rows {
matched, matchErr := matchDistinct(rows, used, func(row store.Note) (bool, error) {
return w.noteStateMatches(row, expected)
})
if matchErr != nil {
fail("invalid note expectation %+v: %v", expected, matchErr)
} else if !matched {
fail("no distinct note matches %+v; notes: %+v", expected, rows)
}
}
}
}
if want.Tasks != nil {
rows, err := w.store.ListTasks(ctx, "")
if err != nil {
fail("read tasks for store assertion: %v", err)
} else {
if want.Tasks.Count != nil && *want.Tasks.Count > store.MaxTaskRows {
fail("task count assertion %d exceeds the store read bound %d", *want.Tasks.Count, store.MaxTaskRows)
} else {
assertStateCount("tasks", want.Tasks.Count, len(rows), fail)
}
used := make([]bool, len(rows))
for _, expected := range want.Tasks.Rows {
matched, matchErr := matchDistinct(rows, used, func(row store.Task) (bool, error) {
return w.taskStateMatches(row, expected)
})
if matchErr != nil {
fail("invalid task expectation %+v: %v", expected, matchErr)
} else if !matched {
fail("no distinct task matches %+v; tasks: %+v", expected, rows)
}
}
}
}
if want.Reminders != nil {
rows, err := w.store.ListReminders(ctx, simStateReadLimit)
if err != nil {
fail("read reminders for store assertion: %v", err)
} else {
assertStateCount("reminders", want.Reminders.Count, len(rows), fail)
used := make([]bool, len(rows))
for _, expected := range want.Reminders.Rows {
matched, matchErr := matchDistinct(rows, used, func(row store.Reminder) (bool, error) {
return w.reminderStateMatches(row, expected)
})
if matchErr != nil {
fail("invalid reminder expectation %+v: %v", expected, matchErr)
} else if !matched {
fail("no distinct reminder matches %+v; reminders: %+v", expected, rows)
}
}
}
}
if want.Facts != nil {
rows, err := w.store.RecentFacts(ctx, simStateReadLimit)
if err != nil {
fail("read facts for store assertion: %v", err)
} else {
assertStateCount("facts", want.Facts.Count, len(rows), fail)
used := make([]bool, len(rows))
for _, expected := range want.Facts.Rows {
matched, matchErr := matchDistinct(rows, used, func(row store.Fact) (bool, error) {
return w.factStateMatches(row, expected)
})
if matchErr != nil {
fail("invalid fact expectation %+v: %v", expected, matchErr)
} else if !matched {
fail("no distinct fact matches %+v; facts: %+v", expected, rows)
}
}
}
}
}
func assertStateCount(kind string, want *int, got int, fail func(string, ...any)) {
if want != nil && got != *want {
fail("%s count = %d, want %d", kind, got, *want)
}
}
// matchDistinct prevents two expectations from being satisfied by the same
// durable row. This is important for identity assertions where two records may
// intentionally carry the same text but have different lifecycle states.
func matchDistinct[T any](rows []T, used []bool, matches func(T) (bool, error)) (bool, error) {
for i, row := range rows {
if used[i] {
continue
}
ok, err := matches(row)
if err != nil {
return false, err
}
if ok {
used[i] = true
return true, nil
}
}
return false, nil
}
func (w *simWorld) noteStateMatches(got store.Note, want noteRowExpectation) (bool, error) {
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text != want.Text ||
want.Source != "" && got.Source != want.Source {
return false, nil
}
return w.stateTimeMatches(got.Ts, want.At)
}
func (w *simWorld) taskStateMatches(got store.Task, want taskRowExpectation) (bool, error) {
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text != want.Text ||
want.Source != "" && got.Source != want.Source || want.Status != "" && got.Status != want.Status ||
want.ResolvedBy != "" && got.ResolvedBy != want.ResolvedBy {
return false, nil
}
if ok, err := w.stateTimeMatches(got.CreatedTs, want.CreatedAt); err != nil || !ok {
return ok, err
}
if want.ResolvedAt == "" {
return true, nil
}
if got.ResolvedTs == nil {
return false, nil
}
return w.stateTimeMatches(*got.ResolvedTs, want.ResolvedAt)
}
func (w *simWorld) reminderStateMatches(got store.Reminder, want reminderRowExpectation) (bool, error) {
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text() != want.Text ||
want.Status != "" && got.Status != want.Status {
return false, nil
}
return w.stateTimeMatches(got.FireTs, want.FireAt)
}
func (w *simWorld) factStateMatches(got store.Fact, want factRowExpectation) (bool, error) {
if want.ID != 0 && got.ID != want.ID || want.Kind != "" && string(got.Kind) != want.Kind ||
want.Key != "" && got.Key != want.Key || want.Value != "" && got.Value != want.Value ||
want.Source != "" && got.Source != want.Source ||
want.Confidence != nil && got.Confidence != *want.Confidence {
return false, nil
}
return w.stateTimeMatches(got.Ts, want.At)
}
func (w *simWorld) stateTimeMatches(got time.Time, raw string) (bool, error) {
if raw == "" {
return true, nil
}
want, err := w.stateTime(raw)
if err != nil {
return false, err
}
return got.Equal(want), nil
}
// stateTime accepts either an absolute RFC3339 instant or the same local
// HH:MM[:SS] shape scenario steps use. The latter keeps fixtures readable
// while still comparing exact instants after the store normalises to UTC.
func (w *simWorld) stateTime(raw string) (time.Time, error) {
if strings.Contains(raw, "T") {
return time.Parse(time.RFC3339, raw)
}
layout := "15:04"
if strings.Count(raw, ":") == 2 {
layout = "15:04:05"
}
hm, err := time.Parse(layout, raw)
if err != nil {
return time.Time{}, fmt.Errorf("expected HH:MM[:SS] or RFC3339, got %q: %w", raw, err)
}
return time.Date(w.start.Year(), w.start.Month(), w.start.Day(),
hm.Hour(), hm.Minute(), hm.Second(), 0, w.loc), nil
}
func sendableTexts(sends []delivery.Sendable) []string {
@@ -933,6 +1205,64 @@ func TestSimulatorScenarios(t *testing.T) {
}
}
// TestSimulatorWorldMirrorsProductionConversationSeams pins the two daemon
// constructor upgrades the continuous scenario needs. A bare store API cannot
// answer DayPlan, and a nil handler extractor cannot complete a parked reminder
// from the next turn; either drift would make the simulator exercise a smaller
// system than production while still producing plausible replies.
func TestSimulatorWorldMirrorsProductionConversationSeams(t *testing.T) {
sc := scenario{SchemaVersion: 1, Name: "constructor-seams", Start: "2026-08-15T08:00:00+04:00"}
w := newSimWorld(t, sc)
if w.handler.api != w.api {
t.Fatal("handler did not receive the simulator's upgraded daemon API")
}
plan, err := w.handler.api.DayPlan(context.Background())
if err != nil {
t.Fatalf("simulator day-plan seam is unavailable: %v", err)
}
planY, planM, planD := plan.Date.In(w.loc).Date()
wantY, wantM, wantD := w.start.Date()
if plan.Date.IsZero() || planY != wantY || planM != wantM || planD != wantD {
t.Fatalf("day plan date = %v, want the fake-clock day %v", plan.Date, w.start)
}
if w.handler.extractor.Time == nil || w.handler.timeParser == nil {
t.Fatal("simulator left the clarify time parser unwired")
}
slots := w.handler.extractor.Extract(context.Background(), router.IntentReminder,
"сегодня в 10:00", w.clock.Now())
if !slots.HasTime || !slots.Time.Equal(w.timeOf("10:00")) {
t.Fatalf("clarify extractor parsed %+v, want the fake-clock day at 10:00", slots)
}
}
// The reported duplicate transcript was a diagnostic artefact: two adjacent
// sed ranges both included boundary line 620. Source had one log call. Keep an
// executable exact-count assertion so a real duplicate cannot be introduced
// later and mistaken for another display artefact.
func TestSimulatorTranscriptRecordsEachSpokenTurnOnce(t *testing.T) {
sc := scenario{
SchemaVersion: 1,
Name: "transcript-count",
Start: "2026-08-15T08:00:00+04:00",
Script: []scriptEntry{{
Match: "привет", Route: `[{"intent":"chat","text":"привет"}]`,
Reply: `{"response":"Привет.","mood":"happy"}`,
}},
}
w := newSimWorld(t, sc)
w.stimulate(context.Background(), step{Say: "привет"})
want := "08:00:00 он: привет"
count := 0
for _, line := range w.transcript {
if line == want {
count++
}
}
if count != 1 {
t.Fatalf("owner transcript line occurred %d times, want exactly once: %v", count, w.transcript)
}
}
func loadScenario(t *testing.T, path string) scenario {
t.Helper()
raw, err := os.ReadFile(path)
+16 -9
View File
@@ -31,6 +31,12 @@ func TestSlotsParity(t *testing.T) {
t.Errorf("router.Slots.%s (%s) missing from dialogue.Slots", name, typ)
continue
}
// ResolvedBy is ActionResolutionMethod in router and string in dialogue
// (dialogue cannot import router: import cycle). The underlying type is
// string in both; skip the reflect-type check for this field.
if name == "ResolvedBy" {
continue
}
if dt != typ {
t.Errorf("field %s: router has %s, dialogue has %s", name, typ, dt)
}
@@ -47,15 +53,16 @@ func TestSlotsParity(t *testing.T) {
// populated value and compare.
func TestSlotsRoundTrip(t *testing.T) {
full := router.Slots{
Time: time.Date(2026, 8, 2, 11, 0, 0, 0, time.UTC),
HasTime: true,
Fn: "restart",
Args: []string{"nginx"},
HasFn: true,
Key: "water",
Value: `"drank"`,
HasKey: true,
Text: "выпил воды",
Time: time.Date(2026, 8, 2, 11, 0, 0, 0, time.UTC),
HasTime: true,
Fn: "restart",
Args: []string{"nginx"},
HasFn: true,
ResolvedBy: router.ActionResolutionGrammarMatcher,
Key: "water",
Value: `"drank"`,
HasKey: true,
Text: "выпил воды",
}
// Every field must be non-zero, or the round-trip proves nothing.
rv := reflect.ValueOf(full)
+219
View File
@@ -0,0 +1,219 @@
{
"schema_version": 1,
"name": "assistant_workday",
"description": "One continuous, deterministic workday through Maven's real conversation pipeline. It proves note capture and grounded high-overlap recall; a reminder that remains uncommitted while Maven clarifies its day, then survives a reported-action no-op, is cancelled exactly once, and stays cancelled on a repeated command; task capture, listing, completion, and a second live task; and the same calendar fact read through both agenda and composed day-plan sources. Store assertions are primary: every mutation and no-op pins exact row count, identity, lifecycle state, provenance, and fake-clock time.",
"start": "2026-08-15T08:00:00+04:00",
"script": [
{
"match": "запомни: запасной ключ лежит",
"route": "[{\"intent\":\"note\",\"text\":\"запомни: запасной ключ лежит в синей коробке\"}]"
},
{
"match": "запасной ключ лежит в синей коробке",
"route": "[{\"intent\":\"query\",\"text\":\"запасной ключ лежит в синей коробке?\",\"source\":\"recall\"}]"
},
{
"match": "я отменил напоминание",
"route": "[{\"intent\":\"chat\",\"text\":\"я отменил напоминание про молоко\"}]",
"reply": "{\"response\":\"Поняла.\",\"mood\":\"neutral\"}"
},
{
"match": "",
"route": "[{\"intent\":\"chat\",\"text\":\"\"}]",
"reply": "{\"response\":\"Поняла.\",\"mood\":\"neutral\"}"
}
],
"steps": [
{
"at": "08:00",
"note": "Capture only the dictated body as one durable note: the command frame is not memory, and no task, reminder, or fact is created.",
"say": "запомни: запасной ключ лежит в синей коробке",
"expect_no_send": true,
"expect_store": {
"notes": { "count": 1, "rows": [{ "id": 1, "at": "08:00", "text": "запасной ключ лежит в синей коробке", "source": "tap:voice" }] },
"tasks": { "count": 0 },
"reminders": { "count": 0 },
"facts": { "count": 0 }
}
},
{
"at": "08:01",
"note": "Recall reads the stored note and does not create a second row.",
"say": "запасной ключ лежит в синей коробке?",
"expect_reply_contains": ["синей коробке"],
"expect_no_send": true,
"expect_store": {
"notes": { "count": 1, "rows": [{ "id": 1, "text": "запасной ключ лежит в синей коробке", "source": "tap:voice" }] }
}
},
{
"at": "08:02",
"note": "A clock without a day is not a committed reminder. Maven asks, and the reminder table remains empty.",
"say": "напомни купить молоко в 10:00",
"expect_reply_contains": ["В какой день"],
"expect_no_send": true,
"expect_store": {
"reminders": { "count": 0 }
}
},
{
"at": "08:03",
"note": "The clarification completes the parked request against the fake clock and creates exactly one pending reminder.",
"say": "сегодня",
"expect_reply_contains": ["10:00"],
"expect_no_send": true,
"expect_store": {
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "pending" }] }
}
},
{
"at": "08:04",
"note": "A first-person report is not another cancellation command and cannot mutate the pending row.",
"say": "я отменил напоминание про молоко",
"expect_reply_contains": ["Поняла"],
"expect_no_send": true,
"expect_store": {
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "pending" }] }
}
},
{
"at": "08:05",
"note": "The addressed imperative cancels that exact durable reminder in place.",
"say": "отмени напоминание про молоко",
"expect_reply_contains": ["отменила напоминание", "купить молоко"],
"expect_no_send": true,
"expect_store": {
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "cancelled" }] }
}
},
{
"at": "08:06",
"note": "Repeating the cancellation is an explicit no-op: no replacement row and no resurrection.",
"say": "отмени напоминание про молоко",
"expect_reply_contains": ["ожидающих напоминаний нет"],
"expect_no_send": true,
"expect_store": {
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "cancelled" }] }
}
},
{
"at": "08:07",
"note": "An explicit task marker creates one open task, not a note.",
"say": "добавь в задачи настроить бэкапы",
"expect_reply_contains": ["настроить бэкапы"],
"expect_no_send": true,
"expect_store": {
"notes": { "count": 1 },
"tasks": { "count": 1, "rows": [{ "id": 1, "created_at": "08:07", "text": "настроить бэкапы", "source": "tap:voice", "status": "open" }] }
}
},
{
"at": "08:08",
"note": "Listing is read-only and returns the live task without duplicating it.",
"say": "какие у меня задачи?",
"expect_reply_contains": ["настроить бэкапы"],
"expect_no_send": true,
"expect_store": {
"tasks": { "count": 1, "rows": [{ "id": 1, "text": "настроить бэкапы", "status": "open" }] }
}
},
{
"at": "08:09",
"note": "Naming the task moves the same row forward to done and records who resolved it.",
"say": "закрой задачу настроить бэкапы",
"expect_reply_contains": ["настроить бэкапы"],
"expect_no_send": true,
"expect_store": {
"tasks": { "count": 1, "rows": [{ "id": 1, "created_at": "08:07", "text": "настроить бэкапы", "source": "tap:voice", "status": "done", "resolved_at": "08:09", "resolved_by": "tap:voice" }] }
}
},
{
"at": "08:10",
"note": "A second task remains live for the rest of the workday while the completed row remains durable history.",
"say": "добавь в задачи отправить отчёт",
"expect_reply_contains": ["отправить отчёт"],
"expect_no_send": true,
"expect_store": {
"tasks": { "count": 2, "rows": [
{ "id": 1, "text": "настроить бэкапы", "status": "done", "resolved_by": "tap:voice" },
{ "id": 2, "created_at": "08:10", "text": "отправить отчёт", "source": "tap:voice", "status": "open" }
] }
}
},
{
"at": "08:11",
"note": "A fully specified reminder commits directly and coexists with the cancelled history row.",
"say": "напомни сегодня в 12:00 размяться",
"expect_reply_contains": ["12:00"],
"expect_no_send": true,
"expect_store": {
"reminders": { "count": 2, "rows": [
{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "cancelled" },
{ "id": 2, "fire_at": "12:00", "text": "размяться", "status": "pending" }
] }
}
},
{
"at": "08:12",
"note": "A calendar poll contributes one exact env fact at the event instant.",
"arrive": {
"source": "poll:caldav",
"as_of": "11:00",
"fact": {
"key": "calendar_event_20260815_Планёрка",
"value": "Планёрка @ 11:00-11:30",
"kind": "env"
}
},
"expect_events": ["calendar_event_20260815_Планёрка"],
"expect_no_send": true,
"expect_store": {
"facts": { "count": 1, "rows": [{ "id": 1, "at": "11:00", "kind": "env", "key": "calendar_event_20260815_Планёрка", "value": "Планёрка @ 11:00-11:30", "source": "poll:caldav", "confidence": 1.0 }] }
}
},
{
"at": "08:13",
"note": "The agenda source reads the calendar fact without changing any durable state.",
"say": "что у меня сегодня?",
"expect_reply_contains": ["Планёрка"],
"expect_no_send": true,
"expect_store": {
"facts": { "count": 1, "rows": [{ "id": 1, "key": "calendar_event_20260815_Планёрка", "source": "poll:caldav" }] },
"tasks": { "count": 2 },
"reminders": { "count": 2 }
}
},
{
"at": "08:14",
"note": "The daemon day-plan seam composes the same calendar fact with the still-pending reminder; cancelled reminders stay out.",
"say": "какие планы на сегодня?",
"expect_reply_contains": ["Планёрка", "размяться"],
"expect_reply_lacks": ["купить молоко"],
"expect_no_send": true,
"expect_store": {
"notes": { "count": 1, "rows": [{ "id": 1, "text": "запасной ключ лежит в синей коробке" }] },
"tasks": { "count": 2, "rows": [
{ "id": 1, "text": "настроить бэкапы", "status": "done" },
{ "id": 2, "text": "отправить отчёт", "status": "open" }
] },
"reminders": { "count": 2, "rows": [
{ "id": 1, "text": "купить молоко", "status": "cancelled" },
{ "id": 2, "text": "размяться", "status": "pending" }
] },
"facts": { "count": 1, "rows": [{ "id": 1, "key": "calendar_event_20260815_Планёрка", "value": "Планёрка @ 11:00-11:30" }] }
}
},
{
"at": "08:15",
"note": "A normal tick after the session remains silent; durable assistant state does not authorize an unsolicited message.",
"tick": true,
"expect_no_send": true,
"expect_store": {
"notes": { "count": 1 },
"tasks": { "count": 2 },
"reminders": { "count": 2 },
"facts": { "count": 1 }
}
}
]
}
+2 -1
View File
@@ -213,7 +213,8 @@ func isPleasantry(text string) bool {
// It is also the whole answer when there is no route to read — the classifier
// is the failure floor and a turn must never break on the model.
func offlineOwnRequest(text string) bool {
return router.IsQuestionShaped(text) || router.CarriesCaptureVerb(text) || carriesReminderVerb(text)
_, cancelsReminder := parseReminderCancelRequest(text)
return router.IsQuestionShaped(text) || router.CarriesCaptureVerb(text) || carriesReminderVerb(text) || cancelsReminder
}
// classifyTurnRole decides what this utterance is against the pending action.
+2 -2
View File
@@ -264,10 +264,10 @@ func TestClarifyCancelEndsTheExchange(t *testing.T) {
// the same memo.
func TestTheTurnIsRoutedOnce(t *testing.T) {
h, _ := newRoutingClarifyHandler(t)
rt := h.newTurnRoute("какая сейчас погода в Риме?", h.now())
rt := h.newTurnRoute(router.NormalizedInput{Text: "какая сейчас погода в Риме?", Source: sourceText}, h.now())
ctx := withTurnRoute(withDialogueID(context.Background(), voiceDialogueID), rt)
first, ok := h.routeForRole(ctx, rt.text)
first, ok := h.routeForRole(ctx, rt.input.Text)
if !ok {
t.Fatal("the cascade must produce a decision to classify against")
}
+11 -8
View File
@@ -18,9 +18,9 @@ import (
// second on the resident model and — worse — could disagree with itself, which
// is exactly the class of bug this task is about.
type turnRoute struct {
h *reactiveHandler
text string
now time.Time
h *reactiveHandler
input router.NormalizedInput
now time.Time
once sync.Once
dec router.Decision
@@ -46,8 +46,8 @@ type turnRoute struct {
type turnRouteKey struct{}
func (h *reactiveHandler) newTurnRoute(text string, now time.Time) *turnRoute {
return &turnRoute{h: h, text: text, now: now}
func (h *reactiveHandler) newTurnRoute(input router.NormalizedInput, now time.Time) *turnRoute {
return &turnRoute{h: h, input: input, now: now}
}
func withTurnRoute(ctx context.Context, rt *turnRoute) context.Context {
@@ -70,7 +70,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
if r.h.dialogueSessions != nil {
r.prev = r.h.dialogueSessions.Get(dialogueIDOf(ctx), r.now)
}
if dec, cont := continuationDecision(r.prev, r.text, r.now); cont {
if dec, cont := continuationDecision(r.prev, r.input.Text, r.now); cont {
log.Printf("voice: continuation of %s from the previous turn", dec.Intent)
r.dec, r.cont = dec, true
return
@@ -79,7 +79,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
r.err = router.ErrNoIntents
return
}
r.dec, r.err = r.h.router.Route(ctx, r.text, r.now)
r.dec, r.err = r.h.router.Route(ctx, r.input, r.now)
})
return r.dec, r.cont, r.prev, r.err
}
@@ -92,7 +92,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
func (h *reactiveHandler) routeForRole(ctx context.Context, text string) (router.Decision, bool) {
rt := turnRouteFrom(ctx)
if rt == nil {
rt = h.newTurnRoute(text, h.now())
rt = h.newTurnRoute(router.NormalizedInput{Text: text, Source: sourceText}, h.now())
}
dec, _, _, err := rt.resolve(ctx)
if err != nil {
@@ -121,5 +121,8 @@ func needsRoute(text string) bool {
if isCancel(text) {
return false
}
if _, ok := parseReminderCancelRequest(text); ok {
return false
}
return len(ownContent(text)) > 0 || router.IsQuestionShaped(text)
}
+40 -18
View File
@@ -211,7 +211,7 @@ func (h *reactiveHandler) HandlePushToTalk(ctx context.Context, req voice.PushTo
// 2-5. the shared turn pipeline (confirm → clarify → route → dialogue →
// action → replier), identical to the text path.
replyText := h.runTurn(ctx, text, sourceVoice)
replyText := h.runTurn(ctx, router.NormalizedInput{Text: text, MatchText: router.NormalizeMatchText(text), Source: sourceVoice})
// 6. tts — synthesise the reply text; return to the voice server which
// ships it back on the conn.
@@ -244,30 +244,30 @@ func (h *reactiveHandler) upgradeAPI(api ipc.CoreAPI) {
// HandlePushToTalk so text channels share the same routing logic.
func (h *reactiveHandler) handleText(ctx context.Context, conversation, text string) string {
log.Printf("voice: handleText: %q", text)
return h.runTurn(withDialogueID(ctx, dialogueIDFor(sourceText, conversation)), text, sourceText)
return h.runTurn(withDialogueID(ctx, dialogueIDFor(sourceText, conversation)), router.NormalizedInput{Text: text, MatchText: router.NormalizeMatchText(text), Source: sourceText})
}
// turnSource — which channel this utterance arrived on, in the same provenance
// vocabulary facts use (internal/event). It is threaded through runTurn because
// a turn can write a fact, and a fact that lies about where it came from is
// worse than no fact: provenance is the first column read when asking why a
// daemon-wide setting is the way it is.
type turnSource string
// turnSource is a local alias for router.InputSource, kept so the daemon code
// reads sourceVoice/sourceText without a package prefix at every call site.
// The canonical type lives in the router package; this is pure convenience.
type turnSource = router.InputSource
const (
sourceVoice turnSource = "tap:voice" // HandlePushToTalk, a real microphone
sourceText turnSource = "tap:text" // handleText: mavweb /api/chat, telegram
sourceVoice = router.InputSourceVoice
sourceText = router.InputSourceText
)
// runTurn — the reactive turn pipeline shared by the voice and text entry
// points: expired-clarify notice → confirm answer → explicit correction →
// clarify answer → quiet toggle → route → dialogue merge → clarify question →
// action → replier.
// Takes the already-transcribed utterance, returns the reply text; the voice
// path wraps it in stt/tts, the text path returns it as-is.
// clarify answer → quiet toggle → reminder cancellation → route → dialogue
// merge → clarify question → action → replier.
// Takes the NormalizedInput (typed ingress boundary), returns the reply text;
// the voice path wraps it in stt/tts, the text path returns it as-is.
//
// The ordering is load-bearing — see the step comments.
func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSource) (reply string) {
func (h *reactiveHandler) runTurn(ctx context.Context, input router.NormalizedInput) (reply string) {
text := input.Text
src := input.Source
// 0. the decision record (V-564). Installed here rather than in the IPC
// entry point, so the mic, telegram and the web all leave the same trail —
// a record only the web produced would be missing exactly the turns that
@@ -275,7 +275,7 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
// on a human-rate path, and no claim site can change a route with it.
if h.decisions != nil {
var rec *decision.Record
ctx, rec = decision.With(ctx, text)
ctx, rec = decision.With(ctx, text, string(src))
decision.Expect(ctx, decision.StagePreRoute, preRouteLadder)
defer func() {
done := rec.Finish(h.now())
@@ -289,7 +289,7 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
// claiming it, and step 5 acts on the same decision — routing twice would
// cost a second on the resident model and could disagree with itself.
now := h.now()
rt := h.newTurnRoute(text, now)
rt := h.newTurnRoute(input, now)
ctx = withTurnRoute(ctx, rt)
// A resolver may suspend an older clarify flow even when it handles this
// turn itself. Finalise that state at one choke point so early returns from
@@ -335,6 +335,16 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
return withNotice(expiredNotice, reply)
}
// 3c. explicit command prohibition — negative authority must be settled
// before a parked slot or candidate can consume these words. In particular,
// "не отменяй напоминание" is not the subject/time answer to an older
// reminder request. Confirmation stays above it: "don't" is already a
// closed no-answer to a destructive confirm, and that narrower stateful
// contract must retain first refusal.
if reply, handled := h.resolveCommandProhibition(ctx, text); notePreRoute(ctx, "command-prohibition", handled) {
return withNotice(expiredNotice, reply)
}
// 4. clarify answer — if she asked a live question last turn, this
// utterance is its answer, not a fresh command. After the confirm check: a
// y/n gate is armed by her own prompt and is the narrower claim on the
@@ -372,7 +382,15 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
return withNotice(expiredNotice, reply)
}
// 5d. ordinal selection — "второй", "первую сделал" pick from the list she
// 5d. committed-reminder cancellation — an explicit cancel verb plus the
// reminder noun resolves against pending rows. It runs before ordinal so a
// clock such as "на девять" cannot be mistaken for a position in an older
// task list; an ambiguous result binds its own list for the next turn.
if reply, handled := h.resolveReminderCancellation(ctx, text); notePreRoute(ctx, "reminder-cancel", handled) {
return withNotice(expiredNotice, reply)
}
// 5e. ordinal selection — "второй", "первую сделал" pick from the list she
// just read (ordinal.go). Before routing, and only when a list is actually
// bound to the session: with nothing offered, "второй" is an ordinary word
// and keeps routing.
@@ -396,6 +414,10 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
return withNotice(expiredNotice, "не получилось разобрать команду.")
}
log.Printf("voice: route result: intent=%s slots=%+v", dec.Intent, dec.Slots)
// Carry the route producer into the decision record for observability.
if rec := decision.From(ctx); rec != nil && dec.Producer != "" {
rec.RouteProducer = string(dec.Producer)
}
// 7. dialogue — fill this turn's missing slots from a prior same-intent
// turn (follow-ups like «напомни завтра» → «…позвонить маме»), then remember
+3 -1
View File
@@ -143,7 +143,9 @@ func main() {
log.Printf("mavweb: ambient notification ingest enabled at POST /api/ambient")
}
// The read surfaces. Every one of them 503s without -core.
// The data surfaces. Every one of them 503s without -core. /reminders also
// accepts an ID-bound cancellation POST. It is deliberately not step-up
// gated: like dismissing a proposed routine, it can only make Maven quieter.
mux.HandleFunc("/dash", corePage(handleDash))
mux.HandleFunc("/history", corePage(handleHistory))
mux.HandleFunc("/trace", corePage(handleTrace))
+111 -15
View File
@@ -3,8 +3,10 @@ package main
import (
_ "embed"
"encoding/json"
"errors"
"fmt"
"net/http"
"strconv"
"strings"
"github.com/kami/maven/internal/ipc"
@@ -23,11 +25,14 @@ var remindersTmpl = parsePage("reminders", remindersHTML, nil)
// (Vikunja #469). Neither is a formatting nicety: the envelope is an internal
// shape he never chose, and a time on a page he reads is the time on his wall.
type reminderRow struct {
Created string
Fires string
Status string
Detail string
Text string
ID int64
Created string
Fires string
Schedule string
Status string
Detail string
Text string
CanCancel bool
}
// reminderText unwraps the {"text":...} payload the router writes.
@@ -51,8 +56,16 @@ func reminderText(payload string) string {
func reminderRows(rs []ipc.Reminder) []reminderRow {
out := make([]reminderRow, 0, len(rs))
for _, r := range rs {
fire := r.NextFireTs
if fire.IsZero() {
fire = r.FireTs
}
status := r.Status
detail := ""
schedule := ""
if r.Cron != "" {
schedule = "recurring · " + r.Cron
}
if !r.DeliveryBlockedTs.IsZero() {
status = "blocked"
detail = r.DeliveryBlockedError
@@ -60,25 +73,108 @@ func reminderRows(rs []ipc.Reminder) []reminderRow {
detail = "retry " + r.NextAttemptTs.Local().Format("02 Jan 15:04")
}
out = append(out, reminderRow{
Created: r.CreatedTs.Local().Format("02 Jan 15:04"),
Fires: r.FireTs.Local().Format("02 Jan 15:04"),
Status: status,
Detail: detail,
Text: reminderText(r.Payload),
ID: r.ID,
Created: r.CreatedTs.Local().Format("02 Jan 15:04"),
Fires: fire.Local().Format("02 Jan 15:04"),
Schedule: schedule,
Status: status,
Detail: detail,
Text: reminderText(r.Payload),
CanCancel: r.Status == "pending",
})
}
return out
}
// remindersForPage keeps every pending row reachable while retaining the
// recent terminal history the page already showed. Pending rows come first in
// firing order (the CoreAPI contract); IDs present in the recent window are not
// duplicated below them.
func remindersForPage(pending, recent []ipc.Reminder) []ipc.Reminder {
out := make([]ipc.Reminder, 0, len(pending)+len(recent))
seen := make(map[int64]bool, len(pending))
for _, reminder := range pending {
out = append(out, reminder)
seen[reminder.ID] = true
}
for _, reminder := range recent {
if seen[reminder.ID] {
continue
}
out = append(out, reminder)
}
return out
}
func handleReminders(w http.ResponseWriter, r *http.Request, core ipc.CoreAPI) {
if !requireCore(w, r, core, "reminders") {
return
}
reminders, err := core.ListReminders(r.Context(), 50)
if err != nil {
writeProblem(w, r, http.StatusBadGateway, problemCoreReadFailed,
"reminders unavailable", fmt.Errorf("list reminders: %w", err))
msg := ""
if r.Method == http.MethodGet && r.URL.Query().Get("cancelled") == "1" {
msg = "reminder cancelled"
}
switch r.Method {
case http.MethodGet:
case http.MethodPost:
if strings.TrimSpace(r.FormValue("action")) != "cancel" {
writeProblem(w, r, http.StatusBadRequest, problemInvalidRequest,
"unknown reminder action", nil)
return
}
id, err := strconv.ParseInt(strings.TrimSpace(r.FormValue("id")), 10, 64)
if err != nil || id <= 0 {
writeProblem(w, r, http.StatusBadRequest, problemInvalidRequest,
"invalid reminder id", err)
return
}
if err := core.CancelReminder(r.Context(), id); err != nil {
switch {
case errors.Is(err, ipc.ErrReminderNotFound):
writeProblem(w, r, http.StatusNotFound, problemResourceNotFound,
"reminder not found", err)
case errors.Is(err, ipc.ErrReminderInFlight):
writeProblem(w, r, http.StatusConflict, problemCoreChangeFailed,
"reminder delivery has already started", err)
case errors.Is(err, ipc.ErrReminderState):
writeProblem(w, r, http.StatusConflict, problemCoreChangeFailed,
"reminder is no longer pending", err)
default:
writeProblem(w, r, http.StatusBadGateway, problemCoreChangeFailed,
"reminder cancellation failed", fmt.Errorf("cancel reminder %d: %w", id, err))
}
return
}
http.Redirect(w, r, "/reminders?cancelled=1", http.StatusSeeOther)
return
default:
writeProblem(w, r, http.StatusMethodNotAllowed, problemMethodNotAllowed,
"method not allowed", nil)
return
}
renderPage(w, remindersTmpl, map[string]any{"Reminders": reminderRows(reminders)})
pending, err := core.ListPendingReminders(r.Context(), 0)
if err != nil {
public := "reminders unavailable"
if msg != "" {
public = "reminder cancelled; refreshed list unavailable"
}
writeProblem(w, r, http.StatusBadGateway, problemCoreReadFailed,
public, fmt.Errorf("list pending reminders: %w", err))
return
}
recent, err := core.ListReminders(r.Context(), 50)
if err != nil {
public := "reminders unavailable"
if msg != "" {
public = "reminder cancelled; refreshed list unavailable"
}
writeProblem(w, r, http.StatusBadGateway, problemCoreReadFailed,
public, fmt.Errorf("list reminders: %w", err))
return
}
reminders := remindersForPage(pending, recent)
renderPage(w, remindersTmpl, struct {
Msg string
Reminders []reminderRow
}{msg, reminderRows(reminders)})
}
+7 -2
View File
@@ -1,12 +1,17 @@
{{template "shellTop" "reminders"}}
<h1>Reminders</h1>
{{if .Msg}}<div class="msg msg-ok">{{.Msg}}</div>{{end}}
{{if .Reminders}}<div class=scroll><table>
<tr><th>created</th><th>fires</th><th>status</th><th>what</th></tr>
<tr><th>created</th><th>fires</th><th>status</th><th>what</th><th>action</th></tr>
{{range .Reminders}}<tr>
<td class=hint>{{.Created}}</td>
<td>{{.Fires}}</td>
<td>{{.Fires}}{{if .Schedule}}<div class=hint>{{.Schedule}}</div>{{end}}</td>
<td><span class="badge {{.Status}}">{{.Status}}</span>{{if .Detail}}<div class=hint>{{.Detail}}</div>{{end}}</td>
<td class=text-max>{{.Text}}</td>
<td>{{if .CanCancel}}<form method="post" action="/reminders">
<input type="hidden" name="id" value="{{.ID}}">
<button type="submit" name="action" value="cancel">cancel</button>
</form>{{end}}</td>
</tr>{{end}}</table></div>
{{else}}<div class=empty>
<svg class=icon width="24" height="24"><use href="/ethos-icons.svg#i-calendar"/></svg>
+202
View File
@@ -1,6 +1,11 @@
package main
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
@@ -12,6 +17,7 @@ import (
func TestReminderRowsUnwrapAndLocalise(t *testing.T) {
fire := time.Date(2026, 8, 4, 18, 30, 0, 0, time.UTC)
rows := reminderRows([]ipc.Reminder{{
ID: 17,
CreatedTs: fire.Add(-time.Hour),
FireTs: fire,
Status: "pending",
@@ -29,6 +35,9 @@ func TestReminderRowsUnwrapAndLocalise(t *testing.T) {
if strings.Contains(rows[0].Text, "{") {
t.Errorf("Text still carries JSON: %q", rows[0].Text)
}
if rows[0].ID != 17 || !rows[0].CanCancel {
t.Errorf("pending reminder action binding = %+v, want id 17 cancellable", rows[0])
}
}
func TestReminderRowsExposeBlockedDelivery(t *testing.T) {
@@ -42,6 +51,33 @@ func TestReminderRowsExposeBlockedDelivery(t *testing.T) {
if len(rows) != 1 || rows[0].Status != "blocked" || rows[0].Detail != "ntfy credentials rejected" {
t.Fatalf("blocked reminder is not visible: %+v", rows)
}
if !rows[0].CanCancel {
t.Fatal("a blocked but still-pending reminder must remain cancellable")
}
}
func TestReminderRowsShowTheCurrentRecurringOccurrence(t *testing.T) {
original := time.Date(2026, 8, 1, 9, 0, 0, 0, time.UTC)
next := time.Date(2026, 8, 16, 9, 0, 0, 0, time.UTC)
rows := reminderRows([]ipc.Reminder{{
ID: 42, FireTs: original, NextFireTs: next, Cron: "0 9 * * *",
Status: "pending", Payload: `{"text":"принять лекарство"}`,
}})
if len(rows) != 1 || rows[0].Fires != next.Local().Format("02 Jan 15:04") {
t.Fatalf("recurring fire = %+v, want current occurrence %s", rows, next)
}
if rows[0].Schedule != "recurring · 0 9 * * *" || !rows[0].CanCancel {
t.Fatalf("recurring identity/action = %+v", rows[0])
}
}
func TestReminderRowsOnlyPendingCanCancel(t *testing.T) {
rows := reminderRows([]ipc.Reminder{{Status: "cancelled"}, {Status: "fired"}})
for _, row := range rows {
if row.CanCancel {
t.Errorf("terminal row %+v exposed a cancel action", row)
}
}
}
// A payload that is not the envelope is his own words, so it is shown as it is.
@@ -57,3 +93,169 @@ func TestReminderTextKeepsPlainPayload(t *testing.T) {
}
}
}
type reminderCore struct {
ipc.UnimplementedCoreAPI
reminders []ipc.Reminder
pending []ipc.Reminder
listErr error
pendingErr error
cancelErr error
cancelID int64
}
func (c *reminderCore) ListReminders(context.Context, int) ([]ipc.Reminder, error) {
return c.reminders, c.listErr
}
func (c *reminderCore) ListPendingReminders(context.Context, int) ([]ipc.Reminder, error) {
return c.pending, c.pendingErr
}
func (c *reminderCore) CancelReminder(_ context.Context, id int64) error {
c.cancelID = id
if c.cancelErr != nil {
return c.cancelErr
}
for i := range c.reminders {
if c.reminders[i].ID == id {
c.reminders[i].Status = "cancelled"
}
}
return nil
}
func reminderPost(action, id string) *http.Request {
form := url.Values{"action": {action}, "id": {id}}
req := httptest.NewRequest(http.MethodPost, "/reminders", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
return req
}
func TestHandleRemindersCancel(t *testing.T) {
core := &reminderCore{reminders: []ipc.Reminder{{
ID: 23, Status: "pending", Payload: `{"text":"позвонить врачу"}`,
}}}
rr := httptest.NewRecorder()
handleReminders(rr, reminderPost("cancel", "23"), core)
if rr.Code != http.StatusSeeOther || rr.Header().Get("Location") != "/reminders?cancelled=1" {
t.Fatalf("status/location = %d %q, want 303 PRG", rr.Code, rr.Header().Get("Location"))
}
if core.cancelID != 23 {
t.Fatalf("cancel id = %d, want 23", core.cancelID)
}
rr = httptest.NewRecorder()
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders?cancelled=1", nil), core)
if rr.Code != http.StatusOK || !strings.Contains(rr.Body.String(), "reminder cancelled") || strings.Contains(rr.Body.String(), "value=\"23\"") {
t.Fatalf("redirect outcome rendered incorrectly: status=%d body=%s", rr.Code, rr.Body.String())
}
}
func TestHandleRemindersSuccessfulMutationDoesNotDependOnRefresh(t *testing.T) {
core := &reminderCore{listErr: errors.New("offline"), pendingErr: errors.New("offline")}
rr := httptest.NewRecorder()
handleReminders(rr, reminderPost("cancel", "23"), core)
if rr.Code != http.StatusSeeOther || core.cancelID != 23 {
t.Fatalf("successful cancellation became refresh failure: status=%d id=%d body=%s", rr.Code, core.cancelID, rr.Body.String())
}
}
func TestHandleRemindersIncludesPendingRowsOutsideRecentWindow(t *testing.T) {
old := ipc.Reminder{ID: 1, Status: "pending", Payload: `{"text":"old but pending"}`}
recent := make([]ipc.Reminder, 50)
for i := range recent {
recent[i] = ipc.Reminder{ID: int64(i + 2), Status: "fired", Payload: `{"text":"history"}`}
}
core := &reminderCore{pending: []ipc.Reminder{old}, reminders: recent}
rr := httptest.NewRecorder()
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders", nil), core)
if rr.Code != http.StatusOK || !strings.Contains(rr.Body.String(), "old but pending") ||
!strings.Contains(rr.Body.String(), `value="1"`) {
t.Fatalf("old pending reminder is not reachable: status=%d body=%s", rr.Code, rr.Body.String())
}
}
func TestHandleRemindersRejectsMalformedPosts(t *testing.T) {
for _, tc := range []struct {
name string
action string
id string
}{
{"unknown action", "delete", "23"},
{"missing id", "cancel", ""},
{"non-numeric id", "cancel", "twenty-three"},
{"non-positive id", "cancel", "0"},
} {
t.Run(tc.name, func(t *testing.T) {
core := &reminderCore{}
rr := httptest.NewRecorder()
handleReminders(rr, reminderPost(tc.action, tc.id), core)
if rr.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400; body=%s", rr.Code, rr.Body.String())
}
if core.cancelID != 0 {
t.Fatalf("CancelReminder called with %d for malformed post", core.cancelID)
}
})
}
}
func TestHandleRemindersCancelErrors(t *testing.T) {
for _, tc := range []struct {
name string
err error
status int
public string
}{
{"missing", ipc.ErrReminderNotFound, http.StatusNotFound, "reminder not found"},
{"delivery started", ipc.ErrReminderInFlight, http.StatusConflict, "reminder delivery has already started"},
{"terminal", ipc.ErrReminderState, http.StatusConflict, "reminder is no longer pending"},
{"transport", errors.New("socket closed"), http.StatusBadGateway, "reminder cancellation failed"},
} {
t.Run(tc.name, func(t *testing.T) {
core := &reminderCore{cancelErr: tc.err}
rr := httptest.NewRecorder()
handleReminders(rr, reminderPost("cancel", "23"), core)
if rr.Code != tc.status || !strings.Contains(rr.Body.String(), tc.public) {
t.Fatalf("status/body = %d %q, want %d containing %q", rr.Code, rr.Body.String(), tc.status, tc.public)
}
if got := rr.Header().Get("Content-Type"); !strings.HasPrefix(got, "application/problem+json") {
t.Fatalf("content type = %q, want problem JSON", got)
}
})
}
}
func TestHandleRemindersMethodAndListErrors(t *testing.T) {
t.Run("method", func(t *testing.T) {
rr := httptest.NewRecorder()
handleReminders(rr, httptest.NewRequest(http.MethodDelete, "/reminders", nil), &reminderCore{})
if rr.Code != http.StatusMethodNotAllowed {
t.Fatalf("status = %d, want 405", rr.Code)
}
})
t.Run("pending list", func(t *testing.T) {
rr := httptest.NewRecorder()
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders", nil), &reminderCore{pendingErr: errors.New("offline")})
if rr.Code != http.StatusBadGateway || !strings.Contains(rr.Body.String(), "reminders unavailable") {
t.Fatalf("status/body = %d %q, want 502 problem", rr.Code, rr.Body.String())
}
})
t.Run("recent list", func(t *testing.T) {
rr := httptest.NewRecorder()
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders", nil), &reminderCore{listErr: errors.New("offline")})
if rr.Code != http.StatusBadGateway || !strings.Contains(rr.Body.String(), "reminders unavailable") {
t.Fatalf("status/body = %d %q, want 502 problem", rr.Code, rr.Body.String())
}
})
t.Run("successful outcome remains explicit when redirected refresh fails", func(t *testing.T) {
rr := httptest.NewRecorder()
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders?cancelled=1", nil), &reminderCore{listErr: errors.New("offline")})
if rr.Code != http.StatusBadGateway || !strings.Contains(rr.Body.String(), "reminder cancelled; refreshed list unavailable") {
t.Fatalf("status/body = %d %q, want truthful refresh problem", rr.Code, rr.Body.String())
}
})
}
+178
View File
@@ -0,0 +1,178 @@
// merge-corpus merges the frozen holdout from corpus_v1.json with the
// expanded v2 development pool, writing the result back to corpus_v1.json.
//
// Usage:
//
// go run ./cmd/merge-corpus/ -v1 internal/router/semantic/corpus_v1.json \
// -v2 /tmp/corpus_v2.json -out internal/router/semantic/corpus_v1.json
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"flag"
"fmt"
"log"
"os"
"sort"
"strings"
"github.com/kami/maven/internal/router/semantic"
)
func main() {
v1Path := flag.String("v1", "internal/router/semantic/corpus_v1.json", "v1 corpus path")
v2Path := flag.String("v2", "/tmp/corpus_v2.json", "v2 factory output path")
outPath := flag.String("out", "internal/router/semantic/corpus_v1.json", "output path")
flag.Parse()
// 1. Load v1
v1Data, err := os.ReadFile(*v1Path)
if err != nil {
log.Fatalf("read v1: %v", err)
}
var v1Env semantic.CorpusEnvelope
if err := json.Unmarshal(v1Data, &v1Env); err != nil {
log.Fatalf("parse v1: %v", err)
}
fmt.Fprintf(os.Stderr, "v1: %d examples\n", len(v1Env.Examples))
// 2. Identify frozen holdout from v1
frozen, _, _ := semantic.FrozenHoldoutSplit(v1Env.Examples)
frozenTexts := make(map[string]bool)
for _, e := range frozen {
frozenTexts[strings.TrimSpace(e.Text)] = true
}
fmt.Fprintf(os.Stderr, "v1 frozen holdout: %d examples\n", len(frozen))
// 3. Load v2
v2Data, err := os.ReadFile(*v2Path)
if err != nil {
log.Fatalf("read v2: %v", err)
}
var v2Env semantic.CorpusEnvelope
if err := json.Unmarshal(v2Data, &v2Env); err != nil {
log.Fatalf("parse v2: %v", err)
}
fmt.Fprintf(os.Stderr, "v2: %d examples\n", len(v2Env.Examples))
// 4. Merge: frozen from v1 + all from v2
// Dedup by normalized text
seen := make(map[string]bool)
var merged []semantic.RouteExample
// Frozen holdout first
for _, e := range frozen {
norm := strings.TrimSpace(e.Text)
if seen[norm] {
fmt.Fprintf(os.Stderr, "SKIP v1 frozen dup: %q\n", norm)
continue
}
seen[norm] = true
merged = append(merged, e)
}
// V2 examples
skipped := 0
for _, e := range v2Env.Examples {
norm := strings.TrimSpace(e.Text)
if seen[norm] {
skipped++
continue
}
// Check if this text conflicts with a frozen holdout entry
if frozenTexts[norm] {
// Text exists in frozen — skip v2 version to preserve frozen
skipped++
continue
}
seen[norm] = true
merged = append(merged, e)
}
fmt.Fprintf(os.Stderr, "merged: %d examples (skipped %d duplicates)\n", len(merged), skipped)
// 5. Fast-path metadata must match the real router. V2 rows are derived
// by interactively-labeled construction; a stale manually-supplied value
// is a build error, never silently rewritten. Frozen holdout rows are
// preserved verbatim — report drift, do not touch them.
v2Checked, frozenChecked, frozenDrift := 0, 0, 0
for _, e := range merged {
derived := semantic.DeriveFastPath(e.Text).Matched
if frozenTexts[strings.TrimSpace(e.Text)] {
frozenChecked++
if derived != e.FastPathResolved {
frozenDrift++
fmt.Fprintf(os.Stderr, "frozen drift: %q stored=%v derived=%v\n", e.Text, e.FastPathResolved, derived)
}
continue
}
v2Checked++
if derived != e.FastPathResolved {
log.Fatalf("v2 row (source=%s source_id=%s) fast_path_resolved=%v but router derives %v: %q",
e.Source, e.SourceID, e.FastPathResolved, derived, e.Text)
}
}
fmt.Fprintf(os.Stderr, "fast-path check: v2 rows %d OK, frozen rows %d (drift %d, reported only)\n",
v2Checked, frozenChecked, frozenDrift)
// 6. Validate
if err := semantic.ValidateCorpus(merged); err != nil {
log.Fatalf("validation failed: %v", err)
}
fmt.Fprintf(os.Stderr, "validation: OK\n")
// 7. Compute dataset hash
texts := make([]string, len(merged))
for i, e := range merged {
texts[i] = e.Text
}
sort.Strings(texts)
h := sha256.Sum256([]byte(strings.Join(texts, "\n")))
datasetHash := hex.EncodeToString(h[:16])
// 8. Stats
routeCounts := make(map[semantic.SemanticRoute]int)
fpCount, resCount := 0, 0
for _, e := range merged {
routeCounts[e.Route]++
if e.FastPathResolved {
fpCount++
} else {
resCount++
}
}
fmt.Fprintf(os.Stderr, "\nRoute distribution:\n")
for _, r := range semantic.AllRoutes {
fmt.Fprintf(os.Stderr, " %-15s %d\n", r, routeCounts[r])
}
fmt.Fprintf(os.Stderr, "fast-path: %d residual: %d\n", fpCount, resCount)
// 9. Write merged corpus
outEnv := semantic.CorpusEnvelope{
SchemaVersion: 1,
Name: "semantic_coarse_route_v1",
Notes: []string{
"Coarse semantic route corpus for the learned-router experiment.",
"Frozen holdout preserved from v1. Development pool expanded by corpus-factory v2.",
"Every row carries provenance (source + source_id) and a split_group.",
"Labels come from explicit mapping rules, not model output.",
},
Reproducibility: &semantic.ReproducibilityMeta{
SourceFixtureHash: "v1-frozen + corpus-factory-v2",
ContrastGeneratorVersion: "v2-direct-generation",
SplitAlgorithm: "grouped-cv-v1",
DatasetHash: datasetHash,
},
Examples: merged,
}
data, err := json.MarshalIndent(outEnv, "", " ")
if err != nil {
log.Fatalf("marshal: %v", err)
}
if err := os.WriteFile(*outPath, data, 0644); err != nil {
log.Fatalf("write %s: %v", *outPath, err)
}
fmt.Fprintf(os.Stderr, "\nOutput: %s (%d bytes)\n", *outPath, len(data))
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,856 @@
#!/usr/bin/env python3
"""
Semantic Router Linear Head Experiment
======================================
Evaluates whether the six-way residual routing boundary is linearly learnable
from Maven's existing e5-small representation.
Architecture under test:
NormalizedInput.MatchText
existing multilingual-e5-small embedder
tiny 6-class linear head (softmax logistic regression)
SemanticRouteDecision
No new encoder. No fine-tuning. Linear separability floor only.
"""
import json
import hashlib
import sys
import warnings
from collections import defaultdict
from pathlib import Path
import numpy as np
from sklearn.exceptions import ConvergenceWarning
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
accuracy_score,
f1_score,
precision_recall_fscore_support,
confusion_matrix,
brier_score_loss,
log_loss,
)
from sklearn.preprocessing import LabelEncoder
warnings.filterwarnings("ignore", category=ConvergenceWarning)
# ─── Constants ──────────────────────────────────────────────────────────────
ROUTES = ["action", "conversation", "knowledge", "memory_write", "system", "uncertain"]
ROUTE_IDX = {r: i for i, r in enumerate(ROUTES)}
# Regularization grid
C_VALUES = [0.01, 0.1, 1.0, 10.0, 100.0]
# Abstention thresholds
THRESHOLDS = [0.40, 0.50, 0.60, 0.70, 0.80, 0.90]
# Contrast families
CONTRAST_FAMILIES = [
"negation", "question", "reported_speech", "quotation",
"hypothetical", "capability_question",
]
# ─── Data Loading ───────────────────────────────────────────────────────────
def load_embeddings(path):
with open(path) as f:
data = json.load(f)
meta = data["meta"]
examples = data["examples"]
return meta, examples
def filter_dev_pool(examples):
"""Return only development pool examples (not frozen holdout)."""
return [e for e in examples if e["dev_pool"]]
def filter_residual(examples):
"""Return only router-residual examples (fast_path_resolved == false)."""
return [e for e in examples if not e["fast_path_resolved"]]
def extract_Xy(examples):
"""Extract feature matrix and label vector from cached examples."""
X = np.array([e["embedding"] for e in examples])
y = np.array([e["route"] for e in examples])
return X, y
def get_fold_groups(examples):
"""Return fold assignment array matching the Go-generated CV folds."""
return np.array([e["cv_fold"] for e in examples])
# ─── Grouped CV ─────────────────────────────────────────────────────────────
def grouped_cv_experiment(X, y, fold_ids, C_values, examples_meta):
"""
Run grouped cross-validation with logistic regression.
Returns best C, per-fold results, and out-of-fold predictions.
"""
unique_folds = sorted(set(fold_ids))
n_classes = len(ROUTES)
results_by_C = {}
for C in C_values:
fold_metrics = []
oof_rows = []
for test_fold in unique_folds:
train_mask = fold_ids != test_fold
test_mask = fold_ids == test_fold
X_train, y_train = X[train_mask], y[train_mask]
X_test, y_test = X[test_mask], y[test_mask]
model = LogisticRegression(
C=C, max_iter=2000, solver="lbfgs",
random_state=42,
)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)
classes = model.classes_
acc = accuracy_score(y_test, y_pred)
macro_f1 = f1_score(y_test, y_pred, average="macro", zero_division=0)
prec, rec, f1, sup = precision_recall_fscore_support(
y_test, y_pred, labels=ROUTES, zero_division=0
)
false_action = 0
for true, pred in zip(y_test, y_pred):
if true != "action" and pred == "action":
false_action += 1
fold_metrics.append({
"fold": test_fold,
"train_size": len(X_train),
"test_size": len(X_test),
"accuracy": acc,
"macro_f1": macro_f1,
"per_class": {
route: {"precision": prec[i], "recall": rec[i], "f1": f1[i]}
for i, route in enumerate(ROUTES)
},
"false_action": false_action,
})
for i, (true, pred) in enumerate(zip(y_test, y_pred)):
test_indices = np.where(test_mask)[0]
meta = examples_meta[test_indices[i]]
proba_dict = {cls: float(y_proba[i][j]) for j, cls in enumerate(classes)}
max_proba = max(proba_dict.values()) if proba_dict else 0.0
oof_rows.append({
"source_id": meta["source_id"],
"fold": test_fold,
"true": true,
"predicted": pred,
"correct": true == pred,
"max_proba": max_proba,
"proba": proba_dict,
"fast_path_resolved": meta.get("fast_path_resolved", False),
"tags": meta.get("tags", []),
"text": meta["text"],
})
# Aggregate across folds
mean_acc = np.mean([m["accuracy"] for m in fold_metrics])
mean_f1 = np.mean([m["macro_f1"] for m in fold_metrics])
std_acc = np.std([m["accuracy"] for m in fold_metrics])
std_f1 = np.std([m["macro_f1"] for m in fold_metrics])
total_fa = sum(m["false_action"] for m in fold_metrics)
results_by_C[C] = {
"mean_accuracy": mean_acc,
"std_accuracy": std_acc,
"mean_macro_f1": mean_f1,
"std_macro_f1": std_f1,
"total_false_action": total_fa,
"fold_metrics": fold_metrics,
"oof_predictions": oof_rows,
}
# Select best C by mean macro F1
best_C = max(results_by_C, key=lambda c: results_by_C[c]["mean_macro_f1"])
return best_C, results_by_C
# ─── Metrics Computation ────────────────────────────────────────────────────
def compute_full_metrics(y_true, y_pred, y_proba=None):
"""Compute all required metrics from out-of-fold predictions."""
acc = accuracy_score(y_true, y_pred)
macro_f1 = f1_score(y_true, y_pred, average="macro", zero_division=0)
# Per-class P/R/F1
prec, rec, f1, sup = precision_recall_fscore_support(
y_true, y_pred, labels=ROUTES, zero_division=0
)
# Confusion matrix
cm = confusion_matrix(y_true, y_pred, labels=ROUTES)
# False action
false_action = 0
false_action_predicted = []
for true, pred in zip(y_true, y_pred):
if true != "action" and pred == "action":
false_action += 1
# Action precision/recall
action_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p == "action")
action_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
action_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p != "action")
action_precision = action_tp / max(action_tp + action_fp, 1)
action_recall = action_tp / max(action_tp + action_fn, 1)
# Uncertain precision/recall
unc_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "uncertain" and p == "uncertain")
unc_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "uncertain" and p == "uncertain")
unc_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "uncertain" and p != "uncertain")
unc_precision = unc_tp / max(unc_tp + unc_fp, 1)
unc_recall = unc_tp / max(unc_tp + unc_fn, 1)
metrics = {
"accuracy": acc,
"macro_f1": macro_f1,
"false_action_count": false_action,
"false_action_rate": false_action / max(len(y_true), 1),
"action_precision": action_precision,
"action_recall": action_recall,
"uncertain_precision": unc_precision,
"uncertain_recall": unc_recall,
"per_class": {},
"confusion_matrix": cm.tolist(),
}
for i, route in enumerate(ROUTES):
metrics["per_class"][route] = {
"precision": float(prec[i]),
"recall": float(rec[i]),
"f1": float(f1[i]),
"support": int(sup[i]),
}
return metrics
def compute_calibration(y_true, y_pred, oof_rows):
"""Compute ECE, Brier score, and per-threshold abstention curves."""
# ECE (Expected Calibration Error) with 10 bins
n_bins = 10
bin_boundaries = np.linspace(0, 1, n_bins + 1)
ece = 0.0
total = len(y_true)
confidences = np.array([r["max_proba"] for r in oof_rows])
correct = np.array([r["correct"] for r in oof_rows])
for i in range(n_bins):
lo, hi = bin_boundaries[i], bin_boundaries[i + 1]
mask = (confidences > lo) & (confidences <= hi)
if mask.sum() == 0:
continue
bin_acc = correct[mask].mean()
bin_conf = confidences[mask].mean()
ece += mask.sum() / total * abs(bin_acc - bin_conf)
# Brier score (multiclass one-hot encoding)
y_true_idx = np.array([ROUTES.index(t) for t in y_true])
n_samples = len(y_true)
n_classes = len(ROUTES)
y_true_oh = np.zeros((n_samples, n_classes))
y_true_oh[np.arange(n_samples), y_true_idx] = 1.0
# Build probability matrix
y_proba_matrix = np.zeros((n_samples, n_classes))
for i, r in enumerate(oof_rows):
for j, route in enumerate(ROUTES):
y_proba_matrix[i, j] = r["proba"].get(route, 0.0)
brier = np.mean(np.sum((y_proba_matrix - y_true_oh) ** 2, axis=1))
ll = log_loss(y_true, y_proba_matrix, labels=ROUTES)
# Abstention curves
abstention_curves = []
for thr in THRESHOLDS:
accepted_mask = confidences >= thr
n_accepted = accepted_mask.sum()
coverage = n_accepted / max(total, 1)
if n_accepted > 0:
acc_accepted = accuracy_score(y_true[accepted_mask], y_pred[accepted_mask])
f1_accepted = f1_score(
y_true[accepted_mask], y_pred[accepted_mask],
average="macro", zero_division=0,
)
fa_count = sum(
1 for t, p in zip(y_true[accepted_mask], y_pred[accepted_mask])
if t != "action" and p == "action"
)
else:
acc_accepted = 0.0
f1_accepted = 0.0
fa_count = 0
abstention_curves.append({
"threshold": thr,
"n_accepted": int(n_accepted),
"coverage": coverage,
"accuracy": acc_accepted,
"macro_f1": f1_accepted,
"false_action_count": fa_count,
})
return {
"ece": float(ece),
"brier": float(brier),
"log_loss": float(ll),
"abstention_curves": abstention_curves,
}
def compute_action_threshold(oof_rows):
"""Evaluate action-specific safety gate thresholds."""
action_curves = []
for thr in THRESHOLDS:
action_pred = []
for r in oof_rows:
p = r["predicted"]
proba = r["proba"].get("action", 0.0)
if p == "action" and proba < thr:
# Demote action prediction
# Find next best route that isn't action
sorted_routes = sorted(r["proba"].items(), key=lambda x: -x[1])
for route, _ in sorted_routes:
if route != "action":
p = route
break
action_pred.append(p)
y_true = np.array([r["true"] for r in oof_rows])
y_pred = np.array(action_pred)
action_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p == "action")
action_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
action_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p != "action")
false_action = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
action_curves.append({
"threshold": thr,
"action_precision": action_tp / max(action_tp + action_fp, 1),
"action_recall": action_tp / max(action_tp + action_fn, 1),
"false_action_count": false_action,
})
return action_curves
def compute_disagreement(y_true, y_pred, legacy_pred, oof_rows):
"""Analyze disagreements between legacy and learned router."""
results = {
"legacy_wrong_learned_right": [],
"legacy_right_learned_wrong": [],
"both_wrong_differently": [],
}
legacy_false_actions = []
learned_false_actions = []
shared_false_actions = []
for i, r in enumerate(oof_rows):
sid = r["source_id"]
text = r["text"]
true = y_true[i]
learned = y_pred[i]
legacy = legacy_pred[i]
if legacy != true and learned == true:
results["legacy_wrong_learned_right"].append({
"source_id": sid, "text": text,
"true": true, "legacy": legacy, "learned": learned,
})
elif legacy == true and learned != true:
results["legacy_right_learned_wrong"].append({
"source_id": sid, "text": text,
"true": true, "legacy": legacy, "learned": learned,
})
elif legacy != true and learned != true and legacy != learned:
results["both_wrong_differently"].append({
"source_id": sid, "text": text,
"true": true, "legacy": legacy, "learned": learned,
})
# False action tracking
if true != "action" and legacy == "action":
legacy_false_actions.append(sid)
if true != "action" and learned == "action":
learned_false_actions.append(sid)
if true != "action" and legacy == "action" and learned == "action":
shared_false_actions.append(sid)
# Repaired false actions
repaired = [sid for sid in legacy_false_actions if sid not in learned_false_actions]
new_errors = [sid for sid in learned_false_actions if sid not in legacy_false_actions]
shared = shared_false_actions
return {
"details": results,
"legacy_false_actions": legacy_false_actions,
"learned_false_actions": learned_false_actions,
"repaired": repaired,
"new_errors": new_errors,
"shared": shared,
}
def compute_contrast_family(oof_rows):
"""Analyze performance per contrast family."""
family_results = {}
for family in CONTRAST_FAMILIES:
members = [r for r in oof_rows if family in r.get("tags", [])]
if not members:
continue
y_true = [r["true"] for r in members]
y_pred = [r["predicted"] for r in members]
correct = sum(1 for t, p in zip(y_true, y_pred) if t == p)
false_act = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
family_results[family] = {
"count": len(members),
"correct": correct,
"accuracy": correct / len(members),
"false_action": false_act,
}
return family_results
def compute_legacy_baseline(examples):
"""
Compute legacy baseline by mapping each example through the known
fast-path and classifier behavior. Since we don't have the actual
router running, we use the corpus metadata:
- fast_path_resolved examples are correct (grammar handles them)
- We simulate the legacy baseline from the test output numbers
"""
# The actual legacy baseline was measured in TestLegacyBaseline:
# overall accuracy: 52.2%, residual: 40.8%, false-action rate: 19.9%
# We need per-example predictions. We'll approximate from the corpus structure.
#
# For fast_path_resolved examples, the legacy router is correct (stage-0 grammar).
# For residual examples, we need to simulate the hash-embedder classifier.
# Since we don't have the hash embedder running, we use the known aggregate.
#
# This is a limitation: the legacy baseline numbers come from the Go test,
# and we only have the aggregate. We'll use the aggregate for comparison.
pass
# ─── Report Generation ──────────────────────────────────────────────────────
def fmt_pct(v, decimals=1):
return f"{100 * v:.{decimals}f}%"
def fmt_float(v, decimals=3):
return f"{v:.{decimals}f}"
def generate_report(meta, all_results, residual_results, oof_all, oof_residual):
"""Generate the full experiment report."""
lines = []
lines.append("# Semantic Router Linear Head Experiment — Report")
lines.append("")
lines.append("## 1. Exact e5 representation used")
lines.append("")
lines.append(f"- **Model**: {meta['embedder_id']}")
lines.append(f"- **Checkpoint**: {meta['model_path']}")
lines.append(f"- **Tokenizer**: {meta['tokenizer_path']}")
lines.append(f"- **Dimension**: {meta['dimension']}")
lines.append(f"- **Pooling**: {meta['pooling']}")
lines.append(f"- **Normalization**: {meta['normalization']}")
lines.append(f"- **Input template**: {meta['input_template']}")
lines.append("")
lines.append("## 2. Development/residual row counts")
lines.append("")
lines.append(f"- Total corpus: {meta['total_examples']}")
lines.append(f"- Frozen holdout: {meta['frozen_count']}")
lines.append(f"- Development pool: {meta['dev_count']}")
lines.append(f"- Fast-path resolved: {meta['fast_path_count']}")
lines.append(f"- Router-residual: {meta['residual_count']}")
lines.append("")
lines.append("Route distribution (full corpus):")
for route, count in sorted(meta["route_counts"].items()):
lines.append(f" - {route}: {count}")
lines.append("")
lines.append("## 3. Grouped fold composition")
lines.append("")
lines.append(f"Folds: {meta['cv_folds']}")
for fold_id, stats in sorted(meta["fold_composition"].items()):
lines.append(f" - Fold {fold_id}: eval={stats['eval_count']} train={stats['train_count']} routes={stats['eval_routes']}")
lines.append("")
# Regularization selection
lines.append("## 4. Selected regularization")
lines.append("")
lines.append("### Experiment A: All development examples")
best_C_all = all_results["best_C"]
lines.append(f"- Best C: {best_C_all}")
lines.append(f"- Mean accuracy: {fmt_pct(all_results['results_by_C'][best_C_all]['mean_accuracy'])} ± {fmt_pct(all_results['results_by_C'][best_C_all]['std_accuracy'])}")
lines.append(f"- Mean macro F1: {fmt_float(all_results['results_by_C'][best_C_all]['mean_macro_f1'])} ± {fmt_float(all_results['results_by_C'][best_C_all]['std_macro_f1'])}")
lines.append(f"- Total false actions (CV): {all_results['results_by_C'][best_C_all]['total_false_action']}")
lines.append("")
lines.append("### Experiment B: Router-residual only")
best_C_res = residual_results["best_C"]
lines.append(f"- Best C: {best_C_res}")
lines.append(f"- Mean accuracy: {fmt_pct(residual_results['results_by_C'][best_C_res]['mean_accuracy'])} ± {fmt_pct(residual_results['results_by_C'][best_C_res]['std_accuracy'])}")
lines.append(f"- Mean macro F1: {fmt_float(residual_results['results_by_C'][best_C_res]['mean_macro_f1'])} ± {fmt_float(residual_results['results_by_C'][best_C_res]['std_macro_f1'])}")
lines.append(f"- Total false actions (CV): {residual_results['results_by_C'][best_C_res]['total_false_action']}")
lines.append("")
# Stability across folds
lines.append("### Stability across folds")
lines.append("")
for C_val in C_VALUES:
r = all_results["results_by_C"][C_val]
fold_accs = [m["accuracy"] for m in r["fold_metrics"]]
fold_f1s = [m["macro_f1"] for m in r["fold_metrics"]]
lines.append(f" C={C_val:<6} acc={fmt_pct(r['mean_accuracy'])}±{fmt_pct(r['std_accuracy'])} f1={fmt_float(r['mean_macro_f1'])}±{fmt_float(r['std_macro_f1'])} folds_acc={[fmt_pct(a) for a in fold_accs]}")
lines.append("")
# Experiment A metrics
lines.append("## 5. All-example CV metrics")
lines.append("")
metrics_all = all_results["full_metrics"]
lines.append(f"- Accuracy: {fmt_pct(metrics_all['accuracy'])}")
lines.append(f"- Macro F1: {fmt_float(metrics_all['macro_f1'])}")
lines.append(f"- False-action count: {metrics_all['false_action_count']}")
lines.append(f"- False-action rate: {fmt_pct(metrics_all['false_action_rate'])}")
lines.append(f"- Action precision: {fmt_float(metrics_all['action_precision'])}")
lines.append(f"- Action recall: {fmt_float(metrics_all['action_recall'])}")
lines.append(f"- Uncertain precision: {fmt_float(metrics_all['uncertain_precision'])}")
lines.append(f"- Uncertain recall: {fmt_float(metrics_all['uncertain_recall'])}")
lines.append("")
lines.append("Per-class metrics:")
for route in ROUTES:
pc = metrics_all["per_class"][route]
lines.append(f" {route:<15} P={fmt_float(pc['precision'])} R={fmt_float(pc['recall'])} F1={fmt_float(pc['f1'])} (n={pc['support']})")
lines.append("")
lines.append("Confusion matrix (rows=expected, cols=predicted):")
header = f"{'':>15}" + "".join(f"{r:>15}" for r in ROUTES)
lines.append(header)
for i, route in enumerate(ROUTES):
row = f"{route:>15}" + "".join(f"{metrics_all['confusion_matrix'][i][j]:>15}" for j in range(len(ROUTES)))
lines.append(row)
lines.append("")
# Experiment B metrics
lines.append("## 6. Residual-only CV metrics")
lines.append("")
metrics_res = residual_results["full_metrics"]
lines.append(f"- Accuracy: {fmt_pct(metrics_res['accuracy'])}")
lines.append(f"- Macro F1: {fmt_float(metrics_res['macro_f1'])}")
lines.append(f"- False-action count: {metrics_res['false_action_count']}")
lines.append(f"- False-action rate: {fmt_pct(metrics_res['false_action_rate'])}")
lines.append(f"- Action precision: {fmt_float(metrics_res['action_precision'])}")
lines.append(f"- Action recall: {fmt_float(metrics_res['action_recall'])}")
lines.append(f"- Uncertain precision: {fmt_float(metrics_res['uncertain_precision'])}")
lines.append(f"- Uncertain recall: {fmt_float(metrics_res['uncertain_recall'])}")
lines.append("")
lines.append("Per-class metrics:")
for route in ROUTES:
pc = metrics_res["per_class"][route]
lines.append(f" {route:<15} P={fmt_float(pc['precision'])} R={fmt_float(pc['recall'])} F1={fmt_float(pc['f1'])} (n={pc['support']})")
lines.append("")
lines.append("Confusion matrix (rows=expected, cols=predicted):")
header = f"{'':>15}" + "".join(f"{r:>15}" for r in ROUTES)
lines.append(header)
for i, route in enumerate(ROUTES):
row = f"{route:>15}" + "".join(f"{metrics_res['confusion_matrix'][i][j]:>15}" for j in range(len(ROUTES)))
lines.append(row)
lines.append("")
# Legacy comparison
lines.append("## 7. Legacy-vs-linear comparison")
lines.append("")
lines.append("### All examples")
lines.append(f"{'metric':<25} {'legacy':>10} {'linear e5':>10} {'delta':>10}")
lines.append("-" * 55)
# Legacy baseline from test: 52.2% overall, 40.8% residual, 19.9% false-action
# These are approximate since we don't have per-example legacy predictions
legacy_acc = 0.522
legacy_fa_rate = 0.199
legacy_macro_f1 = 0.0 # unknown precisely
lines.append(f"{'accuracy':<25} {fmt_pct(legacy_acc):>10} {fmt_pct(metrics_all['accuracy']):>10} {fmt_pct(metrics_all['accuracy'] - legacy_acc):>10}")
lines.append(f"{'macro F1':<25} {'':>10} {fmt_float(metrics_all['macro_f1']):>10} {'':>10}")
lines.append(f"{'action precision':<25} {'':>10} {fmt_float(metrics_all['action_precision']):>10} {'':>10}")
lines.append(f"{'false-action rate':<25} {fmt_pct(legacy_fa_rate):>10} {fmt_pct(metrics_all['false_action_rate']):>10} {fmt_pct(metrics_all['false_action_rate'] - legacy_fa_rate):>10}")
lines.append(f"{'uncertain F1':<25} {fmt_float(0.0):>10} {fmt_float(metrics_all['per_class']['uncertain']['f1']):>10} {fmt_float(metrics_all['per_class']['uncertain']['f1']):>10}")
lines.append("")
lines.append("### Router-residual only")
lines.append(f"{'metric':<25} {'legacy':>10} {'linear e5':>10} {'delta':>10}")
lines.append("-" * 55)
legacy_res_acc = 0.408
lines.append(f"{'accuracy':<25} {fmt_pct(legacy_res_acc):>10} {fmt_pct(metrics_res['accuracy']):>10} {fmt_pct(metrics_res['accuracy'] - legacy_res_acc):>10}")
lines.append(f"{'macro F1':<25} {'':>10} {fmt_float(metrics_res['macro_f1']):>10} {'':>10}")
lines.append(f"{'false-action rate':<25} {'':>10} {fmt_pct(metrics_res['false_action_rate']):>10} {'':>10}")
lines.append("")
# Fold variance
lines.append("## 8. Fold variance")
lines.append("")
lines.append("All-example CV:")
for m in all_results["results_by_C"][best_C_all]["fold_metrics"]:
lines.append(f" Fold {m['fold']}: acc={fmt_pct(m['accuracy'])} f1={fmt_float(m['macro_f1'])} false_action={m['false_action']}")
lines.append("")
lines.append("Residual-only CV:")
for m in residual_results["results_by_C"][best_C_res]["fold_metrics"]:
lines.append(f" Fold {m['fold']}: acc={fmt_pct(m['accuracy'])} f1={fmt_float(m['macro_f1'])} false_action={m['false_action']}")
lines.append("")
# Disagreement analysis
lines.append("## 9. False-action repair/new-error analysis")
lines.append("")
# The disagreement analysis requires legacy per-example predictions.
# Since we don't have those, we report what we can from the out-of-fold data.
lines.append("Note: Legacy per-example predictions were not available for this experiment.")
lines.append("The legacy baseline was measured in aggregate in the Go test suite.")
lines.append("")
lines.append("Learned router false-action cases (out-of-fold):")
for r in oof_all:
if r["true"] != "action" and r["predicted"] == "action":
lines.append(f" {r['source_id']}: '{r['text']}' (true={r['true']}, proba(action)={r['proba'].get('action', 0):.3f})")
lines.append("")
# Contrast family
lines.append("## 10. Contrast-family results")
lines.append("")
lines.append("### Experiment A (all dev)")
contrast_all = all_results["contrast_family"]
lines.append(f"{'family':<25} {'count':>6} {'correct':>8} {'accuracy':>10} {'false_act':>10}")
lines.append("-" * 60)
for family in CONTRAST_FAMILIES:
if family in contrast_all:
c = contrast_all[family]
lines.append(f"{family:<25} {c['count']:>6} {c['correct']:>8} {fmt_pct(c['accuracy']):>10} {c['false_action']:>10}")
lines.append("")
lines.append("### Experiment B (residual only)")
contrast_res = residual_results["contrast_family"]
lines.append(f"{'family':<25} {'count':>6} {'correct':>8} {'accuracy':>10} {'false_act':>10}")
lines.append("-" * 60)
for family in CONTRAST_FAMILIES:
if family in contrast_res:
c = contrast_res[family]
lines.append(f"{family:<25} {c['count']:>6} {c['correct']:>8} {fmt_pct(c['accuracy']):>10} {c['false_action']:>10}")
lines.append("")
# Calibration
lines.append("## 11. Calibration metrics")
lines.append("")
lines.append("### Experiment A")
cal_all = all_results["calibration"]
lines.append(f"- ECE: {fmt_float(cal_all['ece'])}")
lines.append(f"- Brier score: {fmt_float(cal_all['brier'])}")
lines.append(f"- Log loss: {fmt_float(cal_all['log_loss'])}")
lines.append("")
lines.append("### Experiment B")
cal_res = residual_results["calibration"]
lines.append(f"- ECE: {fmt_float(cal_res['ece'])}")
lines.append(f"- Brier score: {fmt_float(cal_res['brier'])}")
lines.append(f"- Log loss: {fmt_float(cal_res['log_loss'])}")
lines.append("")
# Abstention curves
lines.append("## 12. Abstention curves")
lines.append("")
lines.append("### Experiment A (all dev)")
lines.append(f"{'threshold':>10} {'n_accepted':>11} {'coverage':>10} {'accuracy':>10} {'macro_f1':>10} {'false_act':>10}")
lines.append("-" * 62)
for curve in cal_all["abstention_curves"]:
lines.append(f"{curve['threshold']:>10.2f} {curve['n_accepted']:>11} {fmt_pct(curve['coverage']):>10} {fmt_pct(curve['accuracy']):>10} {fmt_float(curve['macro_f1']):>10} {curve['false_action_count']:>10}")
lines.append("")
lines.append("### Experiment B (residual only)")
lines.append(f"{'threshold':>10} {'n_accepted':>11} {'coverage':>10} {'accuracy':>10} {'macro_f1':>10} {'false_act':>10}")
lines.append("-" * 62)
for curve in cal_res["abstention_curves"]:
lines.append(f"{curve['threshold']:>10.2f} {curve['n_accepted']:>11} {fmt_pct(curve['coverage']):>10} {fmt_pct(curve['accuracy']):>10} {fmt_float(curve['macro_f1']):>10} {curve['false_action_count']:>10}")
lines.append("")
# Action threshold
lines.append("## 13. Action-threshold curve")
lines.append("")
lines.append("### Experiment A")
lines.append(f"{'threshold':>10} {'action_P':>10} {'action_R':>10} {'false_act':>10}")
lines.append("-" * 40)
for curve in all_results["action_threshold"]:
lines.append(f"{curve['threshold']:>10.2f} {fmt_float(curve['action_precision']):>10} {fmt_float(curve['action_recall']):>10} {curve['false_action_count']:>10}")
lines.append("")
lines.append("### Experiment B")
lines.append(f"{'threshold':>10} {'action_P':>10} {'action_R':>10} {'false_act':>10}")
lines.append("-" * 40)
for curve in residual_results["action_threshold"]:
lines.append(f"{curve['threshold']:>10.2f} {fmt_float(curve['action_precision']):>10} {fmt_float(curve['action_recall']):>10} {curve['false_action_count']:>10}")
lines.append("")
# Model artifact size
lines.append("## 14. Model artifact size and runtime cost")
lines.append("")
# Logistic regression: 6 classes × 384 features + 6 biases = 2310 parameters
n_params = len(ROUTES) * meta["dimension"] + len(ROUTES)
serialized_bytes = n_params * 4 # float32
lines.append(f"- Trainable parameters: {n_params}")
lines.append(f" - {len(ROUTES)} classes × {meta['dimension']} features = {len(ROUTES) * meta['dimension']} weights")
lines.append(f" - {len(ROUTES)} bias terms")
lines.append(f"- Serialized head size: {serialized_bytes} bytes ({serialized_bytes / 1024:.1f} KB)")
lines.append(f"- Additional inference FLOPs: {len(ROUTES) * meta['dimension']} multiply-accumulates")
lines.append(f"- Incremental cost (e5 already computed): ~{len(ROUTES) * meta['dimension']} FLOPs, <1µs")
lines.append(f"- Cost if semantic router must trigger its own e5: full ONNX inference (~{meta['dimension']} × 128 × 12 = ~590K FLOPs)")
lines.append("")
# Recommendation
lines.append("## 16. Recommendation")
lines.append("")
# Decision logic
all_f1 = metrics_all["macro_f1"]
res_f1 = metrics_res["macro_f1"]
res_acc = metrics_res["accuracy"]
if res_f1 > 0.5 and res_acc > 0.55:
verdict = "linear head sufficient"
detail = (f"Residual macro F1 of {fmt_float(res_f1)} and accuracy of {fmt_pct(res_acc)} "
f"exceed the legacy baseline (40.8% residual accuracy) by a meaningful margin. "
f"A linear head over frozen e5-small embeddings is a viable first production candidate.")
elif all_f1 > 0.5 and res_f1 < 0.5:
verdict = "need more data"
detail = (f"All-example F1 ({fmt_float(all_f1)}) is acceptable but residual-only F1 "
f"({fmt_float(res_f1)}) drops, suggesting the contrast-family examples are "
f"hard for a linear classifier. More contrastive training data may help.")
else:
verdict = "representation inadequate"
detail = (f"Neither all-example ({fmt_float(all_f1)}) nor residual-only ({fmt_float(res_f1)}) "
f"F1 reaches the minimum viable threshold. The e5-small linear separability floor "
f"is insufficient for this 6-way task. Consider a non-linear head or a different encoder.")
lines.append(f"**{verdict}**")
lines.append("")
lines.append(detail)
lines.append("")
# Commit hash
lines.append("## 17. Commit hash for experiment tooling")
lines.append("")
import subprocess
try:
result = subprocess.run(
["git", "rev-parse", "HEAD"],
capture_output=True, text=True, cwd="/home/kami/apps/Maven"
)
lines.append(f"`{result.stdout.strip()}`")
except Exception:
lines.append("(unable to determine)")
lines.append("")
return "\n".join(lines)
# ─── Main ───────────────────────────────────────────────────────────────────
def run_experiment(X, y, fold_ids, examples_meta, label):
"""Run the full experiment pipeline for one population."""
print(f"\n{'='*60}")
print(f" Experiment: {label}")
print(f" Samples: {len(y)} Features: {X.shape[1]} Folds: {len(set(fold_ids))}")
print(f"{'='*60}")
# 1. Grouped CV with regularization search
best_C, results_by_C = grouped_cv_experiment(X, y, fold_ids, C_VALUES, examples_meta)
# 2. Use best C to get full metrics from OOF predictions
oof_rows = results_by_C[best_C]["oof_predictions"]
oof_true = np.array([r["true"] for r in oof_rows])
oof_pred = np.array([r["predicted"] for r in oof_rows])
full_metrics = compute_full_metrics(oof_true, oof_pred)
# 3. Calibration
calibration = compute_calibration(oof_true, oof_pred, oof_rows)
# 4. Action threshold
action_threshold = compute_action_threshold(oof_rows)
# 5. Contrast family
contrast_family = compute_contrast_family(oof_rows)
return {
"best_C": best_C,
"results_by_C": results_by_C,
"full_metrics": full_metrics,
"calibration": calibration,
"action_threshold": action_threshold,
"contrast_family": contrast_family,
"oof_rows": oof_rows,
}
if __name__ == "__main__":
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
REPORT_PATH = "/tmp/mvn-experiment/report.md"
# Load data
meta, examples = load_embeddings(EMBEDDING_PATH)
print(f"Loaded {len(examples)} examples, embedder={meta['embedder_id']}, dim={meta['dimension']}")
# Development pool only
dev_examples = filter_dev_pool(examples)
print(f"Development pool: {len(dev_examples)} examples")
# Extract features
X_all, y_all = extract_Xy(dev_examples)
fold_ids_all = get_fold_groups(dev_examples)
# Experiment A: all dev examples
all_results = run_experiment(X_all, y_all, fold_ids_all, dev_examples, "All development examples")
# Experiment B: residual only
dev_residual = filter_residual(dev_examples)
X_res, y_res = extract_Xy(dev_residual)
fold_ids_res = get_fold_groups(dev_residual)
residual_results = run_experiment(X_res, y_res, fold_ids_res, dev_residual, "Router-residual only")
# Generate report
report = generate_report(
meta, all_results, residual_results,
all_results["oof_rows"], residual_results["oof_rows"],
)
with open(REPORT_PATH, "w") as f:
f.write(report)
print(f"\nReport written to {REPORT_PATH}")
# Also print summary
print("\n" + "="*60)
print(" SUMMARY")
print("="*60)
print(f" All-example: acc={fmt_pct(all_results['full_metrics']['accuracy'])} macro_f1={fmt_float(all_results['full_metrics']['macro_f1'])} false_action={all_results['full_metrics']['false_action_count']}")
print(f" Residual: acc={fmt_pct(residual_results['full_metrics']['accuracy'])} macro_f1={fmt_float(residual_results['full_metrics']['macro_f1'])} false_action={residual_results['full_metrics']['false_action_count']}")
print(f" Best C (all): {all_results['best_C']}")
print(f" Best C (res): {residual_results['best_C']}")
print(f" ECE (all): {fmt_float(all_results['calibration']['ece'])}")
print(f" ECE (res): {fmt_float(residual_results['calibration']['ece'])}")
print(f" Brier (all): {fmt_float(all_results['calibration']['brier'])}")
print(f" Brier (res): {fmt_float(residual_results['calibration']['brier'])}")
+273
View File
@@ -0,0 +1,273 @@
// semantic-router-experiment computes embeddings for the semantic coarse-route
// corpus using the deployed multilingual-e5-small ONNX model. It outputs a
// JSON file containing every corpus row with its embedding vector, split
// assignment, and fold membership for grouped cross-validation.
//
// Usage:
//
// MAVEN_ONNX_LIB=/path/to/libonnxruntime.so \
// go run ./cmd/semantic-router-experiment/ -out embeddings.json
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"flag"
"fmt"
"log"
"os"
"sort"
"strings"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/router/semantic"
)
// CachedRow is one corpus row with its precomputed embedding and split metadata.
type CachedRow struct {
Text string `json:"text"`
Route string `json:"route"`
Source string `json:"source"`
SourceID string `json:"source_id"`
SplitGroup string `json:"split_group"`
Tags []string `json:"tags,omitempty"`
FastPathResolved bool `json:"fast_path_resolved"`
RouterResidual *bool `json:"router_residual,omitempty"`
TextHash string `json:"text_hash"`
Embedding []float32 `json:"embedding"`
EmbedderID string `json:"embedder_id"`
FrozenHoldout bool `json:"frozen_holdout"`
CVFold int `json:"cv_fold"`
DevPool bool `json:"dev_pool"`
FamilyID string `json:"family_id"`
}
// ExperimentMeta carries metadata about the experiment run.
type ExperimentMeta struct {
EmbedderID string `json:"embedder_id"`
ModelPath string `json:"model_path"`
TokenizerPath string `json:"tokenizer_path"`
Dimension int `json:"dimension"`
Pooling string `json:"pooling"`
Normalization string `json:"normalization"`
InputTemplate string `json:"input_template"`
TotalExamples int `json:"total_examples"`
FrozenCount int `json:"frozen_count"`
DevCount int `json:"dev_count"`
CVFolds int `json:"cv_folds"`
FoldComposition map[int]FoldStats `json:"fold_composition"`
RouteCounts map[string]int `json:"route_counts"`
SourceCounts map[string]int `json:"source_counts"`
FastPathCount int `json:"fast_path_count"`
ResidualCount int `json:"residual_count"`
HoldoutHash string `json:"holdout_hash"`
}
// FoldStats describes one CV fold.
type FoldStats struct {
EvalCount int `json:"eval_count"`
TrainCount int `json:"train_count"`
Routes map[string]int `json:"eval_routes"`
}
func main() {
outPath := flag.String("out", "embeddings.json", "output JSON path")
folds := flag.Int("folds", 5, "number of CV folds")
flag.Parse()
libPath := os.Getenv("MAVEN_ONNX_LIB")
if libPath == "" {
log.Fatal("MAVEN_ONNX_LIB must be set to the libonnxruntime.so path")
}
// Resolve model paths relative to the module root (cwd when running with go run).
modelPath := "models/embedder/multilingual-e5-small/model_quantized.onnx"
tokPath := "models/embedder/multilingual-e5-small/tokenizer.json"
for _, p := range []string{libPath, modelPath, tokPath} {
if _, err := os.Stat(p); err != nil {
log.Fatalf("missing %s: %v", p, err)
}
}
// Load corpus.
exs, err := semantic.LoadCorpus()
if err != nil {
log.Fatalf("load corpus: %v", err)
}
fmt.Fprintf(os.Stderr, "corpus: %d examples\n", len(exs))
// Initialize embedder.
emb, err := router.NewONNXEmbedder(modelPath, tokPath, libPath)
if err != nil {
log.Fatalf("init embedder: %v", err)
}
defer emb.Close()
fmt.Fprintf(os.Stderr, "embedder: %s (dim=%d)\n", emb.ID(), emb.Dim())
// Compute frozen holdout / dev pool split.
_, devPool, holdoutHash := semantic.FrozenHoldoutSplit(exs)
fmt.Fprintf(os.Stderr, "frozen holdout: hash=%s, dev pool: %d examples\n", holdoutHash, len(devPool))
// Compute grouped CV folds on dev pool only.
cvFolds := semantic.GroupedCVFolds(devPool, *folds)
fmt.Fprintf(os.Stderr, "cv folds: %d\n", len(cvFolds))
// Build a lookup: source_id → cv_fold (from dev pool only).
foldLookup := make(map[string]int)
for _, f := range cvFolds {
for _, e := range f.Eval {
foldLookup[e.SourceID] = f.Fold
}
}
// Build dev set membership lookup.
_, devSet, _ := semantic.FrozenHoldoutSplit(exs)
devIDs := make(map[string]bool)
for _, e := range devSet {
devIDs[e.SourceID] = true
}
// Embed all examples.
ctx := context.Background()
var cached []CachedRow
foldComp := make(map[int]*FoldStats)
for i := 0; i < *folds; i++ {
foldComp[i] = &FoldStats{Routes: make(map[string]int)}
}
routeCounts := make(map[string]int)
sourceCounts := make(map[string]int)
fpCount, resCount := 0, 0
for i, e := range exs {
textHash := sha256.Sum256([]byte(e.Text))
embedding, err := emb.EmbedQuery(ctx, e.Text)
if err != nil {
log.Fatalf("embed row %d (%s): %v", i, e.SourceID, err)
}
inDev := devIDs[e.SourceID]
fold := -1
if inDev {
if f, ok := foldLookup[e.SourceID]; ok {
fold = f
}
}
isFrozen := !inDev
cr := CachedRow{
Text: e.Text,
Route: string(e.Route),
Source: e.Source,
SourceID: e.SourceID,
SplitGroup: e.SplitGroup,
Tags: e.Tags,
FastPathResolved: e.FastPathResolved,
RouterResidual: e.RouterResidual,
TextHash: hex.EncodeToString(textHash[:]),
Embedding: embedding,
EmbedderID: emb.ID(),
FrozenHoldout: isFrozen,
CVFold: fold,
DevPool: inDev,
FamilyID: e.SplitGroup,
}
cached = append(cached, cr)
routeCounts[cr.Route]++
sourceCounts[cr.Source]++
if e.FastPathResolved {
fpCount++
} else {
resCount++
}
if inDev && fold >= 0 {
foldComp[fold].EvalCount++
foldComp[fold].Routes[cr.Route]++
}
}
// Compute train counts per fold.
for i := 0; i < *folds; i++ {
foldComp[i].TrainCount = len(devPool) - foldComp[i].EvalCount
}
// Sort route counts for deterministic output.
sortedRoutes := make([]string, 0, len(routeCounts))
for r := range routeCounts {
sortedRoutes = append(sortedRoutes, r)
}
sort.Strings(sortedRoutes)
sortedRouteCounts := make(map[string]int)
for _, r := range sortedRoutes {
sortedRouteCounts[r] = routeCounts[r]
}
// Build fold stats with sorted keys.
finalFoldComp := make(map[int]FoldStats)
for i := 0; i < *folds; i++ {
finalFoldComp[i] = *foldComp[i]
}
meta := ExperimentMeta{
EmbedderID: emb.ID(),
ModelPath: modelPath,
TokenizerPath: tokPath,
Dimension: emb.Dim(),
Pooling: "mean-pool + L2-normalize",
Normalization: "L2",
InputTemplate: "query: <text>",
TotalExamples: len(exs),
FrozenCount: len(exs) - len(devPool),
DevCount: len(devPool),
CVFolds: *folds,
FoldComposition: finalFoldComp,
RouteCounts: sortedRouteCounts,
SourceCounts: sourceCounts,
FastPathCount: fpCount,
ResidualCount: resCount,
HoldoutHash: holdoutHash,
}
// Output.
output := map[string]any{
"meta": meta,
"examples": cached,
}
data, err := json.MarshalIndent(output, "", " ")
if err != nil {
log.Fatalf("marshal: %v", err)
}
if err := os.WriteFile(*outPath, data, 0644); err != nil {
log.Fatalf("write %s: %v", *outPath, err)
}
// Print summary.
fmt.Fprintf(os.Stderr, "\n=== experiment metadata ===\n")
fmt.Fprintf(os.Stderr, "embedder: %s\n", meta.EmbedderID)
fmt.Fprintf(os.Stderr, "dimension: %d\n", meta.Dimension)
fmt.Fprintf(os.Stderr, "total: %d frozen: %d dev: %d\n", meta.TotalExamples, meta.FrozenCount, meta.DevCount)
fmt.Fprintf(os.Stderr, "fast-path: %d residual: %d\n", meta.FastPathCount, meta.ResidualCount)
fmt.Fprintf(os.Stderr, "routes: %s\n", formatMap(sortedRouteCounts))
fmt.Fprintf(os.Stderr, "fold composition:\n")
for i := 0; i < *folds; i++ {
fs := finalFoldComp[i]
fmt.Fprintf(os.Stderr, " fold %d: eval=%d train=%d routes=%s\n",
i, fs.EvalCount, fs.TrainCount, formatMap(fs.Routes))
}
fmt.Fprintf(os.Stderr, "output: %s (%d bytes)\n", *outPath, len(data))
}
func formatMap(m map[string]int) string {
var parts []string
for k, v := range m {
parts = append(parts, fmt.Sprintf("%s=%d", k, v))
}
sort.Strings(parts)
return "{" + strings.Join(parts, ", ") + "}"
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,665 @@
#!/usr/bin/env python3
"""
Slice 18: Sparse Lexical Action/Non-Action Gate
================================================
Answer: can Maven reliably distinguish executable requests from semantically
similar non-actions using lexical/local-order features alone?
Representations under test (all frozen-population, no e5):
A. word 1-2 grams, TF-IDF
B. character 3-5 grams, TF-IDF (Unicode, no transliteration)
C. [word ; char] combined TF-IDF
Population reused from slices 15-17: development corpus v2 (dev_pool), router
labels, SplitGroup, cv_fold, tags. The e5 embedding vectors are ignored.
"""
import io
import json
import re
import sys
import time
import unicodedata
import warnings
from collections import Counter, defaultdict
import numpy as np
from sklearn.exceptions import ConvergenceWarning
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
accuracy_score,
average_precision_score,
confusion_matrix,
f1_score,
precision_recall_fscore_support,
roc_auc_score,
)
from sklearn.pipeline import make_pipeline
from scipy import sparse
warnings.filterwarnings("ignore", category=ConvergenceWarning)
warnings.filterwarnings("ignore", category=UserWarning)
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
# Present generator/contrast families in the v2 dev pool (for leave-family-out)
PRESENT_FAMILIES = [
"polite_request",
"modal_request",
"first_person_request",
"reordered_target",
"capability_question",
"question",
]
FAMILY_ALIASES = {
"direct_imperative": "direct_imperative",
"polite_request": "polite_request",
"modal_request": "modal_request",
"first_person_request": "first_person_request",
"reordered_target": "reordered_target",
"capability_question": "capability_question",
"question": "question",
}
# ─── NormalizeMatchText (replicated from internal/router/matchtext.go) ─────
def normalize_match_text(s: str) -> str:
"""NFKC → lowercase → collapse Unicode whitespace. Keeps punctuation, ё."""
out = unicodedata.normalize("NFKC", s).strip().lower()
out = re.sub(r"\s+", " ", out)
return out
# ─── Data Loading ───────────────────────────────────────────────────────────
def load_data():
with open(EMBEDDING_PATH) as f:
data = json.load(f)
return data["meta"], data["examples"]
def filter_dev_pool(examples):
return [e for e in examples if e["dev_pool"]]
def residual_only(examples):
return [e for e in examples if not e["fast_path_resolved"]]
def fmt_pct(v, d=1):
return f"{100*v:.{d}f}%"
def ff(v, d=3):
return f"{v:.{d}f}"
def strip_punct(text: str) -> str:
"""Remove all punctuation (shared with slice 16/17 apply_voice_stress)."""
t = re.sub(r"[?.!,;:]+$", "", text.strip())
t = re.sub(r"[^\w\s]", "", t)
t = t.lower()
t = re.sub(r"\s+", " ", t).strip()
return t
# ─── Feature Builders ───────────────────────────────────────────────────────
def build_features(texts, kind):
"""Build a TF-IDF matrix for the given representation kind.
kind in {'word','char','both'}. Returns (X_sparse, vectorizer)."""
if kind == "word":
vec = TfidfVectorizer(
ngram_range=(1, 2), analyzer="word",
strip_accents=None, lowercase=False,
min_df=2, sublinear_tf=True,
)
elif kind == "char":
# preserve case (already lowered) and identity of missing diacritics;
# token_pattern null => char analyzer
vec = TfidfVectorizer(
ngram_range=(3, 5), analyzer="char",
strip_accents=None, lowercase=False,
min_df=2, sublinear_tf=True,
)
elif kind == "both":
vec_word = TfidfVectorizer(
ngram_range=(1, 2), analyzer="word",
strip_accents=None, lowercase=False, min_df=2, sublinear_tf=True,
)
vec_char = TfidfVectorizer(
ngram_range=(3, 5), analyzer="char",
strip_accents=None, lowercase=False, min_df=2, sublinear_tf=True,
)
Xw = vec_word.fit_transform(texts)
Xc = vec_char.fit_transform(texts)
X = sparse.hstack([Xw, Xc]).tocsr()
return X, ("both", vec_word, vec_char)
X = vec.fit_transform(texts)
return X, vec
def vocab_size(vectorizer):
if isinstance(vectorizer, tuple):
_, vw, vc = vectorizer
return vw.get_feature_names_out().shape[0] + vc.get_feature_names_out().shape[0]
return vectorizer.get_feature_names_out().shape[0]
def transform_texts(texts, vectorizer):
"""Apply an already-fitted vectorizer (handles the 2-tuple 'both' case)."""
if isinstance(vectorizer, tuple):
_, vw, vc = vectorizer
Xw = vw.transform(texts)
Xc = vc.transform(texts)
return sparse.hstack([Xw, Xc]).tocsr()
return vectorizer.transform(texts)
# ─── Grouped CV ─────────────────────────────────────────────────────────────
def run_binary_grouped_cv(X, y, fold_ids, C=1.0):
"""Grouped CV for binary action vs not_action. Returns OOF rows + fold metrics."""
yb = np.array([1 if t == "action" else 0 for t in y])
fold_ids = np.asarray(fold_ids)
unique_folds = sorted(set(fold_ids.tolist()))
oof_rows = []
fold_metrics = []
for test_fold in unique_folds:
tr = fold_ids != test_fold
te = fold_ids == test_fold
clf = LogisticRegression(C=C, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yb[tr])
proba = clf.predict_proba(X[te])[:, 1]
pred = (proba >= 0.5).astype(int)
yt = yb[te]
fp = int(((yt == 0) & (pred == 1)).sum())
fn = int(((yt == 1) & (pred == 0)).sum())
tp = int(((yt == 1) & (pred == 1)).sum())
tn = int(((yt == 0) & (pred == 0)).sum())
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
acc = accuracy_score(yt, pred)
prec = tp / max(tp + fp, 1)
rec = tp / max(tp + fn, 1)
fold_metrics.append({
"fold": int(test_fold), "n": int(len(yt)),
"roc_auc": roc, "pr_auc": pr,
"action_precision": prec, "action_recall": rec,
"fp": fp, "fn": fn, "tp": tp, "tn": tn,
"acc": acc,
})
te_idx = np.where(te)[0]
for i in range(len(yt)):
oof_rows.append({
"fold": int(test_fold),
"proba": float(proba[i]),
"pred": int(pred[i]),
"true": int(yt[i]),
})
return oof_rows, fold_metrics
# ─── Metrics from OOF ───────────────────────────────────────────────────────
def binary_metrics_from_oof(oof_rows):
yt = np.array([r["true"] for r in oof_rows])
yp = np.array([r["pred"] for r in oof_rows])
proba = np.array([r["proba"] for r in oof_rows])
n = len(yt)
tp = int(((yt == 1) & (yp == 1)).sum())
fp = int(((yt == 0) & (yp == 1)).sum())
fn = int(((yt == 1) & (yp == 0)).sum())
prec = tp / max(tp + fp, 1)
rec = tp / max(tp + fn, 1)
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
return {
"n": n, "tp": tp, "fp": fp, "fn": fn,
"action_precision": prec, "action_recall": rec,
"fa_rate": fp / max(n, 1),
"roc_auc": roc, "pr_auc": pr,
}
def threshold_curve(oof_rows, thresholds):
yt = np.array([r["true"] for r in oof_rows])
proba = np.array([r["proba"] for r in oof_rows])
rows = []
for thr in thresholds:
yp = (proba >= thr).astype(int)
tp = int(((yt == 1) & (yp == 1)).sum())
fp = int(((yt == 0) & (yp == 1)).sum())
fn = int(((yt == 1) & (yp == 0)).sum())
rows.append({
"threshold": round(float(thr), 4),
"action_precision": round(tp / max(tp + fp, 1), 4),
"action_recall": round(tp / max(tp + fn, 1), 4),
"fa_count": fp,
"fa_rate": round(fp / max(len(yt), 1), 4),
})
return rows
# ─── Leave-Family-Out ───────────────────────────────────────────────────────
def run_leave_family_out(texts, y, family_per_row, family, kind):
"""Train without `family`, evaluate on `family` only."""
mask_members = family_per_row == family
if mask_members.sum() == 0:
return None
only_family = (mask_members).astype(bool)
train_idx = np.where(~only_family)[0]
test_idx = np.where(only_family)[0]
X, _ = build_features([texts[i] for i in train_idx], kind)
# map test rows onto the full vocabulary
tr_texts = [texts[i] for i in train_idx]
te_texts = [texts[i] for i in test_idx]
all_texts = tr_texts + te_texts
Xall, _ = build_features(all_texts, kind)
Xtr = Xall[:len(tr_texts)]
Xte = Xall[len(tr_texts):]
ytr = np.array([1 if y[i] == "action" else 0 for i in train_idx])
yte = np.array([1 if y[i] == "action" else 0 for i in test_idx])
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(Xtr, ytr)
pred = clf.predict(Xte)
tp = int(((yte == 1) & (pred == 1)).sum())
fp = int(((yte == 0) & (pred == 1)).sum())
fn = int(((yte == 1) & (pred == 0)).sum())
prec = tp / max(tp + fp, 1)
rec = tp / max(tp + fn, 1)
acc = accuracy_score(yte, pred)
return {
"family": family, "rows": int(len(yte)),
"action_precision": prec, "action_recall": rec,
"fa_count": fp, "acc": acc,
"yticks": f"pos={int(yte.sum())} neg={int((yte==0).sum())}",
}
# ─── Paired Action/Capability Test ──────────────────────────────────────────
def paired_action_capability(texts, y, tags_per_row_by_idx, kind, C=1.0):
"""For the best sparse representation, evaluate pairwise ordering."""
idx = list(range(len(texts)))
emb = build_features(texts, kind)[0]
clf = LogisticRegression(C=C, max_iter=2000, solver="lbfgs", random_state=42)
yb = np.array([1 if t == "action" else 0 for t in y])
# use OOF-style: fit on full then derive? We report the pairwise score test;
# to avoid leakage we use grouped OOF proba via grouped cv.
return None
# ─── Main ──────────────────────────────────────────────────────────────────
def main():
meta, examples = load_data()
dev = filter_dev_pool(examples)
dev_res = residual_only(dev)
# normalized texts
for e in dev:
e["n_text"] = normalize_match_text(e["text"])
texts = [e["n_text"] for e in dev]
y = [e["route"] for e in dev]
folds = [e["cv_fold"] for e in dev]
tags = [set(e.get("tags", [])) for e in dev]
split_groups = [e["split_group"] for e in dev]
source_ids = [e["source_id"] for e in dev]
n_action = sum(1 for r in y if r == "action")
n_not = len(y) - n_action
print(f"Dev pool: {len(dev)} action={n_action} not_action={n_not}")
print()
results = {}
# ── 1. Build features and run grouped CV for each representation ──────
for kind in ["word", "char", "both"]:
print(f"\n=== {kind} TF-IDF ===")
t0 = time.time()
X, vec = build_features(texts, kind)
build_t = time.time() - t0
vs = vocab_size(vec)
print(f" vocab={vs} X.shape={X.shape} build={build_t:.2f}s")
oof, folds_m = run_binary_grouped_cv(X, y, folds, C=1.0)
m = binary_metrics_from_oof(oof)
results[kind] = {
"vs": vs, "build_t": build_t, "oof": oof, "fold_metrics": folds_m,
"metrics": m, "X": X, "vec": vec,
"texts": texts, "y": y, "folds": folds,
}
print(f" ROC={ff(m['roc_auc'])} PR={ff(m['pr_auc'])} P={ff(m['action_precision'])} "
f"R={ff(m['action_recall'])} FA={m['fp']} ({fmt_pct(m['fa_rate'])})")
# fold-level
for fm in folds_m:
print(f" fold {fm['fold']}: ROC={ff(fm['roc_auc'])} PR={ff(fm['pr_auc'])} "
f"P={ff(fm['action_precision'])} R={ff(fm['action_recall'])} "
f"FP={fm['fp']} FN={fm['fn']} n={fm['n']}")
# pick best by PR-AUC
best_kind = max(["word", "char", "both"], key=lambda k: results[k]["metrics"]["pr_auc"])
print(f"\nBest representation by PR-AUC: {best_kind}")
# ── 2. Leave-family-out for best kind ─────────────────────────────────
print(f"\n=== Leave-family-out ({best_kind}) ===")
fam_per_row = []
for tg in tags:
fam = None
# prefer the more specific contrast/request families first (a single
# utterance may carry several generator tags, e.g. capability_question
# plus direct_imperative). Check the informative ones before the
# generic direct_imperative fallback.
for f in ["capability_question", "question", "first_person_request",
"modal_request", "polite_request", "reordered_target",
"direct_imperative"]:
if f in tg:
fam = f
break
fam_per_row.append(fam)
fam_per_row = np.array(fam_per_row, dtype=object)
lfo = {}
for fam in PRESENT_FAMILIES:
r = run_leave_family_out(texts, y, fam_per_row, fam, best_kind)
if r is None:
print(f" {fam}: (no rows)")
continue
lfo[fam] = r
print(f" {fam}: rows={r['rows']} ({r['yticks']}) P={ff(r['action_precision'])} "
f"R={ff(r['action_recall'])} FA={r['fa_count']} acc={fmt_pct(r['acc'])}")
# ── 3. Action/capability paired test ─────────────────────────────────
print(f"\n=== Paired action/capability test ({best_kind}) ===")
# Use grouped-CV OOF proba for ordering (no leakage)
oof = results[best_kind]["oof"]
# map oof rows back by source_id order
# oof rows are appended per fold in dev order; reconstruct
# We'll re-embed and get proba via grouped CV with proba recorded per row.
# Re-run grouped CV capturing per-row proba aligned to dev indices.
X = results[best_kind]["X"]
yb = np.array([1 if r == "action" else 0 for r in y])
folds_arr = np.array(folds)
dev_proba = np.zeros(len(dev))
for te_fold in sorted(set(folds)):
tr = folds_arr != te_fold
te = folds_arr == te_fold
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yb[tr])
dev_proba[te] = clf.predict_proba(X[te])[:, 1]
# group action seeds: pair each capability-question row with action rows
# sharing the same target object noun and the same act (turn_on/turn_off/…).
# Both were generated over a common entity+event inventory, so the object
# lexeme is the semantic link between a knowledge question and its
# executable sibling.
DEVICES = [
"свет", "люстру", "люстра", "жалюзи", "вытяжку", "вытяжка",
"вентилятор", "кондиционер", "телевизор", "лампу", "лампа",
"музыку", "музыка", "плеер", "колонку", "колонки", "чайник",
"бойлер", "обогреватель", "пылесос", "пылесосом", "пол",
"поливалки", "полив", "арка", "шторы", "штору", "штору",
"динамики", "дверь", "двери", "замок", "гараж", "ворота",
"кофе", "пасту", "зубы", "крючки", "лаймо", "куртку",
"будильник", "таймер", "напоминание", "расписание",
]
def object_nouns(t):
found = set()
tl = t.lower()
for d in DEVICES:
# match as standalone word (handle Russian case endings loosely via prefix)
if re.search(r"\b" + re.escape(d), tl):
found.add(d)
return found
# verb/act family per row from source_id (e.g. ha-light-off -> off)
def act_family(src):
# pull the 3rd token-ish: ha-light-off => 'off'; ha-light-on => 'on'
m = re.search(r"^(\w+)-([a-z_]+)-([a-zA-Z_]+)", src)
if m:
return f"{m.group(1)}-{m.group(2)}-{m.group(3)}"
# generic fallback
return src.split("-")[0]
cap_rows = [i for i in range(len(dev)) if "capability_question" in tags[i]]
act_idxs = [i for i in range(len(dev)) if y[i] == "action"]
# For each capability row, candidate sibling actions: same object noun
# AND same act (turn_on vs turn_off), i.e. same domain+object. We accept
# any action row sharing an object and matching the on/off sense if present.
pairs = []
for cidx in cap_rows:
c_obj = object_nouns(texts[cidx])
if not c_obj:
continue
# cap rows are kq-cap-<domain>-<n>; domain token after 'kq-cap-'
dom = re.search(r"kq-cap-([^-]+)", source_ids[cidx])
dom = dom.group(1) if dom else None
for aidx in act_idxs:
a_obj = object_nouns(texts[aidx])
if not (c_obj & a_obj):
continue
# require same home domain when both carry one
a_dom = re.search(r"^([a-z]+)-", source_ids[aidx])
a_dom = a_dom.group(1) if a_dom else None
if dom and a_dom and dom != a_dom:
continue
pairs.append((cidx, aidx))
# keep it bounded: cap row pairs with many actions (one per room); that's fine
order_ok = 0
margins = []
reversed_pairs = []
for cidx, aidx in pairs:
pc = dev_proba[cidx]
pa = dev_proba[aidx]
margins.append(pa - pc)
if pa > pc:
order_ok += 1
else:
reversed_pairs.append((texts[cidx][:40], pc, texts[aidx][:40], pa))
if pairs:
order_acc = order_ok / len(pairs)
margins_arr = np.array(margins)
print(f" pairs={len(pairs)} order_acc={ff(order_acc)} mean_margin={ff(margins_arr.mean())} "
f"median_margin={ff(np.median(margins_arr))} reversed={len(reversed_pairs)}")
for rev in reversed_pairs[:12]:
print(f" REV: cap '{rev[0]}' P={rev[1]:.3f} < act '{rev[2]}' P={rev[3]:.3f}")
else:
order_acc = None
print(" (no matched pairs)")
paired = {
"pairs": len(pairs), "order_acc": order_acc,
"mean_margin": float(np.mean(margins)) if margins else None,
"median_margin": float(np.median(margins)) if margins else None,
"reversed": len(reversed_pairs),
}
# ── 4. Voice stress scoring ───────────────────────────────────────────
print(f"\n=== Voice-like stress ({best_kind}) ===")
# Build stress variants, embed, score with a model trained on normal text
# Train one model on full dev (normal punctuation); score stressed variants.
clf_full = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf_full.fit(X, yb)
stress_texts = []
stress_meta = [] # (orig_index, mode)
for i in range(len(dev)):
orig = texts[i]
s_nofinal = re.sub(r"[?.!,;:]+$", "", orig)
s_all = strip_punct(dev[i]["text"])
if s_nofinal != orig:
stress_texts.append(s_nofinal); stress_meta.append((i, "no_final"))
if s_all != orig:
stress_texts.append(s_all); stress_meta.append((i, "all"))
if stress_texts:
# reuse the already-fitted best-kind vectorizer to keep the feature space
# identical to what the full model was trained on.
X_stress = transform_texts(stress_texts, results[best_kind]["vec"])
proba_stress = clf_full.predict_proba(X_stress)[:, 1]
else:
proba_stress = []
# classify ability: capability-question false-action rate and modal-action recall
stress_by_mode = defaultdict(list)
for (i, mode), p in zip(stress_meta, proba_stress):
stress_by_mode[mode].append((i, p, y[i], tags[i]))
voice = {}
for mode, rows in stress_by_mode.items():
cap_q = [p for (i, p, lbl, tg) in rows if "capability_question" in tg]
cap_fa = sum(1 for p in cap_q if p >= 0.5)
mod_act = [p for (i, p, lbl, tg) in rows if lbl == "action" and ("polite_request" in tg or "modal_request" in tg)]
mod_rec = sum(1 for p in mod_act if p >= 0.5) / max(len(mod_act), 1)
voice[mode] = {
"n": len(rows),
"cap_q_fa": cap_fa / max(len(cap_q), 1), "cap_q_n": len(cap_q),
"modal_action_recall": mod_rec, "modal_n": len(mod_act),
}
print(f" {mode}: n={len(rows)} cap_q_FA={fmt_pct(voice[mode]['cap_q_fa'])} ({voice[mode]['cap_q_n']}) "
f"modal_recall={ff(voice[mode]['modal_action_recall'])} ({voice[mode]['modal_n']})")
# ── 5. Punctuation ablation ───────────────────────────────────────────
print(f"\n=== Punctuation ablation ({best_kind}) ===")
texts_stripped = [strip_punct(dev[i]["text"]) for i in range(len(dev))]
X_stripped_train, vec_stripped = build_features(texts_stripped, best_kind)
clf_stripped = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf_stripped.fit(X_stripped_train, yb)
# eval on stripped (in-train) and punctuated (out-of-train) via same vectorizer
X_orig = transform_texts(texts, vec_stripped)
p_orig = clf_stripped.predict_proba(X_orig)[:, 1]
p_strip = clf_stripped.predict_proba(X_stripped_train)[:, 1]
def report_ablation(proba, name):
yp = (proba >= 0.5).astype(int)
tp = int(((yb == 1) & (yp == 1)).sum())
fp = int(((yb == 0) & (yp == 1)).sum())
fn = int(((yb == 1) & (yp == 0)).sum())
prec = tp / max(tp + fp, 1); rec = tp / max(tp + fn, 1)
print(f" trained-stripped, eval {name}: P={ff(prec)} R={ff(rec)} FA={fp} ({fmt_pct(fp/len(yb))})")
report_ablation(p_orig, "punctuated")
report_ablation(p_strip, "stripped")
# ── 6. Threshold curve for best kind ──────────────────────────────────
print(f"\n=== Threshold curve ({best_kind}) ===")
oof = results[best_kind]["oof"]
thresh = np.arange(0.10, 0.995, 0.015).tolist()
tcurve = threshold_curve(oof, thresh)
print(f"{'thr':>6} {'P':>6} {'R':>6} {'FA':>5} {'FArate':>8}")
for t in tcurve:
marker = ""
if t["action_precision"] >= 0.95:
marker = " ← P>=0.95"
print(f"{t['threshold']:>6.3f} {t['action_precision']:>6.3f} {t['action_recall']:>6.3f} "
f"{t['fa_count']:>5} {t['fa_rate']:>8.4f}{marker}")
p95 = [t for t in tcurve if t["action_precision"] >= 0.95 and t["action_recall"] > 0.01]
print(f"\nP>=0.95 region: {len(p95)} points; best recall there = "
f"{max((t['action_recall'] for t in p95), default=0.0):.4f}")
# threshold curve using the (no_leak) dev proba instead of .5-threshold OOF
# OOF pred used fixed 0.5; curve above re-derives from proba. Good.
# ── 7. False-action decomposition ────────────────────────────────────
print(f"\n=== False-action decomposition ({best_kind}) ===")
# recompute OOF predictions at 0.5 from stored 'pred'
fa_by_family = Counter()
fa_by_group = Counter()
# need oof aligned to source_ids — oof stored without source id; rebuild
# Re-do grouped cv capturing source_id + tag + split_group
fake_oof = []
for te_fold in sorted(set(folds_arr)):
tr = folds_arr != te_fold; te = folds_arr == te_fold
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yb[tr])
p = clf.predict_proba(X[te])[:, 1]
pr = (p >= 0.5).astype(int)
te_idx = np.where(te)[0]
for k, i in enumerate(te_idx):
fake_oof.append({
"source_id": source_ids[i], "split_group": split_groups[i],
"tags": tags[i], "true": y[i], "proba": float(p[k]),
"pred": int(pr[k]), "fold": int(te_fold),
})
def classify_semantic_family(row):
tg = row["tags"]
if "capability_question" in tg: return "capability_question"
if "question" in tg: return "ordinary_question"
if "remember" in tg or "note" in tg or "idea" in tg or "free_form" in tg: return "memory_write"
if "version" in tg or "system" in tg or "health" in tg or "status" in tg: return "system"
if "recall" in tg or "world" in tg or "definition" in tg or "aggregate" in tg: return "knowledge_general"
if "greeting" in tg or "goodbye" in tg or "thanks" in tg: return "conversation"
if row["true"].startswith("system"): return "system"
return "other"
for r in fake_oof:
if r["pred"] == 1 and r["true"] != "action":
fam = classify_semantic_family(r)
fa_by_family[fam] += 1
fa_by_group[r["split_group"]] += 1
print("By family:")
for k, v in fa_by_family.most_common():
print(f" {k}: {v}")
print("By SplitGroup (top 15):")
for k, v in fa_by_group.most_common(15):
print(f" {k}: {v}")
# ── 8. Six-way probe on best representation ──────────────────────────
print(f"\n=== Six-way probe ({best_kind}) ===")
routes = ["action", "conversation", "knowledge", "memory_write", "system", "uncertain"]
y6 = np.array(y)
# fit grouped cv 6-way
oof6 = []
fold6 = []
for te_fold in sorted(set(folds_arr)):
tr = folds_arr != te_fold; te = folds_arr == te_fold
clf = LogisticRegression(C=1.0, max_iter=3000, solver="lbfgs", random_state=42)
clf.fit(X[tr], y6[tr])
pr = clf.predict(X[te])
p6 = clf.predict_proba(X[te])
classes = clf.classes_
te_idx = np.where(te)[0]
for k, i in enumerate(te_idx):
oof6.append({
"true": y6[i], "pred": pr[k], "proba": {c: float(p6[k][j]) for j, c in enumerate(classes)},
"tags": tags[i], "split_group": split_groups[i], "source_id": source_ids[i],
})
yt6 = [r["true"] for r in oof6]; yp6 = [r["pred"] for r in oof6]
acc6 = accuracy_score(yt6, yp6)
macro6 = f1_score(yt6, yp6, average="macro", zero_division=0)
prec6, rec6, f16, sup6 = precision_recall_fscore_support(yt6, yp6, labels=routes, zero_division=0)
fa6 = sum(1 for t, p in zip(yt6, yp6) if t != "action" and p == "action")
ap6 = sum(1 for t, p in zip(yt6, yp6) if t == "action" and p == "action") / max(sum(1 for p in yp6 if p == "action"), 1)
ar6 = sum(1 for t, p in zip(yt6, yp6) if t == "action" and p == "action") / max(sum(1 for t in yt6 if t == "action"), 1)
print(f" acc={fmt_pct(acc6)} macroF1={ff(macro6)} actionP={ff(ap6)} actionR={ff(ar6)} FA={fa6} ({fmt_pct(fa6/len(yt6))})")
for i, r in enumerate(routes):
print(f" {r:<12} P={ff(prec6[i])} R={ff(rec6[i])} F1={ff(f16[i])} n={int(sup6[i])}")
# ── 9. Artifact size / runtime ───────────────────────────────────────
print(f"\n=== Artifact size / runtime ===")
for kind in ["word", "char", "both"]:
rr = results[kind]
vec = rr["vec"]
ncoef = rr["metrics"]["n"]
# non-zero coefficients = vocab (TF-IDF), logistic has 1 weight per vocab
print(f" {kind}: vocab={rr['vs']} fp32 model bytes={rr['vs']*4} "
f"build={rr['build_t']:.3f}s")
# done
print("\nDone.")
if __name__ == "__main__":
main()
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""
Slice 19 tokenizers, derived from the development corpus only.
A. CharVocab codepoint ids over the dev corpus (deterministic order)
B. BpeVocab2048 byte-level BPE, vocab ~2048, trained on dev corpus only
Records for §4 of the brief: vocab size, OOV behaviour, serialized tokenizer size.
"""
import re
import unicodedata
from tokenizers import Tokenizer
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import ByteLevel as ByteLevelPreTokenizer
from tokenizers.trainers import BpeTrainer
class CharVocab:
"""Codepoint ids from the dev corpus, sorted by codepoint value."""
def __init__(self, texts):
chars = set()
for t in texts:
chars.update(t)
self.id_to_char = sorted(chars)
self.char_to_id = {c: i + 1 for i, c in enumerate(self.id_to_char)} # 0 = PAD
self.pad = 0
@property
def size(self):
return len(self.id_to_char) + 1
def encode(self, text, max_len):
ids = [self.char_to_id.get(c, 0) for c in text] # 0 doubles as UNK/OOV
return ids[:max_len]
class BpeVocab:
"""Byte-level BPE, trained only on the strings it is given."""
def __init__(self, texts, vocab_size=2048, sep=""):
self.tok = Tokenizer(BPE())
self.tok.pre_tokenizer = ByteLevelPreTokenizer(trim_offsets=False)
self.tok.decoder = ByteLevelDecoder()
trainer = BpeTrainer(vocab_size=vocab_size, special_tokens=["[PAD]"],
show_progress=False)
# train on the corpus *strings*, byte-level BPE handles all codepoints
self.tok.train_from_iterator(texts, trainer=trainer)
self.pad_id = self.tok.token_to_id("[PAD]")
self._vocab = self.tok.get_vocab()
self._n = len(self._vocab)
@property
def size(self):
return self._n
def encode(self, text):
return self.tok.encode(text).ids
def serialized_bytes(self):
# measure the serialized tokenizer size on disk
import os
d = self.tok.to_str()
return len(d.encode("utf-8"))
def normalize_match_text(s: str) -> str:
"""NFKC → lowercase → collapse whitespace. Punctuation kept."""
out = unicodedata.normalize("NFKC", s).strip().lower()
out = re.sub(r"\s+", " ", out)
return out
def strip_punct(text: str) -> str:
"""Remove safe punctuation from an already-normalized text."""
t = re.sub(r"[^\w\s]", " ", text)
t = re.sub(r"\s+", " ", t).strip()
return t
@@ -0,0 +1,640 @@
#!/usr/bin/env python3
"""
Slice 19: tiny sequence-sensitive pragmatics specialists (binary action gate)
=============================================================================
A narrow binary specialist: executable request vs semantically similar
non-executable utterance, generalizing across surface-generator families.
Commands
grouped grouped semantic CV (existing cv_fold), all arch/sizes, saves
per-config per-fold model checkpoints + OOF proba per variant
metrics aggregate saved grouped-CV results into the report tables
(binary metrics, threshold curves, pair ordering, stress)
lfo leave-generator-out. All sizes on capability_question; the
other present families for the leading config only.
e5baseline frozen-e5 logistic + MLP baselines: grouped, cap-Q LOFO, pair
ordering (stress flagged NA no re-embed on this box)
runtime params / sizes / latency / tokenization for each candidate
Primary metrics (brief §2): cap-Q LOFO FA, pair ordering acc, pair margin,
grouped P/R, voice stress, fold variance. Aggregate accuracy is secondary.
"""
import argparse
import json
import os
import re
import resource
import subprocess
import sys
import time
import warnings
from collections import Counter
import numpy as np
warnings.filterwarnings("ignore")
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from slice19_bpe import BpeVocab, CharVocab, normalize_match_text, strip_punct
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
RESULTS_DIR = "/tmp/mvn-s19"
SEED = 42
MAX_CHAR = 64
MAX_BPE = 24
BPE_VOCAB = 2048
PRESENT_FAMILIES = [
"polite_request",
"modal_request",
"first_person_request",
"reordered_target",
"capability_question",
"question",
]
ABSENT_FAMILIES = ["negation", "reported_speech", "quotation", "hypothetical"]
FAMILY_PRIORITY = [
"capability_question", "question", "first_person_request",
"modal_request", "polite_request", "reordered_target", "direct_imperative",
]
DEVICES = [
"свет", "люстру", "люстра", "жалюзи", "вытяжку", "вытяжка",
"вентилятор", "кондиционер", "телевизор", "лампу", "лампа",
"музыку", "музыка", "плеер", "колонку", "колонки", "чайник",
"бойлер", "обогреватель", "пылесос", "пылесосом", "пол",
"поливалки", "полив", "арка", "шторы", "штору",
"динамики", "дверь", "двери", "замок", "гараж", "ворота",
"кофе", "пасту", "зубы", "крючки", "лаймо", "куртку",
"будильник", "таймер", "напоминание", "расписание",
]
ARCH_CONFIGS = {
"char_cnn": ["tiny", "medium"],
"bigru": ["tiny", "medium", "large"],
"tiny_transformer": ["small", "medium"],
}
TRAIN_HYPER = {
"char_cnn": dict(epochs=20, lr=1e-3, bs=64, clip=None),
"bigru": dict(epochs=15, lr=5e-4, bs=64, clip=1.0),
"tiny_transformer": dict(epochs=25, lr=5e-4, bs=64, clip=1.0),
}
VARIANTS = ["orig", "nofinal", "strip"]
# ─── data ───────────────────────────────────────────────────────────────────
def load_dev():
with open(EMBEDDING_PATH) as f:
data = json.load(f)
rows = [e for e in data["examples"] if e["dev_pool"]]
out = []
for e in rows:
n_text = normalize_match_text(e["text"])
out.append({
"text_orig": n_text,
"text_nofinal": re.sub(r"[?.!,;:]+$", "", n_text),
"text_strip": strip_punct(n_text),
"route": e["route"],
"y": 1 if e["route"] == "action" else 0,
"cv_fold": e["cv_fold"],
"split_group": e["split_group"],
"tags": set(e.get("tags", [])),
"source_id": e["source_id"],
"emb": np.asarray(e["embedding"], dtype=np.float32),
})
return out
def family_of(tags):
for f in FAMILY_PRIORITY:
if f in tags:
return f
return None
def build_pairs(rows, texts):
"""Capability-question vs action-by-shared-object pairs (eval only)."""
def object_nouns(t):
found = set()
tl = t.lower()
for d in DEVICES:
if re.search(r"\b" + re.escape(d), tl):
found.add(d)
return found
pairs = []
for cidx, r in enumerate(rows):
if "capability_question" not in r["tags"]:
continue
c_obj = object_nouns(texts[cidx])
if not c_obj:
continue
dom = re.search(r"kq-cap-([^-]+)", r["source_id"])
dom = dom.group(1) if dom else None
for aidx, ra in enumerate(rows):
if ra["y"] != 1:
continue
a_obj = object_nouns(texts[aidx])
if not (c_obj & a_obj):
continue
a_dom = re.search(r"^([a-z]+)-", ra["source_id"])
a_dom = a_dom.group(1) if a_dom else None
if dom and a_dom and dom != a_dom:
continue
pairs.append((cidx, aidx))
return pairs
def pair_metrics(pairs, proba):
if not pairs:
return {"pairs": 0}
margins = []
ties = 0
order = 0
rev = 0
for cidx, aidx in pairs:
pc, pa = proba[cidx], proba[aidx]
margins.append(pa - pc)
if pa > pc:
order += 1
elif pa == pc:
ties += 1
else:
rev += 1
m = np.array(margins)
return {
"pairs": len(pairs),
"ordering_acc": order / len(pairs),
"mean_margin": float(m.mean()),
"median_margin": float(np.median(m)),
"ties": ties,
"reversed": rev,
}
# ─── tokenizers ─────────────────────────────────────────────────────────────
def build_tokenizers(rows):
nat_texts = [r["text_orig"] for r in rows]
char_vocab = CharVocab(nat_texts)
bpe = BpeVocab(nat_texts, vocab_size=BPE_VOCAB)
return char_vocab, bpe
def encode_all(rows, tokenizer, kind):
"""Return dict variant -> (N, max_len) int64 array."""
max_len = MAX_CHAR if kind == "char" else MAX_BPE
out = {}
for v in VARIANTS:
arr = np.zeros((len(rows), max_len), dtype=np.int64)
for i, r in enumerate(rows):
if kind == "char":
ids = tokenizer.encode(r[f"text_{v}"], max_len)
arr[i, :len(ids)] = ids
else:
ids = tokenizer.encode(r[f"text_{v}"])[:max_len]
arr[i, :len(ids)] = ids
out[v] = arr
return out
# ─── training ───────────────────────────────────────────────────────────────
def build_model(arch, size, vocab_size, max_len):
import torch
from slice19_models import CharCNN, BiGRU, TinyTransformer
if arch == "char_cnn":
c = (dict(embed_dim=32, filters=64, widths=[3, 4, 5]) if size == "tiny"
else dict(embed_dim=64, filters=160, widths=[2, 3, 4, 5]))
return CharCNN(vocab_size, c["embed_dim"], c["filters"], c["widths"])
if arch == "bigru":
c = (dict(embed_dim=64, hidden=64) if size == "tiny" else
(dict(embed_dim=128, hidden=128) if size == "medium" else
dict(embed_dim=256, hidden=256)))
return BiGRU(vocab_size, c["embed_dim"], c["hidden"])
c = (dict(d_model=128, n_layers=2, n_heads=4) if size == "small" else
dict(d_model=192, n_layers=4, n_heads=4))
return TinyTransformer(vocab_size, c["d_model"], c["n_layers"], c["n_heads"],
max_len=max_len)
def train_binary(X, y, arch, size, vocab_size, seed_offset=0, log=False):
import torch
torch.manual_seed(SEED + seed_offset)
np.random.seed(SEED + seed_offset)
Xt = torch.from_numpy(X)
yt = torch.from_numpy(y.astype(np.float32))
model = build_model(arch, size, vocab_size, X.shape[1])
h = TRAIN_HYPER[arch]
opt = torch.optim.AdamW(model.parameters(), lr=h["lr"], weight_decay=1e-4)
lossf = torch.nn.BCEWithLogitsLoss()
n = X.shape[0]
model.train()
t0 = time.time()
for epoch in range(h["epochs"]):
perm = torch.randperm(n)
running = 0.0
n_b = 0
for start in range(0, n, h["bs"]):
idx = perm[start:start + h["bs"]]
xb = Xt[idx]
if xb.dim() == 1:
xb = xb.unsqueeze(0)
logits = model(xb)
loss = lossf(logits, yt[idx])
opt.zero_grad()
loss.backward()
if h["clip"]:
torch.nn.utils.clip_grad_norm_(model.parameters(), h["clip"])
opt.step()
running += float(loss)
n_b += 1
if log and (epoch + 1) % 5 == 0:
print(f" epoch {epoch+1}/{h['epochs']} loss {running/max(n_b,1):.4f}")
return model, time.time() - t0
def predict_proba(model, X, bs=256):
import torch
model.eval()
out = []
with torch.no_grad():
Xt = torch.from_numpy(X)
for start in range(0, X.shape[0], bs):
xb = Xt[start:start + bs]
if xb.dim() == 1:
xb = xb.unsqueeze(0)
logits = model(xb)
out.append(torch.sigmoid(logits).numpy())
return np.concatenate(out)
# ─── metrics helpers ────────────────────────────────────────────────────────
def binary_metrics(yt, proba, thr=0.5):
yp = (proba >= thr).astype(int)
tp = int(((yt == 1) & (yp == 1)).sum())
fp = int(((yt == 0) & (yp == 1)).sum())
fn = int(((yt == 1) & (yp == 0)).sum())
from sklearn.metrics import roc_auc_score, average_precision_score
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
return {
"n": int(len(yt)), "tp": tp, "fp": fp, "fn": fn,
"P": tp / max(tp + fp, 1), "R": tp / max(tp + fn, 1),
"FA": fp, "FA_rate": fp / max(len(yt), 1),
"ROC_AUC": float(roc), "PR_AUC": float(pr),
}
def threshold_curve(yt, proba, thr_grid):
rows = []
for thr in thr_grid:
yp = (proba >= thr).astype(int)
tp = int(((yt == 1) & (yp == 1)).sum())
fp = int(((yt == 0) & (yp == 1)).sum())
fn = int(((yt == 1) & (yp == 0)).sum())
P = tp / max(tp + fp, 1)
R = tp / max(tp + fn, 1)
rows.append({
"thr": round(float(thr), 4), "P": round(P, 4), "R": round(R, 4),
"FA": fp, "FA_rate": round(fp / max(len(yt), 1), 4),
})
return rows
def operating_points(rows, pair_proba, pairs):
"""Report P>=0.95/0.98/0.99 points with pair separation at threshold."""
res = {}
for target in (0.95, 0.98, 0.99):
pts = [r for r in rows if r["P"] >= target and r["R"] > 0.0]
if not pts:
res[str(target)] = None
continue
best = max(pts, key=lambda r: r["R"])
# pair separation at that operating point
sep = pair_sep_at(best["thr"], pair_proba, pairs)
best = dict(best); best["pair_sep"] = round(sep, 4)
res[str(target)] = best
return res
def pair_sep_at(thr, pair_proba, pairs):
"""Fraction of pairs where action>=thr and cap<thr."""
if not pairs:
return 0.0
ok = 0
for cidx, aidx in pairs:
if pair_proba[aidx] >= thr and pair_proba[cidx] < thr:
ok += 1
return ok / len(pairs)
def fold_variance(fold_rows):
return {
"folds": [
{
"fold": fr["fold"],
"ROC_AUC": fr["ROC_AUC"], "PR_AUC": fr["PR_AUC"],
"P": fr["P"], "R": fr["R"], "FA": fr["FA"], "n": fr["n"],
}
for fr in fold_rows
]
}
# ─── subcommands ────────────────────────────────────────────────────────────
def _result_path():
os.makedirs(RESULTS_DIR, exist_ok=True)
return RESULTS_DIR
def cmd_grouped(args):
rows = load_dev()
char_vocab, bpe = build_tokenizers(rows)
y = np.array([r["y"] for r in rows])
folds = np.array([r["cv_fold"] for r in rows])
X_char = encode_all(rows, char_vocab, "char")
X_bpe = encode_all(rows, bpe, "bpe")
tokenizers = {"char": char_vocab, "bpe": bpe}
os.makedirs(RESULTS_DIR, exist_ok=True)
# save tokenizer metadata for reproducibility
meta = {
"char_vocab": char_vocab.size,
"char_vocab_sample": char_vocab.id_to_char[:50],
"bpe_vocab": bpe.size,
"bpe_serialized_bytes": bpe.serialized_bytes(),
"max_char": MAX_CHAR, "max_bpe": MAX_BPE,
"n": len(rows),
}
with open(os.path.join(RESULTS_DIR, "corpus_meta.json"), "w") as f:
json.dump(meta, f)
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
for arch, sizes in ARCH_CONFIGS.items():
kind = kind_of[arch]
X = X_char if kind == "char" else X_bpe
for size in sizes:
name = f"{arch}_{size}"
os.makedirs(os.path.join(RESULTS_DIR, "models"), exist_ok=True)
probs = {v: np.zeros(len(rows)) for v in VARIANTS}
fold_rows = []
for fold in range(5):
tr = folds != fold
te = folds == fold
Xtr = X["strip"][tr]
model, secs = train_binary(Xtr, y[tr], arch, size,
vocab_size=tokenizers[kind].size,
seed_offset=fold, log=args.verbose)
torch_models_dir = os.path.join(RESULTS_DIR, "models")
import torch
torch.save(model.state_dict(), os.path.join(torch_models_dir, f"{name}_fold{fold}.pt"))
for v in VARIANTS:
probs[v][te] = predict_proba(model, X[v][te])
fold_m = binary_metrics(y[te], probs["strip"][te])
fold_m["fold"] = fold
fold_rows.append(fold_m)
print(f" {name} fold {fold}: ROC={fold_m['ROC_AUC']:.3f} "
f"PR={fold_m['PR_AUC']:.3f} P={fold_m['P']:.3f} R={fold_m['R']:.3f} "
f"FA={fold_m['FA']} n={fold_m['n']} ({secs:.1f}s)")
np.savez(os.path.join(RESULTS_DIR, f"{name}_probs.npz"),
var_orig=probs["orig"], var_nofinal=probs["nofinal"],
var_strip=probs["strip"])
summary = binary_metrics(y, probs["strip"])
print(f" {name} OOF: ROC={summary['ROC_AUC']:.3f} PR={summary['PR_AUC']:.3f} "
f"P={summary['P']:.3f} R={summary['R']:.3f} FA={summary['FA']}")
print("grouped done")
def cmd_metrics(args):
rows = load_dev()
y = np.array([r["y"] for r in rows])
tags = [r["tags"] for r in rows]
pairs = build_pairs(rows, [r["text_strip"] for r in rows])
print(f"pairs={len(pairs)}")
out = {}
for arch, sizes in ARCH_CONFIGS.items():
for size in sizes:
name = f"{arch}_{size}"
fp = os.path.join(RESULTS_DIR, f"{name}_probs.npz")
if not os.path.exists(fp):
continue
z = np.load(fp)
entry = {"name": name, "arch": arch, "size": size}
# OOF binary on strip variant (primary training input)
entry["strip"] = binary_metrics(y, z["var_strip"])
entry["pairs"] = {}
entry["pairs"]["strip"] = pair_metrics(pairs, z["var_strip"])
entry["pairs"]["orig"] = pair_metrics(pairs, z["var_orig"])
entry["pairs"]["nofinal"] = pair_metrics(pairs, z["var_nofinal"])
# stress: same OOF models, per-variant metrics
entry["stress"] = {}
for v in VARIANTS:
p = z[f"var_{v}"]
entry["stress"][v] = {
"all_FA": binary_metrics(y, p)["FA_rate"],
"capQ_FA": capq_fa(tags, y, p),
"modal_recall": modal_recall(tags, y, p),
}
# threshold curve + operating points on strip
entry["curve"] = threshold_curve(y, z["var_strip"],
np.arange(0.30, 1.0, 0.02))
entry["ops"] = operating_points(entry["curve"], z["var_strip"], pairs)
out[name] = entry
with open(os.path.join(RESULTS_DIR, "metrics.json"), "w") as f:
json.dump(out, f, indent=2, default=str)
print(json.dumps(out, indent=2, default=str))
def capq_fa(tags, y, proba):
mask = np.array(["capability_question" in t for t in tags])
if mask.sum() == 0:
return 0.0
sub = proba[mask]
return float((sub >= 0.5).sum() / mask.sum())
def modal_recall(tags, y, proba):
mask = np.array([
(y[i] == 1 and ("polite_request" in tags[i] or "modal_request" in tags[i]))
for i in range(len(y))
])
if mask.sum() == 0:
return 0.0
sub = proba[mask]
return float((sub >= 0.5).sum() / mask.sum())
def cmd_lfo(args):
rows = load_dev()
char_vocab, bpe = build_tokenizers(rows)
y = np.array([r["y"] for r in rows])
tags = [r["tags"] for r in rows]
X_char = encode_all(rows, char_vocab, "char")["strip"]
X_bpe = encode_all(rows, bpe, "bpe")["strip"]
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
results = {}
tokenizers = {"char": char_vocab, "bpe": bpe}
# capability_question LOFO for every config/size (the critical case)
for arch, sizes in ARCH_CONFIGS.items():
X = X_char if kind_of[arch] == "char" else X_bpe
src_idx = np.array(["capability_question" not in t for t in tags])
tgt_idx = np.array(["capability_question" in t for t in tags])
for size in sizes:
model, _ = train_binary(X[src_idx], y[src_idx], arch, size,
vocab_size=tokenizers[kind_of[arch]].size,
seed_offset=17)
p = predict_proba(model, X[tgt_idx])
m = binary_metrics(y[tgt_idx], p)
results[f"{arch}_{size}:capability_question"] = m
print(f"LFO ability {arch}_{size}: cap rows={m['n']} "
f"pos={int(y[tgt_idx].sum())} FA={m['FA']} FA_rate={m['FA_rate']:.3f} "
f"P={m['P']:.3f} R={m['R']:.3f} acc={1-m['FA_rate']:.3f}")
# full family LOFO for the leading config per architecture
leading = {"char_cnn": "char_cnn_medium", "bigru": "bigru_tiny",
"tiny_transformer": "tiny_transformer_small"}
for arch, name in leading.items():
X = X_char if kind_of[arch] == "char" else X_bpe
for fam in PRESENT_FAMILIES:
src = np.array([fam not in t for t in tags])
tgt = np.array([fam in t for t in tags])
model, _ = train_binary(X[src], y[src], arch, name.split("_")[-1],
vocab_size=tokenizers[kind_of[arch]].size,
seed_offset=41)
p = predict_proba(model, X[tgt])
m = binary_metrics(y[tgt], p)
results[f"{name}:{fam}"] = m
print(f"LFO {fam}: {name} rows={m['n']} pos={int(y[tgt].sum())} "
f"P={m['P']:.3f} R={m['R']:.3f} FA={m['FA']} acc={1-m['FA_rate']:.3f}")
with open(os.path.join(RESULTS_DIR, "lfo.json"), "w") as f:
json.dump(results, f, indent=2, default=str)
print("lfo done")
def cmd_e5baseline(args):
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
rows = load_dev()
y = np.array([r["y"] for r in rows])
folds = np.array([r["cv_fold"] for r in rows])
X = np.vstack([r["emb"] for r in rows])
tags = [r["tags"] for r in rows]
pairs = build_pairs(rows, [r["text_strip"] for r in rows])
out = {}
for model_name, model, extra in [
("e5_linear", LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42), {}),
("e5_mlp_h32", MLPClassifier(hidden_layer_sizes=(32,), activation="relu",
solver="adam", alpha=0.01, random_state=42,
max_iter=800, early_stopping=True,
validation_fraction=0.15, n_iter_no_change=10), {}),
]:
proba = np.zeros(len(rows))
for fold in range(5):
tr = folds != fold
te = folds == fold
m2 = type(model)(**{k: v for k, v in model.get_params().items()})
m2.fit(X[tr], y[tr])
proba[te] = m2.predict_proba(X[te])[:, 1]
entry = {
"grouped": binary_metrics(y, proba),
"pairs": pair_metrics(pairs, proba),
}
# cap-Q leave-generator-out (train without the family)
src = np.array(["capability_question" not in t for t in tags])
tgt = np.array(["capability_question" in t for t in tags])
m3 = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42) \
if model_name == "e5_linear" else \
MLPClassifier(hidden_layer_sizes=(32,), alpha=0.01, random_state=42, max_iter=800)
m3.fit(X[src], y[src])
p3 = m3.predict_proba(X[tgt])[:, 1]
entry["capq_lofo"] = binary_metrics(y[tgt], p3)
entry["stress"] = "NA (no re-embed on this box)"
out[model_name] = entry
print(f"{model_name}: grouped FA_rate={entry['grouped']['FA_rate']:.4f} "
f"PR={entry['grouped']['PR_AUC']:.3f} capQ_LOFO_FA_rate={entry['capq_lofo']['FA_rate']:.4f} "
f"pairs={entry['pairs']['ordering_acc']:.3f}")
with open(os.path.join(RESULTS_DIR, "e5baseline.json"), "w") as f:
json.dump(out, f, indent=2, default=str)
print("e5baseline done")
def cmd_runtime(args):
import torch
rows = load_dev()
char_vocab, bpe = build_tokenizers(rows)
X_char = encode_all(rows, char_vocab, "char")["strip"]
X_bpe = encode_all(rows, bpe, "bpe")["strip"]
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
report = {}
for arch, sizes in ARCH_CONFIGS.items():
X = X_char if kind_of[arch] == "char" else X_bpe
for size in sizes:
name = f"{arch}_{size}"
model = build_model(arch, size, int(X.max()) + 1, X.shape[1])
n_params = sum(p.numel() for p in model.parameters())
fp32 = n_params * 4
int8 = n_params
model.eval()
# warmup + latency (batch-1, eval mode)
xb = torch.from_numpy(X[:1])
with torch.no_grad():
for _ in range(20):
model(xb)
# tokenization latency
if kind_of[arch] == "char":
t0 = time.perf_counter()
for r in rows[:1000]:
char_vocab.encode(r["text_strip"], MAX_CHAR)
tl = (time.perf_counter() - t0) / 1000
else:
t0 = time.perf_counter()
for r in rows[:1000]:
bpe.encode(r["text_strip"])
tl = (time.perf_counter() - t0) / 1000
lat = []
for _ in range(300):
t0 = time.perf_counter()
model(xb)
lat.append(time.perf_counter() - t0)
lat = np.array(lat) * 1e6
report[name] = {
"params": n_params, "fp32_bytes": fp32, "int8_bytes": int8,
"latency_us_mean": float(lat.mean()), "latency_us_p50": float(np.median(lat)),
"latency_us_p95": float(np.percentile(lat, 95)),
"throughput_b1": round(1e6 / float(lat.mean()), 1),
"tok_us": round(tl * 1e6, 1),
"tokenizer": "char" if kind_of[arch] == "char" else "bpe",
}
print(f"{name}: {n_params} params fp32={fp32/1024:.0f}KiB "
f"lat={lat.mean():.0f}us tok={tl*1e6:.1f}us")
with open(os.path.join(RESULTS_DIR, "runtime.json"), "w") as f:
json.dump(report, f, indent=2)
print("runtime done")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("cmd", choices=["grouped", "metrics", "lfo", "e5baseline", "runtime"])
ap.add_argument("--verbose", action="store_true")
args = ap.parse_args()
t0 = time.time()
globals()[f"cmd_{args.cmd}"](args)
print(f"elapsed {time.time()-t0:.1f}s")
if __name__ == "__main__":
main()
@@ -0,0 +1,158 @@
#!/usr/bin/env python3
"""
Slice 19 model zoo: three genuinely sequence-sensitive tiny models, trained
from scratch on Maven's narrow binary pragmatics task.
A. CharCNN codepoint ids char embedding parallel small 1D convs
(several kernel widths) global max-pool linear head
B. BiGRU subword ids token embedding 1-layer BiGRU
maxpool[final] linear head
C. TinyTransformer subword ids token embedding + sine position
N self-attention encoder blocks (heads, FFN 4x, PreNorm)
CLS linear head
All expose :forward(ids) returning the binary logit, plus .n_params().
Deterministic: everything is plain torch ops.
"""
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
class CharCNN(nn.Module):
def __init__(self, vocab_size, embed_dim, filters, widths, pad_idx=0, dropout=0.3):
super().__init__()
self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
self.convs = nn.ModuleList([
nn.Conv1d(embed_dim, filters, k, padding=(k - 1) // 2)
for k in widths
])
self.dropout = nn.Dropout(dropout)
self.head = nn.Linear(filters * len(widths), 1)
def forward(self, ids):
# ids: (B, T)
x = self.embed(ids).transpose(1, 2) # (B, D, T)
hiddens = [F.relu(conv(x)) for conv in self.convs] # each (B, F, T)
pooled = torch.cat([h.max(dim=2).values for h in hiddens], dim=1) # (B, F*W)
return self.head(self.dropout(pooled)).squeeze(-1)
def n_params(self):
return sum(p.numel() for p in self.parameters())
class BiGRU(nn.Module):
def __init__(self, vocab_size, embed_dim, hidden, pad_idx=0, dropout=0.3):
super().__init__()
self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
self.encoder = nn.GRU(embed_dim, hidden, num_layers=1, bidirectional=True,
batch_first=True)
self.dropout = nn.Dropout(dropout)
self.head = nn.Linear(hidden * 2, 1)
def forward(self, ids):
mask = (ids != 0).float() # (B, T)
x = self.embed(ids)
lens = mask.sum(dim=1).clamp(min=1).long()
x_p = nn.utils.rnn.pack_padded_sequence(x, lens.cpu(), batch_first=True,
enforce_sorted=False)
out, _ = self.encoder(x_p)
out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True,
total_length=mask.size(1))
out = out * mask.unsqueeze(-1)
maxed = out.max(dim=1).values # (B, 2H)
return self.head(self.dropout(maxed)).squeeze(-1)
def n_params(self):
return sum(p.numel() for p in self.parameters())
class TinyTransformer(nn.Module):
def __init__(self, vocab_size, d_model, n_layers, n_heads, ff_mult=4,
max_len=64, pad_idx=0, dropout=0.1):
super().__init__()
self.d_model = d_model
self.embed = nn.Embedding(vocab_size, d_model, padding_idx=pad_idx)
self.dropout = nn.Dropout(dropout)
self.pos = nn.Parameter(torch.empty(1, max_len, d_model))
nn.init.normal_(self.pos, std=0.02)
blocks = []
for _ in range(n_layers):
blocks.append(TransformerBlock(d_model, n_heads, ff_mult, dropout))
self.blocks = nn.ModuleList(blocks)
self.ln_out = nn.LayerNorm(d_model)
self.head = nn.Linear(d_model, 1)
def forward(self, ids):
B, T = ids.shape
mask = (ids != 0)
x = self.embed(ids) * math.sqrt(self.d_model) + self.pos[:, :T, :]
x = self.dropout(x)
for blk in self.blocks:
x = blk(x, mask)
x = self.ln_out(x)
pooled = x.masked_fill(~mask.unsqueeze(-1), float("-inf")).max(dim=1).values
return self.head(pooled).squeeze(-1)
def n_params(self):
return sum(p.numel() for p in self.parameters())
class TransformerBlock(nn.Module):
def __init__(self, d_model, n_heads, ff_mult, dropout):
super().__init__()
self.attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout,
batch_first=True)
self.ln1 = nn.LayerNorm(d_model)
self.ff = nn.Sequential(
nn.Linear(d_model, d_model * ff_mult),
nn.GELU(),
nn.Linear(d_model * ff_mult, d_model),
)
self.ln2 = nn.LayerNorm(d_model)
self.dropout = nn.Dropout(dropout)
def forward(self, x, mask):
# mask: (B, T) bool of non-pad; MultiheadAttention uses key_padding_mask
h = self.ln1(x)
h = self.attn(h, h, h, key_padding_mask=~mask,
need_weights=False, is_causal=False)[0]
x = x + self.dropout(h)
h = self.ln2(x)
x = x + self.dropout(self.ff(h))
return x
# ─── Sizes ladder ───────────────────────────────────────────────────────────
def make_model(arch, size, char_vocab, bpe_vocab):
if arch == "char_cnn":
configs = {
"tiny": dict(embed_dim=32, filters=64, widths=[3, 4, 5]),
"medium": dict(embed_dim=64, filters=160, widths=[2, 3, 4, 5]),
}
c = configs[size]
return CharCNN(char_vocab, c["embed_dim"], c["filters"], c["widths"])
if arch == "bigru":
configs = {
"tiny": dict(embed_dim=64, hidden=64),
"medium": dict(embed_dim=128, hidden=128),
"large": dict(embed_dim=256, hidden=256),
}
c = configs[size]
return BiGRU(bpe_vocab, c["embed_dim"], c["hidden"])
if arch == "tiny_transformer":
configs = {
"small": dict(d_model=128, n_layers=2, n_heads=4),
"medium": dict(d_model=192, n_layers=4, n_heads=4),
}
c = configs[size]
return TinyTransformer(bpe_vocab, c["d_model"], c["n_layers"], c["n_heads"])
raise ValueError(arch)
def n_params_of(arch, size, char_vocab, bpe_vocab):
return make_model(arch, size, char_vocab, bpe_vocab).n_params()
@@ -0,0 +1,152 @@
#!/usr/bin/env python3
"""
Slice 20 tokenizer audit (§2 of the brief) and corpus sequence-length
statistics (§3). Runs before any training. If the audit shows catastrophic
Cyrillic / mixed-identifier loss it is the gate to stop.
Loads the frozen dev corpus exactly like slice 19 (same normalization),
so regime A (natural text) is `text_orig` and regime B (punct-stripped) is
`text_strip`.
"""
import json
import os
import re
import statistics
from transformers import AutoTokenizer
import slice19_main as s19
RESULTS_DIR = "/tmp/mvn-s20"
MODEL_DIR = ("/tmp/mvn-s20/hf-cache/models--cointegrated--rubert-tiny/"
"snapshots/5441c5ea8026d4f6d7505ec004845409f1259fb1")
MIXED_CYRILLIC_LATIN = re.compile(r"[а-яёА-ЯЁ]+[a-zA-Z]+|[a-zA-Z]+[а-яёА-ЯЁ]+")
HAS_CYRILLIC = re.compile(r"[а-яёА-ЯЁ]")
HAS_LATIN = re.compile(r"[a-zA-Z]")
NUMERIC = re.compile(r"[0-9]")
TOKEN_RE = re.compile(r"[^\W\d_]+", re.UNICODE)
SAMPLES = [
"выключи свет в спальне пожалуйста",
"turn off the lights",
"перезапусти сервис mavend",
"что такое Nexus",
"включи телевизор, пожалуйста",
"как дела у Мэйвен",
"поставь таймер на 5 минут",
"кто такой Home Assistant",
"открой настройки устройства ha_cam_12",
"Покажи статус сервера Proxmox",
"аутентифицируй на сайте 2fa.ru",
"сообщи погоду завтра в 18:30",
]
def is_toolish(word):
# Maven sibling service names / HA-like identifiers: mixed case, digits,
# underscores, or short Latin words that are not in the vocab as whole
# words. Rough heuristic for the fragmentation probe.
return bool(re.search(r"[A-Z0-9_/.-]", word))
def main():
os.makedirs(RESULTS_DIR, exist_ok=True)
tok = AutoTokenizer.from_pretrained(MODEL_DIR)
rows = s19.load_dev()
texts = {
"orig": [r["text_orig"] for r in rows],
"nofinal": [r["text_nofinal"] for r in rows],
"strip": [r["text_strip"] for r in rows],
}
vv = tok.vocab_size
unk = tok.unk_token_id
rep = {"model": "cointegrated/rubert-tiny",
"sha": "5441c5ea8026d4f6d7505ec004845409f1259fb1",
"tokenizer": type(tok).__name__,
"vocab_size": vv}
# per-character stats (natural texts)
chars = [len(t) for t in texts["orig"]]
rep["char_len"] = {
"mean": round(statistics.mean(chars), 2),
"p50": int(sorted(chars)[len(chars) // 2]),
"p95": sorted(chars)[int(len(chars) * .95)],
"p99": sorted(chars)[int(len(chars) * .99)],
"max": max(chars),
}
for view in ("orig", "strip"):
ids = tok(texts[view], add_special_tokens=True, padding=False,
truncation=False)["input_ids"]
lens = [len(x) for x in ids]
n_tok = sum(lens)
n_unk = sum(x.count(unk) for x in ids)
n_chars = sum(len(t) for t in texts[view])
rep[view] = {
"tokens_per_utt_mean": round(n_tok / len(rows), 2),
"tokens_per_char": round(n_tok / max(n_chars, 1), 4),
"unk_count": n_unk,
"unk_rate": round(n_unk / max(n_tok, 1), 5),
"seq_len_p50": int(sorted(lens)[len(lens) // 2]),
"seq_len_p90": sorted(lens)[int(len(lens) * .90)],
"seq_len_p95": sorted(lens)[int(len(lens) * .95)],
"seq_len_p99": sorted(lens)[int(len(lens) * .99)],
"seq_len_max": max(lens),
"above_96": sum(1 for x in lens if x > 96),
"above_128": sum(1 for x in lens if x > 128),
}
rep[view]["p99_plus_margin"] = rep[view]["seq_len_p99"] + 6
# mixed Cyrillic/Latin behaviour over natural texts
mixed_words = []
for t in texts["orig"]:
for w in t.split():
if MIXED_CYRILLIC_LATIN.search(w):
mixed_words.append(w)
rep["mixed_cyr_lat_rows"] = len({w for w in mixed_words})
rep["mixed_cyr_lat_stats"] = {"word_count": len(mixed_words),
"unique_words": len(set(mixed_words))}
# entity/tool-name fragmentation: unique word-like tokens containing a digit
# or underscore, or non-trivial Latin, and how many BPE/WordPiece pieces they
# split into. Sample the extremes.
fragments = []
vocab = set(tok.get_vocab().keys())
for t in texts["orig"]:
# split into "clean" tokens (word chars + _ / digit boundaries)
for w in re.findall(r"[A-Za-z0-9_]+\b", t):
w2 = re.sub(r"_\b", "", w)
if len(w2) < 3 or not is_toolish(w2):
continue
n_pieces = len(tok.tokenize(w2).replace("##", "_").rstrip())
fragments.append((n_pieces, w2))
frag = sorted(set(fragments))[-30:]
rep["entity_fragment_examples"] = [
{"token": w, "pieces": n} for n, w in frag
]
# representative samples: full tokenization
rep["samples"] = []
for s in SAMPLES:
e = tok(s, add_special_tokens=True, padding=False, truncation=False)
rep["samples"].append({
"text": s,
"tokens": tok.convert_ids_to_tokens(e["input_ids"]),
"pieces": len(e["input_ids"]),
"unk": e["input_ids"].count(unk),
})
with open(os.path.join(RESULTS_DIR, "tokenizer_audit.json"), "w") as f:
json.dump(rep, f, indent=2, ensure_ascii=False)
print(json.dumps({k: v for k, v in rep.items() if k not in ("samples",)}, indent=2, ensure_ascii=False))
print("\n--- samples ---")
for s in rep["samples"]:
print(f'{s["pieces"]:>3} unk={s["unk"]} {s["text"]:50} ->'
f' {" ".join(s["tokens"])}')
if __name__ == "__main__":
main()
@@ -0,0 +1,517 @@
#!/usr/bin/env python3
"""
Slice 20: fine-tune cointegrated/rubert-tiny (11.9M, 3-layer BERT) end-to-end
for the binary executable-intent boundary (action vs not_action) on the frozen
v2 dev corpus, following the slice 20 brief.
Rules honoured:
- full end-to-end fine-tuning, CLS-pooled native classification head
- tokenizer used unchanged (audit in slice20_audit.py)
- max length from corpus stats (p99+margin, cap 128): 25 here
- narrow search: LR in {1e-5, 2e-5, 5e-5}, <= 6 epochs, early stop on a
development (within-fold) split, best checkpoint restored
- >= 3 seeds (42/17/7) for every config
- grouped 5-fold CV reuse; cap-Q leave-generator-out as primary stress case
- two input regimes: A = natural text (orig), B = punctuation-stripped (strip)
Artifacts under /tmp/mvn-s20/:
pre/{regime}_ids.npy, _attn.npy tokenized corpus (all three views)
oof/{regime}_{lr}_{seed}_probs.npz OOF probs per view (var_orig/nofinal/strip)
oof/{regime}_{lr}_{seed}_metrics.json
lfo/{regime}_{lr}_{seed}.json capability-Q LOFO (held-out family)
results/summary.json
models/{regime}_{lr}_{seed}_fold{i}.pt, lfo_{seed}.pt
CLI: slice20_pretrained.py {pre, grouped, lfo, metrics, runtime, onnx}
"""
import argparse
import importlib.util
import json
import os
import sys
import time
import numpy as np
import torch
from torch import nn
from transformers import AutoConfig, AutoTokenizer
from transformers import BertForSequenceClassification
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import slice19_main as s19
RESULTS_DIR = "/tmp/mvn-s20"
MODEL_DIR = ("/tmp/mvn-s20/hf-cache/models--cointegrated--rubert-tiny/"
"snapshots/5441c5ea8026d4f6d7505ec004845409f1259fb1")
MODEL_NAME = "cointegrated/rubert-tiny"
MODEL_SHA = "5441c5ea8026d4f6d7505ec004845409f1259fb1"
REGIMES = ["A", "B"]
LRS = [1e-5, 2e-5, 5e-5]
SEEDS = [42, 17, 7]
VIEWS = ["orig", "nofinal", "strip"]
MAX_LEN = 25
BATCH = 32
MAX_EPOCHS = 4
EARLY_STOP = 1 # patience in epochs on val PR-AUC
VAL_FRACTION = 0.12
WEIGHT_DECAY = 0.01
torch.set_num_threads(4)
# ─── tokenizer / input preparation ──────────────────────────────────────────
def _load_tokenizer():
return AutoTokenizer.from_pretrained(MODEL_DIR)
def tokenize(texts, tok):
e = tok(list(texts), add_special_tokens=True, padding="max_length",
truncation=True, max_length=MAX_LEN)
return np.array(e["input_ids"], np.int64), np.array(e["attention_mask"], np.int64)
def cmd_pre(args):
os.makedirs(os.path.join(RESULTS_DIR, "pre"), exist_ok=True)
tok = _load_tokenizer()
rows = s19.load_dev()
# distributed over all variants, all rows, both regimes
for regime in REGIMES:
train_view = "orig" if regime == "A" else "strip"
tsrc = [r[f"text_{train_view}"] for r in rows]
ids, attn = tokenize(tsrc, tok)
np.save(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"), ids)
np.save(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"), attn)
# eval stress views tokenized under the same regime's vocab/format
for v in VIEWS:
ids_v, attn_v = tokenize([r[f"text_{v}"] for r in rows], tok)
np.save(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_ids.npy"), ids_v)
np.save(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_attn.npy"), attn_v)
print(f"regime {regime} done, train view={train_view}")
def _make_model():
cfg = AutoConfig.from_pretrained(MODEL_DIR)
cfg.num_labels = 1 # sine logit, BCEWithLogits — matches slice 19 head
model = BertForSequenceClassification.from_pretrained(MODEL_DIR, config=cfg)
return model
# ─── training ───────────────────────────────────────────────────────────────
def _val_split(rows_idx, y, seed):
rng = np.random.RandomState(seed)
idx = rng.permutation(rows_idx)
# stratified: keep the y-ratio in both parts
pos = idx[y[idx] == 1]
neg = idx[y[idx] == 0]
nv_pos = max(1, int(round(len(pos) * VAL_FRACTION)))
nv_neg = max(1, int(round(len(neg) * VAL_FRACTION)))
v = np.concatenate([pos[:nv_pos], neg[:nv_neg]])
t = np.concatenate([pos[nv_pos:], neg[nv_neg:]])
return t, v
def train_model(id_arr, attn, y, tr_idx, val_idx, lr, seed, builder=None):
"""Fine-tune the full encoder; early-stop on val PR-AUC; return best state."""
net = (builder or _make_model)()
opt = torch.optim.AdamW([p for p in net.parameters()],
lr=lr, weight_decay=WEIGHT_DECAY)
lossf = nn.BCEWithLogitsLoss()
from sklearn.metrics import average_precision_score
tr = torch.from_numpy(np.ascontiguousarray(id_arr[tr_idx]))
ta = torch.from_numpy(np.ascontiguousarray(attn[tr_idx]))
ty = torch.from_numpy(y[tr_idx].astype(np.float32))
va = torch.from_numpy(np.ascontiguousarray(id_arr[val_idx]))
vaa = torch.from_numpy(np.ascontiguousarray(attn[val_idx]))
vy = y[val_idx]
best_pr = -1.0
best_state = None
best_epoch = 0
patience = 0
n = len(tr_idx)
rng = np.random.RandomState(seed * 97 % 2**31)
for epoch in range(MAX_EPOCHS):
net.train()
perm = rng.permutation(n)
running = 0.0
nb = 0
for st in range(0, n, BATCH):
bidx = torch.from_numpy(perm[st:st + BATCH])
logits = net(input_ids=tr[bidx], attention_mask=ta[bidx]).logits.squeeze(-1)
loss = lossf(logits, ty[bidx])
opt.zero_grad()
loss.backward()
opt.step()
running += float(loss)
nb += 1
net.eval()
with torch.no_grad():
pval = torch.sigmoid(net(input_ids=va, attention_mask=vaa).logits.squeeze(-1)).numpy()
if len(np.unique(vy)) > 1:
pr = average_precision_score(vy, pval)
else:
pr = 0.0
if pr > best_pr:
best_pr = pr
best_state = {k: v.detach().clone() for k, v in net.state_dict().items()}
best_epoch = epoch + 1
patience = 0
else:
patience += 1
if patience >= EARLY_STOP:
break
net.load_state_dict(best_state)
return net, best_epoch, best_pr, running / max(nb, 1)
def predict_proba(net, id_arr, attn, idx=None):
net.eval()
idx = np.arange(len(id_arr)) if idx is None else idx
out = []
with torch.no_grad():
for st in range(0, len(idx), BATCH * 4):
bi = idx[st:st + BATCH * 4]
iid = torch.from_numpy(np.ascontiguousarray(id_arr[bi]))
att = torch.from_numpy(np.ascontiguousarray(attn[bi]))
out.append(torch.sigmoid(net(input_ids=iid, attention_mask=att).logits.squeeze(-1)).numpy())
return np.concatenate(out)
# ─── grouped CV ─────────────────────────────────────────────────────────────
def cmd_grouped(args):
os.makedirs(os.path.join(RESULTS_DIR, "oof"), exist_ok=True)
os.makedirs(os.path.join(RESULTS_DIR, "models"), exist_ok=True)
rows = s19.load_dev()
y = np.array([r["y"] for r in rows])
folds = np.array([r["cv_fold"] for r in rows])
for regime in REGIMES:
ids = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"))
attn = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"))
ev = {v: (np.load(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_ids.npy")),
np.load(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_attn.npy")))
for v in VIEWS}
for lr in LRS:
for seed in SEEDS:
probs = {v: np.zeros(len(rows)) for v in VIEWS}
fold_records = []
for fold in range(5):
tr = np.where(folds != fold)[0]
te = np.where(folds == fold)[0]
t_idx, v_idx = _val_split(tr, y, seed + 100 * fold)
net, ep, best_pr, _ = train_model(ids, attn, y, t_idx, v_idx, lr, seed + fold)
torch.save(net.state_dict(),
os.path.join(RESULTS_DIR, "models",
f"{regime}_{lr}_{seed}_fold{fold}.pt"))
for v in VIEWS:
probs[v][te] = predict_proba(net, *ev[v], te)
fold_records.append({"fold": fold, "epochs": ep, "val_pr": best_pr})
np.savez(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_probs.npz"),
var_orig=probs["orig"], var_nofinal=probs["nofinal"],
var_strip=probs["strip"])
with open(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_metrics.json"), "w") as f:
json.dump({"folds": fold_records}, f, indent=2)
m = s19.binary_metrics(y, probs["strip"])
print(f"[{regime}] lr={lr:.0e} seed={seed} "
f"PR={m['PR_AUC']:.3f} P={m['P']:.3f} R={m['R']:.3f} "
f"FA={m['FA']} epochs={[fr['epochs'] for fr in fold_records]}",
flush=True)
print("grouped done")
# ─── cap-Q leave-generator-out ──────────────────────────────────────────────
def cmd_lfo(args):
os.makedirs(os.path.join(RESULTS_DIR, "lfo"), exist_ok=True)
rows = s19.load_dev()
y = np.array([r["y"] for r in rows])
tags = [r["tags"] for r in rows]
src = np.array(["capability_question" not in t for t in tags])
tgt = ~src
for regime in REGIMES:
ids = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"))
attn = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"))
for lr in LRS:
for seed in SEEDS:
s_idx = np.where(src)[0]
t_idx, v_idx = _val_split(s_idx, y, seed + 7)
net, ep, best_pr, _ = train_model(ids, attn, y, t_idx, v_idx, lr, seed)
p = predict_proba(net, ids, attn, np.where(tgt)[0])
yt = y[tgt]
out = {
"regime": regime, "lr": lr, "seed": seed,
"rows": int(tgt.sum()), "epochs": ep, "val_pr": best_pr,
"mean_action_proba": float(np.mean(p)),
"max_action_proba": float(np.max(p)),
"acc": float(((p >= 0.5) == (yt == 1)).mean()),
"FA": int(((p >= 0.5) & (yt == 0)).sum()),
"FA_rate": float(((p >= 0.5) & (yt == 0)).mean()),
}
with open(os.path.join(RESULTS_DIR, "lfo", f"{regime}_{lr}_{seed}.json"), "w") as f:
json.dump(out, f, indent=2)
print(f"[{regime}] lr={lr:.0e} seed={seed} capQ LOFO "
f"acc={out['acc']:.3f} FA_rate={out['FA_rate']:.3f} "
f"mean_p={out['mean_action_proba']:.3f} epochs={ep}", flush=True)
print("lfo done")
# ─── metrics aggregation ────────────────────────────────────────────────────
def cmd_metrics(args):
rows = s19.load_dev()
y = np.array([r["y"] for r in rows])
tags = [r["tags"] for r in rows]
pairs = s19.build_pairs(rows, [r["text_strip"] for r in rows])
summary = {}
for regime in REGIMES:
summary[regime] = {}
for lr in LRS:
per_seed = []
for seed in SEEDS:
z = np.load(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_probs.npz"))
entry = {"seed": seed,
"views": {v: {"FA_rate": s19.binary_metrics(y, z[f"var_{v}"])["FA_rate"],
"PR": s19.binary_metrics(y, z[f"var_{v}"])["PR_AUC"],
"capQ_FA": s19.capq_fa(tags, y, z[f"var_{v}"])}
for v in VIEWS},
"strip": s19.binary_metrics(y, z["var_strip"]),
"pairs": {v: s19.pair_metrics(pairs, z[f"var_{v}"]) for v in VIEWS},
"curve": s19.threshold_curve(y, z["var_strip"], np.arange(0.30, 1.0, 0.02)),
"ops": s19.operating_points(s19.threshold_curve(
y, z["var_strip"], np.arange(0.30, 1.0, 0.02)),
z["var_strip"], pairs),
}
with open(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_metrics.json")) as f:
entry["folds"] = json.load(f)["folds"]
per_seed.append(entry)
# LOFO
lfos = []
for seed in SEEDS:
with open(os.path.join(RESULTS_DIR, "lfo", f"{regime}_{lr}_{seed}.json")) as f:
lfos.append(json.load(f))
summary[regime][str(lr)] = {"seeds": per_seed, "lfo": lfos}
# primary selection: min mean cap-Q LOFO FA_rate across seeds
sel = {}
for regime in REGIMES:
best_lr, best_mean = None, 9e9
for lr in LRS:
mean_fa = np.mean([l["FA_rate"] for l in summary[regime][str(lr)]["lfo"]])
sel[f"{regime}_{lr}"] = round(float(mean_fa), 4)
if mean_fa < best_mean:
best_lr, best_mean = lr, mean_fa
sel[f"{regime}_selected"] = str(best_lr)
summary["selection"] = sel
os.makedirs(os.path.join(RESULTS_DIR, "results"), exist_ok=True)
with open(os.path.join(RESULTS_DIR, "results", "summary.json"), "w") as f:
json.dump(summary, f, indent=2, default=str)
print(json.dumps(sel))
print("metrics done")
def _tokenizer_bytes():
return sum(os.path.getsize(os.path.join(MODEL_DIR, f))
for f in ["vocab.txt", "tokenizer.json"]
if os.path.exists(os.path.join(MODEL_DIR, f)))
# ─── runtime + ONNX ─────────────────────────────────────────────────────────
def cmd_runtime(args):
import time
net = _make_model()
tok = _load_tokenizer()
rows = s19.load_dev()
x = [r["text_orig"] for r in rows][:200]
e = tok(x, padding="max_length", truncation=True,
max_length=MAX_LEN, return_tensors="pt")
ids, attn = e["input_ids"], e["attention_mask"]
net.eval()
with torch.no_grad():
# warmup
for _ in range(3):
net(input_ids=ids[:1], attention_mask=attn[:1])
# batch-1 latency
lat = []
for i in range(200):
t0 = time.perf_counter()
net(input_ids=ids[i:i + 1], attention_mask=attn[i:i + 1])
lat.append((time.perf_counter() - t0) * 1e6)
# tokenization latency
t0 = time.perf_counter()
for i in range(200):
tok(x[i])
tok_us = (time.perf_counter() - t0) / 200 * 1e6
n_params = sum(p.numel() for p in net.parameters())
fp32 = n_params * 4
rep = {
"model": MODEL_NAME, "sha": MODEL_SHA,
"params": n_params, "fp32_bytes": fp32,
"fp16_bytes": fp32 // 2, "int8_bytes": n_params,
"tokenizer_bytes": _tokenizer_bytes(),
"latency_us_mean": float(np.mean(lat)),
"latency_us_p50": float(np.median(lat)),
"latency_us_p95": float(np.percentile(lat, 95)),
"max_len": MAX_LEN,
"tok_us": round(tok_us, 2),
"num_threads": 12,
}
with open(os.path.join(RESULTS_DIR, "runtime.json"), "w") as f:
json.dump(rep, f, indent=2)
print(json.dumps(rep, indent=2))
print("runtime done")
def cmd_onnx(args):
net = _make_model()
net.eval()
tok = _load_tokenizer()
rows = s19.load_dev()
try:
import torch.onnx
dummy = {
"input_ids": torch.zeros(1, MAX_LEN, dtype=torch.long),
"attention_mask": torch.ones(1, MAX_LEN, dtype=torch.long),
}
with torch.no_grad():
torch.onnx.export(net, (dummy,), os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"),
input_names=["input_ids", "attention_mask"],
output_names=["logits"], opset_version=14,
dynamic_axes={"input_ids": {0: "batch"},
"attention_mask": {0: "batch"}})
# parity on a fixed sample
import numpy as np
samp = [(r["text_orig"], r["y"]) for r in rows[:200]]
e = tok([s[0] for s in samp], padding="max_length", truncation=True,
max_length=MAX_LEN, return_tensors="pt")
with torch.no_grad():
pt = torch.sigmoid(net(**e).logits.squeeze(-1)).numpy()
import onnxruntime as ort
so = ort.SessionOptions()
so.intra_op_num_threads = 12
sess = ort.InferenceSession(os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"),
sess_options=so, providers=["CPUExecutionProvider"])
on = sess.run(None, {"input_ids": e["input_ids"].numpy(),
"attention_mask": e["attention_mask"].numpy()})[0]
on = 1 / (1 + np.exp(-on).squeeze(-1))
mx = float(np.max(np.abs(pt - on)))
size = os.path.getsize(os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"))
rep = {"max_logit_diff": mx, "onnx_bytes": size,
"parity_n": len(samp), "provider": "CPUExecutionProvider"}
with open(os.path.join(RESULTS_DIR, "onnx.json"), "w") as f:
json.dump(rep, f, indent=2)
print(json.dumps(rep))
except Exception as ex:
print("onnx export/parity failed:", ex)
with open(os.path.join(RESULTS_DIR, "onnx.json"), "w") as f:
json.dump({"error": str(ex)}, f, indent=2)
print("onnx done")
# ─── optional capacity/pretraining ceiling (brief §13) ──────────────────────
# Trigger: tiny1 clearly improved over from-scratch on in-pool but missed the
# LOFO boundary. tiny2 is the same 3-layer 312-hidden BERT family; it tests
# whether a *newer, larger-vocab* pretraining of the same family generalises
# where tiny1 failed — disambiguating "this family is the wrong prior" from a
# one-off pretraining. It cannot test capacity (same depth/size).
MODEL2_DIR = ("/tmp/mvn-s20/hf-tiny2-cache/models--cointegrated--rubert-tiny2/"
"snapshots/e8ed3b0c8bbf4fb6984c3de043bf7d2f4e5969ae")
MODEL2_SHA = "e8ed3b0c8bbf4fb6984c3de043bf7d2f4e5969ae"
CEIL_RESULTS = os.path.join(RESULTS_DIR, "tiny2")
def cmd_ceiling(args):
import torch
os.makedirs(CEIL_RESULTS, exist_ok=True)
os.makedirs(os.path.join(CEIL_RESULTS, "models"), exist_ok=True)
tok = AutoTokenizer.from_pretrained(MODEL2_DIR)
rows = s19.load_dev()
y = np.array([r["y"] for r in rows])
tags = [r["tags"] for r in rows]
folds = np.array([r["cv_fold"] for r in rows])
pairs = s19.build_pairs(rows, [r["text_strip"] for r in rows])
# audit: is the tiny2 tokenizer sane on the corpus before anything else
n_unk = 0
n_tok = 0
lens = []
for r in rows:
e = tok(r["text_orig"])
n_unk += e["input_ids"].count(tok.unk_token_id)
n_tok += len(e["input_ids"])
lens.append(len(e["input_ids"]))
audit = {"vocab_size": tok.vocab_size,
"unk_count": int(n_unk),
"unk_rate": round(n_unk / max(n_tok, 1), 5),
"seq_len_p99": sorted(lens)[int(len(lens) * .99)],
"seq_len_max": max(lens)}
with open(os.path.join(CEIL_RESULTS, "audit.json"), "w") as f:
json.dump(audit, f, indent=2)
print("tiny2 audit:", audit)
# tokenize the corpus (regime A only — natural text, the in-pool best)
ids_a, attn_a = tokenize([r["text_orig"] for r in rows], tok)
ev = {v: tokenize([r[f"text_{v}"] for r in rows], tok) for v in VIEWS}
def make2():
cfg = AutoConfig.from_pretrained(MODEL2_DIR)
cfg.num_labels = 1
m = BertForSequenceClassification.from_pretrained(MODEL2_DIR, config=cfg)
return m
# cap-Q LOFO, 3 seeds, matching the A@2e-5 tiny1 config
lfors = []
src = np.where(np.array(["capability_question" not in t for t in tags]))[0]
tgt = np.where(np.array(["capability_question" in t for t in tags]))[0]
for seed in SEEDS:
t_idx, v_idx = _val_split(src, y, seed + 7)
net, ep, best_pr, _ = train_model(ids_a, attn_a, y, t_idx, v_idx, 2e-5, seed, builder=make2)
p = predict_proba(net, ids_a, attn_a, tgt)
yt = y[tgt]
lfors.append({"regime": "A(tiny2)", "lr": 2e-5, "seed": seed,
"rows": int(len(tgt)), "epochs": ep,
"mean_action_proba": float(np.mean(p)),
"acc": float(((p >= 0.5) == (yt == 1)).mean()),
"FA": int(((p >= 0.5) & (yt == 0)).sum()),
"FA_rate": float(((p >= 0.5) & (yt == 0)).mean())})
with open(os.path.join(CEIL_RESULTS, "lfo.json"), "w") as f:
json.dump(lfors, f, indent=2)
print("tiny2 LOFO:", [round(l["FA_rate"], 3) for l in lfors])
# grouped CV for the same best config + in-pool pairs / capQ
oof = {v: np.zeros(len(rows)) for v in VIEWS}
for fold in range(5):
tr = np.where(folds != fold)[0]
te = np.where(folds == fold)[0]
t_idx, v_idx = _val_split(tr, y, 42 + 100 * fold)
net, _, _, _ = train_model(ids_a, attn_a, y, t_idx, v_idx, 2e-5, 42 + fold, builder=make2)
for v in VIEWS:
oof[v][te] = predict_proba(net, *ev[v], te)
inpool = {"strip": s19.binary_metrics(y, oof["strip"]),
"pairs": {v: s19.pair_metrics(pairs, oof[v]) for v in VIEWS},
"capQ_inpool": {v: s19.capq_fa(tags, y, oof[v]) for v in VIEWS}}
with open(os.path.join(CEIL_RESULTS, "grouped.json"), "w") as f:
json.dump(inpool, f, indent=2, default=str)
b = inpool["strip"]
print(f"tiny2 grouped A@2e-5: PR={b['PR_AUC']:.3f} P={b['P']:.3f} R={b['R']:.3f} "
f"FA={b['FA']} pairs_strip={inpool['pairs']['strip']['ordering_acc']:.3f} "
f"capQ_strip={inpool['capQ_inpool']['strip']:.3f}", flush=True)
print("ceiling done")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("cmd", choices=["pre", "grouped", "lfo", "metrics", "runtime", "onnx", "ceiling"])
args = ap.parse_args()
t0 = time.time()
globals()[f"cmd_{args.cmd}"](args)
print(f"elapsed {time.time()-t0:.1f}s")
if __name__ == "__main__":
main()
@@ -0,0 +1,95 @@
package main
// Fixture is one brief-specified utterance and the eligibility it must earn.
// Fixtures are EXACTLY the slice-21 brief examples plus the corpus-verified
// structural discriminators. They are eval material, not corpus additions:
// nothing here is inserted into any pool file (brief §2, §18).
type Fixture struct {
Utterance string
Want Eligibility
Family string
Note string
}
// Fixtures covers every family the brief's §15 list exercises, including the
// four that have no dev-pool rows at all (negation, reported_speech,
// quotation, hypothetical):
//
// direct imperative already plural in dev; fixtures pin the canonical
// cases together with the politeness-negative modal.
// polite request polite imperative + bare "можешь …, пожалуйста".
// modal request "можно …", English "can you …, please".
// first-person request "я хочу …", "мне нужно …", "надо …".
// reordered target "свет выключи, пожалуйста" (target-first command).
// capability question ты/умеешь/сможешь/способна + "… ли" permission.
// ordinary question "что запущено" — no execution pressure.
// negation "не выключай свет" (parser-covered); advisory "не
// надо выключать свет".
// reported speech past/third-person report verbs over a command.
// quotation quoted command, with and without a reporting frame.
// hypothetical если-scopes that are not real condition→command.
var Fixtures = []Fixture{
// negation — direct commands the prosecutor forbids
{"не выключай свет", Blocked, "negation", "direct prohibition"},
{"не перезапускай сервер", Blocked, "negation", "direct prohibition"},
{"не включай nginx", Blocked, "negation", "direct prohibition"},
{"не надо выключать свет", Blocked, "negation", "advisory negative"},
{"не стоит перезапускать nginx", Blocked, "negation", "advisory negative"},
{"не забудь напомнить про свет", Permissive, "negation", "prohibition-parser reminder exemption"},
// reported speech — reports an order, does not issue one
{"он сказал выключить свет", Blocked, "reported_speech", "past report verb + infinitive"},
{"она попросила перезапустить nginx", Blocked, "reported_speech", "past report verb + infinitive"},
{"мне сказали включить свет", Blocked, "reported_speech", "passive report + infinitive"},
{"он написал: «перезапусти nginx»", Blocked, "reported_speech", "report verb + quoted imperative"},
{"скажи мне, что он сказал про свет", Permissive, "reported_speech", "request to report, no commanded clause"},
{"расскажи про свет", Permissive, "reported_speech", "narrative request, not a reported order"},
// quotation — quoted text is referenced, not issued
{"фраза «выключи свет»", Blocked, "quotation", "reporting noun + quoted imperative"},
{"он сказал «выключи свет»", Blocked, "quotation", "report verb + quoted imperative"},
{"«выключи свет»", Ambiguous, "quotation", "bare quoted command, no frame"},
{"выключи свет", Permissive, "quotation", "unquoted imperative is a live command"},
// hypothetical
{"если выключить свет...", Blocked, "hypothetical", "conditional + infinitive + ellipsis"},
{"если бы перезапустить nginx...", Blocked, "hypothetical", "conditional + бы + infinitive"},
{"что будет если выключить свет", Blocked, "hypothetical", "question-scoped conditional"},
{"если будет дождь, выключи полив", Permissive, "hypothetical", "real condition → imperative"},
{"выключи свет если будет дождь", Permissive, "hypothetical", "imperative → real condition"},
// capability question — blocked even polite
{"ты можешь выключить свет?", Blocked, "capability_question", "ты + можешь + ?"},
{"ты можешь выключить свет", Blocked, "capability_question", "ты + можешь, no ?"},
{"ты можешь выключить свет, пожалуйста", Blocked, "capability_question", "ты + можешь + politeness (42/42 non-action)"},
{"умеешь ли ты выключить свет", Blocked, "capability_question", "ability form + ли"},
{"сможешь открыть окно, пожалуйста", Blocked, "capability_question", "bare future + politeness (7/7 non-action)"},
{"ты способна выключить свет", Blocked, "capability_question", "ты + способна"},
{"могу ли я выключить свет", Blocked, "capability_question", "first-person can + ли"},
{"можно ли выключить свет", Blocked, "capability_question", "можно + ли permission question"},
{"ты выключишь свет?", Ambiguous, "capability_question", "future tense + ? without can-form"},
// modal / polite requests — permissive
{"можешь выключить свет, пожалуйста", Permissive, "modal_request", "bare можешь + politeness (127/127 action)"},
{"пожалуйста, выключи свет", Permissive, "polite_request", "leading politeness + imperative"},
{"выключи свет, пожалуйста", Permissive, "polite_request", "imperative + trailing politeness"},
{"выключи свет", Permissive, "direct_imperative", "plain imperative"},
{"свет выключи, пожалуйста", Permissive, "reordered_target", "target-first imperative"},
{"can you выключи свет, please", Permissive, "modal_request", "English frame + Russian imperative + please (96/96 action)"},
{"can you выключи свет", Ambiguous, "modal_request", "English can without politeness"},
{"не мог бы ты выключить свет", Permissive, "polite_request", "conditional politeness, prohibition-parser exemption"},
{"можно выключить свет", Permissive, "modal_request", "можно + infinitive permission-implicature request"},
// first-person requests
{"я хочу выключить свет", Permissive, "first_person_request", "first-person + illocution"},
{"я хочу чтобы ты выключил свет", Permissive, "first_person_request", "first-person + embedded ya-you wish"},
{"надо выключить свет", Permissive, "first_person_request", "impersonal need"},
{"мне нужно включить свет", Permissive, "first_person_request", "first-person oblique + need"},
// ordinary questions — no execution pressure even when answerable
{"что запущено", Ambiguous, "question", "status question, no request evidence"},
{"какие службы работают", Ambiguous, "question", "question, no request evidence"},
{"сколько ламп включено", Ambiguous, "question", "question, no request evidence"},
{"что ты можешь включить", Blocked, "capability_question", "open question with ты + можешь"},
{"покажи что запущено", Permissive, "first_person_request", "imperative lead over a status question"},
}
@@ -0,0 +1,659 @@
// Guard is the slice-21 deterministic execution-frame engine (experiment-only).
//
// It answers one question: given an utterance, what is its execution-frame
// eligibility as a three-way gate — permissive, blocked, ambiguous — and why.
// It never decides what an utterance IS (that stays with the route classifier);
// it only decides whether an utterance may become an executable action at all.
// The policy is asymmetric on purpose: blocked and ambiguous must never
// execute, and permissive only means "no blocking speech-act evidence exists",
// not "execute this".
//
// It reuses the shipped deterministic routers rather than inventing new ones:
//
// router.ParseCommandProhibition / IsCommandProhibition direct negative commands
// morph.IsVerbForm / morph.Lemma verb mood and finiteness
// lexicon.IsFillerParticle / FirstPerson() politeness and first-person frames
//
// Everything else is closed-class evidence measured on the frozen slice-20 dev
// pool (§"measured discriminators" in the brief): 126 capability-question rows
// split 42/42/42 across ты-addressed, bare ability (умеешь), and bare future
// (сможешь) modality; 127 bare "можешь, пожалуйста" rows are 100% action;
// "can you … , please" (English frame + Russian imperative) is 100% action.
// The rules below are the encoding of precisely those numbers.
//
// The reason vocabulary is a closed set. Additions are design decisions that
// must land in the report, not silent new branches.
package main
import (
"strings"
"unicode"
"github.com/kami/maven/internal/lexicon"
"github.com/kami/maven/internal/morph"
"github.com/kami/maven/internal/router"
)
// Eligibility is the three-way execution-frame verdict.
type Eligibility int
const (
Permissive Eligibility = iota // no blocking speech-act evidence; downstream route decides
Blocked // a speech act forbids execution (negation, question, report, …)
Ambiguous // not enough evidence either way; must not execute
)
func (e Eligibility) String() string {
switch e {
case Permissive:
return "permissive"
case Blocked:
return "blocked"
default:
return "ambiguous"
}
}
// Reason is a closed set of structural explanations for a verdict.
type Reason string
const (
ReasonCommandProhibition Reason = "command_prohibition"
ReasonCapabilityQuestion Reason = "capability_question"
ReasonReportedSpeech Reason = "reported_speech"
ReasonQuotation Reason = "quotation"
ReasonHypothetical Reason = "hypothetical"
ReasonNegatedCommand Reason = "negated_command"
ReasonExplicitRequest Reason = "explicit_request"
ReasonAmbiguousModal Reason = "ambiguous_modal"
ReasonNoRequestEvidence Reason = "no_request_evidence"
)
func (r Reason) String() string { return string(r) }
// Frame is the verdict for one utterance. Eligibility decides; Reasons explain.
// A frame may carry more than one reason (e.g. a quoted reported command).
type Frame struct {
Eligibility Eligibility
Reasons []Reason
}
// maybeWord is a single-token or multi-token closed expression, e.g. the
// token "не мог бы" covers the three tokens не мog бы when matched as a
// contiguous run ("бы" is itself a bound marker). Multi-token members are
// matched over the reconstructed token text, never over raw text, so
// punctuation boundaries do not defeat them.
type maybeWord struct {
single []string
multi []string // matched as contiguous lowercased token runs
}
func (w maybeWord) in(toks []string, joined string) bool {
if hasAny(toks, w.single) {
return true
}
for _, m := range w.multi {
tm := strings.Join(tokens(m), " ")
if tm != "" && strings.Contains(joined, tm) {
return true
}
}
return false
}
// ── closed evidence sets (all measured on the slice-20 dev pool) ──────────
// wakeAddr is stripped from the left of an utterance before command-form
// detection: "мавен, выключи свет" and "выключи свет" must ride the same
// frame. Closed: the names Maven answers to in the dev pool.
var wakeAddr = []string{"мавен", "maven", "мавэн", "алекса", "алиса", "окей", "эй", "hey"}
// ruAddress are the second-person Russian address tokens. "ты можешь …"
// (with or without politeness) is 42/42 capability-question in the dev pool,
// so any addressed Russian can-form is a capability question, never a request.
var ruAddress = []string{"ты", "тебе", "тебя", "тобой", "тобою", "вы", "вас", "вам", "вами"}
// enAddress is the English second-person address. Unlike Russian, "can you …
// , please" is 96/96 action in the dev pool (English modal frame around a
// Russian imperative), so English address alone never blocks: it routes to the
// politeness arm.
var enAddress = []string{"you", "u", "your"}
// ruCanForms are the present-can verb forms. Bare (no address) "можешь …,
// пожалуйста" is 127/127 action; bare "можешь …" with no politeness is the
// ambiguous bucket (no such rows exist in dev — conservative default).
var ruCanForms = []string{"можешь", "можете", "могу", "можем"}
// ruAbilityForms are future/ability modal forms that read as a question of
// capability regardless of politeness: "сможешь открыть окно, пожалуйста" and
// "умеешь ли ты …" are 0/84 action in the dev pool, so even a polite bare
// form never grants execution. "мог(ла) бы …" and "смог(ла) бы …" are the
// conditional-politeness mask over the same boundary — except the leading
// politeness construction "не мог бы ты …", which the prohibition parser
// already classifies as ordinary modal politeness and must stay permissive.
var ruAbilityForms = maybeWord{
single: []string{
"сможешь", "сможете", "смогу", "сможем", "сумеешь", "сумеете",
"умеешь", "умеете", "способна", "способен", "способно", "способны",
"смог", "смогла", "смогли", "мог", "могла", "могли",
},
multi: []string{
"смог бы", "смогла бы", "смогли бы", "мог бы", "могла бы", "могли бы",
"смочь бы", "мочь бы",
},
}
// politeNegativeModal is the leading "не мог бы ты/вы …" politeness framing the
// prohibition parser exempts as ordinary modal politeness. When it leads the
// utterance the capability stage declines and the frame reads as a request.
var politeNegativeModal = []string{
"не мог бы", "не могла бы", "не могли бы", "не смог бы", "не смогла бы", "не смогли бы",
}
// enCanForms are the English modal can/could tokens.
var enCanForms = []string{"can", "could"}
// politeness is the closed set of politeness fillers. пожалуйста/плиз/please
// are already closed-class filler particles in the lexicon; the добр-forms
// are the only additions the dev pool exercises.
var politeness = maybeWord{
single: []string{"пожалуйста", "плиз", "please"},
multi: []string{"будь добр", "будьте добры", "был бы добр", "были бы добры"},
}
// reportVerbs are the past/third-person report verbs — the frame that reports
// a command rather than issuing it. Second-person imperatives ("скажи",
// "расскажи", "напомни") are deliberately absent: those are requests to
// report, and their clause forms part of the current utterance, not a
// replayed order. Matched as closed list (a report verb outside it is a data
// gap, noted in the report).
var reportVerbs = []string{
"сказал", "сказала", "сказали", "говорил", "говорила", "говорили",
"говорит", "говорят", "попросил", "попросила", "попросили",
"просил", "просила", "просили", "написал", "написала", "написали",
"пишет", "приказал", "приказала", "приказали", "велел", "велела",
"велели", "скомандовал", "скомандовала", "рекомендовал", "рекомендовала",
"посоветовал", "посоветовала", "сообщил", "сообщила", "сообщили",
"объявил", "объявила", "велено", "сказано", "написано", "записано",
}
// reportNouns name a quoted or reported text: "фраза «выключи свет»" is a
// quotation, not a command.
var reportNouns = []string{
"фраза", "фразы", "фразе", "фразу", "слово", "слова", "слове", "словом",
"выражение", "выражения", "цитата", "цитату", "цитате",
"название", "текст", "сообщение", "письмо", "заметка", "заметку",
}
// hypothesisMarkers open a conditional scope.
var hypothesisMarkers = []string{"если", "ежели", "коли", "кабы", "if"}
// illocutionVerbs make a first-person or impersonal clause a request even
// without an imperative form ("я хочу …", "мне нужно …", "надо …").
var illocutionVerbs = maybeWord{
single: []string{
"хочу", "хотел", "хотела", "хотелось", "желаю", "прошу", "просим",
"просил", "просила", "просили", "попросить",
"надо", "нужно", "следует", "пора", "требуется", "придётся", "придется",
"могу", "давай", "давайте",
},
multi: []string{
"хотел бы", "хотела бы", "хочу чтобы", "хотел чтобы", "хотела чтобы",
"могу ли",
},
}
// ── token helpers ─────────────────────────────────────────────────────────
// tokens lowercases and splits on anything that is not a letter or digit,
// matching the router's planTokens discipline ("что-дальше" tokenises like
// "что дальше").
func tokens(text string) []string {
return strings.FieldsFunc(strings.ToLower(text), func(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
})
}
func hasTok(toks []string, w string) bool {
for _, t := range toks {
if t == w {
return true
}
}
return false
}
func hasAny(toks, ws []string) bool {
for _, w := range ws {
if hasTok(toks, w) {
return true
}
}
return false
}
func indexTok(toks []string, w string) int {
for i, t := range toks {
if t == w {
return i
}
}
return -1
}
// isFiniteVerb reports a verb form that is not the dictionary (infinitive)
// form: "выключи" is finite, "выключить" is not. A finite verb at command
// position is positive request evidence; an infinitive is not.
func isFiniteVerb(tok string) bool {
if !morph.IsVerbForm(tok) {
return false
}
return morph.Lemma(tok) != tok
}
// isInfinitive reports a token that morph resolves to its own dictionary form
// (the lemma ends in the infinitive ending by construction).
func isInfinitive(tok string) bool {
if !morph.IsVerbForm(tok) {
return false
}
return morph.Lemma(tok) == tok
}
func anyInfinitive(toks []string) bool {
for _, t := range toks {
if isInfinitive(t) {
return true
}
}
return false
}
func anyFiniteVerb(toks []string) bool {
for _, t := range toks {
if isFiniteVerb(t) {
return true
}
}
return false
}
func hasAnyVerb(toks []string) bool {
for _, t := range toks {
if morph.IsVerbForm(t) {
return true
}
}
return false
}
func blocked(rs ...Reason) Frame { return Frame{Eligibility: Blocked, Reasons: rs} }
func ambiguous(rs ...Reason) Frame {
return Frame{Eligibility: Ambiguous, Reasons: rs}
}
func permissive(rs ...Reason) Frame {
return Frame{Eligibility: Permissive, Reasons: rs}
}
// ── quoted spans ─────────────────────────────────────────────────────────
// quotedSpan is a maximal quoted interval in the normalized text.
type quotedSpan struct{ content string }
// quotePairs covers the quoting styles the dev pool and brief fixtures use:
// Russian guillemets, curly double/single quotes, and straight quotes.
var quotePairs = []struct{ open, close string }{
{"«", "»"}, {"„", "\""}, {"“", "”"}, {"", ""}, {"", ""}, {"'", "'"}, {"\"", "\""},
}
// extractQuotedSpans returns the contents of quoted spans in order, in rune
// index space (the text is normalized, so glyphs are single runes). An
// unbalanced delimiter yields no span (best-effort; the conservative
// fallback then applies).
func extractQuotedSpans(t string) []quotedSpan {
runes := []rune(t)
var out []quotedSpan
i := 0
for i < len(runes) {
matched := false
for _, p := range quotePairs {
po := []rune(p.open)
pc := []rune(p.close)
if i+len(po) > len(runes) || string(runes[i:i+len(po)]) != p.open {
continue
}
j := i + len(po)
for j+len(pc) <= len(runes) && string(runes[j:j+len(pc)]) != p.close {
j++
}
out = append(out, quotedSpan{content: string(runes[i+len(po) : j])})
i = j + len(pc)
matched = true
break
}
if !matched {
i++
}
}
return out
}
// ── rule stages (evaluated in this order; a decision is final) ────────────
// Evaluate derives the execution-frame verdict for one utterance.
func Evaluate(text string) Frame {
t := router.NormalizeMatchText(text)
if strings.TrimSpace(t) == "" {
return ambiguous(ReasonNoRequestEvidence)
}
toks := tokens(t)
joined := strings.Join(toks, " ")
// 1. Quotation: a command inside a quoted span is not a command being
// issued now. With a reporting frame outside it is a quotation; a bare
// quote is at best ambiguous.
if f, ok := stageQuotation(t, toks, joined); ok {
return f
}
// 2. Reported speech: a past/third-person report verb governing a command
// clause reports an order to someone else, it does not issue one.
if f, ok := stageReport(t, toks, joined); ok {
return f
}
// 3. Hypothetical: a command scope opened by "если/if" that does not
// continue as a real condition→command is not an execution request.
if f, ok := stageHypothesis(toks, joined); ok {
return f
}
// 4. Direct negative commands: the shipped prohibition parser.
if router.IsCommandProhibition(t) {
return blocked(ReasonCommandProhibition)
}
// 5. Advisory negatives: "не надо/не стоит/не нужно …".
if f, ok := stageAdvisoryNegation(toks); ok {
return f
}
// 6. Capability and permission modality (the measured core).
if f, ok := stageCapability(toks, joined); ok {
return f
}
// 7. Trailing question mark with no modal at play: an uncertain posture,
// never a confirmed executable request.
if strings.HasSuffix(t, "?") {
return ambiguous(ReasonAmbiguousModal)
}
// 8. Positive request evidence.
if hasRequestEvidence(toks, joined) {
return permissive(ReasonExplicitRequest)
}
// 9. No execution pressure at all.
return ambiguous(ReasonNoRequestEvidence)
}
// stageQuotation blocks a quoted command when a reporting frame surrounds it.
func stageQuotation(t string, toks []string, joined string) (Frame, bool) {
spans := extractQuotedSpans(t)
if len(spans) == 0 {
return Frame{}, false
}
commandSpan := false
for _, sp := range spans {
if isCommandishWithin(tokens(sp.content), strings.Join(tokens(sp.content), " ")) {
commandSpan = true
break
}
}
if !commandSpan {
return Frame{}, false
}
reasons := []Reason{ReasonQuotation}
if hasReportFrame(toks, joined) {
reasons = append(reasons, ReasonReportedSpeech)
return Frame{Eligibility: Blocked, Reasons: reasons}, true
}
// a bare quoted command has no reporting frame: refusable but not a
// definite prohibition either (it is at least ambiguous)
return Frame{Eligibility: Ambiguous, Reasons: reasons}, true
}
// isCommandishWithin reports the span content carrying command or capability
// polarity itself — imperative, prohibition, or a can-form.
func isCommandishWithin(toks []string, joined string) bool {
if len(toks) == 0 {
return false
}
if router.IsCommandProhibition(strings.Join(toks, " ")) {
return true
}
if hasAny(toks, ruCanForms) || ruAbilityForms.in(toks, joined) || hasAny(toks, enCanForms) {
return true
}
return anyFiniteVerb(toks)
}
func hasReportFrame(toks []string, joined string) bool {
if hasAny(toks, reportVerbs) {
return true
}
return hasAny(toks, reportNouns)
}
// stageReport blocks when a report frame governs a command clause: an
// infinitive after the report verb, or a quoted imperative. Second-person
// imperatives like "скажи/расскажи" are not in reportVerbs, so a request to
// report ("расскажи мне, что сказал папа") passes through.
func stageReport(t string, toks []string, joined string) (Frame, bool) {
if !hasReportFrame(toks, joined) {
return Frame{}, false
}
last := -1
for i, w := range toks {
if hasTok(reportVerbs, w) || hasTok(reportNouns, w) {
last = i
}
}
if last < 0 {
return Frame{}, false
}
after := toks[last+1:]
if len(after) == 0 {
return Frame{}, false
}
// a quoted command after the frame counts as the governed clause
for _, sp := range extractQuotedSpans(t) {
if isCommandishWithin(tokens(sp.content), strings.Join(tokens(sp.content), " ")) {
return blocked(ReasonReportedSpeech, ReasonQuotation), true
}
}
if anyInfinitive(after) || hasAny(after, []string{"что", "чтобы", "чтоб"}) {
return blocked(ReasonReportedSpeech), true
}
return Frame{}, false
}
// stageHypothesis blocks a conditional scope whose clauses are hypothetical
// (infinitive or subjunctive "бы") rather than a real condition→command.
// "если будет дождь, выключи полив" keeps its imperative continuation and
// passes through; it is a real conditional request, not a hypothetical.
func stageHypothesis(toks []string, joined string) (Frame, bool) {
idx := -1
for _, m := range hypothesisMarkers {
if i := indexTok(toks, m); i >= 0 && (idx < 0 || i < idx) {
idx = i
}
}
if idx < 0 {
return Frame{}, false
}
post := toks[idx+1:]
if len(post) == 0 || hasTok(post, "бы") || anyInfinitive(post) {
return blocked(ReasonHypothetical), true
}
// a real condition clause is not hypothetical: «если будет дождь,
// выключи полив» is a request. The dev dict does not cover «будет», so
// the imperative is looked for anywhere, not just after the marker
// («выключи свет, если будет дождь»).
if !anyFiniteVerb(toks) {
return blocked(ReasonHypothetical), true
}
return permissive(ReasonExplicitRequest), true
}
// stageAdvisoryNegation blocks "не надо/не нужно/не стоит/не следует …".
// (absent from the dev pool; covered by brief fixtures)
func stageAdvisoryNegation(toks []string) (Frame, bool) {
if len(toks) < 3 || toks[0] != "не" {
return Frame{}, false
}
if !hasTok(toks[1:2], "надо") && !hasTok(toks[1:2], "нужно") &&
!hasTok(toks[1:2], "стоит") && !hasTok(toks[1:2], "следует") &&
!hasTok(toks[1:2], "требуется") {
return Frame{}, false
}
rest := toks[2:]
if anyInfinitive(rest) || anyFiniteVerb(rest) || hasAnyVerb(rest) {
return blocked(ReasonNegatedCommand), true
}
return Frame{}, false
}
// stageCapability encodes the measured modal matrix. Returns a decision when
// modality alone settles the frame.
func stageCapability(toks []string, joined string) (Frame, bool) {
// "не мог бы ты …, пожалуйста" style conditional politeness is ordinary
// modal politeness (the prohibition parser exempts it as such): a request.
for _, pref := range politeNegativeModal {
if strings.HasPrefix(joined, pref) {
return permissive(ReasonExplicitRequest), true
}
}
// "… ли" directly after a can-form is a polar capability question:
// "могу ли я …", "можешь ли ты …", "умеешь ли ты …", "можно ли …".
// Checked before the modality arms so the polar reading wins.
if hasTok(toks, "ли") {
for i := 1; i < len(toks); i++ {
if toks[i] != "ли" {
continue
}
prev := toks[i-1]
if hasTok(ruCanForms, prev) || prev == "можно" || hasTok(ruAbilityForms.single, prev) {
return blocked(ReasonCapabilityQuestion), true
}
}
}
ruAddr := hasAny(toks, ruAddress)
ruCan := hasAny(toks, ruCanForms)
ruAbil := ruAbilityForms.in(toks, joined)
enCan := hasAny(toks, enCanForms)
polite := politeness.in(toks, joined)
// addressed Russian can-form: capability question, always blocked.
// ("ты можешь выключить свет, пожалуйста" included — 42/42 non-action.)
if ruAddr && (ruCan || ruAbil) {
return blocked(ReasonCapabilityQuestion), true
}
// ability forms (future/conditional/умеешь) are capability even bare and
// even polite: "сможешь открыть окно, пожалуйста" is 7/7 non-action.
if ruAbil && !ruCan {
return blocked(ReasonCapabilityQuestion), true
}
// bare Russian present can-form: politeness is the request marker.
if ruCan && !ruAddr {
if polite {
return Frame{}, false // modal-request positive evidence is found later
}
return ambiguous(ReasonAmbiguousModal), true
}
// English can/could: "can you …, please" is a request (96/96 action in the
// dev pool; the frame wraps a Russian imperative). Without politeness it
// reads as a capability question and stays ambiguous.
if enCan && !ruAddr {
if polite {
return Frame{}, false // positive modal-request evidence later
}
return ambiguous(ReasonAmbiguousModal), true
}
// "можно" (permission): "можно ли …" is a permission question; a bare
// "можно …" is a politeness-implicature request.
if hasTok(toks, "можно") {
if hasTok(toks, "ли") {
return blocked(ReasonCapabilityQuestion), true
}
return Frame{}, false
}
// bare "могу": a self-capability statement, not a request.
if hasTok(toks, "могу") && !hasTok(toks, "ли") {
return ambiguous(ReasonAmbiguousModal), true
}
return Frame{}, false
}
// hasRequestEvidence is the positive permissive trigger, reached only after
// every block/ambiguity stage above has declined.
func hasRequestEvidence(toks []string, joined string) bool {
polite := politeness.in(toks, joined)
enCan := hasAny(toks, enCanForms)
// 1. politeness + a verb (or an English modal) is explicit request
// evidence: "можешь выключить свет, пожалуйста", "can you останови …,
// please", "выключи свет, пожалуйста".
if polite && (hasAnyVerb(toks) || enCan) {
return true
}
// 2. first-person illocution frame: "я хочу …", "мне нужно …".
if hasAny(toks, lexicon.FirstPerson()) && illocutionVerbs.in(toks, joined) {
return true
}
// 3. impersonal need: "надо …", "нужно …", "пора …".
if hasAny(toks, []string{"надо", "нужно", "следует", "пора", "требуется", "придётся", "придется"}) {
return true
}
// 4. permission-implicature request: "можно выключить свет".
if hasTok(toks, "можно") && !hasTok(toks, "ли") {
return true
}
// 5. reminder request in the parser's own exemption scope: the
// prohibition parser declines «не забудь напомнить про свет» as a
// reminder, not a prohibition — carry that into a request.
if strings.HasPrefix(joined, "не забудь") && hasAny(toks, lexicon.ReminderVerbs()) {
return true
}
// 6. leading finite verb (imperative or otherwise tensed verb at command
// position): "выключи свет", "покажи что запущено". Address and filler
// particles are stripped first, so "мавен, выключи свет" rides the same
// frame.
lead := toks
for len(lead) > 0 {
first := lead[0]
if !lexicon.IsFillerParticle(first) && !hasTok(wakeAddr, first) && !hasTok(ruAddress, first) {
break
}
lead = lead[1:]
}
if len(lead) > 0 && isFiniteVerb(lead[0]) {
return true
}
return false
}
@@ -0,0 +1,139 @@
package main
import (
"encoding/json"
"os"
"strings"
"testing"
"github.com/kami/maven/internal/morph"
)
func TestEvaluateFixtures(t *testing.T) {
for _, fx := range Fixtures {
got := Evaluate(fx.Utterance)
if got.Eligibility != fx.Want {
t.Errorf("%s: want %s got %s (reasons %v)", fx.Utterance, fx.Want, got.Eligibility, got.Reasons)
}
}
}
// TestMeasuredDiscriminators pins the corpus-verified numbers on the frozen
// pool. These are the exact measurements the rules were built on, so a change
// that moves them is a rule regression visible in the slice.
func loadPool(t *testing.T) []Row {
t.Helper()
b, err := os.ReadFile("/tmp/mvn-s21/pool.json")
if os.IsNotExist(err) {
t.Skip("pool.json missing; run slice21_emit.py first")
}
if err != nil {
t.Fatal(err)
}
var rows []Row
if err := json.Unmarshal(b, &rows); err != nil {
t.Fatal(err)
}
return rows
}
func TestDevCapabilityProhibition(t *testing.T) {
if !morph.Available() {
t.Skip("morph dict unavailable")
}
rows := loadPool(t)
var n, pass, blockedN, ambig int
for _, r := range rows {
if !hasTok(r.Tags, "capability_question") {
continue
}
n++
switch Evaluate(variants(r.NText)[vOrig]).Eligibility {
case Permissive:
pass++
case Blocked:
blockedN++
case Ambiguous:
ambig++
}
}
if n != 126 {
t.Fatalf("capability-question rows = %d, want 126", n)
}
if pass != 0 {
t.Fatalf("capability-question dangerous pass = %d, want 0", pass)
}
if blockedN != 126 && ambig != 0 {
t.Errorf("blocked=%d ambig=%d, expect all 126 blocked", blockedN, ambig)
}
}
func TestDevBareCanPoliteIsAction(t *testing.T) {
if !morph.Available() {
t.Skip("morph dict unavailable")
}
rows := loadPool(t)
total, action, perm := 0, 0, 0
for _, r := range rows {
toks := tokens(r.NText)
if !hasTok(toks, "можешь") || hasAny(toks, ruAddress) ||
hasAny(toks, append([]string{}, ruAbilityForms.single...)) {
continue
}
total++
if r.Route == "action" {
action++
}
if Evaluate(r.NText).Eligibility == Permissive {
perm++
}
}
if total != 127 || action != 127 {
t.Fatalf("bare-можешь+polite: n=%d action=%d, want 127/127", total, action)
}
if perm != 127 {
t.Fatalf("bare-можешь+polite permissive=%d, want 127", perm)
}
}
func TestDevEnglishCanPoliteIsAction(t *testing.T) {
if !morph.Available() {
t.Skip("morph dict unavailable")
}
rows := loadPool(t)
total, action, perm := 0, 0, 0
for _, r := range rows {
toks := tokens(r.NText)
if !hasAny(toks, enCanForms) {
continue
}
total++
if r.Route == "action" {
action++
}
if Evaluate(r.NText).Eligibility == Permissive {
perm++
}
}
if total != 96 || action != 96 {
t.Fatalf("can-rows: n=%d action=%d, want 96/96", total, action)
}
if perm != 96 {
t.Fatalf("can-rows permissive=%d, want 96", perm)
}
}
func TestNoPermissiveRowEndsInQuestion(t *testing.T) {
// asymmetric posture: the guard must never approve a trailing-? frame --
// the corpus has 0 action rows ending in "?", and approving any would
// bet on punctuation the slice has ruled uncertain.
rows := loadPool(t)
for _, r := range rows {
if !strings.HasSuffix(r.NText, "?") {
continue
}
if got := Evaluate(r.NText).Eligibility; got == Permissive {
t.Errorf("canonical '?': %q approved (%s)", r.Text, got)
}
}
}
@@ -0,0 +1,566 @@
// Slice 21 runner: report the deterministic execution-frame guard against the
// frozen slice-20 dev pool.
//
// Reads the emit step's compact files (pool.json, pairs.json, sparse_oof.json
// in /tmp/mvn-s21) and prints the report tables plus a machine-readable
// guard_results.json. Reuses router/morph/lexicon parsers live inside this
// module — the pool texts are the only data, no embedding is recomputed.
//
// Usage: go run ./cmd/semantic-router-experiment/slice21
package main
import (
"encoding/json"
"flag"
"fmt"
"os"
"regexp"
"sort"
"strings"
"unicode"
)
const sparseThreshold = 0.715 // slice-18 §4 strict operating point (P>=0.95 best recall)
var familyPriority = []string{
"capability_question", "question", "first_person_request",
"modal_request", "polite_request", "reordered_target", "direct_imperative",
}
func familyOf(tags []string) string {
for _, f := range familyPriority {
if hasTok(tags, f) {
return f
}
}
return "other"
}
// ── pool row ──────────────────────────────────────────────────────────────
type Row struct {
Idx int `json:"idx"`
Text string `json:"text"`
NText string `json:"n_text"`
Route string `json:"route"`
Y int `json:"y"`
Tags []string `json:"tags"`
CVFold int `json:"cv_fold"`
SplitGp string `json:"split_group"`
SourceID string `json:"source_id"`
Family string
VariantOf int
}
type Pair struct{ Cap, Act int }
// ── stress variants ───────────────────────────────────────────────────────
var nofinalRe = regexp.MustCompile(`[?.!,;:]+$`)
// strip_punct mirrors the slice-18/19 python strip_punct: trailing sentence
// punctuation, then every non-word/non-space rune.
func stripPunct(t string) string {
t = nofinalRe.ReplaceAllString(strings.TrimSpace(t), "")
out := make([]rune, 0, len(t))
var prevSpace bool
for _, r := range t {
if unicode.IsLetter(r) || unicode.IsNumber(r) {
out = append(out, r)
prevSpace = false
} else if !prevSpace {
out = append(out, ' ')
prevSpace = true
}
}
return strings.TrimSpace(string(out))
}
func variants(nText string) [3]string {
return [3]string{
nText,
nofinalRe.ReplaceAllString(strings.TrimSpace(nText), ""),
stripPunct(nText),
}
}
const (
vOrig = iota
vNofinal
vStrip
)
var variantName = [3]string{"orig", "nofinal", "strip"}
type result struct {
Frame Frame `json:"frame"`
}
// ── metrics ──────────────────────────────────────────────────────────────
type triTab struct {
Permissive, Blocked, Ambiguous int
PermNonact, BlockedAction, AmbAction int
}
type runAgg struct {
n, action, tp, fp, fn int
approved int
capPermissive int
}
func (a *runAgg) addApproved(approved bool, route string) {
a.n++
if route == "action" {
a.action++
}
if approved {
a.approved++
if route == "action" {
a.tp++
} else {
a.fp++
}
} else if route == "action" {
a.fn++
}
}
func (a *runAgg) P() string { return fmtPct(frac(a.tp, a.tp+a.fp)) }
func (a *runAgg) R() string { return fmtPct(frac(a.tp, a.action)) }
func (a *runAgg) FA() int { return a.fp }
func (a *runAgg) FArate() string {
return fmtPct(frac(a.fp, a.n))
}
func maxi(a, b int) int {
if a > b {
return a
}
return b
}
// frac is the guarded ratio the tables print (0/0 is 0).
func frac(num, den int) float64 { return float64(num) / float64(maxi(den, 1)) }
func fmtPct(v float64) string { return fmt.Sprintf("%.1f%%", 100*v) }
// ── main ─────────────────────────────────────────────────────────────────
func main() {
poolPath := flag.String("pool", "/tmp/mvn-s21/pool.json", "dev pool rows")
pairsPath := flag.String("pairs", "/tmp/mvn-s21/pairs.json", "cap-vs-action pairs")
sparsePath := flag.String("sparse", "/tmp/mvn-s21/sparse_oof.json", "slice-18 both OOF proba")
outPath := flag.String("out", "/tmp/mvn-s21/guard_results.json", "machine-readable results")
flag.Parse()
rows := mustLoad[[]Row](*poolPath)
// pairs.json is bare [cap, act] index pairs; adapt into typed pairs.
rawPairs := mustLoad[[][2]int](*pairsPath)
pairs := make([]Pair, 0, len(rawPairs))
for _, rp := range rawPairs {
pairs = append(pairs, Pair{Cap: rp[0], Act: rp[1]})
}
sparseOOF := mustLoad[[]struct {
Idx int `json:"idx"`
Proba float64 `json:"proba"`
}](*sparsePath)
proba := make([]float64, len(rows))
for _, s := range sparseOOF {
proba[s.Idx] = s.Proba
}
famPrio := 0
for i := range rows {
rows[i].Family = familyOf(rows[i].Tags)
if rows[i].Family != "other" {
famPrio++
}
}
_ = famPrio
// verdicts per variant
type rowRes struct {
Idx int `json:"idx"`
Text string `json:"text"`
Route string `json:"route"`
Family string `json:"family"`
Tags []string `json:"tags"`
Frames map[string]string `json:"frames"` // variant -> eligibility
}
perVariant := make([][3]Frame, len(rows))
fmt.Println("slice 21 — deterministic execution-frame guard on slice-20 dev pool")
fmt.Println("==================================================================")
for i, r := range rows {
vs := variants(r.NText)
var fr [3]Frame
for vi := 0; vi < 3; vi++ {
fr[vi] = Evaluate(vs[vi])
}
perVariant[i] = fr
}
// ── §3 three-way cross-tab (orig) ─────────────────────────────────────
fmt.Println("\n## 1. Three-way eligibility × route (orig)")
tab := triTab{}
for i, r := range rows {
switch perVariant[i][vOrig].Eligibility {
case Permissive:
tab.Permissive++
if r.Route != "action" {
tab.PermNonact++
}
case Blocked:
tab.Blocked++
if r.Route == "action" {
tab.BlockedAction++
}
case Ambiguous:
tab.Ambiguous++
if r.Route == "action" {
tab.AmbAction++
}
}
}
fmt.Printf("permissive: %d blocked: %d ambiguous: %d\n", tab.Permissive, tab.Blocked, tab.Ambiguous)
fmt.Printf(" permissive non-action: %d blocked action: %d ambiguous action: %d\n",
tab.PermNonact, tab.BlockedAction, tab.AmbAction)
// ── §4 binary executable-gate metrics on orig ─────────────────────────
fmt.Println("\n## 2. Binary executable gate (approve = permissive; deny = blocked|ambiguous)")
a := runAgg{}
capCov, capPerm, capAmb := 0, 0, 0
for i, r := range rows {
el := perVariant[i][vOrig].Eligibility
a.addApproved(el == Permissive, r.Route)
if hasTok(r.Tags, "capability_question") {
capCov++
switch el {
case Permissive:
capPerm++
case Ambiguous:
capAmb++
}
}
}
fmt.Printf("approved: %d denied: %d (n=%d, action=%d)\n", a.approved, a.n-a.approved, a.n, a.action)
fmt.Printf("action precision %s recall %s FA %d (%s)\n", a.P(), a.R(), a.FA(), a.FArate())
fmt.Printf("capability-question dangerous pass: %d / %d (rate %s)\n",
capPerm, capCov, fmtPct(frac(capPerm, capCov)))
fmt.Printf("capability-question blocked %d, ambiguous %d\n", capCov-capPerm-capAmb, capAmb)
// ── §15 family stress (orig) ──────────────────────────────────────────
fmt.Println("\n## 3. Family stress (orig; counts per eligibility)")
fmt.Printf("%-24s %8s %8s %8s %8s\n", "family", "n", "perm", "block", "ambig")
famOrder := []string{"direct_imperative", "polite_request", "modal_request", "first_person_request",
"reordered_target", "capability_question", "question", "other"}
famAgg := map[string]*triTab{}
for _, f := range famOrder {
famAgg[f] = &triTab{}
}
for i, r := range rows {
t := famAgg[r.Family]
if t == nil {
continue
}
switch perVariant[i][vOrig].Eligibility {
case Permissive:
t.Permissive++
if r.Route != "action" {
t.PermNonact++
}
case Blocked:
t.Blocked++
case Ambiguous:
t.Ambiguous++
}
}
for _, f := range famOrder {
t := famAgg[f]
if t == nil {
continue
}
n := t.Permissive + t.Blocked + t.Ambiguous
if n == 0 {
continue
}
fmt.Printf("%-24s %8d %8d %8d %8d\n", f, n, t.Permissive, t.Blocked, t.Ambiguous)
}
// ── §cap-Q LOFO across stress variants ────────────────────────────────
fmt.Println("\n## 4. Capability-question dangerous pass by stress variant")
for vi := 0; vi < 3; vi++ {
cp, cb, ca := 0, 0, 0
for i, r := range rows {
if !hasTok(r.Tags, "capability_question") {
continue
}
switch perVariant[i][vi].Eligibility {
case Permissive:
cp++
case Blocked:
cb++
case Ambiguous:
ca++
}
}
fmt.Printf(" %-8s dangerous-pass %d blocked %d ambiguous %d\n",
variantName[vi], cp, cb, ca)
}
// ── §pair test ────────────────────────────────────────────────────────
fmt.Println("\n## 5. Paired action/capability (cap row must never clear)")
capClear, actPerm, actAmbig, actBlock := 0, 0, 0, 0
for _, p := range pairs {
cel := perVariant[p.Cap][vOrig].Eligibility
ael := perVariant[p.Act][vOrig].Eligibility
if cel == Permissive {
capClear++
}
switch ael {
case Permissive:
actPerm++
case Ambiguous:
actAmbig++
case Blocked:
actBlock++
}
}
fmt.Printf("pairs %d: cap cleared %d (rate %s), action permissive %d, action ambiguous %d, action blocked %d\n",
len(pairs), capClear, fmtPct(frac(capClear, len(pairs))),
actPerm, actAmbig, actBlock)
// ── safe composition §17 ─────────────────────────────────────────────
fmt.Println("\n## 6. Composition: guard-alone / sparse-alone / guard→sparse (orig)")
compose := map[string]*runAgg{
"guard_alone": {},
"sparse_alone": {},
"guard_sparse": {},
}
for i, r := range rows {
gPerm := perVariant[i][vOrig].Eligibility == Permissive
sPerm := proba[i] >= sparseThreshold
compose["guard_alone"].addApproved(gPerm, r.Route)
compose["sparse_alone"].addApproved(sPerm, r.Route)
compose["guard_sparse"].addApproved(gPerm && sPerm, r.Route)
}
fmt.Printf("%-14s %8s %8s %6s %10s %6s %10s\n", "policy", "P", "R", "FA", "FA rate", "capQ", "capQ rate")
for _, name := range []string{"guard_alone", "sparse_alone", "guard_sparse"} {
agg := compose[name]
capQ := 0
for i, r := range rows {
if !hasTok(r.Tags, "capability_question") {
continue
}
ok := false
switch name {
case "guard_alone":
ok = perVariant[i][vOrig].Eligibility == Permissive
case "sparse_alone":
ok = proba[i] >= sparseThreshold
case "guard_sparse":
ok = perVariant[i][vOrig].Eligibility == Permissive && proba[i] >= sparseThreshold
}
if ok {
capQ++
}
}
fmt.Printf("%-14s %8s %8s %6d %10s %6d %10s\n", name, agg.P(), agg.R(), agg.FA(),
agg.FArate(), capQ, fmtPct(float64(capQ)/126))
}
// composition on strip too (brief §16 voice stress)
fmt.Println("\n## 7. Composition on punctuation-stripped text (strip)")
c2 := runAgg{}
capQ2 := 0
for i, r := range rows {
gPerm := perVariant[i][vStrip].Eligibility == Permissive
ok := gPerm && proba[i] >= sparseThreshold
c2.addApproved(ok, r.Route)
if hasTok(r.Tags, "capability_question") && ok {
capQ2++
}
}
fmt.Printf("guard→sparse strip: P %s R %s FA %d (%s) capQ pass %d\n",
c2.P(), c2.R(), c2.FA(), c2.FArate(), capQ2)
// ── §19 manual classification scratch ─────────────────────────────────
fmt.Println("\n## 8. Manual classification (scan material written to manual_class.json)")
var dangerous []map[string]any
var permNonact []map[string]any
var deniedAction []map[string]any
for i, r := range rows {
fr := perVariant[i][vOrig]
if hasTok(r.Tags, "capability_question") && fr.Eligibility == Permissive {
dangerous = append(dangerous, map[string]any{
"idx": r.Idx, "text": r.Text, "route": r.Route,
"reasons": fr.Reasons,
})
}
if fr.Eligibility == Permissive && r.Route != "action" {
permNonact = append(permNonact, map[string]any{
"idx": r.Idx, "text": r.Text, "route": r.Route,
"family": r.Family, "reasons": fr.Reasons,
})
}
if fr.Eligibility != Permissive && r.Route == "action" {
deniedAction = append(deniedAction, map[string]any{
"idx": r.Idx, "text": r.Text, "family": r.Family,
"eligibility": fr.Eligibility.String(), "reasons": fr.Reasons,
})
}
}
writeManual(permNonact, deniedAction, dangerous)
fmt.Printf("dangerous passes: %d permissive non-action: %d denied action: %d\n",
len(dangerous), len(permNonact), len(deniedAction))
groupAndSample("permissive non-action by reason+family", permNonact, 4)
groupAndSample("denied action by reason+family", deniedAction, 4)
// ── fixtures ──────────────────────────────────────────────────────────
fmt.Println("\n## 9. Brief fixtures")
pass := 0
for _, fx := range Fixtures {
got := Evaluate(fx.Utterance)
mark := "ok "
if got.Eligibility != fx.Want {
mark = "FAIL"
} else {
pass++
}
if got.Eligibility != fx.Want {
fmt.Printf(" %s %-14s want %-10s got %-10s %s\n", mark, fx.Family,
fx.Want, got.Eligibility.String(), fx.Utterance)
}
}
fmt.Printf("fixtures: %d/%d passed\n", pass, len(Fixtures))
// write result file
rr := make([]rowRes, 0, len(rows))
for i, r := range rows {
fr := [3]string{"", "", ""}
for vi := 0; vi < 3; vi++ {
fr[vi] = perVariant[i][vi].Eligibility.String()
}
rr = append(rr, rowRes{
Idx: r.Idx, Text: r.Text, Route: r.Route, Family: r.Family, Tags: r.Tags,
Frames: map[string]string{
"orig": fr[vOrig], "nofinal": fr[vNofinal], "strip": fr[vStrip],
},
})
}
if *outPath != "" {
mustSave(*outPath, map[string]any{
"pool": "/tmp/mvn-s21/pool.json",
"rows": rr,
"aggregates": map[string]any{
"tab": tab,
"capq_pass": capPerm,
"capq_blocked": capCov - capPerm - capAmb,
"capq_ambiguous": capAmb,
"binary": map[string]any{"tp": a.tp, "fp": a.fp, "fn": a.fn, "approved": a.approved, "n": a.n},
"pairs": map[string]any{"n": len(pairs), "cap_cleared": capClear, "act_permissive": actPerm},
"guard_sparse": map[string]any{"tp": compose["guard_sparse"].tp, "fp": compose["guard_sparse"].fp, "fn": compose["guard_sparse"].fn},
"dangerous_passes": len(dangerous),
"perm_nonact_count": len(permNonact),
"denied_action": len(deniedAction),
},
})
fmt.Println("wrote", *outPath)
}
}
// ── manual classification helpers ─────────────────────────────────────────
func writeManual(permNonact, deniedAction, dangerous []map[string]any) {
writeJSON("/tmp/mvn-s21/manual_class.json", map[string]any{
"dangerous_passes": dangerous,
"permissive_non_action": permNonact,
"denied_action": deniedAction,
})
}
func groupAndSample(title string, rows []map[string]any, sample int) {
type g struct {
key string
n int
texts []string
}
groups := map[string]*g{}
var order []string
for _, r := range rows {
var family, reason, el string
if v, ok := r["family"].(string); ok {
family = v
}
if v, ok := r["eligibility"].(string); ok {
el = v
}
if rs, ok := r["reasons"].([]Reason); ok {
rs2 := make([]string, len(rs))
for k, rr := range rs {
rs2[k] = rr.String()
}
reason = strings.Join(rs2, ",")
} else if rs, ok := r["reasons"].([]string); ok {
reason = strings.Join(rs, ",")
}
key := fmt.Sprintf("family=%s elig=%s reason=%s", family, el, reason)
if _, ok := groups[key]; !ok {
groups[key] = &g{key: key}
order = append(order, key)
}
groups[key].n++
if len(groups[key].texts) < sample {
groups[key].texts = append(groups[key].texts, firstN(fmt.Sprint(r["text"]), 60))
}
}
fmt.Printf("%s (%d rows):\n", title, len(rows))
for _, key := range order {
gr := groups[key]
fmt.Printf(" %-58s n=%d %s\n", gr.key, gr.n, strings.Join(gr.texts, " | "))
}
}
func firstN(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n] + "…"
}
// ── io helpers ────────────────────────────────────────────────────────────
func mustLoad[T any](path string) T {
b, err := os.ReadFile(path)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
var v T
if err := json.Unmarshal(b, &v); err != nil {
fmt.Fprintln(os.Stderr, "json:", err)
os.Exit(1)
}
return v
}
func mustSave(path string, v any) {
b, err := json.MarshalIndent(v, "", " ")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if err := os.WriteFile(path, b, 0o644); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
func writeJSON(path string, v any) {
b, _ := json.MarshalIndent(v, "", " ")
_ = os.WriteFile(path, b, 0o644)
}
var _ = sort.Strings
@@ -0,0 +1,135 @@
#!/usr/bin/env python3
"""
Slice 21 emit: deterministic execution-frame guard data files for the Go harness
==================================================================================
Slice 18 showed the sparse lexical gate owns the aggregate boundary (PR-AUC
0.838, strict operating point at threshold 0.715 with P>=0.95 | R=0.264) and
slice 19/20 showed learning heads collapse on capability-question LOFO. Slice 21
tests the deterministic alternative: a rule engine over existing parsers that
decides execution eligibility as a three-way gate (permissive / blocked /
ambiguous), never itself routing.
This script only repackages the frozen dev pool for the Go harness. It reuses
slice 18's feature builders and grouped-CV and slice 19's pair builder verbatim,
so the numbers the Go side reports are the same populations the accepts
measured. It writes:
/tmp/mvn-s21/pool.json dev rows: idx, text, n_text, route, y, tags,
cv_fold, split_group, source_id, family
/tmp/mvn-s21/pairs.json capability-vs-action pairs (slice-19 builder)
/tmp/mvn-s21/sparse_oof.json slice-18 "both" grouped-CV OOF proba per row
(for the §17 guard+sparse composition)
No training happens here and no label is changed. The guard itself is Go.
"""
import json
import os
import re
import sys
import numpy as np
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import slice18_sparse # noqa: E402
import slice19_main # noqa: E402
OUT_DIR = "/tmp/mvn-s21"
SPARSE_THRESHOLD = 0.715 # slice-18 §4 strict-operating-point (P>=0.95 best recall)
def main():
# Population = the exact slice-20 dev pool (s19.load_dev): every dev_pool
# row, fast-path included. The guard is evaluated on what slice 20 measured.
meta, examples = slice18_sparse.load_data()
dev = slice18_sparse.filter_dev_pool(examples)
print(f"dev pool (all dev_pool rows): {len(dev)} rows")
print(f"corpus meta: {meta.get('dev_count', '?')} dev rows declared, "
f"{meta.get('route_counts', {}).get('action', '?')} action declared")
n_texts = [slice18_sparse.normalize_match_text(e["text"]) for e in dev]
rows = []
by_route = {}
by_family = {}
for i, (e, nt) in enumerate(zip(dev, n_texts)):
tags = sorted(set(e.get("tags", [])))
route = e["route"]
fam = slice19_main.family_of(set(tags))
by_route[route] = by_route.get(route, 0) + 1
by_family[fam] = by_family.get(fam, 0) + 1
rows.append({
"idx": i,
"text": e["text"],
"n_text": nt,
"route": route,
"y": 1 if route == "action" else 0,
"tags": tags,
"cv_fold": e["cv_fold"],
"split_group": e["split_group"],
"source_id": e["source_id"],
})
print("routes:", by_route)
print("families:", by_family)
# ── pairs (slice-19 builder, exact population) ─────────────────────────
ldev = [{
"text_orig": nt,
"route": r["route"],
"y": r["y"],
"cv_fold": r["cv_fold"],
"tags": set(r["tags"]),
"source_id": r["source_id"],
} for r, nt in zip(rows, n_texts)]
pairs = slice19_main.build_pairs(ldev, n_texts)
print(f"pairs: {len(pairs)}")
# ── slice-18 "both" grouped-CV OOF proba, aligned to row index ────────
y = [1 if r["route"] == "action" else 0 for r in rows]
folds = [r["cv_fold"] for r in rows]
X, _vec = slice18_sparse.build_features(n_texts, "both")
print(f"sparse 'both' X: {X.shape}")
yb = np.array(y)
folds_arr = np.array(folds)
idx_proba = {}
for te_fold in sorted(set(folds)):
tr = folds_arr != te_fold
te = folds_arr == te_fold
clf = slice18_sparse.LogisticRegression(
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yb[tr])
p = clf.predict_proba(X[te])[:, 1]
te_idx = np.where(te)[0]
for k, i in enumerate(te_idx):
idx_proba[int(i)] = float(p[k])
assert len(idx_proba) == len(rows)
sparse_oof = [{"idx": i, "proba": idx_proba[i]} for i in range(len(rows))]
pred = [1 if idx_proba[i] >= 0.5 else 0 for i in range(len(rows))]
tp = sum(1 for i in range(len(rows)) if y[i] == 1 and pred[i] == 1)
fp = sum(1 for i in range(len(rows)) if y[i] == 0 and pred[i] == 1)
fn = sum(1 for i in range(len(rows)) if y[i] == 1 and pred[i] == 0)
print(f"sparse both OOF @0.5: P={tp/max(tp+fp,1):.3f} R={tp/max(tp+fn,1):.3f} "
f"FA={fp} ({fp/len(rows):.4f})")
pred21 = [1 if idx_proba[i] >= SPARSE_THRESHOLD else 0 for i in range(len(rows))]
tp = sum(1 for i in range(len(rows)) if y[i] == 1 and pred21[i] == 1)
fp = sum(1 for i in range(len(rows)) if y[i] == 0 and pred21[i] == 1)
fn = sum(1 for i in range(len(rows)) if y[i] == 1 and pred21[i] == 0)
print(f"sparse both OOF @{SPARSE_THRESHOLD}: P={tp/max(tp+fp,1):.3f} "
f"R={tp/max(tp+fn,1):.3f} FA={fp} ({fp/len(rows):.4f})")
os.makedirs(OUT_DIR, exist_ok=True)
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
json.dump(rows, f, ensure_ascii=False, indent=1)
with open(os.path.join(OUT_DIR, "pairs.json"), "w") as f:
json.dump([[c, a] for c, a in pairs], f)
with open(os.path.join(OUT_DIR, "sparse_oof.json"), "w") as f:
json.dump(sparse_oof, f)
print(f"wrote {OUT_DIR}/{{pool,pairs,sparse_oof}}.json")
if __name__ == "__main__":
main()
@@ -0,0 +1,65 @@
package main
import (
"fmt"
"os"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/router/semantic"
)
// headsMain runs the deployed cascade minus the resident LLM: stage-0
// grammars → routing heads (fine-tuned e5 copy + softmax, router_heads.onnx,
// 0.6 decline threshold) → ONNX-embedder nearest-centroid classifier →
// 0.55 confidence gate. This is what a production turn takes when the model
// server is out (docs/routing.md: pickLLMRouter degrades to the classifier).
//
// The classifier is seeded from models/seeds like the daemon's seedClassifier,
// embedded with the real multilingual-e5-small model rather than the block
// hash, so this is the closest headless reproduction of the authoritative
// router output the slice-22 report can run.
//
// Requires the ONNX model files and a libonnxruntime.so. Pass the library via
// the MAVEN_ONNX_LIB environment variable, exactly as the daemon does.
func headsMain(poolPath, outPath string) {
lib := os.Getenv("MAVEN_ONNX_LIB")
if lib == "" {
fmt.Fprintln(os.Stderr, "heads mode needs MAVEN_ONNX_LIB pointing at libonnxruntime.so")
os.Exit(2)
}
const (
embedModel = "models/embedder/multilingual-e5-small/model_quantized.onnx"
tokPath = "models/embedder/multilingual-e5-small/tokenizer.json"
headsModel = "models/embedder/router-heads/router_heads.onnx"
)
emb, err := router.NewONNXEmbedder(embedModel, tokPath, lib)
if err != nil {
fmt.Fprintf(os.Stderr, "heads: embedder: %v\n", err)
os.Exit(1)
}
defer emb.Close()
cls := router.NewClassifier(emb)
seedClassifier(cls)
heads, err := router.NewRouterHeads(headsModel, tokPath)
if err != nil {
fmt.Fprintf(os.Stderr, "heads: %v\n", err)
os.Exit(1)
}
defer heads.Close()
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
r := router.New(router.Config{
Grammars: router.StageZeroGrammars(acts),
Classifier: cls,
Extractor: router.Extractor{
Time: router.StubDateTimeParser{},
Acts: acts,
Facts: router.DefaultFactParser{},
},
Threshold: 0.55,
Heads: heads,
})
runOverPool(r, poolPath, outPath)
}
@@ -0,0 +1,87 @@
package main
import (
"bufio"
"context"
"log"
"os"
"path/filepath"
"sort"
"strings"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/router/semantic"
)
// Seed loading replicated from cmd/mavend/voicewire.go (seedClassifier) and
// internal/router/semantic/helpers_test.go, which build the same classifier
// from models/seeds/<intent>.txt. The daemon and the eval fixture must agree
// on the seeds; so must a measurement.
const seedDir = "models/seeds"
var seedIntents = []router.Intent{
router.IntentAct, router.IntentReminder, router.IntentFact,
router.IntentNote, router.IntentQuery, router.IntentChat, router.IntentSystem,
}
// buildMinimalRouter reproduces internal/router/semantic/buildMinimalRouter:
// the daemon's grammar set, a hash-embedder classifier seeded from
// models/seeds, and the deployed 0.55 threshold. Deterministic and
// reproducible. The ONNX embedder and the routing heads score elsewhere;
// this is the floor the eval fixture reports as the legacy baseline.
func buildMinimalRouter() *router.Router {
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
cls := router.NewClassifier(router.NewHashEmbedder(1024))
seedClassifier(cls)
return router.New(router.Config{
Grammars: router.StageZeroGrammars(acts),
Classifier: cls,
Extractor: router.Extractor{
Time: router.StubDateTimeParser{},
Acts: acts,
Facts: router.DefaultFactParser{},
},
Threshold: 0.55,
})
}
func seedClassifier(c *router.Classifier) {
// Walk up to find models/seeds like the daemon's seedPath, so the program
// can run from any depth of the repo tree.
dir := seedDir
for i := 0; i < 5; i++ {
if st, err := os.Stat(dir); err == nil && st.IsDir() {
break
}
dir = filepath.Join("..", dir)
}
ctx := context.Background()
total := 0
for _, intent := range seedIntents {
path := filepath.Join(dir, string(intent)+".txt")
f, err := os.Open(path)
if err != nil {
log.Printf("legacy: open seed %s: %v", path, err)
continue
}
sc := bufio.NewScanner(f)
lines := []string{}
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
lines = append(lines, line)
}
f.Close()
sort.Strings(lines)
for _, line := range lines {
if err := c.AddExample(ctx, intent, line); err != nil {
log.Printf("legacy: seed %s %q: %v", intent, line, err)
continue
}
total++
}
}
log.Printf("legacy: loaded %d seed examples from %s", total, dir)
}
@@ -0,0 +1,192 @@
package main
import (
"bufio"
"context"
"encoding/json"
"flag"
"fmt"
"os"
"time"
"github.com/kami/maven/internal/router"
)
// Legacy-baseline runner for slice 22: run the actual router cascade (stage 0
// grammars → hash-embedder nearest-centroid classifier → 0.55 confidence gate)
// over the frozen residual non-action dev pool and project each decision into
// the five-way non-action semantic space.
//
// Projection rules (the daemon's behaviour, not just ScoreLegacy's):
// - route error → uncertain
// - Clarify=true (stage 3) → uncertain: the daemon asks, it does not commit
// to a semantic bucket
// - chat/query/fact+note/system → conversation/knowledge/memory_write/system
// - act/reminder on a trusted non-action row → class "action" recorded
// VERBATIM with illegal_action_prediction=true; never mapped to uncertain
// - anything else → uncertain
//
// Reads /tmp/mvn-s22/pool.json (emit step) and writes /tmp/mvn-s22/legacy.json
// with both the raw decision fields and the projected class, plus a summary
// printout. No embedding is recomputed and no label is changed.
type poolRow struct {
IDX int `json:"idx"`
Text string `json:"text"`
NText string `json:"n_text"`
Route string `json:"route"`
Tags []string `json:"tags"`
CVFold int `json:"cv_fold"`
SplitGroup string `json:"split_group"`
FamilyID string `json:"family_id"`
SourceID string `json:"source_id"`
}
type legacyRow struct {
IDX int `json:"idx"`
Text string `json:"text"`
Route string `json:"route"`
Intent string `json:"intent"`
Class string `json:"class"`
Illegal bool `json:"illegal_action_prediction"`
Confidence float64 `json:"confidence"`
Stage int `json:"stage"`
Clarify bool `json:"clarify"`
Producer string `json:"producer"`
Error string `json:"error,omitempty"`
SourceID string `json:"source_id"`
}
func main() {
var mode, poolPath, outPath string
flag.StringVar(&mode, "mode", "legacy", "baseline mode: legacy (hash classifier) or heads (ONNX cascade minus LLM)")
flag.StringVar(&poolPath, "pool", "/tmp/mvn-s22/pool.json", "emit-step pool.json")
flag.StringVar(&outPath, "out", "/tmp/mvn-s22/legacy.json", "output path")
flag.Parse()
switch mode {
case "legacy":
legacyMain(poolPath, outPath)
case "heads":
headsMain(poolPath, outPath)
default:
fmt.Fprintf(os.Stderr, "unknown -mode %q\n", mode)
os.Exit(2)
}
}
func legacyMain(poolPath, outPath string) {
runOverPool(buildMinimalRouter(), poolPath, outPath)
}
func runOverPool(r *router.Router, poolPath, outPath string) {
raw, err := os.ReadFile(poolPath)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
var rows []poolRow
if err := json.Unmarshal(raw, &rows); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
ctx := context.Background()
now := time.Now()
out := make([]legacyRow, 0, len(rows))
classCount := map[string]int{}
for _, pr := range rows {
d, err := r.Route(ctx, router.NormalizedInput{Text: pr.Text}, now)
lr := legacyRow{
IDX: pr.IDX,
Text: pr.Text,
Route: pr.Route,
SourceID: pr.SourceID,
}
if err != nil {
lr.Class = "uncertain"
lr.Error = err.Error()
} else {
lr.Intent = string(d.Intent)
lr.Confidence = d.Confidence
lr.Stage = d.Stage
lr.Clarify = d.Clarify
lr.Producer = string(d.Producer)
}
lr.Class, lr.Illegal = project(d, err)
classCount[lr.Class]++
out = append(out, lr)
}
if err := writeJSON(outPath, out); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("legacy baseline over %d residual non-action rows:\n", len(out))
for _, c := range []string{"conversation", "knowledge", "memory_write", "system", "uncertain", "action"} {
fmt.Printf(" %-14s %d (%.1f%%)\n", c, classCount[c], 100*float64(classCount[c])/float64(len(out)))
}
fmt.Printf(" illegal_action_prediction: %d\n", classCount["action"])
// Grammar hits inside a corpus-residual population would be a
// corpus/harness disagreement worth telling the report about: the corpus
// marked each row not-fast-path-resolved, so a current stage-0 rule
// resolving it means the corpus's fast-path mirror is stale or a grammar
// landed after the corpus froze.
gh := 0
ghByRoute := map[string]int{}
ghByIntent := map[string]int{}
for _, lr := range out {
if lr.Producer == string(router.RouteProducerGrammar) {
gh++
ghByRoute[lr.Route]++
ghByIntent[lr.Intent]++
}
}
fmt.Printf(" stage-0 grammar hits: %d\n", gh)
if gh > 0 {
fmt.Printf(" by ground-truth route: %v\n", ghByRoute)
fmt.Printf(" by grammar intent: %v\n", ghByIntent)
}
}
// project maps the router's authoritative output into the five-way non-action
// space, or to the "action" bucket verbatim when the router calls an act or a
// reminder on a non-action row.
func project(d router.Decision, err error) (string, bool) {
if err != nil {
return "uncertain", false
}
if d.Clarify {
return "uncertain", false
}
switch d.Intent {
case router.IntentChat:
return "conversation", false
case router.IntentQuery:
return "knowledge", false
case router.IntentFact, router.IntentNote:
return "memory_write", false
case router.IntentSystem:
return "system", false
case router.IntentAct, router.IntentReminder:
return "action", true
default:
return "uncertain", false
}
}
func writeJSON(path string, v any) error {
fh, err := os.Create(path)
if err != nil {
return err
}
defer fh.Close()
w := bufio.NewWriter(fh)
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
if err := enc.Encode(v); err != nil {
return err
}
return w.Flush()
}
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""
Slice 22 emit: five-way residual non-action semantic router data files
========================================================================
Slice 21 accepted the deterministic execution-frame guard (docs/evals/
2026-09-07-execution-frame-guard.md). Slice 22 returns to the coarse non-action
router that the guard hands to: after TryFastPath misses and the guard passes,
the remaining utterance is one of five non-action semantics conversation,
knowledge, memory_write, system, uncertain. Action rows never reach this
router; they are usable only as out-of-domain probes, never in primary metrics.
This script only repackages the frozen dev pool for the Go legacy baseline and
the Python experiment. It reuses slice 18's loader/filters and slice 19's
normalizers verbatim, so the population here is the same one slices 18-21
measured. It writes:
/tmp/mvn-s22/pool.json residual non-action dev rows: idx, text, n_text,
route, tags, cv_fold, split_group, family_id,
source_id (1652 rows)
/tmp/mvn-s22/ood.json residual ACTION dev rows (766): same shape; OOD
probes only, never primary metrics
/tmp/mvn-s22/stats.json population summary (routes, families, folds)
idx is the row's position among dev_pool rows in dev-pool order, so the Python
experiment can align the embedding vectors from /tmp/mvn-experiment/embeddings.json
by index exactly as slice19.load_dev does.
No training happens here and no label is changed.
Population (verified 2026-09-08 from the frozen file):
dev_pool 2490
dev residual 2418 (= dev_pool minus fast_path_resolved)
residual non-action 1652 knowledge 715 / memory_write 553 / system 184 /
uncertain 107 / conversation 93
residual action 766 (OOD probes only)
"""
import json
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import slice18_sparse # noqa: E402 (normalize_match_text, load_data, filters)
OUT_DIR = "/tmp/mvn-s22"
ROUTES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
def main():
meta, examples = slice18_sparse.load_data()
dev = slice18_sparse.filter_dev_pool(examples)
print(f"dev pool: {len(dev)} rows "
f"(meta declares dev_count={meta.get('dev_count')})")
rows = []
for i, e in enumerate(dev):
if not e["fast_path_resolved"]:
rows.append({
"idx": i,
"text": e["text"],
"n_text": slice18_sparse.normalize_match_text(e["text"]),
"route": e["route"],
"tags": sorted(set(e.get("tags", []))),
"cv_fold": e["cv_fold"],
"split_group": e["split_group"],
"family_id": e["family_id"],
"source_id": e["source_id"],
})
na = [r for r in rows if r["route"] != "action"]
ood = [r for r in rows if r["route"] == "action"]
print(f"residual rows: {len(rows)} non-action: {len(na)} action(OOD): {len(ood)}")
by_route = {}
for r in na:
by_route[r["route"]] = by_route.get(r["route"], 0) + 1
print("routes:", by_route)
assert sum(by_route.values()) == len(na)
assert set(ROUTES) == set(by_route), "route set must be the five-way"
by_family = {}
for r in na:
by_family[r["family_id"]] = by_family.get(r["family_id"], 0) + 1
by_fold = {}
for r in na:
by_fold[r["cv_fold"]] = by_fold.get(r["cv_fold"], 0) + 1
print(f"family_ids: {len(by_family)} split_groups: {len(set(r['split_group'] for r in na))}")
print("folds:", by_fold)
os.makedirs(OUT_DIR, exist_ok=True)
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
json.dump(na, f, ensure_ascii=False, indent=1)
with open(os.path.join(OUT_DIR, "ood.json"), "w") as f:
json.dump(ood, f, ensure_ascii=False, indent=1)
with open(os.path.join(OUT_DIR, "stats.json"), "w") as f:
json.dump({
"dev_count": len(dev),
"residual_count": len(rows),
"non_action_count": len(na),
"action_ood_count": len(ood),
"routes": by_route,
"family_ids": len(by_family),
"folds": by_fold,
"top_family": dict(sorted(by_family.items(), key=lambda kv: -kv[1])[:15]),
}, f, ensure_ascii=False, indent=1)
print(f"wrote {OUT_DIR}/{{pool,ood,stats}}.json")
if __name__ == "__main__":
main()
@@ -0,0 +1,560 @@
#!/usr/bin/env python3
"""
Slice 22: five-way residual non-action semantic router (experiment)
==================================================================
After TryFastPath misses and the ExecutionFrameGuard passes, the residual
utterance is one of five non-action semantics: conversation, knowledge,
memory_write, system, uncertain. This measures whether the deployed e5-small
embeddings (384-d, query-prefixed, mean-pooled, L2, frozen) fed to a linear
softmax head suffice, and how they compare to the legacy router, to floors,
and to the deployed routing heads.
Population: the frozen dev-pool residual non-action rows (1652; the pool
written by slice22_emit.py). Action rows (766) are out-of-domain probes only.
Metrics written to /tmp/mvn-s22/results.json:
§1 population
§2 legacy baseline (legacy.json / legacy_heads.json): acc, macro-F1,
per-class P/R/F1, confusion, illegal_action_prediction count
§3 e5-linear primary head: C grid, grouped CV OOF, per-fold P/R/F1 +
variance + composition
§4 floors: majority, centroid (cosine nearest-mean), sparse word+char
TF-IDF logistic (slice18 builder), all grouped CV
§5 route-family (family_id) leave-family-out
§6 knowledge vs memory_write: matched pairs (water/homelab/task) ordering
§7 uncertain as an explicit class: P/R/F1 + top confusions
§8 OOF confidence: max-softmax correct/wrong, ECE, log-loss, Brier,
coverage/accuracy/macro-F1 abstention curves (no threshold chosen)
§9 action OOD probes: fold models applied to the 766 action rows
§10 artifact cost: head params, serialized bytes, incremental head latency
No corpus label is changed. No frozen-holdout rows are inspected.
"""
import json
import os
import sys
import time
import numpy as np
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import slice18_sparse # noqa: E402
import slice19_main # noqa: E402
EMB_PATH = "/tmp/mvn-experiment/embeddings.json"
OUT_DIR = "/tmp/mvn-s22"
CLASSES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
CLASS_PREFIX = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
C_GRID = [0.1, 1.0, 10.0]
# Route-family holdouts the report calls out by name (slice-22 brief): every
# family that is not part of the shared subject inventory on either side.
HOLDOUT_GROUPS = {
"capability": ["knowledge:capability-ha", "knowledge:capability-tool"],
"world": ["knowledge:world-def", "knowledge:world-explain"],
"calendar": ["knowledge:calendar", "knowledge:calendar-time", "knowledge:calendar-next"],
"recall": ["knowledge:recall-fact", "knowledge:recall-note", "knowledge:recall-possessive"],
"fact": ["fact:meal", "fact:water", "fact:sleep", "fact:shower", "fact:break", "fact:pills", "fact:exercise"],
"note": ["note:idea", "note:homelab", "note:task"],
"remember": ["free:remember"],
"system": None, # all system:*
"conversation": None,
"uncertain": None,
}
def load_pool_and_embeds():
with open(os.path.join(OUT_DIR, "pool.json")) as f:
pool = json.load(f)
meta, examples = slice18_sparse.load_data()
dev = slice18_sparse.filter_dev_pool(examples)
by_idx = {e["dev_idx"]: e for e in dev} if "dev_idx" in dev[0] else None
# pool rows carry idx = position among dev_pool rows in dev order
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
for i, e in enumerate(dev)}
for r in pool:
r["emb"] = emb_by_idx[r["idx"]]
r["y"] = r["route"]
return pool, meta
def oof_proba_grouped(X, y, folds, C=1.0):
"""Grouped OOF probability matrix (n×5, class order CLASSES)."""
y_idx = np.array([CLASSES.index(c) for c in y])
folds = np.asarray(folds)
proba = np.zeros((len(y_idx), len(CLASSES)))
for te_fold in sorted(set(folds.tolist())):
tr = folds != te_fold
te = folds == te_fold
clf = slice18_sparse.LogisticRegression(
C=C, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], y_idx[tr])
proba[te] = clf.predict_proba(X[te])
return proba
def cls_metrics(yt, yp):
import sklearn.metrics as m
yt = np.asarray(yt)
yp = np.asarray(yp)
if yt.dtype != np.int64 and yt.dtype != np.int32:
yt = np.array([CLASSES.index(c) for c in yt])
if yp.dtype != np.int64 and yp.dtype != np.int32:
yp = np.array([CLASSES.index(c) for c in yp])
labels = list(range(len(CLASSES)))
n = len(yt)
acc = m.accuracy_score(yt, yp)
macro = m.f1_score(yt, yp, average="macro", labels=labels, zero_division=0)
pr, rc, f1, sup = m.precision_recall_fscore_support(
yt, yp, labels=labels, zero_division=0)
per = {c: {"p": float(pr[i]), "r": float(rc[i]), "f1": float(f1[i]), "n": int(sup[i])}
for i, c in enumerate(CLASSES)}
conf = m.confusion_matrix(yt, yp, labels=labels).tolist()
return {"n": n, "acc": acc, "macro_f1": macro, "per_class": per, "confusion": conf}
def fold_report(yt, proba, folds, true_y):
out = {}
folds_arr = np.asarray(folds)
comp = {}
for f in sorted(set(folds_arr.tolist())):
mask = folds_arr == f
yt_f = [CLASSES.index(y) for y in true_y[mask]]
comp[f] = {c: int((np.array(true_y[mask]) == c).sum()) for c in CLASSES}
per_fold = {}
for f in sorted(set(folds_arr.tolist())):
mask = folds_arr == f
yp = proba[mask].argmax(1).tolist()
m = cls_metrics([yt[i] for i in np.where(mask)[0].tolist()], yp)
per_fold[f] = {"acc": m["acc"], "macro_f1": m["macro_f1"]}
out["composition"] = comp
out["per_fold"] = per_fold
accs = [v["acc"] for v in per_fold.values()]
macros = [v["macro_f1"] for v in per_fold.values()]
out["acc_mean"] = float(np.mean(accs))
out["acc_std"] = float(np.std(accs))
out["macro_f1_mean"] = float(np.mean(macros))
out["macro_f1_std"] = float(np.std(macros))
return out
def ece(yt, proba, n_bins=15):
conf = proba.max(1)
pred = proba.argmax(1)
acc = (pred == yt).astype(float)
bins = np.linspace(0, 1, n_bins + 1)
tot = 0.0
details = []
counts = 0
for i in range(n_bins):
lo, hi = bins[i], bins[i + 1]
m = (conf >= lo) & (conf < hi) if i < n_bins - 1 else conf >= lo
if m.sum() == 0:
continue
acc_m = acc[m].mean()
conf_m = conf[m].mean()
w = m.sum() / len(conf)
tot += w * abs(acc_m - conf_m)
counts += int(m.sum())
details.append({"bin": i, "lo": lo, "hi": hi, "conf": float(conf_m),
"acc": float(acc_m), "n": int(m.sum())})
return {"ece": float(tot), "n_bins": n_bins, "counted": counts, "bins": details}
def main():
pool, meta = load_pool_and_embeds()
pool.sort(key=lambda r: r["idx"])
print(f"pool: {len(pool)} rows")
from sklearn.metrics import brier_score_loss, log_loss
report = {"population": {}, "legacy": {}, "e5_linear": {}, "floors": {},
"family_holdouts": {}, "kmw": {}, "uncertain": {}, "confidence": {},
"ood": {}, "artifact": {}}
# ── §1 population ──────────────────────────────────────────────────────
cnt = {}
for r in pool:
cnt[r["y"]] = cnt.get(r["y"], 0) + 1
report["population"] = {
"n": len(pool),
"routes": cnt,
"family_ids": len(set(r["family_id"] for r in pool)),
"split_groups": len(set(r["split_group"] for r in pool)),
"folds": {str(f): int(sum(1 for r in pool if r["cv_fold"] == f)) for f in sorted(set(r["cv_fold"] for r in pool))},
"corpus": {k: v for k, v in meta.items() if k in
("dev_count", "residual_count", "fast_path_count",
"dimension", "embedder_id", "input_template", "pooling", "normalization")},
}
print("\n§1 population:", report["population"])
X = np.vstack([r["emb"] for r in pool])
y = np.array([r["y"] for r in pool])
folds = np.array([r["cv_fold"] for r in pool])
yt = np.array([CLASSES.index(c) for c in y])
# ── §2 legacy baselines ────────────────────────────────────────────────
import collections
for tag, fname in [("hash", "legacy.json"), ("heads", "legacy_heads.json")]:
path = os.path.join(OUT_DIR, fname)
if not os.path.exists(path):
continue
leg = json.load(open(path))
leg_by_idx = {r["idx"]: r for r in leg}
yp_leg = []
illegal = []
for r in pool:
lr = leg_by_idx[r["idx"]]
if lr["illegal_action_prediction"]:
illegal.append(lr)
yp_leg.append("action")
else:
yp_leg.append(lr["class"])
yp_leg = np.array(yp_leg)
# five-way: an 'action' prediction is an error (outside the label set)
yp5 = np.array([("uncertain" if p == "action" else p) for p in yp_leg])
m = cls_metrics(y, yp5)
m["illegal_action_prediction"] = len(illegal)
m["illegal_cases"] = [{"idx": i["idx"], "text": i["text"], "route": i["route"],
"intent": i["intent"], "producer": i["producer"],
"confidence": i["confidence"]} for i in illegal]
# per-cell confusion also shows 'action' column
conf_counts = collections.Counter(zip(y, yp_leg))
m["confusion_with_action"] = {f"{a}->{b}": int(c) for (a, b), c in conf_counts.items()}
report["legacy"][tag] = m
print(f"\n§2 legacy ({tag}) acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f} "
f"illegal={len(illegal)}")
for c in CLASSES:
p = m["per_class"][c]
print(f" {c:<14} P={p['p']:.3f} R={p['r']:.3f} F1={p['f1']:.3f} n={p['n']}")
# grammar-pure residual: rows not resolved by any current stage-0 grammar
if os.path.exists(os.path.join(OUT_DIR, "legacy.json")):
leg = json.load(open(os.path.join(OUT_DIR, "legacy.json")))
gh = {r["idx"] for r in leg if r["producer"] == "grammar"}
gp_mask = np.array([r["idx"] not in gh for r in pool])
report["grammar_drift"] = {
"grammar_hits_in_pool": len(gh),
"grammar_pure_n": int(gp_mask.sum()),
}
# ── §3 e5-linear primary head ─────────────────────────────────────────
print("\n§3 e5-linear")
bestC, bestMac = 1.0, -1.0
grid = {}
for C in C_GRID:
p = oof_proba_grouped(X, y, folds, C=C)
mp = cls_metrics(y, p.argmax(1).tolist())
grid[float(C)] = {"acc": mp["acc"], "macro_f1": mp["macro_f1"]}
print(f" C={C} acc={mp['acc']:.4f} macroF1={mp['macro_f1']:.4f}")
if mp["macro_f1"] > bestMac:
bestMac, bestC = mp["macro_f1"], C
print(f" -> best C={bestC}")
p_best = oof_proba_grouped(X, y, folds, C=bestC)
m_best = cls_metrics(y, p_best.argmax(1).tolist())
m_best["C"] = bestC
m_best["C_grid"] = grid
m_best["folds"] = fold_report(yt, p_best, folds, y)
report["e5_linear"] = m_best
for f, v in m_best["folds"]["per_fold"].items():
print(f" fold {f}: acc={v['acc']:.4f} macroF1={v['macro_f1']:.4f}")
print(f" fold acc mean={m_best['folds']['acc_mean']:.4f} "
f"std={m_best['folds']['acc_std']:.4f}; "
f"macroF1 mean={m_best['folds']['macro_f1_mean']:.4f} "
f"std={m_best['folds']['macro_f1_std']:.4f}")
for c in CLASSES:
p_ = m_best["per_class"][c]
print(f" {c:<14} P={p_['p']:.3f} R={p_['r']:.3f} F1={p_['f1']:.3f} n={p_['n']}")
# grammar-pure sensitivity for the primary head
if "grammar_drift" in report:
mp_gp = cls_metrics(y[gp_mask], p_best[gp_mask].argmax(1).tolist())
report["e5_linear"]["grammar_pure"] = {
"acc": mp_gp["acc"], "macro_f1": mp_gp["macro_f1"], "n": int(gp_mask.sum())}
# ── §4 floors ─────────────────────────────────────────────────────────
print("\n§4 floors")
# majority floor
maj = CLASSES.index("knowledge")
ym = np.full(len(y), maj)
mm = cls_metrics(y, ym)
report["floors"]["majority"] = {"acc": mm["acc"], "macro_f1": mm["macro_f1"],
"per_class": mm["per_class"]}
print(f" majority (predict {CLASSES[maj]}): acc={mm['acc']:.4f} macroF1={mm['macro_f1']:.4f}")
# centroid floor: cosine to per-class mean of the training folds' embeddings
cf_proba = np.zeros((len(yt), len(CLASSES)))
folds_arr = np.asarray(folds)
for te_fold in sorted(set(folds_arr.tolist())):
tr = folds_arr != te_fold
te = folds_arr == te_fold
centroids = []
for c in CLASSES:
idxs = np.where(tr & (y == c))[0]
ctr = X[idxs].mean(axis=0)
ctr = ctr / np.linalg.norm(ctr)
centroids.append(ctr)
Cm = np.vstack(centroids)
sims = X[te] @ Cm.T
cf_proba[te] = sims
yc = cf_proba.argmax(1)
# accuracy + macroF1 with the same 5-way
mc = cls_metrics(y, yc.tolist())
report["floors"]["centroid"] = {"acc": mc["acc"], "macro_f1": mc["macro_f1"],
"per_class": mc["per_class"]}
print(f" centroid cosine: acc={mc['acc']:.4f} macroF1={mc['macro_f1']:.4f}")
# sparse word+char logistic (slice18 builder, grouped CV, five-way)
texts = [r["n_text"] for r in pool]
Xs, _vec = slice18_sparse.build_features(texts, "both")
psp = np.zeros((len(yt), len(CLASSES)))
for te_fold in sorted(set(folds_arr.tolist())):
tr = folds_arr != te_fold
te = folds_arr == te_fold
clf = slice18_sparse.LogisticRegression(
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(Xs[tr], yt[tr])
psp[te] = clf.predict_proba(Xs[te])
msp = cls_metrics(y, psp.argmax(1).tolist())
report["floors"]["sparse_word_char"] = {
"acc": msp["acc"], "macro_f1": msp["macro_f1"], "per_class": msp["per_class"],
"vocab": slice18_sparse.vocab_size(_vec)}
print(f" sparse both: acc={msp['acc']:.4f} macroF1={msp['macro_f1']:.4f} "
f"vocab={report['floors']['sparse_word_char']['vocab']}")
# ── §5 route-family holdouts ─────────────────────────────────────────
print("\n§5 route-family holdouts")
fam = np.array([r["family_id"] for r in pool])
holdouts = {}
all_fams = sorted(set(fam.tolist()))
for grp, fams in HOLDOUT_GROUPS.items():
if fams is None:
fams = [f for f in all_fams if f.startswith(grp + ":")]
mask = np.isin(fam, fams)
if mask.sum() == 0:
continue
tr = ~mask
clf = slice18_sparse.LogisticRegression(
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yt[tr])
ypgrp = clf.predict(X[mask])
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypgrp.tolist())
m["families"] = fams
m["rows"] = int(mask.sum())
holdouts[grp] = {"acc": m["acc"], "macro_f1": m["macro_f1"], "n": int(mask.sum()),
"per_class": m["per_class"]}
print(f" {grp:<14} n={m['rows']} acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f}")
# full leave-one-family-out summary
lofo_accs = []
lofo_f1s = []
for f in all_fams:
mask = fam == f
tr = ~mask
clf = slice18_sparse.LogisticRegression(
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yt[tr])
ypf = clf.predict(X[mask])
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypf.tolist())
lofo_accs.append(m["acc"])
lofo_f1s.append(m["macro_f1"])
holdouts["_all_49_lo_"] = {"n_families": len(all_fams),
"acc_mean": float(np.mean(lofo_accs)),
"macro_f1_mean": float(np.mean(lofo_f1s))}
report["family_holdouts"] = holdouts
print(f" leave-one-family-out over {len(all_fams)} families: "
f"acc mean={np.mean(lofo_accs):.4f} macroF1 mean={np.mean(lofo_f1s):.4f}")
# ── §6 knowledge vs memory_write ─────────────────────────────────────
print("\n§6 knowledge vs memory_write")
# reuse e5-linear OOF: does the model put the higher probability on the
# right side (memory_write for a write, knowledge for a recall)?
conf_km = np.zeros((2, 2))
pk = p_best[:, CLASSES.index("knowledge")]
pmw = p_best[:, CLASSES.index("memory_write")]
for i in range(len(yt)):
t = y[i]
if t == "knowledge":
conf_km[0, 1 if pmw[i] > pk[i] else 0] += 1
elif t == "memory_write":
conf_km[1, 1 if pmw[i] >= pk[i] else 0] += 1
report["kmw"] = {"confusion_p_ordered": conf_km.tolist()}
# matched pairs with shared subject lexemes, corpus-justified
def build_pairs(subject, fam_k, fam_mw):
kr = [r for r in pool if r["family_id"] in fam_k]
mr = [r for r in pool if r["family_id"] in fam_mw]
pairs = []
for mw in mr:
for k in kr:
if subject in mw["n_text"] and subject in k["n_text"]:
pairs.append((mw["idx"], k["idx"], mw["n_text"], k["n_text"]))
return pairs
sets = {
"water": build_pairs("вод", ["knowledge:recall-fact"], ["fact:water"]),
"homelab": build_pairs("dns", ["knowledge:homelab-status"], ["note:homelab"])
+ build_pairs("сервер", ["knowledge:homelab-status"], ["note:homelab"])
+ build_pairs("vlan", ["knowledge:homelab-status"], ["note:homelab"]),
"task": build_pairs("задач", ["knowledge:task-check", "knowledge:deadline"],
["note:task"]),
}
idx_of = {r["idx"]: i for i, r in enumerate(pool)}
pair_rep = {}
for name, pairs in sets.items():
if not pairs:
continue
ok = 0
margins = []
bad = []
for mi, ki, mx, kx in pairs:
mi_i, ki_i = idx_of[mi], idx_of[ki]
# MW row should get a higher memory_write probability than the K row
mk = (pmw[mi_i] + 0.0)
if pmw[mi_i] > pmw[ki_i]:
ok += 1
else:
bad.append((mx[:46], round(float(pmw[mi_i]), 3), kx[:46], round(float(pmw[ki_i]), 3)))
margins.append(pmw[mi_i] - pmw[ki_i])
pair_rep[name] = {
"pairs": len(pairs),
"mw_over_k_order_acc": ok / len(pairs),
"mean_margin": float(np.mean(margins)),
"reversed_examples": bad[:6],
}
print(f" {name}: pairs={len(pairs)} order_acc={ok/len(pairs):.3f} "
f"mean_margin={np.mean(margins):+.3f}")
report["kmw"]["matched_pairs"] = pair_rep
# ── §7 uncertain as explicit class ────────────────────────────────────
print("\n§7 uncertain")
up = m_best["per_class"]["uncertain"]
uc = m_best["confusion"][CLASSES.index("uncertain")]
report["uncertain"] = {
"per_class": up,
"row_from_uncertain": {CLASSES[j]: int(uc[j]) for j in range(5)},
"row_to_uncertain": {CLASSES[j]: int(m_best["confusion"][j][CLASSES.index("uncertain")])
for j in range(5)},
}
print(f" uncertain n={up['n']} P={up['p']:.3f} R={up['r']:.3f} F1={up['f1']:.3f}")
print(" wrong-→label pulled from uncertain:", report["uncertain"]["row_from_uncertain"])
print(" →uncertain pulled from:", report["uncertain"]["row_to_uncertain"])
# ── §8 OOF confidence / calibration / abstention ─────────────────────
print("\n§8 confidence / calibration")
conf = p_best.max(1)
right = (p_best.argmax(1) == yt)
cer = {
"correct_conf_mean": float(conf[right].mean()),
"correct_conf_median": float(np.median(conf[right])),
"wrong_conf_mean": float(conf[~right].mean()),
"wrong_conf_median": float(np.median(conf[~right])),
"ece": ece(yt, p_best)["ece"],
"ece_bins": ece(yt, p_best)["bins"],
"log_loss": float(log_loss(yt, p_best, labels=[0, 1, 2, 3, 4])),
}
# Brier is label-set specific: one-vs-rest mean
briers = []
for i in range(5):
briers.append(brier_score_loss((yt == i).astype(int), p_best[:, i]))
cer["brier_macro"] = float(np.mean(briers))
report["confidence"] = cer
print(f" right conf mean={cer['correct_conf_mean']:.3f} "
f"wrong conf mean={cer['wrong_conf_mean']:.3f} ECE={cer['ece']:.4f}")
print(f" log_loss={cer['log_loss']:.4f} brier_macro={cer['brier_macro']:.4f}")
thr_grid = np.linspace(0.10, 0.98, 45)
abst = []
for t in thr_grid:
cov = (conf >= t).mean()
if cov == 0:
continue
keep = conf >= t
yt_k = yt[keep]
yp_k = p_best[keep].argmax(1)
mk_ = cls_metrics(yt_k.tolist(), yp_k.tolist())
abst.append({"threshold": round(float(t), 3), "coverage": float(cov),
"accuracy": mk_["acc"], "macro_f1": mk_["macro_f1"]})
report["confidence"]["abstention_curve"] = abst
print(" threshold | coverage | accuracy | macroF1 (first 6/45 + knee)")
for row in abst[::9]:
print(f" {row['threshold']:.2f} | {row['coverage']:.3f} | "
f"{row['accuracy']:.3f} | {row['macro_f1']:.3f}")
# ── §9 action OOD probes ──────────────────────────────────────────────
print("\n§9 action OOD")
ood_rows = [r for r in json.load(open(os.path.join(OUT_DIR, "ood.json")))]
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
for i, e in enumerate(slice18_sparse.filter_dev_pool(
slice18_sparse.load_data()[1]))}
Xo = np.vstack([emb_by_idx[r["idx"]] for r in ood_rows])
fold_models = []
for te_fold in sorted(set(folds_arr.tolist())):
tr = folds_arr != te_fold
clf = slice18_sparse.LogisticRegression(
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yt[tr])
fold_models.append(clf)
# OOD rows are not in folds; use the full-train model to keep it simple and
# comparable to the non-action in-fold behaviour
po = np.zeros((len(Xo), 5))
for clf in fold_models:
po += clf.predict_proba(Xo)
po /= len(fold_models)
ood_top = int(np.argmax(po.mean(0)))
ood_conf = po.max(1)
ood_pred = po.argmax(1)
top_dist = {CLASSES[i]: int((ood_pred == i).sum()) for i in range(5)}
confident_na = int((ood_conf > 0.9).sum())
report["ood"] = {
"n": len(ood_rows),
"top_class": CLASSES[int(ood_top)],
"top_class_dist": top_dist,
"conf_gt_0.9": confident_na,
"conf_gt_0.9_frac": float(confident_na / len(ood_rows)),
"conf_mean": float(ood_conf.mean()),
"conf_median": float(np.median(ood_conf)),
}
print(f" action OOD n={len(ood_rows)}: most-confident class={report['ood']['top_class']} "
f"dist={top_dist}")
print(f" conf>0.9: {confident_na} ({confident_na/len(ood_rows):.3f}) "
f"conf mean={report['ood']['conf_mean']:.3f}")
# ── §10 artifact cost ────────────────────────────────────────────────
print("\n§10 artifact")
n_params = len(CLASSES) * X.shape[1] + len(CLASSES)
fp32 = n_params * 4
report["artifact"] = {
"e5_dim": X.shape[1],
"head_params": n_params,
"head_fp32_bytes": fp32,
"head_fp32_kib": fp32 / 1024,
"head_int8_bytes": n_params,
}
# incremental latency of the linear head over a batch of 1 (µs)
clf = slice18_sparse.LogisticRegression(C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X, yt)
x1 = X[:1]
for _ in range(50):
clf.predict_proba(x1)
lat = []
for _ in range(2000):
t0 = time.perf_counter_ns()
clf.predict_proba(x1)
lat.append((time.perf_counter_ns() - t0) / 1e3)
lat = np.array(lat)
report["artifact"]["head_latency_us_mean"] = float(lat.mean())
report["artifact"]["head_latency_us_p50"] = float(np.median(lat))
print(f" head params={n_params} fp32={fp32/1024:.2f}KiB "
f"lat mean={lat.mean():.2f}us p50={np.median(lat):.2f}us")
with open(os.path.join(OUT_DIR, "results.json"), "w") as f:
json.dump(report, f, ensure_ascii=False, indent=1, default=float)
print(f"\nwrote {OUT_DIR}/results.json")
if __name__ == "__main__":
main()
@@ -0,0 +1,243 @@
// slice23 — fast-path metadata reconciliation diagnostic.
//
// Classifies every disagreement between the corpus's stored fast_path_resolved
// flag and what the production fast path derives today (TryFastPath over the
// stage-0 grammars with the experiment's act allowlist). Outputs a JSON
// decomposition and a console summary for docs/evals reports.
//
// Usage:
//
// go run ./cmd/semantic-router-experiment/slice23/ -out /tmp/mvn-s23/drift.json
package main
import (
"encoding/json"
"flag"
"fmt"
"os"
"sort"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/router/semantic"
)
type conflict struct {
Text string `json:"text"`
SourceID string `json:"source_id"`
Route string `json:"route"`
Source string `json:"source"`
Group string `json:"split_group"`
Tags []string `json:"tags,omitempty"`
Dev bool `json:"dev"`
// Direction: claimed_fast_now_miss = stored fast, runtime residual;
// mirror_missed = stored residual, runtime fast.
Direction string `json:"direction"`
Grammar string `json:"grammar,omitempty"`
// ShapeDeclined is true when at least one stage-0 grammar matched the
// utterance's shape but refused the content (falls through like the router).
ShapeDeclined bool `json:"shape_declined"`
// DeclinedGrammars names every stage-0 grammar that matched the shape but
// declined the content, for claimed_fast_now_miss rows.
DeclinedGrammars []string `json:"declined_grammars,omitempty"`
}
type report struct {
Meta metaSummary `json:"meta"`
Pop popSummary `json:"population"`
Conflicts []conflict `json:"conflicts"`
ByRoute map[string]map[string]int `json:"by_route"`
ByGrammar map[string]int `json:"by_grammar"`
BySource map[string]map[string]int `json:"by_source"`
ByFamily map[string]map[string]int `json:"by_family"`
Direction map[string]int `json:"by_direction"`
Declined int `json:"claimed_fast_with_declined_shape"`
NoShape int `json:"claimed_fast_with_no_shape"`
}
type metaSummary struct {
Total int `json:"total"`
DevCount int `json:"dev_count"`
Frozen int `json:"frozen_count"`
}
type popSummary struct {
StoredFast int `json:"stored_fast"`
StoredResid int `json:"stored_residual"`
DerivedFast int `json:"derived_fast"`
DerivedResid int `json:"derived_residual"`
// Dev-pool residual route counts derived as the router sees them today.
DevResidualByRoute map[string]int `json:"dev_residual_by_route"`
// Dev-pool residual non-action + action OOD as derived.
DevResidualNonAction int `json:"dev_residual_non_action"`
DevResidualAction int `json:"dev_residual_action"`
// Fast rows in the dev pool, derived.
DevFast int `json:"dev_fast"`
}
func main() {
outPath := flag.String("out", "/tmp/mvn-s23/drift.json", "output JSON path")
flag.Parse()
exs, err := semantic.LoadCorpus()
if err != nil {
fmt.Fprintf(os.Stderr, "load corpus: %v\n", err)
os.Exit(1)
}
_, dev, _ := semantic.FrozenHoldoutSplit(exs)
devSet := make(map[string]bool, len(dev))
for _, e := range dev {
devSet[e.SourceID] = true
}
// Stage-zero grammar list for shape/declined attribution (same list the
// derivation walks).
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
gs := router.StageZeroGrammars(acts)
var (
conflicts []conflict
byRoute = map[string]map[string]int{}
byGrammar = map[string]int{}
bySource = map[string]map[string]int{}
byFamily = map[string]map[string]int{}
byDirection = map[string]int{}
storedFast, derivedFast, declined, noShape int
devResidByRoute = map[string]int{}
devResidNonAct, devResidAct, devFast int
)
for _, e := range exs {
o := semantic.DeriveFastPath(e.Text)
inDev := devSet[e.SourceID]
st := e.FastPathResolved
if st {
storedFast++
}
if o.Matched {
derivedFast++
}
var c *conflict
switch {
case st && o.Matched:
case st && !o.Matched:
// Stored fast but the runtime misses. Attribute why.
shapeDeclined := false
var declinedNames []string
for _, g := range gs {
_, matched, ok := g.Evaluate(e.Text)
if matched && !ok {
shapeDeclined = true
declinedNames = append(declinedNames, g.Name)
}
}
if shapeDeclined {
declined++
} else {
noShape++
}
c = &conflict{Direction: "claimed_fast_now_miss", ShapeDeclined: shapeDeclined, DeclinedGrammars: declinedNames}
case !st && o.Matched:
c = &conflict{Direction: "mirror_missed", Grammar: o.Grammar}
}
if c != nil {
c.Text = e.Text
c.SourceID = e.SourceID
c.Route = string(e.Route)
c.Source = e.Source
c.Group = e.SplitGroup
c.Tags = e.Tags
c.Dev = inDev
conflicts = append(conflicts, *c)
byDirection[c.Direction]++
byGrammar[c.Grammar]++
if byRoute[c.Direction] == nil {
byRoute[c.Direction] = map[string]int{}
}
byRoute[c.Direction][c.Route]++
if bySource[c.Direction] == nil {
bySource[c.Direction] = map[string]int{}
}
bySource[c.Direction][c.Source]++
if byFamily[c.Direction] == nil {
byFamily[c.Direction] = map[string]int{}
}
byFamily[c.Direction][c.Group]++
}
if inDev {
if o.Matched {
devFast++
} else {
devResidByRoute[string(e.Route)]++
if e.Route == semantic.RouteAction {
devResidAct++
} else {
devResidNonAct++
}
}
}
}
sort.Slice(conflicts, func(i, j int) bool { return conflicts[i].SourceID < conflicts[j].SourceID })
rep := report{
Meta: metaSummary{Total: len(exs), DevCount: len(dev), Frozen: len(exs) - len(dev)},
Pop: popSummary{
StoredFast: storedFast, StoredResid: len(exs) - storedFast,
DerivedFast: derivedFast, DerivedResid: len(exs) - derivedFast,
DevResidualByRoute: devResidByRoute,
DevResidualNonAction: devResidNonAct, DevResidualAction: devResidAct,
DevFast: devFast,
},
Conflicts: conflicts,
ByRoute: byRoute, ByGrammar: byGrammar, BySource: bySource, ByFamily: byFamily,
Direction: byDirection, Declined: declined, NoShape: noShape,
}
data, err := json.MarshalIndent(rep, "", " ")
if err != nil {
fmt.Fprintf(os.Stderr, "marshal: %v\n", err)
os.Exit(1)
}
if err := os.WriteFile(*outPath, data, 0644); err != nil {
fmt.Fprintf(os.Stderr, "write %s: %v\n", *outPath, err)
os.Exit(1)
}
fmt.Printf("total %d (dev %d, frozen %d)\n", rep.Meta.Total, rep.Meta.DevCount, rep.Meta.Frozen)
fmt.Printf("stored fast=%d residual=%d\n", rep.Pop.StoredFast, rep.Pop.StoredResid)
fmt.Printf("derived fast=%d residual=%d\n", rep.Pop.DerivedFast, rep.Pop.DerivedResid)
fmt.Printf("disagreements total %d\n", len(conflicts))
for _, d := range []string{"claimed_fast_now_miss", "mirror_missed"} {
fmt.Printf(" %-22s %d\n", d, byDirection[d])
if d == "claimed_fast_now_miss" {
fmt.Printf(" with declined shape: %d no shape: %d\n", declined, noShape)
}
}
fmt.Println(" mirror-missed by grammar:")
for _, k := range sortedKeys(byGrammar) {
fmt.Printf(" %-28s %d\n", k, byGrammar[k])
}
fmt.Println(" by route:")
for _, d := range sortedKeys(byRoute) {
fmt.Printf(" %-22s %v\n", d, byRoute[d])
}
fmt.Printf("dev pool derived: fast=%d residual=%d (non-action=%d action=%d)\n",
rep.Pop.DevFast, rep.Pop.DevResidualNonAction+rep.Pop.DevResidualAction,
rep.Pop.DevResidualNonAction, rep.Pop.DevResidualAction)
fmt.Printf("dev residual by route: %v\n", rep.Pop.DevResidualByRoute)
fmt.Printf("wrote %s\n", *outPath)
}
func sortedKeys[T any](m map[string]T) []string {
ks := make([]string, 0, len(m))
for k := range m {
ks = append(ks, k)
}
sort.Strings(ks)
return ks
}
@@ -0,0 +1,98 @@
#!/usr/bin/env python3
"""
Slice 23 emit: five-way residual non-action semantic router data files
========================================================================
Slice 23 reconciles corpus fast-path metadata with the production router
(TryFastPath over stage-0 grammars). The corpus builder no longer mirrors the
grammars by hand; fast_path_resolved is derived from the router, so this emit
flags exactly the rows the router genuinely leaves for the general cascade.
This script only repackages the frozen dev pool for the Go legacy baseline and
the Python experiment, writing into /tmp/mvn-s23 so the slice-22 artifacts
stay untouched. Logic is slice22_emit.py verbatim; only OUT_DIR differs.
/tmp/mvn-s23/pool.json residual non-action dev rows: idx, text, n_text,
route, tags, cv_fold, split_group, family_id,
source_id (1509 rows)
/tmp/mvn-s23/ood.json residual ACTION dev rows (720): same shape; OOD
probes only, never primary metrics
/tmp/mvn-s23/stats.json population summary (routes, families, folds)
"""
import json
import os
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import slice18_sparse # noqa: E402 (normalize_match_text, load_data, filters)
OUT_DIR = "/tmp/mvn-s23"
ROUTES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
def main():
meta, examples = slice18_sparse.load_data()
dev = slice18_sparse.filter_dev_pool(examples)
print(f"dev pool: {len(dev)} rows "
f"(meta declares dev_count={meta.get('dev_count')})")
rows = []
for i, e in enumerate(dev):
if not e["fast_path_resolved"]:
rows.append({
"idx": i,
"text": e["text"],
"n_text": slice18_sparse.normalize_match_text(e["text"]),
"route": e["route"],
"tags": sorted(set(e.get("tags", []))),
"cv_fold": e["cv_fold"],
"split_group": e["split_group"],
"family_id": e["family_id"],
"source_id": e["source_id"],
})
na = [r for r in rows if r["route"] != "action"]
ood = [r for r in rows if r["route"] == "action"]
print(f"residual rows: {len(rows)} non-action: {len(na)} action(OOD): {len(ood)}")
by_route = {}
for r in na:
by_route[r["route"]] = by_route.get(r["route"], 0) + 1
print("routes:", by_route)
assert sum(by_route.values()) == len(na)
assert set(ROUTES) == set(by_route), "route set must be the five-way"
by_family = {}
for r in na:
by_family[r["family_id"]] = by_family.get(r["family_id"], 0) + 1
by_fold = {}
for r in na:
by_fold[r["cv_fold"]] = by_fold.get(r["cv_fold"], 0) + 1
print(f"family_ids: {len(by_family)} split_groups: {len(set(r['split_group'] for r in na))}")
print("folds:", by_fold)
os.makedirs(OUT_DIR, exist_ok=True)
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
json.dump(na, f, ensure_ascii=False, indent=1)
with open(os.path.join(OUT_DIR, "ood.json"), "w") as f:
json.dump(ood, f, ensure_ascii=False, indent=1)
with open(os.path.join(OUT_DIR, "stats.json"), "w") as f:
json.dump({
"dev_count": len(dev),
"residual_count": len(rows),
"non_action_count": len(na),
"action_ood_count": len(ood),
"routes": by_route,
"family_ids": len(by_family),
"folds": by_fold,
"top_family": dict(sorted(by_family.items(), key=lambda kv: -kv[1])[:15]),
}, f, ensure_ascii=False, indent=1)
print(f"wrote {OUT_DIR}/{{pool,ood,stats}}.json")
if __name__ == "__main__":
main()
@@ -0,0 +1,559 @@
#!/usr/bin/env python3
"""
Slice 23: five-way residual non-action semantic router corrected population
============================================================================
Slice 22 reported the corpus's fast-path mirror was stale next to the
production stage-0 grammars (185 residual rows resolved at runtime). Slice 23
derives fast_path_resolved from the real router, rebuilds the corpus, and
re-measures the primary slice-22 results on the corrected residual pool.
Logic and configs are slice22_main.py verbatim; only OUT_DIR differs.
Population: the corrected dev-pool residual non-action rows (1509; the pool
written by slice23_emit.py). Corrected action rows (720) are OOD probes only.
Metrics written to /tmp/mvn-s23/results.json:
§1 population
§2 legacy baseline (legacy.json / legacy_heads.json): acc, macro-F1,
per-class P/R/F1, confusion, illegal_action_prediction count
§3 e5-linear primary head: C grid, grouped CV OOF, per-fold P/R/F1 +
variance + composition
§4 floors: majority, centroid (cosine nearest-mean), sparse word+char
TF-IDF logistic (slice18 builder), all grouped CV
§5 route-family (family_id) leave-family-out
§6 knowledge vs memory_write: matched pairs (water/homelab/task) ordering
§7 uncertain as an explicit class: P/R/F1 + top confusions
§8 OOF confidence: max-softmax correct/wrong, ECE, log-loss, Brier,
coverage/accuracy/macro-F1 abstention curves (no threshold chosen)
§9 action OOD probes: fold models applied to the corrected action rows
§10 artifact cost: head params, serialized bytes, incremental head latency
No corpus label is changed. No frozen-holdout rows are inspected.
"""
import json
import os
import sys
import time
import numpy as np
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import slice18_sparse # noqa: E402
import slice19_main # noqa: E402
EMB_PATH = "/tmp/mvn-experiment/embeddings.json"
OUT_DIR = "/tmp/mvn-s23"
CLASSES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
CLASS_PREFIX = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
C_GRID = [0.1, 1.0, 10.0]
# Route-family holdouts the report calls out by name (slice-22 brief): every
# family that is not part of the shared subject inventory on either side.
HOLDOUT_GROUPS = {
"capability": ["knowledge:capability-ha", "knowledge:capability-tool"],
"world": ["knowledge:world-def", "knowledge:world-explain"],
"calendar": ["knowledge:calendar", "knowledge:calendar-time", "knowledge:calendar-next"],
"recall": ["knowledge:recall-fact", "knowledge:recall-note", "knowledge:recall-possessive"],
"fact": ["fact:meal", "fact:water", "fact:sleep", "fact:shower", "fact:break", "fact:pills", "fact:exercise"],
"note": ["note:idea", "note:homelab", "note:task"],
"remember": ["free:remember"],
"system": None, # all system:*
"conversation": None,
"uncertain": None,
}
def load_pool_and_embeds():
with open(os.path.join(OUT_DIR, "pool.json")) as f:
pool = json.load(f)
meta, examples = slice18_sparse.load_data()
dev = slice18_sparse.filter_dev_pool(examples)
by_idx = {e["dev_idx"]: e for e in dev} if "dev_idx" in dev[0] else None
# pool rows carry idx = position among dev_pool rows in dev order
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
for i, e in enumerate(dev)}
for r in pool:
r["emb"] = emb_by_idx[r["idx"]]
r["y"] = r["route"]
return pool, meta
def oof_proba_grouped(X, y, folds, C=1.0):
"""Grouped OOF probability matrix (n×5, class order CLASSES)."""
y_idx = np.array([CLASSES.index(c) for c in y])
folds = np.asarray(folds)
proba = np.zeros((len(y_idx), len(CLASSES)))
for te_fold in sorted(set(folds.tolist())):
tr = folds != te_fold
te = folds == te_fold
clf = slice18_sparse.LogisticRegression(
C=C, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], y_idx[tr])
proba[te] = clf.predict_proba(X[te])
return proba
def cls_metrics(yt, yp):
import sklearn.metrics as m
yt = np.asarray(yt)
yp = np.asarray(yp)
if yt.dtype != np.int64 and yt.dtype != np.int32:
yt = np.array([CLASSES.index(c) for c in yt])
if yp.dtype != np.int64 and yp.dtype != np.int32:
yp = np.array([CLASSES.index(c) for c in yp])
labels = list(range(len(CLASSES)))
n = len(yt)
acc = m.accuracy_score(yt, yp)
macro = m.f1_score(yt, yp, average="macro", labels=labels, zero_division=0)
pr, rc, f1, sup = m.precision_recall_fscore_support(
yt, yp, labels=labels, zero_division=0)
per = {c: {"p": float(pr[i]), "r": float(rc[i]), "f1": float(f1[i]), "n": int(sup[i])}
for i, c in enumerate(CLASSES)}
conf = m.confusion_matrix(yt, yp, labels=labels).tolist()
return {"n": n, "acc": acc, "macro_f1": macro, "per_class": per, "confusion": conf}
def fold_report(yt, proba, folds, true_y):
out = {}
folds_arr = np.asarray(folds)
comp = {}
for f in sorted(set(folds_arr.tolist())):
mask = folds_arr == f
yt_f = [CLASSES.index(y) for y in true_y[mask]]
comp[f] = {c: int((np.array(true_y[mask]) == c).sum()) for c in CLASSES}
per_fold = {}
for f in sorted(set(folds_arr.tolist())):
mask = folds_arr == f
yp = proba[mask].argmax(1).tolist()
m = cls_metrics([yt[i] for i in np.where(mask)[0].tolist()], yp)
per_fold[f] = {"acc": m["acc"], "macro_f1": m["macro_f1"]}
out["composition"] = comp
out["per_fold"] = per_fold
accs = [v["acc"] for v in per_fold.values()]
macros = [v["macro_f1"] for v in per_fold.values()]
out["acc_mean"] = float(np.mean(accs))
out["acc_std"] = float(np.std(accs))
out["macro_f1_mean"] = float(np.mean(macros))
out["macro_f1_std"] = float(np.std(macros))
return out
def ece(yt, proba, n_bins=15):
conf = proba.max(1)
pred = proba.argmax(1)
acc = (pred == yt).astype(float)
bins = np.linspace(0, 1, n_bins + 1)
tot = 0.0
details = []
counts = 0
for i in range(n_bins):
lo, hi = bins[i], bins[i + 1]
m = (conf >= lo) & (conf < hi) if i < n_bins - 1 else conf >= lo
if m.sum() == 0:
continue
acc_m = acc[m].mean()
conf_m = conf[m].mean()
w = m.sum() / len(conf)
tot += w * abs(acc_m - conf_m)
counts += int(m.sum())
details.append({"bin": i, "lo": lo, "hi": hi, "conf": float(conf_m),
"acc": float(acc_m), "n": int(m.sum())})
return {"ece": float(tot), "n_bins": n_bins, "counted": counts, "bins": details}
def main():
pool, meta = load_pool_and_embeds()
pool.sort(key=lambda r: r["idx"])
print(f"pool: {len(pool)} rows")
from sklearn.metrics import brier_score_loss, log_loss
report = {"population": {}, "legacy": {}, "e5_linear": {}, "floors": {},
"family_holdouts": {}, "kmw": {}, "uncertain": {}, "confidence": {},
"ood": {}, "artifact": {}}
# ── §1 population ──────────────────────────────────────────────────────
cnt = {}
for r in pool:
cnt[r["y"]] = cnt.get(r["y"], 0) + 1
report["population"] = {
"n": len(pool),
"routes": cnt,
"family_ids": len(set(r["family_id"] for r in pool)),
"split_groups": len(set(r["split_group"] for r in pool)),
"folds": {str(f): int(sum(1 for r in pool if r["cv_fold"] == f)) for f in sorted(set(r["cv_fold"] for r in pool))},
"corpus": {k: v for k, v in meta.items() if k in
("dev_count", "residual_count", "fast_path_count",
"dimension", "embedder_id", "input_template", "pooling", "normalization")},
}
print("\n§1 population:", report["population"])
X = np.vstack([r["emb"] for r in pool])
y = np.array([r["y"] for r in pool])
folds = np.array([r["cv_fold"] for r in pool])
yt = np.array([CLASSES.index(c) for c in y])
# ── §2 legacy baselines ────────────────────────────────────────────────
import collections
for tag, fname in [("hash", "legacy.json"), ("heads", "legacy_heads.json")]:
path = os.path.join(OUT_DIR, fname)
if not os.path.exists(path):
continue
leg = json.load(open(path))
leg_by_idx = {r["idx"]: r for r in leg}
yp_leg = []
illegal = []
for r in pool:
lr = leg_by_idx[r["idx"]]
if lr["illegal_action_prediction"]:
illegal.append(lr)
yp_leg.append("action")
else:
yp_leg.append(lr["class"])
yp_leg = np.array(yp_leg)
# five-way: an 'action' prediction is an error (outside the label set)
yp5 = np.array([("uncertain" if p == "action" else p) for p in yp_leg])
m = cls_metrics(y, yp5)
m["illegal_action_prediction"] = len(illegal)
m["illegal_cases"] = [{"idx": i["idx"], "text": i["text"], "route": i["route"],
"intent": i["intent"], "producer": i["producer"],
"confidence": i["confidence"]} for i in illegal]
# per-cell confusion also shows 'action' column
conf_counts = collections.Counter(zip(y, yp_leg))
m["confusion_with_action"] = {f"{a}->{b}": int(c) for (a, b), c in conf_counts.items()}
report["legacy"][tag] = m
print(f"\n§2 legacy ({tag}) acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f} "
f"illegal={len(illegal)}")
for c in CLASSES:
p = m["per_class"][c]
print(f" {c:<14} P={p['p']:.3f} R={p['r']:.3f} F1={p['f1']:.3f} n={p['n']}")
# grammar-pure residual: rows not resolved by any current stage-0 grammar
if os.path.exists(os.path.join(OUT_DIR, "legacy.json")):
leg = json.load(open(os.path.join(OUT_DIR, "legacy.json")))
gh = {r["idx"] for r in leg if r["producer"] == "grammar"}
gp_mask = np.array([r["idx"] not in gh for r in pool])
report["grammar_drift"] = {
"grammar_hits_in_pool": len(gh),
"grammar_pure_n": int(gp_mask.sum()),
}
# ── §3 e5-linear primary head ─────────────────────────────────────────
print("\n§3 e5-linear")
bestC, bestMac = 1.0, -1.0
grid = {}
for C in C_GRID:
p = oof_proba_grouped(X, y, folds, C=C)
mp = cls_metrics(y, p.argmax(1).tolist())
grid[float(C)] = {"acc": mp["acc"], "macro_f1": mp["macro_f1"]}
print(f" C={C} acc={mp['acc']:.4f} macroF1={mp['macro_f1']:.4f}")
if mp["macro_f1"] > bestMac:
bestMac, bestC = mp["macro_f1"], C
print(f" -> best C={bestC}")
p_best = oof_proba_grouped(X, y, folds, C=bestC)
m_best = cls_metrics(y, p_best.argmax(1).tolist())
m_best["C"] = bestC
m_best["C_grid"] = grid
m_best["folds"] = fold_report(yt, p_best, folds, y)
report["e5_linear"] = m_best
for f, v in m_best["folds"]["per_fold"].items():
print(f" fold {f}: acc={v['acc']:.4f} macroF1={v['macro_f1']:.4f}")
print(f" fold acc mean={m_best['folds']['acc_mean']:.4f} "
f"std={m_best['folds']['acc_std']:.4f}; "
f"macroF1 mean={m_best['folds']['macro_f1_mean']:.4f} "
f"std={m_best['folds']['macro_f1_std']:.4f}")
for c in CLASSES:
p_ = m_best["per_class"][c]
print(f" {c:<14} P={p_['p']:.3f} R={p_['r']:.3f} F1={p_['f1']:.3f} n={p_['n']}")
# grammar-pure sensitivity for the primary head
if "grammar_drift" in report:
mp_gp = cls_metrics(y[gp_mask], p_best[gp_mask].argmax(1).tolist())
report["e5_linear"]["grammar_pure"] = {
"acc": mp_gp["acc"], "macro_f1": mp_gp["macro_f1"], "n": int(gp_mask.sum())}
# ── §4 floors ─────────────────────────────────────────────────────────
print("\n§4 floors")
# majority floor
maj = CLASSES.index("knowledge")
ym = np.full(len(y), maj)
mm = cls_metrics(y, ym)
report["floors"]["majority"] = {"acc": mm["acc"], "macro_f1": mm["macro_f1"],
"per_class": mm["per_class"]}
print(f" majority (predict {CLASSES[maj]}): acc={mm['acc']:.4f} macroF1={mm['macro_f1']:.4f}")
# centroid floor: cosine to per-class mean of the training folds' embeddings
cf_proba = np.zeros((len(yt), len(CLASSES)))
folds_arr = np.asarray(folds)
for te_fold in sorted(set(folds_arr.tolist())):
tr = folds_arr != te_fold
te = folds_arr == te_fold
centroids = []
for c in CLASSES:
idxs = np.where(tr & (y == c))[0]
ctr = X[idxs].mean(axis=0)
ctr = ctr / np.linalg.norm(ctr)
centroids.append(ctr)
Cm = np.vstack(centroids)
sims = X[te] @ Cm.T
cf_proba[te] = sims
yc = cf_proba.argmax(1)
# accuracy + macroF1 with the same 5-way
mc = cls_metrics(y, yc.tolist())
report["floors"]["centroid"] = {"acc": mc["acc"], "macro_f1": mc["macro_f1"],
"per_class": mc["per_class"]}
print(f" centroid cosine: acc={mc['acc']:.4f} macroF1={mc['macro_f1']:.4f}")
# sparse word+char logistic (slice18 builder, grouped CV, five-way)
texts = [r["n_text"] for r in pool]
Xs, _vec = slice18_sparse.build_features(texts, "both")
psp = np.zeros((len(yt), len(CLASSES)))
for te_fold in sorted(set(folds_arr.tolist())):
tr = folds_arr != te_fold
te = folds_arr == te_fold
clf = slice18_sparse.LogisticRegression(
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(Xs[tr], yt[tr])
psp[te] = clf.predict_proba(Xs[te])
msp = cls_metrics(y, psp.argmax(1).tolist())
report["floors"]["sparse_word_char"] = {
"acc": msp["acc"], "macro_f1": msp["macro_f1"], "per_class": msp["per_class"],
"vocab": slice18_sparse.vocab_size(_vec)}
print(f" sparse both: acc={msp['acc']:.4f} macroF1={msp['macro_f1']:.4f} "
f"vocab={report['floors']['sparse_word_char']['vocab']}")
# ── §5 route-family holdouts ─────────────────────────────────────────
print("\n§5 route-family holdouts")
fam = np.array([r["family_id"] for r in pool])
holdouts = {}
all_fams = sorted(set(fam.tolist()))
for grp, fams in HOLDOUT_GROUPS.items():
if fams is None:
fams = [f for f in all_fams if f.startswith(grp + ":")]
mask = np.isin(fam, fams)
if mask.sum() == 0:
continue
tr = ~mask
clf = slice18_sparse.LogisticRegression(
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yt[tr])
ypgrp = clf.predict(X[mask])
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypgrp.tolist())
m["families"] = fams
m["rows"] = int(mask.sum())
holdouts[grp] = {"acc": m["acc"], "macro_f1": m["macro_f1"], "n": int(mask.sum()),
"per_class": m["per_class"]}
print(f" {grp:<14} n={m['rows']} acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f}")
# full leave-one-family-out summary
lofo_accs = []
lofo_f1s = []
for f in all_fams:
mask = fam == f
tr = ~mask
clf = slice18_sparse.LogisticRegression(
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yt[tr])
ypf = clf.predict(X[mask])
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypf.tolist())
lofo_accs.append(m["acc"])
lofo_f1s.append(m["macro_f1"])
holdouts["_all_49_lo_"] = {"n_families": len(all_fams),
"acc_mean": float(np.mean(lofo_accs)),
"macro_f1_mean": float(np.mean(lofo_f1s))}
report["family_holdouts"] = holdouts
print(f" leave-one-family-out over {len(all_fams)} families: "
f"acc mean={np.mean(lofo_accs):.4f} macroF1 mean={np.mean(lofo_f1s):.4f}")
# ── §6 knowledge vs memory_write ─────────────────────────────────────
print("\n§6 knowledge vs memory_write")
# reuse e5-linear OOF: does the model put the higher probability on the
# right side (memory_write for a write, knowledge for a recall)?
conf_km = np.zeros((2, 2))
pk = p_best[:, CLASSES.index("knowledge")]
pmw = p_best[:, CLASSES.index("memory_write")]
for i in range(len(yt)):
t = y[i]
if t == "knowledge":
conf_km[0, 1 if pmw[i] > pk[i] else 0] += 1
elif t == "memory_write":
conf_km[1, 1 if pmw[i] >= pk[i] else 0] += 1
report["kmw"] = {"confusion_p_ordered": conf_km.tolist()}
# matched pairs with shared subject lexemes, corpus-justified
def build_pairs(subject, fam_k, fam_mw):
kr = [r for r in pool if r["family_id"] in fam_k]
mr = [r for r in pool if r["family_id"] in fam_mw]
pairs = []
for mw in mr:
for k in kr:
if subject in mw["n_text"] and subject in k["n_text"]:
pairs.append((mw["idx"], k["idx"], mw["n_text"], k["n_text"]))
return pairs
sets = {
"water": build_pairs("вод", ["knowledge:recall-fact"], ["fact:water"]),
"homelab": build_pairs("dns", ["knowledge:homelab-status"], ["note:homelab"])
+ build_pairs("сервер", ["knowledge:homelab-status"], ["note:homelab"])
+ build_pairs("vlan", ["knowledge:homelab-status"], ["note:homelab"]),
"task": build_pairs("задач", ["knowledge:task-check", "knowledge:deadline"],
["note:task"]),
}
idx_of = {r["idx"]: i for i, r in enumerate(pool)}
pair_rep = {}
for name, pairs in sets.items():
if not pairs:
continue
ok = 0
margins = []
bad = []
for mi, ki, mx, kx in pairs:
mi_i, ki_i = idx_of[mi], idx_of[ki]
# MW row should get a higher memory_write probability than the K row
mk = (pmw[mi_i] + 0.0)
if pmw[mi_i] > pmw[ki_i]:
ok += 1
else:
bad.append((mx[:46], round(float(pmw[mi_i]), 3), kx[:46], round(float(pmw[ki_i]), 3)))
margins.append(pmw[mi_i] - pmw[ki_i])
pair_rep[name] = {
"pairs": len(pairs),
"mw_over_k_order_acc": ok / len(pairs),
"mean_margin": float(np.mean(margins)),
"reversed_examples": bad[:6],
}
print(f" {name}: pairs={len(pairs)} order_acc={ok/len(pairs):.3f} "
f"mean_margin={np.mean(margins):+.3f}")
report["kmw"]["matched_pairs"] = pair_rep
# ── §7 uncertain as explicit class ────────────────────────────────────
print("\n§7 uncertain")
up = m_best["per_class"]["uncertain"]
uc = m_best["confusion"][CLASSES.index("uncertain")]
report["uncertain"] = {
"per_class": up,
"row_from_uncertain": {CLASSES[j]: int(uc[j]) for j in range(5)},
"row_to_uncertain": {CLASSES[j]: int(m_best["confusion"][j][CLASSES.index("uncertain")])
for j in range(5)},
}
print(f" uncertain n={up['n']} P={up['p']:.3f} R={up['r']:.3f} F1={up['f1']:.3f}")
print(" wrong-→label pulled from uncertain:", report["uncertain"]["row_from_uncertain"])
print(" →uncertain pulled from:", report["uncertain"]["row_to_uncertain"])
# ── §8 OOF confidence / calibration / abstention ─────────────────────
print("\n§8 confidence / calibration")
conf = p_best.max(1)
right = (p_best.argmax(1) == yt)
cer = {
"correct_conf_mean": float(conf[right].mean()),
"correct_conf_median": float(np.median(conf[right])),
"wrong_conf_mean": float(conf[~right].mean()),
"wrong_conf_median": float(np.median(conf[~right])),
"ece": ece(yt, p_best)["ece"],
"ece_bins": ece(yt, p_best)["bins"],
"log_loss": float(log_loss(yt, p_best, labels=[0, 1, 2, 3, 4])),
}
# Brier is label-set specific: one-vs-rest mean
briers = []
for i in range(5):
briers.append(brier_score_loss((yt == i).astype(int), p_best[:, i]))
cer["brier_macro"] = float(np.mean(briers))
report["confidence"] = cer
print(f" right conf mean={cer['correct_conf_mean']:.3f} "
f"wrong conf mean={cer['wrong_conf_mean']:.3f} ECE={cer['ece']:.4f}")
print(f" log_loss={cer['log_loss']:.4f} brier_macro={cer['brier_macro']:.4f}")
thr_grid = np.linspace(0.10, 0.98, 45)
abst = []
for t in thr_grid:
cov = (conf >= t).mean()
if cov == 0:
continue
keep = conf >= t
yt_k = yt[keep]
yp_k = p_best[keep].argmax(1)
mk_ = cls_metrics(yt_k.tolist(), yp_k.tolist())
abst.append({"threshold": round(float(t), 3), "coverage": float(cov),
"accuracy": mk_["acc"], "macro_f1": mk_["macro_f1"]})
report["confidence"]["abstention_curve"] = abst
print(" threshold | coverage | accuracy | macroF1 (first 6/45 + knee)")
for row in abst[::9]:
print(f" {row['threshold']:.2f} | {row['coverage']:.3f} | "
f"{row['accuracy']:.3f} | {row['macro_f1']:.3f}")
# ── §9 action OOD probes ──────────────────────────────────────────────
print("\n§9 action OOD")
ood_rows = [r for r in json.load(open(os.path.join(OUT_DIR, "ood.json")))]
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
for i, e in enumerate(slice18_sparse.filter_dev_pool(
slice18_sparse.load_data()[1]))}
Xo = np.vstack([emb_by_idx[r["idx"]] for r in ood_rows])
fold_models = []
for te_fold in sorted(set(folds_arr.tolist())):
tr = folds_arr != te_fold
clf = slice18_sparse.LogisticRegression(
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X[tr], yt[tr])
fold_models.append(clf)
# XX
# comparable to the non-action in-fold behaviour
po = np.zeros((len(Xo), 5))
for clf in fold_models:
po += clf.predict_proba(Xo)
po /= len(fold_models)
ood_top = int(np.argmax(po.mean(0)))
ood_conf = po.max(1)
ood_pred = po.argmax(1)
top_dist = {CLASSES[i]: int((ood_pred == i).sum()) for i in range(5)}
confident_na = int((ood_conf > 0.9).sum())
report["ood"] = {
"n": len(ood_rows),
"top_class": CLASSES[int(ood_top)],
"top_class_dist": top_dist,
"conf_gt_0.9": confident_na,
"conf_gt_0.9_frac": float(confident_na / len(ood_rows)),
"conf_mean": float(ood_conf.mean()),
"conf_median": float(np.median(ood_conf)),
}
print(f" action OOD n={len(ood_rows)}: most-confident class={report['ood']['top_class']} "
f"dist={top_dist}")
print(f" conf>0.9: {confident_na} ({confident_na/len(ood_rows):.3f}) "
f"conf mean={report['ood']['conf_mean']:.3f}")
# ── §10 artifact cost ────────────────────────────────────────────────
print("\n§10 artifact")
n_params = len(CLASSES) * X.shape[1] + len(CLASSES)
fp32 = n_params * 4
report["artifact"] = {
"e5_dim": X.shape[1],
"head_params": n_params,
"head_fp32_bytes": fp32,
"head_fp32_kib": fp32 / 1024,
"head_int8_bytes": n_params,
}
# incremental latency of the linear head over a batch of 1 (µs)
clf = slice18_sparse.LogisticRegression(C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
clf.fit(X, yt)
x1 = X[:1]
for _ in range(50):
clf.predict_proba(x1)
lat = []
for _ in range(2000):
t0 = time.perf_counter_ns()
clf.predict_proba(x1)
lat.append((time.perf_counter_ns() - t0) / 1e3)
lat = np.array(lat)
report["artifact"]["head_latency_us_mean"] = float(lat.mean())
report["artifact"]["head_latency_us_p50"] = float(np.median(lat))
print(f" head params={n_params} fp32={fp32/1024:.2f}KiB "
f"lat mean={lat.mean():.2f}us p50={np.median(lat):.2f}us")
with open(os.path.join(OUT_DIR, "results.json"), "w") as f:
json.dump(report, f, ensure_ascii=False, indent=1, default=float)
print(f"\nwrote {OUT_DIR}/results.json")
if __name__ == "__main__":
main()
+1 -1
View File
@@ -259,7 +259,7 @@
"heads_path": "/opt/maven/models/embedder/router-heads/router_heads.onnx"
},
"llm_router": true,
"query_min_score": 0.55,
"query_min_score": 0.80,
"query_min_margin": 0.008,
"clarify_max_attempts": 3,
"tool_timeout": "30s",
+293
View File
@@ -0,0 +1,293 @@
{
"db_path": "/var/lib/maven/maven.db.enc",
"db_tmpfs": "/dev/shm/maven-plain.db",
"db_key_env": "MAVEN_DB_KEY",
"socket_path": "/run/maven/mavend.sock",
"state_dir": "/var/lib/maven",
"//disabled_rules": [
"Nudge rules that are not wired at all. Names come from loop.DefaultRules:",
"water, meal, break, service_down, netdata_critical.",
"service_down is back on: mavpoll now writes one fact per kuma monitor",
"(service_down:<name>), so the nudge names the service and pausing a monitor",
"in kuma silences that monitor. It is also edge-triggered, so a service that",
"stays down is one nudge, not one every fifteen minutes."
],
"disabled_rules": [],
"phraser": {
"model_path": "/opt/maven/models/llm/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf",
"bin_path": "llama-server",
"n_gpu_layers": 99,
"n_ctx": 4096,
"cache_ram_mib": 512,
"timeout": "60s",
"llm_nudges": false
},
"//ntfy": [
"The second reach (V-649). Until 07-08-2026 telegram was the only one, and",
"telegram needs api.telegram.org, the socks relay below and a matching ufw",
"rule — three things in series that have each failed once, and when they do",
"a sev4 nudge has nowhere to go. ntfy shares none of them: it is reached",
"directly, no relay.",
"It is the preferred away reach. Reminders fall back to Telegram in order,",
"and every missing reach is recorded in the outbox rather than disappearing.",
"The credential is an ntfy access token, scoped write-only to this one",
"topic, so a popped sink can push to it and cannot read it back. Set it in",
"deploy/telegram.env beside the telegram secrets; that file is gitignored."
],
"ntfy": {
"disabled": true,
"base_url": "https://ntfy.kvmx.ru",
"topic": "maven",
"token": "${NTFY_TOKEN}"
},
"telegram": {
"bot_token": "${TELEGRAM_BOT_TOKEN}",
"chat_id": "${TELEGRAM_CHAT_ID}",
"//proxy": [
"api.telegram.org is not reachable directly from this box, so every send",
"timed out. The relay is the x-ui socks inbound on the host, port 10808;",
"192.168.240.1 is the maven_default bridge gateway, which is how a",
"container addresses the host. mavend is on that network.",
"This needs a matching ufw rule or the container's SYN is dropped:",
" ufw allow from 192.168.240.0/20 to any port 10808 proto tcp"
],
"proxy": "socks5://192.168.240.1:10808",
"//intake": [
"Read the chat as well as write to it (V-637). The poller long-polls",
"getUpdates through the same relay and accepts chat_id as the only",
"sender. Deleting this key turns inbound off again.",
"chat_id must be numeric here or the daemon refuses to start: an inbound",
"update names its chat by number, so an @-name would match nothing."
],
"intake": true
},
"//workstation": [
"The big model on the desk PC (workpc, 7900 GRE 16GB), fronted by",
"mavgpud on port 8080. It runs gemma-4-12b and it is preferred over the",
"resident Qwen3-1.7B for routing and replies whenever the card is free.",
"The machine is never assumed up: it sleeps, and the card is often held by",
"a CPT run, in which case mavgpud answers 503 and Maven falls back to the",
"resident model without saying so. Deleting this block restores exactly",
"the behaviour homesrv had before it existed.",
"Addressed by LAN address, not container name: mavgpud runs on another",
"machine and there is no shared docker network to name it on.",
"model_disabled keeps only this model arm dark until MAVEN_GPU_TOKEN is",
"provisioned; the independently authenticated STT arm below stays live."
],
"//workstation.stt": [
"CrisperWhisper 2.0 turbo on the same machine, a second service on port",
"8081 and not a second endpoint on mavgpud. whisper.cpp cannot load CW2 at",
"all: it derives its language count from the vocabulary size, and CW2's",
"51897 tokens shift seven special token ids. So it runs under transformers",
"there and mavsttd stays whisper.cpp here.",
"Worth the second service: CW2 turbo scores 10.4% WER in Russian against",
"27.5% for the ggml-small.bin mavsttd loads, measured on 200 Golos clips",
"in docs/evals/2026-08-09-crisperwhisper2-russian-wer.md.",
"Deleting this block sends every utterance to mavsttd, which is what the",
"box did before it existed. A worse transcript is still a turn, so the",
"fallback is silent and Kami is never told which machine heard him.",
"The token is what stops anything on the LAN posting audio to that port."
],
"workstation": {
"model_disabled": true,
"url": "http://192.168.1.105:8080",
"token": "${MAVEN_GPU_TOKEN}",
"probe": "15s",
"timeout": "90s",
"stt": {
"url": "http://192.168.1.105:8081/transcribe",
"token": "${MAVEN_STT_TOKEN}",
"probe": "15s",
"timeout": "10s"
}
},
"//search": [
"The live web, searched after his own notes and before Kiwix. Only the",
"query string leaves the box — never a note, a fact, the persona block or",
"the history — and a question about him never reaches here at all.",
"The instance must have `json` in search.formats (settings.yml); a stock",
"SearXNG answers 403 to format=json and every search then fails. It is",
"addressed by container name, so it needs the same maven_default",
"attachment kiwix has, and it must listen on 9563: 8080 is taken several",
"times over on this box. No instance reachable ⇒ she falls through to the",
"ZIMs and never says the search failed."
],
"search": {
"url": "http://searxng:9563",
"max_results": 4,
"snippet_runes": 1500,
"language": "auto",
"timeout": "8s"
},
"//kiwix": [
"The offline encyclopedia, searched after his own notes and before anything",
"on the network. kiwix-server publishes 8034 on loopback only, so a container",
"cannot reach it by address; it is attached to the maven_default network",
"instead and addressed by container name. That attachment is imperative and",
"does not survive recreating the kiwix stack — make it declarative there:",
" networks: [default, maven_default] # maven_default: external: true",
"The book is the catalog name from the /content/... href in",
"/catalog/v2/entries, not the display title. Others on the box:",
"ifixit_en_all_2025-06, devdocs_en_ansible_2025-10."
],
"kiwix": {
"url": "http://kiwix-server:8080",
"book": "wikipedia_en_all_maxi_2026-02",
"book_ru": "wikipedia_ru_all_maxi_2026-02",
"max_results": 5,
"snippet_runes": 1500
},
"//morning_routines": [
"The daily checklist (Vikunja #280). Each item is done when its fact_key",
"gets a non-voided fact inside the window, so 'выпил воды' closes water and",
"nothing has to be ticked by hand. nudge_at fires once, at the end of the",
"window, and only for what is still open. Weekdays empty = every day."
],
"morning_routines": [
{
"name": "утро",
"window_start": "08:00",
"window_end": "11:00",
"nudge_at": "10:30",
"severity": 1,
"items": [
{ "key": "medicine", "fact_key": "medicine", "label": "лекарство" },
{ "key": "water", "fact_key": "water", "label": "вода" },
{ "key": "pets", "fact_key": "pets", "label": "покормить кота" }
]
}
],
"//feeds": [
"RSS reading (Vikunja #258). Every item lands as a note with source",
"rss:<name>, which is also what puts entries in the intake journal that",
"/events reads. Only the feed URL leaves the box.",
"This is a starting pair, not a curated set — trim or extend it."
],
"feeds": {
"poll_interval": "30m",
"max_items": 5,
"max_age": "24h",
"sources": [
{ "name": "lwn", "url": "https://lwn.net/headlines/newrss", "category": "технологии" },
{ "name": "archlinux", "url": "https://archlinux.org/feeds/news/", "category": "технологии" }
]
},
"//crawl": [
"Reading a web page (Vikunja #259). on_demand answers 'посмотри <URL>'.",
"No allow_hosts, so any public host he names is readable; private",
"addresses are refused unconditionally by internal/webfetch and do not",
"need listing. Setting allow_hosts here would also narrow on-demand,",
"which is the point of leaving it empty."
],
"crawl": {
"on_demand": true,
"timeout": "10s",
"max_runes": 4000
},
"digest": {
"enabled": true,
"window": "30m",
"max_items": 5,
"severity_ceiling": 2
},
"pattern_proposals": {
"notify": false,
"cooldown": "24h"
},
"mcp": {
"timeout": "15s",
"servers": [
{
"name": "vikunja",
"url": "http://192.168.1.104:9100/mcp",
"allow_private": true,
"allow_tools": ["list_projects", "list_tasks", "get_task_details", "create_task"],
"max_tools": 6,
"enabled": false
}
]
},
"smarthome": {
"provider": "homeassistant",
"url": "http://192.168.1.50:8123",
"token": "${HA_TOKEN}",
"domains": ["light", "switch", "sensor"],
"max_entities": 40,
"timeout": "10s",
"refresh": "15m",
"enabled": false
},
"netscan": {
"subnets": ["192.168.1.0/24"],
"ports": [22, 80, 443, 8080],
"timeout": "400ms",
"rate": 100,
"max_hosts": 256,
"enabled": true
},
"nexus": { "url": "http://nexus:9740" },
"praxis": { "url": "http://praxis:8989" },
"hexis": { "url": "http://hexis:9741" },
"voice": {
"enabled": true,
"bind": "0.0.0.0:9100",
"lang": "ru",
"stt": { "socket": "/run/maven/stt.sock", "lang": "ru" },
"tts": { "socket": "/run/maven/tts.sock", "lang": "ru" },
"embedder": {
"model_path": "/opt/maven/models/embedder/multilingual-e5-small/model_quantized.onnx",
"tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/opt/maven/lib/libonnxruntime.so",
"heads_path": "/opt/maven/models/embedder/router-heads/router_heads.onnx"
},
"llm_router": true,
"query_min_score": 0.80,
"query_min_margin": 0.008,
"clarify_max_attempts": 3,
"tool_timeout": "30s",
"tools": [
{ "name": "status", "cmd": ["systemctl", "status"], "scope": "homelab", "destructive": false,
"aliases": ["статус", "покажи статус", "проверь статус"] },
{ "name": "ps", "cmd": ["docker", "ps"], "scope": "homelab", "destructive": false,
"aliases": ["статус докера", "лог докера", "покажи запущенные контейнеры", "покажи контейнеры", "список контейнеров", "что запущено"] },
{ "name": "uptime", "cmd": ["uptime"], "scope": "homelab", "destructive": false,
"aliases": ["покажи uptime", "аптайм", "как работает сервер", "сколько работает сервер"] },
{ "name": "disk", "cmd": ["df", "-h"], "scope": "homelab", "destructive": false,
"aliases": ["сколько места на диске", "сколько свободного места на диске", "место на диске", "покажи диск"] },
{ "name": "memory", "cmd": ["free", "-h"], "scope": "homelab", "destructive": false,
"aliases": ["свободная память", "сколько оперативной памяти свободно", "покажи память"] },
{ "name": "logs", "cmd": ["journalctl", "-n", "50", "-u"], "scope": "homelab", "destructive": false,
"aliases": ["покажи логи", "логи", "лог"] },
{ "name": "restart", "cmd": ["systemctl", "restart"], "scope": "homelab", "destructive": true,
"aliases": ["перезапусти", "перезагрузи", "рестарт"] },
{ "name": "stop", "cmd": ["systemctl", "stop"], "scope": "homelab", "destructive": true,
"aliases": ["останови", "останови сервис"] },
{ "name": "start", "cmd": ["systemctl", "start"], "scope": "homelab", "destructive": true,
"aliases": ["запусти", "запусти сервис"] },
{ "name": "docker-restart", "cmd": ["docker", "restart"], "scope": "homelab", "destructive": true,
"aliases": ["перезапусти контейнер", "перезагрузи контейнер"] },
{ "name": "docker-stop", "cmd": ["docker", "stop"], "scope": "homelab", "destructive": true,
"aliases": ["останови контейнер"] },
{ "name": "reboot", "cmd": ["systemctl", "reboot"], "scope": "homelab", "destructive": true,
"aliases": ["перезагрузи сервер", "перезагрузи хост"] }
]
}
}
+6
View File
@@ -11,6 +11,8 @@ The tier is the path, so staleness is visible from the filename.
| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. Indexed in `docs/evals/CLAUDE.md`, which marks each one live or superseded. | forever |
| `docs/caveats/` | known limits, one entry per limit, each with a task id and a revisit trigger. Indexed in `docs/caveats/CLAUDE.md`. | until fixed, then deleted |
| `docs/plans/` | the plan for one piece of work, frozen once it starts. Indexed in `docs/plans/CLAUDE.md`. | until the work lands |
| `docs/capabilities/` | generated. The capability ledger and its probe harness, regenerated from `docs/spec.md` plus a named eval. **Never hand-edited**, except `domains.yaml`, `probes_field.json` and the scripts, which are its sources. Indexed in `docs/capabilities/README.md`. | until the spec or the measurement moves |
| `docs/architecture/` | generated. The architecture observation and its evidence pack, rebuilt from source by the scripts beside it. Indexed in `docs/architecture/README.md`. | until the shape changes |
| `docs/archive/` | dead. Read by nobody by default. | forever |
## Rules for this directory
@@ -21,6 +23,10 @@ The tier is the path, so staleness is visible from the filename.
correction. Do not append a changelog to it.
* A number in prose with no `docs/evals/` file behind it is an opinion.
* Fixing something deletes its caveat. It does not edit the eval that found it.
* **A generated tier is rebuilt, never corrected.** A wrong row in
`docs/capabilities/` or `docs/architecture/` is a bug in the generator or in
one of its hand-written inputs. Editing the output makes the next rebuild
silently undo the fix.
## Where a subsystem's reasoning lives
+143
View File
@@ -0,0 +1,143 @@
# docs/architecture
An observation of Maven as built, read at `5cae33a` on 2026-08-25. It describes
what the code does today. It proposes nothing.
This directory is a build output plus its sources. `index.html`,
`maven-architecture.json`, `anchors.md` and `diagrams/*.svg` are generated.
## Read it
| file | what it is |
|---|---|
| `index.html` | the viewer. Open it from the filesystem, no server needed. Seven views, the rendered diagram above each, click a component for its record. |
| `check_viewer.js` | the viewer's only check. Runs every view against a DOM stub, because a TypeError in a renderer shows as a blank panel and not as an error. |
| `findings.md` | the analysis. Kept apart from the facts on purpose. |
| `maven-architecture.json` | the inventory. 160 components, 204 relations. The factual source for everything else. |
| `anchors.md` | every symbol the inventory names, resolved to `path:line` with the line quoted. |
| `diagrams/*.mmd` | the five views as Mermaid source. `03a`, `03b` and `03c` are the three traced requests. |
| `diagrams/*.svg` | the same, rendered. |
## Rebuild it
```sh
python3 docs/architecture/build_inventory.py # → maven-architecture.json
python3 docs/architecture/verify_anchors.py # → anchors.md, exit 1 if stale
sh docs/architecture/render.sh # → diagrams/*.svg, then index.html, then check_viewer.js
python3 docs/architecture/build_viewer.py # → index.html alone
node docs/architecture/check_viewer.js # → every view rendered, no throw
```
Views 6 and 7 read `docs/capabilities/`, not this directory. View 6 is the
capability matrix, 51 rows against seven dimensions. View 7 is the twelve
cross-cutting invariants and the components that participate in each. Both are
inlined by `build_viewer.py`, which reads `ledger.yaml` and `invariants.yaml`
rather than deriving anything itself: the ledger's build is the only thing
allowed to decide a dimension.
`render.sh` drives mermaid-cli through the system chromium rather than letting
puppeteer download its own. It is also the only syntax check this repo has for a
`.mmd`.
## What is verified, and what is not
**Verified mechanically.** `verify_anchors.py` resolves all 692 claimed symbols
against the files their component names. Current state: 681 resolved to a line
and 0 unresolved, with 0 missing files. The other 11 are config keys and make
targets rather than Go identifiers, so they are skipped. The script exits
non-zero on any failure, which makes it a staleness gate.
Writing it caught 29 symbols filed under the wrong component and seven names
that were wrong outright. Two examples: `Store.RecordEvent` for what is really
`Store.CreateEvent`, and `media.Keeper` for what is really `media.Store`.
**Not verified.** That a symbol means what its `responsibility` says. An anchor
proves the identifier is on that line and nothing more. Judgements about
ownership, coupling and enforcement are readings of the code. A reading can be
wrong in a way grep cannot catch.
**Marked, not resolved.** Relations carry a `confidence` field. `medium` means
the wiring is in the source and the call path was not traced end to end. `low`
means it was inferred from one reference. The viewer can hide both. Four
relations are `medium` and one is `low`.
**Deployment-specific.** Sixteen components are `configured-off` against
`deploy/mavend.json` as it stood on the day, and that file was dirty in the
working tree. A different config makes different components live. `status` says
which, per component.
## The one thing to check first
`findings.md` 6.3 through 6.3d. They say the system has no single point that
decides whether an origin may cause an effect, and that the pieces which look
like that point are each answering a different question.
Revised on 2026-08-25 after an independent second pass. Four readings changed
and one earlier statement was wrong. Section 6.3 marks the corrections.
Start at `internal/tool/tool.go:181`, `cmd/mavend/ecosystem_acts.go:158` and
`internal/router/claim.go:38`.
## The evidence pack
`sh docs/architecture/pack_evidence.sh` builds `maven-evidence.zip` at the repo
root: this directory, the structural context, and whole source files for the
architectural seams. Whole files, never snippets, because a cut-down file loses
the call path that makes a claim checkable.
The path list is an allowlist, not an exclusion list. A denylist ships whatever
nobody thought to exclude, and this tree has a database key in it.
`architecture-evidence.txt` is the reviewer's index. It resolves a named symbol
list against the checkout and says plainly when a requested name does not exist.
It also re-runs the probe under every contradiction, so a claim and its grep
cannot drift apart.
One file is not verbatim. `docker-compose.yml` carries an uptime-kuma API key,
so a redacted copy ships in its place with that one value replaced. The script
diffs the two and aborts if anything else changed.
The scan at the end refuses to build on a credential-shaped hit rather than
printing a warning. Both of its first two versions were wrong in instructive
ways. The name filter deleted `internal/router/singletoken.go` for matching
`*token*`. The value scan flagged docker volume lines that name where a secret
would live and contain none.
## The authorization function as implemented
The reconstruction, at the one decision point that gates an act
(`internal/tool/tool.go:156`):
```
permit(tool, confirmed) =
row.status == "enabled" tool.go:164
AND tier != irreversible risk.go:74 VoiceMayRun:false
AND (tier == safe OR confirmed) risk.go:72,76
```
`tier` is `RiskOf(row)`. The reach is not an input: `Executor.Exec` takes
`(ctx, name, args, confirmed)` and no surface.
`confirmed` is unproven at this boundary too. The invariant that a confirmation
binds one capability, one target and an expiry lives in `pendingAct` and
`resolveConfirm`. `Exec` trusts the boolean.
The expression covers two of the three act paths. Hexis reuses it deliberately
(`cmd/mavend/ecosystem_acts.go:768`). The Praxis lifecycle path has no tier and
no confirm turn: `praxisItemAction.handle` calls straight through at
`ecosystem_acts.go:158`.
Behind the IPC boundary, `auth.Can(method, scope, params)` runs with
`scope.Surface` always `SurfaceCoreProcess` (`internal/auth/enrollment.go:65`)
and step-up held as one global timestamp that ignores `Scope`
(`internal/webauthn/session.go:38` and `:62`).
`auth` answers who may carry what authority. `tool` answers what effect a
capability has and what proof it demands. Those are orthogonal, not competing.
The decision combining them does not exist.
Two representations of reach exist and both are ignored. `server.go:198` only
defaults an empty `p.Surface`, so a client-asserted one survives and nothing
reads it. `server.go:148` hardcodes `Session.Surface`. Since `req.Surface` is
request payload on an unauthenticated wire, it must not become an authorization
input as it stands.
+290
View File
@@ -0,0 +1,290 @@
#!/usr/bin/env python3
"""Write architecture-evidence.txt: the reviewer's index into the pack.
It resolves a named symbol list against this checkout and prints where each one
is, or says plainly that it does not exist. A requested name that is absent is
evidence too, so nothing here is silently dropped or silently corrected.
Every contradiction is re-checked at generation time by running its own probe,
so the claim and the grep that supports it cannot drift apart in the pack.
python3 docs/architecture/build_evidence.py
"""
import os
import re
import subprocess
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
TREES = ["cmd", "internal"]
# The reviewer's list, verbatim on the left. Where a name does not exist in this
# repo, the right side is what it appears to mean. Resolution below reports both
# so a wrong name is visible rather than quietly fixed.
REQUESTED = [
("auth.TierFor", "auth.MaxLayer"),
("auth.Surface", None),
("auth.Layer", None),
("tool.Executor.Exec", None),
("tool.PolicyFor", None),
("tool.RiskOf", None),
("tool.RiskSafe", "tool.TierSafe"),
("tool.RiskDestructive", "tool.TierDestructive"),
("tool.RiskIrreversible", "tool.TierIrreversible"),
("router.ClaimOf", None),
("reactiveHandler", None),
("tickLoop", None),
]
# Added because the authorization function the reviewer wants to reconstruct
# runs through these and the list above does not reach them.
ALSO = [
"auth.Gate", "auth.Can", "auth.Requirement", "auth.Authority",
"auth.NewFloorEnrollment", "auth.StaticEnrollment", "auth.Scope",
"tool.Policy", "tool.RiskOfCapability", "tool.irreversibleVerbs",
"tool.ErrNeedsConfirm", "tool.ErrNeedsAuthedSurface", "tool.ErrNotEnabled",
"ipc.Server.Check", "ipc.CheckFunc",
"voice.PushToTalkReq", "voice.Sessions.Add", "voice.Session",
"PolicyFor", "RiskOf", "Executor.Exec",
"pendingAct", "resolveConfirm", "actionAct", "runTurn", "applyAction",
"querySources", "queryWalk", "StageZeroGrammars", "Router.Route",
]
PKG_DIR = {
"auth": "internal/auth", "tool": "internal/tool", "claim": "internal/claim",
"modes": "internal/modes", "router": "internal/router", "voice": "internal/voice",
"ipc": "internal/ipc", "store": "internal/store",
}
def go_files(rel):
full = os.path.join(ROOT, rel)
out = []
for base, _, names in os.walk(full):
for n in sorted(names):
if n.endswith(".go"):
out.append(os.path.relpath(os.path.join(base, n), ROOT))
return sorted(out)
def all_go():
out = []
for t in TREES:
out.extend(go_files(t))
return out
_CACHE = {}
def lines_of(rel):
"""Read once. resolve() sweeps every file per pattern per symbol, and
re-reading cmd/ and internal/ that many times took minutes."""
if rel not in _CACHE:
try:
_CACHE[rel] = open(os.path.join(ROOT, rel), errors="replace").read().splitlines()
except OSError:
_CACHE[rel] = []
return _CACHE[rel]
def resolve(sym):
"""Find the declaration of sym. Returns (path, line, text) or (None,)*3."""
tail = sym.split(".")[-1]
recv = sym.split(".")[-2] if sym.count(".") >= 1 else None
pats = [
re.compile(r"^func\s+\(\w+\s+\*?" + re.escape(recv or "\x00") + r"\)\s+" + re.escape(tail) + r"\b"),
re.compile(r"^func\s+" + re.escape(tail) + r"\b"),
re.compile(r"^type\s+" + re.escape(tail) + r"\b"),
re.compile(r"^\s*" + re.escape(tail) + r"\s+\w+\s*=\s"), # typed const
re.compile(r"^\s*" + re.escape(tail) + r"\s*=\s"),
re.compile(r"^(var|const)\s+" + re.escape(tail) + r"\b"),
re.compile(r"^\s*" + re.escape(tail) + r"\s+\w"), # struct field
]
pkg = sym.split(".")[0]
files = go_files(PKG_DIR[pkg]) if pkg in PKG_DIR else all_go()
files = [f for f in files if not f.endswith("_test.go")]
for pat in pats:
for rel in files:
for i, line in enumerate(lines_of(rel), 1):
if pat.match(line):
return rel, i, line.strip()
return None, None, None
def exported(pkg_rel):
"""Every exported declaration in a package, for the `claim.*` / `modes.*` asks."""
out = []
pat = re.compile(r"^(func|type|const|var)\s+\(?[^)]*\)?\s*([A-Z]\w*)")
fn = re.compile(r"^func\s+(\([^)]*\)\s*)?([A-Z]\w*)")
for rel in go_files(pkg_rel):
if rel.endswith("_test.go"):
continue
for i, line in enumerate(lines_of(rel), 1):
m = fn.match(line) or pat.match(line)
if m:
name = m.group(m.lastindex)
if name and name[0].isupper():
out.append((name, f"{rel}:{i}", line.strip()))
return out
def sh(cmd):
return subprocess.run(cmd, shell=True, cwd=ROOT, capture_output=True,
text=True).stdout.strip()
# Each probe is a shell command whose output IS the evidence. Re-run at pack
# time so the pack cannot claim something the checkout no longer shows.
CONTRADICTIONS = [
("auth surface/layer documented as control, not consumed on turn path",
# px.Surface is the Praxis lifecycle verb, an unrelated name collision, and
# Surfaced* are the read-out-item helpers. Excluded by name so the absence
# this probe reports is the auth Surface and not a filtering accident.
"grep -rnE 'req\\.Surface|sess\\.Surface|Session\\.Surface|auth\\.Surface|voice\\.Surface' cmd/mavend/*.go "
"| grep -v _test | grep -vE 'px\\.Surface|Surfaced' "
"|| echo '(no match: no file in cmd/mavend reads the auth Surface of a request or a session)'"),
("auth.Can runs only behind the IPC boundary",
"grep -rn 'auth\\.' cmd/ internal/ --include='*.go' | grep -v _test "
"| grep -v '^internal/auth/' | grep -vE ':[0-9]+:\\s*(//|\\*)'"),
("tool risk policy live on execution path",
"sed -n '176,190p' internal/tool/tool.go"),
("tool executor receives no reach/surface",
"grep -n 'func (e \\*Executor) Exec' internal/tool/tool.go"),
("voice server normalizes incoming surface to pc-client",
"grep -n 'SurfacePCClient' internal/voice/server.go"),
("claim abstraction exists but Route does not consume it",
"grep -rn 'ClaimOf' --include='*.go' cmd internal | grep -v _test || echo '(only the definition; no caller)'"),
("modes package is imported by nothing",
"grep -rn 'internal/modes' --include='*.go' cmd internal | grep -v '^internal/modes/' || echo '(no importer)'"),
("voice server DEFAULTS an empty surface, it does not overwrite a sent one",
"sed -n '193,201p' internal/voice/server.go"),
("session surface is hardcoded, independently of the request field",
"sed -n '146,149p' internal/voice/server.go"),
("HandlePushToTalk never reads req.Surface",
"sed -n '200,203p' cmd/mavend/voice.go"),
("hexis reuses the same risk policy",
"grep -n 'RiskOfCapability\\|PolicyFor' cmd/mavend/ecosystem_acts.go"),
("praxis lifecycle mutations bypass the risk policy entirely",
"sed -n '156,172p' cmd/mavend/ecosystem_acts.go"),
("Exec trusts a confirmed bool it cannot verify was bound",
"grep -n 'func (e \\*Executor) Exec' internal/tool/tool.go; grep -rn 'tools.Exec(' cmd/mavend/*.go | grep -v _test"),
("FloorEnrollment maps every same-uid caller to one surface",
"sed -n '63,72p' internal/auth/enrollment.go"),
("PasskeySession ignores Scope in both methods",
"grep -n 'func (s \\*PasskeySession) CurrentLayer\\|func (s \\*PasskeySession) Assert' internal/webauthn/session.go"),
("Claim.Coverage can be 1.0 with nothing extracted",
"grep -n 'func claimSpans' -A 4 internal/router/claim.go; grep -n 'd.Slots.Text = ex.Text' -B 2 internal/router/router.go; grep -n 'func (c Claim) Coverage' -A 7 internal/claim/claim.go"),
("claim_test asserts Band only, and every case sets Text == Utterance",
"grep -n 'Utterance:\\|Text:\\|want:' internal/router/claim_test.go | head -20"),
("systemctl reboot is destructive, not irreversible",
"grep -n 'irreversibleVerbs = map' -A 8 internal/tool/risk.go; grep -n 'reboot' deploy/mavend.json"),
]
def main() -> int:
out = []
w = out.append
w("architecture evidence pack")
w("=" * 72)
w("")
w("commit: " + sh("git rev-parse HEAD"))
w("date: " + sh("git log -1 --format=%cd --date=short"))
w("branch: " + sh("git rev-parse --abbrev-ref HEAD"))
w("")
w("working tree at pack time (git status --short):")
for line in (sh("git status --short") or "(clean)").splitlines():
w(" " + line)
w("")
w("The pack is built from the WORKING TREE, not from the commit. The lines")
w("above are the difference. deploy/mavend.json in particular is modified:")
w("phraser.model_path points at maven-instruct-b2, the committed value was")
w("Qwen3-1.7B-UD-Q4_K_XL. Sixteen 'configured-off' claims read this file.")
w("")
w("requested symbols")
w("-" * 72)
for name, actual in REQUESTED:
rel, line, text = resolve(name)
if rel:
w(f"- {name}")
w(f" {rel}:{line} {text}")
elif actual:
arel, aline, atext = resolve(actual)
w(f"- {name} -> DOES NOT EXIST in this repo")
if arel:
w(f" the name appears to be {actual}")
w(f" {arel}:{aline} {atext}")
else:
w(f" and neither does {actual}")
else:
w(f"- {name} -> NOT FOUND")
w("")
for pkg, rel in (("claim.*", "internal/claim"), ("modes.*", "internal/modes")):
w(f"{pkg} ({rel})")
w("-" * 72)
for name, anchor, text in exported(rel):
w(f"- {name}")
w(f" {anchor} {text}")
w("")
w("additional symbols on the authorization path")
w("-" * 72)
for name in ALSO:
rel, line, text = resolve(name)
w(f"- {name}")
w(f" {rel}:{line} {text}" if rel else " NOT FOUND")
w("")
w("known contradictions, each re-checked at pack time")
w("=" * 72)
w("The command under each claim was run against this checkout just now.")
w("Its output is what follows. Nothing here is transcribed by hand.")
w("")
for claim, cmd in CONTRADICTIONS:
w("- " + claim)
w(" $ " + cmd)
res = sh(cmd)
for line in (res or "(no output)").splitlines():
w(" " + line)
w("")
w("what the pack does NOT contain, and why")
w("=" * 72)
w("- .git, so no history and no gitignored working files travel with it.")
w("- deploy/telegram.env and deploy/db_key.env. The second holds the")
w(" database key. Both are gitignored and present in the working tree.")
w("- docker-compose.yml verbatim. It carries one live-looking credential on")
w(" line 132 (an uptime-kuma API key). The pack ships")
w(" docker-compose.redacted.yml with that one value replaced and nothing")
w(" else changed, so the mavcaldav and mavmaild claims stay checkable.")
w("- models/, deps/, *.db, *.onnx, *.gguf, certs, logs, node_modules.")
w("- internal/session, internal/db and tests/ from the requested list: none")
w(" of the three exists. Sessions live in internal/voice/session.go, the")
w(" store is internal/store, and tests sit beside their code as *_test.go.")
w("")
w("included test files, since the ask named them by subject:")
for pat, label in (
("internal/router", "routing and arbitration"),
("internal/tool", "risk and confirmation"),
("internal/auth", "authorization"),
("internal/claim", "claim"),
("cmd/mavend", "turn path, confirm gate, query chain"),
):
n = sh(f"find {pat} -name '*_test.go' | wc -l")
w(f" {label}: {n} *_test.go under {pat}/")
w("")
w("fixtures are synthetic, not captured speech: internal/router/eval/*.json")
w("and cmd/mavend/testdata/**.json are hand-written contracts. Named here")
w("because they are Russian utterances and look like personal data.")
path = os.path.join(HERE, "architecture-evidence.txt")
open(path, "w").write("\n".join(out) + "\n")
print(f"architecture-evidence.txt: {os.path.getsize(path)} bytes, {len(out)} lines")
return 0
if __name__ == "__main__":
sys.exit(main())
File diff suppressed because it is too large Load Diff
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""Assemble index.html from the template, the inventory and the diagrams.
index.html is self-contained on purpose: it opens from the filesystem with no
server, and a browser at file:// refuses to fetch a sibling JSON. So the
inventory, every .mmd source and every rendered .svg are inlined here rather
than loaded at runtime.
Run it through docs/architecture/render.sh, which re-renders the SVGs first.
Running it alone rebuilds the viewer against whatever SVGs are already there.
"""
import json
import os
import yaml
HERE = os.path.dirname(os.path.abspath(__file__))
DIA = os.path.join(HERE, "diagrams")
CAPDIR = os.path.join(os.path.dirname(HERE), "capabilities")
def main() -> None:
arch = json.load(open(os.path.join(HERE, "maven-architecture.json")))
mermaid, svg = {}, {}
for name in sorted(os.listdir(DIA)):
path = os.path.join(DIA, name)
if name.endswith(".mmd"):
mermaid[name] = open(path).read()
elif name.endswith(".svg"):
svg[name] = open(path).read()
# The capability half. Generated beside this one and read here rather than
# re-derived: the ledger's build is the only thing allowed to decide a
# dimension, and a second derivation would drift from it silently.
ledger = yaml.safe_load(open(os.path.join(CAPDIR, "ledger.yaml")))
invariants = yaml.safe_load(
open(os.path.join(CAPDIR, "invariants.yaml")))["invariants"]
payload = (
"const ARCH = " + json.dumps(arch, ensure_ascii=False) + ";\n"
"const CAPS = " + json.dumps(ledger, ensure_ascii=False) + ";\n"
"const INV = " + json.dumps(invariants, ensure_ascii=False) + ";\n"
"const MERMAID = " + json.dumps(mermaid, ensure_ascii=False) + ";\n"
"const SVG = " + json.dumps(svg, ensure_ascii=False) + ";\n"
)
template = open(os.path.join(HERE, "viewer.template.html")).read()
if "/*__DATA__*/" not in template:
raise SystemExit("viewer.template.html has no /*__DATA__*/ marker")
out = os.path.join(HERE, "index.html")
open(out, "w").write(template.replace("/*__DATA__*/", payload))
print(
"index.html: %d bytes, %d components, %d relations, %d diagrams, "
"%d rendered, %d capabilities, %d invariants"
% (os.path.getsize(out), len(arch["components"]), len(arch["edges"]),
len(mermaid), len(svg), len(ledger["capabilities"]), len(invariants))
)
if __name__ == "__main__":
main()
+64
View File
@@ -0,0 +1,64 @@
// Smoke test for index.html's renderers, run by render.sh when node is present.
//
// The viewer has no test harness and a TypeError in a renderer produces a blank
// panel, not an error anyone sees. This runs every view's render function
// against a DOM stub and fails loudly on the first throw. It checks that the
// renderers run over the real data, not that the result looks right.
//
// node docs/architecture/check_viewer.js [path/to/index.html]
const fs = require('fs');
const path = process.argv[2] || __dirname + '/index.html';
const src = fs.readFileSync(path, 'utf8');
const js = src.match(/<script>([\s\S]*)<\/script>/)[1];
const el = () => {
const e = {
innerHTML: '', textContent: '', style: {}, checked: true, value: '',
_written: 0,
dataset: {}, classList: { add(){}, remove(){}, toggle(){} },
querySelectorAll: () => [], querySelector: () => null,
appendChild(){}, addEventListener(){}, scrollIntoView(){},
getBoundingClientRect: () => ({top:0,left:0,width:0,height:0}),
};
return e;
};
// One shared element per id, so a renderer's output can be read back. A stub
// that silently swallows innerHTML would let an empty render pass.
const els = {};
const document = {
getElementById: id => (els[id] = els[id] || el()), querySelectorAll: () => [], querySelector: () => null,
createElementNS: el, createElement: el, addEventListener(){},
};
const window = { addEventListener(){} };
const requestAnimationFrame = () => {};
// `const` inside a direct eval stays in the eval's own scope, so the checks are
// appended to the source and evaluated with it rather than run beside it.
const checks = `
let n = 0;
for (const v of VIEWS) {
setView(v.id);
n++;
}
// Every flow, and every capability's side panel: the branch a click takes.
for (const k of Object.keys(FLOWS)) { S.flow = k; renderFlow(el()); n++; }
setView('c1');
if (els.main.innerHTML.length < 5000) throw new Error('capability matrix rendered ' + els.main.innerHTML.length + ' chars');
for (const c of CAPS.capabilities) {
renderCapSide(c.id);
if (els.side.innerHTML.length < 400) throw new Error('thin panel for ' + c.id);
n++;
}
S.capBy = 'domain'; renderCaps(el()); n++;
setView('c2');
if (els.main.innerHTML.length < 4000) throw new Error('invariants view rendered ' + els.main.innerHTML.length + ' chars');
for (const iv of INV) { invRollup(iv); n++; }
for (const c of ARCH.components) { renderSide(c.id); n++; }
console.log('viewer: ' + n + ' render calls, ' + VIEWS.length + ' views, ' +
CAPS.capabilities.length + ' capabilities, ' + INV.length +
' invariants, no throw');
`;
eval(js + checks);
@@ -0,0 +1,120 @@
%% View 1 — System topology.
%% Runtime processes and external systems, with process boundaries drawn explicitly.
%% mavend is the centre because the code makes it one: it is the only key holder,
%% it owns the store, the IPC socket, the voice listener, the tick loop, eight
%% in-process background workers and the child llama-server.
%% Evidence: docker-compose.yml, cmd/mavend/main.go, cmd/mavend/boot.go,
%% deploy/mavwaked.service, deploy/mavgpud.service.
flowchart LR
subgraph WORKPC["workpc — systemd user units, never in docker-compose"]
direction TB
MAVWAKED["mavwaked<br/>process<br/>arecord · silero VAD · keyword head"]
MAVGPUD["mavgpud<br/>process<br/>GPU supervisor"]
LLAMA_W["llama-server<br/>model · workstation card"]
CW2["CrisperWhisper2 turbo<br/>model · port 8081"]
ALSA["arecord / aplay<br/>external"]
TUNNEL["maven-voice-tunnel.service<br/>ssh · the only path in"]
end
subgraph HOMESRV["homesrv — docker compose project `maven`"]
direction TB
subgraph MAVEND_P["mavend — process boundary · the only key holder"]
direction TB
IPCSRV["IPC server<br/>unix /run/maven/mavend.sock"]
VOICESRV["voice server<br/>TCP 0.0.0.0:9100"]
TURN["reactive handler<br/>the turn pipeline"]
TICK["tick loop<br/>60s"]
WORKERS["8 background workers<br/>tick · fact-enrichment · feed · crawl<br/>voice · mcp · home · memory-eval"]
STORE[("store<br/>sqlite, MaxOpenConns=1")]
end
LLAMA_H["llama-server<br/>model · resident<br/>child process of mavend"]
MAVSTTD["mavsttd<br/>process<br/>whisper.cpp"]
MAVTTSD["mavttsd<br/>process<br/>piper"]
MAVWEB["mavweb<br/>process<br/>HTTP 127.0.0.1:9201"]
MAVPOLL["mavpoll<br/>process<br/>network_mode: host"]
SEARX["SearXNG<br/>external"]
KIWIX["kiwix-server<br/>external"]
NETDATA["netdata<br/>external"]
KUMA["uptime-kuma<br/>external"]
end
subgraph OFF["built, not deployed — commented out in docker-compose.yml"]
direction TB
MAVCALDAV["mavcaldav<br/>process"]
MAVMAILD["mavmaild<br/>process"]
end
subgraph ECO["ecosystem network — external compose project"]
direction TB
NEXUS["Nexus<br/>external · identity"]
PRAXIS["Praxis<br/>external · attention"]
HEXIS["Hexis<br/>external · capabilities"]
end
subgraph NET["internet"]
direction TB
TG["Telegram Bot API<br/>external · via SOCKS relay"]
NTFY["ntfy<br/>external · DISABLED in config"]
ZM["zenmoney<br/>external · no token mounted"]
end
HA["Home Assistant<br/>external · enabled:false"]
%% ---- voice path
ALSA --- MAVWAKED
MAVWAKED -->|"PushToTalk · TCP"| TUNNEL
TUNNEL -->|"ssh to 127.0.0.1:9110"| VOICESRV
VOICESRV -->|"proactive Push on the same conn"| MAVWAKED
%% ---- module IPC
MAVWEB -->|"3 × ipc.Client · unix"| IPCSRV
MAVWEB -->|"POST /api/ptt · TCP mavend:9100"| VOICESRV
MAVPOLL -->|"WriteFact · unix"| IPCSRV
MAVCALDAV -.->|"WriteFact · unix"| IPCSRV
MAVMAILD -.->|"IngestMail · unix"| IPCSRV
%% ---- worker sockets
TURN -->|"worker · unix stt.sock"| MAVSTTD
TURN -->|"worker · unix tts.sock"| MAVTTSD
TURN -->|"HTTP · preferred, silent fallback"| CW2
%% ---- models
MAVEND_P ---|"spawns and owns"| LLAMA_H
MAVGPUD ---|"spawns and supervises"| LLAMA_W
MAVGPUD ---|"spawns and supervises"| CW2
TURN -.->|"llm.Pair · model_disabled:true"| MAVGPUD
%% ---- world and ecosystem
TURN -->|"HTTP · query string only"| SEARX
TURN -->|"HTTP"| KIWIX
TURN -->|"HTTP · v1 contract, correlation id"| NEXUS
TURN -->|"HTTP"| PRAXIS
TURN -->|"HTTP"| HEXIS
MAVWEB -->|"HTTP · read-only panel"| NEXUS
MAVWEB -->|"HTTP · read-only panel"| PRAXIS
MAVWEB -->|"HTTP · read-only panel"| HEXIS
TURN -.->|"HTTP · enabled:false"| HA
%% ---- reaches
TICK -->|"telegram sink"| TG
TG -->|"getUpdates long poll"| TURN
TICK -.->|"ntfy sink · nil, disabled"| NTFY
%% ---- pollers
MAVPOLL --> NETDATA
MAVPOLL --> KUMA
MAVPOLL -.-> ZM
classDef proc fill:#1f3a5f,stroke:#7fb3ff,color:#eaf2ff
classDef ext fill:#3d2f4f,stroke:#c39bd3,color:#f4ecf7
classDef model fill:#4a3a1f,stroke:#e0b050,color:#fff6e0
classDef store fill:#1f4a3a,stroke:#6ed0a8,color:#e8fff5
classDef off fill:#3a3a3a,stroke:#888,color:#ccc,stroke-dasharray:4 3
class MAVWAKED,MAVGPUD,MAVSTTD,MAVTTSD,MAVWEB,MAVPOLL,IPCSRV,VOICESRV,TURN,TICK,WORKERS proc
class ALSA,SEARX,KIWIX,NETDATA,KUMA,NEXUS,PRAXIS,HEXIS,TG,TUNNEL ext
class LLAMA_H,LLAMA_W,CW2 model
class STORE store
class MAVCALDAV,MAVMAILD,NTFY,ZM,HA off
@@ -0,0 +1,160 @@
%% View 2 — Core internals of mavend.
%% The real path, in the order runTurn actually runs it. The sequence is NOT
%% input → routing → intent → state → tools → response: eleven stateful
%% pre-emptors get first refusal BEFORE routing, and a query intent then enters
%% a second, longer arbitration of its own.
%% Evidence: cmd/mavend/voice.go runTurn, cmd/mavend/turnroute.go,
%% cmd/mavend/actions.go, cmd/mavend/actions_query.go, internal/router/router.go.
flowchart TB
subgraph IN["input — three reaches, one pipeline"]
A1["voice.Server<br/>HandlePushToTalk"]
A2["daemonAPI.Chat<br/>mavweb /api/chat"]
A3["telegram poller<br/>getUpdates"]
STT["stt seam<br/>Remote mavsttd · CW2 · Stub"]
end
A1 --> STT --> RT
A2 --> RT
A3 --> A2
RT["runTurn<br/>cmd/mavend/voice.go"]
RT --> D0["decision.With<br/>one arbitration record per turn"]
D0 --> TR0["turnRoute created<br/>sync.Once, on the context"]
subgraph PRE["pre-route ladder — 11 rungs, order load-bearing"]
direction TB
P1["1 expired-clarify notice"]
P2["2 confirm answer<br/>resolveConfirm"]
P3["3 targeted repair"]
P4["3b untargeted repair"]
P5["3c command prohibition"]
P6["4 clarify answer"]
P7["5 quiet toggle"]
P8["5b snooze"]
P9["5c ack"]
P10["5d reminder cancellation"]
P11["5e ordinal selection"]
P1-->P2-->P3-->P4-->P5-->P6-->P7-->P8-->P9-->P10-->P11
end
TR0 --> PRE
PRE -->|"any rung claims"| OUT
subgraph ROUTE["step 6 — the cascade · internal/router"]
direction TB
CONT["continuationDecision<br/>an elliptical follow-up is answered<br/>from the previous turn, not routed"]
S0["stage 0 grammars<br/>StageZeroGrammars · first match wins at 1.0<br/>the ONLY arm that may set SourceAnchored"]
SH["stage 0b routing heads<br/>ONNX softmax over the label set"]
SL["stage 1a LLM router<br/>resident model, grammar-constrained"]
SC["stage 1 classifier<br/>nearest centroid · THE FLOOR<br/>names no destination"]
SE["stage 2 extractor + stage 3 gate"]
CONT -->|"not a continuation"| S0
S0 -->|"no match"| SH
SH -->|"declines"| SL
SL -->|"error or unparsable"| SC
SH --> SE
SL --> SE
SC --> SE
end
PRE -->|"nobody claimed"| ROUTE
ROUTE --> DLG["step 7 dialogue merge<br/>followUpMerge · rememberTurn"]
DLG --> CLAR{"step 8<br/>dec.Clarify OR a required slot missing?"}
CLAR -->|"yes"| ASK["askClarify<br/>park the request, ask one question"]
ASK --> OUT
CLAR -->|"no"| ACT
subgraph ACT["step 9 — actionHandlers, 7 intents"]
direction TB
HF["fact<br/>actions_fact.go"]
HR["reminder<br/>actions_reminder.go"]
HA["act<br/>actions_act.go"]
HN["note"]
HC["chat"]
HS["system"]
HQ["query → the chain"]
end
subgraph QC["the query chain — a SECOND arbitration, 22 sources"]
direction TB
QW["queryWalk<br/>removes only guesses:true sources<br/>when the cascade named a destination"]
Q1["his data<br/>fact-by-key · day-plan · habits · tasks<br/>attention · list · money · history · feeds<br/>home · network · calendar · weather · self<br/>embed · memory · notes"]
QB["personal boundary<br/>the only source a stage 0 anchor may drop"]
Q2["the world<br/>search → kiwix → web → general-knowledge"]
QW --> Q1 --> QB --> Q2
end
HQ --> QC
HF -->|"question-shaped ⇒ re-route"| QC
HF -->|"complaint ⇒ re-route"| HC
subgraph STATE["state and memory"]
direction TB
DB[("store · sqlite<br/>facts · reminders · notes · tools<br/>tasks · lists · nudges")]
VEC[("memory_vectors<br/>brute-force cosine")]
DLGS[("dialogue_sessions<br/>persisted, TTL 2m")]
CLS["clarifyStore<br/>IN MEMORY ONLY, by design"]
PEND["pending act / routine / hexis<br/>3 single-slot registers, one mutex"]
SURF["surfacedItems<br/>last Praxis read-out order"]
RING["decision.Ring<br/>bounded, in memory"]
end
HF --> DB
HF --> VEC
HR --> DB
HN --> DB
HN --> VEC
Q1 --> DB
Q1 --> VEC
DLG --> DLGS
ASK --> CLS
HA --> PEND
QC --> SURF
D0 --> RING
subgraph TOOLS["act execution"]
direction TB
ALLOW[("tools table<br/>only status='enabled' runs")]
EXEC["tool.Executor<br/>+ MCP + Home Assistant callers"]
CONF["destructive confirm turn<br/>binds capability, entity, args, requester, expiry"]
HEX["Hexis capability<br/>entity id resolved via Nexus first"]
end
HA --> ALLOW --> EXEC
HA --> CONF
HA --> HEX
subgraph RESP["response generation"]
direction TB
REP["replier<br/>only when the handler returned \"\""]
PHR["phraser<br/>parseResponseMood is the one parser"]
TTS["tts seam<br/>Remote mavttsd · Stub"]
end
ACT --> RESP
QC --> RESP
REP --> PHR
OUT["reply text<br/>+ notice + resumed question"]
RESP --> OUT
OUT -->|"voice path only"| TTS
subgraph PROACT["the other half of the process — nothing above touches it"]
direction TB
TICKL["tick loop · 60s<br/>13 jobs in one function"]
GTH["loop.Gatherer<br/>one consistent snapshot"]
RUL["loop rules + restraint gate<br/>pure"]
DISP["delivery.Dispatcher<br/>ChannelsFor severity,presence"]
SNK["sinks: voice · ntfy · telegram"]
TICKL --> GTH --> RUL --> TICKL
TICKL --> DISP --> SNK
end
TICKL --> DB
SNK -->|"PushToMostRecent on the request conn"| A1
classDef stage fill:#1f3a5f,stroke:#7fb3ff,color:#eaf2ff
classDef store fill:#1f4a3a,stroke:#6ed0a8,color:#e8fff5
classDef mem fill:#4a3a1f,stroke:#e0b050,color:#fff6e0
classDef danger fill:#4f2626,stroke:#e08080,color:#ffecec
class S0,SH,SL,SC,SE,CONT stage
class DB,VEC,DLGS,ALLOW store
class CLS,PEND,SURF,RING mem
class QB,CONF danger
@@ -0,0 +1,78 @@
%% View 3a — Runtime flow: a reminder request.
%% Traced through cmd/mavend/voice.go runTurn, internal/router/stagezero.go,
%% cmd/mavend/clarify.go, cmd/mavend/actions_reminder.go, cmd/mavend/tick.go,
%% internal/loop/loop.go and internal/delivery/dispatcher.go.
%% Shows the branch where the hour is missing, the parked clarify, the answer
%% turn, the write and the eventual delivery with durable retry.
sequenceDiagram
autonumber
participant K as Owner
participant W as mavwaked
participant V as voice.Server
participant H as reactiveHandler.runTurn
participant PRE as pre-route ladder
participant R as router cascade
participant CL as clarifyStore
participant AR as actionReminder
participant DB as store
participant T as tick loop
participant D as dispatcher
Note over K,W: "Мэйвен, напомни позвонить маме"
K->>W: speech
W->>W: silero VAD + keyword head, score ≥ 0.999
W->>V: PushToTalkReq, one utterance
V->>H: HandlePushToTalk
H->>H: stt seam → text
H->>H: decision.With, turnRoute created
H->>PRE: 11 rungs
PRE-->>H: nobody claims
H->>R: rt.resolve
R->>R: stage 0 ReminderGrammar matches, Stage=0, conf 1.0
R-->>H: IntentReminder, Slots.Text="позвонить маме", HasTime=false
rect rgb(70,40,40)
Note over H,CL: BRANCH — missingFor names `time`, whatever the confidence
H->>H: dec.Clarify false BUT len missingFor > 0 → step 8 fires
H->>CL: Push a PendingQuestion, park the request
H-->>V: "во сколько напомнить?"
V-->>W: reply audio + text
end
Note over K,W: "в семь вечера"
K->>W: speech
W->>V: PushToTalkReq
V->>H: runTurn
H->>PRE: rung 4, resolveClarifyAnswer
PRE->>CL: Pop the parked question
PRE->>R: extractor parses the hour with the SAME parsers stage 2 uses
PRE->>AR: finishClarified → applyAction
Note right of AR: filling in an argument never grants authority
AR->>AR: router.ResolvedTheHour guard
AR->>DB: CreateReminder fire_ts, payload
DB-->>AR: id
AR-->>H: reminderConfirm, phrased FROM THE ROW not the utterance
H-->>V: "хорошо, напомню сегодня в 19:00."
Note over T,D: later — the proactive half, no shared code with the turn path
loop every 60s
T->>DB: Gatherer.GatherState, due reminders, collapsed by group
T->>T: loop.RemindDecisions — reminders BYPASS the restraint gate
alt not cached
T->>T: phraser.PhraseReminder
end
T->>D: DispatchReminder
D->>DB: BeginDeliveryAttempt BEFORE the external send
alt a voice session is live
D->>V: voicesink push on the request conn
else away
D->>D: ntfy is nil (disabled) → telegram
end
alt success
D->>DB: CompleteSuccessfulReminderAttempt + fire the originals, one txn
else failure
D->>DB: advance the persisted bounded backoff, next_attempt_ts
end
end
Note over T,DB: Recurring is NOT on this path. reminders.cron and next_fire_ts<br/>exist since migration #2 and no spoken path writes them.
@@ -0,0 +1,61 @@
%% View 3b — Runtime flow: a factual / state update.
%% Traced through cmd/mavend/actions_fact.go, cmd/mavend/ack.go,
%% cmd/mavend/patterns.go, cmd/mavend/factenrichment.go, cmd/mavend/intake.go
%% and internal/morning.
%% Shows the two re-route branches this handler owns, the vector prune-and-insert,
%% the nudge it can close, and the async entity resolution behind it.
sequenceDiagram
autonumber
participant K as Owner
participant H as runTurn
participant R as router cascade
participant AF as actionFact
participant API as CoreAPI · intakeAPI then storeAPI
participant DB as facts table
participant VEC as memory_vectors
participant BUS as event.Bus
participant FE as fact-enrichment worker
participant NX as Nexus
participant T as tick loop
Note over K,H: "выпил воды"
K->>H: utterance, src=tap:voice
H->>R: rt.resolve
R->>R: stage 0 declines → heads → LLM router → classifier
R-->>H: IntentFact, Slots.Key="water", Slots.Value=...
rect rgb(70,40,40)
Note over AF: two guards that RE-ROUTE rather than write
AF->>AF: router.IsQuestionShaped? → becomes actionQuery, Key cleared
AF->>AF: router.IsTransientComplaint? → becomes actionChat, nothing stored
end
AF->>AF: factConfidence — 1.0 only for a value he actually said
AF->>API: WriteFact kind=self, source=tap:voice, Subject=Key
API->>DB: append-only row
API->>BUS: publish one intake envelope
API-->>AF: factID
AF->>VEC: pruneFactVectors by key
AF->>VEC: EmbedPassage(FactRecallText) then Insert "fact:<key>:<unix>"
Note right of VEC: the FACT is embedded, not the utterance.<br/>The utterance rides along as provenance only
AF->>DB: RecordEvent action+object, for pattern detection
H->>H: step 9b ackFromFact — a fact answering a live nudge closes it as `acted`, silently
par asynchronous, minutes later
FE->>DB: read facts with resolution_state='pending'
FE->>NX: Resolve(Subject)
alt resolved
NX-->>FE: entity_id
FE->>DB: UPDATE entity_id, resolution_state='resolved'
else ambiguous
Note right of FE: candidates are NOT stored —<br/>ambiguity blocks, it does not pick
end
and the next tick
T->>DB: Gatherer reads the same row
T->>T: morning routine item `water` is now evidenced, so it will not nudge
T->>T: detectPatterns scans events for a stable interval
end
Note over DB,VEC: A wrong value is superseded, never overwritten:<br/>voids_id points at the row it cancels, and CorrectValue /<br/>VoidLatestFact drop the key's vectors so recall keeps exactly one.
@@ -0,0 +1,70 @@
%% View 3c — Runtime flow: a world query, tool-backed.
%% Traced through internal/router/worldquery.go, internal/router/source.go,
%% cmd/mavend/actions_query.go queryWalk + querySources, cmd/mavend/personalboundary.go,
%% cmd/mavend/searchwire.go, cmd/mavend/kiwixwire.go.
%% Shows destination anchoring, which sources are skipped and why, and the
%% four-step fallback to the model's own weights.
sequenceDiagram
autonumber
participant K as Owner
participant H as runTurn
participant R as router cascade
participant QC as actionQuery
participant W as queryWalk
participant LOC as local sources
participant PB as personal boundary
participant SX as SearXNG
participant KX as kiwix-server
participant PH as phraser / resident model
participant REC as decision record
Note over K,H: "что такое TCP?"
K->>H: utterance
H->>R: rt.resolve
R->>R: stage 0 — WorldQueryGrammars matches a literal definition frame
R->>R: d.SourceAnchored = true, set HERE and nowhere else
R-->>H: IntentQuery, Source=SourceWorld, anchored
H->>QC: applyAction → actionQuery
QC->>REC: Expect the full 22-source roster
QC->>W: queryWalk(SourceWorld, anchored=true)
rect rgb(70,40,40)
Note over W: removes ONLY sources with guesses:true whose dest ≠ world
W-->>REC: skipped: attention, list, feeds, home, network, weather, self
W-->>REC: skipped: personal boundary — anchored, so a literal pattern may drop it
Note right of W: a model or a softmax naming SourceWorld<br/>would NOT drop the boundary (V-666)
end
W-->>QC: the sources that LOOK still walk, in table order
loop first source to claim answers the turn
QC->>LOC: fact-by-key, day-plan, habits, tasks, money, history, calendar
LOC-->>QC: no rows → pass
QC->>LOC: embed → memory → notes (vector recall, gated by min score + margin)
LOC-->>QC: below the gate → pass
QC->>PB: personal boundary
PB-->>QC: SKIPPED this turn
QC->>SX: Search(utterance verbatim, max 4)
alt results
SX-->>QC: snippets
QC->>PH: phraseSource("search", utterance, evidence)
PH-->>QC: reply
QC->>REC: claimed by "search", and everyone below is NeverAsked
else empty or unreachable
QC->>KX: ZIM search, ru then en
alt hit
KX-->>QC: article snippet
QC->>PH: phraseSource("kiwix", ...)
else miss
QC->>QC: "web" claims only if he named a URL out loud
QC->>PH: queryGeneral — the model answers from its own weights, LAST
end
end
end
QC-->>H: reply text
H-->>K: spoken or written answer
Note over LOC,SX: What leaves the box is the query string and nothing else.<br/>His notes, his facts, the persona block and the history never travel.
Note over W,PB: With no destination named — the classifier arm sets none —<br/>the whole chain walks in table order. That is the floor.
@@ -0,0 +1,115 @@
%% View 4 — State ownership.
%% Every persistent and shared store, its authoritative owner, its writers and
%% readers, its synchronisation boundary and its lifecycle.
%% Red = written by components that do not know about each other.
%% Evidence: internal/store/schema.sql, internal/store/migrations.go,
%% internal/store/crypt.go, cmd/mavend/voice.go, cmd/mavend/tick.go,
%% cmd/mavweb/*.go, cmd/mavpoll/main.go, cmd/mavcaldav/main.go.
flowchart LR
subgraph OWNER["authoritative owner — mavend, the only key holder"]
STORE[("store.Store<br/>SetMaxOpenConns(1)<br/>every write serialised at the db")]
end
subgraph LIFE["lifecycle of the database itself"]
direction TB
ENC[("maven.db.enc<br/>AES-256-GCM at rest<br/>volume dbdata")]
TMP[("/dev/shm/maven-plain.db<br/>tmpfs working copy<br/>dies with the container")]
ENC -->|"Open: decrypt"| TMP
TMP -->|"Close: checkpoint, re-encrypt, atomic rename"| ENC
SEAL["mavseal<br/>recovery only, VACUUM INTO"]
TMP -.->|"when mavend was killed, not stopped"| SEAL
SEAL -.-> ENC
end
STORE --- TMP
%% ------------- multiply written tables
FACTS[("facts<br/>append-only, ts = valid-time<br/>correction sets voids_id")]:::multi
NOTES[("notes<br/>float32 blob, brute-force scan")]:::multi
TOOLS[("tools<br/>only status='enabled' executes")]:::multi
%% ------------- singly owned tables
REM[("reminders")]
NUD[("nudges — the restraint memory<br/>AND the only feedback input")]
VEC[("memory_vectors<br/>marked with the embedder id")]
PRES[("presence_state — singleton row")]
EV[("events")]
PROP[("proposed_routines")]
DIG[("digest_entries — gate-BLOCKED candidates")]
DEL[("delivery_attempts — the outbox")]
ACK[("ack_sends")]
DLGS[("dialogue_sessions — TTL 2m")]
TASKS[("tasks")]
LISTS[("list_items")]
RTR[("routing_traces — 14-day retention")]
RLB[("routing_labels")]
ETR[("ecosystem_traces")]
META[("meta — schema version + embedder marker")]
STORE --- FACTS & NOTES & TOOLS & REM & NUD & VEC & PRES & EV & PROP & DIG & DEL & ACK & DLGS & TASKS & LISTS & RTR & RLB & ETR & META
%% ------------- writers into facts
WF1["actionFact — tap:voice / tap:text"] --> FACTS
WF2["quiet toggle — config fact"] --> FACTS
WF3["mavpoll — poll:netdata, poll:uptimekuma,<br/>infer:wg, poll:zenmoney"] --> FACTS
WF4["mavcaldav — poll:caldav<br/>NOT DEPLOYED"]:::off -.-> FACTS
WF5["mavweb — /api/signal presence,<br/>/api/ambient meeting time"] --> FACTS
WF6["feed + crawl watermarks<br/>crawl:hash:*"] --> FACTS
WF7["fact-enrichment worker<br/>entity_id, resolution_state"] --> FACTS
WF8["tick loop tune()<br/>cooldown:<rule> feedback fact"] --> FACTS
WF9["mavweb /api/revert<br/>voids the latest fact for a key"] --> FACTS
%% ------------- writers into notes
WN1["actionNote"] --> NOTES
WN2["RSS poller — source rss:*"] --> NOTES
WN3["crawl watcher"] --> NOTES
WN4["meeting capture"]:::off -.-> NOTES
WN5["image description"]:::off -.-> NOTES
WN6["netscan record"] --> NOTES
%% ------------- writers into tools
WT1["seedTools from mavend.json"] --> TOOLS
WT2["MCP discovery — proposed"]:::off -.-> TOOLS
WT3["Home Assistant discovery<br/>proposed, always destructive"]:::off -.-> TOOLS
WT4["mavweb POST /tools<br/>the ONLY enable path"] --> TOOLS
%% ------------- readers
FACTS --> RD1["loop.Gatherer — the tick snapshot"]
FACTS --> RD2["queryFactByKey · money · history · morning"]
NOTES --> RD3["queryNotes · queryFeeds · recall"]
VEC --> RD4["queryMemory · queryEmbed"]
TOOLS --> RD5["tool.Matcher + tool.Executor"]
NUD --> RD6["restraint gate · TuneCooldown · UnackedTelegramRules"]
%% ------------- in-memory shared state
subgraph MEM["shared mutable state — process-local, no synchronisation boundary beyond a mutex"]
direction TB
CLS["clarifyStore<br/>per-reach stack · NOT persisted on purpose:<br/>a restart expires the open question"]
PEND["pending act / pendingRoutine / pendingHexis<br/>3 single-slot registers under handler.mu<br/>last-asked wins, TTL each"]
SURF["surfacedItems<br/>replaced by the next digest, NO TTL"]
RING["decision.Ring — bounded, diagnosis only"]
BUS["event.Bus — bounded journal, read surface only"]
TICKM["tickLoop: lastPhrase, lastTrace, digestQ,<br/>routineLast, morningLast, lastProposalAt"]
LASTR["lastRouted — the previous acted turn, for a spoken correction"]
end
H1["reactiveHandler<br/>one instance, called from per-conn goroutines"] --- CLS
H1 --- PEND
H1 --- SURF
H1 --- RING
H1 --- LASTR
TICKL["tickLoop"] --- TICKM
INTAKE["intakeAPI decorator"] --- BUS
%% ------------- outside the database
subgraph OUT["state outside the database"]
direction TB
PK[("passkeys.json<br/>OWNED BY mavweb, not mavend")]:::multi
WK[("wrapped key blob<br/>written by mavend WrapKeyFn,<br/>triggered by mavweb")]:::multi
MAIL[("mavmaild seen-UID file<br/>own volume · NOT DEPLOYED")]:::off
BLOB[("media blobs · retention loop")]:::off
end
PK -.->|"a v1 blob + this file together<br/>recover the database key with no authenticator"| WK
classDef multi fill:#4f2626,stroke:#e08080,color:#ffecec
classDef off fill:#3a3a3a,stroke:#888,color:#ccc,stroke-dasharray:4 3

Some files were not shown because too many files have changed in this diff Show More