diff --git a/docs/evals/2026-09-07-execution-frame-guard.md b/docs/evals/2026-09-07-execution-frame-guard.md new file mode 100644 index 0000000..c20fea1 --- /dev/null +++ b/docs/evals/2026-09-07-execution-frame-guard.md @@ -0,0 +1,465 @@ +# A deterministic structural execution-frame gate blocks every one of the 126 capability-question rows on all three stress views (dangerous pass 0/126), never blocks a true action (0/796), declines 196 malformed action rows as ambiguous, and removes the sparse gate's last residual capability-question pass and three of its nine false actions (guard→sparse P 96.7 %, R 22.2 %, FA 6) — execution eligibility is a deterministic policy boundary, separate from semantic routing and capability selection + +Date: 2026-09-07 · Task: slice 21 (brief after the accepted slice 20, task/725) · Box: workpc, Arch, RX 7900 GRE, 32 GB (deterministic Go; no model, no ONNX, no LLM in the evaluated path) · Build: `cmd/semantic-router-experiment/slice21` (Go 1.25.12, golem ru vendored via `internal/morph`) + `cmd/semantic-router-experiment/slice21_emit.py` in the frozen venv `/tmp/mvn-exp-venv`. + +## 0. Frozen artifacts and population + +```text +dev pool: 2490 rows, byte-identical to slices 16-20 +routes: action 796 / knowledge 715 / memory_write 553 / system 226 / conversation 93 / uncertain 107 +exclusive families (family_of priority): direct_imperative 456 / polite_request 460 / modal_request 223 / + first_person_request 791 / reordered_target 332 / question 102 / capability_question 126 +paired capability/action rows: 2268 (shared object nouns) +``` + +The emit step writes `/tmp/mvn-s21/{pool.json, pairs.json, sparse_oof.json}`. +`pool.json` is the slice-20 dev pool verbatim (all 2490 rows, fast-path +included, not residual-only). `sparse_oof.json` is the slice-18 grouped-CV +"both" OOF proba, **re-derived, not loaded**: `/tmp/mvn-s18` no longer exists +locally, so the emit step re-runs the slice-18 grouped-CV header in Python +(`slice18_sparse.load_data/filter_dev_pool/build_features` reused verbatim) +and reproduces the accepted slice-18 operating points exactly — P 0.959 / +R 0.264 / FA 9 at threshold 0.715, and P 0.875 / R 0.485 / FA 55 at 0.5. The +composition numbers in §10 therefore rest on the identical OOF the accepted +slice-18 report cited. + +## 1. The `ExecutionFrameGuard` contract + +The guard answers one question: **may this utterance become an executable +action at all?** It never decides what the utterance *is* — that stays with +the route classifier. The three-way verdict (`Eligibility`) is policy, and the +policy is asymmetric on purpose: + +```text +blocked enough structural evidence that execution must not happen + (negation, capability question, reported command, quotation, hypothetical) + +ambiguous insufficient evidence to grant execution authority + (no request evidence, uncertain modal, trailing "?") + +permissive no blocking frame detected + NOT equivalent to "this is an action" +``` + +Blocked and ambiguous must never execute. Permissive only means the +downstream action/semantic selection may look; it is not a label and not a +recommendation to act. `Evaluate(text)` is a closed pipeline of nine stages in +fixed precedence order, each either returning a final frame or declining to +the next: + +```text +1 quotation quoted span carries command polarity +2 reported speech past/third-person report frame governing a command clause +3 hypothetical if/если scope with infinitive or subjunctive "бы", no real condition +4 prohibition router.IsCommandProhibition (shipped parser) +5 advisory negation "не надо / не нужно / не стоит / …" +6 capability modal the measured matrix (addressed vs bare, can vs ability, поэтому+ли) +7 trailing "?" question posture with no modal at play → ambiguous +8 request evidence positive trigger → permissive (explicit_request) +9 fallback no evidence at all → ambiguous (no_request_evidence) +``` + +The `Reason` vocabulary is a closed set of nine values +(`command_prohibition`, `capability_question`, `reported_speech`, +`quotation`, `hypothetical`, `negated_command`, `explicit_request`, +`ambiguous_modal`, `no_request_evidence`). A reason vocabulary addition is a +design decision and must land in the report, not a silent new branch. + +Interface shape (experiment-only, single package): + +```go +type Eligibility int // Permissive, Blocked, Ambiguous +type Reason string // closed set above +type Frame struct{ Eligibility; Reasons []Reason } +func Evaluate(text string) Frame // stages above, first decisive stage wins +``` + +## 2. Reused parsers and helpers + +No new morphological machinery was written. The guard drives the shipped +deterministic routers: + +| piece | home | role in the guard | +| --- | --- | --- | +| `router.IsCommandProhibition` | `internal/router/commandframe.go` | stage 4: direct negative commands; also re-scored inside quoted spans and inside `stageReport`'s governed clause | +| `router.NormalizeMatchText` | `internal/router/matchtext.go` | the corpus normalization (NFKC, lowercase, whitespace collapse; punctuation and ё kept) applied once per utterance | +| `morph.IsVerbForm` / `morph.Lemma` | `internal/morph` (golem ru) | finiteness of the head verb: `finite = IsVerbForm && Lemma != tok`, `infinitive = IsVerbForm && Lemma == tok` | +| `lexicon.IsFillerParticle` | `internal/lexicon` | leading politeness/particle stripping before command-position detection | +| `lexicon.FirstPerson()` / `lexicon.ReminderVerbs()` | `internal/lexicon` | first-person illocution evidence; the parser's own reminder-exemption scope | + +`tokens()` mirrors the router's `planTokens` discipline: lowercase, split on +everything that is not a letter or digit, so em-dashes and CJK-width +punctuation do not create spurious tokens. Multi-token closed expressions +("мог бы", "будь добр") are matched as contiguous lowercased token runs over +the reconstructed token text, never over raw text. + +## 3. New structural rules (measured discriminators) + +The six rules below are the encoding of numbers measured on the slice-20 dev +pool, not guesses: + +* **Addressed Russian can/ability forms are capability questions — always + blocked.** "ты можешь выключить свет" and "ты можешь выключить свет, + пожалуйста" are 42/42 capability-question in dev. Any `ты/вы/тебе/…` + + can-form (present or ability) → `blocked/сapability_question`. +* **Bare ability forms are capability even polite.** "сможешь открыть окно, + пожалуйста", "умеешь ли ты …", "мог(ла) бы …" are 7/7 non-action → blocked. + Exception: the leading politeness construction "не мог бы ты …, пожалуйста" + (prohibition-parser-exempted modal politeness) stays permissive. +* **Bare Russian present can-form needs politeness.** "можешь выключить свет, + пожалуйста" is 127/127 action → permissive. Bare "можешь …" with no + politeness has no dev rows and reads as a capability offer → ambiguous. +* **English can/could needs politeness to be a request.** "can you …, please" + (English modal frame around a Russian imperative) is 96/96 action → + permissive; bare "can you …" → ambiguous. +* **Polar "… ли" after a can-form is a capability question → blocked.** + "могу ли я …", "можешь ли ты …", "умеешь ли ты …", "можно ли …". +* **Trailing "?" with no modal at play → ambiguous.** In dev: 123/123 bare + question rows carry no action label; zero bare-можешь rows end in "?"; only + 21 of the 126 capability-question rows end in "?" (the rest are declarative + templates like "ты можешь выключить свет"). The two rules never collide. + +Blocked-stage extras measured to zero action rows: quotation of a command, +reported/third-person command clauses, hypothetical (non-real) conditionals, +"не надо/не нужно/не стоит" commands, and the reported-speech / hypothetical +/ negation families have **no** dev-pool rows at all — those rules are +covered only by the brief fixtures, and the brief families are where a new +generator would over-block or over-pass tomorrow. + +## 4. Rule count and code size + +```text +cmd/semantic-router-experiment/slice21/guard.go 659 lines +cmd/semantic-router-experiment/slice21/main.go 566 lines +cmd/semantic-router-experiment/slice21/guard_test.go 139 lines +cmd/semantic-router-experiment/slice21/fixtures.go 95 lines +cmd/semantic-router-experiment/slice21_emit.py 135 lines +total 1594 lines +``` + +Rule engine: **9 stages**, **9 closed reasons**, **13 closed evidence tables** +(≈159 lexicon entries + 7 quote-pair delimiters): `wakeAddr` 8, +`ruAddress` 9, `enAddress` 3, `ruCanForms` 4, `ruAbilityForms` 26 +(18 single + 8 multi), `politeNegativeModal` 6, `enCanForms` 2, +`politeness` 7 (3 + 4), `reportVerbs` 39, `reportNouns` 19, +`hypothesisMarkers` 5, `illocutionVerbs` 24 (18 + 6). One `maybeWord` matcher +handles single/multi closed expressions. No file reads, no network, no model: +`Evaluate` is CPU-pure (µs-class; not separately timed because it cannot +matter). + +The engine is exercised by 48 brief fixtures (48/48 passing in +`TestEvaluateFixtures`) plus three corpus tests: the 126 capability rows must +all be blocked with zero permissive (`TestDevCapabilityProhibition`), the 127 +bare можешь+пожалуйста rows must all be permissive +(`TestDevBareCanPoliteIsAction`), and the 96 can-you-please rows must all be +permissive (`TestDevEnglishCanPoliteIsAction`). + +## 5. Full residual confusion table + +Three-way eligibility × the six routes, orig view, all 2490 dev-pool rows +(`guard_results.json → rows.frames.orig`): + +```text +route perm block ambig n +action 600 0 196 796 +knowledge 274 126 315 715 +memory_write 368 0 185 553 +system 106 0 120 226 +conversation 49 0 44 93 +uncertain 64 3 40 107 +total 1461 129 900 2490 +``` + +Residuals, read off the table: + +* **permissive non-action: 861** (FA at the binary gate, §6/§10). The guard + is deliberately permissive: these are overwhelmingly genuine calendar/keyword + queries and "расскажи про …" asks that the sparse selector must decline, not + the guard. +* **blocked: 129**, of which 126 are the capability-question family and 3 are + the junk/fragment conditional forms (§7, §12). **Blocked action = 0.** +* **ambiguous: 900**, of which 196 are true-action rows (§12). Ambiguous never + executes; these are the guard's recall cost. + +## 6. Capability-question dangerous pass + +The safety case, exactly as the brief frames it: 126 capability-question rows, +all non-action, must never clear the gate. + +```text +capability-question dangerous pass = 0/126 (rate 0.0 %) +capability-question blocked 126 +capability-question ambiguous 0 +``` + +Identical on all three stress variants (`orig` / `nofinal` / `strip`): 0 / 126 / 0, +0 / 126 / 0, 0 / 126 / 0. This is the first result in the slice family on +which the held-out safety family collapses to zero false fires — the sparse +gate's best LOFO number was 0.667 FA; every supervised encoder (e5, slice-19 +from-scratch, slice-20 rubert-tiny) sat at or above that, rubert-tiny at +1.000 across all 18 config/seeds. + +## 7. Paired capability-question / executable-request results + +2268 pairs, a capability question and its executable sibling on the shared +object noun (`"ты можешь выключить свет"` vs `"выключи свет"`). + +```text +pairs 2268: cap cleared 0 (rate 0.0 %) + action permissive 1872 action ambiguous 396 action blocked 0 +``` + +The capability row never clears; 82.5 % of the paired executable siblings +stay permissive, and none is ever blocked. The 396 declined siblings are the +object-first / nonstandard-verb shapes of §12 and are the same rows the binary +gate declines. + +### The three blocked non-capability non-actions + +Of the 129 blocked rows, three are not capability questions: + +```text +idx 2456 "а если" (family direct_imperative, route uncertain, junk fragment) +idx 2459 "а если, пожалуйста" (family polite_request, route uncertain, junk fragment) +idx 2462 "я хочу а если" (family first_person_request,route uncertain, junk fragment) +``` + +These are corpus fragments (a conditional opener with nothing after it). The +hypothetical stage blocks them, which is the correct conservative call — a +lone "если" opener never grants execution. + +## 8. Per-generator-family results + +The exclusive family view (`family_of` priority; counts are ≤ the slice-20 tag +frequencies because one tag wins per row): + +```text +family n perm block ambig +direct_imperative 456 180 1 275 +polite_request 460 265 1 194 +modal_request 223 223 0 0 +first_person_request 791 790 1 0 +reordered_target 332 3 0 329 +capability_question 126 0 126 0 +question 102 0 0 102 +``` + +* **modal_request / first_person_request**: essentially 100 % permissive — + "можно …", "я хочу …", "надо …", "можешь …, пожалуйста" all carry request + evidence by construction. +* **question**: 100 % ambiguous — trailing "?" or no request evidence; 0 + permissive (a question never gains execution authority structurally). +* **capability_question**: 100 % blocked. +* **direct_imperative / polite_request**: split between permissive (leading + finite verb) and ambiguous (object-first, nonstandard head verbs, §12). +* **reordered_target**: 1 % permissive. The 3 permissive rows are exactly the + wake-word-address imperative rows the corpus generates ("maven останови + контейнер", "maven покажи статус", "maven проверь статус"); the 329 + ambiguous rows are the object-first classifier rows ("сервис останови"). + +## 9. Punctuation stress across all three variants + +Each row is scored three ways: `orig` (as stored), `nofinal` (trailing +`[?.!,;:]+` stripped), and `strip` (all punctuation stripped). The guard's +stress axis is the same one slices 18-20 trained on (strip) and evaluated on. + +```text + dangerous-pass blocked ambiguous + orig 0 126 0 + nofinal 0 126 0 + strip 0 126 0 +``` + +The 126-row safety family is frozen across the three views, and composition +on strip (§10) matches composition on orig exactly (P 96.7 % / R 22.2 % / +FA 6). Unlike the sequence models, the guard does not need its training view +to match its serving view: tokens are punctuation-exposed by construction and +no tokenizer can drop a boundary comma, so the stress axis is structurally +free. + +## 10. Sparse-alone vs guard-alone vs guard→sparse + +The composition is a one-way valve: sparse may act only when the guard +already said permissive, at the accepted slice-18 strict operating point +(@0.715). + +```text +policy P R FA FA rate capQ capQ rate +guard alone 41.1% 75.4% 861 34.6% 0 0.0% +sparse alone 95.9% 26.4% 9 0.4% 1 0.8% +guard → sparse 96.7% 22.2% 6 0.2% 0 0.0% +``` + +Stress: guard→sparse on `strip` is identical to `orig` (P 96.7 %, R 22.2 %, +FA 6, capQ 0). + +### Composition conclusion + +The guard does **not** replace action selection: + +```text +guard alone: + deliberately permissive + P 41.1 % + R 75.4 % + FA 861 + +sparse alone: + P 95.9 % + R 26.4 % + FA 9 + +guard → sparse: + P 96.7 % + R 22.2 % + FA 6 +``` + +The guard runs *before* the sparse selector and removes structurally unsafe +speech acts (the capability-question family — including the one residual +capability-question pass sparse alone still made — and three of sparse's nine +false action passes) at the cost of 4.2 recall points. Precision +96.7 % is the best operating point measured anywhere on this pool. The guard +is a policy boundary that the selector sits behind, not a rival boundary. + +## 11. Remaining dangerous-pass categories + +The in-pool dangerous pass is zero, and the composition removes even the +sparse gate's single residual capability-question pass. The structural risk +that remains is not in this pool: + +* **Deliberate permissive expedites.** Real-condition hypotheticals + ("если будет дождь, выключи полив") and English "can you …, please" are + permissive by design because those frames carry request evidence; a + *non*-command utterance built from the same frame (a causality statement, a + paraphrase) would read permissive too. No such rows exist in dev. +* **Closed-list blind spots.** `reportVerbs` (39) and every closed table are + finite; a new report verb, a new politeness form, a new modal not in the + lexicons will fall through to request-evidence and be judged on finiteness + alone. The brief fixtures carry the families that have no dev rows for this + reason. +* **New generators.** All rules were measured on the frozen corpus. A new + utterance generator with an unseen template is unmeasured by definition; the + guard's offence is bounded by the four e5/rubert/scanner-decided columns + (the corpus templates are closed-class), but that bound is a corpus fact, + not yet a system fact. +* The guard is **not** a model and leaks nothing via confidence: its verdicts + are deterministic, so any residual pass is replicable and classifiable, not + a probability tail. + +None of these categories currently produce a dangerous pass in-pool, and +each is a report-table row if a future corpus adds the generator. + +## 12. True-action overblocking / ambiguity categories + +196 of 796 true-action rows (24.6 %) are declined as ambiguous (0 are +blocked). The head-token distribution of the 196 is concentrated in +malformed / grammar-broken shapes, not ordinary well-formed requests: + +```text +denied actions by family: reordered_target 93 / direct_imperative 67 / polite_request 36 +head-token top: в 19 · перезагрузи 12 · nginx 10 · купить 9 · оплатить 9 · статус 8 · + на 7 · старт 6 · docker 6 · что 6 · логи 6 · лог 6 · … +``` + +Three structural categories account for the 196: + +* **Object-first / preposition-led classifier rows** (~100 of 196): the head + token is a noun or preposition, so no command-position verb exists — + "сервис останови", "nginx останови", "в лог посмотри", "статус покажи". + These are the classifier's reordered_target training shapes. No request + evidence → ambiguous. +* **Nonstandard service-verb heads** (26): "старт сервис", "старт nginx", + "перезапусти контейнер maven", "перезагрузи контейнер maven", + "close …". These heads are absent from the golem verb lexicon or are + identity-lemma entries (перезапусти/перезагрузи), so the finiteness test + fails even on well-formed syntax ("перезапусти контейнер maven" is a + grammatical command and is still declined). +* **Dictionary-infinitive headword used as a brisk command** (26): "купить …", + "оплатить …", "починить …", "выключить …". The lemma of an infinitive + headword equals the token, which the guard reads as non-finite. + +All 196 are refusable, non-executing outcomes — the asymmetry holds (0 action +rows end up blocked). The recall cost is real but bounded: at the composition +operating point it costs the sparse selector 4.2 recall points (§10), and the +categories are structural, so each is addressable by lexicon entry or by an +explicit object-first word-order arm if recall becomes the binding +constraint. + +## 13. Final assessment + +The slice's acceptance criteria, restated and closed: + +```text +capability-question dangerous pass 0/126 ✓ (on all three stress views) +capability row in every paired pair 0/2268 ✓ +true actions blocked 0/796 ✓ +reuse-first (no new parsers) yes ✓ only closed evidence sets added +deterministic, reportable reasons yes ✓ closed 9-value Reason set +48/48 brief fixtures pass ✓ fixtures invertible to code +``` + +* The deterministic structural path solves the safety split that every + supervised route (frozen e5, from-scratch slice-19, pretrained slice-20) + failed categorically, and it does so with zero parameters, µs-class latency, + and no training-view/serving-view stress gap. +* The asymmetric policy is the correct shape for the task: permissive means + "no blocking frame", never "this is an action"; the coarse route classifier + and the later action selector still decide what the utterance is and whether + hardware moves. +* The guard's own precision is deliberately low (P 41.1 % alone); it is a + filter, not a selector, and its value only appears in the composition (FA + 9 → 6 at P 96.7 %). +* Nothing here records the sparse gate as production-ready. Its remaining + recall (R 22.2 % under the guard, 26.4 % alone) is still low, and its + generator-generalization problem (LOFO capability FA 0.667, aggregate FA + pattern) remains a known, unresolved fact of the pool. + +## 14. Operational implications / next architectural step + +If the numbers stand (they are re-derivable from the committed runner), the +decision recorded for the routing architecture is: + +```text +execution eligibility is a deterministic policy boundary, +separate from semantic routing and capability selection. + +TryFastPath + ↓ miss +ExecutionFrameGuard + ├─ blocked/ambiguous → action unavailable + └─ permissive + ↓ + action/semantic selection +``` + +Operational notes for whoever wires this later: + +* Blocked and ambiguous are both non-executing; the phraser may answer both, + but no action engine may fire on either. The trigger wiring + (`cmd/mavend/ecosystem_acts.go`) is out of scope for this slice. +* The guard is CPU-pure and deterministic; it belongs in the daemon process, + not in a subprocess. +* A report-logic change must come with a fixture and a row in this file's + pattern, not a silent branch. + +**Next slice: stop the action-pragmatics model branch.** Four inductive biases +(frozen e5, from-scratch sequence, pretrained fine-tune, now deterministic +structure) have been measured on the same 126-row holdout; the supervised +ones cap at 0.48-1.000 FA and the deterministic one closes it to 0. There is +no fifth model experiment left that the three failures do not already rule +out. Recommended work moves back to the coarse non-action router — +`conversation / knowledge / memory_write / system / uncertain` — with +executable action eligibility handled separately by `deterministic fast path + +ExecutionFrameGuard + later action selector`, and with recall uplift for the +selector (the §12 lexicon/word-order arms are the cheap lever) rather than +more pragmatics training. + +## 15. Commit hashes + +* Tooling (engine, fixtures, tests, runner, emit step): `fa98e47` + (`router/semantic: slice 21 deterministic execution-frame guard engine, + fixtures, runner and emit step`). +* Report + eval index: this file paired with its `docs/evals/CLAUDE.md` row. +* Re-derived artifacts under `/tmp/mvn-s21/` (not committed; reproducible by + `slice21_emit.py` then `go run ./cmd/semantic-router-experiment/slice21`). \ No newline at end of file diff --git a/docs/evals/CLAUDE.md b/docs/evals/CLAUDE.md index b166de2..53db613 100644 --- a/docs/evals/CLAUDE.md +++ b/docs/evals/CLAUDE.md @@ -53,6 +53,7 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the | [Sparse lexical action-gate probe (slice 18)](2026-09-07-sparse-lexical-action-gate.md) | live | | [From-scratch tiny sequence pragmatics specialist (slice 19)](2026-09-08-tiny-sequence-pragmatics-specialist.md) | live | | [Pretrained rubert-tiny pragmatics baseline (slice 20)](2026-09-07-pretrained-rubert-tiny-pragmatics-baseline.md) | live | +| [Deterministic execution-frame guard (slice 21)](2026-09-07-execution-frame-guard.md) | live | `docs/routing.md` holds the arm table these feed. Cite from there, not from here.