Compare commits
79 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1a309b283b | |||
| be71ac406b | |||
| bb6bd8efb9 | |||
| 29f74dd3cc | |||
| bdced83b46 | |||
| fa98e4722e | |||
| de1cb40456 | |||
| dc73fc4e31 | |||
| cf2af6ce1e | |||
| 5397f09785 | |||
| 7d31de5a6b | |||
| 20215650f2 | |||
| f2b65cd5d2 | |||
| 9a25945567 | |||
| e80d45f4b3 | |||
| 4ed0238a25 | |||
| 774c0660b0 | |||
| cbac8b9c4f | |||
| ad3f2d6e99 | |||
| 87411e5fb1 | |||
| 4bb7555459 | |||
| 9df223907b | |||
| f8ec77de8a | |||
| 6397ea1cc9 | |||
| 07bfcea8cc | |||
| 59a0a08d32 | |||
| 1b8ae3c3e5 | |||
| 56051e58c0 | |||
| d63619bd0e | |||
| fb9b719f0b | |||
| 2b38390c87 | |||
| 0ff4562e60 | |||
| bb7c460d12 | |||
| dd98da5c42 | |||
| 9e2abea16b | |||
| b481792c02 | |||
| 5865699a5a | |||
| b6a21c17eb | |||
| 494c719a5d | |||
| dad3cd0738 | |||
| a1e4743492 | |||
| 2ef470c2ba | |||
| 80f807d2ad | |||
| e9049ad27e | |||
| 431e052e6f | |||
| 0a2e194e76 | |||
| 18900dd613 | |||
| 2d42e98871 | |||
| 4a17821b06 | |||
| 4b24fbad98 | |||
| 8d83c66d11 | |||
| a6a62a2d56 | |||
| 05f791735d | |||
| bdd79ad585 | |||
| 66f06796cb | |||
| 06adc4702d | |||
| 1d02ba8936 | |||
| 66c578a6f4 | |||
| 356766bce1 | |||
| 6a402bf556 | |||
| f6d7b05161 | |||
| 025f81e961 | |||
| 064747f192 | |||
| a55d90954a | |||
| 87a3b163e7 | |||
| 2f338a1ab6 | |||
| be062b2d48 | |||
| 7f804b84e7 | |||
| bae81b66c8 | |||
| 8153e5eaa5 | |||
| 41c97bba8d | |||
| f9b0a96d9d | |||
| 40ec0c0d4b | |||
| af6e6c9979 | |||
| 3cced9a2e9 | |||
| bc1ef0f57f | |||
| f002ce0e9c | |||
| 3adfc3e0f9 | |||
| b6666196c1 |
@@ -11,14 +11,18 @@ started with an agent that inferred the goal instead of stating it back.
|
||||
## 0. Get on the branch
|
||||
|
||||
```sh
|
||||
task start <vikunja-id>
|
||||
task start <vikunja-id> # with an id
|
||||
git checkout -b task/<slug> # without one
|
||||
```
|
||||
|
||||
`~/.local/bin/task` owns the branch, the identity and the PR. It cuts
|
||||
`task/<id>-<slug>` off `origin/master` and sets the commit author to the `claude`
|
||||
gitea user. It writes `TASK.md` from the Vikunja task, and pulls any waiting
|
||||
An id is optional (owner's call, 2026-08-25). With one, `~/.local/bin/task` owns
|
||||
the branch, the identity and the PR. It cuts `task/<id>-<slug>` off
|
||||
`origin/master` and sets the commit author to the `claude` gitea user. It writes `TASK.md` from the Vikunja task, and pulls any waiting
|
||||
review comments into `.task/review-comments.md`. Do not hand-roll any of that.
|
||||
|
||||
Without an id, branch by hand and skip `TASK.md`. The user's own brief is then
|
||||
the goal, and step 4 restates it back to him instead.
|
||||
|
||||
`TASK.md` is the brief and it is immutable. If it says a PR already exists, this
|
||||
is a review-fix session and not new work. Read the comments first.
|
||||
|
||||
@@ -34,9 +38,12 @@ If there is no handoff, that is normal. It means the last session closed clean.
|
||||
|
||||
In this order, and stop as soon as you have enough:
|
||||
|
||||
- The Vikunja task, by id. Project Maven is ID 2, MCP at `http://localhost:9100/mcp`.
|
||||
The task description and its comments hold the goal, the constraints, and the
|
||||
assumption ledger. This outranks the handoff on every conflict.
|
||||
- The Vikunja task, if there is one. Project Maven is ID 2, MCP at
|
||||
`http://localhost:9100/mcp`, reachable from workpc only through
|
||||
`ssh -N -f -L 9100:127.0.0.1:9100 kami@192.168.1.104`. A refused connection is
|
||||
the missing tunnel, not an outage. The task description and its comments hold
|
||||
the goal, the constraints, and the assumption ledger. This outranks the handoff
|
||||
on every conflict.
|
||||
- `CLAUDE.md`, the section that covers the area you are about to touch.
|
||||
- The one file under `docs/` that owns the area. Check its `Last verified` line.
|
||||
If the sha is behind the code you are reading, say so in step 4 and trust the code.
|
||||
@@ -44,8 +51,8 @@ In this order, and stop as soon as you have enough:
|
||||
Do not read the dated files under `docs/evals/`. They are measurements from one day,
|
||||
never updated. Read one only when you need the number it recorded.
|
||||
|
||||
If no task id is known, ask for one before doing anything else. Work without a task
|
||||
is work nobody can resume.
|
||||
With no task id, do not ask for one and do not stall. State it in step 4 as
|
||||
`Task: unfiled` and carry on.
|
||||
|
||||
## 3. Look at the ground
|
||||
|
||||
@@ -58,7 +65,7 @@ Write at most five bullets and stop. Do not write code, do not open files to "ch
|
||||
one thing first", do not start with a small safe change.
|
||||
|
||||
```
|
||||
Task: V-359, one line.
|
||||
Task: V-359, one line. `unfiled` when there is no id.
|
||||
Done: what is already on the branch.
|
||||
Next: the one thing this session does.
|
||||
Constraints: what would make this wrong.
|
||||
@@ -67,8 +74,9 @@ Assuming: the beliefs that, if false, waste the session.
|
||||
|
||||
Then ask: is this right? Wait for the answer.
|
||||
|
||||
A corrected assumption goes into the Vikunja task as a comment, not into the handoff.
|
||||
The handoff dies tonight. The task does not.
|
||||
A corrected assumption goes into the Vikunja task as a comment where there is a
|
||||
task, because the handoff dies tonight and the task does not. Unfiled, it goes
|
||||
into the handoff and nowhere else.
|
||||
|
||||
## 5. Then begin
|
||||
|
||||
|
||||
@@ -36,11 +36,13 @@ a commit message, not into a comment in the code.
|
||||
Under 300 changed lines per commit in non-markdown files, enforced by `.githooks/pre-commit`.
|
||||
Markdown is exempt and may land as one batch.
|
||||
|
||||
Each commit is one idea, subject in the repo's voice, lowercase area prefix, and it
|
||||
ends with the Vikunja ref:
|
||||
Each commit is one idea, subject in the repo's voice, lowercase area prefix. A
|
||||
Vikunja ref is welcome where a task exists and is required nowhere: the
|
||||
`commit-msg` hook that demanded it was deleted on 2026-08-25.
|
||||
|
||||
```
|
||||
router: narrow the single-token rule (V-359)
|
||||
router: narrow the single-token rule
|
||||
```
|
||||
|
||||
If a change genuinely cannot split under 300 lines, say why in the commit body before
|
||||
@@ -56,13 +58,17 @@ It refuses a dirty tree, pushes, opens or refreshes the PR against the repo defa
|
||||
branch, labels the Vikunja task in-review, comments the PR url on it, and pushes an
|
||||
ntfy. Do not push by hand and do not call `tea` yourself.
|
||||
|
||||
`task pr` needs an id. On a hand-cut branch with no task, push the branch and open
|
||||
the PR by hand, and skip step 5.
|
||||
|
||||
## 5. Record what `task pr` cannot know
|
||||
|
||||
Comment on the Vikunja task: what you measured, what is still open. List every
|
||||
assumption that turned out to be wrong. If the session found new work, create a task
|
||||
for it now rather than describing it in prose.
|
||||
|
||||
This step is what makes the handoff disposable.
|
||||
This step is what makes the handoff disposable. With no task, it cannot run, so the
|
||||
handoff carries that content instead and stops being disposable. Say so in it.
|
||||
|
||||
## 6. Leave the handoff, or leave none
|
||||
|
||||
@@ -75,7 +81,7 @@ resume, and no history:
|
||||
```markdown
|
||||
# Handoff — <date>
|
||||
|
||||
Task: V-359 <one line>
|
||||
Task: V-359 <one line>, or `unfiled`
|
||||
Branch: task/359-<slug>, cut from master
|
||||
|
||||
## Where I stopped
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Every commit names the Vikunja task it belongs to.
|
||||
#
|
||||
# router: narrow the single-token rule (V-359)
|
||||
#
|
||||
# V- and not #, because Gitea autolinks #359 to a Gitea issue, which is a
|
||||
# different tracker and a wrong link.
|
||||
#
|
||||
# Exempt: merges, reverts, fixup/squash, and the initial commit.
|
||||
|
||||
msg_file=$1
|
||||
subject=$(sed -n '1p' "$msg_file")
|
||||
|
||||
case "$subject" in
|
||||
Merge\ *|Revert\ *|fixup!\ *|squash!\ *|amend!\ *) exit 0 ;;
|
||||
esac
|
||||
|
||||
if [ -f "$(git rev-parse --git-dir)/MERGE_HEAD" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if printf '%s' "$subject" | grep -qE '\(V-[0-9]+\)$'; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "commit-msg: subject must end with a Vikunja task ref." >&2
|
||||
echo " got: $subject" >&2
|
||||
echo " want: router: narrow the single-token rule (V-359)" >&2
|
||||
echo " No task yet? Create one. Work without a task is work nobody can resume." >&2
|
||||
exit 1
|
||||
+15
@@ -76,3 +76,18 @@ __pycache__/
|
||||
.env
|
||||
# silero-vad, downloaded (see AGENTS.md)
|
||||
/models/vad/
|
||||
|
||||
# Go build cache and GOPATH from the containerised e2eprobe build. Created by
|
||||
# the command in docs/capabilities/README.md, which runs as root in a container
|
||||
# and so cannot share the host cache. Multi-GB, entirely reproducible.
|
||||
/.cache/
|
||||
|
||||
# docs/architecture/ derived output. The sources, findings.md and the inventory
|
||||
# JSON are tracked; these rebuild from them with pack_evidence.sh and are large.
|
||||
/docs/architecture/index.html
|
||||
/docs/architecture/anchors.md
|
||||
/docs/architecture/architecture-evidence.txt
|
||||
/docs/architecture/tree.txt
|
||||
/docs/architecture/docker-compose.redacted.yml
|
||||
/docs/architecture/diagrams/*.svg
|
||||
/maven-evidence.zip
|
||||
|
||||
@@ -201,14 +201,14 @@ ToolSearch("select:mcp__vikunja__list_tasks,mcp__vikunja__get_task_details,mcp__
|
||||
```
|
||||
|
||||
- This repo is Vikunja project **Maven** (ID 2), MCP at
|
||||
`http://localhost:9100/mcp`, or `http://192.168.1.104:9100/mcp` from workpc.
|
||||
- **A session with no task id asks for one before it starts**, because work
|
||||
without one is work nobody can resume.
|
||||
`http://localhost:9100/mcp` on homesrv. **`vikunja-mcp` publishes to
|
||||
`127.0.0.1:9100` only, so the LAN address never answers from workpc.** A
|
||||
refused connection is that, not an outage: three sessions read it as "Vikunja
|
||||
is down" and filed nothing. Tunnel first, then use `localhost`:
|
||||
`ssh -N -f -L 9100:127.0.0.1:9100 kami@192.168.1.104`.
|
||||
- **Close a finished task with `done: true` and nothing else** (owner's call,
|
||||
2026-08-07). `update_task` carrying a `description` resets `done` to false.
|
||||
- **`pre-commit` refuses master** and more than 300 changed lines in
|
||||
non-markdown files. Markdown is exempt and may land as one batch.
|
||||
- **`commit-msg` requires the subject to end with `(V-<id>)`.** `V-` and not
|
||||
`#`, because Gitea autolinks `#123` to the wrong tracker.
|
||||
- **`diff-budget.sh` blocks edits past 600 changed lines** on a `task/` branch.
|
||||
- **`--no-verify` exists.** Using it means saying why in the commit body.
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# Handoff
|
||||
|
||||
Master is at `5cae33a`, pushed, tree clean apart from this file. Working on master
|
||||
raw by the owner's call: no branch, `--no-verify` on every commit with the reason
|
||||
in the body.
|
||||
|
||||
## Landed this session
|
||||
|
||||
Twelve commits pushed. Nine were the previous session's tree, already described in
|
||||
the commit log. Three are new:
|
||||
|
||||
- `78a9c61` `docs/spec.md`, 51 capabilities with a DoD each.
|
||||
- `02e3d27` `docs/roadmap.md`, seven milestones, plus both pointer rows in `CLAUDE.md`.
|
||||
- `5cae33a` honesty split into three milestones, five capabilities deferred past v1.
|
||||
|
||||
**Read `docs/spec.md` and `docs/roadmap.md` before anything else.** Every decision
|
||||
from this session is in them. This file holds only what they do not.
|
||||
|
||||
## The two documents
|
||||
|
||||
`docs/spec.md` is the union of the 39 audited rows
|
||||
(`docs/evals/2026-08-13-capability-audit.md`) and the owner's 18-item v1 list.
|
||||
Twelve of his items had no audit row, so the file has 51 entries. Each entry
|
||||
carries three parts. State is a reference to the living doc that owns it. DoD is
|
||||
a plain list observable on the running box. Scenario names a file in
|
||||
`cmd/mavend/testdata/scenarios/`.
|
||||
|
||||
`docs/roadmap.md` orders them into nine milestones. Honesty, then reach, then
|
||||
breadth. Not ordered by code work, because none of the four broken capabilities is
|
||||
a code defect.
|
||||
|
||||
## Decided, do not re-ask
|
||||
|
||||
- **v1 is a voice assistant, minimum viable.** Each DoD is written at
|
||||
"voice-reachable and honest", not "feature-complete".
|
||||
- **Honesty splits into three.** M1 the turn path, M2 memory he cannot correct,
|
||||
M3 step-up. M1 and M2 touch different code and owe different docs. Step-up is
|
||||
configuration, not honesty, and sits before M4 because M4 is what first makes
|
||||
acts real.
|
||||
- **Five capabilities deferred past v1** (owner's call, 2026-08-15): speaker
|
||||
recognition, smart home, bluetooth control, model swap, self-update. Bluetooth
|
||||
was on the v1 list and came off it. Their spec entries keep their DoD.
|
||||
- **A milestone closes its own doc gaps and writes its own scenarios.** Neither
|
||||
becomes a milestone of its own. Otherwise the 17 missing docs and 46 missing
|
||||
scenario files collect at the end.
|
||||
- **Learning means behavioral, not weights.** Stored outcomes only. No adapter, no
|
||||
training set.
|
||||
- **Email and calendar need the product decision before deploying.** Both are
|
||||
built and neither is in `docker-compose.yml`. That is M8.
|
||||
|
||||
## Findings the spec pass produced
|
||||
|
||||
- **Recurring reminders do not exist on the spoken path.** `store.Reminder` carries
|
||||
`Cron` and `ipc.CreateReminder` takes one. `grep "Cron:" --include='*.go'`
|
||||
outside tests returns only `internal/ipc/client.go`, `internal/ipc/storeapi.go`
|
||||
and `cmd/mavend/tick_routines.go`, and the last is routines, a separate
|
||||
mechanism. Pills, the dog, the vet and the kibble are unbuilt on top of finished
|
||||
storage and delivery. This is M6.
|
||||
- **Seventeen capabilities have no living doc.** Memory is the worst cluster:
|
||||
facts, notes and the digestion worker have no owning document at all.
|
||||
- **Webhooks barely exist.** The only one in the tree is
|
||||
`internal/delivery/telegramsink/intake.go`, Telegram's own inbound hook.
|
||||
- **Command chaining does not exist.** The `chain` in `internal/router` is the
|
||||
world chain and the source chain.
|
||||
- **Desk notifications are inbound only.** `ambient:notif` reads his desktop.
|
||||
There is no outbound desk reach, and which direction he meant is undecided.
|
||||
- **Only 5 of 51 spec entries cite a scenario that exists.**
|
||||
|
||||
## Not filed, and this is the risk
|
||||
|
||||
Vikunja returned 503 across this session and the last, so **none of this has a
|
||||
task id**. The three commits above are tagged `V-719`, which is the
|
||||
reminder-cancellation task, not this work. Retag or file when Vikunja is back.
|
||||
|
||||
Unfiled, listed again at the end of `docs/roadmap.md`:
|
||||
|
||||
1. The capability audit itself, headed "unfiled".
|
||||
2. Remember-versus-query misroute, two of seven audit probes.
|
||||
3. The masculine reply on the wire, caught live while `CheckFeminine` passed.
|
||||
4. Recurring reminders having no caller.
|
||||
5. The seventeen capabilities with no living doc.
|
||||
6. The 46 scenario files the spec names and does not have.
|
||||
|
||||
## Next
|
||||
|
||||
Open a session on M1, which is three gate items on the turn path and owes no new
|
||||
doc. If Vikunja is up, file the six above first and give M1 a real id.
|
||||
|
||||
One command still outstanding from the last session, cheap and unrelated:
|
||||
|
||||
```sh
|
||||
docker compose up -d --force-recreate mavsttd mavttsd mavpoll
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,92 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/kami/maven/internal/decision"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// resolveAction produces an ActionCandidate from a routing decision. It is the
|
||||
// single boundary between routing and action execution: everything downstream
|
||||
// (refusesCommand, task-status, Praxis, Hexis, proposeGap, tool.Executor.Exec)
|
||||
// consumes the candidate rather than re-resolving the function.
|
||||
//
|
||||
// Delegates to router.ResolveActionCandidate for the resolution logic, then
|
||||
// records the outcome in the decision trace.
|
||||
func (h *reactiveHandler) resolveAction(ctx context.Context, dec router.Decision) router.ActionCandidate {
|
||||
candidate := router.ResolveActionCandidate(dec, h.matcher)
|
||||
|
||||
// Record the resolution outcome in the decision trace.
|
||||
if dec.Intent == router.IntentAct {
|
||||
if candidate.ActionResolved() {
|
||||
noteActionResolution(ctx, string(candidate.Source), candidate.Fn, true)
|
||||
} else {
|
||||
noteActionResolution(ctx, "matcher", "", false)
|
||||
}
|
||||
}
|
||||
|
||||
return candidate
|
||||
}
|
||||
|
||||
// noteActionResolution records the action resolution outcome in the decision
|
||||
// trace. A nil recorder is the normal case in tests.
|
||||
func noteActionResolution(ctx context.Context, source, fn string, resolved bool) {
|
||||
rec := decision.From(ctx)
|
||||
if rec == nil {
|
||||
return
|
||||
}
|
||||
outcome := decision.Declined
|
||||
reason := "no match"
|
||||
if resolved {
|
||||
outcome = decision.Won
|
||||
reason = "resolved via " + source
|
||||
if fn != "" {
|
||||
reason += ": " + fn
|
||||
}
|
||||
}
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-resolve",
|
||||
Outcome: outcome,
|
||||
Reason: reason,
|
||||
})
|
||||
}
|
||||
|
||||
// noteActionValidation records the structural validation outcome in the
|
||||
// decision trace. Five outcomes: unresolved (matcher miss), valid
|
||||
// (structurally admissible), invalid_argument, missing_argument, or
|
||||
// ambiguous_target (structurally malformed).
|
||||
func noteActionValidation(ctx context.Context, v router.ActionValidationResult) {
|
||||
rec := decision.From(ctx)
|
||||
if rec == nil {
|
||||
return
|
||||
}
|
||||
switch v.Status {
|
||||
case router.ActionUnresolved:
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-validation",
|
||||
Outcome: decision.Declined,
|
||||
Reason: "unresolved",
|
||||
})
|
||||
case router.ActionValid:
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-validation",
|
||||
Outcome: decision.Won,
|
||||
Reason: "valid",
|
||||
})
|
||||
default:
|
||||
reason := string(v.Status)
|
||||
if len(v.Issues) > 0 {
|
||||
reason = string(v.Status) + ":" + v.Issues[0].Reason
|
||||
}
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-validation",
|
||||
Outcome: decision.Declined,
|
||||
Reason: reason,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,556 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/decision"
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
"github.com/kami/maven/internal/tool"
|
||||
)
|
||||
|
||||
// newActHandler builds a handler with the act path wired: a matcher over
|
||||
// whatever tools the test enabled, no model, no ecosystem.
|
||||
func newActHandler(t *testing.T) (*reactiveHandler, *store.Store) {
|
||||
t.Helper()
|
||||
st := newTestStore(t)
|
||||
api := ipc.NewStoreAPI(st)
|
||||
matcher := tool.NewMatcher(api)
|
||||
h := &reactiveHandler{
|
||||
api: api,
|
||||
tools: tool.NewExecutor(api, 2*time.Second),
|
||||
matcher: matcher,
|
||||
now: func() time.Time { return time.Date(2026, 8, 1, 9, 0, 0, 0, time.UTC) },
|
||||
}
|
||||
return h, st
|
||||
}
|
||||
|
||||
// TestActRouteSource_NoMatcherInvoke pins that an act with HasFn=true
|
||||
// produces a candidate from the route and does not invoke the matcher.
|
||||
func TestActRouteSource_NoMatcherInvoke(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Enable a tool so the matcher has something to match against.
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act with HasFn=true: the candidate must come from the route.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("route-sourced act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActMatcherSource_FallbackMatch pins that an act without Fn invokes
|
||||
// the matcher and produces a matcher-sourced candidate.
|
||||
func TestActMatcherSource_FallbackMatch(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Enable a tool so the matcher can find it.
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act without HasFn: the matcher must resolve "status" from the text.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "check status",
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("matcher-sourced act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActMatcherMiss_ProposeGap pins that a matcher miss produces the
|
||||
// same propose-gap behavior as before.
|
||||
func TestActMatcherMiss_ProposeGap(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Enable one tool so the matcher has an allowlist, but not the one asked for.
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act without HasFn and text that doesn't match any tool.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "deploy the thing",
|
||||
Slots: router.Slots{Text: "deploy the thing"},
|
||||
})
|
||||
if !strings.Contains(strings.ToLower(reply), "предлож") {
|
||||
t.Errorf("matcher miss replied %q; want propose-gap behavior", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActDestructive_ConfirmationUnchanged pins that a destructive tool
|
||||
// still triggers the confirmation flow.
|
||||
func TestActDestructive_ConfirmationUnchanged(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart",
|
||||
Slots: router.Slots{Fn: "restart", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("destructive act replied %q; want a confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActTaskStatus_InterceptUnchanged pins that task_status is intercepted
|
||||
// before reaching the tool executor.
|
||||
func TestActTaskStatus_InterceptUnchanged(t *testing.T) {
|
||||
h, _ := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "task status",
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true,
|
||||
Text: "task status"},
|
||||
})
|
||||
// task_status is intercepted by resolveTaskStatus, which returns a
|
||||
// status phrase. The exact reply depends on the store state, but it
|
||||
// must not be a tool execution result.
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Errorf("task_status was not intercepted, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActStage0_SameResult pins that a stage-0 act (grammar match with
|
||||
// HasFn=true) produces the same tool execution as before.
|
||||
func TestActStage0_SameResult(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Stage: 0,
|
||||
Confidence: 1.0,
|
||||
Utterance: "maven, restart nginx",
|
||||
Slots: router.Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
Producer: router.RouteProducerGrammar,
|
||||
})
|
||||
if !strings.Contains(reply, "сделала") && !strings.Contains(reply, "готово") {
|
||||
t.Errorf("stage-0 act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActLearnedRouter_NoFn_FallbackMatch pins that a learned-router act
|
||||
// without Fn falls through to the matcher and produces the same result.
|
||||
func TestActLearnedRouter_NoFn_FallbackMatch(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// LLM routed the act but did not fill Fn (common when the model returns
|
||||
// the verb in Text but not in Fn).
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Stage: 1,
|
||||
Confidence: 0.85,
|
||||
Utterance: "could you restart nginx",
|
||||
Slots: router.Slots{Text: "restart nginx"},
|
||||
Producer: router.RouteProducerLLM,
|
||||
})
|
||||
if !strings.Contains(reply, "сделала") && !strings.Contains(reply, "готово") {
|
||||
t.Errorf("learned-router act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveAction_CandidateSource_Verified pins the candidate source
|
||||
// for both route-resolved and matcher-resolved actions.
|
||||
func TestResolveAction_CandidateSource_Verified(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Route-resolved: HasFn=true.
|
||||
c1 := h.resolveAction(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
if c1.Source != router.ActionSourceRoute {
|
||||
t.Errorf("route candidate source = %q, want route", c1.Source)
|
||||
}
|
||||
if c1.Fn != "status" {
|
||||
t.Errorf("route candidate Fn = %q, want status", c1.Fn)
|
||||
}
|
||||
|
||||
// Matcher-resolved: no Fn, text matches.
|
||||
c2 := h.resolveAction(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if c2.Source != router.ActionSourceMatcher {
|
||||
t.Errorf("matcher candidate source = %q, want matcher", c2.Source)
|
||||
}
|
||||
if c2.Fn != "status" {
|
||||
t.Errorf("matcher candidate Fn = %q, want status", c2.Fn)
|
||||
}
|
||||
|
||||
// Matcher miss: no Fn, text doesn't match.
|
||||
c3 := h.resolveAction(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "deploy everything"},
|
||||
})
|
||||
if c3.ActionResolved() {
|
||||
t.Errorf("miss candidate resolved = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
// --- structural validation integration tests ---
|
||||
|
||||
// TestActValidation_MalformedCandidate_BlankFn pins that a resolved
|
||||
// candidate with a blank (whitespace-only) Fn does not execute and
|
||||
// produces a failure response.
|
||||
func TestActValidation_MalformedCandidate_BlankFn(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Simulate a malformed candidate by writing a blank Fn into Slots
|
||||
// after resolution. This tests that the validation layer catches
|
||||
// structurally invalid candidates.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: " ", HasFn: true},
|
||||
})
|
||||
// The blank Fn should not reach tool execution. It either hits
|
||||
// the validation gate (ActFail) or the existing error paths.
|
||||
if reply == "" {
|
||||
t.Error("expected a response, got empty string")
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_UnresolvedCandidate_ProposeGap pins that an unresolved
|
||||
// candidate (matcher miss) still flows to proposeGap, unchanged.
|
||||
func TestActValidation_UnresolvedCandidate_ProposeGap(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "deploy everything",
|
||||
Slots: router.Slots{Text: "deploy everything"},
|
||||
})
|
||||
if !strings.Contains(strings.ToLower(reply), "предлож") {
|
||||
t.Errorf("unresolved candidate replied %q; want propose-gap behavior", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_DestructiveValid_StillConfirms pins that a destructive
|
||||
// valid action still reaches the confirmation path through validation.
|
||||
func TestActValidation_DestructiveValid_StillConfirms(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart",
|
||||
Slots: router.Slots{Fn: "restart", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("destructive valid act replied %q; want confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_IrreversibleValid_NeedsAuthedSurface pins that an
|
||||
// irreversible valid action still reaches ErrNeedsAuthedSurface.
|
||||
func TestActValidation_IrreversibleValid_NeedsAuthedSurface(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Register an irreversible tool: cmd containing "drop" triggers the
|
||||
// irreversible tier via RiskOf → isIrreversible.
|
||||
if err := st.EnableTool(ctx, "drop_table", []string{"drop"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "drop_table",
|
||||
Slots: router.Slots{Fn: "drop_table", HasFn: true},
|
||||
})
|
||||
// Irreversible tools return ErrNeedsAuthedSurface, which produces
|
||||
// a specific phraser response.
|
||||
if !strings.Contains(reply, "выполню") && !strings.Contains(reply, "запусти") {
|
||||
t.Errorf("irreversible valid act replied %q; want authed-surface response", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_ValidationTracing pins that validation outcomes are
|
||||
// recorded in the decision trace.
|
||||
func TestActValidation_ValidationTracing(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
now := h.now()
|
||||
|
||||
// Valid candidate: trace should show action-validation:won.
|
||||
ctx, rec := decision.With(context.Background(), "status", "tap:text")
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
records := rec.Claims
|
||||
found := false
|
||||
for _, c := range records {
|
||||
if c.Claimant == "action-validation" && c.Outcome == decision.Won {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("expected action-validation:won in trace, got %v", records)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActExecutionFromCandidateNotSlots pins that downstream execution reads
|
||||
// resolved action data from ActionCandidate, not from Decision.Slots. The
|
||||
// decision has empty Fn/Args/HasFn — the bridge used to copy candidate values
|
||||
// back into these fields. After the bridge removal, execution must still
|
||||
// succeed because the candidate carries the resolved function.
|
||||
func TestActExecutionFromCandidateNotSlots(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act without any Fn/Args/HasFn in Slots — the matcher resolves from Text.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "check status",
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("execution from candidate replied %q; want tool success", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// --- validation status boundary tests ---
|
||||
|
||||
// TestActValidation_StatusValidRoute pins that a route-resolved valid action
|
||||
// produces ActionValid status and reaches execution.
|
||||
func TestActValidation_StatusValidRoute(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("valid route act replied %q; want tool success", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_StatusValidMatcher pins that a matcher-resolved valid
|
||||
// action produces ActionValid status and reaches execution.
|
||||
func TestActValidation_StatusValidMatcher(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "check status",
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("valid matcher act replied %q; want tool success", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_StatusUnresolved pins that an unresolved candidate
|
||||
// produces ActionUnresolved status and flows to proposeGap.
|
||||
func TestActValidation_StatusUnresolved(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "deploy everything",
|
||||
Slots: router.Slots{Text: "deploy everything"},
|
||||
})
|
||||
if !strings.Contains(strings.ToLower(reply), "предлож") {
|
||||
t.Errorf("unresolved act replied %q; want propose-gap", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_StatusInvalid pins that a structurally invalid candidate
|
||||
// produces ActionInvalidArgument status and refuses execution.
|
||||
func TestActValidation_StatusInvalid(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: " ", HasFn: true},
|
||||
})
|
||||
if reply == "" {
|
||||
t.Error("expected a response for invalid candidate")
|
||||
}
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Error("invalid candidate should not reach tool execution")
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_DestructiveValidStatus pins that a destructive but
|
||||
// structurally valid action still produces ActionValid status and reaches
|
||||
// the confirmation path (not validation failure).
|
||||
func TestActValidation_DestructiveValidStatus(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart",
|
||||
Slots: router.Slots{Fn: "restart", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("destructive valid act replied %q; want confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_ConfirmationUnchanged pins that the confirmation flow
|
||||
// is unchanged by validation.
|
||||
func TestActValidation_ConfirmationUnchanged(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart nginx",
|
||||
Slots: router.Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("confirmation act replied %q; want confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_TaskStatusInterceptUnchanged pins that task_status
|
||||
// interception is unchanged by validation.
|
||||
func TestActValidation_TaskStatusInterceptUnchanged(t *testing.T) {
|
||||
h, _ := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "task status",
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Text: "task status"},
|
||||
})
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Errorf("task_status was not intercepted, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_NoExecutionOnFailure pins that validation failure
|
||||
// prevents downstream execution.
|
||||
func TestActValidation_NoExecutionOnFailure(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: " ", HasFn: true},
|
||||
})
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Error("validation failure should not reach tool execution")
|
||||
}
|
||||
}
|
||||
+33
-21
@@ -11,9 +11,14 @@ import (
|
||||
"github.com/kami/maven/internal/tool"
|
||||
)
|
||||
|
||||
// actionAct handles router.IntentAct: match a verb to an enabled tool, offer
|
||||
// it to the ecosystems first, and run it behind the confirm gate and the
|
||||
// allowlist. proposeGap and the confirm gate itself live in confirm.go.
|
||||
// actionAct handles router.IntentAct: resolve the action, offer it to the
|
||||
// ecosystems first, and run it behind the confirm gate and the allowlist.
|
||||
// proposeGap and the confirm gate itself live in confirm.go.
|
||||
//
|
||||
// Action resolution happens in resolveAction (actionresolve.go) — a single
|
||||
// boundary that produces an ActionCandidate before execution. This function
|
||||
// consumes the candidate; it no longer decides which function/tool the user
|
||||
// meant.
|
||||
func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) string {
|
||||
// An allowlist or a model route is evidence about WHAT could run, never
|
||||
// authority to run it. Keep the user's negative command at the execution
|
||||
@@ -23,26 +28,33 @@ func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) st
|
||||
return commandProhibitionReply
|
||||
}
|
||||
|
||||
// tool executor: run the matched fn against the enabled allowlist.
|
||||
// HasFn=false ⇒ try the matcher (for LLM-routed acts where the verb
|
||||
// didn't go through the stage-0 act grammar).
|
||||
if !dec.Slots.HasFn && dec.Slots.Text != "" && h.matcher != nil {
|
||||
if fn, args, ok := h.matcher.Match(dec.Slots.Text); ok {
|
||||
dec.Slots.Fn, dec.Slots.Args, dec.Slots.HasFn = fn, args, true
|
||||
}
|
||||
// Resolve the action: produce an ActionCandidate from the routing
|
||||
// decision. The candidate carries the resolved function, its arguments,
|
||||
// and where the resolution came from (route or matcher).
|
||||
candidate := h.resolveAction(ctx, dec)
|
||||
|
||||
// Structural validation: is this candidate complete enough to proceed?
|
||||
// Unresolved (Fn empty) flows to proposeGap; invalid (Fn present but
|
||||
// malformed) is refused; valid proceeds to execution.
|
||||
validation := router.ValidateActionCandidate(candidate)
|
||||
noteActionValidation(ctx, validation)
|
||||
|
||||
if !validation.Unresolved() && !validation.Valid() {
|
||||
// Resolved but structurally malformed: refuse execution.
|
||||
return phraser.A(phraser.ActFail, nil)
|
||||
}
|
||||
|
||||
// The board is Maven's own store, so a spoken status change is answered here
|
||||
// and never offered to an ecosystem client (Vikunja #512). First, because
|
||||
// task_status is on no allowlist and no capability registry: reaching either
|
||||
// of them would answer a turn about his own task list with a gap.
|
||||
if dec.Slots.Fn == router.TaskStatusFn {
|
||||
return h.resolveTaskStatus(ctx, dec)
|
||||
if candidate.Fn == router.TaskStatusFn {
|
||||
return h.resolveTaskStatus(ctx, dec, candidate)
|
||||
}
|
||||
|
||||
// Praxis ecosystem tools: intercept before the system command executor.
|
||||
if h.ecosystem != nil && h.ecosystem.praxis != nil && dec.Slots.HasFn {
|
||||
if reply := h.handlePraxisAct(ctx, dec); reply != "" {
|
||||
if h.ecosystem != nil && h.ecosystem.praxis != nil && candidate.ActionResolved() {
|
||||
if reply := h.handlePraxisAct(ctx, dec, candidate); reply != "" {
|
||||
return reply
|
||||
}
|
||||
}
|
||||
@@ -50,23 +62,23 @@ func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) st
|
||||
// Hexis ecosystem action: if ecosystem is configured and we have a verb
|
||||
// + entity text, try to resolve the entity and execute via Hexis.
|
||||
if h.ecosystem != nil && h.ecosystem.hexis != nil && router.ActHasEntityTarget(dec) {
|
||||
if reply := h.handleHexisAct(ctx, dec); reply != "" {
|
||||
if reply := h.handleHexisAct(ctx, dec, candidate); reply != "" {
|
||||
return reply
|
||||
}
|
||||
}
|
||||
|
||||
// HasFn still false ⇒ no allowlist match: scaffold a 'proposed' tool
|
||||
// Unresolved candidate ⇒ no allowlist match: scaffold a 'proposed' tool
|
||||
// the user can enable on the authed surface ("earn the right to ask").
|
||||
if !dec.Slots.HasFn {
|
||||
if !candidate.ActionResolved() {
|
||||
return h.proposeGap(ctx, dec)
|
||||
}
|
||||
out, err := h.tools.Exec(ctx, dec.Slots.Fn, dec.Slots.Args, false)
|
||||
out, err := h.tools.Exec(ctx, candidate.Fn, candidate.Args, false)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, tool.ErrNeedsConfirm):
|
||||
// destructive: park it and ask. The next utterance answers.
|
||||
phrase := actPhrase(dec.Slots.Fn, dec.Slots.Args)
|
||||
h.park(dec.Slots.Fn, dec.Slots.Args, phrase)
|
||||
phrase := actPhrase(candidate.Fn, candidate.Args)
|
||||
h.park(candidate.Fn, candidate.Args, phrase)
|
||||
return phraser.A(phraser.ActConfirm, map[string]string{"name": phrase})
|
||||
case errors.Is(err, tool.ErrUnknownTarget):
|
||||
// The verb reached a tool and the tail did not reach a target, so
|
||||
@@ -101,7 +113,7 @@ func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) st
|
||||
// where a human types them.
|
||||
return phraser.A(phraser.ActNeedsArgs, nil)
|
||||
}
|
||||
log.Printf("voice: tool %s: %v", dec.Slots.Fn, err)
|
||||
log.Printf("voice: tool %s: %v", candidate.Fn, err)
|
||||
if out != "" {
|
||||
return phraser.A(phraser.ActFailOut, map[string]string{"out": firstLine(out)})
|
||||
}
|
||||
|
||||
@@ -106,7 +106,7 @@ func (h *reactiveHandler) queryTasks(ctx context.Context, t *queryTurn) (string,
|
||||
// match on more than one asks which, because closing the wrong task is work he
|
||||
// never finished being marked done. No task named asks which too, since the
|
||||
// router claims the turn without the referent and the list lives here.
|
||||
func (h *reactiveHandler) resolveTaskStatus(ctx context.Context, dec router.Decision) string {
|
||||
func (h *reactiveHandler) resolveTaskStatus(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
|
||||
live, err := h.api.ListTasks(ctx, "live")
|
||||
if err != nil {
|
||||
log.Printf("voice: task status: list: %v", err)
|
||||
|
||||
@@ -279,7 +279,7 @@ func TestResolveTaskStatusMovesTheNamedTask(t *testing.T) {
|
||||
reply := h.resolveTaskStatus(context.Background(), router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: "молоко"},
|
||||
})
|
||||
}, routeCandidate(router.TaskStatusFn))
|
||||
if api.listArg != "live" {
|
||||
t.Errorf("listed %q, want live — a resolved task cannot be resolved again", api.listArg)
|
||||
}
|
||||
@@ -344,7 +344,7 @@ func TestResolveTaskStatusRefusesToGuess(t *testing.T) {
|
||||
h := taskHandler(api)
|
||||
reply := h.resolveTaskStatus(context.Background(), router.Decision{
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: c.named},
|
||||
})
|
||||
}, routeCandidate(router.TaskStatusFn))
|
||||
if len(api.moved) != 0 {
|
||||
t.Errorf("moved %+v — closing the wrong task is the failure this arm exists to avoid", api.moved)
|
||||
}
|
||||
@@ -362,7 +362,7 @@ func TestResolveTaskStatusOpensACandidateFirst(t *testing.T) {
|
||||
h := taskHandler(api)
|
||||
h.resolveTaskStatus(context.Background(), router.Decision{
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: "продлить домен"},
|
||||
})
|
||||
}, routeCandidate(router.TaskStatusFn))
|
||||
if len(api.moved) != 2 {
|
||||
t.Fatalf("moved %+v, want open then done", api.moved)
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ func TestAttentionEmptyWithHealthySourcesIsAllClear(t *testing.T) {
|
||||
praxis := newFakePraxisWithSources(t, `[]`, `[{"source_id":"src_ntfy","health":"ok"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("healthy and quiet should be an all-clear, got %q", reply)
|
||||
}
|
||||
@@ -28,7 +28,7 @@ func TestAttentionEmptyWithAFailedSourceHedges(t *testing.T) {
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("a failed source must not read as all-clear, got %q", reply)
|
||||
}
|
||||
@@ -47,7 +47,7 @@ func TestAttentionEmptyWithNoSourcesHedges(t *testing.T) {
|
||||
praxis := newFakePraxisWithSources(t, `[]`, `[]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("a Praxis with no sources must not answer all-clear, got %q", reply)
|
||||
}
|
||||
@@ -64,7 +64,7 @@ func TestAttentionDegradedEnvelopeIsReadWithoutASourcesCall(t *testing.T) {
|
||||
`[{"source_id":"src_ntfy","health":"ok"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "src_metrics") {
|
||||
t.Fatalf("the envelope's degraded source is not named: %q", reply)
|
||||
}
|
||||
@@ -82,7 +82,7 @@ func TestAttentionKeepsAllClearWhenSourcesCannotBeRead(t *testing.T) {
|
||||
praxis.SetRouteFault("/api/v1/sources", 500)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("an unreadable sources list should leave the answer alone, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -63,6 +63,9 @@ func (h *reactiveHandler) queryAttention(ctx context.Context, t *queryTurn) (str
|
||||
Utterance: t.dec.Utterance,
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: "list_attention", HasFn: true},
|
||||
}, router.ActionCandidate{
|
||||
Fn: "list_attention",
|
||||
Source: router.ActionSourceRoute,
|
||||
})
|
||||
if reply == "" {
|
||||
return "", false
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/decision"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// TestTextAndVoiceConvergeOnNormalizedInput — both entry points construct a
|
||||
// NormalizedInput and pass it to runTurn. The same utterance produces the same
|
||||
// route intent regardless of whether it arrived as text or voice.
|
||||
func TestTextAndVoiceConvergeOnNormalizedInput(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
|
||||
utterance := "который час"
|
||||
voiceCtx := withDialogueID(ctx, dialogueIDFor(sourceVoice, ""))
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
|
||||
voiceReply := h.runTurn(voiceCtx, router.NormalizedInput{Text: utterance, Source: sourceVoice})
|
||||
textReply := h.runTurn(textCtx, router.NormalizedInput{Text: utterance, Source: sourceText})
|
||||
|
||||
// Both paths should produce the same kind of reply (time answer).
|
||||
for _, pair := range []struct {
|
||||
label, reply string
|
||||
}{
|
||||
{"voice", voiceReply},
|
||||
{"text", textReply},
|
||||
} {
|
||||
if !strings.Contains(pair.reply, "час") && !strings.Contains(pair.reply, "время") {
|
||||
t.Errorf("%s reply %q does not look like a time answer", pair.label, pair.reply)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestNormalizedInputSourcePreserved — the source survives into the decision
|
||||
// record so a trace can tell voice from text.
|
||||
func TestNormalizedInputSourcePreserved(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
h.runTurn(textCtx, router.NormalizedInput{Text: "привет", Source: sourceText})
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Fatal("no decision record")
|
||||
}
|
||||
if recs[0].InputSource != string(sourceText) {
|
||||
t.Errorf("InputSource = %q, want %q", recs[0].InputSource, sourceText)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRouteProducerOnDecisionRecord — the producer is carried from the router
|
||||
// decision into the decision record for observability.
|
||||
func TestRouteProducerOnDecisionRecord(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
h.runTurn(textCtx, router.NormalizedInput{Text: "который час", Source: sourceText})
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Fatal("no decision record")
|
||||
}
|
||||
// A time query is a stage-0 grammar match.
|
||||
if recs[0].RouteProducer != string(router.RouteProducerGrammar) {
|
||||
t.Errorf("RouteProducer = %q, want %q", recs[0].RouteProducer, router.RouteProducerGrammar)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPreRouteClaimHasNoRouteProducer — a turn claimed by a pre-route resolver
|
||||
// never reaches the router, so the record's RouteProducer must be empty.
|
||||
func TestPreRouteClaimHasNoRouteProducer(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
// Park a confirm so the next "да" is consumed before routing.
|
||||
// newRoutingClarifyHandler uses a fixed clock at 2026-07-31 09:00 UTC.
|
||||
h.pending = &pendingAct{
|
||||
fn: "test",
|
||||
phrase: "delete everything",
|
||||
expiry: time.Date(2026, 7, 31, 9, 1, 0, 0, time.UTC),
|
||||
}
|
||||
ctx := context.Background()
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
h.runTurn(textCtx, router.NormalizedInput{Text: "да", Source: sourceText})
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Fatal("no decision record")
|
||||
}
|
||||
if recs[0].RouteProducer != "" {
|
||||
t.Errorf("RouteProducer = %q, want empty (pre-route claimed the turn)", recs[0].RouteProducer)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStage0ProducerUnchanged — grammars still produce the exact same intents
|
||||
// at confidence 1.0. This pins stage-0 behavior through the new boundary.
|
||||
func TestStage0ProducerUnchanged(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
|
||||
cases := []struct {
|
||||
utterance string
|
||||
intent router.Intent
|
||||
}{
|
||||
{"напомни позвонить маме завтра", router.IntentReminder},
|
||||
{"который час", router.IntentSystem},
|
||||
}
|
||||
for _, c := range cases {
|
||||
reply := h.runTurn(textCtx, router.NormalizedInput{Text: c.utterance, Source: sourceText})
|
||||
_ = reply // behavior unchanged; we test the record, not the reply text.
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Errorf("%s: no decision record", c.utterance)
|
||||
continue
|
||||
}
|
||||
rec := recs[0]
|
||||
if rec.RouteProducer != string(router.RouteProducerGrammar) {
|
||||
t.Errorf("%s: RouteProducer = %q, want %q", c.utterance, rec.RouteProducer, router.RouteProducerGrammar)
|
||||
}
|
||||
// Clear the ring for the next case.
|
||||
h.decisions = decision.NewRing()
|
||||
}
|
||||
}
|
||||
@@ -439,7 +439,7 @@ func TestClarifySecondGapExhaustionResumesLowerFlow(t *testing.T) {
|
||||
h.clarifyStore.Push(voiceDialogueID, older)
|
||||
h.clarifyStore.Push(voiceDialogueID, top)
|
||||
|
||||
reply := h.runTurn(ctx, "купить хлеб", sourceText)
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "купить хлеб", Source: sourceText})
|
||||
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
|
||||
want := withResumed(clarifyGaveUp, resumed)
|
||||
if reply != want {
|
||||
|
||||
@@ -45,5 +45,5 @@ func (h *reactiveHandler) resolveCommandProhibition(ctx context.Context, text st
|
||||
// The sentinel cannot be renamed into an enabled function, and the original
|
||||
// utterance remains the authority even when a model rewrites Slots.Text.
|
||||
func refusesCommand(dec router.Decision) bool {
|
||||
return dec.Slots.Fn == router.ProhibitedActFn || router.IsCommandProhibition(dec.Utterance)
|
||||
return dec.CapabilitySelection.Fn == router.ProhibitedActFn || router.IsCommandProhibition(dec.Utterance)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// TestRefusesCommandUsesCapabilitySelection proves that refusesCommand reads
|
||||
// CapabilitySelection.Fn rather than the compatibility Slots.Fn. When
|
||||
// CapabilitySelection is populated with the prohibited sentinel but Slots.Fn
|
||||
// is blank, the refusal must still fire.
|
||||
func TestRefusesCommandUsesCapabilitySelection(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
CapabilitySelection: router.CapabilitySelection{
|
||||
Fn: router.ProhibitedActFn,
|
||||
Resolved: true,
|
||||
},
|
||||
// Slots compatibility fields deliberately blank.
|
||||
}
|
||||
if !refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return true when CapabilitySelection.Fn == ProhibitedActFn")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusesCommandUtteranceFallback proves that the utterance-based
|
||||
// prohibition check still works as a defense-in-depth belt when
|
||||
// CapabilitySelection does not carry the sentinel.
|
||||
func TestRefusesCommandUtteranceFallback(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Utterance: "don't restart nginx",
|
||||
}
|
||||
if !refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return true for a prohibited utterance")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusesCommandNonProhibitedCapability proves that an act with a
|
||||
// non-prohibited capability is NOT refused even when Slots.Fn happens to
|
||||
// carry the prohibited sentinel (cross-contamination).
|
||||
func TestRefusesCommandNonProhibitedCapability(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart nginx",
|
||||
CapabilitySelection: router.CapabilitySelection{
|
||||
Fn: "restart",
|
||||
Resolved: true,
|
||||
},
|
||||
}
|
||||
if refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return false for a non-prohibited capability")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusesCommandProhibitedSentinelPreservedThroughPipeline proves that the
|
||||
// command-prohibition grammar sentinel survives through SelectCapability into
|
||||
// CapabilitySelection byte-for-byte.
|
||||
func TestRefusesCommandProhibitedSentinelPreservedThroughPipeline(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{
|
||||
Fn: router.ProhibitedActFn,
|
||||
HasFn: true,
|
||||
},
|
||||
}
|
||||
sel := router.SelectCapability(dec, nil)
|
||||
dec.CapabilitySelection = sel
|
||||
|
||||
if dec.CapabilitySelection.Fn != router.ProhibitedActFn {
|
||||
t.Errorf("CapabilitySelection.Fn = %q, want %q", dec.CapabilitySelection.Fn, router.ProhibitedActFn)
|
||||
}
|
||||
if !dec.CapabilitySelection.Resolved {
|
||||
t.Error("CapabilitySelection.Resolved should be true")
|
||||
}
|
||||
if !refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return true after pipeline preserves the sentinel")
|
||||
}
|
||||
}
|
||||
@@ -107,7 +107,7 @@ var praxisCapabilities = []praxisCapability{
|
||||
// handlePraxisAct — dispatches ecosystem tool acts through the Praxis tools API.
|
||||
// Returns "" when the act is not a Praxis verb (the caller falls through to the
|
||||
// system command executor). Returns a reply string otherwise.
|
||||
func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decision) string {
|
||||
func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
|
||||
if h.ecosystem == nil || h.ecosystem.praxis == nil {
|
||||
return ""
|
||||
}
|
||||
@@ -127,7 +127,7 @@ func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decisi
|
||||
}
|
||||
for _, capability := range praxisCapabilities {
|
||||
for _, alias := range capability.aliases() {
|
||||
if alias == dec.Slots.Fn {
|
||||
if alias == candidate.Fn {
|
||||
return capability.handle(ctx, h, px, dec)
|
||||
}
|
||||
}
|
||||
@@ -657,7 +657,7 @@ func (h *reactiveHandler) resolveEntityCandidates(ctx context.Context, refs []st
|
||||
// handleHexisAct — resolves entity references through Nexus and executes
|
||||
// matching capabilities through Hexis. Returns a reply string when handled,
|
||||
// or "" to fall through to the system command executor.
|
||||
func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decision) string {
|
||||
func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
|
||||
// This method is intentionally callable outside runTurn by ecosystem
|
||||
// harnesses. Refuse before correlation ids, Nexus resolution or capability
|
||||
// discovery so the no-op sentinel can never leak into Hexis as a verb.
|
||||
@@ -721,7 +721,7 @@ func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decisio
|
||||
// Match the user's verb to a capability by name/description. Collect all
|
||||
// matches: more than one is itself ambiguous, so we ask rather than pick
|
||||
// the first (ecosystem invariant: no arbitrary target for mutation).
|
||||
verb := dec.Slots.Fn
|
||||
verb := candidate.Fn
|
||||
if verb == "" {
|
||||
verb = dec.Slots.Text
|
||||
}
|
||||
@@ -732,7 +732,7 @@ func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decisio
|
||||
// round then: the phrase is the haystack and the capability name is what we
|
||||
// look for in it (Vikunja #476). Only when the fn slot is empty — a matched
|
||||
// fn is a single verb and containment already means what it says.
|
||||
loose := !dec.Slots.HasFn
|
||||
loose := !candidate.ActionResolved()
|
||||
var matches []*hexisclient.Capability
|
||||
for i, c := range caps {
|
||||
name := strings.ToLower(c.Name)
|
||||
@@ -858,7 +858,16 @@ func (h *reactiveHandler) hexisBeforeClarify(ctx context.Context, dec router.Dec
|
||||
if dec.Intent != router.IntentAct || dec.Slots.HasFn || !router.ActHasEntityTarget(dec) {
|
||||
return ""
|
||||
}
|
||||
return h.handleHexisAct(ctx, dec)
|
||||
// Resolve the action candidate. Use the matcher when available; when the
|
||||
// handler has no matcher (ecosystem-only test harnesses), build an
|
||||
// unresolved candidate directly — the matcher would not have matched either.
|
||||
var candidate router.ActionCandidate
|
||||
if h.matcher != nil {
|
||||
candidate = h.resolveAction(ctx, dec)
|
||||
} else {
|
||||
candidate = router.ResolveActionCandidate(dec, nil)
|
||||
}
|
||||
return h.handleHexisAct(ctx, dec, candidate)
|
||||
}
|
||||
|
||||
// attentionCannotTell returns the hedge to say instead of an all-clear, or ""
|
||||
|
||||
@@ -96,7 +96,7 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
|
||||
// A Nexus outage during a Hexis act writes a failure trace, and a shared
|
||||
// store is the one thing the Praxis path could inherit it through.
|
||||
nexus.SetFault(503)
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); actRan(reply) {
|
||||
t.Fatalf("nexus outage must not report success, got %q", reply)
|
||||
}
|
||||
if len(tracesFor(t, h, "nexus", "resolve")) == 0 {
|
||||
@@ -104,7 +104,7 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
|
||||
}
|
||||
|
||||
nexus.SetFault(0)
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("a recorded nexus failure must not degrade the praxis digest, got %q", reply)
|
||||
}
|
||||
@@ -114,10 +114,10 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
|
||||
|
||||
// And the reverse: a Praxis outage mid-session leaves the Hexis path whole.
|
||||
praxis.SetFault(503)
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); strings.Contains(reply, "disk") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); strings.Contains(reply, "disk") {
|
||||
t.Fatalf("praxis outage must not serve content, got %q", reply)
|
||||
}
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("a praxis outage must not block the hexis path, got %q", reply)
|
||||
}
|
||||
}
|
||||
@@ -132,7 +132,7 @@ func TestEcosystem_OneEndpointDownDoesNotMuteTheService(t *testing.T) {
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
praxis.SetRouteFault("/api/v1/tools/surface", 503)
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("a downed surface endpoint must not mute the digest, got %q", reply)
|
||||
}
|
||||
@@ -150,7 +150,7 @@ func TestEcosystem_ResolvedWithoutEntityFailsClosed(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("a resolve with no entity must degrade, not fall through to local execution")
|
||||
}
|
||||
@@ -172,7 +172,7 @@ func TestEcosystem_RejectedCredentialSaysSo(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
nexus.SetFault(status)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "токен") {
|
||||
t.Fatalf("http %d must read as a credential problem, got %q", status, reply)
|
||||
}
|
||||
@@ -193,7 +193,7 @@ func TestEcosystem_MalformedPraxisBodyDegrades(t *testing.T) {
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
praxis.SetBody(`[{"title":`)
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if reply == "" {
|
||||
t.Fatal("a malformed praxis body must not answer with silence")
|
||||
}
|
||||
@@ -211,7 +211,7 @@ func TestEcosystem_MalformedNexusResponseFailsClosed(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
nexus.SetBody(`{"status":"resolved","entity":`)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" || actRan(reply) {
|
||||
t.Fatalf("malformed nexus body must degrade, got %q", reply)
|
||||
}
|
||||
@@ -232,7 +232,7 @@ func TestEcosystem_UnknownContractFieldsTolerated(t *testing.T) {
|
||||
nexus := newFakeNexus(t, body)
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("%s contract shape must still resolve and execute, got %q", name, reply)
|
||||
}
|
||||
})
|
||||
@@ -249,7 +249,7 @@ func TestEcosystem_CancelledContextDegrades(t *testing.T) {
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" || actRan(reply) {
|
||||
t.Fatalf("cancelled resolve must degrade, got %q", reply)
|
||||
}
|
||||
@@ -267,7 +267,7 @@ func TestEcosystem_ExecutionFailureIsNotSuccess(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecutionFailed("exec_1", "unit not found"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if actRan(reply) {
|
||||
t.Fatalf("failed execution must not read as success, got %q", reply)
|
||||
}
|
||||
@@ -291,7 +291,7 @@ func TestEcosystem_SuccessfulActionWritesATrace(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("setup: expected success, got %q", reply)
|
||||
}
|
||||
exec := tracesFor(t, h, "hexis", "execute")
|
||||
@@ -313,7 +313,7 @@ func TestEcosystem_TracesStayOutOfFacts(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("setup: expected success, got %q", reply)
|
||||
}
|
||||
if len(traces(t, h)) == 0 {
|
||||
@@ -339,7 +339,7 @@ func TestEcosystem_AmbiguousTargetBlocksExecution(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Muzick web") {
|
||||
t.Fatalf("ambiguous resolve must list candidates, got %q", reply)
|
||||
}
|
||||
@@ -360,7 +360,7 @@ func TestEcosystem_NoAutonomousPraxisToHexis(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, praxis, hexis)
|
||||
|
||||
_ = h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
_ = h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if hexis.Count("", "/api/v1") != 0 {
|
||||
t.Fatal("attention digest must not contact hexis on its own")
|
||||
}
|
||||
@@ -378,7 +378,7 @@ func TestEcosystem_MutatingCapabilityWaitsForConfirmation(t *testing.T) {
|
||||
hexis := newFakeHexis(t, caps, fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("restart"))
|
||||
reply := h.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "restart") || !strings.Contains(reply, "да") {
|
||||
t.Fatalf("mutating capability must ask for confirmation, got %q", reply)
|
||||
}
|
||||
@@ -404,7 +404,7 @@ func TestEcosystem_SurfaceFailureStillDelivers(t *testing.T) {
|
||||
praxis.SetRouteFault("/api/v1/tools/surface", 500)
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("failed surface must not swallow the digest, got %q", reply)
|
||||
}
|
||||
@@ -426,10 +426,10 @@ func TestEcosystem_TotalOutageSaysSoForEveryPath(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, praxis, hexis)
|
||||
|
||||
for name, reply := range map[string]string{
|
||||
"hexis act": h.handleHexisAct(ctx, actDec("muzick indexer")),
|
||||
"attention": h.handlePraxisAct(ctx, praxisActDec("list_attention")),
|
||||
"changes": h.handlePraxisAct(ctx, praxisActDec("list_changes")),
|
||||
"acknowledge": h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1")),
|
||||
"hexis act": h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")),
|
||||
"attention": h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")),
|
||||
"changes": h.handlePraxisAct(ctx, praxisActDec("list_changes"), routeCandidate("list_changes")),
|
||||
"acknowledge": h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item")),
|
||||
} {
|
||||
if reply == "" {
|
||||
t.Errorf("%s: total outage must not answer with silence", name)
|
||||
@@ -458,11 +458,11 @@ func TestEcosystem_RecoveryAfterOutageNeedsNoRestart(t *testing.T) {
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
praxis.SetFault(503)
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); strings.Contains(reply, "disk") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); strings.Contains(reply, "disk") {
|
||||
t.Fatalf("outage must not serve content, got %q", reply)
|
||||
}
|
||||
praxis.SetFault(0)
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("recovery must work on the next turn, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -59,7 +59,7 @@ func TestHexisDiscovery401IsDeniedNotDown(t *testing.T) {
|
||||
h := hexisGapHandler(t, nexus.URL, hexis.URL)
|
||||
|
||||
hexis.SetFault(401)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !denied(serviceHexis, reply) {
|
||||
t.Fatalf("401 from hexis discovery: got %q, want the denied line naming Hexis", reply)
|
||||
}
|
||||
@@ -75,7 +75,7 @@ func TestHexisDiscoveryOutageIsDownNotDenied(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusResolved("ent_muzick", muzickIndexer, "service"))
|
||||
h := hexisGapHandler(t, nexus.URL, unreachableURL)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !down(serviceHexis, reply) {
|
||||
t.Fatalf("connection refused from hexis: got %q, want the outage line naming Hexis", reply)
|
||||
}
|
||||
@@ -96,7 +96,7 @@ func TestHexisExecute401IsDeniedNotCommandFailure(t *testing.T) {
|
||||
// Discovery stays healthy; only the execute endpoint refuses. A blanket
|
||||
// fault would never reach the site under test.
|
||||
hexis.SetRouteFault("/api/v1/execute", 401)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !denied(serviceHexis, reply) {
|
||||
t.Fatalf("401 from hexis execute: got %q, want the denied line naming Hexis", reply)
|
||||
}
|
||||
@@ -129,7 +129,7 @@ func TestHexisExecuteOutageIsDown(t *testing.T) {
|
||||
t.Cleanup(hexis.Close)
|
||||
h := hexisGapHandler(t, nexus.URL, hexis.URL)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !down(serviceHexis, reply) {
|
||||
t.Fatalf("dropped connection on hexis execute: got %q, want the outage line", reply)
|
||||
}
|
||||
@@ -149,7 +149,7 @@ func TestHexisExecutionFailedStaysCommandFailure(t *testing.T) {
|
||||
hexis := newFakeHexis(t, caps, fixtureHexisExecutionFailed("exec_1", "unit refused to start"))
|
||||
h := hexisGapHandler(t, nexus.URL, hexis.URL)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if down(serviceHexis, reply) || denied(serviceHexis, reply) {
|
||||
t.Fatalf("a failed execution must not be reported as an ecosystem gap, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -19,6 +19,11 @@ func praxisActDec(fn string) router.Decision {
|
||||
return router.Decision{Intent: router.IntentAct, Slots: router.Slots{Fn: fn, HasFn: true}}
|
||||
}
|
||||
|
||||
// routeCandidate builds an ActionCandidate matching a route-resolved Decision.
|
||||
func routeCandidate(fn string) router.ActionCandidate {
|
||||
return router.ActionCandidate{Fn: fn, Source: router.ActionSourceRoute}
|
||||
}
|
||||
|
||||
// praxisItemDec is praxisActDec for the lifecycle verbs, which need an item id
|
||||
// in the value slot. Without one they answer "which item?" and never reach
|
||||
// Praxis at all, which makes them useless for testing a Praxis outage.
|
||||
@@ -46,7 +51,7 @@ func TestPraxisAttention_HappyPathSurfacesItems(t *testing.T) {
|
||||
praxis := newFakePraxis(t, items)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("expected attention digest to mention the item, got %q", reply)
|
||||
}
|
||||
@@ -77,7 +82,7 @@ func TestPraxisAttention_DegradedFailsClosedNotEmpty(t *testing.T) {
|
||||
praxis.SetFault(500)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if reply == "" {
|
||||
t.Fatal("praxis outage must not produce an empty reply")
|
||||
}
|
||||
@@ -104,13 +109,13 @@ func TestFakeNexus_FaultInjectionThenRecovery(t *testing.T) {
|
||||
}
|
||||
|
||||
nexus.SetFault(503)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if actRan(reply) {
|
||||
t.Fatalf("nexus outage must not report success, got %q", reply)
|
||||
}
|
||||
|
||||
nexus.SetFault(0)
|
||||
reply = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !actRan(reply) {
|
||||
t.Fatalf("expected success once nexus recovers, got %q", reply)
|
||||
}
|
||||
@@ -134,7 +139,7 @@ func TestPraxisEntityAttention_RemembersWhatItReadOut(t *testing.T) {
|
||||
reply := h.handlePraxisAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: "entity_attention", HasFn: true, Value: "muzick indexer"},
|
||||
})
|
||||
}, routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "indexer wedged") {
|
||||
t.Fatalf("expected the scoped item to be read out, got %q", reply)
|
||||
}
|
||||
@@ -147,7 +152,7 @@ func TestPraxisEntityAttention_RemembersWhatItReadOut(t *testing.T) {
|
||||
}
|
||||
|
||||
// The follow-up resolves against what he just heard, not the stale list.
|
||||
if reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "last")); reply == "" {
|
||||
if reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "last"), routeCandidate("resolve_item")); reply == "" {
|
||||
t.Fatal("positional follow-up should have been claimed by praxis")
|
||||
}
|
||||
var body string
|
||||
@@ -175,7 +180,7 @@ func TestHexisConfirm_KeepsOneCorrelationIDPerAction(t *testing.T) {
|
||||
hexis := newFakeHexis(t, caps, fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("restart")); !strings.Contains(reply, "да") {
|
||||
if reply := h.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart")); !strings.Contains(reply, "да") {
|
||||
t.Fatalf("mutating capability must ask for confirmation, got %q", reply)
|
||||
}
|
||||
resolve := findTrace(t, h, "nexus", "resolve")
|
||||
|
||||
@@ -73,7 +73,7 @@ func TestHexisMutatingRequiresConfirm(t *testing.T) {
|
||||
caps := `[{"id":"cap_restart","name":"restart","read_only":false,"risk":"high"}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "да") {
|
||||
t.Fatalf("mutating cap should ask to confirm, got %q", reply)
|
||||
}
|
||||
@@ -103,7 +103,7 @@ func TestHexisConfirmNoDoesNotExecute(t *testing.T) {
|
||||
caps := `[{"id":"cap_restart","name":"restart","read_only":false}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
reply, handled := h.resolveConfirm(ctx, "нет")
|
||||
if !handled || !strings.Contains(reply, "отменила") {
|
||||
t.Fatalf("no should cancel, got handled=%v reply=%q", handled, reply)
|
||||
@@ -119,7 +119,7 @@ func TestHexisReadOnlyExecutesImmediately(t *testing.T) {
|
||||
caps := `[{"id":"cap_status","name":"restart","read_only":true}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !*executed {
|
||||
t.Fatal("read-only cap should execute without confirmation")
|
||||
}
|
||||
@@ -136,7 +136,7 @@ func TestHexisAmbiguousAsksClarification(t *testing.T) {
|
||||
ambiguous := `{"status":"ambiguous","candidates":[{"entity_id":"ent_muzick","display_name":"Muzick indexer"},{"entity_id":"ent_manga","display_name":"Manga indexer"}]}`
|
||||
h, executed := newHexisTestHandler(t, ambiguous, `[]`)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("the indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("the indexer"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Manga indexer") {
|
||||
t.Fatalf("ambiguous should list candidates, got %q", reply)
|
||||
}
|
||||
@@ -154,7 +154,7 @@ func TestHexisResolveFlatShapeAccepted(t *testing.T) {
|
||||
caps := `[{"id":"cap_status","name":"restart","read_only":true}]`
|
||||
h, executed := newHexisTestHandler(t, flat, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !*executed {
|
||||
t.Fatalf("flat-shaped resolved entity should still execute, got reply %q", reply)
|
||||
}
|
||||
@@ -183,7 +183,7 @@ func TestHexisNexusErrorFailsClosed(t *testing.T) {
|
||||
ecosystem: stubEcosystem(nexus.URL, hexis.URL),
|
||||
}
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("nexus dependency failure must not fall through with an empty reply")
|
||||
}
|
||||
@@ -216,7 +216,7 @@ func TestHexisUnavailableFailsClosed(t *testing.T) {
|
||||
ecosystem: stubEcosystem(nexus.URL, hexis.URL),
|
||||
}
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("hexis dependency failure must not fall through with an empty reply")
|
||||
}
|
||||
@@ -234,7 +234,7 @@ func TestHexisNotFoundStillFallsThrough(t *testing.T) {
|
||||
notFound := `{"status":"not_found"}`
|
||||
h, executed := newHexisTestHandler(t, notFound, `[]`)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("turn off the lights"))
|
||||
reply := h.handleHexisAct(ctx, actDec("turn off the lights"), routeCandidate("restart"))
|
||||
if reply != "" {
|
||||
t.Fatalf("not_found resolution should fall through with empty reply, got %q", reply)
|
||||
}
|
||||
@@ -264,7 +264,7 @@ func TestHexisIrreversibleCapabilityIsNotRunFromVoice(t *testing.T) {
|
||||
caps := `[{"id":"cap_wipe","name":"restart","read_only":false,"risk":"irreversible","requires_confirmation":true}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if *executed {
|
||||
t.Fatal("an irreversible capability ran from the voice path")
|
||||
}
|
||||
@@ -284,7 +284,7 @@ func TestHexisSafeCapabilityRunsOnItsDeclaredTier(t *testing.T) {
|
||||
caps := `[{"id":"cap_status","name":"restart","read_only":true,"risk":"safe"}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !*executed {
|
||||
t.Fatal("a capability Hexis calls safe should run")
|
||||
}
|
||||
@@ -301,7 +301,7 @@ func TestHexisUndeclaredTierStillConfirms(t *testing.T) {
|
||||
caps := `[{"id":"cap_restart","name":"restart","read_only":false}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if *executed {
|
||||
t.Fatal("a mutating capability ran without a confirm")
|
||||
}
|
||||
|
||||
@@ -142,7 +142,7 @@ func TestEcosystemTrace_SuccessfulActionTracesEveryHop(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("setup: expected success, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -186,7 +186,7 @@ func TestEcosystemTrace_OneCorrelationIDPerPraxisAction(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("setup: expected the digest, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -218,7 +218,7 @@ func TestEcosystemTrace_FailuresAreTracedToo(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
nexus.SetFault(401)
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
|
||||
d := findTrace(t, h, "nexus", "resolve")
|
||||
if d == nil {
|
||||
@@ -242,7 +242,7 @@ func TestEcosystemTrace_UnreachableIsNotRefused(t *testing.T) {
|
||||
h := ecoHandler(t, nil, nil, nil)
|
||||
h.ecosystem.nexus = newNexusClient("http://127.0.0.1:1")
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
|
||||
d := findTrace(t, h, "nexus", "resolve")
|
||||
if d == nil {
|
||||
@@ -263,7 +263,7 @@ func TestEcosystemTrace_RedactsTheUtterance(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusNotFound())
|
||||
h := ecoHandler(t, nexus, nil, nil)
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("перезапусти кофемашину"))
|
||||
_ = h.handleHexisAct(ctx, actDec("перезапусти кофемашину"), routeCandidate("restart"))
|
||||
|
||||
recorded := traces(t, h)
|
||||
if len(recorded) == 0 {
|
||||
@@ -295,7 +295,7 @@ func TestEcosystemTrace_AmbiguityAndConfirmationAreRecorded(t *testing.T) {
|
||||
))
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, ambig, nil, hexis)
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick"), routeCandidate("restart"))
|
||||
if d := findTrace(t, h, "nexus", "resolve"); d == nil || d.Status != traceAmbig {
|
||||
t.Fatalf("ambiguous resolve must be traced as such, got %+v", d)
|
||||
}
|
||||
@@ -303,7 +303,7 @@ func TestEcosystemTrace_AmbiguityAndConfirmationAreRecorded(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusResolved("ent_muzick", "Muzick indexer", "service"))
|
||||
mutating := fixtureHexisCapabilities(map[string]any{"id": "cap_restart", "name": "restart", "read_only": false})
|
||||
h2 := ecoHandler(t, nexus, nil, newFakeHexis(t, mutating, fixtureHexisExecuted("exec_1", "succeeded")))
|
||||
_ = h2.handleHexisAct(ctx, actDec("restart"))
|
||||
_ = h2.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart"))
|
||||
d := findTrace(t, h2, "hexis", "confirmation")
|
||||
if d == nil || d.Status != tracePending {
|
||||
t.Fatalf("a parked confirmation must be traced, got %+v", d)
|
||||
|
||||
@@ -91,7 +91,7 @@ func TestNexusIsAskedForTheNameHeSaid(t *testing.T) {
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "перезагрузить музик индексер", Fn: "restart", HasFn: true},
|
||||
}
|
||||
h.handleHexisAct(ctx, dec)
|
||||
h.handleHexisAct(ctx, dec, routeCandidate("restart"))
|
||||
|
||||
reqs := nexus.Requests()
|
||||
if len(reqs) == 0 {
|
||||
@@ -226,7 +226,7 @@ func TestTwoResolvedNamesAsk(t *testing.T) {
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "перезагрузить нгинкс", Fn: "restart", HasFn: true},
|
||||
}
|
||||
reply := h.handleHexisAct(ctx, dec)
|
||||
reply := h.handleHexisAct(ctx, dec, routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "nginx") || !strings.Contains(reply, "Muzick indexer") {
|
||||
t.Fatalf("reply = %q, want both names she found", reply)
|
||||
}
|
||||
@@ -251,7 +251,7 @@ func TestTheNameNexusKnowsWins(t *testing.T) {
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "перезагрузить нгинкс", Fn: "restart", HasFn: true},
|
||||
}
|
||||
reply := h.handleHexisAct(ctx, dec)
|
||||
reply := h.handleHexisAct(ctx, dec, routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("the resolvable name must carry the act")
|
||||
}
|
||||
|
||||
@@ -33,7 +33,7 @@ func TestEntityAttention_ScopesPraxisByCanonicalID(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "indexer queue is backing up") {
|
||||
t.Fatalf("expected the scoped item in the reply, got %q", reply)
|
||||
}
|
||||
@@ -70,7 +70,7 @@ func TestEntityAttention_FoldsInLocalFactsForSameEntity(t *testing.T) {
|
||||
t.Fatalf("ResolveFactEntity: %v", err)
|
||||
}
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "descaled in june") {
|
||||
t.Fatalf("expected entity-scoped local facts in the reply, got %q", reply)
|
||||
}
|
||||
@@ -88,7 +88,7 @@ func TestEntityAttention_UnscopedPraxisResponseIsRefused(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("an unscoped response must not be read back as entity-scoped, got %q", reply)
|
||||
}
|
||||
@@ -112,7 +112,7 @@ func TestEntityAttention_ForeignItemsAreDropped(t *testing.T) {
|
||||
praxis := newFakePraxis(t, mustJSON(mixed))
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "indexer queue is backing up") {
|
||||
t.Fatalf("the matching item must be spoken, got %q", reply)
|
||||
}
|
||||
@@ -140,7 +140,7 @@ func TestEntityAttention_TruncationIsNamed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "и это не всё") {
|
||||
t.Fatalf("a truncated recall must say it is truncated, got %q", reply)
|
||||
}
|
||||
@@ -156,7 +156,7 @@ func TestEntityAttention_AmbiguousAsksInsteadOfGuessing(t *testing.T) {
|
||||
praxis := newFakePraxis(t, fixturePraxisAttentionItems())
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Muzick web") {
|
||||
t.Fatalf("ambiguous subject must ask, got %q", reply)
|
||||
}
|
||||
@@ -173,13 +173,13 @@ func TestEntityAttention_MissingAndDegradedAreDistinct(t *testing.T) {
|
||||
praxis := newFakePraxis(t, fixturePraxisAttentionItems())
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
missing := h.handlePraxisAct(ctx, entityAttentionDec("нечто"))
|
||||
missing := h.handlePraxisAct(ctx, entityAttentionDec("нечто"), routeCandidate("entity_attention"))
|
||||
if missing == "" {
|
||||
t.Fatal("an unknown entity must still get an answer")
|
||||
}
|
||||
|
||||
nexus.SetFault(503)
|
||||
degraded := h.handlePraxisAct(ctx, entityAttentionDec("нечто"))
|
||||
degraded := h.handlePraxisAct(ctx, entityAttentionDec("нечто"), routeCandidate("entity_attention"))
|
||||
if degraded == missing {
|
||||
t.Fatalf("outage and unknown-entity must not read the same: %q", degraded)
|
||||
}
|
||||
@@ -195,7 +195,7 @@ func TestEntityAttention_DelayedNexusDegradesNotHangs(t *testing.T) {
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if reply == "" {
|
||||
t.Fatal("a delayed resolve must still answer")
|
||||
}
|
||||
@@ -213,7 +213,7 @@ func TestEntityAttention_WithoutNexusSaysSo(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("without nexus, items must not be passed off as entity-scoped, got %q", reply)
|
||||
}
|
||||
|
||||
+11
-9
@@ -72,15 +72,16 @@ func dialogueIDOf(ctx context.Context) string {
|
||||
// toDialogueSlots projects the router's slots onto the dialogue layer's copy.
|
||||
func toDialogueSlots(s router.Slots) dialogue.Slots {
|
||||
return dialogue.Slots{
|
||||
Time: s.Time,
|
||||
HasTime: s.HasTime,
|
||||
Key: s.Key,
|
||||
Value: s.Value,
|
||||
HasKey: s.HasKey,
|
||||
Text: s.Text,
|
||||
Fn: s.Fn,
|
||||
Args: s.Args,
|
||||
HasFn: s.HasFn,
|
||||
Time: s.Time,
|
||||
HasTime: s.HasTime,
|
||||
Key: s.Key,
|
||||
Value: s.Value,
|
||||
HasKey: s.HasKey,
|
||||
Text: s.Text,
|
||||
Fn: s.Fn,
|
||||
Args: s.Args,
|
||||
HasFn: s.HasFn,
|
||||
ResolvedBy: string(s.ResolvedBy),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -90,6 +91,7 @@ func applyDialogueSlots(base router.Slots, d dialogue.Slots) router.Slots {
|
||||
base.Key, base.Value, base.HasKey = d.Key, d.Value, d.HasKey
|
||||
base.Text = d.Text
|
||||
base.Fn, base.Args, base.HasFn = d.Fn, d.Args, d.HasFn
|
||||
base.ResolvedBy = router.ActionResolutionMethod(d.ResolvedBy)
|
||||
return base
|
||||
}
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ func TestPraxisLifecycle401NamesPraxis(t *testing.T) {
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
praxis.SetFault(401)
|
||||
reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "item_1"))
|
||||
reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "item_1"), routeCandidate("resolve_item"))
|
||||
if !strings.Contains(reply, servicePraxis) {
|
||||
t.Fatalf("praxis failure does not name Praxis: %q", reply)
|
||||
}
|
||||
@@ -40,10 +40,10 @@ func TestPraxisLifecycleOutageDiffersFrom401(t *testing.T) {
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
praxis.SetFault(401)
|
||||
refused := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"))
|
||||
refused := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item"))
|
||||
|
||||
h.ecosystem = &ecosystemWiring{praxis: newPraxisClient(unreachableURL)}
|
||||
outage := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"))
|
||||
outage := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item"))
|
||||
|
||||
if refused == outage {
|
||||
t.Fatalf("a refused token and an outage still say the same thing: %q", refused)
|
||||
|
||||
@@ -17,7 +17,7 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention")); reply == "" {
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention")); reply == "" {
|
||||
t.Fatal("attention returned nothing")
|
||||
}
|
||||
|
||||
@@ -28,7 +28,7 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
|
||||
}
|
||||
for _, c := range cases {
|
||||
praxis.ResetRequests()
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", c.ref))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", c.ref), routeCandidate("acknowledge_item"))
|
||||
if !strings.Contains(reply, "принято") {
|
||||
t.Errorf("ref %q: reply %q", c.ref, reply)
|
||||
}
|
||||
@@ -42,10 +42,10 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
|
||||
func TestPositionPastTheEndAsksInsteadOfGuessing(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[{"id":"item_a","title":"диск заканчивается"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "4"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "4"), routeCandidate("resolve_item"))
|
||||
if !strings.Contains(reply, "какой пункт") {
|
||||
t.Errorf("a position with no item should ask, got %q", reply)
|
||||
}
|
||||
@@ -59,7 +59,7 @@ func TestPositionWithNoSpokenListAsks(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"), routeCandidate("acknowledge_item"))
|
||||
if !strings.Contains(reply, "какой пункт") {
|
||||
t.Errorf("want the ask, got %q", reply)
|
||||
}
|
||||
@@ -69,10 +69,10 @@ func TestPositionWithNoSpokenListAsks(t *testing.T) {
|
||||
func TestExplicitItemIDIsNotRewritten(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[{"id":"item_a","title":"диск"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("pin_item", "item_zz"))
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("pin_item", "item_zz"), routeCandidate("pin_item"))
|
||||
if !requestedPathContaining(praxis, "item_zz") {
|
||||
t.Errorf("the id he gave was not the one called; paths %v", paths(praxis))
|
||||
}
|
||||
@@ -85,10 +85,10 @@ func TestUnspokenItemsHoldNoPosition(t *testing.T) {
|
||||
{"id":"item_said","title":"бэкап не прошёл"}
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"))
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"), routeCandidate("acknowledge_item"))
|
||||
if !requestedPathContaining(praxis, "item_said") {
|
||||
t.Errorf("position 1 is the first item she SAID; paths %v", paths(praxis))
|
||||
}
|
||||
@@ -116,10 +116,10 @@ func requestedPathContaining(f *fakeServer, want string) bool {
|
||||
func TestDemonstrativeResolvesWhenOneItemWasSpoken(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[{"id":"item_only","title":"бэкап не прошёл"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "this"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "this"), routeCandidate("acknowledge_item"))
|
||||
if !strings.Contains(reply, "принято") {
|
||||
t.Errorf("reply %q", reply)
|
||||
}
|
||||
@@ -136,10 +136,10 @@ func TestDemonstrativeWithSeveralItemsGivesTheTurnBack(t *testing.T) {
|
||||
{"id":"item_b","title":"бэкап"}
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this")); reply != "" {
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this"), routeCandidate("resolve_item")); reply != "" {
|
||||
t.Errorf("want a fall-through, got %q", reply)
|
||||
}
|
||||
for _, p := range paths(praxis) {
|
||||
@@ -154,7 +154,7 @@ func TestDemonstrativeWithNoDigestGivesTheTurnBack(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this")); reply != "" {
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this"), routeCandidate("resolve_item")); reply != "" {
|
||||
t.Errorf("want a fall-through, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -126,7 +126,7 @@ func TestRunTurnExplicitNoteStoresOnlyTheBody(t *testing.T) {
|
||||
}
|
||||
|
||||
const utterance = "запомни: запасной ключ лежит в синей коробке"
|
||||
if reply := h.runTurn(ctx, utterance, sourceText); reply != "сохранила заметку." {
|
||||
if reply := h.runTurn(ctx, router.NormalizedInput{Text: utterance, Source: sourceText}); reply != "сохранила заметку." {
|
||||
t.Fatalf("reply = %q, want the fixed feminine acknowledgement", reply)
|
||||
}
|
||||
if model.calls != 0 {
|
||||
|
||||
@@ -382,7 +382,7 @@ func TestReminderCancellationIsAPreRouteTurnAndDoesNotGetSwallowedByClarify(t *t
|
||||
Utterance: "напомни позвонить маме", Asked: h.now(), TTL: clarifyTTL,
|
||||
})
|
||||
|
||||
reply := h.runTurn(ctx, "отмени напоминание про врача", sourceText)
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "отмени напоминание про врача", Source: sourceText})
|
||||
if !strings.Contains(reply, clarifyDropped) || !strings.Contains(reply, "отменила напоминание") {
|
||||
t.Fatalf("reply = %q, want dropped clarify notice and cancellation", reply)
|
||||
}
|
||||
|
||||
@@ -149,7 +149,7 @@ func TestRepairResumesQuestionParkedAfterTheCorrectedTurn(t *testing.T) {
|
||||
t.Fatal("expected a parked reminder question")
|
||||
}
|
||||
|
||||
reply := h.runTurn(ctx, "нет, это был вопрос", sourceText)
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "нет, это был вопрос", Source: sourceText})
|
||||
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
|
||||
if !strings.HasSuffix(reply, resumed) {
|
||||
t.Fatalf("the correction hid the still-live question: reply=%q want suffix=%q", reply, resumed)
|
||||
@@ -176,14 +176,14 @@ func TestRepairedClarifyCompletesWithoutDroppingTheOlderQuestion(t *testing.T) {
|
||||
t.Fatal("expected the older reminder question")
|
||||
}
|
||||
|
||||
if reply := h.runTurn(ctx, "нет, это было напоминание", sourceText); !strings.Contains(reply, "Когда") {
|
||||
if reply := h.runTurn(ctx, router.NormalizedInput{Text: "нет, это было напоминание", Source: sourceText}); !strings.Contains(reply, "Когда") {
|
||||
t.Fatalf("the repaired reminder did not ask for its missing time: %q", reply)
|
||||
}
|
||||
if depth := h.clarifyStore.Depth(voiceDialogueID); depth != 2 {
|
||||
t.Fatalf("the repaired question overwrote the older one: depth=%d want=2", depth)
|
||||
}
|
||||
|
||||
reply := h.runTurn(ctx, "сегодня в 15:00", sourceText)
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "сегодня в 15:00", Source: sourceText})
|
||||
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
|
||||
if !strings.HasSuffix(reply, resumed) {
|
||||
t.Fatalf("completing the repaired request did not resume the older one: reply=%q", reply)
|
||||
|
||||
@@ -120,6 +120,7 @@ func (h *reactiveHandler) persistDecision(turnCtx context.Context, rec *decision
|
||||
Source: string(src),
|
||||
Winner: rec.Winner,
|
||||
Intent: wonIntent(rec),
|
||||
RouteProducer: rec.RouteProducer,
|
||||
ClaimedBeforeHead: claimedBeforeHead(rec),
|
||||
EncoderID: h.encoderID,
|
||||
Outcome: wonAt(rec, decision.StageAction),
|
||||
|
||||
@@ -696,7 +696,7 @@ func (w *simWorld) stimulate(ctx context.Context, s step) {
|
||||
|
||||
switch {
|
||||
case s.Say != "":
|
||||
reply := w.handler.runTurn(ctx, s.Say, sourceText)
|
||||
reply := w.handler.runTurn(ctx, router.NormalizedInput{Text: s.Say, Source: sourceText})
|
||||
w.replies = append(w.replies, reply)
|
||||
w.logf("он: %s", s.Say)
|
||||
w.logf("она: %s", reply)
|
||||
|
||||
@@ -31,6 +31,12 @@ func TestSlotsParity(t *testing.T) {
|
||||
t.Errorf("router.Slots.%s (%s) missing from dialogue.Slots", name, typ)
|
||||
continue
|
||||
}
|
||||
// ResolvedBy is ActionResolutionMethod in router and string in dialogue
|
||||
// (dialogue cannot import router: import cycle). The underlying type is
|
||||
// string in both; skip the reflect-type check for this field.
|
||||
if name == "ResolvedBy" {
|
||||
continue
|
||||
}
|
||||
if dt != typ {
|
||||
t.Errorf("field %s: router has %s, dialogue has %s", name, typ, dt)
|
||||
}
|
||||
@@ -47,15 +53,16 @@ func TestSlotsParity(t *testing.T) {
|
||||
// populated value and compare.
|
||||
func TestSlotsRoundTrip(t *testing.T) {
|
||||
full := router.Slots{
|
||||
Time: time.Date(2026, 8, 2, 11, 0, 0, 0, time.UTC),
|
||||
HasTime: true,
|
||||
Fn: "restart",
|
||||
Args: []string{"nginx"},
|
||||
HasFn: true,
|
||||
Key: "water",
|
||||
Value: `"drank"`,
|
||||
HasKey: true,
|
||||
Text: "выпил воды",
|
||||
Time: time.Date(2026, 8, 2, 11, 0, 0, 0, time.UTC),
|
||||
HasTime: true,
|
||||
Fn: "restart",
|
||||
Args: []string{"nginx"},
|
||||
HasFn: true,
|
||||
ResolvedBy: router.ActionResolutionGrammarMatcher,
|
||||
Key: "water",
|
||||
Value: `"drank"`,
|
||||
HasKey: true,
|
||||
Text: "выпил воды",
|
||||
}
|
||||
// Every field must be non-zero, or the round-trip proves nothing.
|
||||
rv := reflect.ValueOf(full)
|
||||
|
||||
@@ -264,10 +264,10 @@ func TestClarifyCancelEndsTheExchange(t *testing.T) {
|
||||
// the same memo.
|
||||
func TestTheTurnIsRoutedOnce(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
rt := h.newTurnRoute("какая сейчас погода в Риме?", h.now())
|
||||
rt := h.newTurnRoute(router.NormalizedInput{Text: "какая сейчас погода в Риме?", Source: sourceText}, h.now())
|
||||
ctx := withTurnRoute(withDialogueID(context.Background(), voiceDialogueID), rt)
|
||||
|
||||
first, ok := h.routeForRole(ctx, rt.text)
|
||||
first, ok := h.routeForRole(ctx, rt.input.Text)
|
||||
if !ok {
|
||||
t.Fatal("the cascade must produce a decision to classify against")
|
||||
}
|
||||
|
||||
@@ -18,9 +18,9 @@ import (
|
||||
// second on the resident model and — worse — could disagree with itself, which
|
||||
// is exactly the class of bug this task is about.
|
||||
type turnRoute struct {
|
||||
h *reactiveHandler
|
||||
text string
|
||||
now time.Time
|
||||
h *reactiveHandler
|
||||
input router.NormalizedInput
|
||||
now time.Time
|
||||
|
||||
once sync.Once
|
||||
dec router.Decision
|
||||
@@ -46,8 +46,8 @@ type turnRoute struct {
|
||||
|
||||
type turnRouteKey struct{}
|
||||
|
||||
func (h *reactiveHandler) newTurnRoute(text string, now time.Time) *turnRoute {
|
||||
return &turnRoute{h: h, text: text, now: now}
|
||||
func (h *reactiveHandler) newTurnRoute(input router.NormalizedInput, now time.Time) *turnRoute {
|
||||
return &turnRoute{h: h, input: input, now: now}
|
||||
}
|
||||
|
||||
func withTurnRoute(ctx context.Context, rt *turnRoute) context.Context {
|
||||
@@ -70,7 +70,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
|
||||
if r.h.dialogueSessions != nil {
|
||||
r.prev = r.h.dialogueSessions.Get(dialogueIDOf(ctx), r.now)
|
||||
}
|
||||
if dec, cont := continuationDecision(r.prev, r.text, r.now); cont {
|
||||
if dec, cont := continuationDecision(r.prev, r.input.Text, r.now); cont {
|
||||
log.Printf("voice: continuation of %s from the previous turn", dec.Intent)
|
||||
r.dec, r.cont = dec, true
|
||||
return
|
||||
@@ -79,7 +79,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
|
||||
r.err = router.ErrNoIntents
|
||||
return
|
||||
}
|
||||
r.dec, r.err = r.h.router.Route(ctx, r.text, r.now)
|
||||
r.dec, r.err = r.h.router.Route(ctx, r.input, r.now)
|
||||
})
|
||||
return r.dec, r.cont, r.prev, r.err
|
||||
}
|
||||
@@ -92,7 +92,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
|
||||
func (h *reactiveHandler) routeForRole(ctx context.Context, text string) (router.Decision, bool) {
|
||||
rt := turnRouteFrom(ctx)
|
||||
if rt == nil {
|
||||
rt = h.newTurnRoute(text, h.now())
|
||||
rt = h.newTurnRoute(router.NormalizedInput{Text: text, Source: sourceText}, h.now())
|
||||
}
|
||||
dec, _, _, err := rt.resolve(ctx)
|
||||
if err != nil {
|
||||
|
||||
+19
-15
@@ -211,7 +211,7 @@ func (h *reactiveHandler) HandlePushToTalk(ctx context.Context, req voice.PushTo
|
||||
|
||||
// 2-5. the shared turn pipeline (confirm → clarify → route → dialogue →
|
||||
// action → replier), identical to the text path.
|
||||
replyText := h.runTurn(ctx, text, sourceVoice)
|
||||
replyText := h.runTurn(ctx, router.NormalizedInput{Text: text, MatchText: router.NormalizeMatchText(text), Source: sourceVoice})
|
||||
|
||||
// 6. tts — synthesise the reply text; return to the voice server which
|
||||
// ships it back on the conn.
|
||||
@@ -244,30 +244,30 @@ func (h *reactiveHandler) upgradeAPI(api ipc.CoreAPI) {
|
||||
// HandlePushToTalk so text channels share the same routing logic.
|
||||
func (h *reactiveHandler) handleText(ctx context.Context, conversation, text string) string {
|
||||
log.Printf("voice: handleText: %q", text)
|
||||
return h.runTurn(withDialogueID(ctx, dialogueIDFor(sourceText, conversation)), text, sourceText)
|
||||
return h.runTurn(withDialogueID(ctx, dialogueIDFor(sourceText, conversation)), router.NormalizedInput{Text: text, MatchText: router.NormalizeMatchText(text), Source: sourceText})
|
||||
}
|
||||
|
||||
// turnSource — which channel this utterance arrived on, in the same provenance
|
||||
// vocabulary facts use (internal/event). It is threaded through runTurn because
|
||||
// a turn can write a fact, and a fact that lies about where it came from is
|
||||
// worse than no fact: provenance is the first column read when asking why a
|
||||
// daemon-wide setting is the way it is.
|
||||
type turnSource string
|
||||
// turnSource is a local alias for router.InputSource, kept so the daemon code
|
||||
// reads sourceVoice/sourceText without a package prefix at every call site.
|
||||
// The canonical type lives in the router package; this is pure convenience.
|
||||
type turnSource = router.InputSource
|
||||
|
||||
const (
|
||||
sourceVoice turnSource = "tap:voice" // HandlePushToTalk, a real microphone
|
||||
sourceText turnSource = "tap:text" // handleText: mavweb /api/chat, telegram
|
||||
sourceVoice = router.InputSourceVoice
|
||||
sourceText = router.InputSourceText
|
||||
)
|
||||
|
||||
// runTurn — the reactive turn pipeline shared by the voice and text entry
|
||||
// points: expired-clarify notice → confirm answer → explicit correction →
|
||||
// clarify answer → quiet toggle → reminder cancellation → route → dialogue
|
||||
// merge → clarify question → action → replier.
|
||||
// Takes the already-transcribed utterance, returns the reply text; the voice
|
||||
// path wraps it in stt/tts, the text path returns it as-is.
|
||||
// Takes the NormalizedInput (typed ingress boundary), returns the reply text;
|
||||
// the voice path wraps it in stt/tts, the text path returns it as-is.
|
||||
//
|
||||
// The ordering is load-bearing — see the step comments.
|
||||
func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSource) (reply string) {
|
||||
func (h *reactiveHandler) runTurn(ctx context.Context, input router.NormalizedInput) (reply string) {
|
||||
text := input.Text
|
||||
src := input.Source
|
||||
// 0. the decision record (V-564). Installed here rather than in the IPC
|
||||
// entry point, so the mic, telegram and the web all leave the same trail —
|
||||
// a record only the web produced would be missing exactly the turns that
|
||||
@@ -275,7 +275,7 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
// on a human-rate path, and no claim site can change a route with it.
|
||||
if h.decisions != nil {
|
||||
var rec *decision.Record
|
||||
ctx, rec = decision.With(ctx, text)
|
||||
ctx, rec = decision.With(ctx, text, string(src))
|
||||
decision.Expect(ctx, decision.StagePreRoute, preRouteLadder)
|
||||
defer func() {
|
||||
done := rec.Finish(h.now())
|
||||
@@ -289,7 +289,7 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
// claiming it, and step 5 acts on the same decision — routing twice would
|
||||
// cost a second on the resident model and could disagree with itself.
|
||||
now := h.now()
|
||||
rt := h.newTurnRoute(text, now)
|
||||
rt := h.newTurnRoute(input, now)
|
||||
ctx = withTurnRoute(ctx, rt)
|
||||
// A resolver may suspend an older clarify flow even when it handles this
|
||||
// turn itself. Finalise that state at one choke point so early returns from
|
||||
@@ -414,6 +414,10 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
return withNotice(expiredNotice, "не получилось разобрать команду.")
|
||||
}
|
||||
log.Printf("voice: route result: intent=%s slots=%+v", dec.Intent, dec.Slots)
|
||||
// Carry the route producer into the decision record for observability.
|
||||
if rec := decision.From(ctx); rec != nil && dec.Producer != "" {
|
||||
rec.RouteProducer = string(dec.Producer)
|
||||
}
|
||||
|
||||
// 7. dialogue — fill this turn's missing slots from a prior same-intent
|
||||
// turn (follow-ups like «напомни завтра» → «…позвонить маме»), then remember
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
// merge-corpus merges the frozen holdout from corpus_v1.json with the
|
||||
// expanded v2 development pool, writing the result back to corpus_v1.json.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/merge-corpus/ -v1 internal/router/semantic/corpus_v1.json \
|
||||
// -v2 /tmp/corpus_v2.json -out internal/router/semantic/corpus_v1.json
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
func main() {
|
||||
v1Path := flag.String("v1", "internal/router/semantic/corpus_v1.json", "v1 corpus path")
|
||||
v2Path := flag.String("v2", "/tmp/corpus_v2.json", "v2 factory output path")
|
||||
outPath := flag.String("out", "internal/router/semantic/corpus_v1.json", "output path")
|
||||
flag.Parse()
|
||||
|
||||
// 1. Load v1
|
||||
v1Data, err := os.ReadFile(*v1Path)
|
||||
if err != nil {
|
||||
log.Fatalf("read v1: %v", err)
|
||||
}
|
||||
var v1Env semantic.CorpusEnvelope
|
||||
if err := json.Unmarshal(v1Data, &v1Env); err != nil {
|
||||
log.Fatalf("parse v1: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "v1: %d examples\n", len(v1Env.Examples))
|
||||
|
||||
// 2. Identify frozen holdout from v1
|
||||
frozen, _, _ := semantic.FrozenHoldoutSplit(v1Env.Examples)
|
||||
frozenTexts := make(map[string]bool)
|
||||
for _, e := range frozen {
|
||||
frozenTexts[strings.TrimSpace(e.Text)] = true
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "v1 frozen holdout: %d examples\n", len(frozen))
|
||||
|
||||
// 3. Load v2
|
||||
v2Data, err := os.ReadFile(*v2Path)
|
||||
if err != nil {
|
||||
log.Fatalf("read v2: %v", err)
|
||||
}
|
||||
var v2Env semantic.CorpusEnvelope
|
||||
if err := json.Unmarshal(v2Data, &v2Env); err != nil {
|
||||
log.Fatalf("parse v2: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "v2: %d examples\n", len(v2Env.Examples))
|
||||
|
||||
// 4. Merge: frozen from v1 + all from v2
|
||||
// Dedup by normalized text
|
||||
seen := make(map[string]bool)
|
||||
var merged []semantic.RouteExample
|
||||
|
||||
// Frozen holdout first
|
||||
for _, e := range frozen {
|
||||
norm := strings.TrimSpace(e.Text)
|
||||
if seen[norm] {
|
||||
fmt.Fprintf(os.Stderr, "SKIP v1 frozen dup: %q\n", norm)
|
||||
continue
|
||||
}
|
||||
seen[norm] = true
|
||||
merged = append(merged, e)
|
||||
}
|
||||
|
||||
// V2 examples
|
||||
skipped := 0
|
||||
for _, e := range v2Env.Examples {
|
||||
norm := strings.TrimSpace(e.Text)
|
||||
if seen[norm] {
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
// Check if this text conflicts with a frozen holdout entry
|
||||
if frozenTexts[norm] {
|
||||
// Text exists in frozen — skip v2 version to preserve frozen
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
seen[norm] = true
|
||||
merged = append(merged, e)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "merged: %d examples (skipped %d duplicates)\n", len(merged), skipped)
|
||||
|
||||
// 5. Fast-path metadata must match the real router. V2 rows are derived
|
||||
// by interactively-labeled construction; a stale manually-supplied value
|
||||
// is a build error, never silently rewritten. Frozen holdout rows are
|
||||
// preserved verbatim — report drift, do not touch them.
|
||||
v2Checked, frozenChecked, frozenDrift := 0, 0, 0
|
||||
for _, e := range merged {
|
||||
derived := semantic.DeriveFastPath(e.Text).Matched
|
||||
if frozenTexts[strings.TrimSpace(e.Text)] {
|
||||
frozenChecked++
|
||||
if derived != e.FastPathResolved {
|
||||
frozenDrift++
|
||||
fmt.Fprintf(os.Stderr, "frozen drift: %q stored=%v derived=%v\n", e.Text, e.FastPathResolved, derived)
|
||||
}
|
||||
continue
|
||||
}
|
||||
v2Checked++
|
||||
if derived != e.FastPathResolved {
|
||||
log.Fatalf("v2 row (source=%s source_id=%s) fast_path_resolved=%v but router derives %v: %q",
|
||||
e.Source, e.SourceID, e.FastPathResolved, derived, e.Text)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "fast-path check: v2 rows %d OK, frozen rows %d (drift %d, reported only)\n",
|
||||
v2Checked, frozenChecked, frozenDrift)
|
||||
|
||||
// 6. Validate
|
||||
if err := semantic.ValidateCorpus(merged); err != nil {
|
||||
log.Fatalf("validation failed: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "validation: OK\n")
|
||||
|
||||
// 7. Compute dataset hash
|
||||
texts := make([]string, len(merged))
|
||||
for i, e := range merged {
|
||||
texts[i] = e.Text
|
||||
}
|
||||
sort.Strings(texts)
|
||||
h := sha256.Sum256([]byte(strings.Join(texts, "\n")))
|
||||
datasetHash := hex.EncodeToString(h[:16])
|
||||
|
||||
// 8. Stats
|
||||
routeCounts := make(map[semantic.SemanticRoute]int)
|
||||
fpCount, resCount := 0, 0
|
||||
for _, e := range merged {
|
||||
routeCounts[e.Route]++
|
||||
if e.FastPathResolved {
|
||||
fpCount++
|
||||
} else {
|
||||
resCount++
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "\nRoute distribution:\n")
|
||||
for _, r := range semantic.AllRoutes {
|
||||
fmt.Fprintf(os.Stderr, " %-15s %d\n", r, routeCounts[r])
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "fast-path: %d residual: %d\n", fpCount, resCount)
|
||||
|
||||
// 9. Write merged corpus
|
||||
outEnv := semantic.CorpusEnvelope{
|
||||
SchemaVersion: 1,
|
||||
Name: "semantic_coarse_route_v1",
|
||||
Notes: []string{
|
||||
"Coarse semantic route corpus for the learned-router experiment.",
|
||||
"Frozen holdout preserved from v1. Development pool expanded by corpus-factory v2.",
|
||||
"Every row carries provenance (source + source_id) and a split_group.",
|
||||
"Labels come from explicit mapping rules, not model output.",
|
||||
},
|
||||
Reproducibility: &semantic.ReproducibilityMeta{
|
||||
SourceFixtureHash: "v1-frozen + corpus-factory-v2",
|
||||
ContrastGeneratorVersion: "v2-direct-generation",
|
||||
SplitAlgorithm: "grouped-cv-v1",
|
||||
DatasetHash: datasetHash,
|
||||
},
|
||||
Examples: merged,
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(outEnv, "", " ")
|
||||
if err != nil {
|
||||
log.Fatalf("marshal: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(*outPath, data, 0644); err != nil {
|
||||
log.Fatalf("write %s: %v", *outPath, err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "\nOutput: %s (%d bytes)\n", *outPath, len(data))
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,856 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Semantic Router Linear Head Experiment
|
||||
======================================
|
||||
|
||||
Evaluates whether the six-way residual routing boundary is linearly learnable
|
||||
from Maven's existing e5-small representation.
|
||||
|
||||
Architecture under test:
|
||||
NormalizedInput.MatchText
|
||||
→ existing multilingual-e5-small embedder
|
||||
→ tiny 6-class linear head (softmax logistic regression)
|
||||
→ SemanticRouteDecision
|
||||
|
||||
No new encoder. No fine-tuning. Linear separability floor only.
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
import sys
|
||||
import warnings
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from sklearn.exceptions import ConvergenceWarning
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.metrics import (
|
||||
accuracy_score,
|
||||
f1_score,
|
||||
precision_recall_fscore_support,
|
||||
confusion_matrix,
|
||||
brier_score_loss,
|
||||
log_loss,
|
||||
)
|
||||
from sklearn.preprocessing import LabelEncoder
|
||||
|
||||
warnings.filterwarnings("ignore", category=ConvergenceWarning)
|
||||
|
||||
# ─── Constants ──────────────────────────────────────────────────────────────
|
||||
|
||||
ROUTES = ["action", "conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
ROUTE_IDX = {r: i for i, r in enumerate(ROUTES)}
|
||||
|
||||
# Regularization grid
|
||||
C_VALUES = [0.01, 0.1, 1.0, 10.0, 100.0]
|
||||
|
||||
# Abstention thresholds
|
||||
THRESHOLDS = [0.40, 0.50, 0.60, 0.70, 0.80, 0.90]
|
||||
|
||||
# Contrast families
|
||||
CONTRAST_FAMILIES = [
|
||||
"negation", "question", "reported_speech", "quotation",
|
||||
"hypothetical", "capability_question",
|
||||
]
|
||||
|
||||
# ─── Data Loading ───────────────────────────────────────────────────────────
|
||||
|
||||
def load_embeddings(path):
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
meta = data["meta"]
|
||||
examples = data["examples"]
|
||||
return meta, examples
|
||||
|
||||
|
||||
def filter_dev_pool(examples):
|
||||
"""Return only development pool examples (not frozen holdout)."""
|
||||
return [e for e in examples if e["dev_pool"]]
|
||||
|
||||
|
||||
def filter_residual(examples):
|
||||
"""Return only router-residual examples (fast_path_resolved == false)."""
|
||||
return [e for e in examples if not e["fast_path_resolved"]]
|
||||
|
||||
|
||||
def extract_Xy(examples):
|
||||
"""Extract feature matrix and label vector from cached examples."""
|
||||
X = np.array([e["embedding"] for e in examples])
|
||||
y = np.array([e["route"] for e in examples])
|
||||
return X, y
|
||||
|
||||
|
||||
def get_fold_groups(examples):
|
||||
"""Return fold assignment array matching the Go-generated CV folds."""
|
||||
return np.array([e["cv_fold"] for e in examples])
|
||||
|
||||
|
||||
# ─── Grouped CV ─────────────────────────────────────────────────────────────
|
||||
|
||||
def grouped_cv_experiment(X, y, fold_ids, C_values, examples_meta):
|
||||
"""
|
||||
Run grouped cross-validation with logistic regression.
|
||||
Returns best C, per-fold results, and out-of-fold predictions.
|
||||
"""
|
||||
unique_folds = sorted(set(fold_ids))
|
||||
n_classes = len(ROUTES)
|
||||
|
||||
results_by_C = {}
|
||||
for C in C_values:
|
||||
fold_metrics = []
|
||||
oof_rows = []
|
||||
|
||||
for test_fold in unique_folds:
|
||||
train_mask = fold_ids != test_fold
|
||||
test_mask = fold_ids == test_fold
|
||||
|
||||
X_train, y_train = X[train_mask], y[train_mask]
|
||||
X_test, y_test = X[test_mask], y[test_mask]
|
||||
|
||||
model = LogisticRegression(
|
||||
C=C, max_iter=2000, solver="lbfgs",
|
||||
random_state=42,
|
||||
)
|
||||
model.fit(X_train, y_train)
|
||||
|
||||
y_pred = model.predict(X_test)
|
||||
y_proba = model.predict_proba(X_test)
|
||||
classes = model.classes_
|
||||
|
||||
acc = accuracy_score(y_test, y_pred)
|
||||
macro_f1 = f1_score(y_test, y_pred, average="macro", zero_division=0)
|
||||
|
||||
prec, rec, f1, sup = precision_recall_fscore_support(
|
||||
y_test, y_pred, labels=ROUTES, zero_division=0
|
||||
)
|
||||
|
||||
false_action = 0
|
||||
for true, pred in zip(y_test, y_pred):
|
||||
if true != "action" and pred == "action":
|
||||
false_action += 1
|
||||
|
||||
fold_metrics.append({
|
||||
"fold": test_fold,
|
||||
"train_size": len(X_train),
|
||||
"test_size": len(X_test),
|
||||
"accuracy": acc,
|
||||
"macro_f1": macro_f1,
|
||||
"per_class": {
|
||||
route: {"precision": prec[i], "recall": rec[i], "f1": f1[i]}
|
||||
for i, route in enumerate(ROUTES)
|
||||
},
|
||||
"false_action": false_action,
|
||||
})
|
||||
|
||||
for i, (true, pred) in enumerate(zip(y_test, y_pred)):
|
||||
test_indices = np.where(test_mask)[0]
|
||||
meta = examples_meta[test_indices[i]]
|
||||
proba_dict = {cls: float(y_proba[i][j]) for j, cls in enumerate(classes)}
|
||||
max_proba = max(proba_dict.values()) if proba_dict else 0.0
|
||||
oof_rows.append({
|
||||
"source_id": meta["source_id"],
|
||||
"fold": test_fold,
|
||||
"true": true,
|
||||
"predicted": pred,
|
||||
"correct": true == pred,
|
||||
"max_proba": max_proba,
|
||||
"proba": proba_dict,
|
||||
"fast_path_resolved": meta.get("fast_path_resolved", False),
|
||||
"tags": meta.get("tags", []),
|
||||
"text": meta["text"],
|
||||
})
|
||||
|
||||
# Aggregate across folds
|
||||
mean_acc = np.mean([m["accuracy"] for m in fold_metrics])
|
||||
mean_f1 = np.mean([m["macro_f1"] for m in fold_metrics])
|
||||
std_acc = np.std([m["accuracy"] for m in fold_metrics])
|
||||
std_f1 = np.std([m["macro_f1"] for m in fold_metrics])
|
||||
total_fa = sum(m["false_action"] for m in fold_metrics)
|
||||
|
||||
results_by_C[C] = {
|
||||
"mean_accuracy": mean_acc,
|
||||
"std_accuracy": std_acc,
|
||||
"mean_macro_f1": mean_f1,
|
||||
"std_macro_f1": std_f1,
|
||||
"total_false_action": total_fa,
|
||||
"fold_metrics": fold_metrics,
|
||||
"oof_predictions": oof_rows,
|
||||
}
|
||||
|
||||
# Select best C by mean macro F1
|
||||
best_C = max(results_by_C, key=lambda c: results_by_C[c]["mean_macro_f1"])
|
||||
return best_C, results_by_C
|
||||
|
||||
|
||||
# ─── Metrics Computation ────────────────────────────────────────────────────
|
||||
|
||||
def compute_full_metrics(y_true, y_pred, y_proba=None):
|
||||
"""Compute all required metrics from out-of-fold predictions."""
|
||||
acc = accuracy_score(y_true, y_pred)
|
||||
macro_f1 = f1_score(y_true, y_pred, average="macro", zero_division=0)
|
||||
|
||||
# Per-class P/R/F1
|
||||
prec, rec, f1, sup = precision_recall_fscore_support(
|
||||
y_true, y_pred, labels=ROUTES, zero_division=0
|
||||
)
|
||||
|
||||
# Confusion matrix
|
||||
cm = confusion_matrix(y_true, y_pred, labels=ROUTES)
|
||||
|
||||
# False action
|
||||
false_action = 0
|
||||
false_action_predicted = []
|
||||
for true, pred in zip(y_true, y_pred):
|
||||
if true != "action" and pred == "action":
|
||||
false_action += 1
|
||||
|
||||
# Action precision/recall
|
||||
action_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p == "action")
|
||||
action_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
action_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p != "action")
|
||||
action_precision = action_tp / max(action_tp + action_fp, 1)
|
||||
action_recall = action_tp / max(action_tp + action_fn, 1)
|
||||
|
||||
# Uncertain precision/recall
|
||||
unc_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "uncertain" and p == "uncertain")
|
||||
unc_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "uncertain" and p == "uncertain")
|
||||
unc_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "uncertain" and p != "uncertain")
|
||||
unc_precision = unc_tp / max(unc_tp + unc_fp, 1)
|
||||
unc_recall = unc_tp / max(unc_tp + unc_fn, 1)
|
||||
|
||||
metrics = {
|
||||
"accuracy": acc,
|
||||
"macro_f1": macro_f1,
|
||||
"false_action_count": false_action,
|
||||
"false_action_rate": false_action / max(len(y_true), 1),
|
||||
"action_precision": action_precision,
|
||||
"action_recall": action_recall,
|
||||
"uncertain_precision": unc_precision,
|
||||
"uncertain_recall": unc_recall,
|
||||
"per_class": {},
|
||||
"confusion_matrix": cm.tolist(),
|
||||
}
|
||||
|
||||
for i, route in enumerate(ROUTES):
|
||||
metrics["per_class"][route] = {
|
||||
"precision": float(prec[i]),
|
||||
"recall": float(rec[i]),
|
||||
"f1": float(f1[i]),
|
||||
"support": int(sup[i]),
|
||||
}
|
||||
|
||||
return metrics
|
||||
|
||||
|
||||
def compute_calibration(y_true, y_pred, oof_rows):
|
||||
"""Compute ECE, Brier score, and per-threshold abstention curves."""
|
||||
# ECE (Expected Calibration Error) with 10 bins
|
||||
n_bins = 10
|
||||
bin_boundaries = np.linspace(0, 1, n_bins + 1)
|
||||
ece = 0.0
|
||||
total = len(y_true)
|
||||
|
||||
confidences = np.array([r["max_proba"] for r in oof_rows])
|
||||
correct = np.array([r["correct"] for r in oof_rows])
|
||||
|
||||
for i in range(n_bins):
|
||||
lo, hi = bin_boundaries[i], bin_boundaries[i + 1]
|
||||
mask = (confidences > lo) & (confidences <= hi)
|
||||
if mask.sum() == 0:
|
||||
continue
|
||||
bin_acc = correct[mask].mean()
|
||||
bin_conf = confidences[mask].mean()
|
||||
ece += mask.sum() / total * abs(bin_acc - bin_conf)
|
||||
|
||||
# Brier score (multiclass one-hot encoding)
|
||||
y_true_idx = np.array([ROUTES.index(t) for t in y_true])
|
||||
n_samples = len(y_true)
|
||||
n_classes = len(ROUTES)
|
||||
y_true_oh = np.zeros((n_samples, n_classes))
|
||||
y_true_oh[np.arange(n_samples), y_true_idx] = 1.0
|
||||
|
||||
# Build probability matrix
|
||||
y_proba_matrix = np.zeros((n_samples, n_classes))
|
||||
for i, r in enumerate(oof_rows):
|
||||
for j, route in enumerate(ROUTES):
|
||||
y_proba_matrix[i, j] = r["proba"].get(route, 0.0)
|
||||
|
||||
brier = np.mean(np.sum((y_proba_matrix - y_true_oh) ** 2, axis=1))
|
||||
ll = log_loss(y_true, y_proba_matrix, labels=ROUTES)
|
||||
|
||||
# Abstention curves
|
||||
abstention_curves = []
|
||||
for thr in THRESHOLDS:
|
||||
accepted_mask = confidences >= thr
|
||||
n_accepted = accepted_mask.sum()
|
||||
coverage = n_accepted / max(total, 1)
|
||||
|
||||
if n_accepted > 0:
|
||||
acc_accepted = accuracy_score(y_true[accepted_mask], y_pred[accepted_mask])
|
||||
f1_accepted = f1_score(
|
||||
y_true[accepted_mask], y_pred[accepted_mask],
|
||||
average="macro", zero_division=0,
|
||||
)
|
||||
fa_count = sum(
|
||||
1 for t, p in zip(y_true[accepted_mask], y_pred[accepted_mask])
|
||||
if t != "action" and p == "action"
|
||||
)
|
||||
else:
|
||||
acc_accepted = 0.0
|
||||
f1_accepted = 0.0
|
||||
fa_count = 0
|
||||
|
||||
abstention_curves.append({
|
||||
"threshold": thr,
|
||||
"n_accepted": int(n_accepted),
|
||||
"coverage": coverage,
|
||||
"accuracy": acc_accepted,
|
||||
"macro_f1": f1_accepted,
|
||||
"false_action_count": fa_count,
|
||||
})
|
||||
|
||||
return {
|
||||
"ece": float(ece),
|
||||
"brier": float(brier),
|
||||
"log_loss": float(ll),
|
||||
"abstention_curves": abstention_curves,
|
||||
}
|
||||
|
||||
|
||||
def compute_action_threshold(oof_rows):
|
||||
"""Evaluate action-specific safety gate thresholds."""
|
||||
action_curves = []
|
||||
for thr in THRESHOLDS:
|
||||
action_pred = []
|
||||
for r in oof_rows:
|
||||
p = r["predicted"]
|
||||
proba = r["proba"].get("action", 0.0)
|
||||
if p == "action" and proba < thr:
|
||||
# Demote action prediction
|
||||
# Find next best route that isn't action
|
||||
sorted_routes = sorted(r["proba"].items(), key=lambda x: -x[1])
|
||||
for route, _ in sorted_routes:
|
||||
if route != "action":
|
||||
p = route
|
||||
break
|
||||
action_pred.append(p)
|
||||
|
||||
y_true = np.array([r["true"] for r in oof_rows])
|
||||
y_pred = np.array(action_pred)
|
||||
|
||||
action_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p == "action")
|
||||
action_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
action_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p != "action")
|
||||
false_action = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
|
||||
action_curves.append({
|
||||
"threshold": thr,
|
||||
"action_precision": action_tp / max(action_tp + action_fp, 1),
|
||||
"action_recall": action_tp / max(action_tp + action_fn, 1),
|
||||
"false_action_count": false_action,
|
||||
})
|
||||
|
||||
return action_curves
|
||||
|
||||
|
||||
def compute_disagreement(y_true, y_pred, legacy_pred, oof_rows):
|
||||
"""Analyze disagreements between legacy and learned router."""
|
||||
results = {
|
||||
"legacy_wrong_learned_right": [],
|
||||
"legacy_right_learned_wrong": [],
|
||||
"both_wrong_differently": [],
|
||||
}
|
||||
|
||||
legacy_false_actions = []
|
||||
learned_false_actions = []
|
||||
shared_false_actions = []
|
||||
|
||||
for i, r in enumerate(oof_rows):
|
||||
sid = r["source_id"]
|
||||
text = r["text"]
|
||||
true = y_true[i]
|
||||
learned = y_pred[i]
|
||||
legacy = legacy_pred[i]
|
||||
|
||||
if legacy != true and learned == true:
|
||||
results["legacy_wrong_learned_right"].append({
|
||||
"source_id": sid, "text": text,
|
||||
"true": true, "legacy": legacy, "learned": learned,
|
||||
})
|
||||
elif legacy == true and learned != true:
|
||||
results["legacy_right_learned_wrong"].append({
|
||||
"source_id": sid, "text": text,
|
||||
"true": true, "legacy": legacy, "learned": learned,
|
||||
})
|
||||
elif legacy != true and learned != true and legacy != learned:
|
||||
results["both_wrong_differently"].append({
|
||||
"source_id": sid, "text": text,
|
||||
"true": true, "legacy": legacy, "learned": learned,
|
||||
})
|
||||
|
||||
# False action tracking
|
||||
if true != "action" and legacy == "action":
|
||||
legacy_false_actions.append(sid)
|
||||
if true != "action" and learned == "action":
|
||||
learned_false_actions.append(sid)
|
||||
if true != "action" and legacy == "action" and learned == "action":
|
||||
shared_false_actions.append(sid)
|
||||
|
||||
# Repaired false actions
|
||||
repaired = [sid for sid in legacy_false_actions if sid not in learned_false_actions]
|
||||
new_errors = [sid for sid in learned_false_actions if sid not in legacy_false_actions]
|
||||
shared = shared_false_actions
|
||||
|
||||
return {
|
||||
"details": results,
|
||||
"legacy_false_actions": legacy_false_actions,
|
||||
"learned_false_actions": learned_false_actions,
|
||||
"repaired": repaired,
|
||||
"new_errors": new_errors,
|
||||
"shared": shared,
|
||||
}
|
||||
|
||||
|
||||
def compute_contrast_family(oof_rows):
|
||||
"""Analyze performance per contrast family."""
|
||||
family_results = {}
|
||||
for family in CONTRAST_FAMILIES:
|
||||
members = [r for r in oof_rows if family in r.get("tags", [])]
|
||||
if not members:
|
||||
continue
|
||||
y_true = [r["true"] for r in members]
|
||||
y_pred = [r["predicted"] for r in members]
|
||||
correct = sum(1 for t, p in zip(y_true, y_pred) if t == p)
|
||||
false_act = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
family_results[family] = {
|
||||
"count": len(members),
|
||||
"correct": correct,
|
||||
"accuracy": correct / len(members),
|
||||
"false_action": false_act,
|
||||
}
|
||||
return family_results
|
||||
|
||||
|
||||
def compute_legacy_baseline(examples):
|
||||
"""
|
||||
Compute legacy baseline by mapping each example through the known
|
||||
fast-path and classifier behavior. Since we don't have the actual
|
||||
router running, we use the corpus metadata:
|
||||
- fast_path_resolved examples are correct (grammar handles them)
|
||||
- We simulate the legacy baseline from the test output numbers
|
||||
"""
|
||||
# The actual legacy baseline was measured in TestLegacyBaseline:
|
||||
# overall accuracy: 52.2%, residual: 40.8%, false-action rate: 19.9%
|
||||
# We need per-example predictions. We'll approximate from the corpus structure.
|
||||
#
|
||||
# For fast_path_resolved examples, the legacy router is correct (stage-0 grammar).
|
||||
# For residual examples, we need to simulate the hash-embedder classifier.
|
||||
# Since we don't have the hash embedder running, we use the known aggregate.
|
||||
#
|
||||
# This is a limitation: the legacy baseline numbers come from the Go test,
|
||||
# and we only have the aggregate. We'll use the aggregate for comparison.
|
||||
pass
|
||||
|
||||
|
||||
# ─── Report Generation ──────────────────────────────────────────────────────
|
||||
|
||||
def fmt_pct(v, decimals=1):
|
||||
return f"{100 * v:.{decimals}f}%"
|
||||
|
||||
|
||||
def fmt_float(v, decimals=3):
|
||||
return f"{v:.{decimals}f}"
|
||||
|
||||
|
||||
def generate_report(meta, all_results, residual_results, oof_all, oof_residual):
|
||||
"""Generate the full experiment report."""
|
||||
lines = []
|
||||
lines.append("# Semantic Router Linear Head Experiment — Report")
|
||||
lines.append("")
|
||||
lines.append("## 1. Exact e5 representation used")
|
||||
lines.append("")
|
||||
lines.append(f"- **Model**: {meta['embedder_id']}")
|
||||
lines.append(f"- **Checkpoint**: {meta['model_path']}")
|
||||
lines.append(f"- **Tokenizer**: {meta['tokenizer_path']}")
|
||||
lines.append(f"- **Dimension**: {meta['dimension']}")
|
||||
lines.append(f"- **Pooling**: {meta['pooling']}")
|
||||
lines.append(f"- **Normalization**: {meta['normalization']}")
|
||||
lines.append(f"- **Input template**: {meta['input_template']}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("## 2. Development/residual row counts")
|
||||
lines.append("")
|
||||
lines.append(f"- Total corpus: {meta['total_examples']}")
|
||||
lines.append(f"- Frozen holdout: {meta['frozen_count']}")
|
||||
lines.append(f"- Development pool: {meta['dev_count']}")
|
||||
lines.append(f"- Fast-path resolved: {meta['fast_path_count']}")
|
||||
lines.append(f"- Router-residual: {meta['residual_count']}")
|
||||
lines.append("")
|
||||
lines.append("Route distribution (full corpus):")
|
||||
for route, count in sorted(meta["route_counts"].items()):
|
||||
lines.append(f" - {route}: {count}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("## 3. Grouped fold composition")
|
||||
lines.append("")
|
||||
lines.append(f"Folds: {meta['cv_folds']}")
|
||||
for fold_id, stats in sorted(meta["fold_composition"].items()):
|
||||
lines.append(f" - Fold {fold_id}: eval={stats['eval_count']} train={stats['train_count']} routes={stats['eval_routes']}")
|
||||
lines.append("")
|
||||
|
||||
# Regularization selection
|
||||
lines.append("## 4. Selected regularization")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment A: All development examples")
|
||||
best_C_all = all_results["best_C"]
|
||||
lines.append(f"- Best C: {best_C_all}")
|
||||
lines.append(f"- Mean accuracy: {fmt_pct(all_results['results_by_C'][best_C_all]['mean_accuracy'])} ± {fmt_pct(all_results['results_by_C'][best_C_all]['std_accuracy'])}")
|
||||
lines.append(f"- Mean macro F1: {fmt_float(all_results['results_by_C'][best_C_all]['mean_macro_f1'])} ± {fmt_float(all_results['results_by_C'][best_C_all]['std_macro_f1'])}")
|
||||
lines.append(f"- Total false actions (CV): {all_results['results_by_C'][best_C_all]['total_false_action']}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B: Router-residual only")
|
||||
best_C_res = residual_results["best_C"]
|
||||
lines.append(f"- Best C: {best_C_res}")
|
||||
lines.append(f"- Mean accuracy: {fmt_pct(residual_results['results_by_C'][best_C_res]['mean_accuracy'])} ± {fmt_pct(residual_results['results_by_C'][best_C_res]['std_accuracy'])}")
|
||||
lines.append(f"- Mean macro F1: {fmt_float(residual_results['results_by_C'][best_C_res]['mean_macro_f1'])} ± {fmt_float(residual_results['results_by_C'][best_C_res]['std_macro_f1'])}")
|
||||
lines.append(f"- Total false actions (CV): {residual_results['results_by_C'][best_C_res]['total_false_action']}")
|
||||
lines.append("")
|
||||
|
||||
# Stability across folds
|
||||
lines.append("### Stability across folds")
|
||||
lines.append("")
|
||||
for C_val in C_VALUES:
|
||||
r = all_results["results_by_C"][C_val]
|
||||
fold_accs = [m["accuracy"] for m in r["fold_metrics"]]
|
||||
fold_f1s = [m["macro_f1"] for m in r["fold_metrics"]]
|
||||
lines.append(f" C={C_val:<6} acc={fmt_pct(r['mean_accuracy'])}±{fmt_pct(r['std_accuracy'])} f1={fmt_float(r['mean_macro_f1'])}±{fmt_float(r['std_macro_f1'])} folds_acc={[fmt_pct(a) for a in fold_accs]}")
|
||||
lines.append("")
|
||||
|
||||
# Experiment A metrics
|
||||
lines.append("## 5. All-example CV metrics")
|
||||
lines.append("")
|
||||
metrics_all = all_results["full_metrics"]
|
||||
lines.append(f"- Accuracy: {fmt_pct(metrics_all['accuracy'])}")
|
||||
lines.append(f"- Macro F1: {fmt_float(metrics_all['macro_f1'])}")
|
||||
lines.append(f"- False-action count: {metrics_all['false_action_count']}")
|
||||
lines.append(f"- False-action rate: {fmt_pct(metrics_all['false_action_rate'])}")
|
||||
lines.append(f"- Action precision: {fmt_float(metrics_all['action_precision'])}")
|
||||
lines.append(f"- Action recall: {fmt_float(metrics_all['action_recall'])}")
|
||||
lines.append(f"- Uncertain precision: {fmt_float(metrics_all['uncertain_precision'])}")
|
||||
lines.append(f"- Uncertain recall: {fmt_float(metrics_all['uncertain_recall'])}")
|
||||
lines.append("")
|
||||
lines.append("Per-class metrics:")
|
||||
for route in ROUTES:
|
||||
pc = metrics_all["per_class"][route]
|
||||
lines.append(f" {route:<15} P={fmt_float(pc['precision'])} R={fmt_float(pc['recall'])} F1={fmt_float(pc['f1'])} (n={pc['support']})")
|
||||
lines.append("")
|
||||
lines.append("Confusion matrix (rows=expected, cols=predicted):")
|
||||
header = f"{'':>15}" + "".join(f"{r:>15}" for r in ROUTES)
|
||||
lines.append(header)
|
||||
for i, route in enumerate(ROUTES):
|
||||
row = f"{route:>15}" + "".join(f"{metrics_all['confusion_matrix'][i][j]:>15}" for j in range(len(ROUTES)))
|
||||
lines.append(row)
|
||||
lines.append("")
|
||||
|
||||
# Experiment B metrics
|
||||
lines.append("## 6. Residual-only CV metrics")
|
||||
lines.append("")
|
||||
metrics_res = residual_results["full_metrics"]
|
||||
lines.append(f"- Accuracy: {fmt_pct(metrics_res['accuracy'])}")
|
||||
lines.append(f"- Macro F1: {fmt_float(metrics_res['macro_f1'])}")
|
||||
lines.append(f"- False-action count: {metrics_res['false_action_count']}")
|
||||
lines.append(f"- False-action rate: {fmt_pct(metrics_res['false_action_rate'])}")
|
||||
lines.append(f"- Action precision: {fmt_float(metrics_res['action_precision'])}")
|
||||
lines.append(f"- Action recall: {fmt_float(metrics_res['action_recall'])}")
|
||||
lines.append(f"- Uncertain precision: {fmt_float(metrics_res['uncertain_precision'])}")
|
||||
lines.append(f"- Uncertain recall: {fmt_float(metrics_res['uncertain_recall'])}")
|
||||
lines.append("")
|
||||
lines.append("Per-class metrics:")
|
||||
for route in ROUTES:
|
||||
pc = metrics_res["per_class"][route]
|
||||
lines.append(f" {route:<15} P={fmt_float(pc['precision'])} R={fmt_float(pc['recall'])} F1={fmt_float(pc['f1'])} (n={pc['support']})")
|
||||
lines.append("")
|
||||
lines.append("Confusion matrix (rows=expected, cols=predicted):")
|
||||
header = f"{'':>15}" + "".join(f"{r:>15}" for r in ROUTES)
|
||||
lines.append(header)
|
||||
for i, route in enumerate(ROUTES):
|
||||
row = f"{route:>15}" + "".join(f"{metrics_res['confusion_matrix'][i][j]:>15}" for j in range(len(ROUTES)))
|
||||
lines.append(row)
|
||||
lines.append("")
|
||||
|
||||
# Legacy comparison
|
||||
lines.append("## 7. Legacy-vs-linear comparison")
|
||||
lines.append("")
|
||||
lines.append("### All examples")
|
||||
lines.append(f"{'metric':<25} {'legacy':>10} {'linear e5':>10} {'delta':>10}")
|
||||
lines.append("-" * 55)
|
||||
# Legacy baseline from test: 52.2% overall, 40.8% residual, 19.9% false-action
|
||||
# These are approximate since we don't have per-example legacy predictions
|
||||
legacy_acc = 0.522
|
||||
legacy_fa_rate = 0.199
|
||||
legacy_macro_f1 = 0.0 # unknown precisely
|
||||
lines.append(f"{'accuracy':<25} {fmt_pct(legacy_acc):>10} {fmt_pct(metrics_all['accuracy']):>10} {fmt_pct(metrics_all['accuracy'] - legacy_acc):>10}")
|
||||
lines.append(f"{'macro F1':<25} {'—':>10} {fmt_float(metrics_all['macro_f1']):>10} {'—':>10}")
|
||||
lines.append(f"{'action precision':<25} {'—':>10} {fmt_float(metrics_all['action_precision']):>10} {'—':>10}")
|
||||
lines.append(f"{'false-action rate':<25} {fmt_pct(legacy_fa_rate):>10} {fmt_pct(metrics_all['false_action_rate']):>10} {fmt_pct(metrics_all['false_action_rate'] - legacy_fa_rate):>10}")
|
||||
lines.append(f"{'uncertain F1':<25} {fmt_float(0.0):>10} {fmt_float(metrics_all['per_class']['uncertain']['f1']):>10} {fmt_float(metrics_all['per_class']['uncertain']['f1']):>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Router-residual only")
|
||||
lines.append(f"{'metric':<25} {'legacy':>10} {'linear e5':>10} {'delta':>10}")
|
||||
lines.append("-" * 55)
|
||||
legacy_res_acc = 0.408
|
||||
lines.append(f"{'accuracy':<25} {fmt_pct(legacy_res_acc):>10} {fmt_pct(metrics_res['accuracy']):>10} {fmt_pct(metrics_res['accuracy'] - legacy_res_acc):>10}")
|
||||
lines.append(f"{'macro F1':<25} {'—':>10} {fmt_float(metrics_res['macro_f1']):>10} {'—':>10}")
|
||||
lines.append(f"{'false-action rate':<25} {'—':>10} {fmt_pct(metrics_res['false_action_rate']):>10} {'—':>10}")
|
||||
lines.append("")
|
||||
|
||||
# Fold variance
|
||||
lines.append("## 8. Fold variance")
|
||||
lines.append("")
|
||||
lines.append("All-example CV:")
|
||||
for m in all_results["results_by_C"][best_C_all]["fold_metrics"]:
|
||||
lines.append(f" Fold {m['fold']}: acc={fmt_pct(m['accuracy'])} f1={fmt_float(m['macro_f1'])} false_action={m['false_action']}")
|
||||
lines.append("")
|
||||
lines.append("Residual-only CV:")
|
||||
for m in residual_results["results_by_C"][best_C_res]["fold_metrics"]:
|
||||
lines.append(f" Fold {m['fold']}: acc={fmt_pct(m['accuracy'])} f1={fmt_float(m['macro_f1'])} false_action={m['false_action']}")
|
||||
lines.append("")
|
||||
|
||||
# Disagreement analysis
|
||||
lines.append("## 9. False-action repair/new-error analysis")
|
||||
lines.append("")
|
||||
# The disagreement analysis requires legacy per-example predictions.
|
||||
# Since we don't have those, we report what we can from the out-of-fold data.
|
||||
lines.append("Note: Legacy per-example predictions were not available for this experiment.")
|
||||
lines.append("The legacy baseline was measured in aggregate in the Go test suite.")
|
||||
lines.append("")
|
||||
lines.append("Learned router false-action cases (out-of-fold):")
|
||||
for r in oof_all:
|
||||
if r["true"] != "action" and r["predicted"] == "action":
|
||||
lines.append(f" {r['source_id']}: '{r['text']}' (true={r['true']}, proba(action)={r['proba'].get('action', 0):.3f})")
|
||||
lines.append("")
|
||||
|
||||
# Contrast family
|
||||
lines.append("## 10. Contrast-family results")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A (all dev)")
|
||||
contrast_all = all_results["contrast_family"]
|
||||
lines.append(f"{'family':<25} {'count':>6} {'correct':>8} {'accuracy':>10} {'false_act':>10}")
|
||||
lines.append("-" * 60)
|
||||
for family in CONTRAST_FAMILIES:
|
||||
if family in contrast_all:
|
||||
c = contrast_all[family]
|
||||
lines.append(f"{family:<25} {c['count']:>6} {c['correct']:>8} {fmt_pct(c['accuracy']):>10} {c['false_action']:>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B (residual only)")
|
||||
contrast_res = residual_results["contrast_family"]
|
||||
lines.append(f"{'family':<25} {'count':>6} {'correct':>8} {'accuracy':>10} {'false_act':>10}")
|
||||
lines.append("-" * 60)
|
||||
for family in CONTRAST_FAMILIES:
|
||||
if family in contrast_res:
|
||||
c = contrast_res[family]
|
||||
lines.append(f"{family:<25} {c['count']:>6} {c['correct']:>8} {fmt_pct(c['accuracy']):>10} {c['false_action']:>10}")
|
||||
lines.append("")
|
||||
|
||||
# Calibration
|
||||
lines.append("## 11. Calibration metrics")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A")
|
||||
cal_all = all_results["calibration"]
|
||||
lines.append(f"- ECE: {fmt_float(cal_all['ece'])}")
|
||||
lines.append(f"- Brier score: {fmt_float(cal_all['brier'])}")
|
||||
lines.append(f"- Log loss: {fmt_float(cal_all['log_loss'])}")
|
||||
lines.append("")
|
||||
lines.append("### Experiment B")
|
||||
cal_res = residual_results["calibration"]
|
||||
lines.append(f"- ECE: {fmt_float(cal_res['ece'])}")
|
||||
lines.append(f"- Brier score: {fmt_float(cal_res['brier'])}")
|
||||
lines.append(f"- Log loss: {fmt_float(cal_res['log_loss'])}")
|
||||
lines.append("")
|
||||
|
||||
# Abstention curves
|
||||
lines.append("## 12. Abstention curves")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A (all dev)")
|
||||
lines.append(f"{'threshold':>10} {'n_accepted':>11} {'coverage':>10} {'accuracy':>10} {'macro_f1':>10} {'false_act':>10}")
|
||||
lines.append("-" * 62)
|
||||
for curve in cal_all["abstention_curves"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {curve['n_accepted']:>11} {fmt_pct(curve['coverage']):>10} {fmt_pct(curve['accuracy']):>10} {fmt_float(curve['macro_f1']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B (residual only)")
|
||||
lines.append(f"{'threshold':>10} {'n_accepted':>11} {'coverage':>10} {'accuracy':>10} {'macro_f1':>10} {'false_act':>10}")
|
||||
lines.append("-" * 62)
|
||||
for curve in cal_res["abstention_curves"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {curve['n_accepted']:>11} {fmt_pct(curve['coverage']):>10} {fmt_pct(curve['accuracy']):>10} {fmt_float(curve['macro_f1']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
# Action threshold
|
||||
lines.append("## 13. Action-threshold curve")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A")
|
||||
lines.append(f"{'threshold':>10} {'action_P':>10} {'action_R':>10} {'false_act':>10}")
|
||||
lines.append("-" * 40)
|
||||
for curve in all_results["action_threshold"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {fmt_float(curve['action_precision']):>10} {fmt_float(curve['action_recall']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B")
|
||||
lines.append(f"{'threshold':>10} {'action_P':>10} {'action_R':>10} {'false_act':>10}")
|
||||
lines.append("-" * 40)
|
||||
for curve in residual_results["action_threshold"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {fmt_float(curve['action_precision']):>10} {fmt_float(curve['action_recall']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
# Model artifact size
|
||||
lines.append("## 14. Model artifact size and runtime cost")
|
||||
lines.append("")
|
||||
# Logistic regression: 6 classes × 384 features + 6 biases = 2310 parameters
|
||||
n_params = len(ROUTES) * meta["dimension"] + len(ROUTES)
|
||||
serialized_bytes = n_params * 4 # float32
|
||||
lines.append(f"- Trainable parameters: {n_params}")
|
||||
lines.append(f" - {len(ROUTES)} classes × {meta['dimension']} features = {len(ROUTES) * meta['dimension']} weights")
|
||||
lines.append(f" - {len(ROUTES)} bias terms")
|
||||
lines.append(f"- Serialized head size: {serialized_bytes} bytes ({serialized_bytes / 1024:.1f} KB)")
|
||||
lines.append(f"- Additional inference FLOPs: {len(ROUTES) * meta['dimension']} multiply-accumulates")
|
||||
lines.append(f"- Incremental cost (e5 already computed): ~{len(ROUTES) * meta['dimension']} FLOPs, <1µs")
|
||||
lines.append(f"- Cost if semantic router must trigger its own e5: full ONNX inference (~{meta['dimension']} × 128 × 12 = ~590K FLOPs)")
|
||||
lines.append("")
|
||||
|
||||
# Recommendation
|
||||
lines.append("## 16. Recommendation")
|
||||
lines.append("")
|
||||
# Decision logic
|
||||
all_f1 = metrics_all["macro_f1"]
|
||||
res_f1 = metrics_res["macro_f1"]
|
||||
res_acc = metrics_res["accuracy"]
|
||||
|
||||
if res_f1 > 0.5 and res_acc > 0.55:
|
||||
verdict = "linear head sufficient"
|
||||
detail = (f"Residual macro F1 of {fmt_float(res_f1)} and accuracy of {fmt_pct(res_acc)} "
|
||||
f"exceed the legacy baseline (40.8% residual accuracy) by a meaningful margin. "
|
||||
f"A linear head over frozen e5-small embeddings is a viable first production candidate.")
|
||||
elif all_f1 > 0.5 and res_f1 < 0.5:
|
||||
verdict = "need more data"
|
||||
detail = (f"All-example F1 ({fmt_float(all_f1)}) is acceptable but residual-only F1 "
|
||||
f"({fmt_float(res_f1)}) drops, suggesting the contrast-family examples are "
|
||||
f"hard for a linear classifier. More contrastive training data may help.")
|
||||
else:
|
||||
verdict = "representation inadequate"
|
||||
detail = (f"Neither all-example ({fmt_float(all_f1)}) nor residual-only ({fmt_float(res_f1)}) "
|
||||
f"F1 reaches the minimum viable threshold. The e5-small linear separability floor "
|
||||
f"is insufficient for this 6-way task. Consider a non-linear head or a different encoder.")
|
||||
|
||||
lines.append(f"**{verdict}**")
|
||||
lines.append("")
|
||||
lines.append(detail)
|
||||
lines.append("")
|
||||
|
||||
# Commit hash
|
||||
lines.append("## 17. Commit hash for experiment tooling")
|
||||
lines.append("")
|
||||
import subprocess
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True, cwd="/home/kami/apps/Maven"
|
||||
)
|
||||
lines.append(f"`{result.stdout.strip()}`")
|
||||
except Exception:
|
||||
lines.append("(unable to determine)")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def run_experiment(X, y, fold_ids, examples_meta, label):
|
||||
"""Run the full experiment pipeline for one population."""
|
||||
print(f"\n{'='*60}")
|
||||
print(f" Experiment: {label}")
|
||||
print(f" Samples: {len(y)} Features: {X.shape[1]} Folds: {len(set(fold_ids))}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# 1. Grouped CV with regularization search
|
||||
best_C, results_by_C = grouped_cv_experiment(X, y, fold_ids, C_VALUES, examples_meta)
|
||||
|
||||
# 2. Use best C to get full metrics from OOF predictions
|
||||
oof_rows = results_by_C[best_C]["oof_predictions"]
|
||||
oof_true = np.array([r["true"] for r in oof_rows])
|
||||
oof_pred = np.array([r["predicted"] for r in oof_rows])
|
||||
|
||||
full_metrics = compute_full_metrics(oof_true, oof_pred)
|
||||
|
||||
# 3. Calibration
|
||||
calibration = compute_calibration(oof_true, oof_pred, oof_rows)
|
||||
|
||||
# 4. Action threshold
|
||||
action_threshold = compute_action_threshold(oof_rows)
|
||||
|
||||
# 5. Contrast family
|
||||
contrast_family = compute_contrast_family(oof_rows)
|
||||
|
||||
return {
|
||||
"best_C": best_C,
|
||||
"results_by_C": results_by_C,
|
||||
"full_metrics": full_metrics,
|
||||
"calibration": calibration,
|
||||
"action_threshold": action_threshold,
|
||||
"contrast_family": contrast_family,
|
||||
"oof_rows": oof_rows,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
REPORT_PATH = "/tmp/mvn-experiment/report.md"
|
||||
|
||||
# Load data
|
||||
meta, examples = load_embeddings(EMBEDDING_PATH)
|
||||
print(f"Loaded {len(examples)} examples, embedder={meta['embedder_id']}, dim={meta['dimension']}")
|
||||
|
||||
# Development pool only
|
||||
dev_examples = filter_dev_pool(examples)
|
||||
print(f"Development pool: {len(dev_examples)} examples")
|
||||
|
||||
# Extract features
|
||||
X_all, y_all = extract_Xy(dev_examples)
|
||||
fold_ids_all = get_fold_groups(dev_examples)
|
||||
|
||||
# Experiment A: all dev examples
|
||||
all_results = run_experiment(X_all, y_all, fold_ids_all, dev_examples, "All development examples")
|
||||
|
||||
# Experiment B: residual only
|
||||
dev_residual = filter_residual(dev_examples)
|
||||
X_res, y_res = extract_Xy(dev_residual)
|
||||
fold_ids_res = get_fold_groups(dev_residual)
|
||||
residual_results = run_experiment(X_res, y_res, fold_ids_res, dev_residual, "Router-residual only")
|
||||
|
||||
# Generate report
|
||||
report = generate_report(
|
||||
meta, all_results, residual_results,
|
||||
all_results["oof_rows"], residual_results["oof_rows"],
|
||||
)
|
||||
|
||||
with open(REPORT_PATH, "w") as f:
|
||||
f.write(report)
|
||||
print(f"\nReport written to {REPORT_PATH}")
|
||||
|
||||
# Also print summary
|
||||
print("\n" + "="*60)
|
||||
print(" SUMMARY")
|
||||
print("="*60)
|
||||
print(f" All-example: acc={fmt_pct(all_results['full_metrics']['accuracy'])} macro_f1={fmt_float(all_results['full_metrics']['macro_f1'])} false_action={all_results['full_metrics']['false_action_count']}")
|
||||
print(f" Residual: acc={fmt_pct(residual_results['full_metrics']['accuracy'])} macro_f1={fmt_float(residual_results['full_metrics']['macro_f1'])} false_action={residual_results['full_metrics']['false_action_count']}")
|
||||
print(f" Best C (all): {all_results['best_C']}")
|
||||
print(f" Best C (res): {residual_results['best_C']}")
|
||||
print(f" ECE (all): {fmt_float(all_results['calibration']['ece'])}")
|
||||
print(f" ECE (res): {fmt_float(residual_results['calibration']['ece'])}")
|
||||
print(f" Brier (all): {fmt_float(all_results['calibration']['brier'])}")
|
||||
print(f" Brier (res): {fmt_float(residual_results['calibration']['brier'])}")
|
||||
@@ -0,0 +1,273 @@
|
||||
// semantic-router-experiment computes embeddings for the semantic coarse-route
|
||||
// corpus using the deployed multilingual-e5-small ONNX model. It outputs a
|
||||
// JSON file containing every corpus row with its embedding vector, split
|
||||
// assignment, and fold membership for grouped cross-validation.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// MAVEN_ONNX_LIB=/path/to/libonnxruntime.so \
|
||||
// go run ./cmd/semantic-router-experiment/ -out embeddings.json
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
// CachedRow is one corpus row with its precomputed embedding and split metadata.
|
||||
type CachedRow struct {
|
||||
Text string `json:"text"`
|
||||
Route string `json:"route"`
|
||||
Source string `json:"source"`
|
||||
SourceID string `json:"source_id"`
|
||||
SplitGroup string `json:"split_group"`
|
||||
Tags []string `json:"tags,omitempty"`
|
||||
FastPathResolved bool `json:"fast_path_resolved"`
|
||||
RouterResidual *bool `json:"router_residual,omitempty"`
|
||||
TextHash string `json:"text_hash"`
|
||||
Embedding []float32 `json:"embedding"`
|
||||
EmbedderID string `json:"embedder_id"`
|
||||
FrozenHoldout bool `json:"frozen_holdout"`
|
||||
CVFold int `json:"cv_fold"`
|
||||
DevPool bool `json:"dev_pool"`
|
||||
FamilyID string `json:"family_id"`
|
||||
}
|
||||
|
||||
// ExperimentMeta carries metadata about the experiment run.
|
||||
type ExperimentMeta struct {
|
||||
EmbedderID string `json:"embedder_id"`
|
||||
ModelPath string `json:"model_path"`
|
||||
TokenizerPath string `json:"tokenizer_path"`
|
||||
Dimension int `json:"dimension"`
|
||||
Pooling string `json:"pooling"`
|
||||
Normalization string `json:"normalization"`
|
||||
InputTemplate string `json:"input_template"`
|
||||
TotalExamples int `json:"total_examples"`
|
||||
FrozenCount int `json:"frozen_count"`
|
||||
DevCount int `json:"dev_count"`
|
||||
CVFolds int `json:"cv_folds"`
|
||||
FoldComposition map[int]FoldStats `json:"fold_composition"`
|
||||
RouteCounts map[string]int `json:"route_counts"`
|
||||
SourceCounts map[string]int `json:"source_counts"`
|
||||
FastPathCount int `json:"fast_path_count"`
|
||||
ResidualCount int `json:"residual_count"`
|
||||
HoldoutHash string `json:"holdout_hash"`
|
||||
}
|
||||
|
||||
// FoldStats describes one CV fold.
|
||||
type FoldStats struct {
|
||||
EvalCount int `json:"eval_count"`
|
||||
TrainCount int `json:"train_count"`
|
||||
Routes map[string]int `json:"eval_routes"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
outPath := flag.String("out", "embeddings.json", "output JSON path")
|
||||
folds := flag.Int("folds", 5, "number of CV folds")
|
||||
flag.Parse()
|
||||
|
||||
libPath := os.Getenv("MAVEN_ONNX_LIB")
|
||||
if libPath == "" {
|
||||
log.Fatal("MAVEN_ONNX_LIB must be set to the libonnxruntime.so path")
|
||||
}
|
||||
|
||||
// Resolve model paths relative to the module root (cwd when running with go run).
|
||||
modelPath := "models/embedder/multilingual-e5-small/model_quantized.onnx"
|
||||
tokPath := "models/embedder/multilingual-e5-small/tokenizer.json"
|
||||
|
||||
for _, p := range []string{libPath, modelPath, tokPath} {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
log.Fatalf("missing %s: %v", p, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Load corpus.
|
||||
exs, err := semantic.LoadCorpus()
|
||||
if err != nil {
|
||||
log.Fatalf("load corpus: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "corpus: %d examples\n", len(exs))
|
||||
|
||||
// Initialize embedder.
|
||||
emb, err := router.NewONNXEmbedder(modelPath, tokPath, libPath)
|
||||
if err != nil {
|
||||
log.Fatalf("init embedder: %v", err)
|
||||
}
|
||||
defer emb.Close()
|
||||
fmt.Fprintf(os.Stderr, "embedder: %s (dim=%d)\n", emb.ID(), emb.Dim())
|
||||
|
||||
// Compute frozen holdout / dev pool split.
|
||||
_, devPool, holdoutHash := semantic.FrozenHoldoutSplit(exs)
|
||||
fmt.Fprintf(os.Stderr, "frozen holdout: hash=%s, dev pool: %d examples\n", holdoutHash, len(devPool))
|
||||
|
||||
// Compute grouped CV folds on dev pool only.
|
||||
cvFolds := semantic.GroupedCVFolds(devPool, *folds)
|
||||
fmt.Fprintf(os.Stderr, "cv folds: %d\n", len(cvFolds))
|
||||
|
||||
// Build a lookup: source_id → cv_fold (from dev pool only).
|
||||
foldLookup := make(map[string]int)
|
||||
for _, f := range cvFolds {
|
||||
for _, e := range f.Eval {
|
||||
foldLookup[e.SourceID] = f.Fold
|
||||
}
|
||||
}
|
||||
|
||||
// Build dev set membership lookup.
|
||||
_, devSet, _ := semantic.FrozenHoldoutSplit(exs)
|
||||
devIDs := make(map[string]bool)
|
||||
for _, e := range devSet {
|
||||
devIDs[e.SourceID] = true
|
||||
}
|
||||
|
||||
// Embed all examples.
|
||||
ctx := context.Background()
|
||||
var cached []CachedRow
|
||||
foldComp := make(map[int]*FoldStats)
|
||||
for i := 0; i < *folds; i++ {
|
||||
foldComp[i] = &FoldStats{Routes: make(map[string]int)}
|
||||
}
|
||||
routeCounts := make(map[string]int)
|
||||
sourceCounts := make(map[string]int)
|
||||
fpCount, resCount := 0, 0
|
||||
|
||||
for i, e := range exs {
|
||||
textHash := sha256.Sum256([]byte(e.Text))
|
||||
embedding, err := emb.EmbedQuery(ctx, e.Text)
|
||||
if err != nil {
|
||||
log.Fatalf("embed row %d (%s): %v", i, e.SourceID, err)
|
||||
}
|
||||
|
||||
inDev := devIDs[e.SourceID]
|
||||
fold := -1
|
||||
if inDev {
|
||||
if f, ok := foldLookup[e.SourceID]; ok {
|
||||
fold = f
|
||||
}
|
||||
}
|
||||
|
||||
isFrozen := !inDev
|
||||
|
||||
cr := CachedRow{
|
||||
Text: e.Text,
|
||||
Route: string(e.Route),
|
||||
Source: e.Source,
|
||||
SourceID: e.SourceID,
|
||||
SplitGroup: e.SplitGroup,
|
||||
Tags: e.Tags,
|
||||
FastPathResolved: e.FastPathResolved,
|
||||
RouterResidual: e.RouterResidual,
|
||||
TextHash: hex.EncodeToString(textHash[:]),
|
||||
Embedding: embedding,
|
||||
EmbedderID: emb.ID(),
|
||||
FrozenHoldout: isFrozen,
|
||||
CVFold: fold,
|
||||
DevPool: inDev,
|
||||
FamilyID: e.SplitGroup,
|
||||
}
|
||||
cached = append(cached, cr)
|
||||
|
||||
routeCounts[cr.Route]++
|
||||
sourceCounts[cr.Source]++
|
||||
if e.FastPathResolved {
|
||||
fpCount++
|
||||
} else {
|
||||
resCount++
|
||||
}
|
||||
|
||||
if inDev && fold >= 0 {
|
||||
foldComp[fold].EvalCount++
|
||||
foldComp[fold].Routes[cr.Route]++
|
||||
}
|
||||
}
|
||||
|
||||
// Compute train counts per fold.
|
||||
for i := 0; i < *folds; i++ {
|
||||
foldComp[i].TrainCount = len(devPool) - foldComp[i].EvalCount
|
||||
}
|
||||
|
||||
// Sort route counts for deterministic output.
|
||||
sortedRoutes := make([]string, 0, len(routeCounts))
|
||||
for r := range routeCounts {
|
||||
sortedRoutes = append(sortedRoutes, r)
|
||||
}
|
||||
sort.Strings(sortedRoutes)
|
||||
sortedRouteCounts := make(map[string]int)
|
||||
for _, r := range sortedRoutes {
|
||||
sortedRouteCounts[r] = routeCounts[r]
|
||||
}
|
||||
|
||||
// Build fold stats with sorted keys.
|
||||
finalFoldComp := make(map[int]FoldStats)
|
||||
for i := 0; i < *folds; i++ {
|
||||
finalFoldComp[i] = *foldComp[i]
|
||||
}
|
||||
|
||||
meta := ExperimentMeta{
|
||||
EmbedderID: emb.ID(),
|
||||
ModelPath: modelPath,
|
||||
TokenizerPath: tokPath,
|
||||
Dimension: emb.Dim(),
|
||||
Pooling: "mean-pool + L2-normalize",
|
||||
Normalization: "L2",
|
||||
InputTemplate: "query: <text>",
|
||||
TotalExamples: len(exs),
|
||||
FrozenCount: len(exs) - len(devPool),
|
||||
DevCount: len(devPool),
|
||||
CVFolds: *folds,
|
||||
FoldComposition: finalFoldComp,
|
||||
RouteCounts: sortedRouteCounts,
|
||||
SourceCounts: sourceCounts,
|
||||
FastPathCount: fpCount,
|
||||
ResidualCount: resCount,
|
||||
HoldoutHash: holdoutHash,
|
||||
}
|
||||
|
||||
// Output.
|
||||
output := map[string]any{
|
||||
"meta": meta,
|
||||
"examples": cached,
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(output, "", " ")
|
||||
if err != nil {
|
||||
log.Fatalf("marshal: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(*outPath, data, 0644); err != nil {
|
||||
log.Fatalf("write %s: %v", *outPath, err)
|
||||
}
|
||||
|
||||
// Print summary.
|
||||
fmt.Fprintf(os.Stderr, "\n=== experiment metadata ===\n")
|
||||
fmt.Fprintf(os.Stderr, "embedder: %s\n", meta.EmbedderID)
|
||||
fmt.Fprintf(os.Stderr, "dimension: %d\n", meta.Dimension)
|
||||
fmt.Fprintf(os.Stderr, "total: %d frozen: %d dev: %d\n", meta.TotalExamples, meta.FrozenCount, meta.DevCount)
|
||||
fmt.Fprintf(os.Stderr, "fast-path: %d residual: %d\n", meta.FastPathCount, meta.ResidualCount)
|
||||
fmt.Fprintf(os.Stderr, "routes: %s\n", formatMap(sortedRouteCounts))
|
||||
fmt.Fprintf(os.Stderr, "fold composition:\n")
|
||||
for i := 0; i < *folds; i++ {
|
||||
fs := finalFoldComp[i]
|
||||
fmt.Fprintf(os.Stderr, " fold %d: eval=%d train=%d routes=%s\n",
|
||||
i, fs.EvalCount, fs.TrainCount, formatMap(fs.Routes))
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "output: %s (%d bytes)\n", *outPath, len(data))
|
||||
}
|
||||
|
||||
func formatMap(m map[string]int) string {
|
||||
var parts []string
|
||||
for k, v := range m {
|
||||
parts = append(parts, fmt.Sprintf("%s=%d", k, v))
|
||||
}
|
||||
sort.Strings(parts)
|
||||
return "{" + strings.Join(parts, ", ") + "}"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,665 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 18: Sparse Lexical Action/Non-Action Gate
|
||||
================================================
|
||||
|
||||
Answer: can Maven reliably distinguish executable requests from semantically
|
||||
similar non-actions using lexical/local-order features alone?
|
||||
|
||||
Representations under test (all frozen-population, no e5):
|
||||
A. word 1-2 grams, TF-IDF
|
||||
B. character 3-5 grams, TF-IDF (Unicode, no transliteration)
|
||||
C. [word ; char] combined TF-IDF
|
||||
|
||||
Population reused from slices 15-17: development corpus v2 (dev_pool), router
|
||||
labels, SplitGroup, cv_fold, tags. The e5 embedding vectors are ignored.
|
||||
"""
|
||||
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import unicodedata
|
||||
import warnings
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
import numpy as np
|
||||
from sklearn.exceptions import ConvergenceWarning
|
||||
from sklearn.feature_extraction.text import TfidfVectorizer
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.metrics import (
|
||||
accuracy_score,
|
||||
average_precision_score,
|
||||
confusion_matrix,
|
||||
f1_score,
|
||||
precision_recall_fscore_support,
|
||||
roc_auc_score,
|
||||
)
|
||||
from sklearn.pipeline import make_pipeline
|
||||
from scipy import sparse
|
||||
|
||||
warnings.filterwarnings("ignore", category=ConvergenceWarning)
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
|
||||
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
|
||||
# Present generator/contrast families in the v2 dev pool (for leave-family-out)
|
||||
PRESENT_FAMILIES = [
|
||||
"polite_request",
|
||||
"modal_request",
|
||||
"first_person_request",
|
||||
"reordered_target",
|
||||
"capability_question",
|
||||
"question",
|
||||
]
|
||||
|
||||
FAMILY_ALIASES = {
|
||||
"direct_imperative": "direct_imperative",
|
||||
"polite_request": "polite_request",
|
||||
"modal_request": "modal_request",
|
||||
"first_person_request": "first_person_request",
|
||||
"reordered_target": "reordered_target",
|
||||
"capability_question": "capability_question",
|
||||
"question": "question",
|
||||
}
|
||||
|
||||
|
||||
# ─── NormalizeMatchText (replicated from internal/router/matchtext.go) ─────
|
||||
|
||||
def normalize_match_text(s: str) -> str:
|
||||
"""NFKC → lowercase → collapse Unicode whitespace. Keeps punctuation, ё."""
|
||||
out = unicodedata.normalize("NFKC", s).strip().lower()
|
||||
out = re.sub(r"\s+", " ", out)
|
||||
return out
|
||||
|
||||
|
||||
# ─── Data Loading ───────────────────────────────────────────────────────────
|
||||
|
||||
def load_data():
|
||||
with open(EMBEDDING_PATH) as f:
|
||||
data = json.load(f)
|
||||
return data["meta"], data["examples"]
|
||||
|
||||
|
||||
def filter_dev_pool(examples):
|
||||
return [e for e in examples if e["dev_pool"]]
|
||||
|
||||
|
||||
def residual_only(examples):
|
||||
return [e for e in examples if not e["fast_path_resolved"]]
|
||||
|
||||
|
||||
def fmt_pct(v, d=1):
|
||||
return f"{100*v:.{d}f}%"
|
||||
|
||||
|
||||
def ff(v, d=3):
|
||||
return f"{v:.{d}f}"
|
||||
|
||||
|
||||
def strip_punct(text: str) -> str:
|
||||
"""Remove all punctuation (shared with slice 16/17 apply_voice_stress)."""
|
||||
t = re.sub(r"[?.!,;:]+$", "", text.strip())
|
||||
t = re.sub(r"[^\w\s]", "", t)
|
||||
t = t.lower()
|
||||
t = re.sub(r"\s+", " ", t).strip()
|
||||
return t
|
||||
|
||||
|
||||
# ─── Feature Builders ───────────────────────────────────────────────────────
|
||||
|
||||
def build_features(texts, kind):
|
||||
"""Build a TF-IDF matrix for the given representation kind.
|
||||
kind in {'word','char','both'}. Returns (X_sparse, vectorizer)."""
|
||||
if kind == "word":
|
||||
vec = TfidfVectorizer(
|
||||
ngram_range=(1, 2), analyzer="word",
|
||||
strip_accents=None, lowercase=False,
|
||||
min_df=2, sublinear_tf=True,
|
||||
)
|
||||
elif kind == "char":
|
||||
# preserve case (already lowered) and identity of missing diacritics;
|
||||
# token_pattern null => char analyzer
|
||||
vec = TfidfVectorizer(
|
||||
ngram_range=(3, 5), analyzer="char",
|
||||
strip_accents=None, lowercase=False,
|
||||
min_df=2, sublinear_tf=True,
|
||||
)
|
||||
elif kind == "both":
|
||||
vec_word = TfidfVectorizer(
|
||||
ngram_range=(1, 2), analyzer="word",
|
||||
strip_accents=None, lowercase=False, min_df=2, sublinear_tf=True,
|
||||
)
|
||||
vec_char = TfidfVectorizer(
|
||||
ngram_range=(3, 5), analyzer="char",
|
||||
strip_accents=None, lowercase=False, min_df=2, sublinear_tf=True,
|
||||
)
|
||||
Xw = vec_word.fit_transform(texts)
|
||||
Xc = vec_char.fit_transform(texts)
|
||||
X = sparse.hstack([Xw, Xc]).tocsr()
|
||||
return X, ("both", vec_word, vec_char)
|
||||
X = vec.fit_transform(texts)
|
||||
return X, vec
|
||||
|
||||
|
||||
def vocab_size(vectorizer):
|
||||
if isinstance(vectorizer, tuple):
|
||||
_, vw, vc = vectorizer
|
||||
return vw.get_feature_names_out().shape[0] + vc.get_feature_names_out().shape[0]
|
||||
return vectorizer.get_feature_names_out().shape[0]
|
||||
|
||||
|
||||
def transform_texts(texts, vectorizer):
|
||||
"""Apply an already-fitted vectorizer (handles the 2-tuple 'both' case)."""
|
||||
if isinstance(vectorizer, tuple):
|
||||
_, vw, vc = vectorizer
|
||||
Xw = vw.transform(texts)
|
||||
Xc = vc.transform(texts)
|
||||
return sparse.hstack([Xw, Xc]).tocsr()
|
||||
return vectorizer.transform(texts)
|
||||
|
||||
|
||||
# ─── Grouped CV ─────────────────────────────────────────────────────────────
|
||||
|
||||
def run_binary_grouped_cv(X, y, fold_ids, C=1.0):
|
||||
"""Grouped CV for binary action vs not_action. Returns OOF rows + fold metrics."""
|
||||
yb = np.array([1 if t == "action" else 0 for t in y])
|
||||
fold_ids = np.asarray(fold_ids)
|
||||
unique_folds = sorted(set(fold_ids.tolist()))
|
||||
oof_rows = []
|
||||
fold_metrics = []
|
||||
|
||||
for test_fold in unique_folds:
|
||||
tr = fold_ids != test_fold
|
||||
te = fold_ids == test_fold
|
||||
clf = LogisticRegression(C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
proba = clf.predict_proba(X[te])[:, 1]
|
||||
pred = (proba >= 0.5).astype(int)
|
||||
yt = yb[te]
|
||||
|
||||
fp = int(((yt == 0) & (pred == 1)).sum())
|
||||
fn = int(((yt == 1) & (pred == 0)).sum())
|
||||
tp = int(((yt == 1) & (pred == 1)).sum())
|
||||
tn = int(((yt == 0) & (pred == 0)).sum())
|
||||
|
||||
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
acc = accuracy_score(yt, pred)
|
||||
prec = tp / max(tp + fp, 1)
|
||||
rec = tp / max(tp + fn, 1)
|
||||
|
||||
fold_metrics.append({
|
||||
"fold": int(test_fold), "n": int(len(yt)),
|
||||
"roc_auc": roc, "pr_auc": pr,
|
||||
"action_precision": prec, "action_recall": rec,
|
||||
"fp": fp, "fn": fn, "tp": tp, "tn": tn,
|
||||
"acc": acc,
|
||||
})
|
||||
te_idx = np.where(te)[0]
|
||||
for i in range(len(yt)):
|
||||
oof_rows.append({
|
||||
"fold": int(test_fold),
|
||||
"proba": float(proba[i]),
|
||||
"pred": int(pred[i]),
|
||||
"true": int(yt[i]),
|
||||
})
|
||||
|
||||
return oof_rows, fold_metrics
|
||||
|
||||
|
||||
# ─── Metrics from OOF ───────────────────────────────────────────────────────
|
||||
|
||||
def binary_metrics_from_oof(oof_rows):
|
||||
yt = np.array([r["true"] for r in oof_rows])
|
||||
yp = np.array([r["pred"] for r in oof_rows])
|
||||
proba = np.array([r["proba"] for r in oof_rows])
|
||||
n = len(yt)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
prec = tp / max(tp + fp, 1)
|
||||
rec = tp / max(tp + fn, 1)
|
||||
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
return {
|
||||
"n": n, "tp": tp, "fp": fp, "fn": fn,
|
||||
"action_precision": prec, "action_recall": rec,
|
||||
"fa_rate": fp / max(n, 1),
|
||||
"roc_auc": roc, "pr_auc": pr,
|
||||
}
|
||||
|
||||
|
||||
def threshold_curve(oof_rows, thresholds):
|
||||
yt = np.array([r["true"] for r in oof_rows])
|
||||
proba = np.array([r["proba"] for r in oof_rows])
|
||||
rows = []
|
||||
for thr in thresholds:
|
||||
yp = (proba >= thr).astype(int)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
rows.append({
|
||||
"threshold": round(float(thr), 4),
|
||||
"action_precision": round(tp / max(tp + fp, 1), 4),
|
||||
"action_recall": round(tp / max(tp + fn, 1), 4),
|
||||
"fa_count": fp,
|
||||
"fa_rate": round(fp / max(len(yt), 1), 4),
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
# ─── Leave-Family-Out ───────────────────────────────────────────────────────
|
||||
|
||||
def run_leave_family_out(texts, y, family_per_row, family, kind):
|
||||
"""Train without `family`, evaluate on `family` only."""
|
||||
mask_members = family_per_row == family
|
||||
if mask_members.sum() == 0:
|
||||
return None
|
||||
only_family = (mask_members).astype(bool)
|
||||
train_idx = np.where(~only_family)[0]
|
||||
test_idx = np.where(only_family)[0]
|
||||
X, _ = build_features([texts[i] for i in train_idx], kind)
|
||||
# map test rows onto the full vocabulary
|
||||
tr_texts = [texts[i] for i in train_idx]
|
||||
te_texts = [texts[i] for i in test_idx]
|
||||
all_texts = tr_texts + te_texts
|
||||
Xall, _ = build_features(all_texts, kind)
|
||||
Xtr = Xall[:len(tr_texts)]
|
||||
Xte = Xall[len(tr_texts):]
|
||||
ytr = np.array([1 if y[i] == "action" else 0 for i in train_idx])
|
||||
yte = np.array([1 if y[i] == "action" else 0 for i in test_idx])
|
||||
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(Xtr, ytr)
|
||||
pred = clf.predict(Xte)
|
||||
tp = int(((yte == 1) & (pred == 1)).sum())
|
||||
fp = int(((yte == 0) & (pred == 1)).sum())
|
||||
fn = int(((yte == 1) & (pred == 0)).sum())
|
||||
prec = tp / max(tp + fp, 1)
|
||||
rec = tp / max(tp + fn, 1)
|
||||
acc = accuracy_score(yte, pred)
|
||||
return {
|
||||
"family": family, "rows": int(len(yte)),
|
||||
"action_precision": prec, "action_recall": rec,
|
||||
"fa_count": fp, "acc": acc,
|
||||
"yticks": f"pos={int(yte.sum())} neg={int((yte==0).sum())}",
|
||||
}
|
||||
|
||||
|
||||
# ─── Paired Action/Capability Test ──────────────────────────────────────────
|
||||
|
||||
def paired_action_capability(texts, y, tags_per_row_by_idx, kind, C=1.0):
|
||||
"""For the best sparse representation, evaluate pairwise ordering."""
|
||||
idx = list(range(len(texts)))
|
||||
emb = build_features(texts, kind)[0]
|
||||
clf = LogisticRegression(C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
yb = np.array([1 if t == "action" else 0 for t in y])
|
||||
# use OOF-style: fit on full then derive? We report the pairwise score test;
|
||||
# to avoid leakage we use grouped OOF proba via grouped cv.
|
||||
return None
|
||||
|
||||
|
||||
# ─── Main ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
meta, examples = load_data()
|
||||
dev = filter_dev_pool(examples)
|
||||
dev_res = residual_only(dev)
|
||||
|
||||
# normalized texts
|
||||
for e in dev:
|
||||
e["n_text"] = normalize_match_text(e["text"])
|
||||
|
||||
texts = [e["n_text"] for e in dev]
|
||||
y = [e["route"] for e in dev]
|
||||
folds = [e["cv_fold"] for e in dev]
|
||||
tags = [set(e.get("tags", [])) for e in dev]
|
||||
split_groups = [e["split_group"] for e in dev]
|
||||
source_ids = [e["source_id"] for e in dev]
|
||||
|
||||
n_action = sum(1 for r in y if r == "action")
|
||||
n_not = len(y) - n_action
|
||||
print(f"Dev pool: {len(dev)} action={n_action} not_action={n_not}")
|
||||
print()
|
||||
|
||||
results = {}
|
||||
|
||||
# ── 1. Build features and run grouped CV for each representation ──────
|
||||
for kind in ["word", "char", "both"]:
|
||||
print(f"\n=== {kind} TF-IDF ===")
|
||||
t0 = time.time()
|
||||
X, vec = build_features(texts, kind)
|
||||
build_t = time.time() - t0
|
||||
vs = vocab_size(vec)
|
||||
print(f" vocab={vs} X.shape={X.shape} build={build_t:.2f}s")
|
||||
|
||||
oof, folds_m = run_binary_grouped_cv(X, y, folds, C=1.0)
|
||||
m = binary_metrics_from_oof(oof)
|
||||
results[kind] = {
|
||||
"vs": vs, "build_t": build_t, "oof": oof, "fold_metrics": folds_m,
|
||||
"metrics": m, "X": X, "vec": vec,
|
||||
"texts": texts, "y": y, "folds": folds,
|
||||
}
|
||||
print(f" ROC={ff(m['roc_auc'])} PR={ff(m['pr_auc'])} P={ff(m['action_precision'])} "
|
||||
f"R={ff(m['action_recall'])} FA={m['fp']} ({fmt_pct(m['fa_rate'])})")
|
||||
|
||||
# fold-level
|
||||
for fm in folds_m:
|
||||
print(f" fold {fm['fold']}: ROC={ff(fm['roc_auc'])} PR={ff(fm['pr_auc'])} "
|
||||
f"P={ff(fm['action_precision'])} R={ff(fm['action_recall'])} "
|
||||
f"FP={fm['fp']} FN={fm['fn']} n={fm['n']}")
|
||||
|
||||
# pick best by PR-AUC
|
||||
best_kind = max(["word", "char", "both"], key=lambda k: results[k]["metrics"]["pr_auc"])
|
||||
print(f"\nBest representation by PR-AUC: {best_kind}")
|
||||
|
||||
# ── 2. Leave-family-out for best kind ─────────────────────────────────
|
||||
print(f"\n=== Leave-family-out ({best_kind}) ===")
|
||||
fam_per_row = []
|
||||
for tg in tags:
|
||||
fam = None
|
||||
# prefer the more specific contrast/request families first (a single
|
||||
# utterance may carry several generator tags, e.g. capability_question
|
||||
# plus direct_imperative). Check the informative ones before the
|
||||
# generic direct_imperative fallback.
|
||||
for f in ["capability_question", "question", "first_person_request",
|
||||
"modal_request", "polite_request", "reordered_target",
|
||||
"direct_imperative"]:
|
||||
if f in tg:
|
||||
fam = f
|
||||
break
|
||||
fam_per_row.append(fam)
|
||||
fam_per_row = np.array(fam_per_row, dtype=object)
|
||||
lfo = {}
|
||||
for fam in PRESENT_FAMILIES:
|
||||
r = run_leave_family_out(texts, y, fam_per_row, fam, best_kind)
|
||||
if r is None:
|
||||
print(f" {fam}: (no rows)")
|
||||
continue
|
||||
lfo[fam] = r
|
||||
print(f" {fam}: rows={r['rows']} ({r['yticks']}) P={ff(r['action_precision'])} "
|
||||
f"R={ff(r['action_recall'])} FA={r['fa_count']} acc={fmt_pct(r['acc'])}")
|
||||
|
||||
# ── 3. Action/capability paired test ─────────────────────────────────
|
||||
print(f"\n=== Paired action/capability test ({best_kind}) ===")
|
||||
# Use grouped-CV OOF proba for ordering (no leakage)
|
||||
oof = results[best_kind]["oof"]
|
||||
# map oof rows back by source_id order
|
||||
# oof rows are appended per fold in dev order; reconstruct
|
||||
# We'll re-embed and get proba via grouped CV with proba recorded per row.
|
||||
# Re-run grouped CV capturing per-row proba aligned to dev indices.
|
||||
X = results[best_kind]["X"]
|
||||
yb = np.array([1 if r == "action" else 0 for r in y])
|
||||
folds_arr = np.array(folds)
|
||||
dev_proba = np.zeros(len(dev))
|
||||
for te_fold in sorted(set(folds)):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
dev_proba[te] = clf.predict_proba(X[te])[:, 1]
|
||||
|
||||
# group action seeds: pair each capability-question row with action rows
|
||||
# sharing the same target object noun and the same act (turn_on/turn_off/…).
|
||||
# Both were generated over a common entity+event inventory, so the object
|
||||
# lexeme is the semantic link between a knowledge question and its
|
||||
# executable sibling.
|
||||
DEVICES = [
|
||||
"свет", "люстру", "люстра", "жалюзи", "вытяжку", "вытяжка",
|
||||
"вентилятор", "кондиционер", "телевизор", "лампу", "лампа",
|
||||
"музыку", "музыка", "плеер", "колонку", "колонки", "чайник",
|
||||
"бойлер", "обогреватель", "пылесос", "пылесосом", "пол",
|
||||
"поливалки", "полив", "арка", "шторы", "штору", "штору",
|
||||
"динамики", "дверь", "двери", "замок", "гараж", "ворота",
|
||||
"кофе", "пасту", "зубы", "крючки", "лаймо", "куртку",
|
||||
"будильник", "таймер", "напоминание", "расписание",
|
||||
]
|
||||
|
||||
def object_nouns(t):
|
||||
found = set()
|
||||
tl = t.lower()
|
||||
for d in DEVICES:
|
||||
# match as standalone word (handle Russian case endings loosely via prefix)
|
||||
if re.search(r"\b" + re.escape(d), tl):
|
||||
found.add(d)
|
||||
return found
|
||||
|
||||
# verb/act family per row from source_id (e.g. ha-light-off -> off)
|
||||
def act_family(src):
|
||||
# pull the 3rd token-ish: ha-light-off => 'off'; ha-light-on => 'on'
|
||||
m = re.search(r"^(\w+)-([a-z_]+)-([a-zA-Z_]+)", src)
|
||||
if m:
|
||||
return f"{m.group(1)}-{m.group(2)}-{m.group(3)}"
|
||||
# generic fallback
|
||||
return src.split("-")[0]
|
||||
|
||||
cap_rows = [i for i in range(len(dev)) if "capability_question" in tags[i]]
|
||||
act_idxs = [i for i in range(len(dev)) if y[i] == "action"]
|
||||
|
||||
# For each capability row, candidate sibling actions: same object noun
|
||||
# AND same act (turn_on vs turn_off), i.e. same domain+object. We accept
|
||||
# any action row sharing an object and matching the on/off sense if present.
|
||||
pairs = []
|
||||
for cidx in cap_rows:
|
||||
c_obj = object_nouns(texts[cidx])
|
||||
if not c_obj:
|
||||
continue
|
||||
# cap rows are kq-cap-<domain>-<n>; domain token after 'kq-cap-'
|
||||
dom = re.search(r"kq-cap-([^-]+)", source_ids[cidx])
|
||||
dom = dom.group(1) if dom else None
|
||||
for aidx in act_idxs:
|
||||
a_obj = object_nouns(texts[aidx])
|
||||
if not (c_obj & a_obj):
|
||||
continue
|
||||
# require same home domain when both carry one
|
||||
a_dom = re.search(r"^([a-z]+)-", source_ids[aidx])
|
||||
a_dom = a_dom.group(1) if a_dom else None
|
||||
if dom and a_dom and dom != a_dom:
|
||||
continue
|
||||
pairs.append((cidx, aidx))
|
||||
|
||||
# keep it bounded: cap row pairs with many actions (one per room); that's fine
|
||||
order_ok = 0
|
||||
margins = []
|
||||
reversed_pairs = []
|
||||
for cidx, aidx in pairs:
|
||||
pc = dev_proba[cidx]
|
||||
pa = dev_proba[aidx]
|
||||
margins.append(pa - pc)
|
||||
if pa > pc:
|
||||
order_ok += 1
|
||||
else:
|
||||
reversed_pairs.append((texts[cidx][:40], pc, texts[aidx][:40], pa))
|
||||
if pairs:
|
||||
order_acc = order_ok / len(pairs)
|
||||
margins_arr = np.array(margins)
|
||||
print(f" pairs={len(pairs)} order_acc={ff(order_acc)} mean_margin={ff(margins_arr.mean())} "
|
||||
f"median_margin={ff(np.median(margins_arr))} reversed={len(reversed_pairs)}")
|
||||
for rev in reversed_pairs[:12]:
|
||||
print(f" REV: cap '{rev[0]}' P={rev[1]:.3f} < act '{rev[2]}' P={rev[3]:.3f}")
|
||||
else:
|
||||
order_acc = None
|
||||
print(" (no matched pairs)")
|
||||
|
||||
paired = {
|
||||
"pairs": len(pairs), "order_acc": order_acc,
|
||||
"mean_margin": float(np.mean(margins)) if margins else None,
|
||||
"median_margin": float(np.median(margins)) if margins else None,
|
||||
"reversed": len(reversed_pairs),
|
||||
}
|
||||
|
||||
# ── 4. Voice stress scoring ───────────────────────────────────────────
|
||||
print(f"\n=== Voice-like stress ({best_kind}) ===")
|
||||
# Build stress variants, embed, score with a model trained on normal text
|
||||
# Train one model on full dev (normal punctuation); score stressed variants.
|
||||
clf_full = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf_full.fit(X, yb)
|
||||
|
||||
stress_texts = []
|
||||
stress_meta = [] # (orig_index, mode)
|
||||
for i in range(len(dev)):
|
||||
orig = texts[i]
|
||||
s_nofinal = re.sub(r"[?.!,;:]+$", "", orig)
|
||||
s_all = strip_punct(dev[i]["text"])
|
||||
if s_nofinal != orig:
|
||||
stress_texts.append(s_nofinal); stress_meta.append((i, "no_final"))
|
||||
if s_all != orig:
|
||||
stress_texts.append(s_all); stress_meta.append((i, "all"))
|
||||
if stress_texts:
|
||||
# reuse the already-fitted best-kind vectorizer to keep the feature space
|
||||
# identical to what the full model was trained on.
|
||||
X_stress = transform_texts(stress_texts, results[best_kind]["vec"])
|
||||
proba_stress = clf_full.predict_proba(X_stress)[:, 1]
|
||||
else:
|
||||
proba_stress = []
|
||||
# classify ability: capability-question false-action rate and modal-action recall
|
||||
stress_by_mode = defaultdict(list)
|
||||
for (i, mode), p in zip(stress_meta, proba_stress):
|
||||
stress_by_mode[mode].append((i, p, y[i], tags[i]))
|
||||
voice = {}
|
||||
for mode, rows in stress_by_mode.items():
|
||||
cap_q = [p for (i, p, lbl, tg) in rows if "capability_question" in tg]
|
||||
cap_fa = sum(1 for p in cap_q if p >= 0.5)
|
||||
mod_act = [p for (i, p, lbl, tg) in rows if lbl == "action" and ("polite_request" in tg or "modal_request" in tg)]
|
||||
mod_rec = sum(1 for p in mod_act if p >= 0.5) / max(len(mod_act), 1)
|
||||
voice[mode] = {
|
||||
"n": len(rows),
|
||||
"cap_q_fa": cap_fa / max(len(cap_q), 1), "cap_q_n": len(cap_q),
|
||||
"modal_action_recall": mod_rec, "modal_n": len(mod_act),
|
||||
}
|
||||
print(f" {mode}: n={len(rows)} cap_q_FA={fmt_pct(voice[mode]['cap_q_fa'])} ({voice[mode]['cap_q_n']}) "
|
||||
f"modal_recall={ff(voice[mode]['modal_action_recall'])} ({voice[mode]['modal_n']})")
|
||||
|
||||
# ── 5. Punctuation ablation ───────────────────────────────────────────
|
||||
print(f"\n=== Punctuation ablation ({best_kind}) ===")
|
||||
texts_stripped = [strip_punct(dev[i]["text"]) for i in range(len(dev))]
|
||||
X_stripped_train, vec_stripped = build_features(texts_stripped, best_kind)
|
||||
clf_stripped = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf_stripped.fit(X_stripped_train, yb)
|
||||
# eval on stripped (in-train) and punctuated (out-of-train) via same vectorizer
|
||||
X_orig = transform_texts(texts, vec_stripped)
|
||||
p_orig = clf_stripped.predict_proba(X_orig)[:, 1]
|
||||
p_strip = clf_stripped.predict_proba(X_stripped_train)[:, 1]
|
||||
def report_ablation(proba, name):
|
||||
yp = (proba >= 0.5).astype(int)
|
||||
tp = int(((yb == 1) & (yp == 1)).sum())
|
||||
fp = int(((yb == 0) & (yp == 1)).sum())
|
||||
fn = int(((yb == 1) & (yp == 0)).sum())
|
||||
prec = tp / max(tp + fp, 1); rec = tp / max(tp + fn, 1)
|
||||
print(f" trained-stripped, eval {name}: P={ff(prec)} R={ff(rec)} FA={fp} ({fmt_pct(fp/len(yb))})")
|
||||
report_ablation(p_orig, "punctuated")
|
||||
report_ablation(p_strip, "stripped")
|
||||
|
||||
# ── 6. Threshold curve for best kind ──────────────────────────────────
|
||||
print(f"\n=== Threshold curve ({best_kind}) ===")
|
||||
oof = results[best_kind]["oof"]
|
||||
thresh = np.arange(0.10, 0.995, 0.015).tolist()
|
||||
tcurve = threshold_curve(oof, thresh)
|
||||
print(f"{'thr':>6} {'P':>6} {'R':>6} {'FA':>5} {'FArate':>8}")
|
||||
for t in tcurve:
|
||||
marker = ""
|
||||
if t["action_precision"] >= 0.95:
|
||||
marker = " ← P>=0.95"
|
||||
print(f"{t['threshold']:>6.3f} {t['action_precision']:>6.3f} {t['action_recall']:>6.3f} "
|
||||
f"{t['fa_count']:>5} {t['fa_rate']:>8.4f}{marker}")
|
||||
p95 = [t for t in tcurve if t["action_precision"] >= 0.95 and t["action_recall"] > 0.01]
|
||||
print(f"\nP>=0.95 region: {len(p95)} points; best recall there = "
|
||||
f"{max((t['action_recall'] for t in p95), default=0.0):.4f}")
|
||||
|
||||
# threshold curve using the (no_leak) dev proba instead of .5-threshold OOF
|
||||
# OOF pred used fixed 0.5; curve above re-derives from proba. Good.
|
||||
|
||||
# ── 7. False-action decomposition ────────────────────────────────────
|
||||
print(f"\n=== False-action decomposition ({best_kind}) ===")
|
||||
# recompute OOF predictions at 0.5 from stored 'pred'
|
||||
fa_by_family = Counter()
|
||||
fa_by_group = Counter()
|
||||
# need oof aligned to source_ids — oof stored without source id; rebuild
|
||||
# Re-do grouped cv capturing source_id + tag + split_group
|
||||
fake_oof = []
|
||||
for te_fold in sorted(set(folds_arr)):
|
||||
tr = folds_arr != te_fold; te = folds_arr == te_fold
|
||||
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
p = clf.predict_proba(X[te])[:, 1]
|
||||
pr = (p >= 0.5).astype(int)
|
||||
te_idx = np.where(te)[0]
|
||||
for k, i in enumerate(te_idx):
|
||||
fake_oof.append({
|
||||
"source_id": source_ids[i], "split_group": split_groups[i],
|
||||
"tags": tags[i], "true": y[i], "proba": float(p[k]),
|
||||
"pred": int(pr[k]), "fold": int(te_fold),
|
||||
})
|
||||
|
||||
def classify_semantic_family(row):
|
||||
tg = row["tags"]
|
||||
if "capability_question" in tg: return "capability_question"
|
||||
if "question" in tg: return "ordinary_question"
|
||||
if "remember" in tg or "note" in tg or "idea" in tg or "free_form" in tg: return "memory_write"
|
||||
if "version" in tg or "system" in tg or "health" in tg or "status" in tg: return "system"
|
||||
if "recall" in tg or "world" in tg or "definition" in tg or "aggregate" in tg: return "knowledge_general"
|
||||
if "greeting" in tg or "goodbye" in tg or "thanks" in tg: return "conversation"
|
||||
if row["true"].startswith("system"): return "system"
|
||||
return "other"
|
||||
|
||||
for r in fake_oof:
|
||||
if r["pred"] == 1 and r["true"] != "action":
|
||||
fam = classify_semantic_family(r)
|
||||
fa_by_family[fam] += 1
|
||||
fa_by_group[r["split_group"]] += 1
|
||||
|
||||
print("By family:")
|
||||
for k, v in fa_by_family.most_common():
|
||||
print(f" {k}: {v}")
|
||||
print("By SplitGroup (top 15):")
|
||||
for k, v in fa_by_group.most_common(15):
|
||||
print(f" {k}: {v}")
|
||||
|
||||
# ── 8. Six-way probe on best representation ──────────────────────────
|
||||
print(f"\n=== Six-way probe ({best_kind}) ===")
|
||||
routes = ["action", "conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
y6 = np.array(y)
|
||||
# fit grouped cv 6-way
|
||||
oof6 = []
|
||||
fold6 = []
|
||||
for te_fold in sorted(set(folds_arr)):
|
||||
tr = folds_arr != te_fold; te = folds_arr == te_fold
|
||||
clf = LogisticRegression(C=1.0, max_iter=3000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], y6[tr])
|
||||
pr = clf.predict(X[te])
|
||||
p6 = clf.predict_proba(X[te])
|
||||
classes = clf.classes_
|
||||
te_idx = np.where(te)[0]
|
||||
for k, i in enumerate(te_idx):
|
||||
oof6.append({
|
||||
"true": y6[i], "pred": pr[k], "proba": {c: float(p6[k][j]) for j, c in enumerate(classes)},
|
||||
"tags": tags[i], "split_group": split_groups[i], "source_id": source_ids[i],
|
||||
})
|
||||
yt6 = [r["true"] for r in oof6]; yp6 = [r["pred"] for r in oof6]
|
||||
acc6 = accuracy_score(yt6, yp6)
|
||||
macro6 = f1_score(yt6, yp6, average="macro", zero_division=0)
|
||||
prec6, rec6, f16, sup6 = precision_recall_fscore_support(yt6, yp6, labels=routes, zero_division=0)
|
||||
fa6 = sum(1 for t, p in zip(yt6, yp6) if t != "action" and p == "action")
|
||||
ap6 = sum(1 for t, p in zip(yt6, yp6) if t == "action" and p == "action") / max(sum(1 for p in yp6 if p == "action"), 1)
|
||||
ar6 = sum(1 for t, p in zip(yt6, yp6) if t == "action" and p == "action") / max(sum(1 for t in yt6 if t == "action"), 1)
|
||||
print(f" acc={fmt_pct(acc6)} macroF1={ff(macro6)} actionP={ff(ap6)} actionR={ff(ar6)} FA={fa6} ({fmt_pct(fa6/len(yt6))})")
|
||||
for i, r in enumerate(routes):
|
||||
print(f" {r:<12} P={ff(prec6[i])} R={ff(rec6[i])} F1={ff(f16[i])} n={int(sup6[i])}")
|
||||
|
||||
# ── 9. Artifact size / runtime ───────────────────────────────────────
|
||||
print(f"\n=== Artifact size / runtime ===")
|
||||
for kind in ["word", "char", "both"]:
|
||||
rr = results[kind]
|
||||
vec = rr["vec"]
|
||||
ncoef = rr["metrics"]["n"]
|
||||
# non-zero coefficients = vocab (TF-IDF), logistic has 1 weight per vocab
|
||||
print(f" {kind}: vocab={rr['vs']} fp32 model bytes={rr['vs']*4} "
|
||||
f"build={rr['build_t']:.3f}s")
|
||||
|
||||
# done
|
||||
print("\nDone.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 19 tokenizers, derived from the development corpus only.
|
||||
|
||||
A. CharVocab — codepoint ids over the dev corpus (deterministic order)
|
||||
B. BpeVocab2048 — byte-level BPE, vocab ~2048, trained on dev corpus only
|
||||
|
||||
Records for §4 of the brief: vocab size, OOV behaviour, serialized tokenizer size.
|
||||
"""
|
||||
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
from tokenizers import Tokenizer
|
||||
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
|
||||
from tokenizers.models import BPE
|
||||
from tokenizers.pre_tokenizers import ByteLevel as ByteLevelPreTokenizer
|
||||
from tokenizers.trainers import BpeTrainer
|
||||
|
||||
|
||||
class CharVocab:
|
||||
"""Codepoint ids from the dev corpus, sorted by codepoint value."""
|
||||
|
||||
def __init__(self, texts):
|
||||
chars = set()
|
||||
for t in texts:
|
||||
chars.update(t)
|
||||
self.id_to_char = sorted(chars)
|
||||
self.char_to_id = {c: i + 1 for i, c in enumerate(self.id_to_char)} # 0 = PAD
|
||||
self.pad = 0
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return len(self.id_to_char) + 1
|
||||
|
||||
def encode(self, text, max_len):
|
||||
ids = [self.char_to_id.get(c, 0) for c in text] # 0 doubles as UNK/OOV
|
||||
return ids[:max_len]
|
||||
|
||||
|
||||
class BpeVocab:
|
||||
"""Byte-level BPE, trained only on the strings it is given."""
|
||||
|
||||
def __init__(self, texts, vocab_size=2048, sep="▁"):
|
||||
self.tok = Tokenizer(BPE())
|
||||
self.tok.pre_tokenizer = ByteLevelPreTokenizer(trim_offsets=False)
|
||||
self.tok.decoder = ByteLevelDecoder()
|
||||
trainer = BpeTrainer(vocab_size=vocab_size, special_tokens=["[PAD]"],
|
||||
show_progress=False)
|
||||
# train on the corpus *strings*, byte-level BPE handles all codepoints
|
||||
self.tok.train_from_iterator(texts, trainer=trainer)
|
||||
self.pad_id = self.tok.token_to_id("[PAD]")
|
||||
self._vocab = self.tok.get_vocab()
|
||||
self._n = len(self._vocab)
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return self._n
|
||||
|
||||
def encode(self, text):
|
||||
return self.tok.encode(text).ids
|
||||
|
||||
def serialized_bytes(self):
|
||||
# measure the serialized tokenizer size on disk
|
||||
import os
|
||||
d = self.tok.to_str()
|
||||
return len(d.encode("utf-8"))
|
||||
|
||||
|
||||
def normalize_match_text(s: str) -> str:
|
||||
"""NFKC → lowercase → collapse whitespace. Punctuation kept."""
|
||||
out = unicodedata.normalize("NFKC", s).strip().lower()
|
||||
out = re.sub(r"\s+", " ", out)
|
||||
return out
|
||||
|
||||
|
||||
def strip_punct(text: str) -> str:
|
||||
"""Remove safe punctuation from an already-normalized text."""
|
||||
t = re.sub(r"[^\w\s]", " ", text)
|
||||
t = re.sub(r"\s+", " ", t).strip()
|
||||
return t
|
||||
@@ -0,0 +1,640 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 19: tiny sequence-sensitive pragmatics specialists (binary action gate)
|
||||
=============================================================================
|
||||
|
||||
A narrow binary specialist: executable request vs semantically similar
|
||||
non-executable utterance, generalizing across surface-generator families.
|
||||
|
||||
Commands
|
||||
grouped — grouped semantic CV (existing cv_fold), all arch/sizes, saves
|
||||
per-config per-fold model checkpoints + OOF proba per variant
|
||||
metrics — aggregate saved grouped-CV results into the report tables
|
||||
(binary metrics, threshold curves, pair ordering, stress)
|
||||
lfo — leave-generator-out. All sizes on capability_question; the
|
||||
other present families for the leading config only.
|
||||
e5baseline— frozen-e5 logistic + MLP baselines: grouped, cap-Q LOFO, pair
|
||||
ordering (stress flagged NA — no re-embed on this box)
|
||||
runtime — params / sizes / latency / tokenization for each candidate
|
||||
|
||||
Primary metrics (brief §2): cap-Q LOFO FA, pair ordering acc, pair margin,
|
||||
grouped P/R, voice stress, fold variance. Aggregate accuracy is secondary.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import resource
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from collections import Counter
|
||||
|
||||
import numpy as np
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from slice19_bpe import BpeVocab, CharVocab, normalize_match_text, strip_punct
|
||||
|
||||
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
RESULTS_DIR = "/tmp/mvn-s19"
|
||||
SEED = 42
|
||||
MAX_CHAR = 64
|
||||
MAX_BPE = 24
|
||||
BPE_VOCAB = 2048
|
||||
|
||||
PRESENT_FAMILIES = [
|
||||
"polite_request",
|
||||
"modal_request",
|
||||
"first_person_request",
|
||||
"reordered_target",
|
||||
"capability_question",
|
||||
"question",
|
||||
]
|
||||
ABSENT_FAMILIES = ["negation", "reported_speech", "quotation", "hypothetical"]
|
||||
|
||||
FAMILY_PRIORITY = [
|
||||
"capability_question", "question", "first_person_request",
|
||||
"modal_request", "polite_request", "reordered_target", "direct_imperative",
|
||||
]
|
||||
|
||||
DEVICES = [
|
||||
"свет", "люстру", "люстра", "жалюзи", "вытяжку", "вытяжка",
|
||||
"вентилятор", "кондиционер", "телевизор", "лампу", "лампа",
|
||||
"музыку", "музыка", "плеер", "колонку", "колонки", "чайник",
|
||||
"бойлер", "обогреватель", "пылесос", "пылесосом", "пол",
|
||||
"поливалки", "полив", "арка", "шторы", "штору",
|
||||
"динамики", "дверь", "двери", "замок", "гараж", "ворота",
|
||||
"кофе", "пасту", "зубы", "крючки", "лаймо", "куртку",
|
||||
"будильник", "таймер", "напоминание", "расписание",
|
||||
]
|
||||
|
||||
ARCH_CONFIGS = {
|
||||
"char_cnn": ["tiny", "medium"],
|
||||
"bigru": ["tiny", "medium", "large"],
|
||||
"tiny_transformer": ["small", "medium"],
|
||||
}
|
||||
|
||||
TRAIN_HYPER = {
|
||||
"char_cnn": dict(epochs=20, lr=1e-3, bs=64, clip=None),
|
||||
"bigru": dict(epochs=15, lr=5e-4, bs=64, clip=1.0),
|
||||
"tiny_transformer": dict(epochs=25, lr=5e-4, bs=64, clip=1.0),
|
||||
}
|
||||
|
||||
VARIANTS = ["orig", "nofinal", "strip"]
|
||||
|
||||
|
||||
# ─── data ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def load_dev():
|
||||
with open(EMBEDDING_PATH) as f:
|
||||
data = json.load(f)
|
||||
rows = [e for e in data["examples"] if e["dev_pool"]]
|
||||
out = []
|
||||
for e in rows:
|
||||
n_text = normalize_match_text(e["text"])
|
||||
out.append({
|
||||
"text_orig": n_text,
|
||||
"text_nofinal": re.sub(r"[?.!,;:]+$", "", n_text),
|
||||
"text_strip": strip_punct(n_text),
|
||||
"route": e["route"],
|
||||
"y": 1 if e["route"] == "action" else 0,
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"tags": set(e.get("tags", [])),
|
||||
"source_id": e["source_id"],
|
||||
"emb": np.asarray(e["embedding"], dtype=np.float32),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def family_of(tags):
|
||||
for f in FAMILY_PRIORITY:
|
||||
if f in tags:
|
||||
return f
|
||||
return None
|
||||
|
||||
|
||||
def build_pairs(rows, texts):
|
||||
"""Capability-question vs action-by-shared-object pairs (eval only)."""
|
||||
def object_nouns(t):
|
||||
found = set()
|
||||
tl = t.lower()
|
||||
for d in DEVICES:
|
||||
if re.search(r"\b" + re.escape(d), tl):
|
||||
found.add(d)
|
||||
return found
|
||||
|
||||
pairs = []
|
||||
for cidx, r in enumerate(rows):
|
||||
if "capability_question" not in r["tags"]:
|
||||
continue
|
||||
c_obj = object_nouns(texts[cidx])
|
||||
if not c_obj:
|
||||
continue
|
||||
dom = re.search(r"kq-cap-([^-]+)", r["source_id"])
|
||||
dom = dom.group(1) if dom else None
|
||||
for aidx, ra in enumerate(rows):
|
||||
if ra["y"] != 1:
|
||||
continue
|
||||
a_obj = object_nouns(texts[aidx])
|
||||
if not (c_obj & a_obj):
|
||||
continue
|
||||
a_dom = re.search(r"^([a-z]+)-", ra["source_id"])
|
||||
a_dom = a_dom.group(1) if a_dom else None
|
||||
if dom and a_dom and dom != a_dom:
|
||||
continue
|
||||
pairs.append((cidx, aidx))
|
||||
return pairs
|
||||
|
||||
|
||||
def pair_metrics(pairs, proba):
|
||||
if not pairs:
|
||||
return {"pairs": 0}
|
||||
margins = []
|
||||
ties = 0
|
||||
order = 0
|
||||
rev = 0
|
||||
for cidx, aidx in pairs:
|
||||
pc, pa = proba[cidx], proba[aidx]
|
||||
margins.append(pa - pc)
|
||||
if pa > pc:
|
||||
order += 1
|
||||
elif pa == pc:
|
||||
ties += 1
|
||||
else:
|
||||
rev += 1
|
||||
m = np.array(margins)
|
||||
return {
|
||||
"pairs": len(pairs),
|
||||
"ordering_acc": order / len(pairs),
|
||||
"mean_margin": float(m.mean()),
|
||||
"median_margin": float(np.median(m)),
|
||||
"ties": ties,
|
||||
"reversed": rev,
|
||||
}
|
||||
|
||||
|
||||
# ─── tokenizers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def build_tokenizers(rows):
|
||||
nat_texts = [r["text_orig"] for r in rows]
|
||||
char_vocab = CharVocab(nat_texts)
|
||||
bpe = BpeVocab(nat_texts, vocab_size=BPE_VOCAB)
|
||||
return char_vocab, bpe
|
||||
|
||||
|
||||
def encode_all(rows, tokenizer, kind):
|
||||
"""Return dict variant -> (N, max_len) int64 array."""
|
||||
max_len = MAX_CHAR if kind == "char" else MAX_BPE
|
||||
out = {}
|
||||
for v in VARIANTS:
|
||||
arr = np.zeros((len(rows), max_len), dtype=np.int64)
|
||||
for i, r in enumerate(rows):
|
||||
if kind == "char":
|
||||
ids = tokenizer.encode(r[f"text_{v}"], max_len)
|
||||
arr[i, :len(ids)] = ids
|
||||
else:
|
||||
ids = tokenizer.encode(r[f"text_{v}"])[:max_len]
|
||||
arr[i, :len(ids)] = ids
|
||||
out[v] = arr
|
||||
return out
|
||||
|
||||
|
||||
# ─── training ───────────────────────────────────────────────────────────────
|
||||
|
||||
def build_model(arch, size, vocab_size, max_len):
|
||||
import torch
|
||||
from slice19_models import CharCNN, BiGRU, TinyTransformer
|
||||
if arch == "char_cnn":
|
||||
c = (dict(embed_dim=32, filters=64, widths=[3, 4, 5]) if size == "tiny"
|
||||
else dict(embed_dim=64, filters=160, widths=[2, 3, 4, 5]))
|
||||
return CharCNN(vocab_size, c["embed_dim"], c["filters"], c["widths"])
|
||||
if arch == "bigru":
|
||||
c = (dict(embed_dim=64, hidden=64) if size == "tiny" else
|
||||
(dict(embed_dim=128, hidden=128) if size == "medium" else
|
||||
dict(embed_dim=256, hidden=256)))
|
||||
return BiGRU(vocab_size, c["embed_dim"], c["hidden"])
|
||||
c = (dict(d_model=128, n_layers=2, n_heads=4) if size == "small" else
|
||||
dict(d_model=192, n_layers=4, n_heads=4))
|
||||
return TinyTransformer(vocab_size, c["d_model"], c["n_layers"], c["n_heads"],
|
||||
max_len=max_len)
|
||||
|
||||
|
||||
def train_binary(X, y, arch, size, vocab_size, seed_offset=0, log=False):
|
||||
import torch
|
||||
torch.manual_seed(SEED + seed_offset)
|
||||
np.random.seed(SEED + seed_offset)
|
||||
Xt = torch.from_numpy(X)
|
||||
yt = torch.from_numpy(y.astype(np.float32))
|
||||
model = build_model(arch, size, vocab_size, X.shape[1])
|
||||
h = TRAIN_HYPER[arch]
|
||||
opt = torch.optim.AdamW(model.parameters(), lr=h["lr"], weight_decay=1e-4)
|
||||
lossf = torch.nn.BCEWithLogitsLoss()
|
||||
n = X.shape[0]
|
||||
model.train()
|
||||
t0 = time.time()
|
||||
for epoch in range(h["epochs"]):
|
||||
perm = torch.randperm(n)
|
||||
running = 0.0
|
||||
n_b = 0
|
||||
for start in range(0, n, h["bs"]):
|
||||
idx = perm[start:start + h["bs"]]
|
||||
xb = Xt[idx]
|
||||
if xb.dim() == 1:
|
||||
xb = xb.unsqueeze(0)
|
||||
logits = model(xb)
|
||||
loss = lossf(logits, yt[idx])
|
||||
opt.zero_grad()
|
||||
loss.backward()
|
||||
if h["clip"]:
|
||||
torch.nn.utils.clip_grad_norm_(model.parameters(), h["clip"])
|
||||
opt.step()
|
||||
running += float(loss)
|
||||
n_b += 1
|
||||
if log and (epoch + 1) % 5 == 0:
|
||||
print(f" epoch {epoch+1}/{h['epochs']} loss {running/max(n_b,1):.4f}")
|
||||
return model, time.time() - t0
|
||||
|
||||
|
||||
def predict_proba(model, X, bs=256):
|
||||
import torch
|
||||
model.eval()
|
||||
out = []
|
||||
with torch.no_grad():
|
||||
Xt = torch.from_numpy(X)
|
||||
for start in range(0, X.shape[0], bs):
|
||||
xb = Xt[start:start + bs]
|
||||
if xb.dim() == 1:
|
||||
xb = xb.unsqueeze(0)
|
||||
logits = model(xb)
|
||||
out.append(torch.sigmoid(logits).numpy())
|
||||
return np.concatenate(out)
|
||||
|
||||
|
||||
# ─── metrics helpers ────────────────────────────────────────────────────────
|
||||
|
||||
def binary_metrics(yt, proba, thr=0.5):
|
||||
yp = (proba >= thr).astype(int)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
from sklearn.metrics import roc_auc_score, average_precision_score
|
||||
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
return {
|
||||
"n": int(len(yt)), "tp": tp, "fp": fp, "fn": fn,
|
||||
"P": tp / max(tp + fp, 1), "R": tp / max(tp + fn, 1),
|
||||
"FA": fp, "FA_rate": fp / max(len(yt), 1),
|
||||
"ROC_AUC": float(roc), "PR_AUC": float(pr),
|
||||
}
|
||||
|
||||
|
||||
def threshold_curve(yt, proba, thr_grid):
|
||||
rows = []
|
||||
for thr in thr_grid:
|
||||
yp = (proba >= thr).astype(int)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
P = tp / max(tp + fp, 1)
|
||||
R = tp / max(tp + fn, 1)
|
||||
rows.append({
|
||||
"thr": round(float(thr), 4), "P": round(P, 4), "R": round(R, 4),
|
||||
"FA": fp, "FA_rate": round(fp / max(len(yt), 1), 4),
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def operating_points(rows, pair_proba, pairs):
|
||||
"""Report P>=0.95/0.98/0.99 points with pair separation at threshold."""
|
||||
res = {}
|
||||
for target in (0.95, 0.98, 0.99):
|
||||
pts = [r for r in rows if r["P"] >= target and r["R"] > 0.0]
|
||||
if not pts:
|
||||
res[str(target)] = None
|
||||
continue
|
||||
best = max(pts, key=lambda r: r["R"])
|
||||
# pair separation at that operating point
|
||||
sep = pair_sep_at(best["thr"], pair_proba, pairs)
|
||||
best = dict(best); best["pair_sep"] = round(sep, 4)
|
||||
res[str(target)] = best
|
||||
return res
|
||||
|
||||
|
||||
def pair_sep_at(thr, pair_proba, pairs):
|
||||
"""Fraction of pairs where action>=thr and cap<thr."""
|
||||
if not pairs:
|
||||
return 0.0
|
||||
ok = 0
|
||||
for cidx, aidx in pairs:
|
||||
if pair_proba[aidx] >= thr and pair_proba[cidx] < thr:
|
||||
ok += 1
|
||||
return ok / len(pairs)
|
||||
|
||||
|
||||
def fold_variance(fold_rows):
|
||||
return {
|
||||
"folds": [
|
||||
{
|
||||
"fold": fr["fold"],
|
||||
"ROC_AUC": fr["ROC_AUC"], "PR_AUC": fr["PR_AUC"],
|
||||
"P": fr["P"], "R": fr["R"], "FA": fr["FA"], "n": fr["n"],
|
||||
}
|
||||
for fr in fold_rows
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
# ─── subcommands ────────────────────────────────────────────────────────────
|
||||
|
||||
def _result_path():
|
||||
os.makedirs(RESULTS_DIR, exist_ok=True)
|
||||
return RESULTS_DIR
|
||||
|
||||
|
||||
def cmd_grouped(args):
|
||||
rows = load_dev()
|
||||
char_vocab, bpe = build_tokenizers(rows)
|
||||
y = np.array([r["y"] for r in rows])
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
X_char = encode_all(rows, char_vocab, "char")
|
||||
X_bpe = encode_all(rows, bpe, "bpe")
|
||||
tokenizers = {"char": char_vocab, "bpe": bpe}
|
||||
|
||||
os.makedirs(RESULTS_DIR, exist_ok=True)
|
||||
# save tokenizer metadata for reproducibility
|
||||
meta = {
|
||||
"char_vocab": char_vocab.size,
|
||||
"char_vocab_sample": char_vocab.id_to_char[:50],
|
||||
"bpe_vocab": bpe.size,
|
||||
"bpe_serialized_bytes": bpe.serialized_bytes(),
|
||||
"max_char": MAX_CHAR, "max_bpe": MAX_BPE,
|
||||
"n": len(rows),
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "corpus_meta.json"), "w") as f:
|
||||
json.dump(meta, f)
|
||||
|
||||
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
kind = kind_of[arch]
|
||||
X = X_char if kind == "char" else X_bpe
|
||||
for size in sizes:
|
||||
name = f"{arch}_{size}"
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "models"), exist_ok=True)
|
||||
probs = {v: np.zeros(len(rows)) for v in VARIANTS}
|
||||
fold_rows = []
|
||||
for fold in range(5):
|
||||
tr = folds != fold
|
||||
te = folds == fold
|
||||
Xtr = X["strip"][tr]
|
||||
model, secs = train_binary(Xtr, y[tr], arch, size,
|
||||
vocab_size=tokenizers[kind].size,
|
||||
seed_offset=fold, log=args.verbose)
|
||||
torch_models_dir = os.path.join(RESULTS_DIR, "models")
|
||||
import torch
|
||||
torch.save(model.state_dict(), os.path.join(torch_models_dir, f"{name}_fold{fold}.pt"))
|
||||
for v in VARIANTS:
|
||||
probs[v][te] = predict_proba(model, X[v][te])
|
||||
fold_m = binary_metrics(y[te], probs["strip"][te])
|
||||
fold_m["fold"] = fold
|
||||
fold_rows.append(fold_m)
|
||||
print(f" {name} fold {fold}: ROC={fold_m['ROC_AUC']:.3f} "
|
||||
f"PR={fold_m['PR_AUC']:.3f} P={fold_m['P']:.3f} R={fold_m['R']:.3f} "
|
||||
f"FA={fold_m['FA']} n={fold_m['n']} ({secs:.1f}s)")
|
||||
np.savez(os.path.join(RESULTS_DIR, f"{name}_probs.npz"),
|
||||
var_orig=probs["orig"], var_nofinal=probs["nofinal"],
|
||||
var_strip=probs["strip"])
|
||||
summary = binary_metrics(y, probs["strip"])
|
||||
print(f" {name} OOF: ROC={summary['ROC_AUC']:.3f} PR={summary['PR_AUC']:.3f} "
|
||||
f"P={summary['P']:.3f} R={summary['R']:.3f} FA={summary['FA']}")
|
||||
print("grouped done")
|
||||
|
||||
|
||||
def cmd_metrics(args):
|
||||
rows = load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
pairs = build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
print(f"pairs={len(pairs)}")
|
||||
|
||||
out = {}
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
for size in sizes:
|
||||
name = f"{arch}_{size}"
|
||||
fp = os.path.join(RESULTS_DIR, f"{name}_probs.npz")
|
||||
if not os.path.exists(fp):
|
||||
continue
|
||||
z = np.load(fp)
|
||||
entry = {"name": name, "arch": arch, "size": size}
|
||||
# OOF binary on strip variant (primary training input)
|
||||
entry["strip"] = binary_metrics(y, z["var_strip"])
|
||||
entry["pairs"] = {}
|
||||
entry["pairs"]["strip"] = pair_metrics(pairs, z["var_strip"])
|
||||
entry["pairs"]["orig"] = pair_metrics(pairs, z["var_orig"])
|
||||
entry["pairs"]["nofinal"] = pair_metrics(pairs, z["var_nofinal"])
|
||||
# stress: same OOF models, per-variant metrics
|
||||
entry["stress"] = {}
|
||||
for v in VARIANTS:
|
||||
p = z[f"var_{v}"]
|
||||
entry["stress"][v] = {
|
||||
"all_FA": binary_metrics(y, p)["FA_rate"],
|
||||
"capQ_FA": capq_fa(tags, y, p),
|
||||
"modal_recall": modal_recall(tags, y, p),
|
||||
}
|
||||
# threshold curve + operating points on strip
|
||||
entry["curve"] = threshold_curve(y, z["var_strip"],
|
||||
np.arange(0.30, 1.0, 0.02))
|
||||
entry["ops"] = operating_points(entry["curve"], z["var_strip"], pairs)
|
||||
out[name] = entry
|
||||
|
||||
with open(os.path.join(RESULTS_DIR, "metrics.json"), "w") as f:
|
||||
json.dump(out, f, indent=2, default=str)
|
||||
print(json.dumps(out, indent=2, default=str))
|
||||
|
||||
|
||||
def capq_fa(tags, y, proba):
|
||||
mask = np.array(["capability_question" in t for t in tags])
|
||||
if mask.sum() == 0:
|
||||
return 0.0
|
||||
sub = proba[mask]
|
||||
return float((sub >= 0.5).sum() / mask.sum())
|
||||
|
||||
|
||||
def modal_recall(tags, y, proba):
|
||||
mask = np.array([
|
||||
(y[i] == 1 and ("polite_request" in tags[i] or "modal_request" in tags[i]))
|
||||
for i in range(len(y))
|
||||
])
|
||||
if mask.sum() == 0:
|
||||
return 0.0
|
||||
sub = proba[mask]
|
||||
return float((sub >= 0.5).sum() / mask.sum())
|
||||
|
||||
|
||||
def cmd_lfo(args):
|
||||
rows = load_dev()
|
||||
char_vocab, bpe = build_tokenizers(rows)
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
X_char = encode_all(rows, char_vocab, "char")["strip"]
|
||||
X_bpe = encode_all(rows, bpe, "bpe")["strip"]
|
||||
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
|
||||
|
||||
results = {}
|
||||
tokenizers = {"char": char_vocab, "bpe": bpe}
|
||||
# capability_question LOFO for every config/size (the critical case)
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
X = X_char if kind_of[arch] == "char" else X_bpe
|
||||
src_idx = np.array(["capability_question" not in t for t in tags])
|
||||
tgt_idx = np.array(["capability_question" in t for t in tags])
|
||||
for size in sizes:
|
||||
model, _ = train_binary(X[src_idx], y[src_idx], arch, size,
|
||||
vocab_size=tokenizers[kind_of[arch]].size,
|
||||
seed_offset=17)
|
||||
p = predict_proba(model, X[tgt_idx])
|
||||
m = binary_metrics(y[tgt_idx], p)
|
||||
results[f"{arch}_{size}:capability_question"] = m
|
||||
print(f"LFO ability {arch}_{size}: cap rows={m['n']} "
|
||||
f"pos={int(y[tgt_idx].sum())} FA={m['FA']} FA_rate={m['FA_rate']:.3f} "
|
||||
f"P={m['P']:.3f} R={m['R']:.3f} acc={1-m['FA_rate']:.3f}")
|
||||
|
||||
# full family LOFO for the leading config per architecture
|
||||
leading = {"char_cnn": "char_cnn_medium", "bigru": "bigru_tiny",
|
||||
"tiny_transformer": "tiny_transformer_small"}
|
||||
for arch, name in leading.items():
|
||||
X = X_char if kind_of[arch] == "char" else X_bpe
|
||||
for fam in PRESENT_FAMILIES:
|
||||
src = np.array([fam not in t for t in tags])
|
||||
tgt = np.array([fam in t for t in tags])
|
||||
model, _ = train_binary(X[src], y[src], arch, name.split("_")[-1],
|
||||
vocab_size=tokenizers[kind_of[arch]].size,
|
||||
seed_offset=41)
|
||||
p = predict_proba(model, X[tgt])
|
||||
m = binary_metrics(y[tgt], p)
|
||||
results[f"{name}:{fam}"] = m
|
||||
print(f"LFO {fam}: {name} rows={m['n']} pos={int(y[tgt].sum())} "
|
||||
f"P={m['P']:.3f} R={m['R']:.3f} FA={m['FA']} acc={1-m['FA_rate']:.3f}")
|
||||
|
||||
with open(os.path.join(RESULTS_DIR, "lfo.json"), "w") as f:
|
||||
json.dump(results, f, indent=2, default=str)
|
||||
print("lfo done")
|
||||
|
||||
|
||||
def cmd_e5baseline(args):
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.neural_network import MLPClassifier
|
||||
rows = load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
X = np.vstack([r["emb"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
pairs = build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
|
||||
out = {}
|
||||
for model_name, model, extra in [
|
||||
("e5_linear", LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42), {}),
|
||||
("e5_mlp_h32", MLPClassifier(hidden_layer_sizes=(32,), activation="relu",
|
||||
solver="adam", alpha=0.01, random_state=42,
|
||||
max_iter=800, early_stopping=True,
|
||||
validation_fraction=0.15, n_iter_no_change=10), {}),
|
||||
]:
|
||||
proba = np.zeros(len(rows))
|
||||
for fold in range(5):
|
||||
tr = folds != fold
|
||||
te = folds == fold
|
||||
m2 = type(model)(**{k: v for k, v in model.get_params().items()})
|
||||
m2.fit(X[tr], y[tr])
|
||||
proba[te] = m2.predict_proba(X[te])[:, 1]
|
||||
entry = {
|
||||
"grouped": binary_metrics(y, proba),
|
||||
"pairs": pair_metrics(pairs, proba),
|
||||
}
|
||||
# cap-Q leave-generator-out (train without the family)
|
||||
src = np.array(["capability_question" not in t for t in tags])
|
||||
tgt = np.array(["capability_question" in t for t in tags])
|
||||
m3 = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42) \
|
||||
if model_name == "e5_linear" else \
|
||||
MLPClassifier(hidden_layer_sizes=(32,), alpha=0.01, random_state=42, max_iter=800)
|
||||
m3.fit(X[src], y[src])
|
||||
p3 = m3.predict_proba(X[tgt])[:, 1]
|
||||
entry["capq_lofo"] = binary_metrics(y[tgt], p3)
|
||||
entry["stress"] = "NA (no re-embed on this box)"
|
||||
out[model_name] = entry
|
||||
print(f"{model_name}: grouped FA_rate={entry['grouped']['FA_rate']:.4f} "
|
||||
f"PR={entry['grouped']['PR_AUC']:.3f} capQ_LOFO_FA_rate={entry['capq_lofo']['FA_rate']:.4f} "
|
||||
f"pairs={entry['pairs']['ordering_acc']:.3f}")
|
||||
with open(os.path.join(RESULTS_DIR, "e5baseline.json"), "w") as f:
|
||||
json.dump(out, f, indent=2, default=str)
|
||||
print("e5baseline done")
|
||||
|
||||
|
||||
def cmd_runtime(args):
|
||||
import torch
|
||||
rows = load_dev()
|
||||
char_vocab, bpe = build_tokenizers(rows)
|
||||
X_char = encode_all(rows, char_vocab, "char")["strip"]
|
||||
X_bpe = encode_all(rows, bpe, "bpe")["strip"]
|
||||
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
|
||||
report = {}
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
X = X_char if kind_of[arch] == "char" else X_bpe
|
||||
for size in sizes:
|
||||
name = f"{arch}_{size}"
|
||||
model = build_model(arch, size, int(X.max()) + 1, X.shape[1])
|
||||
n_params = sum(p.numel() for p in model.parameters())
|
||||
fp32 = n_params * 4
|
||||
int8 = n_params
|
||||
model.eval()
|
||||
# warmup + latency (batch-1, eval mode)
|
||||
xb = torch.from_numpy(X[:1])
|
||||
with torch.no_grad():
|
||||
for _ in range(20):
|
||||
model(xb)
|
||||
# tokenization latency
|
||||
if kind_of[arch] == "char":
|
||||
t0 = time.perf_counter()
|
||||
for r in rows[:1000]:
|
||||
char_vocab.encode(r["text_strip"], MAX_CHAR)
|
||||
tl = (time.perf_counter() - t0) / 1000
|
||||
else:
|
||||
t0 = time.perf_counter()
|
||||
for r in rows[:1000]:
|
||||
bpe.encode(r["text_strip"])
|
||||
tl = (time.perf_counter() - t0) / 1000
|
||||
lat = []
|
||||
for _ in range(300):
|
||||
t0 = time.perf_counter()
|
||||
model(xb)
|
||||
lat.append(time.perf_counter() - t0)
|
||||
lat = np.array(lat) * 1e6
|
||||
report[name] = {
|
||||
"params": n_params, "fp32_bytes": fp32, "int8_bytes": int8,
|
||||
"latency_us_mean": float(lat.mean()), "latency_us_p50": float(np.median(lat)),
|
||||
"latency_us_p95": float(np.percentile(lat, 95)),
|
||||
"throughput_b1": round(1e6 / float(lat.mean()), 1),
|
||||
"tok_us": round(tl * 1e6, 1),
|
||||
"tokenizer": "char" if kind_of[arch] == "char" else "bpe",
|
||||
}
|
||||
print(f"{name}: {n_params} params fp32={fp32/1024:.0f}KiB "
|
||||
f"lat={lat.mean():.0f}us tok={tl*1e6:.1f}us")
|
||||
with open(os.path.join(RESULTS_DIR, "runtime.json"), "w") as f:
|
||||
json.dump(report, f, indent=2)
|
||||
print("runtime done")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("cmd", choices=["grouped", "metrics", "lfo", "e5baseline", "runtime"])
|
||||
ap.add_argument("--verbose", action="store_true")
|
||||
args = ap.parse_args()
|
||||
t0 = time.time()
|
||||
globals()[f"cmd_{args.cmd}"](args)
|
||||
print(f"elapsed {time.time()-t0:.1f}s")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 19 model zoo: three genuinely sequence-sensitive tiny models, trained
|
||||
from scratch on Maven's narrow binary pragmatics task.
|
||||
|
||||
A. CharCNN — codepoint ids → char embedding → parallel small 1D convs
|
||||
(several kernel widths) → global max-pool → linear head
|
||||
B. BiGRU — subword ids → token embedding → 1-layer BiGRU →
|
||||
maxpool[final] → linear head
|
||||
C. TinyTransformer — subword ids → token embedding + sine position →
|
||||
N self-attention encoder blocks (heads, FFN 4x, PreNorm) →
|
||||
CLS → linear head
|
||||
|
||||
All expose :forward(ids) returning the binary logit, plus .n_params().
|
||||
Deterministic: everything is plain torch ops.
|
||||
"""
|
||||
|
||||
import math
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
class CharCNN(nn.Module):
|
||||
def __init__(self, vocab_size, embed_dim, filters, widths, pad_idx=0, dropout=0.3):
|
||||
super().__init__()
|
||||
self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
|
||||
self.convs = nn.ModuleList([
|
||||
nn.Conv1d(embed_dim, filters, k, padding=(k - 1) // 2)
|
||||
for k in widths
|
||||
])
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
self.head = nn.Linear(filters * len(widths), 1)
|
||||
|
||||
def forward(self, ids):
|
||||
# ids: (B, T)
|
||||
x = self.embed(ids).transpose(1, 2) # (B, D, T)
|
||||
hiddens = [F.relu(conv(x)) for conv in self.convs] # each (B, F, T)
|
||||
pooled = torch.cat([h.max(dim=2).values for h in hiddens], dim=1) # (B, F*W)
|
||||
return self.head(self.dropout(pooled)).squeeze(-1)
|
||||
|
||||
def n_params(self):
|
||||
return sum(p.numel() for p in self.parameters())
|
||||
|
||||
|
||||
class BiGRU(nn.Module):
|
||||
def __init__(self, vocab_size, embed_dim, hidden, pad_idx=0, dropout=0.3):
|
||||
super().__init__()
|
||||
self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
|
||||
self.encoder = nn.GRU(embed_dim, hidden, num_layers=1, bidirectional=True,
|
||||
batch_first=True)
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
self.head = nn.Linear(hidden * 2, 1)
|
||||
|
||||
def forward(self, ids):
|
||||
mask = (ids != 0).float() # (B, T)
|
||||
x = self.embed(ids)
|
||||
lens = mask.sum(dim=1).clamp(min=1).long()
|
||||
x_p = nn.utils.rnn.pack_padded_sequence(x, lens.cpu(), batch_first=True,
|
||||
enforce_sorted=False)
|
||||
out, _ = self.encoder(x_p)
|
||||
out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True,
|
||||
total_length=mask.size(1))
|
||||
out = out * mask.unsqueeze(-1)
|
||||
maxed = out.max(dim=1).values # (B, 2H)
|
||||
return self.head(self.dropout(maxed)).squeeze(-1)
|
||||
|
||||
def n_params(self):
|
||||
return sum(p.numel() for p in self.parameters())
|
||||
|
||||
|
||||
class TinyTransformer(nn.Module):
|
||||
def __init__(self, vocab_size, d_model, n_layers, n_heads, ff_mult=4,
|
||||
max_len=64, pad_idx=0, dropout=0.1):
|
||||
super().__init__()
|
||||
self.d_model = d_model
|
||||
self.embed = nn.Embedding(vocab_size, d_model, padding_idx=pad_idx)
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
self.pos = nn.Parameter(torch.empty(1, max_len, d_model))
|
||||
nn.init.normal_(self.pos, std=0.02)
|
||||
blocks = []
|
||||
for _ in range(n_layers):
|
||||
blocks.append(TransformerBlock(d_model, n_heads, ff_mult, dropout))
|
||||
self.blocks = nn.ModuleList(blocks)
|
||||
self.ln_out = nn.LayerNorm(d_model)
|
||||
self.head = nn.Linear(d_model, 1)
|
||||
|
||||
def forward(self, ids):
|
||||
B, T = ids.shape
|
||||
mask = (ids != 0)
|
||||
x = self.embed(ids) * math.sqrt(self.d_model) + self.pos[:, :T, :]
|
||||
x = self.dropout(x)
|
||||
for blk in self.blocks:
|
||||
x = blk(x, mask)
|
||||
x = self.ln_out(x)
|
||||
pooled = x.masked_fill(~mask.unsqueeze(-1), float("-inf")).max(dim=1).values
|
||||
return self.head(pooled).squeeze(-1)
|
||||
|
||||
def n_params(self):
|
||||
return sum(p.numel() for p in self.parameters())
|
||||
|
||||
|
||||
class TransformerBlock(nn.Module):
|
||||
def __init__(self, d_model, n_heads, ff_mult, dropout):
|
||||
super().__init__()
|
||||
self.attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout,
|
||||
batch_first=True)
|
||||
self.ln1 = nn.LayerNorm(d_model)
|
||||
self.ff = nn.Sequential(
|
||||
nn.Linear(d_model, d_model * ff_mult),
|
||||
nn.GELU(),
|
||||
nn.Linear(d_model * ff_mult, d_model),
|
||||
)
|
||||
self.ln2 = nn.LayerNorm(d_model)
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
|
||||
def forward(self, x, mask):
|
||||
# mask: (B, T) bool of non-pad; MultiheadAttention uses key_padding_mask
|
||||
h = self.ln1(x)
|
||||
h = self.attn(h, h, h, key_padding_mask=~mask,
|
||||
need_weights=False, is_causal=False)[0]
|
||||
x = x + self.dropout(h)
|
||||
h = self.ln2(x)
|
||||
x = x + self.dropout(self.ff(h))
|
||||
return x
|
||||
|
||||
|
||||
# ─── Sizes ladder ───────────────────────────────────────────────────────────
|
||||
|
||||
def make_model(arch, size, char_vocab, bpe_vocab):
|
||||
if arch == "char_cnn":
|
||||
configs = {
|
||||
"tiny": dict(embed_dim=32, filters=64, widths=[3, 4, 5]),
|
||||
"medium": dict(embed_dim=64, filters=160, widths=[2, 3, 4, 5]),
|
||||
}
|
||||
c = configs[size]
|
||||
return CharCNN(char_vocab, c["embed_dim"], c["filters"], c["widths"])
|
||||
if arch == "bigru":
|
||||
configs = {
|
||||
"tiny": dict(embed_dim=64, hidden=64),
|
||||
"medium": dict(embed_dim=128, hidden=128),
|
||||
"large": dict(embed_dim=256, hidden=256),
|
||||
}
|
||||
c = configs[size]
|
||||
return BiGRU(bpe_vocab, c["embed_dim"], c["hidden"])
|
||||
if arch == "tiny_transformer":
|
||||
configs = {
|
||||
"small": dict(d_model=128, n_layers=2, n_heads=4),
|
||||
"medium": dict(d_model=192, n_layers=4, n_heads=4),
|
||||
}
|
||||
c = configs[size]
|
||||
return TinyTransformer(bpe_vocab, c["d_model"], c["n_layers"], c["n_heads"])
|
||||
raise ValueError(arch)
|
||||
|
||||
|
||||
def n_params_of(arch, size, char_vocab, bpe_vocab):
|
||||
return make_model(arch, size, char_vocab, bpe_vocab).n_params()
|
||||
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 20 tokenizer audit (§2 of the brief) and corpus sequence-length
|
||||
statistics (§3). Runs before any training. If the audit shows catastrophic
|
||||
Cyrillic / mixed-identifier loss it is the gate to stop.
|
||||
|
||||
Loads the frozen dev corpus exactly like slice 19 (same normalization),
|
||||
so regime A (natural text) is `text_orig` and regime B (punct-stripped) is
|
||||
`text_strip`.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
import slice19_main as s19
|
||||
|
||||
RESULTS_DIR = "/tmp/mvn-s20"
|
||||
MODEL_DIR = ("/tmp/mvn-s20/hf-cache/models--cointegrated--rubert-tiny/"
|
||||
"snapshots/5441c5ea8026d4f6d7505ec004845409f1259fb1")
|
||||
|
||||
MIXED_CYRILLIC_LATIN = re.compile(r"[а-яёА-ЯЁ]+[a-zA-Z]+|[a-zA-Z]+[а-яёА-ЯЁ]+")
|
||||
HAS_CYRILLIC = re.compile(r"[а-яёА-ЯЁ]")
|
||||
HAS_LATIN = re.compile(r"[a-zA-Z]")
|
||||
NUMERIC = re.compile(r"[0-9]")
|
||||
TOKEN_RE = re.compile(r"[^\W\d_]+", re.UNICODE)
|
||||
|
||||
SAMPLES = [
|
||||
"выключи свет в спальне пожалуйста",
|
||||
"turn off the lights",
|
||||
"перезапусти сервис mavend",
|
||||
"что такое Nexus",
|
||||
"включи телевизор, пожалуйста",
|
||||
"как дела у Мэйвен",
|
||||
"поставь таймер на 5 минут",
|
||||
"кто такой Home Assistant",
|
||||
"открой настройки устройства ha_cam_12",
|
||||
"Покажи статус сервера Proxmox",
|
||||
"аутентифицируй на сайте 2fa.ru",
|
||||
"сообщи погоду завтра в 18:30",
|
||||
]
|
||||
|
||||
|
||||
def is_toolish(word):
|
||||
# Maven sibling service names / HA-like identifiers: mixed case, digits,
|
||||
# underscores, or short Latin words that are not in the vocab as whole
|
||||
# words. Rough heuristic for the fragmentation probe.
|
||||
return bool(re.search(r"[A-Z0-9_/.-]", word))
|
||||
|
||||
|
||||
def main():
|
||||
os.makedirs(RESULTS_DIR, exist_ok=True)
|
||||
tok = AutoTokenizer.from_pretrained(MODEL_DIR)
|
||||
rows = s19.load_dev()
|
||||
texts = {
|
||||
"orig": [r["text_orig"] for r in rows],
|
||||
"nofinal": [r["text_nofinal"] for r in rows],
|
||||
"strip": [r["text_strip"] for r in rows],
|
||||
}
|
||||
vv = tok.vocab_size
|
||||
unk = tok.unk_token_id
|
||||
|
||||
rep = {"model": "cointegrated/rubert-tiny",
|
||||
"sha": "5441c5ea8026d4f6d7505ec004845409f1259fb1",
|
||||
"tokenizer": type(tok).__name__,
|
||||
"vocab_size": vv}
|
||||
|
||||
# per-character stats (natural texts)
|
||||
chars = [len(t) for t in texts["orig"]]
|
||||
rep["char_len"] = {
|
||||
"mean": round(statistics.mean(chars), 2),
|
||||
"p50": int(sorted(chars)[len(chars) // 2]),
|
||||
"p95": sorted(chars)[int(len(chars) * .95)],
|
||||
"p99": sorted(chars)[int(len(chars) * .99)],
|
||||
"max": max(chars),
|
||||
}
|
||||
|
||||
for view in ("orig", "strip"):
|
||||
ids = tok(texts[view], add_special_tokens=True, padding=False,
|
||||
truncation=False)["input_ids"]
|
||||
lens = [len(x) for x in ids]
|
||||
n_tok = sum(lens)
|
||||
n_unk = sum(x.count(unk) for x in ids)
|
||||
n_chars = sum(len(t) for t in texts[view])
|
||||
rep[view] = {
|
||||
"tokens_per_utt_mean": round(n_tok / len(rows), 2),
|
||||
"tokens_per_char": round(n_tok / max(n_chars, 1), 4),
|
||||
"unk_count": n_unk,
|
||||
"unk_rate": round(n_unk / max(n_tok, 1), 5),
|
||||
"seq_len_p50": int(sorted(lens)[len(lens) // 2]),
|
||||
"seq_len_p90": sorted(lens)[int(len(lens) * .90)],
|
||||
"seq_len_p95": sorted(lens)[int(len(lens) * .95)],
|
||||
"seq_len_p99": sorted(lens)[int(len(lens) * .99)],
|
||||
"seq_len_max": max(lens),
|
||||
"above_96": sum(1 for x in lens if x > 96),
|
||||
"above_128": sum(1 for x in lens if x > 128),
|
||||
}
|
||||
rep[view]["p99_plus_margin"] = rep[view]["seq_len_p99"] + 6
|
||||
|
||||
# mixed Cyrillic/Latin behaviour over natural texts
|
||||
mixed_words = []
|
||||
for t in texts["orig"]:
|
||||
for w in t.split():
|
||||
if MIXED_CYRILLIC_LATIN.search(w):
|
||||
mixed_words.append(w)
|
||||
rep["mixed_cyr_lat_rows"] = len({w for w in mixed_words})
|
||||
rep["mixed_cyr_lat_stats"] = {"word_count": len(mixed_words),
|
||||
"unique_words": len(set(mixed_words))}
|
||||
|
||||
# entity/tool-name fragmentation: unique word-like tokens containing a digit
|
||||
# or underscore, or non-trivial Latin, and how many BPE/WordPiece pieces they
|
||||
# split into. Sample the extremes.
|
||||
fragments = []
|
||||
vocab = set(tok.get_vocab().keys())
|
||||
for t in texts["orig"]:
|
||||
# split into "clean" tokens (word chars + _ / digit boundaries)
|
||||
for w in re.findall(r"[A-Za-z0-9_]+\b", t):
|
||||
w2 = re.sub(r"_\b", "", w)
|
||||
if len(w2) < 3 or not is_toolish(w2):
|
||||
continue
|
||||
n_pieces = len(tok.tokenize(w2).replace("##", "_").rstrip())
|
||||
fragments.append((n_pieces, w2))
|
||||
frag = sorted(set(fragments))[-30:]
|
||||
rep["entity_fragment_examples"] = [
|
||||
{"token": w, "pieces": n} for n, w in frag
|
||||
]
|
||||
|
||||
# representative samples: full tokenization
|
||||
rep["samples"] = []
|
||||
for s in SAMPLES:
|
||||
e = tok(s, add_special_tokens=True, padding=False, truncation=False)
|
||||
rep["samples"].append({
|
||||
"text": s,
|
||||
"tokens": tok.convert_ids_to_tokens(e["input_ids"]),
|
||||
"pieces": len(e["input_ids"]),
|
||||
"unk": e["input_ids"].count(unk),
|
||||
})
|
||||
|
||||
with open(os.path.join(RESULTS_DIR, "tokenizer_audit.json"), "w") as f:
|
||||
json.dump(rep, f, indent=2, ensure_ascii=False)
|
||||
print(json.dumps({k: v for k, v in rep.items() if k not in ("samples",)}, indent=2, ensure_ascii=False))
|
||||
print("\n--- samples ---")
|
||||
for s in rep["samples"]:
|
||||
print(f'{s["pieces"]:>3} unk={s["unk"]} {s["text"]:50} ->'
|
||||
f' {" ".join(s["tokens"])}')
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,517 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 20: fine-tune cointegrated/rubert-tiny (11.9M, 3-layer BERT) end-to-end
|
||||
for the binary executable-intent boundary (action vs not_action) on the frozen
|
||||
v2 dev corpus, following the slice 20 brief.
|
||||
|
||||
Rules honoured:
|
||||
- full end-to-end fine-tuning, CLS-pooled native classification head
|
||||
- tokenizer used unchanged (audit in slice20_audit.py)
|
||||
- max length from corpus stats (p99+margin, cap 128): 25 here
|
||||
- narrow search: LR in {1e-5, 2e-5, 5e-5}, <= 6 epochs, early stop on a
|
||||
development (within-fold) split, best checkpoint restored
|
||||
- >= 3 seeds (42/17/7) for every config
|
||||
- grouped 5-fold CV reuse; cap-Q leave-generator-out as primary stress case
|
||||
- two input regimes: A = natural text (orig), B = punctuation-stripped (strip)
|
||||
|
||||
Artifacts under /tmp/mvn-s20/:
|
||||
pre/{regime}_ids.npy, _attn.npy tokenized corpus (all three views)
|
||||
oof/{regime}_{lr}_{seed}_probs.npz OOF probs per view (var_orig/nofinal/strip)
|
||||
oof/{regime}_{lr}_{seed}_metrics.json
|
||||
lfo/{regime}_{lr}_{seed}.json capability-Q LOFO (held-out family)
|
||||
results/summary.json
|
||||
models/{regime}_{lr}_{seed}_fold{i}.pt, lfo_{seed}.pt
|
||||
|
||||
CLI: slice20_pretrained.py {pre, grouped, lfo, metrics, runtime, onnx}
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch import nn
|
||||
from transformers import AutoConfig, AutoTokenizer
|
||||
from transformers import BertForSequenceClassification
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import slice19_main as s19
|
||||
|
||||
RESULTS_DIR = "/tmp/mvn-s20"
|
||||
MODEL_DIR = ("/tmp/mvn-s20/hf-cache/models--cointegrated--rubert-tiny/"
|
||||
"snapshots/5441c5ea8026d4f6d7505ec004845409f1259fb1")
|
||||
|
||||
MODEL_NAME = "cointegrated/rubert-tiny"
|
||||
MODEL_SHA = "5441c5ea8026d4f6d7505ec004845409f1259fb1"
|
||||
|
||||
REGIMES = ["A", "B"]
|
||||
LRS = [1e-5, 2e-5, 5e-5]
|
||||
SEEDS = [42, 17, 7]
|
||||
VIEWS = ["orig", "nofinal", "strip"]
|
||||
MAX_LEN = 25
|
||||
BATCH = 32
|
||||
MAX_EPOCHS = 4
|
||||
EARLY_STOP = 1 # patience in epochs on val PR-AUC
|
||||
VAL_FRACTION = 0.12
|
||||
WEIGHT_DECAY = 0.01
|
||||
|
||||
torch.set_num_threads(4)
|
||||
|
||||
|
||||
# ─── tokenizer / input preparation ──────────────────────────────────────────
|
||||
|
||||
def _load_tokenizer():
|
||||
return AutoTokenizer.from_pretrained(MODEL_DIR)
|
||||
|
||||
|
||||
def tokenize(texts, tok):
|
||||
e = tok(list(texts), add_special_tokens=True, padding="max_length",
|
||||
truncation=True, max_length=MAX_LEN)
|
||||
return np.array(e["input_ids"], np.int64), np.array(e["attention_mask"], np.int64)
|
||||
|
||||
|
||||
def cmd_pre(args):
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "pre"), exist_ok=True)
|
||||
tok = _load_tokenizer()
|
||||
rows = s19.load_dev()
|
||||
# distributed over all variants, all rows, both regimes
|
||||
for regime in REGIMES:
|
||||
train_view = "orig" if regime == "A" else "strip"
|
||||
tsrc = [r[f"text_{train_view}"] for r in rows]
|
||||
ids, attn = tokenize(tsrc, tok)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"), ids)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"), attn)
|
||||
# eval stress views tokenized under the same regime's vocab/format
|
||||
for v in VIEWS:
|
||||
ids_v, attn_v = tokenize([r[f"text_{v}"] for r in rows], tok)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_ids.npy"), ids_v)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_attn.npy"), attn_v)
|
||||
print(f"regime {regime} done, train view={train_view}")
|
||||
|
||||
|
||||
def _make_model():
|
||||
cfg = AutoConfig.from_pretrained(MODEL_DIR)
|
||||
cfg.num_labels = 1 # sine logit, BCEWithLogits — matches slice 19 head
|
||||
model = BertForSequenceClassification.from_pretrained(MODEL_DIR, config=cfg)
|
||||
return model
|
||||
|
||||
|
||||
# ─── training ───────────────────────────────────────────────────────────────
|
||||
|
||||
def _val_split(rows_idx, y, seed):
|
||||
rng = np.random.RandomState(seed)
|
||||
idx = rng.permutation(rows_idx)
|
||||
# stratified: keep the y-ratio in both parts
|
||||
pos = idx[y[idx] == 1]
|
||||
neg = idx[y[idx] == 0]
|
||||
nv_pos = max(1, int(round(len(pos) * VAL_FRACTION)))
|
||||
nv_neg = max(1, int(round(len(neg) * VAL_FRACTION)))
|
||||
v = np.concatenate([pos[:nv_pos], neg[:nv_neg]])
|
||||
t = np.concatenate([pos[nv_pos:], neg[nv_neg:]])
|
||||
return t, v
|
||||
|
||||
|
||||
def train_model(id_arr, attn, y, tr_idx, val_idx, lr, seed, builder=None):
|
||||
"""Fine-tune the full encoder; early-stop on val PR-AUC; return best state."""
|
||||
net = (builder or _make_model)()
|
||||
opt = torch.optim.AdamW([p for p in net.parameters()],
|
||||
lr=lr, weight_decay=WEIGHT_DECAY)
|
||||
lossf = nn.BCEWithLogitsLoss()
|
||||
from sklearn.metrics import average_precision_score
|
||||
tr = torch.from_numpy(np.ascontiguousarray(id_arr[tr_idx]))
|
||||
ta = torch.from_numpy(np.ascontiguousarray(attn[tr_idx]))
|
||||
ty = torch.from_numpy(y[tr_idx].astype(np.float32))
|
||||
va = torch.from_numpy(np.ascontiguousarray(id_arr[val_idx]))
|
||||
vaa = torch.from_numpy(np.ascontiguousarray(attn[val_idx]))
|
||||
vy = y[val_idx]
|
||||
|
||||
best_pr = -1.0
|
||||
best_state = None
|
||||
best_epoch = 0
|
||||
patience = 0
|
||||
n = len(tr_idx)
|
||||
rng = np.random.RandomState(seed * 97 % 2**31)
|
||||
|
||||
for epoch in range(MAX_EPOCHS):
|
||||
net.train()
|
||||
perm = rng.permutation(n)
|
||||
running = 0.0
|
||||
nb = 0
|
||||
for st in range(0, n, BATCH):
|
||||
bidx = torch.from_numpy(perm[st:st + BATCH])
|
||||
logits = net(input_ids=tr[bidx], attention_mask=ta[bidx]).logits.squeeze(-1)
|
||||
loss = lossf(logits, ty[bidx])
|
||||
opt.zero_grad()
|
||||
loss.backward()
|
||||
opt.step()
|
||||
running += float(loss)
|
||||
nb += 1
|
||||
net.eval()
|
||||
with torch.no_grad():
|
||||
pval = torch.sigmoid(net(input_ids=va, attention_mask=vaa).logits.squeeze(-1)).numpy()
|
||||
if len(np.unique(vy)) > 1:
|
||||
pr = average_precision_score(vy, pval)
|
||||
else:
|
||||
pr = 0.0
|
||||
if pr > best_pr:
|
||||
best_pr = pr
|
||||
best_state = {k: v.detach().clone() for k, v in net.state_dict().items()}
|
||||
best_epoch = epoch + 1
|
||||
patience = 0
|
||||
else:
|
||||
patience += 1
|
||||
if patience >= EARLY_STOP:
|
||||
break
|
||||
net.load_state_dict(best_state)
|
||||
return net, best_epoch, best_pr, running / max(nb, 1)
|
||||
|
||||
|
||||
def predict_proba(net, id_arr, attn, idx=None):
|
||||
net.eval()
|
||||
idx = np.arange(len(id_arr)) if idx is None else idx
|
||||
out = []
|
||||
with torch.no_grad():
|
||||
for st in range(0, len(idx), BATCH * 4):
|
||||
bi = idx[st:st + BATCH * 4]
|
||||
iid = torch.from_numpy(np.ascontiguousarray(id_arr[bi]))
|
||||
att = torch.from_numpy(np.ascontiguousarray(attn[bi]))
|
||||
out.append(torch.sigmoid(net(input_ids=iid, attention_mask=att).logits.squeeze(-1)).numpy())
|
||||
return np.concatenate(out)
|
||||
|
||||
|
||||
# ─── grouped CV ─────────────────────────────────────────────────────────────
|
||||
|
||||
def cmd_grouped(args):
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "oof"), exist_ok=True)
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "models"), exist_ok=True)
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
for regime in REGIMES:
|
||||
ids = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"))
|
||||
attn = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"))
|
||||
ev = {v: (np.load(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_ids.npy")),
|
||||
np.load(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_attn.npy")))
|
||||
for v in VIEWS}
|
||||
for lr in LRS:
|
||||
for seed in SEEDS:
|
||||
probs = {v: np.zeros(len(rows)) for v in VIEWS}
|
||||
fold_records = []
|
||||
for fold in range(5):
|
||||
tr = np.where(folds != fold)[0]
|
||||
te = np.where(folds == fold)[0]
|
||||
t_idx, v_idx = _val_split(tr, y, seed + 100 * fold)
|
||||
net, ep, best_pr, _ = train_model(ids, attn, y, t_idx, v_idx, lr, seed + fold)
|
||||
torch.save(net.state_dict(),
|
||||
os.path.join(RESULTS_DIR, "models",
|
||||
f"{regime}_{lr}_{seed}_fold{fold}.pt"))
|
||||
for v in VIEWS:
|
||||
probs[v][te] = predict_proba(net, *ev[v], te)
|
||||
fold_records.append({"fold": fold, "epochs": ep, "val_pr": best_pr})
|
||||
np.savez(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_probs.npz"),
|
||||
var_orig=probs["orig"], var_nofinal=probs["nofinal"],
|
||||
var_strip=probs["strip"])
|
||||
with open(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_metrics.json"), "w") as f:
|
||||
json.dump({"folds": fold_records}, f, indent=2)
|
||||
m = s19.binary_metrics(y, probs["strip"])
|
||||
print(f"[{regime}] lr={lr:.0e} seed={seed} "
|
||||
f"PR={m['PR_AUC']:.3f} P={m['P']:.3f} R={m['R']:.3f} "
|
||||
f"FA={m['FA']} epochs={[fr['epochs'] for fr in fold_records]}",
|
||||
flush=True)
|
||||
print("grouped done")
|
||||
|
||||
|
||||
# ─── cap-Q leave-generator-out ──────────────────────────────────────────────
|
||||
|
||||
def cmd_lfo(args):
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "lfo"), exist_ok=True)
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
src = np.array(["capability_question" not in t for t in tags])
|
||||
tgt = ~src
|
||||
for regime in REGIMES:
|
||||
ids = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"))
|
||||
attn = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"))
|
||||
for lr in LRS:
|
||||
for seed in SEEDS:
|
||||
s_idx = np.where(src)[0]
|
||||
t_idx, v_idx = _val_split(s_idx, y, seed + 7)
|
||||
net, ep, best_pr, _ = train_model(ids, attn, y, t_idx, v_idx, lr, seed)
|
||||
p = predict_proba(net, ids, attn, np.where(tgt)[0])
|
||||
yt = y[tgt]
|
||||
out = {
|
||||
"regime": regime, "lr": lr, "seed": seed,
|
||||
"rows": int(tgt.sum()), "epochs": ep, "val_pr": best_pr,
|
||||
"mean_action_proba": float(np.mean(p)),
|
||||
"max_action_proba": float(np.max(p)),
|
||||
"acc": float(((p >= 0.5) == (yt == 1)).mean()),
|
||||
"FA": int(((p >= 0.5) & (yt == 0)).sum()),
|
||||
"FA_rate": float(((p >= 0.5) & (yt == 0)).mean()),
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "lfo", f"{regime}_{lr}_{seed}.json"), "w") as f:
|
||||
json.dump(out, f, indent=2)
|
||||
print(f"[{regime}] lr={lr:.0e} seed={seed} capQ LOFO "
|
||||
f"acc={out['acc']:.3f} FA_rate={out['FA_rate']:.3f} "
|
||||
f"mean_p={out['mean_action_proba']:.3f} epochs={ep}", flush=True)
|
||||
print("lfo done")
|
||||
|
||||
|
||||
# ─── metrics aggregation ────────────────────────────────────────────────────
|
||||
|
||||
def cmd_metrics(args):
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
pairs = s19.build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
summary = {}
|
||||
for regime in REGIMES:
|
||||
summary[regime] = {}
|
||||
for lr in LRS:
|
||||
per_seed = []
|
||||
for seed in SEEDS:
|
||||
z = np.load(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_probs.npz"))
|
||||
entry = {"seed": seed,
|
||||
"views": {v: {"FA_rate": s19.binary_metrics(y, z[f"var_{v}"])["FA_rate"],
|
||||
"PR": s19.binary_metrics(y, z[f"var_{v}"])["PR_AUC"],
|
||||
"capQ_FA": s19.capq_fa(tags, y, z[f"var_{v}"])}
|
||||
for v in VIEWS},
|
||||
"strip": s19.binary_metrics(y, z["var_strip"]),
|
||||
"pairs": {v: s19.pair_metrics(pairs, z[f"var_{v}"]) for v in VIEWS},
|
||||
"curve": s19.threshold_curve(y, z["var_strip"], np.arange(0.30, 1.0, 0.02)),
|
||||
"ops": s19.operating_points(s19.threshold_curve(
|
||||
y, z["var_strip"], np.arange(0.30, 1.0, 0.02)),
|
||||
z["var_strip"], pairs),
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_metrics.json")) as f:
|
||||
entry["folds"] = json.load(f)["folds"]
|
||||
per_seed.append(entry)
|
||||
# LOFO
|
||||
lfos = []
|
||||
for seed in SEEDS:
|
||||
with open(os.path.join(RESULTS_DIR, "lfo", f"{regime}_{lr}_{seed}.json")) as f:
|
||||
lfos.append(json.load(f))
|
||||
summary[regime][str(lr)] = {"seeds": per_seed, "lfo": lfos}
|
||||
# primary selection: min mean cap-Q LOFO FA_rate across seeds
|
||||
sel = {}
|
||||
for regime in REGIMES:
|
||||
best_lr, best_mean = None, 9e9
|
||||
for lr in LRS:
|
||||
mean_fa = np.mean([l["FA_rate"] for l in summary[regime][str(lr)]["lfo"]])
|
||||
sel[f"{regime}_{lr}"] = round(float(mean_fa), 4)
|
||||
if mean_fa < best_mean:
|
||||
best_lr, best_mean = lr, mean_fa
|
||||
sel[f"{regime}_selected"] = str(best_lr)
|
||||
summary["selection"] = sel
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "results"), exist_ok=True)
|
||||
with open(os.path.join(RESULTS_DIR, "results", "summary.json"), "w") as f:
|
||||
json.dump(summary, f, indent=2, default=str)
|
||||
print(json.dumps(sel))
|
||||
print("metrics done")
|
||||
|
||||
|
||||
def _tokenizer_bytes():
|
||||
return sum(os.path.getsize(os.path.join(MODEL_DIR, f))
|
||||
for f in ["vocab.txt", "tokenizer.json"]
|
||||
if os.path.exists(os.path.join(MODEL_DIR, f)))
|
||||
|
||||
|
||||
# ─── runtime + ONNX ─────────────────────────────────────────────────────────
|
||||
|
||||
def cmd_runtime(args):
|
||||
import time
|
||||
net = _make_model()
|
||||
tok = _load_tokenizer()
|
||||
rows = s19.load_dev()
|
||||
x = [r["text_orig"] for r in rows][:200]
|
||||
e = tok(x, padding="max_length", truncation=True,
|
||||
max_length=MAX_LEN, return_tensors="pt")
|
||||
ids, attn = e["input_ids"], e["attention_mask"]
|
||||
net.eval()
|
||||
with torch.no_grad():
|
||||
# warmup
|
||||
for _ in range(3):
|
||||
net(input_ids=ids[:1], attention_mask=attn[:1])
|
||||
# batch-1 latency
|
||||
lat = []
|
||||
for i in range(200):
|
||||
t0 = time.perf_counter()
|
||||
net(input_ids=ids[i:i + 1], attention_mask=attn[i:i + 1])
|
||||
lat.append((time.perf_counter() - t0) * 1e6)
|
||||
# tokenization latency
|
||||
t0 = time.perf_counter()
|
||||
for i in range(200):
|
||||
tok(x[i])
|
||||
tok_us = (time.perf_counter() - t0) / 200 * 1e6
|
||||
n_params = sum(p.numel() for p in net.parameters())
|
||||
fp32 = n_params * 4
|
||||
rep = {
|
||||
"model": MODEL_NAME, "sha": MODEL_SHA,
|
||||
"params": n_params, "fp32_bytes": fp32,
|
||||
"fp16_bytes": fp32 // 2, "int8_bytes": n_params,
|
||||
"tokenizer_bytes": _tokenizer_bytes(),
|
||||
"latency_us_mean": float(np.mean(lat)),
|
||||
"latency_us_p50": float(np.median(lat)),
|
||||
"latency_us_p95": float(np.percentile(lat, 95)),
|
||||
"max_len": MAX_LEN,
|
||||
"tok_us": round(tok_us, 2),
|
||||
"num_threads": 12,
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "runtime.json"), "w") as f:
|
||||
json.dump(rep, f, indent=2)
|
||||
print(json.dumps(rep, indent=2))
|
||||
print("runtime done")
|
||||
|
||||
|
||||
def cmd_onnx(args):
|
||||
net = _make_model()
|
||||
net.eval()
|
||||
tok = _load_tokenizer()
|
||||
rows = s19.load_dev()
|
||||
try:
|
||||
import torch.onnx
|
||||
dummy = {
|
||||
"input_ids": torch.zeros(1, MAX_LEN, dtype=torch.long),
|
||||
"attention_mask": torch.ones(1, MAX_LEN, dtype=torch.long),
|
||||
}
|
||||
with torch.no_grad():
|
||||
torch.onnx.export(net, (dummy,), os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"),
|
||||
input_names=["input_ids", "attention_mask"],
|
||||
output_names=["logits"], opset_version=14,
|
||||
dynamic_axes={"input_ids": {0: "batch"},
|
||||
"attention_mask": {0: "batch"}})
|
||||
# parity on a fixed sample
|
||||
import numpy as np
|
||||
samp = [(r["text_orig"], r["y"]) for r in rows[:200]]
|
||||
e = tok([s[0] for s in samp], padding="max_length", truncation=True,
|
||||
max_length=MAX_LEN, return_tensors="pt")
|
||||
with torch.no_grad():
|
||||
pt = torch.sigmoid(net(**e).logits.squeeze(-1)).numpy()
|
||||
import onnxruntime as ort
|
||||
so = ort.SessionOptions()
|
||||
so.intra_op_num_threads = 12
|
||||
sess = ort.InferenceSession(os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"),
|
||||
sess_options=so, providers=["CPUExecutionProvider"])
|
||||
on = sess.run(None, {"input_ids": e["input_ids"].numpy(),
|
||||
"attention_mask": e["attention_mask"].numpy()})[0]
|
||||
on = 1 / (1 + np.exp(-on).squeeze(-1))
|
||||
mx = float(np.max(np.abs(pt - on)))
|
||||
size = os.path.getsize(os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"))
|
||||
rep = {"max_logit_diff": mx, "onnx_bytes": size,
|
||||
"parity_n": len(samp), "provider": "CPUExecutionProvider"}
|
||||
with open(os.path.join(RESULTS_DIR, "onnx.json"), "w") as f:
|
||||
json.dump(rep, f, indent=2)
|
||||
print(json.dumps(rep))
|
||||
except Exception as ex:
|
||||
print("onnx export/parity failed:", ex)
|
||||
with open(os.path.join(RESULTS_DIR, "onnx.json"), "w") as f:
|
||||
json.dump({"error": str(ex)}, f, indent=2)
|
||||
print("onnx done")
|
||||
|
||||
|
||||
# ─── optional capacity/pretraining ceiling (brief §13) ──────────────────────
|
||||
# Trigger: tiny1 clearly improved over from-scratch on in-pool but missed the
|
||||
# LOFO boundary. tiny2 is the same 3-layer 312-hidden BERT family; it tests
|
||||
# whether a *newer, larger-vocab* pretraining of the same family generalises
|
||||
# where tiny1 failed — disambiguating "this family is the wrong prior" from a
|
||||
# one-off pretraining. It cannot test capacity (same depth/size).
|
||||
|
||||
MODEL2_DIR = ("/tmp/mvn-s20/hf-tiny2-cache/models--cointegrated--rubert-tiny2/"
|
||||
"snapshots/e8ed3b0c8bbf4fb6984c3de043bf7d2f4e5969ae")
|
||||
MODEL2_SHA = "e8ed3b0c8bbf4fb6984c3de043bf7d2f4e5969ae"
|
||||
CEIL_RESULTS = os.path.join(RESULTS_DIR, "tiny2")
|
||||
|
||||
|
||||
def cmd_ceiling(args):
|
||||
import torch
|
||||
os.makedirs(CEIL_RESULTS, exist_ok=True)
|
||||
os.makedirs(os.path.join(CEIL_RESULTS, "models"), exist_ok=True)
|
||||
tok = AutoTokenizer.from_pretrained(MODEL2_DIR)
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
pairs = s19.build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
# audit: is the tiny2 tokenizer sane on the corpus before anything else
|
||||
n_unk = 0
|
||||
n_tok = 0
|
||||
lens = []
|
||||
for r in rows:
|
||||
e = tok(r["text_orig"])
|
||||
n_unk += e["input_ids"].count(tok.unk_token_id)
|
||||
n_tok += len(e["input_ids"])
|
||||
lens.append(len(e["input_ids"]))
|
||||
audit = {"vocab_size": tok.vocab_size,
|
||||
"unk_count": int(n_unk),
|
||||
"unk_rate": round(n_unk / max(n_tok, 1), 5),
|
||||
"seq_len_p99": sorted(lens)[int(len(lens) * .99)],
|
||||
"seq_len_max": max(lens)}
|
||||
with open(os.path.join(CEIL_RESULTS, "audit.json"), "w") as f:
|
||||
json.dump(audit, f, indent=2)
|
||||
print("tiny2 audit:", audit)
|
||||
|
||||
# tokenize the corpus (regime A only — natural text, the in-pool best)
|
||||
ids_a, attn_a = tokenize([r["text_orig"] for r in rows], tok)
|
||||
ev = {v: tokenize([r[f"text_{v}"] for r in rows], tok) for v in VIEWS}
|
||||
|
||||
def make2():
|
||||
cfg = AutoConfig.from_pretrained(MODEL2_DIR)
|
||||
cfg.num_labels = 1
|
||||
m = BertForSequenceClassification.from_pretrained(MODEL2_DIR, config=cfg)
|
||||
return m
|
||||
|
||||
# cap-Q LOFO, 3 seeds, matching the A@2e-5 tiny1 config
|
||||
lfors = []
|
||||
src = np.where(np.array(["capability_question" not in t for t in tags]))[0]
|
||||
tgt = np.where(np.array(["capability_question" in t for t in tags]))[0]
|
||||
for seed in SEEDS:
|
||||
t_idx, v_idx = _val_split(src, y, seed + 7)
|
||||
net, ep, best_pr, _ = train_model(ids_a, attn_a, y, t_idx, v_idx, 2e-5, seed, builder=make2)
|
||||
p = predict_proba(net, ids_a, attn_a, tgt)
|
||||
yt = y[tgt]
|
||||
lfors.append({"regime": "A(tiny2)", "lr": 2e-5, "seed": seed,
|
||||
"rows": int(len(tgt)), "epochs": ep,
|
||||
"mean_action_proba": float(np.mean(p)),
|
||||
"acc": float(((p >= 0.5) == (yt == 1)).mean()),
|
||||
"FA": int(((p >= 0.5) & (yt == 0)).sum()),
|
||||
"FA_rate": float(((p >= 0.5) & (yt == 0)).mean())})
|
||||
with open(os.path.join(CEIL_RESULTS, "lfo.json"), "w") as f:
|
||||
json.dump(lfors, f, indent=2)
|
||||
print("tiny2 LOFO:", [round(l["FA_rate"], 3) for l in lfors])
|
||||
|
||||
# grouped CV for the same best config + in-pool pairs / capQ
|
||||
oof = {v: np.zeros(len(rows)) for v in VIEWS}
|
||||
for fold in range(5):
|
||||
tr = np.where(folds != fold)[0]
|
||||
te = np.where(folds == fold)[0]
|
||||
t_idx, v_idx = _val_split(tr, y, 42 + 100 * fold)
|
||||
net, _, _, _ = train_model(ids_a, attn_a, y, t_idx, v_idx, 2e-5, 42 + fold, builder=make2)
|
||||
for v in VIEWS:
|
||||
oof[v][te] = predict_proba(net, *ev[v], te)
|
||||
inpool = {"strip": s19.binary_metrics(y, oof["strip"]),
|
||||
"pairs": {v: s19.pair_metrics(pairs, oof[v]) for v in VIEWS},
|
||||
"capQ_inpool": {v: s19.capq_fa(tags, y, oof[v]) for v in VIEWS}}
|
||||
with open(os.path.join(CEIL_RESULTS, "grouped.json"), "w") as f:
|
||||
json.dump(inpool, f, indent=2, default=str)
|
||||
b = inpool["strip"]
|
||||
print(f"tiny2 grouped A@2e-5: PR={b['PR_AUC']:.3f} P={b['P']:.3f} R={b['R']:.3f} "
|
||||
f"FA={b['FA']} pairs_strip={inpool['pairs']['strip']['ordering_acc']:.3f} "
|
||||
f"capQ_strip={inpool['capQ_inpool']['strip']:.3f}", flush=True)
|
||||
print("ceiling done")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("cmd", choices=["pre", "grouped", "lfo", "metrics", "runtime", "onnx", "ceiling"])
|
||||
args = ap.parse_args()
|
||||
t0 = time.time()
|
||||
globals()[f"cmd_{args.cmd}"](args)
|
||||
print(f"elapsed {time.time()-t0:.1f}s")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,95 @@
|
||||
package main
|
||||
|
||||
// Fixture is one brief-specified utterance and the eligibility it must earn.
|
||||
// Fixtures are EXACTLY the slice-21 brief examples plus the corpus-verified
|
||||
// structural discriminators. They are eval material, not corpus additions:
|
||||
// nothing here is inserted into any pool file (brief §2, §18).
|
||||
type Fixture struct {
|
||||
Utterance string
|
||||
Want Eligibility
|
||||
Family string
|
||||
Note string
|
||||
}
|
||||
|
||||
// Fixtures covers every family the brief's §15 list exercises, including the
|
||||
// four that have no dev-pool rows at all (negation, reported_speech,
|
||||
// quotation, hypothetical):
|
||||
//
|
||||
// direct imperative already plural in dev; fixtures pin the canonical
|
||||
// cases together with the politeness-negative modal.
|
||||
// polite request polite imperative + bare "можешь …, пожалуйста".
|
||||
// modal request "можно …", English "can you …, please".
|
||||
// first-person request "я хочу …", "мне нужно …", "надо …".
|
||||
// reordered target "свет выключи, пожалуйста" (target-first command).
|
||||
// capability question ты/умеешь/сможешь/способна + "… ли" permission.
|
||||
// ordinary question "что запущено" — no execution pressure.
|
||||
// negation "не выключай свет" (parser-covered); advisory "не
|
||||
// надо выключать свет".
|
||||
// reported speech past/third-person report verbs over a command.
|
||||
// quotation quoted command, with and without a reporting frame.
|
||||
// hypothetical если-scopes that are not real condition→command.
|
||||
var Fixtures = []Fixture{
|
||||
// negation — direct commands the prosecutor forbids
|
||||
{"не выключай свет", Blocked, "negation", "direct prohibition"},
|
||||
{"не перезапускай сервер", Blocked, "negation", "direct prohibition"},
|
||||
{"не включай nginx", Blocked, "negation", "direct prohibition"},
|
||||
{"не надо выключать свет", Blocked, "negation", "advisory negative"},
|
||||
{"не стоит перезапускать nginx", Blocked, "negation", "advisory negative"},
|
||||
{"не забудь напомнить про свет", Permissive, "negation", "prohibition-parser reminder exemption"},
|
||||
|
||||
// reported speech — reports an order, does not issue one
|
||||
{"он сказал выключить свет", Blocked, "reported_speech", "past report verb + infinitive"},
|
||||
{"она попросила перезапустить nginx", Blocked, "reported_speech", "past report verb + infinitive"},
|
||||
{"мне сказали включить свет", Blocked, "reported_speech", "passive report + infinitive"},
|
||||
{"он написал: «перезапусти nginx»", Blocked, "reported_speech", "report verb + quoted imperative"},
|
||||
{"скажи мне, что он сказал про свет", Permissive, "reported_speech", "request to report, no commanded clause"},
|
||||
{"расскажи про свет", Permissive, "reported_speech", "narrative request, not a reported order"},
|
||||
|
||||
// quotation — quoted text is referenced, not issued
|
||||
{"фраза «выключи свет»", Blocked, "quotation", "reporting noun + quoted imperative"},
|
||||
{"он сказал «выключи свет»", Blocked, "quotation", "report verb + quoted imperative"},
|
||||
{"«выключи свет»", Ambiguous, "quotation", "bare quoted command, no frame"},
|
||||
{"выключи свет", Permissive, "quotation", "unquoted imperative is a live command"},
|
||||
|
||||
// hypothetical
|
||||
{"если выключить свет...", Blocked, "hypothetical", "conditional + infinitive + ellipsis"},
|
||||
{"если бы перезапустить nginx...", Blocked, "hypothetical", "conditional + бы + infinitive"},
|
||||
{"что будет если выключить свет", Blocked, "hypothetical", "question-scoped conditional"},
|
||||
{"если будет дождь, выключи полив", Permissive, "hypothetical", "real condition → imperative"},
|
||||
{"выключи свет если будет дождь", Permissive, "hypothetical", "imperative → real condition"},
|
||||
|
||||
// capability question — blocked even polite
|
||||
{"ты можешь выключить свет?", Blocked, "capability_question", "ты + можешь + ?"},
|
||||
{"ты можешь выключить свет", Blocked, "capability_question", "ты + можешь, no ?"},
|
||||
{"ты можешь выключить свет, пожалуйста", Blocked, "capability_question", "ты + можешь + politeness (42/42 non-action)"},
|
||||
{"умеешь ли ты выключить свет", Blocked, "capability_question", "ability form + ли"},
|
||||
{"сможешь открыть окно, пожалуйста", Blocked, "capability_question", "bare future + politeness (7/7 non-action)"},
|
||||
{"ты способна выключить свет", Blocked, "capability_question", "ты + способна"},
|
||||
{"могу ли я выключить свет", Blocked, "capability_question", "first-person can + ли"},
|
||||
{"можно ли выключить свет", Blocked, "capability_question", "можно + ли permission question"},
|
||||
{"ты выключишь свет?", Ambiguous, "capability_question", "future tense + ? without can-form"},
|
||||
|
||||
// modal / polite requests — permissive
|
||||
{"можешь выключить свет, пожалуйста", Permissive, "modal_request", "bare можешь + politeness (127/127 action)"},
|
||||
{"пожалуйста, выключи свет", Permissive, "polite_request", "leading politeness + imperative"},
|
||||
{"выключи свет, пожалуйста", Permissive, "polite_request", "imperative + trailing politeness"},
|
||||
{"выключи свет", Permissive, "direct_imperative", "plain imperative"},
|
||||
{"свет выключи, пожалуйста", Permissive, "reordered_target", "target-first imperative"},
|
||||
{"can you выключи свет, please", Permissive, "modal_request", "English frame + Russian imperative + please (96/96 action)"},
|
||||
{"can you выключи свет", Ambiguous, "modal_request", "English can without politeness"},
|
||||
{"не мог бы ты выключить свет", Permissive, "polite_request", "conditional politeness, prohibition-parser exemption"},
|
||||
{"можно выключить свет", Permissive, "modal_request", "можно + infinitive permission-implicature request"},
|
||||
|
||||
// first-person requests
|
||||
{"я хочу выключить свет", Permissive, "first_person_request", "first-person + illocution"},
|
||||
{"я хочу чтобы ты выключил свет", Permissive, "first_person_request", "first-person + embedded ya-you wish"},
|
||||
{"надо выключить свет", Permissive, "first_person_request", "impersonal need"},
|
||||
{"мне нужно включить свет", Permissive, "first_person_request", "first-person oblique + need"},
|
||||
|
||||
// ordinary questions — no execution pressure even when answerable
|
||||
{"что запущено", Ambiguous, "question", "status question, no request evidence"},
|
||||
{"какие службы работают", Ambiguous, "question", "question, no request evidence"},
|
||||
{"сколько ламп включено", Ambiguous, "question", "question, no request evidence"},
|
||||
{"что ты можешь включить", Blocked, "capability_question", "open question with ты + можешь"},
|
||||
{"покажи что запущено", Permissive, "first_person_request", "imperative lead over a status question"},
|
||||
}
|
||||
@@ -0,0 +1,659 @@
|
||||
// Guard is the slice-21 deterministic execution-frame engine (experiment-only).
|
||||
//
|
||||
// It answers one question: given an utterance, what is its execution-frame
|
||||
// eligibility as a three-way gate — permissive, blocked, ambiguous — and why.
|
||||
// It never decides what an utterance IS (that stays with the route classifier);
|
||||
// it only decides whether an utterance may become an executable action at all.
|
||||
// The policy is asymmetric on purpose: blocked and ambiguous must never
|
||||
// execute, and permissive only means "no blocking speech-act evidence exists",
|
||||
// not "execute this".
|
||||
//
|
||||
// It reuses the shipped deterministic routers rather than inventing new ones:
|
||||
//
|
||||
// router.ParseCommandProhibition / IsCommandProhibition direct negative commands
|
||||
// morph.IsVerbForm / morph.Lemma verb mood and finiteness
|
||||
// lexicon.IsFillerParticle / FirstPerson() politeness and first-person frames
|
||||
//
|
||||
// Everything else is closed-class evidence measured on the frozen slice-20 dev
|
||||
// pool (§"measured discriminators" in the brief): 126 capability-question rows
|
||||
// split 42/42/42 across ты-addressed, bare ability (умеешь), and bare future
|
||||
// (сможешь) modality; 127 bare "можешь, пожалуйста" rows are 100% action;
|
||||
// "can you … , please" (English frame + Russian imperative) is 100% action.
|
||||
// The rules below are the encoding of precisely those numbers.
|
||||
//
|
||||
// The reason vocabulary is a closed set. Additions are design decisions that
|
||||
// must land in the report, not silent new branches.
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"github.com/kami/maven/internal/lexicon"
|
||||
"github.com/kami/maven/internal/morph"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// Eligibility is the three-way execution-frame verdict.
|
||||
type Eligibility int
|
||||
|
||||
const (
|
||||
Permissive Eligibility = iota // no blocking speech-act evidence; downstream route decides
|
||||
Blocked // a speech act forbids execution (negation, question, report, …)
|
||||
Ambiguous // not enough evidence either way; must not execute
|
||||
)
|
||||
|
||||
func (e Eligibility) String() string {
|
||||
switch e {
|
||||
case Permissive:
|
||||
return "permissive"
|
||||
case Blocked:
|
||||
return "blocked"
|
||||
default:
|
||||
return "ambiguous"
|
||||
}
|
||||
}
|
||||
|
||||
// Reason is a closed set of structural explanations for a verdict.
|
||||
type Reason string
|
||||
|
||||
const (
|
||||
ReasonCommandProhibition Reason = "command_prohibition"
|
||||
ReasonCapabilityQuestion Reason = "capability_question"
|
||||
ReasonReportedSpeech Reason = "reported_speech"
|
||||
ReasonQuotation Reason = "quotation"
|
||||
ReasonHypothetical Reason = "hypothetical"
|
||||
ReasonNegatedCommand Reason = "negated_command"
|
||||
ReasonExplicitRequest Reason = "explicit_request"
|
||||
ReasonAmbiguousModal Reason = "ambiguous_modal"
|
||||
ReasonNoRequestEvidence Reason = "no_request_evidence"
|
||||
)
|
||||
|
||||
func (r Reason) String() string { return string(r) }
|
||||
|
||||
// Frame is the verdict for one utterance. Eligibility decides; Reasons explain.
|
||||
// A frame may carry more than one reason (e.g. a quoted reported command).
|
||||
type Frame struct {
|
||||
Eligibility Eligibility
|
||||
Reasons []Reason
|
||||
}
|
||||
|
||||
// maybeWord is a single-token or multi-token closed expression, e.g. the
|
||||
// token "не мог бы" covers the three tokens не мog бы when matched as a
|
||||
// contiguous run ("бы" is itself a bound marker). Multi-token members are
|
||||
// matched over the reconstructed token text, never over raw text, so
|
||||
// punctuation boundaries do not defeat them.
|
||||
type maybeWord struct {
|
||||
single []string
|
||||
multi []string // matched as contiguous lowercased token runs
|
||||
}
|
||||
|
||||
func (w maybeWord) in(toks []string, joined string) bool {
|
||||
if hasAny(toks, w.single) {
|
||||
return true
|
||||
}
|
||||
for _, m := range w.multi {
|
||||
tm := strings.Join(tokens(m), " ")
|
||||
if tm != "" && strings.Contains(joined, tm) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ── closed evidence sets (all measured on the slice-20 dev pool) ──────────
|
||||
|
||||
// wakeAddr is stripped from the left of an utterance before command-form
|
||||
// detection: "мавен, выключи свет" and "выключи свет" must ride the same
|
||||
// frame. Closed: the names Maven answers to in the dev pool.
|
||||
var wakeAddr = []string{"мавен", "maven", "мавэн", "алекса", "алиса", "окей", "эй", "hey"}
|
||||
|
||||
// ruAddress are the second-person Russian address tokens. "ты можешь …"
|
||||
// (with or without politeness) is 42/42 capability-question in the dev pool,
|
||||
// so any addressed Russian can-form is a capability question, never a request.
|
||||
var ruAddress = []string{"ты", "тебе", "тебя", "тобой", "тобою", "вы", "вас", "вам", "вами"}
|
||||
|
||||
// enAddress is the English second-person address. Unlike Russian, "can you …
|
||||
// , please" is 96/96 action in the dev pool (English modal frame around a
|
||||
// Russian imperative), so English address alone never blocks: it routes to the
|
||||
// politeness arm.
|
||||
var enAddress = []string{"you", "u", "your"}
|
||||
|
||||
// ruCanForms are the present-can verb forms. Bare (no address) "можешь …,
|
||||
// пожалуйста" is 127/127 action; bare "можешь …" with no politeness is the
|
||||
// ambiguous bucket (no such rows exist in dev — conservative default).
|
||||
var ruCanForms = []string{"можешь", "можете", "могу", "можем"}
|
||||
|
||||
// ruAbilityForms are future/ability modal forms that read as a question of
|
||||
// capability regardless of politeness: "сможешь открыть окно, пожалуйста" and
|
||||
// "умеешь ли ты …" are 0/84 action in the dev pool, so even a polite bare
|
||||
// form never grants execution. "мог(ла) бы …" and "смог(ла) бы …" are the
|
||||
// conditional-politeness mask over the same boundary — except the leading
|
||||
// politeness construction "не мог бы ты …", which the prohibition parser
|
||||
// already classifies as ordinary modal politeness and must stay permissive.
|
||||
var ruAbilityForms = maybeWord{
|
||||
single: []string{
|
||||
"сможешь", "сможете", "смогу", "сможем", "сумеешь", "сумеете",
|
||||
"умеешь", "умеете", "способна", "способен", "способно", "способны",
|
||||
"смог", "смогла", "смогли", "мог", "могла", "могли",
|
||||
},
|
||||
multi: []string{
|
||||
"смог бы", "смогла бы", "смогли бы", "мог бы", "могла бы", "могли бы",
|
||||
"смочь бы", "мочь бы",
|
||||
},
|
||||
}
|
||||
|
||||
// politeNegativeModal is the leading "не мог бы ты/вы …" politeness framing the
|
||||
// prohibition parser exempts as ordinary modal politeness. When it leads the
|
||||
// utterance the capability stage declines and the frame reads as a request.
|
||||
var politeNegativeModal = []string{
|
||||
"не мог бы", "не могла бы", "не могли бы", "не смог бы", "не смогла бы", "не смогли бы",
|
||||
}
|
||||
|
||||
// enCanForms are the English modal can/could tokens.
|
||||
var enCanForms = []string{"can", "could"}
|
||||
|
||||
// politeness is the closed set of politeness fillers. пожалуйста/плиз/please
|
||||
// are already closed-class filler particles in the lexicon; the добр-forms
|
||||
// are the only additions the dev pool exercises.
|
||||
var politeness = maybeWord{
|
||||
single: []string{"пожалуйста", "плиз", "please"},
|
||||
multi: []string{"будь добр", "будьте добры", "был бы добр", "были бы добры"},
|
||||
}
|
||||
|
||||
// reportVerbs are the past/third-person report verbs — the frame that reports
|
||||
// a command rather than issuing it. Second-person imperatives ("скажи",
|
||||
// "расскажи", "напомни") are deliberately absent: those are requests to
|
||||
// report, and their clause forms part of the current utterance, not a
|
||||
// replayed order. Matched as closed list (a report verb outside it is a data
|
||||
// gap, noted in the report).
|
||||
var reportVerbs = []string{
|
||||
"сказал", "сказала", "сказали", "говорил", "говорила", "говорили",
|
||||
"говорит", "говорят", "попросил", "попросила", "попросили",
|
||||
"просил", "просила", "просили", "написал", "написала", "написали",
|
||||
"пишет", "приказал", "приказала", "приказали", "велел", "велела",
|
||||
"велели", "скомандовал", "скомандовала", "рекомендовал", "рекомендовала",
|
||||
"посоветовал", "посоветовала", "сообщил", "сообщила", "сообщили",
|
||||
"объявил", "объявила", "велено", "сказано", "написано", "записано",
|
||||
}
|
||||
|
||||
// reportNouns name a quoted or reported text: "фраза «выключи свет»" is a
|
||||
// quotation, not a command.
|
||||
var reportNouns = []string{
|
||||
"фраза", "фразы", "фразе", "фразу", "слово", "слова", "слове", "словом",
|
||||
"выражение", "выражения", "цитата", "цитату", "цитате",
|
||||
"название", "текст", "сообщение", "письмо", "заметка", "заметку",
|
||||
}
|
||||
|
||||
// hypothesisMarkers open a conditional scope.
|
||||
var hypothesisMarkers = []string{"если", "ежели", "коли", "кабы", "if"}
|
||||
|
||||
// illocutionVerbs make a first-person or impersonal clause a request even
|
||||
// without an imperative form ("я хочу …", "мне нужно …", "надо …").
|
||||
var illocutionVerbs = maybeWord{
|
||||
single: []string{
|
||||
"хочу", "хотел", "хотела", "хотелось", "желаю", "прошу", "просим",
|
||||
"просил", "просила", "просили", "попросить",
|
||||
"надо", "нужно", "следует", "пора", "требуется", "придётся", "придется",
|
||||
"могу", "давай", "давайте",
|
||||
},
|
||||
multi: []string{
|
||||
"хотел бы", "хотела бы", "хочу чтобы", "хотел чтобы", "хотела чтобы",
|
||||
"могу ли",
|
||||
},
|
||||
}
|
||||
|
||||
// ── token helpers ─────────────────────────────────────────────────────────
|
||||
|
||||
// tokens lowercases and splits on anything that is not a letter or digit,
|
||||
// matching the router's planTokens discipline ("что-дальше" tokenises like
|
||||
// "что дальше").
|
||||
func tokens(text string) []string {
|
||||
return strings.FieldsFunc(strings.ToLower(text), func(r rune) bool {
|
||||
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
|
||||
})
|
||||
}
|
||||
|
||||
func hasTok(toks []string, w string) bool {
|
||||
for _, t := range toks {
|
||||
if t == w {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func hasAny(toks, ws []string) bool {
|
||||
for _, w := range ws {
|
||||
if hasTok(toks, w) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func indexTok(toks []string, w string) int {
|
||||
for i, t := range toks {
|
||||
if t == w {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
// isFiniteVerb reports a verb form that is not the dictionary (infinitive)
|
||||
// form: "выключи" is finite, "выключить" is not. A finite verb at command
|
||||
// position is positive request evidence; an infinitive is not.
|
||||
func isFiniteVerb(tok string) bool {
|
||||
if !morph.IsVerbForm(tok) {
|
||||
return false
|
||||
}
|
||||
return morph.Lemma(tok) != tok
|
||||
}
|
||||
|
||||
// isInfinitive reports a token that morph resolves to its own dictionary form
|
||||
// (the lemma ends in the infinitive ending by construction).
|
||||
func isInfinitive(tok string) bool {
|
||||
if !morph.IsVerbForm(tok) {
|
||||
return false
|
||||
}
|
||||
return morph.Lemma(tok) == tok
|
||||
}
|
||||
|
||||
func anyInfinitive(toks []string) bool {
|
||||
for _, t := range toks {
|
||||
if isInfinitive(t) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func anyFiniteVerb(toks []string) bool {
|
||||
for _, t := range toks {
|
||||
if isFiniteVerb(t) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func hasAnyVerb(toks []string) bool {
|
||||
for _, t := range toks {
|
||||
if morph.IsVerbForm(t) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func blocked(rs ...Reason) Frame { return Frame{Eligibility: Blocked, Reasons: rs} }
|
||||
func ambiguous(rs ...Reason) Frame {
|
||||
return Frame{Eligibility: Ambiguous, Reasons: rs}
|
||||
}
|
||||
func permissive(rs ...Reason) Frame {
|
||||
return Frame{Eligibility: Permissive, Reasons: rs}
|
||||
}
|
||||
|
||||
// ── quoted spans ─────────────────────────────────────────────────────────
|
||||
|
||||
// quotedSpan is a maximal quoted interval in the normalized text.
|
||||
type quotedSpan struct{ content string }
|
||||
|
||||
// quotePairs covers the quoting styles the dev pool and brief fixtures use:
|
||||
// Russian guillemets, curly double/single quotes, and straight quotes.
|
||||
var quotePairs = []struct{ open, close string }{
|
||||
{"«", "»"}, {"„", "\""}, {"“", "”"}, {"‚", "‘"}, {"‘", "’"}, {"'", "'"}, {"\"", "\""},
|
||||
}
|
||||
|
||||
// extractQuotedSpans returns the contents of quoted spans in order, in rune
|
||||
// index space (the text is normalized, so glyphs are single runes). An
|
||||
// unbalanced delimiter yields no span (best-effort; the conservative
|
||||
// fallback then applies).
|
||||
func extractQuotedSpans(t string) []quotedSpan {
|
||||
runes := []rune(t)
|
||||
var out []quotedSpan
|
||||
i := 0
|
||||
for i < len(runes) {
|
||||
matched := false
|
||||
for _, p := range quotePairs {
|
||||
po := []rune(p.open)
|
||||
pc := []rune(p.close)
|
||||
if i+len(po) > len(runes) || string(runes[i:i+len(po)]) != p.open {
|
||||
continue
|
||||
}
|
||||
j := i + len(po)
|
||||
for j+len(pc) <= len(runes) && string(runes[j:j+len(pc)]) != p.close {
|
||||
j++
|
||||
}
|
||||
out = append(out, quotedSpan{content: string(runes[i+len(po) : j])})
|
||||
i = j + len(pc)
|
||||
matched = true
|
||||
break
|
||||
}
|
||||
if !matched {
|
||||
i++
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── rule stages (evaluated in this order; a decision is final) ────────────
|
||||
|
||||
// Evaluate derives the execution-frame verdict for one utterance.
|
||||
func Evaluate(text string) Frame {
|
||||
t := router.NormalizeMatchText(text)
|
||||
if strings.TrimSpace(t) == "" {
|
||||
return ambiguous(ReasonNoRequestEvidence)
|
||||
}
|
||||
toks := tokens(t)
|
||||
joined := strings.Join(toks, " ")
|
||||
|
||||
// 1. Quotation: a command inside a quoted span is not a command being
|
||||
// issued now. With a reporting frame outside it is a quotation; a bare
|
||||
// quote is at best ambiguous.
|
||||
if f, ok := stageQuotation(t, toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 2. Reported speech: a past/third-person report verb governing a command
|
||||
// clause reports an order to someone else, it does not issue one.
|
||||
if f, ok := stageReport(t, toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 3. Hypothetical: a command scope opened by "если/if" that does not
|
||||
// continue as a real condition→command is not an execution request.
|
||||
if f, ok := stageHypothesis(toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 4. Direct negative commands: the shipped prohibition parser.
|
||||
if router.IsCommandProhibition(t) {
|
||||
return blocked(ReasonCommandProhibition)
|
||||
}
|
||||
|
||||
// 5. Advisory negatives: "не надо/не стоит/не нужно …".
|
||||
if f, ok := stageAdvisoryNegation(toks); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 6. Capability and permission modality (the measured core).
|
||||
if f, ok := stageCapability(toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 7. Trailing question mark with no modal at play: an uncertain posture,
|
||||
// never a confirmed executable request.
|
||||
if strings.HasSuffix(t, "?") {
|
||||
return ambiguous(ReasonAmbiguousModal)
|
||||
}
|
||||
|
||||
// 8. Positive request evidence.
|
||||
if hasRequestEvidence(toks, joined) {
|
||||
return permissive(ReasonExplicitRequest)
|
||||
}
|
||||
|
||||
// 9. No execution pressure at all.
|
||||
return ambiguous(ReasonNoRequestEvidence)
|
||||
}
|
||||
|
||||
// stageQuotation blocks a quoted command when a reporting frame surrounds it.
|
||||
func stageQuotation(t string, toks []string, joined string) (Frame, bool) {
|
||||
spans := extractQuotedSpans(t)
|
||||
if len(spans) == 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
commandSpan := false
|
||||
for _, sp := range spans {
|
||||
if isCommandishWithin(tokens(sp.content), strings.Join(tokens(sp.content), " ")) {
|
||||
commandSpan = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !commandSpan {
|
||||
return Frame{}, false
|
||||
}
|
||||
reasons := []Reason{ReasonQuotation}
|
||||
if hasReportFrame(toks, joined) {
|
||||
reasons = append(reasons, ReasonReportedSpeech)
|
||||
return Frame{Eligibility: Blocked, Reasons: reasons}, true
|
||||
}
|
||||
// a bare quoted command has no reporting frame: refusable but not a
|
||||
// definite prohibition either (it is at least ambiguous)
|
||||
return Frame{Eligibility: Ambiguous, Reasons: reasons}, true
|
||||
}
|
||||
|
||||
// isCommandishWithin reports the span content carrying command or capability
|
||||
// polarity itself — imperative, prohibition, or a can-form.
|
||||
func isCommandishWithin(toks []string, joined string) bool {
|
||||
if len(toks) == 0 {
|
||||
return false
|
||||
}
|
||||
if router.IsCommandProhibition(strings.Join(toks, " ")) {
|
||||
return true
|
||||
}
|
||||
if hasAny(toks, ruCanForms) || ruAbilityForms.in(toks, joined) || hasAny(toks, enCanForms) {
|
||||
return true
|
||||
}
|
||||
return anyFiniteVerb(toks)
|
||||
}
|
||||
|
||||
func hasReportFrame(toks []string, joined string) bool {
|
||||
if hasAny(toks, reportVerbs) {
|
||||
return true
|
||||
}
|
||||
return hasAny(toks, reportNouns)
|
||||
}
|
||||
|
||||
// stageReport blocks when a report frame governs a command clause: an
|
||||
// infinitive after the report verb, or a quoted imperative. Second-person
|
||||
// imperatives like "скажи/расскажи" are not in reportVerbs, so a request to
|
||||
// report ("расскажи мне, что сказал папа") passes through.
|
||||
func stageReport(t string, toks []string, joined string) (Frame, bool) {
|
||||
if !hasReportFrame(toks, joined) {
|
||||
return Frame{}, false
|
||||
}
|
||||
last := -1
|
||||
for i, w := range toks {
|
||||
if hasTok(reportVerbs, w) || hasTok(reportNouns, w) {
|
||||
last = i
|
||||
}
|
||||
}
|
||||
if last < 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
after := toks[last+1:]
|
||||
if len(after) == 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
// a quoted command after the frame counts as the governed clause
|
||||
for _, sp := range extractQuotedSpans(t) {
|
||||
if isCommandishWithin(tokens(sp.content), strings.Join(tokens(sp.content), " ")) {
|
||||
return blocked(ReasonReportedSpeech, ReasonQuotation), true
|
||||
}
|
||||
}
|
||||
if anyInfinitive(after) || hasAny(after, []string{"что", "чтобы", "чтоб"}) {
|
||||
return blocked(ReasonReportedSpeech), true
|
||||
}
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// stageHypothesis blocks a conditional scope whose clauses are hypothetical
|
||||
// (infinitive or subjunctive "бы") rather than a real condition→command.
|
||||
// "если будет дождь, выключи полив" keeps its imperative continuation and
|
||||
// passes through; it is a real conditional request, not a hypothetical.
|
||||
func stageHypothesis(toks []string, joined string) (Frame, bool) {
|
||||
idx := -1
|
||||
for _, m := range hypothesisMarkers {
|
||||
if i := indexTok(toks, m); i >= 0 && (idx < 0 || i < idx) {
|
||||
idx = i
|
||||
}
|
||||
}
|
||||
if idx < 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
post := toks[idx+1:]
|
||||
if len(post) == 0 || hasTok(post, "бы") || anyInfinitive(post) {
|
||||
return blocked(ReasonHypothetical), true
|
||||
}
|
||||
// a real condition clause is not hypothetical: «если будет дождь,
|
||||
// выключи полив» is a request. The dev dict does not cover «будет», so
|
||||
// the imperative is looked for anywhere, not just after the marker
|
||||
// («выключи свет, если будет дождь»).
|
||||
if !anyFiniteVerb(toks) {
|
||||
return blocked(ReasonHypothetical), true
|
||||
}
|
||||
return permissive(ReasonExplicitRequest), true
|
||||
}
|
||||
|
||||
// stageAdvisoryNegation blocks "не надо/не нужно/не стоит/не следует …".
|
||||
// (absent from the dev pool; covered by brief fixtures)
|
||||
func stageAdvisoryNegation(toks []string) (Frame, bool) {
|
||||
if len(toks) < 3 || toks[0] != "не" {
|
||||
return Frame{}, false
|
||||
}
|
||||
if !hasTok(toks[1:2], "надо") && !hasTok(toks[1:2], "нужно") &&
|
||||
!hasTok(toks[1:2], "стоит") && !hasTok(toks[1:2], "следует") &&
|
||||
!hasTok(toks[1:2], "требуется") {
|
||||
return Frame{}, false
|
||||
}
|
||||
rest := toks[2:]
|
||||
if anyInfinitive(rest) || anyFiniteVerb(rest) || hasAnyVerb(rest) {
|
||||
return blocked(ReasonNegatedCommand), true
|
||||
}
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// stageCapability encodes the measured modal matrix. Returns a decision when
|
||||
// modality alone settles the frame.
|
||||
func stageCapability(toks []string, joined string) (Frame, bool) {
|
||||
// "не мог бы ты …, пожалуйста" style conditional politeness is ordinary
|
||||
// modal politeness (the prohibition parser exempts it as such): a request.
|
||||
for _, pref := range politeNegativeModal {
|
||||
if strings.HasPrefix(joined, pref) {
|
||||
return permissive(ReasonExplicitRequest), true
|
||||
}
|
||||
}
|
||||
|
||||
// "… ли" directly after a can-form is a polar capability question:
|
||||
// "могу ли я …", "можешь ли ты …", "умеешь ли ты …", "можно ли …".
|
||||
// Checked before the modality arms so the polar reading wins.
|
||||
if hasTok(toks, "ли") {
|
||||
for i := 1; i < len(toks); i++ {
|
||||
if toks[i] != "ли" {
|
||||
continue
|
||||
}
|
||||
prev := toks[i-1]
|
||||
if hasTok(ruCanForms, prev) || prev == "можно" || hasTok(ruAbilityForms.single, prev) {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
ruAddr := hasAny(toks, ruAddress)
|
||||
ruCan := hasAny(toks, ruCanForms)
|
||||
ruAbil := ruAbilityForms.in(toks, joined)
|
||||
enCan := hasAny(toks, enCanForms)
|
||||
polite := politeness.in(toks, joined)
|
||||
|
||||
// addressed Russian can-form: capability question, always blocked.
|
||||
// ("ты можешь выключить свет, пожалуйста" included — 42/42 non-action.)
|
||||
if ruAddr && (ruCan || ruAbil) {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
|
||||
// ability forms (future/conditional/умеешь) are capability even bare and
|
||||
// even polite: "сможешь открыть окно, пожалуйста" is 7/7 non-action.
|
||||
if ruAbil && !ruCan {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
|
||||
// bare Russian present can-form: politeness is the request marker.
|
||||
if ruCan && !ruAddr {
|
||||
if polite {
|
||||
return Frame{}, false // modal-request positive evidence is found later
|
||||
}
|
||||
return ambiguous(ReasonAmbiguousModal), true
|
||||
}
|
||||
|
||||
// English can/could: "can you …, please" is a request (96/96 action in the
|
||||
// dev pool; the frame wraps a Russian imperative). Without politeness it
|
||||
// reads as a capability question and stays ambiguous.
|
||||
if enCan && !ruAddr {
|
||||
if polite {
|
||||
return Frame{}, false // positive modal-request evidence later
|
||||
}
|
||||
return ambiguous(ReasonAmbiguousModal), true
|
||||
}
|
||||
|
||||
// "можно" (permission): "можно ли …" is a permission question; a bare
|
||||
// "можно …" is a politeness-implicature request.
|
||||
if hasTok(toks, "можно") {
|
||||
if hasTok(toks, "ли") {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// bare "могу": a self-capability statement, not a request.
|
||||
if hasTok(toks, "могу") && !hasTok(toks, "ли") {
|
||||
return ambiguous(ReasonAmbiguousModal), true
|
||||
}
|
||||
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// hasRequestEvidence is the positive permissive trigger, reached only after
|
||||
// every block/ambiguity stage above has declined.
|
||||
func hasRequestEvidence(toks []string, joined string) bool {
|
||||
polite := politeness.in(toks, joined)
|
||||
enCan := hasAny(toks, enCanForms)
|
||||
|
||||
// 1. politeness + a verb (or an English modal) is explicit request
|
||||
// evidence: "можешь выключить свет, пожалуйста", "can you останови …,
|
||||
// please", "выключи свет, пожалуйста".
|
||||
if polite && (hasAnyVerb(toks) || enCan) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 2. first-person illocution frame: "я хочу …", "мне нужно …".
|
||||
if hasAny(toks, lexicon.FirstPerson()) && illocutionVerbs.in(toks, joined) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 3. impersonal need: "надо …", "нужно …", "пора …".
|
||||
if hasAny(toks, []string{"надо", "нужно", "следует", "пора", "требуется", "придётся", "придется"}) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 4. permission-implicature request: "можно выключить свет".
|
||||
if hasTok(toks, "можно") && !hasTok(toks, "ли") {
|
||||
return true
|
||||
}
|
||||
|
||||
// 5. reminder request in the parser's own exemption scope: the
|
||||
// prohibition parser declines «не забудь напомнить про свет» as a
|
||||
// reminder, not a prohibition — carry that into a request.
|
||||
if strings.HasPrefix(joined, "не забудь") && hasAny(toks, lexicon.ReminderVerbs()) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 6. leading finite verb (imperative or otherwise tensed verb at command
|
||||
// position): "выключи свет", "покажи что запущено". Address and filler
|
||||
// particles are stripped first, so "мавен, выключи свет" rides the same
|
||||
// frame.
|
||||
lead := toks
|
||||
for len(lead) > 0 {
|
||||
first := lead[0]
|
||||
if !lexicon.IsFillerParticle(first) && !hasTok(wakeAddr, first) && !hasTok(ruAddress, first) {
|
||||
break
|
||||
}
|
||||
lead = lead[1:]
|
||||
}
|
||||
if len(lead) > 0 && isFiniteVerb(lead[0]) {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/kami/maven/internal/morph"
|
||||
)
|
||||
|
||||
func TestEvaluateFixtures(t *testing.T) {
|
||||
for _, fx := range Fixtures {
|
||||
got := Evaluate(fx.Utterance)
|
||||
if got.Eligibility != fx.Want {
|
||||
t.Errorf("%s: want %s got %s (reasons %v)", fx.Utterance, fx.Want, got.Eligibility, got.Reasons)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestMeasuredDiscriminators pins the corpus-verified numbers on the frozen
|
||||
// pool. These are the exact measurements the rules were built on, so a change
|
||||
// that moves them is a rule regression visible in the slice.
|
||||
func loadPool(t *testing.T) []Row {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile("/tmp/mvn-s21/pool.json")
|
||||
if os.IsNotExist(err) {
|
||||
t.Skip("pool.json missing; run slice21_emit.py first")
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var rows []Row
|
||||
if err := json.Unmarshal(b, &rows); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
func TestDevCapabilityProhibition(t *testing.T) {
|
||||
if !morph.Available() {
|
||||
t.Skip("morph dict unavailable")
|
||||
}
|
||||
rows := loadPool(t)
|
||||
var n, pass, blockedN, ambig int
|
||||
for _, r := range rows {
|
||||
if !hasTok(r.Tags, "capability_question") {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
switch Evaluate(variants(r.NText)[vOrig]).Eligibility {
|
||||
case Permissive:
|
||||
pass++
|
||||
case Blocked:
|
||||
blockedN++
|
||||
case Ambiguous:
|
||||
ambig++
|
||||
}
|
||||
}
|
||||
if n != 126 {
|
||||
t.Fatalf("capability-question rows = %d, want 126", n)
|
||||
}
|
||||
if pass != 0 {
|
||||
t.Fatalf("capability-question dangerous pass = %d, want 0", pass)
|
||||
}
|
||||
if blockedN != 126 && ambig != 0 {
|
||||
t.Errorf("blocked=%d ambig=%d, expect all 126 blocked", blockedN, ambig)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevBareCanPoliteIsAction(t *testing.T) {
|
||||
if !morph.Available() {
|
||||
t.Skip("morph dict unavailable")
|
||||
}
|
||||
rows := loadPool(t)
|
||||
total, action, perm := 0, 0, 0
|
||||
for _, r := range rows {
|
||||
toks := tokens(r.NText)
|
||||
if !hasTok(toks, "можешь") || hasAny(toks, ruAddress) ||
|
||||
hasAny(toks, append([]string{}, ruAbilityForms.single...)) {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
if r.Route == "action" {
|
||||
action++
|
||||
}
|
||||
if Evaluate(r.NText).Eligibility == Permissive {
|
||||
perm++
|
||||
}
|
||||
}
|
||||
if total != 127 || action != 127 {
|
||||
t.Fatalf("bare-можешь+polite: n=%d action=%d, want 127/127", total, action)
|
||||
}
|
||||
if perm != 127 {
|
||||
t.Fatalf("bare-можешь+polite permissive=%d, want 127", perm)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevEnglishCanPoliteIsAction(t *testing.T) {
|
||||
if !morph.Available() {
|
||||
t.Skip("morph dict unavailable")
|
||||
}
|
||||
rows := loadPool(t)
|
||||
total, action, perm := 0, 0, 0
|
||||
for _, r := range rows {
|
||||
toks := tokens(r.NText)
|
||||
if !hasAny(toks, enCanForms) {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
if r.Route == "action" {
|
||||
action++
|
||||
}
|
||||
if Evaluate(r.NText).Eligibility == Permissive {
|
||||
perm++
|
||||
}
|
||||
}
|
||||
if total != 96 || action != 96 {
|
||||
t.Fatalf("can-rows: n=%d action=%d, want 96/96", total, action)
|
||||
}
|
||||
if perm != 96 {
|
||||
t.Fatalf("can-rows permissive=%d, want 96", perm)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoPermissiveRowEndsInQuestion(t *testing.T) {
|
||||
// asymmetric posture: the guard must never approve a trailing-? frame --
|
||||
// the corpus has 0 action rows ending in "?", and approving any would
|
||||
// bet on punctuation the slice has ruled uncertain.
|
||||
rows := loadPool(t)
|
||||
for _, r := range rows {
|
||||
if !strings.HasSuffix(r.NText, "?") {
|
||||
continue
|
||||
}
|
||||
if got := Evaluate(r.NText).Eligibility; got == Permissive {
|
||||
t.Errorf("canonical '?': %q approved (%s)", r.Text, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,566 @@
|
||||
// Slice 21 runner: report the deterministic execution-frame guard against the
|
||||
// frozen slice-20 dev pool.
|
||||
//
|
||||
// Reads the emit step's compact files (pool.json, pairs.json, sparse_oof.json
|
||||
// in /tmp/mvn-s21) and prints the report tables plus a machine-readable
|
||||
// guard_results.json. Reuses router/morph/lexicon parsers live inside this
|
||||
// module — the pool texts are the only data, no embedding is recomputed.
|
||||
//
|
||||
// Usage: go run ./cmd/semantic-router-experiment/slice21
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
const sparseThreshold = 0.715 // slice-18 §4 strict operating point (P>=0.95 best recall)
|
||||
|
||||
var familyPriority = []string{
|
||||
"capability_question", "question", "first_person_request",
|
||||
"modal_request", "polite_request", "reordered_target", "direct_imperative",
|
||||
}
|
||||
|
||||
func familyOf(tags []string) string {
|
||||
for _, f := range familyPriority {
|
||||
if hasTok(tags, f) {
|
||||
return f
|
||||
}
|
||||
}
|
||||
return "other"
|
||||
}
|
||||
|
||||
// ── pool row ──────────────────────────────────────────────────────────────
|
||||
|
||||
type Row struct {
|
||||
Idx int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
NText string `json:"n_text"`
|
||||
Route string `json:"route"`
|
||||
Y int `json:"y"`
|
||||
Tags []string `json:"tags"`
|
||||
CVFold int `json:"cv_fold"`
|
||||
SplitGp string `json:"split_group"`
|
||||
SourceID string `json:"source_id"`
|
||||
Family string
|
||||
VariantOf int
|
||||
}
|
||||
|
||||
type Pair struct{ Cap, Act int }
|
||||
|
||||
// ── stress variants ───────────────────────────────────────────────────────
|
||||
|
||||
var nofinalRe = regexp.MustCompile(`[?.!,;:]+$`)
|
||||
|
||||
// strip_punct mirrors the slice-18/19 python strip_punct: trailing sentence
|
||||
// punctuation, then every non-word/non-space rune.
|
||||
func stripPunct(t string) string {
|
||||
t = nofinalRe.ReplaceAllString(strings.TrimSpace(t), "")
|
||||
out := make([]rune, 0, len(t))
|
||||
var prevSpace bool
|
||||
for _, r := range t {
|
||||
if unicode.IsLetter(r) || unicode.IsNumber(r) {
|
||||
out = append(out, r)
|
||||
prevSpace = false
|
||||
} else if !prevSpace {
|
||||
out = append(out, ' ')
|
||||
prevSpace = true
|
||||
}
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
|
||||
func variants(nText string) [3]string {
|
||||
return [3]string{
|
||||
nText,
|
||||
nofinalRe.ReplaceAllString(strings.TrimSpace(nText), ""),
|
||||
stripPunct(nText),
|
||||
}
|
||||
}
|
||||
|
||||
const (
|
||||
vOrig = iota
|
||||
vNofinal
|
||||
vStrip
|
||||
)
|
||||
|
||||
var variantName = [3]string{"orig", "nofinal", "strip"}
|
||||
|
||||
type result struct {
|
||||
Frame Frame `json:"frame"`
|
||||
}
|
||||
|
||||
// ── metrics ──────────────────────────────────────────────────────────────
|
||||
|
||||
type triTab struct {
|
||||
Permissive, Blocked, Ambiguous int
|
||||
PermNonact, BlockedAction, AmbAction int
|
||||
}
|
||||
|
||||
type runAgg struct {
|
||||
n, action, tp, fp, fn int
|
||||
approved int
|
||||
capPermissive int
|
||||
}
|
||||
|
||||
func (a *runAgg) addApproved(approved bool, route string) {
|
||||
a.n++
|
||||
if route == "action" {
|
||||
a.action++
|
||||
}
|
||||
if approved {
|
||||
a.approved++
|
||||
if route == "action" {
|
||||
a.tp++
|
||||
} else {
|
||||
a.fp++
|
||||
}
|
||||
} else if route == "action" {
|
||||
a.fn++
|
||||
}
|
||||
}
|
||||
|
||||
func (a *runAgg) P() string { return fmtPct(frac(a.tp, a.tp+a.fp)) }
|
||||
func (a *runAgg) R() string { return fmtPct(frac(a.tp, a.action)) }
|
||||
func (a *runAgg) FA() int { return a.fp }
|
||||
func (a *runAgg) FArate() string {
|
||||
return fmtPct(frac(a.fp, a.n))
|
||||
}
|
||||
|
||||
func maxi(a, b int) int {
|
||||
if a > b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// frac is the guarded ratio the tables print (0/0 is 0).
|
||||
func frac(num, den int) float64 { return float64(num) / float64(maxi(den, 1)) }
|
||||
|
||||
func fmtPct(v float64) string { return fmt.Sprintf("%.1f%%", 100*v) }
|
||||
|
||||
// ── main ─────────────────────────────────────────────────────────────────
|
||||
|
||||
func main() {
|
||||
poolPath := flag.String("pool", "/tmp/mvn-s21/pool.json", "dev pool rows")
|
||||
pairsPath := flag.String("pairs", "/tmp/mvn-s21/pairs.json", "cap-vs-action pairs")
|
||||
sparsePath := flag.String("sparse", "/tmp/mvn-s21/sparse_oof.json", "slice-18 both OOF proba")
|
||||
outPath := flag.String("out", "/tmp/mvn-s21/guard_results.json", "machine-readable results")
|
||||
flag.Parse()
|
||||
|
||||
rows := mustLoad[[]Row](*poolPath)
|
||||
// pairs.json is bare [cap, act] index pairs; adapt into typed pairs.
|
||||
rawPairs := mustLoad[[][2]int](*pairsPath)
|
||||
pairs := make([]Pair, 0, len(rawPairs))
|
||||
for _, rp := range rawPairs {
|
||||
pairs = append(pairs, Pair{Cap: rp[0], Act: rp[1]})
|
||||
}
|
||||
sparseOOF := mustLoad[[]struct {
|
||||
Idx int `json:"idx"`
|
||||
Proba float64 `json:"proba"`
|
||||
}](*sparsePath)
|
||||
proba := make([]float64, len(rows))
|
||||
for _, s := range sparseOOF {
|
||||
proba[s.Idx] = s.Proba
|
||||
}
|
||||
|
||||
famPrio := 0
|
||||
for i := range rows {
|
||||
rows[i].Family = familyOf(rows[i].Tags)
|
||||
if rows[i].Family != "other" {
|
||||
famPrio++
|
||||
}
|
||||
}
|
||||
_ = famPrio
|
||||
|
||||
// verdicts per variant
|
||||
type rowRes struct {
|
||||
Idx int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
Route string `json:"route"`
|
||||
Family string `json:"family"`
|
||||
Tags []string `json:"tags"`
|
||||
Frames map[string]string `json:"frames"` // variant -> eligibility
|
||||
}
|
||||
|
||||
perVariant := make([][3]Frame, len(rows))
|
||||
fmt.Println("slice 21 — deterministic execution-frame guard on slice-20 dev pool")
|
||||
fmt.Println("==================================================================")
|
||||
|
||||
for i, r := range rows {
|
||||
vs := variants(r.NText)
|
||||
var fr [3]Frame
|
||||
for vi := 0; vi < 3; vi++ {
|
||||
fr[vi] = Evaluate(vs[vi])
|
||||
}
|
||||
perVariant[i] = fr
|
||||
}
|
||||
|
||||
// ── §3 three-way cross-tab (orig) ─────────────────────────────────────
|
||||
fmt.Println("\n## 1. Three-way eligibility × route (orig)")
|
||||
tab := triTab{}
|
||||
for i, r := range rows {
|
||||
switch perVariant[i][vOrig].Eligibility {
|
||||
case Permissive:
|
||||
tab.Permissive++
|
||||
if r.Route != "action" {
|
||||
tab.PermNonact++
|
||||
}
|
||||
case Blocked:
|
||||
tab.Blocked++
|
||||
if r.Route == "action" {
|
||||
tab.BlockedAction++
|
||||
}
|
||||
case Ambiguous:
|
||||
tab.Ambiguous++
|
||||
if r.Route == "action" {
|
||||
tab.AmbAction++
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Printf("permissive: %d blocked: %d ambiguous: %d\n", tab.Permissive, tab.Blocked, tab.Ambiguous)
|
||||
fmt.Printf(" permissive non-action: %d blocked action: %d ambiguous action: %d\n",
|
||||
tab.PermNonact, tab.BlockedAction, tab.AmbAction)
|
||||
|
||||
// ── §4 binary executable-gate metrics on orig ─────────────────────────
|
||||
fmt.Println("\n## 2. Binary executable gate (approve = permissive; deny = blocked|ambiguous)")
|
||||
a := runAgg{}
|
||||
capCov, capPerm, capAmb := 0, 0, 0
|
||||
for i, r := range rows {
|
||||
el := perVariant[i][vOrig].Eligibility
|
||||
a.addApproved(el == Permissive, r.Route)
|
||||
if hasTok(r.Tags, "capability_question") {
|
||||
capCov++
|
||||
switch el {
|
||||
case Permissive:
|
||||
capPerm++
|
||||
case Ambiguous:
|
||||
capAmb++
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Printf("approved: %d denied: %d (n=%d, action=%d)\n", a.approved, a.n-a.approved, a.n, a.action)
|
||||
fmt.Printf("action precision %s recall %s FA %d (%s)\n", a.P(), a.R(), a.FA(), a.FArate())
|
||||
fmt.Printf("capability-question dangerous pass: %d / %d (rate %s)\n",
|
||||
capPerm, capCov, fmtPct(frac(capPerm, capCov)))
|
||||
fmt.Printf("capability-question blocked %d, ambiguous %d\n", capCov-capPerm-capAmb, capAmb)
|
||||
|
||||
// ── §15 family stress (orig) ──────────────────────────────────────────
|
||||
fmt.Println("\n## 3. Family stress (orig; counts per eligibility)")
|
||||
fmt.Printf("%-24s %8s %8s %8s %8s\n", "family", "n", "perm", "block", "ambig")
|
||||
famOrder := []string{"direct_imperative", "polite_request", "modal_request", "first_person_request",
|
||||
"reordered_target", "capability_question", "question", "other"}
|
||||
famAgg := map[string]*triTab{}
|
||||
for _, f := range famOrder {
|
||||
famAgg[f] = &triTab{}
|
||||
}
|
||||
for i, r := range rows {
|
||||
t := famAgg[r.Family]
|
||||
if t == nil {
|
||||
continue
|
||||
}
|
||||
switch perVariant[i][vOrig].Eligibility {
|
||||
case Permissive:
|
||||
t.Permissive++
|
||||
if r.Route != "action" {
|
||||
t.PermNonact++
|
||||
}
|
||||
case Blocked:
|
||||
t.Blocked++
|
||||
case Ambiguous:
|
||||
t.Ambiguous++
|
||||
}
|
||||
}
|
||||
for _, f := range famOrder {
|
||||
t := famAgg[f]
|
||||
if t == nil {
|
||||
continue
|
||||
}
|
||||
n := t.Permissive + t.Blocked + t.Ambiguous
|
||||
if n == 0 {
|
||||
continue
|
||||
}
|
||||
fmt.Printf("%-24s %8d %8d %8d %8d\n", f, n, t.Permissive, t.Blocked, t.Ambiguous)
|
||||
}
|
||||
|
||||
// ── §cap-Q LOFO across stress variants ────────────────────────────────
|
||||
fmt.Println("\n## 4. Capability-question dangerous pass by stress variant")
|
||||
for vi := 0; vi < 3; vi++ {
|
||||
cp, cb, ca := 0, 0, 0
|
||||
for i, r := range rows {
|
||||
if !hasTok(r.Tags, "capability_question") {
|
||||
continue
|
||||
}
|
||||
switch perVariant[i][vi].Eligibility {
|
||||
case Permissive:
|
||||
cp++
|
||||
case Blocked:
|
||||
cb++
|
||||
case Ambiguous:
|
||||
ca++
|
||||
}
|
||||
}
|
||||
fmt.Printf(" %-8s dangerous-pass %d blocked %d ambiguous %d\n",
|
||||
variantName[vi], cp, cb, ca)
|
||||
}
|
||||
|
||||
// ── §pair test ────────────────────────────────────────────────────────
|
||||
fmt.Println("\n## 5. Paired action/capability (cap row must never clear)")
|
||||
capClear, actPerm, actAmbig, actBlock := 0, 0, 0, 0
|
||||
for _, p := range pairs {
|
||||
cel := perVariant[p.Cap][vOrig].Eligibility
|
||||
ael := perVariant[p.Act][vOrig].Eligibility
|
||||
if cel == Permissive {
|
||||
capClear++
|
||||
}
|
||||
switch ael {
|
||||
case Permissive:
|
||||
actPerm++
|
||||
case Ambiguous:
|
||||
actAmbig++
|
||||
case Blocked:
|
||||
actBlock++
|
||||
}
|
||||
}
|
||||
fmt.Printf("pairs %d: cap cleared %d (rate %s), action permissive %d, action ambiguous %d, action blocked %d\n",
|
||||
len(pairs), capClear, fmtPct(frac(capClear, len(pairs))),
|
||||
actPerm, actAmbig, actBlock)
|
||||
|
||||
// ── �safe composition §17 ─────────────────────────────────────────────
|
||||
fmt.Println("\n## 6. Composition: guard-alone / sparse-alone / guard→sparse (orig)")
|
||||
compose := map[string]*runAgg{
|
||||
"guard_alone": {},
|
||||
"sparse_alone": {},
|
||||
"guard_sparse": {},
|
||||
}
|
||||
for i, r := range rows {
|
||||
gPerm := perVariant[i][vOrig].Eligibility == Permissive
|
||||
sPerm := proba[i] >= sparseThreshold
|
||||
compose["guard_alone"].addApproved(gPerm, r.Route)
|
||||
compose["sparse_alone"].addApproved(sPerm, r.Route)
|
||||
compose["guard_sparse"].addApproved(gPerm && sPerm, r.Route)
|
||||
}
|
||||
fmt.Printf("%-14s %8s %8s %6s %10s %6s %10s\n", "policy", "P", "R", "FA", "FA rate", "capQ", "capQ rate")
|
||||
for _, name := range []string{"guard_alone", "sparse_alone", "guard_sparse"} {
|
||||
agg := compose[name]
|
||||
capQ := 0
|
||||
for i, r := range rows {
|
||||
if !hasTok(r.Tags, "capability_question") {
|
||||
continue
|
||||
}
|
||||
ok := false
|
||||
switch name {
|
||||
case "guard_alone":
|
||||
ok = perVariant[i][vOrig].Eligibility == Permissive
|
||||
case "sparse_alone":
|
||||
ok = proba[i] >= sparseThreshold
|
||||
case "guard_sparse":
|
||||
ok = perVariant[i][vOrig].Eligibility == Permissive && proba[i] >= sparseThreshold
|
||||
}
|
||||
if ok {
|
||||
capQ++
|
||||
}
|
||||
}
|
||||
fmt.Printf("%-14s %8s %8s %6d %10s %6d %10s\n", name, agg.P(), agg.R(), agg.FA(),
|
||||
agg.FArate(), capQ, fmtPct(float64(capQ)/126))
|
||||
}
|
||||
|
||||
// composition on strip too (brief §16 voice stress)
|
||||
fmt.Println("\n## 7. Composition on punctuation-stripped text (strip)")
|
||||
c2 := runAgg{}
|
||||
capQ2 := 0
|
||||
for i, r := range rows {
|
||||
gPerm := perVariant[i][vStrip].Eligibility == Permissive
|
||||
ok := gPerm && proba[i] >= sparseThreshold
|
||||
c2.addApproved(ok, r.Route)
|
||||
if hasTok(r.Tags, "capability_question") && ok {
|
||||
capQ2++
|
||||
}
|
||||
}
|
||||
fmt.Printf("guard→sparse strip: P %s R %s FA %d (%s) capQ pass %d\n",
|
||||
c2.P(), c2.R(), c2.FA(), c2.FArate(), capQ2)
|
||||
|
||||
// ── §19 manual classification scratch ─────────────────────────────────
|
||||
fmt.Println("\n## 8. Manual classification (scan material written to manual_class.json)")
|
||||
var dangerous []map[string]any
|
||||
var permNonact []map[string]any
|
||||
var deniedAction []map[string]any
|
||||
for i, r := range rows {
|
||||
fr := perVariant[i][vOrig]
|
||||
if hasTok(r.Tags, "capability_question") && fr.Eligibility == Permissive {
|
||||
dangerous = append(dangerous, map[string]any{
|
||||
"idx": r.Idx, "text": r.Text, "route": r.Route,
|
||||
"reasons": fr.Reasons,
|
||||
})
|
||||
}
|
||||
if fr.Eligibility == Permissive && r.Route != "action" {
|
||||
permNonact = append(permNonact, map[string]any{
|
||||
"idx": r.Idx, "text": r.Text, "route": r.Route,
|
||||
"family": r.Family, "reasons": fr.Reasons,
|
||||
})
|
||||
}
|
||||
if fr.Eligibility != Permissive && r.Route == "action" {
|
||||
deniedAction = append(deniedAction, map[string]any{
|
||||
"idx": r.Idx, "text": r.Text, "family": r.Family,
|
||||
"eligibility": fr.Eligibility.String(), "reasons": fr.Reasons,
|
||||
})
|
||||
}
|
||||
}
|
||||
writeManual(permNonact, deniedAction, dangerous)
|
||||
fmt.Printf("dangerous passes: %d permissive non-action: %d denied action: %d\n",
|
||||
len(dangerous), len(permNonact), len(deniedAction))
|
||||
groupAndSample("permissive non-action by reason+family", permNonact, 4)
|
||||
groupAndSample("denied action by reason+family", deniedAction, 4)
|
||||
|
||||
// ── fixtures ──────────────────────────────────────────────────────────
|
||||
fmt.Println("\n## 9. Brief fixtures")
|
||||
pass := 0
|
||||
for _, fx := range Fixtures {
|
||||
got := Evaluate(fx.Utterance)
|
||||
mark := "ok "
|
||||
if got.Eligibility != fx.Want {
|
||||
mark = "FAIL"
|
||||
} else {
|
||||
pass++
|
||||
}
|
||||
if got.Eligibility != fx.Want {
|
||||
fmt.Printf(" %s %-14s want %-10s got %-10s %s\n", mark, fx.Family,
|
||||
fx.Want, got.Eligibility.String(), fx.Utterance)
|
||||
}
|
||||
}
|
||||
fmt.Printf("fixtures: %d/%d passed\n", pass, len(Fixtures))
|
||||
|
||||
// write result file
|
||||
rr := make([]rowRes, 0, len(rows))
|
||||
for i, r := range rows {
|
||||
fr := [3]string{"", "", ""}
|
||||
for vi := 0; vi < 3; vi++ {
|
||||
fr[vi] = perVariant[i][vi].Eligibility.String()
|
||||
}
|
||||
rr = append(rr, rowRes{
|
||||
Idx: r.Idx, Text: r.Text, Route: r.Route, Family: r.Family, Tags: r.Tags,
|
||||
Frames: map[string]string{
|
||||
"orig": fr[vOrig], "nofinal": fr[vNofinal], "strip": fr[vStrip],
|
||||
},
|
||||
})
|
||||
}
|
||||
if *outPath != "" {
|
||||
mustSave(*outPath, map[string]any{
|
||||
"pool": "/tmp/mvn-s21/pool.json",
|
||||
"rows": rr,
|
||||
"aggregates": map[string]any{
|
||||
"tab": tab,
|
||||
"capq_pass": capPerm,
|
||||
"capq_blocked": capCov - capPerm - capAmb,
|
||||
"capq_ambiguous": capAmb,
|
||||
"binary": map[string]any{"tp": a.tp, "fp": a.fp, "fn": a.fn, "approved": a.approved, "n": a.n},
|
||||
"pairs": map[string]any{"n": len(pairs), "cap_cleared": capClear, "act_permissive": actPerm},
|
||||
"guard_sparse": map[string]any{"tp": compose["guard_sparse"].tp, "fp": compose["guard_sparse"].fp, "fn": compose["guard_sparse"].fn},
|
||||
"dangerous_passes": len(dangerous),
|
||||
"perm_nonact_count": len(permNonact),
|
||||
"denied_action": len(deniedAction),
|
||||
},
|
||||
})
|
||||
fmt.Println("wrote", *outPath)
|
||||
}
|
||||
}
|
||||
|
||||
// ── manual classification helpers ─────────────────────────────────────────
|
||||
|
||||
func writeManual(permNonact, deniedAction, dangerous []map[string]any) {
|
||||
writeJSON("/tmp/mvn-s21/manual_class.json", map[string]any{
|
||||
"dangerous_passes": dangerous,
|
||||
"permissive_non_action": permNonact,
|
||||
"denied_action": deniedAction,
|
||||
})
|
||||
}
|
||||
|
||||
func groupAndSample(title string, rows []map[string]any, sample int) {
|
||||
type g struct {
|
||||
key string
|
||||
n int
|
||||
texts []string
|
||||
}
|
||||
groups := map[string]*g{}
|
||||
var order []string
|
||||
for _, r := range rows {
|
||||
var family, reason, el string
|
||||
if v, ok := r["family"].(string); ok {
|
||||
family = v
|
||||
}
|
||||
if v, ok := r["eligibility"].(string); ok {
|
||||
el = v
|
||||
}
|
||||
if rs, ok := r["reasons"].([]Reason); ok {
|
||||
rs2 := make([]string, len(rs))
|
||||
for k, rr := range rs {
|
||||
rs2[k] = rr.String()
|
||||
}
|
||||
reason = strings.Join(rs2, ",")
|
||||
} else if rs, ok := r["reasons"].([]string); ok {
|
||||
reason = strings.Join(rs, ",")
|
||||
}
|
||||
key := fmt.Sprintf("family=%s elig=%s reason=%s", family, el, reason)
|
||||
if _, ok := groups[key]; !ok {
|
||||
groups[key] = &g{key: key}
|
||||
order = append(order, key)
|
||||
}
|
||||
groups[key].n++
|
||||
if len(groups[key].texts) < sample {
|
||||
groups[key].texts = append(groups[key].texts, firstN(fmt.Sprint(r["text"]), 60))
|
||||
}
|
||||
}
|
||||
fmt.Printf("%s (%d rows):\n", title, len(rows))
|
||||
for _, key := range order {
|
||||
gr := groups[key]
|
||||
fmt.Printf(" %-58s n=%d %s\n", gr.key, gr.n, strings.Join(gr.texts, " | "))
|
||||
}
|
||||
}
|
||||
|
||||
func firstN(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n] + "…"
|
||||
}
|
||||
|
||||
// ── io helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
func mustLoad[T any](path string) T {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var v T
|
||||
if err := json.Unmarshal(b, &v); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "json:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func mustSave(path string, v any) {
|
||||
b, err := json.MarshalIndent(v, "", " ")
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := os.WriteFile(path, b, 0o644); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSON(path string, v any) {
|
||||
b, _ := json.MarshalIndent(v, "", " ")
|
||||
_ = os.WriteFile(path, b, 0o644)
|
||||
}
|
||||
|
||||
var _ = sort.Strings
|
||||
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 21 emit: deterministic execution-frame guard — data files for the Go harness
|
||||
==================================================================================
|
||||
|
||||
Slice 18 showed the sparse lexical gate owns the aggregate boundary (PR-AUC
|
||||
0.838, strict operating point at threshold 0.715 with P>=0.95 | R=0.264) and
|
||||
slice 19/20 showed learning heads collapse on capability-question LOFO. Slice 21
|
||||
tests the deterministic alternative: a rule engine over existing parsers that
|
||||
decides execution eligibility as a three-way gate (permissive / blocked /
|
||||
ambiguous), never itself routing.
|
||||
|
||||
This script only repackages the frozen dev pool for the Go harness. It reuses
|
||||
slice 18's feature builders and grouped-CV and slice 19's pair builder verbatim,
|
||||
so the numbers the Go side reports are the same populations the accepts
|
||||
measured. It writes:
|
||||
|
||||
/tmp/mvn-s21/pool.json dev rows: idx, text, n_text, route, y, tags,
|
||||
cv_fold, split_group, source_id, family
|
||||
/tmp/mvn-s21/pairs.json capability-vs-action pairs (slice-19 builder)
|
||||
/tmp/mvn-s21/sparse_oof.json slice-18 "both" grouped-CV OOF proba per row
|
||||
(for the §17 guard+sparse composition)
|
||||
|
||||
No training happens here and no label is changed. The guard itself is Go.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402
|
||||
import slice19_main # noqa: E402
|
||||
|
||||
OUT_DIR = "/tmp/mvn-s21"
|
||||
SPARSE_THRESHOLD = 0.715 # slice-18 §4 strict-operating-point (P>=0.95 best recall)
|
||||
|
||||
|
||||
def main():
|
||||
# Population = the exact slice-20 dev pool (s19.load_dev): every dev_pool
|
||||
# row, fast-path included. The guard is evaluated on what slice 20 measured.
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
print(f"dev pool (all dev_pool rows): {len(dev)} rows")
|
||||
print(f"corpus meta: {meta.get('dev_count', '?')} dev rows declared, "
|
||||
f"{meta.get('route_counts', {}).get('action', '?')} action declared")
|
||||
|
||||
n_texts = [slice18_sparse.normalize_match_text(e["text"]) for e in dev]
|
||||
|
||||
rows = []
|
||||
by_route = {}
|
||||
by_family = {}
|
||||
for i, (e, nt) in enumerate(zip(dev, n_texts)):
|
||||
tags = sorted(set(e.get("tags", [])))
|
||||
route = e["route"]
|
||||
fam = slice19_main.family_of(set(tags))
|
||||
by_route[route] = by_route.get(route, 0) + 1
|
||||
by_family[fam] = by_family.get(fam, 0) + 1
|
||||
rows.append({
|
||||
"idx": i,
|
||||
"text": e["text"],
|
||||
"n_text": nt,
|
||||
"route": route,
|
||||
"y": 1 if route == "action" else 0,
|
||||
"tags": tags,
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"source_id": e["source_id"],
|
||||
})
|
||||
|
||||
print("routes:", by_route)
|
||||
print("families:", by_family)
|
||||
|
||||
# ── pairs (slice-19 builder, exact population) ─────────────────────────
|
||||
ldev = [{
|
||||
"text_orig": nt,
|
||||
"route": r["route"],
|
||||
"y": r["y"],
|
||||
"cv_fold": r["cv_fold"],
|
||||
"tags": set(r["tags"]),
|
||||
"source_id": r["source_id"],
|
||||
} for r, nt in zip(rows, n_texts)]
|
||||
pairs = slice19_main.build_pairs(ldev, n_texts)
|
||||
print(f"pairs: {len(pairs)}")
|
||||
|
||||
# ── slice-18 "both" grouped-CV OOF proba, aligned to row index ────────
|
||||
y = [1 if r["route"] == "action" else 0 for r in rows]
|
||||
folds = [r["cv_fold"] for r in rows]
|
||||
X, _vec = slice18_sparse.build_features(n_texts, "both")
|
||||
print(f"sparse 'both' X: {X.shape}")
|
||||
yb = np.array(y)
|
||||
folds_arr = np.array(folds)
|
||||
idx_proba = {}
|
||||
for te_fold in sorted(set(folds)):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
p = clf.predict_proba(X[te])[:, 1]
|
||||
te_idx = np.where(te)[0]
|
||||
for k, i in enumerate(te_idx):
|
||||
idx_proba[int(i)] = float(p[k])
|
||||
assert len(idx_proba) == len(rows)
|
||||
sparse_oof = [{"idx": i, "proba": idx_proba[i]} for i in range(len(rows))]
|
||||
pred = [1 if idx_proba[i] >= 0.5 else 0 for i in range(len(rows))]
|
||||
tp = sum(1 for i in range(len(rows)) if y[i] == 1 and pred[i] == 1)
|
||||
fp = sum(1 for i in range(len(rows)) if y[i] == 0 and pred[i] == 1)
|
||||
fn = sum(1 for i in range(len(rows)) if y[i] == 1 and pred[i] == 0)
|
||||
print(f"sparse both OOF @0.5: P={tp/max(tp+fp,1):.3f} R={tp/max(tp+fn,1):.3f} "
|
||||
f"FA={fp} ({fp/len(rows):.4f})")
|
||||
pred21 = [1 if idx_proba[i] >= SPARSE_THRESHOLD else 0 for i in range(len(rows))]
|
||||
tp = sum(1 for i in range(len(rows)) if y[i] == 1 and pred21[i] == 1)
|
||||
fp = sum(1 for i in range(len(rows)) if y[i] == 0 and pred21[i] == 1)
|
||||
fn = sum(1 for i in range(len(rows)) if y[i] == 1 and pred21[i] == 0)
|
||||
print(f"sparse both OOF @{SPARSE_THRESHOLD}: P={tp/max(tp+fp,1):.3f} "
|
||||
f"R={tp/max(tp+fn,1):.3f} FA={fp} ({fp/len(rows):.4f})")
|
||||
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
|
||||
json.dump(rows, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "pairs.json"), "w") as f:
|
||||
json.dump([[c, a] for c, a in pairs], f)
|
||||
with open(os.path.join(OUT_DIR, "sparse_oof.json"), "w") as f:
|
||||
json.dump(sparse_oof, f)
|
||||
print(f"wrote {OUT_DIR}/{{pool,pairs,sparse_oof}}.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,65 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
// headsMain runs the deployed cascade minus the resident LLM: stage-0
|
||||
// grammars → routing heads (fine-tuned e5 copy + softmax, router_heads.onnx,
|
||||
// 0.6 decline threshold) → ONNX-embedder nearest-centroid classifier →
|
||||
// 0.55 confidence gate. This is what a production turn takes when the model
|
||||
// server is out (docs/routing.md: pickLLMRouter degrades to the classifier).
|
||||
//
|
||||
// The classifier is seeded from models/seeds like the daemon's seedClassifier,
|
||||
// embedded with the real multilingual-e5-small model rather than the block
|
||||
// hash, so this is the closest headless reproduction of the authoritative
|
||||
// router output the slice-22 report can run.
|
||||
//
|
||||
// Requires the ONNX model files and a libonnxruntime.so. Pass the library via
|
||||
// the MAVEN_ONNX_LIB environment variable, exactly as the daemon does.
|
||||
func headsMain(poolPath, outPath string) {
|
||||
lib := os.Getenv("MAVEN_ONNX_LIB")
|
||||
if lib == "" {
|
||||
fmt.Fprintln(os.Stderr, "heads mode needs MAVEN_ONNX_LIB pointing at libonnxruntime.so")
|
||||
os.Exit(2)
|
||||
}
|
||||
const (
|
||||
embedModel = "models/embedder/multilingual-e5-small/model_quantized.onnx"
|
||||
tokPath = "models/embedder/multilingual-e5-small/tokenizer.json"
|
||||
headsModel = "models/embedder/router-heads/router_heads.onnx"
|
||||
)
|
||||
emb, err := router.NewONNXEmbedder(embedModel, tokPath, lib)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "heads: embedder: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer emb.Close()
|
||||
|
||||
cls := router.NewClassifier(emb)
|
||||
seedClassifier(cls)
|
||||
|
||||
heads, err := router.NewRouterHeads(headsModel, tokPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "heads: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer heads.Close()
|
||||
|
||||
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
|
||||
r := router.New(router.Config{
|
||||
Grammars: router.StageZeroGrammars(acts),
|
||||
Classifier: cls,
|
||||
Extractor: router.Extractor{
|
||||
Time: router.StubDateTimeParser{},
|
||||
Acts: acts,
|
||||
Facts: router.DefaultFactParser{},
|
||||
},
|
||||
Threshold: 0.55,
|
||||
Heads: heads,
|
||||
})
|
||||
runOverPool(r, poolPath, outPath)
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
// Seed loading replicated from cmd/mavend/voicewire.go (seedClassifier) and
|
||||
// internal/router/semantic/helpers_test.go, which build the same classifier
|
||||
// from models/seeds/<intent>.txt. The daemon and the eval fixture must agree
|
||||
// on the seeds; so must a measurement.
|
||||
const seedDir = "models/seeds"
|
||||
|
||||
var seedIntents = []router.Intent{
|
||||
router.IntentAct, router.IntentReminder, router.IntentFact,
|
||||
router.IntentNote, router.IntentQuery, router.IntentChat, router.IntentSystem,
|
||||
}
|
||||
|
||||
// buildMinimalRouter reproduces internal/router/semantic/buildMinimalRouter:
|
||||
// the daemon's grammar set, a hash-embedder classifier seeded from
|
||||
// models/seeds, and the deployed 0.55 threshold. Deterministic and
|
||||
// reproducible. The ONNX embedder and the routing heads score elsewhere;
|
||||
// this is the floor the eval fixture reports as the legacy baseline.
|
||||
func buildMinimalRouter() *router.Router {
|
||||
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
|
||||
cls := router.NewClassifier(router.NewHashEmbedder(1024))
|
||||
seedClassifier(cls)
|
||||
return router.New(router.Config{
|
||||
Grammars: router.StageZeroGrammars(acts),
|
||||
Classifier: cls,
|
||||
Extractor: router.Extractor{
|
||||
Time: router.StubDateTimeParser{},
|
||||
Acts: acts,
|
||||
Facts: router.DefaultFactParser{},
|
||||
},
|
||||
Threshold: 0.55,
|
||||
})
|
||||
}
|
||||
|
||||
func seedClassifier(c *router.Classifier) {
|
||||
// Walk up to find models/seeds like the daemon's seedPath, so the program
|
||||
// can run from any depth of the repo tree.
|
||||
dir := seedDir
|
||||
for i := 0; i < 5; i++ {
|
||||
if st, err := os.Stat(dir); err == nil && st.IsDir() {
|
||||
break
|
||||
}
|
||||
dir = filepath.Join("..", dir)
|
||||
}
|
||||
ctx := context.Background()
|
||||
total := 0
|
||||
for _, intent := range seedIntents {
|
||||
path := filepath.Join(dir, string(intent)+".txt")
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
log.Printf("legacy: open seed %s: %v", path, err)
|
||||
continue
|
||||
}
|
||||
sc := bufio.NewScanner(f)
|
||||
lines := []string{}
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, line)
|
||||
}
|
||||
f.Close()
|
||||
sort.Strings(lines)
|
||||
for _, line := range lines {
|
||||
if err := c.AddExample(ctx, intent, line); err != nil {
|
||||
log.Printf("legacy: seed %s %q: %v", intent, line, err)
|
||||
continue
|
||||
}
|
||||
total++
|
||||
}
|
||||
}
|
||||
log.Printf("legacy: loaded %d seed examples from %s", total, dir)
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// Legacy-baseline runner for slice 22: run the actual router cascade (stage 0
|
||||
// grammars → hash-embedder nearest-centroid classifier → 0.55 confidence gate)
|
||||
// over the frozen residual non-action dev pool and project each decision into
|
||||
// the five-way non-action semantic space.
|
||||
//
|
||||
// Projection rules (the daemon's behaviour, not just ScoreLegacy's):
|
||||
// - route error → uncertain
|
||||
// - Clarify=true (stage 3) → uncertain: the daemon asks, it does not commit
|
||||
// to a semantic bucket
|
||||
// - chat/query/fact+note/system → conversation/knowledge/memory_write/system
|
||||
// - act/reminder on a trusted non-action row → class "action" recorded
|
||||
// VERBATIM with illegal_action_prediction=true; never mapped to uncertain
|
||||
// - anything else → uncertain
|
||||
//
|
||||
// Reads /tmp/mvn-s22/pool.json (emit step) and writes /tmp/mvn-s22/legacy.json
|
||||
// with both the raw decision fields and the projected class, plus a summary
|
||||
// printout. No embedding is recomputed and no label is changed.
|
||||
|
||||
type poolRow struct {
|
||||
IDX int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
NText string `json:"n_text"`
|
||||
Route string `json:"route"`
|
||||
Tags []string `json:"tags"`
|
||||
CVFold int `json:"cv_fold"`
|
||||
SplitGroup string `json:"split_group"`
|
||||
FamilyID string `json:"family_id"`
|
||||
SourceID string `json:"source_id"`
|
||||
}
|
||||
|
||||
type legacyRow struct {
|
||||
IDX int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
Route string `json:"route"`
|
||||
Intent string `json:"intent"`
|
||||
Class string `json:"class"`
|
||||
Illegal bool `json:"illegal_action_prediction"`
|
||||
Confidence float64 `json:"confidence"`
|
||||
Stage int `json:"stage"`
|
||||
Clarify bool `json:"clarify"`
|
||||
Producer string `json:"producer"`
|
||||
Error string `json:"error,omitempty"`
|
||||
SourceID string `json:"source_id"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
var mode, poolPath, outPath string
|
||||
flag.StringVar(&mode, "mode", "legacy", "baseline mode: legacy (hash classifier) or heads (ONNX cascade minus LLM)")
|
||||
flag.StringVar(&poolPath, "pool", "/tmp/mvn-s22/pool.json", "emit-step pool.json")
|
||||
flag.StringVar(&outPath, "out", "/tmp/mvn-s22/legacy.json", "output path")
|
||||
flag.Parse()
|
||||
switch mode {
|
||||
case "legacy":
|
||||
legacyMain(poolPath, outPath)
|
||||
case "heads":
|
||||
headsMain(poolPath, outPath)
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "unknown -mode %q\n", mode)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func legacyMain(poolPath, outPath string) {
|
||||
runOverPool(buildMinimalRouter(), poolPath, outPath)
|
||||
}
|
||||
|
||||
func runOverPool(r *router.Router, poolPath, outPath string) {
|
||||
raw, err := os.ReadFile(poolPath)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var rows []poolRow
|
||||
if err := json.Unmarshal(raw, &rows); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
out := make([]legacyRow, 0, len(rows))
|
||||
classCount := map[string]int{}
|
||||
for _, pr := range rows {
|
||||
d, err := r.Route(ctx, router.NormalizedInput{Text: pr.Text}, now)
|
||||
lr := legacyRow{
|
||||
IDX: pr.IDX,
|
||||
Text: pr.Text,
|
||||
Route: pr.Route,
|
||||
SourceID: pr.SourceID,
|
||||
}
|
||||
if err != nil {
|
||||
lr.Class = "uncertain"
|
||||
lr.Error = err.Error()
|
||||
} else {
|
||||
lr.Intent = string(d.Intent)
|
||||
lr.Confidence = d.Confidence
|
||||
lr.Stage = d.Stage
|
||||
lr.Clarify = d.Clarify
|
||||
lr.Producer = string(d.Producer)
|
||||
}
|
||||
lr.Class, lr.Illegal = project(d, err)
|
||||
classCount[lr.Class]++
|
||||
out = append(out, lr)
|
||||
}
|
||||
|
||||
if err := writeJSON(outPath, out); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Printf("legacy baseline over %d residual non-action rows:\n", len(out))
|
||||
for _, c := range []string{"conversation", "knowledge", "memory_write", "system", "uncertain", "action"} {
|
||||
fmt.Printf(" %-14s %d (%.1f%%)\n", c, classCount[c], 100*float64(classCount[c])/float64(len(out)))
|
||||
}
|
||||
fmt.Printf(" illegal_action_prediction: %d\n", classCount["action"])
|
||||
// Grammar hits inside a corpus-residual population would be a
|
||||
// corpus/harness disagreement worth telling the report about: the corpus
|
||||
// marked each row not-fast-path-resolved, so a current stage-0 rule
|
||||
// resolving it means the corpus's fast-path mirror is stale or a grammar
|
||||
// landed after the corpus froze.
|
||||
gh := 0
|
||||
ghByRoute := map[string]int{}
|
||||
ghByIntent := map[string]int{}
|
||||
for _, lr := range out {
|
||||
if lr.Producer == string(router.RouteProducerGrammar) {
|
||||
gh++
|
||||
ghByRoute[lr.Route]++
|
||||
ghByIntent[lr.Intent]++
|
||||
}
|
||||
}
|
||||
fmt.Printf(" stage-0 grammar hits: %d\n", gh)
|
||||
if gh > 0 {
|
||||
fmt.Printf(" by ground-truth route: %v\n", ghByRoute)
|
||||
fmt.Printf(" by grammar intent: %v\n", ghByIntent)
|
||||
}
|
||||
}
|
||||
|
||||
// project maps the router's authoritative output into the five-way non-action
|
||||
// space, or to the "action" bucket verbatim when the router calls an act or a
|
||||
// reminder on a non-action row.
|
||||
func project(d router.Decision, err error) (string, bool) {
|
||||
if err != nil {
|
||||
return "uncertain", false
|
||||
}
|
||||
if d.Clarify {
|
||||
return "uncertain", false
|
||||
}
|
||||
switch d.Intent {
|
||||
case router.IntentChat:
|
||||
return "conversation", false
|
||||
case router.IntentQuery:
|
||||
return "knowledge", false
|
||||
case router.IntentFact, router.IntentNote:
|
||||
return "memory_write", false
|
||||
case router.IntentSystem:
|
||||
return "system", false
|
||||
case router.IntentAct, router.IntentReminder:
|
||||
return "action", true
|
||||
default:
|
||||
return "uncertain", false
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSON(path string, v any) error {
|
||||
fh, err := os.Create(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer fh.Close()
|
||||
w := bufio.NewWriter(fh)
|
||||
enc := json.NewEncoder(w)
|
||||
enc.SetIndent("", " ")
|
||||
if err := enc.Encode(v); err != nil {
|
||||
return err
|
||||
}
|
||||
return w.Flush()
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 22 emit: five-way residual non-action semantic router — data files
|
||||
========================================================================
|
||||
|
||||
Slice 21 accepted the deterministic execution-frame guard (docs/evals/
|
||||
2026-09-07-execution-frame-guard.md). Slice 22 returns to the coarse non-action
|
||||
router that the guard hands to: after TryFastPath misses and the guard passes,
|
||||
the remaining utterance is one of five non-action semantics — conversation,
|
||||
knowledge, memory_write, system, uncertain. Action rows never reach this
|
||||
router; they are usable only as out-of-domain probes, never in primary metrics.
|
||||
|
||||
This script only repackages the frozen dev pool for the Go legacy baseline and
|
||||
the Python experiment. It reuses slice 18's loader/filters and slice 19's
|
||||
normalizers verbatim, so the population here is the same one slices 18-21
|
||||
measured. It writes:
|
||||
|
||||
/tmp/mvn-s22/pool.json residual non-action dev rows: idx, text, n_text,
|
||||
route, tags, cv_fold, split_group, family_id,
|
||||
source_id (1652 rows)
|
||||
/tmp/mvn-s22/ood.json residual ACTION dev rows (766): same shape; OOD
|
||||
probes only, never primary metrics
|
||||
/tmp/mvn-s22/stats.json population summary (routes, families, folds)
|
||||
|
||||
idx is the row's position among dev_pool rows in dev-pool order, so the Python
|
||||
experiment can align the embedding vectors from /tmp/mvn-experiment/embeddings.json
|
||||
by index exactly as slice19.load_dev does.
|
||||
|
||||
No training happens here and no label is changed.
|
||||
|
||||
Population (verified 2026-09-08 from the frozen file):
|
||||
dev_pool 2490
|
||||
dev residual 2418 (= dev_pool minus fast_path_resolved)
|
||||
residual non-action 1652 knowledge 715 / memory_write 553 / system 184 /
|
||||
uncertain 107 / conversation 93
|
||||
residual action 766 (OOD probes only)
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402 (normalize_match_text, load_data, filters)
|
||||
|
||||
OUT_DIR = "/tmp/mvn-s22"
|
||||
|
||||
ROUTES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
|
||||
|
||||
def main():
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
print(f"dev pool: {len(dev)} rows "
|
||||
f"(meta declares dev_count={meta.get('dev_count')})")
|
||||
|
||||
rows = []
|
||||
for i, e in enumerate(dev):
|
||||
if not e["fast_path_resolved"]:
|
||||
rows.append({
|
||||
"idx": i,
|
||||
"text": e["text"],
|
||||
"n_text": slice18_sparse.normalize_match_text(e["text"]),
|
||||
"route": e["route"],
|
||||
"tags": sorted(set(e.get("tags", []))),
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"family_id": e["family_id"],
|
||||
"source_id": e["source_id"],
|
||||
})
|
||||
|
||||
na = [r for r in rows if r["route"] != "action"]
|
||||
ood = [r for r in rows if r["route"] == "action"]
|
||||
print(f"residual rows: {len(rows)} non-action: {len(na)} action(OOD): {len(ood)}")
|
||||
|
||||
by_route = {}
|
||||
for r in na:
|
||||
by_route[r["route"]] = by_route.get(r["route"], 0) + 1
|
||||
print("routes:", by_route)
|
||||
assert sum(by_route.values()) == len(na)
|
||||
assert set(ROUTES) == set(by_route), "route set must be the five-way"
|
||||
|
||||
by_family = {}
|
||||
for r in na:
|
||||
by_family[r["family_id"]] = by_family.get(r["family_id"], 0) + 1
|
||||
by_fold = {}
|
||||
for r in na:
|
||||
by_fold[r["cv_fold"]] = by_fold.get(r["cv_fold"], 0) + 1
|
||||
print(f"family_ids: {len(by_family)} split_groups: {len(set(r['split_group'] for r in na))}")
|
||||
print("folds:", by_fold)
|
||||
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
|
||||
json.dump(na, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "ood.json"), "w") as f:
|
||||
json.dump(ood, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "stats.json"), "w") as f:
|
||||
json.dump({
|
||||
"dev_count": len(dev),
|
||||
"residual_count": len(rows),
|
||||
"non_action_count": len(na),
|
||||
"action_ood_count": len(ood),
|
||||
"routes": by_route,
|
||||
"family_ids": len(by_family),
|
||||
"folds": by_fold,
|
||||
"top_family": dict(sorted(by_family.items(), key=lambda kv: -kv[1])[:15]),
|
||||
}, f, ensure_ascii=False, indent=1)
|
||||
print(f"wrote {OUT_DIR}/{{pool,ood,stats}}.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,560 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 22: five-way residual non-action semantic router (experiment)
|
||||
==================================================================
|
||||
|
||||
After TryFastPath misses and the ExecutionFrameGuard passes, the residual
|
||||
utterance is one of five non-action semantics: conversation, knowledge,
|
||||
memory_write, system, uncertain. This measures whether the deployed e5-small
|
||||
embeddings (384-d, query-prefixed, mean-pooled, L2, frozen) fed to a linear
|
||||
softmax head suffice, and how they compare to the legacy router, to floors,
|
||||
and to the deployed routing heads.
|
||||
|
||||
Population: the frozen dev-pool residual non-action rows (1652; the pool
|
||||
written by slice22_emit.py). Action rows (766) are out-of-domain probes only.
|
||||
|
||||
Metrics written to /tmp/mvn-s22/results.json:
|
||||
§1 population
|
||||
§2 legacy baseline (legacy.json / legacy_heads.json): acc, macro-F1,
|
||||
per-class P/R/F1, confusion, illegal_action_prediction count
|
||||
§3 e5-linear primary head: C grid, grouped CV OOF, per-fold P/R/F1 +
|
||||
variance + composition
|
||||
§4 floors: majority, centroid (cosine nearest-mean), sparse word+char
|
||||
TF-IDF logistic (slice18 builder), all grouped CV
|
||||
§5 route-family (family_id) leave-family-out
|
||||
§6 knowledge vs memory_write: matched pairs (water/homelab/task) ordering
|
||||
§7 uncertain as an explicit class: P/R/F1 + top confusions
|
||||
§8 OOF confidence: max-softmax correct/wrong, ECE, log-loss, Brier,
|
||||
coverage/accuracy/macro-F1 abstention curves (no threshold chosen)
|
||||
§9 action OOD probes: fold models applied to the 766 action rows
|
||||
§10 artifact cost: head params, serialized bytes, incremental head latency
|
||||
|
||||
No corpus label is changed. No frozen-holdout rows are inspected.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402
|
||||
import slice19_main # noqa: E402
|
||||
|
||||
EMB_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
OUT_DIR = "/tmp/mvn-s22"
|
||||
CLASSES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
CLASS_PREFIX = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
C_GRID = [0.1, 1.0, 10.0]
|
||||
|
||||
# Route-family holdouts the report calls out by name (slice-22 brief): every
|
||||
# family that is not part of the shared subject inventory on either side.
|
||||
HOLDOUT_GROUPS = {
|
||||
"capability": ["knowledge:capability-ha", "knowledge:capability-tool"],
|
||||
"world": ["knowledge:world-def", "knowledge:world-explain"],
|
||||
"calendar": ["knowledge:calendar", "knowledge:calendar-time", "knowledge:calendar-next"],
|
||||
"recall": ["knowledge:recall-fact", "knowledge:recall-note", "knowledge:recall-possessive"],
|
||||
"fact": ["fact:meal", "fact:water", "fact:sleep", "fact:shower", "fact:break", "fact:pills", "fact:exercise"],
|
||||
"note": ["note:idea", "note:homelab", "note:task"],
|
||||
"remember": ["free:remember"],
|
||||
"system": None, # all system:*
|
||||
"conversation": None,
|
||||
"uncertain": None,
|
||||
}
|
||||
|
||||
|
||||
def load_pool_and_embeds():
|
||||
with open(os.path.join(OUT_DIR, "pool.json")) as f:
|
||||
pool = json.load(f)
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
by_idx = {e["dev_idx"]: e for e in dev} if "dev_idx" in dev[0] else None
|
||||
# pool rows carry idx = position among dev_pool rows in dev order
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(dev)}
|
||||
for r in pool:
|
||||
r["emb"] = emb_by_idx[r["idx"]]
|
||||
r["y"] = r["route"]
|
||||
return pool, meta
|
||||
|
||||
|
||||
def oof_proba_grouped(X, y, folds, C=1.0):
|
||||
"""Grouped OOF probability matrix (n×5, class order CLASSES)."""
|
||||
y_idx = np.array([CLASSES.index(c) for c in y])
|
||||
folds = np.asarray(folds)
|
||||
proba = np.zeros((len(y_idx), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds.tolist())):
|
||||
tr = folds != te_fold
|
||||
te = folds == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], y_idx[tr])
|
||||
proba[te] = clf.predict_proba(X[te])
|
||||
return proba
|
||||
|
||||
|
||||
def cls_metrics(yt, yp):
|
||||
import sklearn.metrics as m
|
||||
yt = np.asarray(yt)
|
||||
yp = np.asarray(yp)
|
||||
if yt.dtype != np.int64 and yt.dtype != np.int32:
|
||||
yt = np.array([CLASSES.index(c) for c in yt])
|
||||
if yp.dtype != np.int64 and yp.dtype != np.int32:
|
||||
yp = np.array([CLASSES.index(c) for c in yp])
|
||||
labels = list(range(len(CLASSES)))
|
||||
n = len(yt)
|
||||
acc = m.accuracy_score(yt, yp)
|
||||
macro = m.f1_score(yt, yp, average="macro", labels=labels, zero_division=0)
|
||||
pr, rc, f1, sup = m.precision_recall_fscore_support(
|
||||
yt, yp, labels=labels, zero_division=0)
|
||||
per = {c: {"p": float(pr[i]), "r": float(rc[i]), "f1": float(f1[i]), "n": int(sup[i])}
|
||||
for i, c in enumerate(CLASSES)}
|
||||
conf = m.confusion_matrix(yt, yp, labels=labels).tolist()
|
||||
return {"n": n, "acc": acc, "macro_f1": macro, "per_class": per, "confusion": conf}
|
||||
|
||||
|
||||
def fold_report(yt, proba, folds, true_y):
|
||||
out = {}
|
||||
folds_arr = np.asarray(folds)
|
||||
comp = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yt_f = [CLASSES.index(y) for y in true_y[mask]]
|
||||
comp[f] = {c: int((np.array(true_y[mask]) == c).sum()) for c in CLASSES}
|
||||
per_fold = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yp = proba[mask].argmax(1).tolist()
|
||||
m = cls_metrics([yt[i] for i in np.where(mask)[0].tolist()], yp)
|
||||
per_fold[f] = {"acc": m["acc"], "macro_f1": m["macro_f1"]}
|
||||
out["composition"] = comp
|
||||
out["per_fold"] = per_fold
|
||||
accs = [v["acc"] for v in per_fold.values()]
|
||||
macros = [v["macro_f1"] for v in per_fold.values()]
|
||||
out["acc_mean"] = float(np.mean(accs))
|
||||
out["acc_std"] = float(np.std(accs))
|
||||
out["macro_f1_mean"] = float(np.mean(macros))
|
||||
out["macro_f1_std"] = float(np.std(macros))
|
||||
return out
|
||||
|
||||
|
||||
def ece(yt, proba, n_bins=15):
|
||||
conf = proba.max(1)
|
||||
pred = proba.argmax(1)
|
||||
acc = (pred == yt).astype(float)
|
||||
bins = np.linspace(0, 1, n_bins + 1)
|
||||
tot = 0.0
|
||||
details = []
|
||||
counts = 0
|
||||
for i in range(n_bins):
|
||||
lo, hi = bins[i], bins[i + 1]
|
||||
m = (conf >= lo) & (conf < hi) if i < n_bins - 1 else conf >= lo
|
||||
if m.sum() == 0:
|
||||
continue
|
||||
acc_m = acc[m].mean()
|
||||
conf_m = conf[m].mean()
|
||||
w = m.sum() / len(conf)
|
||||
tot += w * abs(acc_m - conf_m)
|
||||
counts += int(m.sum())
|
||||
details.append({"bin": i, "lo": lo, "hi": hi, "conf": float(conf_m),
|
||||
"acc": float(acc_m), "n": int(m.sum())})
|
||||
return {"ece": float(tot), "n_bins": n_bins, "counted": counts, "bins": details}
|
||||
|
||||
|
||||
def main():
|
||||
pool, meta = load_pool_and_embeds()
|
||||
pool.sort(key=lambda r: r["idx"])
|
||||
print(f"pool: {len(pool)} rows")
|
||||
|
||||
from sklearn.metrics import brier_score_loss, log_loss
|
||||
|
||||
report = {"population": {}, "legacy": {}, "e5_linear": {}, "floors": {},
|
||||
"family_holdouts": {}, "kmw": {}, "uncertain": {}, "confidence": {},
|
||||
"ood": {}, "artifact": {}}
|
||||
|
||||
# ── §1 population ──────────────────────────────────────────────────────
|
||||
cnt = {}
|
||||
for r in pool:
|
||||
cnt[r["y"]] = cnt.get(r["y"], 0) + 1
|
||||
report["population"] = {
|
||||
"n": len(pool),
|
||||
"routes": cnt,
|
||||
"family_ids": len(set(r["family_id"] for r in pool)),
|
||||
"split_groups": len(set(r["split_group"] for r in pool)),
|
||||
"folds": {str(f): int(sum(1 for r in pool if r["cv_fold"] == f)) for f in sorted(set(r["cv_fold"] for r in pool))},
|
||||
"corpus": {k: v for k, v in meta.items() if k in
|
||||
("dev_count", "residual_count", "fast_path_count",
|
||||
"dimension", "embedder_id", "input_template", "pooling", "normalization")},
|
||||
}
|
||||
print("\n§1 population:", report["population"])
|
||||
|
||||
X = np.vstack([r["emb"] for r in pool])
|
||||
y = np.array([r["y"] for r in pool])
|
||||
folds = np.array([r["cv_fold"] for r in pool])
|
||||
yt = np.array([CLASSES.index(c) for c in y])
|
||||
|
||||
# ── §2 legacy baselines ────────────────────────────────────────────────
|
||||
import collections
|
||||
for tag, fname in [("hash", "legacy.json"), ("heads", "legacy_heads.json")]:
|
||||
path = os.path.join(OUT_DIR, fname)
|
||||
if not os.path.exists(path):
|
||||
continue
|
||||
leg = json.load(open(path))
|
||||
leg_by_idx = {r["idx"]: r for r in leg}
|
||||
yp_leg = []
|
||||
illegal = []
|
||||
for r in pool:
|
||||
lr = leg_by_idx[r["idx"]]
|
||||
if lr["illegal_action_prediction"]:
|
||||
illegal.append(lr)
|
||||
yp_leg.append("action")
|
||||
else:
|
||||
yp_leg.append(lr["class"])
|
||||
yp_leg = np.array(yp_leg)
|
||||
# five-way: an 'action' prediction is an error (outside the label set)
|
||||
yp5 = np.array([("uncertain" if p == "action" else p) for p in yp_leg])
|
||||
m = cls_metrics(y, yp5)
|
||||
m["illegal_action_prediction"] = len(illegal)
|
||||
m["illegal_cases"] = [{"idx": i["idx"], "text": i["text"], "route": i["route"],
|
||||
"intent": i["intent"], "producer": i["producer"],
|
||||
"confidence": i["confidence"]} for i in illegal]
|
||||
# per-cell confusion also shows 'action' column
|
||||
conf_counts = collections.Counter(zip(y, yp_leg))
|
||||
m["confusion_with_action"] = {f"{a}->{b}": int(c) for (a, b), c in conf_counts.items()}
|
||||
report["legacy"][tag] = m
|
||||
print(f"\n§2 legacy ({tag}) acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f} "
|
||||
f"illegal={len(illegal)}")
|
||||
for c in CLASSES:
|
||||
p = m["per_class"][c]
|
||||
print(f" {c:<14} P={p['p']:.3f} R={p['r']:.3f} F1={p['f1']:.3f} n={p['n']}")
|
||||
|
||||
# grammar-pure residual: rows not resolved by any current stage-0 grammar
|
||||
if os.path.exists(os.path.join(OUT_DIR, "legacy.json")):
|
||||
leg = json.load(open(os.path.join(OUT_DIR, "legacy.json")))
|
||||
gh = {r["idx"] for r in leg if r["producer"] == "grammar"}
|
||||
gp_mask = np.array([r["idx"] not in gh for r in pool])
|
||||
report["grammar_drift"] = {
|
||||
"grammar_hits_in_pool": len(gh),
|
||||
"grammar_pure_n": int(gp_mask.sum()),
|
||||
}
|
||||
|
||||
# ── §3 e5-linear primary head ─────────────────────────────────────────
|
||||
print("\n§3 e5-linear")
|
||||
bestC, bestMac = 1.0, -1.0
|
||||
grid = {}
|
||||
for C in C_GRID:
|
||||
p = oof_proba_grouped(X, y, folds, C=C)
|
||||
mp = cls_metrics(y, p.argmax(1).tolist())
|
||||
grid[float(C)] = {"acc": mp["acc"], "macro_f1": mp["macro_f1"]}
|
||||
print(f" C={C} acc={mp['acc']:.4f} macroF1={mp['macro_f1']:.4f}")
|
||||
if mp["macro_f1"] > bestMac:
|
||||
bestMac, bestC = mp["macro_f1"], C
|
||||
print(f" -> best C={bestC}")
|
||||
p_best = oof_proba_grouped(X, y, folds, C=bestC)
|
||||
m_best = cls_metrics(y, p_best.argmax(1).tolist())
|
||||
m_best["C"] = bestC
|
||||
m_best["C_grid"] = grid
|
||||
m_best["folds"] = fold_report(yt, p_best, folds, y)
|
||||
report["e5_linear"] = m_best
|
||||
for f, v in m_best["folds"]["per_fold"].items():
|
||||
print(f" fold {f}: acc={v['acc']:.4f} macroF1={v['macro_f1']:.4f}")
|
||||
print(f" fold acc mean={m_best['folds']['acc_mean']:.4f} "
|
||||
f"std={m_best['folds']['acc_std']:.4f}; "
|
||||
f"macroF1 mean={m_best['folds']['macro_f1_mean']:.4f} "
|
||||
f"std={m_best['folds']['macro_f1_std']:.4f}")
|
||||
for c in CLASSES:
|
||||
p_ = m_best["per_class"][c]
|
||||
print(f" {c:<14} P={p_['p']:.3f} R={p_['r']:.3f} F1={p_['f1']:.3f} n={p_['n']}")
|
||||
|
||||
# grammar-pure sensitivity for the primary head
|
||||
if "grammar_drift" in report:
|
||||
mp_gp = cls_metrics(y[gp_mask], p_best[gp_mask].argmax(1).tolist())
|
||||
report["e5_linear"]["grammar_pure"] = {
|
||||
"acc": mp_gp["acc"], "macro_f1": mp_gp["macro_f1"], "n": int(gp_mask.sum())}
|
||||
|
||||
# ── §4 floors ─────────────────────────────────────────────────────────
|
||||
print("\n§4 floors")
|
||||
# majority floor
|
||||
maj = CLASSES.index("knowledge")
|
||||
ym = np.full(len(y), maj)
|
||||
mm = cls_metrics(y, ym)
|
||||
report["floors"]["majority"] = {"acc": mm["acc"], "macro_f1": mm["macro_f1"],
|
||||
"per_class": mm["per_class"]}
|
||||
print(f" majority (predict {CLASSES[maj]}): acc={mm['acc']:.4f} macroF1={mm['macro_f1']:.4f}")
|
||||
|
||||
# centroid floor: cosine to per-class mean of the training folds' embeddings
|
||||
cf_proba = np.zeros((len(yt), len(CLASSES)))
|
||||
folds_arr = np.asarray(folds)
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
centroids = []
|
||||
for c in CLASSES:
|
||||
idxs = np.where(tr & (y == c))[0]
|
||||
ctr = X[idxs].mean(axis=0)
|
||||
ctr = ctr / np.linalg.norm(ctr)
|
||||
centroids.append(ctr)
|
||||
Cm = np.vstack(centroids)
|
||||
sims = X[te] @ Cm.T
|
||||
cf_proba[te] = sims
|
||||
yc = cf_proba.argmax(1)
|
||||
# accuracy + macroF1 with the same 5-way
|
||||
mc = cls_metrics(y, yc.tolist())
|
||||
report["floors"]["centroid"] = {"acc": mc["acc"], "macro_f1": mc["macro_f1"],
|
||||
"per_class": mc["per_class"]}
|
||||
print(f" centroid cosine: acc={mc['acc']:.4f} macroF1={mc['macro_f1']:.4f}")
|
||||
|
||||
# sparse word+char logistic (slice18 builder, grouped CV, five-way)
|
||||
texts = [r["n_text"] for r in pool]
|
||||
Xs, _vec = slice18_sparse.build_features(texts, "both")
|
||||
psp = np.zeros((len(yt), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(Xs[tr], yt[tr])
|
||||
psp[te] = clf.predict_proba(Xs[te])
|
||||
msp = cls_metrics(y, psp.argmax(1).tolist())
|
||||
report["floors"]["sparse_word_char"] = {
|
||||
"acc": msp["acc"], "macro_f1": msp["macro_f1"], "per_class": msp["per_class"],
|
||||
"vocab": slice18_sparse.vocab_size(_vec)}
|
||||
print(f" sparse both: acc={msp['acc']:.4f} macroF1={msp['macro_f1']:.4f} "
|
||||
f"vocab={report['floors']['sparse_word_char']['vocab']}")
|
||||
|
||||
# ── §5 route-family holdouts ─────────────────────────────────────────
|
||||
print("\n§5 route-family holdouts")
|
||||
fam = np.array([r["family_id"] for r in pool])
|
||||
holdouts = {}
|
||||
all_fams = sorted(set(fam.tolist()))
|
||||
for grp, fams in HOLDOUT_GROUPS.items():
|
||||
if fams is None:
|
||||
fams = [f for f in all_fams if f.startswith(grp + ":")]
|
||||
mask = np.isin(fam, fams)
|
||||
if mask.sum() == 0:
|
||||
continue
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypgrp = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypgrp.tolist())
|
||||
m["families"] = fams
|
||||
m["rows"] = int(mask.sum())
|
||||
holdouts[grp] = {"acc": m["acc"], "macro_f1": m["macro_f1"], "n": int(mask.sum()),
|
||||
"per_class": m["per_class"]}
|
||||
print(f" {grp:<14} n={m['rows']} acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f}")
|
||||
# full leave-one-family-out summary
|
||||
lofo_accs = []
|
||||
lofo_f1s = []
|
||||
for f in all_fams:
|
||||
mask = fam == f
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypf = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypf.tolist())
|
||||
lofo_accs.append(m["acc"])
|
||||
lofo_f1s.append(m["macro_f1"])
|
||||
holdouts["_all_49_lo_"] = {"n_families": len(all_fams),
|
||||
"acc_mean": float(np.mean(lofo_accs)),
|
||||
"macro_f1_mean": float(np.mean(lofo_f1s))}
|
||||
report["family_holdouts"] = holdouts
|
||||
print(f" leave-one-family-out over {len(all_fams)} families: "
|
||||
f"acc mean={np.mean(lofo_accs):.4f} macroF1 mean={np.mean(lofo_f1s):.4f}")
|
||||
|
||||
# ── §6 knowledge vs memory_write ─────────────────────────────────────
|
||||
print("\n§6 knowledge vs memory_write")
|
||||
# reuse e5-linear OOF: does the model put the higher probability on the
|
||||
# right side (memory_write for a write, knowledge for a recall)?
|
||||
conf_km = np.zeros((2, 2))
|
||||
pk = p_best[:, CLASSES.index("knowledge")]
|
||||
pmw = p_best[:, CLASSES.index("memory_write")]
|
||||
for i in range(len(yt)):
|
||||
t = y[i]
|
||||
if t == "knowledge":
|
||||
conf_km[0, 1 if pmw[i] > pk[i] else 0] += 1
|
||||
elif t == "memory_write":
|
||||
conf_km[1, 1 if pmw[i] >= pk[i] else 0] += 1
|
||||
report["kmw"] = {"confusion_p_ordered": conf_km.tolist()}
|
||||
|
||||
# matched pairs with shared subject lexemes, corpus-justified
|
||||
def build_pairs(subject, fam_k, fam_mw):
|
||||
kr = [r for r in pool if r["family_id"] in fam_k]
|
||||
mr = [r for r in pool if r["family_id"] in fam_mw]
|
||||
pairs = []
|
||||
for mw in mr:
|
||||
for k in kr:
|
||||
if subject in mw["n_text"] and subject in k["n_text"]:
|
||||
pairs.append((mw["idx"], k["idx"], mw["n_text"], k["n_text"]))
|
||||
return pairs
|
||||
|
||||
sets = {
|
||||
"water": build_pairs("вод", ["knowledge:recall-fact"], ["fact:water"]),
|
||||
"homelab": build_pairs("dns", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("сервер", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("vlan", ["knowledge:homelab-status"], ["note:homelab"]),
|
||||
"task": build_pairs("задач", ["knowledge:task-check", "knowledge:deadline"],
|
||||
["note:task"]),
|
||||
}
|
||||
idx_of = {r["idx"]: i for i, r in enumerate(pool)}
|
||||
pair_rep = {}
|
||||
for name, pairs in sets.items():
|
||||
if not pairs:
|
||||
continue
|
||||
ok = 0
|
||||
margins = []
|
||||
bad = []
|
||||
for mi, ki, mx, kx in pairs:
|
||||
mi_i, ki_i = idx_of[mi], idx_of[ki]
|
||||
# MW row should get a higher memory_write probability than the K row
|
||||
mk = (pmw[mi_i] + 0.0)
|
||||
if pmw[mi_i] > pmw[ki_i]:
|
||||
ok += 1
|
||||
else:
|
||||
bad.append((mx[:46], round(float(pmw[mi_i]), 3), kx[:46], round(float(pmw[ki_i]), 3)))
|
||||
margins.append(pmw[mi_i] - pmw[ki_i])
|
||||
pair_rep[name] = {
|
||||
"pairs": len(pairs),
|
||||
"mw_over_k_order_acc": ok / len(pairs),
|
||||
"mean_margin": float(np.mean(margins)),
|
||||
"reversed_examples": bad[:6],
|
||||
}
|
||||
print(f" {name}: pairs={len(pairs)} order_acc={ok/len(pairs):.3f} "
|
||||
f"mean_margin={np.mean(margins):+.3f}")
|
||||
report["kmw"]["matched_pairs"] = pair_rep
|
||||
|
||||
# ── §7 uncertain as explicit class ────────────────────────────────────
|
||||
print("\n§7 uncertain")
|
||||
up = m_best["per_class"]["uncertain"]
|
||||
uc = m_best["confusion"][CLASSES.index("uncertain")]
|
||||
report["uncertain"] = {
|
||||
"per_class": up,
|
||||
"row_from_uncertain": {CLASSES[j]: int(uc[j]) for j in range(5)},
|
||||
"row_to_uncertain": {CLASSES[j]: int(m_best["confusion"][j][CLASSES.index("uncertain")])
|
||||
for j in range(5)},
|
||||
}
|
||||
print(f" uncertain n={up['n']} P={up['p']:.3f} R={up['r']:.3f} F1={up['f1']:.3f}")
|
||||
print(" wrong-→label pulled from uncertain:", report["uncertain"]["row_from_uncertain"])
|
||||
print(" →uncertain pulled from:", report["uncertain"]["row_to_uncertain"])
|
||||
|
||||
# ── §8 OOF confidence / calibration / abstention ─────────────────────
|
||||
print("\n§8 confidence / calibration")
|
||||
conf = p_best.max(1)
|
||||
right = (p_best.argmax(1) == yt)
|
||||
cer = {
|
||||
"correct_conf_mean": float(conf[right].mean()),
|
||||
"correct_conf_median": float(np.median(conf[right])),
|
||||
"wrong_conf_mean": float(conf[~right].mean()),
|
||||
"wrong_conf_median": float(np.median(conf[~right])),
|
||||
"ece": ece(yt, p_best)["ece"],
|
||||
"ece_bins": ece(yt, p_best)["bins"],
|
||||
"log_loss": float(log_loss(yt, p_best, labels=[0, 1, 2, 3, 4])),
|
||||
}
|
||||
# Brier is label-set specific: one-vs-rest mean
|
||||
briers = []
|
||||
for i in range(5):
|
||||
briers.append(brier_score_loss((yt == i).astype(int), p_best[:, i]))
|
||||
cer["brier_macro"] = float(np.mean(briers))
|
||||
report["confidence"] = cer
|
||||
print(f" right conf mean={cer['correct_conf_mean']:.3f} "
|
||||
f"wrong conf mean={cer['wrong_conf_mean']:.3f} ECE={cer['ece']:.4f}")
|
||||
print(f" log_loss={cer['log_loss']:.4f} brier_macro={cer['brier_macro']:.4f}")
|
||||
|
||||
thr_grid = np.linspace(0.10, 0.98, 45)
|
||||
abst = []
|
||||
for t in thr_grid:
|
||||
cov = (conf >= t).mean()
|
||||
if cov == 0:
|
||||
continue
|
||||
keep = conf >= t
|
||||
yt_k = yt[keep]
|
||||
yp_k = p_best[keep].argmax(1)
|
||||
mk_ = cls_metrics(yt_k.tolist(), yp_k.tolist())
|
||||
abst.append({"threshold": round(float(t), 3), "coverage": float(cov),
|
||||
"accuracy": mk_["acc"], "macro_f1": mk_["macro_f1"]})
|
||||
report["confidence"]["abstention_curve"] = abst
|
||||
print(" threshold | coverage | accuracy | macroF1 (first 6/45 + knee)")
|
||||
for row in abst[::9]:
|
||||
print(f" {row['threshold']:.2f} | {row['coverage']:.3f} | "
|
||||
f"{row['accuracy']:.3f} | {row['macro_f1']:.3f}")
|
||||
|
||||
# ── §9 action OOD probes ──────────────────────────────────────────────
|
||||
print("\n§9 action OOD")
|
||||
ood_rows = [r for r in json.load(open(os.path.join(OUT_DIR, "ood.json")))]
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(slice18_sparse.filter_dev_pool(
|
||||
slice18_sparse.load_data()[1]))}
|
||||
Xo = np.vstack([emb_by_idx[r["idx"]] for r in ood_rows])
|
||||
fold_models = []
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
fold_models.append(clf)
|
||||
# OOD rows are not in folds; use the full-train model to keep it simple and
|
||||
# comparable to the non-action in-fold behaviour
|
||||
po = np.zeros((len(Xo), 5))
|
||||
for clf in fold_models:
|
||||
po += clf.predict_proba(Xo)
|
||||
po /= len(fold_models)
|
||||
ood_top = int(np.argmax(po.mean(0)))
|
||||
ood_conf = po.max(1)
|
||||
ood_pred = po.argmax(1)
|
||||
top_dist = {CLASSES[i]: int((ood_pred == i).sum()) for i in range(5)}
|
||||
confident_na = int((ood_conf > 0.9).sum())
|
||||
report["ood"] = {
|
||||
"n": len(ood_rows),
|
||||
"top_class": CLASSES[int(ood_top)],
|
||||
"top_class_dist": top_dist,
|
||||
"conf_gt_0.9": confident_na,
|
||||
"conf_gt_0.9_frac": float(confident_na / len(ood_rows)),
|
||||
"conf_mean": float(ood_conf.mean()),
|
||||
"conf_median": float(np.median(ood_conf)),
|
||||
}
|
||||
print(f" action OOD n={len(ood_rows)}: most-confident class={report['ood']['top_class']} "
|
||||
f"dist={top_dist}")
|
||||
print(f" conf>0.9: {confident_na} ({confident_na/len(ood_rows):.3f}) "
|
||||
f"conf mean={report['ood']['conf_mean']:.3f}")
|
||||
|
||||
# ── §10 artifact cost ────────────────────────────────────────────────
|
||||
print("\n§10 artifact")
|
||||
n_params = len(CLASSES) * X.shape[1] + len(CLASSES)
|
||||
fp32 = n_params * 4
|
||||
report["artifact"] = {
|
||||
"e5_dim": X.shape[1],
|
||||
"head_params": n_params,
|
||||
"head_fp32_bytes": fp32,
|
||||
"head_fp32_kib": fp32 / 1024,
|
||||
"head_int8_bytes": n_params,
|
||||
}
|
||||
# incremental latency of the linear head over a batch of 1 (µs)
|
||||
clf = slice18_sparse.LogisticRegression(C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X, yt)
|
||||
x1 = X[:1]
|
||||
for _ in range(50):
|
||||
clf.predict_proba(x1)
|
||||
lat = []
|
||||
for _ in range(2000):
|
||||
t0 = time.perf_counter_ns()
|
||||
clf.predict_proba(x1)
|
||||
lat.append((time.perf_counter_ns() - t0) / 1e3)
|
||||
lat = np.array(lat)
|
||||
report["artifact"]["head_latency_us_mean"] = float(lat.mean())
|
||||
report["artifact"]["head_latency_us_p50"] = float(np.median(lat))
|
||||
print(f" head params={n_params} fp32={fp32/1024:.2f}KiB "
|
||||
f"lat mean={lat.mean():.2f}us p50={np.median(lat):.2f}us")
|
||||
|
||||
with open(os.path.join(OUT_DIR, "results.json"), "w") as f:
|
||||
json.dump(report, f, ensure_ascii=False, indent=1, default=float)
|
||||
print(f"\nwrote {OUT_DIR}/results.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,243 @@
|
||||
// slice23 — fast-path metadata reconciliation diagnostic.
|
||||
//
|
||||
// Classifies every disagreement between the corpus's stored fast_path_resolved
|
||||
// flag and what the production fast path derives today (TryFastPath over the
|
||||
// stage-0 grammars with the experiment's act allowlist). Outputs a JSON
|
||||
// decomposition and a console summary for docs/evals reports.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/semantic-router-experiment/slice23/ -out /tmp/mvn-s23/drift.json
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
type conflict struct {
|
||||
Text string `json:"text"`
|
||||
SourceID string `json:"source_id"`
|
||||
Route string `json:"route"`
|
||||
Source string `json:"source"`
|
||||
Group string `json:"split_group"`
|
||||
Tags []string `json:"tags,omitempty"`
|
||||
Dev bool `json:"dev"`
|
||||
// Direction: claimed_fast_now_miss = stored fast, runtime residual;
|
||||
// mirror_missed = stored residual, runtime fast.
|
||||
Direction string `json:"direction"`
|
||||
Grammar string `json:"grammar,omitempty"`
|
||||
// ShapeDeclined is true when at least one stage-0 grammar matched the
|
||||
// utterance's shape but refused the content (falls through like the router).
|
||||
ShapeDeclined bool `json:"shape_declined"`
|
||||
// DeclinedGrammars names every stage-0 grammar that matched the shape but
|
||||
// declined the content, for claimed_fast_now_miss rows.
|
||||
DeclinedGrammars []string `json:"declined_grammars,omitempty"`
|
||||
}
|
||||
|
||||
type report struct {
|
||||
Meta metaSummary `json:"meta"`
|
||||
Pop popSummary `json:"population"`
|
||||
Conflicts []conflict `json:"conflicts"`
|
||||
ByRoute map[string]map[string]int `json:"by_route"`
|
||||
ByGrammar map[string]int `json:"by_grammar"`
|
||||
BySource map[string]map[string]int `json:"by_source"`
|
||||
ByFamily map[string]map[string]int `json:"by_family"`
|
||||
Direction map[string]int `json:"by_direction"`
|
||||
Declined int `json:"claimed_fast_with_declined_shape"`
|
||||
NoShape int `json:"claimed_fast_with_no_shape"`
|
||||
}
|
||||
|
||||
type metaSummary struct {
|
||||
Total int `json:"total"`
|
||||
DevCount int `json:"dev_count"`
|
||||
Frozen int `json:"frozen_count"`
|
||||
}
|
||||
|
||||
type popSummary struct {
|
||||
StoredFast int `json:"stored_fast"`
|
||||
StoredResid int `json:"stored_residual"`
|
||||
DerivedFast int `json:"derived_fast"`
|
||||
DerivedResid int `json:"derived_residual"`
|
||||
// Dev-pool residual route counts derived as the router sees them today.
|
||||
DevResidualByRoute map[string]int `json:"dev_residual_by_route"`
|
||||
// Dev-pool residual non-action + action OOD as derived.
|
||||
DevResidualNonAction int `json:"dev_residual_non_action"`
|
||||
DevResidualAction int `json:"dev_residual_action"`
|
||||
// Fast rows in the dev pool, derived.
|
||||
DevFast int `json:"dev_fast"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
outPath := flag.String("out", "/tmp/mvn-s23/drift.json", "output JSON path")
|
||||
flag.Parse()
|
||||
|
||||
exs, err := semantic.LoadCorpus()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "load corpus: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
_, dev, _ := semantic.FrozenHoldoutSplit(exs)
|
||||
devSet := make(map[string]bool, len(dev))
|
||||
for _, e := range dev {
|
||||
devSet[e.SourceID] = true
|
||||
}
|
||||
|
||||
// Stage-zero grammar list for shape/declined attribution (same list the
|
||||
// derivation walks).
|
||||
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
|
||||
gs := router.StageZeroGrammars(acts)
|
||||
|
||||
var (
|
||||
conflicts []conflict
|
||||
byRoute = map[string]map[string]int{}
|
||||
byGrammar = map[string]int{}
|
||||
bySource = map[string]map[string]int{}
|
||||
byFamily = map[string]map[string]int{}
|
||||
byDirection = map[string]int{}
|
||||
storedFast, derivedFast, declined, noShape int
|
||||
devResidByRoute = map[string]int{}
|
||||
devResidNonAct, devResidAct, devFast int
|
||||
)
|
||||
|
||||
for _, e := range exs {
|
||||
o := semantic.DeriveFastPath(e.Text)
|
||||
inDev := devSet[e.SourceID]
|
||||
|
||||
st := e.FastPathResolved
|
||||
if st {
|
||||
storedFast++
|
||||
}
|
||||
if o.Matched {
|
||||
derivedFast++
|
||||
}
|
||||
|
||||
var c *conflict
|
||||
switch {
|
||||
case st && o.Matched:
|
||||
case st && !o.Matched:
|
||||
// Stored fast but the runtime misses. Attribute why.
|
||||
shapeDeclined := false
|
||||
var declinedNames []string
|
||||
for _, g := range gs {
|
||||
_, matched, ok := g.Evaluate(e.Text)
|
||||
if matched && !ok {
|
||||
shapeDeclined = true
|
||||
declinedNames = append(declinedNames, g.Name)
|
||||
}
|
||||
}
|
||||
if shapeDeclined {
|
||||
declined++
|
||||
} else {
|
||||
noShape++
|
||||
}
|
||||
c = &conflict{Direction: "claimed_fast_now_miss", ShapeDeclined: shapeDeclined, DeclinedGrammars: declinedNames}
|
||||
case !st && o.Matched:
|
||||
c = &conflict{Direction: "mirror_missed", Grammar: o.Grammar}
|
||||
}
|
||||
|
||||
if c != nil {
|
||||
c.Text = e.Text
|
||||
c.SourceID = e.SourceID
|
||||
c.Route = string(e.Route)
|
||||
c.Source = e.Source
|
||||
c.Group = e.SplitGroup
|
||||
c.Tags = e.Tags
|
||||
c.Dev = inDev
|
||||
conflicts = append(conflicts, *c)
|
||||
byDirection[c.Direction]++
|
||||
byGrammar[c.Grammar]++
|
||||
if byRoute[c.Direction] == nil {
|
||||
byRoute[c.Direction] = map[string]int{}
|
||||
}
|
||||
byRoute[c.Direction][c.Route]++
|
||||
if bySource[c.Direction] == nil {
|
||||
bySource[c.Direction] = map[string]int{}
|
||||
}
|
||||
bySource[c.Direction][c.Source]++
|
||||
if byFamily[c.Direction] == nil {
|
||||
byFamily[c.Direction] = map[string]int{}
|
||||
}
|
||||
byFamily[c.Direction][c.Group]++
|
||||
}
|
||||
|
||||
if inDev {
|
||||
if o.Matched {
|
||||
devFast++
|
||||
} else {
|
||||
devResidByRoute[string(e.Route)]++
|
||||
if e.Route == semantic.RouteAction {
|
||||
devResidAct++
|
||||
} else {
|
||||
devResidNonAct++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sort.Slice(conflicts, func(i, j int) bool { return conflicts[i].SourceID < conflicts[j].SourceID })
|
||||
|
||||
rep := report{
|
||||
Meta: metaSummary{Total: len(exs), DevCount: len(dev), Frozen: len(exs) - len(dev)},
|
||||
Pop: popSummary{
|
||||
StoredFast: storedFast, StoredResid: len(exs) - storedFast,
|
||||
DerivedFast: derivedFast, DerivedResid: len(exs) - derivedFast,
|
||||
DevResidualByRoute: devResidByRoute,
|
||||
DevResidualNonAction: devResidNonAct, DevResidualAction: devResidAct,
|
||||
DevFast: devFast,
|
||||
},
|
||||
Conflicts: conflicts,
|
||||
ByRoute: byRoute, ByGrammar: byGrammar, BySource: bySource, ByFamily: byFamily,
|
||||
Direction: byDirection, Declined: declined, NoShape: noShape,
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(rep, "", " ")
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "marshal: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := os.WriteFile(*outPath, data, 0644); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "write %s: %v\n", *outPath, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Printf("total %d (dev %d, frozen %d)\n", rep.Meta.Total, rep.Meta.DevCount, rep.Meta.Frozen)
|
||||
fmt.Printf("stored fast=%d residual=%d\n", rep.Pop.StoredFast, rep.Pop.StoredResid)
|
||||
fmt.Printf("derived fast=%d residual=%d\n", rep.Pop.DerivedFast, rep.Pop.DerivedResid)
|
||||
fmt.Printf("disagreements total %d\n", len(conflicts))
|
||||
for _, d := range []string{"claimed_fast_now_miss", "mirror_missed"} {
|
||||
fmt.Printf(" %-22s %d\n", d, byDirection[d])
|
||||
if d == "claimed_fast_now_miss" {
|
||||
fmt.Printf(" with declined shape: %d no shape: %d\n", declined, noShape)
|
||||
}
|
||||
}
|
||||
fmt.Println(" mirror-missed by grammar:")
|
||||
for _, k := range sortedKeys(byGrammar) {
|
||||
fmt.Printf(" %-28s %d\n", k, byGrammar[k])
|
||||
}
|
||||
fmt.Println(" by route:")
|
||||
for _, d := range sortedKeys(byRoute) {
|
||||
fmt.Printf(" %-22s %v\n", d, byRoute[d])
|
||||
}
|
||||
fmt.Printf("dev pool derived: fast=%d residual=%d (non-action=%d action=%d)\n",
|
||||
rep.Pop.DevFast, rep.Pop.DevResidualNonAction+rep.Pop.DevResidualAction,
|
||||
rep.Pop.DevResidualNonAction, rep.Pop.DevResidualAction)
|
||||
fmt.Printf("dev residual by route: %v\n", rep.Pop.DevResidualByRoute)
|
||||
fmt.Printf("wrote %s\n", *outPath)
|
||||
}
|
||||
|
||||
func sortedKeys[T any](m map[string]T) []string {
|
||||
ks := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
ks = append(ks, k)
|
||||
}
|
||||
sort.Strings(ks)
|
||||
return ks
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 23 emit: five-way residual non-action semantic router — data files
|
||||
========================================================================
|
||||
|
||||
Slice 23 reconciles corpus fast-path metadata with the production router
|
||||
(TryFastPath over stage-0 grammars). The corpus builder no longer mirrors the
|
||||
grammars by hand; fast_path_resolved is derived from the router, so this emit
|
||||
flags exactly the rows the router genuinely leaves for the general cascade.
|
||||
|
||||
This script only repackages the frozen dev pool for the Go legacy baseline and
|
||||
the Python experiment, writing into /tmp/mvn-s23 so the slice-22 artifacts
|
||||
stay untouched. Logic is slice22_emit.py verbatim; only OUT_DIR differs.
|
||||
|
||||
/tmp/mvn-s23/pool.json residual non-action dev rows: idx, text, n_text,
|
||||
route, tags, cv_fold, split_group, family_id,
|
||||
source_id (1509 rows)
|
||||
/tmp/mvn-s23/ood.json residual ACTION dev rows (720): same shape; OOD
|
||||
probes only, never primary metrics
|
||||
/tmp/mvn-s23/stats.json population summary (routes, families, folds)
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402 (normalize_match_text, load_data, filters)
|
||||
|
||||
OUT_DIR = "/tmp/mvn-s23"
|
||||
|
||||
ROUTES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
|
||||
|
||||
def main():
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
print(f"dev pool: {len(dev)} rows "
|
||||
f"(meta declares dev_count={meta.get('dev_count')})")
|
||||
|
||||
rows = []
|
||||
for i, e in enumerate(dev):
|
||||
if not e["fast_path_resolved"]:
|
||||
rows.append({
|
||||
"idx": i,
|
||||
"text": e["text"],
|
||||
"n_text": slice18_sparse.normalize_match_text(e["text"]),
|
||||
"route": e["route"],
|
||||
"tags": sorted(set(e.get("tags", []))),
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"family_id": e["family_id"],
|
||||
"source_id": e["source_id"],
|
||||
})
|
||||
|
||||
na = [r for r in rows if r["route"] != "action"]
|
||||
ood = [r for r in rows if r["route"] == "action"]
|
||||
print(f"residual rows: {len(rows)} non-action: {len(na)} action(OOD): {len(ood)}")
|
||||
|
||||
by_route = {}
|
||||
for r in na:
|
||||
by_route[r["route"]] = by_route.get(r["route"], 0) + 1
|
||||
print("routes:", by_route)
|
||||
assert sum(by_route.values()) == len(na)
|
||||
assert set(ROUTES) == set(by_route), "route set must be the five-way"
|
||||
|
||||
by_family = {}
|
||||
for r in na:
|
||||
by_family[r["family_id"]] = by_family.get(r["family_id"], 0) + 1
|
||||
by_fold = {}
|
||||
for r in na:
|
||||
by_fold[r["cv_fold"]] = by_fold.get(r["cv_fold"], 0) + 1
|
||||
print(f"family_ids: {len(by_family)} split_groups: {len(set(r['split_group'] for r in na))}")
|
||||
print("folds:", by_fold)
|
||||
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
|
||||
json.dump(na, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "ood.json"), "w") as f:
|
||||
json.dump(ood, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "stats.json"), "w") as f:
|
||||
json.dump({
|
||||
"dev_count": len(dev),
|
||||
"residual_count": len(rows),
|
||||
"non_action_count": len(na),
|
||||
"action_ood_count": len(ood),
|
||||
"routes": by_route,
|
||||
"family_ids": len(by_family),
|
||||
"folds": by_fold,
|
||||
"top_family": dict(sorted(by_family.items(), key=lambda kv: -kv[1])[:15]),
|
||||
}, f, ensure_ascii=False, indent=1)
|
||||
print(f"wrote {OUT_DIR}/{{pool,ood,stats}}.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,559 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 23: five-way residual non-action semantic router — corrected population
|
||||
============================================================================
|
||||
|
||||
Slice 22 reported the corpus's fast-path mirror was stale next to the
|
||||
production stage-0 grammars (185 residual rows resolved at runtime). Slice 23
|
||||
derives fast_path_resolved from the real router, rebuilds the corpus, and
|
||||
re-measures the primary slice-22 results on the corrected residual pool.
|
||||
Logic and configs are slice22_main.py verbatim; only OUT_DIR differs.
|
||||
|
||||
Population: the corrected dev-pool residual non-action rows (1509; the pool
|
||||
written by slice23_emit.py). Corrected action rows (720) are OOD probes only.
|
||||
|
||||
Metrics written to /tmp/mvn-s23/results.json:
|
||||
§1 population
|
||||
§2 legacy baseline (legacy.json / legacy_heads.json): acc, macro-F1,
|
||||
per-class P/R/F1, confusion, illegal_action_prediction count
|
||||
§3 e5-linear primary head: C grid, grouped CV OOF, per-fold P/R/F1 +
|
||||
variance + composition
|
||||
§4 floors: majority, centroid (cosine nearest-mean), sparse word+char
|
||||
TF-IDF logistic (slice18 builder), all grouped CV
|
||||
§5 route-family (family_id) leave-family-out
|
||||
§6 knowledge vs memory_write: matched pairs (water/homelab/task) ordering
|
||||
§7 uncertain as an explicit class: P/R/F1 + top confusions
|
||||
§8 OOF confidence: max-softmax correct/wrong, ECE, log-loss, Brier,
|
||||
coverage/accuracy/macro-F1 abstention curves (no threshold chosen)
|
||||
§9 action OOD probes: fold models applied to the corrected action rows
|
||||
§10 artifact cost: head params, serialized bytes, incremental head latency
|
||||
|
||||
No corpus label is changed. No frozen-holdout rows are inspected.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402
|
||||
import slice19_main # noqa: E402
|
||||
|
||||
EMB_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
OUT_DIR = "/tmp/mvn-s23"
|
||||
CLASSES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
CLASS_PREFIX = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
C_GRID = [0.1, 1.0, 10.0]
|
||||
|
||||
# Route-family holdouts the report calls out by name (slice-22 brief): every
|
||||
# family that is not part of the shared subject inventory on either side.
|
||||
HOLDOUT_GROUPS = {
|
||||
"capability": ["knowledge:capability-ha", "knowledge:capability-tool"],
|
||||
"world": ["knowledge:world-def", "knowledge:world-explain"],
|
||||
"calendar": ["knowledge:calendar", "knowledge:calendar-time", "knowledge:calendar-next"],
|
||||
"recall": ["knowledge:recall-fact", "knowledge:recall-note", "knowledge:recall-possessive"],
|
||||
"fact": ["fact:meal", "fact:water", "fact:sleep", "fact:shower", "fact:break", "fact:pills", "fact:exercise"],
|
||||
"note": ["note:idea", "note:homelab", "note:task"],
|
||||
"remember": ["free:remember"],
|
||||
"system": None, # all system:*
|
||||
"conversation": None,
|
||||
"uncertain": None,
|
||||
}
|
||||
|
||||
|
||||
def load_pool_and_embeds():
|
||||
with open(os.path.join(OUT_DIR, "pool.json")) as f:
|
||||
pool = json.load(f)
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
by_idx = {e["dev_idx"]: e for e in dev} if "dev_idx" in dev[0] else None
|
||||
# pool rows carry idx = position among dev_pool rows in dev order
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(dev)}
|
||||
for r in pool:
|
||||
r["emb"] = emb_by_idx[r["idx"]]
|
||||
r["y"] = r["route"]
|
||||
return pool, meta
|
||||
|
||||
|
||||
def oof_proba_grouped(X, y, folds, C=1.0):
|
||||
"""Grouped OOF probability matrix (n×5, class order CLASSES)."""
|
||||
y_idx = np.array([CLASSES.index(c) for c in y])
|
||||
folds = np.asarray(folds)
|
||||
proba = np.zeros((len(y_idx), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds.tolist())):
|
||||
tr = folds != te_fold
|
||||
te = folds == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], y_idx[tr])
|
||||
proba[te] = clf.predict_proba(X[te])
|
||||
return proba
|
||||
|
||||
|
||||
def cls_metrics(yt, yp):
|
||||
import sklearn.metrics as m
|
||||
yt = np.asarray(yt)
|
||||
yp = np.asarray(yp)
|
||||
if yt.dtype != np.int64 and yt.dtype != np.int32:
|
||||
yt = np.array([CLASSES.index(c) for c in yt])
|
||||
if yp.dtype != np.int64 and yp.dtype != np.int32:
|
||||
yp = np.array([CLASSES.index(c) for c in yp])
|
||||
labels = list(range(len(CLASSES)))
|
||||
n = len(yt)
|
||||
acc = m.accuracy_score(yt, yp)
|
||||
macro = m.f1_score(yt, yp, average="macro", labels=labels, zero_division=0)
|
||||
pr, rc, f1, sup = m.precision_recall_fscore_support(
|
||||
yt, yp, labels=labels, zero_division=0)
|
||||
per = {c: {"p": float(pr[i]), "r": float(rc[i]), "f1": float(f1[i]), "n": int(sup[i])}
|
||||
for i, c in enumerate(CLASSES)}
|
||||
conf = m.confusion_matrix(yt, yp, labels=labels).tolist()
|
||||
return {"n": n, "acc": acc, "macro_f1": macro, "per_class": per, "confusion": conf}
|
||||
|
||||
|
||||
def fold_report(yt, proba, folds, true_y):
|
||||
out = {}
|
||||
folds_arr = np.asarray(folds)
|
||||
comp = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yt_f = [CLASSES.index(y) for y in true_y[mask]]
|
||||
comp[f] = {c: int((np.array(true_y[mask]) == c).sum()) for c in CLASSES}
|
||||
per_fold = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yp = proba[mask].argmax(1).tolist()
|
||||
m = cls_metrics([yt[i] for i in np.where(mask)[0].tolist()], yp)
|
||||
per_fold[f] = {"acc": m["acc"], "macro_f1": m["macro_f1"]}
|
||||
out["composition"] = comp
|
||||
out["per_fold"] = per_fold
|
||||
accs = [v["acc"] for v in per_fold.values()]
|
||||
macros = [v["macro_f1"] for v in per_fold.values()]
|
||||
out["acc_mean"] = float(np.mean(accs))
|
||||
out["acc_std"] = float(np.std(accs))
|
||||
out["macro_f1_mean"] = float(np.mean(macros))
|
||||
out["macro_f1_std"] = float(np.std(macros))
|
||||
return out
|
||||
|
||||
|
||||
def ece(yt, proba, n_bins=15):
|
||||
conf = proba.max(1)
|
||||
pred = proba.argmax(1)
|
||||
acc = (pred == yt).astype(float)
|
||||
bins = np.linspace(0, 1, n_bins + 1)
|
||||
tot = 0.0
|
||||
details = []
|
||||
counts = 0
|
||||
for i in range(n_bins):
|
||||
lo, hi = bins[i], bins[i + 1]
|
||||
m = (conf >= lo) & (conf < hi) if i < n_bins - 1 else conf >= lo
|
||||
if m.sum() == 0:
|
||||
continue
|
||||
acc_m = acc[m].mean()
|
||||
conf_m = conf[m].mean()
|
||||
w = m.sum() / len(conf)
|
||||
tot += w * abs(acc_m - conf_m)
|
||||
counts += int(m.sum())
|
||||
details.append({"bin": i, "lo": lo, "hi": hi, "conf": float(conf_m),
|
||||
"acc": float(acc_m), "n": int(m.sum())})
|
||||
return {"ece": float(tot), "n_bins": n_bins, "counted": counts, "bins": details}
|
||||
|
||||
|
||||
def main():
|
||||
pool, meta = load_pool_and_embeds()
|
||||
pool.sort(key=lambda r: r["idx"])
|
||||
print(f"pool: {len(pool)} rows")
|
||||
|
||||
from sklearn.metrics import brier_score_loss, log_loss
|
||||
|
||||
report = {"population": {}, "legacy": {}, "e5_linear": {}, "floors": {},
|
||||
"family_holdouts": {}, "kmw": {}, "uncertain": {}, "confidence": {},
|
||||
"ood": {}, "artifact": {}}
|
||||
|
||||
# ── §1 population ──────────────────────────────────────────────────────
|
||||
cnt = {}
|
||||
for r in pool:
|
||||
cnt[r["y"]] = cnt.get(r["y"], 0) + 1
|
||||
report["population"] = {
|
||||
"n": len(pool),
|
||||
"routes": cnt,
|
||||
"family_ids": len(set(r["family_id"] for r in pool)),
|
||||
"split_groups": len(set(r["split_group"] for r in pool)),
|
||||
"folds": {str(f): int(sum(1 for r in pool if r["cv_fold"] == f)) for f in sorted(set(r["cv_fold"] for r in pool))},
|
||||
"corpus": {k: v for k, v in meta.items() if k in
|
||||
("dev_count", "residual_count", "fast_path_count",
|
||||
"dimension", "embedder_id", "input_template", "pooling", "normalization")},
|
||||
}
|
||||
print("\n§1 population:", report["population"])
|
||||
|
||||
X = np.vstack([r["emb"] for r in pool])
|
||||
y = np.array([r["y"] for r in pool])
|
||||
folds = np.array([r["cv_fold"] for r in pool])
|
||||
yt = np.array([CLASSES.index(c) for c in y])
|
||||
|
||||
# ── §2 legacy baselines ────────────────────────────────────────────────
|
||||
import collections
|
||||
for tag, fname in [("hash", "legacy.json"), ("heads", "legacy_heads.json")]:
|
||||
path = os.path.join(OUT_DIR, fname)
|
||||
if not os.path.exists(path):
|
||||
continue
|
||||
leg = json.load(open(path))
|
||||
leg_by_idx = {r["idx"]: r for r in leg}
|
||||
yp_leg = []
|
||||
illegal = []
|
||||
for r in pool:
|
||||
lr = leg_by_idx[r["idx"]]
|
||||
if lr["illegal_action_prediction"]:
|
||||
illegal.append(lr)
|
||||
yp_leg.append("action")
|
||||
else:
|
||||
yp_leg.append(lr["class"])
|
||||
yp_leg = np.array(yp_leg)
|
||||
# five-way: an 'action' prediction is an error (outside the label set)
|
||||
yp5 = np.array([("uncertain" if p == "action" else p) for p in yp_leg])
|
||||
m = cls_metrics(y, yp5)
|
||||
m["illegal_action_prediction"] = len(illegal)
|
||||
m["illegal_cases"] = [{"idx": i["idx"], "text": i["text"], "route": i["route"],
|
||||
"intent": i["intent"], "producer": i["producer"],
|
||||
"confidence": i["confidence"]} for i in illegal]
|
||||
# per-cell confusion also shows 'action' column
|
||||
conf_counts = collections.Counter(zip(y, yp_leg))
|
||||
m["confusion_with_action"] = {f"{a}->{b}": int(c) for (a, b), c in conf_counts.items()}
|
||||
report["legacy"][tag] = m
|
||||
print(f"\n§2 legacy ({tag}) acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f} "
|
||||
f"illegal={len(illegal)}")
|
||||
for c in CLASSES:
|
||||
p = m["per_class"][c]
|
||||
print(f" {c:<14} P={p['p']:.3f} R={p['r']:.3f} F1={p['f1']:.3f} n={p['n']}")
|
||||
|
||||
# grammar-pure residual: rows not resolved by any current stage-0 grammar
|
||||
if os.path.exists(os.path.join(OUT_DIR, "legacy.json")):
|
||||
leg = json.load(open(os.path.join(OUT_DIR, "legacy.json")))
|
||||
gh = {r["idx"] for r in leg if r["producer"] == "grammar"}
|
||||
gp_mask = np.array([r["idx"] not in gh for r in pool])
|
||||
report["grammar_drift"] = {
|
||||
"grammar_hits_in_pool": len(gh),
|
||||
"grammar_pure_n": int(gp_mask.sum()),
|
||||
}
|
||||
|
||||
# ── §3 e5-linear primary head ─────────────────────────────────────────
|
||||
print("\n§3 e5-linear")
|
||||
bestC, bestMac = 1.0, -1.0
|
||||
grid = {}
|
||||
for C in C_GRID:
|
||||
p = oof_proba_grouped(X, y, folds, C=C)
|
||||
mp = cls_metrics(y, p.argmax(1).tolist())
|
||||
grid[float(C)] = {"acc": mp["acc"], "macro_f1": mp["macro_f1"]}
|
||||
print(f" C={C} acc={mp['acc']:.4f} macroF1={mp['macro_f1']:.4f}")
|
||||
if mp["macro_f1"] > bestMac:
|
||||
bestMac, bestC = mp["macro_f1"], C
|
||||
print(f" -> best C={bestC}")
|
||||
p_best = oof_proba_grouped(X, y, folds, C=bestC)
|
||||
m_best = cls_metrics(y, p_best.argmax(1).tolist())
|
||||
m_best["C"] = bestC
|
||||
m_best["C_grid"] = grid
|
||||
m_best["folds"] = fold_report(yt, p_best, folds, y)
|
||||
report["e5_linear"] = m_best
|
||||
for f, v in m_best["folds"]["per_fold"].items():
|
||||
print(f" fold {f}: acc={v['acc']:.4f} macroF1={v['macro_f1']:.4f}")
|
||||
print(f" fold acc mean={m_best['folds']['acc_mean']:.4f} "
|
||||
f"std={m_best['folds']['acc_std']:.4f}; "
|
||||
f"macroF1 mean={m_best['folds']['macro_f1_mean']:.4f} "
|
||||
f"std={m_best['folds']['macro_f1_std']:.4f}")
|
||||
for c in CLASSES:
|
||||
p_ = m_best["per_class"][c]
|
||||
print(f" {c:<14} P={p_['p']:.3f} R={p_['r']:.3f} F1={p_['f1']:.3f} n={p_['n']}")
|
||||
|
||||
# grammar-pure sensitivity for the primary head
|
||||
if "grammar_drift" in report:
|
||||
mp_gp = cls_metrics(y[gp_mask], p_best[gp_mask].argmax(1).tolist())
|
||||
report["e5_linear"]["grammar_pure"] = {
|
||||
"acc": mp_gp["acc"], "macro_f1": mp_gp["macro_f1"], "n": int(gp_mask.sum())}
|
||||
|
||||
# ── §4 floors ─────────────────────────────────────────────────────────
|
||||
print("\n§4 floors")
|
||||
# majority floor
|
||||
maj = CLASSES.index("knowledge")
|
||||
ym = np.full(len(y), maj)
|
||||
mm = cls_metrics(y, ym)
|
||||
report["floors"]["majority"] = {"acc": mm["acc"], "macro_f1": mm["macro_f1"],
|
||||
"per_class": mm["per_class"]}
|
||||
print(f" majority (predict {CLASSES[maj]}): acc={mm['acc']:.4f} macroF1={mm['macro_f1']:.4f}")
|
||||
|
||||
# centroid floor: cosine to per-class mean of the training folds' embeddings
|
||||
cf_proba = np.zeros((len(yt), len(CLASSES)))
|
||||
folds_arr = np.asarray(folds)
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
centroids = []
|
||||
for c in CLASSES:
|
||||
idxs = np.where(tr & (y == c))[0]
|
||||
ctr = X[idxs].mean(axis=0)
|
||||
ctr = ctr / np.linalg.norm(ctr)
|
||||
centroids.append(ctr)
|
||||
Cm = np.vstack(centroids)
|
||||
sims = X[te] @ Cm.T
|
||||
cf_proba[te] = sims
|
||||
yc = cf_proba.argmax(1)
|
||||
# accuracy + macroF1 with the same 5-way
|
||||
mc = cls_metrics(y, yc.tolist())
|
||||
report["floors"]["centroid"] = {"acc": mc["acc"], "macro_f1": mc["macro_f1"],
|
||||
"per_class": mc["per_class"]}
|
||||
print(f" centroid cosine: acc={mc['acc']:.4f} macroF1={mc['macro_f1']:.4f}")
|
||||
|
||||
# sparse word+char logistic (slice18 builder, grouped CV, five-way)
|
||||
texts = [r["n_text"] for r in pool]
|
||||
Xs, _vec = slice18_sparse.build_features(texts, "both")
|
||||
psp = np.zeros((len(yt), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(Xs[tr], yt[tr])
|
||||
psp[te] = clf.predict_proba(Xs[te])
|
||||
msp = cls_metrics(y, psp.argmax(1).tolist())
|
||||
report["floors"]["sparse_word_char"] = {
|
||||
"acc": msp["acc"], "macro_f1": msp["macro_f1"], "per_class": msp["per_class"],
|
||||
"vocab": slice18_sparse.vocab_size(_vec)}
|
||||
print(f" sparse both: acc={msp['acc']:.4f} macroF1={msp['macro_f1']:.4f} "
|
||||
f"vocab={report['floors']['sparse_word_char']['vocab']}")
|
||||
|
||||
# ── §5 route-family holdouts ─────────────────────────────────────────
|
||||
print("\n§5 route-family holdouts")
|
||||
fam = np.array([r["family_id"] for r in pool])
|
||||
holdouts = {}
|
||||
all_fams = sorted(set(fam.tolist()))
|
||||
for grp, fams in HOLDOUT_GROUPS.items():
|
||||
if fams is None:
|
||||
fams = [f for f in all_fams if f.startswith(grp + ":")]
|
||||
mask = np.isin(fam, fams)
|
||||
if mask.sum() == 0:
|
||||
continue
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypgrp = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypgrp.tolist())
|
||||
m["families"] = fams
|
||||
m["rows"] = int(mask.sum())
|
||||
holdouts[grp] = {"acc": m["acc"], "macro_f1": m["macro_f1"], "n": int(mask.sum()),
|
||||
"per_class": m["per_class"]}
|
||||
print(f" {grp:<14} n={m['rows']} acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f}")
|
||||
# full leave-one-family-out summary
|
||||
lofo_accs = []
|
||||
lofo_f1s = []
|
||||
for f in all_fams:
|
||||
mask = fam == f
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypf = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypf.tolist())
|
||||
lofo_accs.append(m["acc"])
|
||||
lofo_f1s.append(m["macro_f1"])
|
||||
holdouts["_all_49_lo_"] = {"n_families": len(all_fams),
|
||||
"acc_mean": float(np.mean(lofo_accs)),
|
||||
"macro_f1_mean": float(np.mean(lofo_f1s))}
|
||||
report["family_holdouts"] = holdouts
|
||||
print(f" leave-one-family-out over {len(all_fams)} families: "
|
||||
f"acc mean={np.mean(lofo_accs):.4f} macroF1 mean={np.mean(lofo_f1s):.4f}")
|
||||
|
||||
# ── §6 knowledge vs memory_write ─────────────────────────────────────
|
||||
print("\n§6 knowledge vs memory_write")
|
||||
# reuse e5-linear OOF: does the model put the higher probability on the
|
||||
# right side (memory_write for a write, knowledge for a recall)?
|
||||
conf_km = np.zeros((2, 2))
|
||||
pk = p_best[:, CLASSES.index("knowledge")]
|
||||
pmw = p_best[:, CLASSES.index("memory_write")]
|
||||
for i in range(len(yt)):
|
||||
t = y[i]
|
||||
if t == "knowledge":
|
||||
conf_km[0, 1 if pmw[i] > pk[i] else 0] += 1
|
||||
elif t == "memory_write":
|
||||
conf_km[1, 1 if pmw[i] >= pk[i] else 0] += 1
|
||||
report["kmw"] = {"confusion_p_ordered": conf_km.tolist()}
|
||||
|
||||
# matched pairs with shared subject lexemes, corpus-justified
|
||||
def build_pairs(subject, fam_k, fam_mw):
|
||||
kr = [r for r in pool if r["family_id"] in fam_k]
|
||||
mr = [r for r in pool if r["family_id"] in fam_mw]
|
||||
pairs = []
|
||||
for mw in mr:
|
||||
for k in kr:
|
||||
if subject in mw["n_text"] and subject in k["n_text"]:
|
||||
pairs.append((mw["idx"], k["idx"], mw["n_text"], k["n_text"]))
|
||||
return pairs
|
||||
|
||||
sets = {
|
||||
"water": build_pairs("вод", ["knowledge:recall-fact"], ["fact:water"]),
|
||||
"homelab": build_pairs("dns", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("сервер", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("vlan", ["knowledge:homelab-status"], ["note:homelab"]),
|
||||
"task": build_pairs("задач", ["knowledge:task-check", "knowledge:deadline"],
|
||||
["note:task"]),
|
||||
}
|
||||
idx_of = {r["idx"]: i for i, r in enumerate(pool)}
|
||||
pair_rep = {}
|
||||
for name, pairs in sets.items():
|
||||
if not pairs:
|
||||
continue
|
||||
ok = 0
|
||||
margins = []
|
||||
bad = []
|
||||
for mi, ki, mx, kx in pairs:
|
||||
mi_i, ki_i = idx_of[mi], idx_of[ki]
|
||||
# MW row should get a higher memory_write probability than the K row
|
||||
mk = (pmw[mi_i] + 0.0)
|
||||
if pmw[mi_i] > pmw[ki_i]:
|
||||
ok += 1
|
||||
else:
|
||||
bad.append((mx[:46], round(float(pmw[mi_i]), 3), kx[:46], round(float(pmw[ki_i]), 3)))
|
||||
margins.append(pmw[mi_i] - pmw[ki_i])
|
||||
pair_rep[name] = {
|
||||
"pairs": len(pairs),
|
||||
"mw_over_k_order_acc": ok / len(pairs),
|
||||
"mean_margin": float(np.mean(margins)),
|
||||
"reversed_examples": bad[:6],
|
||||
}
|
||||
print(f" {name}: pairs={len(pairs)} order_acc={ok/len(pairs):.3f} "
|
||||
f"mean_margin={np.mean(margins):+.3f}")
|
||||
report["kmw"]["matched_pairs"] = pair_rep
|
||||
|
||||
# ── §7 uncertain as explicit class ────────────────────────────────────
|
||||
print("\n§7 uncertain")
|
||||
up = m_best["per_class"]["uncertain"]
|
||||
uc = m_best["confusion"][CLASSES.index("uncertain")]
|
||||
report["uncertain"] = {
|
||||
"per_class": up,
|
||||
"row_from_uncertain": {CLASSES[j]: int(uc[j]) for j in range(5)},
|
||||
"row_to_uncertain": {CLASSES[j]: int(m_best["confusion"][j][CLASSES.index("uncertain")])
|
||||
for j in range(5)},
|
||||
}
|
||||
print(f" uncertain n={up['n']} P={up['p']:.3f} R={up['r']:.3f} F1={up['f1']:.3f}")
|
||||
print(" wrong-→label pulled from uncertain:", report["uncertain"]["row_from_uncertain"])
|
||||
print(" →uncertain pulled from:", report["uncertain"]["row_to_uncertain"])
|
||||
|
||||
# ── §8 OOF confidence / calibration / abstention ─────────────────────
|
||||
print("\n§8 confidence / calibration")
|
||||
conf = p_best.max(1)
|
||||
right = (p_best.argmax(1) == yt)
|
||||
cer = {
|
||||
"correct_conf_mean": float(conf[right].mean()),
|
||||
"correct_conf_median": float(np.median(conf[right])),
|
||||
"wrong_conf_mean": float(conf[~right].mean()),
|
||||
"wrong_conf_median": float(np.median(conf[~right])),
|
||||
"ece": ece(yt, p_best)["ece"],
|
||||
"ece_bins": ece(yt, p_best)["bins"],
|
||||
"log_loss": float(log_loss(yt, p_best, labels=[0, 1, 2, 3, 4])),
|
||||
}
|
||||
# Brier is label-set specific: one-vs-rest mean
|
||||
briers = []
|
||||
for i in range(5):
|
||||
briers.append(brier_score_loss((yt == i).astype(int), p_best[:, i]))
|
||||
cer["brier_macro"] = float(np.mean(briers))
|
||||
report["confidence"] = cer
|
||||
print(f" right conf mean={cer['correct_conf_mean']:.3f} "
|
||||
f"wrong conf mean={cer['wrong_conf_mean']:.3f} ECE={cer['ece']:.4f}")
|
||||
print(f" log_loss={cer['log_loss']:.4f} brier_macro={cer['brier_macro']:.4f}")
|
||||
|
||||
thr_grid = np.linspace(0.10, 0.98, 45)
|
||||
abst = []
|
||||
for t in thr_grid:
|
||||
cov = (conf >= t).mean()
|
||||
if cov == 0:
|
||||
continue
|
||||
keep = conf >= t
|
||||
yt_k = yt[keep]
|
||||
yp_k = p_best[keep].argmax(1)
|
||||
mk_ = cls_metrics(yt_k.tolist(), yp_k.tolist())
|
||||
abst.append({"threshold": round(float(t), 3), "coverage": float(cov),
|
||||
"accuracy": mk_["acc"], "macro_f1": mk_["macro_f1"]})
|
||||
report["confidence"]["abstention_curve"] = abst
|
||||
print(" threshold | coverage | accuracy | macroF1 (first 6/45 + knee)")
|
||||
for row in abst[::9]:
|
||||
print(f" {row['threshold']:.2f} | {row['coverage']:.3f} | "
|
||||
f"{row['accuracy']:.3f} | {row['macro_f1']:.3f}")
|
||||
|
||||
# ── §9 action OOD probes ──────────────────────────────────────────────
|
||||
print("\n§9 action OOD")
|
||||
ood_rows = [r for r in json.load(open(os.path.join(OUT_DIR, "ood.json")))]
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(slice18_sparse.filter_dev_pool(
|
||||
slice18_sparse.load_data()[1]))}
|
||||
Xo = np.vstack([emb_by_idx[r["idx"]] for r in ood_rows])
|
||||
fold_models = []
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
fold_models.append(clf)
|
||||
# XX
|
||||
# comparable to the non-action in-fold behaviour
|
||||
po = np.zeros((len(Xo), 5))
|
||||
for clf in fold_models:
|
||||
po += clf.predict_proba(Xo)
|
||||
po /= len(fold_models)
|
||||
ood_top = int(np.argmax(po.mean(0)))
|
||||
ood_conf = po.max(1)
|
||||
ood_pred = po.argmax(1)
|
||||
top_dist = {CLASSES[i]: int((ood_pred == i).sum()) for i in range(5)}
|
||||
confident_na = int((ood_conf > 0.9).sum())
|
||||
report["ood"] = {
|
||||
"n": len(ood_rows),
|
||||
"top_class": CLASSES[int(ood_top)],
|
||||
"top_class_dist": top_dist,
|
||||
"conf_gt_0.9": confident_na,
|
||||
"conf_gt_0.9_frac": float(confident_na / len(ood_rows)),
|
||||
"conf_mean": float(ood_conf.mean()),
|
||||
"conf_median": float(np.median(ood_conf)),
|
||||
}
|
||||
print(f" action OOD n={len(ood_rows)}: most-confident class={report['ood']['top_class']} "
|
||||
f"dist={top_dist}")
|
||||
print(f" conf>0.9: {confident_na} ({confident_na/len(ood_rows):.3f}) "
|
||||
f"conf mean={report['ood']['conf_mean']:.3f}")
|
||||
|
||||
# ── §10 artifact cost ────────────────────────────────────────────────
|
||||
print("\n§10 artifact")
|
||||
n_params = len(CLASSES) * X.shape[1] + len(CLASSES)
|
||||
fp32 = n_params * 4
|
||||
report["artifact"] = {
|
||||
"e5_dim": X.shape[1],
|
||||
"head_params": n_params,
|
||||
"head_fp32_bytes": fp32,
|
||||
"head_fp32_kib": fp32 / 1024,
|
||||
"head_int8_bytes": n_params,
|
||||
}
|
||||
# incremental latency of the linear head over a batch of 1 (µs)
|
||||
clf = slice18_sparse.LogisticRegression(C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X, yt)
|
||||
x1 = X[:1]
|
||||
for _ in range(50):
|
||||
clf.predict_proba(x1)
|
||||
lat = []
|
||||
for _ in range(2000):
|
||||
t0 = time.perf_counter_ns()
|
||||
clf.predict_proba(x1)
|
||||
lat.append((time.perf_counter_ns() - t0) / 1e3)
|
||||
lat = np.array(lat)
|
||||
report["artifact"]["head_latency_us_mean"] = float(lat.mean())
|
||||
report["artifact"]["head_latency_us_p50"] = float(np.median(lat))
|
||||
print(f" head params={n_params} fp32={fp32/1024:.2f}KiB "
|
||||
f"lat mean={lat.mean():.2f}us p50={np.median(lat):.2f}us")
|
||||
|
||||
with open(os.path.join(OUT_DIR, "results.json"), "w") as f:
|
||||
json.dump(report, f, ensure_ascii=False, indent=1, default=float)
|
||||
print(f"\nwrote {OUT_DIR}/results.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,293 @@
|
||||
{
|
||||
"db_path": "/var/lib/maven/maven.db.enc",
|
||||
"db_tmpfs": "/dev/shm/maven-plain.db",
|
||||
"db_key_env": "MAVEN_DB_KEY",
|
||||
"socket_path": "/run/maven/mavend.sock",
|
||||
"state_dir": "/var/lib/maven",
|
||||
|
||||
"//disabled_rules": [
|
||||
"Nudge rules that are not wired at all. Names come from loop.DefaultRules:",
|
||||
"water, meal, break, service_down, netdata_critical.",
|
||||
"service_down is back on: mavpoll now writes one fact per kuma monitor",
|
||||
"(service_down:<name>), so the nudge names the service and pausing a monitor",
|
||||
"in kuma silences that monitor. It is also edge-triggered, so a service that",
|
||||
"stays down is one nudge, not one every fifteen minutes."
|
||||
],
|
||||
"disabled_rules": [],
|
||||
|
||||
"phraser": {
|
||||
"model_path": "/opt/maven/models/llm/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf",
|
||||
"bin_path": "llama-server",
|
||||
"n_gpu_layers": 99,
|
||||
"n_ctx": 4096,
|
||||
"cache_ram_mib": 512,
|
||||
"timeout": "60s",
|
||||
"llm_nudges": false
|
||||
},
|
||||
|
||||
"//ntfy": [
|
||||
"The second reach (V-649). Until 07-08-2026 telegram was the only one, and",
|
||||
"telegram needs api.telegram.org, the socks relay below and a matching ufw",
|
||||
"rule — three things in series that have each failed once, and when they do",
|
||||
"a sev4 nudge has nowhere to go. ntfy shares none of them: it is reached",
|
||||
"directly, no relay.",
|
||||
"It is the preferred away reach. Reminders fall back to Telegram in order,",
|
||||
"and every missing reach is recorded in the outbox rather than disappearing.",
|
||||
"The credential is an ntfy access token, scoped write-only to this one",
|
||||
"topic, so a popped sink can push to it and cannot read it back. Set it in",
|
||||
"deploy/telegram.env beside the telegram secrets; that file is gitignored."
|
||||
],
|
||||
"ntfy": {
|
||||
"disabled": true,
|
||||
"base_url": "https://ntfy.kvmx.ru",
|
||||
"topic": "maven",
|
||||
"token": "${NTFY_TOKEN}"
|
||||
},
|
||||
|
||||
"telegram": {
|
||||
"bot_token": "${TELEGRAM_BOT_TOKEN}",
|
||||
"chat_id": "${TELEGRAM_CHAT_ID}",
|
||||
|
||||
"//proxy": [
|
||||
"api.telegram.org is not reachable directly from this box, so every send",
|
||||
"timed out. The relay is the x-ui socks inbound on the host, port 10808;",
|
||||
"192.168.240.1 is the maven_default bridge gateway, which is how a",
|
||||
"container addresses the host. mavend is on that network.",
|
||||
"This needs a matching ufw rule or the container's SYN is dropped:",
|
||||
" ufw allow from 192.168.240.0/20 to any port 10808 proto tcp"
|
||||
],
|
||||
"proxy": "socks5://192.168.240.1:10808",
|
||||
"//intake": [
|
||||
"Read the chat as well as write to it (V-637). The poller long-polls",
|
||||
"getUpdates through the same relay and accepts chat_id as the only",
|
||||
"sender. Deleting this key turns inbound off again.",
|
||||
"chat_id must be numeric here or the daemon refuses to start: an inbound",
|
||||
"update names its chat by number, so an @-name would match nothing."
|
||||
],
|
||||
"intake": true
|
||||
},
|
||||
|
||||
"//workstation": [
|
||||
"The big model on the desk PC (workpc, 7900 GRE 16GB), fronted by",
|
||||
"mavgpud on port 8080. It runs gemma-4-12b and it is preferred over the",
|
||||
"resident Qwen3-1.7B for routing and replies whenever the card is free.",
|
||||
"The machine is never assumed up: it sleeps, and the card is often held by",
|
||||
"a CPT run, in which case mavgpud answers 503 and Maven falls back to the",
|
||||
"resident model without saying so. Deleting this block restores exactly",
|
||||
"the behaviour homesrv had before it existed.",
|
||||
"Addressed by LAN address, not container name: mavgpud runs on another",
|
||||
"machine and there is no shared docker network to name it on.",
|
||||
"model_disabled keeps only this model arm dark until MAVEN_GPU_TOKEN is",
|
||||
"provisioned; the independently authenticated STT arm below stays live."
|
||||
],
|
||||
"//workstation.stt": [
|
||||
"CrisperWhisper 2.0 turbo on the same machine, a second service on port",
|
||||
"8081 and not a second endpoint on mavgpud. whisper.cpp cannot load CW2 at",
|
||||
"all: it derives its language count from the vocabulary size, and CW2's",
|
||||
"51897 tokens shift seven special token ids. So it runs under transformers",
|
||||
"there and mavsttd stays whisper.cpp here.",
|
||||
"Worth the second service: CW2 turbo scores 10.4% WER in Russian against",
|
||||
"27.5% for the ggml-small.bin mavsttd loads, measured on 200 Golos clips",
|
||||
"in docs/evals/2026-08-09-crisperwhisper2-russian-wer.md.",
|
||||
"Deleting this block sends every utterance to mavsttd, which is what the",
|
||||
"box did before it existed. A worse transcript is still a turn, so the",
|
||||
"fallback is silent and Kami is never told which machine heard him.",
|
||||
"The token is what stops anything on the LAN posting audio to that port."
|
||||
],
|
||||
"workstation": {
|
||||
"model_disabled": true,
|
||||
"url": "http://192.168.1.105:8080",
|
||||
"token": "${MAVEN_GPU_TOKEN}",
|
||||
"probe": "15s",
|
||||
"timeout": "90s",
|
||||
"stt": {
|
||||
"url": "http://192.168.1.105:8081/transcribe",
|
||||
"token": "${MAVEN_STT_TOKEN}",
|
||||
"probe": "15s",
|
||||
"timeout": "10s"
|
||||
}
|
||||
},
|
||||
|
||||
"//search": [
|
||||
"The live web, searched after his own notes and before Kiwix. Only the",
|
||||
"query string leaves the box — never a note, a fact, the persona block or",
|
||||
"the history — and a question about him never reaches here at all.",
|
||||
"The instance must have `json` in search.formats (settings.yml); a stock",
|
||||
"SearXNG answers 403 to format=json and every search then fails. It is",
|
||||
"addressed by container name, so it needs the same maven_default",
|
||||
"attachment kiwix has, and it must listen on 9563: 8080 is taken several",
|
||||
"times over on this box. No instance reachable ⇒ she falls through to the",
|
||||
"ZIMs and never says the search failed."
|
||||
],
|
||||
"search": {
|
||||
"url": "http://searxng:9563",
|
||||
"max_results": 4,
|
||||
"snippet_runes": 1500,
|
||||
"language": "auto",
|
||||
"timeout": "8s"
|
||||
},
|
||||
|
||||
"//kiwix": [
|
||||
"The offline encyclopedia, searched after his own notes and before anything",
|
||||
"on the network. kiwix-server publishes 8034 on loopback only, so a container",
|
||||
"cannot reach it by address; it is attached to the maven_default network",
|
||||
"instead and addressed by container name. That attachment is imperative and",
|
||||
"does not survive recreating the kiwix stack — make it declarative there:",
|
||||
" networks: [default, maven_default] # maven_default: external: true",
|
||||
"The book is the catalog name from the /content/... href in",
|
||||
"/catalog/v2/entries, not the display title. Others on the box:",
|
||||
"ifixit_en_all_2025-06, devdocs_en_ansible_2025-10."
|
||||
],
|
||||
"kiwix": {
|
||||
"url": "http://kiwix-server:8080",
|
||||
"book": "wikipedia_en_all_maxi_2026-02",
|
||||
"book_ru": "wikipedia_ru_all_maxi_2026-02",
|
||||
"max_results": 5,
|
||||
"snippet_runes": 1500
|
||||
},
|
||||
|
||||
"//morning_routines": [
|
||||
"The daily checklist (Vikunja #280). Each item is done when its fact_key",
|
||||
"gets a non-voided fact inside the window, so 'выпил воды' closes water and",
|
||||
"nothing has to be ticked by hand. nudge_at fires once, at the end of the",
|
||||
"window, and only for what is still open. Weekdays empty = every day."
|
||||
],
|
||||
"morning_routines": [
|
||||
{
|
||||
"name": "утро",
|
||||
"window_start": "08:00",
|
||||
"window_end": "11:00",
|
||||
"nudge_at": "10:30",
|
||||
"severity": 1,
|
||||
"items": [
|
||||
{ "key": "medicine", "fact_key": "medicine", "label": "лекарство" },
|
||||
{ "key": "water", "fact_key": "water", "label": "вода" },
|
||||
{ "key": "pets", "fact_key": "pets", "label": "покормить кота" }
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
"//feeds": [
|
||||
"RSS reading (Vikunja #258). Every item lands as a note with source",
|
||||
"rss:<name>, which is also what puts entries in the intake journal that",
|
||||
"/events reads. Only the feed URL leaves the box.",
|
||||
"This is a starting pair, not a curated set — trim or extend it."
|
||||
],
|
||||
"feeds": {
|
||||
"poll_interval": "30m",
|
||||
"max_items": 5,
|
||||
"max_age": "24h",
|
||||
"sources": [
|
||||
{ "name": "lwn", "url": "https://lwn.net/headlines/newrss", "category": "технологии" },
|
||||
{ "name": "archlinux", "url": "https://archlinux.org/feeds/news/", "category": "технологии" }
|
||||
]
|
||||
},
|
||||
|
||||
"//crawl": [
|
||||
"Reading a web page (Vikunja #259). on_demand answers 'посмотри <URL>'.",
|
||||
"No allow_hosts, so any public host he names is readable; private",
|
||||
"addresses are refused unconditionally by internal/webfetch and do not",
|
||||
"need listing. Setting allow_hosts here would also narrow on-demand,",
|
||||
"which is the point of leaving it empty."
|
||||
],
|
||||
"crawl": {
|
||||
"on_demand": true,
|
||||
"timeout": "10s",
|
||||
"max_runes": 4000
|
||||
},
|
||||
|
||||
"digest": {
|
||||
"enabled": true,
|
||||
"window": "30m",
|
||||
"max_items": 5,
|
||||
"severity_ceiling": 2
|
||||
},
|
||||
|
||||
"pattern_proposals": {
|
||||
"notify": false,
|
||||
"cooldown": "24h"
|
||||
},
|
||||
|
||||
"mcp": {
|
||||
"timeout": "15s",
|
||||
"servers": [
|
||||
{
|
||||
"name": "vikunja",
|
||||
"url": "http://192.168.1.104:9100/mcp",
|
||||
"allow_private": true,
|
||||
"allow_tools": ["list_projects", "list_tasks", "get_task_details", "create_task"],
|
||||
"max_tools": 6,
|
||||
"enabled": false
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
"smarthome": {
|
||||
"provider": "homeassistant",
|
||||
"url": "http://192.168.1.50:8123",
|
||||
"token": "${HA_TOKEN}",
|
||||
"domains": ["light", "switch", "sensor"],
|
||||
"max_entities": 40,
|
||||
"timeout": "10s",
|
||||
"refresh": "15m",
|
||||
"enabled": false
|
||||
},
|
||||
|
||||
"netscan": {
|
||||
"subnets": ["192.168.1.0/24"],
|
||||
"ports": [22, 80, 443, 8080],
|
||||
"timeout": "400ms",
|
||||
"rate": 100,
|
||||
"max_hosts": 256,
|
||||
"enabled": true
|
||||
},
|
||||
|
||||
"nexus": { "url": "http://nexus:9740" },
|
||||
"praxis": { "url": "http://praxis:8989" },
|
||||
"hexis": { "url": "http://hexis:9741" },
|
||||
|
||||
"voice": {
|
||||
"enabled": true,
|
||||
"bind": "0.0.0.0:9100",
|
||||
"lang": "ru",
|
||||
"stt": { "socket": "/run/maven/stt.sock", "lang": "ru" },
|
||||
"tts": { "socket": "/run/maven/tts.sock", "lang": "ru" },
|
||||
"embedder": {
|
||||
"model_path": "/opt/maven/models/embedder/multilingual-e5-small/model_quantized.onnx",
|
||||
"tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json",
|
||||
"lib_path": "/opt/maven/lib/libonnxruntime.so",
|
||||
"heads_path": "/opt/maven/models/embedder/router-heads/router_heads.onnx"
|
||||
},
|
||||
"llm_router": true,
|
||||
"query_min_score": 0.80,
|
||||
"query_min_margin": 0.008,
|
||||
"clarify_max_attempts": 3,
|
||||
"tool_timeout": "30s",
|
||||
"tools": [
|
||||
{ "name": "status", "cmd": ["systemctl", "status"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["статус", "покажи статус", "проверь статус"] },
|
||||
{ "name": "ps", "cmd": ["docker", "ps"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["статус докера", "лог докера", "покажи запущенные контейнеры", "покажи контейнеры", "список контейнеров", "что запущено"] },
|
||||
{ "name": "uptime", "cmd": ["uptime"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["покажи uptime", "аптайм", "как работает сервер", "сколько работает сервер"] },
|
||||
{ "name": "disk", "cmd": ["df", "-h"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["сколько места на диске", "сколько свободного места на диске", "место на диске", "покажи диск"] },
|
||||
{ "name": "memory", "cmd": ["free", "-h"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["свободная память", "сколько оперативной памяти свободно", "покажи память"] },
|
||||
{ "name": "logs", "cmd": ["journalctl", "-n", "50", "-u"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["покажи логи", "логи", "лог"] },
|
||||
{ "name": "restart", "cmd": ["systemctl", "restart"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["перезапусти", "перезагрузи", "рестарт"] },
|
||||
{ "name": "stop", "cmd": ["systemctl", "stop"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["останови", "останови сервис"] },
|
||||
{ "name": "start", "cmd": ["systemctl", "start"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["запусти", "запусти сервис"] },
|
||||
{ "name": "docker-restart", "cmd": ["docker", "restart"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["перезапусти контейнер", "перезагрузи контейнер"] },
|
||||
{ "name": "docker-stop", "cmd": ["docker", "stop"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["останови контейнер"] },
|
||||
{ "name": "reboot", "cmd": ["systemctl", "reboot"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["перезагрузи сервер", "перезагрузи хост"] }
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,8 @@ The tier is the path, so staleness is visible from the filename.
|
||||
| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. Indexed in `docs/evals/CLAUDE.md`, which marks each one live or superseded. | forever |
|
||||
| `docs/caveats/` | known limits, one entry per limit, each with a task id and a revisit trigger. Indexed in `docs/caveats/CLAUDE.md`. | until fixed, then deleted |
|
||||
| `docs/plans/` | the plan for one piece of work, frozen once it starts. Indexed in `docs/plans/CLAUDE.md`. | until the work lands |
|
||||
| `docs/capabilities/` | generated. The capability ledger and its probe harness, regenerated from `docs/spec.md` plus a named eval. **Never hand-edited**, except `domains.yaml`, `probes_field.json` and the scripts, which are its sources. Indexed in `docs/capabilities/README.md`. | until the spec or the measurement moves |
|
||||
| `docs/architecture/` | generated. The architecture observation and its evidence pack, rebuilt from source by the scripts beside it. Indexed in `docs/architecture/README.md`. | until the shape changes |
|
||||
| `docs/archive/` | dead. Read by nobody by default. | forever |
|
||||
|
||||
## Rules for this directory
|
||||
@@ -21,6 +23,10 @@ The tier is the path, so staleness is visible from the filename.
|
||||
correction. Do not append a changelog to it.
|
||||
* A number in prose with no `docs/evals/` file behind it is an opinion.
|
||||
* Fixing something deletes its caveat. It does not edit the eval that found it.
|
||||
* **A generated tier is rebuilt, never corrected.** A wrong row in
|
||||
`docs/capabilities/` or `docs/architecture/` is a bug in the generator or in
|
||||
one of its hand-written inputs. Editing the output makes the next rebuild
|
||||
silently undo the fix.
|
||||
|
||||
## Where a subsystem's reasoning lives
|
||||
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# docs/architecture
|
||||
|
||||
An observation of Maven as built, read at `5cae33a` on 2026-08-25. It describes
|
||||
what the code does today. It proposes nothing.
|
||||
|
||||
This directory is a build output plus its sources. `index.html`,
|
||||
`maven-architecture.json`, `anchors.md` and `diagrams/*.svg` are generated.
|
||||
|
||||
## Read it
|
||||
|
||||
| file | what it is |
|
||||
|---|---|
|
||||
| `index.html` | the viewer. Open it from the filesystem, no server needed. Seven views, the rendered diagram above each, click a component for its record. |
|
||||
| `check_viewer.js` | the viewer's only check. Runs every view against a DOM stub, because a TypeError in a renderer shows as a blank panel and not as an error. |
|
||||
| `findings.md` | the analysis. Kept apart from the facts on purpose. |
|
||||
| `maven-architecture.json` | the inventory. 160 components, 204 relations. The factual source for everything else. |
|
||||
| `anchors.md` | every symbol the inventory names, resolved to `path:line` with the line quoted. |
|
||||
| `diagrams/*.mmd` | the five views as Mermaid source. `03a`, `03b` and `03c` are the three traced requests. |
|
||||
| `diagrams/*.svg` | the same, rendered. |
|
||||
|
||||
## Rebuild it
|
||||
|
||||
```sh
|
||||
python3 docs/architecture/build_inventory.py # → maven-architecture.json
|
||||
python3 docs/architecture/verify_anchors.py # → anchors.md, exit 1 if stale
|
||||
sh docs/architecture/render.sh # → diagrams/*.svg, then index.html, then check_viewer.js
|
||||
python3 docs/architecture/build_viewer.py # → index.html alone
|
||||
node docs/architecture/check_viewer.js # → every view rendered, no throw
|
||||
```
|
||||
|
||||
Views 6 and 7 read `docs/capabilities/`, not this directory. View 6 is the
|
||||
capability matrix, 51 rows against seven dimensions. View 7 is the twelve
|
||||
cross-cutting invariants and the components that participate in each. Both are
|
||||
inlined by `build_viewer.py`, which reads `ledger.yaml` and `invariants.yaml`
|
||||
rather than deriving anything itself: the ledger's build is the only thing
|
||||
allowed to decide a dimension.
|
||||
|
||||
`render.sh` drives mermaid-cli through the system chromium rather than letting
|
||||
puppeteer download its own. It is also the only syntax check this repo has for a
|
||||
`.mmd`.
|
||||
|
||||
## What is verified, and what is not
|
||||
|
||||
**Verified mechanically.** `verify_anchors.py` resolves all 692 claimed symbols
|
||||
against the files their component names. Current state: 681 resolved to a line
|
||||
and 0 unresolved, with 0 missing files. The other 11 are config keys and make
|
||||
targets rather than Go identifiers, so they are skipped. The script exits
|
||||
non-zero on any failure, which makes it a staleness gate.
|
||||
|
||||
Writing it caught 29 symbols filed under the wrong component and seven names
|
||||
that were wrong outright. Two examples: `Store.RecordEvent` for what is really
|
||||
`Store.CreateEvent`, and `media.Keeper` for what is really `media.Store`.
|
||||
|
||||
**Not verified.** That a symbol means what its `responsibility` says. An anchor
|
||||
proves the identifier is on that line and nothing more. Judgements about
|
||||
ownership, coupling and enforcement are readings of the code. A reading can be
|
||||
wrong in a way grep cannot catch.
|
||||
|
||||
**Marked, not resolved.** Relations carry a `confidence` field. `medium` means
|
||||
the wiring is in the source and the call path was not traced end to end. `low`
|
||||
means it was inferred from one reference. The viewer can hide both. Four
|
||||
relations are `medium` and one is `low`.
|
||||
|
||||
**Deployment-specific.** Sixteen components are `configured-off` against
|
||||
`deploy/mavend.json` as it stood on the day, and that file was dirty in the
|
||||
working tree. A different config makes different components live. `status` says
|
||||
which, per component.
|
||||
|
||||
## The one thing to check first
|
||||
|
||||
`findings.md` 6.3 through 6.3d. They say the system has no single point that
|
||||
decides whether an origin may cause an effect, and that the pieces which look
|
||||
like that point are each answering a different question.
|
||||
|
||||
Revised on 2026-08-25 after an independent second pass. Four readings changed
|
||||
and one earlier statement was wrong. Section 6.3 marks the corrections.
|
||||
|
||||
Start at `internal/tool/tool.go:181`, `cmd/mavend/ecosystem_acts.go:158` and
|
||||
`internal/router/claim.go:38`.
|
||||
|
||||
## The evidence pack
|
||||
|
||||
`sh docs/architecture/pack_evidence.sh` builds `maven-evidence.zip` at the repo
|
||||
root: this directory, the structural context, and whole source files for the
|
||||
architectural seams. Whole files, never snippets, because a cut-down file loses
|
||||
the call path that makes a claim checkable.
|
||||
|
||||
The path list is an allowlist, not an exclusion list. A denylist ships whatever
|
||||
nobody thought to exclude, and this tree has a database key in it.
|
||||
|
||||
`architecture-evidence.txt` is the reviewer's index. It resolves a named symbol
|
||||
list against the checkout and says plainly when a requested name does not exist.
|
||||
It also re-runs the probe under every contradiction, so a claim and its grep
|
||||
cannot drift apart.
|
||||
|
||||
One file is not verbatim. `docker-compose.yml` carries an uptime-kuma API key,
|
||||
so a redacted copy ships in its place with that one value replaced. The script
|
||||
diffs the two and aborts if anything else changed.
|
||||
|
||||
The scan at the end refuses to build on a credential-shaped hit rather than
|
||||
printing a warning. Both of its first two versions were wrong in instructive
|
||||
ways. The name filter deleted `internal/router/singletoken.go` for matching
|
||||
`*token*`. The value scan flagged docker volume lines that name where a secret
|
||||
would live and contain none.
|
||||
|
||||
## The authorization function as implemented
|
||||
|
||||
The reconstruction, at the one decision point that gates an act
|
||||
(`internal/tool/tool.go:156`):
|
||||
|
||||
```
|
||||
permit(tool, confirmed) =
|
||||
row.status == "enabled" tool.go:164
|
||||
AND tier != irreversible risk.go:74 VoiceMayRun:false
|
||||
AND (tier == safe OR confirmed) risk.go:72,76
|
||||
```
|
||||
|
||||
`tier` is `RiskOf(row)`. The reach is not an input: `Executor.Exec` takes
|
||||
`(ctx, name, args, confirmed)` and no surface.
|
||||
|
||||
`confirmed` is unproven at this boundary too. The invariant that a confirmation
|
||||
binds one capability, one target and an expiry lives in `pendingAct` and
|
||||
`resolveConfirm`. `Exec` trusts the boolean.
|
||||
|
||||
The expression covers two of the three act paths. Hexis reuses it deliberately
|
||||
(`cmd/mavend/ecosystem_acts.go:768`). The Praxis lifecycle path has no tier and
|
||||
no confirm turn: `praxisItemAction.handle` calls straight through at
|
||||
`ecosystem_acts.go:158`.
|
||||
|
||||
Behind the IPC boundary, `auth.Can(method, scope, params)` runs with
|
||||
`scope.Surface` always `SurfaceCoreProcess` (`internal/auth/enrollment.go:65`)
|
||||
and step-up held as one global timestamp that ignores `Scope`
|
||||
(`internal/webauthn/session.go:38` and `:62`).
|
||||
|
||||
`auth` answers who may carry what authority. `tool` answers what effect a
|
||||
capability has and what proof it demands. Those are orthogonal, not competing.
|
||||
The decision combining them does not exist.
|
||||
|
||||
Two representations of reach exist and both are ignored. `server.go:198` only
|
||||
defaults an empty `p.Surface`, so a client-asserted one survives and nothing
|
||||
reads it. `server.go:148` hardcodes `Session.Surface`. Since `req.Surface` is
|
||||
request payload on an unauthenticated wire, it must not become an authorization
|
||||
input as it stands.
|
||||
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write architecture-evidence.txt: the reviewer's index into the pack.
|
||||
|
||||
It resolves a named symbol list against this checkout and prints where each one
|
||||
is, or says plainly that it does not exist. A requested name that is absent is
|
||||
evidence too, so nothing here is silently dropped or silently corrected.
|
||||
|
||||
Every contradiction is re-checked at generation time by running its own probe,
|
||||
so the claim and the grep that supports it cannot drift apart in the pack.
|
||||
|
||||
python3 docs/architecture/build_evidence.py
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
|
||||
TREES = ["cmd", "internal"]
|
||||
|
||||
# The reviewer's list, verbatim on the left. Where a name does not exist in this
|
||||
# repo, the right side is what it appears to mean. Resolution below reports both
|
||||
# so a wrong name is visible rather than quietly fixed.
|
||||
REQUESTED = [
|
||||
("auth.TierFor", "auth.MaxLayer"),
|
||||
("auth.Surface", None),
|
||||
("auth.Layer", None),
|
||||
("tool.Executor.Exec", None),
|
||||
("tool.PolicyFor", None),
|
||||
("tool.RiskOf", None),
|
||||
("tool.RiskSafe", "tool.TierSafe"),
|
||||
("tool.RiskDestructive", "tool.TierDestructive"),
|
||||
("tool.RiskIrreversible", "tool.TierIrreversible"),
|
||||
("router.ClaimOf", None),
|
||||
("reactiveHandler", None),
|
||||
("tickLoop", None),
|
||||
]
|
||||
|
||||
# Added because the authorization function the reviewer wants to reconstruct
|
||||
# runs through these and the list above does not reach them.
|
||||
ALSO = [
|
||||
"auth.Gate", "auth.Can", "auth.Requirement", "auth.Authority",
|
||||
"auth.NewFloorEnrollment", "auth.StaticEnrollment", "auth.Scope",
|
||||
"tool.Policy", "tool.RiskOfCapability", "tool.irreversibleVerbs",
|
||||
"tool.ErrNeedsConfirm", "tool.ErrNeedsAuthedSurface", "tool.ErrNotEnabled",
|
||||
"ipc.Server.Check", "ipc.CheckFunc",
|
||||
"voice.PushToTalkReq", "voice.Sessions.Add", "voice.Session",
|
||||
"PolicyFor", "RiskOf", "Executor.Exec",
|
||||
"pendingAct", "resolveConfirm", "actionAct", "runTurn", "applyAction",
|
||||
"querySources", "queryWalk", "StageZeroGrammars", "Router.Route",
|
||||
]
|
||||
|
||||
PKG_DIR = {
|
||||
"auth": "internal/auth", "tool": "internal/tool", "claim": "internal/claim",
|
||||
"modes": "internal/modes", "router": "internal/router", "voice": "internal/voice",
|
||||
"ipc": "internal/ipc", "store": "internal/store",
|
||||
}
|
||||
|
||||
|
||||
def go_files(rel):
|
||||
full = os.path.join(ROOT, rel)
|
||||
out = []
|
||||
for base, _, names in os.walk(full):
|
||||
for n in sorted(names):
|
||||
if n.endswith(".go"):
|
||||
out.append(os.path.relpath(os.path.join(base, n), ROOT))
|
||||
return sorted(out)
|
||||
|
||||
|
||||
def all_go():
|
||||
out = []
|
||||
for t in TREES:
|
||||
out.extend(go_files(t))
|
||||
return out
|
||||
|
||||
|
||||
_CACHE = {}
|
||||
|
||||
|
||||
def lines_of(rel):
|
||||
"""Read once. resolve() sweeps every file per pattern per symbol, and
|
||||
re-reading cmd/ and internal/ that many times took minutes."""
|
||||
if rel not in _CACHE:
|
||||
try:
|
||||
_CACHE[rel] = open(os.path.join(ROOT, rel), errors="replace").read().splitlines()
|
||||
except OSError:
|
||||
_CACHE[rel] = []
|
||||
return _CACHE[rel]
|
||||
|
||||
|
||||
def resolve(sym):
|
||||
"""Find the declaration of sym. Returns (path, line, text) or (None,)*3."""
|
||||
tail = sym.split(".")[-1]
|
||||
recv = sym.split(".")[-2] if sym.count(".") >= 1 else None
|
||||
pats = [
|
||||
re.compile(r"^func\s+\(\w+\s+\*?" + re.escape(recv or "\x00") + r"\)\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^func\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^type\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^\s*" + re.escape(tail) + r"\s+\w+\s*=\s"), # typed const
|
||||
re.compile(r"^\s*" + re.escape(tail) + r"\s*=\s"),
|
||||
re.compile(r"^(var|const)\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^\s*" + re.escape(tail) + r"\s+\w"), # struct field
|
||||
]
|
||||
pkg = sym.split(".")[0]
|
||||
files = go_files(PKG_DIR[pkg]) if pkg in PKG_DIR else all_go()
|
||||
files = [f for f in files if not f.endswith("_test.go")]
|
||||
for pat in pats:
|
||||
for rel in files:
|
||||
for i, line in enumerate(lines_of(rel), 1):
|
||||
if pat.match(line):
|
||||
return rel, i, line.strip()
|
||||
return None, None, None
|
||||
|
||||
|
||||
def exported(pkg_rel):
|
||||
"""Every exported declaration in a package, for the `claim.*` / `modes.*` asks."""
|
||||
out = []
|
||||
pat = re.compile(r"^(func|type|const|var)\s+\(?[^)]*\)?\s*([A-Z]\w*)")
|
||||
fn = re.compile(r"^func\s+(\([^)]*\)\s*)?([A-Z]\w*)")
|
||||
for rel in go_files(pkg_rel):
|
||||
if rel.endswith("_test.go"):
|
||||
continue
|
||||
for i, line in enumerate(lines_of(rel), 1):
|
||||
m = fn.match(line) or pat.match(line)
|
||||
if m:
|
||||
name = m.group(m.lastindex)
|
||||
if name and name[0].isupper():
|
||||
out.append((name, f"{rel}:{i}", line.strip()))
|
||||
return out
|
||||
|
||||
|
||||
def sh(cmd):
|
||||
return subprocess.run(cmd, shell=True, cwd=ROOT, capture_output=True,
|
||||
text=True).stdout.strip()
|
||||
|
||||
|
||||
# Each probe is a shell command whose output IS the evidence. Re-run at pack
|
||||
# time so the pack cannot claim something the checkout no longer shows.
|
||||
CONTRADICTIONS = [
|
||||
("auth surface/layer documented as control, not consumed on turn path",
|
||||
# px.Surface is the Praxis lifecycle verb, an unrelated name collision, and
|
||||
# Surfaced* are the read-out-item helpers. Excluded by name so the absence
|
||||
# this probe reports is the auth Surface and not a filtering accident.
|
||||
"grep -rnE 'req\\.Surface|sess\\.Surface|Session\\.Surface|auth\\.Surface|voice\\.Surface' cmd/mavend/*.go "
|
||||
"| grep -v _test | grep -vE 'px\\.Surface|Surfaced' "
|
||||
"|| echo '(no match: no file in cmd/mavend reads the auth Surface of a request or a session)'"),
|
||||
("auth.Can runs only behind the IPC boundary",
|
||||
"grep -rn 'auth\\.' cmd/ internal/ --include='*.go' | grep -v _test "
|
||||
"| grep -v '^internal/auth/' | grep -vE ':[0-9]+:\\s*(//|\\*)'"),
|
||||
("tool risk policy live on execution path",
|
||||
"sed -n '176,190p' internal/tool/tool.go"),
|
||||
("tool executor receives no reach/surface",
|
||||
"grep -n 'func (e \\*Executor) Exec' internal/tool/tool.go"),
|
||||
("voice server normalizes incoming surface to pc-client",
|
||||
"grep -n 'SurfacePCClient' internal/voice/server.go"),
|
||||
("claim abstraction exists but Route does not consume it",
|
||||
"grep -rn 'ClaimOf' --include='*.go' cmd internal | grep -v _test || echo '(only the definition; no caller)'"),
|
||||
("modes package is imported by nothing",
|
||||
"grep -rn 'internal/modes' --include='*.go' cmd internal | grep -v '^internal/modes/' || echo '(no importer)'"),
|
||||
("voice server DEFAULTS an empty surface, it does not overwrite a sent one",
|
||||
"sed -n '193,201p' internal/voice/server.go"),
|
||||
("session surface is hardcoded, independently of the request field",
|
||||
"sed -n '146,149p' internal/voice/server.go"),
|
||||
("HandlePushToTalk never reads req.Surface",
|
||||
"sed -n '200,203p' cmd/mavend/voice.go"),
|
||||
("hexis reuses the same risk policy",
|
||||
"grep -n 'RiskOfCapability\\|PolicyFor' cmd/mavend/ecosystem_acts.go"),
|
||||
("praxis lifecycle mutations bypass the risk policy entirely",
|
||||
"sed -n '156,172p' cmd/mavend/ecosystem_acts.go"),
|
||||
("Exec trusts a confirmed bool it cannot verify was bound",
|
||||
"grep -n 'func (e \\*Executor) Exec' internal/tool/tool.go; grep -rn 'tools.Exec(' cmd/mavend/*.go | grep -v _test"),
|
||||
("FloorEnrollment maps every same-uid caller to one surface",
|
||||
"sed -n '63,72p' internal/auth/enrollment.go"),
|
||||
("PasskeySession ignores Scope in both methods",
|
||||
"grep -n 'func (s \\*PasskeySession) CurrentLayer\\|func (s \\*PasskeySession) Assert' internal/webauthn/session.go"),
|
||||
("Claim.Coverage can be 1.0 with nothing extracted",
|
||||
"grep -n 'func claimSpans' -A 4 internal/router/claim.go; grep -n 'd.Slots.Text = ex.Text' -B 2 internal/router/router.go; grep -n 'func (c Claim) Coverage' -A 7 internal/claim/claim.go"),
|
||||
("claim_test asserts Band only, and every case sets Text == Utterance",
|
||||
"grep -n 'Utterance:\\|Text:\\|want:' internal/router/claim_test.go | head -20"),
|
||||
("systemctl reboot is destructive, not irreversible",
|
||||
"grep -n 'irreversibleVerbs = map' -A 8 internal/tool/risk.go; grep -n 'reboot' deploy/mavend.json"),
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
out = []
|
||||
w = out.append
|
||||
w("architecture evidence pack")
|
||||
w("=" * 72)
|
||||
w("")
|
||||
w("commit: " + sh("git rev-parse HEAD"))
|
||||
w("date: " + sh("git log -1 --format=%cd --date=short"))
|
||||
w("branch: " + sh("git rev-parse --abbrev-ref HEAD"))
|
||||
w("")
|
||||
w("working tree at pack time (git status --short):")
|
||||
for line in (sh("git status --short") or "(clean)").splitlines():
|
||||
w(" " + line)
|
||||
w("")
|
||||
w("The pack is built from the WORKING TREE, not from the commit. The lines")
|
||||
w("above are the difference. deploy/mavend.json in particular is modified:")
|
||||
w("phraser.model_path points at maven-instruct-b2, the committed value was")
|
||||
w("Qwen3-1.7B-UD-Q4_K_XL. Sixteen 'configured-off' claims read this file.")
|
||||
w("")
|
||||
|
||||
w("requested symbols")
|
||||
w("-" * 72)
|
||||
for name, actual in REQUESTED:
|
||||
rel, line, text = resolve(name)
|
||||
if rel:
|
||||
w(f"- {name}")
|
||||
w(f" {rel}:{line} {text}")
|
||||
elif actual:
|
||||
arel, aline, atext = resolve(actual)
|
||||
w(f"- {name} -> DOES NOT EXIST in this repo")
|
||||
if arel:
|
||||
w(f" the name appears to be {actual}")
|
||||
w(f" {arel}:{aline} {atext}")
|
||||
else:
|
||||
w(f" and neither does {actual}")
|
||||
else:
|
||||
w(f"- {name} -> NOT FOUND")
|
||||
w("")
|
||||
|
||||
for pkg, rel in (("claim.*", "internal/claim"), ("modes.*", "internal/modes")):
|
||||
w(f"{pkg} ({rel})")
|
||||
w("-" * 72)
|
||||
for name, anchor, text in exported(rel):
|
||||
w(f"- {name}")
|
||||
w(f" {anchor} {text}")
|
||||
w("")
|
||||
|
||||
w("additional symbols on the authorization path")
|
||||
w("-" * 72)
|
||||
for name in ALSO:
|
||||
rel, line, text = resolve(name)
|
||||
w(f"- {name}")
|
||||
w(f" {rel}:{line} {text}" if rel else " NOT FOUND")
|
||||
w("")
|
||||
|
||||
w("known contradictions, each re-checked at pack time")
|
||||
w("=" * 72)
|
||||
w("The command under each claim was run against this checkout just now.")
|
||||
w("Its output is what follows. Nothing here is transcribed by hand.")
|
||||
w("")
|
||||
for claim, cmd in CONTRADICTIONS:
|
||||
w("- " + claim)
|
||||
w(" $ " + cmd)
|
||||
res = sh(cmd)
|
||||
for line in (res or "(no output)").splitlines():
|
||||
w(" " + line)
|
||||
w("")
|
||||
|
||||
w("what the pack does NOT contain, and why")
|
||||
w("=" * 72)
|
||||
w("- .git, so no history and no gitignored working files travel with it.")
|
||||
w("- deploy/telegram.env and deploy/db_key.env. The second holds the")
|
||||
w(" database key. Both are gitignored and present in the working tree.")
|
||||
w("- docker-compose.yml verbatim. It carries one live-looking credential on")
|
||||
w(" line 132 (an uptime-kuma API key). The pack ships")
|
||||
w(" docker-compose.redacted.yml with that one value replaced and nothing")
|
||||
w(" else changed, so the mavcaldav and mavmaild claims stay checkable.")
|
||||
w("- models/, deps/, *.db, *.onnx, *.gguf, certs, logs, node_modules.")
|
||||
w("- internal/session, internal/db and tests/ from the requested list: none")
|
||||
w(" of the three exists. Sessions live in internal/voice/session.go, the")
|
||||
w(" store is internal/store, and tests sit beside their code as *_test.go.")
|
||||
w("")
|
||||
w("included test files, since the ask named them by subject:")
|
||||
for pat, label in (
|
||||
("internal/router", "routing and arbitration"),
|
||||
("internal/tool", "risk and confirmation"),
|
||||
("internal/auth", "authorization"),
|
||||
("internal/claim", "claim"),
|
||||
("cmd/mavend", "turn path, confirm gate, query chain"),
|
||||
):
|
||||
n = sh(f"find {pat} -name '*_test.go' | wc -l")
|
||||
w(f" {label}: {n} *_test.go under {pat}/")
|
||||
w("")
|
||||
w("fixtures are synthetic, not captured speech: internal/router/eval/*.json")
|
||||
w("and cmd/mavend/testdata/**.json are hand-written contracts. Named here")
|
||||
w("because they are Russian utterances and look like personal data.")
|
||||
|
||||
path = os.path.join(HERE, "architecture-evidence.txt")
|
||||
open(path, "w").write("\n".join(out) + "\n")
|
||||
print(f"architecture-evidence.txt: {os.path.getsize(path)} bytes, {len(out)} lines")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Assemble index.html from the template, the inventory and the diagrams.
|
||||
|
||||
index.html is self-contained on purpose: it opens from the filesystem with no
|
||||
server, and a browser at file:// refuses to fetch a sibling JSON. So the
|
||||
inventory, every .mmd source and every rendered .svg are inlined here rather
|
||||
than loaded at runtime.
|
||||
|
||||
Run it through docs/architecture/render.sh, which re-renders the SVGs first.
|
||||
Running it alone rebuilds the viewer against whatever SVGs are already there.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
import yaml
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DIA = os.path.join(HERE, "diagrams")
|
||||
CAPDIR = os.path.join(os.path.dirname(HERE), "capabilities")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
arch = json.load(open(os.path.join(HERE, "maven-architecture.json")))
|
||||
mermaid, svg = {}, {}
|
||||
for name in sorted(os.listdir(DIA)):
|
||||
path = os.path.join(DIA, name)
|
||||
if name.endswith(".mmd"):
|
||||
mermaid[name] = open(path).read()
|
||||
elif name.endswith(".svg"):
|
||||
svg[name] = open(path).read()
|
||||
|
||||
# The capability half. Generated beside this one and read here rather than
|
||||
# re-derived: the ledger's build is the only thing allowed to decide a
|
||||
# dimension, and a second derivation would drift from it silently.
|
||||
ledger = yaml.safe_load(open(os.path.join(CAPDIR, "ledger.yaml")))
|
||||
invariants = yaml.safe_load(
|
||||
open(os.path.join(CAPDIR, "invariants.yaml")))["invariants"]
|
||||
|
||||
payload = (
|
||||
"const ARCH = " + json.dumps(arch, ensure_ascii=False) + ";\n"
|
||||
"const CAPS = " + json.dumps(ledger, ensure_ascii=False) + ";\n"
|
||||
"const INV = " + json.dumps(invariants, ensure_ascii=False) + ";\n"
|
||||
"const MERMAID = " + json.dumps(mermaid, ensure_ascii=False) + ";\n"
|
||||
"const SVG = " + json.dumps(svg, ensure_ascii=False) + ";\n"
|
||||
)
|
||||
template = open(os.path.join(HERE, "viewer.template.html")).read()
|
||||
if "/*__DATA__*/" not in template:
|
||||
raise SystemExit("viewer.template.html has no /*__DATA__*/ marker")
|
||||
out = os.path.join(HERE, "index.html")
|
||||
open(out, "w").write(template.replace("/*__DATA__*/", payload))
|
||||
print(
|
||||
"index.html: %d bytes, %d components, %d relations, %d diagrams, "
|
||||
"%d rendered, %d capabilities, %d invariants"
|
||||
% (os.path.getsize(out), len(arch["components"]), len(arch["edges"]),
|
||||
len(mermaid), len(svg), len(ledger["capabilities"]), len(invariants))
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,64 @@
|
||||
// Smoke test for index.html's renderers, run by render.sh when node is present.
|
||||
//
|
||||
// The viewer has no test harness and a TypeError in a renderer produces a blank
|
||||
// panel, not an error anyone sees. This runs every view's render function
|
||||
// against a DOM stub and fails loudly on the first throw. It checks that the
|
||||
// renderers run over the real data, not that the result looks right.
|
||||
//
|
||||
// node docs/architecture/check_viewer.js [path/to/index.html]
|
||||
|
||||
const fs = require('fs');
|
||||
const path = process.argv[2] || __dirname + '/index.html';
|
||||
const src = fs.readFileSync(path, 'utf8');
|
||||
const js = src.match(/<script>([\s\S]*)<\/script>/)[1];
|
||||
|
||||
const el = () => {
|
||||
const e = {
|
||||
innerHTML: '', textContent: '', style: {}, checked: true, value: '',
|
||||
_written: 0,
|
||||
dataset: {}, classList: { add(){}, remove(){}, toggle(){} },
|
||||
querySelectorAll: () => [], querySelector: () => null,
|
||||
appendChild(){}, addEventListener(){}, scrollIntoView(){},
|
||||
getBoundingClientRect: () => ({top:0,left:0,width:0,height:0}),
|
||||
};
|
||||
return e;
|
||||
};
|
||||
// One shared element per id, so a renderer's output can be read back. A stub
|
||||
// that silently swallows innerHTML would let an empty render pass.
|
||||
const els = {};
|
||||
const document = {
|
||||
getElementById: id => (els[id] = els[id] || el()), querySelectorAll: () => [], querySelector: () => null,
|
||||
createElementNS: el, createElement: el, addEventListener(){},
|
||||
};
|
||||
const window = { addEventListener(){} };
|
||||
const requestAnimationFrame = () => {};
|
||||
|
||||
// `const` inside a direct eval stays in the eval's own scope, so the checks are
|
||||
// appended to the source and evaluated with it rather than run beside it.
|
||||
const checks = `
|
||||
let n = 0;
|
||||
for (const v of VIEWS) {
|
||||
setView(v.id);
|
||||
n++;
|
||||
}
|
||||
// Every flow, and every capability's side panel: the branch a click takes.
|
||||
for (const k of Object.keys(FLOWS)) { S.flow = k; renderFlow(el()); n++; }
|
||||
setView('c1');
|
||||
if (els.main.innerHTML.length < 5000) throw new Error('capability matrix rendered ' + els.main.innerHTML.length + ' chars');
|
||||
for (const c of CAPS.capabilities) {
|
||||
renderCapSide(c.id);
|
||||
if (els.side.innerHTML.length < 400) throw new Error('thin panel for ' + c.id);
|
||||
n++;
|
||||
}
|
||||
S.capBy = 'domain'; renderCaps(el()); n++;
|
||||
setView('c2');
|
||||
if (els.main.innerHTML.length < 4000) throw new Error('invariants view rendered ' + els.main.innerHTML.length + ' chars');
|
||||
for (const iv of INV) { invRollup(iv); n++; }
|
||||
for (const c of ARCH.components) { renderSide(c.id); n++; }
|
||||
|
||||
console.log('viewer: ' + n + ' render calls, ' + VIEWS.length + ' views, ' +
|
||||
CAPS.capabilities.length + ' capabilities, ' + INV.length +
|
||||
' invariants, no throw');
|
||||
`;
|
||||
|
||||
eval(js + checks);
|
||||
@@ -0,0 +1,120 @@
|
||||
%% View 1 — System topology.
|
||||
%% Runtime processes and external systems, with process boundaries drawn explicitly.
|
||||
%% mavend is the centre because the code makes it one: it is the only key holder,
|
||||
%% it owns the store, the IPC socket, the voice listener, the tick loop, eight
|
||||
%% in-process background workers and the child llama-server.
|
||||
%% Evidence: docker-compose.yml, cmd/mavend/main.go, cmd/mavend/boot.go,
|
||||
%% deploy/mavwaked.service, deploy/mavgpud.service.
|
||||
flowchart LR
|
||||
|
||||
subgraph WORKPC["workpc — systemd user units, never in docker-compose"]
|
||||
direction TB
|
||||
MAVWAKED["mavwaked<br/>process<br/>arecord · silero VAD · keyword head"]
|
||||
MAVGPUD["mavgpud<br/>process<br/>GPU supervisor"]
|
||||
LLAMA_W["llama-server<br/>model · workstation card"]
|
||||
CW2["CrisperWhisper2 turbo<br/>model · port 8081"]
|
||||
ALSA["arecord / aplay<br/>external"]
|
||||
TUNNEL["maven-voice-tunnel.service<br/>ssh · the only path in"]
|
||||
end
|
||||
|
||||
subgraph HOMESRV["homesrv — docker compose project `maven`"]
|
||||
direction TB
|
||||
|
||||
subgraph MAVEND_P["mavend — process boundary · the only key holder"]
|
||||
direction TB
|
||||
IPCSRV["IPC server<br/>unix /run/maven/mavend.sock"]
|
||||
VOICESRV["voice server<br/>TCP 0.0.0.0:9100"]
|
||||
TURN["reactive handler<br/>the turn pipeline"]
|
||||
TICK["tick loop<br/>60s"]
|
||||
WORKERS["8 background workers<br/>tick · fact-enrichment · feed · crawl<br/>voice · mcp · home · memory-eval"]
|
||||
STORE[("store<br/>sqlite, MaxOpenConns=1")]
|
||||
end
|
||||
|
||||
LLAMA_H["llama-server<br/>model · resident<br/>child process of mavend"]
|
||||
MAVSTTD["mavsttd<br/>process<br/>whisper.cpp"]
|
||||
MAVTTSD["mavttsd<br/>process<br/>piper"]
|
||||
MAVWEB["mavweb<br/>process<br/>HTTP 127.0.0.1:9201"]
|
||||
MAVPOLL["mavpoll<br/>process<br/>network_mode: host"]
|
||||
SEARX["SearXNG<br/>external"]
|
||||
KIWIX["kiwix-server<br/>external"]
|
||||
NETDATA["netdata<br/>external"]
|
||||
KUMA["uptime-kuma<br/>external"]
|
||||
end
|
||||
|
||||
subgraph OFF["built, not deployed — commented out in docker-compose.yml"]
|
||||
direction TB
|
||||
MAVCALDAV["mavcaldav<br/>process"]
|
||||
MAVMAILD["mavmaild<br/>process"]
|
||||
end
|
||||
|
||||
subgraph ECO["ecosystem network — external compose project"]
|
||||
direction TB
|
||||
NEXUS["Nexus<br/>external · identity"]
|
||||
PRAXIS["Praxis<br/>external · attention"]
|
||||
HEXIS["Hexis<br/>external · capabilities"]
|
||||
end
|
||||
|
||||
subgraph NET["internet"]
|
||||
direction TB
|
||||
TG["Telegram Bot API<br/>external · via SOCKS relay"]
|
||||
NTFY["ntfy<br/>external · DISABLED in config"]
|
||||
ZM["zenmoney<br/>external · no token mounted"]
|
||||
end
|
||||
|
||||
HA["Home Assistant<br/>external · enabled:false"]
|
||||
|
||||
%% ---- voice path
|
||||
ALSA --- MAVWAKED
|
||||
MAVWAKED -->|"PushToTalk · TCP"| TUNNEL
|
||||
TUNNEL -->|"ssh to 127.0.0.1:9110"| VOICESRV
|
||||
VOICESRV -->|"proactive Push on the same conn"| MAVWAKED
|
||||
|
||||
%% ---- module IPC
|
||||
MAVWEB -->|"3 × ipc.Client · unix"| IPCSRV
|
||||
MAVWEB -->|"POST /api/ptt · TCP mavend:9100"| VOICESRV
|
||||
MAVPOLL -->|"WriteFact · unix"| IPCSRV
|
||||
MAVCALDAV -.->|"WriteFact · unix"| IPCSRV
|
||||
MAVMAILD -.->|"IngestMail · unix"| IPCSRV
|
||||
|
||||
%% ---- worker sockets
|
||||
TURN -->|"worker · unix stt.sock"| MAVSTTD
|
||||
TURN -->|"worker · unix tts.sock"| MAVTTSD
|
||||
TURN -->|"HTTP · preferred, silent fallback"| CW2
|
||||
|
||||
%% ---- models
|
||||
MAVEND_P ---|"spawns and owns"| LLAMA_H
|
||||
MAVGPUD ---|"spawns and supervises"| LLAMA_W
|
||||
MAVGPUD ---|"spawns and supervises"| CW2
|
||||
TURN -.->|"llm.Pair · model_disabled:true"| MAVGPUD
|
||||
|
||||
%% ---- world and ecosystem
|
||||
TURN -->|"HTTP · query string only"| SEARX
|
||||
TURN -->|"HTTP"| KIWIX
|
||||
TURN -->|"HTTP · v1 contract, correlation id"| NEXUS
|
||||
TURN -->|"HTTP"| PRAXIS
|
||||
TURN -->|"HTTP"| HEXIS
|
||||
MAVWEB -->|"HTTP · read-only panel"| NEXUS
|
||||
MAVWEB -->|"HTTP · read-only panel"| PRAXIS
|
||||
MAVWEB -->|"HTTP · read-only panel"| HEXIS
|
||||
TURN -.->|"HTTP · enabled:false"| HA
|
||||
|
||||
%% ---- reaches
|
||||
TICK -->|"telegram sink"| TG
|
||||
TG -->|"getUpdates long poll"| TURN
|
||||
TICK -.->|"ntfy sink · nil, disabled"| NTFY
|
||||
|
||||
%% ---- pollers
|
||||
MAVPOLL --> NETDATA
|
||||
MAVPOLL --> KUMA
|
||||
MAVPOLL -.-> ZM
|
||||
|
||||
classDef proc fill:#1f3a5f,stroke:#7fb3ff,color:#eaf2ff
|
||||
classDef ext fill:#3d2f4f,stroke:#c39bd3,color:#f4ecf7
|
||||
classDef model fill:#4a3a1f,stroke:#e0b050,color:#fff6e0
|
||||
classDef store fill:#1f4a3a,stroke:#6ed0a8,color:#e8fff5
|
||||
classDef off fill:#3a3a3a,stroke:#888,color:#ccc,stroke-dasharray:4 3
|
||||
class MAVWAKED,MAVGPUD,MAVSTTD,MAVTTSD,MAVWEB,MAVPOLL,IPCSRV,VOICESRV,TURN,TICK,WORKERS proc
|
||||
class ALSA,SEARX,KIWIX,NETDATA,KUMA,NEXUS,PRAXIS,HEXIS,TG,TUNNEL ext
|
||||
class LLAMA_H,LLAMA_W,CW2 model
|
||||
class STORE store
|
||||
class MAVCALDAV,MAVMAILD,NTFY,ZM,HA off
|
||||
@@ -0,0 +1,160 @@
|
||||
%% View 2 — Core internals of mavend.
|
||||
%% The real path, in the order runTurn actually runs it. The sequence is NOT
|
||||
%% input → routing → intent → state → tools → response: eleven stateful
|
||||
%% pre-emptors get first refusal BEFORE routing, and a query intent then enters
|
||||
%% a second, longer arbitration of its own.
|
||||
%% Evidence: cmd/mavend/voice.go runTurn, cmd/mavend/turnroute.go,
|
||||
%% cmd/mavend/actions.go, cmd/mavend/actions_query.go, internal/router/router.go.
|
||||
flowchart TB
|
||||
|
||||
subgraph IN["input — three reaches, one pipeline"]
|
||||
A1["voice.Server<br/>HandlePushToTalk"]
|
||||
A2["daemonAPI.Chat<br/>mavweb /api/chat"]
|
||||
A3["telegram poller<br/>getUpdates"]
|
||||
STT["stt seam<br/>Remote mavsttd · CW2 · Stub"]
|
||||
end
|
||||
|
||||
A1 --> STT --> RT
|
||||
A2 --> RT
|
||||
A3 --> A2
|
||||
|
||||
RT["runTurn<br/>cmd/mavend/voice.go"]
|
||||
|
||||
RT --> D0["decision.With<br/>one arbitration record per turn"]
|
||||
D0 --> TR0["turnRoute created<br/>sync.Once, on the context"]
|
||||
|
||||
subgraph PRE["pre-route ladder — 11 rungs, order load-bearing"]
|
||||
direction TB
|
||||
P1["1 expired-clarify notice"]
|
||||
P2["2 confirm answer<br/>resolveConfirm"]
|
||||
P3["3 targeted repair"]
|
||||
P4["3b untargeted repair"]
|
||||
P5["3c command prohibition"]
|
||||
P6["4 clarify answer"]
|
||||
P7["5 quiet toggle"]
|
||||
P8["5b snooze"]
|
||||
P9["5c ack"]
|
||||
P10["5d reminder cancellation"]
|
||||
P11["5e ordinal selection"]
|
||||
P1-->P2-->P3-->P4-->P5-->P6-->P7-->P8-->P9-->P10-->P11
|
||||
end
|
||||
TR0 --> PRE
|
||||
PRE -->|"any rung claims"| OUT
|
||||
|
||||
subgraph ROUTE["step 6 — the cascade · internal/router"]
|
||||
direction TB
|
||||
CONT["continuationDecision<br/>an elliptical follow-up is answered<br/>from the previous turn, not routed"]
|
||||
S0["stage 0 grammars<br/>StageZeroGrammars · first match wins at 1.0<br/>the ONLY arm that may set SourceAnchored"]
|
||||
SH["stage 0b routing heads<br/>ONNX softmax over the label set"]
|
||||
SL["stage 1a LLM router<br/>resident model, grammar-constrained"]
|
||||
SC["stage 1 classifier<br/>nearest centroid · THE FLOOR<br/>names no destination"]
|
||||
SE["stage 2 extractor + stage 3 gate"]
|
||||
CONT -->|"not a continuation"| S0
|
||||
S0 -->|"no match"| SH
|
||||
SH -->|"declines"| SL
|
||||
SL -->|"error or unparsable"| SC
|
||||
SH --> SE
|
||||
SL --> SE
|
||||
SC --> SE
|
||||
end
|
||||
PRE -->|"nobody claimed"| ROUTE
|
||||
|
||||
ROUTE --> DLG["step 7 dialogue merge<br/>followUpMerge · rememberTurn"]
|
||||
DLG --> CLAR{"step 8<br/>dec.Clarify OR a required slot missing?"}
|
||||
CLAR -->|"yes"| ASK["askClarify<br/>park the request, ask one question"]
|
||||
ASK --> OUT
|
||||
CLAR -->|"no"| ACT
|
||||
|
||||
subgraph ACT["step 9 — actionHandlers, 7 intents"]
|
||||
direction TB
|
||||
HF["fact<br/>actions_fact.go"]
|
||||
HR["reminder<br/>actions_reminder.go"]
|
||||
HA["act<br/>actions_act.go"]
|
||||
HN["note"]
|
||||
HC["chat"]
|
||||
HS["system"]
|
||||
HQ["query → the chain"]
|
||||
end
|
||||
|
||||
subgraph QC["the query chain — a SECOND arbitration, 22 sources"]
|
||||
direction TB
|
||||
QW["queryWalk<br/>removes only guesses:true sources<br/>when the cascade named a destination"]
|
||||
Q1["his data<br/>fact-by-key · day-plan · habits · tasks<br/>attention · list · money · history · feeds<br/>home · network · calendar · weather · self<br/>embed · memory · notes"]
|
||||
QB["personal boundary<br/>the only source a stage 0 anchor may drop"]
|
||||
Q2["the world<br/>search → kiwix → web → general-knowledge"]
|
||||
QW --> Q1 --> QB --> Q2
|
||||
end
|
||||
HQ --> QC
|
||||
|
||||
HF -->|"question-shaped ⇒ re-route"| QC
|
||||
HF -->|"complaint ⇒ re-route"| HC
|
||||
|
||||
subgraph STATE["state and memory"]
|
||||
direction TB
|
||||
DB[("store · sqlite<br/>facts · reminders · notes · tools<br/>tasks · lists · nudges")]
|
||||
VEC[("memory_vectors<br/>brute-force cosine")]
|
||||
DLGS[("dialogue_sessions<br/>persisted, TTL 2m")]
|
||||
CLS["clarifyStore<br/>IN MEMORY ONLY, by design"]
|
||||
PEND["pending act / routine / hexis<br/>3 single-slot registers, one mutex"]
|
||||
SURF["surfacedItems<br/>last Praxis read-out order"]
|
||||
RING["decision.Ring<br/>bounded, in memory"]
|
||||
end
|
||||
|
||||
HF --> DB
|
||||
HF --> VEC
|
||||
HR --> DB
|
||||
HN --> DB
|
||||
HN --> VEC
|
||||
Q1 --> DB
|
||||
Q1 --> VEC
|
||||
DLG --> DLGS
|
||||
ASK --> CLS
|
||||
HA --> PEND
|
||||
QC --> SURF
|
||||
D0 --> RING
|
||||
|
||||
subgraph TOOLS["act execution"]
|
||||
direction TB
|
||||
ALLOW[("tools table<br/>only status='enabled' runs")]
|
||||
EXEC["tool.Executor<br/>+ MCP + Home Assistant callers"]
|
||||
CONF["destructive confirm turn<br/>binds capability, entity, args, requester, expiry"]
|
||||
HEX["Hexis capability<br/>entity id resolved via Nexus first"]
|
||||
end
|
||||
HA --> ALLOW --> EXEC
|
||||
HA --> CONF
|
||||
HA --> HEX
|
||||
|
||||
subgraph RESP["response generation"]
|
||||
direction TB
|
||||
REP["replier<br/>only when the handler returned \"\""]
|
||||
PHR["phraser<br/>parseResponseMood is the one parser"]
|
||||
TTS["tts seam<br/>Remote mavttsd · Stub"]
|
||||
end
|
||||
ACT --> RESP
|
||||
QC --> RESP
|
||||
REP --> PHR
|
||||
OUT["reply text<br/>+ notice + resumed question"]
|
||||
RESP --> OUT
|
||||
OUT -->|"voice path only"| TTS
|
||||
|
||||
subgraph PROACT["the other half of the process — nothing above touches it"]
|
||||
direction TB
|
||||
TICKL["tick loop · 60s<br/>13 jobs in one function"]
|
||||
GTH["loop.Gatherer<br/>one consistent snapshot"]
|
||||
RUL["loop rules + restraint gate<br/>pure"]
|
||||
DISP["delivery.Dispatcher<br/>ChannelsFor severity,presence"]
|
||||
SNK["sinks: voice · ntfy · telegram"]
|
||||
TICKL --> GTH --> RUL --> TICKL
|
||||
TICKL --> DISP --> SNK
|
||||
end
|
||||
TICKL --> DB
|
||||
SNK -->|"PushToMostRecent on the request conn"| A1
|
||||
|
||||
classDef stage fill:#1f3a5f,stroke:#7fb3ff,color:#eaf2ff
|
||||
classDef store fill:#1f4a3a,stroke:#6ed0a8,color:#e8fff5
|
||||
classDef mem fill:#4a3a1f,stroke:#e0b050,color:#fff6e0
|
||||
classDef danger fill:#4f2626,stroke:#e08080,color:#ffecec
|
||||
class S0,SH,SL,SC,SE,CONT stage
|
||||
class DB,VEC,DLGS,ALLOW store
|
||||
class CLS,PEND,SURF,RING mem
|
||||
class QB,CONF danger
|
||||
@@ -0,0 +1,78 @@
|
||||
%% View 3a — Runtime flow: a reminder request.
|
||||
%% Traced through cmd/mavend/voice.go runTurn, internal/router/stagezero.go,
|
||||
%% cmd/mavend/clarify.go, cmd/mavend/actions_reminder.go, cmd/mavend/tick.go,
|
||||
%% internal/loop/loop.go and internal/delivery/dispatcher.go.
|
||||
%% Shows the branch where the hour is missing, the parked clarify, the answer
|
||||
%% turn, the write and the eventual delivery with durable retry.
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant K as Owner
|
||||
participant W as mavwaked
|
||||
participant V as voice.Server
|
||||
participant H as reactiveHandler.runTurn
|
||||
participant PRE as pre-route ladder
|
||||
participant R as router cascade
|
||||
participant CL as clarifyStore
|
||||
participant AR as actionReminder
|
||||
participant DB as store
|
||||
participant T as tick loop
|
||||
participant D as dispatcher
|
||||
|
||||
Note over K,W: "Мэйвен, напомни позвонить маме"
|
||||
K->>W: speech
|
||||
W->>W: silero VAD + keyword head, score ≥ 0.999
|
||||
W->>V: PushToTalkReq, one utterance
|
||||
V->>H: HandlePushToTalk
|
||||
H->>H: stt seam → text
|
||||
H->>H: decision.With, turnRoute created
|
||||
H->>PRE: 11 rungs
|
||||
PRE-->>H: nobody claims
|
||||
H->>R: rt.resolve
|
||||
R->>R: stage 0 ReminderGrammar matches, Stage=0, conf 1.0
|
||||
R-->>H: IntentReminder, Slots.Text="позвонить маме", HasTime=false
|
||||
|
||||
rect rgb(70,40,40)
|
||||
Note over H,CL: BRANCH — missingFor names `time`, whatever the confidence
|
||||
H->>H: dec.Clarify false BUT len missingFor > 0 → step 8 fires
|
||||
H->>CL: Push a PendingQuestion, park the request
|
||||
H-->>V: "во сколько напомнить?"
|
||||
V-->>W: reply audio + text
|
||||
end
|
||||
|
||||
Note over K,W: "в семь вечера"
|
||||
K->>W: speech
|
||||
W->>V: PushToTalkReq
|
||||
V->>H: runTurn
|
||||
H->>PRE: rung 4, resolveClarifyAnswer
|
||||
PRE->>CL: Pop the parked question
|
||||
PRE->>R: extractor parses the hour with the SAME parsers stage 2 uses
|
||||
PRE->>AR: finishClarified → applyAction
|
||||
Note right of AR: filling in an argument never grants authority
|
||||
AR->>AR: router.ResolvedTheHour guard
|
||||
AR->>DB: CreateReminder fire_ts, payload
|
||||
DB-->>AR: id
|
||||
AR-->>H: reminderConfirm, phrased FROM THE ROW not the utterance
|
||||
H-->>V: "хорошо, напомню сегодня в 19:00."
|
||||
|
||||
Note over T,D: later — the proactive half, no shared code with the turn path
|
||||
loop every 60s
|
||||
T->>DB: Gatherer.GatherState, due reminders, collapsed by group
|
||||
T->>T: loop.RemindDecisions — reminders BYPASS the restraint gate
|
||||
alt not cached
|
||||
T->>T: phraser.PhraseReminder
|
||||
end
|
||||
T->>D: DispatchReminder
|
||||
D->>DB: BeginDeliveryAttempt BEFORE the external send
|
||||
alt a voice session is live
|
||||
D->>V: voicesink push on the request conn
|
||||
else away
|
||||
D->>D: ntfy is nil (disabled) → telegram
|
||||
end
|
||||
alt success
|
||||
D->>DB: CompleteSuccessfulReminderAttempt + fire the originals, one txn
|
||||
else failure
|
||||
D->>DB: advance the persisted bounded backoff, next_attempt_ts
|
||||
end
|
||||
end
|
||||
|
||||
Note over T,DB: Recurring is NOT on this path. reminders.cron and next_fire_ts<br/>exist since migration #2 and no spoken path writes them.
|
||||
@@ -0,0 +1,61 @@
|
||||
%% View 3b — Runtime flow: a factual / state update.
|
||||
%% Traced through cmd/mavend/actions_fact.go, cmd/mavend/ack.go,
|
||||
%% cmd/mavend/patterns.go, cmd/mavend/factenrichment.go, cmd/mavend/intake.go
|
||||
%% and internal/morning.
|
||||
%% Shows the two re-route branches this handler owns, the vector prune-and-insert,
|
||||
%% the nudge it can close, and the async entity resolution behind it.
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant K as Owner
|
||||
participant H as runTurn
|
||||
participant R as router cascade
|
||||
participant AF as actionFact
|
||||
participant API as CoreAPI · intakeAPI then storeAPI
|
||||
participant DB as facts table
|
||||
participant VEC as memory_vectors
|
||||
participant BUS as event.Bus
|
||||
participant FE as fact-enrichment worker
|
||||
participant NX as Nexus
|
||||
participant T as tick loop
|
||||
|
||||
Note over K,H: "выпил воды"
|
||||
K->>H: utterance, src=tap:voice
|
||||
H->>R: rt.resolve
|
||||
R->>R: stage 0 declines → heads → LLM router → classifier
|
||||
R-->>H: IntentFact, Slots.Key="water", Slots.Value=...
|
||||
|
||||
rect rgb(70,40,40)
|
||||
Note over AF: two guards that RE-ROUTE rather than write
|
||||
AF->>AF: router.IsQuestionShaped? → becomes actionQuery, Key cleared
|
||||
AF->>AF: router.IsTransientComplaint? → becomes actionChat, nothing stored
|
||||
end
|
||||
|
||||
AF->>AF: factConfidence — 1.0 only for a value he actually said
|
||||
AF->>API: WriteFact kind=self, source=tap:voice, Subject=Key
|
||||
API->>DB: append-only row
|
||||
API->>BUS: publish one intake envelope
|
||||
API-->>AF: factID
|
||||
|
||||
AF->>VEC: pruneFactVectors by key
|
||||
AF->>VEC: EmbedPassage(FactRecallText) then Insert "fact:<key>:<unix>"
|
||||
Note right of VEC: the FACT is embedded, not the utterance.<br/>The utterance rides along as provenance only
|
||||
|
||||
AF->>DB: RecordEvent action+object, for pattern detection
|
||||
H->>H: step 9b ackFromFact — a fact answering a live nudge closes it as `acted`, silently
|
||||
|
||||
par asynchronous, minutes later
|
||||
FE->>DB: read facts with resolution_state='pending'
|
||||
FE->>NX: Resolve(Subject)
|
||||
alt resolved
|
||||
NX-->>FE: entity_id
|
||||
FE->>DB: UPDATE entity_id, resolution_state='resolved'
|
||||
else ambiguous
|
||||
Note right of FE: candidates are NOT stored —<br/>ambiguity blocks, it does not pick
|
||||
end
|
||||
and the next tick
|
||||
T->>DB: Gatherer reads the same row
|
||||
T->>T: morning routine item `water` is now evidenced, so it will not nudge
|
||||
T->>T: detectPatterns scans events for a stable interval
|
||||
end
|
||||
|
||||
Note over DB,VEC: A wrong value is superseded, never overwritten:<br/>voids_id points at the row it cancels, and CorrectValue /<br/>VoidLatestFact drop the key's vectors so recall keeps exactly one.
|
||||
@@ -0,0 +1,70 @@
|
||||
%% View 3c — Runtime flow: a world query, tool-backed.
|
||||
%% Traced through internal/router/worldquery.go, internal/router/source.go,
|
||||
%% cmd/mavend/actions_query.go queryWalk + querySources, cmd/mavend/personalboundary.go,
|
||||
%% cmd/mavend/searchwire.go, cmd/mavend/kiwixwire.go.
|
||||
%% Shows destination anchoring, which sources are skipped and why, and the
|
||||
%% four-step fallback to the model's own weights.
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant K as Owner
|
||||
participant H as runTurn
|
||||
participant R as router cascade
|
||||
participant QC as actionQuery
|
||||
participant W as queryWalk
|
||||
participant LOC as local sources
|
||||
participant PB as personal boundary
|
||||
participant SX as SearXNG
|
||||
participant KX as kiwix-server
|
||||
participant PH as phraser / resident model
|
||||
participant REC as decision record
|
||||
|
||||
Note over K,H: "что такое TCP?"
|
||||
K->>H: utterance
|
||||
H->>R: rt.resolve
|
||||
R->>R: stage 0 — WorldQueryGrammars matches a literal definition frame
|
||||
R->>R: d.SourceAnchored = true, set HERE and nowhere else
|
||||
R-->>H: IntentQuery, Source=SourceWorld, anchored
|
||||
|
||||
H->>QC: applyAction → actionQuery
|
||||
QC->>REC: Expect the full 22-source roster
|
||||
QC->>W: queryWalk(SourceWorld, anchored=true)
|
||||
|
||||
rect rgb(70,40,40)
|
||||
Note over W: removes ONLY sources with guesses:true whose dest ≠ world
|
||||
W-->>REC: skipped: attention, list, feeds, home, network, weather, self
|
||||
W-->>REC: skipped: personal boundary — anchored, so a literal pattern may drop it
|
||||
Note right of W: a model or a softmax naming SourceWorld<br/>would NOT drop the boundary (V-666)
|
||||
end
|
||||
|
||||
W-->>QC: the sources that LOOK still walk, in table order
|
||||
|
||||
loop first source to claim answers the turn
|
||||
QC->>LOC: fact-by-key, day-plan, habits, tasks, money, history, calendar
|
||||
LOC-->>QC: no rows → pass
|
||||
QC->>LOC: embed → memory → notes (vector recall, gated by min score + margin)
|
||||
LOC-->>QC: below the gate → pass
|
||||
QC->>PB: personal boundary
|
||||
PB-->>QC: SKIPPED this turn
|
||||
QC->>SX: Search(utterance verbatim, max 4)
|
||||
alt results
|
||||
SX-->>QC: snippets
|
||||
QC->>PH: phraseSource("search", utterance, evidence)
|
||||
PH-->>QC: reply
|
||||
QC->>REC: claimed by "search", and everyone below is NeverAsked
|
||||
else empty or unreachable
|
||||
QC->>KX: ZIM search, ru then en
|
||||
alt hit
|
||||
KX-->>QC: article snippet
|
||||
QC->>PH: phraseSource("kiwix", ...)
|
||||
else miss
|
||||
QC->>QC: "web" claims only if he named a URL out loud
|
||||
QC->>PH: queryGeneral — the model answers from its own weights, LAST
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
QC-->>H: reply text
|
||||
H-->>K: spoken or written answer
|
||||
|
||||
Note over LOC,SX: What leaves the box is the query string and nothing else.<br/>His notes, his facts, the persona block and the history never travel.
|
||||
Note over W,PB: With no destination named — the classifier arm sets none —<br/>the whole chain walks in table order. That is the floor.
|
||||
@@ -0,0 +1,115 @@
|
||||
%% View 4 — State ownership.
|
||||
%% Every persistent and shared store, its authoritative owner, its writers and
|
||||
%% readers, its synchronisation boundary and its lifecycle.
|
||||
%% Red = written by components that do not know about each other.
|
||||
%% Evidence: internal/store/schema.sql, internal/store/migrations.go,
|
||||
%% internal/store/crypt.go, cmd/mavend/voice.go, cmd/mavend/tick.go,
|
||||
%% cmd/mavweb/*.go, cmd/mavpoll/main.go, cmd/mavcaldav/main.go.
|
||||
flowchart LR
|
||||
|
||||
subgraph OWNER["authoritative owner — mavend, the only key holder"]
|
||||
STORE[("store.Store<br/>SetMaxOpenConns(1)<br/>every write serialised at the db")]
|
||||
end
|
||||
|
||||
subgraph LIFE["lifecycle of the database itself"]
|
||||
direction TB
|
||||
ENC[("maven.db.enc<br/>AES-256-GCM at rest<br/>volume dbdata")]
|
||||
TMP[("/dev/shm/maven-plain.db<br/>tmpfs working copy<br/>dies with the container")]
|
||||
ENC -->|"Open: decrypt"| TMP
|
||||
TMP -->|"Close: checkpoint, re-encrypt, atomic rename"| ENC
|
||||
SEAL["mavseal<br/>recovery only, VACUUM INTO"]
|
||||
TMP -.->|"when mavend was killed, not stopped"| SEAL
|
||||
SEAL -.-> ENC
|
||||
end
|
||||
STORE --- TMP
|
||||
|
||||
%% ------------- multiply written tables
|
||||
FACTS[("facts<br/>append-only, ts = valid-time<br/>correction sets voids_id")]:::multi
|
||||
NOTES[("notes<br/>float32 blob, brute-force scan")]:::multi
|
||||
TOOLS[("tools<br/>only status='enabled' executes")]:::multi
|
||||
|
||||
%% ------------- singly owned tables
|
||||
REM[("reminders")]
|
||||
NUD[("nudges — the restraint memory<br/>AND the only feedback input")]
|
||||
VEC[("memory_vectors<br/>marked with the embedder id")]
|
||||
PRES[("presence_state — singleton row")]
|
||||
EV[("events")]
|
||||
PROP[("proposed_routines")]
|
||||
DIG[("digest_entries — gate-BLOCKED candidates")]
|
||||
DEL[("delivery_attempts — the outbox")]
|
||||
ACK[("ack_sends")]
|
||||
DLGS[("dialogue_sessions — TTL 2m")]
|
||||
TASKS[("tasks")]
|
||||
LISTS[("list_items")]
|
||||
RTR[("routing_traces — 14-day retention")]
|
||||
RLB[("routing_labels")]
|
||||
ETR[("ecosystem_traces")]
|
||||
META[("meta — schema version + embedder marker")]
|
||||
|
||||
STORE --- FACTS & NOTES & TOOLS & REM & NUD & VEC & PRES & EV & PROP & DIG & DEL & ACK & DLGS & TASKS & LISTS & RTR & RLB & ETR & META
|
||||
|
||||
%% ------------- writers into facts
|
||||
WF1["actionFact — tap:voice / tap:text"] --> FACTS
|
||||
WF2["quiet toggle — config fact"] --> FACTS
|
||||
WF3["mavpoll — poll:netdata, poll:uptimekuma,<br/>infer:wg, poll:zenmoney"] --> FACTS
|
||||
WF4["mavcaldav — poll:caldav<br/>NOT DEPLOYED"]:::off -.-> FACTS
|
||||
WF5["mavweb — /api/signal presence,<br/>/api/ambient meeting time"] --> FACTS
|
||||
WF6["feed + crawl watermarks<br/>crawl:hash:*"] --> FACTS
|
||||
WF7["fact-enrichment worker<br/>entity_id, resolution_state"] --> FACTS
|
||||
WF8["tick loop tune()<br/>cooldown:<rule> feedback fact"] --> FACTS
|
||||
WF9["mavweb /api/revert<br/>voids the latest fact for a key"] --> FACTS
|
||||
|
||||
%% ------------- writers into notes
|
||||
WN1["actionNote"] --> NOTES
|
||||
WN2["RSS poller — source rss:*"] --> NOTES
|
||||
WN3["crawl watcher"] --> NOTES
|
||||
WN4["meeting capture"]:::off -.-> NOTES
|
||||
WN5["image description"]:::off -.-> NOTES
|
||||
WN6["netscan record"] --> NOTES
|
||||
|
||||
%% ------------- writers into tools
|
||||
WT1["seedTools from mavend.json"] --> TOOLS
|
||||
WT2["MCP discovery — proposed"]:::off -.-> TOOLS
|
||||
WT3["Home Assistant discovery<br/>proposed, always destructive"]:::off -.-> TOOLS
|
||||
WT4["mavweb POST /tools<br/>the ONLY enable path"] --> TOOLS
|
||||
|
||||
%% ------------- readers
|
||||
FACTS --> RD1["loop.Gatherer — the tick snapshot"]
|
||||
FACTS --> RD2["queryFactByKey · money · history · morning"]
|
||||
NOTES --> RD3["queryNotes · queryFeeds · recall"]
|
||||
VEC --> RD4["queryMemory · queryEmbed"]
|
||||
TOOLS --> RD5["tool.Matcher + tool.Executor"]
|
||||
NUD --> RD6["restraint gate · TuneCooldown · UnackedTelegramRules"]
|
||||
|
||||
%% ------------- in-memory shared state
|
||||
subgraph MEM["shared mutable state — process-local, no synchronisation boundary beyond a mutex"]
|
||||
direction TB
|
||||
CLS["clarifyStore<br/>per-reach stack · NOT persisted on purpose:<br/>a restart expires the open question"]
|
||||
PEND["pending act / pendingRoutine / pendingHexis<br/>3 single-slot registers under handler.mu<br/>last-asked wins, TTL each"]
|
||||
SURF["surfacedItems<br/>replaced by the next digest, NO TTL"]
|
||||
RING["decision.Ring — bounded, diagnosis only"]
|
||||
BUS["event.Bus — bounded journal, read surface only"]
|
||||
TICKM["tickLoop: lastPhrase, lastTrace, digestQ,<br/>routineLast, morningLast, lastProposalAt"]
|
||||
LASTR["lastRouted — the previous acted turn, for a spoken correction"]
|
||||
end
|
||||
|
||||
H1["reactiveHandler<br/>one instance, called from per-conn goroutines"] --- CLS
|
||||
H1 --- PEND
|
||||
H1 --- SURF
|
||||
H1 --- RING
|
||||
H1 --- LASTR
|
||||
TICKL["tickLoop"] --- TICKM
|
||||
INTAKE["intakeAPI decorator"] --- BUS
|
||||
|
||||
%% ------------- outside the database
|
||||
subgraph OUT["state outside the database"]
|
||||
direction TB
|
||||
PK[("passkeys.json<br/>OWNED BY mavweb, not mavend")]:::multi
|
||||
WK[("wrapped key blob<br/>written by mavend WrapKeyFn,<br/>triggered by mavweb")]:::multi
|
||||
MAIL[("mavmaild seen-UID file<br/>own volume · NOT DEPLOYED")]:::off
|
||||
BLOB[("media blobs · retention loop")]:::off
|
||||
end
|
||||
PK -.->|"a v1 blob + this file together<br/>recover the database key with no authenticator"| WK
|
||||
|
||||
classDef multi fill:#4f2626,stroke:#e08080,color:#ffecec
|
||||
classDef off fill:#3a3a3a,stroke:#888,color:#ccc,stroke-dasharray:4 3
|
||||
@@ -0,0 +1,141 @@
|
||||
%% View 5 — Dependency and boundary map.
|
||||
%% Architectural components, not classes. Highlights the cycle, the cross-layer
|
||||
%% calls, the duplicated responsibilities, the fan-in and fan-out hotspots, the
|
||||
%% process and IPC boundaries, and where a failure propagates.
|
||||
%% Evidence: cmd/mavend/boot.go, cmd/mavend/tick_api.go, cmd/mavend/voice.go,
|
||||
%% cmd/mavend/voicewire.go, internal/ipc/server.go, internal/delivery/channel.go.
|
||||
flowchart TB
|
||||
|
||||
subgraph B1["process boundary — mavend"]
|
||||
direction TB
|
||||
|
||||
subgraph L_EDGE["entry layer"]
|
||||
IPCS["ipc.Server<br/>fan-in: 6 processes<br/>+ 8 bypass function fields"]
|
||||
VSRV["voice.Server"]
|
||||
HTTPIN["telegram poller"]
|
||||
end
|
||||
|
||||
subgraph L_API["API layer"]
|
||||
DAPI["daemonAPI<br/>store adapter + 8 closures"]
|
||||
IAPI["intakeAPI decorator"]
|
||||
SAPI["ipc.NewStoreAPI"]
|
||||
end
|
||||
|
||||
subgraph L_TURN["turn layer"]
|
||||
RH["reactiveHandler<br/>GOD COMPONENT<br/>34 fields · fan-out ≈ 20"]
|
||||
TRT["turnRoute"]
|
||||
PRE["pre-route ladder · 11 rungs"]
|
||||
ATBL["actionHandlers · 7"]
|
||||
QCH["querySources · 22"]
|
||||
end
|
||||
|
||||
subgraph L_ROUTE["routing layer"]
|
||||
RTR["router.Router cascade"]
|
||||
G0["stage 0 grammars · 22+"]
|
||||
HDS["routing heads"]
|
||||
LLMR["LLM router"]
|
||||
CLF["classifier"]
|
||||
end
|
||||
|
||||
subgraph L_PROACT["proactive layer"]
|
||||
TICK["tickLoop<br/>13 jobs, one function<br/>fan-out ≈ 10"]
|
||||
GATH["loop.Gatherer"]
|
||||
RULES["loop rules + gate · pure"]
|
||||
DISP["delivery.Dispatcher"]
|
||||
end
|
||||
|
||||
subgraph L_WIRE["construction layer"]
|
||||
WIRE["wireVoice<br/>builds 17 subsystems<br/>returns voiceWiring"]
|
||||
BOOT["boot.go<br/>newDaemonAPI + startBackground"]
|
||||
end
|
||||
|
||||
subgraph L_STATE["state layer"]
|
||||
ST[("store.Store")]
|
||||
end
|
||||
end
|
||||
|
||||
subgraph B2["process boundary — modules"]
|
||||
MSTT["mavsttd"]
|
||||
MTTS["mavttsd"]
|
||||
MWEB["mavweb"]
|
||||
MPOLL["mavpoll"]
|
||||
end
|
||||
|
||||
subgraph B3["process boundary — workstation"]
|
||||
MWAKE["mavwaked"]
|
||||
MGPU["mavgpud"]
|
||||
end
|
||||
|
||||
subgraph B4["external services"]
|
||||
EXT["SearXNG · kiwix · Nexus · Praxis · Hexis<br/>Telegram · ntfy · Home Assistant"]
|
||||
end
|
||||
|
||||
%% ---------- boundaries
|
||||
MWEB -.->|"UNIX IPC · 3 conns"| IPCS
|
||||
MPOLL -.->|"UNIX IPC"| IPCS
|
||||
MWAKE -.->|"TCP over ssh · plaintext, no auth"| VSRV
|
||||
MWEB -.->|"TCP · /api/ptt"| VSRV
|
||||
RH -.->|"UNIX worker"| MSTT
|
||||
RH -.->|"UNIX worker"| MTTS
|
||||
RH -.->|"HTTP"| EXT
|
||||
RH -.->|"HTTP"| MGPU
|
||||
DISP -.->|"HTTP"| EXT
|
||||
|
||||
%% ---------- the cycle
|
||||
IPCS --> DAPI
|
||||
DAPI -->|"chatFn = handler.handleText"| RH
|
||||
RH -->|"h.api, back-patched by upgradeAPI"| DAPI
|
||||
|
||||
DAPI --> IAPI --> SAPI --> ST
|
||||
|
||||
%% ---------- turn layer
|
||||
VSRV --> RH
|
||||
HTTPIN --> DAPI
|
||||
RH --> TRT --> RTR
|
||||
RH --> PRE --> TRT
|
||||
RH --> ATBL --> QCH
|
||||
QCH --> ST
|
||||
ATBL --> ST
|
||||
RTR --> G0 & HDS & LLMR & CLF
|
||||
|
||||
%% ---------- cross-layer calls
|
||||
QCH -->|"CROSS-LAYER: a query source reads the tick loop"| TICK
|
||||
RH -->|"CROSS-LAYER: dataStore, the raw store beside the CoreAPI"| ST
|
||||
DAPI -->|"reads tick state"| TICK
|
||||
WIRE --> RH
|
||||
WIRE --> RTR
|
||||
WIRE --> DISP
|
||||
BOOT --> DAPI
|
||||
BOOT --> TICK
|
||||
|
||||
%% ---------- proactive
|
||||
TICK --> GATH --> ST
|
||||
TICK --> RULES
|
||||
TICK --> DISP
|
||||
DISP --> ST
|
||||
DISP -->|"voicesink pushes on the request conn"| VSRV
|
||||
|
||||
%% ---------- annotations
|
||||
DUP1["DUPLICATED RESPONSIBILITY<br/>two independent arbitrations decide a turn:<br/>the 22-grammar cascade, then the 22-source chain.<br/>Both are ordered lists; neither can compare scores."]:::note
|
||||
DUP1 -.- RTR
|
||||
DUP1 -.- QCH
|
||||
|
||||
DUP2["DUPLICATED RESPONSIBILITY<br/>restraint is decided twice:<br/>loop.Gate says whether a rule EMITS,<br/>delivery.ChannelsFor says where it LANDS.<br/>Deliberate, and documented in channel.go."]:::note
|
||||
DUP2 -.- RULES
|
||||
DUP2 -.- DISP
|
||||
|
||||
DUP3["DUPLICATED RESPONSIBILITY<br/>three unrelated components propose tool rows:<br/>config seeding, MCP discovery, HA discovery."]:::note
|
||||
DUP3 -.- ST
|
||||
|
||||
FRAG1["FRAGILE PATH<br/>4 seams degrade silently:<br/>workstation model → resident model,<br/>CW2 → mavsttd, heads → LLM → classifier,<br/>search → kiwix → weights.<br/>Nothing on the turn says which one answered."]:::warn
|
||||
FRAG1 -.- RTR
|
||||
FRAG1 -.- QCH
|
||||
|
||||
FRAG2["FAILURE PROPAGATION<br/>ipc.Server holds long-lived conns from 4 modules.<br/>Before V-638 that deadlocked EVERY shutdown and<br/>the deployed ciphertext went 11 days stale."]:::warn
|
||||
FRAG2 -.- IPCS
|
||||
|
||||
FRAG3["PLANNED, UNWIRED<br/>internal/claim + router.ClaimOf: a comparable<br/>unit of evidence for exactly the two arbitrations above.<br/>Nothing calls it. internal/modes: nothing imports it."]:::warn
|
||||
FRAG3 -.- RTR
|
||||
|
||||
classDef note fill:#2a3f2a,stroke:#7fbf7f,color:#eaffea
|
||||
classDef warn fill:#4f2626,stroke:#e08080,color:#ffecec
|
||||
@@ -0,0 +1,699 @@
|
||||
# Architecture findings: Maven as built
|
||||
|
||||
Read at commit `5cae33a`, 2026-08-25. Working tree dirty: `deploy/mavend.json`
|
||||
swaps `phraser.model_path` to `maven-instruct-b2-Q4_K_XL.gguf`, plus an edited
|
||||
`docs/evals/CLAUDE.md` and two untracked files.
|
||||
|
||||
This file is analysis. The factual inventory is
|
||||
`docs/architecture/maven-architecture.json` and the diagrams under
|
||||
`docs/architecture/diagrams/`. Nothing here proposes a new architecture.
|
||||
|
||||
**Revised 2026-08-25 after an independent second pass over the evidence pack.**
|
||||
Four readings changed, and section 6.3 contained one statement that was wrong:
|
||||
the voice server defaults an empty `Surface`, it does not overwrite the client's.
|
||||
The sections marked below carry the corrections.
|
||||
|
||||
**The ranking changed with them.** The missing end-to-end authority model
|
||||
(6.3 through 6.3d) is the first architectural issue, ahead of `reactiveHandler`
|
||||
size (4.1) and the process boundaries (section 5). Those are refactors. This one
|
||||
is a property nobody can state.
|
||||
|
||||
Each finding cites what it was read from. Where the repository already names a
|
||||
problem in its own comments, that is said. A known defect and an undiscovered
|
||||
one are different facts.
|
||||
|
||||
---
|
||||
|
||||
## 1. Unclear ownership
|
||||
|
||||
### 1.1 The `facts` table has nine writers and no owner
|
||||
|
||||
`internal/store/schema.sql` calls facts "substrate, all observations". Nine
|
||||
components append to it, and no component owns the key namespace:
|
||||
|
||||
| Writer | Source tag | Evidence |
|
||||
|---|---|---|
|
||||
| `actionFact` | `tap:voice`, `tap:text` | `cmd/mavend/actions_fact.go` |
|
||||
| quiet-hours toggle | `config` | `cmd/mavend/quiet_toggle.go` |
|
||||
| mavpoll | `poll:netdata`, `poll:uptimekuma`, `infer:wg`, `poll:zenmoney` | `cmd/mavpoll/main.go` |
|
||||
| mavcaldav | `poll:caldav` | `cmd/mavcaldav/main.go`, not deployed |
|
||||
| mavweb | presence, ambient meeting time | `cmd/mavweb/facts.go`, `cmd/mavweb/ambient.go` |
|
||||
| feed worker | RSS watermark | `cmd/mavend/feeds.go` |
|
||||
| crawl worker | `crawl:hash:<name>` | `cmd/mavend/crawls.go` `hashKey` |
|
||||
| fact-enrichment worker | mutates `entity_id`, `resolution_state` | `cmd/mavend/factenrichment.go` |
|
||||
| tick loop autotune | `cooldown:<rule>` | `cmd/mavend/tick.go` `tune`, `internal/loop/feedback.go` `FeedbackKey` |
|
||||
|
||||
Two of these are not observations at all. `crawl:hash:*` is a fetch watermark
|
||||
and `cooldown:<rule>` is a tuning parameter. Both live in the same append-only
|
||||
table that recall embeds and that `queryFactByKey` reads back as an answer. The
|
||||
`source` column is what keeps them apart, and it is a convention, not a
|
||||
constraint: `schema.sql` documents the vocabulary in a comment and the `CHECK`
|
||||
covers only `kind`.
|
||||
|
||||
### 1.2 `notes` has six writers and one of them is a LAN scan
|
||||
|
||||
`cmd/mavend/netscan.go` `writeScanRecord` writes a scan result as a note. Notes
|
||||
are the recall corpus: `queryNotes` and `queryMemory` answer from them. So a
|
||||
network scan record competes by cosine similarity with things he said.
|
||||
|
||||
### 1.3 `tools` is proposed by three unrelated components
|
||||
|
||||
Config seeding (`seedTools`), MCP discovery (`cmd/mavend/mcp.go` `propose`) and
|
||||
Home Assistant discovery (`cmd/mavend/smarthome.go` `propose`) all write rows.
|
||||
Only `mavweb` `POST /tools` can enable one, which is the invariant that holds.
|
||||
But nothing arbitrates a name collision between the three proposers, and
|
||||
`tools.name` is the primary key.
|
||||
|
||||
### 1.4 The day plan has no store and two owners
|
||||
|
||||
`queryDayPlan` is a query source. The day plan it reads is assembled by the tick
|
||||
loop (`cmd/mavend/tick_morning.go` `dayPlan`). The bare store adapter cannot
|
||||
answer it, which is why `upgradeAPI` exists at all (finding 3.1). So a read of
|
||||
his calendar depends on a proactive scheduler being wired.
|
||||
|
||||
---
|
||||
|
||||
## 2. Duplicated responsibilities
|
||||
|
||||
### 2.1 Two independent arbitrations decide one turn
|
||||
|
||||
The cascade sorts an utterance into one of seven intents through four arms
|
||||
(`internal/router/router.go` `Route`). An `IntentQuery` then enters a second
|
||||
arbitration of twenty-two ordered sources (`cmd/mavend/actions_query.go`
|
||||
`querySources`, counted in the source). Both are ordered lists. Neither can
|
||||
compare scores across arms.
|
||||
|
||||
The repository states this itself, in `internal/router/source.go`:
|
||||
|
||||
> The cascade sorted an utterance into one of seven intents with stage 0 rules,
|
||||
> the resident model and the classifier behind it, a fixture measuring it and
|
||||
> the decision trace recording it. Then IntentQuery handed the turn to
|
||||
> querySources in the daemon, a chain of twenty-two branches deciding by seed
|
||||
> similarity in a fixed order, with none of that.
|
||||
|
||||
`Source` and `queryWalk` narrow the second arbitration with a decision from the
|
||||
first. They do not merge the two.
|
||||
|
||||
### 2.2 A third arbitration runs before both
|
||||
|
||||
`runTurn` steps 1 through 5e are eleven stateful pre-emptors, each answering "is
|
||||
this mine?" alone (`cmd/mavend/voice.go`, `preRouteLadder` in
|
||||
`cmd/mavend/decisiontrace.go`). Their order is argued rung by rung in comments.
|
||||
That is three ordered lists deciding one utterance, in three files, with three
|
||||
different notions of confidence.
|
||||
|
||||
`internal/claim/claim.go` names exactly this and counts it:
|
||||
|
||||
> Maven's cascade has twenty-two stage-0 grammars, seven router intents,
|
||||
> twenty-two query sources and seven stateful pre-emptors, and every one of them
|
||||
> answers "is this mine?" alone. None can answer "is this more mine than
|
||||
> yours?" … So list order is the whole arbitration.
|
||||
|
||||
The unit that would fix it is written, tested and called by nothing. See 6.1.
|
||||
|
||||
### 2.3 Restraint is decided twice, deliberately
|
||||
|
||||
`internal/loop/loop.go` `Gate` decides whether a rule emits.
|
||||
`internal/delivery/channel.go` `ChannelsFor` decides where it lands, and drops
|
||||
care nudges on away for its own reasons. `channel.go` argues the duplication:
|
||||
|
||||
> double authority is intentional: the gate decides whether a rule EMITS;
|
||||
> delivery decides where it LANDS.
|
||||
|
||||
Recorded here as duplication that is owned, not as a defect.
|
||||
|
||||
### 2.4 Two digest mechanisms with the same word in the name
|
||||
|
||||
`tickLoop.digestQ` is an in-memory queue batching candidates the gate **allowed**.
|
||||
`digest_entries` is a table durably holding candidates the gate **blocked**. Both
|
||||
are flushed in the same `tick()` body, six lines apart
|
||||
(`cmd/mavend/tick_digest.go`). The distinction is carried entirely by a comment.
|
||||
|
||||
---
|
||||
|
||||
## 3. Accidental coupling
|
||||
|
||||
### 3.1 A construction cycle between the API layer and the turn layer
|
||||
|
||||
Two back-patches, each documented, together forming a cycle:
|
||||
|
||||
- `cmd/mavend/boot.go`: `api.chatFn = d.voiceW.handler.handleText`
|
||||
- `cmd/mavend/voice.go` `upgradeAPI`: `h.api = api`, the daemon's own CoreAPI
|
||||
|
||||
So `daemonAPI` holds the handler and the handler holds `daemonAPI`. The comment
|
||||
on `upgradeAPI` states the reason and the safety argument:
|
||||
|
||||
> Wiring order forces this. wireVoice runs before the tick loop exists … main
|
||||
> already back-patches the other direction … this is the same seam in reverse.
|
||||
> Safe against the obvious loop: nothing in the voice path calls api.Chat.
|
||||
|
||||
The safety rests on a negative that nothing enforces. Adding a query source that
|
||||
calls `api.Chat` would recurse.
|
||||
|
||||
### 3.2 The handler holds the raw store beside the mediated one
|
||||
|
||||
`reactiveHandler` carries both `api ipc.CoreAPI` and
|
||||
`dataStore *store.Store`, "direct store access for event extraction + pattern
|
||||
detection" (`cmd/mavend/voice.go`). `internal/ipc/frame.go` states the opposing
|
||||
rule for the boundary:
|
||||
|
||||
> Core mediates, never hands back a db handle … Anything needing raw db access
|
||||
> lives in core and is unreachable.
|
||||
|
||||
That holds across the process boundary and not inside it. The turn path has two
|
||||
ways to reach the same tables, with different auditing.
|
||||
|
||||
### 3.3 The intake journal is bypassed by the one path that needed it
|
||||
|
||||
`cmd/mavend/intake.go` decorates `CoreAPI` so every intake write narrates
|
||||
itself, and names its own exception:
|
||||
|
||||
> The exception is cmd/mavend/mail.go, which reaches past the interface to
|
||||
> st.CaptureTask directly. It publishes explicitly.
|
||||
|
||||
One caller reaching past a decorator means the decorator is not the boundary it
|
||||
claims to be.
|
||||
|
||||
### 3.4 A query source reads the proactive scheduler
|
||||
|
||||
`queryDayPlan` → `tickLoop.dayPlan`. The reactive and proactive halves otherwise
|
||||
share only the store. This is the single call across that line, and it is the
|
||||
reason for the `upgradeAPI` back-patch in 3.1.
|
||||
|
||||
---
|
||||
|
||||
## 4. God components
|
||||
|
||||
### 4.1 `reactiveHandler` has 34 fields
|
||||
|
||||
`cmd/mavend/voice.go:75`. One struct holds stt, tts, the router, the CoreAPI,
|
||||
the raw store, the tool executor and matcher, the phraser, the replier, the
|
||||
recall wiring, the crawler, the search client, the Kiwix client, the feeds flag,
|
||||
the Home Assistant wiring, the LAN scanner, the weather provider and its default
|
||||
location, the time parser, the dialogue session store, the decision ring, the
|
||||
trace writer, the encoder id, the clarify store and its attempt cap, the
|
||||
extractor, a mutex, `lastRouted`, three pending-confirmation registers,
|
||||
`surfacedItems`, and the ecosystem clients.
|
||||
|
||||
`docs/handler-wiring.md` exists because grouping five of these into `recall`
|
||||
was itself a task (Vikunja #433).
|
||||
|
||||
Every query source, every action handler and every pre-route resolver is a
|
||||
method on this one type. There is no seam between "the thing that routes a
|
||||
turn" and "the thing that knows the house is a Home Assistant".
|
||||
|
||||
### 4.2 `runTurn` is one function with eleven early returns
|
||||
|
||||
`cmd/mavend/voice.go:270`, about 226 lines. Two deferred finalisers, six numbered
|
||||
steps with lettered sub-steps up to `5e`, and an explicit statement that the
|
||||
ordering is load-bearing. Eleven of the returns are `return withNotice(...)`
|
||||
from a pre-emptor.
|
||||
|
||||
### 4.3 `tick` runs thirteen jobs in one function
|
||||
|
||||
`cmd/mavend/tick.go:160`. Gather, save presence, pick a candidate, queue or
|
||||
phrase-and-dispatch, flush the digest, enqueue gate-suppressed candidates,
|
||||
expire stale digest, drain digest, fire routines, fire accepted routines, fire
|
||||
morning routines, detect patterns, deliver reminders, repeat un-acked sev4
|
||||
alarms. One 60s ticker drives all of it, so a slow phraser call delays every job
|
||||
after it.
|
||||
|
||||
### 4.4 `wireVoice` is one constructor for seventeen subsystems
|
||||
|
||||
`cmd/mavend/voicewire.go:108`, about 270 lines, returning a `voiceWiring` struct
|
||||
whose fields the rest of the daemon reaches into (`embedderOf`, `nexusOf`,
|
||||
`d.voiceW.mcp`, `d.voiceW.home`, `d.voiceW.server`, `d.voiceW.handler`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Process boundaries
|
||||
|
||||
### 5.1 Unnecessary: `mavsttd` and `mavttsd` at current scale
|
||||
|
||||
Both are justified in their own headers as "restart-free, key-free,
|
||||
fail-independent". Both run in the same container image, on the same host, as
|
||||
the same user, over a socket in a shared volume, and both are hard dependencies
|
||||
of a turn: `HandlePushToTalk` returns an error reply when either is unavailable.
|
||||
The key argument is real but partial. `internal/ipc/frame.go` says "a crashing
|
||||
tts can't read the key page", and the same holds for any goroutine that never
|
||||
touches the key.
|
||||
|
||||
The boundary earns itself for a different reason the docs do not lead with:
|
||||
whisper.cpp and piper are cgo and subprocess dependencies, so an in-process
|
||||
crash would be a daemon crash. Recorded as a boundary whose stated reason and
|
||||
real reason differ.
|
||||
|
||||
### 5.2 Unnecessary: three IPC connections from one process
|
||||
|
||||
`cmd/mavweb/main.go` opens `core`, `swapConn` and `turnConn` to the same socket,
|
||||
because `ipc.Client` serialises every call on one mutex and a model swap or a
|
||||
chat turn would otherwise freeze every page. The comments say so. Connection
|
||||
count is standing in for request concurrency.
|
||||
|
||||
### 5.3 Missing: the turn path and the tick loop are one process
|
||||
|
||||
They share `store.Store` at `SetMaxOpenConns(1)`, one `phraser.Phraser` and one
|
||||
`llm.Gate`. A reminder being phrased and a spoken turn being answered contend
|
||||
for the same llama-server through `internal/llm/gate.go`. Nothing isolates a
|
||||
foreground turn from a background job beyond that gate.
|
||||
|
||||
### 5.4 Missing: the act executor runs in the key holder
|
||||
|
||||
`internal/tool/tool.go:238` is `exec.CommandContext(ctx, argv[0], argv[1:]...)`,
|
||||
running inside mavend, the only process holding the database key.
|
||||
`deploy/mavend.json` seeds twelve rows, five of them destructive, including
|
||||
`systemctl restart`, `docker restart` and `systemctl reboot`.
|
||||
|
||||
The controls are the enabled allowlist, the risk tier (6.3b) and the confirm
|
||||
turn. The process boundary is not one of them. `internal/tool/risk.go:84` says
|
||||
so directly: "It is not a sandbox and it does not try to be one. An enabled row
|
||||
can already run anything the daemon's user can run."
|
||||
|
||||
### 5.5 The one boundary that is load-bearing and undefended by itself
|
||||
|
||||
The voice TCP wire is plaintext with no auth (`internal/voice/server.go`). Its
|
||||
security argument is entirely external: loopback publish plus an ssh tunnel
|
||||
(`docker-compose.yml` `ports: ["127.0.0.1:9110:9100"]`,
|
||||
`deploy/mavwaked.service` `Requires=maven-voice-tunnel.service`). Correct, and
|
||||
it means a single compose edit silently removes the whole control.
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation disagreeing with apparent responsibility
|
||||
|
||||
### 6.1 `internal/claim` and `router.ClaimOf` are called by nothing
|
||||
|
||||
`internal/router/claim.go` says so in its own doc comment:
|
||||
|
||||
> Nothing in Route calls this yet. The arbiter that reads claims is V-560.
|
||||
|
||||
V-560 landed as `turnRoute` (memoise the route), not as an arbiter. The package
|
||||
and its `router` adapter are complete and tested and are on no path.
|
||||
|
||||
### 6.2 `internal/modes` is imported by nothing outside itself
|
||||
|
||||
`grep -rn "internal/modes"` over `cmd/` and `internal/` returns only its own
|
||||
test. It describes itself as "the roughly thirty distinct downstream behaviours
|
||||
mavend has". That is an inventory of the very thing findings 2.1 and 2.2 are
|
||||
about.
|
||||
|
||||
### 6.3 The auth tier system does not bind the turn path
|
||||
|
||||
`internal/auth/tier.go` documents "voice can never reach EnableTool, not
|
||||
because we check the method, but because the surface can't carry the layer",
|
||||
and `MaxLayer(SurfaceVoice)` returns `Layer0`. `cmd/mavwaked/main.go:275` duly
|
||||
sends `Surface: voice.SurfaceVoice` on the wire.
|
||||
|
||||
Nothing in `cmd/mavend` reads it. `grep -rn "internal/auth" cmd/ internal/`
|
||||
outside tests returns `cmd/mavend/main.go` (building the IPC `Gate`),
|
||||
`cmd/mavweb/webauthn.go`, `internal/webauthn/session.go` and
|
||||
`internal/voice/wire.go` (type aliases only). `actionAct`
|
||||
(`cmd/mavend/actions_act.go`) contains no surface check.
|
||||
|
||||
**Two representations of reach exist, and both are ignored.** An earlier draft
|
||||
of this file said the server overwrites the client's value. It does not.
|
||||
|
||||
1. **Client-asserted, and it survives.** `internal/voice/server.go:198` reads
|
||||
`if p.Surface == "" { p.Surface = SurfacePCClient }`. That defaults an empty
|
||||
field. `mavwaked`'s `SurfaceVoice` arrives intact and reaches
|
||||
`HandlePushToTalk`, which ignores it (`cmd/mavend/voice.go:200`, the
|
||||
parameter is `req` and only `req.Audio` is read).
|
||||
2. **Server-created, and it is wrong.** `internal/voice/server.go:148` is
|
||||
`sess := s.sessions.Add(c, SurfacePCClient)`, hardcoded for every connection
|
||||
whatever the peer is. Nothing reads that either.
|
||||
|
||||
The consequence matters more than the finding. `req.Surface` is request payload
|
||||
on a plaintext wire with no auth, so **any voice-wire client can claim
|
||||
`"pc_client"`**. It must not become an authorization input as it stands. A reach
|
||||
has to be derived from the transport or the session, never trusted from the
|
||||
body.
|
||||
|
||||
`auth.Can` runs only in `ipc.Server.Check`, and `FloorEnrollment` maps every
|
||||
same-uid caller there to `SurfaceCoreProcess` / `Layer3`
|
||||
(`internal/auth/enrollment.go:65`).
|
||||
|
||||
The comments in `cmd/mavwaked/main.go`, `deploy/mavwaked.service` and
|
||||
`CLAUDE.md` all present "SurfaceVoice caps acts at L0" as a live control. On the
|
||||
reactive turn path, `internal/auth` is not what enforces it. Finding 6.3b is.
|
||||
|
||||
### 6.3b There is a second tier system, it is live, and it is not keyed on the reach
|
||||
|
||||
`internal/tool/risk.go` carries its own two-axis policy, and this one runs on
|
||||
every act:
|
||||
|
||||
```go
|
||||
policy := PolicyFor(RiskOf(t)) // internal/tool/tool.go:181
|
||||
if !policy.VoiceMayRun { return "", ErrNeedsAuthedSurface }
|
||||
if policy.Confirm && !confirmed { return "", ErrNeedsConfirm }
|
||||
```
|
||||
|
||||
`RiskOf` sorts a row into `TierSafe`, `TierDestructive` or `TierIrreversible`.
|
||||
`PolicyFor` maps those to `{Confirm:false, VoiceMayRun:true}`,
|
||||
`{Confirm:true, VoiceMayRun:true}` and `{Confirm:true, VoiceMayRun:false}`
|
||||
(`internal/tool/risk.go:69`). So the control that actually stops an act is real,
|
||||
well argued, and fails safe on an unknown shape.
|
||||
|
||||
Two observations about it:
|
||||
|
||||
1. **`VoiceMayRun` is not conditioned on voice.** `Executor.Exec` takes
|
||||
`(ctx, name, args, confirmed)` and no surface. The same policy is applied to
|
||||
the mic, to telegram inbound and to `POST /api/chat` on the authed page. A
|
||||
field named for a reach is evaluated identically for every reach.
|
||||
2. **`systemctl reboot` is `TierDestructive`, not `TierIrreversible`.**
|
||||
`irreversibleVerbs` (`internal/tool/risk.go:88`) lists `rm`, `mkfs`, `dd`,
|
||||
`prune`, `truncate` and eleven more. `reboot` is not among them, and
|
||||
`deploy/mavend.json` seeds it as an enabled row with `destructive: true`. So
|
||||
it runs on the reactive path after one spoken "да", which is exactly what
|
||||
`PolicyFor(TierDestructive)` says and is worth stating out loud.
|
||||
|
||||
So the repository has two tier systems: `Surface × Layer` in `internal/auth`,
|
||||
unread on the turn path, and `Risk × Policy` in `internal/tool`, live.
|
||||
|
||||
They are **not two implementations of one idea**, which is how an earlier draft
|
||||
of this file read. They are two orthogonal dimensions that never meet. `auth`
|
||||
answers who or where may carry what authority. `tool` answers what effect a
|
||||
capability has and what proof it demands. The decision that combines them does
|
||||
not exist anywhere.
|
||||
|
||||
That both dimensions are also thin today makes the gap easier to see:
|
||||
|
||||
- `FloorEnrollment.Lookup` maps **every** same-uid IPC caller to
|
||||
`SurfaceCoreProcess` (`internal/auth/enrollment.go:65`), so the process-radius
|
||||
distinction behind IPC is a future contract, not a live one.
|
||||
- `PasskeySession` is one global timestamp. `CurrentLayer` and `Assert` both
|
||||
take a `Scope` and both ignore it (`internal/webauthn/session.go:38` and
|
||||
`:62`), so step-up is per-daemon rather than per-scope.
|
||||
|
||||
### 6.3c The act policy is more distributed than one gate
|
||||
|
||||
`RiskOf → PolicyFor → Executor.Exec` is one of three act paths, not the act
|
||||
path.
|
||||
|
||||
| path | risk policy? | evidence |
|
||||
|---|---|---|
|
||||
| local tool row | yes | `internal/tool/tool.go:181` |
|
||||
| Hexis capability | yes, explicitly reused | `cmd/mavend/ecosystem_acts.go:768` `tool.RiskOfCapability` then `tool.PolicyFor` |
|
||||
| Praxis lifecycle | **no** | `cmd/mavend/ecosystem_acts.go:158` `praxisItemAction.handle` calls `a.call(ctx, px, id)` directly |
|
||||
|
||||
Acknowledge, resolve, ignore and pin are remote mutations that run on first
|
||||
hearing, with no tier and no confirm turn. They are reversible on the Praxis
|
||||
side, which is a reason, and it is a reason nothing in the code states.
|
||||
|
||||
`Exec` also has no proof that its `confirmed bool` was bound correctly. The
|
||||
invariant that a confirmation names one capability, one target and an expiry
|
||||
lives in `pendingAct` and `resolveConfirm` (`cmd/mavend/confirm.go`), not at the
|
||||
boundary that acts on it. `Exec` trusts the boolean because only two callers
|
||||
exist today.
|
||||
|
||||
So the authorization function is spread across origin handling, routing, parked
|
||||
confirm state, risk classification, allowlist state and execution. Section
|
||||
"The authorization function as implemented" in `README.md` writes down the part
|
||||
that is one expression. The rest is not.
|
||||
|
||||
### 6.3d `Claim.Coverage` returns 1.0 for a claim that extracted nothing
|
||||
|
||||
`ClaimOf` builds its consumed span from `claimSpans`, which includes
|
||||
`d.Slots.Text` unconditionally (`internal/router/claim.go:38`).
|
||||
`Router.fillSlots` backfills the raw utterance into `Text` for a note, a query
|
||||
and a chat turn (`internal/router/router.go:334`, `if d.Slots.Text == "" &&
|
||||
d.Intent != IntentReminder`).
|
||||
|
||||
`claim.Split` then marks every token of the utterance explained, and
|
||||
`Coverage()` is `len(Consumed) / total` (`internal/claim/claim.go:122`). A query
|
||||
claim that extracted nothing scores 1.0, and `MoreSpecificThan` reads coverage
|
||||
first.
|
||||
|
||||
`filledSlots` in the same file already knows about this: it counts `Text` "only
|
||||
when it differs from the whole utterance". `claimSpans`, four functions above
|
||||
it, does not.
|
||||
|
||||
`internal/router/claim_test.go` does not catch it. All five cases in
|
||||
`TestClaimOfBands` set `Text` equal to `Utterance`, and the test asserts `Band`
|
||||
only. Coverage is never asserted anywhere.
|
||||
|
||||
This is why `internal/claim` is not yet an answer to "what competes for a turn".
|
||||
It is the beginning of a vocabulary. It also has no production callers, no
|
||||
builders for query sources or pre-route claimants, and it identifies only the
|
||||
seven-intent destination rather than the roughly thirty behaviours
|
||||
`internal/modes` enumerates. Keeping it unwired is the right state until that is
|
||||
resolved, and the file's own comment already warns against it becoming a fourth
|
||||
arbitration layer.
|
||||
|
||||
### 6.4 Two query sources do not do what their names say
|
||||
|
||||
Two of the twenty-two "query sources" have side effects or read a different
|
||||
substrate than their name implies. `queryNetwork` triggers a live LAN scan
|
||||
inside a read path (`cmd/mavend/netscan.go` `scanSummary`), and the scan writes
|
||||
a note.
|
||||
|
||||
### 6.5 `actionFact` answers queries and chat
|
||||
|
||||
`cmd/mavend/actions_fact.go` re-routes a question-shaped utterance into
|
||||
`actionQuery` and a complaint into `actionChat`. Both re-routes are argued and
|
||||
correct in effect. The consequence is that the fact handler is one of three
|
||||
entry points into the query chain.
|
||||
|
||||
### 6.6 `mavgpud`'s model arm is off and its STT arm is on
|
||||
|
||||
`deploy/mavend.json` sets `workstation.model_disabled: true` while
|
||||
`workstation.stt` is live. One config block, two independently authenticated
|
||||
services, one flag that turns off half of it. The block's own comment explains
|
||||
this. A reader of the topology would not guess it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Hidden shared state
|
||||
|
||||
### 7.1 Six context keys carry per-turn state
|
||||
|
||||
`querySourceKey`, `turnRouteKey`, `dialogueKey`, `ecosystemCorrelationKey`,
|
||||
`traceIDKey` (all `cmd/mavend/`), and `recorderKey`
|
||||
(`internal/decision/decision.go`). Plus `callerKey` in `internal/ipc/api.go`.
|
||||
|
||||
Every one is invisible in a function signature. `turnRouteFrom` returns nil
|
||||
"when the caller is not inside runTurn, a unit test calling one resolver
|
||||
directly, most often". That is the shape of the problem: a resolver behaves
|
||||
differently depending on invisible context.
|
||||
|
||||
### 7.2 Three single-slot confirmation registers under one mutex
|
||||
|
||||
`reactiveHandler.pending`, `pendingRoutine`, `pendingHexis`
|
||||
(`cmd/mavend/voice.go:170-180`). The comment states the posture: "single slot,
|
||||
single-user box, a second act while one waits overwrites it (last-asked wins)".
|
||||
Three separate registers, one shared mutex, and the pre-route ladder decides
|
||||
between them by position rather than by comparing them.
|
||||
|
||||
### 7.3 `surfacedItems` has no TTL
|
||||
|
||||
Same struct. The comment argues it: a stale position resolves to an item Praxis
|
||||
reports as already acknowledged, "which is a harmless answer, unlike a stale
|
||||
confirmation". That is correct given Praxis is the arbiter. It also means an
|
||||
ordinal can refer to a list read out an arbitrarily long time ago.
|
||||
|
||||
### 7.4 The tick loop's memory is in-process and unbounded in one place
|
||||
|
||||
`tickLoop.lastPhrase` is a `map[string]delivery.PhrasedNudge` keyed by rule
|
||||
name, and rules are a fixed set, so it is bounded. `digestQ` is a slice with a
|
||||
config `MaxItems`. `lastProposalAt` is deliberately not persisted: "a restart is
|
||||
allowed to permit one more announcement".
|
||||
|
||||
### 7.5 The clarify store is deliberately not persisted, while the dialogue store is
|
||||
|
||||
`cmd/mavend/voicewire.go`: `dialogue.NewPersistentSessionStore` for follow-up
|
||||
slots, `dialogue.NewClarifyStore` for the parked question. The reasoning is
|
||||
recorded (Vikunja #385). The consequence is that a restart mid-clarify silently
|
||||
drops a request the user believes is parked, and the "expired clarify notice"
|
||||
path in `runTurn` step 1 cannot fire for it, because the store it reads is gone
|
||||
too.
|
||||
|
||||
---
|
||||
|
||||
## 8. Fragile request paths
|
||||
|
||||
### 8.1 Four silent degradations stacked on one turn
|
||||
|
||||
| Seam | Falls back to | Told to the user? |
|
||||
|---|---|---|
|
||||
| workstation model → resident model | `internal/llm/remote.go` `Pair.Complete` | no, by design (`docs/offload.md`) |
|
||||
| CW2 → mavsttd | `cmd/mavend/voicewire.go` `sttSeam` | no |
|
||||
| routing heads → LLM router → classifier | `internal/router/router.go` | no |
|
||||
| search → kiwix → named page → model weights | `cmd/mavend/actions_query.go` | no |
|
||||
|
||||
Each is individually argued. Together, a single answer can be the resident model
|
||||
routing a worse transcript with the classifier as a floor and answering from its
|
||||
own weights, and nothing in the reply distinguishes that from the best case. The
|
||||
only instrument is the decision record and the query-source log line.
|
||||
|
||||
### 8.2 The reminder path depends on a table nobody writes
|
||||
|
||||
`queryCalendar` reads `facts(kind=env, source=caldav:*)`, and `mavcaldav` is
|
||||
commented out in `docker-compose.yml`. `loop.State.CalendarBusy` reads the same
|
||||
facts, so the "do not nag mid-meeting" suppressor is permanently false. The
|
||||
compose comment says both of these explicitly, which makes it a known gap rather
|
||||
than a hidden one.
|
||||
|
||||
### 8.3 Recurring reminders have storage, an IPC parameter, and no caller
|
||||
|
||||
`reminders.cron` and `reminders.next_fire_ts` exist since migration #2
|
||||
(`internal/store/migrations.go`). `ipc.CreateReminder` takes a cron argument.
|
||||
`actionReminder` passes `""`. Nothing on the spoken path can create one.
|
||||
|
||||
### 8.4 Shutdown is a known past failure with a bounded workaround
|
||||
|
||||
`cmd/mavend/main.go` carries the history: long-lived module connections
|
||||
deadlocked every shutdown, `run()` never returned, `defer st.Close()` never
|
||||
sealed, and "the deployed ciphertext was eleven days stale before anyone
|
||||
noticed". The fix is `workerGrace = 4 * time.Second` plus tracked connections.
|
||||
A worker parked in a model call still loses its tick, and the seal proceeds
|
||||
without it.
|
||||
|
||||
### 8.5 One inbound worker is outside the assertable worker set
|
||||
|
||||
`backgroundWorkers` in `cmd/mavend/boot.go` exists so "a test can compare the
|
||||
set the two paths would start without standing a daemon up". `wireTelegramIntake`
|
||||
starts its poller with `wg.Add(1)` and a bare goroutine
|
||||
(`cmd/mavend/telegramintake.go:41`), so it is not in that set. It is at least on
|
||||
the outer `WaitGroup`, unlike the seven workers V-639 fixed.
|
||||
|
||||
### 8.6 The daemon is wired twice, in two places
|
||||
|
||||
`run()` wires everything at boot. `srv.UnlockFn` wires everything again after a
|
||||
passkey assertion. `boot.go` exists because those two lists had already drifted:
|
||||
"seven workers started untracked on the unlock path and two daemonAPI fields
|
||||
were never set there, silently". Both paths now funnel through `newDaemonAPI`
|
||||
and `startBackground`. But `wireRules`, `wireGatherer`, `wirePhraser`,
|
||||
`wireEcosystem`, `wireVoice`, `wireDispatcher`, `wireTickLoop`, the four worker
|
||||
constructors, `wireMailIntake`, `wireModelSwap`, `wireTelegramIntake`,
|
||||
`wireVision`, `wireCapture` and `wireSpeaker` are still listed twice, by hand,
|
||||
in the same file.
|
||||
|
||||
---
|
||||
|
||||
## 9. Difficult-to-test boundaries
|
||||
|
||||
### 9.1 A resolver's behaviour depends on invisible context
|
||||
|
||||
See 7.1. `turnRouteFrom(ctx)` returning nil is the documented test case, and it
|
||||
changes what the resolver does.
|
||||
|
||||
### 9.2 The single-instance handler is the unit under test for ~60 behaviours
|
||||
|
||||
Twenty-two query sources, seven action handlers, eleven pre-route resolvers and
|
||||
the recall gate are all methods on `*reactiveHandler`. Testing one requires
|
||||
constructing a struct with 34 fields, most of them nil.
|
||||
|
||||
### 9.3 The static gates pass against a baseline, and the baseline records the debt
|
||||
|
||||
`scripts/analyzers/deadcode.baseline` accepts thirteen unreachable symbols,
|
||||
eleven of them from the 2026-08-10 audit (V-686), with three marked as
|
||||
"must stay". `make audit` is a git-grep inventory and is explicitly not a
|
||||
reachability check (`CLAUDE.md`).
|
||||
|
||||
### 9.4 Measurement needs weights that are not in the tree
|
||||
|
||||
`make t` self-skips the four `TestONNX*` measurements without `MAVEN_ONNX_LIB`,
|
||||
and still prints `ok` (`CLAUDE.md`). The routing heads, the embedder, silero and
|
||||
the keyword head are all ONNX files under `models/`, bind-mounted from
|
||||
`/mnt/hdd1/llms` in the case of the gguf. A checkout alone cannot reproduce a
|
||||
routing measurement.
|
||||
|
||||
### 9.5 Only 5 of 51 spec entries cite a scenario that exists
|
||||
|
||||
Recorded in the previous session's handoff, from `docs/spec.md` and
|
||||
`cmd/mavend/testdata/scenarios/`. Not re-verified here.
|
||||
|
||||
---
|
||||
|
||||
## 10. Excessive fan-in and fan-out
|
||||
|
||||
**Fan-in.** `ipc.Server` is reached by six processes (mavweb ×3 connections,
|
||||
mavpoll, mavcaldav, mavmaild, mavupdate, e2eprobe) and carries eight function
|
||||
fields that bypass `CoreAPI` entirely: `StepUp`, `UnlockFn`, `WrapKeyFn`,
|
||||
`IngestMailFn`, `SwapModelFn`, `ModelStatusFn`, `DescribeImageFn` and the four
|
||||
`Capture*` fields. Each is nil unless its config block exists, so the wire
|
||||
surface of the daemon depends on `deploy/mavend.json`.
|
||||
|
||||
**Fan-out.** `reactiveHandler` reaches roughly twenty distinct subsystems
|
||||
(4.1). `tickLoop` reaches ten (4.3). `wireVoice` constructs seventeen (4.4).
|
||||
|
||||
**Failure propagation.** The store is the shared point: `SetMaxOpenConns(1)`
|
||||
means every writer in the daemon and every module over IPC serialises through
|
||||
one connection. The measurement backing that cap is
|
||||
`docs/evals/2026-08-07-store-connection-cap.md` (V-642), cited in
|
||||
`internal/ipc/server.go` and not re-run here.
|
||||
|
||||
---
|
||||
|
||||
## 11. What is dark, and what that costs
|
||||
|
||||
Sixteen components are wired in code and off in the deployed configuration:
|
||||
`ntfy`, `zenmoney`, Home Assistant, MCP, the weather provider, the workstation
|
||||
model arm, vision, meeting capture, speaker identification, mail intake, model
|
||||
swap, memory evaluation, and the `mavcaldav` and `mavmaild` services.
|
||||
|
||||
Three of these have a visible cost:
|
||||
|
||||
1. **ntfy disabled** means the away reach is telegram alone, through a SOCKS
|
||||
relay, through `api.telegram.org`. `deploy/mavend.json` documents that this
|
||||
was exactly the fragility ntfy was added to remove: "three things in series
|
||||
that have each failed once, and when they do a sev4 nudge has nowhere to go."
|
||||
2. **mavcaldav absent** disables both the calendar answer and the busy
|
||||
suppressor (8.2).
|
||||
3. **The weather provider is a stub.** `wireVoice` selects Open-Meteo only when
|
||||
`cfg.Voice.Weather.Provider == "open-meteo"`, and the deployed `voice` block
|
||||
has no `weather` key at all. `weather` is nevertheless a live query source with
|
||||
`guesses: true`, so it can claim a turn and answer it from a stub.
|
||||
|
||||
---
|
||||
|
||||
# Questions the current architecture raises
|
||||
|
||||
1. **Which of the three ordered lists is the arbiter?** Stage 0 grammars, the
|
||||
query-source chain and the pre-route ladder each decide by position. If
|
||||
`internal/claim` is the answer, what stops it being a fourth list rather than
|
||||
the thing that collapses the other three?
|
||||
|
||||
2. **Where is the one point that decides whether this authenticated origin may
|
||||
perform this specific effect using this specific evidence?** Today there is
|
||||
no such point. `reboot` shows why the question is not "which tier system
|
||||
wins": it is correctly classified as not irreversible, and that does not
|
||||
imply a room microphone plus "да" should carry reboot authority.
|
||||
Reversibility, effect severity, reach authority and confirmation strength are
|
||||
four dimensions, and `TierDestructive → VoiceMayRun:true` collapses them into
|
||||
one.
|
||||
|
||||
3. **What owns the `facts` key namespace?** Nine writers, two of which store
|
||||
watermarks and tuning parameters in the table that recall embeds. Is `source`
|
||||
meant to be a partition, and if so what enforces it?
|
||||
|
||||
4. **Should the executor live in the key holder?** `systemctl reboot` is a
|
||||
seeded, enabled row in a process holding the unlocked database. The controls
|
||||
are an allowlist and a spoken confirm. Is that the intended trust boundary,
|
||||
or the one that happened?
|
||||
|
||||
5. **Should the tick loop and the turn path share one llama-server?**
|
||||
`internal/llm/gate.go` exists to arbitrate them. What is the acceptable
|
||||
latency a foreground turn may pay for a background nudge being phrased?
|
||||
|
||||
6. **Is a silent four-level degradation still honest?** Each fallback is argued
|
||||
separately. Nothing tells the user when all four fire at once. The M1 honesty
|
||||
milestone in `docs/roadmap.md` is about the turn path. Does it cover this?
|
||||
|
||||
7. **What is `docker-compose.yml` the source of truth for?** Two complete
|
||||
services are commented out in it with their reasoning, and one of them
|
||||
silently disables two behaviours elsewhere. Should absence be expressible in
|
||||
`deploy/mavend.json` where the rest of the capability switches live?
|
||||
|
||||
8. **Why is the daemon wired twice?** `boot.go` fixed the drift that had already
|
||||
happened. Fifteen `wire*` calls are still listed by hand on both paths. Is
|
||||
cold-start unlock worth a second wiring path, or should the locked daemon
|
||||
wire everything and gate at the `Check` hook alone?
|
||||
|
||||
9. **What is a query source allowed to do?** One triggers a live LAN scan and
|
||||
writes a note. If a source may have side effects, what does "first source to
|
||||
claim answers the turn" guarantee about the sources that ran before it?
|
||||
|
||||
10. **Is `mavsttd`/`mavttsd`'s process boundary about the key or about cgo?**
|
||||
The stated reason is key isolation. The operative reason looks like crash
|
||||
isolation from cgo and subprocesses. Which one governs whether the next
|
||||
model caller gets its own process?
|
||||
File diff suppressed because it is too large
Load Diff
Executable
+139
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build maven-evidence.zip: the architecture package plus the source seams a
|
||||
# reviewer needs to test its claims, and nothing else.
|
||||
#
|
||||
# sh docs/architecture/pack_evidence.sh
|
||||
#
|
||||
# Three rules this script exists to enforce:
|
||||
#
|
||||
# 1. Whole files, never snippets. A cut-down file loses the call path that
|
||||
# makes a claim checkable, which is the whole point of sending source.
|
||||
# 2. Allowlist, not denylist. Paths are named one by one below. A denylist
|
||||
# ships whatever nobody thought to exclude, and this tree has a database
|
||||
# key in it.
|
||||
# 3. Refuse rather than warn. The scan at the end aborts on a hit instead of
|
||||
# printing something a tired person scrolls past.
|
||||
#
|
||||
# The one file that is not verbatim is docker-compose.yml. It carries a live
|
||||
# uptime-kuma API key, so a redacted copy goes in its place and the redaction is
|
||||
# recorded in architecture-evidence.txt and printed here.
|
||||
set -euo pipefail
|
||||
|
||||
here=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
root=$(CDPATH= cd -- "$here/../.." && pwd)
|
||||
cd "$root"
|
||||
|
||||
out=maven-evidence.zip
|
||||
stage=$(mktemp -d)
|
||||
trap 'rm -rf "$stage"' EXIT
|
||||
|
||||
echo "== regenerating the architecture package"
|
||||
python3 "$here/build_inventory.py"
|
||||
python3 "$here/verify_anchors.py" # exits 1 if any claim no longer resolves
|
||||
python3 "$here/build_evidence.py"
|
||||
python3 "$here/build_viewer.py"
|
||||
|
||||
echo "== structural context"
|
||||
# `tree` here is an eza alias in the owner's shell and absent in a plain sh, so
|
||||
# the listing is generated with find and does not depend on either.
|
||||
{
|
||||
echo "# find -L internal cmd -maxdepth 3 -type d"
|
||||
echo
|
||||
find internal cmd -maxdepth 3 -type d | sort
|
||||
echo
|
||||
echo "# go files per package"
|
||||
echo
|
||||
find internal cmd -name '*.go' ! -name '*_test.go' | sed 's|/[^/]*$||' | sort | uniq -c | sort -rn
|
||||
echo
|
||||
echo "# test files per package"
|
||||
echo
|
||||
find internal cmd -name '*_test.go' | sed 's|/[^/]*$||' | sort | uniq -c | sort -rn
|
||||
} > "$here/tree.txt"
|
||||
|
||||
echo "== redacting the one credential in docker-compose.yml"
|
||||
sed 's/"uk5_[^"]*"/"<REDACTED: uptime-kuma api key>"/' docker-compose.yml \
|
||||
> "$here/docker-compose.redacted.yml"
|
||||
if grep -q 'uk5_' "$here/docker-compose.redacted.yml"; then
|
||||
echo "pack_evidence.sh: redaction failed, refusing to build" >&2; exit 1
|
||||
fi
|
||||
diff <(sed 's/"uk5_[^"]*"/X/' docker-compose.yml) \
|
||||
<(sed 's/"<REDACTED: uptime-kuma api key>"/X/' "$here/docker-compose.redacted.yml") \
|
||||
>/dev/null || { echo "pack_evidence.sh: redacted copy differs by more than the key" >&2; exit 1; }
|
||||
|
||||
# ---- the allowlist -------------------------------------------------------
|
||||
# Requested and present. internal/session, internal/db and tests/ are absent
|
||||
# from this repo; architecture-evidence.txt says where their contents live.
|
||||
paths=(
|
||||
docs/architecture
|
||||
CLAUDE.md
|
||||
docs/CLAUDE.md
|
||||
go.mod
|
||||
deploy/mavend.json # ${VAR} placeholders only; 16 off-claims read it
|
||||
cmd/mavend
|
||||
internal/auth
|
||||
internal/tool
|
||||
internal/claim
|
||||
internal/modes
|
||||
internal/router
|
||||
internal/voice
|
||||
internal/ipc # the boundary auth.Can actually runs on
|
||||
internal/store
|
||||
internal/dialogue # clarify + session state the turn path parks in
|
||||
internal/decision # the arbitration record
|
||||
internal/delivery/channel.go
|
||||
internal/loop
|
||||
internal/webauthn # the other half of the auth story
|
||||
cmd/mavwaked/main.go # the client that sends Surface
|
||||
cmd/mavweb/main.go # the six unguarded surfaces
|
||||
)
|
||||
|
||||
echo "== staging"
|
||||
for p in "${paths[@]}"; do
|
||||
if [ ! -e "$p" ]; then echo " MISSING $p (skipped)"; continue; fi
|
||||
mkdir -p "$stage/$(dirname "$p")"
|
||||
cp -r "$p" "$stage/$(dirname "$p")/"
|
||||
done
|
||||
|
||||
# Generated-in-place files that must not travel, and anything that is a secret,
|
||||
# a model, a database or a build artefact regardless of how it got staged.
|
||||
#
|
||||
# The name filters skip .go on purpose: internal/router/singletoken.go matched
|
||||
# '*token*' and was deleted out of the first build of this pack. That is exactly
|
||||
# the silent hole an allowlist exists to prevent, and a Go source file is never
|
||||
# the thing this clause is for.
|
||||
find "$stage" ! -name '*.go' \( \
|
||||
-name '*.db' -o -name '*.sqlite*' -o -name '*.enc' \
|
||||
-o -name '*.pem' -o -name '*.key' -o -name '*.crt' -o -name '*.p12' \
|
||||
-o -name '.env*' -o -name '*.token' -o -name '*.secret' -o -name '*.password' \
|
||||
-o -name '*.onnx' -o -name '*.gguf' -o -name '*.bin' -o -name '*.wav' \
|
||||
-o -name '*.zip' -o -name '*.log' -o -name '.git' \
|
||||
\) -print -exec rm -rf {} + 2>/dev/null || true
|
||||
|
||||
echo "== scanning the staged tree"
|
||||
# A value-shaped assignment: a credential word, a delimiter, then twelve or more
|
||||
# characters of value. The value must NOT begin with a slash or a dot, because a
|
||||
# docker volume line pairs a host path with a container path and both halves end
|
||||
# in the same secret-sounding filename while containing no secret. Three of those
|
||||
# in docker-compose.yml tripped the first version of this scan.
|
||||
hits=$(grep -rInE '(api[_-]?key|secret|passwo?r?d|bearer|token)["'"'"' ]*[:=]["'"'"' ]*[A-Za-z0-9+_-][A-Za-z0-9/+_-]{11,}' "$stage" \
|
||||
| grep -vE '\$\{|<REDACTED|example|EXAMPLE|xxx|XXX|your-|changeme' \
|
||||
| grep -vE '_test\.go|\.md:' \
|
||||
| grep -vE ':[0-9]+:[[:space:]]*(#|//)' || true)
|
||||
if [ -n "$hits" ]; then
|
||||
echo "pack_evidence.sh: possible credentials in the staged tree, refusing to build:" >&2
|
||||
echo "$hits" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "== building $out"
|
||||
rm -f "$out"
|
||||
( cd "$stage" && zip -qr "$root/$out" . )
|
||||
|
||||
echo
|
||||
printf '%s %s %s files\n' "$out" \
|
||||
"$(du -h "$out" 2>/dev/null | cut -f1)" \
|
||||
"$(unzip -l "$out" | tail -1 | awk '{print $2}')"
|
||||
echo
|
||||
echo "redacted: docker-compose.yml -> docs/architecture/docker-compose.redacted.yml"
|
||||
echo " one uptime-kuma api key, nothing else"
|
||||
echo "excluded: .git, deploy/telegram.env, deploy/db_key.env, models, deps, databases"
|
||||
Executable
+52
@@ -0,0 +1,52 @@
|
||||
#!/bin/sh
|
||||
# Re-render every diagram in diagrams/*.mmd to a committed SVG beside it, then
|
||||
# rebuild index.html so the viewer carries the new pictures.
|
||||
#
|
||||
# mermaid-cli drives a real browser through puppeteer. It downloads its own
|
||||
# chrome-headless-shell by default, which fails behind a proxy and wastes
|
||||
# 150 MB; PUPPETEER_EXECUTABLE_PATH points it at the system chromium instead.
|
||||
# --no-sandbox is required because that chromium is not the one puppeteer
|
||||
# provisioned and has no sandbox helper of its own here.
|
||||
#
|
||||
# sh docs/architecture/render.sh
|
||||
#
|
||||
# Run it from anywhere. A parse error in one file leaves the others alone and
|
||||
# prints FAIL with the mermaid error, which is the only way this repo has to
|
||||
# syntax-check a .mmd.
|
||||
set -eu
|
||||
|
||||
here=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
dia="$here/diagrams"
|
||||
cfg=$(mktemp)
|
||||
trap 'rm -f "$cfg"' EXIT
|
||||
printf '{"args":["--no-sandbox","--disable-gpu"]}' > "$cfg"
|
||||
|
||||
: "${PUPPETEER_EXECUTABLE_PATH:=$(command -v chromium || command -v chromium-browser || command -v google-chrome-stable || true)}"
|
||||
if [ -z "$PUPPETEER_EXECUTABLE_PATH" ]; then
|
||||
echo "render.sh: no chromium found. Install one, or set PUPPETEER_EXECUTABLE_PATH." >&2
|
||||
exit 1
|
||||
fi
|
||||
export PUPPETEER_EXECUTABLE_PATH
|
||||
|
||||
for f in "$dia"/*.mmd; do
|
||||
n=$(basename "$f" .mmd)
|
||||
err=$(mktemp)
|
||||
if npx --yes @mermaid-js/mermaid-cli@11 -p "$cfg" -t dark -b '#0e1116' \
|
||||
-i "$f" -o "$dia/$n.svg" >/dev/null 2>"$err" && [ -s "$dia/$n.svg" ]; then
|
||||
echo "OK $n"
|
||||
else
|
||||
echo "FAIL $n"
|
||||
grep -m1 -A3 'Parse error' "$err" || tail -3 "$err"
|
||||
fi
|
||||
rm -f "$err"
|
||||
done
|
||||
|
||||
python3 "$here/build_viewer.py"
|
||||
|
||||
# The only check the viewer has. A TypeError in a renderer shows as a blank
|
||||
# panel, not as an error, so run every view against a DOM stub before shipping.
|
||||
if command -v node >/dev/null 2>&1; then
|
||||
node "$here/check_viewer.js" || exit 1
|
||||
else
|
||||
echo "SKIP check_viewer.js: no node"
|
||||
fi
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Resolve every claim in maven-architecture.json to a file and a line.
|
||||
|
||||
The inventory names files and symbols. A reader has to take on trust that the
|
||||
symbol is in the file and that the file still exists. This script removes the
|
||||
trust: it looks up every symbol in the component's own files and writes
|
||||
anchors.md, a table of component, symbol, path:line and the verbatim line.
|
||||
|
||||
Exit code is 1 when anything fails to resolve, so it doubles as a staleness
|
||||
gate. A symbol that moved to another file, or a file that was deleted, fails
|
||||
here rather than in a reader's head.
|
||||
|
||||
python3 docs/architecture/verify_anchors.py # write anchors.md
|
||||
python3 docs/architecture/verify_anchors.py --quiet # gate only
|
||||
|
||||
What it deliberately does NOT check: that the symbol means what the
|
||||
responsibility says it means. That is the human pass this file exists to make
|
||||
cheap.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
|
||||
|
||||
# Symbols the inventory names that are not Go identifiers in this repo: config
|
||||
# keys, make targets, flags, wire strings, table names. Looking them up in a .go
|
||||
# file would fail for the wrong reason, so they are resolved against the file
|
||||
# they belong to when possible and skipped when not.
|
||||
NON_GO = re.compile(r"^(make |-|/|\$)|\.(json|sql|service|yml)$| ")
|
||||
|
||||
|
||||
def candidates(sym: str):
|
||||
"""Search forms for one symbol, longest first.
|
||||
|
||||
A dotted symbol like `Store.WriteFact` or `voice.NewServer` is written as a
|
||||
method or a qualified call, so the tail is what appears in a definition and
|
||||
the whole string is what appears at a call site. Try both.
|
||||
"""
|
||||
forms = [sym]
|
||||
if "." in sym:
|
||||
forms.append(sym.split(".")[-1])
|
||||
return forms
|
||||
|
||||
|
||||
def find(paths, sym):
|
||||
for form in candidates(sym):
|
||||
needle = re.compile(r"\b" + re.escape(form) + r"\b")
|
||||
for rel in paths:
|
||||
full = os.path.join(ROOT, rel)
|
||||
if not os.path.isfile(full):
|
||||
continue
|
||||
try:
|
||||
lines = open(full, errors="replace").read().splitlines()
|
||||
except OSError:
|
||||
continue
|
||||
# Three passes, best anchor first: a definition, then any code
|
||||
# line, then a comment. Without the comment pass being last, a
|
||||
# const whose doc comment names it anchors on the prose rather
|
||||
# than on the declaration.
|
||||
for rank in (0, 1, 2):
|
||||
for i, line in enumerate(lines, 1):
|
||||
if not needle.search(line):
|
||||
continue
|
||||
bare = line.strip()
|
||||
comment = bare.startswith(("//", "#", "--", "%%", "*"))
|
||||
isdef = bool(re.match(
|
||||
r"\s*(func|type|const|var)\b", line)) or bool(re.match(
|
||||
r"\s*\"?" + re.escape(form) + r"\"?\s*[:=]", line))
|
||||
got = 2 if comment else (0 if isdef else 1)
|
||||
if got == rank:
|
||||
return rel, i, bare
|
||||
return None, None, None
|
||||
|
||||
|
||||
def expand(rel):
|
||||
"""A directory in the inventory stands for the files under it."""
|
||||
full = os.path.join(ROOT, rel)
|
||||
if os.path.isdir(full):
|
||||
return sorted(
|
||||
os.path.join(rel, f) for f in os.listdir(full)
|
||||
if f.endswith((".go", ".json", ".sql")) and not f.endswith("_test.go")
|
||||
)
|
||||
return [rel]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
quiet = "--quiet" in sys.argv
|
||||
arch = json.load(open(os.path.join(HERE, "maven-architecture.json")))
|
||||
rows, missing_files, unresolved = [], [], []
|
||||
|
||||
for c in arch["components"]:
|
||||
paths = []
|
||||
for f in c["files"]:
|
||||
if not os.path.exists(os.path.join(ROOT, f)):
|
||||
missing_files.append((c["id"], f))
|
||||
continue
|
||||
paths.extend(expand(f))
|
||||
for sym in c["symbols"]:
|
||||
if NON_GO.search(sym):
|
||||
rows.append((c["id"], sym, "", "", "not a Go identifier, not looked up"))
|
||||
continue
|
||||
rel, line, text = find(paths, sym)
|
||||
if rel is None:
|
||||
unresolved.append((c["id"], sym))
|
||||
rows.append((c["id"], sym, "", "", "UNRESOLVED"))
|
||||
else:
|
||||
rows.append((c["id"], sym, f"{rel}:{line}", text, ""))
|
||||
|
||||
resolved = sum(1 for r in rows if r[2])
|
||||
if not quiet:
|
||||
with open(os.path.join(HERE, "anchors.md"), "w") as fh:
|
||||
fh.write("# Claim anchors\n\n")
|
||||
fh.write(
|
||||
"Generated by `docs/architecture/verify_anchors.py`. Every symbol the\n"
|
||||
"inventory names, resolved to a file and a line in this checkout, with the\n"
|
||||
"line quoted. Regenerate after any edit to the inventory or the code.\n\n"
|
||||
)
|
||||
fh.write(
|
||||
f"- components: {len(arch['components'])}\n"
|
||||
f"- symbols claimed: {len(rows)}\n"
|
||||
f"- resolved to a line: {resolved}\n"
|
||||
f"- unresolved: {len(unresolved)}\n"
|
||||
f"- missing files: {len(missing_files)}\n\n"
|
||||
)
|
||||
if unresolved:
|
||||
fh.write("## Unresolved\n\n")
|
||||
for cid, sym in unresolved:
|
||||
fh.write(f"- `{cid}` claims `{sym}` and it is in none of its files\n")
|
||||
fh.write("\n")
|
||||
if missing_files:
|
||||
fh.write("## Missing files\n\n")
|
||||
for cid, f in missing_files:
|
||||
fh.write(f"- `{cid}` names `{f}`, which does not exist\n")
|
||||
fh.write("\n")
|
||||
fh.write("## Anchors\n\n| component | symbol | anchor | line |\n|---|---|---|---|\n")
|
||||
for cid, sym, anchor, text, note in rows:
|
||||
shown = (text or note).replace("|", "\\|")
|
||||
if len(shown) > 120:
|
||||
shown = shown[:117] + "..."
|
||||
fh.write(f"| `{cid}` | `{sym}` | {anchor or '—'} | `{shown}` |\n")
|
||||
|
||||
print(f"symbols {len(rows)}, resolved {resolved}, unresolved {len(unresolved)}, "
|
||||
f"missing files {len(missing_files)}")
|
||||
for cid, sym in unresolved[:20]:
|
||||
print(f" UNRESOLVED {cid} :: {sym}")
|
||||
for cid, f in missing_files[:20]:
|
||||
print(f" MISSING {cid} :: {f}")
|
||||
return 1 if (unresolved or missing_files) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,691 @@
|
||||
<!doctype html>
|
||||
<html lang="en" data-theme="dark">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Maven architecture — as built</title>
|
||||
<style>
|
||||
:root{
|
||||
--bg:#0e1116; --panel:#141922; --panel2:#1a2130; --line:#26304a; --line2:#38456b;
|
||||
--fg:#dfe6f2; --dim:#8d9bb5; --dim2:#5f6c85;
|
||||
--proc:#7fb3ff; --procbg:#16263f;
|
||||
--svc:#8fd0ff; --svcbg:#132433;
|
||||
--store:#6ed0a8; --storebg:#0f2a20;
|
||||
--model:#e0b050; --modelbg:#2a2210;
|
||||
--adapter:#b79bf0; --adapterbg:#221a33;
|
||||
--ext:#c39bd3; --extbg:#241a2b;
|
||||
--bnd:#ff9f6b; --bndbg:#2c1c12;
|
||||
--warn:#e08080; --warnbg:#2e1616;
|
||||
--ok:#7fbf7f;
|
||||
}
|
||||
*{box-sizing:border-box}
|
||||
html,body{margin:0;height:100%}
|
||||
body{background:var(--bg);color:var(--fg);font:14px/1.5 ui-sans-serif,-apple-system,"Segoe UI",Roboto,sans-serif;overflow:hidden}
|
||||
code,.mono{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}
|
||||
|
||||
#app{display:grid;grid-template-columns:250px 1fr 420px;grid-template-rows:auto 1fr;height:100vh}
|
||||
header{grid-column:1/-1;display:flex;align-items:center;gap:18px;padding:10px 16px;border-bottom:1px solid var(--line);background:var(--panel)}
|
||||
header h1{font-size:15px;margin:0;font-weight:650;letter-spacing:.2px}
|
||||
header .meta{color:var(--dim2);font-size:12px}
|
||||
header .meta b{color:var(--dim);font-weight:500}
|
||||
|
||||
nav{border-right:1px solid var(--line);background:var(--panel);overflow-y:auto;padding:12px 10px}
|
||||
nav h2{font-size:10px;letter-spacing:.12em;text-transform:uppercase;color:var(--dim2);margin:14px 6px 6px}
|
||||
nav h2:first-child{margin-top:0}
|
||||
.viewbtn{display:block;width:100%;text-align:left;background:transparent;border:1px solid transparent;color:var(--dim);
|
||||
padding:7px 9px;border-radius:6px;cursor:pointer;font:inherit;font-size:13px}
|
||||
.viewbtn:hover{background:var(--panel2);color:var(--fg)}
|
||||
.viewbtn.on{background:#1d2c47;border-color:var(--line2);color:#fff}
|
||||
.viewbtn small{display:block;color:var(--dim2);font-size:11px;line-height:1.35;margin-top:2px}
|
||||
.toggle{display:flex;align-items:center;gap:8px;padding:5px 7px;color:var(--dim);font-size:12.5px;cursor:pointer;border-radius:5px}
|
||||
.toggle:hover{background:var(--panel2)}
|
||||
.toggle input{accent-color:#5b8ff9}
|
||||
.legend{display:flex;flex-wrap:wrap;gap:5px;padding:4px 6px}
|
||||
.legend span{font-size:10.5px;padding:2px 6px;border-radius:99px;border:1px solid var(--line2);color:var(--dim)}
|
||||
|
||||
/* capability matrix and invariants */
|
||||
.capsel{display:flex;gap:6px;margin:0 0 14px}
|
||||
.capsel button{background:var(--panel2);border:1px solid var(--line);color:var(--dim);padding:5px 11px;border-radius:6px;cursor:pointer;font:inherit;font-size:12.5px}
|
||||
.capsel button.on{background:#1d2c47;border-color:var(--line2);color:#fff}
|
||||
table.mx{border-collapse:collapse;width:100%;font-size:12.5px}
|
||||
table.mx th{text-align:left;font-weight:500;color:var(--dim2);font-size:10px;letter-spacing:.1em;text-transform:uppercase;padding:0 6px 7px;vertical-align:bottom}
|
||||
table.mx th.d{text-align:center;width:64px}
|
||||
table.mx tr.grp td{padding:16px 6px 5px;color:var(--dim);font-size:10.5px;letter-spacing:.1em;text-transform:uppercase;border-bottom:1px solid var(--line)}
|
||||
table.mx tbody tr.cap{cursor:pointer}
|
||||
table.mx tbody tr.cap:hover td{background:var(--panel2)}
|
||||
table.mx tbody tr.cap.sel td{background:#1b2b45}
|
||||
table.mx td{padding:4px 6px;border-bottom:1px solid #1b2230}
|
||||
table.mx td.n{font-weight:600}
|
||||
table.mx td.n em{font-style:normal;color:var(--dim2);font-weight:400;font-size:11px;margin-left:7px}
|
||||
table.mx td.d{text-align:center}
|
||||
.dot{display:inline-block;width:11px;height:11px;border-radius:3px;border:1px solid #0006}
|
||||
.dot.yes{background:#4f9d69}.dot.partial{background:#c8992e}.dot.no{background:#3a4256}
|
||||
.dot.speconly{background:#3a4256;border-style:dashed;border-color:#6b7896}
|
||||
.gapc{font-size:10.5px;color:var(--dim2)}
|
||||
.gapc.missing{color:#e08080}.gapc.unreachable{color:#e0a060}.gapc.partial{color:#c8992e}
|
||||
.mark{font-size:10px;letter-spacing:.06em;text-transform:uppercase;padding:2px 7px;border-radius:99px;border:1px solid var(--line2)}
|
||||
.mark.explicit{color:#7fbf7f;border-color:#3d6b43}
|
||||
.mark.implied{color:#e0b050;border-color:#6b5a26}
|
||||
.mark.unresolved{color:#e08080;border-color:#6b3838}
|
||||
.inv{background:var(--panel2);border:1px solid var(--line);border-radius:9px;padding:12px 14px;margin-bottom:12px}
|
||||
.inv h4{margin:0 0 6px;font-size:14px;display:flex;align-items:center;gap:10px}
|
||||
.inv h4 span.n{color:var(--dim2);font-weight:400}
|
||||
.inv .q{color:#e0b8b8;font-size:12.5px;margin:8px 0 0}
|
||||
.bars{display:flex;gap:14px;margin:9px 0 4px;flex-wrap:wrap}
|
||||
.bar{font-size:10.5px;color:var(--dim2)}
|
||||
.bar b{display:block;font-size:9.5px;letter-spacing:.1em;text-transform:uppercase;color:var(--dim2);font-weight:500;margin-bottom:3px}
|
||||
.bar .v{color:var(--fg);font-size:12px}
|
||||
.bar .v.warn{color:#e08080}.bar .v.mid{color:#c8992e}.bar .v.ok{color:#7fbf7f}
|
||||
.cchip{display:inline-block;background:var(--panel);border:1px solid var(--line);border-radius:6px;padding:3px 8px;margin:3px 4px 0 0;
|
||||
font-size:11.5px;cursor:pointer;color:var(--dim)}
|
||||
.cchip:hover{border-color:var(--line2);color:var(--fg)}
|
||||
.cchip.off{border-color:#6b5a26;color:#e0b050}
|
||||
.cchip.pl{border-color:#6b3838;color:#e08080}
|
||||
.crit{border-left:2px solid var(--line2);padding:0 0 0 10px;margin:0 0 11px}
|
||||
.crit .vd{font-size:10px;letter-spacing:.08em;text-transform:uppercase;margin-right:8px}
|
||||
.crit .vd.pass{color:#7fbf7f}.crit .vd.fail{color:#e08080}.crit .vd.blocked{color:#e0a060}
|
||||
.crit .vd.untested{color:var(--dim2)}.crit .vd.unknown{color:#b79bf0}
|
||||
.crit .rs{color:var(--dim2);font-size:11px}
|
||||
.crit p{margin:5px 0 0;color:var(--dim);font-size:12px}
|
||||
main{position:relative;overflow:auto;padding:18px 20px 60px}
|
||||
.lane{margin-bottom:20px}
|
||||
.lane-h{display:flex;align-items:baseline;gap:10px;margin:0 0 8px;cursor:pointer;user-select:none}
|
||||
.lane-h b{font-size:11px;letter-spacing:.1em;text-transform:uppercase;color:var(--dim)}
|
||||
.lane-h i{font-style:normal;color:var(--dim2);font-size:11px}
|
||||
.lane-h .caret{color:var(--dim2);font-size:11px;width:10px}
|
||||
.chips{display:flex;flex-wrap:wrap;gap:8px}
|
||||
.chip{position:relative;background:var(--panel2);border:1px solid var(--line);border-radius:8px;padding:7px 10px;cursor:pointer;
|
||||
max-width:280px;transition:border-color .12s,background .12s}
|
||||
.chip:hover{border-color:var(--line2)}
|
||||
.chip.sel{border-color:#7fb3ff;background:#1b2b45;box-shadow:0 0 0 1px #7fb3ff44}
|
||||
.chip.rel{border-color:#4a5f8f}
|
||||
.chip.dim{opacity:.28}
|
||||
.chip .nm{font-weight:600;font-size:13px}
|
||||
.chip .ty{font-size:10.5px;color:var(--dim2);letter-spacing:.04em;text-transform:uppercase}
|
||||
.chip .badges{display:flex;gap:4px;margin-top:4px;flex-wrap:wrap}
|
||||
.b{font-size:9.5px;padding:1px 5px;border-radius:99px;border:1px solid currentColor;letter-spacing:.03em}
|
||||
.b.off{color:#c9a227}.b.nd{color:#c98a27}.b.pl{color:#a07fe0}.b.tmp{color:#8d9bb5}.b.pw{color:#e08080}
|
||||
.b.lo{color:#e08080}.b.me{color:#c9a227}
|
||||
.chip[data-t=process]{border-left:3px solid var(--proc)}
|
||||
.chip[data-t=service],.chip[data-t=worker],.chip[data-t=handler]{border-left:3px solid var(--svc)}
|
||||
.chip[data-t=arbitration],.chip[data-t=query_source]{border-left:3px solid #ffd479}
|
||||
.chip[data-t=storage],.chip[data-t=table]{border-left:3px solid var(--store)}
|
||||
.chip[data-t=model]{border-left:3px solid var(--model)}
|
||||
.chip[data-t=adapter]{border-left:3px solid var(--adapter)}
|
||||
.chip[data-t=external]{border-left:3px solid var(--ext)}
|
||||
.chip[data-t=boundary]{border-left:3px solid var(--bnd)}
|
||||
.chip[data-t="shared-state"]{border-left:3px solid #ff9ec7}
|
||||
.chip[data-t=planned]{border-left:3px solid #a07fe0}
|
||||
.chip[data-t=config],.chip[data-t=test]{border-left:3px solid var(--dim2)}
|
||||
svg.wires{position:absolute;inset:0;pointer-events:none;overflow:visible}
|
||||
|
||||
aside{border-left:1px solid var(--line);background:var(--panel);overflow-y:auto;padding:16px 16px 60px}
|
||||
aside .empty{color:var(--dim2);font-size:13px;margin-top:30px;line-height:1.7}
|
||||
aside h3{margin:0 0 2px;font-size:16px}
|
||||
aside .sub{color:var(--dim2);font-size:11px;text-transform:uppercase;letter-spacing:.08em;margin-bottom:10px}
|
||||
aside section{margin-top:16px}
|
||||
aside section > h4{font-size:10px;letter-spacing:.12em;text-transform:uppercase;color:var(--dim2);margin:0 0 6px}
|
||||
aside p{margin:0 0 8px;color:#c9d3e6}
|
||||
.note{background:#1c1f14;border-left:2px solid #c9a227;padding:8px 10px;border-radius:0 5px 5px 0;color:#ded6b6;font-size:12.5px}
|
||||
ul.plain{list-style:none;margin:0;padding:0}
|
||||
ul.plain li{padding:3px 0;border-bottom:1px solid #1d2433;font-size:12.5px}
|
||||
ul.plain li:last-child{border-bottom:0}
|
||||
.rel{display:block;width:100%;text-align:left;background:transparent;border:0;color:#a9c6f5;cursor:pointer;font:inherit;font-size:12.5px;padding:3px 0}
|
||||
.rel:hover{color:#fff;text-decoration:underline}
|
||||
.rel .k{display:inline-block;min-width:66px;color:var(--dim2);font-size:10.5px;text-transform:uppercase;letter-spacing:.05em}
|
||||
.rel .ev{display:block;color:var(--dim2);font-size:11px;margin-left:66px;line-height:1.4}
|
||||
pre.mm{white-space:pre-wrap;word-break:break-word;background:#0b0e13;border:1px solid var(--line);border-radius:6px;
|
||||
padding:12px;font-size:11.5px;color:#b8c6de;overflow-x:auto;max-height:none}
|
||||
.searchbox{width:100%;background:var(--panel2);border:1px solid var(--line);border-radius:6px;color:var(--fg);
|
||||
padding:7px 9px;font:inherit;font-size:12.5px}
|
||||
.searchbox:focus{outline:none;border-color:var(--line2)}
|
||||
.results{margin-top:6px;max-height:280px;overflow:auto}
|
||||
.results button{display:block;width:100%;text-align:left;background:transparent;border:0;color:var(--dim);
|
||||
padding:5px 7px;border-radius:5px;cursor:pointer;font:inherit;font-size:12px}
|
||||
.results button:hover{background:var(--panel2);color:#fff}
|
||||
.results button em{font-style:normal;color:#ffd479}
|
||||
.flowsel{display:flex;gap:6px;margin-bottom:14px;flex-wrap:wrap}
|
||||
.flowsel button{background:var(--panel2);border:1px solid var(--line);color:var(--dim);border-radius:6px;
|
||||
padding:6px 11px;cursor:pointer;font:inherit;font-size:12.5px}
|
||||
.flowsel button.on{background:#1d2c47;border-color:var(--line2);color:#fff}
|
||||
ol.steps{counter-reset:s;list-style:none;margin:0;padding:0;max-width:1000px}
|
||||
ol.steps li{position:relative;padding:9px 12px 9px 44px;border-left:2px solid var(--line);margin-left:14px}
|
||||
ol.steps li:before{counter-increment:s;content:counter(s);position:absolute;left:-13px;top:9px;width:24px;height:24px;
|
||||
border-radius:99px;background:var(--panel2);border:1px solid var(--line2);color:var(--dim);font-size:11px;
|
||||
display:flex;align-items:center;justify-content:center}
|
||||
ol.steps li.branch{border-left-color:var(--warn);background:#1e1414}
|
||||
ol.steps li.branch:before{border-color:var(--warn);color:#e08080}
|
||||
ol.steps b{color:#fff}
|
||||
ol.steps .who{display:inline-block;background:#1d2c47;border:1px solid var(--line2);border-radius:4px;
|
||||
padding:0 6px;font-size:11px;color:#a9c6f5;cursor:pointer;margin-right:8px}
|
||||
ol.steps .who:hover{color:#fff;border-color:#7fb3ff}
|
||||
ol.steps .ev{display:block;color:var(--dim2);font-size:11.5px;margin-top:3px}
|
||||
.viewnote{max-width:1000px;color:var(--dim);font-size:12.5px;background:var(--panel2);border:1px solid var(--line);
|
||||
border-radius:7px;padding:11px 13px;margin-bottom:18px}
|
||||
.viewnote b{color:var(--fg)}
|
||||
.dia{margin-bottom:20px;border:1px solid var(--line);border-radius:8px;background:#0b0e13;overflow:hidden}
|
||||
.dia-h{display:flex;align-items:center;gap:10px;padding:8px 12px;background:var(--panel2);border-bottom:1px solid var(--line);cursor:pointer;user-select:none}
|
||||
.dia-h b{font-size:11px;letter-spacing:.1em;text-transform:uppercase;color:var(--dim)}
|
||||
.dia-h .fn{color:var(--dim2);font-size:11px}
|
||||
.dia-h .zoom{margin-left:auto;display:flex;gap:4px}
|
||||
.dia-h .zoom button{background:var(--panel);border:1px solid var(--line);color:var(--dim);border-radius:4px;
|
||||
width:24px;height:22px;cursor:pointer;font:inherit;font-size:12px;line-height:1}
|
||||
.dia-h .zoom button:hover{color:#fff;border-color:var(--line2)}
|
||||
.dia-body{overflow:auto;max-height:70vh;padding:10px}
|
||||
.dia-body > div{transform-origin:0 0}
|
||||
.dia-body svg{max-width:none;height:auto;display:block}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app">
|
||||
<header>
|
||||
<h1>Maven — architecture as built</h1>
|
||||
<div class="meta">commit <b id="commit"></b> · <b id="gen"></b> · <b id="counts"></b></div>
|
||||
<div class="meta" id="dirty"></div>
|
||||
</header>
|
||||
|
||||
<nav>
|
||||
<h2>Views</h2>
|
||||
<div id="views"></div>
|
||||
<h2>Search</h2>
|
||||
<input class="searchbox" id="q" placeholder="component, file or symbol">
|
||||
<div class="results" id="results"></div>
|
||||
<div id="archctl">
|
||||
<h2>Filters</h2>
|
||||
<label class="toggle"><input type="checkbox" id="tLow" checked> show low-confidence relations</label>
|
||||
<label class="toggle"><input type="checkbox" id="tMed" checked> show medium-confidence relations</label>
|
||||
<label class="toggle"><input type="checkbox" id="tOff" checked> show configured-off</label>
|
||||
<label class="toggle"><input type="checkbox" id="tUndeployed" checked> show built-not-deployed</label>
|
||||
<label class="toggle"><input type="checkbox" id="tPlanned" checked> show planned / unwired</label>
|
||||
<label class="toggle"><input type="checkbox" id="tWires" checked> draw relation wires</label>
|
||||
<h2>Type</h2>
|
||||
<div class="legend" id="legend"></div>
|
||||
</div>
|
||||
</nav>
|
||||
|
||||
<main id="main"></main>
|
||||
<aside id="side"><div class="empty">Select a component to see its responsibility, the files and symbols it was read from, and every relation in and out.<br><br>Every claim here cites a file. Nothing is inferred from a directory name.</div></aside>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
/*__DATA__*/
|
||||
|
||||
const byId = Object.fromEntries(ARCH.components.map(c => [c.id, c]));
|
||||
const S = { view: 'v1', sel: null, selCap: null, capBy: 'section', flow: 'reminder', collapsed: {}, diaClosed: false, diaZoom: 1 };
|
||||
|
||||
/* ---------------- view definitions ---------------- */
|
||||
const VIEWS = [
|
||||
{ id:'v1', name:'1 · System topology', hint:'Processes and external systems, with process boundaries drawn.',
|
||||
note:'<b>mavend is the centre because the code makes it one.</b> It is the only holder of the database key, it owns the store, the IPC socket, the voice TCP listener, the tick loop, eight background workers and the child llama-server. Every other daemon is key-free and fail-independent. Five services run under docker-compose; two more are built and commented out; two run under systemd on the workstation.',
|
||||
lanes:[
|
||||
['homesrv — docker compose', c => c.group==='homesrv' && ['process','model','boundary','external'].includes(c.type)],
|
||||
['workpc — systemd user units', c => c.group==='workpc'],
|
||||
['ecosystem network', c => c.group==='ecosystem'],
|
||||
['internet / LAN', c => ['internet','lan'].includes(c.group)],
|
||||
['dev and recovery binaries', c => c.group==='dev' || ['proc.mavseal','proc.mavupdate'].includes(c.id)],
|
||||
['configuration boundary', c => c.type==='config'],
|
||||
]},
|
||||
{ id:'v2', name:'2 · Core internals', hint:'The real path through mavend, in the order runTurn runs it.',
|
||||
note:'The implementation does <b>not</b> follow input → routing → intent → state → tools → response. Eleven stateful pre-emptors get first refusal <b>before</b> routing; the routing cascade is four arms deep; and an <code>IntentQuery</code> then enters a <b>second</b> arbitration of twenty-two ordered sources. The proactive half shares no code with any of it.',
|
||||
lanes:[
|
||||
['input and entry', c => ['core.voice_server','core.ipc_server','core.daemon_api','core.intake_api','core.store_api','core.stt_seam','core.telegram_intake','core.auth_gate','core.daemon_lock'].includes(c.id)],
|
||||
['turn pipeline', c => ['core.reactive_handler','core.turn_route','core.preroute','core.action_table'].includes(c.id)],
|
||||
['routing cascade', c => c.id.startsWith('router.')],
|
||||
['intent handlers', c => c.type==='handler' || c.id==='core.query_chain'],
|
||||
['query sources — the second arbitration, in table order', c => c.type==='query_source'],
|
||||
['response generation', c => ['core.replier','core.phraser','core.tts_seam','core.model_seam','core.recall','core.topics'].includes(c.id)],
|
||||
['proactive half', c => ['core.tick_loop','core.gatherer','core.rules','core.pattern','core.morning','core.routines','core.dispatcher','core.sink_voice','core.sink_ntfy','core.sink_telegram'].includes(c.id)],
|
||||
['background workers', c => c.type==='worker' && c.id!=='core.tick_loop' && c.id!=='core.telegram_intake'],
|
||||
['dark capabilities — wired, no config block', c => ['core.vision','core.capture','core.speaker','core.mail_intake','core.modelswap','core.netscan','core.memory_eval'].includes(c.id)],
|
||||
['construction and diagnosis', c => ['core.wiring','core.decision_trace','core.event_bus'].includes(c.id)],
|
||||
]},
|
||||
{ id:'v3', name:'3 · Runtime flow', hint:'Three representative requests traced through real code.', flow:true },
|
||||
{ id:'v4', name:'4 · State ownership', hint:'Every persistent and shared store, its owner, writers and readers.',
|
||||
note:'One process owns the database and every write is serialised at it: <code>SetMaxOpenConns(1)</code>. Three tables are nevertheless written by components that do not know about each other — <b>facts</b> by nine, <b>notes</b> by six, <b>tools</b> by four. Two files live outside the database entirely, and together they weaken the at-rest key.',
|
||||
lanes:[
|
||||
['authoritative owner', c => ['proc.mavend','state.db'].includes(c.id)],
|
||||
['database lifecycle', c => ['state.db_file','state.db_tmpfs','state.wrapped_key','proc.mavseal'].includes(c.id)],
|
||||
['tables written by unrelated components', c => ['state.facts','state.notes','state.tools'].includes(c.id)],
|
||||
['singly-owned tables', c => c.type==='table' && !['state.facts','state.notes','state.tools'].includes(c.id)],
|
||||
['shared mutable state — process-local', c => c.type==='shared-state'],
|
||||
['state outside the database', c => ['state.passkey_file','state.maildata','state.media_blobs'].includes(c.id)],
|
||||
['writers', c => (c.writes||[]).length>0 && c.type!=='table'],
|
||||
['readers', c => (c.reads||[]).length>0 && c.type!=='table' && !(c.writes||[]).length],
|
||||
]},
|
||||
{ id:'v5', name:'5 · Dependency and boundary map', hint:'Components, not classes. Cycles, cross-layer calls, fan-in and fan-out.',
|
||||
note:'The one <b>cycle</b> is deliberate and documented at both ends: <code>daemonAPI.chatFn = handler.handleText</code> and <code>handler.api</code> back-patched by <code>upgradeAPI</code>. The <b>fan-in</b> hotspot is <code>ipc.Server</code>, reached by six processes and carrying eight function fields that bypass CoreAPI entirely. The <b>fan-out</b> hotspots are <code>reactiveHandler</code> (34 fields) and <code>tickLoop</code> (thirteen jobs in one function).',
|
||||
lanes:[
|
||||
['process and network boundaries', c => c.type==='boundary'],
|
||||
['entry layer', c => ['core.ipc_server','core.voice_server','core.telegram_intake','core.auth_gate'].includes(c.id)],
|
||||
['API layer', c => c.type==='adapter'],
|
||||
['turn layer — fan-out hotspot', c => ['core.reactive_handler','core.turn_route','core.preroute','core.action_table','core.query_chain'].includes(c.id)],
|
||||
['routing layer', c => c.id.startsWith('router.')],
|
||||
['proactive layer — fan-out hotspot', c => ['core.tick_loop','core.gatherer','core.rules','core.dispatcher'].includes(c.id)],
|
||||
['construction layer', c => ['core.wiring'].includes(c.id)],
|
||||
['state layer', c => ['state.db'].includes(c.id) || c.type==='shared-state'],
|
||||
['evaluation and gates', c => c.type==='test'],
|
||||
]},
|
||||
{ id:'c1', name:'6 · Capabilities', hint:'Every capability against the seven dimensions. Sorted by the spec, or by domain.', caps:true,
|
||||
note:'<b>Nothing here is asserted.</b> The six build dimensions come from the <code>status</code> field of every component the capability maps to, and <code>verified</code> comes from a probe run against the deployed stack. A capability can be coded and unwired, wired and unconfigured, or configured and undeployed, and those are three different pieces of work — which is why this is not one <code>implemented</code> column. Source: <code>docs/capabilities/ledger.yaml</code>.' },
|
||||
{ id:'c2', name:'7 · Invariants', hint:'The twelve cross-cutting rules, and which components participate in each.', inv:true,
|
||||
note:'These rules run across all 51 capabilities and no capability\'s definition of done states any of them, so breaking one breaks many at once without producing a single failing criterion. <b>Target</b> is whether the rule is written down. <b>Implementation</b> is the status of the components that participate. <b>Runtime</b> is what the probe run observed for the capabilities it touches. Source: <code>docs/capabilities/invariants.yaml</code>, prose and evidence in <code>invariants.md</code>.' },
|
||||
];
|
||||
|
||||
/* ---------------- runtime flows ---------------- */
|
||||
const FLOWS = {
|
||||
reminder: { name:'A reminder request', file:'03a-flow-reminder.mmd', steps:[
|
||||
{who:['proc.mavwaked'], t:'The keyword head scores the utterance at or above 0.999 and silero VAD closes it. One clean blob ships over the ssh tunnel.', ev:'cmd/mavwaked/wakeword.go, deploy/mavwaked.service'},
|
||||
{who:['core.voice_server','core.reactive_handler'], t:'The conn already has a Session; HandlePushToTalk transcribes and enters runTurn.', ev:'internal/voice/server.go, cmd/mavend/voice.go'},
|
||||
{who:['core.decision_trace'], t:'A decision record is installed on the context before anything can claim the turn, so the mic, telegram and the web leave the same trail.', ev:'cmd/mavend/voice.go step 0'},
|
||||
{who:['core.preroute'], t:'Eleven rungs get first refusal. None claims "напомни позвонить маме".', ev:'cmd/mavend/voice.go steps 1 to 5e, preRouteLadder'},
|
||||
{who:['router.stage0','router.cascade'], t:'ReminderGrammar matches at stage 0 and wins outright at confidence 1.0. The extractor then fills the slots the grammar did not match.', ev:'internal/router/stagezero.go, router.go fillMatchedSlots'},
|
||||
{who:['core.reactive_handler'], t:'BRANCH — the route is confident and incomplete. missingFor names `time`, so step 8 fires even though dec.Clarify is false.', ev:'cmd/mavend/voice.go step 8, Vikunja #557', branch:true},
|
||||
{who:['state.clarify_store'], t:'The request is parked as a PendingQuestion and she asks one question about one thing. The store is in memory on purpose: a restart expires it.', ev:'internal/dialogue/clarify.go, cmd/mavend/clarify.go'},
|
||||
{who:['core.preroute'], t:'The next utterance is claimed by rung 4, resolveClarifyAnswer, and parsed with the same parsers stage 2 uses.', ev:'cmd/mavend/clarify.go finishClarified'},
|
||||
{who:['core.action_reminder'], t:'ResolvedTheHour guards a time the parser did not really read. The row is written, and the confirmation is phrased FROM THE ROW, never from the utterance.', ev:'cmd/mavend/actions_reminder.go, Vikunja #507'},
|
||||
{who:['state.reminders'], t:'One append. cron and next_fire_ts exist as columns and this path never sets them.', ev:'internal/store/reminders.go, migrations.go #2', branch:true},
|
||||
{who:['core.tick_loop','core.gatherer'], t:'Later, on a 60s ticker: the gatherer collapses due reminders by delivery group and RemindDecisions bypasses the restraint gate.', ev:'internal/loop/gather.go collapseReminders, loop.go RemindDecisions'},
|
||||
{who:['core.dispatcher','state.delivery_attempts'], t:'The outbox records intent BEFORE the external send, so a crash leaves a pending row rather than silence.', ev:'internal/delivery/dispatcher.go beginReminderOutbox'},
|
||||
{who:['core.sink_voice','core.sink_telegram'], t:'Voice when a session is live; away, ntfy is nil because the config disables it, so telegram carries it. A definite failure advances the persisted bounded backoff.', ev:'internal/delivery/channel.go ChannelsFor, cmd/mavend/main.go wireNtfySink'},
|
||||
]},
|
||||
fact: { name:'A factual / state update', file:'03b-flow-fact.mmd', steps:[
|
||||
{who:['core.reactive_handler','router.cascade'], t:'"выпил воды" reaches the cascade. Stage 0 declines, the heads or the LLM router or the classifier names IntentFact with a key and a value.', ev:'internal/router/router.go Route'},
|
||||
{who:['core.action_fact'], t:'BRANCH — a question-shaped utterance is never a fact. It is re-routed into actionQuery with the model-guessed key cleared, and the stage 0 world destination reconstructed so the boundary cannot claim it.', ev:'cmd/mavend/actions_fact.go, Vikunja #470', branch:true},
|
||||
{who:['core.action_fact','core.action_chat'], t:'BRANCH — a passing complaint is not a fact either. It becomes chat and stores nothing, because recall reads a self row back later as if it were still true.', ev:'cmd/mavend/actions_fact.go, Vikunja #481', branch:true},
|
||||
{who:['core.intake_api','state.facts'], t:'WriteFact appends kind=self, source=tap:voice, Subject=Key. Confidence is 1.0 only for a value he actually said. The decorator publishes one intake envelope.', ev:'cmd/mavend/actions_fact.go factConfidence, cmd/mavend/intake.go'},
|
||||
{who:['state.memory_vectors'], t:'The key’s old vectors are pruned, then the FACT text is embedded with the passage prefix and inserted. The utterance rides along as provenance and is never embedded.', ev:'cmd/mavend/actions_fact.go pruneFactVectors, Vikunja #493'},
|
||||
{who:['core.reactive_handler'], t:'Step 9b: a fact that answers a live nudge closes it as `acted`, silently. The fact reply stands.', ev:'cmd/mavend/ack.go ackFromFact'},
|
||||
{who:['core.fact_enrichment','ext.nexus'], t:'Asynchronously, the enrichment worker resolves Subject to a canonical entity id with per-fact backoff. An ambiguous result is NOT stored.', ev:'cmd/mavend/factenrichment.go resolveOne'},
|
||||
{who:['core.tick_loop','core.morning'], t:'On the next tick the morning routine sees the item evidenced inside its window and will not nudge for it.', ev:'cmd/mavend/tick_morning.go gatherMorningFacts'},
|
||||
{who:['core.pattern','state.events'], t:'detectPatterns scans every action+object pair for a stable interval and may propose a routine. notify is false in the deployed config, so it proposes silently.', ev:'cmd/mavend/tick_routines.go, deploy/mavend.json pattern_proposals'},
|
||||
{who:['state.facts'], t:'A wrong value is superseded, never overwritten: voids_id points at the row it cancels, and both correction paths drop the key’s vectors.', ev:'internal/store/schema.sql, internal/store/facts.go'},
|
||||
]},
|
||||
world: { name:'A world query, tool-backed', file:'03c-flow-world-query.mmd', steps:[
|
||||
{who:['router.stage0'], t:'"что такое TCP?" matches WorldQueryGrammars, a literal definition frame. That match sets Source=SourceWorld AND SourceAnchored, which happens here and nowhere else in the cascade.', ev:'internal/router/router.go d.SourceAnchored = d.Source != SourceUnknown'},
|
||||
{who:['core.query_chain','core.decision_trace'], t:'actionQuery declares the full 22-source roster to the record, so a reader can tell "looked and passed" from "never asked".', ev:'cmd/mavend/actions_query.go decision.Expect'},
|
||||
{who:['core.query_chain'], t:'queryWalk removes only the sources marked guesses:true whose destination is not world. Sources that LOOK are all still asked, because a named destination is evidence and not a promise.', ev:'cmd/mavend/actions_query.go queryWalk'},
|
||||
{who:['core.q.personal'], t:'BRANCH — the personal boundary is dropped only because a literal pattern named the destination. A model or a softmax naming SourceWorld would NOT drop it.', ev:'cmd/mavend/actions_query.go queryWalk anchored, V-666', branch:true},
|
||||
{who:['core.q.factbykey','core.q.memory','core.q.notes'], t:'His own data still gets its turn: fact-by-key, the day plan, tasks, money, history, the calendar, then the three recall passes gated by min score 0.80 and min margin 0.008.', ev:'cmd/mavend/actions_query.go querySources, deploy/mavend.json'},
|
||||
{who:['core.q.search','ext.searxng'], t:'SearXNG is asked verbatim, with no rewriter. Only the query string leaves the box: no note, no fact, no persona block, no history.', ev:'cmd/mavend/actions_query.go querySearch'},
|
||||
{who:['core.phraser'], t:'The snippets are handed over as evidence for the question, trimmed under one budget, and phrased. With no phraser the best snippet is read back rather than pretending the search did not happen.', ev:'cmd/mavend/actions_query.go phraseSource, readBack'},
|
||||
{who:['core.q.kiwix','ext.kiwix'], t:'BRANCH — empty or unreachable falls through to the offline ZIMs, Russian first. No results there is not announced.', ev:'cmd/mavend/actions_query.go queryKiwix', branch:true},
|
||||
{who:['core.q.web'], t:'A page he named by URL is read only if he actually said a URL, and it sits AFTER the ZIMs on purpose.', ev:'cmd/mavend/actions_query.go queryWeb, Vikunja #259'},
|
||||
{who:['core.q.general'], t:'Last: the resident model answers from its own weights. Response.Empty() is the whole gate on a world answer; there is no quality threshold in front of it.', ev:'cmd/mavend/actions_query.go queryGeneral, CLAUDE.md'},
|
||||
{who:['core.query_chain'], t:'Whichever source claimed is logged and noted on the turn sink, so /chat can show it. Everyone below the winner is recorded as NeverAsked.', ev:'cmd/mavend/querysource.go noteQuerySource, Vikunja #474'},
|
||||
]},
|
||||
};
|
||||
|
||||
/* ---------------- filters ---------------- */
|
||||
const T = id => document.getElementById(id).checked;
|
||||
function statusHidden(st){
|
||||
if (st==='configured-off') return !T('tOff');
|
||||
if (st==='built-not-deployed') return !T('tUndeployed');
|
||||
if (st==='planned-unwired'||st==='dead') return !T('tPlanned');
|
||||
return false;
|
||||
}
|
||||
function edgeHidden(e){
|
||||
if (e.confidence==='low' && !T('tLow')) return true;
|
||||
if (e.confidence==='medium' && !T('tMed')) return true;
|
||||
return statusHidden(e.status);
|
||||
}
|
||||
const edgesOf = id => ARCH.edges.filter(e => e.from===id || e.to===id);
|
||||
|
||||
/* ---------------- rendering ---------------- */
|
||||
function badges(c){
|
||||
const out=[];
|
||||
if (c.status!=='implemented') out.push(`<span class="b ${ {'configured-off':'off','built-not-deployed':'nd','planned-unwired':'pl','temporary':'tmp','partially-wired':'pw','dead':'pl'}[c.status]||'tmp'}">${c.status}</span>`);
|
||||
if (c.confidence==='low') out.push('<span class="b lo">uncertain</span>');
|
||||
if (c.confidence==='medium') out.push('<span class="b me">medium confidence</span>');
|
||||
return out.length?`<div class="badges">${out.join('')}</div>`:'';
|
||||
}
|
||||
|
||||
function chipHTML(c){
|
||||
return `<div class="chip" data-id="${c.id}" data-t="${c.type}">
|
||||
<div class="nm">${c.id.split('.').pop().replace(/_/g,' ')}</div>
|
||||
<div class="ty">${c.type} · ${c.id}</div>${badges(c)}</div>`;
|
||||
}
|
||||
|
||||
function renderView(){
|
||||
const v = VIEWS.find(x=>x.id===S.view);
|
||||
const main = document.getElementById('main');
|
||||
if (v.flow) return renderFlow(main);
|
||||
if (v.caps) return renderCaps(main);
|
||||
if (v.inv) return renderInv(main);
|
||||
let html = v.note ? `<div class="viewnote">${v.note}</div>` : '';
|
||||
html += diagramPanel(diagramFileFor());
|
||||
const used = new Set();
|
||||
v.lanes.forEach(([title, pred], i) => {
|
||||
const items = ARCH.components.filter(c => !used.has(c.id) && pred(c) && !statusHidden(c.status));
|
||||
items.forEach(c=>used.add(c.id));
|
||||
if (!items.length) return;
|
||||
const key = v.id+':'+i, open = !S.collapsed[key];
|
||||
html += `<div class="lane"><div class="lane-h" data-lane="${key}">
|
||||
<span class="caret">${open?'▾':'▸'}</span><b>${title}</b><i>${items.length}</i></div>
|
||||
<div class="chips" ${open?'':'style="display:none"'}>${items.map(chipHTML).join('')}</div></div>`;
|
||||
});
|
||||
html += `<svg class="wires" id="wires"></svg>`;
|
||||
main.innerHTML = html;
|
||||
main.querySelectorAll('.lane-h').forEach(h=>h.onclick=()=>{ S.collapsed[h.dataset.lane]=!S.collapsed[h.dataset.lane]; renderView(); paint(); });
|
||||
main.querySelectorAll('.chip').forEach(ch=>ch.onclick=()=>select(ch.dataset.id));
|
||||
wireDiagram();
|
||||
requestAnimationFrame(drawWires);
|
||||
}
|
||||
|
||||
/* ---------------- capabilities and invariants ---------------- */
|
||||
const DIMS = ['designed','code_present','wired','configured','deployed','reachable','verified'];
|
||||
const DIMH = {designed:'design',code_present:'code',wired:'wired',configured:'config',
|
||||
deployed:'deploy',reachable:'reach',verified:'verified'};
|
||||
const capById = Object.fromEntries(CAPS.capabilities.map(c => [c.id, c]));
|
||||
// A criterion is blocked when it was observed blocked, or when its reason names
|
||||
// something outside the code as the thing in the way.
|
||||
const BLOCKREASON = new Set(['configuration missing','deployment missing',
|
||||
'external dependency unavailable','scenario missing']);
|
||||
const dotCls = v => v==='spec-only' ? 'speconly' : v;
|
||||
// The ledger carries markdown inline code, because docs/spec.md does. Rendering
|
||||
// it literally puts backticks on screen next to every path.
|
||||
const md = t => (t||'').replace(/[&<>]/g, m => ({'&':'&','<':'<','>':'>'}[m]))
|
||||
.replace(/`([^`]+)`/g, '<code>$1</code>')
|
||||
.replace(/\*\*([^*]+)\*\*/g, '<b>$1</b>');
|
||||
|
||||
function capRow(c){
|
||||
const cls = {'capability missing':'missing','capability exists but unreachable':'unreachable',
|
||||
'capability partial':'partial'}[c.implementation.gap_class] || '';
|
||||
const gap = c.implementation.gap_class==='none' ? '' : c.implementation.gap_class;
|
||||
return `<tr class="cap" data-cap="${c.id}">
|
||||
<td class="n">${c.title}${c.scope!=='v1'?'<em>deferred</em>':''}
|
||||
<em class="gapc ${cls}">${gap}</em></td>
|
||||
${DIMS.map(d=>`<td class="d"><span class="dot ${dotCls(c.implementation[d])}" title="${d}: ${c.implementation[d]}"></span></td>`).join('')}
|
||||
</tr>`;
|
||||
}
|
||||
|
||||
function renderCaps(main){
|
||||
const v = VIEWS.find(x=>x.id===S.view);
|
||||
const by = S.capBy;
|
||||
const groups = {};
|
||||
CAPS.capabilities.forEach(c => {
|
||||
const keys = by==='domain' ? (c.domain.length?c.domain:['unassigned']) : [c.section];
|
||||
keys.forEach(k => (groups[k] = groups[k]||[]).push(c));
|
||||
});
|
||||
const order = by==='domain' ? Object.keys(groups).sort()
|
||||
: [...new Set(CAPS.capabilities.map(c=>c.section))];
|
||||
main.innerHTML = `<div class="viewnote">${v.note}</div>
|
||||
<div class="capsel">
|
||||
<button data-by="section" class="${by==='section'?'on':''}">by spec section</button>
|
||||
<button data-by="domain" class="${by==='domain'?'on':''}">by domain</button>
|
||||
</div>
|
||||
<table class="mx"><thead><tr><th>capability</th>
|
||||
${DIMS.map(d=>`<th class="d">${DIMH[d]}</th>`).join('')}</tr></thead>
|
||||
<tbody>${order.map(g=>`<tr class="grp"><td colspan="8">${g} · ${groups[g].length}</td></tr>`
|
||||
+ groups[g].map(capRow).join('')).join('')}</tbody></table>`;
|
||||
main.querySelectorAll('.capsel button').forEach(b=>b.onclick=()=>{S.capBy=b.dataset.by;renderCaps(main);});
|
||||
main.querySelectorAll('tr.cap').forEach(r=>r.onclick=()=>selectCap(r.dataset.cap));
|
||||
}
|
||||
|
||||
function selectCap(id){
|
||||
S.sel = null; S.selCap = id;
|
||||
document.querySelectorAll('tr.cap').forEach(r=>r.classList.toggle('sel', r.dataset.cap===id));
|
||||
renderCapSide(id);
|
||||
}
|
||||
|
||||
function compChip(cid){
|
||||
const c = byId[cid];
|
||||
const k = c && (c.status==='configured-off' ? 'off'
|
||||
: ['planned-unwired','dead','partially-wired'].includes(c.status) ? 'pl' : '');
|
||||
return `<button class="cchip ${k}" data-id="${cid}" title="${c?c.status:'unknown'}">${cid}</button>`;
|
||||
}
|
||||
|
||||
function renderCapSide(id){
|
||||
const c = capById[id], side = document.getElementById('side');
|
||||
const blockers = c.criteria.filter(cr => cr.verified==='blocked' || BLOCKREASON.has(cr.reason));
|
||||
const qs = INV.filter(iv => iv.capabilities.includes(id) && iv.question);
|
||||
const scen = c.scenarios || [];
|
||||
side.innerHTML = `
|
||||
<h3>${c.title}</h3>
|
||||
<div class="sub">${c.section} · ${c.domain.join(', ')} · ${c.scope}${
|
||||
c.implementation.gap_class==='none'?'':' · '+c.implementation.gap_class}</div>
|
||||
<p>${md(c.state)}</p>
|
||||
<section><h4>Implementation</h4>
|
||||
<div class="bars">${DIMS.map(d=>{
|
||||
const v = c.implementation[d];
|
||||
const k = v==='yes'?'ok':v==='partial'?'mid':'warn';
|
||||
return `<div class="bar"><b>${DIMH[d]}</b><span class="v ${k}">${v}</span></div>`;
|
||||
}).join('')}</div></section>
|
||||
<section><h4>Components — ${(c.components||[]).length}</h4>
|
||||
${(c.components||[]).length ? c.components.map(compChip).join('')
|
||||
: '<div class="empty" style="margin:0">Nothing carries this capability. That is the finding.</div>'}</section>
|
||||
<section><h4>Definition of done — ${c.criteria.length}</h4>
|
||||
${c.criteria.map(cr=>`<div class="crit">
|
||||
<span class="vd ${cr.verified}">${cr.verified}</span><span class="rs">${cr.reason}</span>
|
||||
<p>${md(cr.text)}</p>
|
||||
${cr.detail?`<p style="color:var(--dim2)">${md(cr.detail)}</p>`:''}
|
||||
${(cr.evidence||[]).length?`<p class="mono" style="font-size:11px;color:var(--dim2)">${cr.evidence.join('<br>')}</p>`:''}
|
||||
</div>`).join('')}</section>
|
||||
<section><h4>Blockers — ${blockers.length}</h4>
|
||||
${blockers.length ? '<ul class="plain">'+blockers.map(b=>`<li>${b.reason} · <span class="mono">${b.id}</span></li>`).join('')+'</ul>'
|
||||
: '<div class="empty" style="margin:0">none. Nothing outside the code is in the way.</div>'}</section>
|
||||
<section><h4>Scenarios — ${scen.length}</h4>
|
||||
${scen.length ? '<ul class="plain">'+scen.map(x=>`<li class="mono">${x.name} ${x.exists?'':'<span class="b off">absent from disk</span>'}</li>`).join('')+'</ul>'
|
||||
: '<div class="empty" style="margin:0">none named</div>'}</section>
|
||||
<section><h4>Unresolved product questions — ${qs.length}</h4>
|
||||
${qs.length ? qs.map(iv=>`<div class="crit"><span class="mark ${iv.mark}">invariant ${iv.id}</span>
|
||||
<p>${iv.question}</p></div>`).join('')
|
||||
: '<div class="empty" style="margin:0">none</div>'}</section>`;
|
||||
side.querySelectorAll('.cchip').forEach(b=>b.onclick=()=>{S.sel=b.dataset.id;renderSide(b.dataset.id);});
|
||||
}
|
||||
|
||||
function invRollup(iv){
|
||||
const comps = iv.components.map(x=>byId[x]).filter(Boolean);
|
||||
const bad = comps.filter(c=>c.status!=='implemented'&&c.status!=='temporary').length;
|
||||
const caps = iv.capabilities.map(x=>capById[x]).filter(Boolean);
|
||||
const ver = caps.filter(c=>c.implementation.verified==='yes').length;
|
||||
const part = caps.filter(c=>c.implementation.verified==='partial').length;
|
||||
return {comps, bad, caps, ver, part};
|
||||
}
|
||||
|
||||
function renderInv(main){
|
||||
const v = VIEWS.find(x=>x.id===S.view);
|
||||
main.innerHTML = `<div class="viewnote">${v.note}</div>` + INV.map(iv=>{
|
||||
const r = invRollup(iv);
|
||||
const mk = iv.mark==='explicit'?'ok':iv.mark==='implied'?'mid':'warn';
|
||||
const ik = r.bad?'mid':'ok';
|
||||
const vk = r.ver===r.caps.length?'ok':(r.ver+r.part)?'mid':'warn';
|
||||
return `<div class="inv" data-inv="${iv.id}">
|
||||
<h4><span class="n">${iv.id}</span> ${iv.title}
|
||||
<span class="mark ${iv.mark}">${iv.mark}${iv.split?' · split':''}</span></h4>
|
||||
<div class="bars">
|
||||
<div class="bar"><b>target</b><span class="v ${mk}">${
|
||||
iv.mark==='explicit'?'written down':iv.mark==='implied'?'not stated':'no answer exists'}</span></div>
|
||||
<div class="bar"><b>implementation</b><span class="v ${ik}">${r.comps.length} components, ${r.bad} not live</span></div>
|
||||
<div class="bar"><b>runtime verification</b><span class="v ${vk}">${r.ver} of ${r.caps.length} capabilities verified${r.part?', '+r.part+' partly':''}</span></div>
|
||||
</div>
|
||||
${iv.question?`<p class="q">${iv.question}</p>`:''}
|
||||
<div style="margin-top:8px">${iv.capabilities.map(c=>`<button class="cchip" data-cap="${c}">${c}</button>`).join('')}</div>
|
||||
<div style="margin-top:4px">${iv.components.map(compChip).join('')}</div>
|
||||
</div>`;
|
||||
}).join('');
|
||||
main.querySelectorAll('.cchip[data-id]').forEach(b=>b.onclick=()=>{S.sel=b.dataset.id;renderSide(b.dataset.id);});
|
||||
main.querySelectorAll('.cchip[data-cap]').forEach(b=>b.onclick=()=>renderCapSide(b.dataset.cap));
|
||||
}
|
||||
|
||||
function renderFlow(main){
|
||||
const f = FLOWS[S.flow];
|
||||
main.innerHTML = `${diagramPanel(FLOWS[S.flow].file)}<div class="viewnote"><b>Three requests, traced through real code.</b> Steps marked in red are branches, fallbacks or refusals the implementation actually takes. Click a component name to open its record. The Mermaid sequence source for each flow is under the panel on the right.</div>
|
||||
<div class="flowsel">${Object.entries(FLOWS).map(([k,x])=>`<button data-f="${k}" class="${k===S.flow?'on':''}">${x.name}</button>`).join('')}</div>
|
||||
<ol class="steps">${f.steps.map(s=>`<li class="${s.branch?'branch':''}">
|
||||
${s.who.map(w=>`<span class="who" data-id="${w}">${byId[w]?byId[w].id:w}</span>`).join('')}
|
||||
${s.t}<span class="ev">${s.ev}</span></li>`).join('')}</ol>`;
|
||||
main.querySelectorAll('.flowsel button').forEach(b=>b.onclick=()=>{S.flow=b.dataset.f;renderFlow(main);});
|
||||
main.querySelectorAll('.who').forEach(b=>b.onclick=()=>select(b.dataset.id));
|
||||
wireDiagram();
|
||||
}
|
||||
|
||||
function diagramFileFor(){
|
||||
return {v1:'01-system-topology.mmd',v2:'02-core-internals.mmd',
|
||||
v4:'04-state-ownership.mmd',v5:'05-dependency-boundary.mmd'}[S.view] || null;
|
||||
}
|
||||
|
||||
// The rendered picture, from the committed SVG beside the .mmd. Absent SVG ⇒
|
||||
// no panel at all, rather than an empty frame: `sh docs/architecture/render.sh`
|
||||
// is what fills it, and a missing file means that has not been run.
|
||||
function diagramPanel(mmFile){
|
||||
if (!mmFile) return '';
|
||||
const svg = SVG[mmFile.replace(/\.mmd$/, '.svg')];
|
||||
if (!svg) return '';
|
||||
const open = !S.diaClosed;
|
||||
return `<div class="dia">
|
||||
<div class="dia-h" id="diaH"><span class="caret">${open?'▾':'▸'}</span><b>Rendered diagram</b>
|
||||
<span class="fn">diagrams/${mmFile.replace(/\.mmd$/,'.svg')}</span>
|
||||
<span class="zoom"><button data-z="-1" title="zoom out">−</button><button data-z="0" title="fit">◻</button><button data-z="1" title="zoom in">+</button></span>
|
||||
</div>
|
||||
<div class="dia-body" id="diaBody" ${open?'':'style="display:none"'}><div id="diaScale">${svg}</div></div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function wireDiagram(){
|
||||
const h = document.getElementById('diaH'); if (!h) return;
|
||||
const body = document.getElementById('diaBody'), scale = document.getElementById('diaScale');
|
||||
// The mermaid SVG carries width="100%" and a viewBox, so it fills whatever
|
||||
// box it is given. Widening the wrapper past 100% is the zoom, and the
|
||||
// .dia-body scrollbar is what makes the extra width reachable. A CSS
|
||||
// transform would scale the scrollport too and clip the bottom of a tall
|
||||
// flowchart, which 01-system-topology is at 2304x3542.
|
||||
const apply = () => { scale.style.width = (S.diaZoom*100)+'%'; };
|
||||
h.onclick = ev => {
|
||||
const z = ev.target.closest('button');
|
||||
if (z){ ev.stopPropagation();
|
||||
const d = +z.dataset.z;
|
||||
S.diaZoom = d===0 ? 1 : Math.min(3, Math.max(.25, S.diaZoom + d*0.2));
|
||||
apply(); return; }
|
||||
S.diaClosed = !S.diaClosed; renderView(); if (S.sel) select(S.sel);
|
||||
};
|
||||
apply();
|
||||
}
|
||||
|
||||
function drawWires(){
|
||||
const svg = document.getElementById('wires');
|
||||
if (!svg) return;
|
||||
svg.innerHTML='';
|
||||
if (!T('tWires') || !S.sel) return;
|
||||
const main = document.getElementById('main'), mr = main.getBoundingClientRect();
|
||||
const pos = id => { const el = main.querySelector(`.chip[data-id="${id}"]`); if(!el) return null;
|
||||
const r = el.getBoundingClientRect();
|
||||
return {x:r.left-mr.left+main.scrollLeft+r.width/2, y:r.top-mr.top+main.scrollTop+r.height/2}; };
|
||||
const a = pos(S.sel); if (!a) return;
|
||||
edgesOf(S.sel).filter(e=>!edgeHidden(e)).forEach(e=>{
|
||||
const other = e.from===S.sel ? e.to : e.from, b = pos(other); if (!b) return;
|
||||
const out = e.from===S.sel;
|
||||
const col = e.confidence==='low' ? '#e08080' : e.confidence==='medium' ? '#c9a227' : (out?'#7fb3ff':'#6ed0a8');
|
||||
const mx = (a.x+b.x)/2;
|
||||
const p = document.createElementNS('http://www.w3.org/2000/svg','path');
|
||||
p.setAttribute('d',`M${a.x},${a.y} C${mx},${a.y} ${mx},${b.y} ${b.x},${b.y}`);
|
||||
p.setAttribute('stroke',col); p.setAttribute('stroke-width','1.4'); p.setAttribute('fill','none');
|
||||
p.setAttribute('opacity','.75');
|
||||
if (e.status!=='implemented') p.setAttribute('stroke-dasharray','5 4');
|
||||
svg.appendChild(p);
|
||||
});
|
||||
}
|
||||
|
||||
function select(id){
|
||||
S.sel = id;
|
||||
document.querySelectorAll('.chip').forEach(ch=>{
|
||||
ch.classList.remove('sel','rel','dim');
|
||||
if (ch.dataset.id===id) ch.classList.add('sel');
|
||||
});
|
||||
const rel = new Set(edgesOf(id).filter(e=>!edgeHidden(e)).map(e=>e.from===id?e.to:e.from));
|
||||
document.querySelectorAll('.chip').forEach(ch=>{
|
||||
if (ch.dataset.id!==id) ch.classList.add(rel.has(ch.dataset.id)?'rel':'dim');
|
||||
});
|
||||
renderSide(id);
|
||||
drawWires();
|
||||
}
|
||||
|
||||
function relRow(e, id){
|
||||
const out = e.from===id, other = out?e.to:e.from, oc = byId[other];
|
||||
const marks=[];
|
||||
if (e.confidence!=='high') marks.push(`<span class="b ${e.confidence==='low'?'lo':'me'}">${e.confidence}</span>`);
|
||||
if (e.status!=='implemented') marks.push(`<span class="b off">${e.status}</span>`);
|
||||
return `<button class="rel" data-id="${other}"><span class="k">${out?'→':'←'} ${e.kind}</span>${oc?oc.id:other} ${marks.join('')}
|
||||
<span class="ev">${e.label}${e.evidence?' · '+e.evidence:''}</span></button>`;
|
||||
}
|
||||
|
||||
function renderSide(id){
|
||||
const c = byId[id], side = document.getElementById('side');
|
||||
if (!c){ side.innerHTML = `<div class="empty">No record for <code>${id}</code>.</div>`; return; }
|
||||
const es = edgesOf(id).filter(e=>!edgeHidden(e));
|
||||
const outE = es.filter(e=>e.from===id), inE = es.filter(e=>e.to===id);
|
||||
const mmFile = S.view==='v3' ? FLOWS[S.flow].file : {v1:'01-system-topology.mmd',v2:'02-core-internals.mmd',v4:'04-state-ownership.mmd',v5:'05-dependency-boundary.mmd'}[S.view];
|
||||
side.innerHTML = `
|
||||
<h3>${c.id}</h3>
|
||||
<div class="sub">${c.type} · ${c.group} · ${c.status} · ${c.confidence} confidence</div>
|
||||
<p>${c.responsibility}</p>
|
||||
${c.notes?`<div class="note">${c.notes}</div>`:''}
|
||||
<section><h4>Files</h4><ul class="plain">${c.files.map(f=>`<li class="mono">${f}</li>`).join('')}</ul></section>
|
||||
<section><h4>Symbols</h4><ul class="plain">${c.symbols.map(s=>`<li class="mono">${s}</li>`).join('')}</ul></section>
|
||||
<section><h4>Outgoing — ${outE.length}</h4>${outE.map(e=>relRow(e,id)).join('')||'<div class="empty" style="margin:0">none</div>'}</section>
|
||||
<section><h4>Incoming — ${inE.length}</h4>${inE.map(e=>relRow(e,id)).join('')||'<div class="empty" style="margin:0">none</div>'}</section>
|
||||
${mmFile?`<section><h4>Mermaid source — ${mmFile}</h4><pre class="mm">${MERMAID[mmFile].replace(/[&<>]/g,m=>({'&':'&','<':'<','>':'>'}[m]))}</pre></section>`:''}`;
|
||||
side.querySelectorAll('.rel').forEach(b=>b.onclick=()=>{
|
||||
const t=b.dataset.id;
|
||||
if (!document.querySelector(`.chip[data-id="${t}"]`)) { renderSide(t); S.sel=t; drawWires(); }
|
||||
else select(t);
|
||||
});
|
||||
}
|
||||
|
||||
/* ---------------- search ---------------- */
|
||||
function search(){
|
||||
const q = document.getElementById('q').value.trim().toLowerCase();
|
||||
const box = document.getElementById('results');
|
||||
if (q.length<2){ box.innerHTML=''; return; }
|
||||
const hits=[];
|
||||
for (const c of ARCH.components){
|
||||
const where=[];
|
||||
if (c.id.toLowerCase().includes(q)) where.push('id');
|
||||
if (c.responsibility.toLowerCase().includes(q)) where.push('responsibility');
|
||||
const f = c.files.filter(x=>x.toLowerCase().includes(q));
|
||||
const s = c.symbols.filter(x=>x.toLowerCase().includes(q));
|
||||
if (f.length) where.push('file: '+f[0]);
|
||||
if (s.length) where.push('symbol: '+s[0]);
|
||||
if ((c.notes||'').toLowerCase().includes(q)) where.push('note');
|
||||
if (where.length) hits.push([c, where]);
|
||||
}
|
||||
box.innerHTML = hits.slice(0,60).map(([c,w])=>`<button data-id="${c.id}">${c.id}<br><em>${w.join(' · ')}</em></button>`).join('')
|
||||
|| '<button disabled style="color:#5f6c85">no match</button>';
|
||||
box.querySelectorAll('button[data-id]').forEach(b=>b.onclick=()=>{
|
||||
const el = document.querySelector(`.chip[data-id="${b.dataset.id}"]`);
|
||||
if (el){ select(b.dataset.id); el.scrollIntoView({block:'center',behavior:'smooth'}); }
|
||||
else { S.sel=b.dataset.id; renderSide(b.dataset.id); }
|
||||
});
|
||||
}
|
||||
|
||||
/* ---------------- boot ---------------- */
|
||||
document.getElementById('commit').textContent = ARCH.commit.slice(0,7);
|
||||
document.getElementById('gen').textContent = ARCH.generated;
|
||||
document.getElementById('counts').textContent = `${ARCH.components.length} components · ${ARCH.edges.length} relations`;
|
||||
document.getElementById('dirty').textContent = ARCH.working_tree;
|
||||
document.getElementById('views').innerHTML = VIEWS.map(v=>`<button class="viewbtn" data-v="${v.id}">${v.name}<small>${v.hint}</small></button>`).join('');
|
||||
document.getElementById('legend').innerHTML = [...new Set(ARCH.components.map(c=>c.type))].sort().map(t=>`<span>${t}</span>`).join('');
|
||||
|
||||
function setView(id){ S.view=id; S.sel=null; S.selCap=null;
|
||||
// The relation filters and the component-type legend do nothing in the
|
||||
// capability views. Leaving them visible reads as controls that are broken.
|
||||
const vw = VIEWS.find(x=>x.id===id);
|
||||
document.getElementById('archctl').style.display = (vw.caps || vw.inv) ? 'none' : '';
|
||||
document.querySelectorAll('.viewbtn').forEach(b=>b.classList.toggle('on', b.dataset.v===id));
|
||||
renderView();
|
||||
const v = VIEWS.find(x=>x.id===id);
|
||||
document.getElementById('side').innerHTML = v.caps
|
||||
? '<div class="empty">Select a capability to see its definition of done, every verdict and its evidence, the components that carry it, its blockers and the product questions it waits on.<br><br>A dot is never an opinion. Six of the seven come from component status, the seventh from a probe run.</div>'
|
||||
: v.inv
|
||||
? '<div class="empty">Twelve rules that run across all 51 capabilities. Click a capability or a component to open its record.<br><br>An unresolved rule is a product question, not a defect.</div>'
|
||||
: '<div class="empty">Select a component to see its responsibility, the files and symbols it was read from, and every relation in and out.</div>';
|
||||
}
|
||||
document.querySelectorAll('.viewbtn').forEach(b=>b.onclick=()=>setView(b.dataset.v));
|
||||
['tLow','tMed','tOff','tUndeployed','tPlanned','tWires'].forEach(k=>
|
||||
document.getElementById(k).onchange=()=>{ renderView(); if(S.sel) select(S.sel); });
|
||||
document.getElementById('q').oninput = search;
|
||||
document.getElementById('main').addEventListener('scroll', drawWires);
|
||||
window.addEventListener('resize', drawWires);
|
||||
setView('v1');
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,626 @@
|
||||
# Maven Current-State Audit
|
||||
|
||||
**Date:** 2026-09-05
|
||||
**Scope:** Read-only code + tests investigation. No inference from names or docs.
|
||||
**Source of truth:** code and tests only.
|
||||
|
||||
---
|
||||
|
||||
## A. Current End-to-End Flow Diagram
|
||||
|
||||
```
|
||||
Telegram Bot API (long-poll)
|
||||
|
|
||||
| api.Chat(ctx, "telegram:<id>", text)
|
||||
v
|
||||
mavweb POST /api/chat --> ipc.Client (unix socket) --> daemonAPI.Chat()
|
||||
|
|
||||
| chatFn = handler.handleText
|
||||
v
|
||||
Voice TCP :9100 --> HandlePushToTalk --> STT --> +------------------+
|
||||
| runTurn() |
|
||||
| voice.go:270 |
|
||||
+------------------+
|
||||
| 0. decision record (ctx)
|
||||
| 0b. turnRoute (computed once)
|
||||
| 1. expired clarify notice
|
||||
| 2. resolveConfirm (y/n)
|
||||
| 3. resolveRepair (correction)
|
||||
| 3b. resolveUntargetedRepair
|
||||
| 3c. resolveCommandProhibition
|
||||
| 4. resolveClarifyAnswer
|
||||
| 5. resolveQuietToggle
|
||||
| 5b. resolveSnooze
|
||||
| 5c. resolveAck
|
||||
| 5d. resolveReminderCancellation
|
||||
| 5e. resolveCandidate (ordinal)
|
||||
| 6. ROUTE (cascade)
|
||||
| 7. dialogue merge (followUpMerge)
|
||||
| 8. clarify (missing slots)
|
||||
| 9. applyAction (per-intent dispatch)
|
||||
| 10. replier (LLM or stub)
|
||||
+------------------+
|
||||
|
|
||||
v
|
||||
reply string
|
||||
```
|
||||
|
||||
### The routing cascade (step 6) in detail:
|
||||
|
||||
```
|
||||
utterance
|
||||
|
|
||||
v
|
||||
Stage 0: Grammars (stagezero.go:24-91)
|
||||
| 22 ordered regex/structural grammars. First match wins.
|
||||
| Confidence = 1.0, Stage = 0. fillMatchedSlots runs.
|
||||
| NO MATCH -> fall through
|
||||
v
|
||||
Stage 0b: Routing Heads (heads.go, ONNX)
|
||||
| 4 linear heads over mean-pooled e5-small: intent, destination, slot BIO, clarify
|
||||
| Below headsThreshold (0.6) -> decline, cascade continues
|
||||
| Intent head softmax -> Decision{Intent, Confidence, Source, Clarify}
|
||||
| completeParsedSingleVerbFact can overrule clarify
|
||||
| NO WIRE / ERROR / DECLINE -> fall through
|
||||
v
|
||||
Stage 1a: LLM Router (llmrouter.go, Qwen3-1.7B via llama-server)
|
||||
| GBNF grammar-constrained JSON output
|
||||
| gateLLMDecision: thin confidence (0.3) for incomplete slots
|
||||
| ERROR / PARSE FAIL / "unknown" -> fall through
|
||||
v
|
||||
Stage 1: Nearest-Centroid Classifier (classifier.go)
|
||||
| Cosine similarity over ONNX embeddings against seed centroids
|
||||
| Best intent wins (ties broken by name)
|
||||
v
|
||||
Stage 2: Slot Extraction (slots.go:55-91)
|
||||
| Dispatch on intent: DateTimeParser (reminder), ActMatcher (act), FactParser (fact)
|
||||
v
|
||||
Stage 3: Confidence Gate (router.go:244-248)
|
||||
| Confidence < threshold (0.55) -> Clarify = true, Stage = 3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## B. Package/File Ownership Map
|
||||
|
||||
### Command packages (cmd/)
|
||||
|
||||
| Package | Binary | Role |
|
||||
|---------|--------|------|
|
||||
| cmd/mavend | mavend | Core daemon: DB, IPC, routing, actions, voice server, tick loop |
|
||||
| cmd/mavweb | mavweb | Web UI + HTTP server (serves /, /dash, /history, /trace, /notifications, /tools) |
|
||||
| cmd/mavwaked | mavwaked | Wake-word + voice activity detection (silero VAD, energy threshold) |
|
||||
| cmd/mavsttd | mavsttd | Speech-to-text daemon (whisper.cpp / remote worker) |
|
||||
| cmd/mavttsd | mavttsd | Text-to-speech daemon (piper) |
|
||||
| cmd/mavgpud | mavgpud | GPU proxy daemon (workpc inference) |
|
||||
| cmd/mavcaldav | mavcaldav | CalDAV sync daemon |
|
||||
| cmd/mavmaild | mavmaild | Mail ingestion daemon |
|
||||
| cmd/mavpoll | mavpoll | Polling daemon (Kuma monitors) |
|
||||
| cmd/mavenclient | mavenclient | CLI client |
|
||||
| cmd/mavupdate | mavupdate | Self-update tool |
|
||||
| cmd/mavseal | mavseal | Seal/encryption tool |
|
||||
| cmd/e2eprobe | e2eprobe | End-to-end probe |
|
||||
| cmd/labelgen | labelgen | Label generation tool |
|
||||
| cmd/mavend/seedtest | (test helper) | Test seeder |
|
||||
|
||||
### Internal packages relevant to routing/action
|
||||
|
||||
| Package | Key files | Role |
|
||||
|---------|-----------|------|
|
||||
| internal/router | router.go, intent.go, stage0.go, stagezero.go, heads.go, llmrouter.go, classifier.go, slots.go, source.go, embedder.go, onnxembedder.go, question.go, singletoken.go, notecapture.go, claim.go (package-level), decisiontrace.go | Routing cascade, intent taxonomy, slot extraction, confidence |
|
||||
| internal/claim | claim.go | Band/Claim/arbitration types (imported by router and dialogue) |
|
||||
| internal/dialogue | session.go, clarify.go, pending.go, pending.go | Session state, clarification, PendingAction/Capability |
|
||||
| internal/tool | tool.go, risk.go | Tool execution, risk tiers, policy |
|
||||
| internal/phraser | phraser.go, llmphraser.go, fallbacks.go, replier.go | Response phrasing (LLM or stub) |
|
||||
| internal/voice | server.go, wire.go, replier.go | Voice TCP server, wire protocol |
|
||||
| internal/memory | store.go | Vector memory (cosine similarity search) |
|
||||
| internal/llm | remote.go | Two-tier LLM client (resident + workstation) |
|
||||
| internal/lexicon | lexicon.go | Closed Russian word sets (embedded JSON) |
|
||||
| internal/morph | morph.go | Russian morphology (golem lemmatizer) |
|
||||
| internal/loop | loop.go | Proactive nudge rules, gates |
|
||||
| internal/store | facts.go, reminders.go, tools.go | SQLite persistence |
|
||||
| internal/ipc | coreapi.go, api.go | IPC interface + wire types |
|
||||
| internal/modes | modes.go | Mode inventory (30 routing classes) |
|
||||
| internal/mcp | manager.go, client.go, allowlist.go | MCP tool discovery and execution |
|
||||
| internal/smarthome | client.go | Home Assistant service calls |
|
||||
|
||||
### Key cmd/mavend files (the "glue" layer)
|
||||
|
||||
| File | Responsibility |
|
||||
|------|---------------|
|
||||
| voice.go | runTurn() pipeline, reactiveHandler, replySystem, applyAction |
|
||||
| turnroute.go | turnRoute memo (computed once per turn) |
|
||||
| actions.go | actionHandlers dispatch table (7 intents) |
|
||||
| actions_act.go | actionAct: tool matching, ecosystem interception, execution |
|
||||
| actions_fact.go | actionFact: write fact + embed + vector insert |
|
||||
| actions_reminder.go | actionReminder: parse time + create reminder |
|
||||
| actions_note.go | actionNote: write note + embed + vector insert |
|
||||
| actions_query.go | actionQuery: 20+ source chain, queryTurn, queryWalk |
|
||||
| confirm.go | resolveConfirm: y/n for destructive acts, Hexis, routines |
|
||||
| clarify.go | askClarify, resolveClarifyAnswer, wantedSlots |
|
||||
| followup.go | followUpMerge: slot inheritance across turns |
|
||||
| continuation.go | continuationDecision: elliptical follow-ups |
|
||||
| boot.go | Daemon wiring: connects all pieces |
|
||||
| voicewire.go | Voice server wiring: STT, router, replier, tools, sessions |
|
||||
| tick.go | Proactive tick loop: nudge delivery, reminders, routines |
|
||||
| tick_routines.go | Routine firing, pattern detection |
|
||||
| tick_digest.go | Digest queue and flush |
|
||||
|
||||
---
|
||||
|
||||
## C. Ingress -> Routing -> Action Call Trace
|
||||
|
||||
### Text path (web/telegram)
|
||||
|
||||
```
|
||||
1. mavweb POST /api/chat (cmd/mavweb/chat.go:84)
|
||||
-> strings.TrimSpace(r.FormValue("text"))
|
||||
-> core.Chat(ctx, "web", text) (ipc/coreapi.go:205-213)
|
||||
|
||||
2. daemonAPI.Chat (cmd/mavend/tick_api.go:95)
|
||||
-> chatFn(ctx, conversation, text)
|
||||
-> handler.handleText(ctx, conversation, text) (voice.go:245)
|
||||
|
||||
3. handleText (voice.go:245-248)
|
||||
-> runTurn(withDialogueID(ctx, ...), text, sourceText)
|
||||
|
||||
4. runTurn (voice.go:270-481) -- see Section A for step-by-step
|
||||
```
|
||||
|
||||
### Voice path (mic)
|
||||
|
||||
```
|
||||
1. Voice TCP Server receives PushToTalk frame (internal/voice/server.go:187-215)
|
||||
-> handler.HandlePushToTalk(ctx, req, sid) (cmd/mavend/voice.go:200-218)
|
||||
|
||||
2. HandlePushToTalk:
|
||||
-> stt.Transcribe(ctx, req.Audio) -- whisper.cpp or remote worker
|
||||
-> runTurn(ctx, text, sourceVoice) -- same pipeline as text
|
||||
-> tts.Synthesize(ctx, replyText) -- piper
|
||||
-> return voice.PushToTalkResp{ReplyText, ReplyAudio}
|
||||
```
|
||||
|
||||
### runTurn step-by-step (voice.go:270-481)
|
||||
|
||||
```
|
||||
Step 0 (276): decision.Record installed on context (V-564)
|
||||
Step 0b (292): turnRoute computed once, shared via context
|
||||
Step 1 (313): clarifyExpiredNotice -- parked question TTL ran out
|
||||
Step 2 (318): resolveConfirm -- y/n for parked destructive act
|
||||
-> classifyConfirm(text) (confirm.go:264)
|
||||
-> confirmResolvers chain (confirm.go:114):
|
||||
1. pendingRoutineConfirm
|
||||
2. pendingHexisExec
|
||||
3. pendingAct -> tools.Exec(ctx, fn, args, true)
|
||||
Step 3 (327): resolveRepair -- "нет, это был вопрос"
|
||||
Step 3b (334): resolveUntargetedRepair -- "нет, не так"
|
||||
Step 3c (344): resolveCommandProhibition -- "не отменяй..."
|
||||
Step 4 (356): resolveClarifyAnswer -- answer to parked question
|
||||
Step 5 (366): resolveQuietToggle -- "тихий режим"
|
||||
Step 5b (374): resolveSnooze -- "не сейчас" / "потом"
|
||||
Step 5c (381): resolveAck -- "готово"
|
||||
Step 5d (389): resolveReminderCancellation -- cancel verb + noun
|
||||
Step 5e (397): resolveCandidate -- "второй" (ordinal)
|
||||
Step 6 (406): ROUTE
|
||||
-> turnRoute.resolve(ctx) (turnroute.go:68)
|
||||
-> continuationDecision(prev, text, now) OR router.Route(ctx, text, now)
|
||||
Step 7 (425): followUpMerge(prev, dec, now) -- slot inheritance
|
||||
Step 8 (445): clarify -- missing required slots
|
||||
-> hexisBeforeClarify -- try Hexis before asking
|
||||
-> askClarify -> park PendingQuestion
|
||||
Step 9 (466): applyAction -> actionHandlers[dec.Intent]
|
||||
Step 9b (474): ackFromFact -- close live nudge
|
||||
Step 10 (477): replier -- phrase the reply (LLM or stub)
|
||||
```
|
||||
|
||||
### Action dispatch (voice.go:496-504 -> actions.go:48-56)
|
||||
|
||||
```
|
||||
applyAction(ctx, dec):
|
||||
if dec.Clarify -> return "" (Replier phrases)
|
||||
actionHandlers[dec.Intent](h, ctx, dec):
|
||||
|
||||
IntentFact -> actionFact (actions_fact.go:17)
|
||||
-> coreAPI.WriteFact + memStore.Insert + detectPattern
|
||||
|
||||
IntentReminder -> actionReminder (actions_reminder.go:16)
|
||||
-> coreAPI.CreateReminder
|
||||
|
||||
IntentAct -> actionAct (actions_act.go:17)
|
||||
-> refusesCommand check
|
||||
-> matcher.Match (fuzzy prefix over enabled tool names)
|
||||
-> task_status intercept
|
||||
-> praxis intercept
|
||||
-> hexis intercept (resolve entity -> discover capabilities -> risk -> exec)
|
||||
-> proposeGap (if no match)
|
||||
-> tools.Exec (tool/tool.go:156):
|
||||
LookupTool -> RiskOf -> PolicyFor(tier) -> dispatch:
|
||||
MCP -> mcp.CallPositional
|
||||
HA -> smarthome.CallService
|
||||
Process -> exec.CommandContext
|
||||
|
||||
IntentChat -> actionChat (actions.go:58)
|
||||
-> phraser.PhraseChat(ctx, utterance, history)
|
||||
|
||||
IntentSystem -> actionSystem (actions.go:77)
|
||||
-> replySystem: keyword match on utterance
|
||||
|
||||
IntentNote -> actionNote (actions_note.go:24)
|
||||
-> coreAPI.WriteNote + memStore.Insert
|
||||
|
||||
IntentQuery -> actionQuery (actions_query.go:varies)
|
||||
-> querySources chain (20+ sources, first claim wins)
|
||||
-> queryWalk narrows by destination
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## D. Inventory of Existing Machinery
|
||||
|
||||
### D1. Deterministic fast-path recognizers
|
||||
|
||||
**Status: EXISTS, extensive, production-critical**
|
||||
|
||||
| Component | File:line | What it does |
|
||||
|-----------|-----------|-------------|
|
||||
| 22 stage-0 grammars | router/stagezero.go:24-91 | Ordered regex/structural rules. First match wins at confidence 1.0 |
|
||||
| Grammar type | router/stage0.go:20-29 | Pattern+Build (regex) or Decide (structural) |
|
||||
| Wake-word strip | router/stage0.go:66-78 | StripWakeToken: removes "мавен" in any script |
|
||||
| Act allowlist fast path | router/stage0.go:55,80 | "мавен, restart nginx" -> act at stage 0 |
|
||||
| Command prohibition | stagezero.go:30 | "don't restart nginx" -> refusal sentinel |
|
||||
| System time/date | stagezero.go:35 | "сколько времени", "который час" |
|
||||
| Agenda query | stagezero.go:39 | "что у меня сегодня" |
|
||||
| Reminder | stagezero.go:59 | "напомни через час" with time extraction |
|
||||
| Fact capture | stagezero.go:80 | "запиши купить молоко" with note body extraction |
|
||||
| Possession statement | stagezero.go:89 | "у меня кончилась вода" |
|
||||
| Lexicon (closed word sets) | lexicon/lexicon.go | ~40 embedded Russian word sets |
|
||||
| Morphology | morph/morph.go | Lemma(), IsVerbForm(), SameWord() via golem |
|
||||
| Question detection | router/question.go | IsQuestionShaped(), IsOpenQuestionShaped(), CarriesCaptureVerb() |
|
||||
| Single-token analysis | router/singletoken.go | thinSingleToken() with completeSingles escape |
|
||||
| Note capture parser | router/notecapture.go | ParseNoteCapture(): strips capture frame |
|
||||
| DateTimeParser | router/slots.go:19-21 | Interface; production is dateparser (not shown) |
|
||||
| DefaultFactParser | router/slots.go:154-180 | Lemma-based fact extraction (water/meal/shower/break/sleep) |
|
||||
| DefaultActMatcher | router/slots.go:106-149 | Exact phrase prefix + aliases, longest-first |
|
||||
|
||||
### D2. Learned routing/NLU
|
||||
|
||||
**Status: EXISTS, multi-layered**
|
||||
|
||||
| Component | File:line | What it does |
|
||||
|-----------|-----------|-------------|
|
||||
| RouterHeads (ONNX) | router/heads.go:55-64 | 4 heads over e5-small: intent, destination, slot BIO, clarify |
|
||||
| RouterHeads.Route | router/heads.go:120+ | Softmax classification, threshold=0.6, single forward pass |
|
||||
| LLMRouter | router/llmrouter.go:20-22 | GBNF-constrained JSON from Qwen3-1.7B |
|
||||
| LLMRouter.Route | router/llmrouter.go:120+ | System prompt + grammar -> routeAction structs |
|
||||
| gateLLMDecision | router/router.go:352-373 | Thin confidence (0.3) for incomplete slots |
|
||||
| Classifier | router/classifier.go:40-45 | Nearest-centroid over ONNX embeddings |
|
||||
| Classifier.Classify | router/classifier.go:50+ | Cosine similarity, sorted best-first |
|
||||
| Classifier.AddExample | router/classifier.go | Append-only correction (grows classifier) |
|
||||
| ONNX Embedder | router/onnxembedder.go | multilingual-e5-small, 384-dim, query/passage prefix |
|
||||
| HashEmbedder | router/embedder.go:79-116 | Fallback bag-of-words (deterministic, weak) |
|
||||
| Embedder interface | router/embedder.go:17-21 | Dim(), Embed(), Close() |
|
||||
| Intent taxonomy | router/intent.go:38-63 | 7 intents: act, reminder, fact, note, query, chat, system |
|
||||
| Source taxonomy | router/source.go | 12 destinations: recall, calendar, tasks, list, money, weather, home, network, feeds, attention, self, world |
|
||||
| Slots struct | router/intent.go:66-94 | Time, Fn, Args, Key, Value, Text + Has* flags |
|
||||
| Decision struct | router/intent.go:100-128 | Utterance, Stage, Intent, Confidence, Slots, Clarify, Source, SourceAnchored |
|
||||
| Modes inventory | modes/modes.go | ~30 distinct downstream behaviors, embedded JSON |
|
||||
| Confidence gate | router/router.go:244-248 | threshold=0.55, below -> Clarify=true |
|
||||
|
||||
### D3. Claim/arbitration system
|
||||
|
||||
**Status: EXISTS, not wired into cascade (per claim.go:138-140)**
|
||||
|
||||
| Component | File:line | What it does |
|
||||
|-----------|-----------|-------------|
|
||||
| Band enum | claim/claim.go:38-69 | BandUnknown, BandVetoed, BandNearest, BandStructural, BandAnchored |
|
||||
| Claim struct | claim/claim.go:88-117 | Claimant, Intent, Filled, Consumed, Unexplained, Band, Veto |
|
||||
| Coverage() | claim/claim.go:122-128 | Consumed / (Consumed + Unexplained) |
|
||||
| MoreSpecificThan() | claim/claim.go:141-147 | Coverage first, Band breaks ties |
|
||||
| Tokens() | claim/claim.go:156-169 | Tokenize utterance for coverage |
|
||||
| Split() | claim/claim.go:178-192 | Partition tokens into consumed/unexplained |
|
||||
|
||||
Note: claim.go:138-140 explicitly states this is "Deliberately NOT wired into the cascade by V-565." It is here so the ordering is one function with tests rather than duplicated logic.
|
||||
|
||||
### D4. Action path
|
||||
|
||||
**Status: EXISTS, with multiple dispatch paths**
|
||||
|
||||
| Component | File:line | What it does |
|
||||
|-----------|-----------|-------------|
|
||||
| actionHandlers table | actions.go:48-56 | 7-intent dispatch map |
|
||||
| applyAction | voice.go:496-504 | Short-circuits on Clarify, dispatches via table |
|
||||
| actionAct | actions_act.go:17-114 | Full act cascade: refuse -> match -> task_status -> praxis -> hexis -> propose -> exec |
|
||||
| tool.Executor.Exec | tool/tool.go:156-235 | LookupTool -> RiskOf -> PolicyFor -> dispatch (MCP/HA/process) |
|
||||
| RiskOf | tool/risk.go | Derives tier from Tool row |
|
||||
| PolicyFor | tool/risk.go | TierSafe/TierDestructive/TierIrreversible -> Confirm/VoiceMayRun |
|
||||
| Matcher.Match | tool/tool.go:251+ (tool package) | Fuzzy prefix over enabled tool names |
|
||||
| MCP dispatch | tool/tool.go:191-198 | mcp.ParseCmd -> mcp.CallPositional |
|
||||
| HA dispatch | tool/tool.go:205-221 | smarthome.ParseCmd -> smarthome.CallService |
|
||||
| Process dispatch | tool/tool.go:231-234 | exec.CommandContext (no shell) |
|
||||
| Hexis integration | ecosystem_acts.go:660+ | resolve entity -> discover capabilities -> risk -> confirm/exec |
|
||||
| Praxis integration | ecosystem_acts.go (handlePraxisAct) | Attention/item lifecycle |
|
||||
| UnknownTargetError | tool/tool.go:90-96 | Named error with target word |
|
||||
| Tool store | store/tools.go | ProposeTool, EnableTool, DisableTool, LookupTool, ReconcileMCPTool |
|
||||
|
||||
### D5. Confirmation/risk handling
|
||||
|
||||
**Status: EXISTS, comprehensive**
|
||||
|
||||
| Component | File:line | What it does |
|
||||
|-----------|-----------|-------------|
|
||||
| resolveConfirm | confirm.go:78-100 | classifyConfirm + chain of resolvers |
|
||||
| classifyConfirm | confirm.go:264+ | Closed yes/no lexicon, entire utterance must match |
|
||||
| confirmResolvers | confirm.go:114-186 | 3 slots: routine proposal, Hexis exec, local tool |
|
||||
| confirmTTL | confirm.go:59 | 90s |
|
||||
| pendingAct | confirm.go:48-54 | fn, args, phrase, expiry |
|
||||
| pendingHexisExec | confirm.go:21-35 | capabilityID, entityID, correlationID, expiry |
|
||||
| pendingRoutineConfirm | confirm.go:39-46 | routineID, action, object, interval, phrase, expiry |
|
||||
| park() | confirm.go:63-67 | Stores pendingAct |
|
||||
|
||||
### D6. Non-action paths
|
||||
|
||||
| Component | File:line | What it does |
|
||||
|-----------|-----------|-------------|
|
||||
| actionChat | actions.go:58-75 | phraser.PhraseChat with dialogue history |
|
||||
| replySystem | voice.go:509-569 | Keyword matching on utterance for time/date/status |
|
||||
| actionQuery chain | actions_query.go:99-188 | 20+ sources, first claim wins |
|
||||
| queryWalk | actions_query.go:190+ | Narrows chain by destination |
|
||||
| Best recall | recall.go:8-65 | Vector search with confidence gate (minScore + minMargin) |
|
||||
| Clarify store | dialogue/clarify.go | PendingQuestion stack (max depth 2), 90s TTL |
|
||||
| Follow-up merge | followup.go:99-194 | Slot inheritance across same-intent turns |
|
||||
| Continuation | continuation.go:57-98 | Elliptical follow-ups ("а завтра?") |
|
||||
| Tick loop | tick.go | Proactive nudge delivery, reminders, routines, patterns, digest |
|
||||
|
||||
### D7. Tests/data inventory
|
||||
|
||||
**Status: Extensive (354 test files)**
|
||||
|
||||
Key test fixtures:
|
||||
|
||||
| Fixture | File | Cases |
|
||||
|---------|------|-------|
|
||||
| Routing contract | internal/router/eval/ru_routing_v1.json | ~70 held-out utterances with intent/source/time/clarity expectations |
|
||||
| Ecosystem reach | internal/router/eval/ru_ecosystem_v1.json | ~50 act utterances with service/capability expectations |
|
||||
| Nudge phrasing | internal/phraser/eval/nudges_v1.json | 15 cases, on-topic + property checks |
|
||||
| Talk phrasing | internal/phraser/eval/talk_v1.json | 27 cases (chat/query/knowledge) |
|
||||
| Recall contract | internal/memory/recalleval/ru_recall_v1.json | ~40 cases with note sets |
|
||||
| Personal boundary | cmd/mavend/testdata/personal_boundary_v1.json | 72 cases |
|
||||
| Safety scenarios | cmd/mavend/testdata/system_safety_scenarios.json | 4 scenarios |
|
||||
| Simulator scenarios | cmd/mavend/testdata/scenarios/*.json | 5 full scripted scenarios |
|
||||
| Usage transcript | scripts/testdata/usage-turns.txt | 167-line simulated usage |
|
||||
| STT golden | cmd/mavsttd/testdata/golden_v1.json | 4 cases with WER bounds |
|
||||
|
||||
Key eval harnesses:
|
||||
|
||||
| Harness | File |
|
||||
|---------|------|
|
||||
| Routing eval | internal/router/eval/eval.go |
|
||||
| Heads eval | internal/router/eval/heads_test.go |
|
||||
| LLM router eval | internal/router/eval/llmrouter_test.go |
|
||||
| Ecosystem reach eval | internal/router/eval/reach_test.go |
|
||||
| Claim scoring | internal/router/eval/claims_test.go |
|
||||
| Nudge phrasing eval | internal/phraser/eval/eval.go |
|
||||
| Talk phrasing eval | internal/phraser/eval/talk.go |
|
||||
| Recall eval | internal/memory/recalleval/recalleval.go |
|
||||
| Kiwix rewrite eval | internal/kiwix/rewrite_eval.go |
|
||||
| Simulator | cmd/mavend/simulator_test.go |
|
||||
|
||||
Classifier seed phrases: models/seeds/{query,fact,chat,act,note,reminder,system}.txt
|
||||
|
||||
---
|
||||
|
||||
## E. Existing Contracts/Types We Can Reuse
|
||||
|
||||
### Already well-typed (reuse as-is or thin wrapper)
|
||||
|
||||
| Type | File:line | Notes |
|
||||
|------|-----------|-------|
|
||||
| `router.Intent` | intent.go:38 | String enum: act, reminder, fact, note, query, chat, system |
|
||||
| `router.Slots` | intent.go:66-94 | Typed: Time, Fn, Args, Key, Value, Text + Has* flags |
|
||||
| `router.Decision` | intent.go:100-128 | Utterance, Stage, Intent, Confidence, Slots, Clarify, Source, SourceAnchored |
|
||||
| `router.Source` | source.go | String enum: 12 destinations |
|
||||
| `router.Grammar` | stage0.go:20-29 | Pattern+Build or Decide |
|
||||
| `router.Extractor` | slots.go:45-49 | Time, Acts, Facts parsers |
|
||||
| `claim.Claim` | claim.go:88-117 | Band-based arbitration (not wired yet) |
|
||||
| `claim.Band` | claim.go:36-69 | Ordinal evidence kinds |
|
||||
| `dialogue.PendingAction` | pending.go:58 | Capability, slots, missing, utterance, TTL |
|
||||
| `dialogue.Capability` | pending.go:16-46 | 7 capability strings |
|
||||
| `tool.Executor` | tool/tool.go:116-122 | Exec with policy checks |
|
||||
| `tool.Policy` | tool/risk.go | Confirm, VoiceMayRun per tier |
|
||||
| `ipc.ChatReply` | ipc/api.go:752-761 | Reply, Source, TraceID |
|
||||
| `ipc.Tool` | ipc/api.go:699 | Wire shape of tool row |
|
||||
|
||||
### Partially exists (needs extension)
|
||||
|
||||
| Concept | Current form | Gap |
|
||||
|---------|-------------|-----|
|
||||
| NormalizedInput | Raw string in, `text string` parameter | No NormalizedInput struct; STT output passes as-is |
|
||||
| FastPathResult | Stage-0 grammar Decision | Not a separate type; embedded in Decision |
|
||||
| RouteDecision | `router.Decision` | Already carries Stage (0/1/2/3), Intent, Confidence, Slots. Could become RouteDecision |
|
||||
| ActionCandidate | `router.Decision` + `actionHandlers` dispatch | No explicit ActionCandidate type; intent + slots + handler selection are implicit |
|
||||
|
||||
### Missing (must be created if needed)
|
||||
|
||||
| Concept | Notes |
|
||||
|---------|-------|
|
||||
| Schema validation | No JSON-schema or struct validation on incoming slots before execution |
|
||||
| Confidence/risk policy | Risk tiers exist for tools but not for routing confidence. Stage 3 gate exists but is a simple threshold |
|
||||
| Post-execution verification | No explicit verification step after tool execution (success/failure is the extent) |
|
||||
|
||||
---
|
||||
|
||||
## F. Gaps Against the Proposed First-Stage Design
|
||||
|
||||
### Proposed pipeline vs current reality
|
||||
|
||||
| Proposed stage | Current state | Gap |
|
||||
|---------------|---------------|-----|
|
||||
| **NormalizedInput** | Raw `text string` everywhere | No normalization struct. Preprocessing is scattered: StripWakeToken in router, lowercase in matchers, trim in entry points |
|
||||
| **Deterministic fast-path** | Stage-0 grammars (22 rules) | EXISTS and is production-critical. However: grammars produce `Decision` directly, not a separate FastPathResult type. No schema validation on slots before returning |
|
||||
| **Learned routing** | Heads -> LLM -> Classifier cascade | EXISTS and complex. But: multiple confidence scales (stage-0=1.0, heads=softmax, LLM=1.0/0.3, classifier=cosine). No unified confidence model |
|
||||
| **RouteDecision** | `router.Decision` | EXISTS under another name. Carries Stage, Intent, Confidence, Slots, Clarify, Source. Could be wrapped/renamed |
|
||||
| **ActionCandidate** | Implicit in `router.Decision` + `actionHandlers` | Missing as explicit type. The decision arrives at applyAction and is dispatched by intent. No schema validation of slots before dispatch |
|
||||
| **Schema validation** | NONE | Slots are filled by extractors and used directly. No validation that e.g. reminder has both Text and Time before actionReminder runs |
|
||||
| **Confidence/risk policy** | Threshold gate (0.55) for routing; risk tiers for tools | No unified confidence policy. Routing confidence and tool risk are separate systems. No policy that says "if confidence < X, require confirmation for action" |
|
||||
| **Confirmation** | resolveConfirm with 3 pending slots | EXISTS for destructive tools, Hexis, and routines. Not applied to routing confidence (a low-confidence act just gets proposed, not confirmed) |
|
||||
| **Execution** | tool.Executor.Exec | EXISTS with MCP/HA/process dispatch. Well-structured. But: actionFact, actionReminder, actionNote bypass tool.Executor entirely (they call CoreAPI directly) |
|
||||
| **Post-execution verification** | Success/failure error handling in actionAct | Partially EXISTS. Tool execution returns (out, err). Error types are handled specifically. But: no structured verification step, no retry policy, no rollback |
|
||||
|
||||
### Architectural problems identified
|
||||
|
||||
1. **Fast paths that execute directly**: actionFact, actionReminder, actionNote call CoreAPI.WriteFact/CreateReminder/WriteNote directly from the action handler, bypassing tool.Executor. This means they skip the risk tier system, the confirm gate, and the allowlist. This is by design (facts/reminders are user-stated, not tool invocations) but means the "all actions through one pipeline" goal requires either wrapping these in tool-like abstractions or explicitly exempting them.
|
||||
|
||||
2. **Multiple confidence scales**: Stage-0 = 1.0 (hardcode), heads = softmax float, LLM = 1.0 or 0.3 (thin), classifier = cosine similarity. These are not on the same scale and cannot be compared. The claim system (claim.Band) explicitly addresses this by making confidence ordinal (Band) rather than graded. The proposed design should preserve this insight.
|
||||
|
||||
3. **Routing code that also performs tool selection**: actionAct at actions_act.go:29-33 runs the matcher inline when HasFn is false. The matcher is also the stage-2 ActMatcher. So tool selection happens both in the router (stage 2) and in the action handler (actionAct). The actionAct path is the fallback for LLM-routed acts where the verb didn't go through stage-0.
|
||||
|
||||
4. **Implicit fallthrough**: The router cascade is explicitly designed as fallthrough (each stage may decline). The query source chain is also fallthrough (first claim wins). The confirm resolver chain is also fallthrough. This is a consistent pattern, not a bug, but means the "stage-to-stage" architecture must preserve explicit decline semantics.
|
||||
|
||||
5. **Clarify bypasses action pipeline**: When dec.Clarify is true, applyAction returns "" immediately (voice.go:497-499). The clarifier can also call hexisBeforeClarify (voice.go:446) to try Hexis before asking, which is a hidden action path that runs before the normal action dispatch.
|
||||
|
||||
6. **No schema validation**: Slots filled by stage-2 extraction or stage-0 grammars are used directly by action handlers. actionReminder (actions_reminder.go) checks HasTime itself. actionFact checks HasKey. But there is no shared validation layer; each handler does its own checks.
|
||||
|
||||
7. **Voice-specific behavior**: HandlePushToTalk wraps runTurn with STT before and TTS after. The turnSource tag ("tap:voice" vs "tap:text") propagates into fact sources but does not change routing or action behavior. However: the voice path has barge-in, session management, and wake-word detection that the text path lacks entirely. The semantic behavior is the same; the infrastructure is different.
|
||||
|
||||
8. **Existing useful code to preserve**:
|
||||
- Stage-0 grammars: 22 ordered rules, battle-tested, each with extensive comments about why it sits where it sits. Moving or reordering them breaks routing.
|
||||
- The claim/Band system: explicitly designed for the problem of incomparable confidence scales. Not wired yet but well-tested.
|
||||
- The clarify store with stack support: handles nested clarification flows.
|
||||
- The query source chain with destination narrowing: 20+ sources with guessers vs lookups distinction.
|
||||
- The tool risk tier system: well-tested, with voice-specific authority limits.
|
||||
- The turnRoute memo pattern (V-560): computed once, shared via context, prevents routing disagreement.
|
||||
|
||||
---
|
||||
|
||||
## G. Smallest Behavior-Preserving Refactor Boundary
|
||||
|
||||
The smallest refactor that aligns with the proposed architecture without changing behavior:
|
||||
|
||||
**Wrap Decision in RouteDecision + add NormalizedInput as thin alias**
|
||||
|
||||
```
|
||||
Current: router.Route(ctx, utterance, now) -> (Decision, error)
|
||||
Proposed: router.Route(ctx, NormalizedInput, now) -> (RouteDecision, error)
|
||||
```
|
||||
|
||||
Where:
|
||||
- `NormalizedInput` is `type NormalizedInput struct { Text string; Source string }` -- a thin wrapper, not a transformation
|
||||
- `RouteDecision` is `type RouteDecision Decision` -- or just `Decision` with a type alias
|
||||
- The existing Stage field (0/1/2/3) already encodes which stage produced the result
|
||||
- The existing Confidence field already carries the per-stage confidence
|
||||
|
||||
This changes zero behavior. It names what exists. It creates the typed boundary the future stages need.
|
||||
|
||||
**Second step: extract action candidates**
|
||||
|
||||
Currently actionAct does tool matching inline. The matcher result (fn, args) should be a typed ActionCandidate returned by the router or by a post-route step, not discovered inside the action handler. But this changes the call structure of actionAct, which is a larger refactor.
|
||||
|
||||
**Third step: schema validation**
|
||||
|
||||
Add a Validate(slots) step between routing and action dispatch. Currently each handler validates its own slots; this would centralize it. Minimal behavior change: the same checks, in one place.
|
||||
|
||||
---
|
||||
|
||||
## H. Recommended Implementation Order
|
||||
|
||||
1. **Type the boundaries** (1-2 hours)
|
||||
- Define NormalizedInput, RouteDecision as thin wrappers
|
||||
- Route() signature change (internal callers only)
|
||||
- Zero behavior change
|
||||
|
||||
2. **Pin current behavior with regression tests** (2-3 hours)
|
||||
- Run existing eval fixtures and record baselines
|
||||
- Add integration tests for the full runTurn pipeline (text + voice paths)
|
||||
- Add tests for each action handler with representative inputs
|
||||
|
||||
3. **Extract ActionCandidate from actionAct** (3-4 hours)
|
||||
- Move tool matching out of actionAct into a post-route step
|
||||
- Return ActionCandidate{Fn, Args, Source} from routing
|
||||
- actionAct consumes ActionCandidate instead of re-matching
|
||||
|
||||
4. **Add schema validation layer** (2-3 hours)
|
||||
- Validate slots before action dispatch
|
||||
- Centralize the per-handler checks
|
||||
- Fail-closed: missing required slot -> clarify, not runtime error
|
||||
|
||||
5. **Unify confidence presentation** (3-4 hours)
|
||||
- Map per-stage confidence to ordinal Band (leverage existing claim.Band)
|
||||
- Expose in RouteDecision for downstream policy
|
||||
- Do NOT try to make confidence comparable across stages
|
||||
|
||||
6. **Wire claim.Band into cascade** (4-6 hours)
|
||||
- Replace ad-hoc precedence with MoreSpecificThan
|
||||
- This is the V-558/V-565 work already planned
|
||||
|
||||
---
|
||||
|
||||
## I. Tests That Should Pin Current Behavior Before Refactoring
|
||||
|
||||
### High-value regression pins
|
||||
|
||||
| Test | What it pins | File |
|
||||
|------|-------------|------|
|
||||
| Router cascade stage ordering | Stage-0 wins, heads decline correctly, LLM fallback, classifier floor | internal/router/router_test.go |
|
||||
| Held-out routing contract | ~70 utterances with intent/source/time expectations | internal/router/eval/eval_test.go |
|
||||
| Ecosystem reach contract | ~50 act utterances routing to correct service | internal/router/eval/reach_test.go |
|
||||
| Simulator scripted day | Full pipeline: STT -> router -> store -> phraser | cmd/mavend/simulator_test.go |
|
||||
| Safety scenarios | Destructive acts require confirmation, ambiguous entities clarified | cmd/mavend/eval_scenarios_test.go |
|
||||
| Tool risk assessment | Tier derivation from tool rows | internal/tool/ risk_test.go (implied) |
|
||||
| Confirm flow | y/n for parked acts, TTL expiry, chain ordering | cmd/mavend/confirm_test.go |
|
||||
| Clarify flow | Missing slots -> question -> answer -> continue | cmd/mavend/clarify_test.go |
|
||||
| Follow-up merge | Slot inheritance across turns | cmd/mavend/followup_test.go |
|
||||
| Query source chain | First-claim-wins, destination narrowing | cmd/mavend/querywalk_test.go |
|
||||
| Personal boundary | 72-case held-out fixture | cmd/mavend/personalboundary_test.go |
|
||||
| Recall contract | ~40 cases with paraphrased queries | internal/memory/recalleval/recalleval_test.go |
|
||||
| Action act risk | Destructive/irreversible classification | cmd/mavend/actions_act_risk_test.go |
|
||||
| Degradation | Each ecosystem service unreachable | cmd/mavend/ecosystem_degraded_test.go |
|
||||
|
||||
### What to run before and after each refactor step
|
||||
|
||||
```sh
|
||||
make test # full suite
|
||||
go test ./internal/router/eval/ -run Eval # routing contract
|
||||
go test ./cmd/mavend/ -run Simulator # integration
|
||||
go test ./cmd/mavend/ -run Eval # safety scenarios
|
||||
go test ./cmd/mavend/ -run PersonalBoundary # boundary fixture
|
||||
go test ./internal/phraser/eval/ -run Eval # phrasing contract
|
||||
go test ./internal/memory/recalleval/ -run Eval # recall contract
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## J. Unknowns That Cannot Be Established From Code/Tests
|
||||
|
||||
1. **Actual production accuracy numbers**: The eval fixtures measure held-out accuracy, but production routing traces (routing_traces table) are the real measure. We cannot inspect the production DB from code.
|
||||
|
||||
2. **Whether the LLM router is currently enabled in production**: The config shows `llm_router` settings but we cannot confirm the daemon is running with it wired. The heads may be the actual fast path.
|
||||
|
||||
3. **Real-world confirmation rates**: How often do users get asked to confirm? How often do they decline? This is behavioral data, not code.
|
||||
|
||||
4. **Whether the claim system should be wired**: claim.go says "Deliberately NOT wired by V-565" but the current ad-hoc precedence works. The claim system is tested but untested in production.
|
||||
|
||||
5. **Token budget pressure on the resident model**: The 4096 context window is shared between routing, phraser, and chat. We cannot tell from code whether context pressure causes routing failures in production.
|
||||
|
||||
6. **Whether stage-0 grammars overlap or shadow each other**: The ordering is documented, but no test measures "if grammar A were removed, which utterances would fall through differently." The cascade hides contention by design.
|
||||
|
||||
7. **Performance characteristics of the ONNX embedder in production**: Tests measure p50 (20.6ms for classifier). Production numbers on the actual hardware may differ.
|
||||
|
||||
8. **Whether the ecology of pre-route resolvers (steps 1-5e) can be unified**: Seven resolvers each claim the turn independently, in order. Whether they could be replaced by a single arbiter (the claim system) is a design question, not a code question.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Mapping: Future Stage/Contract -> Current Implementation
|
||||
|
||||
| Future stage/contract | Current implementation | Reuse/wrap/move/replace | Reason |
|
||||
|----------------------|----------------------|------------------------|--------|
|
||||
| **NormalizedInput** | Raw `text string` parameter in handleText/HandlePushToTalk/runTurn | **wrap** | Create struct, pass through. No transformation needed yet. Existing preprocessing (StripWakeToken, trim) stays inside the router. |
|
||||
| **Deterministic fast path** | Stage-0 grammars (router/stagezero.go, stage0.go) | **reuse as-is** | 22 battle-tested rules with load-bearing ordering. Output is Decision at confidence 1.0. Naming it "fast path" is cosmetic. |
|
||||
| **FastPathResult** | Decision with Stage=0 | **wrap** | Type alias or thin struct. The Stage field already identifies the source. |
|
||||
| **RouteDecision** | `router.Decision` (intent.go:100-128) | **reuse (rename or alias)** | Already carries all needed fields: Intent, Confidence, Slots, Clarify, Source, Stage. The Stage field (0/1/2/3) tells which cascade stage produced it. |
|
||||
| **ActionCandidate** | Implicit: Decision.Intent + Decision.Slots + actionHandlers dispatch | **move** | Extract from actionAct into a post-route step. Currently, actionAct:29-33 re-runs the matcher when HasFn is false. This should produce an ActionCandidate that actionAct consumes. |
|
||||
| **Schema validation** | Per-handler checks (actionReminder checks HasTime, actionFact checks HasKey) | **move + centralize** | Currently scattered across action handlers. Centralize into a Validate(Decision) step before applyAction. |
|
||||
| **Confidence/risk policy** | Stage-3 gate (threshold 0.55) for routing; RiskOf/PolicyFor for tools | **extend** | These are separate systems today. The routing gate Clarify flag. The tool policy returns ErrNeedsConfirm/ErrNeedsAuthedSurface. A unified policy would map confidence bands to action policies. |
|
||||
| **Confirmation** | resolveConfirm (confirm.go:78) with 3 pending slots | **reuse** | Already handles destructive tools, Hexis, and routines. Would need extension if low-confidence acts should also confirm. |
|
||||
| **Execution** | tool.Executor.Exec (tool/tool.go:156) + per-intent handlers | **reuse** | Well-structured with MCP/HA/process dispatch. The per-intent handlers (actionFact, actionReminder) bypass Executor by design -- they write to the store, not run tools. |
|
||||
| **Post-execution verification** | Error handling in actionAct (actions_act.go:64-108) | **extend** | Currently: success -> "done", specific error -> specific reply. No structured verification step. Adding one would be a new layer. |
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# docs/capabilities/
|
||||
|
||||
Generated. Regenerated from `docs/spec.md` plus a named eval. Not hand-edited.
|
||||
|
||||
`docs/spec.md` says what Maven should do. This directory says how much of that
|
||||
exists, measured rather than asserted. The predecessor audit had a green test
|
||||
suite while 22 of 39 capabilities were not live, which is the failure mode the
|
||||
whole directory is built against.
|
||||
|
||||
## The files
|
||||
|
||||
| file | what it is | hand-edited |
|
||||
| --- | --- | --- |
|
||||
| `ledger.yaml` | the ledger: 51 capabilities, 156 DoD criteria, one verdict per criterion, seven implementation dimensions per capability | no |
|
||||
| `build_ledger.py` | extracts the ledger from `docs/spec.md` and joins the two inputs | yes, it is the source |
|
||||
| `domains.yaml` | the domain axis, one of the two judgment calls in the extraction | yes |
|
||||
| `implementation.yaml` | capability to component mapping, the other judgment call | yes |
|
||||
| `verdicts.json` | one verdict per criterion id, produced by scoring a probe run | no, scored |
|
||||
| `probes_field.json` | 25 multi-turn probes: the owner's real week | yes |
|
||||
| `probes_dod.json` | probes derived from the ledger's criteria | no, generated |
|
||||
| `run_probes.py` | drives a probe file through the deployed stack | yes |
|
||||
| `store_counts.py` | row counts per store, over IPC | yes |
|
||||
| `invariants.md` | the twelve cross-cutting rules the 51 capabilities imply, with the prose and the evidence | yes |
|
||||
| `invariants.yaml` | the machine-readable half of the same twelve: mark, capabilities, components | yes |
|
||||
| `gaps.md` | eight gap classes and the one ranked priority list | yes, except classes 1-4 |
|
||||
| `out/` | raw probe output, one JSON object per line | no |
|
||||
|
||||
## Rebuilding
|
||||
|
||||
```sh
|
||||
python3 docs/capabilities/build_ledger.py # spec.md + domains.yaml + implementation.yaml + verdicts.json + maven-architecture.json -> ledger.yaml
|
||||
```
|
||||
|
||||
The generator is also the checker. It fails, loudly and non-zero, on a
|
||||
capability with no DoD criteria, a capability with no `State` line, a criterion
|
||||
id collision, a capability with no domain or more than two, an unknown domain
|
||||
name, a `domains.yaml` row naming a capability that does not exist, a
|
||||
`verdicts.json` row scoring a criterion that does not exist, a verdict word
|
||||
outside the five, and a reason outside the plan's list. It caught the domain
|
||||
reconciler silently dropping `recall` from its 51.
|
||||
|
||||
It fails on an invariant whose `## N. Title` heading is absent from
|
||||
`invariants.md`, on a count mismatch between the two files, on an unknown
|
||||
capability or component in `invariants.yaml`, and on an `unresolved` invariant
|
||||
carrying no product question. The two files exist separately so the viewer can
|
||||
read one and a person can read the other, and they drift the moment nothing
|
||||
checks them.
|
||||
|
||||
It also fails on a capability missing from `implementation.yaml`, a component id
|
||||
that `docs/architecture/maven-architecture.json` does not carry, and a component
|
||||
status the dimension table does not know. A capability absent from the mapping
|
||||
would read `no` on every dimension, which is indistinguishable from a capability
|
||||
nothing carries.
|
||||
|
||||
## The seven dimensions
|
||||
|
||||
Never one `implemented` boolean. Coded and unwired, wired and unconfigured, and
|
||||
configured and undeployed are three different pieces of work.
|
||||
|
||||
`designed`, `code_present`, `wired`, `configured`, `deployed` and `reachable`
|
||||
come from the `status` field of every component mapped to the capability, rolled
|
||||
up as all yes, none no, otherwise partial. `verified` comes from the criteria
|
||||
verdicts: yes when every one passes, partial when some do.
|
||||
|
||||
`designed` answers a narrower question, because `docs/spec.md` states all 51 of
|
||||
them. It reads `yes` when a living doc owns the subsystem, and `spec-only` when
|
||||
the State line says no living doc or no package.
|
||||
|
||||
A component the mapping does not use is reported by name at the end of a build.
|
||||
Shared infrastructure is excluded on purpose: mapping `core.reactive_handler` to
|
||||
everything would give all 51 rows the same status and say nothing.
|
||||
|
||||
## Running the probes
|
||||
|
||||
The probes run **on homesrv**, where mavweb is on `127.0.0.1:9201` and the
|
||||
mavend socket is reachable from inside the container.
|
||||
|
||||
```sh
|
||||
# Build the probe binary. It needs CGO and the target's glibc, so build it in a
|
||||
# trixie container: both the golang image and the mavend image are trixie.
|
||||
docker run --rm -v "$PWD":/src -w /src \
|
||||
-e CGO_ENABLED=1 -e GOFLAGS=-mod=vendor \
|
||||
-e GOCACHE=/src/.cache/gocache -e GOPATH=/src/.cache/gopath \
|
||||
golang:1.25-trixie go build -buildvcs=false -o /src/.cache/e2eprobe ./cmd/e2eprobe
|
||||
docker cp .cache/e2eprobe maven-mavend-1:/tmp/e2eprobe
|
||||
|
||||
python3 docs/capabilities/store_counts.py # before
|
||||
python3 docs/capabilities/run_probes.py probes_field.json > out/field.raw.jsonl
|
||||
python3 docs/capabilities/store_counts.py # after
|
||||
```
|
||||
|
||||
## Two things the harness learned the hard way
|
||||
|
||||
**Probes must be isolated.** `mavweb` hardcodes one conversation id for the
|
||||
whole web reach, so a clarify parked by one probe is still parked for the next.
|
||||
The first run measured the previous probe, not the current one: the park set at
|
||||
turn 8 appended `Сейчас 01:07. В какой день?` to turns 9 through 13, five
|
||||
unrelated turns in a row, including plain statements. `run_probes.py` now sends `отмена` before every
|
||||
probe. The contaminated run is kept at `out/field.contaminated.jsonl`, because
|
||||
the leak is a finding and not only an artifact.
|
||||
|
||||
**Readback is the contract, not the file.** The plaintext database copy at
|
||||
`/dev/shm/maven-plain.db` would answer every question faster and would bypass
|
||||
the IPC contract the ledger exists to measure. The mavweb pages are a
|
||||
second-hand rendering of the same thing.
|
||||
|
||||
## What a verdict means
|
||||
|
||||
`pass` comes only from `live` evidence: the deployed build, the real model, real
|
||||
store rows. The scenario harness scripts both `route` and `reply`, so a green
|
||||
scenario proves the wiring around the model and not the turn; it is recorded as
|
||||
implementation evidence and reads `untested`. `simulated` is allowed only where
|
||||
the trigger is anchored to a wall-clock hour or date a probe cannot reach.
|
||||
|
||||
## The viewer
|
||||
|
||||
`docs/architecture/index.html` views 6 and 7 read this directory.
|
||||
`build_viewer.py` inlines `ledger.yaml` and `invariants.yaml` and derives
|
||||
nothing: the ledger's build is the only thing allowed to decide a dimension, and
|
||||
a second derivation would drift from it silently.
|
||||
|
||||
```sh
|
||||
python3 docs/architecture/build_viewer.py
|
||||
node docs/architecture/check_viewer.js
|
||||
```
|
||||
@@ -0,0 +1,592 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract the target side of the capability ledger from docs/spec.md.
|
||||
|
||||
Mechanical. No implementation judgment, no verification status, no ranking.
|
||||
Domain assignment is the one human input and lives in domains.yaml, keyed by
|
||||
capability id; this script only joins it and fails loudly on a mismatch.
|
||||
|
||||
Run from the repo root: python3 docs/capabilities/build_ledger.py
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[2]
|
||||
SPEC = ROOT / "docs" / "spec.md"
|
||||
OUT = ROOT / "docs" / "capabilities" / "ledger.yaml"
|
||||
SCENARIO_DIR = ROOT / "cmd" / "mavend" / "testdata" / "scenarios"
|
||||
DOMAINS = ROOT / "docs" / "capabilities" / "domains.yaml"
|
||||
VERDICTS = ROOT / "docs" / "capabilities" / "verdicts.json"
|
||||
IMPL = ROOT / "docs" / "capabilities" / "implementation.yaml"
|
||||
ARCH = ROOT / "docs" / "architecture" / "maven-architecture.json"
|
||||
INV_YAML = ROOT / "docs" / "capabilities" / "invariants.yaml"
|
||||
INV_MD = ROOT / "docs" / "capabilities" / "invariants.md"
|
||||
|
||||
# Sections of docs/spec.md whose ### headings are capabilities. Every other ##
|
||||
# is prose about how to read the file.
|
||||
CAPABILITY_SECTIONS = {
|
||||
"The turn",
|
||||
"Memory",
|
||||
"Proactive",
|
||||
"Reach",
|
||||
"Speech and senses",
|
||||
"The ecosystem",
|
||||
"Operations",
|
||||
"Undesigned in v1",
|
||||
}
|
||||
|
||||
# Backticked identifiers that appear on a Scenario line and are not scenarios.
|
||||
NOT_SCENARIOS = {"mavseal", "docker"}
|
||||
|
||||
DOMAIN_NAMES = {
|
||||
"perception", "memory", "attention", "deliberation",
|
||||
"initiative", "action", "interaction", "governance", "operations",
|
||||
}
|
||||
|
||||
|
||||
def slug(title):
|
||||
s = re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")
|
||||
return s
|
||||
|
||||
|
||||
def crit_id(cap_slug, text):
|
||||
h = hashlib.sha256(text.encode("utf-8")).hexdigest()[:4]
|
||||
return f"{cap_slug}#{h}"
|
||||
|
||||
|
||||
def unwrap(lines):
|
||||
"""Join a bullet's continuation lines into one string."""
|
||||
return " ".join(l.strip() for l in lines).strip()
|
||||
|
||||
|
||||
def parse():
|
||||
lines = SPEC.read_text(encoding="utf-8").splitlines()
|
||||
caps = []
|
||||
section = None
|
||||
cur = None
|
||||
mode = None # None | 'dod' | 'state' | 'scenario'
|
||||
buf = [] # continuation lines of the bullet being read
|
||||
section_notes = {}
|
||||
|
||||
def flush_bullet():
|
||||
nonlocal buf
|
||||
if not buf or cur is None:
|
||||
buf = []
|
||||
return
|
||||
text = unwrap(buf)
|
||||
buf = []
|
||||
if not text:
|
||||
return
|
||||
if mode == "dod":
|
||||
cur["dod"].append(text)
|
||||
elif mode == "state":
|
||||
cur["state"] = text
|
||||
elif mode == "scenario":
|
||||
cur["scenario_raw"] = text
|
||||
|
||||
for raw in lines:
|
||||
if raw.startswith("## "):
|
||||
flush_bullet()
|
||||
mode = None
|
||||
cur = None
|
||||
section = raw[3:].strip()
|
||||
continue
|
||||
if raw.startswith("### "):
|
||||
flush_bullet()
|
||||
mode = None
|
||||
if section not in CAPABILITY_SECTIONS:
|
||||
cur = None
|
||||
continue
|
||||
title = raw[4:].strip()
|
||||
cur = {
|
||||
"id": slug(title),
|
||||
"title": title,
|
||||
"section": section,
|
||||
"scope": "v1",
|
||||
"state": "",
|
||||
"finding": "",
|
||||
"dod": [],
|
||||
"scenario_raw": "",
|
||||
"body": [],
|
||||
}
|
||||
caps.append(cur)
|
||||
continue
|
||||
if cur is None:
|
||||
# Section-level prose. Keep a Finding paragraph, it belongs to the
|
||||
# whole cluster (Memory has one).
|
||||
if section in CAPABILITY_SECTIONS and "**Finding**" in raw:
|
||||
section_notes.setdefault(section, []).append(raw.strip())
|
||||
elif section in section_notes and raw.strip() and not raw.startswith("#"):
|
||||
# continuation of that paragraph
|
||||
if section_notes[section] and section_notes[section][-1]:
|
||||
section_notes[section][-1] += " " + raw.strip()
|
||||
continue
|
||||
|
||||
cur["body"].append(raw)
|
||||
stripped = raw.strip()
|
||||
|
||||
if stripped.startswith("- **State**:"):
|
||||
flush_bullet()
|
||||
mode = "state"
|
||||
buf = [stripped[len("- **State**:"):]]
|
||||
continue
|
||||
if stripped.startswith("- **DoD**"):
|
||||
flush_bullet()
|
||||
mode = "dod"
|
||||
continue
|
||||
if stripped.startswith("- **Scenario**:"):
|
||||
flush_bullet()
|
||||
mode = "scenario"
|
||||
buf = [stripped[len("- **Scenario**:"):]]
|
||||
continue
|
||||
if stripped.startswith("**Deferred past v1**"):
|
||||
flush_bullet()
|
||||
cur["scope"] = "deferred"
|
||||
cur["deferred_note"] = stripped
|
||||
mode = None
|
||||
continue
|
||||
if not stripped:
|
||||
flush_bullet()
|
||||
continue
|
||||
if mode == "dod":
|
||||
if stripped.startswith("- "):
|
||||
flush_bullet()
|
||||
buf = [stripped[2:]]
|
||||
else:
|
||||
buf.append(stripped)
|
||||
continue
|
||||
if mode in ("state", "scenario"):
|
||||
if stripped.startswith("- "):
|
||||
flush_bullet()
|
||||
mode = None
|
||||
else:
|
||||
buf.append(stripped)
|
||||
continue
|
||||
|
||||
flush_bullet()
|
||||
|
||||
for c in caps:
|
||||
# The Finding sentence lives inside the State bullet in every entry that
|
||||
# has one. Split it out so a gap is a field, not prose.
|
||||
m = re.search(r"\*\*Finding\*\*:\s*(.*)$", c["state"], re.S)
|
||||
if m:
|
||||
c["finding"] = m.group(1).strip()
|
||||
c["state"] = c["state"][: m.start()].strip()
|
||||
c["state"] = c["state"].strip()
|
||||
c["scenarios"] = parse_scenarios(c["scenario_raw"])
|
||||
c["criteria"] = [
|
||||
{"id": crit_id(c["id"], t), "text": t} for t in c["dod"]
|
||||
]
|
||||
del c["body"], c["dod"], c["scenario_raw"]
|
||||
|
||||
return caps, section_notes
|
||||
|
||||
|
||||
def parse_scenarios(raw):
|
||||
"""Names in `backticks`, each flagged exists / to write.
|
||||
|
||||
The scenario line is prose in several entries ("covered by cmd/mavweb
|
||||
tests", "none"). Keep the prose verbatim as `note` rather than guessing.
|
||||
"""
|
||||
out = []
|
||||
for m in re.finditer(r"`([a-z0-9_]+)`(\s*\*\(to write\)\*)?", raw):
|
||||
name = m.group(1)
|
||||
# A path, a package or a binary named in prose is not a scenario name.
|
||||
if "/" in name or "." in name or name in NOT_SCENARIOS:
|
||||
continue
|
||||
claimed = m.group(2) is None
|
||||
on_disk = (SCENARIO_DIR / f"{name}.json").exists()
|
||||
if not claimed and not on_disk:
|
||||
pass # marked (to write) and absent: consistent
|
||||
out.append({"name": name, "claimed": claimed, "exists": on_disk})
|
||||
return {"named": out, "note": raw.strip()}
|
||||
|
||||
|
||||
def y(s, indent):
|
||||
"""Emit one scalar as a YAML block string, no quoting games."""
|
||||
pad = " " * indent
|
||||
body = "\n".join(pad + " " + l for l in s.splitlines()) if s else ""
|
||||
return ">-\n" + body if s else '""'
|
||||
|
||||
|
||||
VERDICT_WORDS = {"pass", "fail", "blocked", "untested", "unknown"}
|
||||
|
||||
# Why a criterion is not passing. The plan's list, and nothing outside it.
|
||||
REASON_KINDS = {
|
||||
"code missing", "wiring missing", "configuration missing",
|
||||
"deployment missing", "external dependency unavailable",
|
||||
"scenario missing", "scenario fails",
|
||||
"implementation exists with no runtime proof",
|
||||
"deferred past v1", "not yet probed", "passes",
|
||||
}
|
||||
|
||||
|
||||
def load_verdicts():
|
||||
if not VERDICTS.exists():
|
||||
return {}
|
||||
return json.loads(VERDICTS.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def emit(caps, section_notes, domains, verdicts, impl, arch):
|
||||
L = []
|
||||
L.append("# Capability ledger, target side.")
|
||||
L.append("#")
|
||||
L.append("# GENERATED by docs/capabilities/build_ledger.py from docs/spec.md.")
|
||||
L.append("# Do not hand-edit. Domain assignment is the one human input and")
|
||||
L.append("# lives in docs/capabilities/domains.yaml.")
|
||||
L.append("#")
|
||||
L.append("# Verification is per criterion, from verdicts.json. Implementation is")
|
||||
L.append("# per capability, seven dimensions derived from the component statuses")
|
||||
L.append("# in docs/architecture/maven-architecture.json through the mapping in")
|
||||
L.append("# docs/capabilities/implementation.yaml. Never one boolean.")
|
||||
L.append("")
|
||||
L.append(f"source: docs/spec.md")
|
||||
L.append(f"capability_count: {len(caps)}")
|
||||
L.append(f"criterion_count: {sum(len(c['criteria']) for c in caps)}")
|
||||
L.append("")
|
||||
if section_notes:
|
||||
L.append("section_findings:")
|
||||
for sec, notes in section_notes.items():
|
||||
L.append(f" - section: {sec!r}")
|
||||
L.append(" finding: " + y(" ".join(notes), 4))
|
||||
L.append("")
|
||||
L.append("capabilities:")
|
||||
for c in caps:
|
||||
L.append(f" - id: {c['id']}")
|
||||
L.append(f" title: {c['title']!r}")
|
||||
L.append(f" section: {c['section']!r}")
|
||||
L.append(f" scope: {c['scope']}")
|
||||
d = domains.get(c["id"], [])
|
||||
L.append(" domain: [" + ", ".join(d) + "]")
|
||||
L.append(" state: " + y(c["state"], 4))
|
||||
if arch:
|
||||
dims = dimensions(c, impl.get(c["id"], []), arch, verdicts)
|
||||
L.append(" implementation:")
|
||||
for k in ("designed", "code_present", "wired", "configured",
|
||||
"deployed", "reachable", "verified", "gap_class"):
|
||||
# Quoted: bare yes/no are YAML booleans and the
|
||||
# round-trip check reads them back as True/False.
|
||||
L.append(f" {k}: {dims[k]!r}")
|
||||
comps = impl.get(c["id"], [])
|
||||
L.append(" components: [" + ", ".join(comps) + "]")
|
||||
if c["finding"]:
|
||||
L.append(" finding: " + y(c["finding"], 4))
|
||||
if c.get("deferred_note"):
|
||||
L.append(" deferred_note: " + y(c["deferred_note"], 4))
|
||||
L.append(" scenarios:")
|
||||
for s in c["scenarios"]["named"]:
|
||||
L.append(f" - name: {s['name']}")
|
||||
L.append(f" exists: {str(s['exists']).lower()}")
|
||||
if s["claimed"] != s["exists"]:
|
||||
L.append(" discrepancy: spec implies it exists and "
|
||||
"cmd/mavend/testdata/scenarios has no such file")
|
||||
L.append(" scenario_note: " + y(c["scenarios"]["note"], 4))
|
||||
L.append(" criteria:")
|
||||
for cr in c["criteria"]:
|
||||
L.append(f" - id: {cr['id']!r}")
|
||||
L.append(" text: " + y(cr["text"], 8))
|
||||
v = verdicts.get(cr["id"])
|
||||
if v is None and c["scope"] == "deferred":
|
||||
v = {"verified": "untested", "reason": "deferred past v1"}
|
||||
if v is None:
|
||||
v = {"verified": "untested", "reason": "not yet probed"}
|
||||
L.append(f" verified: {v['verified']}")
|
||||
L.append(f" reason: {v['reason']!r}")
|
||||
if v.get("detail"):
|
||||
L.append(" detail: " + y(v["detail"], 8))
|
||||
ev = v.get("evidence", [])
|
||||
if ev:
|
||||
L.append(" evidence:")
|
||||
for e in ev:
|
||||
L.append(f" - {e!r}")
|
||||
L.append("")
|
||||
return "\n".join(L) + "\n"
|
||||
|
||||
|
||||
# --- Implementation dimensions -------------------------------------------
|
||||
#
|
||||
# Never one `implemented` boolean. A capability can be coded and unwired, wired
|
||||
# and unconfigured, configured and undeployed, and each of those is a different
|
||||
# piece of work. The four flags below come from the component status in
|
||||
# maven-architecture.json, which was read from code, config and compose.
|
||||
#
|
||||
# wired configured deployed reachable
|
||||
STATUS_DIMS = {
|
||||
"implemented": (1, 1, 1, 1),
|
||||
"temporary": (1, 1, 1, 1),
|
||||
"built-not-deployed": (1, 1, 0, 0),
|
||||
"configured-off": (1, 0, 0, 0),
|
||||
"partially-wired": (0, 0, 0, 0),
|
||||
"planned-unwired": (0, 0, 0, 0),
|
||||
"dead": (0, 0, 0, 0),
|
||||
}
|
||||
DIMS = ("wired", "configured", "deployed", "reachable")
|
||||
|
||||
|
||||
def load_arch():
|
||||
"""Component id -> status, from the architecture inventory."""
|
||||
if not ARCH.exists():
|
||||
return {}
|
||||
d = json.loads(ARCH.read_text(encoding="utf-8"))
|
||||
return {c["id"]: c["status"] for c in d["components"]}
|
||||
|
||||
|
||||
def roll(flags):
|
||||
"""all -> yes, none -> no, some -> partial. Empty -> no."""
|
||||
if not flags:
|
||||
return "no"
|
||||
if all(flags):
|
||||
return "yes"
|
||||
if not any(flags):
|
||||
return "no"
|
||||
return "partial"
|
||||
|
||||
|
||||
def dimensions(cap, comps, arch, verdicts):
|
||||
"""The seven dimensions for one capability. Never collapsed."""
|
||||
known = [c for c in comps if c in arch]
|
||||
out = {}
|
||||
|
||||
# designed: the spec states every one of these, so the question this
|
||||
# dimension answers is narrower. Does a living doc own the subsystem.
|
||||
st = cap["state"].lower()
|
||||
if "no package" in st:
|
||||
out["designed"] = "spec-only"
|
||||
elif "no living doc" in st or "no capture client" in st:
|
||||
out["designed"] = "spec-only"
|
||||
else:
|
||||
out["designed"] = "yes"
|
||||
|
||||
out["code_present"] = roll([1] * len(known)) if comps else "no"
|
||||
|
||||
for i, name in enumerate(DIMS):
|
||||
out[name] = roll([STATUS_DIMS[arch[c]][i] for c in known])
|
||||
|
||||
vs = [verdicts.get(cr["id"], {}).get("verified", "untested")
|
||||
for cr in cap["criteria"]]
|
||||
# The gap class, for docs/capabilities/gaps.md. Order matters: nothing built
|
||||
# outranks nothing reachable, which outranks something reachable and wrong.
|
||||
if out["code_present"] == "no":
|
||||
out["gap_class"] = "capability missing"
|
||||
elif out["reachable"] != "yes":
|
||||
out["gap_class"] = "capability exists but unreachable"
|
||||
elif any(v == "fail" for v in vs):
|
||||
out["gap_class"] = "capability partial"
|
||||
elif all(v == "pass" for v in vs):
|
||||
out["gap_class"] = "none"
|
||||
else:
|
||||
out["gap_class"] = "capability exists but unverified"
|
||||
if vs and all(v == "pass" for v in vs):
|
||||
out["verified"] = "yes"
|
||||
elif any(v == "pass" for v in vs):
|
||||
out["verified"] = "partial"
|
||||
else:
|
||||
out["verified"] = "no"
|
||||
return out
|
||||
|
||||
|
||||
def load_flat(path):
|
||||
"""`key: [a, b]` per line, # comments stripped. domains and implementation."""
|
||||
if not path.exists():
|
||||
return {}
|
||||
out = {}
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.split("#", 1)[0].strip()
|
||||
if not line or ":" not in line:
|
||||
continue
|
||||
k, v = line.split(":", 1)
|
||||
vals = [x.strip() for x in v.strip().strip("[]").split(",") if x.strip()]
|
||||
out[k.strip()] = vals
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
caps, section_notes = parse()
|
||||
domains = load_flat(DOMAINS)
|
||||
impl = load_flat(IMPL)
|
||||
arch = load_arch()
|
||||
verdicts = load_verdicts()
|
||||
|
||||
errs = []
|
||||
seen = {}
|
||||
for c in caps:
|
||||
for cr in c["criteria"]:
|
||||
if cr["id"] in seen:
|
||||
errs.append(f"criterion id collision: {cr['id']}")
|
||||
seen[cr["id"]] = cr["text"]
|
||||
if not c["criteria"]:
|
||||
errs.append(f"{c['id']}: no DoD criteria extracted")
|
||||
if not c["state"]:
|
||||
errs.append(f"{c['id']}: no State line extracted")
|
||||
d = domains.get(c["id"])
|
||||
if domains:
|
||||
if not d:
|
||||
errs.append(f"{c['id']}: no domain assigned")
|
||||
elif len(d) > 2:
|
||||
errs.append(f"{c['id']}: {len(d)} domains, max is 2")
|
||||
else:
|
||||
for x in d:
|
||||
if x not in DOMAIN_NAMES:
|
||||
errs.append(f"{c['id']}: unknown domain {x!r}")
|
||||
cap_ids = {c["id"] for c in caps}
|
||||
for k in domains:
|
||||
if k not in cap_ids:
|
||||
errs.append(f"domains.yaml names unknown capability {k!r}")
|
||||
|
||||
# The mapping is the whole basis of the implementation columns. A capability
|
||||
# missing from it reads as `no` on every dimension, which is indistinguishable
|
||||
# from a capability nothing carries. Refuse rather than guess which.
|
||||
if impl:
|
||||
if not arch:
|
||||
errs.append("implementation.yaml is present and "
|
||||
"docs/architecture/maven-architecture.json is not")
|
||||
for k in impl:
|
||||
if k not in cap_ids:
|
||||
errs.append(f"implementation.yaml names unknown capability {k!r}")
|
||||
for c in caps:
|
||||
if c["id"] not in impl:
|
||||
errs.append(f"{c['id']}: no row in implementation.yaml")
|
||||
for k, comps in impl.items():
|
||||
for comp in comps:
|
||||
if arch and comp not in arch:
|
||||
errs.append(f"{k}: unknown component {comp!r}")
|
||||
for comp, st in arch.items():
|
||||
if st not in STATUS_DIMS:
|
||||
errs.append(f"maven-architecture.json: unknown status {st!r} on {comp}")
|
||||
|
||||
# A verdict cites evidence by path, and a path that resolves to nothing is
|
||||
# worse than no citation: it reads as verified and is not. Section refs are
|
||||
# checked too, because writing "§ Something" that no heading matches is the
|
||||
# easy way to make an unsupported claim look sourced.
|
||||
for cid, v in verdicts.items():
|
||||
for e in v.get("evidence", []):
|
||||
path, _, section = e.partition(" § ")
|
||||
# An evidence string is "<path>[:line] [locator]" or
|
||||
# "<path> § <heading>". The locator points inside the file
|
||||
# (a probe id, a readback key) and is not part of the path.
|
||||
path = path.strip().split()[0].split(":")[0]
|
||||
f = ROOT / path
|
||||
if not f.exists():
|
||||
errs.append(f"{cid}: evidence path does not exist: {path}")
|
||||
elif section and f.suffix == ".md" and section.strip() not in f.read_text(encoding="utf-8"):
|
||||
errs.append(f"{cid}: evidence names a section not in {path}: {section}")
|
||||
|
||||
for cid, v in verdicts.items():
|
||||
if cid not in seen:
|
||||
errs.append(f"verdicts.json scores unknown criterion {cid!r}")
|
||||
elif v.get("verified") not in VERDICT_WORDS:
|
||||
errs.append(f"{cid}: verdict {v.get('verified')!r} is not one of {sorted(VERDICT_WORDS)}")
|
||||
elif v.get("reason") not in REASON_KINDS:
|
||||
errs.append(f"{cid}: reason {v.get('reason')!r} is not one of the plan's kinds")
|
||||
elif v["verified"] == "pass" and v["reason"] != "passes":
|
||||
errs.append(f"{cid}: a pass carries reason {v['reason']!r}")
|
||||
elif v["verified"] != "pass" and v["reason"] == "passes":
|
||||
errs.append(f"{cid}: reason 'passes' on a {v['verified']} verdict")
|
||||
elif v["verified"] == "fail" and v["reason"] == "implementation exists with no runtime proof":
|
||||
# That reason means nothing was observed. A fail was observed, or it
|
||||
# is not a fail. Mixing them is how a wrong diagnosis survives.
|
||||
errs.append(f"{cid}: a fail cannot rest on 'no runtime proof'")
|
||||
|
||||
# The invariants exist twice on purpose: prose and evidence in the .md, the
|
||||
# machine-readable half in the .yaml for the viewer. They drift the moment
|
||||
# nothing checks them, so check them.
|
||||
if INV_YAML.exists():
|
||||
try:
|
||||
import yaml as _y
|
||||
except ImportError:
|
||||
errs.append("pyyaml absent: invariants.yaml was not checked")
|
||||
else:
|
||||
inv = _y.safe_load(INV_YAML.read_text(encoding="utf-8"))["invariants"]
|
||||
md = INV_MD.read_text(encoding="utf-8") if INV_MD.exists() else ""
|
||||
if not md:
|
||||
errs.append("invariants.yaml exists and invariants.md does not")
|
||||
n_md = md.count("\n## ") - md.count("\n## What this file")
|
||||
if md and n_md != len(inv):
|
||||
errs.append(f"invariants: {len(inv)} in the yaml, {n_md} headings in the md")
|
||||
for iv in inv:
|
||||
head = f"## {iv['id']}. {iv['title']}"
|
||||
if md and head not in md:
|
||||
errs.append(f"invariant {iv['id']}: no heading {head!r} in invariants.md")
|
||||
if iv["mark"] not in {"explicit", "implied", "unresolved"}:
|
||||
errs.append(f"invariant {iv['id']}: mark {iv['mark']!r} is not one of three")
|
||||
for c in iv["capabilities"]:
|
||||
if c not in cap_ids:
|
||||
errs.append(f"invariant {iv['id']}: unknown capability {c!r}")
|
||||
for c in iv["components"]:
|
||||
if arch and c not in arch:
|
||||
errs.append(f"invariant {iv['id']}: unknown component {c!r}")
|
||||
if iv["mark"] == "unresolved" and not iv.get("question"):
|
||||
errs.append(f"invariant {iv['id']}: unresolved with no product question")
|
||||
|
||||
OUT.write_text(emit(caps, section_notes, domains, verdicts, impl, arch), encoding="utf-8")
|
||||
|
||||
# The emitter hand-writes YAML, so it can produce something that reads fine
|
||||
# and does not parse. It did once: evidence came out as a bare list item
|
||||
# inside a mapping. Parse what was just written.
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
errs.append("pyyaml absent: the output was written without a parse check")
|
||||
else:
|
||||
try:
|
||||
doc = yaml.safe_load(OUT.read_text(encoding="utf-8"))
|
||||
except yaml.YAMLError as e:
|
||||
errs.append(f"the emitted ledger is not valid YAML: {e}")
|
||||
else:
|
||||
n = sum(len(c["criteria"]) for c in doc["capabilities"])
|
||||
if n != len(seen):
|
||||
errs.append(f"round trip lost criteria: wrote {len(seen)}, read back {n}")
|
||||
for c in doc["capabilities"]:
|
||||
if c["scope"] != "v1":
|
||||
continue
|
||||
for cr in c["criteria"]:
|
||||
if cr["verified"] != "untested" and not cr.get("evidence"):
|
||||
errs.append(f"{cr['id']}: scored {cr['verified']} with no evidence")
|
||||
|
||||
print(f"{len(caps)} capabilities, {len(seen)} criteria -> {OUT.relative_to(ROOT)}")
|
||||
print(f" v1: {sum(1 for c in caps if c['scope'] == 'v1')}, "
|
||||
f"deferred: {sum(1 for c in caps if c['scope'] == 'deferred')}")
|
||||
scen = {s["name"]: s["exists"] for c in caps for s in c["scenarios"]["named"]}
|
||||
ghosts = sorted(n for n, e in scen.items() if not e
|
||||
and any(s["claimed"] for c in caps for s in c["scenarios"]["named"] if s["name"] == n))
|
||||
if ghosts:
|
||||
print(f" named as existing but absent from disk: {', '.join(ghosts)}")
|
||||
print(f" scenarios named: {len(scen)}, existing: {sum(scen.values())}, "
|
||||
f"to write: {len(scen) - sum(scen.values())}")
|
||||
if verdicts:
|
||||
from collections import Counter
|
||||
tally = Counter(v["verified"] for v in verdicts.values())
|
||||
print(" verdicts: " + ", ".join(f"{k} {n}" for k, n in sorted(tally.items())))
|
||||
if arch:
|
||||
from collections import Counter as _C
|
||||
for k in ("code_present", "wired", "configured", "deployed", "reachable"):
|
||||
t = _C(dimensions(c, impl.get(c["id"], []), arch, verdicts)[k] for c in caps)
|
||||
print(f" {k}: " + ", ".join(f"{a} {n}" for a, n in sorted(t.items())))
|
||||
# A capability nothing carries that still scores a pass. Always a
|
||||
# negative criterion passing by absence. Worth seeing, not an error.
|
||||
for c in caps:
|
||||
d_ = dimensions(c, impl.get(c["id"], []), arch, verdicts)
|
||||
if d_["code_present"] == "no" and d_["verified"] != "no":
|
||||
print(f" ANOMALY {c['id']}: nothing carries it and it scores "
|
||||
f"verified={d_['verified']} (a negative criterion passing by absence)")
|
||||
t = _C(dimensions(c, impl.get(c["id"], []), arch, verdicts)["gap_class"]
|
||||
for c in caps if c["scope"] == "v1")
|
||||
print(" v1 gap classes: " + ", ".join(f"{a} {n}" for a, n in sorted(t.items())))
|
||||
used = {x for v in impl.values() for x in v}
|
||||
orphan = sorted(set(arch) - used)
|
||||
print(f" components serving no capability: {len(orphan)}")
|
||||
for o in orphan:
|
||||
print(f" {o} ({arch[o]})")
|
||||
print(f" no living doc: {sum(1 for c in caps if 'No living doc' in c['state'] or 'no living doc' in c['state'].lower())}")
|
||||
if errs:
|
||||
print("\nERRORS:", file=sys.stderr)
|
||||
for e in errs:
|
||||
print(" " + e, file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,166 @@
|
||||
# Domain assignment for the capability ledger. The one human input to
|
||||
# build_ledger.py; everything else in ledger.yaml is mechanical.
|
||||
#
|
||||
# At most two domains, PRIMARY FIRST. Three independent passes ran under
|
||||
# different lenses (bottom-up from the DoD, from the owner's experience,
|
||||
# from state ownership and effect), then one reconciler re-read the DoD of
|
||||
# every contested row and broke the tie. A row marked (contested) is one the
|
||||
# three passes did not agree on; its note names what broke the tie.
|
||||
#
|
||||
# 35 of 51 were unanimous. Goes to the owner once, before any probe runs.
|
||||
|
||||
# Unanimous: every criterion interprets an utterance into intent plus source and records which stage decided.
|
||||
route-an-utterance: [deliberation]
|
||||
|
||||
# Contested; broken by criterion count: two of three DoD lines are parked-turn dialogue lifecycle (cancel by "отмена", survive an interleaved turn and resume), and none is a permission or confirmation, so interaction beats governance for second. (contested)
|
||||
ask-instead-of-guessing: [deliberation, interaction]
|
||||
|
||||
# Unanimous: every criterion polices what wording reaches the outbound wire.
|
||||
speak-as-herself: [interaction]
|
||||
|
||||
# Contested; broken by criterion dbeb, which is the privacy ordering rule (owner's sources before anything outside, every time), not an interpretation step, so governance beats deliberation. (contested)
|
||||
answer-from-your-own-data: [memory, governance]
|
||||
|
||||
# Unanimous: retrieve an external answer, bounded by what may leave the box.
|
||||
answer-from-the-world: [action, governance]
|
||||
|
||||
# Unanimous: pick the right Kiwix book and retrieve a topically correct article — pure retrieval.
|
||||
read-an-encyclopedia: [action]
|
||||
|
||||
# Unanimous: call a configured provider, with the follow-up city parked as a clarify rather than guessed.
|
||||
weather: [action, deliberation]
|
||||
|
||||
# Contested; I overrule the two perception-first votes on the ledger's own weather logic: the DoD is an offloaded vision tool call over content he supplied, with a silent fallback, so it is a tool call first and sensing second. (contested)
|
||||
see-an-image: [action, perception]
|
||||
|
||||
# Contested; broken by reading the criteria: write, honest confirmation, supersede and Nexus-resolved subject are all fact-store integrity, and none is a permission, privacy or confirmation-binding rule, so governance drops. (contested)
|
||||
facts: [memory]
|
||||
|
||||
# Contested; broken by criterion count: capture, recall and delete are the store's lifecycle, and "a question is not stored as a statement" is a routing defect already owned by route-an-utterance, so deliberation drops. (contested)
|
||||
notes: [memory]
|
||||
|
||||
# Unanimous across all three passes: it reads the note and fact store through
|
||||
# the embedder, with the personal boundary deciding what that read may cross
|
||||
# into. Recovered from the three passes' journal: the reconciler dropped this
|
||||
# row from its 51, and build_ledger.py's guard caught the omission.
|
||||
recall: [memory, governance]
|
||||
|
||||
# Contested; broken by the ledger finding that the evaluator cannot speak: nothing surfaces to him, so initiative cannot hold, and the DoD's checkable conclusions over notes it read are deliberation. (contested)
|
||||
memory-evaluation: [memory, deliberation]
|
||||
|
||||
# Unanimous: hold a future commitment, fire it at its time, and get it delivered across reaches.
|
||||
reminders: [attention, interaction]
|
||||
|
||||
# Unanimous: the whole DoD is whether an unprompted item may break in, keyed on presence and severity.
|
||||
interruption-policy: [initiative, perception]
|
||||
|
||||
# Unanimous: a suppressed nudge candidate must resurface unprompted in a later digest, without acting.
|
||||
digest-of-held-nudges: [initiative, attention]
|
||||
|
||||
# Unanimous: an unprompted plan inside its window that must find another reach rather than be dropped.
|
||||
morning-routine: [initiative, interaction]
|
||||
|
||||
# Unanimous: propose from observed repeated behaviour and store the decline.
|
||||
routine-proposals: [initiative, memory]
|
||||
|
||||
# Unanimous: open items ordered by deadline and urgency over a stored work list.
|
||||
tasks: [attention, memory]
|
||||
|
||||
# Unanimous: fetch configured feeds and find the matching item on request, explicitly never unprompted.
|
||||
rss-and-news: [action]
|
||||
|
||||
# Unanimous: outbound delivery with an outbox row and continuous inbound reading — a reach.
|
||||
telegram: [interaction]
|
||||
|
||||
# Unanimous: a push reach whose criteria are its credential and not looping when refused.
|
||||
ntfy: [interaction]
|
||||
|
||||
# Unanimous: a live speech reach, capped at L0 and bound to loopback.
|
||||
voice: [interaction, governance]
|
||||
|
||||
# Unanimous: a page per capability with step-up standing in front of a destructive write.
|
||||
web-ui: [interaction, governance]
|
||||
|
||||
# Unanimous: inbound ingests desktop events as low-confidence facts; the outbound half is an undecided fourth reach.
|
||||
desk-notifications: [perception, interaction]
|
||||
|
||||
# Contested; broken by the DoD being accuracy-enough-to-route plus a silent fallback arm on the voice surface, with nothing about context, so interaction leads and perception stays second. (contested)
|
||||
speech-to-text: [interaction, perception]
|
||||
|
||||
# Unanimous: the reply rendered as Russian speech with times and numbers expanded for the ear.
|
||||
text-to-speech: [interaction]
|
||||
|
||||
# Contested; broken by criterion 10ca being the detection itself ("Мэйвен" wakes her, a near-miss does not) — the session it opens belongs to voice, so perception leads. (contested)
|
||||
wake-word: [perception, interaction]
|
||||
|
||||
# Contested; broken by the DoD being a microphone capture client existing at all — sensing that must exist before any surface — so perception leads over the reach it feeds. (contested)
|
||||
hearing: [perception, interaction]
|
||||
|
||||
# Contested (order only); broken by criterion 3110 being the recognition itself, with the act-path gate the second criterion it feeds, so perception leads. (contested)
|
||||
speaker-recognition: [perception, governance]
|
||||
|
||||
# Unanimous: free text resolves to a canonical entity or she asks, and that resolution precedes any mutating call.
|
||||
nexus: [deliberation, governance]
|
||||
|
||||
# Unanimous: the source of items needing attention, bound by lifecycle words and the no-auto-act rule.
|
||||
praxis: [attention, governance]
|
||||
|
||||
# Unanimous: the only path that changes the world, bound by a confirmation LLM output cannot supply.
|
||||
hexis: [action, governance]
|
||||
|
||||
# Unanimous: control a device through Hexis on a Nexus-resolved entity, never by free text.
|
||||
smart-home: [action, governance]
|
||||
|
||||
# Contested; broken by what the rate-limit criterion actually is — a politeness bound in config, not an authorization gate — so governance drops and action stands alone. (contested)
|
||||
network-scans: [action]
|
||||
|
||||
# Unanimous: connect and disconnect a paired radio device through the act path.
|
||||
bluetooth-control: [action]
|
||||
|
||||
# Unanimous: external tools callable through the act path with the allowlist as the only door.
|
||||
mcps: [action, governance]
|
||||
|
||||
# Unanimous: compose services on the current build and a lossless restart — infrastructure, filed under Operations.
|
||||
the-deployed-stack: [operations]
|
||||
|
||||
# Contested; broken by all three criteria being secrecy mechanisms (at-rest encryption, key held only by mavend, passwords from files), which is governance's privacy clause carried by infrastructure. (contested)
|
||||
encrypted-database: [operations, governance]
|
||||
|
||||
# Unanimous: no privileged gate fail-open and step-up per-request — authorization, filed under Operations.
|
||||
passkey-and-step-up: [governance, operations]
|
||||
|
||||
# Unanimous: which gguf may load and whether the swap survives a restart.
|
||||
model-swap: [operations]
|
||||
|
||||
# Unanimous: updating the deployment from inside it and rolling back a failure.
|
||||
self-update: [operations]
|
||||
|
||||
# Unanimous: the build and analyzer gate itself, no agent behaviour in it.
|
||||
tests-and-analyzers: [operations]
|
||||
|
||||
# Unanimous: a mail workflow whose open criteria are candidates staying candidates and no content leaving the box.
|
||||
email-triage: [action, governance]
|
||||
|
||||
# Contested; broken by criteria 4047 and 5f7c both refusing to proceed on an under-determined request (ask for the slot, name the conflict), which is deliberation, not proactive attention. (contested)
|
||||
calendar-management: [action, deliberation]
|
||||
|
||||
# Contested; broken by criterion count: two of four are constraints (robots and politeness, only the URL and utterance leave) against one for the watch, so governance takes second — the watch does pull at initiative. (contested)
|
||||
web-crawling: [action, governance]
|
||||
|
||||
# Unanimous: pull a named source and condense it, refusing a summary that would invent content.
|
||||
summaries: [action, governance]
|
||||
|
||||
# Contested; broken by criterion 5134 naming authentication for both directions explicitly, which outweighs the inbound half's perception flavour. (contested)
|
||||
webhooks: [interaction, governance]
|
||||
|
||||
# Unanimous: a user-set schedule that runs acts under the same confirmation rules as a spoken act.
|
||||
cron-jobs: [action, governance]
|
||||
|
||||
# Unanimous: a correction stored as a readable, deletable outcome whose only effect is later phrasing.
|
||||
learning-the-style: [memory, interaction]
|
||||
|
||||
# Unanimous: stored outcomes from dismissals and repairs that change the next decision.
|
||||
learning-from-mistakes: [memory, deliberation]
|
||||
|
||||
# Contested; broken by criterion 77b5 stating that a chain containing an act confirms each act separately, an explicit confirmation rule that outranks the generic "performs both" pull toward action. (contested)
|
||||
command-chaining: [deliberation, governance]
|
||||
@@ -0,0 +1,241 @@
|
||||
# Gaps: what is missing, and what the architecture does about it
|
||||
|
||||
Hand-written, except the four capability classes, which are derived.
|
||||
|
||||
This file compares responsibilities. It never compares package names. A package
|
||||
existing is not a capability, and a capability can be spread over six packages
|
||||
and still be missing.
|
||||
|
||||
## Where each class comes from
|
||||
|
||||
Classes 1 through 4 are the `gap_class` field in `docs/capabilities/ledger.yaml`,
|
||||
derived from the seven implementation dimensions and the criteria verdicts.
|
||||
Rebuild them:
|
||||
|
||||
```sh
|
||||
python3 docs/capabilities/build_ledger.py
|
||||
```
|
||||
|
||||
Classes 5 through 8 are read from `docs/architecture/findings.md` and
|
||||
`docs/capabilities/invariants.md`. **Every entry names the capability or
|
||||
invariant it affects.** An entry affecting neither is marked non-blocking
|
||||
cleanup, in those words, and it is the whole of class 8.
|
||||
|
||||
Counts are over the 46 v1 capabilities. The 5 deferred ones are excluded.
|
||||
|
||||
---
|
||||
|
||||
## 1. Capability missing (5)
|
||||
|
||||
Nothing carries it. `code_present: no`.
|
||||
|
||||
| capability | criteria | note |
|
||||
| --- | --- | --- |
|
||||
| `summaries` | 0 pass, 3 fail | no package. Wanted by `email-triage`, `web-crawling`, `hearing` and `rss-and-news`, each of which would consume it |
|
||||
| `webhooks` | 0 pass, 3 fail | no package. Telegram's own inbound channel is mapped to `telegram`, not here |
|
||||
| `command-chaining` | 0 pass, 3 fail | no package. The `chain` in `internal/router` is the world chain and the source chain |
|
||||
| `learning-the-style` | 1 pass, 2 fail | the pass is a negative criterion satisfied by absence. The build reports it as an anomaly |
|
||||
| `learning-from-mistakes` | 0 pass, 2 fail | no package |
|
||||
|
||||
The last two are invariant 10, and it is `unresolved`. Whether behavioural
|
||||
learning is wanted is a product question, so these two are not automatically
|
||||
work.
|
||||
|
||||
## 2. Capability partial (21)
|
||||
|
||||
Reachable, and at least one criterion was observed failing. This is the class
|
||||
that matters most, because a user can get to all 21 today and 21 misbehave.
|
||||
|
||||
`route-an-utterance`, `ask-instead-of-guessing`, `answer-from-your-own-data`,
|
||||
`answer-from-the-world`, `read-an-encyclopedia`, `facts`, `notes`, `recall`,
|
||||
`reminders`, `voice`, `web-ui`, `desk-notifications`, `wake-word`, `nexus`,
|
||||
`praxis`, `the-deployed-stack`, `encrypted-database`, `passkey-and-step-up`,
|
||||
`tests-and-analyzers`, `web-crawling`, `cron-jobs`.
|
||||
|
||||
`wake-word` is the sharpest: reachable on every dimension and 0 of 2 criteria
|
||||
pass.
|
||||
|
||||
## 3. Capability exists but unreachable (9)
|
||||
|
||||
Built, and the deployed configuration does not reach it.
|
||||
|
||||
| capability | why | class of fix |
|
||||
| --- | --- | --- |
|
||||
| `speak-as-herself` | `reachable: partial` on `core.model_seam` | configuration |
|
||||
| `weather` | no `weather` key in the deployed `voice` block | configuration |
|
||||
| `see-an-image` | no media block | configuration |
|
||||
| `memory-evaluation` | worker is `configured-off` | configuration |
|
||||
| `ntfy` | present in the config and disabled there | configuration |
|
||||
| `hearing` | `capture.enabled` false and no capture client ships (V-514) | configuration and code |
|
||||
| `mcps` | no MCP server configured, and V-478 blocks the one candidate | deployment |
|
||||
| `email-triage` | `mavmaild` is not in `docker-compose.yml` | deployment |
|
||||
| `calendar-management` | `mavcaldav` is not in `docker-compose.yml` | deployment |
|
||||
|
||||
None of these nine is a code defect. Seven are one config block and two are one
|
||||
compose entry. `docs/spec.md` says this about the audit's four and it still
|
||||
holds for these nine.
|
||||
|
||||
## 4. Capability exists but unverified (11)
|
||||
|
||||
Reachable, nothing observed failing, and not all criteria pass. These are
|
||||
measurement gaps, not defects.
|
||||
|
||||
`interruption-policy`, `digest-of-held-nudges`, `morning-routine`,
|
||||
`routine-proposals`, `tasks`, `rss-and-news`, `telegram`, `speech-to-text`,
|
||||
`text-to-speech`, `hexis`, `network-scans`.
|
||||
|
||||
Four of them, the whole Proactive cluster, are untested on every criterion,
|
||||
because a proactive behaviour cannot be probed by sending an utterance. That is
|
||||
the shape of the gap and it needs a different harness, not more probes.
|
||||
|
||||
---
|
||||
|
||||
## 5. Duplicated mechanism (6)
|
||||
|
||||
| what | affects | owned? |
|
||||
| --- | --- | --- |
|
||||
| Two independent arbitrations decide one turn: seven intents, then twenty-two ordered query sources (`findings.md` 2.1) | invariant 11, `route-an-utterance`, `answer-from-your-own-data` | no |
|
||||
| A third arbitration runs before both: eleven stateful pre-emptors in the pre-route ladder (`findings.md` 2.2) | invariant 11, invariant 6, `ask-instead-of-guessing` | no |
|
||||
| Two tier systems. `internal/auth` does not bind the turn path, `internal/tool` is not keyed on the reach (`findings.md` 6.3, 6.3b) | invariant 8, `hexis`, `passkey-and-step-up` | no |
|
||||
| Two representations of reach, both ignored (`findings.md` 6.3) | invariant 8, `voice` | no |
|
||||
| Two digest mechanisms with the same word in the name, flushed six lines apart (`findings.md` 2.4) | invariant 5, `digest-of-held-nudges` | no |
|
||||
| Restraint decided twice: the gate decides whether a rule emits, delivery decides where it lands (`findings.md` 2.3) | `interruption-policy` | **yes**, argued in `channel.go` |
|
||||
|
||||
The last row is duplication that is owned. It is listed so it is not
|
||||
rediscovered as a defect.
|
||||
|
||||
## 6. Missing shared mechanism (6)
|
||||
|
||||
| what is missing | affects |
|
||||
| --- | --- |
|
||||
| A single point that decides whether this origin may perform this effect with this evidence. `origin × effect × evidence → permit` is the target and nothing computes it | invariant 8, `hexis`, `praxis`, `voice`, `passkey-and-step-up` |
|
||||
| An owner for a key namespace. `facts` has nine writers, `notes` six, `tools` three unrelated proposers (`findings.md` 1.1, 1.2, 1.3) | invariant 2, `facts`, `notes`, `recall` |
|
||||
| A comparable unit of evidence, so claimants can answer "is this more mine than yours?". `internal/claim` is that unit, written, tested and called by nothing (`findings.md` 6.1) | invariant 11, `command-chaining` |
|
||||
| A conversation that spans reaches. `mavweb` instead hardcodes one conversation id for the whole web reach | invariant 1, `web-ui`, `voice`, `telegram` |
|
||||
| A stated rule for what survives a restart. Six stores, six independent choices, two of them argued | invariant 12, `ask-instead-of-guessing` |
|
||||
| A summariser. Four capabilities would consume one and none exists | `summaries`, `email-triage`, `web-crawling`, `hearing` |
|
||||
|
||||
## 7. Current architecture conflicts with target behavior (10)
|
||||
|
||||
The class where the code works as written and the written thing is not what the
|
||||
spec asks for.
|
||||
|
||||
| conflict | affects |
|
||||
| --- | --- |
|
||||
| Four silent degradations stack on one turn, and `docs/spec.md` writes every v1 DoD at "honest" (`findings.md` 8.1) | invariant 7, `answer-from-the-world`, `speech-to-text`, `route-an-utterance`, `speak-as-herself` |
|
||||
| `praxisItemAction.handle` calls straight through: acknowledge, resolve, ignore and pin run on first hearing with no tier and no confirm turn (`findings.md` 6.3c, `cmd/mavend/ecosystem_acts.go:158`) | invariant 8, `praxis` |
|
||||
| `weather` is a live query source with `guesses: true` and the deployed config selects no provider, so it can claim a turn and answer from a stub (`findings.md` 11.3) | invariant 7, `weather` |
|
||||
| `loop.State.CalendarBusy` reads facts `mavcaldav` never writes, so the do-not-nag-mid-meeting suppressor is permanently false (`findings.md` 8.2) | invariant 3, `interruption-policy`, `calendar-management` |
|
||||
| Recurring reminders have a column, an IPC parameter and no caller. `actionReminder` passes `""` (`findings.md` 8.3) | `reminders`, `cron-jobs` |
|
||||
| The clarify store is not persisted and the expired-clarify notice reads the store that is gone (`findings.md` 7.5) | invariant 6, invariant 12, `ask-instead-of-guessing` |
|
||||
| `Claim.Coverage` returns 1.0 for a claim that extracted nothing (`findings.md` 6.3d) | invariant 11. Latent: it corrupts the fix for class 6 row 3 before that fix ships |
|
||||
| The act executor runs inside the key holder, and the process boundary is not one of the controls (`findings.md` 5.4) | invariant 8, `hexis`, `encrypted-database` |
|
||||
| The voice wire's whole security argument is external: loopback publish plus an ssh tunnel, so one compose edit removes it (`findings.md` 5.5) | invariant 8, `voice` |
|
||||
| `make test` is green with the four `TestONNX*` measurements silently skipped, because the recipe does not set `MAVEN_ONNX_LIB` | `tests-and-analyzers`, `recall`. It is why the predecessor audit had a green suite and 22 dead capabilities |
|
||||
|
||||
## 8. Architecture concern with no current product impact (11)
|
||||
|
||||
**Every row here is non-blocking cleanup.** None names a capability or an
|
||||
invariant, which is the test for belonging in this class rather than in 5, 6 or
|
||||
7.
|
||||
|
||||
- `reactiveHandler` has 34 fields (`findings.md` 4.1).
|
||||
- `runTurn` is one function with eleven early returns (4.2).
|
||||
- `tick` runs thirteen jobs in one function (4.3).
|
||||
- `wireVoice` is one constructor for seventeen subsystems (4.4).
|
||||
- `mavsttd` and `mavttsd` are separate processes at a scale that does not need it (5.1).
|
||||
- Three IPC connections from one process (5.2).
|
||||
- A construction cycle between the API layer and the turn layer (3.1).
|
||||
- The handler holds the raw store beside the mediated one (3.2).
|
||||
- `queryDayPlan` reads the proactive scheduler, the single call across that line (3.4).
|
||||
- `internal/modes` is imported by nothing outside itself (6.2).
|
||||
- The daemon is wired twice, in two places (8.6).
|
||||
|
||||
Two entries were considered for this class and moved out. `queryNetwork`
|
||||
triggering a live LAN scan inside a read path (6.4) affects `network-scans`,
|
||||
whose two remaining criteria are `unknown`. `actionFact` re-routing into the
|
||||
query chain (6.5) affects `facts` and `route-an-utterance`, which is where the
|
||||
"меня зовут Ками" misroute lives.
|
||||
|
||||
---
|
||||
|
||||
# Priority
|
||||
|
||||
One list. The rank is the plan's, and it is about impact today, not about how
|
||||
ugly the code is. An unwired or unreachable future defect never outranks a live
|
||||
user-visible failure because its architecture is offensive.
|
||||
|
||||
## 1. Prevents intended everyday use today
|
||||
|
||||
1. **`speak-as-herself` fails all three criteria.** The deployed resident model,
|
||||
`maven-instruct-b2-Q4_K_XL`, produces Russian sentences that no longer hold
|
||||
together, and the phrasing checks that would catch it run in the eval and not
|
||||
on the outbound path. Everything that asks the model to write a sentence
|
||||
inherits this. Formal `вас` and `вы` reached the wire while `CheckFeminine`
|
||||
passed.
|
||||
2. **His own name is not stored as a fact.** "меня зовут Ками" routes to chat,
|
||||
so nothing is written, and "что ты помнишь обо мне?" routes to chat too.
|
||||
Two of the seven audit probes, still broken and now broken differently.
|
||||
3. **`wake-word` fails both criteria while reachable on every dimension.** Voice
|
||||
is the spine of v1 and the always-on half of it does not work.
|
||||
4. **Nine capabilities are one config block or one compose entry from
|
||||
reachable.** `weather`, `ntfy`, `see-an-image`, `memory-evaluation`,
|
||||
`email-triage`, `calendar-management` and `mcps` are the cheap ones. This is
|
||||
the highest ratio of capability to work in the whole list.
|
||||
|
||||
## 2. Makes existing behavior incorrect or unreliable
|
||||
|
||||
5. **The Praxis lifecycle path has no gate.** Four remote mutations run on first
|
||||
hearing. This is live today and needs no new wiring to matter.
|
||||
6. **`weather` answers from a stub and is allowed to claim the turn.** A source
|
||||
marked `guesses: true` with no provider is worse than a named gap.
|
||||
7. **The busy suppressor is permanently false.** Every interruption decision
|
||||
that should have deferred to a meeting fails open.
|
||||
8. **Four silent degradations stack**, and nothing in a reply distinguishes the
|
||||
worst case from the best. Invariant 7 has no written boundary between "only
|
||||
better" and "cannot do the job".
|
||||
9. **A parked clarify survived five consecutive turns** and was released by a
|
||||
path other than `отмена`. Invariant 6 says nobody owns closing it.
|
||||
|
||||
## 3. Blocks multiple capabilities
|
||||
|
||||
10. **No single authorization point.** Invariant 8, `unresolved`, and the third
|
||||
of the three questions the freeze was called to answer. It blocks `hexis`,
|
||||
`praxis`, `voice` and `passkey-and-step-up`, and it is the one property
|
||||
nobody can currently state.
|
||||
11. **No owner for a key namespace.** A fetch watermark and a tuning parameter
|
||||
live in the table recall embeds and `queryFactByKey` reads back as an
|
||||
answer.
|
||||
12. **No comparable unit of evidence.** Three ordered lists decide one turn.
|
||||
`command-chaining` cannot be built on top of them, and `internal/claim`
|
||||
carries a live defect before it is wired.
|
||||
13. **No summariser.** Four capabilities would consume one.
|
||||
|
||||
## 4. Prevents verification
|
||||
|
||||
14. **`make test` is green with four measurements skipped.** The recipe does not
|
||||
set `MAVEN_ONNX_LIB`. This is the exact trap `CLAUDE.md` describes, and the
|
||||
baseline walked into it while measuring whether other things had.
|
||||
15. **The whole Proactive cluster is untested on every criterion.** A proactive
|
||||
behaviour cannot be probed by sending an utterance. It needs a clock-driving
|
||||
harness, not more probes.
|
||||
16. **26 of 31 named scenarios do not exist on disk.** Only 5 of 51 spec entries
|
||||
cite a scenario that is there (`findings.md` 9.5).
|
||||
17. **`POST /api/ptt` was called unreachable in an earlier draft and is not.**
|
||||
Four speech criteria were filed `deployment missing` when the deployment is
|
||||
present and the probe was never written.
|
||||
|
||||
## 5. Architectural cleanup with no present user impact
|
||||
|
||||
18. Everything in class 8, in any order. None of it blocks a capability or an
|
||||
invariant, and that is why it is last.
|
||||
|
||||
---
|
||||
|
||||
## What this file does not do
|
||||
|
||||
It does not schedule. `docs/roadmap.md` orders the work and this file feeds it.
|
||||
|
||||
It does not decide the four `unresolved` invariants. Authority and confirmation,
|
||||
learning from outcomes, capability composition and the shelf life of a held
|
||||
nudge are the owner's, and items 10, 12 and 13 above stall on them.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Capability -> component mapping. HAND-WRITTEN. This is the judgment call.
|
||||
#
|
||||
# Component ids come from docs/architecture/maven-architecture.json, whose
|
||||
# `status` field was read from code, config and compose and audited against
|
||||
# them. build_ledger.py derives the six implementation dimensions from those
|
||||
# statuses and refuses an id that file does not carry.
|
||||
#
|
||||
# What is mapped is what CARRIES the capability, never the infrastructure every
|
||||
# capability shares. core.reactive_handler, core.wiring, core.action_table,
|
||||
# core.daemon_api and bnd.ipc are deliberately absent: mapping them everywhere
|
||||
# would give all 51 rows the same status and say nothing.
|
||||
#
|
||||
# An empty list means no component carries it. That is the finding, not a hole
|
||||
# in this file.
|
||||
|
||||
# --- The turn ---
|
||||
route-an-utterance: [router.cascade, router.stage0, router.heads, router.llm, router.classifier, router.embedder, router.extractor, core.turn_route, core.topics, core.decision_trace, state.decision_ring, state.routing_traces, state.routing_labels]
|
||||
ask-instead-of-guessing: [core.preroute, state.clarify_store, state.dialogue_sessions]
|
||||
speak-as-herself: [core.phraser, core.replier, core.action_chat, core.model_seam, svc.llama_server, eval.phrasing]
|
||||
answer-from-your-own-data: [core.query_chain, core.q.embed, core.q.memory, core.q.factbykey, core.q.notes, core.q.history, core.q.list, core.q.self, core.q.personal, state.list_items]
|
||||
answer-from-the-world: [core.q.search, core.q.web, core.q.general, core.q.personal, ext.searxng]
|
||||
read-an-encyclopedia: [core.q.kiwix, ext.kiwix]
|
||||
weather: [core.q.weather, ext.openmeteo]
|
||||
see-an-image: [core.vision, state.media_blobs]
|
||||
|
||||
# --- Memory ---
|
||||
facts: [state.facts, core.action_fact, core.fact_enrichment, core.store_api]
|
||||
notes: [state.notes, core.action_note]
|
||||
recall: [core.recall, state.memory_vectors, router.embedder, core.q.memory, core.q.notes]
|
||||
memory-evaluation: [core.memory_eval]
|
||||
|
||||
# --- Proactive ---
|
||||
reminders: [state.reminders, core.action_reminder, core.dispatcher, state.delivery_attempts]
|
||||
interruption-policy: [core.rules, core.dispatcher, core.gatherer, state.presence_state, state.nudges, state.tick_memo]
|
||||
digest-of-held-nudges: [state.digest_entries, core.tick_loop, core.rules]
|
||||
morning-routine: [core.morning, core.q.dayplan]
|
||||
routine-proposals: [core.pattern, core.routines, state.proposed_routines, state.events]
|
||||
tasks: [state.tasks, core.q.tasks]
|
||||
rss-and-news: [core.feed_worker, core.q.feeds]
|
||||
|
||||
# --- Reach ---
|
||||
telegram: [core.sink_telegram, core.telegram_intake, ext.telegram, state.ack_sends]
|
||||
ntfy: [core.sink_ntfy, ext.ntfy]
|
||||
voice: [core.voice_server, bnd.voice_tcp, core.sink_voice, proc.mavenclient]
|
||||
web-ui: [proc.mavweb, bnd.http_web]
|
||||
desk-notifications: [core.event_bus, proc.mavweb]
|
||||
|
||||
# --- Speech and senses ---
|
||||
speech-to-text: [core.stt_seam, proc.mavsttd, ext.whispercpp, ext.cw2_stt, bnd.worker]
|
||||
text-to-speech: [core.tts_seam, proc.mavttsd, ext.piper, bnd.worker]
|
||||
wake-word: [proc.mavwaked, cfg.systemd, ext.alsa]
|
||||
hearing: [core.capture, state.media_blobs]
|
||||
speaker-recognition: [core.speaker]
|
||||
|
||||
# --- The ecosystem ---
|
||||
nexus: [ext.nexus, core.ecosystem, bnd.http_ecosystem, state.ecosystem_traces]
|
||||
praxis: [ext.praxis, core.ecosystem, core.praxis_acts, core.q.attention, state.surfaced_items, state.ecosystem_traces]
|
||||
hexis: [ext.hexis, core.ecosystem, core.ecosystem_hexis_gate, core.action_act, core.risk_policy, state.tools, state.pending_act, state.ecosystem_traces]
|
||||
smart-home: [ext.homeassistant, core.home_worker, core.q.home]
|
||||
network-scans: [core.netscan, core.q.network]
|
||||
# No package, no component. The finding, not an omission.
|
||||
bluetooth-control: []
|
||||
mcps: [core.mcp_worker, ext.vikunja_mcp]
|
||||
|
||||
# --- Operations ---
|
||||
the-deployed-stack: [cfg.compose, cfg.mavend, proc.mavend, proc.mavweb, proc.mavsttd, proc.mavttsd, proc.mavpoll, proc.mavgpud, ext.netdata, ext.uptimekuma]
|
||||
encrypted-database: [state.db_file, state.db, state.db_tmpfs, proc.mavseal]
|
||||
passkey-and-step-up: [state.wrapped_key, state.passkey_file, core.daemon_lock, core.auth_gate, bnd.http_web]
|
||||
model-swap: [core.modelswap, svc.llama_server]
|
||||
self-update: [proc.mavupdate]
|
||||
tests-and-analyzers: [eval.gates, eval.router, eval.phrasing]
|
||||
|
||||
# --- Undesigned in v1 ---
|
||||
email-triage: [proc.mavmaild, core.mail_intake, state.maildata]
|
||||
calendar-management: [proc.mavcaldav, core.q.calendar]
|
||||
web-crawling: [core.crawl_worker, core.q.web]
|
||||
summaries: []
|
||||
# Empty on purpose. core.telegram_intake is Telegram's own inbound channel and
|
||||
# is mapped to `telegram`. Mapping it here too would make this row read as
|
||||
# built and deployed when all three of its criteria fail on code missing.
|
||||
webhooks: []
|
||||
cron-jobs: [core.routines, core.tick_loop]
|
||||
learning-the-style: []
|
||||
# state.routing_labels holds owner corrections of a route and is deliberately
|
||||
# NOT mapped here. It is route learning, not behavioural learning, and mapping
|
||||
# it would make this row read as partially built when nothing reads it back.
|
||||
learning-from-mistakes: []
|
||||
command-chaining: []
|
||||
@@ -0,0 +1,281 @@
|
||||
# The cross-cutting rules the 51 capabilities imply
|
||||
|
||||
Hand-written. The one file in this directory that is not generated.
|
||||
|
||||
`docs/spec.md` states 51 capabilities one at a time. Twelve rules run across all
|
||||
of them, and no capability's definition of done states any of these. A rule
|
||||
broken here breaks many capabilities at once, which is why it does not show up
|
||||
as one failing criterion.
|
||||
|
||||
Each rule carries a mark. A rule split between what is written down and what
|
||||
is not carries both, and says which half is which.
|
||||
|
||||
| mark | meaning |
|
||||
| --- | --- |
|
||||
| `explicit` | a source states the rule and names its enforcement point |
|
||||
| `implied` | capabilities depend on it, no source states it, and the code decides it case by case |
|
||||
| `unresolved` | the sources do not answer it. A product question, not a defect |
|
||||
|
||||
Nothing wanted is invented where the sources are silent. An `unresolved` rule
|
||||
needs the owner, not a commit.
|
||||
|
||||
Evidence is `docs/architecture/findings.md` for the code reading,
|
||||
`docs/evals/2026-08-26-capability-baseline.md` for what ran, and the file itself
|
||||
where the rule is written down.
|
||||
|
||||
---
|
||||
|
||||
## 1. Continuity across turns and across reaches
|
||||
|
||||
**Implied.** Continuity within one reach is built. Continuity across reaches is
|
||||
not, and nothing states whether it should be.
|
||||
|
||||
`dialogue.NewPersistentSessionStore` carries follow-up slots across turns and
|
||||
across a restart. The clarify store is a per-reach stack and is deliberately not
|
||||
persisted (`findings.md` 7.5, Vikunja #385).
|
||||
|
||||
Across reaches there is no shared thread. `mavweb` hardcodes one conversation id
|
||||
for the whole web reach, which is not continuity but the absence of separation:
|
||||
a clarify parked by one probe was still parked for the next, and the first field
|
||||
run had to be discarded for it (`docs/capabilities/README.md`, "Two things the
|
||||
harness learned the hard way").
|
||||
|
||||
**What breaks:** a question asked by voice and answered on the web has no thread
|
||||
to attach to. No capability's DoD asks for one, so nothing scores this.
|
||||
|
||||
**The product question:** is a conversation per reach, or one conversation the
|
||||
reaches are windows onto?
|
||||
|
||||
## 2. Memory and correction semantics
|
||||
|
||||
**Explicit for the row, implied for the namespace.**
|
||||
|
||||
Supersede is written down and enforced: a correction points `voids_id` at the row
|
||||
it replaces, and valid-time is the `ts` column (`internal/store/schema.sql`).
|
||||
`CLAUDE.md` states the embedder contract, `EmbedQuery` and `EmbedPassage`, and
|
||||
calling plain `Embed` on a note is named as a bug.
|
||||
|
||||
Who may write a key is not written anywhere. `facts` has nine writers and no
|
||||
owner, and two of them store things that are not observations: `crawl:hash:*` is
|
||||
a fetch watermark and `cooldown:<rule>` is a tuning parameter (`findings.md`
|
||||
1.1). The `source` column keeps them apart by convention, and the `CHECK`
|
||||
constraint covers only `kind`. `notes` has six writers, one of them a LAN scan
|
||||
whose records then compete by cosine similarity with things he said
|
||||
(`findings.md` 1.2).
|
||||
|
||||
**What breaks:** recall answers a question about him with a fetch watermark.
|
||||
`queryFactByKey` reads the same table back as an answer.
|
||||
|
||||
## 3. Current context and presence
|
||||
|
||||
**Explicit and partly false at runtime.**
|
||||
|
||||
Presence is one hysteresis bucket rewritten each tick (`state.presence_state`),
|
||||
and the dispatcher's routing table is a pure function of severity and presence.
|
||||
|
||||
One input is permanently wrong. `loop.State.CalendarBusy` reads
|
||||
`facts(kind=env, source=caldav:*)` and `mavcaldav` is commented out of
|
||||
`docker-compose.yml`, so the "do not nag mid-meeting" suppressor is always false
|
||||
(`findings.md` 8.2). The compose file says so, which makes it a known gap.
|
||||
|
||||
**What breaks:** every interruption decision that should have deferred to a
|
||||
meeting. It fails open, toward interrupting.
|
||||
|
||||
## 4. Proactive attention
|
||||
|
||||
**Explicit, and the one prohibition is stated.**
|
||||
|
||||
`CLAUDE.md`: no automatic attention-to-action path. Digestion may summarise
|
||||
Praxis and may not call Hexis. At most one nudge candidate per tick, and the
|
||||
restraint gate is a pure function over the rule set (`core.rules`).
|
||||
|
||||
Restraint is decided twice on purpose (`findings.md` 2.3), and blocked candidates
|
||||
are held durably in `digest_entries` rather than dropped.
|
||||
|
||||
**What breaks:** nothing observed. This is the best-specified rule in the list.
|
||||
|
||||
## 5. Interruption policy
|
||||
|
||||
**Explicit for the choice, implied for the outcome.**
|
||||
|
||||
`docs/handler-wiring.md` owns the dispatch decision, and the table over
|
||||
(severity, presence) is pure. Delivery intent is recorded in
|
||||
`delivery_attempts` before the external send, so a crash leaves a pending row
|
||||
rather than a lost one.
|
||||
|
||||
What is not stated is what a held nudge owes the user later. `digest_entries`
|
||||
holds blocked candidates and two separate mechanisms carry the word digest
|
||||
(`findings.md` 2.4). Nothing says when a held item expires instead of
|
||||
resurfacing.
|
||||
|
||||
**The product question:** does a held nudge have a shelf life?
|
||||
|
||||
## 6. Clarification and follow-up ownership
|
||||
|
||||
**Implied.** Who owns an open question, and for how long, is decided by three
|
||||
components and stated by none.
|
||||
|
||||
`runTurn` step 1 fires an expired-clarify notice, the clarify store is a
|
||||
per-reach stack, and the pre-route ladder may claim the turn before routing
|
||||
(`findings.md` 2.2). A restart drops a parked request silently, because the
|
||||
notice path reads the store that is gone (`findings.md` 7.5).
|
||||
|
||||
Measured: one park survived five consecutive turns, turns 9 through 13, and was
|
||||
released by a path other than `отмена`
|
||||
(`docs/evals/2026-08-26-capability-baseline.md`).
|
||||
|
||||
**What breaks:** a question she asked stays open across unrelated turns, and
|
||||
neither the ladder nor the store says whose job it is to close it.
|
||||
|
||||
## 7. Degradation and honesty
|
||||
|
||||
**Explicit as a rule, and the rule contradicts itself in practice.**
|
||||
|
||||
`CLAUDE.md` states both halves. Fall back silently when the fallback would only
|
||||
do the job better. Name the gap when the resident model cannot do the job at
|
||||
all. `docs/spec.md` writes every v1 DoD at "voice-reachable and honest", where
|
||||
honest means naming the gap and never filling it with a guess.
|
||||
|
||||
Four silent degradations stack on one turn: workstation model to resident model,
|
||||
CW2 to mavsttd, routing heads to LLM router to classifier, and search to Kiwix to
|
||||
a named page to the model's own weights (`findings.md` 8.1). Each is argued
|
||||
individually. Together a reply can be the resident model routing a worse
|
||||
transcript with the classifier as a floor, answering from its weights, and
|
||||
nothing in the reply distinguishes that from the best case.
|
||||
|
||||
**What breaks:** the boundary between "only better" and "cannot do the job" is
|
||||
not drawn anywhere, so the stack decides it by accident.
|
||||
|
||||
**The product question:** at what depth of fallback does silence stop being
|
||||
honest?
|
||||
|
||||
## 8. Authority and confirmation
|
||||
|
||||
**Unresolved, and this is the largest hole in the list.**
|
||||
|
||||
Two systems each answer half and never meet. `internal/auth` answers who may
|
||||
carry what authority and does not bind the reactive turn path at all
|
||||
(`findings.md` 6.3). `internal/tool` answers what effect a capability has and
|
||||
what proof it demands, runs on every act, and is not keyed on the reach
|
||||
(`findings.md` 6.3b). Neither has the other's reach.
|
||||
|
||||
Two representations of reach exist and both are ignored.
|
||||
`internal/voice/server.go:198` defaults an empty `p.Surface` and a
|
||||
client-asserted one survives to a handler that never reads it. `:148` hardcodes
|
||||
`SurfacePCClient` for every connection. `req.Surface` is request payload on a
|
||||
plaintext wire with no auth, so any client can claim `pc_client`. It must not
|
||||
become an authorization input as it stands.
|
||||
|
||||
One path has no gate at all. `praxisItemAction.handle`
|
||||
(`cmd/mavend/ecosystem_acts.go:158`) reads `dec.Slots.Value` and calls straight
|
||||
through. Acknowledge, resolve, ignore and pin are remote mutations that run on
|
||||
first hearing, with no tier and no confirm turn.
|
||||
|
||||
`CLAUDE.md` states the rule the code does not implement: LLM output is not
|
||||
authorization, and a confirmation binds capability id, target entity, arguments,
|
||||
requester and expiry.
|
||||
|
||||
**What breaks:** no one can currently state the authority property of a Maven
|
||||
turn. `origin × effect × evidence → permit` is the target shape and nothing
|
||||
computes it.
|
||||
|
||||
## 9. Privacy boundaries
|
||||
|
||||
**Explicit, and it is the best-enforced rule here.**
|
||||
|
||||
`CLAUDE.md`: the owner's data first, then the world. His notes and facts are
|
||||
never search input, only the utterance leaves the box. The personal boundary is a
|
||||
query source with `boundary: true`, and `queryWalk` reads
|
||||
`Decision.SourceAnchored` for that source and no other.
|
||||
|
||||
The exception is deliberate and recorded. The boundary guesses, so naming
|
||||
`SourceWorld` drops it, and only a stage 0 grammar may do that (owner's call,
|
||||
V-666). No component reads another component's database, and Praxis attention
|
||||
comes over HTTP rather than from its SQLite file.
|
||||
|
||||
**What breaks:** nothing observed. The one caveat is that `queryWalk` takes
|
||||
sources out and moves none, which is the safety argument, and it holds only as
|
||||
long as the table's order stays load-bearing.
|
||||
|
||||
## 10. Learning from outcomes
|
||||
|
||||
**Unresolved.** One loop exists, two are specified with no package, and nothing
|
||||
says whether learning is a product goal.
|
||||
|
||||
Built: `state.nudges` is the restraint memory and the only input to the tick
|
||||
loop's autotune, which writes `cooldown:<rule>` back into `facts`.
|
||||
`state.routing_labels` holds owner corrections of a route.
|
||||
|
||||
Not built: `learning-the-style` and `learning-from-mistakes` have no package and
|
||||
no component. `docs/spec.md` gives each a DoD written at what done would look
|
||||
like. Both score `code_present: no`.
|
||||
|
||||
One criterion passes by absence. "No model weights change and no training set is
|
||||
built" is a negative, and nothing being built satisfies it. The generator reports
|
||||
this as an anomaly rather than counting it as progress.
|
||||
|
||||
**The product question:** is behavioural learning wanted, or is the negative
|
||||
criterion the whole of the intent?
|
||||
|
||||
## 11. Capability composition
|
||||
|
||||
**Implied and absent.** Every capability is specified alone and the turn is
|
||||
single-claim by construction.
|
||||
|
||||
`core.action_table` dispatches one intent to one handler, and a handler returning
|
||||
the empty string hands the turn on. `queryWalk` stops at the first source that
|
||||
claims. Two independent arbitrations already decide one turn, with a third
|
||||
running before both (`findings.md` 2.1, 2.2).
|
||||
|
||||
`command-chaining` fails all three of its criteria with reason `wiring missing`.
|
||||
The `chain` in `internal/router` is the world chain and the source chain, not
|
||||
command chaining.
|
||||
|
||||
`internal/claim` is the beginning of a vocabulary for this and is called by
|
||||
nothing (`findings.md` 6.1). It carries a live defect: `Claim.Coverage` returns
|
||||
1.0 for a claim that extracted nothing, because `claimSpans` includes
|
||||
`Slots.Text` unconditionally and `fillSlots` backfills the raw utterance into
|
||||
`Text` (`findings.md` 6.3d).
|
||||
|
||||
**What breaks:** "напомни мне и запиши это" performs one of the two and says
|
||||
nothing about the other.
|
||||
|
||||
**The product question:** what is the single unit that competes for a turn. This
|
||||
is one of the three the freeze was called to answer.
|
||||
|
||||
## 12. Persistence across restart
|
||||
|
||||
**Implied.** Four stores made four different choices and no source states the
|
||||
rule.
|
||||
|
||||
| state | survives a restart | evidence |
|
||||
| --- | --- | --- |
|
||||
| dialogue sessions | yes | `dialogue.NewPersistentSessionStore` |
|
||||
| clarify store | no, deliberately | `findings.md` 7.5, Vikunja #385 |
|
||||
| decision ring | no, in-memory bounded at 25 | `internal/decision/ring.go:11` |
|
||||
| routing traces | yes, retained 14 days | `CLAUDE.md` |
|
||||
| tick memo | no, in-process and argued for one field | `findings.md` 7.4 |
|
||||
| surfaced items | no, and no TTL | `findings.md` 7.3 |
|
||||
|
||||
Two of these are principled. The decision ring holds his words and is bounded on
|
||||
purpose. The clarify store's reasoning is filed. The other four are not decided
|
||||
anywhere.
|
||||
|
||||
**What breaks:** less than it looks. `surfacedItems` has no TTL, and the source
|
||||
comment argues that a stale ordinal resolves to an item Praxis reports as already
|
||||
acknowledged, which is harmless because Praxis is the arbiter (`findings.md`
|
||||
7.3). The cost is that the same absence of a written rule produced one argued
|
||||
choice and three unargued ones.
|
||||
|
||||
---
|
||||
|
||||
## What this file is for
|
||||
|
||||
Session 2 step 2 of `docs/plans/26-capability-ledger-and-baseline.md`. It feeds
|
||||
`docs/capabilities/gaps.md`, where every architecture concern must name the
|
||||
capability or invariant it affects.
|
||||
|
||||
Four rules are `unresolved` and they are the owner's, not a commit's: authority
|
||||
and confirmation, learning from outcomes, capability composition, and the shelf
|
||||
life of a held nudge. Two of the three questions the freeze was called to answer
|
||||
appear here as invariant 8 and invariant 11.
|
||||
@@ -0,0 +1,91 @@
|
||||
# The structured half of docs/capabilities/invariants.md. HAND-WRITTEN.
|
||||
#
|
||||
# The prose, the evidence and the reasoning live in the .md. This file carries
|
||||
# only what a machine needs: the mark, which capabilities the rule touches, and
|
||||
# which components participate in it. build_ledger.py checks the two agree, so
|
||||
# an invariant cannot exist in one and not the other.
|
||||
#
|
||||
# mark: explicit | implied | unresolved. `split` means the rule is written down
|
||||
# in one half and not in the other, and the .md says which half is which.
|
||||
|
||||
invariants:
|
||||
- id: 1
|
||||
title: Continuity across turns and across reaches
|
||||
mark: implied
|
||||
question: Is a conversation per reach, or one conversation the reaches are windows onto?
|
||||
capabilities: [web-ui, voice, telegram, ask-instead-of-guessing]
|
||||
components: [state.dialogue_sessions, state.clarify_store, proc.mavweb, core.voice_server, core.sink_telegram]
|
||||
|
||||
- id: 2
|
||||
title: Memory and correction semantics
|
||||
mark: explicit
|
||||
split: true
|
||||
capabilities: [facts, notes, recall]
|
||||
components: [state.facts, state.notes, state.memory_vectors, core.recall, router.embedder, core.q.factbykey, core.fact_enrichment, core.netscan]
|
||||
|
||||
- id: 3
|
||||
title: Current context and presence
|
||||
mark: explicit
|
||||
capabilities: [interruption-policy, calendar-management, morning-routine]
|
||||
components: [state.presence_state, core.gatherer, core.q.calendar, proc.mavcaldav, core.dispatcher]
|
||||
|
||||
- id: 4
|
||||
title: Proactive attention
|
||||
mark: explicit
|
||||
capabilities: [interruption-policy, digest-of-held-nudges, routine-proposals, praxis]
|
||||
components: [core.tick_loop, core.rules, state.digest_entries, state.nudges, core.q.attention, ext.praxis]
|
||||
|
||||
- id: 5
|
||||
title: Interruption policy
|
||||
mark: explicit
|
||||
split: true
|
||||
question: Does a held nudge have a shelf life?
|
||||
capabilities: [interruption-policy, digest-of-held-nudges, telegram, ntfy]
|
||||
components: [core.dispatcher, state.delivery_attempts, state.digest_entries, core.rules, core.sink_telegram, core.sink_ntfy, core.sink_voice]
|
||||
|
||||
- id: 6
|
||||
title: Clarification and follow-up ownership
|
||||
mark: implied
|
||||
capabilities: [ask-instead-of-guessing, route-an-utterance]
|
||||
components: [core.preroute, state.clarify_store, state.dialogue_sessions, core.turn_route]
|
||||
|
||||
- id: 7
|
||||
title: Degradation and honesty
|
||||
mark: explicit
|
||||
split: true
|
||||
question: At what depth of fallback does silence stop being honest?
|
||||
capabilities: [answer-from-the-world, speech-to-text, route-an-utterance, speak-as-herself, read-an-encyclopedia, weather]
|
||||
components: [core.model_seam, core.stt_seam, router.cascade, router.classifier, core.query_chain, core.phraser, core.q.kiwix, core.q.general]
|
||||
|
||||
- id: 8
|
||||
title: Authority and confirmation
|
||||
mark: unresolved
|
||||
question: Where is the one point that decides whether this origin may perform this effect with this evidence?
|
||||
capabilities: [hexis, praxis, voice, passkey-and-step-up, encrypted-database]
|
||||
components: [core.auth_gate, core.risk_policy, state.pending_act, state.tools, core.action_act, core.ecosystem_hexis_gate, core.praxis_acts, core.voice_server, bnd.voice_tcp, core.daemon_lock]
|
||||
|
||||
- id: 9
|
||||
title: Privacy boundaries
|
||||
mark: explicit
|
||||
capabilities: [answer-from-the-world, answer-from-your-own-data, recall, read-an-encyclopedia]
|
||||
components: [core.q.personal, core.query_chain, core.q.search, core.q.kiwix, bnd.http_ecosystem]
|
||||
|
||||
- id: 10
|
||||
title: Learning from outcomes
|
||||
mark: unresolved
|
||||
question: Is behavioural learning wanted, or is the negative criterion the whole of the intent?
|
||||
capabilities: [learning-the-style, learning-from-mistakes, interruption-policy, route-an-utterance]
|
||||
components: [state.nudges, state.routing_labels, core.tick_loop, core.rules]
|
||||
|
||||
- id: 11
|
||||
title: Capability composition
|
||||
mark: implied
|
||||
question: What is the single unit that competes for a turn?
|
||||
capabilities: [command-chaining, route-an-utterance, answer-from-your-own-data, ask-instead-of-guessing]
|
||||
components: [core.action_table, core.query_chain, core.preroute, router.cascade, router.stage0, router.claim, router.modes]
|
||||
|
||||
- id: 12
|
||||
title: Persistence across restart
|
||||
mark: implied
|
||||
capabilities: [ask-instead-of-guessing, praxis, route-an-utterance]
|
||||
components: [state.dialogue_sessions, state.clarify_store, state.decision_ring, state.routing_traces, state.tick_memo, state.surfaced_items]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"facts": {
|
||||
"count": 13041
|
||||
},
|
||||
"notes": {
|
||||
"count": 185
|
||||
},
|
||||
"reminders": {
|
||||
"count": 98
|
||||
},
|
||||
"pending_reminders": {
|
||||
"error": "e2eprobe: ipc: unknown method:"
|
||||
},
|
||||
"tasks_live": {
|
||||
"count": 6
|
||||
},
|
||||
"nudges": {
|
||||
"count": 5
|
||||
},
|
||||
"tools": {
|
||||
"count": 13
|
||||
},
|
||||
"decisions": {
|
||||
"count": 25
|
||||
},
|
||||
"events": {
|
||||
"count": 62
|
||||
},
|
||||
"eco_traces": {
|
||||
"count": 115
|
||||
},
|
||||
"delivery_attempts": {
|
||||
"count": 200,
|
||||
"note": "at e2eprobe's hardcoded 200 cap, true count is >= 200"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"facts": {
|
||||
"count": 1
|
||||
},
|
||||
"notes": {
|
||||
"count": 4
|
||||
},
|
||||
"reminders": {
|
||||
"count": 0
|
||||
},
|
||||
"pending_reminders": {
|
||||
"error": "e2eprobe: ipc: unknown method:"
|
||||
},
|
||||
"tasks_live": {
|
||||
"count": 0
|
||||
},
|
||||
"nudges": {
|
||||
"count": 0
|
||||
},
|
||||
"tools": {
|
||||
"count": 12
|
||||
},
|
||||
"decisions": {
|
||||
"count": 0
|
||||
},
|
||||
"events": {
|
||||
"count": 4
|
||||
},
|
||||
"eco_traces": {
|
||||
"count": 0
|
||||
},
|
||||
"delivery_attempts": {
|
||||
"count": 0
|
||||
}
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,34 @@
|
||||
field:reminder:create-and-persist напомни мне завтра в 9 утра позвонить в клинику хорошо, напомню завтра в 09:00. 0.29
|
||||
field:reminder:clarify-the-missing-slot напомни мне позвонить маме Сейчас 01:09. Когда? 0.26
|
||||
field:reminder:clarify-the-missing-slot в шесть вечера Сейчас 01:09. Поняла: в шесть вечера. В какой день? 0.62
|
||||
field:reminder:cancel-mid-clarify напомни мне заехать за кормом Сейчас 01:09. Когда? 0.26
|
||||
field:reminder:cancel-mid-clarify отмена Хорошо, отменила. 0.27
|
||||
field:reminder:correct-the-time напомни мне в семь вечера вынести мусор Сейчас 01:09. В какой день? 0.3
|
||||
field:reminder:correct-the-time нет, в восемь Сейчас 01:09. В какой день? 0.6
|
||||
field:reminder:recurring-from-speech напоминай мне каждый день в девять вечера дать собаке таблетку Сейчас 01:09. В какой день? 0.29
|
||||
field:reminder:recurring-from-speech какие у меня повторяющиеся напоминания? Все напоминания: 0. На какое время поставить напоминание? memory 3.36
|
||||
field:fact:record-and-confirm меня зовут Ками У тебя есть 35 сообщений. 4.14
|
||||
field:fact:supersede я работаю в Тинькофф не знаю — не нашла у тебя такой записи. personal 0.12
|
||||
field:fact:supersede я больше не работаю в Тинькофф, теперь я в Яндексе не знаю — не нашла у тебя такой записи. personal 0.13
|
||||
field:fact:supersede где я работаю? не знаю — не нашла у тебя такой записи. personal 0.1
|
||||
field:fact:what-do-you-remember что ты помнишь обо мне? Мне известно о тебе с 2017 года. В настоящее время у меня нет информации об удалении работы в Тинькофф и возобновлений работ по этой компании. 2.98
|
||||
field:note:question-is-not-a-statement я рассказывал тебе про байкал? Давайте проверим. memory 4.78
|
||||
field:note:capture-then-recall запиши: ключи от гаража лежат в верхнем ящике стола "Записала", "проверила" 2.65
|
||||
field:note:capture-then-recall где лежат ключи от гаража? В главе с Долорес он спрашивает: «где лежат ключи от гаража?» Источник утверждает, что они находятся рядом со кухней на первом этаже. search 7.52
|
||||
field:note:recall-miss-is-honest что я говорил тебе про ремонт балкона? Уточни пожалуйста имя человека из которого ты будешь учиться или посмотреть информацию. memory 2.4
|
||||
field:query:own-data-first что у меня сегодня по плану? на 26.08.2026 ничего не запланировано. day-plan 0.01
|
||||
field:query:the-world-summary почему небо голубое? Почему небо голубое? - В статье разберемся, почему днем небо кажется голубым (см., например). Рассказываем про солнечный свет. search 6.29
|
||||
field:query:arithmetic-must-not-escape сколько будет два плюс два? Похоже у вас есть интерес к математике! 2 + 2 = ? Это довольно простая задачка для ребенка с первого класса, а вы можете продолжить? Или вам нужны советы по образованию? search 8.17
|
||||
field:query:follow-up-context какая сейчас погода? для какого города? weather 0.17
|
||||
field:query:follow-up-context Самара Я тебя не разобрала. Повтори, пожалуйста. 0.04
|
||||
field:query:encyclopedia-language расскажи про Байкал Всего лишь одна фразаЕсли у тебя есть желание узнать больше о Байкале или его влиянии как гидроэнергетического комплекс, посетите сайт «Энциклопедии Кирилла и Мефодия» https://www.km.ru/encyclopedia/view.php?ID=602. memory 5.0
|
||||
field:query:feeds-on-request что нового в новостях? вот что нового: mklinux-v7.0-mk2 released; [$] Old-school calendaring at the command line with Remind; Vanilla OS 3 released feeds 1.67
|
||||
field:query:network-scan-as-prose какие устройства сейчас в сети? нашла 3 устройства, из них 2 с вебом, 2 с ssh. список записала. network 27.01
|
||||
field:action:target-must-resolve перезапусти контейнер с базой «с» — не знаю такой цели. назови её как в системе. 0.02
|
||||
field:action:confirm-binds выключи свет в спальне «с» — не знаю такой цели. назови её как в системе. 0.04
|
||||
field:action:report-the-actual-result покажи что запущено в докере «с» — не знаю такой цели. назови её как в системе. 0.06
|
||||
field:action:praxis-lifecycle-gate что требует моего внимания? Нет ничего нового. 3.66
|
||||
field:action:praxis-lifecycle-gate отметь первое как выполненное Такую команду я не знаю. 0.04
|
||||
field:action:capture-a-task добавь в задачи: заказать корм для собаки внесла в список: заказать корм для собаки 0.01
|
||||
field:action:capture-a-task какие у меня задачи? сначала: проверить интейк форму на боксе (просрочено на 17 дней); купить хлеб (давно в списке); купить молоко (давно в списке); полить цветы (давно в списке); починить кран (давно в списке); и ещё 1 задача. 1 задача просрочено; 5 задач лежит дольше 10 дней. tasks 0.01
|
||||
field:proactive:delivery-reaches-telegram напомни мне через две минуты MVNPROBE проверка доставки хорошо, напомню сегодня в 01:13. 0.33
|
||||
|
Can't render this file because it contains an unexpected character in line 16 and column 143.
|
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"note": "field probes: the owner's real week, drawn from the five vertical slices, the three ugly conversations of the 2026-08-13 audit verbatim, the five existing scenarios and what deploy/mavend.json is actually configured for. A failing field probe is an unresolved product question or a missing-criterion finding against docs/spec.md, never a DoD verdict.",
|
||||
"probes": [
|
||||
|
||||
{ "id": "field:reminder:create-and-persist", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне завтра в 9 утра позвонить в клинику"],
|
||||
"readback": {"reminders": ["reminders", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "The reply states the time back, and a reminders row exists for 09:00 tomorrow with that text." },
|
||||
|
||||
{ "id": "field:reminder:clarify-the-missing-slot", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне позвонить маме", "в шесть вечера"],
|
||||
"readback": {"reminders": ["reminders", "10"], "decisions": ["decisions", "2"]},
|
||||
"expect": "Turn 1 asks only for the time, not for all slots. Turn 2 fills it and a row lands at 18:00. The audit saw this pass; it is here to catch a regression." },
|
||||
|
||||
{ "id": "field:reminder:cancel-mid-clarify", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне заехать за кормом", "отмена"],
|
||||
"readback": {"reminders": ["reminders", "10"]},
|
||||
"expect": "Turn 2 drops the parked turn and says so. No reminder row is written for the kibble." },
|
||||
|
||||
{ "id": "field:reminder:correct-the-time", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне в семь вечера вынести мусор", "нет, в восемь"],
|
||||
"readback": {"reminders": ["reminders", "10"]},
|
||||
"expect": "One reminder at 20:00, not two rows and not one at 19:00. Correction of a just-set reminder is the slice's real shape." },
|
||||
|
||||
{ "id": "field:reminder:recurring-from-speech", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напоминай мне каждый день в девять вечера дать собаке таблетку", "какие у меня повторяющиеся напоминания?"],
|
||||
"readback": {"reminders": ["reminders", "20"]},
|
||||
"expect": "The schedule is stated back and a row carries Cron. The spec says storage and delivery are finished and no caller in cmd/mavend passes a cron, so this is expected to fail; the probe records HOW it fails." },
|
||||
|
||||
{ "id": "field:fact:record-and-confirm", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["меня зовут Ками"],
|
||||
"readback": {"facts": ["facts", "20"], "decisions": ["decisions", "1"]},
|
||||
"expect": "The confirmation is feminine, and a facts row actually exists. The audit caught 'я записала информацию о тебе' confirming a write that never happened." },
|
||||
|
||||
{ "id": "field:fact:supersede", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["я работаю в Тинькофф", "я больше не работаю в Тинькофф, теперь я в Яндексе", "где я работаю?"],
|
||||
"readback": {"facts": ["facts", "30"]},
|
||||
"expect": "Turn 3 answers Yandex, not Tinkoff and not both. Both rows are readable and the old value is retired." },
|
||||
|
||||
{ "id": "field:fact:what-do-you-remember", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["что ты помнишь обо мне?"],
|
||||
"readback": {"decisions": ["decisions", "1"], "notes": ["notes", "5"]},
|
||||
"expect": "Routes to query, not remember, and the reply uses informal singular ты. The planning session got formal вас/вы here while the phrasing eval passed. Two of the audit's seven probes misrouted on exactly this shape." },
|
||||
|
||||
{ "id": "field:note:question-is-not-a-statement", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["я рассказывал тебе про байкал?"],
|
||||
"readback": {"notes": ["notes", "5"], "decisions": ["decisions", "1"]},
|
||||
"expect": "No note row is written. The audit stored this question as a statement, and the personal boundary scored it as world." },
|
||||
|
||||
{ "id": "field:note:capture-then-recall", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["запиши: ключи от гаража лежат в верхнем ящике стола", "где лежат ключи от гаража?"],
|
||||
"readback": {"notes": ["notes", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "Turn 2 returns the drawer from the note captured in turn 1, not a world answer and not a recall miss." },
|
||||
|
||||
{ "id": "field:note:recall-miss-is-honest", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["что я говорил тебе про ремонт балкона?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "She says she does not remember. A world answer here is the failure: the personal boundary must stop a question about him from reaching outside." },
|
||||
|
||||
{ "id": "field:query:own-data-first", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["что у меня сегодня по плану?"],
|
||||
"readback": {"decisions": ["decisions", "1"], "plan": ["plan"]},
|
||||
"expect": "The real checklist and its open items come back, and the decision trace shows an owner source winning before any world source was asked." },
|
||||
|
||||
{ "id": "field:query:the-world-summary", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["почему небо голубое?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "A Russian summary that does not invent physics. The audit's answer was 'корочковатые цветы отражают длинноволны'. Verbatim from the audit, so the two are comparable." },
|
||||
|
||||
{ "id": "field:query:arithmetic-must-not-escape", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["сколько будет два плюс два?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "The answer is 4. Measured this session: the arithmetic-query stage 0 grammar declined, the turn reached external search, and the reply was 'Во-первых - это два плюса двойки'." },
|
||||
|
||||
{ "id": "field:query:follow-up-context", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["какая сейчас погода?", "Самара"],
|
||||
"readback": {"decisions": ["decisions", "2"]},
|
||||
"expect": "Turn 2 is understood as the city for turn 1. The audit's follow-up died with 'Я тебя не разобрала'. There is no weather block in the config, so a named gap is the honest pass and a guessed forecast is the failure." },
|
||||
|
||||
{ "id": "field:query:encyclopedia-language", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["расскажи про Байкал"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "A Russian question lands on the Russian book. The claiming source in the trace is kiwix, and the article is about the lake." },
|
||||
|
||||
{ "id": "field:query:feeds-on-request", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["что нового в новостях?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "Configured feed items are read back. Two sources are configured; a dead feed must name itself dead and the other still answer." },
|
||||
|
||||
{ "id": "field:query:network-scan-as-prose", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["какие устройства сейчас в сети?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "Hosts and open ports come back as prose, not as a table dump. netscan.enabled is true with subnets, ports and rate set." },
|
||||
|
||||
{ "id": "field:action:target-must-resolve", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["перезапусти контейнер с базой"],
|
||||
"readback": {"eco-traces": ["eco-traces", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "Free text does not reach a mutating Hexis call. Nexus holds no entities, so the honest outcome is a named gap or a clarify, never a guessed target and never an execution." },
|
||||
|
||||
{ "id": "field:action:confirm-binds", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["выключи свет в спальне"],
|
||||
"readback": {"eco-traces": ["eco-traces", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "smarthome.enabled is false, so this is a named gap. The failure is a reply that claims the light was switched." },
|
||||
|
||||
{ "id": "field:action:report-the-actual-result", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["покажи что запущено в докере"],
|
||||
"readback": {"eco-traces": ["eco-traces", "10"], "tools": ["tools"]},
|
||||
"expect": "Either the real container list or a named gap. A plausible invented list is the failure this slice exists to catch." },
|
||||
|
||||
{ "id": "field:action:praxis-lifecycle-gate", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["что требует моего внимания?", "отметь первое как выполненное"],
|
||||
"readback": {"eco-traces": ["eco-traces", "20"], "decisions": ["decisions", "2"]},
|
||||
"expect": "Turn 1 calls Surface, never Acknowledge. Turn 2 is a remote mutation: the architecture pass found praxisItemAction.handle calls straight through with no tier and no confirm turn. The probe records whether it runs on first hearing." },
|
||||
|
||||
{ "id": "field:action:capture-a-task", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["добавь в задачи: заказать корм для собаки", "какие у меня задачи?"],
|
||||
"readback": {"tasks": ["tasks"], "decisions": ["decisions", "2"]},
|
||||
"expect": "The task appears in the list, ordered by deadline and urgency, and the capture is visible over IPC." },
|
||||
|
||||
{ "id": "field:proactive:delivery-reaches-telegram", "origin": "field", "kind": "live", "slice": "proactive",
|
||||
"utterances": ["напомни мне через две минуты MVNPROBE проверка доставки"],
|
||||
"readback": {"reminders": ["reminders", "10"], "delivery-attempts": ["delivery-attempts"]},
|
||||
"expect": "Within a compressed live horizon the reminder fires and a delivery-attempts row records the reach that took it. TickInterval defaults to 60s and no override is configured, so two minutes is two ticks. The marker keeps it from reading as a real nudge." },
|
||||
|
||||
{ "id": "field:proactive:failure-retries-into-another-reach", "origin": "field", "kind": "live", "slice": "proactive",
|
||||
"utterances": [],
|
||||
"readback": {"delivery-attempts": ["delivery-attempts"], "nudges": ["nudges", "20"], "events": ["events", "50"]},
|
||||
"expect": "Read the existing attempt rows: a failed delivery retried into another reach rather than looping. The audit watched the ntfy failure run once a minute until 03:05, then Telegram took reminder #83 at 03:06 and nothing retried." }
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Drive probes through the deployed stack and record what came back.
|
||||
|
||||
Runs ON homesrv, where 127.0.0.1:9201 is mavweb and the mavend socket is
|
||||
reachable from inside maven-mavend-1.
|
||||
|
||||
Transport is POST /api/chat, which runs a real turn: router, resident model,
|
||||
query walk, act path. The reply rides back on the 303 Location as
|
||||
?q=..&r=..&s=<claiming query source>&t=<trace id>.
|
||||
|
||||
Readback is e2eprobe over the IPC socket, never the plaintext sqlite copy.
|
||||
|
||||
python3 run_probes.py probes.json > raw.jsonl
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
|
||||
WEB = "http://127.0.0.1:9201"
|
||||
SOCK = "/run/maven/mavend.sock"
|
||||
CONTAINER = "maven-mavend-1"
|
||||
PROBE_BIN = "/tmp/e2eprobe"
|
||||
|
||||
|
||||
def sh(args, timeout=180):
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
|
||||
|
||||
def chat(text):
|
||||
"""One turn. Returns the parsed redirect, or the failure verbatim."""
|
||||
t0 = time.time()
|
||||
rc, out, err = sh([
|
||||
"curl", "-s", "-o", "/dev/null", "-w", "%{http_code}\t%{redirect_url}",
|
||||
"-X", "POST", "--data-urlencode", f"text={text}", f"{WEB}/api/chat",
|
||||
])
|
||||
dt = round(time.time() - t0, 2)
|
||||
if rc != 0:
|
||||
return {"utterance": text, "error": err.strip() or f"curl rc={rc}", "seconds": dt}
|
||||
code, _, loc = out.partition("\t")
|
||||
r = {"utterance": text, "http": code, "seconds": dt}
|
||||
if code != "303":
|
||||
r["error"] = f"expected 303, got {code}"
|
||||
return r
|
||||
q = urllib.parse.parse_qs(urllib.parse.urlparse(loc).query)
|
||||
r["reply"] = q.get("r", [""])[0]
|
||||
r["source"] = q.get("s", [""])[0]
|
||||
r["trace"] = q.get("t", [""])[0]
|
||||
return r
|
||||
|
||||
|
||||
def probe(cmd):
|
||||
"""One e2eprobe readback. Returns parsed JSON or the error verbatim."""
|
||||
rc, out, err = sh(["docker", "exec", CONTAINER, PROBE_BIN, "-sock", SOCK] + cmd)
|
||||
if rc != 0:
|
||||
return {"error": (err or out).strip()}
|
||||
try:
|
||||
return json.loads(out)
|
||||
except json.JSONDecodeError:
|
||||
return {"raw": out.strip()}
|
||||
|
||||
|
||||
def shell(cmd):
|
||||
"""One read-only command on homesrv, for a configuration or deployment fact.
|
||||
|
||||
A criterion about whether a config block exists is not observable through a
|
||||
turn, and chasing it through one measures the router instead.
|
||||
"""
|
||||
t0 = time.time()
|
||||
p = subprocess.run(["sh", "-c", cmd], capture_output=True, text=True, timeout=300)
|
||||
return {
|
||||
"cmd": cmd,
|
||||
"rc": p.returncode,
|
||||
"stdout": p.stdout[-8000:],
|
||||
"stderr": p.stderr[-2000:],
|
||||
"seconds": round(time.time() - t0, 2),
|
||||
}
|
||||
|
||||
|
||||
def normalise_readback(rb):
|
||||
"""Accept both shapes: {name: argv} and [{name, argv}]."""
|
||||
if isinstance(rb, dict):
|
||||
return list(rb.items())
|
||||
return [(d["name"], d["argv"]) for d in (rb or [])]
|
||||
|
||||
|
||||
def reset():
|
||||
"""Clear any parked clarify before the next probe.
|
||||
|
||||
mavweb hardcodes one conversation id for the whole web reach, so a clarify
|
||||
parked by one probe is still parked for the next one. Measured: an
|
||||
unanswered park appended "Сейчас 01:07. В какой день?" to eleven unrelated
|
||||
turns in a row, including plain statements the router should have taken as
|
||||
facts. Without this the run measures the previous probe, not this one.
|
||||
|
||||
The leak itself is a finding, recorded separately from a run that isolates.
|
||||
"""
|
||||
r = chat("отмена")
|
||||
return {"reply": r.get("reply", ""), "http": r.get("http", ""), "error": r.get("error", "")}
|
||||
|
||||
|
||||
def main():
|
||||
spec = json.load(open(sys.argv[1]))
|
||||
for p in spec["probes"]:
|
||||
rec = {
|
||||
"id": p["id"],
|
||||
"origin": p.get("origin", "dod" if p["id"].startswith("dod:") else "field"),
|
||||
"criteria": p.get("criteria", []),
|
||||
"slice": p.get("slice", ""),
|
||||
"expect": p.get("expect", ""),
|
||||
"kind": p.get("kind", "live"),
|
||||
"method": p.get("method", "chat"),
|
||||
"turns": [],
|
||||
"readback": {},
|
||||
}
|
||||
if rec["kind"] == "blocked" or rec["method"] == "none":
|
||||
rec["blocked_reason"] = p.get("blocked_reason", "")
|
||||
print(json.dumps(rec, ensure_ascii=False), flush=True)
|
||||
continue
|
||||
if p.get("isolate", True) and rec["method"] == "chat":
|
||||
rec["reset_before"] = reset()
|
||||
time.sleep(1)
|
||||
if rec["method"] == "shell":
|
||||
rec["shell"] = shell(p["shell"])
|
||||
for text in p.get("utterances", []):
|
||||
rec["turns"].append(chat(text))
|
||||
time.sleep(p.get("gap", 1))
|
||||
for name, cmd in normalise_readback(p.get("readback")):
|
||||
rec["readback"][name] = probe(cmd)
|
||||
print(json.dumps(rec, ensure_ascii=False), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Row counts per store, read over IPC. Runs on homesrv.
|
||||
|
||||
The eval's first artifact. Read before any probe writes, and again after, so
|
||||
what the probes added is attributable. Not the sqlite file: the point is to
|
||||
measure the contract, and the plaintext copy in tmpfs bypasses it.
|
||||
"""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
C = "maven-mavend-1"
|
||||
S = "/run/maven/mavend.sock"
|
||||
BIN = "/tmp/e2eprobe"
|
||||
|
||||
# command, argv, and whether the reply is capped by a limit e2eprobe hardcodes.
|
||||
READS = [
|
||||
("facts", ["facts", "100000"], False),
|
||||
("notes", ["notes", "100000"], False),
|
||||
("reminders", ["reminders", "100000"], False),
|
||||
("pending_reminders", ["pending-reminders", "100000"], False),
|
||||
("tasks_live", ["tasks"], False),
|
||||
("nudges", ["nudges", "100000"], False),
|
||||
("tools", ["tools"], False),
|
||||
("decisions", ["decisions", "100000"], False),
|
||||
("events", ["events", "100000"], False),
|
||||
("eco_traces", ["eco-traces", "100000"], False),
|
||||
("delivery_attempts", ["delivery-attempts"], True), # capped at 200 in e2eprobe
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
out = {}
|
||||
for name, argv, capped in READS:
|
||||
p = subprocess.run(["docker", "exec", C, BIN, "-sock", S] + argv,
|
||||
capture_output=True, text=True, timeout=180)
|
||||
if p.returncode != 0:
|
||||
out[name] = {"error": (p.stderr or p.stdout).strip()[:300]}
|
||||
continue
|
||||
try:
|
||||
d = json.loads(p.stdout)
|
||||
except json.JSONDecodeError:
|
||||
out[name] = {"error": "not json", "raw": p.stdout.strip()[:300]}
|
||||
continue
|
||||
if d is None:
|
||||
out[name] = {"count": 0}
|
||||
elif isinstance(d, list):
|
||||
r = {"count": len(d)}
|
||||
if capped and len(d) >= 200:
|
||||
r["note"] = "at e2eprobe's hardcoded 200 cap, true count is >= 200"
|
||||
out[name] = r
|
||||
else:
|
||||
out[name] = {"value": d}
|
||||
json.dump(out, sys.stdout, ensure_ascii=False, indent=2)
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user