Compare commits
91 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1a309b283b | |||
| be71ac406b | |||
| bb6bd8efb9 | |||
| 29f74dd3cc | |||
| bdced83b46 | |||
| fa98e4722e | |||
| de1cb40456 | |||
| dc73fc4e31 | |||
| cf2af6ce1e | |||
| 5397f09785 | |||
| 7d31de5a6b | |||
| 20215650f2 | |||
| f2b65cd5d2 | |||
| 9a25945567 | |||
| e80d45f4b3 | |||
| 4ed0238a25 | |||
| 774c0660b0 | |||
| cbac8b9c4f | |||
| ad3f2d6e99 | |||
| 87411e5fb1 | |||
| 4bb7555459 | |||
| 9df223907b | |||
| f8ec77de8a | |||
| 6397ea1cc9 | |||
| 07bfcea8cc | |||
| 59a0a08d32 | |||
| 1b8ae3c3e5 | |||
| 56051e58c0 | |||
| d63619bd0e | |||
| fb9b719f0b | |||
| 2b38390c87 | |||
| 0ff4562e60 | |||
| bb7c460d12 | |||
| dd98da5c42 | |||
| 9e2abea16b | |||
| b481792c02 | |||
| 5865699a5a | |||
| b6a21c17eb | |||
| 494c719a5d | |||
| dad3cd0738 | |||
| a1e4743492 | |||
| 2ef470c2ba | |||
| 80f807d2ad | |||
| e9049ad27e | |||
| 431e052e6f | |||
| 0a2e194e76 | |||
| 18900dd613 | |||
| 2d42e98871 | |||
| 4a17821b06 | |||
| 4b24fbad98 | |||
| 8d83c66d11 | |||
| a6a62a2d56 | |||
| 05f791735d | |||
| bdd79ad585 | |||
| 66f06796cb | |||
| 06adc4702d | |||
| 1d02ba8936 | |||
| 66c578a6f4 | |||
| 356766bce1 | |||
| 6a402bf556 | |||
| f6d7b05161 | |||
| 025f81e961 | |||
| 064747f192 | |||
| a55d90954a | |||
| 87a3b163e7 | |||
| 2f338a1ab6 | |||
| be062b2d48 | |||
| 7f804b84e7 | |||
| bae81b66c8 | |||
| 8153e5eaa5 | |||
| 41c97bba8d | |||
| f9b0a96d9d | |||
| 40ec0c0d4b | |||
| af6e6c9979 | |||
| 3cced9a2e9 | |||
| bc1ef0f57f | |||
| f002ce0e9c | |||
| 3adfc3e0f9 | |||
| b6666196c1 | |||
| 5cae33a517 | |||
| 02e3d27aa9 | |||
| 78a9c61acb | |||
| 231248a990 | |||
| db50c34c23 | |||
| 1b5d35ad37 | |||
| 85a3397bf4 | |||
| 0b057df2a3 | |||
| 5b0b29dfad | |||
| a97764c5f7 | |||
| 9944ec8c58 | |||
| 98ab646206 |
@@ -11,14 +11,18 @@ started with an agent that inferred the goal instead of stating it back.
|
||||
## 0. Get on the branch
|
||||
|
||||
```sh
|
||||
task start <vikunja-id>
|
||||
task start <vikunja-id> # with an id
|
||||
git checkout -b task/<slug> # without one
|
||||
```
|
||||
|
||||
`~/.local/bin/task` owns the branch, the identity and the PR. It cuts
|
||||
`task/<id>-<slug>` off `origin/master` and sets the commit author to the `claude`
|
||||
gitea user. It writes `TASK.md` from the Vikunja task, and pulls any waiting
|
||||
An id is optional (owner's call, 2026-08-25). With one, `~/.local/bin/task` owns
|
||||
the branch, the identity and the PR. It cuts `task/<id>-<slug>` off
|
||||
`origin/master` and sets the commit author to the `claude` gitea user. It writes `TASK.md` from the Vikunja task, and pulls any waiting
|
||||
review comments into `.task/review-comments.md`. Do not hand-roll any of that.
|
||||
|
||||
Without an id, branch by hand and skip `TASK.md`. The user's own brief is then
|
||||
the goal, and step 4 restates it back to him instead.
|
||||
|
||||
`TASK.md` is the brief and it is immutable. If it says a PR already exists, this
|
||||
is a review-fix session and not new work. Read the comments first.
|
||||
|
||||
@@ -34,9 +38,12 @@ If there is no handoff, that is normal. It means the last session closed clean.
|
||||
|
||||
In this order, and stop as soon as you have enough:
|
||||
|
||||
- The Vikunja task, by id. Project Maven is ID 2, MCP at `http://localhost:9100/mcp`.
|
||||
The task description and its comments hold the goal, the constraints, and the
|
||||
assumption ledger. This outranks the handoff on every conflict.
|
||||
- The Vikunja task, if there is one. Project Maven is ID 2, MCP at
|
||||
`http://localhost:9100/mcp`, reachable from workpc only through
|
||||
`ssh -N -f -L 9100:127.0.0.1:9100 kami@192.168.1.104`. A refused connection is
|
||||
the missing tunnel, not an outage. The task description and its comments hold
|
||||
the goal, the constraints, and the assumption ledger. This outranks the handoff
|
||||
on every conflict.
|
||||
- `CLAUDE.md`, the section that covers the area you are about to touch.
|
||||
- The one file under `docs/` that owns the area. Check its `Last verified` line.
|
||||
If the sha is behind the code you are reading, say so in step 4 and trust the code.
|
||||
@@ -44,8 +51,8 @@ In this order, and stop as soon as you have enough:
|
||||
Do not read the dated files under `docs/evals/`. They are measurements from one day,
|
||||
never updated. Read one only when you need the number it recorded.
|
||||
|
||||
If no task id is known, ask for one before doing anything else. Work without a task
|
||||
is work nobody can resume.
|
||||
With no task id, do not ask for one and do not stall. State it in step 4 as
|
||||
`Task: unfiled` and carry on.
|
||||
|
||||
## 3. Look at the ground
|
||||
|
||||
@@ -58,7 +65,7 @@ Write at most five bullets and stop. Do not write code, do not open files to "ch
|
||||
one thing first", do not start with a small safe change.
|
||||
|
||||
```
|
||||
Task: V-359, one line.
|
||||
Task: V-359, one line. `unfiled` when there is no id.
|
||||
Done: what is already on the branch.
|
||||
Next: the one thing this session does.
|
||||
Constraints: what would make this wrong.
|
||||
@@ -67,8 +74,9 @@ Assuming: the beliefs that, if false, waste the session.
|
||||
|
||||
Then ask: is this right? Wait for the answer.
|
||||
|
||||
A corrected assumption goes into the Vikunja task as a comment, not into the handoff.
|
||||
The handoff dies tonight. The task does not.
|
||||
A corrected assumption goes into the Vikunja task as a comment where there is a
|
||||
task, because the handoff dies tonight and the task does not. Unfiled, it goes
|
||||
into the handoff and nowhere else.
|
||||
|
||||
## 5. Then begin
|
||||
|
||||
|
||||
@@ -36,11 +36,13 @@ a commit message, not into a comment in the code.
|
||||
Under 300 changed lines per commit in non-markdown files, enforced by `.githooks/pre-commit`.
|
||||
Markdown is exempt and may land as one batch.
|
||||
|
||||
Each commit is one idea, subject in the repo's voice, lowercase area prefix, and it
|
||||
ends with the Vikunja ref:
|
||||
Each commit is one idea, subject in the repo's voice, lowercase area prefix. A
|
||||
Vikunja ref is welcome where a task exists and is required nowhere: the
|
||||
`commit-msg` hook that demanded it was deleted on 2026-08-25.
|
||||
|
||||
```
|
||||
router: narrow the single-token rule (V-359)
|
||||
router: narrow the single-token rule
|
||||
```
|
||||
|
||||
If a change genuinely cannot split under 300 lines, say why in the commit body before
|
||||
@@ -56,13 +58,17 @@ It refuses a dirty tree, pushes, opens or refreshes the PR against the repo defa
|
||||
branch, labels the Vikunja task in-review, comments the PR url on it, and pushes an
|
||||
ntfy. Do not push by hand and do not call `tea` yourself.
|
||||
|
||||
`task pr` needs an id. On a hand-cut branch with no task, push the branch and open
|
||||
the PR by hand, and skip step 5.
|
||||
|
||||
## 5. Record what `task pr` cannot know
|
||||
|
||||
Comment on the Vikunja task: what you measured, what is still open. List every
|
||||
assumption that turned out to be wrong. If the session found new work, create a task
|
||||
for it now rather than describing it in prose.
|
||||
|
||||
This step is what makes the handoff disposable.
|
||||
This step is what makes the handoff disposable. With no task, it cannot run, so the
|
||||
handoff carries that content instead and stops being disposable. Say so in it.
|
||||
|
||||
## 6. Leave the handoff, or leave none
|
||||
|
||||
@@ -75,7 +81,7 @@ resume, and no history:
|
||||
```markdown
|
||||
# Handoff — <date>
|
||||
|
||||
Task: V-359 <one line>
|
||||
Task: V-359 <one line>, or `unfiled`
|
||||
Branch: task/359-<slug>, cut from master
|
||||
|
||||
## Where I stopped
|
||||
|
||||
@@ -1,30 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Every commit names the Vikunja task it belongs to.
|
||||
#
|
||||
# router: narrow the single-token rule (V-359)
|
||||
#
|
||||
# V- and not #, because Gitea autolinks #359 to a Gitea issue, which is a
|
||||
# different tracker and a wrong link.
|
||||
#
|
||||
# Exempt: merges, reverts, fixup/squash, and the initial commit.
|
||||
|
||||
msg_file=$1
|
||||
subject=$(sed -n '1p' "$msg_file")
|
||||
|
||||
case "$subject" in
|
||||
Merge\ *|Revert\ *|fixup!\ *|squash!\ *|amend!\ *) exit 0 ;;
|
||||
esac
|
||||
|
||||
if [ -f "$(git rev-parse --git-dir)/MERGE_HEAD" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if printf '%s' "$subject" | grep -qE '\(V-[0-9]+\)$'; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "commit-msg: subject must end with a Vikunja task ref." >&2
|
||||
echo " got: $subject" >&2
|
||||
echo " want: router: narrow the single-token rule (V-359)" >&2
|
||||
echo " No task yet? Create one. Work without a task is work nobody can resume." >&2
|
||||
exit 1
|
||||
+15
@@ -76,3 +76,18 @@ __pycache__/
|
||||
.env
|
||||
# silero-vad, downloaded (see AGENTS.md)
|
||||
/models/vad/
|
||||
|
||||
# Go build cache and GOPATH from the containerised e2eprobe build. Created by
|
||||
# the command in docs/capabilities/README.md, which runs as root in a container
|
||||
# and so cannot share the host cache. Multi-GB, entirely reproducible.
|
||||
/.cache/
|
||||
|
||||
# docs/architecture/ derived output. The sources, findings.md and the inventory
|
||||
# JSON are tracked; these rebuild from them with pack_evidence.sh and are large.
|
||||
/docs/architecture/index.html
|
||||
/docs/architecture/anchors.md
|
||||
/docs/architecture/architecture-evidence.txt
|
||||
/docs/architecture/tree.txt
|
||||
/docs/architecture/docker-compose.redacted.yml
|
||||
/docs/architecture/diagrams/*.svg
|
||||
/maven-evidence.zip
|
||||
|
||||
@@ -15,6 +15,8 @@ under `docs/`. Read that doc before changing the subsystem.
|
||||
| `docs/world.md` | touching search, Kiwix or the world chain |
|
||||
| `docs/language.md` | changing a prompt contract or a Russian word list |
|
||||
| `docs/ecosystem.md` | touching Nexus, Praxis or Hexis |
|
||||
| `docs/spec.md` | asking what a capability is for, or whether it is done |
|
||||
| `docs/roadmap.md` | picking what to work on next |
|
||||
| `docs/rearchitecture.md`, `docs/design.md` | changing the shape of anything |
|
||||
| `docs/workflow.md` | the five stores, the doc tiers, the guards |
|
||||
| `docs/caveats/` | a known limit, its task id and its revisit trigger |
|
||||
@@ -199,14 +201,14 @@ ToolSearch("select:mcp__vikunja__list_tasks,mcp__vikunja__get_task_details,mcp__
|
||||
```
|
||||
|
||||
- This repo is Vikunja project **Maven** (ID 2), MCP at
|
||||
`http://localhost:9100/mcp`, or `http://192.168.1.104:9100/mcp` from workpc.
|
||||
- **A session with no task id asks for one before it starts**, because work
|
||||
without one is work nobody can resume.
|
||||
`http://localhost:9100/mcp` on homesrv. **`vikunja-mcp` publishes to
|
||||
`127.0.0.1:9100` only, so the LAN address never answers from workpc.** A
|
||||
refused connection is that, not an outage: three sessions read it as "Vikunja
|
||||
is down" and filed nothing. Tunnel first, then use `localhost`:
|
||||
`ssh -N -f -L 9100:127.0.0.1:9100 kami@192.168.1.104`.
|
||||
- **Close a finished task with `done: true` and nothing else** (owner's call,
|
||||
2026-08-07). `update_task` carrying a `description` resets `done` to false.
|
||||
- **`pre-commit` refuses master** and more than 300 changed lines in
|
||||
non-markdown files. Markdown is exempt and may land as one batch.
|
||||
- **`commit-msg` requires the subject to end with `(V-<id>)`.** `V-` and not
|
||||
`#`, because Gitea autolinks `#123` to the wrong tracker.
|
||||
- **`diff-budget.sh` blocks edits past 600 changed lines** on a `task/` branch.
|
||||
- **`--no-verify` exists.** Using it means saying why in the commit body.
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
# Handoff
|
||||
|
||||
Master is at `5cae33a`, pushed, tree clean apart from this file. Working on master
|
||||
raw by the owner's call: no branch, `--no-verify` on every commit with the reason
|
||||
in the body.
|
||||
|
||||
## Landed this session
|
||||
|
||||
Twelve commits pushed. Nine were the previous session's tree, already described in
|
||||
the commit log. Three are new:
|
||||
|
||||
- `78a9c61` `docs/spec.md`, 51 capabilities with a DoD each.
|
||||
- `02e3d27` `docs/roadmap.md`, seven milestones, plus both pointer rows in `CLAUDE.md`.
|
||||
- `5cae33a` honesty split into three milestones, five capabilities deferred past v1.
|
||||
|
||||
**Read `docs/spec.md` and `docs/roadmap.md` before anything else.** Every decision
|
||||
from this session is in them. This file holds only what they do not.
|
||||
|
||||
## The two documents
|
||||
|
||||
`docs/spec.md` is the union of the 39 audited rows
|
||||
(`docs/evals/2026-08-13-capability-audit.md`) and the owner's 18-item v1 list.
|
||||
Twelve of his items had no audit row, so the file has 51 entries. Each entry
|
||||
carries three parts. State is a reference to the living doc that owns it. DoD is
|
||||
a plain list observable on the running box. Scenario names a file in
|
||||
`cmd/mavend/testdata/scenarios/`.
|
||||
|
||||
`docs/roadmap.md` orders them into nine milestones. Honesty, then reach, then
|
||||
breadth. Not ordered by code work, because none of the four broken capabilities is
|
||||
a code defect.
|
||||
|
||||
## Decided, do not re-ask
|
||||
|
||||
- **v1 is a voice assistant, minimum viable.** Each DoD is written at
|
||||
"voice-reachable and honest", not "feature-complete".
|
||||
- **Honesty splits into three.** M1 the turn path, M2 memory he cannot correct,
|
||||
M3 step-up. M1 and M2 touch different code and owe different docs. Step-up is
|
||||
configuration, not honesty, and sits before M4 because M4 is what first makes
|
||||
acts real.
|
||||
- **Five capabilities deferred past v1** (owner's call, 2026-08-15): speaker
|
||||
recognition, smart home, bluetooth control, model swap, self-update. Bluetooth
|
||||
was on the v1 list and came off it. Their spec entries keep their DoD.
|
||||
- **A milestone closes its own doc gaps and writes its own scenarios.** Neither
|
||||
becomes a milestone of its own. Otherwise the 17 missing docs and 46 missing
|
||||
scenario files collect at the end.
|
||||
- **Learning means behavioral, not weights.** Stored outcomes only. No adapter, no
|
||||
training set.
|
||||
- **Email and calendar need the product decision before deploying.** Both are
|
||||
built and neither is in `docker-compose.yml`. That is M8.
|
||||
|
||||
## Findings the spec pass produced
|
||||
|
||||
- **Recurring reminders do not exist on the spoken path.** `store.Reminder` carries
|
||||
`Cron` and `ipc.CreateReminder` takes one. `grep "Cron:" --include='*.go'`
|
||||
outside tests returns only `internal/ipc/client.go`, `internal/ipc/storeapi.go`
|
||||
and `cmd/mavend/tick_routines.go`, and the last is routines, a separate
|
||||
mechanism. Pills, the dog, the vet and the kibble are unbuilt on top of finished
|
||||
storage and delivery. This is M6.
|
||||
- **Seventeen capabilities have no living doc.** Memory is the worst cluster:
|
||||
facts, notes and the digestion worker have no owning document at all.
|
||||
- **Webhooks barely exist.** The only one in the tree is
|
||||
`internal/delivery/telegramsink/intake.go`, Telegram's own inbound hook.
|
||||
- **Command chaining does not exist.** The `chain` in `internal/router` is the
|
||||
world chain and the source chain.
|
||||
- **Desk notifications are inbound only.** `ambient:notif` reads his desktop.
|
||||
There is no outbound desk reach, and which direction he meant is undecided.
|
||||
- **Only 5 of 51 spec entries cite a scenario that exists.**
|
||||
|
||||
## Not filed, and this is the risk
|
||||
|
||||
Vikunja returned 503 across this session and the last, so **none of this has a
|
||||
task id**. The three commits above are tagged `V-719`, which is the
|
||||
reminder-cancellation task, not this work. Retag or file when Vikunja is back.
|
||||
|
||||
Unfiled, listed again at the end of `docs/roadmap.md`:
|
||||
|
||||
1. The capability audit itself, headed "unfiled".
|
||||
2. Remember-versus-query misroute, two of seven audit probes.
|
||||
3. The masculine reply on the wire, caught live while `CheckFeminine` passed.
|
||||
4. Recurring reminders having no caller.
|
||||
5. The seventeen capabilities with no living doc.
|
||||
6. The 46 scenario files the spec names and does not have.
|
||||
|
||||
## Next
|
||||
|
||||
Open a session on M1, which is three gate items on the turn path and owes no new
|
||||
doc. If Vikunja is up, file the six above first and give M1 a real id.
|
||||
|
||||
One command still outstanding from the last session, cheap and unrelated:
|
||||
|
||||
```sh
|
||||
docker compose up -d --force-recreate mavsttd mavttsd mavpoll
|
||||
```
|
||||
+58
@@ -281,3 +281,61 @@ expires, and when it has been drained. `./cmd/mavend/ -run TestSuppressedCareDig
|
||||
passes under race in 4.626s, the digest store and loop cases in 4.123s and
|
||||
1.046s, and the three full packages in 264.076s, 64.496s and 4.280s. The caveat
|
||||
`docs/caveats/workers.md#nudges` and the `SA4006` baseline entry are deleted.
|
||||
|
||||
## 2026-08-15 — authoritative-state continuation
|
||||
|
||||
The continuation began by treating the checked-out tree and external task state
|
||||
as authoritative. `master` was clean, identical to `origin/master`, and already
|
||||
contained the V-717 and V-687 branch merges. Their interrupted worker messages
|
||||
were therefore not used as evidence of missing work.
|
||||
|
||||
### Live reminder catch-up closes V-570 and V-715
|
||||
|
||||
The rebuilt stack loaded the resident Qwen model, multilingual ONNX embedder,
|
||||
and routing heads. On its first eligible tick it phrased the three overdue
|
||||
reminders once as one catch-up message. Disabled ntfy left one durable failed
|
||||
attempt for delivery group `reminder #83`; ordered fallback then left exactly
|
||||
one successful Telegram attempt for the same group. All originals became
|
||||
`fired`. Four later ticks produced no second phrase, send, pending attempt, or
|
||||
unknown attempt. `/`, `/reminders`, and `/notifications` each returned HTTP 200
|
||||
with a server-generated request ID. This supplies the live evidence that was
|
||||
still missing in the 2026-08-13 entry; V-715 and the stale-open V-570 are closed.
|
||||
|
||||
The first HTTP repro command also exposed a zsh test-harness trap: `path` is a
|
||||
special array tied to `PATH`, so using it as a loop variable removed command
|
||||
lookup inside the loop. The corrected probe used `probe_path` and `wget
|
||||
--no-proxy`; the deployed web process had not crashed.
|
||||
|
||||
### Stale-open task reconciliation
|
||||
|
||||
The open Vikunja list was read with `done: false` and compared with the current
|
||||
tree before choosing work. V-397 already described itself as done and merged;
|
||||
V-557 is present as `ea0eb16` plus the forced missing-slot dialogue contract;
|
||||
V-570 is covered by the live catch-up proof above. Those three records were
|
||||
closed rather than reimplemented. V-651 remains open: its nil-sink and reminder
|
||||
spin halves landed with V-715, but the first failed severity-4 Telegram send is
|
||||
still not represented in the repeat-until-ack stream.
|
||||
|
||||
### Stable hash-floor performance evidence [V-718]
|
||||
|
||||
`TestPersonalBoundaryHashFloorLatency` coupled correctness to ambient machine
|
||||
load while running a numeric training loop under race and coverage
|
||||
instrumentation. It is now a deterministic fit-and-score test that also proves
|
||||
the 1024-dimensional head was built. Elapsed time remains observable through
|
||||
`BenchmarkPersonalBoundaryHashFloorFitAndScore`, where three one-iteration runs
|
||||
on this host measured 75.1 ms, 76.8 ms and 81.4 ms without making those host
|
||||
figures a CI pass condition.
|
||||
|
||||
The exact race-plus-coverage focused test passed in 9.445s. `make test` then
|
||||
passed on its first run: formatting, vet, CW2 configuration tests, and every
|
||||
internal and command package under race plus coverage; `cmd/mavend` completed
|
||||
in 205.712s. This closes V-718 without raising a brittle timeout.
|
||||
|
||||
### Delegation availability
|
||||
|
||||
All three available subagent slots were filled: clarification exhaustion,
|
||||
durable nudge identity, and a read-only live deployment probe. The first two
|
||||
left complete merges on `master`; all three later reported the same shared
|
||||
Codex usage limit, with capacity unavailable until 2026-08-20. Work continues
|
||||
serially. The temporary constraint and revisit trigger are recorded at
|
||||
`docs/caveats/workers.md#agent-quota` under the V-714 completion umbrella.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,284 @@
|
||||
// e2eprobe is a temporary typed IPC driver used by the 2026-08-15 isolated
|
||||
// whole-Maven acceptance session. It is removed after the session; keeping the
|
||||
// driver inside the module lets it import Maven's internal IPC contract rather
|
||||
// than peeking into sqlite.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"math"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := run(os.Args[1:]); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "e2eprobe:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func run(args []string) error {
|
||||
fs := flag.NewFlagSet("e2eprobe", flag.ContinueOnError)
|
||||
sock := fs.String("sock", "", "mavend unix socket")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
argv := fs.Args()
|
||||
if len(argv) == 0 {
|
||||
return errors.New("usage: e2eprobe -sock PATH COMMAND [ARGS]")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
if argv[0] == "score-pair" {
|
||||
out, err := scorePair(ctx, argv)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return encode(out)
|
||||
}
|
||||
if argv[0] == "parse-task-status" {
|
||||
if len(argv) != 2 {
|
||||
return errors.New("parse-task-status needs TEXT")
|
||||
}
|
||||
parsed, ok := router.ParseTaskStatus(argv[1])
|
||||
return encode(map[string]any{"accepted": ok, "parsed": parsed})
|
||||
}
|
||||
if *sock == "" {
|
||||
return errors.New("usage: e2eprobe -sock PATH COMMAND [ARGS]")
|
||||
}
|
||||
|
||||
cli, err := ipc.DialWait(*sock, 15*time.Second)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cli.Close()
|
||||
|
||||
var out any
|
||||
switch argv[0] {
|
||||
case "ping":
|
||||
out, err = cli.Ping(ctx)
|
||||
case "chat":
|
||||
if len(argv) < 3 {
|
||||
return errors.New("chat needs CONVERSATION TEXT")
|
||||
}
|
||||
out, err = cli.Chat(ctx, argv[1], strings.Join(argv[2:], " "))
|
||||
case "create-reminder":
|
||||
if len(argv) < 3 || len(argv) > 4 {
|
||||
return errors.New("create-reminder needs RFC3339 TEXT [CRON]")
|
||||
}
|
||||
fire, parseErr := time.Parse(time.RFC3339, argv[1])
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
cron := ""
|
||||
if len(argv) == 4 {
|
||||
cron = argv[3]
|
||||
}
|
||||
id, createErr := cli.CreateReminder(ctx, fire, `{"text":`+quote(argv[2])+`}`, cron)
|
||||
out, err = map[string]any{"id": id}, createErr
|
||||
case "cancel-reminder":
|
||||
id, parseErr := oneID(argv)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
err = cli.CancelReminder(ctx, id)
|
||||
out = map[string]any{"cancelled": id}
|
||||
case "mark-reminder":
|
||||
if len(argv) != 3 {
|
||||
return errors.New("mark-reminder needs ID STATUS")
|
||||
}
|
||||
id, parseErr := strconv.ParseInt(argv[1], 10, 64)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
err = cli.MarkReminder(ctx, id, argv[2])
|
||||
out = map[string]any{"marked": id, "status": argv[2]}
|
||||
case "reminders":
|
||||
n, parseErr := optionalN(argv, 200)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.ListReminders(ctx, n)
|
||||
case "pending-reminders":
|
||||
n, parseErr := optionalN(argv, 0)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.ListPendingReminders(ctx, n)
|
||||
case "create-task":
|
||||
if len(argv) != 2 {
|
||||
return errors.New("create-task needs TEXT")
|
||||
}
|
||||
out, err = cli.CaptureTask(ctx, ipc.CaptureTaskReq{
|
||||
Text: argv[1], Source: "tap:web", Status: store.TaskOpen, Ts: time.Now(),
|
||||
})
|
||||
case "tasks":
|
||||
status := "live"
|
||||
if len(argv) == 2 {
|
||||
status = argv[1]
|
||||
} else if len(argv) != 1 {
|
||||
return errors.New("tasks takes optional STATUS")
|
||||
}
|
||||
out, err = cli.ListTasks(ctx, status)
|
||||
case "notes":
|
||||
n, parseErr := optionalN(argv, 50)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.RecentNotes(ctx, n)
|
||||
case "query-notes":
|
||||
if len(argv) != 2 {
|
||||
return errors.New("query-notes needs TEXT")
|
||||
}
|
||||
embedder, embedErr := router.NewONNXEmbedder(
|
||||
"models/embedder/multilingual-e5-small/model_quantized.onnx",
|
||||
"models/embedder/multilingual-e5-small/tokenizer.json",
|
||||
"deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so",
|
||||
)
|
||||
if embedErr != nil {
|
||||
return embedErr
|
||||
}
|
||||
defer embedder.Close()
|
||||
vec, embedErr := router.EmbedQuery(ctx, embedder, argv[1])
|
||||
if embedErr != nil {
|
||||
return embedErr
|
||||
}
|
||||
out, err = cli.QueryNotes(ctx, vec, 10)
|
||||
case "score-pair":
|
||||
out, err = scorePair(ctx, argv)
|
||||
case "facts":
|
||||
n, parseErr := optionalN(argv, 50)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.RecentFacts(ctx, n)
|
||||
case "decisions":
|
||||
n, parseErr := optionalN(argv, 50)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.TurnDecisions(ctx, n)
|
||||
case "events":
|
||||
n, parseErr := optionalN(argv, 50)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.RecentEvents(ctx, n)
|
||||
case "eco-traces":
|
||||
n, parseErr := optionalN(argv, 50)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.RecentEcosystemTraces(ctx, n)
|
||||
case "delivery-attempts":
|
||||
status := ""
|
||||
if len(argv) == 2 {
|
||||
status = argv[1]
|
||||
} else if len(argv) != 1 {
|
||||
return errors.New("delivery-attempts takes optional STATUS")
|
||||
}
|
||||
out, err = cli.DeliveryAttempts(ctx, status, 200)
|
||||
case "nudges":
|
||||
n, parseErr := optionalN(argv, 50)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
out, err = cli.RecentNudges(ctx, n)
|
||||
case "tools":
|
||||
status := ""
|
||||
if len(argv) == 2 {
|
||||
status = argv[1]
|
||||
} else if len(argv) != 1 {
|
||||
return errors.New("tools takes optional STATUS")
|
||||
}
|
||||
out, err = cli.ListTools(ctx, status)
|
||||
case "plan":
|
||||
out, err = cli.DayPlan(ctx)
|
||||
case "correct":
|
||||
if len(argv) != 3 {
|
||||
return errors.New("correct needs TRACE_ID SHOULD_BE")
|
||||
}
|
||||
id, parseErr := strconv.ParseInt(argv[1], 10, 64)
|
||||
if parseErr != nil {
|
||||
return parseErr
|
||||
}
|
||||
err = cli.CorrectTurn(ctx, id, argv[2])
|
||||
out = map[string]any{"corrected": id, "should_be": argv[2]}
|
||||
default:
|
||||
return fmt.Errorf("unknown command %q", argv[0])
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return encode(out)
|
||||
}
|
||||
|
||||
func encode(out any) error {
|
||||
enc := json.NewEncoder(os.Stdout)
|
||||
enc.SetIndent("", " ")
|
||||
return enc.Encode(out)
|
||||
}
|
||||
|
||||
func scorePair(ctx context.Context, argv []string) (any, error) {
|
||||
if len(argv) != 3 {
|
||||
return nil, errors.New("score-pair needs QUERY PASSAGE")
|
||||
}
|
||||
embedder, err := router.NewONNXEmbedder(
|
||||
"models/embedder/multilingual-e5-small/model_quantized.onnx",
|
||||
"models/embedder/multilingual-e5-small/tokenizer.json",
|
||||
"deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so",
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer embedder.Close()
|
||||
qvec, err := router.EmbedQuery(ctx, embedder, argv[1])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pvec, err := router.EmbedPassage(ctx, embedder, argv[2])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(qvec) != len(pvec) {
|
||||
return nil, fmt.Errorf("embedding widths differ: %d != %d", len(qvec), len(pvec))
|
||||
}
|
||||
var dot float64
|
||||
for i := range qvec {
|
||||
dot += float64(qvec[i]) * float64(pvec[i])
|
||||
}
|
||||
return map[string]any{"score": math.Round(dot*1e9) / 1e9}, nil
|
||||
}
|
||||
|
||||
func quote(s string) string {
|
||||
b, _ := json.Marshal(s)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
func oneID(argv []string) (int64, error) {
|
||||
if len(argv) != 2 {
|
||||
return 0, errors.New("command needs ID")
|
||||
}
|
||||
return strconv.ParseInt(argv[1], 10, 64)
|
||||
}
|
||||
|
||||
func optionalN(argv []string, fallback int) (int, error) {
|
||||
if len(argv) == 1 {
|
||||
return fallback, nil
|
||||
}
|
||||
if len(argv) != 2 {
|
||||
return 0, errors.New("command takes optional N")
|
||||
}
|
||||
return strconv.Atoi(argv[1])
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/kami/maven/internal/decision"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// resolveAction produces an ActionCandidate from a routing decision. It is the
|
||||
// single boundary between routing and action execution: everything downstream
|
||||
// (refusesCommand, task-status, Praxis, Hexis, proposeGap, tool.Executor.Exec)
|
||||
// consumes the candidate rather than re-resolving the function.
|
||||
//
|
||||
// Delegates to router.ResolveActionCandidate for the resolution logic, then
|
||||
// records the outcome in the decision trace.
|
||||
func (h *reactiveHandler) resolveAction(ctx context.Context, dec router.Decision) router.ActionCandidate {
|
||||
candidate := router.ResolveActionCandidate(dec, h.matcher)
|
||||
|
||||
// Record the resolution outcome in the decision trace.
|
||||
if dec.Intent == router.IntentAct {
|
||||
if candidate.ActionResolved() {
|
||||
noteActionResolution(ctx, string(candidate.Source), candidate.Fn, true)
|
||||
} else {
|
||||
noteActionResolution(ctx, "matcher", "", false)
|
||||
}
|
||||
}
|
||||
|
||||
return candidate
|
||||
}
|
||||
|
||||
// noteActionResolution records the action resolution outcome in the decision
|
||||
// trace. A nil recorder is the normal case in tests.
|
||||
func noteActionResolution(ctx context.Context, source, fn string, resolved bool) {
|
||||
rec := decision.From(ctx)
|
||||
if rec == nil {
|
||||
return
|
||||
}
|
||||
outcome := decision.Declined
|
||||
reason := "no match"
|
||||
if resolved {
|
||||
outcome = decision.Won
|
||||
reason = "resolved via " + source
|
||||
if fn != "" {
|
||||
reason += ": " + fn
|
||||
}
|
||||
}
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-resolve",
|
||||
Outcome: outcome,
|
||||
Reason: reason,
|
||||
})
|
||||
}
|
||||
|
||||
// noteActionValidation records the structural validation outcome in the
|
||||
// decision trace. Five outcomes: unresolved (matcher miss), valid
|
||||
// (structurally admissible), invalid_argument, missing_argument, or
|
||||
// ambiguous_target (structurally malformed).
|
||||
func noteActionValidation(ctx context.Context, v router.ActionValidationResult) {
|
||||
rec := decision.From(ctx)
|
||||
if rec == nil {
|
||||
return
|
||||
}
|
||||
switch v.Status {
|
||||
case router.ActionUnresolved:
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-validation",
|
||||
Outcome: decision.Declined,
|
||||
Reason: "unresolved",
|
||||
})
|
||||
case router.ActionValid:
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-validation",
|
||||
Outcome: decision.Won,
|
||||
Reason: "valid",
|
||||
})
|
||||
default:
|
||||
reason := string(v.Status)
|
||||
if len(v.Issues) > 0 {
|
||||
reason = string(v.Status) + ":" + v.Issues[0].Reason
|
||||
}
|
||||
rec.Note(decision.Claim{
|
||||
Stage: decision.StageAction,
|
||||
Claimant: "action-validation",
|
||||
Outcome: decision.Declined,
|
||||
Reason: reason,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,556 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/decision"
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
"github.com/kami/maven/internal/tool"
|
||||
)
|
||||
|
||||
// newActHandler builds a handler with the act path wired: a matcher over
|
||||
// whatever tools the test enabled, no model, no ecosystem.
|
||||
func newActHandler(t *testing.T) (*reactiveHandler, *store.Store) {
|
||||
t.Helper()
|
||||
st := newTestStore(t)
|
||||
api := ipc.NewStoreAPI(st)
|
||||
matcher := tool.NewMatcher(api)
|
||||
h := &reactiveHandler{
|
||||
api: api,
|
||||
tools: tool.NewExecutor(api, 2*time.Second),
|
||||
matcher: matcher,
|
||||
now: func() time.Time { return time.Date(2026, 8, 1, 9, 0, 0, 0, time.UTC) },
|
||||
}
|
||||
return h, st
|
||||
}
|
||||
|
||||
// TestActRouteSource_NoMatcherInvoke pins that an act with HasFn=true
|
||||
// produces a candidate from the route and does not invoke the matcher.
|
||||
func TestActRouteSource_NoMatcherInvoke(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Enable a tool so the matcher has something to match against.
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act with HasFn=true: the candidate must come from the route.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("route-sourced act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActMatcherSource_FallbackMatch pins that an act without Fn invokes
|
||||
// the matcher and produces a matcher-sourced candidate.
|
||||
func TestActMatcherSource_FallbackMatch(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Enable a tool so the matcher can find it.
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act without HasFn: the matcher must resolve "status" from the text.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "check status",
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("matcher-sourced act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActMatcherMiss_ProposeGap pins that a matcher miss produces the
|
||||
// same propose-gap behavior as before.
|
||||
func TestActMatcherMiss_ProposeGap(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Enable one tool so the matcher has an allowlist, but not the one asked for.
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act without HasFn and text that doesn't match any tool.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "deploy the thing",
|
||||
Slots: router.Slots{Text: "deploy the thing"},
|
||||
})
|
||||
if !strings.Contains(strings.ToLower(reply), "предлож") {
|
||||
t.Errorf("matcher miss replied %q; want propose-gap behavior", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActDestructive_ConfirmationUnchanged pins that a destructive tool
|
||||
// still triggers the confirmation flow.
|
||||
func TestActDestructive_ConfirmationUnchanged(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart",
|
||||
Slots: router.Slots{Fn: "restart", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("destructive act replied %q; want a confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActTaskStatus_InterceptUnchanged pins that task_status is intercepted
|
||||
// before reaching the tool executor.
|
||||
func TestActTaskStatus_InterceptUnchanged(t *testing.T) {
|
||||
h, _ := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "task status",
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true,
|
||||
Text: "task status"},
|
||||
})
|
||||
// task_status is intercepted by resolveTaskStatus, which returns a
|
||||
// status phrase. The exact reply depends on the store state, but it
|
||||
// must not be a tool execution result.
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Errorf("task_status was not intercepted, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActStage0_SameResult pins that a stage-0 act (grammar match with
|
||||
// HasFn=true) produces the same tool execution as before.
|
||||
func TestActStage0_SameResult(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Stage: 0,
|
||||
Confidence: 1.0,
|
||||
Utterance: "maven, restart nginx",
|
||||
Slots: router.Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
Producer: router.RouteProducerGrammar,
|
||||
})
|
||||
if !strings.Contains(reply, "сделала") && !strings.Contains(reply, "готово") {
|
||||
t.Errorf("stage-0 act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActLearnedRouter_NoFn_FallbackMatch pins that a learned-router act
|
||||
// without Fn falls through to the matcher and produces the same result.
|
||||
func TestActLearnedRouter_NoFn_FallbackMatch(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// LLM routed the act but did not fill Fn (common when the model returns
|
||||
// the verb in Text but not in Fn).
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Stage: 1,
|
||||
Confidence: 0.85,
|
||||
Utterance: "could you restart nginx",
|
||||
Slots: router.Slots{Text: "restart nginx"},
|
||||
Producer: router.RouteProducerLLM,
|
||||
})
|
||||
if !strings.Contains(reply, "сделала") && !strings.Contains(reply, "готово") {
|
||||
t.Errorf("learned-router act replied %q; want it to have run", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveAction_CandidateSource_Verified pins the candidate source
|
||||
// for both route-resolved and matcher-resolved actions.
|
||||
func TestResolveAction_CandidateSource_Verified(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Route-resolved: HasFn=true.
|
||||
c1 := h.resolveAction(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
if c1.Source != router.ActionSourceRoute {
|
||||
t.Errorf("route candidate source = %q, want route", c1.Source)
|
||||
}
|
||||
if c1.Fn != "status" {
|
||||
t.Errorf("route candidate Fn = %q, want status", c1.Fn)
|
||||
}
|
||||
|
||||
// Matcher-resolved: no Fn, text matches.
|
||||
c2 := h.resolveAction(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if c2.Source != router.ActionSourceMatcher {
|
||||
t.Errorf("matcher candidate source = %q, want matcher", c2.Source)
|
||||
}
|
||||
if c2.Fn != "status" {
|
||||
t.Errorf("matcher candidate Fn = %q, want status", c2.Fn)
|
||||
}
|
||||
|
||||
// Matcher miss: no Fn, text doesn't match.
|
||||
c3 := h.resolveAction(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "deploy everything"},
|
||||
})
|
||||
if c3.ActionResolved() {
|
||||
t.Errorf("miss candidate resolved = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
// --- structural validation integration tests ---
|
||||
|
||||
// TestActValidation_MalformedCandidate_BlankFn pins that a resolved
|
||||
// candidate with a blank (whitespace-only) Fn does not execute and
|
||||
// produces a failure response.
|
||||
func TestActValidation_MalformedCandidate_BlankFn(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Simulate a malformed candidate by writing a blank Fn into Slots
|
||||
// after resolution. This tests that the validation layer catches
|
||||
// structurally invalid candidates.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: " ", HasFn: true},
|
||||
})
|
||||
// The blank Fn should not reach tool execution. It either hits
|
||||
// the validation gate (ActFail) or the existing error paths.
|
||||
if reply == "" {
|
||||
t.Error("expected a response, got empty string")
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_UnresolvedCandidate_ProposeGap pins that an unresolved
|
||||
// candidate (matcher miss) still flows to proposeGap, unchanged.
|
||||
func TestActValidation_UnresolvedCandidate_ProposeGap(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "deploy everything",
|
||||
Slots: router.Slots{Text: "deploy everything"},
|
||||
})
|
||||
if !strings.Contains(strings.ToLower(reply), "предлож") {
|
||||
t.Errorf("unresolved candidate replied %q; want propose-gap behavior", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_DestructiveValid_StillConfirms pins that a destructive
|
||||
// valid action still reaches the confirmation path through validation.
|
||||
func TestActValidation_DestructiveValid_StillConfirms(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart",
|
||||
Slots: router.Slots{Fn: "restart", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("destructive valid act replied %q; want confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_IrreversibleValid_NeedsAuthedSurface pins that an
|
||||
// irreversible valid action still reaches ErrNeedsAuthedSurface.
|
||||
func TestActValidation_IrreversibleValid_NeedsAuthedSurface(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
// Register an irreversible tool: cmd containing "drop" triggers the
|
||||
// irreversible tier via RiskOf → isIrreversible.
|
||||
if err := st.EnableTool(ctx, "drop_table", []string{"drop"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "drop_table",
|
||||
Slots: router.Slots{Fn: "drop_table", HasFn: true},
|
||||
})
|
||||
// Irreversible tools return ErrNeedsAuthedSurface, which produces
|
||||
// a specific phraser response.
|
||||
if !strings.Contains(reply, "выполню") && !strings.Contains(reply, "запусти") {
|
||||
t.Errorf("irreversible valid act replied %q; want authed-surface response", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_ValidationTracing pins that validation outcomes are
|
||||
// recorded in the decision trace.
|
||||
func TestActValidation_ValidationTracing(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
now := h.now()
|
||||
|
||||
// Valid candidate: trace should show action-validation:won.
|
||||
ctx, rec := decision.With(context.Background(), "status", "tap:text")
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
records := rec.Claims
|
||||
found := false
|
||||
for _, c := range records {
|
||||
if c.Claimant == "action-validation" && c.Outcome == decision.Won {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("expected action-validation:won in trace, got %v", records)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActExecutionFromCandidateNotSlots pins that downstream execution reads
|
||||
// resolved action data from ActionCandidate, not from Decision.Slots. The
|
||||
// decision has empty Fn/Args/HasFn — the bridge used to copy candidate values
|
||||
// back into these fields. After the bridge removal, execution must still
|
||||
// succeed because the candidate carries the resolved function.
|
||||
func TestActExecutionFromCandidateNotSlots(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Act without any Fn/Args/HasFn in Slots — the matcher resolves from Text.
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "check status",
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("execution from candidate replied %q; want tool success", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// --- validation status boundary tests ---
|
||||
|
||||
// TestActValidation_StatusValidRoute pins that a route-resolved valid action
|
||||
// produces ActionValid status and reaches execution.
|
||||
func TestActValidation_StatusValidRoute(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: "status", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("valid route act replied %q; want tool success", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_StatusValidMatcher pins that a matcher-resolved valid
|
||||
// action produces ActionValid status and reaches execution.
|
||||
func TestActValidation_StatusValidMatcher(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "check status",
|
||||
Slots: router.Slots{Text: "status"},
|
||||
})
|
||||
if !strings.Contains(reply, "готово") {
|
||||
t.Errorf("valid matcher act replied %q; want tool success", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_StatusUnresolved pins that an unresolved candidate
|
||||
// produces ActionUnresolved status and flows to proposeGap.
|
||||
func TestActValidation_StatusUnresolved(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "deploy everything",
|
||||
Slots: router.Slots{Text: "deploy everything"},
|
||||
})
|
||||
if !strings.Contains(strings.ToLower(reply), "предлож") {
|
||||
t.Errorf("unresolved act replied %q; want propose-gap", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_StatusInvalid pins that a structurally invalid candidate
|
||||
// produces ActionInvalidArgument status and refuses execution.
|
||||
func TestActValidation_StatusInvalid(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: " ", HasFn: true},
|
||||
})
|
||||
if reply == "" {
|
||||
t.Error("expected a response for invalid candidate")
|
||||
}
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Error("invalid candidate should not reach tool execution")
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_DestructiveValidStatus pins that a destructive but
|
||||
// structurally valid action still produces ActionValid status and reaches
|
||||
// the confirmation path (not validation failure).
|
||||
func TestActValidation_DestructiveValidStatus(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"true"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart",
|
||||
Slots: router.Slots{Fn: "restart", HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("destructive valid act replied %q; want confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_ConfirmationUnchanged pins that the confirmation flow
|
||||
// is unchanged by validation.
|
||||
func TestActValidation_ConfirmationUnchanged(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "restart", []string{"echo", "ok"}, true, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart nginx",
|
||||
Slots: router.Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
})
|
||||
if !strings.Contains(reply, "да или нет") {
|
||||
t.Errorf("confirmation act replied %q; want confirm turn", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_TaskStatusInterceptUnchanged pins that task_status
|
||||
// interception is unchanged by validation.
|
||||
func TestActValidation_TaskStatusInterceptUnchanged(t *testing.T) {
|
||||
h, _ := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "task status",
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Text: "task status"},
|
||||
})
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Errorf("task_status was not intercepted, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
// TestActValidation_NoExecutionOnFailure pins that validation failure
|
||||
// prevents downstream execution.
|
||||
func TestActValidation_NoExecutionOnFailure(t *testing.T) {
|
||||
h, st := newActHandler(t)
|
||||
ctx := context.Background()
|
||||
now := h.now()
|
||||
|
||||
if err := st.EnableTool(ctx, "status", []string{"true"}, false, "test", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reply := h.actionAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "status",
|
||||
Slots: router.Slots{Fn: " ", HasFn: true},
|
||||
})
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Error("validation failure should not reach tool execution")
|
||||
}
|
||||
}
|
||||
+42
-22
@@ -11,54 +11,74 @@ import (
|
||||
"github.com/kami/maven/internal/tool"
|
||||
)
|
||||
|
||||
// actionAct handles router.IntentAct: match a verb to an enabled tool, offer
|
||||
// it to the ecosystems first, and run it behind the confirm gate and the
|
||||
// allowlist. proposeGap and the confirm gate itself live in confirm.go.
|
||||
// actionAct handles router.IntentAct: resolve the action, offer it to the
|
||||
// ecosystems first, and run it behind the confirm gate and the allowlist.
|
||||
// proposeGap and the confirm gate itself live in confirm.go.
|
||||
//
|
||||
// Action resolution happens in resolveAction (actionresolve.go) — a single
|
||||
// boundary that produces an ActionCandidate before execution. This function
|
||||
// consumes the candidate; it no longer decides which function/tool the user
|
||||
// meant.
|
||||
func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) string {
|
||||
// tool executor: run the matched fn against the enabled allowlist.
|
||||
// HasFn=false ⇒ try the matcher (for LLM-routed acts where the verb
|
||||
// didn't go through the stage-0 act grammar).
|
||||
if !dec.Slots.HasFn && dec.Slots.Text != "" && h.matcher != nil {
|
||||
if fn, args, ok := h.matcher.Match(dec.Slots.Text); ok {
|
||||
dec.Slots.Fn, dec.Slots.Args, dec.Slots.HasFn = fn, args, true
|
||||
}
|
||||
// An allowlist or a model route is evidence about WHAT could run, never
|
||||
// authority to run it. Keep the user's negative command at the execution
|
||||
// boundary too: actionAct is also reached by rebuilt decisions outside the
|
||||
// ordinary pre-route ladder.
|
||||
if refusesCommand(dec) {
|
||||
return commandProhibitionReply
|
||||
}
|
||||
|
||||
// Resolve the action: produce an ActionCandidate from the routing
|
||||
// decision. The candidate carries the resolved function, its arguments,
|
||||
// and where the resolution came from (route or matcher).
|
||||
candidate := h.resolveAction(ctx, dec)
|
||||
|
||||
// Structural validation: is this candidate complete enough to proceed?
|
||||
// Unresolved (Fn empty) flows to proposeGap; invalid (Fn present but
|
||||
// malformed) is refused; valid proceeds to execution.
|
||||
validation := router.ValidateActionCandidate(candidate)
|
||||
noteActionValidation(ctx, validation)
|
||||
|
||||
if !validation.Unresolved() && !validation.Valid() {
|
||||
// Resolved but structurally malformed: refuse execution.
|
||||
return phraser.A(phraser.ActFail, nil)
|
||||
}
|
||||
|
||||
// The board is Maven's own store, so a spoken status change is answered here
|
||||
// and never offered to an ecosystem client (Vikunja #512). First, because
|
||||
// task_status is on no allowlist and no capability registry: reaching either
|
||||
// of them would answer a turn about his own task list with a gap.
|
||||
if dec.Slots.Fn == router.TaskStatusFn {
|
||||
return h.resolveTaskStatus(ctx, dec)
|
||||
if candidate.Fn == router.TaskStatusFn {
|
||||
return h.resolveTaskStatus(ctx, dec, candidate)
|
||||
}
|
||||
|
||||
// Praxis ecosystem tools: intercept before the system command executor.
|
||||
if h.ecosystem != nil && h.ecosystem.praxis != nil && dec.Slots.HasFn {
|
||||
if reply := h.handlePraxisAct(ctx, dec); reply != "" {
|
||||
if h.ecosystem != nil && h.ecosystem.praxis != nil && candidate.ActionResolved() {
|
||||
if reply := h.handlePraxisAct(ctx, dec, candidate); reply != "" {
|
||||
return reply
|
||||
}
|
||||
}
|
||||
|
||||
// Hexis ecosystem action: if ecosystem is configured and we have a verb
|
||||
// + entity text, try to resolve the entity and execute via Hexis.
|
||||
if h.ecosystem != nil && h.ecosystem.hexis != nil && dec.Slots.Text != "" {
|
||||
if reply := h.handleHexisAct(ctx, dec); reply != "" {
|
||||
if h.ecosystem != nil && h.ecosystem.hexis != nil && router.ActHasEntityTarget(dec) {
|
||||
if reply := h.handleHexisAct(ctx, dec, candidate); reply != "" {
|
||||
return reply
|
||||
}
|
||||
}
|
||||
|
||||
// HasFn still false ⇒ no allowlist match: scaffold a 'proposed' tool
|
||||
// Unresolved candidate ⇒ no allowlist match: scaffold a 'proposed' tool
|
||||
// the user can enable on the authed surface ("earn the right to ask").
|
||||
if !dec.Slots.HasFn {
|
||||
if !candidate.ActionResolved() {
|
||||
return h.proposeGap(ctx, dec)
|
||||
}
|
||||
out, err := h.tools.Exec(ctx, dec.Slots.Fn, dec.Slots.Args, false)
|
||||
out, err := h.tools.Exec(ctx, candidate.Fn, candidate.Args, false)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, tool.ErrNeedsConfirm):
|
||||
// destructive: park it and ask. The next utterance answers.
|
||||
phrase := actPhrase(dec.Slots.Fn, dec.Slots.Args)
|
||||
h.park(dec.Slots.Fn, dec.Slots.Args, phrase)
|
||||
phrase := actPhrase(candidate.Fn, candidate.Args)
|
||||
h.park(candidate.Fn, candidate.Args, phrase)
|
||||
return phraser.A(phraser.ActConfirm, map[string]string{"name": phrase})
|
||||
case errors.Is(err, tool.ErrUnknownTarget):
|
||||
// The verb reached a tool and the tail did not reach a target, so
|
||||
@@ -93,7 +113,7 @@ func (h *reactiveHandler) actionAct(ctx context.Context, dec router.Decision) st
|
||||
// where a human types them.
|
||||
return phraser.A(phraser.ActNeedsArgs, nil)
|
||||
}
|
||||
log.Printf("voice: tool %s: %v", dec.Slots.Fn, err)
|
||||
log.Printf("voice: tool %s: %v", candidate.Fn, err)
|
||||
if out != "" {
|
||||
return phraser.A(phraser.ActFailOut, map[string]string{"out": firstLine(out)})
|
||||
}
|
||||
|
||||
@@ -38,6 +38,15 @@ func (h *reactiveHandler) actionFact(ctx context.Context, dec router.Decision) s
|
||||
// claim the turn before any real source ran.
|
||||
q.Slots.Key, q.Slots.HasKey = "", false
|
||||
q.Slots.Value = ""
|
||||
// Defensive reconstruction must preserve the same literal destination
|
||||
// the stage-0 router would have named. A learned fact decision has no
|
||||
// source, and without restoring this anchored world frame the personal
|
||||
// boundary can claim "latest Go version" by similarity and prevent the
|
||||
// live source from ever being asked.
|
||||
if world, ok := router.WorldQueryDecision(dec.Utterance); ok {
|
||||
q.Source = world.Source
|
||||
q.SourceAnchored = world.SourceAnchored
|
||||
}
|
||||
return h.actionQuery(ctx, q)
|
||||
}
|
||||
// A complaint is not a fact either (#481). "сеть какая-то медленная" and
|
||||
|
||||
@@ -16,10 +16,11 @@ const nothingToCorrectReply = "не поняла, что поправить. с
|
||||
// actionNote handles router.IntentNote: embed the note, persist it, and
|
||||
// index it for recall.
|
||||
//
|
||||
// The stored body is dec.Utterance and nothing else (V-576). It is not
|
||||
// Slots.Text, not phraser output and not any other model string: a note is
|
||||
// durable, the embedder indexes it, and it comes back later as recall in his
|
||||
// own words. Phrasing belongs in the spoken confirmation.
|
||||
// The stored body comes only from dec.Utterance (V-576/V-721). An explicit
|
||||
// leading capture frame is structurally removed; an unmarked note is otherwise
|
||||
// byte-for-byte his utterance. It is never Slots.Text, phraser output or any
|
||||
// other model string: a note is durable, the embedder indexes it, and it comes
|
||||
// back later as recall in his own words. Phrasing belongs in the confirmation.
|
||||
func (h *reactiveHandler) actionNote(ctx context.Context, dec router.Decision) string {
|
||||
// A correction with no referent. Everything that could own one has already
|
||||
// run by here: clarify, confirm and repair are all resolved before routing,
|
||||
@@ -39,16 +40,20 @@ func (h *reactiveHandler) actionNote(ctx context.Context, dec router.Decision) s
|
||||
if reply, ok := h.captureListFromNote(ctx, dec); ok {
|
||||
return reply
|
||||
}
|
||||
noteText := dec.Utterance
|
||||
if body, explicit := router.ParseNoteCapture(dec.Utterance); explicit {
|
||||
noteText = body
|
||||
}
|
||||
// embed the note text with the same model the classifier uses, persist
|
||||
// via CoreAPI (source=tap:voice). Semantic recall lives in `notes`, not
|
||||
// facts — no predicate reads it (spec's two-memory split).
|
||||
vec, err := router.EmbedPassage(ctx, h.recall.embedder, dec.Utterance)
|
||||
vec, err := router.EmbedPassage(ctx, h.recall.embedder, noteText)
|
||||
if err != nil {
|
||||
log.Printf("voice: embed note: %v", err)
|
||||
return phraser.Ack(phraser.FailNote, nil)
|
||||
}
|
||||
noteTs := h.now()
|
||||
noteID, err := h.api.WriteNote(ctx, noteTs, dec.Utterance, vec, "tap:voice")
|
||||
noteID, err := h.api.WriteNote(ctx, noteTs, noteText, vec, "tap:voice")
|
||||
if err != nil {
|
||||
log.Printf("voice: write note: %v", err)
|
||||
return phraser.Ack(phraser.FailNote, nil)
|
||||
@@ -59,7 +64,7 @@ func (h *reactiveHandler) actionNote(ctx context.Context, dec router.Decision) s
|
||||
if err := h.recall.memStore.Insert(ctx, "note:"+strconv.FormatInt(noteID, 10), vec, map[string]string{
|
||||
"source": "voice",
|
||||
"type": "note",
|
||||
"text": dec.Utterance,
|
||||
"text": noteText,
|
||||
"ts": strconv.FormatInt(noteTs.Unix(), 10),
|
||||
}); err != nil {
|
||||
log.Printf("voice: memory insert: %v", err)
|
||||
|
||||
@@ -641,10 +641,11 @@ const (
|
||||
// небо синее?", because the right-note and must-be-silent score ranges overlap
|
||||
// and no threshold sits between them.
|
||||
func recallOnTopic(utterance, text string) bool {
|
||||
if memory.RecallAllowed(utterance, text) {
|
||||
if memory.RecallAllowed(utterance, text,
|
||||
router.IsOpenQuestionShaped(utterance), router.IsLocativeQuestionShaped(utterance)) {
|
||||
return true
|
||||
}
|
||||
log.Printf("voice: recall %q rejected for %q: a world question and no shared topic word", text, utterance)
|
||||
log.Printf("voice: recall %q rejected for %q: no structural ask with a shared topic, or a world/locative question with no shared topic", text, utterance)
|
||||
return false
|
||||
}
|
||||
|
||||
|
||||
@@ -14,6 +14,12 @@ import (
|
||||
// actionReminder handles router.IntentReminder: parse the time when stage-0
|
||||
// skipped the extractor, then create the reminder.
|
||||
func (h *reactiveHandler) actionReminder(ctx context.Context, dec router.Decision) string {
|
||||
// The pre-route belt normally answers this before routing. Keep the write
|
||||
// boundary guarded as well: a model calling the sentence a reminder does
|
||||
// not turn "don't ..." into permission to create a row.
|
||||
if refusesCommand(dec) {
|
||||
return commandProhibitionReply
|
||||
}
|
||||
if !dec.Slots.HasTime {
|
||||
// Stage-0 (reminder-wakeword grammar) skips the extractor, so the
|
||||
// time wasn't parsed. Run the parser as a fallback.
|
||||
|
||||
@@ -106,7 +106,7 @@ func (h *reactiveHandler) queryTasks(ctx context.Context, t *queryTurn) (string,
|
||||
// match on more than one asks which, because closing the wrong task is work he
|
||||
// never finished being marked done. No task named asks which too, since the
|
||||
// router claims the turn without the referent and the list lives here.
|
||||
func (h *reactiveHandler) resolveTaskStatus(ctx context.Context, dec router.Decision) string {
|
||||
func (h *reactiveHandler) resolveTaskStatus(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
|
||||
live, err := h.api.ListTasks(ctx, "live")
|
||||
if err != nil {
|
||||
log.Printf("voice: task status: list: %v", err)
|
||||
|
||||
@@ -279,7 +279,7 @@ func TestResolveTaskStatusMovesTheNamedTask(t *testing.T) {
|
||||
reply := h.resolveTaskStatus(context.Background(), router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: "молоко"},
|
||||
})
|
||||
}, routeCandidate(router.TaskStatusFn))
|
||||
if api.listArg != "live" {
|
||||
t.Errorf("listed %q, want live — a resolved task cannot be resolved again", api.listArg)
|
||||
}
|
||||
@@ -294,6 +294,36 @@ func TestResolveTaskStatusMovesTheNamedTask(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The regression crosses the grammar/action seam instead of handing the
|
||||
// action a repaired Decision. The stored title is a normal imperative title,
|
||||
// while the spoken marker names only its topic; framing words must not become
|
||||
// identity and the unrelated live row must remain untouched.
|
||||
func TestActionActMarkerReferentMovesOnlyTheNamedStoredTask(t *testing.T) {
|
||||
api := &taskAPI{tasks: []ipc.Task{
|
||||
{ID: 17, Text: "настроить бэкапы", Status: "open"},
|
||||
{ID: 18, Text: "обновить сертификаты", Status: "open"},
|
||||
}}
|
||||
h := taskHandler(api)
|
||||
dec, matched, accepted := router.TaskStatusGrammar().Evaluate("отметь задачу про бэкапы как сделанную")
|
||||
if !matched || !accepted {
|
||||
t.Fatalf("task-status grammar matched=%v accepted=%v", matched, accepted)
|
||||
}
|
||||
|
||||
reply := h.actionAct(context.Background(), dec)
|
||||
if api.listArg != "live" {
|
||||
t.Errorf("listed %q, want live", api.listArg)
|
||||
}
|
||||
if len(api.moved) != 1 {
|
||||
t.Fatalf("moved %+v, want exactly the named stored task", api.moved)
|
||||
}
|
||||
if got := api.moved[0]; got.id != 17 || got.status != "done" || got.by != "tap:voice" {
|
||||
t.Errorf("moved %+v, want task 17 → done by tap:voice", got)
|
||||
}
|
||||
if !strings.Contains(reply, "настроить бэкапы") {
|
||||
t.Errorf("reply = %q, want the transitioned stored title", reply)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveTaskStatusRefusesToGuess(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
@@ -314,7 +344,7 @@ func TestResolveTaskStatusRefusesToGuess(t *testing.T) {
|
||||
h := taskHandler(api)
|
||||
reply := h.resolveTaskStatus(context.Background(), router.Decision{
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: c.named},
|
||||
})
|
||||
}, routeCandidate(router.TaskStatusFn))
|
||||
if len(api.moved) != 0 {
|
||||
t.Errorf("moved %+v — closing the wrong task is the failure this arm exists to avoid", api.moved)
|
||||
}
|
||||
@@ -332,7 +362,7 @@ func TestResolveTaskStatusOpensACandidateFirst(t *testing.T) {
|
||||
h := taskHandler(api)
|
||||
h.resolveTaskStatus(context.Background(), router.Decision{
|
||||
Slots: router.Slots{Fn: router.TaskStatusFn, HasFn: true, Value: "done", Text: "продлить домен"},
|
||||
})
|
||||
}, routeCandidate(router.TaskStatusFn))
|
||||
if len(api.moved) != 2 {
|
||||
t.Fatalf("moved %+v, want open then done", api.moved)
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ func TestAttentionEmptyWithHealthySourcesIsAllClear(t *testing.T) {
|
||||
praxis := newFakePraxisWithSources(t, `[]`, `[{"source_id":"src_ntfy","health":"ok"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("healthy and quiet should be an all-clear, got %q", reply)
|
||||
}
|
||||
@@ -28,7 +28,7 @@ func TestAttentionEmptyWithAFailedSourceHedges(t *testing.T) {
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("a failed source must not read as all-clear, got %q", reply)
|
||||
}
|
||||
@@ -47,7 +47,7 @@ func TestAttentionEmptyWithNoSourcesHedges(t *testing.T) {
|
||||
praxis := newFakePraxisWithSources(t, `[]`, `[]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("a Praxis with no sources must not answer all-clear, got %q", reply)
|
||||
}
|
||||
@@ -64,7 +64,7 @@ func TestAttentionDegradedEnvelopeIsReadWithoutASourcesCall(t *testing.T) {
|
||||
`[{"source_id":"src_ntfy","health":"ok"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "src_metrics") {
|
||||
t.Fatalf("the envelope's degraded source is not named: %q", reply)
|
||||
}
|
||||
@@ -82,7 +82,7 @@ func TestAttentionKeepsAllClearWhenSourcesCannotBeRead(t *testing.T) {
|
||||
praxis.SetRouteFault("/api/v1/sources", 500)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "ничего не требует внимания") {
|
||||
t.Fatalf("an unreadable sources list should leave the answer alone, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -63,6 +63,9 @@ func (h *reactiveHandler) queryAttention(ctx context.Context, t *queryTurn) (str
|
||||
Utterance: t.dec.Utterance,
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: "list_attention", HasFn: true},
|
||||
}, router.ActionCandidate{
|
||||
Fn: "list_attention",
|
||||
Source: router.ActionSourceRoute,
|
||||
})
|
||||
if reply == "" {
|
||||
return "", false
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/decision"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// TestTextAndVoiceConvergeOnNormalizedInput — both entry points construct a
|
||||
// NormalizedInput and pass it to runTurn. The same utterance produces the same
|
||||
// route intent regardless of whether it arrived as text or voice.
|
||||
func TestTextAndVoiceConvergeOnNormalizedInput(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
|
||||
utterance := "который час"
|
||||
voiceCtx := withDialogueID(ctx, dialogueIDFor(sourceVoice, ""))
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
|
||||
voiceReply := h.runTurn(voiceCtx, router.NormalizedInput{Text: utterance, Source: sourceVoice})
|
||||
textReply := h.runTurn(textCtx, router.NormalizedInput{Text: utterance, Source: sourceText})
|
||||
|
||||
// Both paths should produce the same kind of reply (time answer).
|
||||
for _, pair := range []struct {
|
||||
label, reply string
|
||||
}{
|
||||
{"voice", voiceReply},
|
||||
{"text", textReply},
|
||||
} {
|
||||
if !strings.Contains(pair.reply, "час") && !strings.Contains(pair.reply, "время") {
|
||||
t.Errorf("%s reply %q does not look like a time answer", pair.label, pair.reply)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestNormalizedInputSourcePreserved — the source survives into the decision
|
||||
// record so a trace can tell voice from text.
|
||||
func TestNormalizedInputSourcePreserved(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
h.runTurn(textCtx, router.NormalizedInput{Text: "привет", Source: sourceText})
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Fatal("no decision record")
|
||||
}
|
||||
if recs[0].InputSource != string(sourceText) {
|
||||
t.Errorf("InputSource = %q, want %q", recs[0].InputSource, sourceText)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRouteProducerOnDecisionRecord — the producer is carried from the router
|
||||
// decision into the decision record for observability.
|
||||
func TestRouteProducerOnDecisionRecord(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
h.runTurn(textCtx, router.NormalizedInput{Text: "который час", Source: sourceText})
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Fatal("no decision record")
|
||||
}
|
||||
// A time query is a stage-0 grammar match.
|
||||
if recs[0].RouteProducer != string(router.RouteProducerGrammar) {
|
||||
t.Errorf("RouteProducer = %q, want %q", recs[0].RouteProducer, router.RouteProducerGrammar)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPreRouteClaimHasNoRouteProducer — a turn claimed by a pre-route resolver
|
||||
// never reaches the router, so the record's RouteProducer must be empty.
|
||||
func TestPreRouteClaimHasNoRouteProducer(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
// Park a confirm so the next "да" is consumed before routing.
|
||||
// newRoutingClarifyHandler uses a fixed clock at 2026-07-31 09:00 UTC.
|
||||
h.pending = &pendingAct{
|
||||
fn: "test",
|
||||
phrase: "delete everything",
|
||||
expiry: time.Date(2026, 7, 31, 9, 1, 0, 0, time.UTC),
|
||||
}
|
||||
ctx := context.Background()
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
h.runTurn(textCtx, router.NormalizedInput{Text: "да", Source: sourceText})
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Fatal("no decision record")
|
||||
}
|
||||
if recs[0].RouteProducer != "" {
|
||||
t.Errorf("RouteProducer = %q, want empty (pre-route claimed the turn)", recs[0].RouteProducer)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStage0ProducerUnchanged — grammars still produce the exact same intents
|
||||
// at confidence 1.0. This pins stage-0 behavior through the new boundary.
|
||||
func TestStage0ProducerUnchanged(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
h.decisions = decision.NewRing()
|
||||
ctx := context.Background()
|
||||
textCtx := withDialogueID(ctx, dialogueIDFor(sourceText, "test"))
|
||||
|
||||
cases := []struct {
|
||||
utterance string
|
||||
intent router.Intent
|
||||
}{
|
||||
{"напомни позвонить маме завтра", router.IntentReminder},
|
||||
{"который час", router.IntentSystem},
|
||||
}
|
||||
for _, c := range cases {
|
||||
reply := h.runTurn(textCtx, router.NormalizedInput{Text: c.utterance, Source: sourceText})
|
||||
_ = reply // behavior unchanged; we test the record, not the reply text.
|
||||
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) == 0 {
|
||||
t.Errorf("%s: no decision record", c.utterance)
|
||||
continue
|
||||
}
|
||||
rec := recs[0]
|
||||
if rec.RouteProducer != string(router.RouteProducerGrammar) {
|
||||
t.Errorf("%s: RouteProducer = %q, want %q", c.utterance, rec.RouteProducer, router.RouteProducerGrammar)
|
||||
}
|
||||
// Clear the ring for the next case.
|
||||
h.decisions = decision.NewRing()
|
||||
}
|
||||
}
|
||||
@@ -439,7 +439,7 @@ func TestClarifySecondGapExhaustionResumesLowerFlow(t *testing.T) {
|
||||
h.clarifyStore.Push(voiceDialogueID, older)
|
||||
h.clarifyStore.Push(voiceDialogueID, top)
|
||||
|
||||
reply := h.runTurn(ctx, "купить хлеб", sourceText)
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "купить хлеб", Source: sourceText})
|
||||
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
|
||||
want := withResumed(clarifyGaveUp, resumed)
|
||||
if reply != want {
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// commandProhibitionReply is deliberately operation-neutral. At this boundary
|
||||
// Maven may know only that the user denied authority, not whether the model
|
||||
// would have called it a reminder, board transition, local tool or Hexis act.
|
||||
const commandProhibitionReply = "хорошо, не буду."
|
||||
|
||||
// resolveCommandProhibition is the first mutation boundary in a turn. It runs
|
||||
// before a parked clarify answer or candidate selection can consume the words,
|
||||
// and before any route/model is consulted. A direct prohibition is complete in
|
||||
// itself: it needs no target lookup and makes no external call.
|
||||
//
|
||||
// A parked clarify request is unrelated state. Preserve it and say the pending
|
||||
// question again, using the same bounded suspend policy as every other side
|
||||
// request. Candidate lists likewise remain untouched; no ordinal was selected.
|
||||
func (h *reactiveHandler) resolveCommandProhibition(ctx context.Context, text string) (string, bool) {
|
||||
if !router.IsCommandProhibition(text) {
|
||||
return "", false
|
||||
}
|
||||
// A later bare "да" must not revive authority the user has just revoked.
|
||||
// Confirmation slots are all mutation authority and are process-local, so
|
||||
// clearing the three under their shared mutex is both conservative and
|
||||
// atomic. Clarify questions and candidate lists are not authority and stay.
|
||||
h.mu.Lock()
|
||||
h.pending = nil
|
||||
h.pendingHexis = nil
|
||||
h.pendingRoutine = nil
|
||||
h.mu.Unlock()
|
||||
if h.clarifyStore != nil {
|
||||
if q := h.clarifyStore.Get(dialogueIDOf(ctx), h.now()); q != nil {
|
||||
h.noteSuspended(ctx, q)
|
||||
}
|
||||
}
|
||||
return commandProhibitionReply, true
|
||||
}
|
||||
|
||||
// refusesCommand is the defense-in-depth form for execution entry points which
|
||||
// can also be called with a reconstructed or test decision outside runTurn.
|
||||
// The sentinel cannot be renamed into an enabled function, and the original
|
||||
// utterance remains the authority even when a model rewrites Slots.Text.
|
||||
func refusesCommand(dec router.Decision) bool {
|
||||
return dec.CapabilitySelection.Fn == router.ProhibitedActFn || router.IsCommandProhibition(dec.Utterance)
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// TestRefusesCommandUsesCapabilitySelection proves that refusesCommand reads
|
||||
// CapabilitySelection.Fn rather than the compatibility Slots.Fn. When
|
||||
// CapabilitySelection is populated with the prohibited sentinel but Slots.Fn
|
||||
// is blank, the refusal must still fire.
|
||||
func TestRefusesCommandUsesCapabilitySelection(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
CapabilitySelection: router.CapabilitySelection{
|
||||
Fn: router.ProhibitedActFn,
|
||||
Resolved: true,
|
||||
},
|
||||
// Slots compatibility fields deliberately blank.
|
||||
}
|
||||
if !refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return true when CapabilitySelection.Fn == ProhibitedActFn")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusesCommandUtteranceFallback proves that the utterance-based
|
||||
// prohibition check still works as a defense-in-depth belt when
|
||||
// CapabilitySelection does not carry the sentinel.
|
||||
func TestRefusesCommandUtteranceFallback(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Utterance: "don't restart nginx",
|
||||
}
|
||||
if !refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return true for a prohibited utterance")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusesCommandNonProhibitedCapability proves that an act with a
|
||||
// non-prohibited capability is NOT refused even when Slots.Fn happens to
|
||||
// carry the prohibited sentinel (cross-contamination).
|
||||
func TestRefusesCommandNonProhibitedCapability(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Utterance: "restart nginx",
|
||||
CapabilitySelection: router.CapabilitySelection{
|
||||
Fn: "restart",
|
||||
Resolved: true,
|
||||
},
|
||||
}
|
||||
if refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return false for a non-prohibited capability")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRefusesCommandProhibitedSentinelPreservedThroughPipeline proves that the
|
||||
// command-prohibition grammar sentinel survives through SelectCapability into
|
||||
// CapabilitySelection byte-for-byte.
|
||||
func TestRefusesCommandProhibitedSentinelPreservedThroughPipeline(t *testing.T) {
|
||||
dec := router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{
|
||||
Fn: router.ProhibitedActFn,
|
||||
HasFn: true,
|
||||
},
|
||||
}
|
||||
sel := router.SelectCapability(dec, nil)
|
||||
dec.CapabilitySelection = sel
|
||||
|
||||
if dec.CapabilitySelection.Fn != router.ProhibitedActFn {
|
||||
t.Errorf("CapabilitySelection.Fn = %q, want %q", dec.CapabilitySelection.Fn, router.ProhibitedActFn)
|
||||
}
|
||||
if !dec.CapabilitySelection.Resolved {
|
||||
t.Error("CapabilitySelection.Resolved should be true")
|
||||
}
|
||||
if !refusesCommand(dec) {
|
||||
t.Fatal("refusesCommand should return true after pipeline preserves the sentinel")
|
||||
}
|
||||
}
|
||||
@@ -31,8 +31,8 @@ import (
|
||||
// and nothing should: a missing name costs one line of the record, while a
|
||||
// check that walks the ladder would have to run the ladder.
|
||||
var preRouteLadder = []string{
|
||||
"confirm", "repair", "repair-negative", "clarify-answer", "quiet-toggle",
|
||||
"snooze", "ack", "ordinal",
|
||||
"confirm", "repair", "repair-negative", "command-prohibition", "clarify-answer", "quiet-toggle",
|
||||
"snooze", "ack", "reminder-cancel", "ordinal",
|
||||
}
|
||||
|
||||
// notePreRoute records one rung of that ladder and passes its verdict through
|
||||
|
||||
@@ -107,7 +107,7 @@ var praxisCapabilities = []praxisCapability{
|
||||
// handlePraxisAct — dispatches ecosystem tool acts through the Praxis tools API.
|
||||
// Returns "" when the act is not a Praxis verb (the caller falls through to the
|
||||
// system command executor). Returns a reply string otherwise.
|
||||
func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decision) string {
|
||||
func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
|
||||
if h.ecosystem == nil || h.ecosystem.praxis == nil {
|
||||
return ""
|
||||
}
|
||||
@@ -127,7 +127,7 @@ func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decisi
|
||||
}
|
||||
for _, capability := range praxisCapabilities {
|
||||
for _, alias := range capability.aliases() {
|
||||
if alias == dec.Slots.Fn {
|
||||
if alias == candidate.Fn {
|
||||
return capability.handle(ctx, h, px, dec)
|
||||
}
|
||||
}
|
||||
@@ -657,7 +657,13 @@ func (h *reactiveHandler) resolveEntityCandidates(ctx context.Context, refs []st
|
||||
// handleHexisAct — resolves entity references through Nexus and executes
|
||||
// matching capabilities through Hexis. Returns a reply string when handled,
|
||||
// or "" to fall through to the system command executor.
|
||||
func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decision) string {
|
||||
func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decision, candidate router.ActionCandidate) string {
|
||||
// This method is intentionally callable outside runTurn by ecosystem
|
||||
// harnesses. Refuse before correlation ids, Nexus resolution or capability
|
||||
// discovery so the no-op sentinel can never leak into Hexis as a verb.
|
||||
if refusesCommand(dec) {
|
||||
return commandProhibitionReply
|
||||
}
|
||||
if h.ecosystem == nil {
|
||||
return ""
|
||||
}
|
||||
@@ -715,7 +721,7 @@ func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decisio
|
||||
// Match the user's verb to a capability by name/description. Collect all
|
||||
// matches: more than one is itself ambiguous, so we ask rather than pick
|
||||
// the first (ecosystem invariant: no arbitrary target for mutation).
|
||||
verb := dec.Slots.Fn
|
||||
verb := candidate.Fn
|
||||
if verb == "" {
|
||||
verb = dec.Slots.Text
|
||||
}
|
||||
@@ -726,7 +732,7 @@ func (h *reactiveHandler) handleHexisAct(ctx context.Context, dec router.Decisio
|
||||
// round then: the phrase is the haystack and the capability name is what we
|
||||
// look for in it (Vikunja #476). Only when the fn slot is empty — a matched
|
||||
// fn is a single verb and containment already means what it says.
|
||||
loose := !dec.Slots.HasFn
|
||||
loose := !candidate.ActionResolved()
|
||||
var matches []*hexisclient.Capability
|
||||
for i, c := range caps {
|
||||
name := strings.ToLower(c.Name)
|
||||
@@ -841,13 +847,27 @@ func (h *reactiveHandler) execHexis(ctx context.Context, capID, capName, entityI
|
||||
// resolution stops on ambiguity and a mutating capability still goes through
|
||||
// the spoken confirm in handleHexisAct.
|
||||
func (h *reactiveHandler) hexisBeforeClarify(ctx context.Context, dec router.Decision) string {
|
||||
// A thinned model act reaches this hook before actionAct. Negative authority
|
||||
// must therefore stop here as well, before even a read to Nexus/Hexis.
|
||||
if refusesCommand(dec) {
|
||||
return commandProhibitionReply
|
||||
}
|
||||
if h.ecosystem == nil || h.ecosystem.hexis == nil {
|
||||
return ""
|
||||
}
|
||||
if dec.Intent != router.IntentAct || dec.Slots.HasFn || dec.Slots.Text == "" {
|
||||
if dec.Intent != router.IntentAct || dec.Slots.HasFn || !router.ActHasEntityTarget(dec) {
|
||||
return ""
|
||||
}
|
||||
return h.handleHexisAct(ctx, dec)
|
||||
// Resolve the action candidate. Use the matcher when available; when the
|
||||
// handler has no matcher (ecosystem-only test harnesses), build an
|
||||
// unresolved candidate directly — the matcher would not have matched either.
|
||||
var candidate router.ActionCandidate
|
||||
if h.matcher != nil {
|
||||
candidate = h.resolveAction(ctx, dec)
|
||||
} else {
|
||||
candidate = router.ResolveActionCandidate(dec, nil)
|
||||
}
|
||||
return h.handleHexisAct(ctx, dec, candidate)
|
||||
}
|
||||
|
||||
// attentionCannotTell returns the hedge to say instead of an all-clear, or ""
|
||||
|
||||
@@ -96,7 +96,7 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
|
||||
// A Nexus outage during a Hexis act writes a failure trace, and a shared
|
||||
// store is the one thing the Praxis path could inherit it through.
|
||||
nexus.SetFault(503)
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); actRan(reply) {
|
||||
t.Fatalf("nexus outage must not report success, got %q", reply)
|
||||
}
|
||||
if len(tracesFor(t, h, "nexus", "resolve")) == 0 {
|
||||
@@ -104,7 +104,7 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
|
||||
}
|
||||
|
||||
nexus.SetFault(0)
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("a recorded nexus failure must not degrade the praxis digest, got %q", reply)
|
||||
}
|
||||
@@ -114,10 +114,10 @@ func TestEcosystem_OutagesLeaveNoSharedFailureState(t *testing.T) {
|
||||
|
||||
// And the reverse: a Praxis outage mid-session leaves the Hexis path whole.
|
||||
praxis.SetFault(503)
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); strings.Contains(reply, "disk") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); strings.Contains(reply, "disk") {
|
||||
t.Fatalf("praxis outage must not serve content, got %q", reply)
|
||||
}
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("a praxis outage must not block the hexis path, got %q", reply)
|
||||
}
|
||||
}
|
||||
@@ -132,7 +132,7 @@ func TestEcosystem_OneEndpointDownDoesNotMuteTheService(t *testing.T) {
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
praxis.SetRouteFault("/api/v1/tools/surface", 503)
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("a downed surface endpoint must not mute the digest, got %q", reply)
|
||||
}
|
||||
@@ -150,7 +150,7 @@ func TestEcosystem_ResolvedWithoutEntityFailsClosed(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("a resolve with no entity must degrade, not fall through to local execution")
|
||||
}
|
||||
@@ -172,7 +172,7 @@ func TestEcosystem_RejectedCredentialSaysSo(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
nexus.SetFault(status)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "токен") {
|
||||
t.Fatalf("http %d must read as a credential problem, got %q", status, reply)
|
||||
}
|
||||
@@ -193,7 +193,7 @@ func TestEcosystem_MalformedPraxisBodyDegrades(t *testing.T) {
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
praxis.SetBody(`[{"title":`)
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if reply == "" {
|
||||
t.Fatal("a malformed praxis body must not answer with silence")
|
||||
}
|
||||
@@ -211,7 +211,7 @@ func TestEcosystem_MalformedNexusResponseFailsClosed(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
nexus.SetBody(`{"status":"resolved","entity":`)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" || actRan(reply) {
|
||||
t.Fatalf("malformed nexus body must degrade, got %q", reply)
|
||||
}
|
||||
@@ -232,7 +232,7 @@ func TestEcosystem_UnknownContractFieldsTolerated(t *testing.T) {
|
||||
nexus := newFakeNexus(t, body)
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("%s contract shape must still resolve and execute, got %q", name, reply)
|
||||
}
|
||||
})
|
||||
@@ -249,7 +249,7 @@ func TestEcosystem_CancelledContextDegrades(t *testing.T) {
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" || actRan(reply) {
|
||||
t.Fatalf("cancelled resolve must degrade, got %q", reply)
|
||||
}
|
||||
@@ -267,7 +267,7 @@ func TestEcosystem_ExecutionFailureIsNotSuccess(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecutionFailed("exec_1", "unit not found"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if actRan(reply) {
|
||||
t.Fatalf("failed execution must not read as success, got %q", reply)
|
||||
}
|
||||
@@ -291,7 +291,7 @@ func TestEcosystem_SuccessfulActionWritesATrace(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("setup: expected success, got %q", reply)
|
||||
}
|
||||
exec := tracesFor(t, h, "hexis", "execute")
|
||||
@@ -313,7 +313,7 @@ func TestEcosystem_TracesStayOutOfFacts(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("setup: expected success, got %q", reply)
|
||||
}
|
||||
if len(traces(t, h)) == 0 {
|
||||
@@ -339,7 +339,7 @@ func TestEcosystem_AmbiguousTargetBlocksExecution(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Muzick web") {
|
||||
t.Fatalf("ambiguous resolve must list candidates, got %q", reply)
|
||||
}
|
||||
@@ -360,7 +360,7 @@ func TestEcosystem_NoAutonomousPraxisToHexis(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, praxis, hexis)
|
||||
|
||||
_ = h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
_ = h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if hexis.Count("", "/api/v1") != 0 {
|
||||
t.Fatal("attention digest must not contact hexis on its own")
|
||||
}
|
||||
@@ -378,7 +378,7 @@ func TestEcosystem_MutatingCapabilityWaitsForConfirmation(t *testing.T) {
|
||||
hexis := newFakeHexis(t, caps, fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("restart"))
|
||||
reply := h.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "restart") || !strings.Contains(reply, "да") {
|
||||
t.Fatalf("mutating capability must ask for confirmation, got %q", reply)
|
||||
}
|
||||
@@ -404,7 +404,7 @@ func TestEcosystem_SurfaceFailureStillDelivers(t *testing.T) {
|
||||
praxis.SetRouteFault("/api/v1/tools/surface", 500)
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("failed surface must not swallow the digest, got %q", reply)
|
||||
}
|
||||
@@ -426,10 +426,10 @@ func TestEcosystem_TotalOutageSaysSoForEveryPath(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, praxis, hexis)
|
||||
|
||||
for name, reply := range map[string]string{
|
||||
"hexis act": h.handleHexisAct(ctx, actDec("muzick indexer")),
|
||||
"attention": h.handlePraxisAct(ctx, praxisActDec("list_attention")),
|
||||
"changes": h.handlePraxisAct(ctx, praxisActDec("list_changes")),
|
||||
"acknowledge": h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1")),
|
||||
"hexis act": h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")),
|
||||
"attention": h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")),
|
||||
"changes": h.handlePraxisAct(ctx, praxisActDec("list_changes"), routeCandidate("list_changes")),
|
||||
"acknowledge": h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item")),
|
||||
} {
|
||||
if reply == "" {
|
||||
t.Errorf("%s: total outage must not answer with silence", name)
|
||||
@@ -458,11 +458,11 @@ func TestEcosystem_RecoveryAfterOutageNeedsNoRestart(t *testing.T) {
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
praxis.SetFault(503)
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); strings.Contains(reply, "disk") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); strings.Contains(reply, "disk") {
|
||||
t.Fatalf("outage must not serve content, got %q", reply)
|
||||
}
|
||||
praxis.SetFault(0)
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("recovery must work on the next turn, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -59,7 +59,7 @@ func TestHexisDiscovery401IsDeniedNotDown(t *testing.T) {
|
||||
h := hexisGapHandler(t, nexus.URL, hexis.URL)
|
||||
|
||||
hexis.SetFault(401)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !denied(serviceHexis, reply) {
|
||||
t.Fatalf("401 from hexis discovery: got %q, want the denied line naming Hexis", reply)
|
||||
}
|
||||
@@ -75,7 +75,7 @@ func TestHexisDiscoveryOutageIsDownNotDenied(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusResolved("ent_muzick", muzickIndexer, "service"))
|
||||
h := hexisGapHandler(t, nexus.URL, unreachableURL)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !down(serviceHexis, reply) {
|
||||
t.Fatalf("connection refused from hexis: got %q, want the outage line naming Hexis", reply)
|
||||
}
|
||||
@@ -96,7 +96,7 @@ func TestHexisExecute401IsDeniedNotCommandFailure(t *testing.T) {
|
||||
// Discovery stays healthy; only the execute endpoint refuses. A blanket
|
||||
// fault would never reach the site under test.
|
||||
hexis.SetRouteFault("/api/v1/execute", 401)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !denied(serviceHexis, reply) {
|
||||
t.Fatalf("401 from hexis execute: got %q, want the denied line naming Hexis", reply)
|
||||
}
|
||||
@@ -129,7 +129,7 @@ func TestHexisExecuteOutageIsDown(t *testing.T) {
|
||||
t.Cleanup(hexis.Close)
|
||||
h := hexisGapHandler(t, nexus.URL, hexis.URL)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !down(serviceHexis, reply) {
|
||||
t.Fatalf("dropped connection on hexis execute: got %q, want the outage line", reply)
|
||||
}
|
||||
@@ -149,7 +149,7 @@ func TestHexisExecutionFailedStaysCommandFailure(t *testing.T) {
|
||||
hexis := newFakeHexis(t, caps, fixtureHexisExecutionFailed("exec_1", "unit refused to start"))
|
||||
h := hexisGapHandler(t, nexus.URL, hexis.URL)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if down(serviceHexis, reply) || denied(serviceHexis, reply) {
|
||||
t.Fatalf("a failed execution must not be reported as an ecosystem gap, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -19,6 +19,11 @@ func praxisActDec(fn string) router.Decision {
|
||||
return router.Decision{Intent: router.IntentAct, Slots: router.Slots{Fn: fn, HasFn: true}}
|
||||
}
|
||||
|
||||
// routeCandidate builds an ActionCandidate matching a route-resolved Decision.
|
||||
func routeCandidate(fn string) router.ActionCandidate {
|
||||
return router.ActionCandidate{Fn: fn, Source: router.ActionSourceRoute}
|
||||
}
|
||||
|
||||
// praxisItemDec is praxisActDec for the lifecycle verbs, which need an item id
|
||||
// in the value slot. Without one they answer "which item?" and never reach
|
||||
// Praxis at all, which makes them useless for testing a Praxis outage.
|
||||
@@ -46,7 +51,7 @@ func TestPraxisAttention_HappyPathSurfacesItems(t *testing.T) {
|
||||
praxis := newFakePraxis(t, items)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("expected attention digest to mention the item, got %q", reply)
|
||||
}
|
||||
@@ -77,7 +82,7 @@ func TestPraxisAttention_DegradedFailsClosedNotEmpty(t *testing.T) {
|
||||
praxis.SetFault(500)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"))
|
||||
reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
if reply == "" {
|
||||
t.Fatal("praxis outage must not produce an empty reply")
|
||||
}
|
||||
@@ -104,13 +109,13 @@ func TestFakeNexus_FaultInjectionThenRecovery(t *testing.T) {
|
||||
}
|
||||
|
||||
nexus.SetFault(503)
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if actRan(reply) {
|
||||
t.Fatalf("nexus outage must not report success, got %q", reply)
|
||||
}
|
||||
|
||||
nexus.SetFault(0)
|
||||
reply = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !actRan(reply) {
|
||||
t.Fatalf("expected success once nexus recovers, got %q", reply)
|
||||
}
|
||||
@@ -134,7 +139,7 @@ func TestPraxisEntityAttention_RemembersWhatItReadOut(t *testing.T) {
|
||||
reply := h.handlePraxisAct(ctx, router.Decision{
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: "entity_attention", HasFn: true, Value: "muzick indexer"},
|
||||
})
|
||||
}, routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "indexer wedged") {
|
||||
t.Fatalf("expected the scoped item to be read out, got %q", reply)
|
||||
}
|
||||
@@ -147,7 +152,7 @@ func TestPraxisEntityAttention_RemembersWhatItReadOut(t *testing.T) {
|
||||
}
|
||||
|
||||
// The follow-up resolves against what he just heard, not the stale list.
|
||||
if reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "last")); reply == "" {
|
||||
if reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "last"), routeCandidate("resolve_item")); reply == "" {
|
||||
t.Fatal("positional follow-up should have been claimed by praxis")
|
||||
}
|
||||
var body string
|
||||
@@ -175,7 +180,7 @@ func TestHexisConfirm_KeepsOneCorrelationIDPerAction(t *testing.T) {
|
||||
hexis := newFakeHexis(t, caps, fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("restart")); !strings.Contains(reply, "да") {
|
||||
if reply := h.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart")); !strings.Contains(reply, "да") {
|
||||
t.Fatalf("mutating capability must ask for confirmation, got %q", reply)
|
||||
}
|
||||
resolve := findTrace(t, h, "nexus", "resolve")
|
||||
|
||||
@@ -73,7 +73,7 @@ func TestHexisMutatingRequiresConfirm(t *testing.T) {
|
||||
caps := `[{"id":"cap_restart","name":"restart","read_only":false,"risk":"high"}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "да") {
|
||||
t.Fatalf("mutating cap should ask to confirm, got %q", reply)
|
||||
}
|
||||
@@ -103,7 +103,7 @@ func TestHexisConfirmNoDoesNotExecute(t *testing.T) {
|
||||
caps := `[{"id":"cap_restart","name":"restart","read_only":false}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
reply, handled := h.resolveConfirm(ctx, "нет")
|
||||
if !handled || !strings.Contains(reply, "отменила") {
|
||||
t.Fatalf("no should cancel, got handled=%v reply=%q", handled, reply)
|
||||
@@ -119,7 +119,7 @@ func TestHexisReadOnlyExecutesImmediately(t *testing.T) {
|
||||
caps := `[{"id":"cap_status","name":"restart","read_only":true}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !*executed {
|
||||
t.Fatal("read-only cap should execute without confirmation")
|
||||
}
|
||||
@@ -136,7 +136,7 @@ func TestHexisAmbiguousAsksClarification(t *testing.T) {
|
||||
ambiguous := `{"status":"ambiguous","candidates":[{"entity_id":"ent_muzick","display_name":"Muzick indexer"},{"entity_id":"ent_manga","display_name":"Manga indexer"}]}`
|
||||
h, executed := newHexisTestHandler(t, ambiguous, `[]`)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("the indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("the indexer"), routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Manga indexer") {
|
||||
t.Fatalf("ambiguous should list candidates, got %q", reply)
|
||||
}
|
||||
@@ -154,7 +154,7 @@ func TestHexisResolveFlatShapeAccepted(t *testing.T) {
|
||||
caps := `[{"id":"cap_status","name":"restart","read_only":true}]`
|
||||
h, executed := newHexisTestHandler(t, flat, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !*executed {
|
||||
t.Fatalf("flat-shaped resolved entity should still execute, got reply %q", reply)
|
||||
}
|
||||
@@ -183,7 +183,7 @@ func TestHexisNexusErrorFailsClosed(t *testing.T) {
|
||||
ecosystem: stubEcosystem(nexus.URL, hexis.URL),
|
||||
}
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("nexus dependency failure must not fall through with an empty reply")
|
||||
}
|
||||
@@ -216,7 +216,7 @@ func TestHexisUnavailableFailsClosed(t *testing.T) {
|
||||
ecosystem: stubEcosystem(nexus.URL, hexis.URL),
|
||||
}
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("hexis dependency failure must not fall through with an empty reply")
|
||||
}
|
||||
@@ -234,7 +234,7 @@ func TestHexisNotFoundStillFallsThrough(t *testing.T) {
|
||||
notFound := `{"status":"not_found"}`
|
||||
h, executed := newHexisTestHandler(t, notFound, `[]`)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("turn off the lights"))
|
||||
reply := h.handleHexisAct(ctx, actDec("turn off the lights"), routeCandidate("restart"))
|
||||
if reply != "" {
|
||||
t.Fatalf("not_found resolution should fall through with empty reply, got %q", reply)
|
||||
}
|
||||
@@ -264,7 +264,7 @@ func TestHexisIrreversibleCapabilityIsNotRunFromVoice(t *testing.T) {
|
||||
caps := `[{"id":"cap_wipe","name":"restart","read_only":false,"risk":"irreversible","requires_confirmation":true}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if *executed {
|
||||
t.Fatal("an irreversible capability ran from the voice path")
|
||||
}
|
||||
@@ -284,7 +284,7 @@ func TestHexisSafeCapabilityRunsOnItsDeclaredTier(t *testing.T) {
|
||||
caps := `[{"id":"cap_status","name":"restart","read_only":true,"risk":"safe"}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if !*executed {
|
||||
t.Fatal("a capability Hexis calls safe should run")
|
||||
}
|
||||
@@ -301,7 +301,7 @@ func TestHexisUndeclaredTierStillConfirms(t *testing.T) {
|
||||
caps := `[{"id":"cap_restart","name":"restart","read_only":false}]`
|
||||
h, executed := newHexisTestHandler(t, resolved, caps)
|
||||
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
if *executed {
|
||||
t.Fatal("a mutating capability ran without a confirm")
|
||||
}
|
||||
|
||||
@@ -142,7 +142,7 @@ func TestEcosystemTrace_SuccessfulActionTracesEveryHop(t *testing.T) {
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer")); !actRan(reply) {
|
||||
if reply := h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart")); !actRan(reply) {
|
||||
t.Fatalf("setup: expected success, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -186,7 +186,7 @@ func TestEcosystemTrace_OneCorrelationIDPerPraxisAction(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
if reply := h.handlePraxisAct(ctx, praxisActDec("list_attention"), routeCandidate("list_attention")); !strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("setup: expected the digest, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -218,7 +218,7 @@ func TestEcosystemTrace_FailuresAreTracedToo(t *testing.T) {
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
nexus.SetFault(401)
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
|
||||
d := findTrace(t, h, "nexus", "resolve")
|
||||
if d == nil {
|
||||
@@ -242,7 +242,7 @@ func TestEcosystemTrace_UnreachableIsNotRefused(t *testing.T) {
|
||||
h := ecoHandler(t, nil, nil, nil)
|
||||
h.ecosystem.nexus = newNexusClient("http://127.0.0.1:1")
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick indexer"), routeCandidate("restart"))
|
||||
|
||||
d := findTrace(t, h, "nexus", "resolve")
|
||||
if d == nil {
|
||||
@@ -263,7 +263,7 @@ func TestEcosystemTrace_RedactsTheUtterance(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusNotFound())
|
||||
h := ecoHandler(t, nexus, nil, nil)
|
||||
|
||||
_ = h.handleHexisAct(ctx, actDec("перезапусти кофемашину"))
|
||||
_ = h.handleHexisAct(ctx, actDec("перезапусти кофемашину"), routeCandidate("restart"))
|
||||
|
||||
recorded := traces(t, h)
|
||||
if len(recorded) == 0 {
|
||||
@@ -295,7 +295,7 @@ func TestEcosystemTrace_AmbiguityAndConfirmationAreRecorded(t *testing.T) {
|
||||
))
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, ambig, nil, hexis)
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick"))
|
||||
_ = h.handleHexisAct(ctx, actDec("muzick"), routeCandidate("restart"))
|
||||
if d := findTrace(t, h, "nexus", "resolve"); d == nil || d.Status != traceAmbig {
|
||||
t.Fatalf("ambiguous resolve must be traced as such, got %+v", d)
|
||||
}
|
||||
@@ -303,7 +303,7 @@ func TestEcosystemTrace_AmbiguityAndConfirmationAreRecorded(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusResolved("ent_muzick", "Muzick indexer", "service"))
|
||||
mutating := fixtureHexisCapabilities(map[string]any{"id": "cap_restart", "name": "restart", "read_only": false})
|
||||
h2 := ecoHandler(t, nexus, nil, newFakeHexis(t, mutating, fixtureHexisExecuted("exec_1", "succeeded")))
|
||||
_ = h2.handleHexisAct(ctx, actDec("restart"))
|
||||
_ = h2.handleHexisAct(ctx, actDec("restart"), routeCandidate("restart"))
|
||||
d := findTrace(t, h2, "hexis", "confirmation")
|
||||
if d == nil || d.Status != tracePending {
|
||||
t.Fatalf("a parked confirmation must be traced, got %+v", d)
|
||||
|
||||
@@ -91,7 +91,7 @@ func TestNexusIsAskedForTheNameHeSaid(t *testing.T) {
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "перезагрузить музик индексер", Fn: "restart", HasFn: true},
|
||||
}
|
||||
h.handleHexisAct(ctx, dec)
|
||||
h.handleHexisAct(ctx, dec, routeCandidate("restart"))
|
||||
|
||||
reqs := nexus.Requests()
|
||||
if len(reqs) == 0 {
|
||||
@@ -147,6 +147,48 @@ func TestClarifyStillAsksWithoutHexis(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A verb is not an entity. Before the reach gate, an exact local matcher hit
|
||||
// with no arguments still sent the raw verb to Nexus and could discover a
|
||||
// similarly named entity through Hexis. The local tool lane may handle or
|
||||
// reject it, but the ecosystem must not be consulted without a target.
|
||||
func TestBareMatchedActNeverReachesNexus(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusResolved("ent_power", "Power", "service"))
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h, _, _ := newClarifyHandler(t)
|
||||
h.ecosystem = ecoHandler(t, nexus, nil, hexis).ecosystem
|
||||
|
||||
reply := h.actionAct(context.Background(), router.Decision{
|
||||
Utterance: "выключи",
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Fn: "выключи", HasFn: true, Text: "выключи"},
|
||||
})
|
||||
if len(nexus.Requests()) != 0 {
|
||||
t.Fatalf("bare verb reached Nexus: %+v", nexus.Requests())
|
||||
}
|
||||
if reply == "" {
|
||||
t.Fatal("bare act disappeared instead of staying in Maven's local lane")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnresolvedActNeverReachesNexusBeforeClarify(t *testing.T) {
|
||||
nexus := newFakeNexus(t, fixtureNexusResolved("ent_it", "It", "service"))
|
||||
hexis := newFakeHexis(t, restartCaps(), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h := ecoHandler(t, nexus, nil, hexis)
|
||||
dec := router.Decision{
|
||||
Utterance: "сделай это",
|
||||
Intent: router.IntentAct,
|
||||
Stage: 3,
|
||||
Clarify: true,
|
||||
Slots: router.Slots{Text: "сделай это"},
|
||||
}
|
||||
if reply := h.hexisBeforeClarify(context.Background(), dec); reply != "" {
|
||||
t.Fatalf("unresolved act was answered by Hexis: %q", reply)
|
||||
}
|
||||
if len(nexus.Requests()) != 0 {
|
||||
t.Fatalf("unresolved act reached Nexus: %+v", nexus.Requests())
|
||||
}
|
||||
}
|
||||
|
||||
// nexusInOrder serves one resolve answer per call, in order, so a test can say
|
||||
// what Nexus knows about the first name and what it knows about the second. The
|
||||
// last body repeats once the list runs out.
|
||||
@@ -184,7 +226,7 @@ func TestTwoResolvedNamesAsk(t *testing.T) {
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "перезагрузить нгинкс", Fn: "restart", HasFn: true},
|
||||
}
|
||||
reply := h.handleHexisAct(ctx, dec)
|
||||
reply := h.handleHexisAct(ctx, dec, routeCandidate("restart"))
|
||||
if !strings.Contains(reply, "nginx") || !strings.Contains(reply, "Muzick indexer") {
|
||||
t.Fatalf("reply = %q, want both names she found", reply)
|
||||
}
|
||||
@@ -209,7 +251,7 @@ func TestTheNameNexusKnowsWins(t *testing.T) {
|
||||
Intent: router.IntentAct,
|
||||
Slots: router.Slots{Text: "перезагрузить нгинкс", Fn: "restart", HasFn: true},
|
||||
}
|
||||
reply := h.handleHexisAct(ctx, dec)
|
||||
reply := h.handleHexisAct(ctx, dec, routeCandidate("restart"))
|
||||
if reply == "" {
|
||||
t.Fatal("the resolvable name must carry the act")
|
||||
}
|
||||
|
||||
@@ -33,7 +33,7 @@ func TestEntityAttention_ScopesPraxisByCanonicalID(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "indexer queue is backing up") {
|
||||
t.Fatalf("expected the scoped item in the reply, got %q", reply)
|
||||
}
|
||||
@@ -70,7 +70,7 @@ func TestEntityAttention_FoldsInLocalFactsForSameEntity(t *testing.T) {
|
||||
t.Fatalf("ResolveFactEntity: %v", err)
|
||||
}
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "descaled in june") {
|
||||
t.Fatalf("expected entity-scoped local facts in the reply, got %q", reply)
|
||||
}
|
||||
@@ -88,7 +88,7 @@ func TestEntityAttention_UnscopedPraxisResponseIsRefused(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("an unscoped response must not be read back as entity-scoped, got %q", reply)
|
||||
}
|
||||
@@ -112,7 +112,7 @@ func TestEntityAttention_ForeignItemsAreDropped(t *testing.T) {
|
||||
praxis := newFakePraxis(t, mustJSON(mixed))
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "indexer queue is backing up") {
|
||||
t.Fatalf("the matching item must be spoken, got %q", reply)
|
||||
}
|
||||
@@ -140,7 +140,7 @@ func TestEntityAttention_TruncationIsNamed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("the espresso machine"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "и это не всё") {
|
||||
t.Fatalf("a truncated recall must say it is truncated, got %q", reply)
|
||||
}
|
||||
@@ -156,7 +156,7 @@ func TestEntityAttention_AmbiguousAsksInsteadOfGuessing(t *testing.T) {
|
||||
praxis := newFakePraxis(t, fixturePraxisAttentionItems())
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick"), routeCandidate("entity_attention"))
|
||||
if !strings.Contains(reply, "Muzick indexer") || !strings.Contains(reply, "Muzick web") {
|
||||
t.Fatalf("ambiguous subject must ask, got %q", reply)
|
||||
}
|
||||
@@ -173,13 +173,13 @@ func TestEntityAttention_MissingAndDegradedAreDistinct(t *testing.T) {
|
||||
praxis := newFakePraxis(t, fixturePraxisAttentionItems())
|
||||
h := ecoHandler(t, nexus, praxis, nil)
|
||||
|
||||
missing := h.handlePraxisAct(ctx, entityAttentionDec("нечто"))
|
||||
missing := h.handlePraxisAct(ctx, entityAttentionDec("нечто"), routeCandidate("entity_attention"))
|
||||
if missing == "" {
|
||||
t.Fatal("an unknown entity must still get an answer")
|
||||
}
|
||||
|
||||
nexus.SetFault(503)
|
||||
degraded := h.handlePraxisAct(ctx, entityAttentionDec("нечто"))
|
||||
degraded := h.handlePraxisAct(ctx, entityAttentionDec("нечто"), routeCandidate("entity_attention"))
|
||||
if degraded == missing {
|
||||
t.Fatalf("outage and unknown-entity must not read the same: %q", degraded)
|
||||
}
|
||||
@@ -195,7 +195,7 @@ func TestEntityAttention_DelayedNexusDegradesNotHangs(t *testing.T) {
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if reply == "" {
|
||||
t.Fatal("a delayed resolve must still answer")
|
||||
}
|
||||
@@ -213,7 +213,7 @@ func TestEntityAttention_WithoutNexusSaysSo(t *testing.T) {
|
||||
))
|
||||
h := ecoHandler(t, nil, praxis, nil)
|
||||
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"))
|
||||
reply := h.handlePraxisAct(ctx, entityAttentionDec("muzick indexer"), routeCandidate("entity_attention"))
|
||||
if strings.Contains(reply, "disk almost full") {
|
||||
t.Fatalf("without nexus, items must not be passed off as entity-scoped, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -106,7 +106,12 @@ func TestSystemSafetyScenarios(t *testing.T) {
|
||||
hexis := newFakeHexis(t, fixtureHexisCapabilities(map[string]any{"id": "restart", "name": "restart", "read_only": false}), fixtureHexisExecuted("exec_1", "succeeded"))
|
||||
h, _ := newSafetyHandler(t)
|
||||
h.ecosystem = stubEcosystem(nexus.URL, hexis.URL)
|
||||
reply := h.applyAction(ctx, router.Decision{Intent: router.IntentAct, Slots: router.Slots{Fn: "restart", HasFn: true, Text: "indexer"}})
|
||||
// A matched function carries its entity target in Args. Text may be
|
||||
// model phrasing, but Args is the production matcher contract and the
|
||||
// ecosystem reach gate deliberately requires that evidence.
|
||||
reply := h.applyAction(ctx, router.Decision{Intent: router.IntentAct, Slots: router.Slots{
|
||||
Fn: "restart", HasFn: true, Args: []string{"indexer"}, Text: "indexer",
|
||||
}})
|
||||
if !strings.Contains(reply, "Indexer A") || !strings.Contains(reply, "Indexer B") {
|
||||
t.Fatalf("ambiguous entity must prompt for clarification, got %q", reply)
|
||||
}
|
||||
|
||||
@@ -2,6 +2,9 @@ package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -33,6 +36,10 @@ func newFactGateHandler(t *testing.T, now time.Time) (*reactiveHandler, ipc.Core
|
||||
func TestActionFact_QuestionIsNotWritten(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, api := newFactGateHandler(t, time.Now())
|
||||
searchH, seen := searchHandler(t,
|
||||
`{"answers":["Актуальная версия Go — 1.25."],"results":[]}`,
|
||||
http.StatusOK)
|
||||
h.search = searchH.search
|
||||
|
||||
reply := h.actionFact(ctx, router.Decision{
|
||||
Intent: router.IntentFact,
|
||||
@@ -50,11 +57,14 @@ func TestActionFact_QuestionIsNotWritten(t *testing.T) {
|
||||
if len(hits) != 0 {
|
||||
t.Fatalf("the question was indexed for recall: %+v", hits)
|
||||
}
|
||||
// It went down the query chain instead. Nothing is configured to answer a
|
||||
// world question in this harness, so "не знаю." is the honest outcome —
|
||||
// what matters is that the turn was answered, not stored.
|
||||
if reply == "" {
|
||||
t.Fatal("the turn was neither stored nor answered")
|
||||
// It went down the world query chain instead. This asserts the actual
|
||||
// destination, not merely that the write was refused: the regression was
|
||||
// the personal boundary claiming this question before search.
|
||||
if !strings.Contains(reply, "1.25") {
|
||||
t.Fatalf("reply = %q, want live world evidence", reply)
|
||||
}
|
||||
if !strings.Contains(*seen, "q="+url.QueryEscape("какая последняя версия языка Go?")) {
|
||||
t.Fatalf("search query = %q; world source was not reached verbatim", *seen)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+11
-9
@@ -72,15 +72,16 @@ func dialogueIDOf(ctx context.Context) string {
|
||||
// toDialogueSlots projects the router's slots onto the dialogue layer's copy.
|
||||
func toDialogueSlots(s router.Slots) dialogue.Slots {
|
||||
return dialogue.Slots{
|
||||
Time: s.Time,
|
||||
HasTime: s.HasTime,
|
||||
Key: s.Key,
|
||||
Value: s.Value,
|
||||
HasKey: s.HasKey,
|
||||
Text: s.Text,
|
||||
Fn: s.Fn,
|
||||
Args: s.Args,
|
||||
HasFn: s.HasFn,
|
||||
Time: s.Time,
|
||||
HasTime: s.HasTime,
|
||||
Key: s.Key,
|
||||
Value: s.Value,
|
||||
HasKey: s.HasKey,
|
||||
Text: s.Text,
|
||||
Fn: s.Fn,
|
||||
Args: s.Args,
|
||||
HasFn: s.HasFn,
|
||||
ResolvedBy: string(s.ResolvedBy),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -90,6 +91,7 @@ func applyDialogueSlots(base router.Slots, d dialogue.Slots) router.Slots {
|
||||
base.Key, base.Value, base.HasKey = d.Key, d.Value, d.HasKey
|
||||
base.Text = d.Text
|
||||
base.Fn, base.Args, base.HasFn = d.Fn, d.Args, d.HasFn
|
||||
base.ResolvedBy = router.ActionResolutionMethod(d.ResolvedBy)
|
||||
return base
|
||||
}
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
|
||||
"github.com/kami/maven/internal/dialogue"
|
||||
"github.com/kami/maven/internal/lexicon"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
)
|
||||
|
||||
@@ -67,6 +68,44 @@ func parseOrdinal(text string) (int, bool) {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// parseReminderCancelChoice is intentionally narrower than parseOrdinal. A
|
||||
// task ordinal may appear inside a sentence carrying its transition verb; the
|
||||
// reminder list was already offered specifically for cancellation, so the next
|
||||
// mutation requires the whole turn to be one affirmative position answer.
|
||||
// Questions, negation, two positions and new requests all decline and route as
|
||||
// fresh turns instead of cancelling whichever ordinal happened to appear.
|
||||
func parseReminderCancelChoice(text string) (int, bool) {
|
||||
if router.IsQuestionShaped(text) {
|
||||
return 0, false
|
||||
}
|
||||
tokens := turnTokens(text)
|
||||
nth, positions := 0, 0
|
||||
for _, tok := range tokens {
|
||||
if reminderCancelNegation(tok) {
|
||||
return 0, false
|
||||
}
|
||||
if n, ok := candidateDigits[tok]; ok {
|
||||
nth, positions = n, positions+1
|
||||
continue
|
||||
}
|
||||
if n, ok := lexicon.Ordinal(tok); ok {
|
||||
nth, positions = n, positions+1
|
||||
continue
|
||||
}
|
||||
if lexicon.IsFillerParticle(tok) || reminderCancelVerbs[tok] ||
|
||||
isReminderCancelTarget(tok) || reminderCancelFrame[tok] {
|
||||
continue
|
||||
}
|
||||
switch tok {
|
||||
case "номер", "вариант", "number", "option", "one":
|
||||
continue
|
||||
default:
|
||||
return 0, false
|
||||
}
|
||||
}
|
||||
return nth, positions == 1
|
||||
}
|
||||
|
||||
// candidateVerbs — what he wants done with the one he picked. Nothing here is
|
||||
// destructive: a task moves forward or is dropped, and both are recorded with a
|
||||
// provenance the /tasks page shows.
|
||||
@@ -112,7 +151,21 @@ func (h *reactiveHandler) resolveCandidate(ctx context.Context, text string, src
|
||||
if sess == nil || len(sess.Candidates) == 0 {
|
||||
return "", false
|
||||
}
|
||||
reminderList := true
|
||||
for _, candidate := range sess.Candidates {
|
||||
if candidate.Kind != "reminder-cancel" {
|
||||
reminderList = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if reminderList && classifyConfirm(text) == confirmNo {
|
||||
h.dialogueSessions.SetCandidates(dialogueIDOf(ctx), h.now(), nil)
|
||||
return "хорошо, ничего не отменяю.", true
|
||||
}
|
||||
nth, ok := parseOrdinal(text)
|
||||
if reminderList {
|
||||
nth, ok = parseReminderCancelChoice(text)
|
||||
}
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
@@ -125,6 +178,12 @@ func (h *reactiveHandler) resolveCandidate(ctx context.Context, text string, src
|
||||
return fmt.Sprintf("я назвала только %d.", len(sess.Candidates)), true
|
||||
}
|
||||
pick := sess.Candidates[nth-1]
|
||||
if pick.Kind == "reminder-cancel" {
|
||||
// Unlike a task list, this list was offered in answer to the explicit
|
||||
// question "which reminder should I cancel?" A bare ordinal is the
|
||||
// answer to that question and therefore completes the cancellation.
|
||||
return h.cancelReminderChoice(ctx, pick.Ref, pick.Label), true
|
||||
}
|
||||
status, say, hasVerb := parseCandidateVerb(text)
|
||||
if !hasVerb || pick.Kind != "task" {
|
||||
// Read it back and keep the list: naming one is often the first half of
|
||||
|
||||
@@ -186,22 +186,39 @@ func TestPersonalBoundaryFrozenHeadDecodes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPersonalBoundaryHashFloorLatency(t *testing.T) {
|
||||
func TestPersonalBoundaryHashFloorFitsAndScores(t *testing.T) {
|
||||
b := &personalBoundary{}
|
||||
embedder := router.NewHashEmbedder(1024)
|
||||
query, err := router.EmbedQuery(context.Background(), embedder, "когда моя встреча")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
started := time.Now()
|
||||
b.load(context.Background(), embedder)
|
||||
if _, _, ok := b.score(query); !ok {
|
||||
t.Fatal("hash-floor boundary declined to score")
|
||||
}
|
||||
elapsed := time.Since(started)
|
||||
t.Logf("hash-floor corpus fit+score: %s", elapsed)
|
||||
if elapsed > 2*time.Second {
|
||||
t.Errorf("hash-floor boundary took %s, exceeds 2s local floor ceiling", elapsed)
|
||||
if len(b.head.weights) != 1024 {
|
||||
t.Fatalf("hash-floor boundary has %d weights, want 1024", len(b.head.weights))
|
||||
}
|
||||
}
|
||||
|
||||
// BenchmarkPersonalBoundaryHashFloorFitAndScore keeps startup cost measurable
|
||||
// without making ambient CI load a correctness condition. In particular,
|
||||
// -race and coverage instrumentation both multiply the cost of this numeric
|
||||
// training loop; the functional test above is the deterministic gate.
|
||||
func BenchmarkPersonalBoundaryHashFloorFitAndScore(b *testing.B) {
|
||||
embedder := router.NewHashEmbedder(1024)
|
||||
query, err := router.EmbedQuery(context.Background(), embedder, "когда моя встреча")
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
boundary := &personalBoundary{}
|
||||
boundary.load(context.Background(), embedder)
|
||||
if _, _, ok := boundary.score(query); !ok {
|
||||
b.Fatal("hash-floor boundary declined to score")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ func TestPraxisLifecycle401NamesPraxis(t *testing.T) {
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
praxis.SetFault(401)
|
||||
reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "item_1"))
|
||||
reply := h.handlePraxisAct(ctx, praxisItemDec("resolve_item", "item_1"), routeCandidate("resolve_item"))
|
||||
if !strings.Contains(reply, servicePraxis) {
|
||||
t.Fatalf("praxis failure does not name Praxis: %q", reply)
|
||||
}
|
||||
@@ -40,10 +40,10 @@ func TestPraxisLifecycleOutageDiffersFrom401(t *testing.T) {
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
praxis.SetFault(401)
|
||||
refused := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"))
|
||||
refused := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item"))
|
||||
|
||||
h.ecosystem = &ecosystemWiring{praxis: newPraxisClient(unreachableURL)}
|
||||
outage := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"))
|
||||
outage := h.handlePraxisAct(ctx, praxisItemDec("acknowledge_item", "item_1"), routeCandidate("acknowledge_item"))
|
||||
|
||||
if refused == outage {
|
||||
t.Fatalf("a refused token and an outage still say the same thing: %q", refused)
|
||||
|
||||
@@ -17,7 +17,7 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention")); reply == "" {
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention")); reply == "" {
|
||||
t.Fatal("attention returned nothing")
|
||||
}
|
||||
|
||||
@@ -28,7 +28,7 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
|
||||
}
|
||||
for _, c := range cases {
|
||||
praxis.ResetRequests()
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", c.ref))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", c.ref), routeCandidate("acknowledge_item"))
|
||||
if !strings.Contains(reply, "принято") {
|
||||
t.Errorf("ref %q: reply %q", c.ref, reply)
|
||||
}
|
||||
@@ -42,10 +42,10 @@ func TestPositionResolvesAgainstTheLastSpokenList(t *testing.T) {
|
||||
func TestPositionPastTheEndAsksInsteadOfGuessing(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[{"id":"item_a","title":"диск заканчивается"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "4"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "4"), routeCandidate("resolve_item"))
|
||||
if !strings.Contains(reply, "какой пункт") {
|
||||
t.Errorf("a position with no item should ask, got %q", reply)
|
||||
}
|
||||
@@ -59,7 +59,7 @@ func TestPositionWithNoSpokenListAsks(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"), routeCandidate("acknowledge_item"))
|
||||
if !strings.Contains(reply, "какой пункт") {
|
||||
t.Errorf("want the ask, got %q", reply)
|
||||
}
|
||||
@@ -69,10 +69,10 @@ func TestPositionWithNoSpokenListAsks(t *testing.T) {
|
||||
func TestExplicitItemIDIsNotRewritten(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[{"id":"item_a","title":"диск"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("pin_item", "item_zz"))
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("pin_item", "item_zz"), routeCandidate("pin_item"))
|
||||
if !requestedPathContaining(praxis, "item_zz") {
|
||||
t.Errorf("the id he gave was not the one called; paths %v", paths(praxis))
|
||||
}
|
||||
@@ -85,10 +85,10 @@ func TestUnspokenItemsHoldNoPosition(t *testing.T) {
|
||||
{"id":"item_said","title":"бэкап не прошёл"}
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"))
|
||||
h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "1"), routeCandidate("acknowledge_item"))
|
||||
if !requestedPathContaining(praxis, "item_said") {
|
||||
t.Errorf("position 1 is the first item she SAID; paths %v", paths(praxis))
|
||||
}
|
||||
@@ -116,10 +116,10 @@ func requestedPathContaining(f *fakeServer, want string) bool {
|
||||
func TestDemonstrativeResolvesWhenOneItemWasSpoken(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[{"id":"item_only","title":"бэкап не прошёл"}]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "this"))
|
||||
reply := h.handlePraxisAct(context.Background(), praxisItemDec("acknowledge_item", "this"), routeCandidate("acknowledge_item"))
|
||||
if !strings.Contains(reply, "принято") {
|
||||
t.Errorf("reply %q", reply)
|
||||
}
|
||||
@@ -136,10 +136,10 @@ func TestDemonstrativeWithSeveralItemsGivesTheTurnBack(t *testing.T) {
|
||||
{"id":"item_b","title":"бэкап"}
|
||||
]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"))
|
||||
h.handlePraxisAct(context.Background(), praxisActDec("list_attention"), routeCandidate("list_attention"))
|
||||
|
||||
praxis.ResetRequests()
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this")); reply != "" {
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this"), routeCandidate("resolve_item")); reply != "" {
|
||||
t.Errorf("want a fall-through, got %q", reply)
|
||||
}
|
||||
for _, p := range paths(praxis) {
|
||||
@@ -154,7 +154,7 @@ func TestDemonstrativeWithNoDigestGivesTheTurnBack(t *testing.T) {
|
||||
praxis := newFakePraxis(t, `[]`)
|
||||
h := newPraxisTestHandler(t, praxis)
|
||||
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this")); reply != "" {
|
||||
if reply := h.handlePraxisAct(context.Background(), praxisItemDec("resolve_item", "this"), routeCandidate("resolve_item")); reply != "" {
|
||||
t.Errorf("want a fall-through, got %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -193,9 +193,9 @@ func TestQueryRecallNoteCanWin(t *testing.T) {
|
||||
t.Run("the better-matching fact answers", func(t *testing.T) {
|
||||
h, _ := buildRecallHandler(t, q, []recallCase{
|
||||
{text: "молоко стоит в холодильнике", score: 0.80, kind: "note"},
|
||||
{text: "купил молоко в среду", score: 0.95, kind: "fact"},
|
||||
{text: "молоко было в холодильнике в среду", score: 0.95, kind: "fact"},
|
||||
})
|
||||
if reply := askQuery(t, h, q); reply != "купил молоко в среду" {
|
||||
if reply := askQuery(t, h, q); reply != "молоко было в холодильнике в среду" {
|
||||
t.Errorf("reply %q, want the fact read back", reply)
|
||||
}
|
||||
})
|
||||
@@ -212,3 +212,63 @@ func TestQueryRecallNoteCanWin(t *testing.T) {
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestQueryRecallRequiresStructuralOrTopicEvidence — the whole-assistant
|
||||
// cold-start regression. The routing heads called an ordinary past-tense
|
||||
// report a query; with one note in the store the margin gate has no runner-up,
|
||||
// and cosine 0.825 was enough to speak a completely unrelated spare-key note.
|
||||
// A bare question mark does not turn the proposition into an open information
|
||||
// question, negation must not weaken the refusal, and a locative question must
|
||||
// corroborate the target it asks Maven to locate (V-719).
|
||||
func TestQueryRecallRequiresStructuralOrTopicEvidence(t *testing.T) {
|
||||
const unrelated = "запомни: запасной ключ лежит в синей коробке"
|
||||
for _, tc := range []struct {
|
||||
query string
|
||||
score float64
|
||||
}{
|
||||
{"я отменил напоминание про молоко", 0.825031306},
|
||||
{"я отменил напоминание про молоко?", 0.825031306},
|
||||
{"я не отменил напоминание про молоко", 0.825031306},
|
||||
{"я не отменил напоминание про молоко?", 0.825031306},
|
||||
{"где мой паспорт?", 0.817210},
|
||||
{"где я отменил напоминание про молоко?", 0.805800},
|
||||
{"где лежит синяя рубашка?", 0.837694},
|
||||
{"где лежит синяя папка?", 0.837472},
|
||||
{"где мой запасной паспорт?", 0.831662},
|
||||
{"где лежит запасная флешка?", 0.838980},
|
||||
{"где находится синяя коробка с документами?", 0.866553},
|
||||
{"где лежит ключ от машины?", 0.843853},
|
||||
{"где синяя коробка?", 0.90},
|
||||
} {
|
||||
t.Run(tc.query, func(t *testing.T) {
|
||||
h, phr := buildRecallHandler(t, tc.query, []recallCase{
|
||||
{text: unrelated, score: tc.score, kind: "note"},
|
||||
})
|
||||
reply := askQuery(t, h, tc.query)
|
||||
if strings.Contains(reply, "запасной ключ") {
|
||||
t.Fatalf("unrelated note escaped into reply %q", reply)
|
||||
}
|
||||
if len(phr.notes) != 0 {
|
||||
t.Fatalf("unrelated note reached the phraser: %q", phr.notes)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Voice punctuation is optional. A nominal request with no interrogative
|
||||
// still works when the candidate itself corroborates the named topic.
|
||||
const nominal = "адрес домашнего сервера"
|
||||
h, _ := buildRecallHandler(t, nominal, []recallCase{
|
||||
{text: "домашний сервер на 192.168.1.104", score: 0.90, kind: "note"},
|
||||
})
|
||||
if reply := askQuery(t, h, nominal); !strings.Contains(reply, "домашний сервер") {
|
||||
t.Fatalf("nominal recall lost its shared-topic answer: %q", reply)
|
||||
}
|
||||
|
||||
const locative = "где лежит запасной ключ?"
|
||||
h, _ = buildRecallHandler(t, locative, []recallCase{
|
||||
{text: "запасной ключ лежит в синей коробке", score: 0.90, kind: "note"},
|
||||
})
|
||||
if reply := askQuery(t, h, locative); !strings.Contains(reply, "запасной ключ") {
|
||||
t.Fatalf("locative recall lost its corroborated target: %q", reply)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
"github.com/kami/maven/internal/memory"
|
||||
"github.com/kami/maven/internal/phraser"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
"github.com/kami/maven/internal/tool"
|
||||
@@ -84,12 +85,76 @@ func TestReactiveNotesReminders(t *testing.T) {
|
||||
t.Fatal("expected at least one note, got none")
|
||||
}
|
||||
last := notes[0]
|
||||
if last.Text != "запомни что кофе закончился" {
|
||||
t.Errorf("note text = %q, want %q", last.Text, "запомни что кофе закончился")
|
||||
if last.Text != "кофе закончился" {
|
||||
t.Errorf("note text = %q, want %q", last.Text, "кофе закончился")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestRunTurnExplicitNoteStoresOnlyTheBody pins the live failure end to end:
|
||||
// a routed text turn reaches actionNote, stores only the dictated body in both
|
||||
// durable and vector memory, and cannot ask the resident model to choose the
|
||||
// acknowledgement's grammatical gender (V-721).
|
||||
func TestRunTurnExplicitNoteStoresOnlyTheBody(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
st := newTestStore(t)
|
||||
api := ipc.NewStoreAPI(st)
|
||||
now := time.Date(2026, 8, 15, 8, 0, 0, 0, time.FixedZone("+04", 4*60*60))
|
||||
emb := router.NewHashEmbedder(1024)
|
||||
mem := memory.NewInMemoryStore()
|
||||
rtr := router.New(router.Config{
|
||||
Grammars: []router.Grammar{{
|
||||
Name: "explicit-note-test",
|
||||
Decide: func(string) (router.Decision, bool) {
|
||||
return router.Decision{
|
||||
Stage: 0, Intent: router.IntentNote, Confidence: 1,
|
||||
// Deliberately hostile model slot: neither persistence nor
|
||||
// acknowledgement may use it.
|
||||
Slots: router.Slots{Text: "ты поедешь на дачу"},
|
||||
}, true
|
||||
},
|
||||
}},
|
||||
Threshold: 0.55,
|
||||
})
|
||||
model := &countingCompleter{out: `{"response":"Хорошо, сохранил.","mood":"neutral"}`}
|
||||
h := &reactiveHandler{
|
||||
api: api, router: rtr,
|
||||
recall: recallWiring{embedder: emb, memStore: mem},
|
||||
replier: newLLMReplier(model, nil),
|
||||
now: func() time.Time { return now },
|
||||
dataStore: st,
|
||||
}
|
||||
|
||||
const utterance = "запомни: запасной ключ лежит в синей коробке"
|
||||
if reply := h.runTurn(ctx, router.NormalizedInput{Text: utterance, Source: sourceText}); reply != "сохранила заметку." {
|
||||
t.Fatalf("reply = %q, want the fixed feminine acknowledgement", reply)
|
||||
}
|
||||
if model.calls != 0 {
|
||||
t.Fatalf("resident model was called %d time(s) for a note acknowledgement", model.calls)
|
||||
}
|
||||
notes, err := st.RecentNotes(ctx, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("RecentNotes: %v", err)
|
||||
}
|
||||
const body = "запасной ключ лежит в синей коробке"
|
||||
if len(notes) != 1 || notes[0].Text != body || notes[0].Source != "tap:voice" || !notes[0].Ts.Equal(now) {
|
||||
t.Fatalf("stored notes = %+v, want one exact body at the turn time", notes)
|
||||
}
|
||||
records, err := mem.ByPrefix(ctx, "note:")
|
||||
if err != nil {
|
||||
t.Fatalf("vector catalog: %v", err)
|
||||
}
|
||||
if len(records) != 1 || records[0].Meta["text"] != body {
|
||||
t.Fatalf("vector records = %+v, want the same extracted body", records)
|
||||
}
|
||||
if records[0].Meta["text"] == utterance || records[0].Meta["text"] == "ты поедешь на дачу" {
|
||||
t.Fatalf("vector metadata used a command or model rewrite: %+v", records[0].Meta)
|
||||
}
|
||||
if !phraser.IsAck(phraser.AckNote, nil, "сохранила заметку.") {
|
||||
t.Fatal("fixed acknowledgement is not registered as the note acknowledgement")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSpokenTaskCaptureFilesATask — the whole path, from the utterance to the
|
||||
// task table. It went dead when the router started claiming the marker as an
|
||||
// act: capture rides the note intent, so nothing below actionNote was ever
|
||||
|
||||
@@ -56,7 +56,7 @@ type recallWiring struct {
|
||||
// minScore — the note-recall confidence gate. Top cosine below this ⇒
|
||||
// "I don't know" instead of a guess. Tuned for the ONNX embedder; a knob,
|
||||
// not load-bearing math (same posture as the presence thresholds). Set by
|
||||
// wireVoice from VoiceConfig; default 0.55.
|
||||
// wireVoice from VoiceConfig; default 0.80.
|
||||
minScore float64
|
||||
|
||||
// minMargin — the second half of that gate: how far the top hit must beat
|
||||
|
||||
@@ -0,0 +1,383 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/dialogue"
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
"github.com/kami/maven/internal/lexicon"
|
||||
"github.com/kami/maven/internal/morph"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
)
|
||||
|
||||
// reminderCancelRequest exists to make the parser's contract explicit: a hit
|
||||
// proves only that the turn is an addressed imperative naming the reminder
|
||||
// store. Subject and time are resolved separately after that safety boundary.
|
||||
type reminderCancelRequest struct{}
|
||||
|
||||
var reminderCancelVerbs = func() map[string]bool {
|
||||
out := make(map[string]bool)
|
||||
for _, word := range lexicon.ReminderCancelVerbs() {
|
||||
out[strings.ToLower(word)] = true
|
||||
}
|
||||
return out
|
||||
}()
|
||||
|
||||
var reminderCancelFrame = func() map[string]bool {
|
||||
out := make(map[string]bool)
|
||||
for _, word := range lexicon.ReminderCancelFrame() {
|
||||
out[strings.ToLower(word)] = true
|
||||
}
|
||||
return out
|
||||
}()
|
||||
|
||||
// isReminderCancelTarget is deliberately a noun test, not a substring test.
|
||||
// A committed reminder must be named, otherwise "убери со стола" would reach
|
||||
// the reminder store. Russian cases are grammar and go through morph; the
|
||||
// English singular/plural forms are closed command vocabulary.
|
||||
func isReminderCancelTarget(tok string) bool {
|
||||
if morph.SameWord(tok, "напоминание") || morph.SameWord(tok, "будильник") {
|
||||
return true
|
||||
}
|
||||
switch tok {
|
||||
case "reminder", "reminders", "alarm", "alarms":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// reminderCancelLead reports which words may precede the imperative without
|
||||
// becoming a subject of their own. Filler/politeness vocabulary already has
|
||||
// one home in the lexicon; Maven's name is an address, not a Russian class.
|
||||
func reminderCancelLead(tok string) bool {
|
||||
return lexicon.IsFillerParticle(tok) || tok == "мавен" || tok == "maven"
|
||||
}
|
||||
|
||||
// parseReminderCancelRequest recognizes an exact cancel imperative at the
|
||||
// start of the addressed command plus an explicit reminder noun. Both are
|
||||
// whole tokens. Requiring command position is the safety boundary: infinitive
|
||||
// questions ("как отменить ..."), reported speech ("он сказал: отмени ...")
|
||||
// and past-tense remarks never reach the reminder store. A relative clause
|
||||
// after a real command remains valid even though it may contain a question
|
||||
// pronoun, so this is stronger and more precise than a punctuation test.
|
||||
func parseReminderCancelRequest(text string) (reminderCancelRequest, bool) {
|
||||
tokens := turnTokens(text)
|
||||
verbAt := -1
|
||||
for i, tok := range tokens {
|
||||
if reminderCancelVerbs[tok] {
|
||||
verbAt = i
|
||||
break
|
||||
}
|
||||
}
|
||||
if verbAt < 0 {
|
||||
return reminderCancelRequest{}, false
|
||||
}
|
||||
for _, tok := range tokens[:verbAt] {
|
||||
if !reminderCancelLead(tok) {
|
||||
return reminderCancelRequest{}, false
|
||||
}
|
||||
}
|
||||
for _, tok := range tokens[verbAt+1:] {
|
||||
if isReminderCancelTarget(tok) {
|
||||
return reminderCancelRequest{}, true
|
||||
}
|
||||
}
|
||||
return reminderCancelRequest{}, false
|
||||
}
|
||||
|
||||
func reminderCancelNegation(tok string) bool {
|
||||
switch tok {
|
||||
case "не", "ни", "not", "no", "don't", "dont":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func reminderCancelTimeLead(tok string) bool {
|
||||
switch tok {
|
||||
case "в", "во", "на", "к", "ко", "через", "спустя",
|
||||
"at", "in", "by", "until", "after", "before":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func reminderCancelTimeUnit(tok string) bool {
|
||||
if lexicon.IsHourUnit(tok) || lexicon.IsMinuteUnit(tok) {
|
||||
return true
|
||||
}
|
||||
for _, part := range lexicon.PartsOfDay() {
|
||||
if tok == part {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return tok == "утра" || tok == "дня" || tok == "вечера" || tok == "ночи" ||
|
||||
tok == "am" || tok == "pm" || tok == "noon" || tok == "midnight"
|
||||
}
|
||||
|
||||
func reminderCancelNumeral(tok string) (int, bool) {
|
||||
if n, ok := lexicon.Cardinal(tok); ok {
|
||||
return n, true
|
||||
}
|
||||
if n, ok := lexicon.Ordinal(tok); ok && n > 0 {
|
||||
return n, true
|
||||
}
|
||||
n, err := strconv.Atoi(tok)
|
||||
return n, err == nil
|
||||
}
|
||||
|
||||
// reminderClockTokenBudget records the numeric pieces that came from a written
|
||||
// clock. turnTokens deliberately splits 21:30 into 21 and 30, so a small
|
||||
// multiset lets subject extraction ignore exactly those occurrences without
|
||||
// discarding the same number when it also belongs to the reminder text.
|
||||
func reminderClockTokenBudget(text string) map[string]int {
|
||||
out := make(map[string]int)
|
||||
for _, field := range strings.Fields(strings.ToLower(text)) {
|
||||
field = strings.Trim(field, ".,!?;()[]{}«»\"'")
|
||||
hour, minute, ok := strings.Cut(field, ":")
|
||||
if !ok || len(minute) != 2 {
|
||||
continue
|
||||
}
|
||||
h, herr := strconv.Atoi(hour)
|
||||
m, merr := strconv.Atoi(minute)
|
||||
if herr != nil || merr != nil || h < 0 || h > 23 || m < 0 || m > 59 {
|
||||
continue
|
||||
}
|
||||
out[hour]++
|
||||
out[minute]++
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// reminderCancellationTerms keeps identity-bearing words, including negation
|
||||
// and quantities. The old ownContent shortcut erased both, so "не звонить" and
|
||||
// "звонить", or "одну таблетку" and "две таблетки", could select the same
|
||||
// row. Time framing is removed only after the shared parser proved that this
|
||||
// turn actually carries a readable time; numerals are removed only in a clock
|
||||
// position, never merely because they are numbers.
|
||||
func reminderCancellationTerms(text string, hasTime bool) []string {
|
||||
tokens := turnTokens(text)
|
||||
clockBudget := reminderClockTokenBudget(text)
|
||||
out := make([]string, 0, len(tokens))
|
||||
for i, tok := range tokens {
|
||||
if reminderCancelVerbs[tok] || isReminderCancelTarget(tok) ||
|
||||
reminderCancelFrame[tok] || lexicon.IsFillerParticle(tok) {
|
||||
continue
|
||||
}
|
||||
if !hasTime || reminderCancelNegation(tok) {
|
||||
out = append(out, tok)
|
||||
continue
|
||||
}
|
||||
if clockBudget[tok] > 0 {
|
||||
clockBudget[tok]--
|
||||
continue
|
||||
}
|
||||
if _, numeric := reminderCancelNumeral(tok); numeric {
|
||||
prevTime := i > 0 && reminderCancelTimeLead(tokens[i-1])
|
||||
nextTime := i+1 < len(tokens) && reminderCancelTimeUnit(tokens[i+1])
|
||||
if prevTime || nextTime {
|
||||
continue
|
||||
}
|
||||
}
|
||||
// frameWords is assembled exclusively from the closed time/grammar
|
||||
// lexicons. At this point a time was parsed, and negation has already
|
||||
// been preserved above, so these words identify the time rather than
|
||||
// the stored reminder body.
|
||||
if frameWords[tok] {
|
||||
continue
|
||||
}
|
||||
out = append(out, tok)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// reminderCancellationTime applies the same parse and resolved-hour gate as a
|
||||
// newly created reminder. A time expression that is present but unread is not
|
||||
// silently discarded: the caller asks for a clearer time instead of cancelling
|
||||
// whichever row happens to match the remaining words.
|
||||
func (h *reactiveHandler) reminderCancellationTime(ctx context.Context, text string) (time.Time, bool) {
|
||||
if slots := h.extractor.Extract(ctx, router.IntentReminder, text, h.now()); slots.HasTime {
|
||||
return slots.Time, true
|
||||
}
|
||||
if h.timeParser == nil {
|
||||
return time.Time{}, false
|
||||
}
|
||||
parsed, ok, err := h.timeParser.Parse(ctx, text, h.now())
|
||||
if err != nil || !ok || !router.ResolvedTheHour(text, parsed) {
|
||||
return time.Time{}, false
|
||||
}
|
||||
return parsed, true
|
||||
}
|
||||
|
||||
func reminderNextFire(r ipc.Reminder) time.Time {
|
||||
if !r.NextFireTs.IsZero() {
|
||||
return r.NextFireTs
|
||||
}
|
||||
return r.FireTs
|
||||
}
|
||||
|
||||
// reminderTimeMatches lets state disambiguate a clock when the day was not
|
||||
// named. "На девять" can therefore select the sole 09:00/21:00 reminder, but
|
||||
// if both exist they both remain candidates and Maven asks. A named day or an
|
||||
// interval denotes an absolute minute and must match that minute exactly.
|
||||
func reminderTimeMatches(text string, parsed, fire time.Time) bool {
|
||||
local := fire.In(parsed.Location())
|
||||
if router.NamesADay(text) || router.NamesAnInterval(text) {
|
||||
return local.Truncate(time.Minute).Equal(parsed.Truncate(time.Minute))
|
||||
}
|
||||
if router.HourIsAmbiguous(text) {
|
||||
return local.Minute() == parsed.Minute() && local.Hour()%12 == parsed.Hour()%12
|
||||
}
|
||||
return local.Hour() == parsed.Hour() && local.Minute() == parsed.Minute()
|
||||
}
|
||||
|
||||
func reminderTextMatchesTerms(r ipc.Reminder, terms []string) bool {
|
||||
if len(terms) == 0 {
|
||||
return true
|
||||
}
|
||||
words := turnTokens(store.ReminderText(r.Payload))
|
||||
used := make([]bool, len(words))
|
||||
for _, term := range terms {
|
||||
found := false
|
||||
for i, word := range words {
|
||||
if used[i] {
|
||||
continue
|
||||
}
|
||||
tn, tok := reminderCancelNumeral(term)
|
||||
wn, wok := reminderCancelNumeral(word)
|
||||
if term == word || morph.SameWord(term, word) || (tok && wok && tn == wn) {
|
||||
used[i] = true
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func reminderCancellationLabel(r ipc.Reminder, now time.Time) string {
|
||||
fire := reminderNextFire(r).In(now.Location())
|
||||
when := dayPrefix(now, fire)
|
||||
if when == "это" {
|
||||
when = fmt.Sprintf("%d %s", fire.Day(), lexicon.MonthGenitive(int(fire.Month())))
|
||||
}
|
||||
return fmt.Sprintf("%s в %s — %s", when, fire.Format("15:04"), store.ReminderText(r.Payload))
|
||||
}
|
||||
|
||||
// offerReminderCancellations binds exactly the rows Maven names, in that order.
|
||||
// An ordinal on the next turn therefore points at the spoken list, never at a
|
||||
// fresh query whose order may have changed in between.
|
||||
func (h *reactiveHandler) offerReminderCancellations(ctx context.Context, text string, matches []ipc.Reminder) string {
|
||||
const maxSpoken = 5
|
||||
truncated := len(matches) > maxSpoken
|
||||
if len(matches) > maxSpoken {
|
||||
matches = matches[:maxSpoken]
|
||||
}
|
||||
candidates := make([]dialogue.Candidate, 0, len(matches))
|
||||
parts := make([]string, 0, len(matches))
|
||||
for i, r := range matches {
|
||||
label := reminderCancellationLabel(r, h.now())
|
||||
candidates = append(candidates, dialogue.Candidate{Kind: "reminder-cancel", Ref: r.ID, Label: label})
|
||||
parts = append(parts, fmt.Sprintf("%d: %s", i+1, label))
|
||||
}
|
||||
|
||||
if h.dialogueSessions == nil {
|
||||
return "нашла несколько подходящих напоминаний — уточни текст или время."
|
||||
}
|
||||
id, now := dialogueIDOf(ctx), h.now()
|
||||
// This command is its own turn. Reusing an older session would keep stale
|
||||
// intent/slots alive after the choice and let the next utterance inherit
|
||||
// unrelated state, so the offered list gets a fresh system session.
|
||||
h.dialogueSessions.Put(id, &dialogue.Session{
|
||||
Intent: dialogue.IntentSystem, Utterance: text, Timestamp: now,
|
||||
Candidates: candidates,
|
||||
})
|
||||
prefix := "нашла несколько подходящих. какое отменить? "
|
||||
if truncated {
|
||||
prefix = "нашла больше пяти подходящих; называю первые пять. если нужного здесь нет, уточни текст или время. какое отменить? "
|
||||
}
|
||||
return prefix + strings.Join(parts, "; ") + ". ответь одним порядковым словом, например «второе»."
|
||||
}
|
||||
|
||||
func (h *reactiveHandler) clearReminderCandidates(ctx context.Context) {
|
||||
if h.dialogueSessions != nil {
|
||||
h.dialogueSessions.SetCandidates(dialogueIDOf(ctx), h.now(), nil)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *reactiveHandler) cancelReminderChoice(ctx context.Context, id int64, label string) string {
|
||||
if err := h.api.CancelReminder(ctx, id); err != nil {
|
||||
switch {
|
||||
case errors.Is(err, ipc.ErrReminderNotFound), errors.Is(err, ipc.ErrReminderState):
|
||||
h.clearReminderCandidates(ctx)
|
||||
return "это напоминание уже не ожидает отправки."
|
||||
case errors.Is(err, ipc.ErrReminderInFlight):
|
||||
h.clearReminderCandidates(ctx)
|
||||
return "я уже начала отправлять это напоминание — надёжно отменить его уже нельзя."
|
||||
default:
|
||||
log.Printf("voice: cancel reminder %d: %v", id, err)
|
||||
return "не получилось отменить напоминание."
|
||||
}
|
||||
}
|
||||
h.clearReminderCandidates(ctx)
|
||||
log.Printf("voice: cancelled reminder %d (%q)", id, label)
|
||||
return "отменила напоминание: " + label + "."
|
||||
}
|
||||
|
||||
// resolveReminderCancellation is the stateful pre-route resolver for a
|
||||
// committed reminder. It claims only the explicit structural command above,
|
||||
// resolves against every pending row, and never ranks an ambiguous set down to
|
||||
// one. One match cancels; more than one is an offered, ordinal-bound question.
|
||||
func (h *reactiveHandler) resolveReminderCancellation(ctx context.Context, text string) (string, bool) {
|
||||
_, ok := parseReminderCancelRequest(text)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
rows, err := h.api.ListPendingReminders(ctx, 0)
|
||||
if err != nil {
|
||||
log.Printf("voice: list reminders for cancellation: %v", err)
|
||||
return "не получилось посмотреть напоминания.", true
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return "ожидающих напоминаний нет.", true
|
||||
}
|
||||
|
||||
parsed, hasTime := h.reminderCancellationTime(ctx, text)
|
||||
if router.MentionsTime(text) && !hasTime {
|
||||
return "не смогла разобрать время напоминания — уточни его.", true
|
||||
}
|
||||
terms := reminderCancellationTerms(text, hasTime)
|
||||
matches := make([]ipc.Reminder, 0, len(rows))
|
||||
for _, r := range rows {
|
||||
if !reminderTextMatchesTerms(r, terms) {
|
||||
continue
|
||||
}
|
||||
if hasTime && !reminderTimeMatches(text, parsed, reminderNextFire(r)) {
|
||||
continue
|
||||
}
|
||||
matches = append(matches, r)
|
||||
}
|
||||
|
||||
switch len(matches) {
|
||||
case 0:
|
||||
return "не нашла такого ожидающего напоминания.", true
|
||||
case 1:
|
||||
label := reminderCancellationLabel(matches[0], h.now())
|
||||
return h.cancelReminderChoice(ctx, matches[0].ID, label), true
|
||||
default:
|
||||
return h.offerReminderCancellations(ctx, text, matches), true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,425 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/decision"
|
||||
"github.com/kami/maven/internal/dialogue"
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
"github.com/kami/maven/internal/tts"
|
||||
)
|
||||
|
||||
func TestParseReminderCancelRequest(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
text string
|
||||
ok bool
|
||||
}{
|
||||
{"отмени напоминание про врача", true},
|
||||
{"убери моё напоминание о визите", true},
|
||||
{"удали будильник на девять", true},
|
||||
{"пожалуйста, Maven, cancel the reminder about doctor", true},
|
||||
{"отмени напоминание, которое стоит на завтра", true},
|
||||
{"напоминание про врача", false},
|
||||
{"отмени задачу про врача", false},
|
||||
{"я отменил напоминание про врача", false},
|
||||
{"как отменить напоминание про врача?", false},
|
||||
{"можно отменить напоминание про врача?", false},
|
||||
{"он сказал: отмени напоминание про врача", false},
|
||||
{"how to cancel the reminder about doctor?", false},
|
||||
{"can you cancel the reminder about doctor?", false},
|
||||
{"убери со стола", false},
|
||||
{"отмена", false},
|
||||
} {
|
||||
_, ok := parseReminderCancelRequest(tc.text)
|
||||
if ok != tc.ok {
|
||||
t.Errorf("parseReminderCancelRequest(%q) ok = %v, want %v", tc.text, ok, tc.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationTermsPreserveIdentity(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
text string
|
||||
hasTime bool
|
||||
want []string
|
||||
}{
|
||||
{"отмени напоминание про врача", false, []string{"врача"}},
|
||||
{"отмени напоминание не звонить врачу", false, []string{"не", "звонить", "врачу"}},
|
||||
{"отмени напоминание принять две таблетки", false, []string{"принять", "две", "таблетки"}},
|
||||
{"отмени напоминание принять две таблетки на девять", true, []string{"принять", "две", "таблетки"}},
|
||||
{"cancel the reminder to take 2 pills at 21:30", true, []string{"take", "2", "pills"}},
|
||||
} {
|
||||
got := reminderCancellationTerms(tc.text, tc.hasTime)
|
||||
if strings.Join(got, "|") != strings.Join(tc.want, "|") {
|
||||
t.Errorf("reminderCancellationTerms(%q) = %v, want %v", tc.text, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func seedVoiceReminder(t *testing.T, st *store.Store, fire time.Time, text string) int64 {
|
||||
t.Helper()
|
||||
id, err := st.CreateReminder(context.Background(), fire, `{"text":"`+text+`"}`, "")
|
||||
if err != nil {
|
||||
t.Fatalf("create reminder: %v", err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func reminderStatuses(t *testing.T, st *store.Store) map[int64]string {
|
||||
t.Helper()
|
||||
rows, err := st.ListReminders(context.Background(), 100)
|
||||
if err != nil {
|
||||
t.Fatalf("list reminders: %v", err)
|
||||
}
|
||||
out := make(map[int64]string, len(rows))
|
||||
for _, row := range rows {
|
||||
out[row.ID] = row.Status
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestReminderCancellationResolvesSubjectByMorphology(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
h.timeParser = router.StubDateTimeParser{}
|
||||
doctor := seedVoiceReminder(t, st, now.Add(3*time.Hour), "позвонить врачу")
|
||||
bread := seedVoiceReminder(t, st, now.Add(4*time.Hour), "купить хлеб")
|
||||
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание про врача")
|
||||
if !handled || !strings.Contains(reply, "отменила") || !strings.Contains(reply, "позвонить врачу") {
|
||||
t.Fatalf("reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[doctor] != store.ReminderCancelled || statuses[bread] != store.ReminderPending {
|
||||
t.Fatalf("statuses = %+v, want doctor cancelled and bread pending", statuses)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationKeepsNegationAndQuantityDistinct(t *testing.T) {
|
||||
t.Run("negation", func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
positive := seedVoiceReminder(t, st, now.Add(time.Hour), "звонить врачу")
|
||||
negative := seedVoiceReminder(t, st, now.Add(2*time.Hour), "не звонить врачу")
|
||||
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание не звонить врачу")
|
||||
if !handled || !strings.Contains(reply, "не звонить врачу") {
|
||||
t.Fatalf("reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[positive] != store.ReminderPending || statuses[negative] != store.ReminderCancelled {
|
||||
t.Fatalf("negation selected the wrong row: %+v", statuses)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("quantity", func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
one := seedVoiceReminder(t, st, now.Add(time.Hour), "принять одну таблетку")
|
||||
two := seedVoiceReminder(t, st, now.Add(2*time.Hour), "принять две таблетки")
|
||||
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "удали напоминание принять две таблетки")
|
||||
if !handled || !strings.Contains(reply, "две таблетки") {
|
||||
t.Fatalf("reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[one] != store.ReminderPending || statuses[two] != store.ReminderCancelled {
|
||||
t.Fatalf("quantity selected the wrong row: %+v", statuses)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestReminderCancellationQuestionNeverMutates(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
id := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
|
||||
for _, text := range []string{
|
||||
"как отменить напоминание про врача?",
|
||||
"можно отменить напоминание про врача?",
|
||||
"он сказал: отмени напоминание про врача",
|
||||
} {
|
||||
if reply, handled := h.resolveReminderCancellation(context.Background(), text); handled || reply != "" {
|
||||
t.Fatalf("non-command %q was claimed: reply=%q handled=%v", text, reply, handled)
|
||||
}
|
||||
if got := reminderStatuses(t, st)[id]; got != store.ReminderPending {
|
||||
t.Fatalf("non-command %q changed reminder to %q", text, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationUsesClockAndAsksWhenStateIsAmbiguous(t *testing.T) {
|
||||
t.Run("one matching half of day is enough", func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
h.timeParser = router.StubDateTimeParser{}
|
||||
evening := seedVoiceReminder(t, st, time.Date(now.Year(), now.Month(), now.Day(), 21, 0, 0, 0, now.Location()), "вечернее лекарство")
|
||||
seedVoiceReminder(t, st, now.Add(2*time.Hour), "купить хлеб")
|
||||
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "убери напоминание на девять")
|
||||
if !handled || !strings.Contains(reply, "отменила") {
|
||||
t.Fatalf("reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
if got := reminderStatuses(t, st)[evening]; got != store.ReminderCancelled {
|
||||
t.Fatalf("21:00 status = %q, want cancelled", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("two matching halves are offered and ordinal is bound", func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
h.timeParser = router.StubDateTimeParser{}
|
||||
evening := seedVoiceReminder(t, st, time.Date(now.Year(), now.Month(), now.Day(), 21, 0, 0, 0, now.Location()), "вечернее лекарство")
|
||||
morning := seedVoiceReminder(t, st, time.Date(now.Year(), now.Month(), now.Day()+1, 9, 0, 0, 0, now.Location()), "утреннее лекарство")
|
||||
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "убери напоминание на девять")
|
||||
if !handled || !strings.Contains(reply, "порядковым словом") {
|
||||
t.Fatalf("ambiguous reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[evening] != store.ReminderPending || statuses[morning] != store.ReminderPending {
|
||||
t.Fatalf("ambiguous command mutated rows: %+v", statuses)
|
||||
}
|
||||
sess := h.dialogueSessions.Get(dialogueIDOf(context.Background()), h.now())
|
||||
if sess == nil || len(sess.Candidates) != 2 || sess.Candidates[1].Ref != morning {
|
||||
t.Fatalf("bound candidates = %+v", sess)
|
||||
}
|
||||
|
||||
reply, handled = h.resolveCandidate(context.Background(), "второе", sourceVoice)
|
||||
if !handled || !strings.Contains(reply, "утреннее лекарство") {
|
||||
t.Fatalf("ordinal reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
statuses = reminderStatuses(t, st)
|
||||
if statuses[evening] != store.ReminderPending || statuses[morning] != store.ReminderCancelled {
|
||||
t.Fatalf("ordinal cancelled the wrong row: %+v", statuses)
|
||||
}
|
||||
if sess := h.dialogueSessions.Get(dialogueIDOf(context.Background()), h.now()); sess == nil || len(sess.Candidates) != 0 {
|
||||
t.Fatalf("spent candidates survived: %+v", sess)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestReminderCancellationChoiceRequiresAWholeAffirmativeOrdinal(t *testing.T) {
|
||||
unsafe := []string{
|
||||
"почему второе?",
|
||||
"не второе",
|
||||
"второе не отменяй",
|
||||
"первое и второе",
|
||||
"напомни мне первого сентября оплатить счёт",
|
||||
}
|
||||
for _, answer := range unsafe {
|
||||
t.Run(answer, func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
h.timeParser = router.StubDateTimeParser{}
|
||||
first := seedVoiceReminder(t, st, now.Add(time.Hour), "первое лекарство")
|
||||
second := seedVoiceReminder(t, st, now.Add(2*time.Hour), "второе лекарство")
|
||||
if _, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание"); !handled {
|
||||
t.Fatal("ambiguous cancellation was not offered")
|
||||
}
|
||||
if reply, handled := h.resolveCandidate(context.Background(), answer, sourceVoice); handled || reply != "" {
|
||||
t.Fatalf("unsafe answer was claimed: reply=%q handled=%v", reply, handled)
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[first] != store.ReminderPending || statuses[second] != store.ReminderPending {
|
||||
t.Fatalf("unsafe answer mutated rows: %+v", statuses)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationChoiceCanBeAbandoned(t *testing.T) {
|
||||
for _, answer := range []string{"отмена", "не надо", "no"} {
|
||||
t.Run(answer, func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
first := seedVoiceReminder(t, st, now.Add(time.Hour), "первое")
|
||||
second := seedVoiceReminder(t, st, now.Add(2*time.Hour), "второе")
|
||||
if _, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание"); !handled {
|
||||
t.Fatal("ambiguous cancellation was not offered")
|
||||
}
|
||||
reply, handled := h.resolveCandidate(context.Background(), answer, sourceVoice)
|
||||
if !handled || !strings.Contains(reply, "ничего не отменяю") {
|
||||
t.Fatalf("cancel answer = %q handled=%v", reply, handled)
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[first] != store.ReminderPending || statuses[second] != store.ReminderPending {
|
||||
t.Fatalf("abandoning the choice mutated rows: %+v", statuses)
|
||||
}
|
||||
if sess := h.dialogueSessions.Get(dialogueIDOf(context.Background()), h.now()); sess == nil || len(sess.Candidates) != 0 {
|
||||
t.Fatalf("abandoned candidates survived: %+v", sess)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationOfferStartsFreshAndNamesTruncation(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
id := dialogueIDOf(context.Background())
|
||||
h.dialogueSessions.Put(id, &dialogue.Session{
|
||||
Intent: dialogue.IntentReminder,
|
||||
Slots: dialogue.Slots{Text: "stale subject", HasTime: true, Time: now.Add(time.Hour)},
|
||||
Timestamp: now.Add(-time.Minute),
|
||||
})
|
||||
for i := 0; i < 6; i++ {
|
||||
seedVoiceReminder(t, st, now.Add(time.Duration(i+1)*time.Hour), fmt.Sprintf("row %d", i+1))
|
||||
}
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание")
|
||||
if !handled || !strings.Contains(reply, "первые пять") || !strings.Contains(reply, "уточни текст или время") {
|
||||
t.Fatalf("truncated offer = %q handled=%v", reply, handled)
|
||||
}
|
||||
sess := h.dialogueSessions.Get(id, h.now())
|
||||
if sess == nil || sess.Intent != dialogue.IntentSystem || sess.Slots.Text != "" ||
|
||||
len(sess.Candidates) != 5 || sess.Utterance != "отмени напоминание" {
|
||||
t.Fatalf("offer reused stale dialogue state: %+v", sess)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationNeverGuesses(t *testing.T) {
|
||||
t.Run("bare command over several rows", func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
first := seedVoiceReminder(t, st, now.Add(time.Hour), "первое")
|
||||
second := seedVoiceReminder(t, st, now.Add(2*time.Hour), "второе")
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание")
|
||||
if !handled || !strings.Contains(reply, "порядковым словом") {
|
||||
t.Fatalf("reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[first] != store.ReminderPending || statuses[second] != store.ReminderPending {
|
||||
t.Fatalf("bare ambiguous command mutated rows: %+v", statuses)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("unread time", func(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
h.timeParser = router.StubDateTimeParser{}
|
||||
id := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
|
||||
reply, handled := h.resolveReminderCancellation(context.Background(), "отмени напоминание через вечность")
|
||||
if !handled || !strings.Contains(reply, "не смогла разобрать время") {
|
||||
t.Fatalf("reply = %q, handled=%v", reply, handled)
|
||||
}
|
||||
if got := reminderStatuses(t, st)[id]; got != store.ReminderPending {
|
||||
t.Fatalf("unread time cancelled reminder: %q", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
type cancelReminderAPI struct {
|
||||
ipc.UnimplementedCoreAPI
|
||||
|
||||
rows []ipc.Reminder
|
||||
listErr error
|
||||
cancelErr error
|
||||
calls []int64
|
||||
}
|
||||
|
||||
func (a *cancelReminderAPI) ListPendingReminders(context.Context, int) ([]ipc.Reminder, error) {
|
||||
return a.rows, a.listErr
|
||||
}
|
||||
|
||||
func (a *cancelReminderAPI) CancelReminder(_ context.Context, id int64) error {
|
||||
a.calls = append(a.calls, id)
|
||||
return a.cancelErr
|
||||
}
|
||||
|
||||
func cancelHandler(api ipc.CoreAPI) *reactiveHandler {
|
||||
now := time.Date(2026, 8, 15, 9, 0, 0, 0, time.UTC)
|
||||
parser := router.StubDateTimeParser{}
|
||||
return &reactiveHandler{
|
||||
api: api, now: func() time.Time { return now }, timeParser: parser,
|
||||
extractor: router.Extractor{Time: parser},
|
||||
dialogueSessions: dialogue.NewSessionStore(2 * time.Minute),
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationReportsStoreOutcomes(t *testing.T) {
|
||||
row := ipc.Reminder{
|
||||
ID: 7, FireTs: time.Date(2026, 8, 15, 12, 0, 0, 0, time.UTC),
|
||||
NextFireTs: time.Date(2026, 8, 15, 12, 0, 0, 0, time.UTC),
|
||||
Payload: `{"text":"позвонить врачу"}`, Status: store.ReminderPending,
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
err error
|
||||
want string
|
||||
}{
|
||||
{"already terminal", ipc.ErrReminderState, "уже не ожидает"},
|
||||
{"delivery in flight", ipc.ErrReminderInFlight, "уже начала отправлять"},
|
||||
{"transport", errors.New("socket closed"), "не получилось отменить"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
api := &cancelReminderAPI{rows: []ipc.Reminder{row}, cancelErr: tc.err}
|
||||
reply, handled := cancelHandler(api).resolveReminderCancellation(context.Background(), "отмени напоминание про врача")
|
||||
if !handled || !strings.Contains(reply, tc.want) || len(api.calls) != 1 || api.calls[0] != 7 {
|
||||
t.Fatalf("reply=%q handled=%v calls=%v", reply, handled, api.calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("list failure", func(t *testing.T) {
|
||||
api := &cancelReminderAPI{listErr: errors.New("offline")}
|
||||
reply, handled := cancelHandler(api).resolveReminderCancellation(context.Background(), "отмени напоминание")
|
||||
if !handled || !strings.Contains(reply, "не получилось посмотреть") || len(api.calls) != 0 {
|
||||
t.Fatalf("reply=%q handled=%v calls=%v", reply, handled, api.calls)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("nothing pending", func(t *testing.T) {
|
||||
api := &cancelReminderAPI{}
|
||||
reply, handled := cancelHandler(api).resolveReminderCancellation(context.Background(), "отмени напоминание")
|
||||
if !handled || !strings.Contains(reply, "ожидающих напоминаний нет") || len(api.calls) != 0 {
|
||||
t.Fatalf("reply=%q handled=%v calls=%v", reply, handled, api.calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestReminderCancellationIsAPreRouteTurnAndDoesNotGetSwallowedByClarify(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
h.timeParser = router.StubDateTimeParser{}
|
||||
h.decisions = decision.NewRing()
|
||||
id := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
|
||||
ctx := withDialogueID(context.Background(), dialogueIDFor(sourceText, "web"))
|
||||
h.clarifyStore.Put(dialogueIDOf(ctx), &dialogue.PendingQuestion{
|
||||
Intent: dialogue.IntentReminder, Missing: []dialogue.Slot{dialogue.SlotTime},
|
||||
Utterance: "напомни позвонить маме", Asked: h.now(), TTL: clarifyTTL,
|
||||
})
|
||||
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "отмени напоминание про врача", Source: sourceText})
|
||||
if !strings.Contains(reply, clarifyDropped) || !strings.Contains(reply, "отменила напоминание") {
|
||||
t.Fatalf("reply = %q, want dropped clarify notice and cancellation", reply)
|
||||
}
|
||||
if h.clarifyStore.Get(dialogueIDOf(ctx), h.now()) != nil {
|
||||
t.Fatal("the superseded clarify question survived the cancellation request")
|
||||
}
|
||||
if got := reminderStatuses(t, st)[id]; got != store.ReminderCancelled {
|
||||
t.Fatalf("status = %q, want cancelled", got)
|
||||
}
|
||||
recs := h.decisions.Recent(1)
|
||||
if len(recs) != 1 {
|
||||
t.Fatalf("decision records = %d, want 1", len(recs))
|
||||
}
|
||||
claim := findClaim(recs[0], "reminder-cancel")
|
||||
if claim == nil || claim.Outcome != decision.Won {
|
||||
t.Fatalf("reminder-cancel claim = %+v, want pre-route winner", claim)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderCancellationThroughPushToTalk(t *testing.T) {
|
||||
h, st, now := newClarifyHandler(t)
|
||||
h.stt = simTranscriber{text: "отмени напоминание про врача"}
|
||||
h.tts = tts.NewStub()
|
||||
h.timeParser = router.StubDateTimeParser{}
|
||||
h.router = buildRouter(router.NewHashEmbedder(64), h.matcher, 0.55, nil, nil)
|
||||
doctor := seedVoiceReminder(t, st, now.Add(time.Hour), "позвонить врачу")
|
||||
bread := seedVoiceReminder(t, st, now.Add(2*time.Hour), "купить хлеб")
|
||||
|
||||
resp, err := h.HandlePushToTalk(context.Background(), voicePTT(), 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(resp.ReplyText, "отменила напоминание") || len(resp.ReplyAudio.Bytes) == 0 {
|
||||
t.Fatalf("PTT response = text %q audio=%d bytes", resp.ReplyText, len(resp.ReplyAudio.Bytes))
|
||||
}
|
||||
statuses := reminderStatuses(t, st)
|
||||
if statuses[doctor] != store.ReminderCancelled || statuses[bread] != store.ReminderPending {
|
||||
t.Fatalf("PTT cancellation changed the wrong rows: %+v", statuses)
|
||||
}
|
||||
}
|
||||
@@ -149,7 +149,7 @@ func TestRepairResumesQuestionParkedAfterTheCorrectedTurn(t *testing.T) {
|
||||
t.Fatal("expected a parked reminder question")
|
||||
}
|
||||
|
||||
reply := h.runTurn(ctx, "нет, это был вопрос", sourceText)
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "нет, это был вопрос", Source: sourceText})
|
||||
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
|
||||
if !strings.HasSuffix(reply, resumed) {
|
||||
t.Fatalf("the correction hid the still-live question: reply=%q want suffix=%q", reply, resumed)
|
||||
@@ -176,14 +176,14 @@ func TestRepairedClarifyCompletesWithoutDroppingTheOlderQuestion(t *testing.T) {
|
||||
t.Fatal("expected the older reminder question")
|
||||
}
|
||||
|
||||
if reply := h.runTurn(ctx, "нет, это было напоминание", sourceText); !strings.Contains(reply, "Когда") {
|
||||
if reply := h.runTurn(ctx, router.NormalizedInput{Text: "нет, это было напоминание", Source: sourceText}); !strings.Contains(reply, "Когда") {
|
||||
t.Fatalf("the repaired reminder did not ask for its missing time: %q", reply)
|
||||
}
|
||||
if depth := h.clarifyStore.Depth(voiceDialogueID); depth != 2 {
|
||||
t.Fatalf("the repaired question overwrote the older one: depth=%d want=2", depth)
|
||||
}
|
||||
|
||||
reply := h.runTurn(ctx, "сегодня в 15:00", sourceText)
|
||||
reply := h.runTurn(ctx, router.NormalizedInput{Text: "сегодня в 15:00", Source: sourceText})
|
||||
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
|
||||
if !strings.HasSuffix(reply, resumed) {
|
||||
t.Fatalf("completing the repaired request did not resume the older one: reply=%q", reply)
|
||||
|
||||
@@ -39,6 +39,13 @@ func (r *llmReplier) Reply(ctx context.Context, d router.Decision) string {
|
||||
// что ты выпел стакан воды" for "я выпил воды".
|
||||
return phraser.FactAck(d.Utterance)
|
||||
}
|
||||
if d.Intent == router.IntentNote {
|
||||
// A successful durable write needs no generation. The resident model
|
||||
// answered one live capture with masculine self-reference ("сохранил")
|
||||
// despite the prompt; the hand-written line is both faster and a hard
|
||||
// persona guarantee on the daemon's reply path (V-721).
|
||||
return phraser.Ack(phraser.AckNote, nil)
|
||||
}
|
||||
out, err := r.p.PhraseReply(ctx, d)
|
||||
if err != nil || out == "" {
|
||||
return r.stub.Reply(ctx, d)
|
||||
|
||||
@@ -20,22 +20,46 @@ type stubCompleter struct {
|
||||
|
||||
func (s stubCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return s.out, s.err }
|
||||
|
||||
func TestLLMReplierPassesTheModelReplyThrough(t *testing.T) {
|
||||
func TestLLMReplierPassesTheModelReplyThroughForOtherIntents(t *testing.T) {
|
||||
r := newLLMReplier(stubCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}, nil)
|
||||
got := r.Reply(context.Background(), router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
|
||||
got := r.Reply(context.Background(), router.Decision{Intent: router.IntentReminder, Slots: router.Slots{Text: "кофе закончился"}})
|
||||
if got != "записала, кофе закончился" {
|
||||
t.Errorf("got %q, want %q", got, "записала, кофе закончился")
|
||||
}
|
||||
}
|
||||
|
||||
type countingCompleter struct {
|
||||
out string
|
||||
calls int
|
||||
}
|
||||
|
||||
func (c *countingCompleter) Complete(_ context.Context, _ llm.Req) (string, error) {
|
||||
c.calls++
|
||||
return c.out, nil
|
||||
}
|
||||
|
||||
func TestLLMReplierNoteUsesFixedFeminineAcknowledgement(t *testing.T) {
|
||||
c := &countingCompleter{out: `{"response":"Хорошо, сохранил.","mood":"neutral"}`}
|
||||
r := newLLMReplier(c, nil)
|
||||
got := r.Reply(context.Background(), router.Decision{
|
||||
Intent: router.IntentNote, Slots: router.Slots{Text: "запасной ключ лежит в синей коробке"},
|
||||
})
|
||||
if c.calls != 0 {
|
||||
t.Fatalf("note acknowledgement called the resident model %d time(s), want none", c.calls)
|
||||
}
|
||||
if got != "сохранила заметку." {
|
||||
t.Fatalf("note acknowledgement = %q, want the fixed feminine line", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMReplierFallsBackToStubOnError(t *testing.T) {
|
||||
r := newLLMReplier(stubCompleter{err: errReplierTest}, nil)
|
||||
assertAck(t, r, router.Decision{Intent: router.IntentNote}, phraser.AckNote, "llm error")
|
||||
assertAck(t, r, router.Decision{Intent: router.IntentReminder}, phraser.AckReminder, "llm error")
|
||||
}
|
||||
|
||||
func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) {
|
||||
r := newLLMReplier(stubCompleter{out: ""}, nil)
|
||||
assertAck(t, r, router.Decision{Intent: router.IntentNote}, phraser.AckNote, "empty llm")
|
||||
assertAck(t, r, router.Decision{Intent: router.IntentReminder}, phraser.AckReminder, "empty llm")
|
||||
}
|
||||
|
||||
// A clarify never reaches the model, and since Vikunja #457 it is answered from
|
||||
|
||||
@@ -120,6 +120,7 @@ func (h *reactiveHandler) persistDecision(turnCtx context.Context, rec *decision
|
||||
Source: string(src),
|
||||
Winner: rec.Winner,
|
||||
Intent: wonIntent(rec),
|
||||
RouteProducer: rec.RouteProducer,
|
||||
ClaimedBeforeHead: claimedBeforeHead(rec),
|
||||
EncoderID: h.encoderID,
|
||||
Outcome: wonAt(rec, decision.StageAction),
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"regexp"
|
||||
|
||||
"github.com/kami/maven/internal/phraser"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// A question about her — "что ты умеешь", "кто ты" — used to have no answer at
|
||||
@@ -98,6 +99,16 @@ func selfFloor(utterance string) bool {
|
||||
// asked — "что ты умеешь" and "кто ты" want different halves of it — and falls
|
||||
// back to the text itself, which is already readable, if the model is down.
|
||||
func (h *reactiveHandler) querySelf(ctx context.Context, t *queryTurn) (string, bool) {
|
||||
// Product help is self knowledge too (Vikunja V-720), but unlike the prose description it
|
||||
// must be exact: these examples name the grammar Maven actually accepts.
|
||||
// Answer them before topic scoring so a phrasing such as "как отменить
|
||||
// задачу" cannot leak to SearXNG as generic third-party instructions.
|
||||
switch router.LocalHelpTopic(t.dec.Utterance) {
|
||||
case router.HelpReminderCancel:
|
||||
return "Скажи, например: «отмени напоминание про молоко». Если совпадений несколько, я попрошу выбрать одно.", true
|
||||
case router.HelpTaskDrop:
|
||||
return "Скажи, например: «убери из задач настроить бэкапы». Я уберу задачу из активного списка, не отмечая её выполненной.", true
|
||||
}
|
||||
if !h.turnIsAbout(ctx, t, topicSelf, selfFloor) {
|
||||
return "", false
|
||||
}
|
||||
|
||||
@@ -65,6 +65,36 @@ func TestSelfSourceAnswersFromTheDescription(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// V-720: asking how to operate Maven is never a third-party web-search query.
|
||||
func TestMavenHowToAnswersLocallyWithoutSearch(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
utterance string
|
||||
want string
|
||||
}{
|
||||
{"как отменить напоминание про молоко?", "отмени напоминание"},
|
||||
{"как отменить задачу настроить бэкапы?", "убери из задач"},
|
||||
{"можно ли отменить напоминание?", "отмени напоминание"},
|
||||
{"can I cancel a reminder?", "отмени напоминание"},
|
||||
{"could I cancel a task?", "убери из задач"},
|
||||
} {
|
||||
h, seen := searchHandler(t,
|
||||
`{"answers":["Инструкция стороннего приложения"],"results":[]}`,
|
||||
200)
|
||||
reply := h.actionQuery(context.Background(), router.Decision{
|
||||
Intent: router.IntentQuery,
|
||||
Utterance: testCase.utterance,
|
||||
Source: router.SourceSelf,
|
||||
SourceAnchored: true,
|
||||
})
|
||||
if !strings.Contains(reply, testCase.want) {
|
||||
t.Errorf("%q reply = %q, want local usage example containing %q", testCase.utterance, reply, testCase.want)
|
||||
}
|
||||
if *seen != "" {
|
||||
t.Errorf("%q leaked to search as %q", testCase.utterance, *seen)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSelfDescriptionHoldsThePersona — it is her own text and she reads it out,
|
||||
// so the same rules the phrasing eval enforces apply to it. Feminine
|
||||
// self-reference, informal address, no pet names.
|
||||
|
||||
+342
-12
@@ -12,6 +12,7 @@
|
||||
// what was SENT — every delivery.Sendable the dispatcher emitted
|
||||
// what ARRIVED — the unified intake journal from #283
|
||||
// what TOOLS were called — the recorded requests against fake Praxis/Nexis/Hexis
|
||||
// what is DURABLE — typed notes/tasks/reminders/facts store state
|
||||
// what did NOT happen — expect_no_send / expect_no_call, first-class
|
||||
//
|
||||
// The last one is the point. Maven's hard constraints are mostly negative —
|
||||
@@ -190,14 +191,81 @@ type step struct {
|
||||
|
||||
// --- assertions ---
|
||||
|
||||
ExpectReply []string `json:"expect_reply_contains,omitempty"`
|
||||
ExpectNotReply []string `json:"expect_reply_lacks,omitempty"`
|
||||
ExpectSent []string `json:"expect_sent_contains,omitempty"`
|
||||
ExpectNoSend bool `json:"expect_no_send,omitempty"`
|
||||
ExpectCalled []string `json:"expect_called,omitempty"`
|
||||
ExpectNotCalled []string `json:"expect_not_called,omitempty"`
|
||||
ExpectEvents []string `json:"expect_events,omitempty"`
|
||||
ExpectNoEvents bool `json:"expect_no_events,omitempty"`
|
||||
ExpectReply []string `json:"expect_reply_contains,omitempty"`
|
||||
ExpectNotReply []string `json:"expect_reply_lacks,omitempty"`
|
||||
ExpectSent []string `json:"expect_sent_contains,omitempty"`
|
||||
ExpectNoSend bool `json:"expect_no_send,omitempty"`
|
||||
ExpectCalled []string `json:"expect_called,omitempty"`
|
||||
ExpectNotCalled []string `json:"expect_not_called,omitempty"`
|
||||
ExpectEvents []string `json:"expect_events,omitempty"`
|
||||
ExpectNoEvents bool `json:"expect_no_events,omitempty"`
|
||||
ExpectStore *storeStateExpectation `json:"expect_store,omitempty"`
|
||||
}
|
||||
|
||||
// storeStateExpectation is a typed, exact read of Maven's four user-visible
|
||||
// durable stores. Reply assertions prove what she said; these prove what the
|
||||
// turn actually committed. Each selected collection can assert its total row
|
||||
// count and exact row identity independently, so a duplicate insert cannot be
|
||||
// hidden by finding one matching row.
|
||||
type storeStateExpectation struct {
|
||||
Notes *noteStateExpectation `json:"notes,omitempty"`
|
||||
Tasks *taskStateExpectation `json:"tasks,omitempty"`
|
||||
Reminders *reminderStateExpectation `json:"reminders,omitempty"`
|
||||
Facts *factStateExpectation `json:"facts,omitempty"`
|
||||
}
|
||||
|
||||
type noteStateExpectation struct {
|
||||
Count *int `json:"count,omitempty"`
|
||||
Rows []noteRowExpectation `json:"rows,omitempty"`
|
||||
}
|
||||
|
||||
type noteRowExpectation struct {
|
||||
ID int64 `json:"id,omitempty"`
|
||||
At string `json:"at,omitempty"`
|
||||
Text string `json:"text,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
type taskStateExpectation struct {
|
||||
Count *int `json:"count,omitempty"`
|
||||
Rows []taskRowExpectation `json:"rows,omitempty"`
|
||||
}
|
||||
|
||||
type taskRowExpectation struct {
|
||||
ID int64 `json:"id,omitempty"`
|
||||
CreatedAt string `json:"created_at,omitempty"`
|
||||
Text string `json:"text,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
Status string `json:"status,omitempty"`
|
||||
ResolvedAt string `json:"resolved_at,omitempty"`
|
||||
ResolvedBy string `json:"resolved_by,omitempty"`
|
||||
}
|
||||
|
||||
type reminderStateExpectation struct {
|
||||
Count *int `json:"count,omitempty"`
|
||||
Rows []reminderRowExpectation `json:"rows,omitempty"`
|
||||
}
|
||||
|
||||
type reminderRowExpectation struct {
|
||||
ID int64 `json:"id,omitempty"`
|
||||
FireAt string `json:"fire_at,omitempty"`
|
||||
Text string `json:"text,omitempty"`
|
||||
Status string `json:"status,omitempty"`
|
||||
}
|
||||
|
||||
type factStateExpectation struct {
|
||||
Count *int `json:"count,omitempty"`
|
||||
Rows []factRowExpectation `json:"rows,omitempty"`
|
||||
}
|
||||
|
||||
type factRowExpectation struct {
|
||||
ID int64 `json:"id,omitempty"`
|
||||
At string `json:"at,omitempty"`
|
||||
Kind string `json:"kind,omitempty"`
|
||||
Key string `json:"key,omitempty"`
|
||||
Value string `json:"value,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
Confidence *float64 `json:"confidence,omitempty"`
|
||||
}
|
||||
|
||||
type signalStep struct {
|
||||
@@ -440,6 +508,17 @@ func newSimWorld(t *testing.T, sc scenario) *simWorld {
|
||||
})
|
||||
tl := newTickLoop(st, gatherer, dispatcher, phraser.NewStub(), rules,
|
||||
time.Minute, 5*time.Minute, 0, nil, nil, nil, nil)
|
||||
// Production upgrades the voice handler from the direct store adapter to
|
||||
// daemonAPI after the tick loop exists. Mirror that seam so a simulated
|
||||
// day-plan query reads the real store-backed plan instead of the direct
|
||||
// adapter's "not available" refusal. The fake clock is the one deliberate
|
||||
// difference from production's time.Now.
|
||||
api = &daemonAPI{
|
||||
CoreAPI: api,
|
||||
getDayPlan: func(ctx context.Context) ipc.DayPlan {
|
||||
return tl.dayPlan(ctx, clock.Now())
|
||||
},
|
||||
}
|
||||
|
||||
scripted := &scriptedLLM{entries: sc.Script}
|
||||
|
||||
@@ -492,6 +571,7 @@ func newSimWorld(t *testing.T, sc scenario) *simWorld {
|
||||
// act panicked the moment the matcher was consulted.
|
||||
matcher := tool.NewMatcher(api)
|
||||
rtr := buildRouter(emb, matcher, config.DefaultRouterThreshold, router.NewLLMRouter(scripted), nil)
|
||||
timeParser := router.NewPythonDateParser()
|
||||
|
||||
w.handler = &reactiveHandler{
|
||||
stt: simTranscriber{},
|
||||
@@ -510,10 +590,11 @@ func newSimWorld(t *testing.T, sc scenario) *simWorld {
|
||||
replier: newLLMReplier(scripted, nil),
|
||||
now: clock.Now,
|
||||
dataStore: st,
|
||||
timeParser: router.StubDateTimeParser{},
|
||||
timeParser: timeParser,
|
||||
dialogueSessions: dialogue.NewSessionStore(time.Hour),
|
||||
clarifyStore: dialogue.NewClarifyStore(time.Hour),
|
||||
clarifyMaxAttempts: dialogue.DefaultMaxAttempts,
|
||||
extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}},
|
||||
ecosystem: eco,
|
||||
}
|
||||
return w
|
||||
@@ -599,7 +680,7 @@ func (w *simWorld) run(sc scenario) {
|
||||
eventsBefore := w.publishCount()
|
||||
|
||||
w.stimulate(ctx, s)
|
||||
w.assert(i, s, sendsBefore, callsBefore, eventsBefore)
|
||||
w.assert(ctx, i, s, sendsBefore, callsBefore, eventsBefore)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -615,7 +696,7 @@ func (w *simWorld) stimulate(ctx context.Context, s step) {
|
||||
|
||||
switch {
|
||||
case s.Say != "":
|
||||
reply := w.handler.runTurn(ctx, s.Say, sourceText)
|
||||
reply := w.handler.runTurn(ctx, router.NormalizedInput{Text: s.Say, Source: sourceText})
|
||||
w.replies = append(w.replies, reply)
|
||||
w.logf("он: %s", s.Say)
|
||||
w.logf("она: %s", reply)
|
||||
@@ -800,7 +881,7 @@ func (w *simWorld) callPaths() []string { return w.callPathsSince(nil) }
|
||||
// Assertions
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
func (w *simWorld) assert(i int, s step, sendsBefore int, callsBefore []int, eventsBefore int) {
|
||||
func (w *simWorld) assert(ctx context.Context, i int, s step, sendsBefore int, callsBefore []int, eventsBefore int) {
|
||||
w.t.Helper()
|
||||
where := fmt.Sprintf("step %d (%s)", i+1, s.At)
|
||||
if s.Note != "" {
|
||||
@@ -864,6 +945,197 @@ func (w *simWorld) assert(i int, s step, sendsBefore int, callsBefore []int, eve
|
||||
fail("expected nothing to arrive, %d event(s) were published",
|
||||
w.publishCount()-eventsBefore)
|
||||
}
|
||||
if s.ExpectStore != nil {
|
||||
w.assertStoreState(ctx, *s.ExpectStore, fail)
|
||||
}
|
||||
}
|
||||
|
||||
const simStateReadLimit = 10_000
|
||||
|
||||
func (w *simWorld) assertStoreState(ctx context.Context, want storeStateExpectation, fail func(string, ...any)) {
|
||||
if want.Notes != nil {
|
||||
rows, err := w.store.RecentNotes(ctx, simStateReadLimit)
|
||||
if err != nil {
|
||||
fail("read notes for store assertion: %v", err)
|
||||
} else {
|
||||
assertStateCount("notes", want.Notes.Count, len(rows), fail)
|
||||
used := make([]bool, len(rows))
|
||||
for _, expected := range want.Notes.Rows {
|
||||
matched, matchErr := matchDistinct(rows, used, func(row store.Note) (bool, error) {
|
||||
return w.noteStateMatches(row, expected)
|
||||
})
|
||||
if matchErr != nil {
|
||||
fail("invalid note expectation %+v: %v", expected, matchErr)
|
||||
} else if !matched {
|
||||
fail("no distinct note matches %+v; notes: %+v", expected, rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if want.Tasks != nil {
|
||||
rows, err := w.store.ListTasks(ctx, "")
|
||||
if err != nil {
|
||||
fail("read tasks for store assertion: %v", err)
|
||||
} else {
|
||||
if want.Tasks.Count != nil && *want.Tasks.Count > store.MaxTaskRows {
|
||||
fail("task count assertion %d exceeds the store read bound %d", *want.Tasks.Count, store.MaxTaskRows)
|
||||
} else {
|
||||
assertStateCount("tasks", want.Tasks.Count, len(rows), fail)
|
||||
}
|
||||
used := make([]bool, len(rows))
|
||||
for _, expected := range want.Tasks.Rows {
|
||||
matched, matchErr := matchDistinct(rows, used, func(row store.Task) (bool, error) {
|
||||
return w.taskStateMatches(row, expected)
|
||||
})
|
||||
if matchErr != nil {
|
||||
fail("invalid task expectation %+v: %v", expected, matchErr)
|
||||
} else if !matched {
|
||||
fail("no distinct task matches %+v; tasks: %+v", expected, rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if want.Reminders != nil {
|
||||
rows, err := w.store.ListReminders(ctx, simStateReadLimit)
|
||||
if err != nil {
|
||||
fail("read reminders for store assertion: %v", err)
|
||||
} else {
|
||||
assertStateCount("reminders", want.Reminders.Count, len(rows), fail)
|
||||
used := make([]bool, len(rows))
|
||||
for _, expected := range want.Reminders.Rows {
|
||||
matched, matchErr := matchDistinct(rows, used, func(row store.Reminder) (bool, error) {
|
||||
return w.reminderStateMatches(row, expected)
|
||||
})
|
||||
if matchErr != nil {
|
||||
fail("invalid reminder expectation %+v: %v", expected, matchErr)
|
||||
} else if !matched {
|
||||
fail("no distinct reminder matches %+v; reminders: %+v", expected, rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if want.Facts != nil {
|
||||
rows, err := w.store.RecentFacts(ctx, simStateReadLimit)
|
||||
if err != nil {
|
||||
fail("read facts for store assertion: %v", err)
|
||||
} else {
|
||||
assertStateCount("facts", want.Facts.Count, len(rows), fail)
|
||||
used := make([]bool, len(rows))
|
||||
for _, expected := range want.Facts.Rows {
|
||||
matched, matchErr := matchDistinct(rows, used, func(row store.Fact) (bool, error) {
|
||||
return w.factStateMatches(row, expected)
|
||||
})
|
||||
if matchErr != nil {
|
||||
fail("invalid fact expectation %+v: %v", expected, matchErr)
|
||||
} else if !matched {
|
||||
fail("no distinct fact matches %+v; facts: %+v", expected, rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func assertStateCount(kind string, want *int, got int, fail func(string, ...any)) {
|
||||
if want != nil && got != *want {
|
||||
fail("%s count = %d, want %d", kind, got, *want)
|
||||
}
|
||||
}
|
||||
|
||||
// matchDistinct prevents two expectations from being satisfied by the same
|
||||
// durable row. This is important for identity assertions where two records may
|
||||
// intentionally carry the same text but have different lifecycle states.
|
||||
func matchDistinct[T any](rows []T, used []bool, matches func(T) (bool, error)) (bool, error) {
|
||||
for i, row := range rows {
|
||||
if used[i] {
|
||||
continue
|
||||
}
|
||||
ok, err := matches(row)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if ok {
|
||||
used[i] = true
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
||||
func (w *simWorld) noteStateMatches(got store.Note, want noteRowExpectation) (bool, error) {
|
||||
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text != want.Text ||
|
||||
want.Source != "" && got.Source != want.Source {
|
||||
return false, nil
|
||||
}
|
||||
return w.stateTimeMatches(got.Ts, want.At)
|
||||
}
|
||||
|
||||
func (w *simWorld) taskStateMatches(got store.Task, want taskRowExpectation) (bool, error) {
|
||||
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text != want.Text ||
|
||||
want.Source != "" && got.Source != want.Source || want.Status != "" && got.Status != want.Status ||
|
||||
want.ResolvedBy != "" && got.ResolvedBy != want.ResolvedBy {
|
||||
return false, nil
|
||||
}
|
||||
if ok, err := w.stateTimeMatches(got.CreatedTs, want.CreatedAt); err != nil || !ok {
|
||||
return ok, err
|
||||
}
|
||||
if want.ResolvedAt == "" {
|
||||
return true, nil
|
||||
}
|
||||
if got.ResolvedTs == nil {
|
||||
return false, nil
|
||||
}
|
||||
return w.stateTimeMatches(*got.ResolvedTs, want.ResolvedAt)
|
||||
}
|
||||
|
||||
func (w *simWorld) reminderStateMatches(got store.Reminder, want reminderRowExpectation) (bool, error) {
|
||||
if want.ID != 0 && got.ID != want.ID || want.Text != "" && got.Text() != want.Text ||
|
||||
want.Status != "" && got.Status != want.Status {
|
||||
return false, nil
|
||||
}
|
||||
return w.stateTimeMatches(got.FireTs, want.FireAt)
|
||||
}
|
||||
|
||||
func (w *simWorld) factStateMatches(got store.Fact, want factRowExpectation) (bool, error) {
|
||||
if want.ID != 0 && got.ID != want.ID || want.Kind != "" && string(got.Kind) != want.Kind ||
|
||||
want.Key != "" && got.Key != want.Key || want.Value != "" && got.Value != want.Value ||
|
||||
want.Source != "" && got.Source != want.Source ||
|
||||
want.Confidence != nil && got.Confidence != *want.Confidence {
|
||||
return false, nil
|
||||
}
|
||||
return w.stateTimeMatches(got.Ts, want.At)
|
||||
}
|
||||
|
||||
func (w *simWorld) stateTimeMatches(got time.Time, raw string) (bool, error) {
|
||||
if raw == "" {
|
||||
return true, nil
|
||||
}
|
||||
want, err := w.stateTime(raw)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return got.Equal(want), nil
|
||||
}
|
||||
|
||||
// stateTime accepts either an absolute RFC3339 instant or the same local
|
||||
// HH:MM[:SS] shape scenario steps use. The latter keeps fixtures readable
|
||||
// while still comparing exact instants after the store normalises to UTC.
|
||||
func (w *simWorld) stateTime(raw string) (time.Time, error) {
|
||||
if strings.Contains(raw, "T") {
|
||||
return time.Parse(time.RFC3339, raw)
|
||||
}
|
||||
layout := "15:04"
|
||||
if strings.Count(raw, ":") == 2 {
|
||||
layout = "15:04:05"
|
||||
}
|
||||
hm, err := time.Parse(layout, raw)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("expected HH:MM[:SS] or RFC3339, got %q: %w", raw, err)
|
||||
}
|
||||
return time.Date(w.start.Year(), w.start.Month(), w.start.Day(),
|
||||
hm.Hour(), hm.Minute(), hm.Second(), 0, w.loc), nil
|
||||
}
|
||||
|
||||
func sendableTexts(sends []delivery.Sendable) []string {
|
||||
@@ -933,6 +1205,64 @@ func TestSimulatorScenarios(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSimulatorWorldMirrorsProductionConversationSeams pins the two daemon
|
||||
// constructor upgrades the continuous scenario needs. A bare store API cannot
|
||||
// answer DayPlan, and a nil handler extractor cannot complete a parked reminder
|
||||
// from the next turn; either drift would make the simulator exercise a smaller
|
||||
// system than production while still producing plausible replies.
|
||||
func TestSimulatorWorldMirrorsProductionConversationSeams(t *testing.T) {
|
||||
sc := scenario{SchemaVersion: 1, Name: "constructor-seams", Start: "2026-08-15T08:00:00+04:00"}
|
||||
w := newSimWorld(t, sc)
|
||||
if w.handler.api != w.api {
|
||||
t.Fatal("handler did not receive the simulator's upgraded daemon API")
|
||||
}
|
||||
plan, err := w.handler.api.DayPlan(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("simulator day-plan seam is unavailable: %v", err)
|
||||
}
|
||||
planY, planM, planD := plan.Date.In(w.loc).Date()
|
||||
wantY, wantM, wantD := w.start.Date()
|
||||
if plan.Date.IsZero() || planY != wantY || planM != wantM || planD != wantD {
|
||||
t.Fatalf("day plan date = %v, want the fake-clock day %v", plan.Date, w.start)
|
||||
}
|
||||
if w.handler.extractor.Time == nil || w.handler.timeParser == nil {
|
||||
t.Fatal("simulator left the clarify time parser unwired")
|
||||
}
|
||||
slots := w.handler.extractor.Extract(context.Background(), router.IntentReminder,
|
||||
"сегодня в 10:00", w.clock.Now())
|
||||
if !slots.HasTime || !slots.Time.Equal(w.timeOf("10:00")) {
|
||||
t.Fatalf("clarify extractor parsed %+v, want the fake-clock day at 10:00", slots)
|
||||
}
|
||||
}
|
||||
|
||||
// The reported duplicate transcript was a diagnostic artefact: two adjacent
|
||||
// sed ranges both included boundary line 620. Source had one log call. Keep an
|
||||
// executable exact-count assertion so a real duplicate cannot be introduced
|
||||
// later and mistaken for another display artefact.
|
||||
func TestSimulatorTranscriptRecordsEachSpokenTurnOnce(t *testing.T) {
|
||||
sc := scenario{
|
||||
SchemaVersion: 1,
|
||||
Name: "transcript-count",
|
||||
Start: "2026-08-15T08:00:00+04:00",
|
||||
Script: []scriptEntry{{
|
||||
Match: "привет", Route: `[{"intent":"chat","text":"привет"}]`,
|
||||
Reply: `{"response":"Привет.","mood":"happy"}`,
|
||||
}},
|
||||
}
|
||||
w := newSimWorld(t, sc)
|
||||
w.stimulate(context.Background(), step{Say: "привет"})
|
||||
want := "08:00:00 он: привет"
|
||||
count := 0
|
||||
for _, line := range w.transcript {
|
||||
if line == want {
|
||||
count++
|
||||
}
|
||||
}
|
||||
if count != 1 {
|
||||
t.Fatalf("owner transcript line occurred %d times, want exactly once: %v", count, w.transcript)
|
||||
}
|
||||
}
|
||||
|
||||
func loadScenario(t *testing.T, path string) scenario {
|
||||
t.Helper()
|
||||
raw, err := os.ReadFile(path)
|
||||
|
||||
@@ -31,6 +31,12 @@ func TestSlotsParity(t *testing.T) {
|
||||
t.Errorf("router.Slots.%s (%s) missing from dialogue.Slots", name, typ)
|
||||
continue
|
||||
}
|
||||
// ResolvedBy is ActionResolutionMethod in router and string in dialogue
|
||||
// (dialogue cannot import router: import cycle). The underlying type is
|
||||
// string in both; skip the reflect-type check for this field.
|
||||
if name == "ResolvedBy" {
|
||||
continue
|
||||
}
|
||||
if dt != typ {
|
||||
t.Errorf("field %s: router has %s, dialogue has %s", name, typ, dt)
|
||||
}
|
||||
@@ -47,15 +53,16 @@ func TestSlotsParity(t *testing.T) {
|
||||
// populated value and compare.
|
||||
func TestSlotsRoundTrip(t *testing.T) {
|
||||
full := router.Slots{
|
||||
Time: time.Date(2026, 8, 2, 11, 0, 0, 0, time.UTC),
|
||||
HasTime: true,
|
||||
Fn: "restart",
|
||||
Args: []string{"nginx"},
|
||||
HasFn: true,
|
||||
Key: "water",
|
||||
Value: `"drank"`,
|
||||
HasKey: true,
|
||||
Text: "выпил воды",
|
||||
Time: time.Date(2026, 8, 2, 11, 0, 0, 0, time.UTC),
|
||||
HasTime: true,
|
||||
Fn: "restart",
|
||||
Args: []string{"nginx"},
|
||||
HasFn: true,
|
||||
ResolvedBy: router.ActionResolutionGrammarMatcher,
|
||||
Key: "water",
|
||||
Value: `"drank"`,
|
||||
HasKey: true,
|
||||
Text: "выпил воды",
|
||||
}
|
||||
// Every field must be non-zero, or the round-trip proves nothing.
|
||||
rv := reflect.ValueOf(full)
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"name": "assistant_workday",
|
||||
"description": "One continuous, deterministic workday through Maven's real conversation pipeline. It proves note capture and grounded high-overlap recall; a reminder that remains uncommitted while Maven clarifies its day, then survives a reported-action no-op, is cancelled exactly once, and stays cancelled on a repeated command; task capture, listing, completion, and a second live task; and the same calendar fact read through both agenda and composed day-plan sources. Store assertions are primary: every mutation and no-op pins exact row count, identity, lifecycle state, provenance, and fake-clock time.",
|
||||
"start": "2026-08-15T08:00:00+04:00",
|
||||
"script": [
|
||||
{
|
||||
"match": "запомни: запасной ключ лежит",
|
||||
"route": "[{\"intent\":\"note\",\"text\":\"запомни: запасной ключ лежит в синей коробке\"}]"
|
||||
},
|
||||
{
|
||||
"match": "запасной ключ лежит в синей коробке",
|
||||
"route": "[{\"intent\":\"query\",\"text\":\"запасной ключ лежит в синей коробке?\",\"source\":\"recall\"}]"
|
||||
},
|
||||
{
|
||||
"match": "я отменил напоминание",
|
||||
"route": "[{\"intent\":\"chat\",\"text\":\"я отменил напоминание про молоко\"}]",
|
||||
"reply": "{\"response\":\"Поняла.\",\"mood\":\"neutral\"}"
|
||||
},
|
||||
{
|
||||
"match": "",
|
||||
"route": "[{\"intent\":\"chat\",\"text\":\"\"}]",
|
||||
"reply": "{\"response\":\"Поняла.\",\"mood\":\"neutral\"}"
|
||||
}
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"at": "08:00",
|
||||
"note": "Capture only the dictated body as one durable note: the command frame is not memory, and no task, reminder, or fact is created.",
|
||||
"say": "запомни: запасной ключ лежит в синей коробке",
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"notes": { "count": 1, "rows": [{ "id": 1, "at": "08:00", "text": "запасной ключ лежит в синей коробке", "source": "tap:voice" }] },
|
||||
"tasks": { "count": 0 },
|
||||
"reminders": { "count": 0 },
|
||||
"facts": { "count": 0 }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:01",
|
||||
"note": "Recall reads the stored note and does not create a second row.",
|
||||
"say": "запасной ключ лежит в синей коробке?",
|
||||
"expect_reply_contains": ["синей коробке"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"notes": { "count": 1, "rows": [{ "id": 1, "text": "запасной ключ лежит в синей коробке", "source": "tap:voice" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:02",
|
||||
"note": "A clock without a day is not a committed reminder. Maven asks, and the reminder table remains empty.",
|
||||
"say": "напомни купить молоко в 10:00",
|
||||
"expect_reply_contains": ["В какой день"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"reminders": { "count": 0 }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:03",
|
||||
"note": "The clarification completes the parked request against the fake clock and creates exactly one pending reminder.",
|
||||
"say": "сегодня",
|
||||
"expect_reply_contains": ["10:00"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "pending" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:04",
|
||||
"note": "A first-person report is not another cancellation command and cannot mutate the pending row.",
|
||||
"say": "я отменил напоминание про молоко",
|
||||
"expect_reply_contains": ["Поняла"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "pending" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:05",
|
||||
"note": "The addressed imperative cancels that exact durable reminder in place.",
|
||||
"say": "отмени напоминание про молоко",
|
||||
"expect_reply_contains": ["отменила напоминание", "купить молоко"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "cancelled" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:06",
|
||||
"note": "Repeating the cancellation is an explicit no-op: no replacement row and no resurrection.",
|
||||
"say": "отмени напоминание про молоко",
|
||||
"expect_reply_contains": ["ожидающих напоминаний нет"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"reminders": { "count": 1, "rows": [{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "cancelled" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:07",
|
||||
"note": "An explicit task marker creates one open task, not a note.",
|
||||
"say": "добавь в задачи настроить бэкапы",
|
||||
"expect_reply_contains": ["настроить бэкапы"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"notes": { "count": 1 },
|
||||
"tasks": { "count": 1, "rows": [{ "id": 1, "created_at": "08:07", "text": "настроить бэкапы", "source": "tap:voice", "status": "open" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:08",
|
||||
"note": "Listing is read-only and returns the live task without duplicating it.",
|
||||
"say": "какие у меня задачи?",
|
||||
"expect_reply_contains": ["настроить бэкапы"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"tasks": { "count": 1, "rows": [{ "id": 1, "text": "настроить бэкапы", "status": "open" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:09",
|
||||
"note": "Naming the task moves the same row forward to done and records who resolved it.",
|
||||
"say": "закрой задачу настроить бэкапы",
|
||||
"expect_reply_contains": ["настроить бэкапы"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"tasks": { "count": 1, "rows": [{ "id": 1, "created_at": "08:07", "text": "настроить бэкапы", "source": "tap:voice", "status": "done", "resolved_at": "08:09", "resolved_by": "tap:voice" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:10",
|
||||
"note": "A second task remains live for the rest of the workday while the completed row remains durable history.",
|
||||
"say": "добавь в задачи отправить отчёт",
|
||||
"expect_reply_contains": ["отправить отчёт"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"tasks": { "count": 2, "rows": [
|
||||
{ "id": 1, "text": "настроить бэкапы", "status": "done", "resolved_by": "tap:voice" },
|
||||
{ "id": 2, "created_at": "08:10", "text": "отправить отчёт", "source": "tap:voice", "status": "open" }
|
||||
] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:11",
|
||||
"note": "A fully specified reminder commits directly and coexists with the cancelled history row.",
|
||||
"say": "напомни сегодня в 12:00 размяться",
|
||||
"expect_reply_contains": ["12:00"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"reminders": { "count": 2, "rows": [
|
||||
{ "id": 1, "fire_at": "10:00", "text": "купить молоко", "status": "cancelled" },
|
||||
{ "id": 2, "fire_at": "12:00", "text": "размяться", "status": "pending" }
|
||||
] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:12",
|
||||
"note": "A calendar poll contributes one exact env fact at the event instant.",
|
||||
"arrive": {
|
||||
"source": "poll:caldav",
|
||||
"as_of": "11:00",
|
||||
"fact": {
|
||||
"key": "calendar_event_20260815_Планёрка",
|
||||
"value": "Планёрка @ 11:00-11:30",
|
||||
"kind": "env"
|
||||
}
|
||||
},
|
||||
"expect_events": ["calendar_event_20260815_Планёрка"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"facts": { "count": 1, "rows": [{ "id": 1, "at": "11:00", "kind": "env", "key": "calendar_event_20260815_Планёрка", "value": "Планёрка @ 11:00-11:30", "source": "poll:caldav", "confidence": 1.0 }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:13",
|
||||
"note": "The agenda source reads the calendar fact without changing any durable state.",
|
||||
"say": "что у меня сегодня?",
|
||||
"expect_reply_contains": ["Планёрка"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"facts": { "count": 1, "rows": [{ "id": 1, "key": "calendar_event_20260815_Планёрка", "source": "poll:caldav" }] },
|
||||
"tasks": { "count": 2 },
|
||||
"reminders": { "count": 2 }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:14",
|
||||
"note": "The daemon day-plan seam composes the same calendar fact with the still-pending reminder; cancelled reminders stay out.",
|
||||
"say": "какие планы на сегодня?",
|
||||
"expect_reply_contains": ["Планёрка", "размяться"],
|
||||
"expect_reply_lacks": ["купить молоко"],
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"notes": { "count": 1, "rows": [{ "id": 1, "text": "запасной ключ лежит в синей коробке" }] },
|
||||
"tasks": { "count": 2, "rows": [
|
||||
{ "id": 1, "text": "настроить бэкапы", "status": "done" },
|
||||
{ "id": 2, "text": "отправить отчёт", "status": "open" }
|
||||
] },
|
||||
"reminders": { "count": 2, "rows": [
|
||||
{ "id": 1, "text": "купить молоко", "status": "cancelled" },
|
||||
{ "id": 2, "text": "размяться", "status": "pending" }
|
||||
] },
|
||||
"facts": { "count": 1, "rows": [{ "id": 1, "key": "calendar_event_20260815_Планёрка", "value": "Планёрка @ 11:00-11:30" }] }
|
||||
}
|
||||
},
|
||||
{
|
||||
"at": "08:15",
|
||||
"note": "A normal tick after the session remains silent; durable assistant state does not authorize an unsolicited message.",
|
||||
"tick": true,
|
||||
"expect_no_send": true,
|
||||
"expect_store": {
|
||||
"notes": { "count": 1 },
|
||||
"tasks": { "count": 2 },
|
||||
"reminders": { "count": 2 },
|
||||
"facts": { "count": 1 }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -213,7 +213,8 @@ func isPleasantry(text string) bool {
|
||||
// It is also the whole answer when there is no route to read — the classifier
|
||||
// is the failure floor and a turn must never break on the model.
|
||||
func offlineOwnRequest(text string) bool {
|
||||
return router.IsQuestionShaped(text) || router.CarriesCaptureVerb(text) || carriesReminderVerb(text)
|
||||
_, cancelsReminder := parseReminderCancelRequest(text)
|
||||
return router.IsQuestionShaped(text) || router.CarriesCaptureVerb(text) || carriesReminderVerb(text) || cancelsReminder
|
||||
}
|
||||
|
||||
// classifyTurnRole decides what this utterance is against the pending action.
|
||||
|
||||
@@ -264,10 +264,10 @@ func TestClarifyCancelEndsTheExchange(t *testing.T) {
|
||||
// the same memo.
|
||||
func TestTheTurnIsRoutedOnce(t *testing.T) {
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
rt := h.newTurnRoute("какая сейчас погода в Риме?", h.now())
|
||||
rt := h.newTurnRoute(router.NormalizedInput{Text: "какая сейчас погода в Риме?", Source: sourceText}, h.now())
|
||||
ctx := withTurnRoute(withDialogueID(context.Background(), voiceDialogueID), rt)
|
||||
|
||||
first, ok := h.routeForRole(ctx, rt.text)
|
||||
first, ok := h.routeForRole(ctx, rt.input.Text)
|
||||
if !ok {
|
||||
t.Fatal("the cascade must produce a decision to classify against")
|
||||
}
|
||||
|
||||
+11
-8
@@ -18,9 +18,9 @@ import (
|
||||
// second on the resident model and — worse — could disagree with itself, which
|
||||
// is exactly the class of bug this task is about.
|
||||
type turnRoute struct {
|
||||
h *reactiveHandler
|
||||
text string
|
||||
now time.Time
|
||||
h *reactiveHandler
|
||||
input router.NormalizedInput
|
||||
now time.Time
|
||||
|
||||
once sync.Once
|
||||
dec router.Decision
|
||||
@@ -46,8 +46,8 @@ type turnRoute struct {
|
||||
|
||||
type turnRouteKey struct{}
|
||||
|
||||
func (h *reactiveHandler) newTurnRoute(text string, now time.Time) *turnRoute {
|
||||
return &turnRoute{h: h, text: text, now: now}
|
||||
func (h *reactiveHandler) newTurnRoute(input router.NormalizedInput, now time.Time) *turnRoute {
|
||||
return &turnRoute{h: h, input: input, now: now}
|
||||
}
|
||||
|
||||
func withTurnRoute(ctx context.Context, rt *turnRoute) context.Context {
|
||||
@@ -70,7 +70,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
|
||||
if r.h.dialogueSessions != nil {
|
||||
r.prev = r.h.dialogueSessions.Get(dialogueIDOf(ctx), r.now)
|
||||
}
|
||||
if dec, cont := continuationDecision(r.prev, r.text, r.now); cont {
|
||||
if dec, cont := continuationDecision(r.prev, r.input.Text, r.now); cont {
|
||||
log.Printf("voice: continuation of %s from the previous turn", dec.Intent)
|
||||
r.dec, r.cont = dec, true
|
||||
return
|
||||
@@ -79,7 +79,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
|
||||
r.err = router.ErrNoIntents
|
||||
return
|
||||
}
|
||||
r.dec, r.err = r.h.router.Route(ctx, r.text, r.now)
|
||||
r.dec, r.err = r.h.router.Route(ctx, r.input, r.now)
|
||||
})
|
||||
return r.dec, r.cont, r.prev, r.err
|
||||
}
|
||||
@@ -92,7 +92,7 @@ func (r *turnRoute) resolve(ctx context.Context) (router.Decision, bool, *dialog
|
||||
func (h *reactiveHandler) routeForRole(ctx context.Context, text string) (router.Decision, bool) {
|
||||
rt := turnRouteFrom(ctx)
|
||||
if rt == nil {
|
||||
rt = h.newTurnRoute(text, h.now())
|
||||
rt = h.newTurnRoute(router.NormalizedInput{Text: text, Source: sourceText}, h.now())
|
||||
}
|
||||
dec, _, _, err := rt.resolve(ctx)
|
||||
if err != nil {
|
||||
@@ -121,5 +121,8 @@ func needsRoute(text string) bool {
|
||||
if isCancel(text) {
|
||||
return false
|
||||
}
|
||||
if _, ok := parseReminderCancelRequest(text); ok {
|
||||
return false
|
||||
}
|
||||
return len(ownContent(text)) > 0 || router.IsQuestionShaped(text)
|
||||
}
|
||||
|
||||
+40
-18
@@ -211,7 +211,7 @@ func (h *reactiveHandler) HandlePushToTalk(ctx context.Context, req voice.PushTo
|
||||
|
||||
// 2-5. the shared turn pipeline (confirm → clarify → route → dialogue →
|
||||
// action → replier), identical to the text path.
|
||||
replyText := h.runTurn(ctx, text, sourceVoice)
|
||||
replyText := h.runTurn(ctx, router.NormalizedInput{Text: text, MatchText: router.NormalizeMatchText(text), Source: sourceVoice})
|
||||
|
||||
// 6. tts — synthesise the reply text; return to the voice server which
|
||||
// ships it back on the conn.
|
||||
@@ -244,30 +244,30 @@ func (h *reactiveHandler) upgradeAPI(api ipc.CoreAPI) {
|
||||
// HandlePushToTalk so text channels share the same routing logic.
|
||||
func (h *reactiveHandler) handleText(ctx context.Context, conversation, text string) string {
|
||||
log.Printf("voice: handleText: %q", text)
|
||||
return h.runTurn(withDialogueID(ctx, dialogueIDFor(sourceText, conversation)), text, sourceText)
|
||||
return h.runTurn(withDialogueID(ctx, dialogueIDFor(sourceText, conversation)), router.NormalizedInput{Text: text, MatchText: router.NormalizeMatchText(text), Source: sourceText})
|
||||
}
|
||||
|
||||
// turnSource — which channel this utterance arrived on, in the same provenance
|
||||
// vocabulary facts use (internal/event). It is threaded through runTurn because
|
||||
// a turn can write a fact, and a fact that lies about where it came from is
|
||||
// worse than no fact: provenance is the first column read when asking why a
|
||||
// daemon-wide setting is the way it is.
|
||||
type turnSource string
|
||||
// turnSource is a local alias for router.InputSource, kept so the daemon code
|
||||
// reads sourceVoice/sourceText without a package prefix at every call site.
|
||||
// The canonical type lives in the router package; this is pure convenience.
|
||||
type turnSource = router.InputSource
|
||||
|
||||
const (
|
||||
sourceVoice turnSource = "tap:voice" // HandlePushToTalk, a real microphone
|
||||
sourceText turnSource = "tap:text" // handleText: mavweb /api/chat, telegram
|
||||
sourceVoice = router.InputSourceVoice
|
||||
sourceText = router.InputSourceText
|
||||
)
|
||||
|
||||
// runTurn — the reactive turn pipeline shared by the voice and text entry
|
||||
// points: expired-clarify notice → confirm answer → explicit correction →
|
||||
// clarify answer → quiet toggle → route → dialogue merge → clarify question →
|
||||
// action → replier.
|
||||
// Takes the already-transcribed utterance, returns the reply text; the voice
|
||||
// path wraps it in stt/tts, the text path returns it as-is.
|
||||
// clarify answer → quiet toggle → reminder cancellation → route → dialogue
|
||||
// merge → clarify question → action → replier.
|
||||
// Takes the NormalizedInput (typed ingress boundary), returns the reply text;
|
||||
// the voice path wraps it in stt/tts, the text path returns it as-is.
|
||||
//
|
||||
// The ordering is load-bearing — see the step comments.
|
||||
func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSource) (reply string) {
|
||||
func (h *reactiveHandler) runTurn(ctx context.Context, input router.NormalizedInput) (reply string) {
|
||||
text := input.Text
|
||||
src := input.Source
|
||||
// 0. the decision record (V-564). Installed here rather than in the IPC
|
||||
// entry point, so the mic, telegram and the web all leave the same trail —
|
||||
// a record only the web produced would be missing exactly the turns that
|
||||
@@ -275,7 +275,7 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
// on a human-rate path, and no claim site can change a route with it.
|
||||
if h.decisions != nil {
|
||||
var rec *decision.Record
|
||||
ctx, rec = decision.With(ctx, text)
|
||||
ctx, rec = decision.With(ctx, text, string(src))
|
||||
decision.Expect(ctx, decision.StagePreRoute, preRouteLadder)
|
||||
defer func() {
|
||||
done := rec.Finish(h.now())
|
||||
@@ -289,7 +289,7 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
// claiming it, and step 5 acts on the same decision — routing twice would
|
||||
// cost a second on the resident model and could disagree with itself.
|
||||
now := h.now()
|
||||
rt := h.newTurnRoute(text, now)
|
||||
rt := h.newTurnRoute(input, now)
|
||||
ctx = withTurnRoute(ctx, rt)
|
||||
// A resolver may suspend an older clarify flow even when it handles this
|
||||
// turn itself. Finalise that state at one choke point so early returns from
|
||||
@@ -335,6 +335,16 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
return withNotice(expiredNotice, reply)
|
||||
}
|
||||
|
||||
// 3c. explicit command prohibition — negative authority must be settled
|
||||
// before a parked slot or candidate can consume these words. In particular,
|
||||
// "не отменяй напоминание" is not the subject/time answer to an older
|
||||
// reminder request. Confirmation stays above it: "don't" is already a
|
||||
// closed no-answer to a destructive confirm, and that narrower stateful
|
||||
// contract must retain first refusal.
|
||||
if reply, handled := h.resolveCommandProhibition(ctx, text); notePreRoute(ctx, "command-prohibition", handled) {
|
||||
return withNotice(expiredNotice, reply)
|
||||
}
|
||||
|
||||
// 4. clarify answer — if she asked a live question last turn, this
|
||||
// utterance is its answer, not a fresh command. After the confirm check: a
|
||||
// y/n gate is armed by her own prompt and is the narrower claim on the
|
||||
@@ -372,7 +382,15 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
return withNotice(expiredNotice, reply)
|
||||
}
|
||||
|
||||
// 5d. ordinal selection — "второй", "первую сделал" pick from the list she
|
||||
// 5d. committed-reminder cancellation — an explicit cancel verb plus the
|
||||
// reminder noun resolves against pending rows. It runs before ordinal so a
|
||||
// clock such as "на девять" cannot be mistaken for a position in an older
|
||||
// task list; an ambiguous result binds its own list for the next turn.
|
||||
if reply, handled := h.resolveReminderCancellation(ctx, text); notePreRoute(ctx, "reminder-cancel", handled) {
|
||||
return withNotice(expiredNotice, reply)
|
||||
}
|
||||
|
||||
// 5e. ordinal selection — "второй", "первую сделал" pick from the list she
|
||||
// just read (ordinal.go). Before routing, and only when a list is actually
|
||||
// bound to the session: with nothing offered, "второй" is an ordinary word
|
||||
// and keeps routing.
|
||||
@@ -396,6 +414,10 @@ func (h *reactiveHandler) runTurn(ctx context.Context, text string, src turnSour
|
||||
return withNotice(expiredNotice, "не получилось разобрать команду.")
|
||||
}
|
||||
log.Printf("voice: route result: intent=%s slots=%+v", dec.Intent, dec.Slots)
|
||||
// Carry the route producer into the decision record for observability.
|
||||
if rec := decision.From(ctx); rec != nil && dec.Producer != "" {
|
||||
rec.RouteProducer = string(dec.Producer)
|
||||
}
|
||||
|
||||
// 7. dialogue — fill this turn's missing slots from a prior same-intent
|
||||
// turn (follow-ups like «напомни завтра» → «…позвонить маме»), then remember
|
||||
|
||||
+3
-1
@@ -143,7 +143,9 @@ func main() {
|
||||
log.Printf("mavweb: ambient notification ingest enabled at POST /api/ambient")
|
||||
}
|
||||
|
||||
// The read surfaces. Every one of them 503s without -core.
|
||||
// The data surfaces. Every one of them 503s without -core. /reminders also
|
||||
// accepts an ID-bound cancellation POST. It is deliberately not step-up
|
||||
// gated: like dismissing a proposed routine, it can only make Maven quieter.
|
||||
mux.HandleFunc("/dash", corePage(handleDash))
|
||||
mux.HandleFunc("/history", corePage(handleHistory))
|
||||
mux.HandleFunc("/trace", corePage(handleTrace))
|
||||
|
||||
+111
-15
@@ -3,8 +3,10 @@ package main
|
||||
import (
|
||||
_ "embed"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
@@ -23,11 +25,14 @@ var remindersTmpl = parsePage("reminders", remindersHTML, nil)
|
||||
// (Vikunja #469). Neither is a formatting nicety: the envelope is an internal
|
||||
// shape he never chose, and a time on a page he reads is the time on his wall.
|
||||
type reminderRow struct {
|
||||
Created string
|
||||
Fires string
|
||||
Status string
|
||||
Detail string
|
||||
Text string
|
||||
ID int64
|
||||
Created string
|
||||
Fires string
|
||||
Schedule string
|
||||
Status string
|
||||
Detail string
|
||||
Text string
|
||||
CanCancel bool
|
||||
}
|
||||
|
||||
// reminderText unwraps the {"text":...} payload the router writes.
|
||||
@@ -51,8 +56,16 @@ func reminderText(payload string) string {
|
||||
func reminderRows(rs []ipc.Reminder) []reminderRow {
|
||||
out := make([]reminderRow, 0, len(rs))
|
||||
for _, r := range rs {
|
||||
fire := r.NextFireTs
|
||||
if fire.IsZero() {
|
||||
fire = r.FireTs
|
||||
}
|
||||
status := r.Status
|
||||
detail := ""
|
||||
schedule := ""
|
||||
if r.Cron != "" {
|
||||
schedule = "recurring · " + r.Cron
|
||||
}
|
||||
if !r.DeliveryBlockedTs.IsZero() {
|
||||
status = "blocked"
|
||||
detail = r.DeliveryBlockedError
|
||||
@@ -60,25 +73,108 @@ func reminderRows(rs []ipc.Reminder) []reminderRow {
|
||||
detail = "retry " + r.NextAttemptTs.Local().Format("02 Jan 15:04")
|
||||
}
|
||||
out = append(out, reminderRow{
|
||||
Created: r.CreatedTs.Local().Format("02 Jan 15:04"),
|
||||
Fires: r.FireTs.Local().Format("02 Jan 15:04"),
|
||||
Status: status,
|
||||
Detail: detail,
|
||||
Text: reminderText(r.Payload),
|
||||
ID: r.ID,
|
||||
Created: r.CreatedTs.Local().Format("02 Jan 15:04"),
|
||||
Fires: fire.Local().Format("02 Jan 15:04"),
|
||||
Schedule: schedule,
|
||||
Status: status,
|
||||
Detail: detail,
|
||||
Text: reminderText(r.Payload),
|
||||
CanCancel: r.Status == "pending",
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// remindersForPage keeps every pending row reachable while retaining the
|
||||
// recent terminal history the page already showed. Pending rows come first in
|
||||
// firing order (the CoreAPI contract); IDs present in the recent window are not
|
||||
// duplicated below them.
|
||||
func remindersForPage(pending, recent []ipc.Reminder) []ipc.Reminder {
|
||||
out := make([]ipc.Reminder, 0, len(pending)+len(recent))
|
||||
seen := make(map[int64]bool, len(pending))
|
||||
for _, reminder := range pending {
|
||||
out = append(out, reminder)
|
||||
seen[reminder.ID] = true
|
||||
}
|
||||
for _, reminder := range recent {
|
||||
if seen[reminder.ID] {
|
||||
continue
|
||||
}
|
||||
out = append(out, reminder)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func handleReminders(w http.ResponseWriter, r *http.Request, core ipc.CoreAPI) {
|
||||
if !requireCore(w, r, core, "reminders") {
|
||||
return
|
||||
}
|
||||
reminders, err := core.ListReminders(r.Context(), 50)
|
||||
if err != nil {
|
||||
writeProblem(w, r, http.StatusBadGateway, problemCoreReadFailed,
|
||||
"reminders unavailable", fmt.Errorf("list reminders: %w", err))
|
||||
msg := ""
|
||||
if r.Method == http.MethodGet && r.URL.Query().Get("cancelled") == "1" {
|
||||
msg = "reminder cancelled"
|
||||
}
|
||||
switch r.Method {
|
||||
case http.MethodGet:
|
||||
case http.MethodPost:
|
||||
if strings.TrimSpace(r.FormValue("action")) != "cancel" {
|
||||
writeProblem(w, r, http.StatusBadRequest, problemInvalidRequest,
|
||||
"unknown reminder action", nil)
|
||||
return
|
||||
}
|
||||
id, err := strconv.ParseInt(strings.TrimSpace(r.FormValue("id")), 10, 64)
|
||||
if err != nil || id <= 0 {
|
||||
writeProblem(w, r, http.StatusBadRequest, problemInvalidRequest,
|
||||
"invalid reminder id", err)
|
||||
return
|
||||
}
|
||||
if err := core.CancelReminder(r.Context(), id); err != nil {
|
||||
switch {
|
||||
case errors.Is(err, ipc.ErrReminderNotFound):
|
||||
writeProblem(w, r, http.StatusNotFound, problemResourceNotFound,
|
||||
"reminder not found", err)
|
||||
case errors.Is(err, ipc.ErrReminderInFlight):
|
||||
writeProblem(w, r, http.StatusConflict, problemCoreChangeFailed,
|
||||
"reminder delivery has already started", err)
|
||||
case errors.Is(err, ipc.ErrReminderState):
|
||||
writeProblem(w, r, http.StatusConflict, problemCoreChangeFailed,
|
||||
"reminder is no longer pending", err)
|
||||
default:
|
||||
writeProblem(w, r, http.StatusBadGateway, problemCoreChangeFailed,
|
||||
"reminder cancellation failed", fmt.Errorf("cancel reminder %d: %w", id, err))
|
||||
}
|
||||
return
|
||||
}
|
||||
http.Redirect(w, r, "/reminders?cancelled=1", http.StatusSeeOther)
|
||||
return
|
||||
default:
|
||||
writeProblem(w, r, http.StatusMethodNotAllowed, problemMethodNotAllowed,
|
||||
"method not allowed", nil)
|
||||
return
|
||||
}
|
||||
renderPage(w, remindersTmpl, map[string]any{"Reminders": reminderRows(reminders)})
|
||||
pending, err := core.ListPendingReminders(r.Context(), 0)
|
||||
if err != nil {
|
||||
public := "reminders unavailable"
|
||||
if msg != "" {
|
||||
public = "reminder cancelled; refreshed list unavailable"
|
||||
}
|
||||
writeProblem(w, r, http.StatusBadGateway, problemCoreReadFailed,
|
||||
public, fmt.Errorf("list pending reminders: %w", err))
|
||||
return
|
||||
}
|
||||
recent, err := core.ListReminders(r.Context(), 50)
|
||||
if err != nil {
|
||||
public := "reminders unavailable"
|
||||
if msg != "" {
|
||||
public = "reminder cancelled; refreshed list unavailable"
|
||||
}
|
||||
writeProblem(w, r, http.StatusBadGateway, problemCoreReadFailed,
|
||||
public, fmt.Errorf("list reminders: %w", err))
|
||||
return
|
||||
}
|
||||
reminders := remindersForPage(pending, recent)
|
||||
renderPage(w, remindersTmpl, struct {
|
||||
Msg string
|
||||
Reminders []reminderRow
|
||||
}{msg, reminderRows(reminders)})
|
||||
}
|
||||
|
||||
@@ -1,12 +1,17 @@
|
||||
{{template "shellTop" "reminders"}}
|
||||
<h1>Reminders</h1>
|
||||
{{if .Msg}}<div class="msg msg-ok">{{.Msg}}</div>{{end}}
|
||||
{{if .Reminders}}<div class=scroll><table>
|
||||
<tr><th>created</th><th>fires</th><th>status</th><th>what</th></tr>
|
||||
<tr><th>created</th><th>fires</th><th>status</th><th>what</th><th>action</th></tr>
|
||||
{{range .Reminders}}<tr>
|
||||
<td class=hint>{{.Created}}</td>
|
||||
<td>{{.Fires}}</td>
|
||||
<td>{{.Fires}}{{if .Schedule}}<div class=hint>{{.Schedule}}</div>{{end}}</td>
|
||||
<td><span class="badge {{.Status}}">{{.Status}}</span>{{if .Detail}}<div class=hint>{{.Detail}}</div>{{end}}</td>
|
||||
<td class=text-max>{{.Text}}</td>
|
||||
<td>{{if .CanCancel}}<form method="post" action="/reminders">
|
||||
<input type="hidden" name="id" value="{{.ID}}">
|
||||
<button type="submit" name="action" value="cancel">cancel</button>
|
||||
</form>{{end}}</td>
|
||||
</tr>{{end}}</table></div>
|
||||
{{else}}<div class=empty>
|
||||
<svg class=icon width="24" height="24"><use href="/ethos-icons.svg#i-calendar"/></svg>
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -12,6 +17,7 @@ import (
|
||||
func TestReminderRowsUnwrapAndLocalise(t *testing.T) {
|
||||
fire := time.Date(2026, 8, 4, 18, 30, 0, 0, time.UTC)
|
||||
rows := reminderRows([]ipc.Reminder{{
|
||||
ID: 17,
|
||||
CreatedTs: fire.Add(-time.Hour),
|
||||
FireTs: fire,
|
||||
Status: "pending",
|
||||
@@ -29,6 +35,9 @@ func TestReminderRowsUnwrapAndLocalise(t *testing.T) {
|
||||
if strings.Contains(rows[0].Text, "{") {
|
||||
t.Errorf("Text still carries JSON: %q", rows[0].Text)
|
||||
}
|
||||
if rows[0].ID != 17 || !rows[0].CanCancel {
|
||||
t.Errorf("pending reminder action binding = %+v, want id 17 cancellable", rows[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderRowsExposeBlockedDelivery(t *testing.T) {
|
||||
@@ -42,6 +51,33 @@ func TestReminderRowsExposeBlockedDelivery(t *testing.T) {
|
||||
if len(rows) != 1 || rows[0].Status != "blocked" || rows[0].Detail != "ntfy credentials rejected" {
|
||||
t.Fatalf("blocked reminder is not visible: %+v", rows)
|
||||
}
|
||||
if !rows[0].CanCancel {
|
||||
t.Fatal("a blocked but still-pending reminder must remain cancellable")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderRowsShowTheCurrentRecurringOccurrence(t *testing.T) {
|
||||
original := time.Date(2026, 8, 1, 9, 0, 0, 0, time.UTC)
|
||||
next := time.Date(2026, 8, 16, 9, 0, 0, 0, time.UTC)
|
||||
rows := reminderRows([]ipc.Reminder{{
|
||||
ID: 42, FireTs: original, NextFireTs: next, Cron: "0 9 * * *",
|
||||
Status: "pending", Payload: `{"text":"принять лекарство"}`,
|
||||
}})
|
||||
if len(rows) != 1 || rows[0].Fires != next.Local().Format("02 Jan 15:04") {
|
||||
t.Fatalf("recurring fire = %+v, want current occurrence %s", rows, next)
|
||||
}
|
||||
if rows[0].Schedule != "recurring · 0 9 * * *" || !rows[0].CanCancel {
|
||||
t.Fatalf("recurring identity/action = %+v", rows[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestReminderRowsOnlyPendingCanCancel(t *testing.T) {
|
||||
rows := reminderRows([]ipc.Reminder{{Status: "cancelled"}, {Status: "fired"}})
|
||||
for _, row := range rows {
|
||||
if row.CanCancel {
|
||||
t.Errorf("terminal row %+v exposed a cancel action", row)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A payload that is not the envelope is his own words, so it is shown as it is.
|
||||
@@ -57,3 +93,169 @@ func TestReminderTextKeepsPlainPayload(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type reminderCore struct {
|
||||
ipc.UnimplementedCoreAPI
|
||||
|
||||
reminders []ipc.Reminder
|
||||
pending []ipc.Reminder
|
||||
listErr error
|
||||
pendingErr error
|
||||
cancelErr error
|
||||
cancelID int64
|
||||
}
|
||||
|
||||
func (c *reminderCore) ListReminders(context.Context, int) ([]ipc.Reminder, error) {
|
||||
return c.reminders, c.listErr
|
||||
}
|
||||
|
||||
func (c *reminderCore) ListPendingReminders(context.Context, int) ([]ipc.Reminder, error) {
|
||||
return c.pending, c.pendingErr
|
||||
}
|
||||
|
||||
func (c *reminderCore) CancelReminder(_ context.Context, id int64) error {
|
||||
c.cancelID = id
|
||||
if c.cancelErr != nil {
|
||||
return c.cancelErr
|
||||
}
|
||||
for i := range c.reminders {
|
||||
if c.reminders[i].ID == id {
|
||||
c.reminders[i].Status = "cancelled"
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func reminderPost(action, id string) *http.Request {
|
||||
form := url.Values{"action": {action}, "id": {id}}
|
||||
req := httptest.NewRequest(http.MethodPost, "/reminders", strings.NewReader(form.Encode()))
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
return req
|
||||
}
|
||||
|
||||
func TestHandleRemindersCancel(t *testing.T) {
|
||||
core := &reminderCore{reminders: []ipc.Reminder{{
|
||||
ID: 23, Status: "pending", Payload: `{"text":"позвонить врачу"}`,
|
||||
}}}
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, reminderPost("cancel", "23"), core)
|
||||
|
||||
if rr.Code != http.StatusSeeOther || rr.Header().Get("Location") != "/reminders?cancelled=1" {
|
||||
t.Fatalf("status/location = %d %q, want 303 PRG", rr.Code, rr.Header().Get("Location"))
|
||||
}
|
||||
if core.cancelID != 23 {
|
||||
t.Fatalf("cancel id = %d, want 23", core.cancelID)
|
||||
}
|
||||
|
||||
rr = httptest.NewRecorder()
|
||||
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders?cancelled=1", nil), core)
|
||||
if rr.Code != http.StatusOK || !strings.Contains(rr.Body.String(), "reminder cancelled") || strings.Contains(rr.Body.String(), "value=\"23\"") {
|
||||
t.Fatalf("redirect outcome rendered incorrectly: status=%d body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleRemindersSuccessfulMutationDoesNotDependOnRefresh(t *testing.T) {
|
||||
core := &reminderCore{listErr: errors.New("offline"), pendingErr: errors.New("offline")}
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, reminderPost("cancel", "23"), core)
|
||||
if rr.Code != http.StatusSeeOther || core.cancelID != 23 {
|
||||
t.Fatalf("successful cancellation became refresh failure: status=%d id=%d body=%s", rr.Code, core.cancelID, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleRemindersIncludesPendingRowsOutsideRecentWindow(t *testing.T) {
|
||||
old := ipc.Reminder{ID: 1, Status: "pending", Payload: `{"text":"old but pending"}`}
|
||||
recent := make([]ipc.Reminder, 50)
|
||||
for i := range recent {
|
||||
recent[i] = ipc.Reminder{ID: int64(i + 2), Status: "fired", Payload: `{"text":"history"}`}
|
||||
}
|
||||
core := &reminderCore{pending: []ipc.Reminder{old}, reminders: recent}
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders", nil), core)
|
||||
if rr.Code != http.StatusOK || !strings.Contains(rr.Body.String(), "old but pending") ||
|
||||
!strings.Contains(rr.Body.String(), `value="1"`) {
|
||||
t.Fatalf("old pending reminder is not reachable: status=%d body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleRemindersRejectsMalformedPosts(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
action string
|
||||
id string
|
||||
}{
|
||||
{"unknown action", "delete", "23"},
|
||||
{"missing id", "cancel", ""},
|
||||
{"non-numeric id", "cancel", "twenty-three"},
|
||||
{"non-positive id", "cancel", "0"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
core := &reminderCore{}
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, reminderPost(tc.action, tc.id), core)
|
||||
if rr.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400; body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if core.cancelID != 0 {
|
||||
t.Fatalf("CancelReminder called with %d for malformed post", core.cancelID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleRemindersCancelErrors(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
err error
|
||||
status int
|
||||
public string
|
||||
}{
|
||||
{"missing", ipc.ErrReminderNotFound, http.StatusNotFound, "reminder not found"},
|
||||
{"delivery started", ipc.ErrReminderInFlight, http.StatusConflict, "reminder delivery has already started"},
|
||||
{"terminal", ipc.ErrReminderState, http.StatusConflict, "reminder is no longer pending"},
|
||||
{"transport", errors.New("socket closed"), http.StatusBadGateway, "reminder cancellation failed"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
core := &reminderCore{cancelErr: tc.err}
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, reminderPost("cancel", "23"), core)
|
||||
if rr.Code != tc.status || !strings.Contains(rr.Body.String(), tc.public) {
|
||||
t.Fatalf("status/body = %d %q, want %d containing %q", rr.Code, rr.Body.String(), tc.status, tc.public)
|
||||
}
|
||||
if got := rr.Header().Get("Content-Type"); !strings.HasPrefix(got, "application/problem+json") {
|
||||
t.Fatalf("content type = %q, want problem JSON", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleRemindersMethodAndListErrors(t *testing.T) {
|
||||
t.Run("method", func(t *testing.T) {
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, httptest.NewRequest(http.MethodDelete, "/reminders", nil), &reminderCore{})
|
||||
if rr.Code != http.StatusMethodNotAllowed {
|
||||
t.Fatalf("status = %d, want 405", rr.Code)
|
||||
}
|
||||
})
|
||||
t.Run("pending list", func(t *testing.T) {
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders", nil), &reminderCore{pendingErr: errors.New("offline")})
|
||||
if rr.Code != http.StatusBadGateway || !strings.Contains(rr.Body.String(), "reminders unavailable") {
|
||||
t.Fatalf("status/body = %d %q, want 502 problem", rr.Code, rr.Body.String())
|
||||
}
|
||||
})
|
||||
t.Run("recent list", func(t *testing.T) {
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders", nil), &reminderCore{listErr: errors.New("offline")})
|
||||
if rr.Code != http.StatusBadGateway || !strings.Contains(rr.Body.String(), "reminders unavailable") {
|
||||
t.Fatalf("status/body = %d %q, want 502 problem", rr.Code, rr.Body.String())
|
||||
}
|
||||
})
|
||||
t.Run("successful outcome remains explicit when redirected refresh fails", func(t *testing.T) {
|
||||
rr := httptest.NewRecorder()
|
||||
handleReminders(rr, httptest.NewRequest(http.MethodGet, "/reminders?cancelled=1", nil), &reminderCore{listErr: errors.New("offline")})
|
||||
if rr.Code != http.StatusBadGateway || !strings.Contains(rr.Body.String(), "reminder cancelled; refreshed list unavailable") {
|
||||
t.Fatalf("status/body = %d %q, want truthful refresh problem", rr.Code, rr.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
// merge-corpus merges the frozen holdout from corpus_v1.json with the
|
||||
// expanded v2 development pool, writing the result back to corpus_v1.json.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/merge-corpus/ -v1 internal/router/semantic/corpus_v1.json \
|
||||
// -v2 /tmp/corpus_v2.json -out internal/router/semantic/corpus_v1.json
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
func main() {
|
||||
v1Path := flag.String("v1", "internal/router/semantic/corpus_v1.json", "v1 corpus path")
|
||||
v2Path := flag.String("v2", "/tmp/corpus_v2.json", "v2 factory output path")
|
||||
outPath := flag.String("out", "internal/router/semantic/corpus_v1.json", "output path")
|
||||
flag.Parse()
|
||||
|
||||
// 1. Load v1
|
||||
v1Data, err := os.ReadFile(*v1Path)
|
||||
if err != nil {
|
||||
log.Fatalf("read v1: %v", err)
|
||||
}
|
||||
var v1Env semantic.CorpusEnvelope
|
||||
if err := json.Unmarshal(v1Data, &v1Env); err != nil {
|
||||
log.Fatalf("parse v1: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "v1: %d examples\n", len(v1Env.Examples))
|
||||
|
||||
// 2. Identify frozen holdout from v1
|
||||
frozen, _, _ := semantic.FrozenHoldoutSplit(v1Env.Examples)
|
||||
frozenTexts := make(map[string]bool)
|
||||
for _, e := range frozen {
|
||||
frozenTexts[strings.TrimSpace(e.Text)] = true
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "v1 frozen holdout: %d examples\n", len(frozen))
|
||||
|
||||
// 3. Load v2
|
||||
v2Data, err := os.ReadFile(*v2Path)
|
||||
if err != nil {
|
||||
log.Fatalf("read v2: %v", err)
|
||||
}
|
||||
var v2Env semantic.CorpusEnvelope
|
||||
if err := json.Unmarshal(v2Data, &v2Env); err != nil {
|
||||
log.Fatalf("parse v2: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "v2: %d examples\n", len(v2Env.Examples))
|
||||
|
||||
// 4. Merge: frozen from v1 + all from v2
|
||||
// Dedup by normalized text
|
||||
seen := make(map[string]bool)
|
||||
var merged []semantic.RouteExample
|
||||
|
||||
// Frozen holdout first
|
||||
for _, e := range frozen {
|
||||
norm := strings.TrimSpace(e.Text)
|
||||
if seen[norm] {
|
||||
fmt.Fprintf(os.Stderr, "SKIP v1 frozen dup: %q\n", norm)
|
||||
continue
|
||||
}
|
||||
seen[norm] = true
|
||||
merged = append(merged, e)
|
||||
}
|
||||
|
||||
// V2 examples
|
||||
skipped := 0
|
||||
for _, e := range v2Env.Examples {
|
||||
norm := strings.TrimSpace(e.Text)
|
||||
if seen[norm] {
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
// Check if this text conflicts with a frozen holdout entry
|
||||
if frozenTexts[norm] {
|
||||
// Text exists in frozen — skip v2 version to preserve frozen
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
seen[norm] = true
|
||||
merged = append(merged, e)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "merged: %d examples (skipped %d duplicates)\n", len(merged), skipped)
|
||||
|
||||
// 5. Fast-path metadata must match the real router. V2 rows are derived
|
||||
// by interactively-labeled construction; a stale manually-supplied value
|
||||
// is a build error, never silently rewritten. Frozen holdout rows are
|
||||
// preserved verbatim — report drift, do not touch them.
|
||||
v2Checked, frozenChecked, frozenDrift := 0, 0, 0
|
||||
for _, e := range merged {
|
||||
derived := semantic.DeriveFastPath(e.Text).Matched
|
||||
if frozenTexts[strings.TrimSpace(e.Text)] {
|
||||
frozenChecked++
|
||||
if derived != e.FastPathResolved {
|
||||
frozenDrift++
|
||||
fmt.Fprintf(os.Stderr, "frozen drift: %q stored=%v derived=%v\n", e.Text, e.FastPathResolved, derived)
|
||||
}
|
||||
continue
|
||||
}
|
||||
v2Checked++
|
||||
if derived != e.FastPathResolved {
|
||||
log.Fatalf("v2 row (source=%s source_id=%s) fast_path_resolved=%v but router derives %v: %q",
|
||||
e.Source, e.SourceID, e.FastPathResolved, derived, e.Text)
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "fast-path check: v2 rows %d OK, frozen rows %d (drift %d, reported only)\n",
|
||||
v2Checked, frozenChecked, frozenDrift)
|
||||
|
||||
// 6. Validate
|
||||
if err := semantic.ValidateCorpus(merged); err != nil {
|
||||
log.Fatalf("validation failed: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "validation: OK\n")
|
||||
|
||||
// 7. Compute dataset hash
|
||||
texts := make([]string, len(merged))
|
||||
for i, e := range merged {
|
||||
texts[i] = e.Text
|
||||
}
|
||||
sort.Strings(texts)
|
||||
h := sha256.Sum256([]byte(strings.Join(texts, "\n")))
|
||||
datasetHash := hex.EncodeToString(h[:16])
|
||||
|
||||
// 8. Stats
|
||||
routeCounts := make(map[semantic.SemanticRoute]int)
|
||||
fpCount, resCount := 0, 0
|
||||
for _, e := range merged {
|
||||
routeCounts[e.Route]++
|
||||
if e.FastPathResolved {
|
||||
fpCount++
|
||||
} else {
|
||||
resCount++
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "\nRoute distribution:\n")
|
||||
for _, r := range semantic.AllRoutes {
|
||||
fmt.Fprintf(os.Stderr, " %-15s %d\n", r, routeCounts[r])
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "fast-path: %d residual: %d\n", fpCount, resCount)
|
||||
|
||||
// 9. Write merged corpus
|
||||
outEnv := semantic.CorpusEnvelope{
|
||||
SchemaVersion: 1,
|
||||
Name: "semantic_coarse_route_v1",
|
||||
Notes: []string{
|
||||
"Coarse semantic route corpus for the learned-router experiment.",
|
||||
"Frozen holdout preserved from v1. Development pool expanded by corpus-factory v2.",
|
||||
"Every row carries provenance (source + source_id) and a split_group.",
|
||||
"Labels come from explicit mapping rules, not model output.",
|
||||
},
|
||||
Reproducibility: &semantic.ReproducibilityMeta{
|
||||
SourceFixtureHash: "v1-frozen + corpus-factory-v2",
|
||||
ContrastGeneratorVersion: "v2-direct-generation",
|
||||
SplitAlgorithm: "grouped-cv-v1",
|
||||
DatasetHash: datasetHash,
|
||||
},
|
||||
Examples: merged,
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(outEnv, "", " ")
|
||||
if err != nil {
|
||||
log.Fatalf("marshal: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(*outPath, data, 0644); err != nil {
|
||||
log.Fatalf("write %s: %v", *outPath, err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "\nOutput: %s (%d bytes)\n", *outPath, len(data))
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,856 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Semantic Router Linear Head Experiment
|
||||
======================================
|
||||
|
||||
Evaluates whether the six-way residual routing boundary is linearly learnable
|
||||
from Maven's existing e5-small representation.
|
||||
|
||||
Architecture under test:
|
||||
NormalizedInput.MatchText
|
||||
→ existing multilingual-e5-small embedder
|
||||
→ tiny 6-class linear head (softmax logistic regression)
|
||||
→ SemanticRouteDecision
|
||||
|
||||
No new encoder. No fine-tuning. Linear separability floor only.
|
||||
"""
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
import sys
|
||||
import warnings
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from sklearn.exceptions import ConvergenceWarning
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.metrics import (
|
||||
accuracy_score,
|
||||
f1_score,
|
||||
precision_recall_fscore_support,
|
||||
confusion_matrix,
|
||||
brier_score_loss,
|
||||
log_loss,
|
||||
)
|
||||
from sklearn.preprocessing import LabelEncoder
|
||||
|
||||
warnings.filterwarnings("ignore", category=ConvergenceWarning)
|
||||
|
||||
# ─── Constants ──────────────────────────────────────────────────────────────
|
||||
|
||||
ROUTES = ["action", "conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
ROUTE_IDX = {r: i for i, r in enumerate(ROUTES)}
|
||||
|
||||
# Regularization grid
|
||||
C_VALUES = [0.01, 0.1, 1.0, 10.0, 100.0]
|
||||
|
||||
# Abstention thresholds
|
||||
THRESHOLDS = [0.40, 0.50, 0.60, 0.70, 0.80, 0.90]
|
||||
|
||||
# Contrast families
|
||||
CONTRAST_FAMILIES = [
|
||||
"negation", "question", "reported_speech", "quotation",
|
||||
"hypothetical", "capability_question",
|
||||
]
|
||||
|
||||
# ─── Data Loading ───────────────────────────────────────────────────────────
|
||||
|
||||
def load_embeddings(path):
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
meta = data["meta"]
|
||||
examples = data["examples"]
|
||||
return meta, examples
|
||||
|
||||
|
||||
def filter_dev_pool(examples):
|
||||
"""Return only development pool examples (not frozen holdout)."""
|
||||
return [e for e in examples if e["dev_pool"]]
|
||||
|
||||
|
||||
def filter_residual(examples):
|
||||
"""Return only router-residual examples (fast_path_resolved == false)."""
|
||||
return [e for e in examples if not e["fast_path_resolved"]]
|
||||
|
||||
|
||||
def extract_Xy(examples):
|
||||
"""Extract feature matrix and label vector from cached examples."""
|
||||
X = np.array([e["embedding"] for e in examples])
|
||||
y = np.array([e["route"] for e in examples])
|
||||
return X, y
|
||||
|
||||
|
||||
def get_fold_groups(examples):
|
||||
"""Return fold assignment array matching the Go-generated CV folds."""
|
||||
return np.array([e["cv_fold"] for e in examples])
|
||||
|
||||
|
||||
# ─── Grouped CV ─────────────────────────────────────────────────────────────
|
||||
|
||||
def grouped_cv_experiment(X, y, fold_ids, C_values, examples_meta):
|
||||
"""
|
||||
Run grouped cross-validation with logistic regression.
|
||||
Returns best C, per-fold results, and out-of-fold predictions.
|
||||
"""
|
||||
unique_folds = sorted(set(fold_ids))
|
||||
n_classes = len(ROUTES)
|
||||
|
||||
results_by_C = {}
|
||||
for C in C_values:
|
||||
fold_metrics = []
|
||||
oof_rows = []
|
||||
|
||||
for test_fold in unique_folds:
|
||||
train_mask = fold_ids != test_fold
|
||||
test_mask = fold_ids == test_fold
|
||||
|
||||
X_train, y_train = X[train_mask], y[train_mask]
|
||||
X_test, y_test = X[test_mask], y[test_mask]
|
||||
|
||||
model = LogisticRegression(
|
||||
C=C, max_iter=2000, solver="lbfgs",
|
||||
random_state=42,
|
||||
)
|
||||
model.fit(X_train, y_train)
|
||||
|
||||
y_pred = model.predict(X_test)
|
||||
y_proba = model.predict_proba(X_test)
|
||||
classes = model.classes_
|
||||
|
||||
acc = accuracy_score(y_test, y_pred)
|
||||
macro_f1 = f1_score(y_test, y_pred, average="macro", zero_division=0)
|
||||
|
||||
prec, rec, f1, sup = precision_recall_fscore_support(
|
||||
y_test, y_pred, labels=ROUTES, zero_division=0
|
||||
)
|
||||
|
||||
false_action = 0
|
||||
for true, pred in zip(y_test, y_pred):
|
||||
if true != "action" and pred == "action":
|
||||
false_action += 1
|
||||
|
||||
fold_metrics.append({
|
||||
"fold": test_fold,
|
||||
"train_size": len(X_train),
|
||||
"test_size": len(X_test),
|
||||
"accuracy": acc,
|
||||
"macro_f1": macro_f1,
|
||||
"per_class": {
|
||||
route: {"precision": prec[i], "recall": rec[i], "f1": f1[i]}
|
||||
for i, route in enumerate(ROUTES)
|
||||
},
|
||||
"false_action": false_action,
|
||||
})
|
||||
|
||||
for i, (true, pred) in enumerate(zip(y_test, y_pred)):
|
||||
test_indices = np.where(test_mask)[0]
|
||||
meta = examples_meta[test_indices[i]]
|
||||
proba_dict = {cls: float(y_proba[i][j]) for j, cls in enumerate(classes)}
|
||||
max_proba = max(proba_dict.values()) if proba_dict else 0.0
|
||||
oof_rows.append({
|
||||
"source_id": meta["source_id"],
|
||||
"fold": test_fold,
|
||||
"true": true,
|
||||
"predicted": pred,
|
||||
"correct": true == pred,
|
||||
"max_proba": max_proba,
|
||||
"proba": proba_dict,
|
||||
"fast_path_resolved": meta.get("fast_path_resolved", False),
|
||||
"tags": meta.get("tags", []),
|
||||
"text": meta["text"],
|
||||
})
|
||||
|
||||
# Aggregate across folds
|
||||
mean_acc = np.mean([m["accuracy"] for m in fold_metrics])
|
||||
mean_f1 = np.mean([m["macro_f1"] for m in fold_metrics])
|
||||
std_acc = np.std([m["accuracy"] for m in fold_metrics])
|
||||
std_f1 = np.std([m["macro_f1"] for m in fold_metrics])
|
||||
total_fa = sum(m["false_action"] for m in fold_metrics)
|
||||
|
||||
results_by_C[C] = {
|
||||
"mean_accuracy": mean_acc,
|
||||
"std_accuracy": std_acc,
|
||||
"mean_macro_f1": mean_f1,
|
||||
"std_macro_f1": std_f1,
|
||||
"total_false_action": total_fa,
|
||||
"fold_metrics": fold_metrics,
|
||||
"oof_predictions": oof_rows,
|
||||
}
|
||||
|
||||
# Select best C by mean macro F1
|
||||
best_C = max(results_by_C, key=lambda c: results_by_C[c]["mean_macro_f1"])
|
||||
return best_C, results_by_C
|
||||
|
||||
|
||||
# ─── Metrics Computation ────────────────────────────────────────────────────
|
||||
|
||||
def compute_full_metrics(y_true, y_pred, y_proba=None):
|
||||
"""Compute all required metrics from out-of-fold predictions."""
|
||||
acc = accuracy_score(y_true, y_pred)
|
||||
macro_f1 = f1_score(y_true, y_pred, average="macro", zero_division=0)
|
||||
|
||||
# Per-class P/R/F1
|
||||
prec, rec, f1, sup = precision_recall_fscore_support(
|
||||
y_true, y_pred, labels=ROUTES, zero_division=0
|
||||
)
|
||||
|
||||
# Confusion matrix
|
||||
cm = confusion_matrix(y_true, y_pred, labels=ROUTES)
|
||||
|
||||
# False action
|
||||
false_action = 0
|
||||
false_action_predicted = []
|
||||
for true, pred in zip(y_true, y_pred):
|
||||
if true != "action" and pred == "action":
|
||||
false_action += 1
|
||||
|
||||
# Action precision/recall
|
||||
action_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p == "action")
|
||||
action_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
action_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p != "action")
|
||||
action_precision = action_tp / max(action_tp + action_fp, 1)
|
||||
action_recall = action_tp / max(action_tp + action_fn, 1)
|
||||
|
||||
# Uncertain precision/recall
|
||||
unc_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "uncertain" and p == "uncertain")
|
||||
unc_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "uncertain" and p == "uncertain")
|
||||
unc_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "uncertain" and p != "uncertain")
|
||||
unc_precision = unc_tp / max(unc_tp + unc_fp, 1)
|
||||
unc_recall = unc_tp / max(unc_tp + unc_fn, 1)
|
||||
|
||||
metrics = {
|
||||
"accuracy": acc,
|
||||
"macro_f1": macro_f1,
|
||||
"false_action_count": false_action,
|
||||
"false_action_rate": false_action / max(len(y_true), 1),
|
||||
"action_precision": action_precision,
|
||||
"action_recall": action_recall,
|
||||
"uncertain_precision": unc_precision,
|
||||
"uncertain_recall": unc_recall,
|
||||
"per_class": {},
|
||||
"confusion_matrix": cm.tolist(),
|
||||
}
|
||||
|
||||
for i, route in enumerate(ROUTES):
|
||||
metrics["per_class"][route] = {
|
||||
"precision": float(prec[i]),
|
||||
"recall": float(rec[i]),
|
||||
"f1": float(f1[i]),
|
||||
"support": int(sup[i]),
|
||||
}
|
||||
|
||||
return metrics
|
||||
|
||||
|
||||
def compute_calibration(y_true, y_pred, oof_rows):
|
||||
"""Compute ECE, Brier score, and per-threshold abstention curves."""
|
||||
# ECE (Expected Calibration Error) with 10 bins
|
||||
n_bins = 10
|
||||
bin_boundaries = np.linspace(0, 1, n_bins + 1)
|
||||
ece = 0.0
|
||||
total = len(y_true)
|
||||
|
||||
confidences = np.array([r["max_proba"] for r in oof_rows])
|
||||
correct = np.array([r["correct"] for r in oof_rows])
|
||||
|
||||
for i in range(n_bins):
|
||||
lo, hi = bin_boundaries[i], bin_boundaries[i + 1]
|
||||
mask = (confidences > lo) & (confidences <= hi)
|
||||
if mask.sum() == 0:
|
||||
continue
|
||||
bin_acc = correct[mask].mean()
|
||||
bin_conf = confidences[mask].mean()
|
||||
ece += mask.sum() / total * abs(bin_acc - bin_conf)
|
||||
|
||||
# Brier score (multiclass one-hot encoding)
|
||||
y_true_idx = np.array([ROUTES.index(t) for t in y_true])
|
||||
n_samples = len(y_true)
|
||||
n_classes = len(ROUTES)
|
||||
y_true_oh = np.zeros((n_samples, n_classes))
|
||||
y_true_oh[np.arange(n_samples), y_true_idx] = 1.0
|
||||
|
||||
# Build probability matrix
|
||||
y_proba_matrix = np.zeros((n_samples, n_classes))
|
||||
for i, r in enumerate(oof_rows):
|
||||
for j, route in enumerate(ROUTES):
|
||||
y_proba_matrix[i, j] = r["proba"].get(route, 0.0)
|
||||
|
||||
brier = np.mean(np.sum((y_proba_matrix - y_true_oh) ** 2, axis=1))
|
||||
ll = log_loss(y_true, y_proba_matrix, labels=ROUTES)
|
||||
|
||||
# Abstention curves
|
||||
abstention_curves = []
|
||||
for thr in THRESHOLDS:
|
||||
accepted_mask = confidences >= thr
|
||||
n_accepted = accepted_mask.sum()
|
||||
coverage = n_accepted / max(total, 1)
|
||||
|
||||
if n_accepted > 0:
|
||||
acc_accepted = accuracy_score(y_true[accepted_mask], y_pred[accepted_mask])
|
||||
f1_accepted = f1_score(
|
||||
y_true[accepted_mask], y_pred[accepted_mask],
|
||||
average="macro", zero_division=0,
|
||||
)
|
||||
fa_count = sum(
|
||||
1 for t, p in zip(y_true[accepted_mask], y_pred[accepted_mask])
|
||||
if t != "action" and p == "action"
|
||||
)
|
||||
else:
|
||||
acc_accepted = 0.0
|
||||
f1_accepted = 0.0
|
||||
fa_count = 0
|
||||
|
||||
abstention_curves.append({
|
||||
"threshold": thr,
|
||||
"n_accepted": int(n_accepted),
|
||||
"coverage": coverage,
|
||||
"accuracy": acc_accepted,
|
||||
"macro_f1": f1_accepted,
|
||||
"false_action_count": fa_count,
|
||||
})
|
||||
|
||||
return {
|
||||
"ece": float(ece),
|
||||
"brier": float(brier),
|
||||
"log_loss": float(ll),
|
||||
"abstention_curves": abstention_curves,
|
||||
}
|
||||
|
||||
|
||||
def compute_action_threshold(oof_rows):
|
||||
"""Evaluate action-specific safety gate thresholds."""
|
||||
action_curves = []
|
||||
for thr in THRESHOLDS:
|
||||
action_pred = []
|
||||
for r in oof_rows:
|
||||
p = r["predicted"]
|
||||
proba = r["proba"].get("action", 0.0)
|
||||
if p == "action" and proba < thr:
|
||||
# Demote action prediction
|
||||
# Find next best route that isn't action
|
||||
sorted_routes = sorted(r["proba"].items(), key=lambda x: -x[1])
|
||||
for route, _ in sorted_routes:
|
||||
if route != "action":
|
||||
p = route
|
||||
break
|
||||
action_pred.append(p)
|
||||
|
||||
y_true = np.array([r["true"] for r in oof_rows])
|
||||
y_pred = np.array(action_pred)
|
||||
|
||||
action_tp = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p == "action")
|
||||
action_fp = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
action_fn = sum(1 for t, p in zip(y_true, y_pred) if t == "action" and p != "action")
|
||||
false_action = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
|
||||
action_curves.append({
|
||||
"threshold": thr,
|
||||
"action_precision": action_tp / max(action_tp + action_fp, 1),
|
||||
"action_recall": action_tp / max(action_tp + action_fn, 1),
|
||||
"false_action_count": false_action,
|
||||
})
|
||||
|
||||
return action_curves
|
||||
|
||||
|
||||
def compute_disagreement(y_true, y_pred, legacy_pred, oof_rows):
|
||||
"""Analyze disagreements between legacy and learned router."""
|
||||
results = {
|
||||
"legacy_wrong_learned_right": [],
|
||||
"legacy_right_learned_wrong": [],
|
||||
"both_wrong_differently": [],
|
||||
}
|
||||
|
||||
legacy_false_actions = []
|
||||
learned_false_actions = []
|
||||
shared_false_actions = []
|
||||
|
||||
for i, r in enumerate(oof_rows):
|
||||
sid = r["source_id"]
|
||||
text = r["text"]
|
||||
true = y_true[i]
|
||||
learned = y_pred[i]
|
||||
legacy = legacy_pred[i]
|
||||
|
||||
if legacy != true and learned == true:
|
||||
results["legacy_wrong_learned_right"].append({
|
||||
"source_id": sid, "text": text,
|
||||
"true": true, "legacy": legacy, "learned": learned,
|
||||
})
|
||||
elif legacy == true and learned != true:
|
||||
results["legacy_right_learned_wrong"].append({
|
||||
"source_id": sid, "text": text,
|
||||
"true": true, "legacy": legacy, "learned": learned,
|
||||
})
|
||||
elif legacy != true and learned != true and legacy != learned:
|
||||
results["both_wrong_differently"].append({
|
||||
"source_id": sid, "text": text,
|
||||
"true": true, "legacy": legacy, "learned": learned,
|
||||
})
|
||||
|
||||
# False action tracking
|
||||
if true != "action" and legacy == "action":
|
||||
legacy_false_actions.append(sid)
|
||||
if true != "action" and learned == "action":
|
||||
learned_false_actions.append(sid)
|
||||
if true != "action" and legacy == "action" and learned == "action":
|
||||
shared_false_actions.append(sid)
|
||||
|
||||
# Repaired false actions
|
||||
repaired = [sid for sid in legacy_false_actions if sid not in learned_false_actions]
|
||||
new_errors = [sid for sid in learned_false_actions if sid not in legacy_false_actions]
|
||||
shared = shared_false_actions
|
||||
|
||||
return {
|
||||
"details": results,
|
||||
"legacy_false_actions": legacy_false_actions,
|
||||
"learned_false_actions": learned_false_actions,
|
||||
"repaired": repaired,
|
||||
"new_errors": new_errors,
|
||||
"shared": shared,
|
||||
}
|
||||
|
||||
|
||||
def compute_contrast_family(oof_rows):
|
||||
"""Analyze performance per contrast family."""
|
||||
family_results = {}
|
||||
for family in CONTRAST_FAMILIES:
|
||||
members = [r for r in oof_rows if family in r.get("tags", [])]
|
||||
if not members:
|
||||
continue
|
||||
y_true = [r["true"] for r in members]
|
||||
y_pred = [r["predicted"] for r in members]
|
||||
correct = sum(1 for t, p in zip(y_true, y_pred) if t == p)
|
||||
false_act = sum(1 for t, p in zip(y_true, y_pred) if t != "action" and p == "action")
|
||||
family_results[family] = {
|
||||
"count": len(members),
|
||||
"correct": correct,
|
||||
"accuracy": correct / len(members),
|
||||
"false_action": false_act,
|
||||
}
|
||||
return family_results
|
||||
|
||||
|
||||
def compute_legacy_baseline(examples):
|
||||
"""
|
||||
Compute legacy baseline by mapping each example through the known
|
||||
fast-path and classifier behavior. Since we don't have the actual
|
||||
router running, we use the corpus metadata:
|
||||
- fast_path_resolved examples are correct (grammar handles them)
|
||||
- We simulate the legacy baseline from the test output numbers
|
||||
"""
|
||||
# The actual legacy baseline was measured in TestLegacyBaseline:
|
||||
# overall accuracy: 52.2%, residual: 40.8%, false-action rate: 19.9%
|
||||
# We need per-example predictions. We'll approximate from the corpus structure.
|
||||
#
|
||||
# For fast_path_resolved examples, the legacy router is correct (stage-0 grammar).
|
||||
# For residual examples, we need to simulate the hash-embedder classifier.
|
||||
# Since we don't have the hash embedder running, we use the known aggregate.
|
||||
#
|
||||
# This is a limitation: the legacy baseline numbers come from the Go test,
|
||||
# and we only have the aggregate. We'll use the aggregate for comparison.
|
||||
pass
|
||||
|
||||
|
||||
# ─── Report Generation ──────────────────────────────────────────────────────
|
||||
|
||||
def fmt_pct(v, decimals=1):
|
||||
return f"{100 * v:.{decimals}f}%"
|
||||
|
||||
|
||||
def fmt_float(v, decimals=3):
|
||||
return f"{v:.{decimals}f}"
|
||||
|
||||
|
||||
def generate_report(meta, all_results, residual_results, oof_all, oof_residual):
|
||||
"""Generate the full experiment report."""
|
||||
lines = []
|
||||
lines.append("# Semantic Router Linear Head Experiment — Report")
|
||||
lines.append("")
|
||||
lines.append("## 1. Exact e5 representation used")
|
||||
lines.append("")
|
||||
lines.append(f"- **Model**: {meta['embedder_id']}")
|
||||
lines.append(f"- **Checkpoint**: {meta['model_path']}")
|
||||
lines.append(f"- **Tokenizer**: {meta['tokenizer_path']}")
|
||||
lines.append(f"- **Dimension**: {meta['dimension']}")
|
||||
lines.append(f"- **Pooling**: {meta['pooling']}")
|
||||
lines.append(f"- **Normalization**: {meta['normalization']}")
|
||||
lines.append(f"- **Input template**: {meta['input_template']}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("## 2. Development/residual row counts")
|
||||
lines.append("")
|
||||
lines.append(f"- Total corpus: {meta['total_examples']}")
|
||||
lines.append(f"- Frozen holdout: {meta['frozen_count']}")
|
||||
lines.append(f"- Development pool: {meta['dev_count']}")
|
||||
lines.append(f"- Fast-path resolved: {meta['fast_path_count']}")
|
||||
lines.append(f"- Router-residual: {meta['residual_count']}")
|
||||
lines.append("")
|
||||
lines.append("Route distribution (full corpus):")
|
||||
for route, count in sorted(meta["route_counts"].items()):
|
||||
lines.append(f" - {route}: {count}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("## 3. Grouped fold composition")
|
||||
lines.append("")
|
||||
lines.append(f"Folds: {meta['cv_folds']}")
|
||||
for fold_id, stats in sorted(meta["fold_composition"].items()):
|
||||
lines.append(f" - Fold {fold_id}: eval={stats['eval_count']} train={stats['train_count']} routes={stats['eval_routes']}")
|
||||
lines.append("")
|
||||
|
||||
# Regularization selection
|
||||
lines.append("## 4. Selected regularization")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment A: All development examples")
|
||||
best_C_all = all_results["best_C"]
|
||||
lines.append(f"- Best C: {best_C_all}")
|
||||
lines.append(f"- Mean accuracy: {fmt_pct(all_results['results_by_C'][best_C_all]['mean_accuracy'])} ± {fmt_pct(all_results['results_by_C'][best_C_all]['std_accuracy'])}")
|
||||
lines.append(f"- Mean macro F1: {fmt_float(all_results['results_by_C'][best_C_all]['mean_macro_f1'])} ± {fmt_float(all_results['results_by_C'][best_C_all]['std_macro_f1'])}")
|
||||
lines.append(f"- Total false actions (CV): {all_results['results_by_C'][best_C_all]['total_false_action']}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B: Router-residual only")
|
||||
best_C_res = residual_results["best_C"]
|
||||
lines.append(f"- Best C: {best_C_res}")
|
||||
lines.append(f"- Mean accuracy: {fmt_pct(residual_results['results_by_C'][best_C_res]['mean_accuracy'])} ± {fmt_pct(residual_results['results_by_C'][best_C_res]['std_accuracy'])}")
|
||||
lines.append(f"- Mean macro F1: {fmt_float(residual_results['results_by_C'][best_C_res]['mean_macro_f1'])} ± {fmt_float(residual_results['results_by_C'][best_C_res]['std_macro_f1'])}")
|
||||
lines.append(f"- Total false actions (CV): {residual_results['results_by_C'][best_C_res]['total_false_action']}")
|
||||
lines.append("")
|
||||
|
||||
# Stability across folds
|
||||
lines.append("### Stability across folds")
|
||||
lines.append("")
|
||||
for C_val in C_VALUES:
|
||||
r = all_results["results_by_C"][C_val]
|
||||
fold_accs = [m["accuracy"] for m in r["fold_metrics"]]
|
||||
fold_f1s = [m["macro_f1"] for m in r["fold_metrics"]]
|
||||
lines.append(f" C={C_val:<6} acc={fmt_pct(r['mean_accuracy'])}±{fmt_pct(r['std_accuracy'])} f1={fmt_float(r['mean_macro_f1'])}±{fmt_float(r['std_macro_f1'])} folds_acc={[fmt_pct(a) for a in fold_accs]}")
|
||||
lines.append("")
|
||||
|
||||
# Experiment A metrics
|
||||
lines.append("## 5. All-example CV metrics")
|
||||
lines.append("")
|
||||
metrics_all = all_results["full_metrics"]
|
||||
lines.append(f"- Accuracy: {fmt_pct(metrics_all['accuracy'])}")
|
||||
lines.append(f"- Macro F1: {fmt_float(metrics_all['macro_f1'])}")
|
||||
lines.append(f"- False-action count: {metrics_all['false_action_count']}")
|
||||
lines.append(f"- False-action rate: {fmt_pct(metrics_all['false_action_rate'])}")
|
||||
lines.append(f"- Action precision: {fmt_float(metrics_all['action_precision'])}")
|
||||
lines.append(f"- Action recall: {fmt_float(metrics_all['action_recall'])}")
|
||||
lines.append(f"- Uncertain precision: {fmt_float(metrics_all['uncertain_precision'])}")
|
||||
lines.append(f"- Uncertain recall: {fmt_float(metrics_all['uncertain_recall'])}")
|
||||
lines.append("")
|
||||
lines.append("Per-class metrics:")
|
||||
for route in ROUTES:
|
||||
pc = metrics_all["per_class"][route]
|
||||
lines.append(f" {route:<15} P={fmt_float(pc['precision'])} R={fmt_float(pc['recall'])} F1={fmt_float(pc['f1'])} (n={pc['support']})")
|
||||
lines.append("")
|
||||
lines.append("Confusion matrix (rows=expected, cols=predicted):")
|
||||
header = f"{'':>15}" + "".join(f"{r:>15}" for r in ROUTES)
|
||||
lines.append(header)
|
||||
for i, route in enumerate(ROUTES):
|
||||
row = f"{route:>15}" + "".join(f"{metrics_all['confusion_matrix'][i][j]:>15}" for j in range(len(ROUTES)))
|
||||
lines.append(row)
|
||||
lines.append("")
|
||||
|
||||
# Experiment B metrics
|
||||
lines.append("## 6. Residual-only CV metrics")
|
||||
lines.append("")
|
||||
metrics_res = residual_results["full_metrics"]
|
||||
lines.append(f"- Accuracy: {fmt_pct(metrics_res['accuracy'])}")
|
||||
lines.append(f"- Macro F1: {fmt_float(metrics_res['macro_f1'])}")
|
||||
lines.append(f"- False-action count: {metrics_res['false_action_count']}")
|
||||
lines.append(f"- False-action rate: {fmt_pct(metrics_res['false_action_rate'])}")
|
||||
lines.append(f"- Action precision: {fmt_float(metrics_res['action_precision'])}")
|
||||
lines.append(f"- Action recall: {fmt_float(metrics_res['action_recall'])}")
|
||||
lines.append(f"- Uncertain precision: {fmt_float(metrics_res['uncertain_precision'])}")
|
||||
lines.append(f"- Uncertain recall: {fmt_float(metrics_res['uncertain_recall'])}")
|
||||
lines.append("")
|
||||
lines.append("Per-class metrics:")
|
||||
for route in ROUTES:
|
||||
pc = metrics_res["per_class"][route]
|
||||
lines.append(f" {route:<15} P={fmt_float(pc['precision'])} R={fmt_float(pc['recall'])} F1={fmt_float(pc['f1'])} (n={pc['support']})")
|
||||
lines.append("")
|
||||
lines.append("Confusion matrix (rows=expected, cols=predicted):")
|
||||
header = f"{'':>15}" + "".join(f"{r:>15}" for r in ROUTES)
|
||||
lines.append(header)
|
||||
for i, route in enumerate(ROUTES):
|
||||
row = f"{route:>15}" + "".join(f"{metrics_res['confusion_matrix'][i][j]:>15}" for j in range(len(ROUTES)))
|
||||
lines.append(row)
|
||||
lines.append("")
|
||||
|
||||
# Legacy comparison
|
||||
lines.append("## 7. Legacy-vs-linear comparison")
|
||||
lines.append("")
|
||||
lines.append("### All examples")
|
||||
lines.append(f"{'metric':<25} {'legacy':>10} {'linear e5':>10} {'delta':>10}")
|
||||
lines.append("-" * 55)
|
||||
# Legacy baseline from test: 52.2% overall, 40.8% residual, 19.9% false-action
|
||||
# These are approximate since we don't have per-example legacy predictions
|
||||
legacy_acc = 0.522
|
||||
legacy_fa_rate = 0.199
|
||||
legacy_macro_f1 = 0.0 # unknown precisely
|
||||
lines.append(f"{'accuracy':<25} {fmt_pct(legacy_acc):>10} {fmt_pct(metrics_all['accuracy']):>10} {fmt_pct(metrics_all['accuracy'] - legacy_acc):>10}")
|
||||
lines.append(f"{'macro F1':<25} {'—':>10} {fmt_float(metrics_all['macro_f1']):>10} {'—':>10}")
|
||||
lines.append(f"{'action precision':<25} {'—':>10} {fmt_float(metrics_all['action_precision']):>10} {'—':>10}")
|
||||
lines.append(f"{'false-action rate':<25} {fmt_pct(legacy_fa_rate):>10} {fmt_pct(metrics_all['false_action_rate']):>10} {fmt_pct(metrics_all['false_action_rate'] - legacy_fa_rate):>10}")
|
||||
lines.append(f"{'uncertain F1':<25} {fmt_float(0.0):>10} {fmt_float(metrics_all['per_class']['uncertain']['f1']):>10} {fmt_float(metrics_all['per_class']['uncertain']['f1']):>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Router-residual only")
|
||||
lines.append(f"{'metric':<25} {'legacy':>10} {'linear e5':>10} {'delta':>10}")
|
||||
lines.append("-" * 55)
|
||||
legacy_res_acc = 0.408
|
||||
lines.append(f"{'accuracy':<25} {fmt_pct(legacy_res_acc):>10} {fmt_pct(metrics_res['accuracy']):>10} {fmt_pct(metrics_res['accuracy'] - legacy_res_acc):>10}")
|
||||
lines.append(f"{'macro F1':<25} {'—':>10} {fmt_float(metrics_res['macro_f1']):>10} {'—':>10}")
|
||||
lines.append(f"{'false-action rate':<25} {'—':>10} {fmt_pct(metrics_res['false_action_rate']):>10} {'—':>10}")
|
||||
lines.append("")
|
||||
|
||||
# Fold variance
|
||||
lines.append("## 8. Fold variance")
|
||||
lines.append("")
|
||||
lines.append("All-example CV:")
|
||||
for m in all_results["results_by_C"][best_C_all]["fold_metrics"]:
|
||||
lines.append(f" Fold {m['fold']}: acc={fmt_pct(m['accuracy'])} f1={fmt_float(m['macro_f1'])} false_action={m['false_action']}")
|
||||
lines.append("")
|
||||
lines.append("Residual-only CV:")
|
||||
for m in residual_results["results_by_C"][best_C_res]["fold_metrics"]:
|
||||
lines.append(f" Fold {m['fold']}: acc={fmt_pct(m['accuracy'])} f1={fmt_float(m['macro_f1'])} false_action={m['false_action']}")
|
||||
lines.append("")
|
||||
|
||||
# Disagreement analysis
|
||||
lines.append("## 9. False-action repair/new-error analysis")
|
||||
lines.append("")
|
||||
# The disagreement analysis requires legacy per-example predictions.
|
||||
# Since we don't have those, we report what we can from the out-of-fold data.
|
||||
lines.append("Note: Legacy per-example predictions were not available for this experiment.")
|
||||
lines.append("The legacy baseline was measured in aggregate in the Go test suite.")
|
||||
lines.append("")
|
||||
lines.append("Learned router false-action cases (out-of-fold):")
|
||||
for r in oof_all:
|
||||
if r["true"] != "action" and r["predicted"] == "action":
|
||||
lines.append(f" {r['source_id']}: '{r['text']}' (true={r['true']}, proba(action)={r['proba'].get('action', 0):.3f})")
|
||||
lines.append("")
|
||||
|
||||
# Contrast family
|
||||
lines.append("## 10. Contrast-family results")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A (all dev)")
|
||||
contrast_all = all_results["contrast_family"]
|
||||
lines.append(f"{'family':<25} {'count':>6} {'correct':>8} {'accuracy':>10} {'false_act':>10}")
|
||||
lines.append("-" * 60)
|
||||
for family in CONTRAST_FAMILIES:
|
||||
if family in contrast_all:
|
||||
c = contrast_all[family]
|
||||
lines.append(f"{family:<25} {c['count']:>6} {c['correct']:>8} {fmt_pct(c['accuracy']):>10} {c['false_action']:>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B (residual only)")
|
||||
contrast_res = residual_results["contrast_family"]
|
||||
lines.append(f"{'family':<25} {'count':>6} {'correct':>8} {'accuracy':>10} {'false_act':>10}")
|
||||
lines.append("-" * 60)
|
||||
for family in CONTRAST_FAMILIES:
|
||||
if family in contrast_res:
|
||||
c = contrast_res[family]
|
||||
lines.append(f"{family:<25} {c['count']:>6} {c['correct']:>8} {fmt_pct(c['accuracy']):>10} {c['false_action']:>10}")
|
||||
lines.append("")
|
||||
|
||||
# Calibration
|
||||
lines.append("## 11. Calibration metrics")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A")
|
||||
cal_all = all_results["calibration"]
|
||||
lines.append(f"- ECE: {fmt_float(cal_all['ece'])}")
|
||||
lines.append(f"- Brier score: {fmt_float(cal_all['brier'])}")
|
||||
lines.append(f"- Log loss: {fmt_float(cal_all['log_loss'])}")
|
||||
lines.append("")
|
||||
lines.append("### Experiment B")
|
||||
cal_res = residual_results["calibration"]
|
||||
lines.append(f"- ECE: {fmt_float(cal_res['ece'])}")
|
||||
lines.append(f"- Brier score: {fmt_float(cal_res['brier'])}")
|
||||
lines.append(f"- Log loss: {fmt_float(cal_res['log_loss'])}")
|
||||
lines.append("")
|
||||
|
||||
# Abstention curves
|
||||
lines.append("## 12. Abstention curves")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A (all dev)")
|
||||
lines.append(f"{'threshold':>10} {'n_accepted':>11} {'coverage':>10} {'accuracy':>10} {'macro_f1':>10} {'false_act':>10}")
|
||||
lines.append("-" * 62)
|
||||
for curve in cal_all["abstention_curves"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {curve['n_accepted']:>11} {fmt_pct(curve['coverage']):>10} {fmt_pct(curve['accuracy']):>10} {fmt_float(curve['macro_f1']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B (residual only)")
|
||||
lines.append(f"{'threshold':>10} {'n_accepted':>11} {'coverage':>10} {'accuracy':>10} {'macro_f1':>10} {'false_act':>10}")
|
||||
lines.append("-" * 62)
|
||||
for curve in cal_res["abstention_curves"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {curve['n_accepted']:>11} {fmt_pct(curve['coverage']):>10} {fmt_pct(curve['accuracy']):>10} {fmt_float(curve['macro_f1']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
# Action threshold
|
||||
lines.append("## 13. Action-threshold curve")
|
||||
lines.append("")
|
||||
lines.append("### Experiment A")
|
||||
lines.append(f"{'threshold':>10} {'action_P':>10} {'action_R':>10} {'false_act':>10}")
|
||||
lines.append("-" * 40)
|
||||
for curve in all_results["action_threshold"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {fmt_float(curve['action_precision']):>10} {fmt_float(curve['action_recall']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("### Experiment B")
|
||||
lines.append(f"{'threshold':>10} {'action_P':>10} {'action_R':>10} {'false_act':>10}")
|
||||
lines.append("-" * 40)
|
||||
for curve in residual_results["action_threshold"]:
|
||||
lines.append(f"{curve['threshold']:>10.2f} {fmt_float(curve['action_precision']):>10} {fmt_float(curve['action_recall']):>10} {curve['false_action_count']:>10}")
|
||||
lines.append("")
|
||||
|
||||
# Model artifact size
|
||||
lines.append("## 14. Model artifact size and runtime cost")
|
||||
lines.append("")
|
||||
# Logistic regression: 6 classes × 384 features + 6 biases = 2310 parameters
|
||||
n_params = len(ROUTES) * meta["dimension"] + len(ROUTES)
|
||||
serialized_bytes = n_params * 4 # float32
|
||||
lines.append(f"- Trainable parameters: {n_params}")
|
||||
lines.append(f" - {len(ROUTES)} classes × {meta['dimension']} features = {len(ROUTES) * meta['dimension']} weights")
|
||||
lines.append(f" - {len(ROUTES)} bias terms")
|
||||
lines.append(f"- Serialized head size: {serialized_bytes} bytes ({serialized_bytes / 1024:.1f} KB)")
|
||||
lines.append(f"- Additional inference FLOPs: {len(ROUTES) * meta['dimension']} multiply-accumulates")
|
||||
lines.append(f"- Incremental cost (e5 already computed): ~{len(ROUTES) * meta['dimension']} FLOPs, <1µs")
|
||||
lines.append(f"- Cost if semantic router must trigger its own e5: full ONNX inference (~{meta['dimension']} × 128 × 12 = ~590K FLOPs)")
|
||||
lines.append("")
|
||||
|
||||
# Recommendation
|
||||
lines.append("## 16. Recommendation")
|
||||
lines.append("")
|
||||
# Decision logic
|
||||
all_f1 = metrics_all["macro_f1"]
|
||||
res_f1 = metrics_res["macro_f1"]
|
||||
res_acc = metrics_res["accuracy"]
|
||||
|
||||
if res_f1 > 0.5 and res_acc > 0.55:
|
||||
verdict = "linear head sufficient"
|
||||
detail = (f"Residual macro F1 of {fmt_float(res_f1)} and accuracy of {fmt_pct(res_acc)} "
|
||||
f"exceed the legacy baseline (40.8% residual accuracy) by a meaningful margin. "
|
||||
f"A linear head over frozen e5-small embeddings is a viable first production candidate.")
|
||||
elif all_f1 > 0.5 and res_f1 < 0.5:
|
||||
verdict = "need more data"
|
||||
detail = (f"All-example F1 ({fmt_float(all_f1)}) is acceptable but residual-only F1 "
|
||||
f"({fmt_float(res_f1)}) drops, suggesting the contrast-family examples are "
|
||||
f"hard for a linear classifier. More contrastive training data may help.")
|
||||
else:
|
||||
verdict = "representation inadequate"
|
||||
detail = (f"Neither all-example ({fmt_float(all_f1)}) nor residual-only ({fmt_float(res_f1)}) "
|
||||
f"F1 reaches the minimum viable threshold. The e5-small linear separability floor "
|
||||
f"is insufficient for this 6-way task. Consider a non-linear head or a different encoder.")
|
||||
|
||||
lines.append(f"**{verdict}**")
|
||||
lines.append("")
|
||||
lines.append(detail)
|
||||
lines.append("")
|
||||
|
||||
# Commit hash
|
||||
lines.append("## 17. Commit hash for experiment tooling")
|
||||
lines.append("")
|
||||
import subprocess
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True, cwd="/home/kami/apps/Maven"
|
||||
)
|
||||
lines.append(f"`{result.stdout.strip()}`")
|
||||
except Exception:
|
||||
lines.append("(unable to determine)")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def run_experiment(X, y, fold_ids, examples_meta, label):
|
||||
"""Run the full experiment pipeline for one population."""
|
||||
print(f"\n{'='*60}")
|
||||
print(f" Experiment: {label}")
|
||||
print(f" Samples: {len(y)} Features: {X.shape[1]} Folds: {len(set(fold_ids))}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# 1. Grouped CV with regularization search
|
||||
best_C, results_by_C = grouped_cv_experiment(X, y, fold_ids, C_VALUES, examples_meta)
|
||||
|
||||
# 2. Use best C to get full metrics from OOF predictions
|
||||
oof_rows = results_by_C[best_C]["oof_predictions"]
|
||||
oof_true = np.array([r["true"] for r in oof_rows])
|
||||
oof_pred = np.array([r["predicted"] for r in oof_rows])
|
||||
|
||||
full_metrics = compute_full_metrics(oof_true, oof_pred)
|
||||
|
||||
# 3. Calibration
|
||||
calibration = compute_calibration(oof_true, oof_pred, oof_rows)
|
||||
|
||||
# 4. Action threshold
|
||||
action_threshold = compute_action_threshold(oof_rows)
|
||||
|
||||
# 5. Contrast family
|
||||
contrast_family = compute_contrast_family(oof_rows)
|
||||
|
||||
return {
|
||||
"best_C": best_C,
|
||||
"results_by_C": results_by_C,
|
||||
"full_metrics": full_metrics,
|
||||
"calibration": calibration,
|
||||
"action_threshold": action_threshold,
|
||||
"contrast_family": contrast_family,
|
||||
"oof_rows": oof_rows,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
REPORT_PATH = "/tmp/mvn-experiment/report.md"
|
||||
|
||||
# Load data
|
||||
meta, examples = load_embeddings(EMBEDDING_PATH)
|
||||
print(f"Loaded {len(examples)} examples, embedder={meta['embedder_id']}, dim={meta['dimension']}")
|
||||
|
||||
# Development pool only
|
||||
dev_examples = filter_dev_pool(examples)
|
||||
print(f"Development pool: {len(dev_examples)} examples")
|
||||
|
||||
# Extract features
|
||||
X_all, y_all = extract_Xy(dev_examples)
|
||||
fold_ids_all = get_fold_groups(dev_examples)
|
||||
|
||||
# Experiment A: all dev examples
|
||||
all_results = run_experiment(X_all, y_all, fold_ids_all, dev_examples, "All development examples")
|
||||
|
||||
# Experiment B: residual only
|
||||
dev_residual = filter_residual(dev_examples)
|
||||
X_res, y_res = extract_Xy(dev_residual)
|
||||
fold_ids_res = get_fold_groups(dev_residual)
|
||||
residual_results = run_experiment(X_res, y_res, fold_ids_res, dev_residual, "Router-residual only")
|
||||
|
||||
# Generate report
|
||||
report = generate_report(
|
||||
meta, all_results, residual_results,
|
||||
all_results["oof_rows"], residual_results["oof_rows"],
|
||||
)
|
||||
|
||||
with open(REPORT_PATH, "w") as f:
|
||||
f.write(report)
|
||||
print(f"\nReport written to {REPORT_PATH}")
|
||||
|
||||
# Also print summary
|
||||
print("\n" + "="*60)
|
||||
print(" SUMMARY")
|
||||
print("="*60)
|
||||
print(f" All-example: acc={fmt_pct(all_results['full_metrics']['accuracy'])} macro_f1={fmt_float(all_results['full_metrics']['macro_f1'])} false_action={all_results['full_metrics']['false_action_count']}")
|
||||
print(f" Residual: acc={fmt_pct(residual_results['full_metrics']['accuracy'])} macro_f1={fmt_float(residual_results['full_metrics']['macro_f1'])} false_action={residual_results['full_metrics']['false_action_count']}")
|
||||
print(f" Best C (all): {all_results['best_C']}")
|
||||
print(f" Best C (res): {residual_results['best_C']}")
|
||||
print(f" ECE (all): {fmt_float(all_results['calibration']['ece'])}")
|
||||
print(f" ECE (res): {fmt_float(residual_results['calibration']['ece'])}")
|
||||
print(f" Brier (all): {fmt_float(all_results['calibration']['brier'])}")
|
||||
print(f" Brier (res): {fmt_float(residual_results['calibration']['brier'])}")
|
||||
@@ -0,0 +1,273 @@
|
||||
// semantic-router-experiment computes embeddings for the semantic coarse-route
|
||||
// corpus using the deployed multilingual-e5-small ONNX model. It outputs a
|
||||
// JSON file containing every corpus row with its embedding vector, split
|
||||
// assignment, and fold membership for grouped cross-validation.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// MAVEN_ONNX_LIB=/path/to/libonnxruntime.so \
|
||||
// go run ./cmd/semantic-router-experiment/ -out embeddings.json
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
// CachedRow is one corpus row with its precomputed embedding and split metadata.
|
||||
type CachedRow struct {
|
||||
Text string `json:"text"`
|
||||
Route string `json:"route"`
|
||||
Source string `json:"source"`
|
||||
SourceID string `json:"source_id"`
|
||||
SplitGroup string `json:"split_group"`
|
||||
Tags []string `json:"tags,omitempty"`
|
||||
FastPathResolved bool `json:"fast_path_resolved"`
|
||||
RouterResidual *bool `json:"router_residual,omitempty"`
|
||||
TextHash string `json:"text_hash"`
|
||||
Embedding []float32 `json:"embedding"`
|
||||
EmbedderID string `json:"embedder_id"`
|
||||
FrozenHoldout bool `json:"frozen_holdout"`
|
||||
CVFold int `json:"cv_fold"`
|
||||
DevPool bool `json:"dev_pool"`
|
||||
FamilyID string `json:"family_id"`
|
||||
}
|
||||
|
||||
// ExperimentMeta carries metadata about the experiment run.
|
||||
type ExperimentMeta struct {
|
||||
EmbedderID string `json:"embedder_id"`
|
||||
ModelPath string `json:"model_path"`
|
||||
TokenizerPath string `json:"tokenizer_path"`
|
||||
Dimension int `json:"dimension"`
|
||||
Pooling string `json:"pooling"`
|
||||
Normalization string `json:"normalization"`
|
||||
InputTemplate string `json:"input_template"`
|
||||
TotalExamples int `json:"total_examples"`
|
||||
FrozenCount int `json:"frozen_count"`
|
||||
DevCount int `json:"dev_count"`
|
||||
CVFolds int `json:"cv_folds"`
|
||||
FoldComposition map[int]FoldStats `json:"fold_composition"`
|
||||
RouteCounts map[string]int `json:"route_counts"`
|
||||
SourceCounts map[string]int `json:"source_counts"`
|
||||
FastPathCount int `json:"fast_path_count"`
|
||||
ResidualCount int `json:"residual_count"`
|
||||
HoldoutHash string `json:"holdout_hash"`
|
||||
}
|
||||
|
||||
// FoldStats describes one CV fold.
|
||||
type FoldStats struct {
|
||||
EvalCount int `json:"eval_count"`
|
||||
TrainCount int `json:"train_count"`
|
||||
Routes map[string]int `json:"eval_routes"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
outPath := flag.String("out", "embeddings.json", "output JSON path")
|
||||
folds := flag.Int("folds", 5, "number of CV folds")
|
||||
flag.Parse()
|
||||
|
||||
libPath := os.Getenv("MAVEN_ONNX_LIB")
|
||||
if libPath == "" {
|
||||
log.Fatal("MAVEN_ONNX_LIB must be set to the libonnxruntime.so path")
|
||||
}
|
||||
|
||||
// Resolve model paths relative to the module root (cwd when running with go run).
|
||||
modelPath := "models/embedder/multilingual-e5-small/model_quantized.onnx"
|
||||
tokPath := "models/embedder/multilingual-e5-small/tokenizer.json"
|
||||
|
||||
for _, p := range []string{libPath, modelPath, tokPath} {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
log.Fatalf("missing %s: %v", p, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Load corpus.
|
||||
exs, err := semantic.LoadCorpus()
|
||||
if err != nil {
|
||||
log.Fatalf("load corpus: %v", err)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "corpus: %d examples\n", len(exs))
|
||||
|
||||
// Initialize embedder.
|
||||
emb, err := router.NewONNXEmbedder(modelPath, tokPath, libPath)
|
||||
if err != nil {
|
||||
log.Fatalf("init embedder: %v", err)
|
||||
}
|
||||
defer emb.Close()
|
||||
fmt.Fprintf(os.Stderr, "embedder: %s (dim=%d)\n", emb.ID(), emb.Dim())
|
||||
|
||||
// Compute frozen holdout / dev pool split.
|
||||
_, devPool, holdoutHash := semantic.FrozenHoldoutSplit(exs)
|
||||
fmt.Fprintf(os.Stderr, "frozen holdout: hash=%s, dev pool: %d examples\n", holdoutHash, len(devPool))
|
||||
|
||||
// Compute grouped CV folds on dev pool only.
|
||||
cvFolds := semantic.GroupedCVFolds(devPool, *folds)
|
||||
fmt.Fprintf(os.Stderr, "cv folds: %d\n", len(cvFolds))
|
||||
|
||||
// Build a lookup: source_id → cv_fold (from dev pool only).
|
||||
foldLookup := make(map[string]int)
|
||||
for _, f := range cvFolds {
|
||||
for _, e := range f.Eval {
|
||||
foldLookup[e.SourceID] = f.Fold
|
||||
}
|
||||
}
|
||||
|
||||
// Build dev set membership lookup.
|
||||
_, devSet, _ := semantic.FrozenHoldoutSplit(exs)
|
||||
devIDs := make(map[string]bool)
|
||||
for _, e := range devSet {
|
||||
devIDs[e.SourceID] = true
|
||||
}
|
||||
|
||||
// Embed all examples.
|
||||
ctx := context.Background()
|
||||
var cached []CachedRow
|
||||
foldComp := make(map[int]*FoldStats)
|
||||
for i := 0; i < *folds; i++ {
|
||||
foldComp[i] = &FoldStats{Routes: make(map[string]int)}
|
||||
}
|
||||
routeCounts := make(map[string]int)
|
||||
sourceCounts := make(map[string]int)
|
||||
fpCount, resCount := 0, 0
|
||||
|
||||
for i, e := range exs {
|
||||
textHash := sha256.Sum256([]byte(e.Text))
|
||||
embedding, err := emb.EmbedQuery(ctx, e.Text)
|
||||
if err != nil {
|
||||
log.Fatalf("embed row %d (%s): %v", i, e.SourceID, err)
|
||||
}
|
||||
|
||||
inDev := devIDs[e.SourceID]
|
||||
fold := -1
|
||||
if inDev {
|
||||
if f, ok := foldLookup[e.SourceID]; ok {
|
||||
fold = f
|
||||
}
|
||||
}
|
||||
|
||||
isFrozen := !inDev
|
||||
|
||||
cr := CachedRow{
|
||||
Text: e.Text,
|
||||
Route: string(e.Route),
|
||||
Source: e.Source,
|
||||
SourceID: e.SourceID,
|
||||
SplitGroup: e.SplitGroup,
|
||||
Tags: e.Tags,
|
||||
FastPathResolved: e.FastPathResolved,
|
||||
RouterResidual: e.RouterResidual,
|
||||
TextHash: hex.EncodeToString(textHash[:]),
|
||||
Embedding: embedding,
|
||||
EmbedderID: emb.ID(),
|
||||
FrozenHoldout: isFrozen,
|
||||
CVFold: fold,
|
||||
DevPool: inDev,
|
||||
FamilyID: e.SplitGroup,
|
||||
}
|
||||
cached = append(cached, cr)
|
||||
|
||||
routeCounts[cr.Route]++
|
||||
sourceCounts[cr.Source]++
|
||||
if e.FastPathResolved {
|
||||
fpCount++
|
||||
} else {
|
||||
resCount++
|
||||
}
|
||||
|
||||
if inDev && fold >= 0 {
|
||||
foldComp[fold].EvalCount++
|
||||
foldComp[fold].Routes[cr.Route]++
|
||||
}
|
||||
}
|
||||
|
||||
// Compute train counts per fold.
|
||||
for i := 0; i < *folds; i++ {
|
||||
foldComp[i].TrainCount = len(devPool) - foldComp[i].EvalCount
|
||||
}
|
||||
|
||||
// Sort route counts for deterministic output.
|
||||
sortedRoutes := make([]string, 0, len(routeCounts))
|
||||
for r := range routeCounts {
|
||||
sortedRoutes = append(sortedRoutes, r)
|
||||
}
|
||||
sort.Strings(sortedRoutes)
|
||||
sortedRouteCounts := make(map[string]int)
|
||||
for _, r := range sortedRoutes {
|
||||
sortedRouteCounts[r] = routeCounts[r]
|
||||
}
|
||||
|
||||
// Build fold stats with sorted keys.
|
||||
finalFoldComp := make(map[int]FoldStats)
|
||||
for i := 0; i < *folds; i++ {
|
||||
finalFoldComp[i] = *foldComp[i]
|
||||
}
|
||||
|
||||
meta := ExperimentMeta{
|
||||
EmbedderID: emb.ID(),
|
||||
ModelPath: modelPath,
|
||||
TokenizerPath: tokPath,
|
||||
Dimension: emb.Dim(),
|
||||
Pooling: "mean-pool + L2-normalize",
|
||||
Normalization: "L2",
|
||||
InputTemplate: "query: <text>",
|
||||
TotalExamples: len(exs),
|
||||
FrozenCount: len(exs) - len(devPool),
|
||||
DevCount: len(devPool),
|
||||
CVFolds: *folds,
|
||||
FoldComposition: finalFoldComp,
|
||||
RouteCounts: sortedRouteCounts,
|
||||
SourceCounts: sourceCounts,
|
||||
FastPathCount: fpCount,
|
||||
ResidualCount: resCount,
|
||||
HoldoutHash: holdoutHash,
|
||||
}
|
||||
|
||||
// Output.
|
||||
output := map[string]any{
|
||||
"meta": meta,
|
||||
"examples": cached,
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(output, "", " ")
|
||||
if err != nil {
|
||||
log.Fatalf("marshal: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(*outPath, data, 0644); err != nil {
|
||||
log.Fatalf("write %s: %v", *outPath, err)
|
||||
}
|
||||
|
||||
// Print summary.
|
||||
fmt.Fprintf(os.Stderr, "\n=== experiment metadata ===\n")
|
||||
fmt.Fprintf(os.Stderr, "embedder: %s\n", meta.EmbedderID)
|
||||
fmt.Fprintf(os.Stderr, "dimension: %d\n", meta.Dimension)
|
||||
fmt.Fprintf(os.Stderr, "total: %d frozen: %d dev: %d\n", meta.TotalExamples, meta.FrozenCount, meta.DevCount)
|
||||
fmt.Fprintf(os.Stderr, "fast-path: %d residual: %d\n", meta.FastPathCount, meta.ResidualCount)
|
||||
fmt.Fprintf(os.Stderr, "routes: %s\n", formatMap(sortedRouteCounts))
|
||||
fmt.Fprintf(os.Stderr, "fold composition:\n")
|
||||
for i := 0; i < *folds; i++ {
|
||||
fs := finalFoldComp[i]
|
||||
fmt.Fprintf(os.Stderr, " fold %d: eval=%d train=%d routes=%s\n",
|
||||
i, fs.EvalCount, fs.TrainCount, formatMap(fs.Routes))
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "output: %s (%d bytes)\n", *outPath, len(data))
|
||||
}
|
||||
|
||||
func formatMap(m map[string]int) string {
|
||||
var parts []string
|
||||
for k, v := range m {
|
||||
parts = append(parts, fmt.Sprintf("%s=%d", k, v))
|
||||
}
|
||||
sort.Strings(parts)
|
||||
return "{" + strings.Join(parts, ", ") + "}"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,665 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 18: Sparse Lexical Action/Non-Action Gate
|
||||
================================================
|
||||
|
||||
Answer: can Maven reliably distinguish executable requests from semantically
|
||||
similar non-actions using lexical/local-order features alone?
|
||||
|
||||
Representations under test (all frozen-population, no e5):
|
||||
A. word 1-2 grams, TF-IDF
|
||||
B. character 3-5 grams, TF-IDF (Unicode, no transliteration)
|
||||
C. [word ; char] combined TF-IDF
|
||||
|
||||
Population reused from slices 15-17: development corpus v2 (dev_pool), router
|
||||
labels, SplitGroup, cv_fold, tags. The e5 embedding vectors are ignored.
|
||||
"""
|
||||
|
||||
import io
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import unicodedata
|
||||
import warnings
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
import numpy as np
|
||||
from sklearn.exceptions import ConvergenceWarning
|
||||
from sklearn.feature_extraction.text import TfidfVectorizer
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.metrics import (
|
||||
accuracy_score,
|
||||
average_precision_score,
|
||||
confusion_matrix,
|
||||
f1_score,
|
||||
precision_recall_fscore_support,
|
||||
roc_auc_score,
|
||||
)
|
||||
from sklearn.pipeline import make_pipeline
|
||||
from scipy import sparse
|
||||
|
||||
warnings.filterwarnings("ignore", category=ConvergenceWarning)
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
|
||||
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
|
||||
# Present generator/contrast families in the v2 dev pool (for leave-family-out)
|
||||
PRESENT_FAMILIES = [
|
||||
"polite_request",
|
||||
"modal_request",
|
||||
"first_person_request",
|
||||
"reordered_target",
|
||||
"capability_question",
|
||||
"question",
|
||||
]
|
||||
|
||||
FAMILY_ALIASES = {
|
||||
"direct_imperative": "direct_imperative",
|
||||
"polite_request": "polite_request",
|
||||
"modal_request": "modal_request",
|
||||
"first_person_request": "first_person_request",
|
||||
"reordered_target": "reordered_target",
|
||||
"capability_question": "capability_question",
|
||||
"question": "question",
|
||||
}
|
||||
|
||||
|
||||
# ─── NormalizeMatchText (replicated from internal/router/matchtext.go) ─────
|
||||
|
||||
def normalize_match_text(s: str) -> str:
|
||||
"""NFKC → lowercase → collapse Unicode whitespace. Keeps punctuation, ё."""
|
||||
out = unicodedata.normalize("NFKC", s).strip().lower()
|
||||
out = re.sub(r"\s+", " ", out)
|
||||
return out
|
||||
|
||||
|
||||
# ─── Data Loading ───────────────────────────────────────────────────────────
|
||||
|
||||
def load_data():
|
||||
with open(EMBEDDING_PATH) as f:
|
||||
data = json.load(f)
|
||||
return data["meta"], data["examples"]
|
||||
|
||||
|
||||
def filter_dev_pool(examples):
|
||||
return [e for e in examples if e["dev_pool"]]
|
||||
|
||||
|
||||
def residual_only(examples):
|
||||
return [e for e in examples if not e["fast_path_resolved"]]
|
||||
|
||||
|
||||
def fmt_pct(v, d=1):
|
||||
return f"{100*v:.{d}f}%"
|
||||
|
||||
|
||||
def ff(v, d=3):
|
||||
return f"{v:.{d}f}"
|
||||
|
||||
|
||||
def strip_punct(text: str) -> str:
|
||||
"""Remove all punctuation (shared with slice 16/17 apply_voice_stress)."""
|
||||
t = re.sub(r"[?.!,;:]+$", "", text.strip())
|
||||
t = re.sub(r"[^\w\s]", "", t)
|
||||
t = t.lower()
|
||||
t = re.sub(r"\s+", " ", t).strip()
|
||||
return t
|
||||
|
||||
|
||||
# ─── Feature Builders ───────────────────────────────────────────────────────
|
||||
|
||||
def build_features(texts, kind):
|
||||
"""Build a TF-IDF matrix for the given representation kind.
|
||||
kind in {'word','char','both'}. Returns (X_sparse, vectorizer)."""
|
||||
if kind == "word":
|
||||
vec = TfidfVectorizer(
|
||||
ngram_range=(1, 2), analyzer="word",
|
||||
strip_accents=None, lowercase=False,
|
||||
min_df=2, sublinear_tf=True,
|
||||
)
|
||||
elif kind == "char":
|
||||
# preserve case (already lowered) and identity of missing diacritics;
|
||||
# token_pattern null => char analyzer
|
||||
vec = TfidfVectorizer(
|
||||
ngram_range=(3, 5), analyzer="char",
|
||||
strip_accents=None, lowercase=False,
|
||||
min_df=2, sublinear_tf=True,
|
||||
)
|
||||
elif kind == "both":
|
||||
vec_word = TfidfVectorizer(
|
||||
ngram_range=(1, 2), analyzer="word",
|
||||
strip_accents=None, lowercase=False, min_df=2, sublinear_tf=True,
|
||||
)
|
||||
vec_char = TfidfVectorizer(
|
||||
ngram_range=(3, 5), analyzer="char",
|
||||
strip_accents=None, lowercase=False, min_df=2, sublinear_tf=True,
|
||||
)
|
||||
Xw = vec_word.fit_transform(texts)
|
||||
Xc = vec_char.fit_transform(texts)
|
||||
X = sparse.hstack([Xw, Xc]).tocsr()
|
||||
return X, ("both", vec_word, vec_char)
|
||||
X = vec.fit_transform(texts)
|
||||
return X, vec
|
||||
|
||||
|
||||
def vocab_size(vectorizer):
|
||||
if isinstance(vectorizer, tuple):
|
||||
_, vw, vc = vectorizer
|
||||
return vw.get_feature_names_out().shape[0] + vc.get_feature_names_out().shape[0]
|
||||
return vectorizer.get_feature_names_out().shape[0]
|
||||
|
||||
|
||||
def transform_texts(texts, vectorizer):
|
||||
"""Apply an already-fitted vectorizer (handles the 2-tuple 'both' case)."""
|
||||
if isinstance(vectorizer, tuple):
|
||||
_, vw, vc = vectorizer
|
||||
Xw = vw.transform(texts)
|
||||
Xc = vc.transform(texts)
|
||||
return sparse.hstack([Xw, Xc]).tocsr()
|
||||
return vectorizer.transform(texts)
|
||||
|
||||
|
||||
# ─── Grouped CV ─────────────────────────────────────────────────────────────
|
||||
|
||||
def run_binary_grouped_cv(X, y, fold_ids, C=1.0):
|
||||
"""Grouped CV for binary action vs not_action. Returns OOF rows + fold metrics."""
|
||||
yb = np.array([1 if t == "action" else 0 for t in y])
|
||||
fold_ids = np.asarray(fold_ids)
|
||||
unique_folds = sorted(set(fold_ids.tolist()))
|
||||
oof_rows = []
|
||||
fold_metrics = []
|
||||
|
||||
for test_fold in unique_folds:
|
||||
tr = fold_ids != test_fold
|
||||
te = fold_ids == test_fold
|
||||
clf = LogisticRegression(C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
proba = clf.predict_proba(X[te])[:, 1]
|
||||
pred = (proba >= 0.5).astype(int)
|
||||
yt = yb[te]
|
||||
|
||||
fp = int(((yt == 0) & (pred == 1)).sum())
|
||||
fn = int(((yt == 1) & (pred == 0)).sum())
|
||||
tp = int(((yt == 1) & (pred == 1)).sum())
|
||||
tn = int(((yt == 0) & (pred == 0)).sum())
|
||||
|
||||
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
acc = accuracy_score(yt, pred)
|
||||
prec = tp / max(tp + fp, 1)
|
||||
rec = tp / max(tp + fn, 1)
|
||||
|
||||
fold_metrics.append({
|
||||
"fold": int(test_fold), "n": int(len(yt)),
|
||||
"roc_auc": roc, "pr_auc": pr,
|
||||
"action_precision": prec, "action_recall": rec,
|
||||
"fp": fp, "fn": fn, "tp": tp, "tn": tn,
|
||||
"acc": acc,
|
||||
})
|
||||
te_idx = np.where(te)[0]
|
||||
for i in range(len(yt)):
|
||||
oof_rows.append({
|
||||
"fold": int(test_fold),
|
||||
"proba": float(proba[i]),
|
||||
"pred": int(pred[i]),
|
||||
"true": int(yt[i]),
|
||||
})
|
||||
|
||||
return oof_rows, fold_metrics
|
||||
|
||||
|
||||
# ─── Metrics from OOF ───────────────────────────────────────────────────────
|
||||
|
||||
def binary_metrics_from_oof(oof_rows):
|
||||
yt = np.array([r["true"] for r in oof_rows])
|
||||
yp = np.array([r["pred"] for r in oof_rows])
|
||||
proba = np.array([r["proba"] for r in oof_rows])
|
||||
n = len(yt)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
prec = tp / max(tp + fp, 1)
|
||||
rec = tp / max(tp + fn, 1)
|
||||
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
return {
|
||||
"n": n, "tp": tp, "fp": fp, "fn": fn,
|
||||
"action_precision": prec, "action_recall": rec,
|
||||
"fa_rate": fp / max(n, 1),
|
||||
"roc_auc": roc, "pr_auc": pr,
|
||||
}
|
||||
|
||||
|
||||
def threshold_curve(oof_rows, thresholds):
|
||||
yt = np.array([r["true"] for r in oof_rows])
|
||||
proba = np.array([r["proba"] for r in oof_rows])
|
||||
rows = []
|
||||
for thr in thresholds:
|
||||
yp = (proba >= thr).astype(int)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
rows.append({
|
||||
"threshold": round(float(thr), 4),
|
||||
"action_precision": round(tp / max(tp + fp, 1), 4),
|
||||
"action_recall": round(tp / max(tp + fn, 1), 4),
|
||||
"fa_count": fp,
|
||||
"fa_rate": round(fp / max(len(yt), 1), 4),
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
# ─── Leave-Family-Out ───────────────────────────────────────────────────────
|
||||
|
||||
def run_leave_family_out(texts, y, family_per_row, family, kind):
|
||||
"""Train without `family`, evaluate on `family` only."""
|
||||
mask_members = family_per_row == family
|
||||
if mask_members.sum() == 0:
|
||||
return None
|
||||
only_family = (mask_members).astype(bool)
|
||||
train_idx = np.where(~only_family)[0]
|
||||
test_idx = np.where(only_family)[0]
|
||||
X, _ = build_features([texts[i] for i in train_idx], kind)
|
||||
# map test rows onto the full vocabulary
|
||||
tr_texts = [texts[i] for i in train_idx]
|
||||
te_texts = [texts[i] for i in test_idx]
|
||||
all_texts = tr_texts + te_texts
|
||||
Xall, _ = build_features(all_texts, kind)
|
||||
Xtr = Xall[:len(tr_texts)]
|
||||
Xte = Xall[len(tr_texts):]
|
||||
ytr = np.array([1 if y[i] == "action" else 0 for i in train_idx])
|
||||
yte = np.array([1 if y[i] == "action" else 0 for i in test_idx])
|
||||
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(Xtr, ytr)
|
||||
pred = clf.predict(Xte)
|
||||
tp = int(((yte == 1) & (pred == 1)).sum())
|
||||
fp = int(((yte == 0) & (pred == 1)).sum())
|
||||
fn = int(((yte == 1) & (pred == 0)).sum())
|
||||
prec = tp / max(tp + fp, 1)
|
||||
rec = tp / max(tp + fn, 1)
|
||||
acc = accuracy_score(yte, pred)
|
||||
return {
|
||||
"family": family, "rows": int(len(yte)),
|
||||
"action_precision": prec, "action_recall": rec,
|
||||
"fa_count": fp, "acc": acc,
|
||||
"yticks": f"pos={int(yte.sum())} neg={int((yte==0).sum())}",
|
||||
}
|
||||
|
||||
|
||||
# ─── Paired Action/Capability Test ──────────────────────────────────────────
|
||||
|
||||
def paired_action_capability(texts, y, tags_per_row_by_idx, kind, C=1.0):
|
||||
"""For the best sparse representation, evaluate pairwise ordering."""
|
||||
idx = list(range(len(texts)))
|
||||
emb = build_features(texts, kind)[0]
|
||||
clf = LogisticRegression(C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
yb = np.array([1 if t == "action" else 0 for t in y])
|
||||
# use OOF-style: fit on full then derive? We report the pairwise score test;
|
||||
# to avoid leakage we use grouped OOF proba via grouped cv.
|
||||
return None
|
||||
|
||||
|
||||
# ─── Main ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
meta, examples = load_data()
|
||||
dev = filter_dev_pool(examples)
|
||||
dev_res = residual_only(dev)
|
||||
|
||||
# normalized texts
|
||||
for e in dev:
|
||||
e["n_text"] = normalize_match_text(e["text"])
|
||||
|
||||
texts = [e["n_text"] for e in dev]
|
||||
y = [e["route"] for e in dev]
|
||||
folds = [e["cv_fold"] for e in dev]
|
||||
tags = [set(e.get("tags", [])) for e in dev]
|
||||
split_groups = [e["split_group"] for e in dev]
|
||||
source_ids = [e["source_id"] for e in dev]
|
||||
|
||||
n_action = sum(1 for r in y if r == "action")
|
||||
n_not = len(y) - n_action
|
||||
print(f"Dev pool: {len(dev)} action={n_action} not_action={n_not}")
|
||||
print()
|
||||
|
||||
results = {}
|
||||
|
||||
# ── 1. Build features and run grouped CV for each representation ──────
|
||||
for kind in ["word", "char", "both"]:
|
||||
print(f"\n=== {kind} TF-IDF ===")
|
||||
t0 = time.time()
|
||||
X, vec = build_features(texts, kind)
|
||||
build_t = time.time() - t0
|
||||
vs = vocab_size(vec)
|
||||
print(f" vocab={vs} X.shape={X.shape} build={build_t:.2f}s")
|
||||
|
||||
oof, folds_m = run_binary_grouped_cv(X, y, folds, C=1.0)
|
||||
m = binary_metrics_from_oof(oof)
|
||||
results[kind] = {
|
||||
"vs": vs, "build_t": build_t, "oof": oof, "fold_metrics": folds_m,
|
||||
"metrics": m, "X": X, "vec": vec,
|
||||
"texts": texts, "y": y, "folds": folds,
|
||||
}
|
||||
print(f" ROC={ff(m['roc_auc'])} PR={ff(m['pr_auc'])} P={ff(m['action_precision'])} "
|
||||
f"R={ff(m['action_recall'])} FA={m['fp']} ({fmt_pct(m['fa_rate'])})")
|
||||
|
||||
# fold-level
|
||||
for fm in folds_m:
|
||||
print(f" fold {fm['fold']}: ROC={ff(fm['roc_auc'])} PR={ff(fm['pr_auc'])} "
|
||||
f"P={ff(fm['action_precision'])} R={ff(fm['action_recall'])} "
|
||||
f"FP={fm['fp']} FN={fm['fn']} n={fm['n']}")
|
||||
|
||||
# pick best by PR-AUC
|
||||
best_kind = max(["word", "char", "both"], key=lambda k: results[k]["metrics"]["pr_auc"])
|
||||
print(f"\nBest representation by PR-AUC: {best_kind}")
|
||||
|
||||
# ── 2. Leave-family-out for best kind ─────────────────────────────────
|
||||
print(f"\n=== Leave-family-out ({best_kind}) ===")
|
||||
fam_per_row = []
|
||||
for tg in tags:
|
||||
fam = None
|
||||
# prefer the more specific contrast/request families first (a single
|
||||
# utterance may carry several generator tags, e.g. capability_question
|
||||
# plus direct_imperative). Check the informative ones before the
|
||||
# generic direct_imperative fallback.
|
||||
for f in ["capability_question", "question", "first_person_request",
|
||||
"modal_request", "polite_request", "reordered_target",
|
||||
"direct_imperative"]:
|
||||
if f in tg:
|
||||
fam = f
|
||||
break
|
||||
fam_per_row.append(fam)
|
||||
fam_per_row = np.array(fam_per_row, dtype=object)
|
||||
lfo = {}
|
||||
for fam in PRESENT_FAMILIES:
|
||||
r = run_leave_family_out(texts, y, fam_per_row, fam, best_kind)
|
||||
if r is None:
|
||||
print(f" {fam}: (no rows)")
|
||||
continue
|
||||
lfo[fam] = r
|
||||
print(f" {fam}: rows={r['rows']} ({r['yticks']}) P={ff(r['action_precision'])} "
|
||||
f"R={ff(r['action_recall'])} FA={r['fa_count']} acc={fmt_pct(r['acc'])}")
|
||||
|
||||
# ── 3. Action/capability paired test ─────────────────────────────────
|
||||
print(f"\n=== Paired action/capability test ({best_kind}) ===")
|
||||
# Use grouped-CV OOF proba for ordering (no leakage)
|
||||
oof = results[best_kind]["oof"]
|
||||
# map oof rows back by source_id order
|
||||
# oof rows are appended per fold in dev order; reconstruct
|
||||
# We'll re-embed and get proba via grouped CV with proba recorded per row.
|
||||
# Re-run grouped CV capturing per-row proba aligned to dev indices.
|
||||
X = results[best_kind]["X"]
|
||||
yb = np.array([1 if r == "action" else 0 for r in y])
|
||||
folds_arr = np.array(folds)
|
||||
dev_proba = np.zeros(len(dev))
|
||||
for te_fold in sorted(set(folds)):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
dev_proba[te] = clf.predict_proba(X[te])[:, 1]
|
||||
|
||||
# group action seeds: pair each capability-question row with action rows
|
||||
# sharing the same target object noun and the same act (turn_on/turn_off/…).
|
||||
# Both were generated over a common entity+event inventory, so the object
|
||||
# lexeme is the semantic link between a knowledge question and its
|
||||
# executable sibling.
|
||||
DEVICES = [
|
||||
"свет", "люстру", "люстра", "жалюзи", "вытяжку", "вытяжка",
|
||||
"вентилятор", "кондиционер", "телевизор", "лампу", "лампа",
|
||||
"музыку", "музыка", "плеер", "колонку", "колонки", "чайник",
|
||||
"бойлер", "обогреватель", "пылесос", "пылесосом", "пол",
|
||||
"поливалки", "полив", "арка", "шторы", "штору", "штору",
|
||||
"динамики", "дверь", "двери", "замок", "гараж", "ворота",
|
||||
"кофе", "пасту", "зубы", "крючки", "лаймо", "куртку",
|
||||
"будильник", "таймер", "напоминание", "расписание",
|
||||
]
|
||||
|
||||
def object_nouns(t):
|
||||
found = set()
|
||||
tl = t.lower()
|
||||
for d in DEVICES:
|
||||
# match as standalone word (handle Russian case endings loosely via prefix)
|
||||
if re.search(r"\b" + re.escape(d), tl):
|
||||
found.add(d)
|
||||
return found
|
||||
|
||||
# verb/act family per row from source_id (e.g. ha-light-off -> off)
|
||||
def act_family(src):
|
||||
# pull the 3rd token-ish: ha-light-off => 'off'; ha-light-on => 'on'
|
||||
m = re.search(r"^(\w+)-([a-z_]+)-([a-zA-Z_]+)", src)
|
||||
if m:
|
||||
return f"{m.group(1)}-{m.group(2)}-{m.group(3)}"
|
||||
# generic fallback
|
||||
return src.split("-")[0]
|
||||
|
||||
cap_rows = [i for i in range(len(dev)) if "capability_question" in tags[i]]
|
||||
act_idxs = [i for i in range(len(dev)) if y[i] == "action"]
|
||||
|
||||
# For each capability row, candidate sibling actions: same object noun
|
||||
# AND same act (turn_on vs turn_off), i.e. same domain+object. We accept
|
||||
# any action row sharing an object and matching the on/off sense if present.
|
||||
pairs = []
|
||||
for cidx in cap_rows:
|
||||
c_obj = object_nouns(texts[cidx])
|
||||
if not c_obj:
|
||||
continue
|
||||
# cap rows are kq-cap-<domain>-<n>; domain token after 'kq-cap-'
|
||||
dom = re.search(r"kq-cap-([^-]+)", source_ids[cidx])
|
||||
dom = dom.group(1) if dom else None
|
||||
for aidx in act_idxs:
|
||||
a_obj = object_nouns(texts[aidx])
|
||||
if not (c_obj & a_obj):
|
||||
continue
|
||||
# require same home domain when both carry one
|
||||
a_dom = re.search(r"^([a-z]+)-", source_ids[aidx])
|
||||
a_dom = a_dom.group(1) if a_dom else None
|
||||
if dom and a_dom and dom != a_dom:
|
||||
continue
|
||||
pairs.append((cidx, aidx))
|
||||
|
||||
# keep it bounded: cap row pairs with many actions (one per room); that's fine
|
||||
order_ok = 0
|
||||
margins = []
|
||||
reversed_pairs = []
|
||||
for cidx, aidx in pairs:
|
||||
pc = dev_proba[cidx]
|
||||
pa = dev_proba[aidx]
|
||||
margins.append(pa - pc)
|
||||
if pa > pc:
|
||||
order_ok += 1
|
||||
else:
|
||||
reversed_pairs.append((texts[cidx][:40], pc, texts[aidx][:40], pa))
|
||||
if pairs:
|
||||
order_acc = order_ok / len(pairs)
|
||||
margins_arr = np.array(margins)
|
||||
print(f" pairs={len(pairs)} order_acc={ff(order_acc)} mean_margin={ff(margins_arr.mean())} "
|
||||
f"median_margin={ff(np.median(margins_arr))} reversed={len(reversed_pairs)}")
|
||||
for rev in reversed_pairs[:12]:
|
||||
print(f" REV: cap '{rev[0]}' P={rev[1]:.3f} < act '{rev[2]}' P={rev[3]:.3f}")
|
||||
else:
|
||||
order_acc = None
|
||||
print(" (no matched pairs)")
|
||||
|
||||
paired = {
|
||||
"pairs": len(pairs), "order_acc": order_acc,
|
||||
"mean_margin": float(np.mean(margins)) if margins else None,
|
||||
"median_margin": float(np.median(margins)) if margins else None,
|
||||
"reversed": len(reversed_pairs),
|
||||
}
|
||||
|
||||
# ── 4. Voice stress scoring ───────────────────────────────────────────
|
||||
print(f"\n=== Voice-like stress ({best_kind}) ===")
|
||||
# Build stress variants, embed, score with a model trained on normal text
|
||||
# Train one model on full dev (normal punctuation); score stressed variants.
|
||||
clf_full = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf_full.fit(X, yb)
|
||||
|
||||
stress_texts = []
|
||||
stress_meta = [] # (orig_index, mode)
|
||||
for i in range(len(dev)):
|
||||
orig = texts[i]
|
||||
s_nofinal = re.sub(r"[?.!,;:]+$", "", orig)
|
||||
s_all = strip_punct(dev[i]["text"])
|
||||
if s_nofinal != orig:
|
||||
stress_texts.append(s_nofinal); stress_meta.append((i, "no_final"))
|
||||
if s_all != orig:
|
||||
stress_texts.append(s_all); stress_meta.append((i, "all"))
|
||||
if stress_texts:
|
||||
# reuse the already-fitted best-kind vectorizer to keep the feature space
|
||||
# identical to what the full model was trained on.
|
||||
X_stress = transform_texts(stress_texts, results[best_kind]["vec"])
|
||||
proba_stress = clf_full.predict_proba(X_stress)[:, 1]
|
||||
else:
|
||||
proba_stress = []
|
||||
# classify ability: capability-question false-action rate and modal-action recall
|
||||
stress_by_mode = defaultdict(list)
|
||||
for (i, mode), p in zip(stress_meta, proba_stress):
|
||||
stress_by_mode[mode].append((i, p, y[i], tags[i]))
|
||||
voice = {}
|
||||
for mode, rows in stress_by_mode.items():
|
||||
cap_q = [p for (i, p, lbl, tg) in rows if "capability_question" in tg]
|
||||
cap_fa = sum(1 for p in cap_q if p >= 0.5)
|
||||
mod_act = [p for (i, p, lbl, tg) in rows if lbl == "action" and ("polite_request" in tg or "modal_request" in tg)]
|
||||
mod_rec = sum(1 for p in mod_act if p >= 0.5) / max(len(mod_act), 1)
|
||||
voice[mode] = {
|
||||
"n": len(rows),
|
||||
"cap_q_fa": cap_fa / max(len(cap_q), 1), "cap_q_n": len(cap_q),
|
||||
"modal_action_recall": mod_rec, "modal_n": len(mod_act),
|
||||
}
|
||||
print(f" {mode}: n={len(rows)} cap_q_FA={fmt_pct(voice[mode]['cap_q_fa'])} ({voice[mode]['cap_q_n']}) "
|
||||
f"modal_recall={ff(voice[mode]['modal_action_recall'])} ({voice[mode]['modal_n']})")
|
||||
|
||||
# ── 5. Punctuation ablation ───────────────────────────────────────────
|
||||
print(f"\n=== Punctuation ablation ({best_kind}) ===")
|
||||
texts_stripped = [strip_punct(dev[i]["text"]) for i in range(len(dev))]
|
||||
X_stripped_train, vec_stripped = build_features(texts_stripped, best_kind)
|
||||
clf_stripped = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf_stripped.fit(X_stripped_train, yb)
|
||||
# eval on stripped (in-train) and punctuated (out-of-train) via same vectorizer
|
||||
X_orig = transform_texts(texts, vec_stripped)
|
||||
p_orig = clf_stripped.predict_proba(X_orig)[:, 1]
|
||||
p_strip = clf_stripped.predict_proba(X_stripped_train)[:, 1]
|
||||
def report_ablation(proba, name):
|
||||
yp = (proba >= 0.5).astype(int)
|
||||
tp = int(((yb == 1) & (yp == 1)).sum())
|
||||
fp = int(((yb == 0) & (yp == 1)).sum())
|
||||
fn = int(((yb == 1) & (yp == 0)).sum())
|
||||
prec = tp / max(tp + fp, 1); rec = tp / max(tp + fn, 1)
|
||||
print(f" trained-stripped, eval {name}: P={ff(prec)} R={ff(rec)} FA={fp} ({fmt_pct(fp/len(yb))})")
|
||||
report_ablation(p_orig, "punctuated")
|
||||
report_ablation(p_strip, "stripped")
|
||||
|
||||
# ── 6. Threshold curve for best kind ──────────────────────────────────
|
||||
print(f"\n=== Threshold curve ({best_kind}) ===")
|
||||
oof = results[best_kind]["oof"]
|
||||
thresh = np.arange(0.10, 0.995, 0.015).tolist()
|
||||
tcurve = threshold_curve(oof, thresh)
|
||||
print(f"{'thr':>6} {'P':>6} {'R':>6} {'FA':>5} {'FArate':>8}")
|
||||
for t in tcurve:
|
||||
marker = ""
|
||||
if t["action_precision"] >= 0.95:
|
||||
marker = " ← P>=0.95"
|
||||
print(f"{t['threshold']:>6.3f} {t['action_precision']:>6.3f} {t['action_recall']:>6.3f} "
|
||||
f"{t['fa_count']:>5} {t['fa_rate']:>8.4f}{marker}")
|
||||
p95 = [t for t in tcurve if t["action_precision"] >= 0.95 and t["action_recall"] > 0.01]
|
||||
print(f"\nP>=0.95 region: {len(p95)} points; best recall there = "
|
||||
f"{max((t['action_recall'] for t in p95), default=0.0):.4f}")
|
||||
|
||||
# threshold curve using the (no_leak) dev proba instead of .5-threshold OOF
|
||||
# OOF pred used fixed 0.5; curve above re-derives from proba. Good.
|
||||
|
||||
# ── 7. False-action decomposition ────────────────────────────────────
|
||||
print(f"\n=== False-action decomposition ({best_kind}) ===")
|
||||
# recompute OOF predictions at 0.5 from stored 'pred'
|
||||
fa_by_family = Counter()
|
||||
fa_by_group = Counter()
|
||||
# need oof aligned to source_ids — oof stored without source id; rebuild
|
||||
# Re-do grouped cv capturing source_id + tag + split_group
|
||||
fake_oof = []
|
||||
for te_fold in sorted(set(folds_arr)):
|
||||
tr = folds_arr != te_fold; te = folds_arr == te_fold
|
||||
clf = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
p = clf.predict_proba(X[te])[:, 1]
|
||||
pr = (p >= 0.5).astype(int)
|
||||
te_idx = np.where(te)[0]
|
||||
for k, i in enumerate(te_idx):
|
||||
fake_oof.append({
|
||||
"source_id": source_ids[i], "split_group": split_groups[i],
|
||||
"tags": tags[i], "true": y[i], "proba": float(p[k]),
|
||||
"pred": int(pr[k]), "fold": int(te_fold),
|
||||
})
|
||||
|
||||
def classify_semantic_family(row):
|
||||
tg = row["tags"]
|
||||
if "capability_question" in tg: return "capability_question"
|
||||
if "question" in tg: return "ordinary_question"
|
||||
if "remember" in tg or "note" in tg or "idea" in tg or "free_form" in tg: return "memory_write"
|
||||
if "version" in tg or "system" in tg or "health" in tg or "status" in tg: return "system"
|
||||
if "recall" in tg or "world" in tg or "definition" in tg or "aggregate" in tg: return "knowledge_general"
|
||||
if "greeting" in tg or "goodbye" in tg or "thanks" in tg: return "conversation"
|
||||
if row["true"].startswith("system"): return "system"
|
||||
return "other"
|
||||
|
||||
for r in fake_oof:
|
||||
if r["pred"] == 1 and r["true"] != "action":
|
||||
fam = classify_semantic_family(r)
|
||||
fa_by_family[fam] += 1
|
||||
fa_by_group[r["split_group"]] += 1
|
||||
|
||||
print("By family:")
|
||||
for k, v in fa_by_family.most_common():
|
||||
print(f" {k}: {v}")
|
||||
print("By SplitGroup (top 15):")
|
||||
for k, v in fa_by_group.most_common(15):
|
||||
print(f" {k}: {v}")
|
||||
|
||||
# ── 8. Six-way probe on best representation ──────────────────────────
|
||||
print(f"\n=== Six-way probe ({best_kind}) ===")
|
||||
routes = ["action", "conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
y6 = np.array(y)
|
||||
# fit grouped cv 6-way
|
||||
oof6 = []
|
||||
fold6 = []
|
||||
for te_fold in sorted(set(folds_arr)):
|
||||
tr = folds_arr != te_fold; te = folds_arr == te_fold
|
||||
clf = LogisticRegression(C=1.0, max_iter=3000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], y6[tr])
|
||||
pr = clf.predict(X[te])
|
||||
p6 = clf.predict_proba(X[te])
|
||||
classes = clf.classes_
|
||||
te_idx = np.where(te)[0]
|
||||
for k, i in enumerate(te_idx):
|
||||
oof6.append({
|
||||
"true": y6[i], "pred": pr[k], "proba": {c: float(p6[k][j]) for j, c in enumerate(classes)},
|
||||
"tags": tags[i], "split_group": split_groups[i], "source_id": source_ids[i],
|
||||
})
|
||||
yt6 = [r["true"] for r in oof6]; yp6 = [r["pred"] for r in oof6]
|
||||
acc6 = accuracy_score(yt6, yp6)
|
||||
macro6 = f1_score(yt6, yp6, average="macro", zero_division=0)
|
||||
prec6, rec6, f16, sup6 = precision_recall_fscore_support(yt6, yp6, labels=routes, zero_division=0)
|
||||
fa6 = sum(1 for t, p in zip(yt6, yp6) if t != "action" and p == "action")
|
||||
ap6 = sum(1 for t, p in zip(yt6, yp6) if t == "action" and p == "action") / max(sum(1 for p in yp6 if p == "action"), 1)
|
||||
ar6 = sum(1 for t, p in zip(yt6, yp6) if t == "action" and p == "action") / max(sum(1 for t in yt6 if t == "action"), 1)
|
||||
print(f" acc={fmt_pct(acc6)} macroF1={ff(macro6)} actionP={ff(ap6)} actionR={ff(ar6)} FA={fa6} ({fmt_pct(fa6/len(yt6))})")
|
||||
for i, r in enumerate(routes):
|
||||
print(f" {r:<12} P={ff(prec6[i])} R={ff(rec6[i])} F1={ff(f16[i])} n={int(sup6[i])}")
|
||||
|
||||
# ── 9. Artifact size / runtime ───────────────────────────────────────
|
||||
print(f"\n=== Artifact size / runtime ===")
|
||||
for kind in ["word", "char", "both"]:
|
||||
rr = results[kind]
|
||||
vec = rr["vec"]
|
||||
ncoef = rr["metrics"]["n"]
|
||||
# non-zero coefficients = vocab (TF-IDF), logistic has 1 weight per vocab
|
||||
print(f" {kind}: vocab={rr['vs']} fp32 model bytes={rr['vs']*4} "
|
||||
f"build={rr['build_t']:.3f}s")
|
||||
|
||||
# done
|
||||
print("\nDone.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 19 tokenizers, derived from the development corpus only.
|
||||
|
||||
A. CharVocab — codepoint ids over the dev corpus (deterministic order)
|
||||
B. BpeVocab2048 — byte-level BPE, vocab ~2048, trained on dev corpus only
|
||||
|
||||
Records for §4 of the brief: vocab size, OOV behaviour, serialized tokenizer size.
|
||||
"""
|
||||
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
from tokenizers import Tokenizer
|
||||
from tokenizers.decoders import ByteLevel as ByteLevelDecoder
|
||||
from tokenizers.models import BPE
|
||||
from tokenizers.pre_tokenizers import ByteLevel as ByteLevelPreTokenizer
|
||||
from tokenizers.trainers import BpeTrainer
|
||||
|
||||
|
||||
class CharVocab:
|
||||
"""Codepoint ids from the dev corpus, sorted by codepoint value."""
|
||||
|
||||
def __init__(self, texts):
|
||||
chars = set()
|
||||
for t in texts:
|
||||
chars.update(t)
|
||||
self.id_to_char = sorted(chars)
|
||||
self.char_to_id = {c: i + 1 for i, c in enumerate(self.id_to_char)} # 0 = PAD
|
||||
self.pad = 0
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return len(self.id_to_char) + 1
|
||||
|
||||
def encode(self, text, max_len):
|
||||
ids = [self.char_to_id.get(c, 0) for c in text] # 0 doubles as UNK/OOV
|
||||
return ids[:max_len]
|
||||
|
||||
|
||||
class BpeVocab:
|
||||
"""Byte-level BPE, trained only on the strings it is given."""
|
||||
|
||||
def __init__(self, texts, vocab_size=2048, sep="▁"):
|
||||
self.tok = Tokenizer(BPE())
|
||||
self.tok.pre_tokenizer = ByteLevelPreTokenizer(trim_offsets=False)
|
||||
self.tok.decoder = ByteLevelDecoder()
|
||||
trainer = BpeTrainer(vocab_size=vocab_size, special_tokens=["[PAD]"],
|
||||
show_progress=False)
|
||||
# train on the corpus *strings*, byte-level BPE handles all codepoints
|
||||
self.tok.train_from_iterator(texts, trainer=trainer)
|
||||
self.pad_id = self.tok.token_to_id("[PAD]")
|
||||
self._vocab = self.tok.get_vocab()
|
||||
self._n = len(self._vocab)
|
||||
|
||||
@property
|
||||
def size(self):
|
||||
return self._n
|
||||
|
||||
def encode(self, text):
|
||||
return self.tok.encode(text).ids
|
||||
|
||||
def serialized_bytes(self):
|
||||
# measure the serialized tokenizer size on disk
|
||||
import os
|
||||
d = self.tok.to_str()
|
||||
return len(d.encode("utf-8"))
|
||||
|
||||
|
||||
def normalize_match_text(s: str) -> str:
|
||||
"""NFKC → lowercase → collapse whitespace. Punctuation kept."""
|
||||
out = unicodedata.normalize("NFKC", s).strip().lower()
|
||||
out = re.sub(r"\s+", " ", out)
|
||||
return out
|
||||
|
||||
|
||||
def strip_punct(text: str) -> str:
|
||||
"""Remove safe punctuation from an already-normalized text."""
|
||||
t = re.sub(r"[^\w\s]", " ", text)
|
||||
t = re.sub(r"\s+", " ", t).strip()
|
||||
return t
|
||||
@@ -0,0 +1,640 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 19: tiny sequence-sensitive pragmatics specialists (binary action gate)
|
||||
=============================================================================
|
||||
|
||||
A narrow binary specialist: executable request vs semantically similar
|
||||
non-executable utterance, generalizing across surface-generator families.
|
||||
|
||||
Commands
|
||||
grouped — grouped semantic CV (existing cv_fold), all arch/sizes, saves
|
||||
per-config per-fold model checkpoints + OOF proba per variant
|
||||
metrics — aggregate saved grouped-CV results into the report tables
|
||||
(binary metrics, threshold curves, pair ordering, stress)
|
||||
lfo — leave-generator-out. All sizes on capability_question; the
|
||||
other present families for the leading config only.
|
||||
e5baseline— frozen-e5 logistic + MLP baselines: grouped, cap-Q LOFO, pair
|
||||
ordering (stress flagged NA — no re-embed on this box)
|
||||
runtime — params / sizes / latency / tokenization for each candidate
|
||||
|
||||
Primary metrics (brief §2): cap-Q LOFO FA, pair ordering acc, pair margin,
|
||||
grouped P/R, voice stress, fold variance. Aggregate accuracy is secondary.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import resource
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import warnings
|
||||
from collections import Counter
|
||||
|
||||
import numpy as np
|
||||
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from slice19_bpe import BpeVocab, CharVocab, normalize_match_text, strip_punct
|
||||
|
||||
EMBEDDING_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
RESULTS_DIR = "/tmp/mvn-s19"
|
||||
SEED = 42
|
||||
MAX_CHAR = 64
|
||||
MAX_BPE = 24
|
||||
BPE_VOCAB = 2048
|
||||
|
||||
PRESENT_FAMILIES = [
|
||||
"polite_request",
|
||||
"modal_request",
|
||||
"first_person_request",
|
||||
"reordered_target",
|
||||
"capability_question",
|
||||
"question",
|
||||
]
|
||||
ABSENT_FAMILIES = ["negation", "reported_speech", "quotation", "hypothetical"]
|
||||
|
||||
FAMILY_PRIORITY = [
|
||||
"capability_question", "question", "first_person_request",
|
||||
"modal_request", "polite_request", "reordered_target", "direct_imperative",
|
||||
]
|
||||
|
||||
DEVICES = [
|
||||
"свет", "люстру", "люстра", "жалюзи", "вытяжку", "вытяжка",
|
||||
"вентилятор", "кондиционер", "телевизор", "лампу", "лампа",
|
||||
"музыку", "музыка", "плеер", "колонку", "колонки", "чайник",
|
||||
"бойлер", "обогреватель", "пылесос", "пылесосом", "пол",
|
||||
"поливалки", "полив", "арка", "шторы", "штору",
|
||||
"динамики", "дверь", "двери", "замок", "гараж", "ворота",
|
||||
"кофе", "пасту", "зубы", "крючки", "лаймо", "куртку",
|
||||
"будильник", "таймер", "напоминание", "расписание",
|
||||
]
|
||||
|
||||
ARCH_CONFIGS = {
|
||||
"char_cnn": ["tiny", "medium"],
|
||||
"bigru": ["tiny", "medium", "large"],
|
||||
"tiny_transformer": ["small", "medium"],
|
||||
}
|
||||
|
||||
TRAIN_HYPER = {
|
||||
"char_cnn": dict(epochs=20, lr=1e-3, bs=64, clip=None),
|
||||
"bigru": dict(epochs=15, lr=5e-4, bs=64, clip=1.0),
|
||||
"tiny_transformer": dict(epochs=25, lr=5e-4, bs=64, clip=1.0),
|
||||
}
|
||||
|
||||
VARIANTS = ["orig", "nofinal", "strip"]
|
||||
|
||||
|
||||
# ─── data ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def load_dev():
|
||||
with open(EMBEDDING_PATH) as f:
|
||||
data = json.load(f)
|
||||
rows = [e for e in data["examples"] if e["dev_pool"]]
|
||||
out = []
|
||||
for e in rows:
|
||||
n_text = normalize_match_text(e["text"])
|
||||
out.append({
|
||||
"text_orig": n_text,
|
||||
"text_nofinal": re.sub(r"[?.!,;:]+$", "", n_text),
|
||||
"text_strip": strip_punct(n_text),
|
||||
"route": e["route"],
|
||||
"y": 1 if e["route"] == "action" else 0,
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"tags": set(e.get("tags", [])),
|
||||
"source_id": e["source_id"],
|
||||
"emb": np.asarray(e["embedding"], dtype=np.float32),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def family_of(tags):
|
||||
for f in FAMILY_PRIORITY:
|
||||
if f in tags:
|
||||
return f
|
||||
return None
|
||||
|
||||
|
||||
def build_pairs(rows, texts):
|
||||
"""Capability-question vs action-by-shared-object pairs (eval only)."""
|
||||
def object_nouns(t):
|
||||
found = set()
|
||||
tl = t.lower()
|
||||
for d in DEVICES:
|
||||
if re.search(r"\b" + re.escape(d), tl):
|
||||
found.add(d)
|
||||
return found
|
||||
|
||||
pairs = []
|
||||
for cidx, r in enumerate(rows):
|
||||
if "capability_question" not in r["tags"]:
|
||||
continue
|
||||
c_obj = object_nouns(texts[cidx])
|
||||
if not c_obj:
|
||||
continue
|
||||
dom = re.search(r"kq-cap-([^-]+)", r["source_id"])
|
||||
dom = dom.group(1) if dom else None
|
||||
for aidx, ra in enumerate(rows):
|
||||
if ra["y"] != 1:
|
||||
continue
|
||||
a_obj = object_nouns(texts[aidx])
|
||||
if not (c_obj & a_obj):
|
||||
continue
|
||||
a_dom = re.search(r"^([a-z]+)-", ra["source_id"])
|
||||
a_dom = a_dom.group(1) if a_dom else None
|
||||
if dom and a_dom and dom != a_dom:
|
||||
continue
|
||||
pairs.append((cidx, aidx))
|
||||
return pairs
|
||||
|
||||
|
||||
def pair_metrics(pairs, proba):
|
||||
if not pairs:
|
||||
return {"pairs": 0}
|
||||
margins = []
|
||||
ties = 0
|
||||
order = 0
|
||||
rev = 0
|
||||
for cidx, aidx in pairs:
|
||||
pc, pa = proba[cidx], proba[aidx]
|
||||
margins.append(pa - pc)
|
||||
if pa > pc:
|
||||
order += 1
|
||||
elif pa == pc:
|
||||
ties += 1
|
||||
else:
|
||||
rev += 1
|
||||
m = np.array(margins)
|
||||
return {
|
||||
"pairs": len(pairs),
|
||||
"ordering_acc": order / len(pairs),
|
||||
"mean_margin": float(m.mean()),
|
||||
"median_margin": float(np.median(m)),
|
||||
"ties": ties,
|
||||
"reversed": rev,
|
||||
}
|
||||
|
||||
|
||||
# ─── tokenizers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def build_tokenizers(rows):
|
||||
nat_texts = [r["text_orig"] for r in rows]
|
||||
char_vocab = CharVocab(nat_texts)
|
||||
bpe = BpeVocab(nat_texts, vocab_size=BPE_VOCAB)
|
||||
return char_vocab, bpe
|
||||
|
||||
|
||||
def encode_all(rows, tokenizer, kind):
|
||||
"""Return dict variant -> (N, max_len) int64 array."""
|
||||
max_len = MAX_CHAR if kind == "char" else MAX_BPE
|
||||
out = {}
|
||||
for v in VARIANTS:
|
||||
arr = np.zeros((len(rows), max_len), dtype=np.int64)
|
||||
for i, r in enumerate(rows):
|
||||
if kind == "char":
|
||||
ids = tokenizer.encode(r[f"text_{v}"], max_len)
|
||||
arr[i, :len(ids)] = ids
|
||||
else:
|
||||
ids = tokenizer.encode(r[f"text_{v}"])[:max_len]
|
||||
arr[i, :len(ids)] = ids
|
||||
out[v] = arr
|
||||
return out
|
||||
|
||||
|
||||
# ─── training ───────────────────────────────────────────────────────────────
|
||||
|
||||
def build_model(arch, size, vocab_size, max_len):
|
||||
import torch
|
||||
from slice19_models import CharCNN, BiGRU, TinyTransformer
|
||||
if arch == "char_cnn":
|
||||
c = (dict(embed_dim=32, filters=64, widths=[3, 4, 5]) if size == "tiny"
|
||||
else dict(embed_dim=64, filters=160, widths=[2, 3, 4, 5]))
|
||||
return CharCNN(vocab_size, c["embed_dim"], c["filters"], c["widths"])
|
||||
if arch == "bigru":
|
||||
c = (dict(embed_dim=64, hidden=64) if size == "tiny" else
|
||||
(dict(embed_dim=128, hidden=128) if size == "medium" else
|
||||
dict(embed_dim=256, hidden=256)))
|
||||
return BiGRU(vocab_size, c["embed_dim"], c["hidden"])
|
||||
c = (dict(d_model=128, n_layers=2, n_heads=4) if size == "small" else
|
||||
dict(d_model=192, n_layers=4, n_heads=4))
|
||||
return TinyTransformer(vocab_size, c["d_model"], c["n_layers"], c["n_heads"],
|
||||
max_len=max_len)
|
||||
|
||||
|
||||
def train_binary(X, y, arch, size, vocab_size, seed_offset=0, log=False):
|
||||
import torch
|
||||
torch.manual_seed(SEED + seed_offset)
|
||||
np.random.seed(SEED + seed_offset)
|
||||
Xt = torch.from_numpy(X)
|
||||
yt = torch.from_numpy(y.astype(np.float32))
|
||||
model = build_model(arch, size, vocab_size, X.shape[1])
|
||||
h = TRAIN_HYPER[arch]
|
||||
opt = torch.optim.AdamW(model.parameters(), lr=h["lr"], weight_decay=1e-4)
|
||||
lossf = torch.nn.BCEWithLogitsLoss()
|
||||
n = X.shape[0]
|
||||
model.train()
|
||||
t0 = time.time()
|
||||
for epoch in range(h["epochs"]):
|
||||
perm = torch.randperm(n)
|
||||
running = 0.0
|
||||
n_b = 0
|
||||
for start in range(0, n, h["bs"]):
|
||||
idx = perm[start:start + h["bs"]]
|
||||
xb = Xt[idx]
|
||||
if xb.dim() == 1:
|
||||
xb = xb.unsqueeze(0)
|
||||
logits = model(xb)
|
||||
loss = lossf(logits, yt[idx])
|
||||
opt.zero_grad()
|
||||
loss.backward()
|
||||
if h["clip"]:
|
||||
torch.nn.utils.clip_grad_norm_(model.parameters(), h["clip"])
|
||||
opt.step()
|
||||
running += float(loss)
|
||||
n_b += 1
|
||||
if log and (epoch + 1) % 5 == 0:
|
||||
print(f" epoch {epoch+1}/{h['epochs']} loss {running/max(n_b,1):.4f}")
|
||||
return model, time.time() - t0
|
||||
|
||||
|
||||
def predict_proba(model, X, bs=256):
|
||||
import torch
|
||||
model.eval()
|
||||
out = []
|
||||
with torch.no_grad():
|
||||
Xt = torch.from_numpy(X)
|
||||
for start in range(0, X.shape[0], bs):
|
||||
xb = Xt[start:start + bs]
|
||||
if xb.dim() == 1:
|
||||
xb = xb.unsqueeze(0)
|
||||
logits = model(xb)
|
||||
out.append(torch.sigmoid(logits).numpy())
|
||||
return np.concatenate(out)
|
||||
|
||||
|
||||
# ─── metrics helpers ────────────────────────────────────────────────────────
|
||||
|
||||
def binary_metrics(yt, proba, thr=0.5):
|
||||
yp = (proba >= thr).astype(int)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
from sklearn.metrics import roc_auc_score, average_precision_score
|
||||
roc = roc_auc_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
pr = average_precision_score(yt, proba) if len(np.unique(yt)) > 1 else 0.0
|
||||
return {
|
||||
"n": int(len(yt)), "tp": tp, "fp": fp, "fn": fn,
|
||||
"P": tp / max(tp + fp, 1), "R": tp / max(tp + fn, 1),
|
||||
"FA": fp, "FA_rate": fp / max(len(yt), 1),
|
||||
"ROC_AUC": float(roc), "PR_AUC": float(pr),
|
||||
}
|
||||
|
||||
|
||||
def threshold_curve(yt, proba, thr_grid):
|
||||
rows = []
|
||||
for thr in thr_grid:
|
||||
yp = (proba >= thr).astype(int)
|
||||
tp = int(((yt == 1) & (yp == 1)).sum())
|
||||
fp = int(((yt == 0) & (yp == 1)).sum())
|
||||
fn = int(((yt == 1) & (yp == 0)).sum())
|
||||
P = tp / max(tp + fp, 1)
|
||||
R = tp / max(tp + fn, 1)
|
||||
rows.append({
|
||||
"thr": round(float(thr), 4), "P": round(P, 4), "R": round(R, 4),
|
||||
"FA": fp, "FA_rate": round(fp / max(len(yt), 1), 4),
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def operating_points(rows, pair_proba, pairs):
|
||||
"""Report P>=0.95/0.98/0.99 points with pair separation at threshold."""
|
||||
res = {}
|
||||
for target in (0.95, 0.98, 0.99):
|
||||
pts = [r for r in rows if r["P"] >= target and r["R"] > 0.0]
|
||||
if not pts:
|
||||
res[str(target)] = None
|
||||
continue
|
||||
best = max(pts, key=lambda r: r["R"])
|
||||
# pair separation at that operating point
|
||||
sep = pair_sep_at(best["thr"], pair_proba, pairs)
|
||||
best = dict(best); best["pair_sep"] = round(sep, 4)
|
||||
res[str(target)] = best
|
||||
return res
|
||||
|
||||
|
||||
def pair_sep_at(thr, pair_proba, pairs):
|
||||
"""Fraction of pairs where action>=thr and cap<thr."""
|
||||
if not pairs:
|
||||
return 0.0
|
||||
ok = 0
|
||||
for cidx, aidx in pairs:
|
||||
if pair_proba[aidx] >= thr and pair_proba[cidx] < thr:
|
||||
ok += 1
|
||||
return ok / len(pairs)
|
||||
|
||||
|
||||
def fold_variance(fold_rows):
|
||||
return {
|
||||
"folds": [
|
||||
{
|
||||
"fold": fr["fold"],
|
||||
"ROC_AUC": fr["ROC_AUC"], "PR_AUC": fr["PR_AUC"],
|
||||
"P": fr["P"], "R": fr["R"], "FA": fr["FA"], "n": fr["n"],
|
||||
}
|
||||
for fr in fold_rows
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
# ─── subcommands ────────────────────────────────────────────────────────────
|
||||
|
||||
def _result_path():
|
||||
os.makedirs(RESULTS_DIR, exist_ok=True)
|
||||
return RESULTS_DIR
|
||||
|
||||
|
||||
def cmd_grouped(args):
|
||||
rows = load_dev()
|
||||
char_vocab, bpe = build_tokenizers(rows)
|
||||
y = np.array([r["y"] for r in rows])
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
X_char = encode_all(rows, char_vocab, "char")
|
||||
X_bpe = encode_all(rows, bpe, "bpe")
|
||||
tokenizers = {"char": char_vocab, "bpe": bpe}
|
||||
|
||||
os.makedirs(RESULTS_DIR, exist_ok=True)
|
||||
# save tokenizer metadata for reproducibility
|
||||
meta = {
|
||||
"char_vocab": char_vocab.size,
|
||||
"char_vocab_sample": char_vocab.id_to_char[:50],
|
||||
"bpe_vocab": bpe.size,
|
||||
"bpe_serialized_bytes": bpe.serialized_bytes(),
|
||||
"max_char": MAX_CHAR, "max_bpe": MAX_BPE,
|
||||
"n": len(rows),
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "corpus_meta.json"), "w") as f:
|
||||
json.dump(meta, f)
|
||||
|
||||
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
kind = kind_of[arch]
|
||||
X = X_char if kind == "char" else X_bpe
|
||||
for size in sizes:
|
||||
name = f"{arch}_{size}"
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "models"), exist_ok=True)
|
||||
probs = {v: np.zeros(len(rows)) for v in VARIANTS}
|
||||
fold_rows = []
|
||||
for fold in range(5):
|
||||
tr = folds != fold
|
||||
te = folds == fold
|
||||
Xtr = X["strip"][tr]
|
||||
model, secs = train_binary(Xtr, y[tr], arch, size,
|
||||
vocab_size=tokenizers[kind].size,
|
||||
seed_offset=fold, log=args.verbose)
|
||||
torch_models_dir = os.path.join(RESULTS_DIR, "models")
|
||||
import torch
|
||||
torch.save(model.state_dict(), os.path.join(torch_models_dir, f"{name}_fold{fold}.pt"))
|
||||
for v in VARIANTS:
|
||||
probs[v][te] = predict_proba(model, X[v][te])
|
||||
fold_m = binary_metrics(y[te], probs["strip"][te])
|
||||
fold_m["fold"] = fold
|
||||
fold_rows.append(fold_m)
|
||||
print(f" {name} fold {fold}: ROC={fold_m['ROC_AUC']:.3f} "
|
||||
f"PR={fold_m['PR_AUC']:.3f} P={fold_m['P']:.3f} R={fold_m['R']:.3f} "
|
||||
f"FA={fold_m['FA']} n={fold_m['n']} ({secs:.1f}s)")
|
||||
np.savez(os.path.join(RESULTS_DIR, f"{name}_probs.npz"),
|
||||
var_orig=probs["orig"], var_nofinal=probs["nofinal"],
|
||||
var_strip=probs["strip"])
|
||||
summary = binary_metrics(y, probs["strip"])
|
||||
print(f" {name} OOF: ROC={summary['ROC_AUC']:.3f} PR={summary['PR_AUC']:.3f} "
|
||||
f"P={summary['P']:.3f} R={summary['R']:.3f} FA={summary['FA']}")
|
||||
print("grouped done")
|
||||
|
||||
|
||||
def cmd_metrics(args):
|
||||
rows = load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
pairs = build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
print(f"pairs={len(pairs)}")
|
||||
|
||||
out = {}
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
for size in sizes:
|
||||
name = f"{arch}_{size}"
|
||||
fp = os.path.join(RESULTS_DIR, f"{name}_probs.npz")
|
||||
if not os.path.exists(fp):
|
||||
continue
|
||||
z = np.load(fp)
|
||||
entry = {"name": name, "arch": arch, "size": size}
|
||||
# OOF binary on strip variant (primary training input)
|
||||
entry["strip"] = binary_metrics(y, z["var_strip"])
|
||||
entry["pairs"] = {}
|
||||
entry["pairs"]["strip"] = pair_metrics(pairs, z["var_strip"])
|
||||
entry["pairs"]["orig"] = pair_metrics(pairs, z["var_orig"])
|
||||
entry["pairs"]["nofinal"] = pair_metrics(pairs, z["var_nofinal"])
|
||||
# stress: same OOF models, per-variant metrics
|
||||
entry["stress"] = {}
|
||||
for v in VARIANTS:
|
||||
p = z[f"var_{v}"]
|
||||
entry["stress"][v] = {
|
||||
"all_FA": binary_metrics(y, p)["FA_rate"],
|
||||
"capQ_FA": capq_fa(tags, y, p),
|
||||
"modal_recall": modal_recall(tags, y, p),
|
||||
}
|
||||
# threshold curve + operating points on strip
|
||||
entry["curve"] = threshold_curve(y, z["var_strip"],
|
||||
np.arange(0.30, 1.0, 0.02))
|
||||
entry["ops"] = operating_points(entry["curve"], z["var_strip"], pairs)
|
||||
out[name] = entry
|
||||
|
||||
with open(os.path.join(RESULTS_DIR, "metrics.json"), "w") as f:
|
||||
json.dump(out, f, indent=2, default=str)
|
||||
print(json.dumps(out, indent=2, default=str))
|
||||
|
||||
|
||||
def capq_fa(tags, y, proba):
|
||||
mask = np.array(["capability_question" in t for t in tags])
|
||||
if mask.sum() == 0:
|
||||
return 0.0
|
||||
sub = proba[mask]
|
||||
return float((sub >= 0.5).sum() / mask.sum())
|
||||
|
||||
|
||||
def modal_recall(tags, y, proba):
|
||||
mask = np.array([
|
||||
(y[i] == 1 and ("polite_request" in tags[i] or "modal_request" in tags[i]))
|
||||
for i in range(len(y))
|
||||
])
|
||||
if mask.sum() == 0:
|
||||
return 0.0
|
||||
sub = proba[mask]
|
||||
return float((sub >= 0.5).sum() / mask.sum())
|
||||
|
||||
|
||||
def cmd_lfo(args):
|
||||
rows = load_dev()
|
||||
char_vocab, bpe = build_tokenizers(rows)
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
X_char = encode_all(rows, char_vocab, "char")["strip"]
|
||||
X_bpe = encode_all(rows, bpe, "bpe")["strip"]
|
||||
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
|
||||
|
||||
results = {}
|
||||
tokenizers = {"char": char_vocab, "bpe": bpe}
|
||||
# capability_question LOFO for every config/size (the critical case)
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
X = X_char if kind_of[arch] == "char" else X_bpe
|
||||
src_idx = np.array(["capability_question" not in t for t in tags])
|
||||
tgt_idx = np.array(["capability_question" in t for t in tags])
|
||||
for size in sizes:
|
||||
model, _ = train_binary(X[src_idx], y[src_idx], arch, size,
|
||||
vocab_size=tokenizers[kind_of[arch]].size,
|
||||
seed_offset=17)
|
||||
p = predict_proba(model, X[tgt_idx])
|
||||
m = binary_metrics(y[tgt_idx], p)
|
||||
results[f"{arch}_{size}:capability_question"] = m
|
||||
print(f"LFO ability {arch}_{size}: cap rows={m['n']} "
|
||||
f"pos={int(y[tgt_idx].sum())} FA={m['FA']} FA_rate={m['FA_rate']:.3f} "
|
||||
f"P={m['P']:.3f} R={m['R']:.3f} acc={1-m['FA_rate']:.3f}")
|
||||
|
||||
# full family LOFO for the leading config per architecture
|
||||
leading = {"char_cnn": "char_cnn_medium", "bigru": "bigru_tiny",
|
||||
"tiny_transformer": "tiny_transformer_small"}
|
||||
for arch, name in leading.items():
|
||||
X = X_char if kind_of[arch] == "char" else X_bpe
|
||||
for fam in PRESENT_FAMILIES:
|
||||
src = np.array([fam not in t for t in tags])
|
||||
tgt = np.array([fam in t for t in tags])
|
||||
model, _ = train_binary(X[src], y[src], arch, name.split("_")[-1],
|
||||
vocab_size=tokenizers[kind_of[arch]].size,
|
||||
seed_offset=41)
|
||||
p = predict_proba(model, X[tgt])
|
||||
m = binary_metrics(y[tgt], p)
|
||||
results[f"{name}:{fam}"] = m
|
||||
print(f"LFO {fam}: {name} rows={m['n']} pos={int(y[tgt].sum())} "
|
||||
f"P={m['P']:.3f} R={m['R']:.3f} FA={m['FA']} acc={1-m['FA_rate']:.3f}")
|
||||
|
||||
with open(os.path.join(RESULTS_DIR, "lfo.json"), "w") as f:
|
||||
json.dump(results, f, indent=2, default=str)
|
||||
print("lfo done")
|
||||
|
||||
|
||||
def cmd_e5baseline(args):
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
from sklearn.neural_network import MLPClassifier
|
||||
rows = load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
X = np.vstack([r["emb"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
pairs = build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
|
||||
out = {}
|
||||
for model_name, model, extra in [
|
||||
("e5_linear", LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42), {}),
|
||||
("e5_mlp_h32", MLPClassifier(hidden_layer_sizes=(32,), activation="relu",
|
||||
solver="adam", alpha=0.01, random_state=42,
|
||||
max_iter=800, early_stopping=True,
|
||||
validation_fraction=0.15, n_iter_no_change=10), {}),
|
||||
]:
|
||||
proba = np.zeros(len(rows))
|
||||
for fold in range(5):
|
||||
tr = folds != fold
|
||||
te = folds == fold
|
||||
m2 = type(model)(**{k: v for k, v in model.get_params().items()})
|
||||
m2.fit(X[tr], y[tr])
|
||||
proba[te] = m2.predict_proba(X[te])[:, 1]
|
||||
entry = {
|
||||
"grouped": binary_metrics(y, proba),
|
||||
"pairs": pair_metrics(pairs, proba),
|
||||
}
|
||||
# cap-Q leave-generator-out (train without the family)
|
||||
src = np.array(["capability_question" not in t for t in tags])
|
||||
tgt = np.array(["capability_question" in t for t in tags])
|
||||
m3 = LogisticRegression(C=1.0, max_iter=2000, solver="lbfgs", random_state=42) \
|
||||
if model_name == "e5_linear" else \
|
||||
MLPClassifier(hidden_layer_sizes=(32,), alpha=0.01, random_state=42, max_iter=800)
|
||||
m3.fit(X[src], y[src])
|
||||
p3 = m3.predict_proba(X[tgt])[:, 1]
|
||||
entry["capq_lofo"] = binary_metrics(y[tgt], p3)
|
||||
entry["stress"] = "NA (no re-embed on this box)"
|
||||
out[model_name] = entry
|
||||
print(f"{model_name}: grouped FA_rate={entry['grouped']['FA_rate']:.4f} "
|
||||
f"PR={entry['grouped']['PR_AUC']:.3f} capQ_LOFO_FA_rate={entry['capq_lofo']['FA_rate']:.4f} "
|
||||
f"pairs={entry['pairs']['ordering_acc']:.3f}")
|
||||
with open(os.path.join(RESULTS_DIR, "e5baseline.json"), "w") as f:
|
||||
json.dump(out, f, indent=2, default=str)
|
||||
print("e5baseline done")
|
||||
|
||||
|
||||
def cmd_runtime(args):
|
||||
import torch
|
||||
rows = load_dev()
|
||||
char_vocab, bpe = build_tokenizers(rows)
|
||||
X_char = encode_all(rows, char_vocab, "char")["strip"]
|
||||
X_bpe = encode_all(rows, bpe, "bpe")["strip"]
|
||||
kind_of = {"char_cnn": "char", "bigru": "bpe", "tiny_transformer": "bpe"}
|
||||
report = {}
|
||||
for arch, sizes in ARCH_CONFIGS.items():
|
||||
X = X_char if kind_of[arch] == "char" else X_bpe
|
||||
for size in sizes:
|
||||
name = f"{arch}_{size}"
|
||||
model = build_model(arch, size, int(X.max()) + 1, X.shape[1])
|
||||
n_params = sum(p.numel() for p in model.parameters())
|
||||
fp32 = n_params * 4
|
||||
int8 = n_params
|
||||
model.eval()
|
||||
# warmup + latency (batch-1, eval mode)
|
||||
xb = torch.from_numpy(X[:1])
|
||||
with torch.no_grad():
|
||||
for _ in range(20):
|
||||
model(xb)
|
||||
# tokenization latency
|
||||
if kind_of[arch] == "char":
|
||||
t0 = time.perf_counter()
|
||||
for r in rows[:1000]:
|
||||
char_vocab.encode(r["text_strip"], MAX_CHAR)
|
||||
tl = (time.perf_counter() - t0) / 1000
|
||||
else:
|
||||
t0 = time.perf_counter()
|
||||
for r in rows[:1000]:
|
||||
bpe.encode(r["text_strip"])
|
||||
tl = (time.perf_counter() - t0) / 1000
|
||||
lat = []
|
||||
for _ in range(300):
|
||||
t0 = time.perf_counter()
|
||||
model(xb)
|
||||
lat.append(time.perf_counter() - t0)
|
||||
lat = np.array(lat) * 1e6
|
||||
report[name] = {
|
||||
"params": n_params, "fp32_bytes": fp32, "int8_bytes": int8,
|
||||
"latency_us_mean": float(lat.mean()), "latency_us_p50": float(np.median(lat)),
|
||||
"latency_us_p95": float(np.percentile(lat, 95)),
|
||||
"throughput_b1": round(1e6 / float(lat.mean()), 1),
|
||||
"tok_us": round(tl * 1e6, 1),
|
||||
"tokenizer": "char" if kind_of[arch] == "char" else "bpe",
|
||||
}
|
||||
print(f"{name}: {n_params} params fp32={fp32/1024:.0f}KiB "
|
||||
f"lat={lat.mean():.0f}us tok={tl*1e6:.1f}us")
|
||||
with open(os.path.join(RESULTS_DIR, "runtime.json"), "w") as f:
|
||||
json.dump(report, f, indent=2)
|
||||
print("runtime done")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("cmd", choices=["grouped", "metrics", "lfo", "e5baseline", "runtime"])
|
||||
ap.add_argument("--verbose", action="store_true")
|
||||
args = ap.parse_args()
|
||||
t0 = time.time()
|
||||
globals()[f"cmd_{args.cmd}"](args)
|
||||
print(f"elapsed {time.time()-t0:.1f}s")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 19 model zoo: three genuinely sequence-sensitive tiny models, trained
|
||||
from scratch on Maven's narrow binary pragmatics task.
|
||||
|
||||
A. CharCNN — codepoint ids → char embedding → parallel small 1D convs
|
||||
(several kernel widths) → global max-pool → linear head
|
||||
B. BiGRU — subword ids → token embedding → 1-layer BiGRU →
|
||||
maxpool[final] → linear head
|
||||
C. TinyTransformer — subword ids → token embedding + sine position →
|
||||
N self-attention encoder blocks (heads, FFN 4x, PreNorm) →
|
||||
CLS → linear head
|
||||
|
||||
All expose :forward(ids) returning the binary logit, plus .n_params().
|
||||
Deterministic: everything is plain torch ops.
|
||||
"""
|
||||
|
||||
import math
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
class CharCNN(nn.Module):
|
||||
def __init__(self, vocab_size, embed_dim, filters, widths, pad_idx=0, dropout=0.3):
|
||||
super().__init__()
|
||||
self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
|
||||
self.convs = nn.ModuleList([
|
||||
nn.Conv1d(embed_dim, filters, k, padding=(k - 1) // 2)
|
||||
for k in widths
|
||||
])
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
self.head = nn.Linear(filters * len(widths), 1)
|
||||
|
||||
def forward(self, ids):
|
||||
# ids: (B, T)
|
||||
x = self.embed(ids).transpose(1, 2) # (B, D, T)
|
||||
hiddens = [F.relu(conv(x)) for conv in self.convs] # each (B, F, T)
|
||||
pooled = torch.cat([h.max(dim=2).values for h in hiddens], dim=1) # (B, F*W)
|
||||
return self.head(self.dropout(pooled)).squeeze(-1)
|
||||
|
||||
def n_params(self):
|
||||
return sum(p.numel() for p in self.parameters())
|
||||
|
||||
|
||||
class BiGRU(nn.Module):
|
||||
def __init__(self, vocab_size, embed_dim, hidden, pad_idx=0, dropout=0.3):
|
||||
super().__init__()
|
||||
self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=pad_idx)
|
||||
self.encoder = nn.GRU(embed_dim, hidden, num_layers=1, bidirectional=True,
|
||||
batch_first=True)
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
self.head = nn.Linear(hidden * 2, 1)
|
||||
|
||||
def forward(self, ids):
|
||||
mask = (ids != 0).float() # (B, T)
|
||||
x = self.embed(ids)
|
||||
lens = mask.sum(dim=1).clamp(min=1).long()
|
||||
x_p = nn.utils.rnn.pack_padded_sequence(x, lens.cpu(), batch_first=True,
|
||||
enforce_sorted=False)
|
||||
out, _ = self.encoder(x_p)
|
||||
out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True,
|
||||
total_length=mask.size(1))
|
||||
out = out * mask.unsqueeze(-1)
|
||||
maxed = out.max(dim=1).values # (B, 2H)
|
||||
return self.head(self.dropout(maxed)).squeeze(-1)
|
||||
|
||||
def n_params(self):
|
||||
return sum(p.numel() for p in self.parameters())
|
||||
|
||||
|
||||
class TinyTransformer(nn.Module):
|
||||
def __init__(self, vocab_size, d_model, n_layers, n_heads, ff_mult=4,
|
||||
max_len=64, pad_idx=0, dropout=0.1):
|
||||
super().__init__()
|
||||
self.d_model = d_model
|
||||
self.embed = nn.Embedding(vocab_size, d_model, padding_idx=pad_idx)
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
self.pos = nn.Parameter(torch.empty(1, max_len, d_model))
|
||||
nn.init.normal_(self.pos, std=0.02)
|
||||
blocks = []
|
||||
for _ in range(n_layers):
|
||||
blocks.append(TransformerBlock(d_model, n_heads, ff_mult, dropout))
|
||||
self.blocks = nn.ModuleList(blocks)
|
||||
self.ln_out = nn.LayerNorm(d_model)
|
||||
self.head = nn.Linear(d_model, 1)
|
||||
|
||||
def forward(self, ids):
|
||||
B, T = ids.shape
|
||||
mask = (ids != 0)
|
||||
x = self.embed(ids) * math.sqrt(self.d_model) + self.pos[:, :T, :]
|
||||
x = self.dropout(x)
|
||||
for blk in self.blocks:
|
||||
x = blk(x, mask)
|
||||
x = self.ln_out(x)
|
||||
pooled = x.masked_fill(~mask.unsqueeze(-1), float("-inf")).max(dim=1).values
|
||||
return self.head(pooled).squeeze(-1)
|
||||
|
||||
def n_params(self):
|
||||
return sum(p.numel() for p in self.parameters())
|
||||
|
||||
|
||||
class TransformerBlock(nn.Module):
|
||||
def __init__(self, d_model, n_heads, ff_mult, dropout):
|
||||
super().__init__()
|
||||
self.attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout,
|
||||
batch_first=True)
|
||||
self.ln1 = nn.LayerNorm(d_model)
|
||||
self.ff = nn.Sequential(
|
||||
nn.Linear(d_model, d_model * ff_mult),
|
||||
nn.GELU(),
|
||||
nn.Linear(d_model * ff_mult, d_model),
|
||||
)
|
||||
self.ln2 = nn.LayerNorm(d_model)
|
||||
self.dropout = nn.Dropout(dropout)
|
||||
|
||||
def forward(self, x, mask):
|
||||
# mask: (B, T) bool of non-pad; MultiheadAttention uses key_padding_mask
|
||||
h = self.ln1(x)
|
||||
h = self.attn(h, h, h, key_padding_mask=~mask,
|
||||
need_weights=False, is_causal=False)[0]
|
||||
x = x + self.dropout(h)
|
||||
h = self.ln2(x)
|
||||
x = x + self.dropout(self.ff(h))
|
||||
return x
|
||||
|
||||
|
||||
# ─── Sizes ladder ───────────────────────────────────────────────────────────
|
||||
|
||||
def make_model(arch, size, char_vocab, bpe_vocab):
|
||||
if arch == "char_cnn":
|
||||
configs = {
|
||||
"tiny": dict(embed_dim=32, filters=64, widths=[3, 4, 5]),
|
||||
"medium": dict(embed_dim=64, filters=160, widths=[2, 3, 4, 5]),
|
||||
}
|
||||
c = configs[size]
|
||||
return CharCNN(char_vocab, c["embed_dim"], c["filters"], c["widths"])
|
||||
if arch == "bigru":
|
||||
configs = {
|
||||
"tiny": dict(embed_dim=64, hidden=64),
|
||||
"medium": dict(embed_dim=128, hidden=128),
|
||||
"large": dict(embed_dim=256, hidden=256),
|
||||
}
|
||||
c = configs[size]
|
||||
return BiGRU(bpe_vocab, c["embed_dim"], c["hidden"])
|
||||
if arch == "tiny_transformer":
|
||||
configs = {
|
||||
"small": dict(d_model=128, n_layers=2, n_heads=4),
|
||||
"medium": dict(d_model=192, n_layers=4, n_heads=4),
|
||||
}
|
||||
c = configs[size]
|
||||
return TinyTransformer(bpe_vocab, c["d_model"], c["n_layers"], c["n_heads"])
|
||||
raise ValueError(arch)
|
||||
|
||||
|
||||
def n_params_of(arch, size, char_vocab, bpe_vocab):
|
||||
return make_model(arch, size, char_vocab, bpe_vocab).n_params()
|
||||
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 20 tokenizer audit (§2 of the brief) and corpus sequence-length
|
||||
statistics (§3). Runs before any training. If the audit shows catastrophic
|
||||
Cyrillic / mixed-identifier loss it is the gate to stop.
|
||||
|
||||
Loads the frozen dev corpus exactly like slice 19 (same normalization),
|
||||
so regime A (natural text) is `text_orig` and regime B (punct-stripped) is
|
||||
`text_strip`.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import statistics
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
import slice19_main as s19
|
||||
|
||||
RESULTS_DIR = "/tmp/mvn-s20"
|
||||
MODEL_DIR = ("/tmp/mvn-s20/hf-cache/models--cointegrated--rubert-tiny/"
|
||||
"snapshots/5441c5ea8026d4f6d7505ec004845409f1259fb1")
|
||||
|
||||
MIXED_CYRILLIC_LATIN = re.compile(r"[а-яёА-ЯЁ]+[a-zA-Z]+|[a-zA-Z]+[а-яёА-ЯЁ]+")
|
||||
HAS_CYRILLIC = re.compile(r"[а-яёА-ЯЁ]")
|
||||
HAS_LATIN = re.compile(r"[a-zA-Z]")
|
||||
NUMERIC = re.compile(r"[0-9]")
|
||||
TOKEN_RE = re.compile(r"[^\W\d_]+", re.UNICODE)
|
||||
|
||||
SAMPLES = [
|
||||
"выключи свет в спальне пожалуйста",
|
||||
"turn off the lights",
|
||||
"перезапусти сервис mavend",
|
||||
"что такое Nexus",
|
||||
"включи телевизор, пожалуйста",
|
||||
"как дела у Мэйвен",
|
||||
"поставь таймер на 5 минут",
|
||||
"кто такой Home Assistant",
|
||||
"открой настройки устройства ha_cam_12",
|
||||
"Покажи статус сервера Proxmox",
|
||||
"аутентифицируй на сайте 2fa.ru",
|
||||
"сообщи погоду завтра в 18:30",
|
||||
]
|
||||
|
||||
|
||||
def is_toolish(word):
|
||||
# Maven sibling service names / HA-like identifiers: mixed case, digits,
|
||||
# underscores, or short Latin words that are not in the vocab as whole
|
||||
# words. Rough heuristic for the fragmentation probe.
|
||||
return bool(re.search(r"[A-Z0-9_/.-]", word))
|
||||
|
||||
|
||||
def main():
|
||||
os.makedirs(RESULTS_DIR, exist_ok=True)
|
||||
tok = AutoTokenizer.from_pretrained(MODEL_DIR)
|
||||
rows = s19.load_dev()
|
||||
texts = {
|
||||
"orig": [r["text_orig"] for r in rows],
|
||||
"nofinal": [r["text_nofinal"] for r in rows],
|
||||
"strip": [r["text_strip"] for r in rows],
|
||||
}
|
||||
vv = tok.vocab_size
|
||||
unk = tok.unk_token_id
|
||||
|
||||
rep = {"model": "cointegrated/rubert-tiny",
|
||||
"sha": "5441c5ea8026d4f6d7505ec004845409f1259fb1",
|
||||
"tokenizer": type(tok).__name__,
|
||||
"vocab_size": vv}
|
||||
|
||||
# per-character stats (natural texts)
|
||||
chars = [len(t) for t in texts["orig"]]
|
||||
rep["char_len"] = {
|
||||
"mean": round(statistics.mean(chars), 2),
|
||||
"p50": int(sorted(chars)[len(chars) // 2]),
|
||||
"p95": sorted(chars)[int(len(chars) * .95)],
|
||||
"p99": sorted(chars)[int(len(chars) * .99)],
|
||||
"max": max(chars),
|
||||
}
|
||||
|
||||
for view in ("orig", "strip"):
|
||||
ids = tok(texts[view], add_special_tokens=True, padding=False,
|
||||
truncation=False)["input_ids"]
|
||||
lens = [len(x) for x in ids]
|
||||
n_tok = sum(lens)
|
||||
n_unk = sum(x.count(unk) for x in ids)
|
||||
n_chars = sum(len(t) for t in texts[view])
|
||||
rep[view] = {
|
||||
"tokens_per_utt_mean": round(n_tok / len(rows), 2),
|
||||
"tokens_per_char": round(n_tok / max(n_chars, 1), 4),
|
||||
"unk_count": n_unk,
|
||||
"unk_rate": round(n_unk / max(n_tok, 1), 5),
|
||||
"seq_len_p50": int(sorted(lens)[len(lens) // 2]),
|
||||
"seq_len_p90": sorted(lens)[int(len(lens) * .90)],
|
||||
"seq_len_p95": sorted(lens)[int(len(lens) * .95)],
|
||||
"seq_len_p99": sorted(lens)[int(len(lens) * .99)],
|
||||
"seq_len_max": max(lens),
|
||||
"above_96": sum(1 for x in lens if x > 96),
|
||||
"above_128": sum(1 for x in lens if x > 128),
|
||||
}
|
||||
rep[view]["p99_plus_margin"] = rep[view]["seq_len_p99"] + 6
|
||||
|
||||
# mixed Cyrillic/Latin behaviour over natural texts
|
||||
mixed_words = []
|
||||
for t in texts["orig"]:
|
||||
for w in t.split():
|
||||
if MIXED_CYRILLIC_LATIN.search(w):
|
||||
mixed_words.append(w)
|
||||
rep["mixed_cyr_lat_rows"] = len({w for w in mixed_words})
|
||||
rep["mixed_cyr_lat_stats"] = {"word_count": len(mixed_words),
|
||||
"unique_words": len(set(mixed_words))}
|
||||
|
||||
# entity/tool-name fragmentation: unique word-like tokens containing a digit
|
||||
# or underscore, or non-trivial Latin, and how many BPE/WordPiece pieces they
|
||||
# split into. Sample the extremes.
|
||||
fragments = []
|
||||
vocab = set(tok.get_vocab().keys())
|
||||
for t in texts["orig"]:
|
||||
# split into "clean" tokens (word chars + _ / digit boundaries)
|
||||
for w in re.findall(r"[A-Za-z0-9_]+\b", t):
|
||||
w2 = re.sub(r"_\b", "", w)
|
||||
if len(w2) < 3 or not is_toolish(w2):
|
||||
continue
|
||||
n_pieces = len(tok.tokenize(w2).replace("##", "_").rstrip())
|
||||
fragments.append((n_pieces, w2))
|
||||
frag = sorted(set(fragments))[-30:]
|
||||
rep["entity_fragment_examples"] = [
|
||||
{"token": w, "pieces": n} for n, w in frag
|
||||
]
|
||||
|
||||
# representative samples: full tokenization
|
||||
rep["samples"] = []
|
||||
for s in SAMPLES:
|
||||
e = tok(s, add_special_tokens=True, padding=False, truncation=False)
|
||||
rep["samples"].append({
|
||||
"text": s,
|
||||
"tokens": tok.convert_ids_to_tokens(e["input_ids"]),
|
||||
"pieces": len(e["input_ids"]),
|
||||
"unk": e["input_ids"].count(unk),
|
||||
})
|
||||
|
||||
with open(os.path.join(RESULTS_DIR, "tokenizer_audit.json"), "w") as f:
|
||||
json.dump(rep, f, indent=2, ensure_ascii=False)
|
||||
print(json.dumps({k: v for k, v in rep.items() if k not in ("samples",)}, indent=2, ensure_ascii=False))
|
||||
print("\n--- samples ---")
|
||||
for s in rep["samples"]:
|
||||
print(f'{s["pieces"]:>3} unk={s["unk"]} {s["text"]:50} ->'
|
||||
f' {" ".join(s["tokens"])}')
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,517 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 20: fine-tune cointegrated/rubert-tiny (11.9M, 3-layer BERT) end-to-end
|
||||
for the binary executable-intent boundary (action vs not_action) on the frozen
|
||||
v2 dev corpus, following the slice 20 brief.
|
||||
|
||||
Rules honoured:
|
||||
- full end-to-end fine-tuning, CLS-pooled native classification head
|
||||
- tokenizer used unchanged (audit in slice20_audit.py)
|
||||
- max length from corpus stats (p99+margin, cap 128): 25 here
|
||||
- narrow search: LR in {1e-5, 2e-5, 5e-5}, <= 6 epochs, early stop on a
|
||||
development (within-fold) split, best checkpoint restored
|
||||
- >= 3 seeds (42/17/7) for every config
|
||||
- grouped 5-fold CV reuse; cap-Q leave-generator-out as primary stress case
|
||||
- two input regimes: A = natural text (orig), B = punctuation-stripped (strip)
|
||||
|
||||
Artifacts under /tmp/mvn-s20/:
|
||||
pre/{regime}_ids.npy, _attn.npy tokenized corpus (all three views)
|
||||
oof/{regime}_{lr}_{seed}_probs.npz OOF probs per view (var_orig/nofinal/strip)
|
||||
oof/{regime}_{lr}_{seed}_metrics.json
|
||||
lfo/{regime}_{lr}_{seed}.json capability-Q LOFO (held-out family)
|
||||
results/summary.json
|
||||
models/{regime}_{lr}_{seed}_fold{i}.pt, lfo_{seed}.pt
|
||||
|
||||
CLI: slice20_pretrained.py {pre, grouped, lfo, metrics, runtime, onnx}
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch import nn
|
||||
from transformers import AutoConfig, AutoTokenizer
|
||||
from transformers import BertForSequenceClassification
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import slice19_main as s19
|
||||
|
||||
RESULTS_DIR = "/tmp/mvn-s20"
|
||||
MODEL_DIR = ("/tmp/mvn-s20/hf-cache/models--cointegrated--rubert-tiny/"
|
||||
"snapshots/5441c5ea8026d4f6d7505ec004845409f1259fb1")
|
||||
|
||||
MODEL_NAME = "cointegrated/rubert-tiny"
|
||||
MODEL_SHA = "5441c5ea8026d4f6d7505ec004845409f1259fb1"
|
||||
|
||||
REGIMES = ["A", "B"]
|
||||
LRS = [1e-5, 2e-5, 5e-5]
|
||||
SEEDS = [42, 17, 7]
|
||||
VIEWS = ["orig", "nofinal", "strip"]
|
||||
MAX_LEN = 25
|
||||
BATCH = 32
|
||||
MAX_EPOCHS = 4
|
||||
EARLY_STOP = 1 # patience in epochs on val PR-AUC
|
||||
VAL_FRACTION = 0.12
|
||||
WEIGHT_DECAY = 0.01
|
||||
|
||||
torch.set_num_threads(4)
|
||||
|
||||
|
||||
# ─── tokenizer / input preparation ──────────────────────────────────────────
|
||||
|
||||
def _load_tokenizer():
|
||||
return AutoTokenizer.from_pretrained(MODEL_DIR)
|
||||
|
||||
|
||||
def tokenize(texts, tok):
|
||||
e = tok(list(texts), add_special_tokens=True, padding="max_length",
|
||||
truncation=True, max_length=MAX_LEN)
|
||||
return np.array(e["input_ids"], np.int64), np.array(e["attention_mask"], np.int64)
|
||||
|
||||
|
||||
def cmd_pre(args):
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "pre"), exist_ok=True)
|
||||
tok = _load_tokenizer()
|
||||
rows = s19.load_dev()
|
||||
# distributed over all variants, all rows, both regimes
|
||||
for regime in REGIMES:
|
||||
train_view = "orig" if regime == "A" else "strip"
|
||||
tsrc = [r[f"text_{train_view}"] for r in rows]
|
||||
ids, attn = tokenize(tsrc, tok)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"), ids)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"), attn)
|
||||
# eval stress views tokenized under the same regime's vocab/format
|
||||
for v in VIEWS:
|
||||
ids_v, attn_v = tokenize([r[f"text_{v}"] for r in rows], tok)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_ids.npy"), ids_v)
|
||||
np.save(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_attn.npy"), attn_v)
|
||||
print(f"regime {regime} done, train view={train_view}")
|
||||
|
||||
|
||||
def _make_model():
|
||||
cfg = AutoConfig.from_pretrained(MODEL_DIR)
|
||||
cfg.num_labels = 1 # sine logit, BCEWithLogits — matches slice 19 head
|
||||
model = BertForSequenceClassification.from_pretrained(MODEL_DIR, config=cfg)
|
||||
return model
|
||||
|
||||
|
||||
# ─── training ───────────────────────────────────────────────────────────────
|
||||
|
||||
def _val_split(rows_idx, y, seed):
|
||||
rng = np.random.RandomState(seed)
|
||||
idx = rng.permutation(rows_idx)
|
||||
# stratified: keep the y-ratio in both parts
|
||||
pos = idx[y[idx] == 1]
|
||||
neg = idx[y[idx] == 0]
|
||||
nv_pos = max(1, int(round(len(pos) * VAL_FRACTION)))
|
||||
nv_neg = max(1, int(round(len(neg) * VAL_FRACTION)))
|
||||
v = np.concatenate([pos[:nv_pos], neg[:nv_neg]])
|
||||
t = np.concatenate([pos[nv_pos:], neg[nv_neg:]])
|
||||
return t, v
|
||||
|
||||
|
||||
def train_model(id_arr, attn, y, tr_idx, val_idx, lr, seed, builder=None):
|
||||
"""Fine-tune the full encoder; early-stop on val PR-AUC; return best state."""
|
||||
net = (builder or _make_model)()
|
||||
opt = torch.optim.AdamW([p for p in net.parameters()],
|
||||
lr=lr, weight_decay=WEIGHT_DECAY)
|
||||
lossf = nn.BCEWithLogitsLoss()
|
||||
from sklearn.metrics import average_precision_score
|
||||
tr = torch.from_numpy(np.ascontiguousarray(id_arr[tr_idx]))
|
||||
ta = torch.from_numpy(np.ascontiguousarray(attn[tr_idx]))
|
||||
ty = torch.from_numpy(y[tr_idx].astype(np.float32))
|
||||
va = torch.from_numpy(np.ascontiguousarray(id_arr[val_idx]))
|
||||
vaa = torch.from_numpy(np.ascontiguousarray(attn[val_idx]))
|
||||
vy = y[val_idx]
|
||||
|
||||
best_pr = -1.0
|
||||
best_state = None
|
||||
best_epoch = 0
|
||||
patience = 0
|
||||
n = len(tr_idx)
|
||||
rng = np.random.RandomState(seed * 97 % 2**31)
|
||||
|
||||
for epoch in range(MAX_EPOCHS):
|
||||
net.train()
|
||||
perm = rng.permutation(n)
|
||||
running = 0.0
|
||||
nb = 0
|
||||
for st in range(0, n, BATCH):
|
||||
bidx = torch.from_numpy(perm[st:st + BATCH])
|
||||
logits = net(input_ids=tr[bidx], attention_mask=ta[bidx]).logits.squeeze(-1)
|
||||
loss = lossf(logits, ty[bidx])
|
||||
opt.zero_grad()
|
||||
loss.backward()
|
||||
opt.step()
|
||||
running += float(loss)
|
||||
nb += 1
|
||||
net.eval()
|
||||
with torch.no_grad():
|
||||
pval = torch.sigmoid(net(input_ids=va, attention_mask=vaa).logits.squeeze(-1)).numpy()
|
||||
if len(np.unique(vy)) > 1:
|
||||
pr = average_precision_score(vy, pval)
|
||||
else:
|
||||
pr = 0.0
|
||||
if pr > best_pr:
|
||||
best_pr = pr
|
||||
best_state = {k: v.detach().clone() for k, v in net.state_dict().items()}
|
||||
best_epoch = epoch + 1
|
||||
patience = 0
|
||||
else:
|
||||
patience += 1
|
||||
if patience >= EARLY_STOP:
|
||||
break
|
||||
net.load_state_dict(best_state)
|
||||
return net, best_epoch, best_pr, running / max(nb, 1)
|
||||
|
||||
|
||||
def predict_proba(net, id_arr, attn, idx=None):
|
||||
net.eval()
|
||||
idx = np.arange(len(id_arr)) if idx is None else idx
|
||||
out = []
|
||||
with torch.no_grad():
|
||||
for st in range(0, len(idx), BATCH * 4):
|
||||
bi = idx[st:st + BATCH * 4]
|
||||
iid = torch.from_numpy(np.ascontiguousarray(id_arr[bi]))
|
||||
att = torch.from_numpy(np.ascontiguousarray(attn[bi]))
|
||||
out.append(torch.sigmoid(net(input_ids=iid, attention_mask=att).logits.squeeze(-1)).numpy())
|
||||
return np.concatenate(out)
|
||||
|
||||
|
||||
# ─── grouped CV ─────────────────────────────────────────────────────────────
|
||||
|
||||
def cmd_grouped(args):
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "oof"), exist_ok=True)
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "models"), exist_ok=True)
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
for regime in REGIMES:
|
||||
ids = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"))
|
||||
attn = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"))
|
||||
ev = {v: (np.load(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_ids.npy")),
|
||||
np.load(os.path.join(RESULTS_DIR, "pre", f"ev_{regime}_{v}_attn.npy")))
|
||||
for v in VIEWS}
|
||||
for lr in LRS:
|
||||
for seed in SEEDS:
|
||||
probs = {v: np.zeros(len(rows)) for v in VIEWS}
|
||||
fold_records = []
|
||||
for fold in range(5):
|
||||
tr = np.where(folds != fold)[0]
|
||||
te = np.where(folds == fold)[0]
|
||||
t_idx, v_idx = _val_split(tr, y, seed + 100 * fold)
|
||||
net, ep, best_pr, _ = train_model(ids, attn, y, t_idx, v_idx, lr, seed + fold)
|
||||
torch.save(net.state_dict(),
|
||||
os.path.join(RESULTS_DIR, "models",
|
||||
f"{regime}_{lr}_{seed}_fold{fold}.pt"))
|
||||
for v in VIEWS:
|
||||
probs[v][te] = predict_proba(net, *ev[v], te)
|
||||
fold_records.append({"fold": fold, "epochs": ep, "val_pr": best_pr})
|
||||
np.savez(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_probs.npz"),
|
||||
var_orig=probs["orig"], var_nofinal=probs["nofinal"],
|
||||
var_strip=probs["strip"])
|
||||
with open(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_metrics.json"), "w") as f:
|
||||
json.dump({"folds": fold_records}, f, indent=2)
|
||||
m = s19.binary_metrics(y, probs["strip"])
|
||||
print(f"[{regime}] lr={lr:.0e} seed={seed} "
|
||||
f"PR={m['PR_AUC']:.3f} P={m['P']:.3f} R={m['R']:.3f} "
|
||||
f"FA={m['FA']} epochs={[fr['epochs'] for fr in fold_records]}",
|
||||
flush=True)
|
||||
print("grouped done")
|
||||
|
||||
|
||||
# ─── cap-Q leave-generator-out ──────────────────────────────────────────────
|
||||
|
||||
def cmd_lfo(args):
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "lfo"), exist_ok=True)
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
src = np.array(["capability_question" not in t for t in tags])
|
||||
tgt = ~src
|
||||
for regime in REGIMES:
|
||||
ids = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_ids.npy"))
|
||||
attn = np.load(os.path.join(RESULTS_DIR, "pre", f"{regime}_attn.npy"))
|
||||
for lr in LRS:
|
||||
for seed in SEEDS:
|
||||
s_idx = np.where(src)[0]
|
||||
t_idx, v_idx = _val_split(s_idx, y, seed + 7)
|
||||
net, ep, best_pr, _ = train_model(ids, attn, y, t_idx, v_idx, lr, seed)
|
||||
p = predict_proba(net, ids, attn, np.where(tgt)[0])
|
||||
yt = y[tgt]
|
||||
out = {
|
||||
"regime": regime, "lr": lr, "seed": seed,
|
||||
"rows": int(tgt.sum()), "epochs": ep, "val_pr": best_pr,
|
||||
"mean_action_proba": float(np.mean(p)),
|
||||
"max_action_proba": float(np.max(p)),
|
||||
"acc": float(((p >= 0.5) == (yt == 1)).mean()),
|
||||
"FA": int(((p >= 0.5) & (yt == 0)).sum()),
|
||||
"FA_rate": float(((p >= 0.5) & (yt == 0)).mean()),
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "lfo", f"{regime}_{lr}_{seed}.json"), "w") as f:
|
||||
json.dump(out, f, indent=2)
|
||||
print(f"[{regime}] lr={lr:.0e} seed={seed} capQ LOFO "
|
||||
f"acc={out['acc']:.3f} FA_rate={out['FA_rate']:.3f} "
|
||||
f"mean_p={out['mean_action_proba']:.3f} epochs={ep}", flush=True)
|
||||
print("lfo done")
|
||||
|
||||
|
||||
# ─── metrics aggregation ────────────────────────────────────────────────────
|
||||
|
||||
def cmd_metrics(args):
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
pairs = s19.build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
summary = {}
|
||||
for regime in REGIMES:
|
||||
summary[regime] = {}
|
||||
for lr in LRS:
|
||||
per_seed = []
|
||||
for seed in SEEDS:
|
||||
z = np.load(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_probs.npz"))
|
||||
entry = {"seed": seed,
|
||||
"views": {v: {"FA_rate": s19.binary_metrics(y, z[f"var_{v}"])["FA_rate"],
|
||||
"PR": s19.binary_metrics(y, z[f"var_{v}"])["PR_AUC"],
|
||||
"capQ_FA": s19.capq_fa(tags, y, z[f"var_{v}"])}
|
||||
for v in VIEWS},
|
||||
"strip": s19.binary_metrics(y, z["var_strip"]),
|
||||
"pairs": {v: s19.pair_metrics(pairs, z[f"var_{v}"]) for v in VIEWS},
|
||||
"curve": s19.threshold_curve(y, z["var_strip"], np.arange(0.30, 1.0, 0.02)),
|
||||
"ops": s19.operating_points(s19.threshold_curve(
|
||||
y, z["var_strip"], np.arange(0.30, 1.0, 0.02)),
|
||||
z["var_strip"], pairs),
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "oof", f"{regime}_{lr}_{seed}_metrics.json")) as f:
|
||||
entry["folds"] = json.load(f)["folds"]
|
||||
per_seed.append(entry)
|
||||
# LOFO
|
||||
lfos = []
|
||||
for seed in SEEDS:
|
||||
with open(os.path.join(RESULTS_DIR, "lfo", f"{regime}_{lr}_{seed}.json")) as f:
|
||||
lfos.append(json.load(f))
|
||||
summary[regime][str(lr)] = {"seeds": per_seed, "lfo": lfos}
|
||||
# primary selection: min mean cap-Q LOFO FA_rate across seeds
|
||||
sel = {}
|
||||
for regime in REGIMES:
|
||||
best_lr, best_mean = None, 9e9
|
||||
for lr in LRS:
|
||||
mean_fa = np.mean([l["FA_rate"] for l in summary[regime][str(lr)]["lfo"]])
|
||||
sel[f"{regime}_{lr}"] = round(float(mean_fa), 4)
|
||||
if mean_fa < best_mean:
|
||||
best_lr, best_mean = lr, mean_fa
|
||||
sel[f"{regime}_selected"] = str(best_lr)
|
||||
summary["selection"] = sel
|
||||
os.makedirs(os.path.join(RESULTS_DIR, "results"), exist_ok=True)
|
||||
with open(os.path.join(RESULTS_DIR, "results", "summary.json"), "w") as f:
|
||||
json.dump(summary, f, indent=2, default=str)
|
||||
print(json.dumps(sel))
|
||||
print("metrics done")
|
||||
|
||||
|
||||
def _tokenizer_bytes():
|
||||
return sum(os.path.getsize(os.path.join(MODEL_DIR, f))
|
||||
for f in ["vocab.txt", "tokenizer.json"]
|
||||
if os.path.exists(os.path.join(MODEL_DIR, f)))
|
||||
|
||||
|
||||
# ─── runtime + ONNX ─────────────────────────────────────────────────────────
|
||||
|
||||
def cmd_runtime(args):
|
||||
import time
|
||||
net = _make_model()
|
||||
tok = _load_tokenizer()
|
||||
rows = s19.load_dev()
|
||||
x = [r["text_orig"] for r in rows][:200]
|
||||
e = tok(x, padding="max_length", truncation=True,
|
||||
max_length=MAX_LEN, return_tensors="pt")
|
||||
ids, attn = e["input_ids"], e["attention_mask"]
|
||||
net.eval()
|
||||
with torch.no_grad():
|
||||
# warmup
|
||||
for _ in range(3):
|
||||
net(input_ids=ids[:1], attention_mask=attn[:1])
|
||||
# batch-1 latency
|
||||
lat = []
|
||||
for i in range(200):
|
||||
t0 = time.perf_counter()
|
||||
net(input_ids=ids[i:i + 1], attention_mask=attn[i:i + 1])
|
||||
lat.append((time.perf_counter() - t0) * 1e6)
|
||||
# tokenization latency
|
||||
t0 = time.perf_counter()
|
||||
for i in range(200):
|
||||
tok(x[i])
|
||||
tok_us = (time.perf_counter() - t0) / 200 * 1e6
|
||||
n_params = sum(p.numel() for p in net.parameters())
|
||||
fp32 = n_params * 4
|
||||
rep = {
|
||||
"model": MODEL_NAME, "sha": MODEL_SHA,
|
||||
"params": n_params, "fp32_bytes": fp32,
|
||||
"fp16_bytes": fp32 // 2, "int8_bytes": n_params,
|
||||
"tokenizer_bytes": _tokenizer_bytes(),
|
||||
"latency_us_mean": float(np.mean(lat)),
|
||||
"latency_us_p50": float(np.median(lat)),
|
||||
"latency_us_p95": float(np.percentile(lat, 95)),
|
||||
"max_len": MAX_LEN,
|
||||
"tok_us": round(tok_us, 2),
|
||||
"num_threads": 12,
|
||||
}
|
||||
with open(os.path.join(RESULTS_DIR, "runtime.json"), "w") as f:
|
||||
json.dump(rep, f, indent=2)
|
||||
print(json.dumps(rep, indent=2))
|
||||
print("runtime done")
|
||||
|
||||
|
||||
def cmd_onnx(args):
|
||||
net = _make_model()
|
||||
net.eval()
|
||||
tok = _load_tokenizer()
|
||||
rows = s19.load_dev()
|
||||
try:
|
||||
import torch.onnx
|
||||
dummy = {
|
||||
"input_ids": torch.zeros(1, MAX_LEN, dtype=torch.long),
|
||||
"attention_mask": torch.ones(1, MAX_LEN, dtype=torch.long),
|
||||
}
|
||||
with torch.no_grad():
|
||||
torch.onnx.export(net, (dummy,), os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"),
|
||||
input_names=["input_ids", "attention_mask"],
|
||||
output_names=["logits"], opset_version=14,
|
||||
dynamic_axes={"input_ids": {0: "batch"},
|
||||
"attention_mask": {0: "batch"}})
|
||||
# parity on a fixed sample
|
||||
import numpy as np
|
||||
samp = [(r["text_orig"], r["y"]) for r in rows[:200]]
|
||||
e = tok([s[0] for s in samp], padding="max_length", truncation=True,
|
||||
max_length=MAX_LEN, return_tensors="pt")
|
||||
with torch.no_grad():
|
||||
pt = torch.sigmoid(net(**e).logits.squeeze(-1)).numpy()
|
||||
import onnxruntime as ort
|
||||
so = ort.SessionOptions()
|
||||
so.intra_op_num_threads = 12
|
||||
sess = ort.InferenceSession(os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"),
|
||||
sess_options=so, providers=["CPUExecutionProvider"])
|
||||
on = sess.run(None, {"input_ids": e["input_ids"].numpy(),
|
||||
"attention_mask": e["attention_mask"].numpy()})[0]
|
||||
on = 1 / (1 + np.exp(-on).squeeze(-1))
|
||||
mx = float(np.max(np.abs(pt - on)))
|
||||
size = os.path.getsize(os.path.join(RESULTS_DIR, "rubert-tiny-gate.onnx"))
|
||||
rep = {"max_logit_diff": mx, "onnx_bytes": size,
|
||||
"parity_n": len(samp), "provider": "CPUExecutionProvider"}
|
||||
with open(os.path.join(RESULTS_DIR, "onnx.json"), "w") as f:
|
||||
json.dump(rep, f, indent=2)
|
||||
print(json.dumps(rep))
|
||||
except Exception as ex:
|
||||
print("onnx export/parity failed:", ex)
|
||||
with open(os.path.join(RESULTS_DIR, "onnx.json"), "w") as f:
|
||||
json.dump({"error": str(ex)}, f, indent=2)
|
||||
print("onnx done")
|
||||
|
||||
|
||||
# ─── optional capacity/pretraining ceiling (brief §13) ──────────────────────
|
||||
# Trigger: tiny1 clearly improved over from-scratch on in-pool but missed the
|
||||
# LOFO boundary. tiny2 is the same 3-layer 312-hidden BERT family; it tests
|
||||
# whether a *newer, larger-vocab* pretraining of the same family generalises
|
||||
# where tiny1 failed — disambiguating "this family is the wrong prior" from a
|
||||
# one-off pretraining. It cannot test capacity (same depth/size).
|
||||
|
||||
MODEL2_DIR = ("/tmp/mvn-s20/hf-tiny2-cache/models--cointegrated--rubert-tiny2/"
|
||||
"snapshots/e8ed3b0c8bbf4fb6984c3de043bf7d2f4e5969ae")
|
||||
MODEL2_SHA = "e8ed3b0c8bbf4fb6984c3de043bf7d2f4e5969ae"
|
||||
CEIL_RESULTS = os.path.join(RESULTS_DIR, "tiny2")
|
||||
|
||||
|
||||
def cmd_ceiling(args):
|
||||
import torch
|
||||
os.makedirs(CEIL_RESULTS, exist_ok=True)
|
||||
os.makedirs(os.path.join(CEIL_RESULTS, "models"), exist_ok=True)
|
||||
tok = AutoTokenizer.from_pretrained(MODEL2_DIR)
|
||||
rows = s19.load_dev()
|
||||
y = np.array([r["y"] for r in rows])
|
||||
tags = [r["tags"] for r in rows]
|
||||
folds = np.array([r["cv_fold"] for r in rows])
|
||||
pairs = s19.build_pairs(rows, [r["text_strip"] for r in rows])
|
||||
# audit: is the tiny2 tokenizer sane on the corpus before anything else
|
||||
n_unk = 0
|
||||
n_tok = 0
|
||||
lens = []
|
||||
for r in rows:
|
||||
e = tok(r["text_orig"])
|
||||
n_unk += e["input_ids"].count(tok.unk_token_id)
|
||||
n_tok += len(e["input_ids"])
|
||||
lens.append(len(e["input_ids"]))
|
||||
audit = {"vocab_size": tok.vocab_size,
|
||||
"unk_count": int(n_unk),
|
||||
"unk_rate": round(n_unk / max(n_tok, 1), 5),
|
||||
"seq_len_p99": sorted(lens)[int(len(lens) * .99)],
|
||||
"seq_len_max": max(lens)}
|
||||
with open(os.path.join(CEIL_RESULTS, "audit.json"), "w") as f:
|
||||
json.dump(audit, f, indent=2)
|
||||
print("tiny2 audit:", audit)
|
||||
|
||||
# tokenize the corpus (regime A only — natural text, the in-pool best)
|
||||
ids_a, attn_a = tokenize([r["text_orig"] for r in rows], tok)
|
||||
ev = {v: tokenize([r[f"text_{v}"] for r in rows], tok) for v in VIEWS}
|
||||
|
||||
def make2():
|
||||
cfg = AutoConfig.from_pretrained(MODEL2_DIR)
|
||||
cfg.num_labels = 1
|
||||
m = BertForSequenceClassification.from_pretrained(MODEL2_DIR, config=cfg)
|
||||
return m
|
||||
|
||||
# cap-Q LOFO, 3 seeds, matching the A@2e-5 tiny1 config
|
||||
lfors = []
|
||||
src = np.where(np.array(["capability_question" not in t for t in tags]))[0]
|
||||
tgt = np.where(np.array(["capability_question" in t for t in tags]))[0]
|
||||
for seed in SEEDS:
|
||||
t_idx, v_idx = _val_split(src, y, seed + 7)
|
||||
net, ep, best_pr, _ = train_model(ids_a, attn_a, y, t_idx, v_idx, 2e-5, seed, builder=make2)
|
||||
p = predict_proba(net, ids_a, attn_a, tgt)
|
||||
yt = y[tgt]
|
||||
lfors.append({"regime": "A(tiny2)", "lr": 2e-5, "seed": seed,
|
||||
"rows": int(len(tgt)), "epochs": ep,
|
||||
"mean_action_proba": float(np.mean(p)),
|
||||
"acc": float(((p >= 0.5) == (yt == 1)).mean()),
|
||||
"FA": int(((p >= 0.5) & (yt == 0)).sum()),
|
||||
"FA_rate": float(((p >= 0.5) & (yt == 0)).mean())})
|
||||
with open(os.path.join(CEIL_RESULTS, "lfo.json"), "w") as f:
|
||||
json.dump(lfors, f, indent=2)
|
||||
print("tiny2 LOFO:", [round(l["FA_rate"], 3) for l in lfors])
|
||||
|
||||
# grouped CV for the same best config + in-pool pairs / capQ
|
||||
oof = {v: np.zeros(len(rows)) for v in VIEWS}
|
||||
for fold in range(5):
|
||||
tr = np.where(folds != fold)[0]
|
||||
te = np.where(folds == fold)[0]
|
||||
t_idx, v_idx = _val_split(tr, y, 42 + 100 * fold)
|
||||
net, _, _, _ = train_model(ids_a, attn_a, y, t_idx, v_idx, 2e-5, 42 + fold, builder=make2)
|
||||
for v in VIEWS:
|
||||
oof[v][te] = predict_proba(net, *ev[v], te)
|
||||
inpool = {"strip": s19.binary_metrics(y, oof["strip"]),
|
||||
"pairs": {v: s19.pair_metrics(pairs, oof[v]) for v in VIEWS},
|
||||
"capQ_inpool": {v: s19.capq_fa(tags, y, oof[v]) for v in VIEWS}}
|
||||
with open(os.path.join(CEIL_RESULTS, "grouped.json"), "w") as f:
|
||||
json.dump(inpool, f, indent=2, default=str)
|
||||
b = inpool["strip"]
|
||||
print(f"tiny2 grouped A@2e-5: PR={b['PR_AUC']:.3f} P={b['P']:.3f} R={b['R']:.3f} "
|
||||
f"FA={b['FA']} pairs_strip={inpool['pairs']['strip']['ordering_acc']:.3f} "
|
||||
f"capQ_strip={inpool['capQ_inpool']['strip']:.3f}", flush=True)
|
||||
print("ceiling done")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("cmd", choices=["pre", "grouped", "lfo", "metrics", "runtime", "onnx", "ceiling"])
|
||||
args = ap.parse_args()
|
||||
t0 = time.time()
|
||||
globals()[f"cmd_{args.cmd}"](args)
|
||||
print(f"elapsed {time.time()-t0:.1f}s")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,95 @@
|
||||
package main
|
||||
|
||||
// Fixture is one brief-specified utterance and the eligibility it must earn.
|
||||
// Fixtures are EXACTLY the slice-21 brief examples plus the corpus-verified
|
||||
// structural discriminators. They are eval material, not corpus additions:
|
||||
// nothing here is inserted into any pool file (brief §2, §18).
|
||||
type Fixture struct {
|
||||
Utterance string
|
||||
Want Eligibility
|
||||
Family string
|
||||
Note string
|
||||
}
|
||||
|
||||
// Fixtures covers every family the brief's §15 list exercises, including the
|
||||
// four that have no dev-pool rows at all (negation, reported_speech,
|
||||
// quotation, hypothetical):
|
||||
//
|
||||
// direct imperative already plural in dev; fixtures pin the canonical
|
||||
// cases together with the politeness-negative modal.
|
||||
// polite request polite imperative + bare "можешь …, пожалуйста".
|
||||
// modal request "можно …", English "can you …, please".
|
||||
// first-person request "я хочу …", "мне нужно …", "надо …".
|
||||
// reordered target "свет выключи, пожалуйста" (target-first command).
|
||||
// capability question ты/умеешь/сможешь/способна + "… ли" permission.
|
||||
// ordinary question "что запущено" — no execution pressure.
|
||||
// negation "не выключай свет" (parser-covered); advisory "не
|
||||
// надо выключать свет".
|
||||
// reported speech past/third-person report verbs over a command.
|
||||
// quotation quoted command, with and without a reporting frame.
|
||||
// hypothetical если-scopes that are not real condition→command.
|
||||
var Fixtures = []Fixture{
|
||||
// negation — direct commands the prosecutor forbids
|
||||
{"не выключай свет", Blocked, "negation", "direct prohibition"},
|
||||
{"не перезапускай сервер", Blocked, "negation", "direct prohibition"},
|
||||
{"не включай nginx", Blocked, "negation", "direct prohibition"},
|
||||
{"не надо выключать свет", Blocked, "negation", "advisory negative"},
|
||||
{"не стоит перезапускать nginx", Blocked, "negation", "advisory negative"},
|
||||
{"не забудь напомнить про свет", Permissive, "negation", "prohibition-parser reminder exemption"},
|
||||
|
||||
// reported speech — reports an order, does not issue one
|
||||
{"он сказал выключить свет", Blocked, "reported_speech", "past report verb + infinitive"},
|
||||
{"она попросила перезапустить nginx", Blocked, "reported_speech", "past report verb + infinitive"},
|
||||
{"мне сказали включить свет", Blocked, "reported_speech", "passive report + infinitive"},
|
||||
{"он написал: «перезапусти nginx»", Blocked, "reported_speech", "report verb + quoted imperative"},
|
||||
{"скажи мне, что он сказал про свет", Permissive, "reported_speech", "request to report, no commanded clause"},
|
||||
{"расскажи про свет", Permissive, "reported_speech", "narrative request, not a reported order"},
|
||||
|
||||
// quotation — quoted text is referenced, not issued
|
||||
{"фраза «выключи свет»", Blocked, "quotation", "reporting noun + quoted imperative"},
|
||||
{"он сказал «выключи свет»", Blocked, "quotation", "report verb + quoted imperative"},
|
||||
{"«выключи свет»", Ambiguous, "quotation", "bare quoted command, no frame"},
|
||||
{"выключи свет", Permissive, "quotation", "unquoted imperative is a live command"},
|
||||
|
||||
// hypothetical
|
||||
{"если выключить свет...", Blocked, "hypothetical", "conditional + infinitive + ellipsis"},
|
||||
{"если бы перезапустить nginx...", Blocked, "hypothetical", "conditional + бы + infinitive"},
|
||||
{"что будет если выключить свет", Blocked, "hypothetical", "question-scoped conditional"},
|
||||
{"если будет дождь, выключи полив", Permissive, "hypothetical", "real condition → imperative"},
|
||||
{"выключи свет если будет дождь", Permissive, "hypothetical", "imperative → real condition"},
|
||||
|
||||
// capability question — blocked even polite
|
||||
{"ты можешь выключить свет?", Blocked, "capability_question", "ты + можешь + ?"},
|
||||
{"ты можешь выключить свет", Blocked, "capability_question", "ты + можешь, no ?"},
|
||||
{"ты можешь выключить свет, пожалуйста", Blocked, "capability_question", "ты + можешь + politeness (42/42 non-action)"},
|
||||
{"умеешь ли ты выключить свет", Blocked, "capability_question", "ability form + ли"},
|
||||
{"сможешь открыть окно, пожалуйста", Blocked, "capability_question", "bare future + politeness (7/7 non-action)"},
|
||||
{"ты способна выключить свет", Blocked, "capability_question", "ты + способна"},
|
||||
{"могу ли я выключить свет", Blocked, "capability_question", "first-person can + ли"},
|
||||
{"можно ли выключить свет", Blocked, "capability_question", "можно + ли permission question"},
|
||||
{"ты выключишь свет?", Ambiguous, "capability_question", "future tense + ? without can-form"},
|
||||
|
||||
// modal / polite requests — permissive
|
||||
{"можешь выключить свет, пожалуйста", Permissive, "modal_request", "bare можешь + politeness (127/127 action)"},
|
||||
{"пожалуйста, выключи свет", Permissive, "polite_request", "leading politeness + imperative"},
|
||||
{"выключи свет, пожалуйста", Permissive, "polite_request", "imperative + trailing politeness"},
|
||||
{"выключи свет", Permissive, "direct_imperative", "plain imperative"},
|
||||
{"свет выключи, пожалуйста", Permissive, "reordered_target", "target-first imperative"},
|
||||
{"can you выключи свет, please", Permissive, "modal_request", "English frame + Russian imperative + please (96/96 action)"},
|
||||
{"can you выключи свет", Ambiguous, "modal_request", "English can without politeness"},
|
||||
{"не мог бы ты выключить свет", Permissive, "polite_request", "conditional politeness, prohibition-parser exemption"},
|
||||
{"можно выключить свет", Permissive, "modal_request", "можно + infinitive permission-implicature request"},
|
||||
|
||||
// first-person requests
|
||||
{"я хочу выключить свет", Permissive, "first_person_request", "first-person + illocution"},
|
||||
{"я хочу чтобы ты выключил свет", Permissive, "first_person_request", "first-person + embedded ya-you wish"},
|
||||
{"надо выключить свет", Permissive, "first_person_request", "impersonal need"},
|
||||
{"мне нужно включить свет", Permissive, "first_person_request", "first-person oblique + need"},
|
||||
|
||||
// ordinary questions — no execution pressure even when answerable
|
||||
{"что запущено", Ambiguous, "question", "status question, no request evidence"},
|
||||
{"какие службы работают", Ambiguous, "question", "question, no request evidence"},
|
||||
{"сколько ламп включено", Ambiguous, "question", "question, no request evidence"},
|
||||
{"что ты можешь включить", Blocked, "capability_question", "open question with ты + можешь"},
|
||||
{"покажи что запущено", Permissive, "first_person_request", "imperative lead over a status question"},
|
||||
}
|
||||
@@ -0,0 +1,659 @@
|
||||
// Guard is the slice-21 deterministic execution-frame engine (experiment-only).
|
||||
//
|
||||
// It answers one question: given an utterance, what is its execution-frame
|
||||
// eligibility as a three-way gate — permissive, blocked, ambiguous — and why.
|
||||
// It never decides what an utterance IS (that stays with the route classifier);
|
||||
// it only decides whether an utterance may become an executable action at all.
|
||||
// The policy is asymmetric on purpose: blocked and ambiguous must never
|
||||
// execute, and permissive only means "no blocking speech-act evidence exists",
|
||||
// not "execute this".
|
||||
//
|
||||
// It reuses the shipped deterministic routers rather than inventing new ones:
|
||||
//
|
||||
// router.ParseCommandProhibition / IsCommandProhibition direct negative commands
|
||||
// morph.IsVerbForm / morph.Lemma verb mood and finiteness
|
||||
// lexicon.IsFillerParticle / FirstPerson() politeness and first-person frames
|
||||
//
|
||||
// Everything else is closed-class evidence measured on the frozen slice-20 dev
|
||||
// pool (§"measured discriminators" in the brief): 126 capability-question rows
|
||||
// split 42/42/42 across ты-addressed, bare ability (умеешь), and bare future
|
||||
// (сможешь) modality; 127 bare "можешь, пожалуйста" rows are 100% action;
|
||||
// "can you … , please" (English frame + Russian imperative) is 100% action.
|
||||
// The rules below are the encoding of precisely those numbers.
|
||||
//
|
||||
// The reason vocabulary is a closed set. Additions are design decisions that
|
||||
// must land in the report, not silent new branches.
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"github.com/kami/maven/internal/lexicon"
|
||||
"github.com/kami/maven/internal/morph"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// Eligibility is the three-way execution-frame verdict.
|
||||
type Eligibility int
|
||||
|
||||
const (
|
||||
Permissive Eligibility = iota // no blocking speech-act evidence; downstream route decides
|
||||
Blocked // a speech act forbids execution (negation, question, report, …)
|
||||
Ambiguous // not enough evidence either way; must not execute
|
||||
)
|
||||
|
||||
func (e Eligibility) String() string {
|
||||
switch e {
|
||||
case Permissive:
|
||||
return "permissive"
|
||||
case Blocked:
|
||||
return "blocked"
|
||||
default:
|
||||
return "ambiguous"
|
||||
}
|
||||
}
|
||||
|
||||
// Reason is a closed set of structural explanations for a verdict.
|
||||
type Reason string
|
||||
|
||||
const (
|
||||
ReasonCommandProhibition Reason = "command_prohibition"
|
||||
ReasonCapabilityQuestion Reason = "capability_question"
|
||||
ReasonReportedSpeech Reason = "reported_speech"
|
||||
ReasonQuotation Reason = "quotation"
|
||||
ReasonHypothetical Reason = "hypothetical"
|
||||
ReasonNegatedCommand Reason = "negated_command"
|
||||
ReasonExplicitRequest Reason = "explicit_request"
|
||||
ReasonAmbiguousModal Reason = "ambiguous_modal"
|
||||
ReasonNoRequestEvidence Reason = "no_request_evidence"
|
||||
)
|
||||
|
||||
func (r Reason) String() string { return string(r) }
|
||||
|
||||
// Frame is the verdict for one utterance. Eligibility decides; Reasons explain.
|
||||
// A frame may carry more than one reason (e.g. a quoted reported command).
|
||||
type Frame struct {
|
||||
Eligibility Eligibility
|
||||
Reasons []Reason
|
||||
}
|
||||
|
||||
// maybeWord is a single-token or multi-token closed expression, e.g. the
|
||||
// token "не мог бы" covers the three tokens не мog бы when matched as a
|
||||
// contiguous run ("бы" is itself a bound marker). Multi-token members are
|
||||
// matched over the reconstructed token text, never over raw text, so
|
||||
// punctuation boundaries do not defeat them.
|
||||
type maybeWord struct {
|
||||
single []string
|
||||
multi []string // matched as contiguous lowercased token runs
|
||||
}
|
||||
|
||||
func (w maybeWord) in(toks []string, joined string) bool {
|
||||
if hasAny(toks, w.single) {
|
||||
return true
|
||||
}
|
||||
for _, m := range w.multi {
|
||||
tm := strings.Join(tokens(m), " ")
|
||||
if tm != "" && strings.Contains(joined, tm) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ── closed evidence sets (all measured on the slice-20 dev pool) ──────────
|
||||
|
||||
// wakeAddr is stripped from the left of an utterance before command-form
|
||||
// detection: "мавен, выключи свет" and "выключи свет" must ride the same
|
||||
// frame. Closed: the names Maven answers to in the dev pool.
|
||||
var wakeAddr = []string{"мавен", "maven", "мавэн", "алекса", "алиса", "окей", "эй", "hey"}
|
||||
|
||||
// ruAddress are the second-person Russian address tokens. "ты можешь …"
|
||||
// (with or without politeness) is 42/42 capability-question in the dev pool,
|
||||
// so any addressed Russian can-form is a capability question, never a request.
|
||||
var ruAddress = []string{"ты", "тебе", "тебя", "тобой", "тобою", "вы", "вас", "вам", "вами"}
|
||||
|
||||
// enAddress is the English second-person address. Unlike Russian, "can you …
|
||||
// , please" is 96/96 action in the dev pool (English modal frame around a
|
||||
// Russian imperative), so English address alone never blocks: it routes to the
|
||||
// politeness arm.
|
||||
var enAddress = []string{"you", "u", "your"}
|
||||
|
||||
// ruCanForms are the present-can verb forms. Bare (no address) "можешь …,
|
||||
// пожалуйста" is 127/127 action; bare "можешь …" with no politeness is the
|
||||
// ambiguous bucket (no such rows exist in dev — conservative default).
|
||||
var ruCanForms = []string{"можешь", "можете", "могу", "можем"}
|
||||
|
||||
// ruAbilityForms are future/ability modal forms that read as a question of
|
||||
// capability regardless of politeness: "сможешь открыть окно, пожалуйста" and
|
||||
// "умеешь ли ты …" are 0/84 action in the dev pool, so even a polite bare
|
||||
// form never grants execution. "мог(ла) бы …" and "смог(ла) бы …" are the
|
||||
// conditional-politeness mask over the same boundary — except the leading
|
||||
// politeness construction "не мог бы ты …", which the prohibition parser
|
||||
// already classifies as ordinary modal politeness and must stay permissive.
|
||||
var ruAbilityForms = maybeWord{
|
||||
single: []string{
|
||||
"сможешь", "сможете", "смогу", "сможем", "сумеешь", "сумеете",
|
||||
"умеешь", "умеете", "способна", "способен", "способно", "способны",
|
||||
"смог", "смогла", "смогли", "мог", "могла", "могли",
|
||||
},
|
||||
multi: []string{
|
||||
"смог бы", "смогла бы", "смогли бы", "мог бы", "могла бы", "могли бы",
|
||||
"смочь бы", "мочь бы",
|
||||
},
|
||||
}
|
||||
|
||||
// politeNegativeModal is the leading "не мог бы ты/вы …" politeness framing the
|
||||
// prohibition parser exempts as ordinary modal politeness. When it leads the
|
||||
// utterance the capability stage declines and the frame reads as a request.
|
||||
var politeNegativeModal = []string{
|
||||
"не мог бы", "не могла бы", "не могли бы", "не смог бы", "не смогла бы", "не смогли бы",
|
||||
}
|
||||
|
||||
// enCanForms are the English modal can/could tokens.
|
||||
var enCanForms = []string{"can", "could"}
|
||||
|
||||
// politeness is the closed set of politeness fillers. пожалуйста/плиз/please
|
||||
// are already closed-class filler particles in the lexicon; the добр-forms
|
||||
// are the only additions the dev pool exercises.
|
||||
var politeness = maybeWord{
|
||||
single: []string{"пожалуйста", "плиз", "please"},
|
||||
multi: []string{"будь добр", "будьте добры", "был бы добр", "были бы добры"},
|
||||
}
|
||||
|
||||
// reportVerbs are the past/third-person report verbs — the frame that reports
|
||||
// a command rather than issuing it. Second-person imperatives ("скажи",
|
||||
// "расскажи", "напомни") are deliberately absent: those are requests to
|
||||
// report, and their clause forms part of the current utterance, not a
|
||||
// replayed order. Matched as closed list (a report verb outside it is a data
|
||||
// gap, noted in the report).
|
||||
var reportVerbs = []string{
|
||||
"сказал", "сказала", "сказали", "говорил", "говорила", "говорили",
|
||||
"говорит", "говорят", "попросил", "попросила", "попросили",
|
||||
"просил", "просила", "просили", "написал", "написала", "написали",
|
||||
"пишет", "приказал", "приказала", "приказали", "велел", "велела",
|
||||
"велели", "скомандовал", "скомандовала", "рекомендовал", "рекомендовала",
|
||||
"посоветовал", "посоветовала", "сообщил", "сообщила", "сообщили",
|
||||
"объявил", "объявила", "велено", "сказано", "написано", "записано",
|
||||
}
|
||||
|
||||
// reportNouns name a quoted or reported text: "фраза «выключи свет»" is a
|
||||
// quotation, not a command.
|
||||
var reportNouns = []string{
|
||||
"фраза", "фразы", "фразе", "фразу", "слово", "слова", "слове", "словом",
|
||||
"выражение", "выражения", "цитата", "цитату", "цитате",
|
||||
"название", "текст", "сообщение", "письмо", "заметка", "заметку",
|
||||
}
|
||||
|
||||
// hypothesisMarkers open a conditional scope.
|
||||
var hypothesisMarkers = []string{"если", "ежели", "коли", "кабы", "if"}
|
||||
|
||||
// illocutionVerbs make a first-person or impersonal clause a request even
|
||||
// without an imperative form ("я хочу …", "мне нужно …", "надо …").
|
||||
var illocutionVerbs = maybeWord{
|
||||
single: []string{
|
||||
"хочу", "хотел", "хотела", "хотелось", "желаю", "прошу", "просим",
|
||||
"просил", "просила", "просили", "попросить",
|
||||
"надо", "нужно", "следует", "пора", "требуется", "придётся", "придется",
|
||||
"могу", "давай", "давайте",
|
||||
},
|
||||
multi: []string{
|
||||
"хотел бы", "хотела бы", "хочу чтобы", "хотел чтобы", "хотела чтобы",
|
||||
"могу ли",
|
||||
},
|
||||
}
|
||||
|
||||
// ── token helpers ─────────────────────────────────────────────────────────
|
||||
|
||||
// tokens lowercases and splits on anything that is not a letter or digit,
|
||||
// matching the router's planTokens discipline ("что-дальше" tokenises like
|
||||
// "что дальше").
|
||||
func tokens(text string) []string {
|
||||
return strings.FieldsFunc(strings.ToLower(text), func(r rune) bool {
|
||||
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
|
||||
})
|
||||
}
|
||||
|
||||
func hasTok(toks []string, w string) bool {
|
||||
for _, t := range toks {
|
||||
if t == w {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func hasAny(toks, ws []string) bool {
|
||||
for _, w := range ws {
|
||||
if hasTok(toks, w) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func indexTok(toks []string, w string) int {
|
||||
for i, t := range toks {
|
||||
if t == w {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
// isFiniteVerb reports a verb form that is not the dictionary (infinitive)
|
||||
// form: "выключи" is finite, "выключить" is not. A finite verb at command
|
||||
// position is positive request evidence; an infinitive is not.
|
||||
func isFiniteVerb(tok string) bool {
|
||||
if !morph.IsVerbForm(tok) {
|
||||
return false
|
||||
}
|
||||
return morph.Lemma(tok) != tok
|
||||
}
|
||||
|
||||
// isInfinitive reports a token that morph resolves to its own dictionary form
|
||||
// (the lemma ends in the infinitive ending by construction).
|
||||
func isInfinitive(tok string) bool {
|
||||
if !morph.IsVerbForm(tok) {
|
||||
return false
|
||||
}
|
||||
return morph.Lemma(tok) == tok
|
||||
}
|
||||
|
||||
func anyInfinitive(toks []string) bool {
|
||||
for _, t := range toks {
|
||||
if isInfinitive(t) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func anyFiniteVerb(toks []string) bool {
|
||||
for _, t := range toks {
|
||||
if isFiniteVerb(t) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func hasAnyVerb(toks []string) bool {
|
||||
for _, t := range toks {
|
||||
if morph.IsVerbForm(t) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func blocked(rs ...Reason) Frame { return Frame{Eligibility: Blocked, Reasons: rs} }
|
||||
func ambiguous(rs ...Reason) Frame {
|
||||
return Frame{Eligibility: Ambiguous, Reasons: rs}
|
||||
}
|
||||
func permissive(rs ...Reason) Frame {
|
||||
return Frame{Eligibility: Permissive, Reasons: rs}
|
||||
}
|
||||
|
||||
// ── quoted spans ─────────────────────────────────────────────────────────
|
||||
|
||||
// quotedSpan is a maximal quoted interval in the normalized text.
|
||||
type quotedSpan struct{ content string }
|
||||
|
||||
// quotePairs covers the quoting styles the dev pool and brief fixtures use:
|
||||
// Russian guillemets, curly double/single quotes, and straight quotes.
|
||||
var quotePairs = []struct{ open, close string }{
|
||||
{"«", "»"}, {"„", "\""}, {"“", "”"}, {"‚", "‘"}, {"‘", "’"}, {"'", "'"}, {"\"", "\""},
|
||||
}
|
||||
|
||||
// extractQuotedSpans returns the contents of quoted spans in order, in rune
|
||||
// index space (the text is normalized, so glyphs are single runes). An
|
||||
// unbalanced delimiter yields no span (best-effort; the conservative
|
||||
// fallback then applies).
|
||||
func extractQuotedSpans(t string) []quotedSpan {
|
||||
runes := []rune(t)
|
||||
var out []quotedSpan
|
||||
i := 0
|
||||
for i < len(runes) {
|
||||
matched := false
|
||||
for _, p := range quotePairs {
|
||||
po := []rune(p.open)
|
||||
pc := []rune(p.close)
|
||||
if i+len(po) > len(runes) || string(runes[i:i+len(po)]) != p.open {
|
||||
continue
|
||||
}
|
||||
j := i + len(po)
|
||||
for j+len(pc) <= len(runes) && string(runes[j:j+len(pc)]) != p.close {
|
||||
j++
|
||||
}
|
||||
out = append(out, quotedSpan{content: string(runes[i+len(po) : j])})
|
||||
i = j + len(pc)
|
||||
matched = true
|
||||
break
|
||||
}
|
||||
if !matched {
|
||||
i++
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── rule stages (evaluated in this order; a decision is final) ────────────
|
||||
|
||||
// Evaluate derives the execution-frame verdict for one utterance.
|
||||
func Evaluate(text string) Frame {
|
||||
t := router.NormalizeMatchText(text)
|
||||
if strings.TrimSpace(t) == "" {
|
||||
return ambiguous(ReasonNoRequestEvidence)
|
||||
}
|
||||
toks := tokens(t)
|
||||
joined := strings.Join(toks, " ")
|
||||
|
||||
// 1. Quotation: a command inside a quoted span is not a command being
|
||||
// issued now. With a reporting frame outside it is a quotation; a bare
|
||||
// quote is at best ambiguous.
|
||||
if f, ok := stageQuotation(t, toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 2. Reported speech: a past/third-person report verb governing a command
|
||||
// clause reports an order to someone else, it does not issue one.
|
||||
if f, ok := stageReport(t, toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 3. Hypothetical: a command scope opened by "если/if" that does not
|
||||
// continue as a real condition→command is not an execution request.
|
||||
if f, ok := stageHypothesis(toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 4. Direct negative commands: the shipped prohibition parser.
|
||||
if router.IsCommandProhibition(t) {
|
||||
return blocked(ReasonCommandProhibition)
|
||||
}
|
||||
|
||||
// 5. Advisory negatives: "не надо/не стоит/не нужно …".
|
||||
if f, ok := stageAdvisoryNegation(toks); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 6. Capability and permission modality (the measured core).
|
||||
if f, ok := stageCapability(toks, joined); ok {
|
||||
return f
|
||||
}
|
||||
|
||||
// 7. Trailing question mark with no modal at play: an uncertain posture,
|
||||
// never a confirmed executable request.
|
||||
if strings.HasSuffix(t, "?") {
|
||||
return ambiguous(ReasonAmbiguousModal)
|
||||
}
|
||||
|
||||
// 8. Positive request evidence.
|
||||
if hasRequestEvidence(toks, joined) {
|
||||
return permissive(ReasonExplicitRequest)
|
||||
}
|
||||
|
||||
// 9. No execution pressure at all.
|
||||
return ambiguous(ReasonNoRequestEvidence)
|
||||
}
|
||||
|
||||
// stageQuotation blocks a quoted command when a reporting frame surrounds it.
|
||||
func stageQuotation(t string, toks []string, joined string) (Frame, bool) {
|
||||
spans := extractQuotedSpans(t)
|
||||
if len(spans) == 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
commandSpan := false
|
||||
for _, sp := range spans {
|
||||
if isCommandishWithin(tokens(sp.content), strings.Join(tokens(sp.content), " ")) {
|
||||
commandSpan = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !commandSpan {
|
||||
return Frame{}, false
|
||||
}
|
||||
reasons := []Reason{ReasonQuotation}
|
||||
if hasReportFrame(toks, joined) {
|
||||
reasons = append(reasons, ReasonReportedSpeech)
|
||||
return Frame{Eligibility: Blocked, Reasons: reasons}, true
|
||||
}
|
||||
// a bare quoted command has no reporting frame: refusable but not a
|
||||
// definite prohibition either (it is at least ambiguous)
|
||||
return Frame{Eligibility: Ambiguous, Reasons: reasons}, true
|
||||
}
|
||||
|
||||
// isCommandishWithin reports the span content carrying command or capability
|
||||
// polarity itself — imperative, prohibition, or a can-form.
|
||||
func isCommandishWithin(toks []string, joined string) bool {
|
||||
if len(toks) == 0 {
|
||||
return false
|
||||
}
|
||||
if router.IsCommandProhibition(strings.Join(toks, " ")) {
|
||||
return true
|
||||
}
|
||||
if hasAny(toks, ruCanForms) || ruAbilityForms.in(toks, joined) || hasAny(toks, enCanForms) {
|
||||
return true
|
||||
}
|
||||
return anyFiniteVerb(toks)
|
||||
}
|
||||
|
||||
func hasReportFrame(toks []string, joined string) bool {
|
||||
if hasAny(toks, reportVerbs) {
|
||||
return true
|
||||
}
|
||||
return hasAny(toks, reportNouns)
|
||||
}
|
||||
|
||||
// stageReport blocks when a report frame governs a command clause: an
|
||||
// infinitive after the report verb, or a quoted imperative. Second-person
|
||||
// imperatives like "скажи/расскажи" are not in reportVerbs, so a request to
|
||||
// report ("расскажи мне, что сказал папа") passes through.
|
||||
func stageReport(t string, toks []string, joined string) (Frame, bool) {
|
||||
if !hasReportFrame(toks, joined) {
|
||||
return Frame{}, false
|
||||
}
|
||||
last := -1
|
||||
for i, w := range toks {
|
||||
if hasTok(reportVerbs, w) || hasTok(reportNouns, w) {
|
||||
last = i
|
||||
}
|
||||
}
|
||||
if last < 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
after := toks[last+1:]
|
||||
if len(after) == 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
// a quoted command after the frame counts as the governed clause
|
||||
for _, sp := range extractQuotedSpans(t) {
|
||||
if isCommandishWithin(tokens(sp.content), strings.Join(tokens(sp.content), " ")) {
|
||||
return blocked(ReasonReportedSpeech, ReasonQuotation), true
|
||||
}
|
||||
}
|
||||
if anyInfinitive(after) || hasAny(after, []string{"что", "чтобы", "чтоб"}) {
|
||||
return blocked(ReasonReportedSpeech), true
|
||||
}
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// stageHypothesis blocks a conditional scope whose clauses are hypothetical
|
||||
// (infinitive or subjunctive "бы") rather than a real condition→command.
|
||||
// "если будет дождь, выключи полив" keeps its imperative continuation and
|
||||
// passes through; it is a real conditional request, not a hypothetical.
|
||||
func stageHypothesis(toks []string, joined string) (Frame, bool) {
|
||||
idx := -1
|
||||
for _, m := range hypothesisMarkers {
|
||||
if i := indexTok(toks, m); i >= 0 && (idx < 0 || i < idx) {
|
||||
idx = i
|
||||
}
|
||||
}
|
||||
if idx < 0 {
|
||||
return Frame{}, false
|
||||
}
|
||||
post := toks[idx+1:]
|
||||
if len(post) == 0 || hasTok(post, "бы") || anyInfinitive(post) {
|
||||
return blocked(ReasonHypothetical), true
|
||||
}
|
||||
// a real condition clause is not hypothetical: «если будет дождь,
|
||||
// выключи полив» is a request. The dev dict does not cover «будет», so
|
||||
// the imperative is looked for anywhere, not just after the marker
|
||||
// («выключи свет, если будет дождь»).
|
||||
if !anyFiniteVerb(toks) {
|
||||
return blocked(ReasonHypothetical), true
|
||||
}
|
||||
return permissive(ReasonExplicitRequest), true
|
||||
}
|
||||
|
||||
// stageAdvisoryNegation blocks "не надо/не нужно/не стоит/не следует …".
|
||||
// (absent from the dev pool; covered by brief fixtures)
|
||||
func stageAdvisoryNegation(toks []string) (Frame, bool) {
|
||||
if len(toks) < 3 || toks[0] != "не" {
|
||||
return Frame{}, false
|
||||
}
|
||||
if !hasTok(toks[1:2], "надо") && !hasTok(toks[1:2], "нужно") &&
|
||||
!hasTok(toks[1:2], "стоит") && !hasTok(toks[1:2], "следует") &&
|
||||
!hasTok(toks[1:2], "требуется") {
|
||||
return Frame{}, false
|
||||
}
|
||||
rest := toks[2:]
|
||||
if anyInfinitive(rest) || anyFiniteVerb(rest) || hasAnyVerb(rest) {
|
||||
return blocked(ReasonNegatedCommand), true
|
||||
}
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// stageCapability encodes the measured modal matrix. Returns a decision when
|
||||
// modality alone settles the frame.
|
||||
func stageCapability(toks []string, joined string) (Frame, bool) {
|
||||
// "не мог бы ты …, пожалуйста" style conditional politeness is ordinary
|
||||
// modal politeness (the prohibition parser exempts it as such): a request.
|
||||
for _, pref := range politeNegativeModal {
|
||||
if strings.HasPrefix(joined, pref) {
|
||||
return permissive(ReasonExplicitRequest), true
|
||||
}
|
||||
}
|
||||
|
||||
// "… ли" directly after a can-form is a polar capability question:
|
||||
// "могу ли я …", "можешь ли ты …", "умеешь ли ты …", "можно ли …".
|
||||
// Checked before the modality arms so the polar reading wins.
|
||||
if hasTok(toks, "ли") {
|
||||
for i := 1; i < len(toks); i++ {
|
||||
if toks[i] != "ли" {
|
||||
continue
|
||||
}
|
||||
prev := toks[i-1]
|
||||
if hasTok(ruCanForms, prev) || prev == "можно" || hasTok(ruAbilityForms.single, prev) {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
ruAddr := hasAny(toks, ruAddress)
|
||||
ruCan := hasAny(toks, ruCanForms)
|
||||
ruAbil := ruAbilityForms.in(toks, joined)
|
||||
enCan := hasAny(toks, enCanForms)
|
||||
polite := politeness.in(toks, joined)
|
||||
|
||||
// addressed Russian can-form: capability question, always blocked.
|
||||
// ("ты можешь выключить свет, пожалуйста" included — 42/42 non-action.)
|
||||
if ruAddr && (ruCan || ruAbil) {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
|
||||
// ability forms (future/conditional/умеешь) are capability even bare and
|
||||
// even polite: "сможешь открыть окно, пожалуйста" is 7/7 non-action.
|
||||
if ruAbil && !ruCan {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
|
||||
// bare Russian present can-form: politeness is the request marker.
|
||||
if ruCan && !ruAddr {
|
||||
if polite {
|
||||
return Frame{}, false // modal-request positive evidence is found later
|
||||
}
|
||||
return ambiguous(ReasonAmbiguousModal), true
|
||||
}
|
||||
|
||||
// English can/could: "can you …, please" is a request (96/96 action in the
|
||||
// dev pool; the frame wraps a Russian imperative). Without politeness it
|
||||
// reads as a capability question and stays ambiguous.
|
||||
if enCan && !ruAddr {
|
||||
if polite {
|
||||
return Frame{}, false // positive modal-request evidence later
|
||||
}
|
||||
return ambiguous(ReasonAmbiguousModal), true
|
||||
}
|
||||
|
||||
// "можно" (permission): "можно ли …" is a permission question; a bare
|
||||
// "можно …" is a politeness-implicature request.
|
||||
if hasTok(toks, "можно") {
|
||||
if hasTok(toks, "ли") {
|
||||
return blocked(ReasonCapabilityQuestion), true
|
||||
}
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// bare "могу": a self-capability statement, not a request.
|
||||
if hasTok(toks, "могу") && !hasTok(toks, "ли") {
|
||||
return ambiguous(ReasonAmbiguousModal), true
|
||||
}
|
||||
|
||||
return Frame{}, false
|
||||
}
|
||||
|
||||
// hasRequestEvidence is the positive permissive trigger, reached only after
|
||||
// every block/ambiguity stage above has declined.
|
||||
func hasRequestEvidence(toks []string, joined string) bool {
|
||||
polite := politeness.in(toks, joined)
|
||||
enCan := hasAny(toks, enCanForms)
|
||||
|
||||
// 1. politeness + a verb (or an English modal) is explicit request
|
||||
// evidence: "можешь выключить свет, пожалуйста", "can you останови …,
|
||||
// please", "выключи свет, пожалуйста".
|
||||
if polite && (hasAnyVerb(toks) || enCan) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 2. first-person illocution frame: "я хочу …", "мне нужно …".
|
||||
if hasAny(toks, lexicon.FirstPerson()) && illocutionVerbs.in(toks, joined) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 3. impersonal need: "надо …", "нужно …", "пора …".
|
||||
if hasAny(toks, []string{"надо", "нужно", "следует", "пора", "требуется", "придётся", "придется"}) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 4. permission-implicature request: "можно выключить свет".
|
||||
if hasTok(toks, "можно") && !hasTok(toks, "ли") {
|
||||
return true
|
||||
}
|
||||
|
||||
// 5. reminder request in the parser's own exemption scope: the
|
||||
// prohibition parser declines «не забудь напомнить про свет» as a
|
||||
// reminder, not a prohibition — carry that into a request.
|
||||
if strings.HasPrefix(joined, "не забудь") && hasAny(toks, lexicon.ReminderVerbs()) {
|
||||
return true
|
||||
}
|
||||
|
||||
// 6. leading finite verb (imperative or otherwise tensed verb at command
|
||||
// position): "выключи свет", "покажи что запущено". Address and filler
|
||||
// particles are stripped first, so "мавен, выключи свет" rides the same
|
||||
// frame.
|
||||
lead := toks
|
||||
for len(lead) > 0 {
|
||||
first := lead[0]
|
||||
if !lexicon.IsFillerParticle(first) && !hasTok(wakeAddr, first) && !hasTok(ruAddress, first) {
|
||||
break
|
||||
}
|
||||
lead = lead[1:]
|
||||
}
|
||||
if len(lead) > 0 && isFiniteVerb(lead[0]) {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/kami/maven/internal/morph"
|
||||
)
|
||||
|
||||
func TestEvaluateFixtures(t *testing.T) {
|
||||
for _, fx := range Fixtures {
|
||||
got := Evaluate(fx.Utterance)
|
||||
if got.Eligibility != fx.Want {
|
||||
t.Errorf("%s: want %s got %s (reasons %v)", fx.Utterance, fx.Want, got.Eligibility, got.Reasons)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestMeasuredDiscriminators pins the corpus-verified numbers on the frozen
|
||||
// pool. These are the exact measurements the rules were built on, so a change
|
||||
// that moves them is a rule regression visible in the slice.
|
||||
func loadPool(t *testing.T) []Row {
|
||||
t.Helper()
|
||||
b, err := os.ReadFile("/tmp/mvn-s21/pool.json")
|
||||
if os.IsNotExist(err) {
|
||||
t.Skip("pool.json missing; run slice21_emit.py first")
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var rows []Row
|
||||
if err := json.Unmarshal(b, &rows); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
func TestDevCapabilityProhibition(t *testing.T) {
|
||||
if !morph.Available() {
|
||||
t.Skip("morph dict unavailable")
|
||||
}
|
||||
rows := loadPool(t)
|
||||
var n, pass, blockedN, ambig int
|
||||
for _, r := range rows {
|
||||
if !hasTok(r.Tags, "capability_question") {
|
||||
continue
|
||||
}
|
||||
n++
|
||||
switch Evaluate(variants(r.NText)[vOrig]).Eligibility {
|
||||
case Permissive:
|
||||
pass++
|
||||
case Blocked:
|
||||
blockedN++
|
||||
case Ambiguous:
|
||||
ambig++
|
||||
}
|
||||
}
|
||||
if n != 126 {
|
||||
t.Fatalf("capability-question rows = %d, want 126", n)
|
||||
}
|
||||
if pass != 0 {
|
||||
t.Fatalf("capability-question dangerous pass = %d, want 0", pass)
|
||||
}
|
||||
if blockedN != 126 && ambig != 0 {
|
||||
t.Errorf("blocked=%d ambig=%d, expect all 126 blocked", blockedN, ambig)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevBareCanPoliteIsAction(t *testing.T) {
|
||||
if !morph.Available() {
|
||||
t.Skip("morph dict unavailable")
|
||||
}
|
||||
rows := loadPool(t)
|
||||
total, action, perm := 0, 0, 0
|
||||
for _, r := range rows {
|
||||
toks := tokens(r.NText)
|
||||
if !hasTok(toks, "можешь") || hasAny(toks, ruAddress) ||
|
||||
hasAny(toks, append([]string{}, ruAbilityForms.single...)) {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
if r.Route == "action" {
|
||||
action++
|
||||
}
|
||||
if Evaluate(r.NText).Eligibility == Permissive {
|
||||
perm++
|
||||
}
|
||||
}
|
||||
if total != 127 || action != 127 {
|
||||
t.Fatalf("bare-можешь+polite: n=%d action=%d, want 127/127", total, action)
|
||||
}
|
||||
if perm != 127 {
|
||||
t.Fatalf("bare-можешь+polite permissive=%d, want 127", perm)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDevEnglishCanPoliteIsAction(t *testing.T) {
|
||||
if !morph.Available() {
|
||||
t.Skip("morph dict unavailable")
|
||||
}
|
||||
rows := loadPool(t)
|
||||
total, action, perm := 0, 0, 0
|
||||
for _, r := range rows {
|
||||
toks := tokens(r.NText)
|
||||
if !hasAny(toks, enCanForms) {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
if r.Route == "action" {
|
||||
action++
|
||||
}
|
||||
if Evaluate(r.NText).Eligibility == Permissive {
|
||||
perm++
|
||||
}
|
||||
}
|
||||
if total != 96 || action != 96 {
|
||||
t.Fatalf("can-rows: n=%d action=%d, want 96/96", total, action)
|
||||
}
|
||||
if perm != 96 {
|
||||
t.Fatalf("can-rows permissive=%d, want 96", perm)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoPermissiveRowEndsInQuestion(t *testing.T) {
|
||||
// asymmetric posture: the guard must never approve a trailing-? frame --
|
||||
// the corpus has 0 action rows ending in "?", and approving any would
|
||||
// bet on punctuation the slice has ruled uncertain.
|
||||
rows := loadPool(t)
|
||||
for _, r := range rows {
|
||||
if !strings.HasSuffix(r.NText, "?") {
|
||||
continue
|
||||
}
|
||||
if got := Evaluate(r.NText).Eligibility; got == Permissive {
|
||||
t.Errorf("canonical '?': %q approved (%s)", r.Text, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,566 @@
|
||||
// Slice 21 runner: report the deterministic execution-frame guard against the
|
||||
// frozen slice-20 dev pool.
|
||||
//
|
||||
// Reads the emit step's compact files (pool.json, pairs.json, sparse_oof.json
|
||||
// in /tmp/mvn-s21) and prints the report tables plus a machine-readable
|
||||
// guard_results.json. Reuses router/morph/lexicon parsers live inside this
|
||||
// module — the pool texts are the only data, no embedding is recomputed.
|
||||
//
|
||||
// Usage: go run ./cmd/semantic-router-experiment/slice21
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
const sparseThreshold = 0.715 // slice-18 §4 strict operating point (P>=0.95 best recall)
|
||||
|
||||
var familyPriority = []string{
|
||||
"capability_question", "question", "first_person_request",
|
||||
"modal_request", "polite_request", "reordered_target", "direct_imperative",
|
||||
}
|
||||
|
||||
func familyOf(tags []string) string {
|
||||
for _, f := range familyPriority {
|
||||
if hasTok(tags, f) {
|
||||
return f
|
||||
}
|
||||
}
|
||||
return "other"
|
||||
}
|
||||
|
||||
// ── pool row ──────────────────────────────────────────────────────────────
|
||||
|
||||
type Row struct {
|
||||
Idx int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
NText string `json:"n_text"`
|
||||
Route string `json:"route"`
|
||||
Y int `json:"y"`
|
||||
Tags []string `json:"tags"`
|
||||
CVFold int `json:"cv_fold"`
|
||||
SplitGp string `json:"split_group"`
|
||||
SourceID string `json:"source_id"`
|
||||
Family string
|
||||
VariantOf int
|
||||
}
|
||||
|
||||
type Pair struct{ Cap, Act int }
|
||||
|
||||
// ── stress variants ───────────────────────────────────────────────────────
|
||||
|
||||
var nofinalRe = regexp.MustCompile(`[?.!,;:]+$`)
|
||||
|
||||
// strip_punct mirrors the slice-18/19 python strip_punct: trailing sentence
|
||||
// punctuation, then every non-word/non-space rune.
|
||||
func stripPunct(t string) string {
|
||||
t = nofinalRe.ReplaceAllString(strings.TrimSpace(t), "")
|
||||
out := make([]rune, 0, len(t))
|
||||
var prevSpace bool
|
||||
for _, r := range t {
|
||||
if unicode.IsLetter(r) || unicode.IsNumber(r) {
|
||||
out = append(out, r)
|
||||
prevSpace = false
|
||||
} else if !prevSpace {
|
||||
out = append(out, ' ')
|
||||
prevSpace = true
|
||||
}
|
||||
}
|
||||
return strings.TrimSpace(string(out))
|
||||
}
|
||||
|
||||
func variants(nText string) [3]string {
|
||||
return [3]string{
|
||||
nText,
|
||||
nofinalRe.ReplaceAllString(strings.TrimSpace(nText), ""),
|
||||
stripPunct(nText),
|
||||
}
|
||||
}
|
||||
|
||||
const (
|
||||
vOrig = iota
|
||||
vNofinal
|
||||
vStrip
|
||||
)
|
||||
|
||||
var variantName = [3]string{"orig", "nofinal", "strip"}
|
||||
|
||||
type result struct {
|
||||
Frame Frame `json:"frame"`
|
||||
}
|
||||
|
||||
// ── metrics ──────────────────────────────────────────────────────────────
|
||||
|
||||
type triTab struct {
|
||||
Permissive, Blocked, Ambiguous int
|
||||
PermNonact, BlockedAction, AmbAction int
|
||||
}
|
||||
|
||||
type runAgg struct {
|
||||
n, action, tp, fp, fn int
|
||||
approved int
|
||||
capPermissive int
|
||||
}
|
||||
|
||||
func (a *runAgg) addApproved(approved bool, route string) {
|
||||
a.n++
|
||||
if route == "action" {
|
||||
a.action++
|
||||
}
|
||||
if approved {
|
||||
a.approved++
|
||||
if route == "action" {
|
||||
a.tp++
|
||||
} else {
|
||||
a.fp++
|
||||
}
|
||||
} else if route == "action" {
|
||||
a.fn++
|
||||
}
|
||||
}
|
||||
|
||||
func (a *runAgg) P() string { return fmtPct(frac(a.tp, a.tp+a.fp)) }
|
||||
func (a *runAgg) R() string { return fmtPct(frac(a.tp, a.action)) }
|
||||
func (a *runAgg) FA() int { return a.fp }
|
||||
func (a *runAgg) FArate() string {
|
||||
return fmtPct(frac(a.fp, a.n))
|
||||
}
|
||||
|
||||
func maxi(a, b int) int {
|
||||
if a > b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// frac is the guarded ratio the tables print (0/0 is 0).
|
||||
func frac(num, den int) float64 { return float64(num) / float64(maxi(den, 1)) }
|
||||
|
||||
func fmtPct(v float64) string { return fmt.Sprintf("%.1f%%", 100*v) }
|
||||
|
||||
// ── main ─────────────────────────────────────────────────────────────────
|
||||
|
||||
func main() {
|
||||
poolPath := flag.String("pool", "/tmp/mvn-s21/pool.json", "dev pool rows")
|
||||
pairsPath := flag.String("pairs", "/tmp/mvn-s21/pairs.json", "cap-vs-action pairs")
|
||||
sparsePath := flag.String("sparse", "/tmp/mvn-s21/sparse_oof.json", "slice-18 both OOF proba")
|
||||
outPath := flag.String("out", "/tmp/mvn-s21/guard_results.json", "machine-readable results")
|
||||
flag.Parse()
|
||||
|
||||
rows := mustLoad[[]Row](*poolPath)
|
||||
// pairs.json is bare [cap, act] index pairs; adapt into typed pairs.
|
||||
rawPairs := mustLoad[[][2]int](*pairsPath)
|
||||
pairs := make([]Pair, 0, len(rawPairs))
|
||||
for _, rp := range rawPairs {
|
||||
pairs = append(pairs, Pair{Cap: rp[0], Act: rp[1]})
|
||||
}
|
||||
sparseOOF := mustLoad[[]struct {
|
||||
Idx int `json:"idx"`
|
||||
Proba float64 `json:"proba"`
|
||||
}](*sparsePath)
|
||||
proba := make([]float64, len(rows))
|
||||
for _, s := range sparseOOF {
|
||||
proba[s.Idx] = s.Proba
|
||||
}
|
||||
|
||||
famPrio := 0
|
||||
for i := range rows {
|
||||
rows[i].Family = familyOf(rows[i].Tags)
|
||||
if rows[i].Family != "other" {
|
||||
famPrio++
|
||||
}
|
||||
}
|
||||
_ = famPrio
|
||||
|
||||
// verdicts per variant
|
||||
type rowRes struct {
|
||||
Idx int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
Route string `json:"route"`
|
||||
Family string `json:"family"`
|
||||
Tags []string `json:"tags"`
|
||||
Frames map[string]string `json:"frames"` // variant -> eligibility
|
||||
}
|
||||
|
||||
perVariant := make([][3]Frame, len(rows))
|
||||
fmt.Println("slice 21 — deterministic execution-frame guard on slice-20 dev pool")
|
||||
fmt.Println("==================================================================")
|
||||
|
||||
for i, r := range rows {
|
||||
vs := variants(r.NText)
|
||||
var fr [3]Frame
|
||||
for vi := 0; vi < 3; vi++ {
|
||||
fr[vi] = Evaluate(vs[vi])
|
||||
}
|
||||
perVariant[i] = fr
|
||||
}
|
||||
|
||||
// ── §3 three-way cross-tab (orig) ─────────────────────────────────────
|
||||
fmt.Println("\n## 1. Three-way eligibility × route (orig)")
|
||||
tab := triTab{}
|
||||
for i, r := range rows {
|
||||
switch perVariant[i][vOrig].Eligibility {
|
||||
case Permissive:
|
||||
tab.Permissive++
|
||||
if r.Route != "action" {
|
||||
tab.PermNonact++
|
||||
}
|
||||
case Blocked:
|
||||
tab.Blocked++
|
||||
if r.Route == "action" {
|
||||
tab.BlockedAction++
|
||||
}
|
||||
case Ambiguous:
|
||||
tab.Ambiguous++
|
||||
if r.Route == "action" {
|
||||
tab.AmbAction++
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Printf("permissive: %d blocked: %d ambiguous: %d\n", tab.Permissive, tab.Blocked, tab.Ambiguous)
|
||||
fmt.Printf(" permissive non-action: %d blocked action: %d ambiguous action: %d\n",
|
||||
tab.PermNonact, tab.BlockedAction, tab.AmbAction)
|
||||
|
||||
// ── §4 binary executable-gate metrics on orig ─────────────────────────
|
||||
fmt.Println("\n## 2. Binary executable gate (approve = permissive; deny = blocked|ambiguous)")
|
||||
a := runAgg{}
|
||||
capCov, capPerm, capAmb := 0, 0, 0
|
||||
for i, r := range rows {
|
||||
el := perVariant[i][vOrig].Eligibility
|
||||
a.addApproved(el == Permissive, r.Route)
|
||||
if hasTok(r.Tags, "capability_question") {
|
||||
capCov++
|
||||
switch el {
|
||||
case Permissive:
|
||||
capPerm++
|
||||
case Ambiguous:
|
||||
capAmb++
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Printf("approved: %d denied: %d (n=%d, action=%d)\n", a.approved, a.n-a.approved, a.n, a.action)
|
||||
fmt.Printf("action precision %s recall %s FA %d (%s)\n", a.P(), a.R(), a.FA(), a.FArate())
|
||||
fmt.Printf("capability-question dangerous pass: %d / %d (rate %s)\n",
|
||||
capPerm, capCov, fmtPct(frac(capPerm, capCov)))
|
||||
fmt.Printf("capability-question blocked %d, ambiguous %d\n", capCov-capPerm-capAmb, capAmb)
|
||||
|
||||
// ── §15 family stress (orig) ──────────────────────────────────────────
|
||||
fmt.Println("\n## 3. Family stress (orig; counts per eligibility)")
|
||||
fmt.Printf("%-24s %8s %8s %8s %8s\n", "family", "n", "perm", "block", "ambig")
|
||||
famOrder := []string{"direct_imperative", "polite_request", "modal_request", "first_person_request",
|
||||
"reordered_target", "capability_question", "question", "other"}
|
||||
famAgg := map[string]*triTab{}
|
||||
for _, f := range famOrder {
|
||||
famAgg[f] = &triTab{}
|
||||
}
|
||||
for i, r := range rows {
|
||||
t := famAgg[r.Family]
|
||||
if t == nil {
|
||||
continue
|
||||
}
|
||||
switch perVariant[i][vOrig].Eligibility {
|
||||
case Permissive:
|
||||
t.Permissive++
|
||||
if r.Route != "action" {
|
||||
t.PermNonact++
|
||||
}
|
||||
case Blocked:
|
||||
t.Blocked++
|
||||
case Ambiguous:
|
||||
t.Ambiguous++
|
||||
}
|
||||
}
|
||||
for _, f := range famOrder {
|
||||
t := famAgg[f]
|
||||
if t == nil {
|
||||
continue
|
||||
}
|
||||
n := t.Permissive + t.Blocked + t.Ambiguous
|
||||
if n == 0 {
|
||||
continue
|
||||
}
|
||||
fmt.Printf("%-24s %8d %8d %8d %8d\n", f, n, t.Permissive, t.Blocked, t.Ambiguous)
|
||||
}
|
||||
|
||||
// ── §cap-Q LOFO across stress variants ────────────────────────────────
|
||||
fmt.Println("\n## 4. Capability-question dangerous pass by stress variant")
|
||||
for vi := 0; vi < 3; vi++ {
|
||||
cp, cb, ca := 0, 0, 0
|
||||
for i, r := range rows {
|
||||
if !hasTok(r.Tags, "capability_question") {
|
||||
continue
|
||||
}
|
||||
switch perVariant[i][vi].Eligibility {
|
||||
case Permissive:
|
||||
cp++
|
||||
case Blocked:
|
||||
cb++
|
||||
case Ambiguous:
|
||||
ca++
|
||||
}
|
||||
}
|
||||
fmt.Printf(" %-8s dangerous-pass %d blocked %d ambiguous %d\n",
|
||||
variantName[vi], cp, cb, ca)
|
||||
}
|
||||
|
||||
// ── §pair test ────────────────────────────────────────────────────────
|
||||
fmt.Println("\n## 5. Paired action/capability (cap row must never clear)")
|
||||
capClear, actPerm, actAmbig, actBlock := 0, 0, 0, 0
|
||||
for _, p := range pairs {
|
||||
cel := perVariant[p.Cap][vOrig].Eligibility
|
||||
ael := perVariant[p.Act][vOrig].Eligibility
|
||||
if cel == Permissive {
|
||||
capClear++
|
||||
}
|
||||
switch ael {
|
||||
case Permissive:
|
||||
actPerm++
|
||||
case Ambiguous:
|
||||
actAmbig++
|
||||
case Blocked:
|
||||
actBlock++
|
||||
}
|
||||
}
|
||||
fmt.Printf("pairs %d: cap cleared %d (rate %s), action permissive %d, action ambiguous %d, action blocked %d\n",
|
||||
len(pairs), capClear, fmtPct(frac(capClear, len(pairs))),
|
||||
actPerm, actAmbig, actBlock)
|
||||
|
||||
// ── �safe composition §17 ─────────────────────────────────────────────
|
||||
fmt.Println("\n## 6. Composition: guard-alone / sparse-alone / guard→sparse (orig)")
|
||||
compose := map[string]*runAgg{
|
||||
"guard_alone": {},
|
||||
"sparse_alone": {},
|
||||
"guard_sparse": {},
|
||||
}
|
||||
for i, r := range rows {
|
||||
gPerm := perVariant[i][vOrig].Eligibility == Permissive
|
||||
sPerm := proba[i] >= sparseThreshold
|
||||
compose["guard_alone"].addApproved(gPerm, r.Route)
|
||||
compose["sparse_alone"].addApproved(sPerm, r.Route)
|
||||
compose["guard_sparse"].addApproved(gPerm && sPerm, r.Route)
|
||||
}
|
||||
fmt.Printf("%-14s %8s %8s %6s %10s %6s %10s\n", "policy", "P", "R", "FA", "FA rate", "capQ", "capQ rate")
|
||||
for _, name := range []string{"guard_alone", "sparse_alone", "guard_sparse"} {
|
||||
agg := compose[name]
|
||||
capQ := 0
|
||||
for i, r := range rows {
|
||||
if !hasTok(r.Tags, "capability_question") {
|
||||
continue
|
||||
}
|
||||
ok := false
|
||||
switch name {
|
||||
case "guard_alone":
|
||||
ok = perVariant[i][vOrig].Eligibility == Permissive
|
||||
case "sparse_alone":
|
||||
ok = proba[i] >= sparseThreshold
|
||||
case "guard_sparse":
|
||||
ok = perVariant[i][vOrig].Eligibility == Permissive && proba[i] >= sparseThreshold
|
||||
}
|
||||
if ok {
|
||||
capQ++
|
||||
}
|
||||
}
|
||||
fmt.Printf("%-14s %8s %8s %6d %10s %6d %10s\n", name, agg.P(), agg.R(), agg.FA(),
|
||||
agg.FArate(), capQ, fmtPct(float64(capQ)/126))
|
||||
}
|
||||
|
||||
// composition on strip too (brief §16 voice stress)
|
||||
fmt.Println("\n## 7. Composition on punctuation-stripped text (strip)")
|
||||
c2 := runAgg{}
|
||||
capQ2 := 0
|
||||
for i, r := range rows {
|
||||
gPerm := perVariant[i][vStrip].Eligibility == Permissive
|
||||
ok := gPerm && proba[i] >= sparseThreshold
|
||||
c2.addApproved(ok, r.Route)
|
||||
if hasTok(r.Tags, "capability_question") && ok {
|
||||
capQ2++
|
||||
}
|
||||
}
|
||||
fmt.Printf("guard→sparse strip: P %s R %s FA %d (%s) capQ pass %d\n",
|
||||
c2.P(), c2.R(), c2.FA(), c2.FArate(), capQ2)
|
||||
|
||||
// ── §19 manual classification scratch ─────────────────────────────────
|
||||
fmt.Println("\n## 8. Manual classification (scan material written to manual_class.json)")
|
||||
var dangerous []map[string]any
|
||||
var permNonact []map[string]any
|
||||
var deniedAction []map[string]any
|
||||
for i, r := range rows {
|
||||
fr := perVariant[i][vOrig]
|
||||
if hasTok(r.Tags, "capability_question") && fr.Eligibility == Permissive {
|
||||
dangerous = append(dangerous, map[string]any{
|
||||
"idx": r.Idx, "text": r.Text, "route": r.Route,
|
||||
"reasons": fr.Reasons,
|
||||
})
|
||||
}
|
||||
if fr.Eligibility == Permissive && r.Route != "action" {
|
||||
permNonact = append(permNonact, map[string]any{
|
||||
"idx": r.Idx, "text": r.Text, "route": r.Route,
|
||||
"family": r.Family, "reasons": fr.Reasons,
|
||||
})
|
||||
}
|
||||
if fr.Eligibility != Permissive && r.Route == "action" {
|
||||
deniedAction = append(deniedAction, map[string]any{
|
||||
"idx": r.Idx, "text": r.Text, "family": r.Family,
|
||||
"eligibility": fr.Eligibility.String(), "reasons": fr.Reasons,
|
||||
})
|
||||
}
|
||||
}
|
||||
writeManual(permNonact, deniedAction, dangerous)
|
||||
fmt.Printf("dangerous passes: %d permissive non-action: %d denied action: %d\n",
|
||||
len(dangerous), len(permNonact), len(deniedAction))
|
||||
groupAndSample("permissive non-action by reason+family", permNonact, 4)
|
||||
groupAndSample("denied action by reason+family", deniedAction, 4)
|
||||
|
||||
// ── fixtures ──────────────────────────────────────────────────────────
|
||||
fmt.Println("\n## 9. Brief fixtures")
|
||||
pass := 0
|
||||
for _, fx := range Fixtures {
|
||||
got := Evaluate(fx.Utterance)
|
||||
mark := "ok "
|
||||
if got.Eligibility != fx.Want {
|
||||
mark = "FAIL"
|
||||
} else {
|
||||
pass++
|
||||
}
|
||||
if got.Eligibility != fx.Want {
|
||||
fmt.Printf(" %s %-14s want %-10s got %-10s %s\n", mark, fx.Family,
|
||||
fx.Want, got.Eligibility.String(), fx.Utterance)
|
||||
}
|
||||
}
|
||||
fmt.Printf("fixtures: %d/%d passed\n", pass, len(Fixtures))
|
||||
|
||||
// write result file
|
||||
rr := make([]rowRes, 0, len(rows))
|
||||
for i, r := range rows {
|
||||
fr := [3]string{"", "", ""}
|
||||
for vi := 0; vi < 3; vi++ {
|
||||
fr[vi] = perVariant[i][vi].Eligibility.String()
|
||||
}
|
||||
rr = append(rr, rowRes{
|
||||
Idx: r.Idx, Text: r.Text, Route: r.Route, Family: r.Family, Tags: r.Tags,
|
||||
Frames: map[string]string{
|
||||
"orig": fr[vOrig], "nofinal": fr[vNofinal], "strip": fr[vStrip],
|
||||
},
|
||||
})
|
||||
}
|
||||
if *outPath != "" {
|
||||
mustSave(*outPath, map[string]any{
|
||||
"pool": "/tmp/mvn-s21/pool.json",
|
||||
"rows": rr,
|
||||
"aggregates": map[string]any{
|
||||
"tab": tab,
|
||||
"capq_pass": capPerm,
|
||||
"capq_blocked": capCov - capPerm - capAmb,
|
||||
"capq_ambiguous": capAmb,
|
||||
"binary": map[string]any{"tp": a.tp, "fp": a.fp, "fn": a.fn, "approved": a.approved, "n": a.n},
|
||||
"pairs": map[string]any{"n": len(pairs), "cap_cleared": capClear, "act_permissive": actPerm},
|
||||
"guard_sparse": map[string]any{"tp": compose["guard_sparse"].tp, "fp": compose["guard_sparse"].fp, "fn": compose["guard_sparse"].fn},
|
||||
"dangerous_passes": len(dangerous),
|
||||
"perm_nonact_count": len(permNonact),
|
||||
"denied_action": len(deniedAction),
|
||||
},
|
||||
})
|
||||
fmt.Println("wrote", *outPath)
|
||||
}
|
||||
}
|
||||
|
||||
// ── manual classification helpers ─────────────────────────────────────────
|
||||
|
||||
func writeManual(permNonact, deniedAction, dangerous []map[string]any) {
|
||||
writeJSON("/tmp/mvn-s21/manual_class.json", map[string]any{
|
||||
"dangerous_passes": dangerous,
|
||||
"permissive_non_action": permNonact,
|
||||
"denied_action": deniedAction,
|
||||
})
|
||||
}
|
||||
|
||||
func groupAndSample(title string, rows []map[string]any, sample int) {
|
||||
type g struct {
|
||||
key string
|
||||
n int
|
||||
texts []string
|
||||
}
|
||||
groups := map[string]*g{}
|
||||
var order []string
|
||||
for _, r := range rows {
|
||||
var family, reason, el string
|
||||
if v, ok := r["family"].(string); ok {
|
||||
family = v
|
||||
}
|
||||
if v, ok := r["eligibility"].(string); ok {
|
||||
el = v
|
||||
}
|
||||
if rs, ok := r["reasons"].([]Reason); ok {
|
||||
rs2 := make([]string, len(rs))
|
||||
for k, rr := range rs {
|
||||
rs2[k] = rr.String()
|
||||
}
|
||||
reason = strings.Join(rs2, ",")
|
||||
} else if rs, ok := r["reasons"].([]string); ok {
|
||||
reason = strings.Join(rs, ",")
|
||||
}
|
||||
key := fmt.Sprintf("family=%s elig=%s reason=%s", family, el, reason)
|
||||
if _, ok := groups[key]; !ok {
|
||||
groups[key] = &g{key: key}
|
||||
order = append(order, key)
|
||||
}
|
||||
groups[key].n++
|
||||
if len(groups[key].texts) < sample {
|
||||
groups[key].texts = append(groups[key].texts, firstN(fmt.Sprint(r["text"]), 60))
|
||||
}
|
||||
}
|
||||
fmt.Printf("%s (%d rows):\n", title, len(rows))
|
||||
for _, key := range order {
|
||||
gr := groups[key]
|
||||
fmt.Printf(" %-58s n=%d %s\n", gr.key, gr.n, strings.Join(gr.texts, " | "))
|
||||
}
|
||||
}
|
||||
|
||||
func firstN(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n] + "…"
|
||||
}
|
||||
|
||||
// ── io helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
func mustLoad[T any](path string) T {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var v T
|
||||
if err := json.Unmarshal(b, &v); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "json:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func mustSave(path string, v any) {
|
||||
b, err := json.MarshalIndent(v, "", " ")
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := os.WriteFile(path, b, 0o644); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSON(path string, v any) {
|
||||
b, _ := json.MarshalIndent(v, "", " ")
|
||||
_ = os.WriteFile(path, b, 0o644)
|
||||
}
|
||||
|
||||
var _ = sort.Strings
|
||||
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 21 emit: deterministic execution-frame guard — data files for the Go harness
|
||||
==================================================================================
|
||||
|
||||
Slice 18 showed the sparse lexical gate owns the aggregate boundary (PR-AUC
|
||||
0.838, strict operating point at threshold 0.715 with P>=0.95 | R=0.264) and
|
||||
slice 19/20 showed learning heads collapse on capability-question LOFO. Slice 21
|
||||
tests the deterministic alternative: a rule engine over existing parsers that
|
||||
decides execution eligibility as a three-way gate (permissive / blocked /
|
||||
ambiguous), never itself routing.
|
||||
|
||||
This script only repackages the frozen dev pool for the Go harness. It reuses
|
||||
slice 18's feature builders and grouped-CV and slice 19's pair builder verbatim,
|
||||
so the numbers the Go side reports are the same populations the accepts
|
||||
measured. It writes:
|
||||
|
||||
/tmp/mvn-s21/pool.json dev rows: idx, text, n_text, route, y, tags,
|
||||
cv_fold, split_group, source_id, family
|
||||
/tmp/mvn-s21/pairs.json capability-vs-action pairs (slice-19 builder)
|
||||
/tmp/mvn-s21/sparse_oof.json slice-18 "both" grouped-CV OOF proba per row
|
||||
(for the §17 guard+sparse composition)
|
||||
|
||||
No training happens here and no label is changed. The guard itself is Go.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402
|
||||
import slice19_main # noqa: E402
|
||||
|
||||
OUT_DIR = "/tmp/mvn-s21"
|
||||
SPARSE_THRESHOLD = 0.715 # slice-18 §4 strict-operating-point (P>=0.95 best recall)
|
||||
|
||||
|
||||
def main():
|
||||
# Population = the exact slice-20 dev pool (s19.load_dev): every dev_pool
|
||||
# row, fast-path included. The guard is evaluated on what slice 20 measured.
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
print(f"dev pool (all dev_pool rows): {len(dev)} rows")
|
||||
print(f"corpus meta: {meta.get('dev_count', '?')} dev rows declared, "
|
||||
f"{meta.get('route_counts', {}).get('action', '?')} action declared")
|
||||
|
||||
n_texts = [slice18_sparse.normalize_match_text(e["text"]) for e in dev]
|
||||
|
||||
rows = []
|
||||
by_route = {}
|
||||
by_family = {}
|
||||
for i, (e, nt) in enumerate(zip(dev, n_texts)):
|
||||
tags = sorted(set(e.get("tags", [])))
|
||||
route = e["route"]
|
||||
fam = slice19_main.family_of(set(tags))
|
||||
by_route[route] = by_route.get(route, 0) + 1
|
||||
by_family[fam] = by_family.get(fam, 0) + 1
|
||||
rows.append({
|
||||
"idx": i,
|
||||
"text": e["text"],
|
||||
"n_text": nt,
|
||||
"route": route,
|
||||
"y": 1 if route == "action" else 0,
|
||||
"tags": tags,
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"source_id": e["source_id"],
|
||||
})
|
||||
|
||||
print("routes:", by_route)
|
||||
print("families:", by_family)
|
||||
|
||||
# ── pairs (slice-19 builder, exact population) ─────────────────────────
|
||||
ldev = [{
|
||||
"text_orig": nt,
|
||||
"route": r["route"],
|
||||
"y": r["y"],
|
||||
"cv_fold": r["cv_fold"],
|
||||
"tags": set(r["tags"]),
|
||||
"source_id": r["source_id"],
|
||||
} for r, nt in zip(rows, n_texts)]
|
||||
pairs = slice19_main.build_pairs(ldev, n_texts)
|
||||
print(f"pairs: {len(pairs)}")
|
||||
|
||||
# ── slice-18 "both" grouped-CV OOF proba, aligned to row index ────────
|
||||
y = [1 if r["route"] == "action" else 0 for r in rows]
|
||||
folds = [r["cv_fold"] for r in rows]
|
||||
X, _vec = slice18_sparse.build_features(n_texts, "both")
|
||||
print(f"sparse 'both' X: {X.shape}")
|
||||
yb = np.array(y)
|
||||
folds_arr = np.array(folds)
|
||||
idx_proba = {}
|
||||
for te_fold in sorted(set(folds)):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yb[tr])
|
||||
p = clf.predict_proba(X[te])[:, 1]
|
||||
te_idx = np.where(te)[0]
|
||||
for k, i in enumerate(te_idx):
|
||||
idx_proba[int(i)] = float(p[k])
|
||||
assert len(idx_proba) == len(rows)
|
||||
sparse_oof = [{"idx": i, "proba": idx_proba[i]} for i in range(len(rows))]
|
||||
pred = [1 if idx_proba[i] >= 0.5 else 0 for i in range(len(rows))]
|
||||
tp = sum(1 for i in range(len(rows)) if y[i] == 1 and pred[i] == 1)
|
||||
fp = sum(1 for i in range(len(rows)) if y[i] == 0 and pred[i] == 1)
|
||||
fn = sum(1 for i in range(len(rows)) if y[i] == 1 and pred[i] == 0)
|
||||
print(f"sparse both OOF @0.5: P={tp/max(tp+fp,1):.3f} R={tp/max(tp+fn,1):.3f} "
|
||||
f"FA={fp} ({fp/len(rows):.4f})")
|
||||
pred21 = [1 if idx_proba[i] >= SPARSE_THRESHOLD else 0 for i in range(len(rows))]
|
||||
tp = sum(1 for i in range(len(rows)) if y[i] == 1 and pred21[i] == 1)
|
||||
fp = sum(1 for i in range(len(rows)) if y[i] == 0 and pred21[i] == 1)
|
||||
fn = sum(1 for i in range(len(rows)) if y[i] == 1 and pred21[i] == 0)
|
||||
print(f"sparse both OOF @{SPARSE_THRESHOLD}: P={tp/max(tp+fp,1):.3f} "
|
||||
f"R={tp/max(tp+fn,1):.3f} FA={fp} ({fp/len(rows):.4f})")
|
||||
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
|
||||
json.dump(rows, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "pairs.json"), "w") as f:
|
||||
json.dump([[c, a] for c, a in pairs], f)
|
||||
with open(os.path.join(OUT_DIR, "sparse_oof.json"), "w") as f:
|
||||
json.dump(sparse_oof, f)
|
||||
print(f"wrote {OUT_DIR}/{{pool,pairs,sparse_oof}}.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,65 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
// headsMain runs the deployed cascade minus the resident LLM: stage-0
|
||||
// grammars → routing heads (fine-tuned e5 copy + softmax, router_heads.onnx,
|
||||
// 0.6 decline threshold) → ONNX-embedder nearest-centroid classifier →
|
||||
// 0.55 confidence gate. This is what a production turn takes when the model
|
||||
// server is out (docs/routing.md: pickLLMRouter degrades to the classifier).
|
||||
//
|
||||
// The classifier is seeded from models/seeds like the daemon's seedClassifier,
|
||||
// embedded with the real multilingual-e5-small model rather than the block
|
||||
// hash, so this is the closest headless reproduction of the authoritative
|
||||
// router output the slice-22 report can run.
|
||||
//
|
||||
// Requires the ONNX model files and a libonnxruntime.so. Pass the library via
|
||||
// the MAVEN_ONNX_LIB environment variable, exactly as the daemon does.
|
||||
func headsMain(poolPath, outPath string) {
|
||||
lib := os.Getenv("MAVEN_ONNX_LIB")
|
||||
if lib == "" {
|
||||
fmt.Fprintln(os.Stderr, "heads mode needs MAVEN_ONNX_LIB pointing at libonnxruntime.so")
|
||||
os.Exit(2)
|
||||
}
|
||||
const (
|
||||
embedModel = "models/embedder/multilingual-e5-small/model_quantized.onnx"
|
||||
tokPath = "models/embedder/multilingual-e5-small/tokenizer.json"
|
||||
headsModel = "models/embedder/router-heads/router_heads.onnx"
|
||||
)
|
||||
emb, err := router.NewONNXEmbedder(embedModel, tokPath, lib)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "heads: embedder: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer emb.Close()
|
||||
|
||||
cls := router.NewClassifier(emb)
|
||||
seedClassifier(cls)
|
||||
|
||||
heads, err := router.NewRouterHeads(headsModel, tokPath)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "heads: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer heads.Close()
|
||||
|
||||
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
|
||||
r := router.New(router.Config{
|
||||
Grammars: router.StageZeroGrammars(acts),
|
||||
Classifier: cls,
|
||||
Extractor: router.Extractor{
|
||||
Time: router.StubDateTimeParser{},
|
||||
Acts: acts,
|
||||
Facts: router.DefaultFactParser{},
|
||||
},
|
||||
Threshold: 0.55,
|
||||
Heads: heads,
|
||||
})
|
||||
runOverPool(r, poolPath, outPath)
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
// Seed loading replicated from cmd/mavend/voicewire.go (seedClassifier) and
|
||||
// internal/router/semantic/helpers_test.go, which build the same classifier
|
||||
// from models/seeds/<intent>.txt. The daemon and the eval fixture must agree
|
||||
// on the seeds; so must a measurement.
|
||||
const seedDir = "models/seeds"
|
||||
|
||||
var seedIntents = []router.Intent{
|
||||
router.IntentAct, router.IntentReminder, router.IntentFact,
|
||||
router.IntentNote, router.IntentQuery, router.IntentChat, router.IntentSystem,
|
||||
}
|
||||
|
||||
// buildMinimalRouter reproduces internal/router/semantic/buildMinimalRouter:
|
||||
// the daemon's grammar set, a hash-embedder classifier seeded from
|
||||
// models/seeds, and the deployed 0.55 threshold. Deterministic and
|
||||
// reproducible. The ONNX embedder and the routing heads score elsewhere;
|
||||
// this is the floor the eval fixture reports as the legacy baseline.
|
||||
func buildMinimalRouter() *router.Router {
|
||||
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
|
||||
cls := router.NewClassifier(router.NewHashEmbedder(1024))
|
||||
seedClassifier(cls)
|
||||
return router.New(router.Config{
|
||||
Grammars: router.StageZeroGrammars(acts),
|
||||
Classifier: cls,
|
||||
Extractor: router.Extractor{
|
||||
Time: router.StubDateTimeParser{},
|
||||
Acts: acts,
|
||||
Facts: router.DefaultFactParser{},
|
||||
},
|
||||
Threshold: 0.55,
|
||||
})
|
||||
}
|
||||
|
||||
func seedClassifier(c *router.Classifier) {
|
||||
// Walk up to find models/seeds like the daemon's seedPath, so the program
|
||||
// can run from any depth of the repo tree.
|
||||
dir := seedDir
|
||||
for i := 0; i < 5; i++ {
|
||||
if st, err := os.Stat(dir); err == nil && st.IsDir() {
|
||||
break
|
||||
}
|
||||
dir = filepath.Join("..", dir)
|
||||
}
|
||||
ctx := context.Background()
|
||||
total := 0
|
||||
for _, intent := range seedIntents {
|
||||
path := filepath.Join(dir, string(intent)+".txt")
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
log.Printf("legacy: open seed %s: %v", path, err)
|
||||
continue
|
||||
}
|
||||
sc := bufio.NewScanner(f)
|
||||
lines := []string{}
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, line)
|
||||
}
|
||||
f.Close()
|
||||
sort.Strings(lines)
|
||||
for _, line := range lines {
|
||||
if err := c.AddExample(ctx, intent, line); err != nil {
|
||||
log.Printf("legacy: seed %s %q: %v", intent, line, err)
|
||||
continue
|
||||
}
|
||||
total++
|
||||
}
|
||||
}
|
||||
log.Printf("legacy: loaded %d seed examples from %s", total, dir)
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// Legacy-baseline runner for slice 22: run the actual router cascade (stage 0
|
||||
// grammars → hash-embedder nearest-centroid classifier → 0.55 confidence gate)
|
||||
// over the frozen residual non-action dev pool and project each decision into
|
||||
// the five-way non-action semantic space.
|
||||
//
|
||||
// Projection rules (the daemon's behaviour, not just ScoreLegacy's):
|
||||
// - route error → uncertain
|
||||
// - Clarify=true (stage 3) → uncertain: the daemon asks, it does not commit
|
||||
// to a semantic bucket
|
||||
// - chat/query/fact+note/system → conversation/knowledge/memory_write/system
|
||||
// - act/reminder on a trusted non-action row → class "action" recorded
|
||||
// VERBATIM with illegal_action_prediction=true; never mapped to uncertain
|
||||
// - anything else → uncertain
|
||||
//
|
||||
// Reads /tmp/mvn-s22/pool.json (emit step) and writes /tmp/mvn-s22/legacy.json
|
||||
// with both the raw decision fields and the projected class, plus a summary
|
||||
// printout. No embedding is recomputed and no label is changed.
|
||||
|
||||
type poolRow struct {
|
||||
IDX int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
NText string `json:"n_text"`
|
||||
Route string `json:"route"`
|
||||
Tags []string `json:"tags"`
|
||||
CVFold int `json:"cv_fold"`
|
||||
SplitGroup string `json:"split_group"`
|
||||
FamilyID string `json:"family_id"`
|
||||
SourceID string `json:"source_id"`
|
||||
}
|
||||
|
||||
type legacyRow struct {
|
||||
IDX int `json:"idx"`
|
||||
Text string `json:"text"`
|
||||
Route string `json:"route"`
|
||||
Intent string `json:"intent"`
|
||||
Class string `json:"class"`
|
||||
Illegal bool `json:"illegal_action_prediction"`
|
||||
Confidence float64 `json:"confidence"`
|
||||
Stage int `json:"stage"`
|
||||
Clarify bool `json:"clarify"`
|
||||
Producer string `json:"producer"`
|
||||
Error string `json:"error,omitempty"`
|
||||
SourceID string `json:"source_id"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
var mode, poolPath, outPath string
|
||||
flag.StringVar(&mode, "mode", "legacy", "baseline mode: legacy (hash classifier) or heads (ONNX cascade minus LLM)")
|
||||
flag.StringVar(&poolPath, "pool", "/tmp/mvn-s22/pool.json", "emit-step pool.json")
|
||||
flag.StringVar(&outPath, "out", "/tmp/mvn-s22/legacy.json", "output path")
|
||||
flag.Parse()
|
||||
switch mode {
|
||||
case "legacy":
|
||||
legacyMain(poolPath, outPath)
|
||||
case "heads":
|
||||
headsMain(poolPath, outPath)
|
||||
default:
|
||||
fmt.Fprintf(os.Stderr, "unknown -mode %q\n", mode)
|
||||
os.Exit(2)
|
||||
}
|
||||
}
|
||||
|
||||
func legacyMain(poolPath, outPath string) {
|
||||
runOverPool(buildMinimalRouter(), poolPath, outPath)
|
||||
}
|
||||
|
||||
func runOverPool(r *router.Router, poolPath, outPath string) {
|
||||
raw, err := os.ReadFile(poolPath)
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
var rows []poolRow
|
||||
if err := json.Unmarshal(raw, &rows); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
now := time.Now()
|
||||
|
||||
out := make([]legacyRow, 0, len(rows))
|
||||
classCount := map[string]int{}
|
||||
for _, pr := range rows {
|
||||
d, err := r.Route(ctx, router.NormalizedInput{Text: pr.Text}, now)
|
||||
lr := legacyRow{
|
||||
IDX: pr.IDX,
|
||||
Text: pr.Text,
|
||||
Route: pr.Route,
|
||||
SourceID: pr.SourceID,
|
||||
}
|
||||
if err != nil {
|
||||
lr.Class = "uncertain"
|
||||
lr.Error = err.Error()
|
||||
} else {
|
||||
lr.Intent = string(d.Intent)
|
||||
lr.Confidence = d.Confidence
|
||||
lr.Stage = d.Stage
|
||||
lr.Clarify = d.Clarify
|
||||
lr.Producer = string(d.Producer)
|
||||
}
|
||||
lr.Class, lr.Illegal = project(d, err)
|
||||
classCount[lr.Class]++
|
||||
out = append(out, lr)
|
||||
}
|
||||
|
||||
if err := writeJSON(outPath, out); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Printf("legacy baseline over %d residual non-action rows:\n", len(out))
|
||||
for _, c := range []string{"conversation", "knowledge", "memory_write", "system", "uncertain", "action"} {
|
||||
fmt.Printf(" %-14s %d (%.1f%%)\n", c, classCount[c], 100*float64(classCount[c])/float64(len(out)))
|
||||
}
|
||||
fmt.Printf(" illegal_action_prediction: %d\n", classCount["action"])
|
||||
// Grammar hits inside a corpus-residual population would be a
|
||||
// corpus/harness disagreement worth telling the report about: the corpus
|
||||
// marked each row not-fast-path-resolved, so a current stage-0 rule
|
||||
// resolving it means the corpus's fast-path mirror is stale or a grammar
|
||||
// landed after the corpus froze.
|
||||
gh := 0
|
||||
ghByRoute := map[string]int{}
|
||||
ghByIntent := map[string]int{}
|
||||
for _, lr := range out {
|
||||
if lr.Producer == string(router.RouteProducerGrammar) {
|
||||
gh++
|
||||
ghByRoute[lr.Route]++
|
||||
ghByIntent[lr.Intent]++
|
||||
}
|
||||
}
|
||||
fmt.Printf(" stage-0 grammar hits: %d\n", gh)
|
||||
if gh > 0 {
|
||||
fmt.Printf(" by ground-truth route: %v\n", ghByRoute)
|
||||
fmt.Printf(" by grammar intent: %v\n", ghByIntent)
|
||||
}
|
||||
}
|
||||
|
||||
// project maps the router's authoritative output into the five-way non-action
|
||||
// space, or to the "action" bucket verbatim when the router calls an act or a
|
||||
// reminder on a non-action row.
|
||||
func project(d router.Decision, err error) (string, bool) {
|
||||
if err != nil {
|
||||
return "uncertain", false
|
||||
}
|
||||
if d.Clarify {
|
||||
return "uncertain", false
|
||||
}
|
||||
switch d.Intent {
|
||||
case router.IntentChat:
|
||||
return "conversation", false
|
||||
case router.IntentQuery:
|
||||
return "knowledge", false
|
||||
case router.IntentFact, router.IntentNote:
|
||||
return "memory_write", false
|
||||
case router.IntentSystem:
|
||||
return "system", false
|
||||
case router.IntentAct, router.IntentReminder:
|
||||
return "action", true
|
||||
default:
|
||||
return "uncertain", false
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSON(path string, v any) error {
|
||||
fh, err := os.Create(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer fh.Close()
|
||||
w := bufio.NewWriter(fh)
|
||||
enc := json.NewEncoder(w)
|
||||
enc.SetIndent("", " ")
|
||||
if err := enc.Encode(v); err != nil {
|
||||
return err
|
||||
}
|
||||
return w.Flush()
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 22 emit: five-way residual non-action semantic router — data files
|
||||
========================================================================
|
||||
|
||||
Slice 21 accepted the deterministic execution-frame guard (docs/evals/
|
||||
2026-09-07-execution-frame-guard.md). Slice 22 returns to the coarse non-action
|
||||
router that the guard hands to: after TryFastPath misses and the guard passes,
|
||||
the remaining utterance is one of five non-action semantics — conversation,
|
||||
knowledge, memory_write, system, uncertain. Action rows never reach this
|
||||
router; they are usable only as out-of-domain probes, never in primary metrics.
|
||||
|
||||
This script only repackages the frozen dev pool for the Go legacy baseline and
|
||||
the Python experiment. It reuses slice 18's loader/filters and slice 19's
|
||||
normalizers verbatim, so the population here is the same one slices 18-21
|
||||
measured. It writes:
|
||||
|
||||
/tmp/mvn-s22/pool.json residual non-action dev rows: idx, text, n_text,
|
||||
route, tags, cv_fold, split_group, family_id,
|
||||
source_id (1652 rows)
|
||||
/tmp/mvn-s22/ood.json residual ACTION dev rows (766): same shape; OOD
|
||||
probes only, never primary metrics
|
||||
/tmp/mvn-s22/stats.json population summary (routes, families, folds)
|
||||
|
||||
idx is the row's position among dev_pool rows in dev-pool order, so the Python
|
||||
experiment can align the embedding vectors from /tmp/mvn-experiment/embeddings.json
|
||||
by index exactly as slice19.load_dev does.
|
||||
|
||||
No training happens here and no label is changed.
|
||||
|
||||
Population (verified 2026-09-08 from the frozen file):
|
||||
dev_pool 2490
|
||||
dev residual 2418 (= dev_pool minus fast_path_resolved)
|
||||
residual non-action 1652 knowledge 715 / memory_write 553 / system 184 /
|
||||
uncertain 107 / conversation 93
|
||||
residual action 766 (OOD probes only)
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402 (normalize_match_text, load_data, filters)
|
||||
|
||||
OUT_DIR = "/tmp/mvn-s22"
|
||||
|
||||
ROUTES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
|
||||
|
||||
def main():
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
print(f"dev pool: {len(dev)} rows "
|
||||
f"(meta declares dev_count={meta.get('dev_count')})")
|
||||
|
||||
rows = []
|
||||
for i, e in enumerate(dev):
|
||||
if not e["fast_path_resolved"]:
|
||||
rows.append({
|
||||
"idx": i,
|
||||
"text": e["text"],
|
||||
"n_text": slice18_sparse.normalize_match_text(e["text"]),
|
||||
"route": e["route"],
|
||||
"tags": sorted(set(e.get("tags", []))),
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"family_id": e["family_id"],
|
||||
"source_id": e["source_id"],
|
||||
})
|
||||
|
||||
na = [r for r in rows if r["route"] != "action"]
|
||||
ood = [r for r in rows if r["route"] == "action"]
|
||||
print(f"residual rows: {len(rows)} non-action: {len(na)} action(OOD): {len(ood)}")
|
||||
|
||||
by_route = {}
|
||||
for r in na:
|
||||
by_route[r["route"]] = by_route.get(r["route"], 0) + 1
|
||||
print("routes:", by_route)
|
||||
assert sum(by_route.values()) == len(na)
|
||||
assert set(ROUTES) == set(by_route), "route set must be the five-way"
|
||||
|
||||
by_family = {}
|
||||
for r in na:
|
||||
by_family[r["family_id"]] = by_family.get(r["family_id"], 0) + 1
|
||||
by_fold = {}
|
||||
for r in na:
|
||||
by_fold[r["cv_fold"]] = by_fold.get(r["cv_fold"], 0) + 1
|
||||
print(f"family_ids: {len(by_family)} split_groups: {len(set(r['split_group'] for r in na))}")
|
||||
print("folds:", by_fold)
|
||||
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
|
||||
json.dump(na, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "ood.json"), "w") as f:
|
||||
json.dump(ood, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "stats.json"), "w") as f:
|
||||
json.dump({
|
||||
"dev_count": len(dev),
|
||||
"residual_count": len(rows),
|
||||
"non_action_count": len(na),
|
||||
"action_ood_count": len(ood),
|
||||
"routes": by_route,
|
||||
"family_ids": len(by_family),
|
||||
"folds": by_fold,
|
||||
"top_family": dict(sorted(by_family.items(), key=lambda kv: -kv[1])[:15]),
|
||||
}, f, ensure_ascii=False, indent=1)
|
||||
print(f"wrote {OUT_DIR}/{{pool,ood,stats}}.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,560 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 22: five-way residual non-action semantic router (experiment)
|
||||
==================================================================
|
||||
|
||||
After TryFastPath misses and the ExecutionFrameGuard passes, the residual
|
||||
utterance is one of five non-action semantics: conversation, knowledge,
|
||||
memory_write, system, uncertain. This measures whether the deployed e5-small
|
||||
embeddings (384-d, query-prefixed, mean-pooled, L2, frozen) fed to a linear
|
||||
softmax head suffice, and how they compare to the legacy router, to floors,
|
||||
and to the deployed routing heads.
|
||||
|
||||
Population: the frozen dev-pool residual non-action rows (1652; the pool
|
||||
written by slice22_emit.py). Action rows (766) are out-of-domain probes only.
|
||||
|
||||
Metrics written to /tmp/mvn-s22/results.json:
|
||||
§1 population
|
||||
§2 legacy baseline (legacy.json / legacy_heads.json): acc, macro-F1,
|
||||
per-class P/R/F1, confusion, illegal_action_prediction count
|
||||
§3 e5-linear primary head: C grid, grouped CV OOF, per-fold P/R/F1 +
|
||||
variance + composition
|
||||
§4 floors: majority, centroid (cosine nearest-mean), sparse word+char
|
||||
TF-IDF logistic (slice18 builder), all grouped CV
|
||||
§5 route-family (family_id) leave-family-out
|
||||
§6 knowledge vs memory_write: matched pairs (water/homelab/task) ordering
|
||||
§7 uncertain as an explicit class: P/R/F1 + top confusions
|
||||
§8 OOF confidence: max-softmax correct/wrong, ECE, log-loss, Brier,
|
||||
coverage/accuracy/macro-F1 abstention curves (no threshold chosen)
|
||||
§9 action OOD probes: fold models applied to the 766 action rows
|
||||
§10 artifact cost: head params, serialized bytes, incremental head latency
|
||||
|
||||
No corpus label is changed. No frozen-holdout rows are inspected.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402
|
||||
import slice19_main # noqa: E402
|
||||
|
||||
EMB_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
OUT_DIR = "/tmp/mvn-s22"
|
||||
CLASSES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
CLASS_PREFIX = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
C_GRID = [0.1, 1.0, 10.0]
|
||||
|
||||
# Route-family holdouts the report calls out by name (slice-22 brief): every
|
||||
# family that is not part of the shared subject inventory on either side.
|
||||
HOLDOUT_GROUPS = {
|
||||
"capability": ["knowledge:capability-ha", "knowledge:capability-tool"],
|
||||
"world": ["knowledge:world-def", "knowledge:world-explain"],
|
||||
"calendar": ["knowledge:calendar", "knowledge:calendar-time", "knowledge:calendar-next"],
|
||||
"recall": ["knowledge:recall-fact", "knowledge:recall-note", "knowledge:recall-possessive"],
|
||||
"fact": ["fact:meal", "fact:water", "fact:sleep", "fact:shower", "fact:break", "fact:pills", "fact:exercise"],
|
||||
"note": ["note:idea", "note:homelab", "note:task"],
|
||||
"remember": ["free:remember"],
|
||||
"system": None, # all system:*
|
||||
"conversation": None,
|
||||
"uncertain": None,
|
||||
}
|
||||
|
||||
|
||||
def load_pool_and_embeds():
|
||||
with open(os.path.join(OUT_DIR, "pool.json")) as f:
|
||||
pool = json.load(f)
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
by_idx = {e["dev_idx"]: e for e in dev} if "dev_idx" in dev[0] else None
|
||||
# pool rows carry idx = position among dev_pool rows in dev order
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(dev)}
|
||||
for r in pool:
|
||||
r["emb"] = emb_by_idx[r["idx"]]
|
||||
r["y"] = r["route"]
|
||||
return pool, meta
|
||||
|
||||
|
||||
def oof_proba_grouped(X, y, folds, C=1.0):
|
||||
"""Grouped OOF probability matrix (n×5, class order CLASSES)."""
|
||||
y_idx = np.array([CLASSES.index(c) for c in y])
|
||||
folds = np.asarray(folds)
|
||||
proba = np.zeros((len(y_idx), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds.tolist())):
|
||||
tr = folds != te_fold
|
||||
te = folds == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], y_idx[tr])
|
||||
proba[te] = clf.predict_proba(X[te])
|
||||
return proba
|
||||
|
||||
|
||||
def cls_metrics(yt, yp):
|
||||
import sklearn.metrics as m
|
||||
yt = np.asarray(yt)
|
||||
yp = np.asarray(yp)
|
||||
if yt.dtype != np.int64 and yt.dtype != np.int32:
|
||||
yt = np.array([CLASSES.index(c) for c in yt])
|
||||
if yp.dtype != np.int64 and yp.dtype != np.int32:
|
||||
yp = np.array([CLASSES.index(c) for c in yp])
|
||||
labels = list(range(len(CLASSES)))
|
||||
n = len(yt)
|
||||
acc = m.accuracy_score(yt, yp)
|
||||
macro = m.f1_score(yt, yp, average="macro", labels=labels, zero_division=0)
|
||||
pr, rc, f1, sup = m.precision_recall_fscore_support(
|
||||
yt, yp, labels=labels, zero_division=0)
|
||||
per = {c: {"p": float(pr[i]), "r": float(rc[i]), "f1": float(f1[i]), "n": int(sup[i])}
|
||||
for i, c in enumerate(CLASSES)}
|
||||
conf = m.confusion_matrix(yt, yp, labels=labels).tolist()
|
||||
return {"n": n, "acc": acc, "macro_f1": macro, "per_class": per, "confusion": conf}
|
||||
|
||||
|
||||
def fold_report(yt, proba, folds, true_y):
|
||||
out = {}
|
||||
folds_arr = np.asarray(folds)
|
||||
comp = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yt_f = [CLASSES.index(y) for y in true_y[mask]]
|
||||
comp[f] = {c: int((np.array(true_y[mask]) == c).sum()) for c in CLASSES}
|
||||
per_fold = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yp = proba[mask].argmax(1).tolist()
|
||||
m = cls_metrics([yt[i] for i in np.where(mask)[0].tolist()], yp)
|
||||
per_fold[f] = {"acc": m["acc"], "macro_f1": m["macro_f1"]}
|
||||
out["composition"] = comp
|
||||
out["per_fold"] = per_fold
|
||||
accs = [v["acc"] for v in per_fold.values()]
|
||||
macros = [v["macro_f1"] for v in per_fold.values()]
|
||||
out["acc_mean"] = float(np.mean(accs))
|
||||
out["acc_std"] = float(np.std(accs))
|
||||
out["macro_f1_mean"] = float(np.mean(macros))
|
||||
out["macro_f1_std"] = float(np.std(macros))
|
||||
return out
|
||||
|
||||
|
||||
def ece(yt, proba, n_bins=15):
|
||||
conf = proba.max(1)
|
||||
pred = proba.argmax(1)
|
||||
acc = (pred == yt).astype(float)
|
||||
bins = np.linspace(0, 1, n_bins + 1)
|
||||
tot = 0.0
|
||||
details = []
|
||||
counts = 0
|
||||
for i in range(n_bins):
|
||||
lo, hi = bins[i], bins[i + 1]
|
||||
m = (conf >= lo) & (conf < hi) if i < n_bins - 1 else conf >= lo
|
||||
if m.sum() == 0:
|
||||
continue
|
||||
acc_m = acc[m].mean()
|
||||
conf_m = conf[m].mean()
|
||||
w = m.sum() / len(conf)
|
||||
tot += w * abs(acc_m - conf_m)
|
||||
counts += int(m.sum())
|
||||
details.append({"bin": i, "lo": lo, "hi": hi, "conf": float(conf_m),
|
||||
"acc": float(acc_m), "n": int(m.sum())})
|
||||
return {"ece": float(tot), "n_bins": n_bins, "counted": counts, "bins": details}
|
||||
|
||||
|
||||
def main():
|
||||
pool, meta = load_pool_and_embeds()
|
||||
pool.sort(key=lambda r: r["idx"])
|
||||
print(f"pool: {len(pool)} rows")
|
||||
|
||||
from sklearn.metrics import brier_score_loss, log_loss
|
||||
|
||||
report = {"population": {}, "legacy": {}, "e5_linear": {}, "floors": {},
|
||||
"family_holdouts": {}, "kmw": {}, "uncertain": {}, "confidence": {},
|
||||
"ood": {}, "artifact": {}}
|
||||
|
||||
# ── §1 population ──────────────────────────────────────────────────────
|
||||
cnt = {}
|
||||
for r in pool:
|
||||
cnt[r["y"]] = cnt.get(r["y"], 0) + 1
|
||||
report["population"] = {
|
||||
"n": len(pool),
|
||||
"routes": cnt,
|
||||
"family_ids": len(set(r["family_id"] for r in pool)),
|
||||
"split_groups": len(set(r["split_group"] for r in pool)),
|
||||
"folds": {str(f): int(sum(1 for r in pool if r["cv_fold"] == f)) for f in sorted(set(r["cv_fold"] for r in pool))},
|
||||
"corpus": {k: v for k, v in meta.items() if k in
|
||||
("dev_count", "residual_count", "fast_path_count",
|
||||
"dimension", "embedder_id", "input_template", "pooling", "normalization")},
|
||||
}
|
||||
print("\n§1 population:", report["population"])
|
||||
|
||||
X = np.vstack([r["emb"] for r in pool])
|
||||
y = np.array([r["y"] for r in pool])
|
||||
folds = np.array([r["cv_fold"] for r in pool])
|
||||
yt = np.array([CLASSES.index(c) for c in y])
|
||||
|
||||
# ── §2 legacy baselines ────────────────────────────────────────────────
|
||||
import collections
|
||||
for tag, fname in [("hash", "legacy.json"), ("heads", "legacy_heads.json")]:
|
||||
path = os.path.join(OUT_DIR, fname)
|
||||
if not os.path.exists(path):
|
||||
continue
|
||||
leg = json.load(open(path))
|
||||
leg_by_idx = {r["idx"]: r for r in leg}
|
||||
yp_leg = []
|
||||
illegal = []
|
||||
for r in pool:
|
||||
lr = leg_by_idx[r["idx"]]
|
||||
if lr["illegal_action_prediction"]:
|
||||
illegal.append(lr)
|
||||
yp_leg.append("action")
|
||||
else:
|
||||
yp_leg.append(lr["class"])
|
||||
yp_leg = np.array(yp_leg)
|
||||
# five-way: an 'action' prediction is an error (outside the label set)
|
||||
yp5 = np.array([("uncertain" if p == "action" else p) for p in yp_leg])
|
||||
m = cls_metrics(y, yp5)
|
||||
m["illegal_action_prediction"] = len(illegal)
|
||||
m["illegal_cases"] = [{"idx": i["idx"], "text": i["text"], "route": i["route"],
|
||||
"intent": i["intent"], "producer": i["producer"],
|
||||
"confidence": i["confidence"]} for i in illegal]
|
||||
# per-cell confusion also shows 'action' column
|
||||
conf_counts = collections.Counter(zip(y, yp_leg))
|
||||
m["confusion_with_action"] = {f"{a}->{b}": int(c) for (a, b), c in conf_counts.items()}
|
||||
report["legacy"][tag] = m
|
||||
print(f"\n§2 legacy ({tag}) acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f} "
|
||||
f"illegal={len(illegal)}")
|
||||
for c in CLASSES:
|
||||
p = m["per_class"][c]
|
||||
print(f" {c:<14} P={p['p']:.3f} R={p['r']:.3f} F1={p['f1']:.3f} n={p['n']}")
|
||||
|
||||
# grammar-pure residual: rows not resolved by any current stage-0 grammar
|
||||
if os.path.exists(os.path.join(OUT_DIR, "legacy.json")):
|
||||
leg = json.load(open(os.path.join(OUT_DIR, "legacy.json")))
|
||||
gh = {r["idx"] for r in leg if r["producer"] == "grammar"}
|
||||
gp_mask = np.array([r["idx"] not in gh for r in pool])
|
||||
report["grammar_drift"] = {
|
||||
"grammar_hits_in_pool": len(gh),
|
||||
"grammar_pure_n": int(gp_mask.sum()),
|
||||
}
|
||||
|
||||
# ── §3 e5-linear primary head ─────────────────────────────────────────
|
||||
print("\n§3 e5-linear")
|
||||
bestC, bestMac = 1.0, -1.0
|
||||
grid = {}
|
||||
for C in C_GRID:
|
||||
p = oof_proba_grouped(X, y, folds, C=C)
|
||||
mp = cls_metrics(y, p.argmax(1).tolist())
|
||||
grid[float(C)] = {"acc": mp["acc"], "macro_f1": mp["macro_f1"]}
|
||||
print(f" C={C} acc={mp['acc']:.4f} macroF1={mp['macro_f1']:.4f}")
|
||||
if mp["macro_f1"] > bestMac:
|
||||
bestMac, bestC = mp["macro_f1"], C
|
||||
print(f" -> best C={bestC}")
|
||||
p_best = oof_proba_grouped(X, y, folds, C=bestC)
|
||||
m_best = cls_metrics(y, p_best.argmax(1).tolist())
|
||||
m_best["C"] = bestC
|
||||
m_best["C_grid"] = grid
|
||||
m_best["folds"] = fold_report(yt, p_best, folds, y)
|
||||
report["e5_linear"] = m_best
|
||||
for f, v in m_best["folds"]["per_fold"].items():
|
||||
print(f" fold {f}: acc={v['acc']:.4f} macroF1={v['macro_f1']:.4f}")
|
||||
print(f" fold acc mean={m_best['folds']['acc_mean']:.4f} "
|
||||
f"std={m_best['folds']['acc_std']:.4f}; "
|
||||
f"macroF1 mean={m_best['folds']['macro_f1_mean']:.4f} "
|
||||
f"std={m_best['folds']['macro_f1_std']:.4f}")
|
||||
for c in CLASSES:
|
||||
p_ = m_best["per_class"][c]
|
||||
print(f" {c:<14} P={p_['p']:.3f} R={p_['r']:.3f} F1={p_['f1']:.3f} n={p_['n']}")
|
||||
|
||||
# grammar-pure sensitivity for the primary head
|
||||
if "grammar_drift" in report:
|
||||
mp_gp = cls_metrics(y[gp_mask], p_best[gp_mask].argmax(1).tolist())
|
||||
report["e5_linear"]["grammar_pure"] = {
|
||||
"acc": mp_gp["acc"], "macro_f1": mp_gp["macro_f1"], "n": int(gp_mask.sum())}
|
||||
|
||||
# ── §4 floors ─────────────────────────────────────────────────────────
|
||||
print("\n§4 floors")
|
||||
# majority floor
|
||||
maj = CLASSES.index("knowledge")
|
||||
ym = np.full(len(y), maj)
|
||||
mm = cls_metrics(y, ym)
|
||||
report["floors"]["majority"] = {"acc": mm["acc"], "macro_f1": mm["macro_f1"],
|
||||
"per_class": mm["per_class"]}
|
||||
print(f" majority (predict {CLASSES[maj]}): acc={mm['acc']:.4f} macroF1={mm['macro_f1']:.4f}")
|
||||
|
||||
# centroid floor: cosine to per-class mean of the training folds' embeddings
|
||||
cf_proba = np.zeros((len(yt), len(CLASSES)))
|
||||
folds_arr = np.asarray(folds)
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
centroids = []
|
||||
for c in CLASSES:
|
||||
idxs = np.where(tr & (y == c))[0]
|
||||
ctr = X[idxs].mean(axis=0)
|
||||
ctr = ctr / np.linalg.norm(ctr)
|
||||
centroids.append(ctr)
|
||||
Cm = np.vstack(centroids)
|
||||
sims = X[te] @ Cm.T
|
||||
cf_proba[te] = sims
|
||||
yc = cf_proba.argmax(1)
|
||||
# accuracy + macroF1 with the same 5-way
|
||||
mc = cls_metrics(y, yc.tolist())
|
||||
report["floors"]["centroid"] = {"acc": mc["acc"], "macro_f1": mc["macro_f1"],
|
||||
"per_class": mc["per_class"]}
|
||||
print(f" centroid cosine: acc={mc['acc']:.4f} macroF1={mc['macro_f1']:.4f}")
|
||||
|
||||
# sparse word+char logistic (slice18 builder, grouped CV, five-way)
|
||||
texts = [r["n_text"] for r in pool]
|
||||
Xs, _vec = slice18_sparse.build_features(texts, "both")
|
||||
psp = np.zeros((len(yt), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(Xs[tr], yt[tr])
|
||||
psp[te] = clf.predict_proba(Xs[te])
|
||||
msp = cls_metrics(y, psp.argmax(1).tolist())
|
||||
report["floors"]["sparse_word_char"] = {
|
||||
"acc": msp["acc"], "macro_f1": msp["macro_f1"], "per_class": msp["per_class"],
|
||||
"vocab": slice18_sparse.vocab_size(_vec)}
|
||||
print(f" sparse both: acc={msp['acc']:.4f} macroF1={msp['macro_f1']:.4f} "
|
||||
f"vocab={report['floors']['sparse_word_char']['vocab']}")
|
||||
|
||||
# ── §5 route-family holdouts ─────────────────────────────────────────
|
||||
print("\n§5 route-family holdouts")
|
||||
fam = np.array([r["family_id"] for r in pool])
|
||||
holdouts = {}
|
||||
all_fams = sorted(set(fam.tolist()))
|
||||
for grp, fams in HOLDOUT_GROUPS.items():
|
||||
if fams is None:
|
||||
fams = [f for f in all_fams if f.startswith(grp + ":")]
|
||||
mask = np.isin(fam, fams)
|
||||
if mask.sum() == 0:
|
||||
continue
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypgrp = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypgrp.tolist())
|
||||
m["families"] = fams
|
||||
m["rows"] = int(mask.sum())
|
||||
holdouts[grp] = {"acc": m["acc"], "macro_f1": m["macro_f1"], "n": int(mask.sum()),
|
||||
"per_class": m["per_class"]}
|
||||
print(f" {grp:<14} n={m['rows']} acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f}")
|
||||
# full leave-one-family-out summary
|
||||
lofo_accs = []
|
||||
lofo_f1s = []
|
||||
for f in all_fams:
|
||||
mask = fam == f
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypf = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypf.tolist())
|
||||
lofo_accs.append(m["acc"])
|
||||
lofo_f1s.append(m["macro_f1"])
|
||||
holdouts["_all_49_lo_"] = {"n_families": len(all_fams),
|
||||
"acc_mean": float(np.mean(lofo_accs)),
|
||||
"macro_f1_mean": float(np.mean(lofo_f1s))}
|
||||
report["family_holdouts"] = holdouts
|
||||
print(f" leave-one-family-out over {len(all_fams)} families: "
|
||||
f"acc mean={np.mean(lofo_accs):.4f} macroF1 mean={np.mean(lofo_f1s):.4f}")
|
||||
|
||||
# ── §6 knowledge vs memory_write ─────────────────────────────────────
|
||||
print("\n§6 knowledge vs memory_write")
|
||||
# reuse e5-linear OOF: does the model put the higher probability on the
|
||||
# right side (memory_write for a write, knowledge for a recall)?
|
||||
conf_km = np.zeros((2, 2))
|
||||
pk = p_best[:, CLASSES.index("knowledge")]
|
||||
pmw = p_best[:, CLASSES.index("memory_write")]
|
||||
for i in range(len(yt)):
|
||||
t = y[i]
|
||||
if t == "knowledge":
|
||||
conf_km[0, 1 if pmw[i] > pk[i] else 0] += 1
|
||||
elif t == "memory_write":
|
||||
conf_km[1, 1 if pmw[i] >= pk[i] else 0] += 1
|
||||
report["kmw"] = {"confusion_p_ordered": conf_km.tolist()}
|
||||
|
||||
# matched pairs with shared subject lexemes, corpus-justified
|
||||
def build_pairs(subject, fam_k, fam_mw):
|
||||
kr = [r for r in pool if r["family_id"] in fam_k]
|
||||
mr = [r for r in pool if r["family_id"] in fam_mw]
|
||||
pairs = []
|
||||
for mw in mr:
|
||||
for k in kr:
|
||||
if subject in mw["n_text"] and subject in k["n_text"]:
|
||||
pairs.append((mw["idx"], k["idx"], mw["n_text"], k["n_text"]))
|
||||
return pairs
|
||||
|
||||
sets = {
|
||||
"water": build_pairs("вод", ["knowledge:recall-fact"], ["fact:water"]),
|
||||
"homelab": build_pairs("dns", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("сервер", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("vlan", ["knowledge:homelab-status"], ["note:homelab"]),
|
||||
"task": build_pairs("задач", ["knowledge:task-check", "knowledge:deadline"],
|
||||
["note:task"]),
|
||||
}
|
||||
idx_of = {r["idx"]: i for i, r in enumerate(pool)}
|
||||
pair_rep = {}
|
||||
for name, pairs in sets.items():
|
||||
if not pairs:
|
||||
continue
|
||||
ok = 0
|
||||
margins = []
|
||||
bad = []
|
||||
for mi, ki, mx, kx in pairs:
|
||||
mi_i, ki_i = idx_of[mi], idx_of[ki]
|
||||
# MW row should get a higher memory_write probability than the K row
|
||||
mk = (pmw[mi_i] + 0.0)
|
||||
if pmw[mi_i] > pmw[ki_i]:
|
||||
ok += 1
|
||||
else:
|
||||
bad.append((mx[:46], round(float(pmw[mi_i]), 3), kx[:46], round(float(pmw[ki_i]), 3)))
|
||||
margins.append(pmw[mi_i] - pmw[ki_i])
|
||||
pair_rep[name] = {
|
||||
"pairs": len(pairs),
|
||||
"mw_over_k_order_acc": ok / len(pairs),
|
||||
"mean_margin": float(np.mean(margins)),
|
||||
"reversed_examples": bad[:6],
|
||||
}
|
||||
print(f" {name}: pairs={len(pairs)} order_acc={ok/len(pairs):.3f} "
|
||||
f"mean_margin={np.mean(margins):+.3f}")
|
||||
report["kmw"]["matched_pairs"] = pair_rep
|
||||
|
||||
# ── §7 uncertain as explicit class ────────────────────────────────────
|
||||
print("\n§7 uncertain")
|
||||
up = m_best["per_class"]["uncertain"]
|
||||
uc = m_best["confusion"][CLASSES.index("uncertain")]
|
||||
report["uncertain"] = {
|
||||
"per_class": up,
|
||||
"row_from_uncertain": {CLASSES[j]: int(uc[j]) for j in range(5)},
|
||||
"row_to_uncertain": {CLASSES[j]: int(m_best["confusion"][j][CLASSES.index("uncertain")])
|
||||
for j in range(5)},
|
||||
}
|
||||
print(f" uncertain n={up['n']} P={up['p']:.3f} R={up['r']:.3f} F1={up['f1']:.3f}")
|
||||
print(" wrong-→label pulled from uncertain:", report["uncertain"]["row_from_uncertain"])
|
||||
print(" →uncertain pulled from:", report["uncertain"]["row_to_uncertain"])
|
||||
|
||||
# ── §8 OOF confidence / calibration / abstention ─────────────────────
|
||||
print("\n§8 confidence / calibration")
|
||||
conf = p_best.max(1)
|
||||
right = (p_best.argmax(1) == yt)
|
||||
cer = {
|
||||
"correct_conf_mean": float(conf[right].mean()),
|
||||
"correct_conf_median": float(np.median(conf[right])),
|
||||
"wrong_conf_mean": float(conf[~right].mean()),
|
||||
"wrong_conf_median": float(np.median(conf[~right])),
|
||||
"ece": ece(yt, p_best)["ece"],
|
||||
"ece_bins": ece(yt, p_best)["bins"],
|
||||
"log_loss": float(log_loss(yt, p_best, labels=[0, 1, 2, 3, 4])),
|
||||
}
|
||||
# Brier is label-set specific: one-vs-rest mean
|
||||
briers = []
|
||||
for i in range(5):
|
||||
briers.append(brier_score_loss((yt == i).astype(int), p_best[:, i]))
|
||||
cer["brier_macro"] = float(np.mean(briers))
|
||||
report["confidence"] = cer
|
||||
print(f" right conf mean={cer['correct_conf_mean']:.3f} "
|
||||
f"wrong conf mean={cer['wrong_conf_mean']:.3f} ECE={cer['ece']:.4f}")
|
||||
print(f" log_loss={cer['log_loss']:.4f} brier_macro={cer['brier_macro']:.4f}")
|
||||
|
||||
thr_grid = np.linspace(0.10, 0.98, 45)
|
||||
abst = []
|
||||
for t in thr_grid:
|
||||
cov = (conf >= t).mean()
|
||||
if cov == 0:
|
||||
continue
|
||||
keep = conf >= t
|
||||
yt_k = yt[keep]
|
||||
yp_k = p_best[keep].argmax(1)
|
||||
mk_ = cls_metrics(yt_k.tolist(), yp_k.tolist())
|
||||
abst.append({"threshold": round(float(t), 3), "coverage": float(cov),
|
||||
"accuracy": mk_["acc"], "macro_f1": mk_["macro_f1"]})
|
||||
report["confidence"]["abstention_curve"] = abst
|
||||
print(" threshold | coverage | accuracy | macroF1 (first 6/45 + knee)")
|
||||
for row in abst[::9]:
|
||||
print(f" {row['threshold']:.2f} | {row['coverage']:.3f} | "
|
||||
f"{row['accuracy']:.3f} | {row['macro_f1']:.3f}")
|
||||
|
||||
# ── §9 action OOD probes ──────────────────────────────────────────────
|
||||
print("\n§9 action OOD")
|
||||
ood_rows = [r for r in json.load(open(os.path.join(OUT_DIR, "ood.json")))]
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(slice18_sparse.filter_dev_pool(
|
||||
slice18_sparse.load_data()[1]))}
|
||||
Xo = np.vstack([emb_by_idx[r["idx"]] for r in ood_rows])
|
||||
fold_models = []
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
fold_models.append(clf)
|
||||
# OOD rows are not in folds; use the full-train model to keep it simple and
|
||||
# comparable to the non-action in-fold behaviour
|
||||
po = np.zeros((len(Xo), 5))
|
||||
for clf in fold_models:
|
||||
po += clf.predict_proba(Xo)
|
||||
po /= len(fold_models)
|
||||
ood_top = int(np.argmax(po.mean(0)))
|
||||
ood_conf = po.max(1)
|
||||
ood_pred = po.argmax(1)
|
||||
top_dist = {CLASSES[i]: int((ood_pred == i).sum()) for i in range(5)}
|
||||
confident_na = int((ood_conf > 0.9).sum())
|
||||
report["ood"] = {
|
||||
"n": len(ood_rows),
|
||||
"top_class": CLASSES[int(ood_top)],
|
||||
"top_class_dist": top_dist,
|
||||
"conf_gt_0.9": confident_na,
|
||||
"conf_gt_0.9_frac": float(confident_na / len(ood_rows)),
|
||||
"conf_mean": float(ood_conf.mean()),
|
||||
"conf_median": float(np.median(ood_conf)),
|
||||
}
|
||||
print(f" action OOD n={len(ood_rows)}: most-confident class={report['ood']['top_class']} "
|
||||
f"dist={top_dist}")
|
||||
print(f" conf>0.9: {confident_na} ({confident_na/len(ood_rows):.3f}) "
|
||||
f"conf mean={report['ood']['conf_mean']:.3f}")
|
||||
|
||||
# ── §10 artifact cost ────────────────────────────────────────────────
|
||||
print("\n§10 artifact")
|
||||
n_params = len(CLASSES) * X.shape[1] + len(CLASSES)
|
||||
fp32 = n_params * 4
|
||||
report["artifact"] = {
|
||||
"e5_dim": X.shape[1],
|
||||
"head_params": n_params,
|
||||
"head_fp32_bytes": fp32,
|
||||
"head_fp32_kib": fp32 / 1024,
|
||||
"head_int8_bytes": n_params,
|
||||
}
|
||||
# incremental latency of the linear head over a batch of 1 (µs)
|
||||
clf = slice18_sparse.LogisticRegression(C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X, yt)
|
||||
x1 = X[:1]
|
||||
for _ in range(50):
|
||||
clf.predict_proba(x1)
|
||||
lat = []
|
||||
for _ in range(2000):
|
||||
t0 = time.perf_counter_ns()
|
||||
clf.predict_proba(x1)
|
||||
lat.append((time.perf_counter_ns() - t0) / 1e3)
|
||||
lat = np.array(lat)
|
||||
report["artifact"]["head_latency_us_mean"] = float(lat.mean())
|
||||
report["artifact"]["head_latency_us_p50"] = float(np.median(lat))
|
||||
print(f" head params={n_params} fp32={fp32/1024:.2f}KiB "
|
||||
f"lat mean={lat.mean():.2f}us p50={np.median(lat):.2f}us")
|
||||
|
||||
with open(os.path.join(OUT_DIR, "results.json"), "w") as f:
|
||||
json.dump(report, f, ensure_ascii=False, indent=1, default=float)
|
||||
print(f"\nwrote {OUT_DIR}/results.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,243 @@
|
||||
// slice23 — fast-path metadata reconciliation diagnostic.
|
||||
//
|
||||
// Classifies every disagreement between the corpus's stored fast_path_resolved
|
||||
// flag and what the production fast path derives today (TryFastPath over the
|
||||
// stage-0 grammars with the experiment's act allowlist). Outputs a JSON
|
||||
// decomposition and a console summary for docs/evals reports.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/semantic-router-experiment/slice23/ -out /tmp/mvn-s23/drift.json
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/router/semantic"
|
||||
)
|
||||
|
||||
type conflict struct {
|
||||
Text string `json:"text"`
|
||||
SourceID string `json:"source_id"`
|
||||
Route string `json:"route"`
|
||||
Source string `json:"source"`
|
||||
Group string `json:"split_group"`
|
||||
Tags []string `json:"tags,omitempty"`
|
||||
Dev bool `json:"dev"`
|
||||
// Direction: claimed_fast_now_miss = stored fast, runtime residual;
|
||||
// mirror_missed = stored residual, runtime fast.
|
||||
Direction string `json:"direction"`
|
||||
Grammar string `json:"grammar,omitempty"`
|
||||
// ShapeDeclined is true when at least one stage-0 grammar matched the
|
||||
// utterance's shape but refused the content (falls through like the router).
|
||||
ShapeDeclined bool `json:"shape_declined"`
|
||||
// DeclinedGrammars names every stage-0 grammar that matched the shape but
|
||||
// declined the content, for claimed_fast_now_miss rows.
|
||||
DeclinedGrammars []string `json:"declined_grammars,omitempty"`
|
||||
}
|
||||
|
||||
type report struct {
|
||||
Meta metaSummary `json:"meta"`
|
||||
Pop popSummary `json:"population"`
|
||||
Conflicts []conflict `json:"conflicts"`
|
||||
ByRoute map[string]map[string]int `json:"by_route"`
|
||||
ByGrammar map[string]int `json:"by_grammar"`
|
||||
BySource map[string]map[string]int `json:"by_source"`
|
||||
ByFamily map[string]map[string]int `json:"by_family"`
|
||||
Direction map[string]int `json:"by_direction"`
|
||||
Declined int `json:"claimed_fast_with_declined_shape"`
|
||||
NoShape int `json:"claimed_fast_with_no_shape"`
|
||||
}
|
||||
|
||||
type metaSummary struct {
|
||||
Total int `json:"total"`
|
||||
DevCount int `json:"dev_count"`
|
||||
Frozen int `json:"frozen_count"`
|
||||
}
|
||||
|
||||
type popSummary struct {
|
||||
StoredFast int `json:"stored_fast"`
|
||||
StoredResid int `json:"stored_residual"`
|
||||
DerivedFast int `json:"derived_fast"`
|
||||
DerivedResid int `json:"derived_residual"`
|
||||
// Dev-pool residual route counts derived as the router sees them today.
|
||||
DevResidualByRoute map[string]int `json:"dev_residual_by_route"`
|
||||
// Dev-pool residual non-action + action OOD as derived.
|
||||
DevResidualNonAction int `json:"dev_residual_non_action"`
|
||||
DevResidualAction int `json:"dev_residual_action"`
|
||||
// Fast rows in the dev pool, derived.
|
||||
DevFast int `json:"dev_fast"`
|
||||
}
|
||||
|
||||
func main() {
|
||||
outPath := flag.String("out", "/tmp/mvn-s23/drift.json", "output JSON path")
|
||||
flag.Parse()
|
||||
|
||||
exs, err := semantic.LoadCorpus()
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "load corpus: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
_, dev, _ := semantic.FrozenHoldoutSplit(exs)
|
||||
devSet := make(map[string]bool, len(dev))
|
||||
for _, e := range dev {
|
||||
devSet[e.SourceID] = true
|
||||
}
|
||||
|
||||
// Stage-zero grammar list for shape/declined attribution (same list the
|
||||
// derivation walks).
|
||||
acts := router.DefaultActMatcher{Fns: semantic.ExperimentActVerbs()}
|
||||
gs := router.StageZeroGrammars(acts)
|
||||
|
||||
var (
|
||||
conflicts []conflict
|
||||
byRoute = map[string]map[string]int{}
|
||||
byGrammar = map[string]int{}
|
||||
bySource = map[string]map[string]int{}
|
||||
byFamily = map[string]map[string]int{}
|
||||
byDirection = map[string]int{}
|
||||
storedFast, derivedFast, declined, noShape int
|
||||
devResidByRoute = map[string]int{}
|
||||
devResidNonAct, devResidAct, devFast int
|
||||
)
|
||||
|
||||
for _, e := range exs {
|
||||
o := semantic.DeriveFastPath(e.Text)
|
||||
inDev := devSet[e.SourceID]
|
||||
|
||||
st := e.FastPathResolved
|
||||
if st {
|
||||
storedFast++
|
||||
}
|
||||
if o.Matched {
|
||||
derivedFast++
|
||||
}
|
||||
|
||||
var c *conflict
|
||||
switch {
|
||||
case st && o.Matched:
|
||||
case st && !o.Matched:
|
||||
// Stored fast but the runtime misses. Attribute why.
|
||||
shapeDeclined := false
|
||||
var declinedNames []string
|
||||
for _, g := range gs {
|
||||
_, matched, ok := g.Evaluate(e.Text)
|
||||
if matched && !ok {
|
||||
shapeDeclined = true
|
||||
declinedNames = append(declinedNames, g.Name)
|
||||
}
|
||||
}
|
||||
if shapeDeclined {
|
||||
declined++
|
||||
} else {
|
||||
noShape++
|
||||
}
|
||||
c = &conflict{Direction: "claimed_fast_now_miss", ShapeDeclined: shapeDeclined, DeclinedGrammars: declinedNames}
|
||||
case !st && o.Matched:
|
||||
c = &conflict{Direction: "mirror_missed", Grammar: o.Grammar}
|
||||
}
|
||||
|
||||
if c != nil {
|
||||
c.Text = e.Text
|
||||
c.SourceID = e.SourceID
|
||||
c.Route = string(e.Route)
|
||||
c.Source = e.Source
|
||||
c.Group = e.SplitGroup
|
||||
c.Tags = e.Tags
|
||||
c.Dev = inDev
|
||||
conflicts = append(conflicts, *c)
|
||||
byDirection[c.Direction]++
|
||||
byGrammar[c.Grammar]++
|
||||
if byRoute[c.Direction] == nil {
|
||||
byRoute[c.Direction] = map[string]int{}
|
||||
}
|
||||
byRoute[c.Direction][c.Route]++
|
||||
if bySource[c.Direction] == nil {
|
||||
bySource[c.Direction] = map[string]int{}
|
||||
}
|
||||
bySource[c.Direction][c.Source]++
|
||||
if byFamily[c.Direction] == nil {
|
||||
byFamily[c.Direction] = map[string]int{}
|
||||
}
|
||||
byFamily[c.Direction][c.Group]++
|
||||
}
|
||||
|
||||
if inDev {
|
||||
if o.Matched {
|
||||
devFast++
|
||||
} else {
|
||||
devResidByRoute[string(e.Route)]++
|
||||
if e.Route == semantic.RouteAction {
|
||||
devResidAct++
|
||||
} else {
|
||||
devResidNonAct++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sort.Slice(conflicts, func(i, j int) bool { return conflicts[i].SourceID < conflicts[j].SourceID })
|
||||
|
||||
rep := report{
|
||||
Meta: metaSummary{Total: len(exs), DevCount: len(dev), Frozen: len(exs) - len(dev)},
|
||||
Pop: popSummary{
|
||||
StoredFast: storedFast, StoredResid: len(exs) - storedFast,
|
||||
DerivedFast: derivedFast, DerivedResid: len(exs) - derivedFast,
|
||||
DevResidualByRoute: devResidByRoute,
|
||||
DevResidualNonAction: devResidNonAct, DevResidualAction: devResidAct,
|
||||
DevFast: devFast,
|
||||
},
|
||||
Conflicts: conflicts,
|
||||
ByRoute: byRoute, ByGrammar: byGrammar, BySource: bySource, ByFamily: byFamily,
|
||||
Direction: byDirection, Declined: declined, NoShape: noShape,
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(rep, "", " ")
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "marshal: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if err := os.WriteFile(*outPath, data, 0644); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "write %s: %v\n", *outPath, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
fmt.Printf("total %d (dev %d, frozen %d)\n", rep.Meta.Total, rep.Meta.DevCount, rep.Meta.Frozen)
|
||||
fmt.Printf("stored fast=%d residual=%d\n", rep.Pop.StoredFast, rep.Pop.StoredResid)
|
||||
fmt.Printf("derived fast=%d residual=%d\n", rep.Pop.DerivedFast, rep.Pop.DerivedResid)
|
||||
fmt.Printf("disagreements total %d\n", len(conflicts))
|
||||
for _, d := range []string{"claimed_fast_now_miss", "mirror_missed"} {
|
||||
fmt.Printf(" %-22s %d\n", d, byDirection[d])
|
||||
if d == "claimed_fast_now_miss" {
|
||||
fmt.Printf(" with declined shape: %d no shape: %d\n", declined, noShape)
|
||||
}
|
||||
}
|
||||
fmt.Println(" mirror-missed by grammar:")
|
||||
for _, k := range sortedKeys(byGrammar) {
|
||||
fmt.Printf(" %-28s %d\n", k, byGrammar[k])
|
||||
}
|
||||
fmt.Println(" by route:")
|
||||
for _, d := range sortedKeys(byRoute) {
|
||||
fmt.Printf(" %-22s %v\n", d, byRoute[d])
|
||||
}
|
||||
fmt.Printf("dev pool derived: fast=%d residual=%d (non-action=%d action=%d)\n",
|
||||
rep.Pop.DevFast, rep.Pop.DevResidualNonAction+rep.Pop.DevResidualAction,
|
||||
rep.Pop.DevResidualNonAction, rep.Pop.DevResidualAction)
|
||||
fmt.Printf("dev residual by route: %v\n", rep.Pop.DevResidualByRoute)
|
||||
fmt.Printf("wrote %s\n", *outPath)
|
||||
}
|
||||
|
||||
func sortedKeys[T any](m map[string]T) []string {
|
||||
ks := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
ks = append(ks, k)
|
||||
}
|
||||
sort.Strings(ks)
|
||||
return ks
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 23 emit: five-way residual non-action semantic router — data files
|
||||
========================================================================
|
||||
|
||||
Slice 23 reconciles corpus fast-path metadata with the production router
|
||||
(TryFastPath over stage-0 grammars). The corpus builder no longer mirrors the
|
||||
grammars by hand; fast_path_resolved is derived from the router, so this emit
|
||||
flags exactly the rows the router genuinely leaves for the general cascade.
|
||||
|
||||
This script only repackages the frozen dev pool for the Go legacy baseline and
|
||||
the Python experiment, writing into /tmp/mvn-s23 so the slice-22 artifacts
|
||||
stay untouched. Logic is slice22_emit.py verbatim; only OUT_DIR differs.
|
||||
|
||||
/tmp/mvn-s23/pool.json residual non-action dev rows: idx, text, n_text,
|
||||
route, tags, cv_fold, split_group, family_id,
|
||||
source_id (1509 rows)
|
||||
/tmp/mvn-s23/ood.json residual ACTION dev rows (720): same shape; OOD
|
||||
probes only, never primary metrics
|
||||
/tmp/mvn-s23/stats.json population summary (routes, families, folds)
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402 (normalize_match_text, load_data, filters)
|
||||
|
||||
OUT_DIR = "/tmp/mvn-s23"
|
||||
|
||||
ROUTES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
|
||||
|
||||
def main():
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
print(f"dev pool: {len(dev)} rows "
|
||||
f"(meta declares dev_count={meta.get('dev_count')})")
|
||||
|
||||
rows = []
|
||||
for i, e in enumerate(dev):
|
||||
if not e["fast_path_resolved"]:
|
||||
rows.append({
|
||||
"idx": i,
|
||||
"text": e["text"],
|
||||
"n_text": slice18_sparse.normalize_match_text(e["text"]),
|
||||
"route": e["route"],
|
||||
"tags": sorted(set(e.get("tags", []))),
|
||||
"cv_fold": e["cv_fold"],
|
||||
"split_group": e["split_group"],
|
||||
"family_id": e["family_id"],
|
||||
"source_id": e["source_id"],
|
||||
})
|
||||
|
||||
na = [r for r in rows if r["route"] != "action"]
|
||||
ood = [r for r in rows if r["route"] == "action"]
|
||||
print(f"residual rows: {len(rows)} non-action: {len(na)} action(OOD): {len(ood)}")
|
||||
|
||||
by_route = {}
|
||||
for r in na:
|
||||
by_route[r["route"]] = by_route.get(r["route"], 0) + 1
|
||||
print("routes:", by_route)
|
||||
assert sum(by_route.values()) == len(na)
|
||||
assert set(ROUTES) == set(by_route), "route set must be the five-way"
|
||||
|
||||
by_family = {}
|
||||
for r in na:
|
||||
by_family[r["family_id"]] = by_family.get(r["family_id"], 0) + 1
|
||||
by_fold = {}
|
||||
for r in na:
|
||||
by_fold[r["cv_fold"]] = by_fold.get(r["cv_fold"], 0) + 1
|
||||
print(f"family_ids: {len(by_family)} split_groups: {len(set(r['split_group'] for r in na))}")
|
||||
print("folds:", by_fold)
|
||||
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
with open(os.path.join(OUT_DIR, "pool.json"), "w") as f:
|
||||
json.dump(na, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "ood.json"), "w") as f:
|
||||
json.dump(ood, f, ensure_ascii=False, indent=1)
|
||||
with open(os.path.join(OUT_DIR, "stats.json"), "w") as f:
|
||||
json.dump({
|
||||
"dev_count": len(dev),
|
||||
"residual_count": len(rows),
|
||||
"non_action_count": len(na),
|
||||
"action_ood_count": len(ood),
|
||||
"routes": by_route,
|
||||
"family_ids": len(by_family),
|
||||
"folds": by_fold,
|
||||
"top_family": dict(sorted(by_family.items(), key=lambda kv: -kv[1])[:15]),
|
||||
}, f, ensure_ascii=False, indent=1)
|
||||
print(f"wrote {OUT_DIR}/{{pool,ood,stats}}.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,559 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Slice 23: five-way residual non-action semantic router — corrected population
|
||||
============================================================================
|
||||
|
||||
Slice 22 reported the corpus's fast-path mirror was stale next to the
|
||||
production stage-0 grammars (185 residual rows resolved at runtime). Slice 23
|
||||
derives fast_path_resolved from the real router, rebuilds the corpus, and
|
||||
re-measures the primary slice-22 results on the corrected residual pool.
|
||||
Logic and configs are slice22_main.py verbatim; only OUT_DIR differs.
|
||||
|
||||
Population: the corrected dev-pool residual non-action rows (1509; the pool
|
||||
written by slice23_emit.py). Corrected action rows (720) are OOD probes only.
|
||||
|
||||
Metrics written to /tmp/mvn-s23/results.json:
|
||||
§1 population
|
||||
§2 legacy baseline (legacy.json / legacy_heads.json): acc, macro-F1,
|
||||
per-class P/R/F1, confusion, illegal_action_prediction count
|
||||
§3 e5-linear primary head: C grid, grouped CV OOF, per-fold P/R/F1 +
|
||||
variance + composition
|
||||
§4 floors: majority, centroid (cosine nearest-mean), sparse word+char
|
||||
TF-IDF logistic (slice18 builder), all grouped CV
|
||||
§5 route-family (family_id) leave-family-out
|
||||
§6 knowledge vs memory_write: matched pairs (water/homelab/task) ordering
|
||||
§7 uncertain as an explicit class: P/R/F1 + top confusions
|
||||
§8 OOF confidence: max-softmax correct/wrong, ECE, log-loss, Brier,
|
||||
coverage/accuracy/macro-F1 abstention curves (no threshold chosen)
|
||||
§9 action OOD probes: fold models applied to the corrected action rows
|
||||
§10 artifact cost: head params, serialized bytes, incremental head latency
|
||||
|
||||
No corpus label is changed. No frozen-holdout rows are inspected.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
import slice18_sparse # noqa: E402
|
||||
import slice19_main # noqa: E402
|
||||
|
||||
EMB_PATH = "/tmp/mvn-experiment/embeddings.json"
|
||||
OUT_DIR = "/tmp/mvn-s23"
|
||||
CLASSES = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
CLASS_PREFIX = ["conversation", "knowledge", "memory_write", "system", "uncertain"]
|
||||
C_GRID = [0.1, 1.0, 10.0]
|
||||
|
||||
# Route-family holdouts the report calls out by name (slice-22 brief): every
|
||||
# family that is not part of the shared subject inventory on either side.
|
||||
HOLDOUT_GROUPS = {
|
||||
"capability": ["knowledge:capability-ha", "knowledge:capability-tool"],
|
||||
"world": ["knowledge:world-def", "knowledge:world-explain"],
|
||||
"calendar": ["knowledge:calendar", "knowledge:calendar-time", "knowledge:calendar-next"],
|
||||
"recall": ["knowledge:recall-fact", "knowledge:recall-note", "knowledge:recall-possessive"],
|
||||
"fact": ["fact:meal", "fact:water", "fact:sleep", "fact:shower", "fact:break", "fact:pills", "fact:exercise"],
|
||||
"note": ["note:idea", "note:homelab", "note:task"],
|
||||
"remember": ["free:remember"],
|
||||
"system": None, # all system:*
|
||||
"conversation": None,
|
||||
"uncertain": None,
|
||||
}
|
||||
|
||||
|
||||
def load_pool_and_embeds():
|
||||
with open(os.path.join(OUT_DIR, "pool.json")) as f:
|
||||
pool = json.load(f)
|
||||
meta, examples = slice18_sparse.load_data()
|
||||
dev = slice18_sparse.filter_dev_pool(examples)
|
||||
by_idx = {e["dev_idx"]: e for e in dev} if "dev_idx" in dev[0] else None
|
||||
# pool rows carry idx = position among dev_pool rows in dev order
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(dev)}
|
||||
for r in pool:
|
||||
r["emb"] = emb_by_idx[r["idx"]]
|
||||
r["y"] = r["route"]
|
||||
return pool, meta
|
||||
|
||||
|
||||
def oof_proba_grouped(X, y, folds, C=1.0):
|
||||
"""Grouped OOF probability matrix (n×5, class order CLASSES)."""
|
||||
y_idx = np.array([CLASSES.index(c) for c in y])
|
||||
folds = np.asarray(folds)
|
||||
proba = np.zeros((len(y_idx), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds.tolist())):
|
||||
tr = folds != te_fold
|
||||
te = folds == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=C, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], y_idx[tr])
|
||||
proba[te] = clf.predict_proba(X[te])
|
||||
return proba
|
||||
|
||||
|
||||
def cls_metrics(yt, yp):
|
||||
import sklearn.metrics as m
|
||||
yt = np.asarray(yt)
|
||||
yp = np.asarray(yp)
|
||||
if yt.dtype != np.int64 and yt.dtype != np.int32:
|
||||
yt = np.array([CLASSES.index(c) for c in yt])
|
||||
if yp.dtype != np.int64 and yp.dtype != np.int32:
|
||||
yp = np.array([CLASSES.index(c) for c in yp])
|
||||
labels = list(range(len(CLASSES)))
|
||||
n = len(yt)
|
||||
acc = m.accuracy_score(yt, yp)
|
||||
macro = m.f1_score(yt, yp, average="macro", labels=labels, zero_division=0)
|
||||
pr, rc, f1, sup = m.precision_recall_fscore_support(
|
||||
yt, yp, labels=labels, zero_division=0)
|
||||
per = {c: {"p": float(pr[i]), "r": float(rc[i]), "f1": float(f1[i]), "n": int(sup[i])}
|
||||
for i, c in enumerate(CLASSES)}
|
||||
conf = m.confusion_matrix(yt, yp, labels=labels).tolist()
|
||||
return {"n": n, "acc": acc, "macro_f1": macro, "per_class": per, "confusion": conf}
|
||||
|
||||
|
||||
def fold_report(yt, proba, folds, true_y):
|
||||
out = {}
|
||||
folds_arr = np.asarray(folds)
|
||||
comp = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yt_f = [CLASSES.index(y) for y in true_y[mask]]
|
||||
comp[f] = {c: int((np.array(true_y[mask]) == c).sum()) for c in CLASSES}
|
||||
per_fold = {}
|
||||
for f in sorted(set(folds_arr.tolist())):
|
||||
mask = folds_arr == f
|
||||
yp = proba[mask].argmax(1).tolist()
|
||||
m = cls_metrics([yt[i] for i in np.where(mask)[0].tolist()], yp)
|
||||
per_fold[f] = {"acc": m["acc"], "macro_f1": m["macro_f1"]}
|
||||
out["composition"] = comp
|
||||
out["per_fold"] = per_fold
|
||||
accs = [v["acc"] for v in per_fold.values()]
|
||||
macros = [v["macro_f1"] for v in per_fold.values()]
|
||||
out["acc_mean"] = float(np.mean(accs))
|
||||
out["acc_std"] = float(np.std(accs))
|
||||
out["macro_f1_mean"] = float(np.mean(macros))
|
||||
out["macro_f1_std"] = float(np.std(macros))
|
||||
return out
|
||||
|
||||
|
||||
def ece(yt, proba, n_bins=15):
|
||||
conf = proba.max(1)
|
||||
pred = proba.argmax(1)
|
||||
acc = (pred == yt).astype(float)
|
||||
bins = np.linspace(0, 1, n_bins + 1)
|
||||
tot = 0.0
|
||||
details = []
|
||||
counts = 0
|
||||
for i in range(n_bins):
|
||||
lo, hi = bins[i], bins[i + 1]
|
||||
m = (conf >= lo) & (conf < hi) if i < n_bins - 1 else conf >= lo
|
||||
if m.sum() == 0:
|
||||
continue
|
||||
acc_m = acc[m].mean()
|
||||
conf_m = conf[m].mean()
|
||||
w = m.sum() / len(conf)
|
||||
tot += w * abs(acc_m - conf_m)
|
||||
counts += int(m.sum())
|
||||
details.append({"bin": i, "lo": lo, "hi": hi, "conf": float(conf_m),
|
||||
"acc": float(acc_m), "n": int(m.sum())})
|
||||
return {"ece": float(tot), "n_bins": n_bins, "counted": counts, "bins": details}
|
||||
|
||||
|
||||
def main():
|
||||
pool, meta = load_pool_and_embeds()
|
||||
pool.sort(key=lambda r: r["idx"])
|
||||
print(f"pool: {len(pool)} rows")
|
||||
|
||||
from sklearn.metrics import brier_score_loss, log_loss
|
||||
|
||||
report = {"population": {}, "legacy": {}, "e5_linear": {}, "floors": {},
|
||||
"family_holdouts": {}, "kmw": {}, "uncertain": {}, "confidence": {},
|
||||
"ood": {}, "artifact": {}}
|
||||
|
||||
# ── §1 population ──────────────────────────────────────────────────────
|
||||
cnt = {}
|
||||
for r in pool:
|
||||
cnt[r["y"]] = cnt.get(r["y"], 0) + 1
|
||||
report["population"] = {
|
||||
"n": len(pool),
|
||||
"routes": cnt,
|
||||
"family_ids": len(set(r["family_id"] for r in pool)),
|
||||
"split_groups": len(set(r["split_group"] for r in pool)),
|
||||
"folds": {str(f): int(sum(1 for r in pool if r["cv_fold"] == f)) for f in sorted(set(r["cv_fold"] for r in pool))},
|
||||
"corpus": {k: v for k, v in meta.items() if k in
|
||||
("dev_count", "residual_count", "fast_path_count",
|
||||
"dimension", "embedder_id", "input_template", "pooling", "normalization")},
|
||||
}
|
||||
print("\n§1 population:", report["population"])
|
||||
|
||||
X = np.vstack([r["emb"] for r in pool])
|
||||
y = np.array([r["y"] for r in pool])
|
||||
folds = np.array([r["cv_fold"] for r in pool])
|
||||
yt = np.array([CLASSES.index(c) for c in y])
|
||||
|
||||
# ── §2 legacy baselines ────────────────────────────────────────────────
|
||||
import collections
|
||||
for tag, fname in [("hash", "legacy.json"), ("heads", "legacy_heads.json")]:
|
||||
path = os.path.join(OUT_DIR, fname)
|
||||
if not os.path.exists(path):
|
||||
continue
|
||||
leg = json.load(open(path))
|
||||
leg_by_idx = {r["idx"]: r for r in leg}
|
||||
yp_leg = []
|
||||
illegal = []
|
||||
for r in pool:
|
||||
lr = leg_by_idx[r["idx"]]
|
||||
if lr["illegal_action_prediction"]:
|
||||
illegal.append(lr)
|
||||
yp_leg.append("action")
|
||||
else:
|
||||
yp_leg.append(lr["class"])
|
||||
yp_leg = np.array(yp_leg)
|
||||
# five-way: an 'action' prediction is an error (outside the label set)
|
||||
yp5 = np.array([("uncertain" if p == "action" else p) for p in yp_leg])
|
||||
m = cls_metrics(y, yp5)
|
||||
m["illegal_action_prediction"] = len(illegal)
|
||||
m["illegal_cases"] = [{"idx": i["idx"], "text": i["text"], "route": i["route"],
|
||||
"intent": i["intent"], "producer": i["producer"],
|
||||
"confidence": i["confidence"]} for i in illegal]
|
||||
# per-cell confusion also shows 'action' column
|
||||
conf_counts = collections.Counter(zip(y, yp_leg))
|
||||
m["confusion_with_action"] = {f"{a}->{b}": int(c) for (a, b), c in conf_counts.items()}
|
||||
report["legacy"][tag] = m
|
||||
print(f"\n§2 legacy ({tag}) acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f} "
|
||||
f"illegal={len(illegal)}")
|
||||
for c in CLASSES:
|
||||
p = m["per_class"][c]
|
||||
print(f" {c:<14} P={p['p']:.3f} R={p['r']:.3f} F1={p['f1']:.3f} n={p['n']}")
|
||||
|
||||
# grammar-pure residual: rows not resolved by any current stage-0 grammar
|
||||
if os.path.exists(os.path.join(OUT_DIR, "legacy.json")):
|
||||
leg = json.load(open(os.path.join(OUT_DIR, "legacy.json")))
|
||||
gh = {r["idx"] for r in leg if r["producer"] == "grammar"}
|
||||
gp_mask = np.array([r["idx"] not in gh for r in pool])
|
||||
report["grammar_drift"] = {
|
||||
"grammar_hits_in_pool": len(gh),
|
||||
"grammar_pure_n": int(gp_mask.sum()),
|
||||
}
|
||||
|
||||
# ── §3 e5-linear primary head ─────────────────────────────────────────
|
||||
print("\n§3 e5-linear")
|
||||
bestC, bestMac = 1.0, -1.0
|
||||
grid = {}
|
||||
for C in C_GRID:
|
||||
p = oof_proba_grouped(X, y, folds, C=C)
|
||||
mp = cls_metrics(y, p.argmax(1).tolist())
|
||||
grid[float(C)] = {"acc": mp["acc"], "macro_f1": mp["macro_f1"]}
|
||||
print(f" C={C} acc={mp['acc']:.4f} macroF1={mp['macro_f1']:.4f}")
|
||||
if mp["macro_f1"] > bestMac:
|
||||
bestMac, bestC = mp["macro_f1"], C
|
||||
print(f" -> best C={bestC}")
|
||||
p_best = oof_proba_grouped(X, y, folds, C=bestC)
|
||||
m_best = cls_metrics(y, p_best.argmax(1).tolist())
|
||||
m_best["C"] = bestC
|
||||
m_best["C_grid"] = grid
|
||||
m_best["folds"] = fold_report(yt, p_best, folds, y)
|
||||
report["e5_linear"] = m_best
|
||||
for f, v in m_best["folds"]["per_fold"].items():
|
||||
print(f" fold {f}: acc={v['acc']:.4f} macroF1={v['macro_f1']:.4f}")
|
||||
print(f" fold acc mean={m_best['folds']['acc_mean']:.4f} "
|
||||
f"std={m_best['folds']['acc_std']:.4f}; "
|
||||
f"macroF1 mean={m_best['folds']['macro_f1_mean']:.4f} "
|
||||
f"std={m_best['folds']['macro_f1_std']:.4f}")
|
||||
for c in CLASSES:
|
||||
p_ = m_best["per_class"][c]
|
||||
print(f" {c:<14} P={p_['p']:.3f} R={p_['r']:.3f} F1={p_['f1']:.3f} n={p_['n']}")
|
||||
|
||||
# grammar-pure sensitivity for the primary head
|
||||
if "grammar_drift" in report:
|
||||
mp_gp = cls_metrics(y[gp_mask], p_best[gp_mask].argmax(1).tolist())
|
||||
report["e5_linear"]["grammar_pure"] = {
|
||||
"acc": mp_gp["acc"], "macro_f1": mp_gp["macro_f1"], "n": int(gp_mask.sum())}
|
||||
|
||||
# ── §4 floors ─────────────────────────────────────────────────────────
|
||||
print("\n§4 floors")
|
||||
# majority floor
|
||||
maj = CLASSES.index("knowledge")
|
||||
ym = np.full(len(y), maj)
|
||||
mm = cls_metrics(y, ym)
|
||||
report["floors"]["majority"] = {"acc": mm["acc"], "macro_f1": mm["macro_f1"],
|
||||
"per_class": mm["per_class"]}
|
||||
print(f" majority (predict {CLASSES[maj]}): acc={mm['acc']:.4f} macroF1={mm['macro_f1']:.4f}")
|
||||
|
||||
# centroid floor: cosine to per-class mean of the training folds' embeddings
|
||||
cf_proba = np.zeros((len(yt), len(CLASSES)))
|
||||
folds_arr = np.asarray(folds)
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
centroids = []
|
||||
for c in CLASSES:
|
||||
idxs = np.where(tr & (y == c))[0]
|
||||
ctr = X[idxs].mean(axis=0)
|
||||
ctr = ctr / np.linalg.norm(ctr)
|
||||
centroids.append(ctr)
|
||||
Cm = np.vstack(centroids)
|
||||
sims = X[te] @ Cm.T
|
||||
cf_proba[te] = sims
|
||||
yc = cf_proba.argmax(1)
|
||||
# accuracy + macroF1 with the same 5-way
|
||||
mc = cls_metrics(y, yc.tolist())
|
||||
report["floors"]["centroid"] = {"acc": mc["acc"], "macro_f1": mc["macro_f1"],
|
||||
"per_class": mc["per_class"]}
|
||||
print(f" centroid cosine: acc={mc['acc']:.4f} macroF1={mc['macro_f1']:.4f}")
|
||||
|
||||
# sparse word+char logistic (slice18 builder, grouped CV, five-way)
|
||||
texts = [r["n_text"] for r in pool]
|
||||
Xs, _vec = slice18_sparse.build_features(texts, "both")
|
||||
psp = np.zeros((len(yt), len(CLASSES)))
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
te = folds_arr == te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=1.0, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(Xs[tr], yt[tr])
|
||||
psp[te] = clf.predict_proba(Xs[te])
|
||||
msp = cls_metrics(y, psp.argmax(1).tolist())
|
||||
report["floors"]["sparse_word_char"] = {
|
||||
"acc": msp["acc"], "macro_f1": msp["macro_f1"], "per_class": msp["per_class"],
|
||||
"vocab": slice18_sparse.vocab_size(_vec)}
|
||||
print(f" sparse both: acc={msp['acc']:.4f} macroF1={msp['macro_f1']:.4f} "
|
||||
f"vocab={report['floors']['sparse_word_char']['vocab']}")
|
||||
|
||||
# ── §5 route-family holdouts ─────────────────────────────────────────
|
||||
print("\n§5 route-family holdouts")
|
||||
fam = np.array([r["family_id"] for r in pool])
|
||||
holdouts = {}
|
||||
all_fams = sorted(set(fam.tolist()))
|
||||
for grp, fams in HOLDOUT_GROUPS.items():
|
||||
if fams is None:
|
||||
fams = [f for f in all_fams if f.startswith(grp + ":")]
|
||||
mask = np.isin(fam, fams)
|
||||
if mask.sum() == 0:
|
||||
continue
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypgrp = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypgrp.tolist())
|
||||
m["families"] = fams
|
||||
m["rows"] = int(mask.sum())
|
||||
holdouts[grp] = {"acc": m["acc"], "macro_f1": m["macro_f1"], "n": int(mask.sum()),
|
||||
"per_class": m["per_class"]}
|
||||
print(f" {grp:<14} n={m['rows']} acc={m['acc']:.4f} macroF1={m['macro_f1']:.4f}")
|
||||
# full leave-one-family-out summary
|
||||
lofo_accs = []
|
||||
lofo_f1s = []
|
||||
for f in all_fams:
|
||||
mask = fam == f
|
||||
tr = ~mask
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
ypf = clf.predict(X[mask])
|
||||
m = cls_metrics([CLASSES.index(c) for c in y[mask]], ypf.tolist())
|
||||
lofo_accs.append(m["acc"])
|
||||
lofo_f1s.append(m["macro_f1"])
|
||||
holdouts["_all_49_lo_"] = {"n_families": len(all_fams),
|
||||
"acc_mean": float(np.mean(lofo_accs)),
|
||||
"macro_f1_mean": float(np.mean(lofo_f1s))}
|
||||
report["family_holdouts"] = holdouts
|
||||
print(f" leave-one-family-out over {len(all_fams)} families: "
|
||||
f"acc mean={np.mean(lofo_accs):.4f} macroF1 mean={np.mean(lofo_f1s):.4f}")
|
||||
|
||||
# ── §6 knowledge vs memory_write ─────────────────────────────────────
|
||||
print("\n§6 knowledge vs memory_write")
|
||||
# reuse e5-linear OOF: does the model put the higher probability on the
|
||||
# right side (memory_write for a write, knowledge for a recall)?
|
||||
conf_km = np.zeros((2, 2))
|
||||
pk = p_best[:, CLASSES.index("knowledge")]
|
||||
pmw = p_best[:, CLASSES.index("memory_write")]
|
||||
for i in range(len(yt)):
|
||||
t = y[i]
|
||||
if t == "knowledge":
|
||||
conf_km[0, 1 if pmw[i] > pk[i] else 0] += 1
|
||||
elif t == "memory_write":
|
||||
conf_km[1, 1 if pmw[i] >= pk[i] else 0] += 1
|
||||
report["kmw"] = {"confusion_p_ordered": conf_km.tolist()}
|
||||
|
||||
# matched pairs with shared subject lexemes, corpus-justified
|
||||
def build_pairs(subject, fam_k, fam_mw):
|
||||
kr = [r for r in pool if r["family_id"] in fam_k]
|
||||
mr = [r for r in pool if r["family_id"] in fam_mw]
|
||||
pairs = []
|
||||
for mw in mr:
|
||||
for k in kr:
|
||||
if subject in mw["n_text"] and subject in k["n_text"]:
|
||||
pairs.append((mw["idx"], k["idx"], mw["n_text"], k["n_text"]))
|
||||
return pairs
|
||||
|
||||
sets = {
|
||||
"water": build_pairs("вод", ["knowledge:recall-fact"], ["fact:water"]),
|
||||
"homelab": build_pairs("dns", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("сервер", ["knowledge:homelab-status"], ["note:homelab"])
|
||||
+ build_pairs("vlan", ["knowledge:homelab-status"], ["note:homelab"]),
|
||||
"task": build_pairs("задач", ["knowledge:task-check", "knowledge:deadline"],
|
||||
["note:task"]),
|
||||
}
|
||||
idx_of = {r["idx"]: i for i, r in enumerate(pool)}
|
||||
pair_rep = {}
|
||||
for name, pairs in sets.items():
|
||||
if not pairs:
|
||||
continue
|
||||
ok = 0
|
||||
margins = []
|
||||
bad = []
|
||||
for mi, ki, mx, kx in pairs:
|
||||
mi_i, ki_i = idx_of[mi], idx_of[ki]
|
||||
# MW row should get a higher memory_write probability than the K row
|
||||
mk = (pmw[mi_i] + 0.0)
|
||||
if pmw[mi_i] > pmw[ki_i]:
|
||||
ok += 1
|
||||
else:
|
||||
bad.append((mx[:46], round(float(pmw[mi_i]), 3), kx[:46], round(float(pmw[ki_i]), 3)))
|
||||
margins.append(pmw[mi_i] - pmw[ki_i])
|
||||
pair_rep[name] = {
|
||||
"pairs": len(pairs),
|
||||
"mw_over_k_order_acc": ok / len(pairs),
|
||||
"mean_margin": float(np.mean(margins)),
|
||||
"reversed_examples": bad[:6],
|
||||
}
|
||||
print(f" {name}: pairs={len(pairs)} order_acc={ok/len(pairs):.3f} "
|
||||
f"mean_margin={np.mean(margins):+.3f}")
|
||||
report["kmw"]["matched_pairs"] = pair_rep
|
||||
|
||||
# ── §7 uncertain as explicit class ────────────────────────────────────
|
||||
print("\n§7 uncertain")
|
||||
up = m_best["per_class"]["uncertain"]
|
||||
uc = m_best["confusion"][CLASSES.index("uncertain")]
|
||||
report["uncertain"] = {
|
||||
"per_class": up,
|
||||
"row_from_uncertain": {CLASSES[j]: int(uc[j]) for j in range(5)},
|
||||
"row_to_uncertain": {CLASSES[j]: int(m_best["confusion"][j][CLASSES.index("uncertain")])
|
||||
for j in range(5)},
|
||||
}
|
||||
print(f" uncertain n={up['n']} P={up['p']:.3f} R={up['r']:.3f} F1={up['f1']:.3f}")
|
||||
print(" wrong-→label pulled from uncertain:", report["uncertain"]["row_from_uncertain"])
|
||||
print(" →uncertain pulled from:", report["uncertain"]["row_to_uncertain"])
|
||||
|
||||
# ── §8 OOF confidence / calibration / abstention ─────────────────────
|
||||
print("\n§8 confidence / calibration")
|
||||
conf = p_best.max(1)
|
||||
right = (p_best.argmax(1) == yt)
|
||||
cer = {
|
||||
"correct_conf_mean": float(conf[right].mean()),
|
||||
"correct_conf_median": float(np.median(conf[right])),
|
||||
"wrong_conf_mean": float(conf[~right].mean()),
|
||||
"wrong_conf_median": float(np.median(conf[~right])),
|
||||
"ece": ece(yt, p_best)["ece"],
|
||||
"ece_bins": ece(yt, p_best)["bins"],
|
||||
"log_loss": float(log_loss(yt, p_best, labels=[0, 1, 2, 3, 4])),
|
||||
}
|
||||
# Brier is label-set specific: one-vs-rest mean
|
||||
briers = []
|
||||
for i in range(5):
|
||||
briers.append(brier_score_loss((yt == i).astype(int), p_best[:, i]))
|
||||
cer["brier_macro"] = float(np.mean(briers))
|
||||
report["confidence"] = cer
|
||||
print(f" right conf mean={cer['correct_conf_mean']:.3f} "
|
||||
f"wrong conf mean={cer['wrong_conf_mean']:.3f} ECE={cer['ece']:.4f}")
|
||||
print(f" log_loss={cer['log_loss']:.4f} brier_macro={cer['brier_macro']:.4f}")
|
||||
|
||||
thr_grid = np.linspace(0.10, 0.98, 45)
|
||||
abst = []
|
||||
for t in thr_grid:
|
||||
cov = (conf >= t).mean()
|
||||
if cov == 0:
|
||||
continue
|
||||
keep = conf >= t
|
||||
yt_k = yt[keep]
|
||||
yp_k = p_best[keep].argmax(1)
|
||||
mk_ = cls_metrics(yt_k.tolist(), yp_k.tolist())
|
||||
abst.append({"threshold": round(float(t), 3), "coverage": float(cov),
|
||||
"accuracy": mk_["acc"], "macro_f1": mk_["macro_f1"]})
|
||||
report["confidence"]["abstention_curve"] = abst
|
||||
print(" threshold | coverage | accuracy | macroF1 (first 6/45 + knee)")
|
||||
for row in abst[::9]:
|
||||
print(f" {row['threshold']:.2f} | {row['coverage']:.3f} | "
|
||||
f"{row['accuracy']:.3f} | {row['macro_f1']:.3f}")
|
||||
|
||||
# ── §9 action OOD probes ──────────────────────────────────────────────
|
||||
print("\n§9 action OOD")
|
||||
ood_rows = [r for r in json.load(open(os.path.join(OUT_DIR, "ood.json")))]
|
||||
emb_by_idx = {i: np.asarray(e["embedding"], dtype=np.float64)
|
||||
for i, e in enumerate(slice18_sparse.filter_dev_pool(
|
||||
slice18_sparse.load_data()[1]))}
|
||||
Xo = np.vstack([emb_by_idx[r["idx"]] for r in ood_rows])
|
||||
fold_models = []
|
||||
for te_fold in sorted(set(folds_arr.tolist())):
|
||||
tr = folds_arr != te_fold
|
||||
clf = slice18_sparse.LogisticRegression(
|
||||
C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X[tr], yt[tr])
|
||||
fold_models.append(clf)
|
||||
# XX
|
||||
# comparable to the non-action in-fold behaviour
|
||||
po = np.zeros((len(Xo), 5))
|
||||
for clf in fold_models:
|
||||
po += clf.predict_proba(Xo)
|
||||
po /= len(fold_models)
|
||||
ood_top = int(np.argmax(po.mean(0)))
|
||||
ood_conf = po.max(1)
|
||||
ood_pred = po.argmax(1)
|
||||
top_dist = {CLASSES[i]: int((ood_pred == i).sum()) for i in range(5)}
|
||||
confident_na = int((ood_conf > 0.9).sum())
|
||||
report["ood"] = {
|
||||
"n": len(ood_rows),
|
||||
"top_class": CLASSES[int(ood_top)],
|
||||
"top_class_dist": top_dist,
|
||||
"conf_gt_0.9": confident_na,
|
||||
"conf_gt_0.9_frac": float(confident_na / len(ood_rows)),
|
||||
"conf_mean": float(ood_conf.mean()),
|
||||
"conf_median": float(np.median(ood_conf)),
|
||||
}
|
||||
print(f" action OOD n={len(ood_rows)}: most-confident class={report['ood']['top_class']} "
|
||||
f"dist={top_dist}")
|
||||
print(f" conf>0.9: {confident_na} ({confident_na/len(ood_rows):.3f}) "
|
||||
f"conf mean={report['ood']['conf_mean']:.3f}")
|
||||
|
||||
# ── §10 artifact cost ────────────────────────────────────────────────
|
||||
print("\n§10 artifact")
|
||||
n_params = len(CLASSES) * X.shape[1] + len(CLASSES)
|
||||
fp32 = n_params * 4
|
||||
report["artifact"] = {
|
||||
"e5_dim": X.shape[1],
|
||||
"head_params": n_params,
|
||||
"head_fp32_bytes": fp32,
|
||||
"head_fp32_kib": fp32 / 1024,
|
||||
"head_int8_bytes": n_params,
|
||||
}
|
||||
# incremental latency of the linear head over a batch of 1 (µs)
|
||||
clf = slice18_sparse.LogisticRegression(C=bestC, max_iter=2000, solver="lbfgs", random_state=42)
|
||||
clf.fit(X, yt)
|
||||
x1 = X[:1]
|
||||
for _ in range(50):
|
||||
clf.predict_proba(x1)
|
||||
lat = []
|
||||
for _ in range(2000):
|
||||
t0 = time.perf_counter_ns()
|
||||
clf.predict_proba(x1)
|
||||
lat.append((time.perf_counter_ns() - t0) / 1e3)
|
||||
lat = np.array(lat)
|
||||
report["artifact"]["head_latency_us_mean"] = float(lat.mean())
|
||||
report["artifact"]["head_latency_us_p50"] = float(np.median(lat))
|
||||
print(f" head params={n_params} fp32={fp32/1024:.2f}KiB "
|
||||
f"lat mean={lat.mean():.2f}us p50={np.median(lat):.2f}us")
|
||||
|
||||
with open(os.path.join(OUT_DIR, "results.json"), "w") as f:
|
||||
json.dump(report, f, ensure_ascii=False, indent=1, default=float)
|
||||
print(f"\nwrote {OUT_DIR}/results.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+1
-1
@@ -259,7 +259,7 @@
|
||||
"heads_path": "/opt/maven/models/embedder/router-heads/router_heads.onnx"
|
||||
},
|
||||
"llm_router": true,
|
||||
"query_min_score": 0.55,
|
||||
"query_min_score": 0.80,
|
||||
"query_min_margin": 0.008,
|
||||
"clarify_max_attempts": 3,
|
||||
"tool_timeout": "30s",
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
{
|
||||
"db_path": "/var/lib/maven/maven.db.enc",
|
||||
"db_tmpfs": "/dev/shm/maven-plain.db",
|
||||
"db_key_env": "MAVEN_DB_KEY",
|
||||
"socket_path": "/run/maven/mavend.sock",
|
||||
"state_dir": "/var/lib/maven",
|
||||
|
||||
"//disabled_rules": [
|
||||
"Nudge rules that are not wired at all. Names come from loop.DefaultRules:",
|
||||
"water, meal, break, service_down, netdata_critical.",
|
||||
"service_down is back on: mavpoll now writes one fact per kuma monitor",
|
||||
"(service_down:<name>), so the nudge names the service and pausing a monitor",
|
||||
"in kuma silences that monitor. It is also edge-triggered, so a service that",
|
||||
"stays down is one nudge, not one every fifteen minutes."
|
||||
],
|
||||
"disabled_rules": [],
|
||||
|
||||
"phraser": {
|
||||
"model_path": "/opt/maven/models/llm/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf",
|
||||
"bin_path": "llama-server",
|
||||
"n_gpu_layers": 99,
|
||||
"n_ctx": 4096,
|
||||
"cache_ram_mib": 512,
|
||||
"timeout": "60s",
|
||||
"llm_nudges": false
|
||||
},
|
||||
|
||||
"//ntfy": [
|
||||
"The second reach (V-649). Until 07-08-2026 telegram was the only one, and",
|
||||
"telegram needs api.telegram.org, the socks relay below and a matching ufw",
|
||||
"rule — three things in series that have each failed once, and when they do",
|
||||
"a sev4 nudge has nowhere to go. ntfy shares none of them: it is reached",
|
||||
"directly, no relay.",
|
||||
"It is the preferred away reach. Reminders fall back to Telegram in order,",
|
||||
"and every missing reach is recorded in the outbox rather than disappearing.",
|
||||
"The credential is an ntfy access token, scoped write-only to this one",
|
||||
"topic, so a popped sink can push to it and cannot read it back. Set it in",
|
||||
"deploy/telegram.env beside the telegram secrets; that file is gitignored."
|
||||
],
|
||||
"ntfy": {
|
||||
"disabled": true,
|
||||
"base_url": "https://ntfy.kvmx.ru",
|
||||
"topic": "maven",
|
||||
"token": "${NTFY_TOKEN}"
|
||||
},
|
||||
|
||||
"telegram": {
|
||||
"bot_token": "${TELEGRAM_BOT_TOKEN}",
|
||||
"chat_id": "${TELEGRAM_CHAT_ID}",
|
||||
|
||||
"//proxy": [
|
||||
"api.telegram.org is not reachable directly from this box, so every send",
|
||||
"timed out. The relay is the x-ui socks inbound on the host, port 10808;",
|
||||
"192.168.240.1 is the maven_default bridge gateway, which is how a",
|
||||
"container addresses the host. mavend is on that network.",
|
||||
"This needs a matching ufw rule or the container's SYN is dropped:",
|
||||
" ufw allow from 192.168.240.0/20 to any port 10808 proto tcp"
|
||||
],
|
||||
"proxy": "socks5://192.168.240.1:10808",
|
||||
"//intake": [
|
||||
"Read the chat as well as write to it (V-637). The poller long-polls",
|
||||
"getUpdates through the same relay and accepts chat_id as the only",
|
||||
"sender. Deleting this key turns inbound off again.",
|
||||
"chat_id must be numeric here or the daemon refuses to start: an inbound",
|
||||
"update names its chat by number, so an @-name would match nothing."
|
||||
],
|
||||
"intake": true
|
||||
},
|
||||
|
||||
"//workstation": [
|
||||
"The big model on the desk PC (workpc, 7900 GRE 16GB), fronted by",
|
||||
"mavgpud on port 8080. It runs gemma-4-12b and it is preferred over the",
|
||||
"resident Qwen3-1.7B for routing and replies whenever the card is free.",
|
||||
"The machine is never assumed up: it sleeps, and the card is often held by",
|
||||
"a CPT run, in which case mavgpud answers 503 and Maven falls back to the",
|
||||
"resident model without saying so. Deleting this block restores exactly",
|
||||
"the behaviour homesrv had before it existed.",
|
||||
"Addressed by LAN address, not container name: mavgpud runs on another",
|
||||
"machine and there is no shared docker network to name it on.",
|
||||
"model_disabled keeps only this model arm dark until MAVEN_GPU_TOKEN is",
|
||||
"provisioned; the independently authenticated STT arm below stays live."
|
||||
],
|
||||
"//workstation.stt": [
|
||||
"CrisperWhisper 2.0 turbo on the same machine, a second service on port",
|
||||
"8081 and not a second endpoint on mavgpud. whisper.cpp cannot load CW2 at",
|
||||
"all: it derives its language count from the vocabulary size, and CW2's",
|
||||
"51897 tokens shift seven special token ids. So it runs under transformers",
|
||||
"there and mavsttd stays whisper.cpp here.",
|
||||
"Worth the second service: CW2 turbo scores 10.4% WER in Russian against",
|
||||
"27.5% for the ggml-small.bin mavsttd loads, measured on 200 Golos clips",
|
||||
"in docs/evals/2026-08-09-crisperwhisper2-russian-wer.md.",
|
||||
"Deleting this block sends every utterance to mavsttd, which is what the",
|
||||
"box did before it existed. A worse transcript is still a turn, so the",
|
||||
"fallback is silent and Kami is never told which machine heard him.",
|
||||
"The token is what stops anything on the LAN posting audio to that port."
|
||||
],
|
||||
"workstation": {
|
||||
"model_disabled": true,
|
||||
"url": "http://192.168.1.105:8080",
|
||||
"token": "${MAVEN_GPU_TOKEN}",
|
||||
"probe": "15s",
|
||||
"timeout": "90s",
|
||||
"stt": {
|
||||
"url": "http://192.168.1.105:8081/transcribe",
|
||||
"token": "${MAVEN_STT_TOKEN}",
|
||||
"probe": "15s",
|
||||
"timeout": "10s"
|
||||
}
|
||||
},
|
||||
|
||||
"//search": [
|
||||
"The live web, searched after his own notes and before Kiwix. Only the",
|
||||
"query string leaves the box — never a note, a fact, the persona block or",
|
||||
"the history — and a question about him never reaches here at all.",
|
||||
"The instance must have `json` in search.formats (settings.yml); a stock",
|
||||
"SearXNG answers 403 to format=json and every search then fails. It is",
|
||||
"addressed by container name, so it needs the same maven_default",
|
||||
"attachment kiwix has, and it must listen on 9563: 8080 is taken several",
|
||||
"times over on this box. No instance reachable ⇒ she falls through to the",
|
||||
"ZIMs and never says the search failed."
|
||||
],
|
||||
"search": {
|
||||
"url": "http://searxng:9563",
|
||||
"max_results": 4,
|
||||
"snippet_runes": 1500,
|
||||
"language": "auto",
|
||||
"timeout": "8s"
|
||||
},
|
||||
|
||||
"//kiwix": [
|
||||
"The offline encyclopedia, searched after his own notes and before anything",
|
||||
"on the network. kiwix-server publishes 8034 on loopback only, so a container",
|
||||
"cannot reach it by address; it is attached to the maven_default network",
|
||||
"instead and addressed by container name. That attachment is imperative and",
|
||||
"does not survive recreating the kiwix stack — make it declarative there:",
|
||||
" networks: [default, maven_default] # maven_default: external: true",
|
||||
"The book is the catalog name from the /content/... href in",
|
||||
"/catalog/v2/entries, not the display title. Others on the box:",
|
||||
"ifixit_en_all_2025-06, devdocs_en_ansible_2025-10."
|
||||
],
|
||||
"kiwix": {
|
||||
"url": "http://kiwix-server:8080",
|
||||
"book": "wikipedia_en_all_maxi_2026-02",
|
||||
"book_ru": "wikipedia_ru_all_maxi_2026-02",
|
||||
"max_results": 5,
|
||||
"snippet_runes": 1500
|
||||
},
|
||||
|
||||
"//morning_routines": [
|
||||
"The daily checklist (Vikunja #280). Each item is done when its fact_key",
|
||||
"gets a non-voided fact inside the window, so 'выпил воды' closes water and",
|
||||
"nothing has to be ticked by hand. nudge_at fires once, at the end of the",
|
||||
"window, and only for what is still open. Weekdays empty = every day."
|
||||
],
|
||||
"morning_routines": [
|
||||
{
|
||||
"name": "утро",
|
||||
"window_start": "08:00",
|
||||
"window_end": "11:00",
|
||||
"nudge_at": "10:30",
|
||||
"severity": 1,
|
||||
"items": [
|
||||
{ "key": "medicine", "fact_key": "medicine", "label": "лекарство" },
|
||||
{ "key": "water", "fact_key": "water", "label": "вода" },
|
||||
{ "key": "pets", "fact_key": "pets", "label": "покормить кота" }
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
"//feeds": [
|
||||
"RSS reading (Vikunja #258). Every item lands as a note with source",
|
||||
"rss:<name>, which is also what puts entries in the intake journal that",
|
||||
"/events reads. Only the feed URL leaves the box.",
|
||||
"This is a starting pair, not a curated set — trim or extend it."
|
||||
],
|
||||
"feeds": {
|
||||
"poll_interval": "30m",
|
||||
"max_items": 5,
|
||||
"max_age": "24h",
|
||||
"sources": [
|
||||
{ "name": "lwn", "url": "https://lwn.net/headlines/newrss", "category": "технологии" },
|
||||
{ "name": "archlinux", "url": "https://archlinux.org/feeds/news/", "category": "технологии" }
|
||||
]
|
||||
},
|
||||
|
||||
"//crawl": [
|
||||
"Reading a web page (Vikunja #259). on_demand answers 'посмотри <URL>'.",
|
||||
"No allow_hosts, so any public host he names is readable; private",
|
||||
"addresses are refused unconditionally by internal/webfetch and do not",
|
||||
"need listing. Setting allow_hosts here would also narrow on-demand,",
|
||||
"which is the point of leaving it empty."
|
||||
],
|
||||
"crawl": {
|
||||
"on_demand": true,
|
||||
"timeout": "10s",
|
||||
"max_runes": 4000
|
||||
},
|
||||
|
||||
"digest": {
|
||||
"enabled": true,
|
||||
"window": "30m",
|
||||
"max_items": 5,
|
||||
"severity_ceiling": 2
|
||||
},
|
||||
|
||||
"pattern_proposals": {
|
||||
"notify": false,
|
||||
"cooldown": "24h"
|
||||
},
|
||||
|
||||
"mcp": {
|
||||
"timeout": "15s",
|
||||
"servers": [
|
||||
{
|
||||
"name": "vikunja",
|
||||
"url": "http://192.168.1.104:9100/mcp",
|
||||
"allow_private": true,
|
||||
"allow_tools": ["list_projects", "list_tasks", "get_task_details", "create_task"],
|
||||
"max_tools": 6,
|
||||
"enabled": false
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
"smarthome": {
|
||||
"provider": "homeassistant",
|
||||
"url": "http://192.168.1.50:8123",
|
||||
"token": "${HA_TOKEN}",
|
||||
"domains": ["light", "switch", "sensor"],
|
||||
"max_entities": 40,
|
||||
"timeout": "10s",
|
||||
"refresh": "15m",
|
||||
"enabled": false
|
||||
},
|
||||
|
||||
"netscan": {
|
||||
"subnets": ["192.168.1.0/24"],
|
||||
"ports": [22, 80, 443, 8080],
|
||||
"timeout": "400ms",
|
||||
"rate": 100,
|
||||
"max_hosts": 256,
|
||||
"enabled": true
|
||||
},
|
||||
|
||||
"nexus": { "url": "http://nexus:9740" },
|
||||
"praxis": { "url": "http://praxis:8989" },
|
||||
"hexis": { "url": "http://hexis:9741" },
|
||||
|
||||
"voice": {
|
||||
"enabled": true,
|
||||
"bind": "0.0.0.0:9100",
|
||||
"lang": "ru",
|
||||
"stt": { "socket": "/run/maven/stt.sock", "lang": "ru" },
|
||||
"tts": { "socket": "/run/maven/tts.sock", "lang": "ru" },
|
||||
"embedder": {
|
||||
"model_path": "/opt/maven/models/embedder/multilingual-e5-small/model_quantized.onnx",
|
||||
"tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json",
|
||||
"lib_path": "/opt/maven/lib/libonnxruntime.so",
|
||||
"heads_path": "/opt/maven/models/embedder/router-heads/router_heads.onnx"
|
||||
},
|
||||
"llm_router": true,
|
||||
"query_min_score": 0.80,
|
||||
"query_min_margin": 0.008,
|
||||
"clarify_max_attempts": 3,
|
||||
"tool_timeout": "30s",
|
||||
"tools": [
|
||||
{ "name": "status", "cmd": ["systemctl", "status"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["статус", "покажи статус", "проверь статус"] },
|
||||
{ "name": "ps", "cmd": ["docker", "ps"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["статус докера", "лог докера", "покажи запущенные контейнеры", "покажи контейнеры", "список контейнеров", "что запущено"] },
|
||||
{ "name": "uptime", "cmd": ["uptime"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["покажи uptime", "аптайм", "как работает сервер", "сколько работает сервер"] },
|
||||
{ "name": "disk", "cmd": ["df", "-h"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["сколько места на диске", "сколько свободного места на диске", "место на диске", "покажи диск"] },
|
||||
{ "name": "memory", "cmd": ["free", "-h"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["свободная память", "сколько оперативной памяти свободно", "покажи память"] },
|
||||
{ "name": "logs", "cmd": ["journalctl", "-n", "50", "-u"], "scope": "homelab", "destructive": false,
|
||||
"aliases": ["покажи логи", "логи", "лог"] },
|
||||
{ "name": "restart", "cmd": ["systemctl", "restart"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["перезапусти", "перезагрузи", "рестарт"] },
|
||||
{ "name": "stop", "cmd": ["systemctl", "stop"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["останови", "останови сервис"] },
|
||||
{ "name": "start", "cmd": ["systemctl", "start"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["запусти", "запусти сервис"] },
|
||||
{ "name": "docker-restart", "cmd": ["docker", "restart"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["перезапусти контейнер", "перезагрузи контейнер"] },
|
||||
{ "name": "docker-stop", "cmd": ["docker", "stop"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["останови контейнер"] },
|
||||
{ "name": "reboot", "cmd": ["systemctl", "reboot"], "scope": "homelab", "destructive": true,
|
||||
"aliases": ["перезагрузи сервер", "перезагрузи хост"] }
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,8 @@ The tier is the path, so staleness is visible from the filename.
|
||||
| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. Indexed in `docs/evals/CLAUDE.md`, which marks each one live or superseded. | forever |
|
||||
| `docs/caveats/` | known limits, one entry per limit, each with a task id and a revisit trigger. Indexed in `docs/caveats/CLAUDE.md`. | until fixed, then deleted |
|
||||
| `docs/plans/` | the plan for one piece of work, frozen once it starts. Indexed in `docs/plans/CLAUDE.md`. | until the work lands |
|
||||
| `docs/capabilities/` | generated. The capability ledger and its probe harness, regenerated from `docs/spec.md` plus a named eval. **Never hand-edited**, except `domains.yaml`, `probes_field.json` and the scripts, which are its sources. Indexed in `docs/capabilities/README.md`. | until the spec or the measurement moves |
|
||||
| `docs/architecture/` | generated. The architecture observation and its evidence pack, rebuilt from source by the scripts beside it. Indexed in `docs/architecture/README.md`. | until the shape changes |
|
||||
| `docs/archive/` | dead. Read by nobody by default. | forever |
|
||||
|
||||
## Rules for this directory
|
||||
@@ -21,6 +23,10 @@ The tier is the path, so staleness is visible from the filename.
|
||||
correction. Do not append a changelog to it.
|
||||
* A number in prose with no `docs/evals/` file behind it is an opinion.
|
||||
* Fixing something deletes its caveat. It does not edit the eval that found it.
|
||||
* **A generated tier is rebuilt, never corrected.** A wrong row in
|
||||
`docs/capabilities/` or `docs/architecture/` is a bug in the generator or in
|
||||
one of its hand-written inputs. Editing the output makes the next rebuild
|
||||
silently undo the fix.
|
||||
|
||||
## Where a subsystem's reasoning lives
|
||||
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
# docs/architecture
|
||||
|
||||
An observation of Maven as built, read at `5cae33a` on 2026-08-25. It describes
|
||||
what the code does today. It proposes nothing.
|
||||
|
||||
This directory is a build output plus its sources. `index.html`,
|
||||
`maven-architecture.json`, `anchors.md` and `diagrams/*.svg` are generated.
|
||||
|
||||
## Read it
|
||||
|
||||
| file | what it is |
|
||||
|---|---|
|
||||
| `index.html` | the viewer. Open it from the filesystem, no server needed. Seven views, the rendered diagram above each, click a component for its record. |
|
||||
| `check_viewer.js` | the viewer's only check. Runs every view against a DOM stub, because a TypeError in a renderer shows as a blank panel and not as an error. |
|
||||
| `findings.md` | the analysis. Kept apart from the facts on purpose. |
|
||||
| `maven-architecture.json` | the inventory. 160 components, 204 relations. The factual source for everything else. |
|
||||
| `anchors.md` | every symbol the inventory names, resolved to `path:line` with the line quoted. |
|
||||
| `diagrams/*.mmd` | the five views as Mermaid source. `03a`, `03b` and `03c` are the three traced requests. |
|
||||
| `diagrams/*.svg` | the same, rendered. |
|
||||
|
||||
## Rebuild it
|
||||
|
||||
```sh
|
||||
python3 docs/architecture/build_inventory.py # → maven-architecture.json
|
||||
python3 docs/architecture/verify_anchors.py # → anchors.md, exit 1 if stale
|
||||
sh docs/architecture/render.sh # → diagrams/*.svg, then index.html, then check_viewer.js
|
||||
python3 docs/architecture/build_viewer.py # → index.html alone
|
||||
node docs/architecture/check_viewer.js # → every view rendered, no throw
|
||||
```
|
||||
|
||||
Views 6 and 7 read `docs/capabilities/`, not this directory. View 6 is the
|
||||
capability matrix, 51 rows against seven dimensions. View 7 is the twelve
|
||||
cross-cutting invariants and the components that participate in each. Both are
|
||||
inlined by `build_viewer.py`, which reads `ledger.yaml` and `invariants.yaml`
|
||||
rather than deriving anything itself: the ledger's build is the only thing
|
||||
allowed to decide a dimension.
|
||||
|
||||
`render.sh` drives mermaid-cli through the system chromium rather than letting
|
||||
puppeteer download its own. It is also the only syntax check this repo has for a
|
||||
`.mmd`.
|
||||
|
||||
## What is verified, and what is not
|
||||
|
||||
**Verified mechanically.** `verify_anchors.py` resolves all 692 claimed symbols
|
||||
against the files their component names. Current state: 681 resolved to a line
|
||||
and 0 unresolved, with 0 missing files. The other 11 are config keys and make
|
||||
targets rather than Go identifiers, so they are skipped. The script exits
|
||||
non-zero on any failure, which makes it a staleness gate.
|
||||
|
||||
Writing it caught 29 symbols filed under the wrong component and seven names
|
||||
that were wrong outright. Two examples: `Store.RecordEvent` for what is really
|
||||
`Store.CreateEvent`, and `media.Keeper` for what is really `media.Store`.
|
||||
|
||||
**Not verified.** That a symbol means what its `responsibility` says. An anchor
|
||||
proves the identifier is on that line and nothing more. Judgements about
|
||||
ownership, coupling and enforcement are readings of the code. A reading can be
|
||||
wrong in a way grep cannot catch.
|
||||
|
||||
**Marked, not resolved.** Relations carry a `confidence` field. `medium` means
|
||||
the wiring is in the source and the call path was not traced end to end. `low`
|
||||
means it was inferred from one reference. The viewer can hide both. Four
|
||||
relations are `medium` and one is `low`.
|
||||
|
||||
**Deployment-specific.** Sixteen components are `configured-off` against
|
||||
`deploy/mavend.json` as it stood on the day, and that file was dirty in the
|
||||
working tree. A different config makes different components live. `status` says
|
||||
which, per component.
|
||||
|
||||
## The one thing to check first
|
||||
|
||||
`findings.md` 6.3 through 6.3d. They say the system has no single point that
|
||||
decides whether an origin may cause an effect, and that the pieces which look
|
||||
like that point are each answering a different question.
|
||||
|
||||
Revised on 2026-08-25 after an independent second pass. Four readings changed
|
||||
and one earlier statement was wrong. Section 6.3 marks the corrections.
|
||||
|
||||
Start at `internal/tool/tool.go:181`, `cmd/mavend/ecosystem_acts.go:158` and
|
||||
`internal/router/claim.go:38`.
|
||||
|
||||
## The evidence pack
|
||||
|
||||
`sh docs/architecture/pack_evidence.sh` builds `maven-evidence.zip` at the repo
|
||||
root: this directory, the structural context, and whole source files for the
|
||||
architectural seams. Whole files, never snippets, because a cut-down file loses
|
||||
the call path that makes a claim checkable.
|
||||
|
||||
The path list is an allowlist, not an exclusion list. A denylist ships whatever
|
||||
nobody thought to exclude, and this tree has a database key in it.
|
||||
|
||||
`architecture-evidence.txt` is the reviewer's index. It resolves a named symbol
|
||||
list against the checkout and says plainly when a requested name does not exist.
|
||||
It also re-runs the probe under every contradiction, so a claim and its grep
|
||||
cannot drift apart.
|
||||
|
||||
One file is not verbatim. `docker-compose.yml` carries an uptime-kuma API key,
|
||||
so a redacted copy ships in its place with that one value replaced. The script
|
||||
diffs the two and aborts if anything else changed.
|
||||
|
||||
The scan at the end refuses to build on a credential-shaped hit rather than
|
||||
printing a warning. Both of its first two versions were wrong in instructive
|
||||
ways. The name filter deleted `internal/router/singletoken.go` for matching
|
||||
`*token*`. The value scan flagged docker volume lines that name where a secret
|
||||
would live and contain none.
|
||||
|
||||
## The authorization function as implemented
|
||||
|
||||
The reconstruction, at the one decision point that gates an act
|
||||
(`internal/tool/tool.go:156`):
|
||||
|
||||
```
|
||||
permit(tool, confirmed) =
|
||||
row.status == "enabled" tool.go:164
|
||||
AND tier != irreversible risk.go:74 VoiceMayRun:false
|
||||
AND (tier == safe OR confirmed) risk.go:72,76
|
||||
```
|
||||
|
||||
`tier` is `RiskOf(row)`. The reach is not an input: `Executor.Exec` takes
|
||||
`(ctx, name, args, confirmed)` and no surface.
|
||||
|
||||
`confirmed` is unproven at this boundary too. The invariant that a confirmation
|
||||
binds one capability, one target and an expiry lives in `pendingAct` and
|
||||
`resolveConfirm`. `Exec` trusts the boolean.
|
||||
|
||||
The expression covers two of the three act paths. Hexis reuses it deliberately
|
||||
(`cmd/mavend/ecosystem_acts.go:768`). The Praxis lifecycle path has no tier and
|
||||
no confirm turn: `praxisItemAction.handle` calls straight through at
|
||||
`ecosystem_acts.go:158`.
|
||||
|
||||
Behind the IPC boundary, `auth.Can(method, scope, params)` runs with
|
||||
`scope.Surface` always `SurfaceCoreProcess` (`internal/auth/enrollment.go:65`)
|
||||
and step-up held as one global timestamp that ignores `Scope`
|
||||
(`internal/webauthn/session.go:38` and `:62`).
|
||||
|
||||
`auth` answers who may carry what authority. `tool` answers what effect a
|
||||
capability has and what proof it demands. Those are orthogonal, not competing.
|
||||
The decision combining them does not exist.
|
||||
|
||||
Two representations of reach exist and both are ignored. `server.go:198` only
|
||||
defaults an empty `p.Surface`, so a client-asserted one survives and nothing
|
||||
reads it. `server.go:148` hardcodes `Session.Surface`. Since `req.Surface` is
|
||||
request payload on an unauthenticated wire, it must not become an authorization
|
||||
input as it stands.
|
||||
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write architecture-evidence.txt: the reviewer's index into the pack.
|
||||
|
||||
It resolves a named symbol list against this checkout and prints where each one
|
||||
is, or says plainly that it does not exist. A requested name that is absent is
|
||||
evidence too, so nothing here is silently dropped or silently corrected.
|
||||
|
||||
Every contradiction is re-checked at generation time by running its own probe,
|
||||
so the claim and the grep that supports it cannot drift apart in the pack.
|
||||
|
||||
python3 docs/architecture/build_evidence.py
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
ROOT = os.path.abspath(os.path.join(HERE, "..", ".."))
|
||||
TREES = ["cmd", "internal"]
|
||||
|
||||
# The reviewer's list, verbatim on the left. Where a name does not exist in this
|
||||
# repo, the right side is what it appears to mean. Resolution below reports both
|
||||
# so a wrong name is visible rather than quietly fixed.
|
||||
REQUESTED = [
|
||||
("auth.TierFor", "auth.MaxLayer"),
|
||||
("auth.Surface", None),
|
||||
("auth.Layer", None),
|
||||
("tool.Executor.Exec", None),
|
||||
("tool.PolicyFor", None),
|
||||
("tool.RiskOf", None),
|
||||
("tool.RiskSafe", "tool.TierSafe"),
|
||||
("tool.RiskDestructive", "tool.TierDestructive"),
|
||||
("tool.RiskIrreversible", "tool.TierIrreversible"),
|
||||
("router.ClaimOf", None),
|
||||
("reactiveHandler", None),
|
||||
("tickLoop", None),
|
||||
]
|
||||
|
||||
# Added because the authorization function the reviewer wants to reconstruct
|
||||
# runs through these and the list above does not reach them.
|
||||
ALSO = [
|
||||
"auth.Gate", "auth.Can", "auth.Requirement", "auth.Authority",
|
||||
"auth.NewFloorEnrollment", "auth.StaticEnrollment", "auth.Scope",
|
||||
"tool.Policy", "tool.RiskOfCapability", "tool.irreversibleVerbs",
|
||||
"tool.ErrNeedsConfirm", "tool.ErrNeedsAuthedSurface", "tool.ErrNotEnabled",
|
||||
"ipc.Server.Check", "ipc.CheckFunc",
|
||||
"voice.PushToTalkReq", "voice.Sessions.Add", "voice.Session",
|
||||
"PolicyFor", "RiskOf", "Executor.Exec",
|
||||
"pendingAct", "resolveConfirm", "actionAct", "runTurn", "applyAction",
|
||||
"querySources", "queryWalk", "StageZeroGrammars", "Router.Route",
|
||||
]
|
||||
|
||||
PKG_DIR = {
|
||||
"auth": "internal/auth", "tool": "internal/tool", "claim": "internal/claim",
|
||||
"modes": "internal/modes", "router": "internal/router", "voice": "internal/voice",
|
||||
"ipc": "internal/ipc", "store": "internal/store",
|
||||
}
|
||||
|
||||
|
||||
def go_files(rel):
|
||||
full = os.path.join(ROOT, rel)
|
||||
out = []
|
||||
for base, _, names in os.walk(full):
|
||||
for n in sorted(names):
|
||||
if n.endswith(".go"):
|
||||
out.append(os.path.relpath(os.path.join(base, n), ROOT))
|
||||
return sorted(out)
|
||||
|
||||
|
||||
def all_go():
|
||||
out = []
|
||||
for t in TREES:
|
||||
out.extend(go_files(t))
|
||||
return out
|
||||
|
||||
|
||||
_CACHE = {}
|
||||
|
||||
|
||||
def lines_of(rel):
|
||||
"""Read once. resolve() sweeps every file per pattern per symbol, and
|
||||
re-reading cmd/ and internal/ that many times took minutes."""
|
||||
if rel not in _CACHE:
|
||||
try:
|
||||
_CACHE[rel] = open(os.path.join(ROOT, rel), errors="replace").read().splitlines()
|
||||
except OSError:
|
||||
_CACHE[rel] = []
|
||||
return _CACHE[rel]
|
||||
|
||||
|
||||
def resolve(sym):
|
||||
"""Find the declaration of sym. Returns (path, line, text) or (None,)*3."""
|
||||
tail = sym.split(".")[-1]
|
||||
recv = sym.split(".")[-2] if sym.count(".") >= 1 else None
|
||||
pats = [
|
||||
re.compile(r"^func\s+\(\w+\s+\*?" + re.escape(recv or "\x00") + r"\)\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^func\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^type\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^\s*" + re.escape(tail) + r"\s+\w+\s*=\s"), # typed const
|
||||
re.compile(r"^\s*" + re.escape(tail) + r"\s*=\s"),
|
||||
re.compile(r"^(var|const)\s+" + re.escape(tail) + r"\b"),
|
||||
re.compile(r"^\s*" + re.escape(tail) + r"\s+\w"), # struct field
|
||||
]
|
||||
pkg = sym.split(".")[0]
|
||||
files = go_files(PKG_DIR[pkg]) if pkg in PKG_DIR else all_go()
|
||||
files = [f for f in files if not f.endswith("_test.go")]
|
||||
for pat in pats:
|
||||
for rel in files:
|
||||
for i, line in enumerate(lines_of(rel), 1):
|
||||
if pat.match(line):
|
||||
return rel, i, line.strip()
|
||||
return None, None, None
|
||||
|
||||
|
||||
def exported(pkg_rel):
|
||||
"""Every exported declaration in a package, for the `claim.*` / `modes.*` asks."""
|
||||
out = []
|
||||
pat = re.compile(r"^(func|type|const|var)\s+\(?[^)]*\)?\s*([A-Z]\w*)")
|
||||
fn = re.compile(r"^func\s+(\([^)]*\)\s*)?([A-Z]\w*)")
|
||||
for rel in go_files(pkg_rel):
|
||||
if rel.endswith("_test.go"):
|
||||
continue
|
||||
for i, line in enumerate(lines_of(rel), 1):
|
||||
m = fn.match(line) or pat.match(line)
|
||||
if m:
|
||||
name = m.group(m.lastindex)
|
||||
if name and name[0].isupper():
|
||||
out.append((name, f"{rel}:{i}", line.strip()))
|
||||
return out
|
||||
|
||||
|
||||
def sh(cmd):
|
||||
return subprocess.run(cmd, shell=True, cwd=ROOT, capture_output=True,
|
||||
text=True).stdout.strip()
|
||||
|
||||
|
||||
# Each probe is a shell command whose output IS the evidence. Re-run at pack
|
||||
# time so the pack cannot claim something the checkout no longer shows.
|
||||
CONTRADICTIONS = [
|
||||
("auth surface/layer documented as control, not consumed on turn path",
|
||||
# px.Surface is the Praxis lifecycle verb, an unrelated name collision, and
|
||||
# Surfaced* are the read-out-item helpers. Excluded by name so the absence
|
||||
# this probe reports is the auth Surface and not a filtering accident.
|
||||
"grep -rnE 'req\\.Surface|sess\\.Surface|Session\\.Surface|auth\\.Surface|voice\\.Surface' cmd/mavend/*.go "
|
||||
"| grep -v _test | grep -vE 'px\\.Surface|Surfaced' "
|
||||
"|| echo '(no match: no file in cmd/mavend reads the auth Surface of a request or a session)'"),
|
||||
("auth.Can runs only behind the IPC boundary",
|
||||
"grep -rn 'auth\\.' cmd/ internal/ --include='*.go' | grep -v _test "
|
||||
"| grep -v '^internal/auth/' | grep -vE ':[0-9]+:\\s*(//|\\*)'"),
|
||||
("tool risk policy live on execution path",
|
||||
"sed -n '176,190p' internal/tool/tool.go"),
|
||||
("tool executor receives no reach/surface",
|
||||
"grep -n 'func (e \\*Executor) Exec' internal/tool/tool.go"),
|
||||
("voice server normalizes incoming surface to pc-client",
|
||||
"grep -n 'SurfacePCClient' internal/voice/server.go"),
|
||||
("claim abstraction exists but Route does not consume it",
|
||||
"grep -rn 'ClaimOf' --include='*.go' cmd internal | grep -v _test || echo '(only the definition; no caller)'"),
|
||||
("modes package is imported by nothing",
|
||||
"grep -rn 'internal/modes' --include='*.go' cmd internal | grep -v '^internal/modes/' || echo '(no importer)'"),
|
||||
("voice server DEFAULTS an empty surface, it does not overwrite a sent one",
|
||||
"sed -n '193,201p' internal/voice/server.go"),
|
||||
("session surface is hardcoded, independently of the request field",
|
||||
"sed -n '146,149p' internal/voice/server.go"),
|
||||
("HandlePushToTalk never reads req.Surface",
|
||||
"sed -n '200,203p' cmd/mavend/voice.go"),
|
||||
("hexis reuses the same risk policy",
|
||||
"grep -n 'RiskOfCapability\\|PolicyFor' cmd/mavend/ecosystem_acts.go"),
|
||||
("praxis lifecycle mutations bypass the risk policy entirely",
|
||||
"sed -n '156,172p' cmd/mavend/ecosystem_acts.go"),
|
||||
("Exec trusts a confirmed bool it cannot verify was bound",
|
||||
"grep -n 'func (e \\*Executor) Exec' internal/tool/tool.go; grep -rn 'tools.Exec(' cmd/mavend/*.go | grep -v _test"),
|
||||
("FloorEnrollment maps every same-uid caller to one surface",
|
||||
"sed -n '63,72p' internal/auth/enrollment.go"),
|
||||
("PasskeySession ignores Scope in both methods",
|
||||
"grep -n 'func (s \\*PasskeySession) CurrentLayer\\|func (s \\*PasskeySession) Assert' internal/webauthn/session.go"),
|
||||
("Claim.Coverage can be 1.0 with nothing extracted",
|
||||
"grep -n 'func claimSpans' -A 4 internal/router/claim.go; grep -n 'd.Slots.Text = ex.Text' -B 2 internal/router/router.go; grep -n 'func (c Claim) Coverage' -A 7 internal/claim/claim.go"),
|
||||
("claim_test asserts Band only, and every case sets Text == Utterance",
|
||||
"grep -n 'Utterance:\\|Text:\\|want:' internal/router/claim_test.go | head -20"),
|
||||
("systemctl reboot is destructive, not irreversible",
|
||||
"grep -n 'irreversibleVerbs = map' -A 8 internal/tool/risk.go; grep -n 'reboot' deploy/mavend.json"),
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
out = []
|
||||
w = out.append
|
||||
w("architecture evidence pack")
|
||||
w("=" * 72)
|
||||
w("")
|
||||
w("commit: " + sh("git rev-parse HEAD"))
|
||||
w("date: " + sh("git log -1 --format=%cd --date=short"))
|
||||
w("branch: " + sh("git rev-parse --abbrev-ref HEAD"))
|
||||
w("")
|
||||
w("working tree at pack time (git status --short):")
|
||||
for line in (sh("git status --short") or "(clean)").splitlines():
|
||||
w(" " + line)
|
||||
w("")
|
||||
w("The pack is built from the WORKING TREE, not from the commit. The lines")
|
||||
w("above are the difference. deploy/mavend.json in particular is modified:")
|
||||
w("phraser.model_path points at maven-instruct-b2, the committed value was")
|
||||
w("Qwen3-1.7B-UD-Q4_K_XL. Sixteen 'configured-off' claims read this file.")
|
||||
w("")
|
||||
|
||||
w("requested symbols")
|
||||
w("-" * 72)
|
||||
for name, actual in REQUESTED:
|
||||
rel, line, text = resolve(name)
|
||||
if rel:
|
||||
w(f"- {name}")
|
||||
w(f" {rel}:{line} {text}")
|
||||
elif actual:
|
||||
arel, aline, atext = resolve(actual)
|
||||
w(f"- {name} -> DOES NOT EXIST in this repo")
|
||||
if arel:
|
||||
w(f" the name appears to be {actual}")
|
||||
w(f" {arel}:{aline} {atext}")
|
||||
else:
|
||||
w(f" and neither does {actual}")
|
||||
else:
|
||||
w(f"- {name} -> NOT FOUND")
|
||||
w("")
|
||||
|
||||
for pkg, rel in (("claim.*", "internal/claim"), ("modes.*", "internal/modes")):
|
||||
w(f"{pkg} ({rel})")
|
||||
w("-" * 72)
|
||||
for name, anchor, text in exported(rel):
|
||||
w(f"- {name}")
|
||||
w(f" {anchor} {text}")
|
||||
w("")
|
||||
|
||||
w("additional symbols on the authorization path")
|
||||
w("-" * 72)
|
||||
for name in ALSO:
|
||||
rel, line, text = resolve(name)
|
||||
w(f"- {name}")
|
||||
w(f" {rel}:{line} {text}" if rel else " NOT FOUND")
|
||||
w("")
|
||||
|
||||
w("known contradictions, each re-checked at pack time")
|
||||
w("=" * 72)
|
||||
w("The command under each claim was run against this checkout just now.")
|
||||
w("Its output is what follows. Nothing here is transcribed by hand.")
|
||||
w("")
|
||||
for claim, cmd in CONTRADICTIONS:
|
||||
w("- " + claim)
|
||||
w(" $ " + cmd)
|
||||
res = sh(cmd)
|
||||
for line in (res or "(no output)").splitlines():
|
||||
w(" " + line)
|
||||
w("")
|
||||
|
||||
w("what the pack does NOT contain, and why")
|
||||
w("=" * 72)
|
||||
w("- .git, so no history and no gitignored working files travel with it.")
|
||||
w("- deploy/telegram.env and deploy/db_key.env. The second holds the")
|
||||
w(" database key. Both are gitignored and present in the working tree.")
|
||||
w("- docker-compose.yml verbatim. It carries one live-looking credential on")
|
||||
w(" line 132 (an uptime-kuma API key). The pack ships")
|
||||
w(" docker-compose.redacted.yml with that one value replaced and nothing")
|
||||
w(" else changed, so the mavcaldav and mavmaild claims stay checkable.")
|
||||
w("- models/, deps/, *.db, *.onnx, *.gguf, certs, logs, node_modules.")
|
||||
w("- internal/session, internal/db and tests/ from the requested list: none")
|
||||
w(" of the three exists. Sessions live in internal/voice/session.go, the")
|
||||
w(" store is internal/store, and tests sit beside their code as *_test.go.")
|
||||
w("")
|
||||
w("included test files, since the ask named them by subject:")
|
||||
for pat, label in (
|
||||
("internal/router", "routing and arbitration"),
|
||||
("internal/tool", "risk and confirmation"),
|
||||
("internal/auth", "authorization"),
|
||||
("internal/claim", "claim"),
|
||||
("cmd/mavend", "turn path, confirm gate, query chain"),
|
||||
):
|
||||
n = sh(f"find {pat} -name '*_test.go' | wc -l")
|
||||
w(f" {label}: {n} *_test.go under {pat}/")
|
||||
w("")
|
||||
w("fixtures are synthetic, not captured speech: internal/router/eval/*.json")
|
||||
w("and cmd/mavend/testdata/**.json are hand-written contracts. Named here")
|
||||
w("because they are Russian utterances and look like personal data.")
|
||||
|
||||
path = os.path.join(HERE, "architecture-evidence.txt")
|
||||
open(path, "w").write("\n".join(out) + "\n")
|
||||
print(f"architecture-evidence.txt: {os.path.getsize(path)} bytes, {len(out)} lines")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Assemble index.html from the template, the inventory and the diagrams.
|
||||
|
||||
index.html is self-contained on purpose: it opens from the filesystem with no
|
||||
server, and a browser at file:// refuses to fetch a sibling JSON. So the
|
||||
inventory, every .mmd source and every rendered .svg are inlined here rather
|
||||
than loaded at runtime.
|
||||
|
||||
Run it through docs/architecture/render.sh, which re-renders the SVGs first.
|
||||
Running it alone rebuilds the viewer against whatever SVGs are already there.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
import yaml
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DIA = os.path.join(HERE, "diagrams")
|
||||
CAPDIR = os.path.join(os.path.dirname(HERE), "capabilities")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
arch = json.load(open(os.path.join(HERE, "maven-architecture.json")))
|
||||
mermaid, svg = {}, {}
|
||||
for name in sorted(os.listdir(DIA)):
|
||||
path = os.path.join(DIA, name)
|
||||
if name.endswith(".mmd"):
|
||||
mermaid[name] = open(path).read()
|
||||
elif name.endswith(".svg"):
|
||||
svg[name] = open(path).read()
|
||||
|
||||
# The capability half. Generated beside this one and read here rather than
|
||||
# re-derived: the ledger's build is the only thing allowed to decide a
|
||||
# dimension, and a second derivation would drift from it silently.
|
||||
ledger = yaml.safe_load(open(os.path.join(CAPDIR, "ledger.yaml")))
|
||||
invariants = yaml.safe_load(
|
||||
open(os.path.join(CAPDIR, "invariants.yaml")))["invariants"]
|
||||
|
||||
payload = (
|
||||
"const ARCH = " + json.dumps(arch, ensure_ascii=False) + ";\n"
|
||||
"const CAPS = " + json.dumps(ledger, ensure_ascii=False) + ";\n"
|
||||
"const INV = " + json.dumps(invariants, ensure_ascii=False) + ";\n"
|
||||
"const MERMAID = " + json.dumps(mermaid, ensure_ascii=False) + ";\n"
|
||||
"const SVG = " + json.dumps(svg, ensure_ascii=False) + ";\n"
|
||||
)
|
||||
template = open(os.path.join(HERE, "viewer.template.html")).read()
|
||||
if "/*__DATA__*/" not in template:
|
||||
raise SystemExit("viewer.template.html has no /*__DATA__*/ marker")
|
||||
out = os.path.join(HERE, "index.html")
|
||||
open(out, "w").write(template.replace("/*__DATA__*/", payload))
|
||||
print(
|
||||
"index.html: %d bytes, %d components, %d relations, %d diagrams, "
|
||||
"%d rendered, %d capabilities, %d invariants"
|
||||
% (os.path.getsize(out), len(arch["components"]), len(arch["edges"]),
|
||||
len(mermaid), len(svg), len(ledger["capabilities"]), len(invariants))
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,64 @@
|
||||
// Smoke test for index.html's renderers, run by render.sh when node is present.
|
||||
//
|
||||
// The viewer has no test harness and a TypeError in a renderer produces a blank
|
||||
// panel, not an error anyone sees. This runs every view's render function
|
||||
// against a DOM stub and fails loudly on the first throw. It checks that the
|
||||
// renderers run over the real data, not that the result looks right.
|
||||
//
|
||||
// node docs/architecture/check_viewer.js [path/to/index.html]
|
||||
|
||||
const fs = require('fs');
|
||||
const path = process.argv[2] || __dirname + '/index.html';
|
||||
const src = fs.readFileSync(path, 'utf8');
|
||||
const js = src.match(/<script>([\s\S]*)<\/script>/)[1];
|
||||
|
||||
const el = () => {
|
||||
const e = {
|
||||
innerHTML: '', textContent: '', style: {}, checked: true, value: '',
|
||||
_written: 0,
|
||||
dataset: {}, classList: { add(){}, remove(){}, toggle(){} },
|
||||
querySelectorAll: () => [], querySelector: () => null,
|
||||
appendChild(){}, addEventListener(){}, scrollIntoView(){},
|
||||
getBoundingClientRect: () => ({top:0,left:0,width:0,height:0}),
|
||||
};
|
||||
return e;
|
||||
};
|
||||
// One shared element per id, so a renderer's output can be read back. A stub
|
||||
// that silently swallows innerHTML would let an empty render pass.
|
||||
const els = {};
|
||||
const document = {
|
||||
getElementById: id => (els[id] = els[id] || el()), querySelectorAll: () => [], querySelector: () => null,
|
||||
createElementNS: el, createElement: el, addEventListener(){},
|
||||
};
|
||||
const window = { addEventListener(){} };
|
||||
const requestAnimationFrame = () => {};
|
||||
|
||||
// `const` inside a direct eval stays in the eval's own scope, so the checks are
|
||||
// appended to the source and evaluated with it rather than run beside it.
|
||||
const checks = `
|
||||
let n = 0;
|
||||
for (const v of VIEWS) {
|
||||
setView(v.id);
|
||||
n++;
|
||||
}
|
||||
// Every flow, and every capability's side panel: the branch a click takes.
|
||||
for (const k of Object.keys(FLOWS)) { S.flow = k; renderFlow(el()); n++; }
|
||||
setView('c1');
|
||||
if (els.main.innerHTML.length < 5000) throw new Error('capability matrix rendered ' + els.main.innerHTML.length + ' chars');
|
||||
for (const c of CAPS.capabilities) {
|
||||
renderCapSide(c.id);
|
||||
if (els.side.innerHTML.length < 400) throw new Error('thin panel for ' + c.id);
|
||||
n++;
|
||||
}
|
||||
S.capBy = 'domain'; renderCaps(el()); n++;
|
||||
setView('c2');
|
||||
if (els.main.innerHTML.length < 4000) throw new Error('invariants view rendered ' + els.main.innerHTML.length + ' chars');
|
||||
for (const iv of INV) { invRollup(iv); n++; }
|
||||
for (const c of ARCH.components) { renderSide(c.id); n++; }
|
||||
|
||||
console.log('viewer: ' + n + ' render calls, ' + VIEWS.length + ' views, ' +
|
||||
CAPS.capabilities.length + ' capabilities, ' + INV.length +
|
||||
' invariants, no throw');
|
||||
`;
|
||||
|
||||
eval(js + checks);
|
||||
@@ -0,0 +1,120 @@
|
||||
%% View 1 — System topology.
|
||||
%% Runtime processes and external systems, with process boundaries drawn explicitly.
|
||||
%% mavend is the centre because the code makes it one: it is the only key holder,
|
||||
%% it owns the store, the IPC socket, the voice listener, the tick loop, eight
|
||||
%% in-process background workers and the child llama-server.
|
||||
%% Evidence: docker-compose.yml, cmd/mavend/main.go, cmd/mavend/boot.go,
|
||||
%% deploy/mavwaked.service, deploy/mavgpud.service.
|
||||
flowchart LR
|
||||
|
||||
subgraph WORKPC["workpc — systemd user units, never in docker-compose"]
|
||||
direction TB
|
||||
MAVWAKED["mavwaked<br/>process<br/>arecord · silero VAD · keyword head"]
|
||||
MAVGPUD["mavgpud<br/>process<br/>GPU supervisor"]
|
||||
LLAMA_W["llama-server<br/>model · workstation card"]
|
||||
CW2["CrisperWhisper2 turbo<br/>model · port 8081"]
|
||||
ALSA["arecord / aplay<br/>external"]
|
||||
TUNNEL["maven-voice-tunnel.service<br/>ssh · the only path in"]
|
||||
end
|
||||
|
||||
subgraph HOMESRV["homesrv — docker compose project `maven`"]
|
||||
direction TB
|
||||
|
||||
subgraph MAVEND_P["mavend — process boundary · the only key holder"]
|
||||
direction TB
|
||||
IPCSRV["IPC server<br/>unix /run/maven/mavend.sock"]
|
||||
VOICESRV["voice server<br/>TCP 0.0.0.0:9100"]
|
||||
TURN["reactive handler<br/>the turn pipeline"]
|
||||
TICK["tick loop<br/>60s"]
|
||||
WORKERS["8 background workers<br/>tick · fact-enrichment · feed · crawl<br/>voice · mcp · home · memory-eval"]
|
||||
STORE[("store<br/>sqlite, MaxOpenConns=1")]
|
||||
end
|
||||
|
||||
LLAMA_H["llama-server<br/>model · resident<br/>child process of mavend"]
|
||||
MAVSTTD["mavsttd<br/>process<br/>whisper.cpp"]
|
||||
MAVTTSD["mavttsd<br/>process<br/>piper"]
|
||||
MAVWEB["mavweb<br/>process<br/>HTTP 127.0.0.1:9201"]
|
||||
MAVPOLL["mavpoll<br/>process<br/>network_mode: host"]
|
||||
SEARX["SearXNG<br/>external"]
|
||||
KIWIX["kiwix-server<br/>external"]
|
||||
NETDATA["netdata<br/>external"]
|
||||
KUMA["uptime-kuma<br/>external"]
|
||||
end
|
||||
|
||||
subgraph OFF["built, not deployed — commented out in docker-compose.yml"]
|
||||
direction TB
|
||||
MAVCALDAV["mavcaldav<br/>process"]
|
||||
MAVMAILD["mavmaild<br/>process"]
|
||||
end
|
||||
|
||||
subgraph ECO["ecosystem network — external compose project"]
|
||||
direction TB
|
||||
NEXUS["Nexus<br/>external · identity"]
|
||||
PRAXIS["Praxis<br/>external · attention"]
|
||||
HEXIS["Hexis<br/>external · capabilities"]
|
||||
end
|
||||
|
||||
subgraph NET["internet"]
|
||||
direction TB
|
||||
TG["Telegram Bot API<br/>external · via SOCKS relay"]
|
||||
NTFY["ntfy<br/>external · DISABLED in config"]
|
||||
ZM["zenmoney<br/>external · no token mounted"]
|
||||
end
|
||||
|
||||
HA["Home Assistant<br/>external · enabled:false"]
|
||||
|
||||
%% ---- voice path
|
||||
ALSA --- MAVWAKED
|
||||
MAVWAKED -->|"PushToTalk · TCP"| TUNNEL
|
||||
TUNNEL -->|"ssh to 127.0.0.1:9110"| VOICESRV
|
||||
VOICESRV -->|"proactive Push on the same conn"| MAVWAKED
|
||||
|
||||
%% ---- module IPC
|
||||
MAVWEB -->|"3 × ipc.Client · unix"| IPCSRV
|
||||
MAVWEB -->|"POST /api/ptt · TCP mavend:9100"| VOICESRV
|
||||
MAVPOLL -->|"WriteFact · unix"| IPCSRV
|
||||
MAVCALDAV -.->|"WriteFact · unix"| IPCSRV
|
||||
MAVMAILD -.->|"IngestMail · unix"| IPCSRV
|
||||
|
||||
%% ---- worker sockets
|
||||
TURN -->|"worker · unix stt.sock"| MAVSTTD
|
||||
TURN -->|"worker · unix tts.sock"| MAVTTSD
|
||||
TURN -->|"HTTP · preferred, silent fallback"| CW2
|
||||
|
||||
%% ---- models
|
||||
MAVEND_P ---|"spawns and owns"| LLAMA_H
|
||||
MAVGPUD ---|"spawns and supervises"| LLAMA_W
|
||||
MAVGPUD ---|"spawns and supervises"| CW2
|
||||
TURN -.->|"llm.Pair · model_disabled:true"| MAVGPUD
|
||||
|
||||
%% ---- world and ecosystem
|
||||
TURN -->|"HTTP · query string only"| SEARX
|
||||
TURN -->|"HTTP"| KIWIX
|
||||
TURN -->|"HTTP · v1 contract, correlation id"| NEXUS
|
||||
TURN -->|"HTTP"| PRAXIS
|
||||
TURN -->|"HTTP"| HEXIS
|
||||
MAVWEB -->|"HTTP · read-only panel"| NEXUS
|
||||
MAVWEB -->|"HTTP · read-only panel"| PRAXIS
|
||||
MAVWEB -->|"HTTP · read-only panel"| HEXIS
|
||||
TURN -.->|"HTTP · enabled:false"| HA
|
||||
|
||||
%% ---- reaches
|
||||
TICK -->|"telegram sink"| TG
|
||||
TG -->|"getUpdates long poll"| TURN
|
||||
TICK -.->|"ntfy sink · nil, disabled"| NTFY
|
||||
|
||||
%% ---- pollers
|
||||
MAVPOLL --> NETDATA
|
||||
MAVPOLL --> KUMA
|
||||
MAVPOLL -.-> ZM
|
||||
|
||||
classDef proc fill:#1f3a5f,stroke:#7fb3ff,color:#eaf2ff
|
||||
classDef ext fill:#3d2f4f,stroke:#c39bd3,color:#f4ecf7
|
||||
classDef model fill:#4a3a1f,stroke:#e0b050,color:#fff6e0
|
||||
classDef store fill:#1f4a3a,stroke:#6ed0a8,color:#e8fff5
|
||||
classDef off fill:#3a3a3a,stroke:#888,color:#ccc,stroke-dasharray:4 3
|
||||
class MAVWAKED,MAVGPUD,MAVSTTD,MAVTTSD,MAVWEB,MAVPOLL,IPCSRV,VOICESRV,TURN,TICK,WORKERS proc
|
||||
class ALSA,SEARX,KIWIX,NETDATA,KUMA,NEXUS,PRAXIS,HEXIS,TG,TUNNEL ext
|
||||
class LLAMA_H,LLAMA_W,CW2 model
|
||||
class STORE store
|
||||
class MAVCALDAV,MAVMAILD,NTFY,ZM,HA off
|
||||
@@ -0,0 +1,160 @@
|
||||
%% View 2 — Core internals of mavend.
|
||||
%% The real path, in the order runTurn actually runs it. The sequence is NOT
|
||||
%% input → routing → intent → state → tools → response: eleven stateful
|
||||
%% pre-emptors get first refusal BEFORE routing, and a query intent then enters
|
||||
%% a second, longer arbitration of its own.
|
||||
%% Evidence: cmd/mavend/voice.go runTurn, cmd/mavend/turnroute.go,
|
||||
%% cmd/mavend/actions.go, cmd/mavend/actions_query.go, internal/router/router.go.
|
||||
flowchart TB
|
||||
|
||||
subgraph IN["input — three reaches, one pipeline"]
|
||||
A1["voice.Server<br/>HandlePushToTalk"]
|
||||
A2["daemonAPI.Chat<br/>mavweb /api/chat"]
|
||||
A3["telegram poller<br/>getUpdates"]
|
||||
STT["stt seam<br/>Remote mavsttd · CW2 · Stub"]
|
||||
end
|
||||
|
||||
A1 --> STT --> RT
|
||||
A2 --> RT
|
||||
A3 --> A2
|
||||
|
||||
RT["runTurn<br/>cmd/mavend/voice.go"]
|
||||
|
||||
RT --> D0["decision.With<br/>one arbitration record per turn"]
|
||||
D0 --> TR0["turnRoute created<br/>sync.Once, on the context"]
|
||||
|
||||
subgraph PRE["pre-route ladder — 11 rungs, order load-bearing"]
|
||||
direction TB
|
||||
P1["1 expired-clarify notice"]
|
||||
P2["2 confirm answer<br/>resolveConfirm"]
|
||||
P3["3 targeted repair"]
|
||||
P4["3b untargeted repair"]
|
||||
P5["3c command prohibition"]
|
||||
P6["4 clarify answer"]
|
||||
P7["5 quiet toggle"]
|
||||
P8["5b snooze"]
|
||||
P9["5c ack"]
|
||||
P10["5d reminder cancellation"]
|
||||
P11["5e ordinal selection"]
|
||||
P1-->P2-->P3-->P4-->P5-->P6-->P7-->P8-->P9-->P10-->P11
|
||||
end
|
||||
TR0 --> PRE
|
||||
PRE -->|"any rung claims"| OUT
|
||||
|
||||
subgraph ROUTE["step 6 — the cascade · internal/router"]
|
||||
direction TB
|
||||
CONT["continuationDecision<br/>an elliptical follow-up is answered<br/>from the previous turn, not routed"]
|
||||
S0["stage 0 grammars<br/>StageZeroGrammars · first match wins at 1.0<br/>the ONLY arm that may set SourceAnchored"]
|
||||
SH["stage 0b routing heads<br/>ONNX softmax over the label set"]
|
||||
SL["stage 1a LLM router<br/>resident model, grammar-constrained"]
|
||||
SC["stage 1 classifier<br/>nearest centroid · THE FLOOR<br/>names no destination"]
|
||||
SE["stage 2 extractor + stage 3 gate"]
|
||||
CONT -->|"not a continuation"| S0
|
||||
S0 -->|"no match"| SH
|
||||
SH -->|"declines"| SL
|
||||
SL -->|"error or unparsable"| SC
|
||||
SH --> SE
|
||||
SL --> SE
|
||||
SC --> SE
|
||||
end
|
||||
PRE -->|"nobody claimed"| ROUTE
|
||||
|
||||
ROUTE --> DLG["step 7 dialogue merge<br/>followUpMerge · rememberTurn"]
|
||||
DLG --> CLAR{"step 8<br/>dec.Clarify OR a required slot missing?"}
|
||||
CLAR -->|"yes"| ASK["askClarify<br/>park the request, ask one question"]
|
||||
ASK --> OUT
|
||||
CLAR -->|"no"| ACT
|
||||
|
||||
subgraph ACT["step 9 — actionHandlers, 7 intents"]
|
||||
direction TB
|
||||
HF["fact<br/>actions_fact.go"]
|
||||
HR["reminder<br/>actions_reminder.go"]
|
||||
HA["act<br/>actions_act.go"]
|
||||
HN["note"]
|
||||
HC["chat"]
|
||||
HS["system"]
|
||||
HQ["query → the chain"]
|
||||
end
|
||||
|
||||
subgraph QC["the query chain — a SECOND arbitration, 22 sources"]
|
||||
direction TB
|
||||
QW["queryWalk<br/>removes only guesses:true sources<br/>when the cascade named a destination"]
|
||||
Q1["his data<br/>fact-by-key · day-plan · habits · tasks<br/>attention · list · money · history · feeds<br/>home · network · calendar · weather · self<br/>embed · memory · notes"]
|
||||
QB["personal boundary<br/>the only source a stage 0 anchor may drop"]
|
||||
Q2["the world<br/>search → kiwix → web → general-knowledge"]
|
||||
QW --> Q1 --> QB --> Q2
|
||||
end
|
||||
HQ --> QC
|
||||
|
||||
HF -->|"question-shaped ⇒ re-route"| QC
|
||||
HF -->|"complaint ⇒ re-route"| HC
|
||||
|
||||
subgraph STATE["state and memory"]
|
||||
direction TB
|
||||
DB[("store · sqlite<br/>facts · reminders · notes · tools<br/>tasks · lists · nudges")]
|
||||
VEC[("memory_vectors<br/>brute-force cosine")]
|
||||
DLGS[("dialogue_sessions<br/>persisted, TTL 2m")]
|
||||
CLS["clarifyStore<br/>IN MEMORY ONLY, by design"]
|
||||
PEND["pending act / routine / hexis<br/>3 single-slot registers, one mutex"]
|
||||
SURF["surfacedItems<br/>last Praxis read-out order"]
|
||||
RING["decision.Ring<br/>bounded, in memory"]
|
||||
end
|
||||
|
||||
HF --> DB
|
||||
HF --> VEC
|
||||
HR --> DB
|
||||
HN --> DB
|
||||
HN --> VEC
|
||||
Q1 --> DB
|
||||
Q1 --> VEC
|
||||
DLG --> DLGS
|
||||
ASK --> CLS
|
||||
HA --> PEND
|
||||
QC --> SURF
|
||||
D0 --> RING
|
||||
|
||||
subgraph TOOLS["act execution"]
|
||||
direction TB
|
||||
ALLOW[("tools table<br/>only status='enabled' runs")]
|
||||
EXEC["tool.Executor<br/>+ MCP + Home Assistant callers"]
|
||||
CONF["destructive confirm turn<br/>binds capability, entity, args, requester, expiry"]
|
||||
HEX["Hexis capability<br/>entity id resolved via Nexus first"]
|
||||
end
|
||||
HA --> ALLOW --> EXEC
|
||||
HA --> CONF
|
||||
HA --> HEX
|
||||
|
||||
subgraph RESP["response generation"]
|
||||
direction TB
|
||||
REP["replier<br/>only when the handler returned \"\""]
|
||||
PHR["phraser<br/>parseResponseMood is the one parser"]
|
||||
TTS["tts seam<br/>Remote mavttsd · Stub"]
|
||||
end
|
||||
ACT --> RESP
|
||||
QC --> RESP
|
||||
REP --> PHR
|
||||
OUT["reply text<br/>+ notice + resumed question"]
|
||||
RESP --> OUT
|
||||
OUT -->|"voice path only"| TTS
|
||||
|
||||
subgraph PROACT["the other half of the process — nothing above touches it"]
|
||||
direction TB
|
||||
TICKL["tick loop · 60s<br/>13 jobs in one function"]
|
||||
GTH["loop.Gatherer<br/>one consistent snapshot"]
|
||||
RUL["loop rules + restraint gate<br/>pure"]
|
||||
DISP["delivery.Dispatcher<br/>ChannelsFor severity,presence"]
|
||||
SNK["sinks: voice · ntfy · telegram"]
|
||||
TICKL --> GTH --> RUL --> TICKL
|
||||
TICKL --> DISP --> SNK
|
||||
end
|
||||
TICKL --> DB
|
||||
SNK -->|"PushToMostRecent on the request conn"| A1
|
||||
|
||||
classDef stage fill:#1f3a5f,stroke:#7fb3ff,color:#eaf2ff
|
||||
classDef store fill:#1f4a3a,stroke:#6ed0a8,color:#e8fff5
|
||||
classDef mem fill:#4a3a1f,stroke:#e0b050,color:#fff6e0
|
||||
classDef danger fill:#4f2626,stroke:#e08080,color:#ffecec
|
||||
class S0,SH,SL,SC,SE,CONT stage
|
||||
class DB,VEC,DLGS,ALLOW store
|
||||
class CLS,PEND,SURF,RING mem
|
||||
class QB,CONF danger
|
||||
@@ -0,0 +1,78 @@
|
||||
%% View 3a — Runtime flow: a reminder request.
|
||||
%% Traced through cmd/mavend/voice.go runTurn, internal/router/stagezero.go,
|
||||
%% cmd/mavend/clarify.go, cmd/mavend/actions_reminder.go, cmd/mavend/tick.go,
|
||||
%% internal/loop/loop.go and internal/delivery/dispatcher.go.
|
||||
%% Shows the branch where the hour is missing, the parked clarify, the answer
|
||||
%% turn, the write and the eventual delivery with durable retry.
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant K as Owner
|
||||
participant W as mavwaked
|
||||
participant V as voice.Server
|
||||
participant H as reactiveHandler.runTurn
|
||||
participant PRE as pre-route ladder
|
||||
participant R as router cascade
|
||||
participant CL as clarifyStore
|
||||
participant AR as actionReminder
|
||||
participant DB as store
|
||||
participant T as tick loop
|
||||
participant D as dispatcher
|
||||
|
||||
Note over K,W: "Мэйвен, напомни позвонить маме"
|
||||
K->>W: speech
|
||||
W->>W: silero VAD + keyword head, score ≥ 0.999
|
||||
W->>V: PushToTalkReq, one utterance
|
||||
V->>H: HandlePushToTalk
|
||||
H->>H: stt seam → text
|
||||
H->>H: decision.With, turnRoute created
|
||||
H->>PRE: 11 rungs
|
||||
PRE-->>H: nobody claims
|
||||
H->>R: rt.resolve
|
||||
R->>R: stage 0 ReminderGrammar matches, Stage=0, conf 1.0
|
||||
R-->>H: IntentReminder, Slots.Text="позвонить маме", HasTime=false
|
||||
|
||||
rect rgb(70,40,40)
|
||||
Note over H,CL: BRANCH — missingFor names `time`, whatever the confidence
|
||||
H->>H: dec.Clarify false BUT len missingFor > 0 → step 8 fires
|
||||
H->>CL: Push a PendingQuestion, park the request
|
||||
H-->>V: "во сколько напомнить?"
|
||||
V-->>W: reply audio + text
|
||||
end
|
||||
|
||||
Note over K,W: "в семь вечера"
|
||||
K->>W: speech
|
||||
W->>V: PushToTalkReq
|
||||
V->>H: runTurn
|
||||
H->>PRE: rung 4, resolveClarifyAnswer
|
||||
PRE->>CL: Pop the parked question
|
||||
PRE->>R: extractor parses the hour with the SAME parsers stage 2 uses
|
||||
PRE->>AR: finishClarified → applyAction
|
||||
Note right of AR: filling in an argument never grants authority
|
||||
AR->>AR: router.ResolvedTheHour guard
|
||||
AR->>DB: CreateReminder fire_ts, payload
|
||||
DB-->>AR: id
|
||||
AR-->>H: reminderConfirm, phrased FROM THE ROW not the utterance
|
||||
H-->>V: "хорошо, напомню сегодня в 19:00."
|
||||
|
||||
Note over T,D: later — the proactive half, no shared code with the turn path
|
||||
loop every 60s
|
||||
T->>DB: Gatherer.GatherState, due reminders, collapsed by group
|
||||
T->>T: loop.RemindDecisions — reminders BYPASS the restraint gate
|
||||
alt not cached
|
||||
T->>T: phraser.PhraseReminder
|
||||
end
|
||||
T->>D: DispatchReminder
|
||||
D->>DB: BeginDeliveryAttempt BEFORE the external send
|
||||
alt a voice session is live
|
||||
D->>V: voicesink push on the request conn
|
||||
else away
|
||||
D->>D: ntfy is nil (disabled) → telegram
|
||||
end
|
||||
alt success
|
||||
D->>DB: CompleteSuccessfulReminderAttempt + fire the originals, one txn
|
||||
else failure
|
||||
D->>DB: advance the persisted bounded backoff, next_attempt_ts
|
||||
end
|
||||
end
|
||||
|
||||
Note over T,DB: Recurring is NOT on this path. reminders.cron and next_fire_ts<br/>exist since migration #2 and no spoken path writes them.
|
||||
@@ -0,0 +1,61 @@
|
||||
%% View 3b — Runtime flow: a factual / state update.
|
||||
%% Traced through cmd/mavend/actions_fact.go, cmd/mavend/ack.go,
|
||||
%% cmd/mavend/patterns.go, cmd/mavend/factenrichment.go, cmd/mavend/intake.go
|
||||
%% and internal/morning.
|
||||
%% Shows the two re-route branches this handler owns, the vector prune-and-insert,
|
||||
%% the nudge it can close, and the async entity resolution behind it.
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant K as Owner
|
||||
participant H as runTurn
|
||||
participant R as router cascade
|
||||
participant AF as actionFact
|
||||
participant API as CoreAPI · intakeAPI then storeAPI
|
||||
participant DB as facts table
|
||||
participant VEC as memory_vectors
|
||||
participant BUS as event.Bus
|
||||
participant FE as fact-enrichment worker
|
||||
participant NX as Nexus
|
||||
participant T as tick loop
|
||||
|
||||
Note over K,H: "выпил воды"
|
||||
K->>H: utterance, src=tap:voice
|
||||
H->>R: rt.resolve
|
||||
R->>R: stage 0 declines → heads → LLM router → classifier
|
||||
R-->>H: IntentFact, Slots.Key="water", Slots.Value=...
|
||||
|
||||
rect rgb(70,40,40)
|
||||
Note over AF: two guards that RE-ROUTE rather than write
|
||||
AF->>AF: router.IsQuestionShaped? → becomes actionQuery, Key cleared
|
||||
AF->>AF: router.IsTransientComplaint? → becomes actionChat, nothing stored
|
||||
end
|
||||
|
||||
AF->>AF: factConfidence — 1.0 only for a value he actually said
|
||||
AF->>API: WriteFact kind=self, source=tap:voice, Subject=Key
|
||||
API->>DB: append-only row
|
||||
API->>BUS: publish one intake envelope
|
||||
API-->>AF: factID
|
||||
|
||||
AF->>VEC: pruneFactVectors by key
|
||||
AF->>VEC: EmbedPassage(FactRecallText) then Insert "fact:<key>:<unix>"
|
||||
Note right of VEC: the FACT is embedded, not the utterance.<br/>The utterance rides along as provenance only
|
||||
|
||||
AF->>DB: RecordEvent action+object, for pattern detection
|
||||
H->>H: step 9b ackFromFact — a fact answering a live nudge closes it as `acted`, silently
|
||||
|
||||
par asynchronous, minutes later
|
||||
FE->>DB: read facts with resolution_state='pending'
|
||||
FE->>NX: Resolve(Subject)
|
||||
alt resolved
|
||||
NX-->>FE: entity_id
|
||||
FE->>DB: UPDATE entity_id, resolution_state='resolved'
|
||||
else ambiguous
|
||||
Note right of FE: candidates are NOT stored —<br/>ambiguity blocks, it does not pick
|
||||
end
|
||||
and the next tick
|
||||
T->>DB: Gatherer reads the same row
|
||||
T->>T: morning routine item `water` is now evidenced, so it will not nudge
|
||||
T->>T: detectPatterns scans events for a stable interval
|
||||
end
|
||||
|
||||
Note over DB,VEC: A wrong value is superseded, never overwritten:<br/>voids_id points at the row it cancels, and CorrectValue /<br/>VoidLatestFact drop the key's vectors so recall keeps exactly one.
|
||||
@@ -0,0 +1,70 @@
|
||||
%% View 3c — Runtime flow: a world query, tool-backed.
|
||||
%% Traced through internal/router/worldquery.go, internal/router/source.go,
|
||||
%% cmd/mavend/actions_query.go queryWalk + querySources, cmd/mavend/personalboundary.go,
|
||||
%% cmd/mavend/searchwire.go, cmd/mavend/kiwixwire.go.
|
||||
%% Shows destination anchoring, which sources are skipped and why, and the
|
||||
%% four-step fallback to the model's own weights.
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant K as Owner
|
||||
participant H as runTurn
|
||||
participant R as router cascade
|
||||
participant QC as actionQuery
|
||||
participant W as queryWalk
|
||||
participant LOC as local sources
|
||||
participant PB as personal boundary
|
||||
participant SX as SearXNG
|
||||
participant KX as kiwix-server
|
||||
participant PH as phraser / resident model
|
||||
participant REC as decision record
|
||||
|
||||
Note over K,H: "что такое TCP?"
|
||||
K->>H: utterance
|
||||
H->>R: rt.resolve
|
||||
R->>R: stage 0 — WorldQueryGrammars matches a literal definition frame
|
||||
R->>R: d.SourceAnchored = true, set HERE and nowhere else
|
||||
R-->>H: IntentQuery, Source=SourceWorld, anchored
|
||||
|
||||
H->>QC: applyAction → actionQuery
|
||||
QC->>REC: Expect the full 22-source roster
|
||||
QC->>W: queryWalk(SourceWorld, anchored=true)
|
||||
|
||||
rect rgb(70,40,40)
|
||||
Note over W: removes ONLY sources with guesses:true whose dest ≠ world
|
||||
W-->>REC: skipped: attention, list, feeds, home, network, weather, self
|
||||
W-->>REC: skipped: personal boundary — anchored, so a literal pattern may drop it
|
||||
Note right of W: a model or a softmax naming SourceWorld<br/>would NOT drop the boundary (V-666)
|
||||
end
|
||||
|
||||
W-->>QC: the sources that LOOK still walk, in table order
|
||||
|
||||
loop first source to claim answers the turn
|
||||
QC->>LOC: fact-by-key, day-plan, habits, tasks, money, history, calendar
|
||||
LOC-->>QC: no rows → pass
|
||||
QC->>LOC: embed → memory → notes (vector recall, gated by min score + margin)
|
||||
LOC-->>QC: below the gate → pass
|
||||
QC->>PB: personal boundary
|
||||
PB-->>QC: SKIPPED this turn
|
||||
QC->>SX: Search(utterance verbatim, max 4)
|
||||
alt results
|
||||
SX-->>QC: snippets
|
||||
QC->>PH: phraseSource("search", utterance, evidence)
|
||||
PH-->>QC: reply
|
||||
QC->>REC: claimed by "search", and everyone below is NeverAsked
|
||||
else empty or unreachable
|
||||
QC->>KX: ZIM search, ru then en
|
||||
alt hit
|
||||
KX-->>QC: article snippet
|
||||
QC->>PH: phraseSource("kiwix", ...)
|
||||
else miss
|
||||
QC->>QC: "web" claims only if he named a URL out loud
|
||||
QC->>PH: queryGeneral — the model answers from its own weights, LAST
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
QC-->>H: reply text
|
||||
H-->>K: spoken or written answer
|
||||
|
||||
Note over LOC,SX: What leaves the box is the query string and nothing else.<br/>His notes, his facts, the persona block and the history never travel.
|
||||
Note over W,PB: With no destination named — the classifier arm sets none —<br/>the whole chain walks in table order. That is the floor.
|
||||
@@ -0,0 +1,115 @@
|
||||
%% View 4 — State ownership.
|
||||
%% Every persistent and shared store, its authoritative owner, its writers and
|
||||
%% readers, its synchronisation boundary and its lifecycle.
|
||||
%% Red = written by components that do not know about each other.
|
||||
%% Evidence: internal/store/schema.sql, internal/store/migrations.go,
|
||||
%% internal/store/crypt.go, cmd/mavend/voice.go, cmd/mavend/tick.go,
|
||||
%% cmd/mavweb/*.go, cmd/mavpoll/main.go, cmd/mavcaldav/main.go.
|
||||
flowchart LR
|
||||
|
||||
subgraph OWNER["authoritative owner — mavend, the only key holder"]
|
||||
STORE[("store.Store<br/>SetMaxOpenConns(1)<br/>every write serialised at the db")]
|
||||
end
|
||||
|
||||
subgraph LIFE["lifecycle of the database itself"]
|
||||
direction TB
|
||||
ENC[("maven.db.enc<br/>AES-256-GCM at rest<br/>volume dbdata")]
|
||||
TMP[("/dev/shm/maven-plain.db<br/>tmpfs working copy<br/>dies with the container")]
|
||||
ENC -->|"Open: decrypt"| TMP
|
||||
TMP -->|"Close: checkpoint, re-encrypt, atomic rename"| ENC
|
||||
SEAL["mavseal<br/>recovery only, VACUUM INTO"]
|
||||
TMP -.->|"when mavend was killed, not stopped"| SEAL
|
||||
SEAL -.-> ENC
|
||||
end
|
||||
STORE --- TMP
|
||||
|
||||
%% ------------- multiply written tables
|
||||
FACTS[("facts<br/>append-only, ts = valid-time<br/>correction sets voids_id")]:::multi
|
||||
NOTES[("notes<br/>float32 blob, brute-force scan")]:::multi
|
||||
TOOLS[("tools<br/>only status='enabled' executes")]:::multi
|
||||
|
||||
%% ------------- singly owned tables
|
||||
REM[("reminders")]
|
||||
NUD[("nudges — the restraint memory<br/>AND the only feedback input")]
|
||||
VEC[("memory_vectors<br/>marked with the embedder id")]
|
||||
PRES[("presence_state — singleton row")]
|
||||
EV[("events")]
|
||||
PROP[("proposed_routines")]
|
||||
DIG[("digest_entries — gate-BLOCKED candidates")]
|
||||
DEL[("delivery_attempts — the outbox")]
|
||||
ACK[("ack_sends")]
|
||||
DLGS[("dialogue_sessions — TTL 2m")]
|
||||
TASKS[("tasks")]
|
||||
LISTS[("list_items")]
|
||||
RTR[("routing_traces — 14-day retention")]
|
||||
RLB[("routing_labels")]
|
||||
ETR[("ecosystem_traces")]
|
||||
META[("meta — schema version + embedder marker")]
|
||||
|
||||
STORE --- FACTS & NOTES & TOOLS & REM & NUD & VEC & PRES & EV & PROP & DIG & DEL & ACK & DLGS & TASKS & LISTS & RTR & RLB & ETR & META
|
||||
|
||||
%% ------------- writers into facts
|
||||
WF1["actionFact — tap:voice / tap:text"] --> FACTS
|
||||
WF2["quiet toggle — config fact"] --> FACTS
|
||||
WF3["mavpoll — poll:netdata, poll:uptimekuma,<br/>infer:wg, poll:zenmoney"] --> FACTS
|
||||
WF4["mavcaldav — poll:caldav<br/>NOT DEPLOYED"]:::off -.-> FACTS
|
||||
WF5["mavweb — /api/signal presence,<br/>/api/ambient meeting time"] --> FACTS
|
||||
WF6["feed + crawl watermarks<br/>crawl:hash:*"] --> FACTS
|
||||
WF7["fact-enrichment worker<br/>entity_id, resolution_state"] --> FACTS
|
||||
WF8["tick loop tune()<br/>cooldown:<rule> feedback fact"] --> FACTS
|
||||
WF9["mavweb /api/revert<br/>voids the latest fact for a key"] --> FACTS
|
||||
|
||||
%% ------------- writers into notes
|
||||
WN1["actionNote"] --> NOTES
|
||||
WN2["RSS poller — source rss:*"] --> NOTES
|
||||
WN3["crawl watcher"] --> NOTES
|
||||
WN4["meeting capture"]:::off -.-> NOTES
|
||||
WN5["image description"]:::off -.-> NOTES
|
||||
WN6["netscan record"] --> NOTES
|
||||
|
||||
%% ------------- writers into tools
|
||||
WT1["seedTools from mavend.json"] --> TOOLS
|
||||
WT2["MCP discovery — proposed"]:::off -.-> TOOLS
|
||||
WT3["Home Assistant discovery<br/>proposed, always destructive"]:::off -.-> TOOLS
|
||||
WT4["mavweb POST /tools<br/>the ONLY enable path"] --> TOOLS
|
||||
|
||||
%% ------------- readers
|
||||
FACTS --> RD1["loop.Gatherer — the tick snapshot"]
|
||||
FACTS --> RD2["queryFactByKey · money · history · morning"]
|
||||
NOTES --> RD3["queryNotes · queryFeeds · recall"]
|
||||
VEC --> RD4["queryMemory · queryEmbed"]
|
||||
TOOLS --> RD5["tool.Matcher + tool.Executor"]
|
||||
NUD --> RD6["restraint gate · TuneCooldown · UnackedTelegramRules"]
|
||||
|
||||
%% ------------- in-memory shared state
|
||||
subgraph MEM["shared mutable state — process-local, no synchronisation boundary beyond a mutex"]
|
||||
direction TB
|
||||
CLS["clarifyStore<br/>per-reach stack · NOT persisted on purpose:<br/>a restart expires the open question"]
|
||||
PEND["pending act / pendingRoutine / pendingHexis<br/>3 single-slot registers under handler.mu<br/>last-asked wins, TTL each"]
|
||||
SURF["surfacedItems<br/>replaced by the next digest, NO TTL"]
|
||||
RING["decision.Ring — bounded, diagnosis only"]
|
||||
BUS["event.Bus — bounded journal, read surface only"]
|
||||
TICKM["tickLoop: lastPhrase, lastTrace, digestQ,<br/>routineLast, morningLast, lastProposalAt"]
|
||||
LASTR["lastRouted — the previous acted turn, for a spoken correction"]
|
||||
end
|
||||
|
||||
H1["reactiveHandler<br/>one instance, called from per-conn goroutines"] --- CLS
|
||||
H1 --- PEND
|
||||
H1 --- SURF
|
||||
H1 --- RING
|
||||
H1 --- LASTR
|
||||
TICKL["tickLoop"] --- TICKM
|
||||
INTAKE["intakeAPI decorator"] --- BUS
|
||||
|
||||
%% ------------- outside the database
|
||||
subgraph OUT["state outside the database"]
|
||||
direction TB
|
||||
PK[("passkeys.json<br/>OWNED BY mavweb, not mavend")]:::multi
|
||||
WK[("wrapped key blob<br/>written by mavend WrapKeyFn,<br/>triggered by mavweb")]:::multi
|
||||
MAIL[("mavmaild seen-UID file<br/>own volume · NOT DEPLOYED")]:::off
|
||||
BLOB[("media blobs · retention loop")]:::off
|
||||
end
|
||||
PK -.->|"a v1 blob + this file together<br/>recover the database key with no authenticator"| WK
|
||||
|
||||
classDef multi fill:#4f2626,stroke:#e08080,color:#ffecec
|
||||
classDef off fill:#3a3a3a,stroke:#888,color:#ccc,stroke-dasharray:4 3
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user