Compare commits
16 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| ecb8ba72eb | |||
| 4fdecf9a25 | |||
| beb093aebb | |||
| b1b326018f | |||
| 08889cad88 | |||
| a4630b9314 | |||
| 39d44bb384 | |||
| 65ee0f9c61 | |||
| 76938e206d | |||
| 0b3d81ecbf | |||
| 4be6852b94 | |||
| f7b76c572f | |||
| 05ddc5c92e | |||
| b55e68f98d | |||
| beaa24754c | |||
| aed8cac439 |
@@ -58,16 +58,27 @@ make build-web # single daemon (pure-Go ones: web/waked/poll/caldav build w
|
||||
make test # go test -race across ./internal/... ./cmd/... with CGO env set
|
||||
```
|
||||
|
||||
Run a single test (must carry the CGO env for packages that touch STT/TTS/voice):
|
||||
Run one package or one test with `make t`. **Do not hand-write the CGO preamble.**
|
||||
Past sessions pasted it about 390 times. That is where the shell-quoting failures
|
||||
came from. This box runs zsh, so an unquoted `-run Test*` or `--include=*.go`
|
||||
dies on "no matches found" before `go` is ever reached.
|
||||
|
||||
```sh
|
||||
CGO_CFLAGS="-I$(pwd)/deps/include -I$(pwd)/deps/whisper.cpp/ggml/include" \
|
||||
CGO_LDFLAGS="-L$(pwd)/deps/lib -Wl,-rpath,$(pwd)/deps/lib" \
|
||||
LD_LIBRARY_PATH="$(pwd)/deps/lib" \
|
||||
deps/go/go/bin/go test -run TestName ./internal/router/
|
||||
make t PKG=./internal/router/
|
||||
make t PKG=./cmd/mavend/ RUN=TestSimulator
|
||||
make t PKG=./internal/router/eval/ RUN='TestONNX' V=1 # V=1 for -v, RACE=0 to drop -race
|
||||
```
|
||||
|
||||
Pure-Go packages (`router`, `memory`, `mavweb`, …) run under a plain `go test ./pkg/`.
|
||||
`t` carries `-race`, so a green `make t` cannot turn red under `make test`. It carries
|
||||
`-count=1`, so a cached PASS from before your edit is never mistaken for a result.
|
||||
|
||||
It also sets `MAVEN_ONNX_LIB`, which the hand-written recipe did not. The four
|
||||
`TestONNX*` measurements self-skip when that variable is unset. The run still prints
|
||||
`ok`. So every targeted eval done the old way reported the hash ratchet while reading
|
||||
as a real embedder score.
|
||||
|
||||
Pure-Go packages (`router`, `memory`, `mavweb`, …) also run under a plain `go test ./pkg/`,
|
||||
but `make t` works everywhere and is one thing to remember.
|
||||
|
||||
## The daemons (`cmd/`)
|
||||
|
||||
@@ -101,9 +112,15 @@ protocol; the config in `deploy/mavend.json` (with `${VAR}` env expansion from g
|
||||
Count against compose, not against the table. Four of the nine daemons are absent, and each
|
||||
absence has a different reason.
|
||||
|
||||
`mavmaild` is commented out in compose, with the reason written beside it: it needs a mail
|
||||
account and this box has none. `mavcaldav` appears nowhere at all, and unlike the other
|
||||
three that is an oversight rather than a decision (V-644).
|
||||
`mavmaild` and `mavcaldav` are commented out in compose, each with the reason written
|
||||
beside it: the first needs a mail account, the second a CalDAV account, and this box has
|
||||
neither. `mavcaldav` used to appear nowhere at all, which was an oversight; it became a
|
||||
recorded decision on 07-08-2026 (V-644). Two things ride on that absence and the block
|
||||
names them. Agenda questions route to `IntentQuery` at stage 0 (V-498) and the `calendar`
|
||||
query source then reads a table nobody writes. And `loop.State.CalendarBusy` is fed by the
|
||||
same facts, so the gate's "do not nag mid-meeting" is permanently false. Its password is
|
||||
read from a file (`-pass-file`, and `-render-pass-file` for the render collection), never
|
||||
taken as a flag value, which is the rule `mavpoll` and `mavmaild` follow too.
|
||||
|
||||
**`mavwaked` and `mavenclient` are absent by decision, not oversight** (Vikunja #463,
|
||||
`docs/plans/17-where-the-voice-loop-runs.md`).
|
||||
@@ -454,8 +471,11 @@ start of a session rather than one lookup per first use:
|
||||
ToolSearch("select:mcp__vikunja__list_tasks,mcp__vikunja__get_task_details,mcp__vikunja__create_task,mcp__vikunja__update_task")
|
||||
```
|
||||
|
||||
`update_task` carrying a `description` resets `done` to false, so closing a task with a
|
||||
write-up takes two calls: the description, then `done: true`.
|
||||
**Close a finished task with `done: true` and nothing else** (owner's call, 07-08-2026).
|
||||
Do not write a completion summary into the description on the way out. It is lost anyway,
|
||||
and the durable record is the commit messages and the merged PR. Note that `update_task`
|
||||
carrying a `description` resets `done` to false, which is why a write-up ever took two
|
||||
calls.
|
||||
|
||||
## Session workflow
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ PIPER_BIN := $(shell pwd)/deps/piper/piper
|
||||
PIPER_MODEL := $(shell pwd)/models/tts/ru_RU-irina-medium.onnx
|
||||
PIPER_ESPEAK := $(shell pwd)/deps/piper/espeak-ng-data
|
||||
|
||||
.PHONY: simulate stt-fixtures test-stt-golden all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go deps-sentinel tidy eval-router eval-reach eval-recall eval-phrasing eval-models build-gpud
|
||||
.PHONY: t audit simulate stt-fixtures test-stt-golden all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go deps-sentinel tidy eval-router eval-reach eval-recall eval-phrasing eval-models build-gpud
|
||||
|
||||
all: build
|
||||
|
||||
@@ -128,6 +128,35 @@ test: fmt-check vet
|
||||
CGO_CFLAGS="$(CGO_CFLAGS)" CGO_LDFLAGS="$(CGO_LDFLAGS)" LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
|
||||
$(GO) test -race -coverprofile=coverage.out ./internal/... ./cmd/...
|
||||
|
||||
# t — run ONE package or ONE test with the toolchain env already wired. This is
|
||||
# the iteration target; `test` is the gate. Reach for it instead of pasting the
|
||||
# CGO_CFLAGS/CGO_LDFLAGS/LD_LIBRARY_PATH preamble by hand, which is how it was
|
||||
# done ~390 times across past sessions and is where the shell-quoting failures
|
||||
# came from -- the interactive shell here is zsh, and an unquoted `-run Test*`
|
||||
# or `--include=*.go` dies on "no matches found" before go ever starts.
|
||||
#
|
||||
# make t # whole tree (same scope as `test`)
|
||||
# make t PKG=./internal/router/
|
||||
# make t PKG=./cmd/mavend/ RUN=TestSimulator
|
||||
# make t PKG=./internal/router/eval/ RUN='TestONNX' V=1
|
||||
# make t PKG=./internal/store/ RACE=0 # drop -race when iterating hot
|
||||
#
|
||||
# -race is on by default so a green `make t` cannot turn red under `make test`.
|
||||
# -count=1 because a cached PASS from before your edit is worse than no answer.
|
||||
# MAVEN_ONNX_LIB is set for the same reason: the four TestONNX* measurements
|
||||
# self-skip when it is unset, so a targeted eval run would otherwise report the
|
||||
# deterministic hash ratchet and look like it scored the real embedder.
|
||||
PKG ?= ./internal/... ./cmd/...
|
||||
RUN ?=
|
||||
V ?=
|
||||
RACE ?= 1
|
||||
|
||||
t:
|
||||
CGO_CFLAGS="$(CGO_CFLAGS)" CGO_LDFLAGS="$(CGO_LDFLAGS)" LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
|
||||
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" \
|
||||
$(GO) test $(if $(V),-v,) $(if $(filter-out 0,$(RACE)),-race,) -count=1 \
|
||||
$(if $(RUN),-run '$(RUN)',) $(PKG)
|
||||
|
||||
# eval-router — score the held-out RU routing fixture (internal/router/eval).
|
||||
# Verbose so the report tables land in the terminal. MAVEN_ONNX_LIB points the
|
||||
# prod-representative baseline at the vendored runtime; override it or set it
|
||||
@@ -189,6 +218,16 @@ eval-models:
|
||||
# scores the fixtures against ggml-small and self-skips when the model is
|
||||
# absent, and TestGoldenFixturesAreCanonical, which checks the committed audio
|
||||
# and the manifest with no model at all.
|
||||
# audit — the repo inventory: LOC per package, open TODOs, real stubs, living-doc
|
||||
# staleness, test shape, packages with no test. Read-only, prints, writes nothing.
|
||||
# Run it instead of rebuilding the same greps by hand; past sessions spent 93 of
|
||||
# them on this before their first edit. SECTION=loc|todo|stubs|docs|tests|gaps
|
||||
# narrows it.
|
||||
SECTION ?= all
|
||||
|
||||
audit:
|
||||
@SECTION="$(SECTION)" ./scripts/audit.sh
|
||||
|
||||
stt-fixtures:
|
||||
./scripts/gen-stt-fixtures.sh
|
||||
|
||||
|
||||
+35
-8
@@ -50,10 +50,10 @@ func run(args []string) error {
|
||||
socket := fs.String("socket", "", "core IPC socket path (required)")
|
||||
url := fs.String("url", "", "CalDAV calendar URL, e.g. http://localhost:5232/kami/personal (required)")
|
||||
user := fs.String("user", "", "CalDAV basic-auth username (required)")
|
||||
pass := fs.String("pass", "", "CalDAV basic-auth password (required)")
|
||||
passFile := fs.String("pass-file", "", "file holding the CalDAV basic-auth password (required — never passed as a flag value)")
|
||||
renderURL := fs.String("render-url", "", "CalDAV collection maven publishes her own reminders to; empty disables rendering")
|
||||
renderUser := fs.String("render-user", "", "basic-auth username for -render-url (defaults to -user)")
|
||||
renderPass := fs.String("render-pass", "", "basic-auth password for -render-url (defaults to -pass)")
|
||||
renderPassFile := fs.String("render-pass-file", "", "file holding the password for -render-url (defaults to -pass-file)")
|
||||
renderDur := fs.Duration("render-duration", calendar.DefaultReminderDuration, "how long a rendered reminder occupies")
|
||||
interval := fs.Duration("interval", 5*time.Minute, "poll cadence")
|
||||
timeout := fs.Duration("timeout", 10*time.Second, "per-request HTTP timeout")
|
||||
@@ -63,13 +63,22 @@ func run(args []string) error {
|
||||
if *socket == "" {
|
||||
return fmt.Errorf("-socket is required")
|
||||
}
|
||||
if *url == "" || *user == "" || *pass == "" {
|
||||
return fmt.Errorf("-url, -user, -pass are required")
|
||||
if *url == "" || *user == "" || *passFile == "" {
|
||||
return fmt.Errorf("-url, -user, -pass-file are required")
|
||||
}
|
||||
if err := checkRenderTarget([]string{*url}, *renderURL); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// The password is read from a file, never taken as a flag value: an argv
|
||||
// secret is visible in `ps` to every user on the box and lands in the compose
|
||||
// file and the shell history. Same rule mavmaild and mavpoll follow. Read
|
||||
// once at start, so a rotated password means a restart.
|
||||
pass, err := readSecret(*passFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
@@ -85,17 +94,20 @@ func run(args []string) error {
|
||||
http: hc,
|
||||
url: strings.TrimRight(*url, "/"),
|
||||
user: *user,
|
||||
pass: *pass,
|
||||
pass: pass,
|
||||
}
|
||||
|
||||
var rend *renderer
|
||||
if *renderURL != "" {
|
||||
ru, rp := *renderUser, *renderPass
|
||||
ru, rp := *renderUser, pass
|
||||
if ru == "" {
|
||||
ru = *user
|
||||
}
|
||||
if rp == "" {
|
||||
rp = *pass
|
||||
if *renderPassFile != "" {
|
||||
rp, err = readSecret(*renderPassFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
rend = newRenderer(core, hc, *renderURL, ru, rp, *renderDur)
|
||||
log.Printf("mavcaldav: rendering reminders to %s", *renderURL)
|
||||
@@ -131,6 +143,21 @@ func run(args []string) error {
|
||||
// It takes the whole read set, not one URL. The guarantee in the package
|
||||
// comment is about every calendar maven reads, and a second read target added
|
||||
// later must not quietly fall outside the check.
|
||||
// readSecret reads one credential from a file and refuses an empty one. An
|
||||
// empty file is a deployment mistake, not a password, and CalDAV basic auth
|
||||
// would send it and get a 401 every poll.
|
||||
func readSecret(path string) (string, error) {
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read password file: %w", err)
|
||||
}
|
||||
secret := strings.TrimSpace(string(raw))
|
||||
if secret == "" {
|
||||
return "", fmt.Errorf("password file %s is empty", path)
|
||||
}
|
||||
return secret, nil
|
||||
}
|
||||
|
||||
func checkRenderTarget(readURLs []string, renderURL string) error {
|
||||
if renderURL == "" {
|
||||
return nil
|
||||
|
||||
@@ -5,12 +5,38 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
)
|
||||
|
||||
// The password comes from a file so it never reaches argv. An empty or missing
|
||||
// file must fail at start rather than authenticate as "" against his calendar.
|
||||
func TestReadSecret(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
good := filepath.Join(dir, "ok")
|
||||
if err := os.WriteFile(good, []byte(" hunter2\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, err := readSecret(good); err != nil || got != "hunter2" {
|
||||
t.Fatalf("readSecret(good) = %q, %v; want \"hunter2\", nil", got, err)
|
||||
}
|
||||
|
||||
empty := filepath.Join(dir, "empty")
|
||||
if err := os.WriteFile(empty, []byte("\n \n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := readSecret(empty); err == nil {
|
||||
t.Fatal("readSecret(empty) = nil error, want refusal")
|
||||
}
|
||||
if _, err := readSecret(filepath.Join(dir, "absent")); err == nil {
|
||||
t.Fatal("readSecret(absent) = nil error, want refusal")
|
||||
}
|
||||
}
|
||||
|
||||
type fakeCore struct {
|
||||
ipc.UnimplementedCoreAPI
|
||||
facts map[string]ipc.Fact // composite key "key|source" → Fact
|
||||
|
||||
+42
-22
@@ -6,7 +6,6 @@ import (
|
||||
"math/rand"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/kami/maven/internal/dialogue"
|
||||
@@ -162,12 +161,13 @@ func withNotice(notice, reply string) string {
|
||||
// напоминание?" — answer first, then the open question. A question in front of
|
||||
// its own answer would read as ignoring what he asked.
|
||||
//
|
||||
// A statement's full stop is folded into a comma, so the two acts read as one
|
||||
// sentence — that is the owner's own punctuation, "в Риме сейчас ..., на какое
|
||||
// время поставить напоминание?". An answer that is ITSELF a question keeps its
|
||||
// mark and the resume starts a new sentence: she sometimes answers a side query
|
||||
// by asking him to say it again, and "переформулировать?, на какое время" folds
|
||||
// two questions into one unreadable line.
|
||||
// Two sentences, not one (V-654). This used to fold the answer's full stop into
|
||||
// a comma, on the strength of the owner having written it that way once. Spliced
|
||||
// onto a real answer it reads as one run-on thought — "вот что я нашла: вайфай
|
||||
// пароль лежит в ящике стола, на какое время поставить напоминание?" — and the
|
||||
// question disappears into the tail of a sentence about something else. A reply
|
||||
// with no terminator of its own is given one, so the join never depends on how
|
||||
// the phraser chose to end.
|
||||
//
|
||||
// A resume with no answer in front of it is just the question.
|
||||
func withResumed(reply, resumed string) string {
|
||||
@@ -178,23 +178,17 @@ func withResumed(reply, resumed string) string {
|
||||
if reply == "" {
|
||||
return resumed
|
||||
}
|
||||
if strings.HasSuffix(reply, "?") {
|
||||
return reply + " " + resumed
|
||||
if !endsSentence(reply) {
|
||||
reply += "."
|
||||
}
|
||||
if trimmed := strings.TrimRight(reply, ".!"); trimmed != "" {
|
||||
reply = trimmed
|
||||
}
|
||||
return reply + ", " + lowerFirst(resumed)
|
||||
return reply + " " + resumed
|
||||
}
|
||||
|
||||
// lowerFirst lowercases the opening rune, so a deck line written as a standalone
|
||||
// sentence reads as the second half of one. Only the first rune: "На какое
|
||||
// время" must become "на какое время" and nothing else in it may move.
|
||||
func lowerFirst(s string) string {
|
||||
for i, r := range s {
|
||||
return string(unicode.ToLower(r)) + s[i+utf8.RuneLen(r):]
|
||||
}
|
||||
return s
|
||||
// endsSentence reports whether s already closes itself. The ellipsis counts: a
|
||||
// trailing "…" is a deliberate end, and a full stop after it reads as a typo.
|
||||
func endsSentence(s string) bool {
|
||||
r, _ := utf8.DecodeLastRuneInString(s)
|
||||
return strings.ContainsRune(".!?…", r)
|
||||
}
|
||||
|
||||
// missingFor returns the slots a decision still needs, most important first.
|
||||
@@ -381,6 +375,14 @@ func (h *reactiveHandler) resolveClarifyAnswer(ctx context.Context, text string)
|
||||
return "", false
|
||||
}
|
||||
|
||||
// He is answering, so the run of step-asides is over (V-654). Reset here
|
||||
// rather than where a gap is FILLED: "позвонить маме" against a question
|
||||
// about the time gives her nothing she asked for and still means he is in
|
||||
// the exchange, and the retry it costs is bound enough on its own. The
|
||||
// counter is for the case the bounds miss — he asked for other things and
|
||||
// never came back.
|
||||
q.Suspends = 0
|
||||
|
||||
merged := q.Answer(text, toDialogueSlots(answer))
|
||||
// Fold a newly answered subject into the raw utterance. Downstream actions
|
||||
// phrase from Utterance, not from the text slot — actionReminder stores it
|
||||
@@ -467,6 +469,14 @@ func (h *reactiveHandler) noteDropped(ctx context.Context) {
|
||||
//
|
||||
// A slot with no resumed wording (clarifyResumedFor says so) resumes nothing and
|
||||
// says nothing. She must not claim to be holding a question she cannot re-ask.
|
||||
//
|
||||
// Suspension is bounded, since V-654. Neither of the two things above is a
|
||||
// limit: no attempt is spent, and restarting the clock means the TTL cannot
|
||||
// arrive while he keeps talking. So the count is the only thing that ends it,
|
||||
// and past MaxSuspends she lets the request go and says so with the same line
|
||||
// every other drop uses. The rule is unchanged — a question ends by being
|
||||
// answered or by being let go out loud — this only recognises three unrelated
|
||||
// requests in a row as the second of those.
|
||||
func (h *reactiveHandler) noteSuspended(ctx context.Context, q *dialogue.PendingQuestion) {
|
||||
rt := turnRouteFrom(ctx)
|
||||
if rt == nil || len(q.Missing) == 0 {
|
||||
@@ -476,11 +486,18 @@ func (h *reactiveHandler) noteSuspended(ctx context.Context, q *dialogue.Pending
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !q.CanResume() {
|
||||
h.clarifyStore.Delete(dialogueIDOf(ctx))
|
||||
h.noteDropped(ctx)
|
||||
log.Printf("voice: clarify — the question about %s stepped aside %d times; letting the request go", q.Missing[0], q.Suspends)
|
||||
return
|
||||
}
|
||||
q.Suspends++
|
||||
q.Asked = h.now()
|
||||
h.clarifyStore.Put(dialogueIDOf(ctx), q)
|
||||
rt.resume = question
|
||||
rt.suspended = true
|
||||
log.Printf("voice: clarify — is its own request; suspending the question about %s and resuming it in the same reply", q.Missing[0])
|
||||
log.Printf("voice: clarify — is its own request; suspending the question about %s and resuming it in the same reply (suspend %d of %d)", q.Missing[0], q.Suspends, dialogue.MaxSuspends)
|
||||
}
|
||||
|
||||
// foldAnswerIntoUtterance appends an answered subject to the original words,
|
||||
@@ -520,6 +537,9 @@ func (h *reactiveHandler) askRemainingGap(ctx context.Context, q *dialogue.Pendi
|
||||
if !ok || !q.CanAsk() {
|
||||
return "", false
|
||||
}
|
||||
// Suspends is not carried, and by this point it is already zero: the answer
|
||||
// path resets it (V-654). Left off the literal so the zero is stated where
|
||||
// the struct is built, rather than inherited from a field nobody names.
|
||||
h.clarifyStore.Put(dialogueIDOf(ctx), &dialogue.PendingQuestion{
|
||||
Intent: q.Intent,
|
||||
Slots: merged,
|
||||
|
||||
@@ -740,3 +740,53 @@ func TestACompleteTurnStillDoesNotAsk(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheResumedQuestionIsItsOwnSentence — V-654. The re-ask used to be spliced
|
||||
// onto the answer with a comma, so a real answer and an unrelated open question
|
||||
// read as one run-on thought and the question vanished into its tail.
|
||||
func TestTheResumedQuestionIsItsOwnSentence(t *testing.T) {
|
||||
const resumed = "На какое время поставить напоминание?"
|
||||
cases := []struct {
|
||||
name string
|
||||
reply string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
// The measured line, shortened. Two sentences, and the question keeps
|
||||
// its capital.
|
||||
name: "a statement keeps its full stop",
|
||||
reply: "Вайфай пароль лежит в ящике стола.",
|
||||
want: "Вайфай пароль лежит в ящике стола. " + resumed,
|
||||
},
|
||||
{
|
||||
name: "a reply with no terminator is given one",
|
||||
reply: "Вайфай пароль лежит в ящике стола",
|
||||
want: "Вайфай пароль лежит в ящике стола. " + resumed,
|
||||
},
|
||||
{
|
||||
// She sometimes answers a side query by asking him to say it again.
|
||||
// Two questions, and neither may swallow the other.
|
||||
name: "a question keeps its mark",
|
||||
reply: "Можешь переформулировать?",
|
||||
want: "Можешь переформулировать? " + resumed,
|
||||
},
|
||||
{
|
||||
name: "an ellipsis is already an ending",
|
||||
reply: "Не уверена…",
|
||||
want: "Не уверена… " + resumed,
|
||||
},
|
||||
{
|
||||
name: "a resume with no answer in front of it is just the question",
|
||||
reply: "",
|
||||
want: resumed,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := withResumed(tc.reply, resumed); got != tc.want {
|
||||
t.Errorf("%s: withResumed(%q) = %q, want %q", tc.name, tc.reply, got, tc.want)
|
||||
}
|
||||
}
|
||||
if got := withResumed("Готово.", ""); got != "Готово." {
|
||||
t.Errorf("nothing to resume must leave the reply alone, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -279,3 +279,97 @@ func TestTheTurnIsRoutedOnce(t *testing.T) {
|
||||
t.Fatalf("the pipeline routed again and got something else: %+v vs %+v", second, first)
|
||||
}
|
||||
}
|
||||
|
||||
// TestASuspendedQuestionDoesNotRideForever — V-654, the measured failure of
|
||||
// 2026-08-07 (docs/evals/2026-08-07-week-of-usage-transcript.md, t=51 to t=58).
|
||||
//
|
||||
// A side query suspends the parked question, spends no attempt and restarts the
|
||||
// TTL. Nothing else bounded it, so one unfilled time slot came back on the end
|
||||
// of six consecutive unrelated replies and stopped only when a seventh turn
|
||||
// happened to read as a failed answer. Three step-asides, then she lets it go
|
||||
// and says so.
|
||||
func TestASuspendedQuestionDoesNotRideForever(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, st := newRoutingClarifyHandler(t)
|
||||
id := dialogueIDFor(sourceText, "web")
|
||||
resumed, _ := clarifyResumedFor(dialogue.SlotTime)
|
||||
|
||||
if reply := h.handleText(ctx, "web", "напомни позвонить маме"); !strings.Contains(reply, "?") {
|
||||
t.Fatalf("expected the time question, got %q", reply)
|
||||
}
|
||||
|
||||
// Three questions of his own. Each one is answered as itself and each one
|
||||
// brings the open question back, exactly as V-561 asks.
|
||||
asides := []string{
|
||||
"о чём мы вчера говорили?",
|
||||
"какие у меня напоминания?",
|
||||
"сколько времени?",
|
||||
}
|
||||
for i, text := range asides {
|
||||
reply := h.handleText(ctx, "web", text)
|
||||
if !strings.HasSuffix(reply, resumed) {
|
||||
t.Fatalf("side query %d: the question must come back, got %q", i+1, reply)
|
||||
}
|
||||
if strings.Contains(reply, clarifyDropped) {
|
||||
t.Fatalf("side query %d: nothing was let go yet, so nothing may say so: %q", i+1, reply)
|
||||
}
|
||||
q := h.clarifyStore.Get(id, h.now())
|
||||
if q == nil {
|
||||
t.Fatalf("side query %d: the question was dropped early", i+1)
|
||||
}
|
||||
if q.Attempts != 1 {
|
||||
t.Fatalf("side query %d: a step-aside spent an attempt: %d", i+1, q.Attempts)
|
||||
}
|
||||
if q.Suspends != i+1 {
|
||||
t.Fatalf("side query %d: suspends = %d, want %d", i+1, q.Suspends, i+1)
|
||||
}
|
||||
}
|
||||
|
||||
// The fourth. She has stepped aside as often as she is willing to, so the
|
||||
// request goes — out loud, and without the question on the tail.
|
||||
reply := h.handleText(ctx, "web", "что у меня сегодня?")
|
||||
if !strings.Contains(reply, clarifyDropped) {
|
||||
t.Fatalf("the request was let go in silence: %q", reply)
|
||||
}
|
||||
if strings.HasSuffix(reply, resumed) {
|
||||
t.Fatalf("a question she has let go must not be asked again: %q", reply)
|
||||
}
|
||||
if h.clarifyStore.Get(id, h.now()) != nil {
|
||||
t.Fatal("the question must be gone once she has said she let it go")
|
||||
}
|
||||
if reminders, err := st.DueReminders(ctx, h.now().Add(48*time.Hour)); err != nil || len(reminders) != 0 {
|
||||
t.Fatalf("a reminder was invented for a time nobody gave: %v err=%v", reminders, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAnAnsweredGapResetsTheSuspendBudget — the counter measures CONSECUTIVE
|
||||
// step-asides. He filled a gap, so the run is broken and the next question
|
||||
// starts with its full allowance: a long exchange he is engaged with must not
|
||||
// run out of patience on his behalf.
|
||||
func TestAnAnsweredGapResetsTheSuspendBudget(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, _ := newRoutingClarifyHandler(t)
|
||||
id := dialogueIDFor(sourceText, "web")
|
||||
|
||||
// A bare "напомни" is missing both halves, so answering the subject re-parks
|
||||
// the request with a question about the time.
|
||||
if reply := h.handleText(ctx, "web", "напомни"); !strings.Contains(reply, "?") {
|
||||
t.Fatalf("expected a question, got %q", reply)
|
||||
}
|
||||
if reply := h.handleText(ctx, "web", "какие у меня напоминания?"); reply == "" {
|
||||
t.Fatal("the side query must be answered as itself")
|
||||
}
|
||||
if q := h.clarifyStore.Get(id, h.now()); q == nil || q.Suspends != 1 {
|
||||
t.Fatalf("the side query was not counted: %+v", q)
|
||||
}
|
||||
if reply := h.handleText(ctx, "web", "позвонить маме"); reply == "" {
|
||||
t.Fatal("the answer must be consumed")
|
||||
}
|
||||
q := h.clarifyStore.Get(id, h.now())
|
||||
if q == nil {
|
||||
t.Fatal("a reminder still needs its time, so a question must be parked")
|
||||
}
|
||||
if q.Suspends != 0 {
|
||||
t.Fatalf("answering a gap must reset the suspend budget: suspends = %d", q.Suspends)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,6 +25,25 @@
|
||||
"llm_nudges": false
|
||||
},
|
||||
|
||||
"//ntfy": [
|
||||
"The second reach (V-649). Until 07-08-2026 telegram was the only one, and",
|
||||
"telegram needs api.telegram.org, the socks relay below and a matching ufw",
|
||||
"rule — three things in series that have each failed once, and when they do",
|
||||
"a sev4 nudge has nowhere to go. ntfy shares none of them: it is reached",
|
||||
"directly, no relay.",
|
||||
"It is not only a spare. The routing table sends sev3-away and away",
|
||||
"reminders here and NOWHERE else, so with this block absent those two",
|
||||
"routes hit a nil sink and vanish without a log or an outbox row.",
|
||||
"The credential is an ntfy access token, scoped write-only to this one",
|
||||
"topic, so a popped sink can push to it and cannot read it back. Set it in",
|
||||
"deploy/telegram.env beside the telegram secrets; that file is gitignored."
|
||||
],
|
||||
"ntfy": {
|
||||
"base_url": "https://ntfy.kvmx.ru",
|
||||
"topic": "maven",
|
||||
"token": "${NTFY_TOKEN}"
|
||||
},
|
||||
|
||||
"telegram": {
|
||||
"bot_token": "${TELEGRAM_BOT_TOKEN}",
|
||||
"chat_id": "${TELEGRAM_CHAT_ID}",
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# Telegram bot token and chat ID for mavend's away-channel reach.
|
||||
# Secrets for mavend's away-channel reaches. The file is still called
|
||||
# telegram.env because compose names it that; it holds both reaches now.
|
||||
# Copy this file to deploy/telegram.env and fill in real values.
|
||||
# deploy/telegram.env is gitignored — never commit the real secrets.
|
||||
TELEGRAM_BOT_TOKEN=
|
||||
TELEGRAM_CHAT_ID=
|
||||
|
||||
# ntfy access token for the `maven` topic, the second reach (V-649). Mint it on
|
||||
# the ntfy server with write access to that topic and nothing else:
|
||||
# ntfy token add --expires=never maven
|
||||
# Read access is not needed — mavend publishes and never subscribes.
|
||||
NTFY_TOKEN=
|
||||
|
||||
@@ -157,6 +157,44 @@ services:
|
||||
# - maildata:/var/lib/mavmaild
|
||||
# - ./deploy/imap.password:/run/secrets/imap.password:ro
|
||||
|
||||
# The calendar reader (Vikunja #644) is OFF and commented out: it needs a
|
||||
# CalDAV account, and there is none on this box. It was built, listed in
|
||||
# `make build`, and deployed nowhere, which is the worst of the three states —
|
||||
# this block records the decision instead.
|
||||
#
|
||||
# What its absence costs, so the cost is visible from here:
|
||||
# - Agenda questions route correctly and answer from nothing. Stage 0 sends
|
||||
# "что у меня сегодня" to IntentQuery (V-498) and the `calendar` query
|
||||
# source reads facts(kind=env, source=caldav:*) that nobody writes.
|
||||
# - The nudge gate loses a suppressor. loop.State.CalendarBusy is fed by
|
||||
# those same facts, so "do not nag mid-meeting" is permanently false.
|
||||
#
|
||||
# Core never sees the CalDAV password: the reader polls the collection itself
|
||||
# and hands core one fact per event over WriteFact. Nothing here can create a
|
||||
# reminder, so a misread event cannot fire.
|
||||
#
|
||||
# The password is read from a FILE, so it never appears in `ps`, in this file,
|
||||
# or in shell history — the same rule mavpoll and mavmaild follow.
|
||||
#
|
||||
# To enable: write the password to deploy/caldav.password (0600, gitignored),
|
||||
# point -url at the collection, and uncomment this service. No mavend.json
|
||||
# block is needed — events arrive over IPC as facts. -render-url is optional
|
||||
# and OFF here: it publishes Maven's own reminders back as events, and it must
|
||||
# not name the collection -url reads, or the poller reads its own writes back
|
||||
# in (checkRenderTarget refuses that). It takes -render-pass-file, and falls
|
||||
# back to this password when that is not given.
|
||||
# mavcaldav:
|
||||
# <<: *image
|
||||
# command: ["mavcaldav", "-socket", "/run/maven/mavend.sock",
|
||||
# "-url", "http://localhost:5232/kami/personal",
|
||||
# "-user", "kami",
|
||||
# "-pass-file", "/run/secrets/caldav.password",
|
||||
# "-interval", "5m"]
|
||||
# depends_on: [mavend]
|
||||
# volumes:
|
||||
# - sockets:/run/maven
|
||||
# - ./deploy/caldav.password:/run/secrets/caldav.password:ro
|
||||
|
||||
volumes:
|
||||
dbdata:
|
||||
sockets:
|
||||
|
||||
+34
-1
@@ -1,6 +1,6 @@
|
||||
# Maven — Design
|
||||
|
||||
*Last verified: 2026-08-02 @ 7079a24. Living doc: correct it in place, do not append.*
|
||||
*Last verified: 2026-08-07 @ beb093a. Living doc: correct it in place, do not append.*
|
||||
|
||||
> Folded 2026-07-30 from `SPEC.md` (north star, 2026-07-03), `maven.md`
|
||||
> (consolidated decisions, 2026-06-30) and `ROADMAP.md` (execution plan,
|
||||
@@ -282,6 +282,39 @@ Three reasons, in the order they settle it:
|
||||
So the notice stays what it is: the in-process TTL case, where she really did
|
||||
wait and really did let go.
|
||||
|
||||
#### A parked question may step aside three times
|
||||
|
||||
Decided 2026-08-07 (V-654). A side query or an aside suspends the parked
|
||||
question instead of dropping it. The words are answered as themselves, and the
|
||||
question comes back on the end of the same reply.
|
||||
|
||||
Neither bound on a question reaches that path. No attempt is spent, because a
|
||||
side query is not a failed answer, so `MaxAttempts` never applies.
|
||||
`noteSuspended` also restarts the 90s clock, since she is about to speak the
|
||||
question again. So the TTL cannot arrive while he keeps talking.
|
||||
|
||||
Measured on 2026-08-07: one unfilled time slot rode the tail of six consecutive
|
||||
unrelated replies. It stopped only when a seventh turn happened to read as a
|
||||
failed answer. See `docs/evals/2026-08-07-week-of-usage.md`.
|
||||
|
||||
`PendingQuestion.Suspends` counts the step-asides. `MaxSuspends` is 3, matching
|
||||
`DefaultMaxAttempts`. Past it she lets the request go, with the same
|
||||
`clarifyDropped` line every other drop uses. The owner's rule is unchanged. A
|
||||
question still ends by being answered or by being let go out loud. This only
|
||||
recognises three unrelated requests in a row as the second of those.
|
||||
|
||||
The count is of CONSECUTIVE step-asides. It resets the moment he answers, in
|
||||
`resolveClarifyAnswer`. An answer that gives her nothing she asked for resets it
|
||||
too. "Позвонить маме" against a question about the time is still him in the
|
||||
exchange. The retry it costs is bound enough on its own.
|
||||
|
||||
The re-ask is also two sentences rather than one. It used to be spliced onto the
|
||||
answer with a comma. On a real answer that buries the question in the tail of
|
||||
one run-on thought:
|
||||
|
||||
> вот что я нашла: вайфай пароль лежит в ящике стола, на какое время поставить
|
||||
> напоминание?
|
||||
|
||||
### save-where — the two-memory routing axis
|
||||
|
||||
One discriminator: **does the loop evaluate a predicate against it?**
|
||||
|
||||
+12
-2
@@ -1,6 +1,6 @@
|
||||
# Start Commands
|
||||
|
||||
*Last verified: 2026-08-02 @ 7079a24. Living doc: correct it in place, do not append.*
|
||||
*Last verified: 2026-08-07 @ a4630b9. Living doc: correct it in place, do not append.*
|
||||
|
||||
All commands assume `ROOT=/home/kami/apps/Maven` and the local Go toolchain at `$ROOT/deps/go/go/bin/go`.
|
||||
|
||||
@@ -44,7 +44,8 @@ Config path: `~/.config/maven/mavend.json`. Full example with all options.
|
||||
"repeat_interval": "5m",
|
||||
"ntfy": {
|
||||
"base_url": "https://ntfy.kvmx.ru",
|
||||
"topic": "maven"
|
||||
"topic": "maven",
|
||||
"token": "${NTFY_TOKEN}"
|
||||
},
|
||||
"phraser": {
|
||||
"model_path": "/mnt/hdd1/llms/Qwen3-Maven-1.7B-Q8_0.gguf",
|
||||
@@ -66,6 +67,15 @@ Config path: `~/.config/maven/mavend.json`. Full example with all options.
|
||||
|
||||
Omit the `embedder` block entirely to use the deterministic HashEmbedder floor (no ML, no ONNX runtime dependency). Useful for testing or low-resource setups.
|
||||
|
||||
`${NTFY_TOKEN}` and the `${TELEGRAM_*}` vars are expanded from `deploy/telegram.env`, which is gitignored. Copy `deploy/telegram.env.example` and fill it in. Mint a scoped token rather than reusing an admin one. It needs write access to the `maven` topic and nothing else:
|
||||
|
||||
```sh
|
||||
ntfy access maven maven write-only
|
||||
ntfy token add --expires=never maven
|
||||
```
|
||||
|
||||
Deleting the `ntfy` block turns the reach off, and that is not a no-op. The routing table sends sev3-away nudges and away reminders to ntfy and nowhere else. With no sink wired they hit a nil and vanish, leaving no log line and no `delivery_attempts` row (V-649).
|
||||
|
||||
## mavsttd — STT worker (optional, remote whisper.cpp)
|
||||
|
||||
Requires `LD_LIBRARY_PATH` to include deps/lib (for libwhisper.so, libggml-vulkan.so).
|
||||
|
||||
@@ -45,4 +45,17 @@ func TestDeployConfigLoads(t *testing.T) {
|
||||
if cfg.Voice.RouterThreshold <= 0 {
|
||||
t.Error("router threshold did not get its default")
|
||||
}
|
||||
|
||||
// The second reach (V-649). Deleting this block is how you turn ntfy off,
|
||||
// so its absence has to be loud: sev3-away nudges and away reminders route
|
||||
// to ntfy and to nothing else, and a nil sink drops them with no log and no
|
||||
// outbox row. The token is a ${VAR} that CI cannot resolve, so this checks
|
||||
// the wiring and not the credential.
|
||||
if cfg.Ntfy == nil {
|
||||
t.Fatal("deploy config has no ntfy block — sev3-away and away reminders " +
|
||||
"would have nowhere to land, and would vanish silently rather than fail")
|
||||
}
|
||||
if cfg.Ntfy.BaseURL == "" || cfg.Ntfy.Topic == "" {
|
||||
t.Errorf("ntfy block is incomplete: base_url=%q topic=%q", cfg.Ntfy.BaseURL, cfg.Ntfy.Topic)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,11 +7,17 @@
|
||||
// the relay). the dispatcher already strips detail off away sendables; the
|
||||
// sink uses the same helper so it can't leak the body on its own either.
|
||||
//
|
||||
// ntfy runs locally (docker, 127.0.0.1:8085, deny-all auth). maven publishes
|
||||
// with a dedicated user (write-only to maven-* topics) — the credential is a
|
||||
// delivery-config secret, not a db key; a popped ntfy sink can push spam to
|
||||
// your phone, nothing else. matches the module key-isolation invariant: the
|
||||
// sink never holds the sqlcipher key.
|
||||
// ntfy is a self-hosted server with deny-all auth — ntfy.kvmx.ru as of
|
||||
// 07-08-2026, reached directly, not through the socks relay telegram needs.
|
||||
// maven publishes with a write-only token scoped to its own topic; the
|
||||
// credential is a delivery-config secret, not a db key. a popped ntfy sink
|
||||
// can push spam to that one topic, nothing else — it cannot read the topic
|
||||
// back and it never holds the sqlcipher key.
|
||||
//
|
||||
// this is the second reach, and the reason there is one is that telegram was
|
||||
// the only one (V-649). telegram needs api.telegram.org, a socks relay on the
|
||||
// host and a matching ufw rule, three things in series that have each broken
|
||||
// once. ntfy shares none of them.
|
||||
package ntfysink
|
||||
|
||||
import (
|
||||
@@ -31,11 +37,29 @@ import (
|
||||
// the credential lives in the daemon's config (or a systemd credential),
|
||||
// never in the binary.
|
||||
type Config struct {
|
||||
BaseURL string // e.g. http://127.0.0.1:8085 (no trailing path)
|
||||
Topic string // e.g. maven (all maven notifications land here)
|
||||
Username string // basic auth; empty = anonymous (won't work with deny-all)
|
||||
Password string // basic auth
|
||||
Timeout time.Duration // per-request; 0 = DefaultTimeout
|
||||
// BaseURL — the ntfy server, no trailing path. Required.
|
||||
BaseURL string `json:"base_url"`
|
||||
|
||||
// Topic — where maven publishes. Required. All maven notifications land
|
||||
// on this one topic; severity rides the Priority header, not the topic.
|
||||
Topic string `json:"topic"`
|
||||
|
||||
// Token — an ntfy access token, sent as a bearer. This is the preferred
|
||||
// credential: ntfy scopes a token to a topic and to write-only, so a
|
||||
// popped sink can push to this one topic and cannot read it back or
|
||||
// touch another. Revoking it does not disturb a password anyone else
|
||||
// uses. Mutually exclusive with Username.
|
||||
Token string `json:"token,omitempty"`
|
||||
|
||||
// Username, Password — basic auth, for a server that has no tokens.
|
||||
// Empty username means no credential is sent at all, which a deny-all
|
||||
// server rejects.
|
||||
Username string `json:"username,omitempty"`
|
||||
Password string `json:"password,omitempty"`
|
||||
|
||||
// Timeout — per-request; 0 = DefaultTimeout. A dead server must not hang
|
||||
// the tick loop.
|
||||
Timeout time.Duration `json:"-"`
|
||||
}
|
||||
|
||||
const DefaultTimeout = 10 * time.Second
|
||||
@@ -59,6 +83,12 @@ func New(cfg Config) (*Sink, error) {
|
||||
if cfg.Topic == "" {
|
||||
return nil, fmt.Errorf("ntfysink: Topic is required")
|
||||
}
|
||||
// Refuse rather than pick. Two credentials configured means someone
|
||||
// intended one of them, and guessing which would send the other nowhere
|
||||
// and leave a working config that is not the one they wrote.
|
||||
if cfg.Token != "" && cfg.Username != "" {
|
||||
return nil, fmt.Errorf("ntfysink: set Token or Username, not both")
|
||||
}
|
||||
to := cfg.Timeout
|
||||
if to == 0 {
|
||||
to = DefaultTimeout
|
||||
@@ -84,7 +114,9 @@ func (s *Sink) Send(ctx context.Context, d delivery.Sendable) error {
|
||||
}
|
||||
req.Header.Set("Title", "maven")
|
||||
req.Header.Set("Priority", priorityFor(d).String())
|
||||
if s.cfg.Username != "" {
|
||||
if s.cfg.Token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+s.cfg.Token)
|
||||
} else if s.cfg.Username != "" {
|
||||
req.SetBasicAuth(s.cfg.Username, s.cfg.Password)
|
||||
}
|
||||
|
||||
|
||||
@@ -224,6 +224,37 @@ func TestSendNoAuthWhenUsernameEmpty(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSendSetsBearerToken — the deployed credential (V-649) is an ntfy access
|
||||
// token scoped write-only to the maven topic, not a password. A token sent as
|
||||
// basic auth is rejected by ntfy, so the header shape is the whole test.
|
||||
func TestSendSetsBearerToken(t *testing.T) {
|
||||
rs := newRecordingServer(t, 200, "")
|
||||
srv := httptest.NewServer(rs.handler())
|
||||
defer srv.Close()
|
||||
|
||||
sink, _ := New(Config{BaseURL: srv.URL, Topic: "maven", Token: "tk_secret"})
|
||||
if err := sink.Send(context.Background(), nudgeSendable(loop.Sev3, "down")); err != nil {
|
||||
t.Fatalf("Send: %v", err)
|
||||
}
|
||||
_, _, _, auth, _, _ := rs.snapshot()
|
||||
if auth != "Bearer tk_secret" {
|
||||
t.Fatalf("auth: want 'Bearer tk_secret', got %q", auth)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewRejectsBothCredentials — configuring a token and a username means one
|
||||
// of them was meant and the other is a leftover. Picking either would leave a
|
||||
// server that authenticates against a credential nobody wrote down.
|
||||
func TestNewRejectsBothCredentials(t *testing.T) {
|
||||
_, err := New(Config{BaseURL: "http://x", Topic: "maven", Token: "tk_x", Username: "maven"})
|
||||
if err == nil {
|
||||
t.Fatal("New accepted both a token and a username")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "not both") {
|
||||
t.Errorf("error does not say which to fix: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSendTitleIsMaven(t *testing.T) {
|
||||
rs := newRecordingServer(t, 200, "")
|
||||
srv := httptest.NewServer(rs.handler())
|
||||
|
||||
@@ -41,8 +41,33 @@ type PendingQuestion struct {
|
||||
Attempts int // questions already asked
|
||||
// MaxAttempts caps Attempts. 0 ⇒ DefaultMaxAttempts.
|
||||
MaxAttempts int
|
||||
// Suspends counts how many times this question has stepped aside for
|
||||
// something he asked instead, and come back on the end of the answer. It is
|
||||
// deliberately NOT an attempt: a side query is not a failed answer, and
|
||||
// charging it a retry is the V-554 shape. See CanResume for why it is
|
||||
// counted at all.
|
||||
Suspends int
|
||||
}
|
||||
|
||||
// MaxSuspends — how many times one question may step aside and come back before
|
||||
// she lets the request go (V-654).
|
||||
//
|
||||
// It exists because suspension had no bound of any kind. A side query spends no
|
||||
// attempt, so MaxAttempts never applies to it, and it restarts the 90s clock, so
|
||||
// the TTL never arrives either. Measured on 2026-08-07: one unfilled time slot
|
||||
// rode the end of six consecutive unrelated replies and stopped only when a
|
||||
// seventh turn happened to read as a failed answer.
|
||||
//
|
||||
// Three, matching DefaultMaxAttempts, and for the same reason. Once he has
|
||||
// asked for three other things without touching the question, the likely truth
|
||||
// is that he has moved on and has not said so.
|
||||
const MaxSuspends = 3
|
||||
|
||||
// CanResume reports whether this question may step aside once more. False ⇒ the
|
||||
// caller lets the request go and says so; it must never simply stop resuming,
|
||||
// because a question dropped in silence reads as one that was answered.
|
||||
func (q *PendingQuestion) CanResume() bool { return q.Suspends < MaxSuspends }
|
||||
|
||||
// Action reads the parked question as the typed action it is assembling
|
||||
// (pending.go). Derived rather than stored: the question's fields stay the one
|
||||
// copy of the truth, so a caller that fills them the old way cannot end up with
|
||||
|
||||
+37
-6
@@ -89,8 +89,8 @@ type Poller struct {
|
||||
ranker Ranker
|
||||
cfg Config
|
||||
nextDue map[string]time.Time
|
||||
seen map[string]map[string]bool // feed → item ID, for items with no date
|
||||
polled map[string]bool // feed → polled at least once in THIS process
|
||||
seen map[string]*seenIDs // feed → item IDs, for items with no date
|
||||
polled map[string]bool // feed → polled at least once in THIS process
|
||||
}
|
||||
|
||||
// NewPoller wires a poller. Returns nil when there is nothing to poll — a
|
||||
@@ -122,7 +122,7 @@ func NewPoller(feeds []FeedConfig, fetch Fetcher, notes Notes, marks Marks, embe
|
||||
feeds: valid, fetch: fetch, notes: notes, marks: marks,
|
||||
embed: embed, ranker: ranker, cfg: cfg,
|
||||
nextDue: map[string]time.Time{},
|
||||
seen: map[string]map[string]bool{},
|
||||
seen: map[string]*seenIDs{},
|
||||
polled: map[string]bool{},
|
||||
}
|
||||
}
|
||||
@@ -283,6 +283,38 @@ func (p *Poller) mark(ctx context.Context, feed string, now time.Time) (time.Tim
|
||||
return at, true
|
||||
}
|
||||
|
||||
// maxSeenPerFeed bounds the undated-item set. It has to stay comfortably above
|
||||
// any one feed's front page, or an item still listed there would fall out of the
|
||||
// set and be written a second time. A few hundred entries covers the largest
|
||||
// page anyone publishes, and the set only has to span one poll window plus the
|
||||
// resync guard, not all of history.
|
||||
const maxSeenPerFeed = 512
|
||||
|
||||
// seenIDs is a bounded insertion-ordered set. The map answers the lookup, the
|
||||
// slice remembers what to drop first, so an undated feed cannot grow the poller
|
||||
// for as long as mavend runs.
|
||||
type seenIDs struct {
|
||||
ids map[string]bool
|
||||
order []string
|
||||
}
|
||||
|
||||
// add records id and reports whether it was new.
|
||||
func (s *seenIDs) add(id string) bool {
|
||||
if s.ids == nil {
|
||||
s.ids = make(map[string]bool, maxSeenPerFeed)
|
||||
}
|
||||
if s.ids[id] {
|
||||
return false
|
||||
}
|
||||
s.ids[id] = true
|
||||
s.order = append(s.order, id)
|
||||
if len(s.order) > maxSeenPerFeed {
|
||||
delete(s.ids, s.order[0])
|
||||
s.order = s.order[1:]
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// fresh — two dedup rules, because feeds are inconsistent about dates. A dated
|
||||
// item must be newer than the mark; an undated one is kept once per process by
|
||||
// ID.
|
||||
@@ -308,12 +340,11 @@ func (p *Poller) fresh(f FeedConfig, it Item, mark, now time.Time, resync bool)
|
||||
id = it.Title
|
||||
}
|
||||
if p.seen[f.Name] == nil {
|
||||
p.seen[f.Name] = map[string]bool{}
|
||||
p.seen[f.Name] = &seenIDs{}
|
||||
}
|
||||
if p.seen[f.Name][id] {
|
||||
if !p.seen[f.Name].add(id) {
|
||||
return false
|
||||
}
|
||||
p.seen[f.Name][id] = true
|
||||
return !resync
|
||||
}
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ package rss
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -208,3 +209,27 @@ func TestNoFeedsMeansNoPoller(t *testing.T) {
|
||||
t.Fatal("a feed with no name or url is not a configuration")
|
||||
}
|
||||
}
|
||||
|
||||
// An undated feed used to grow p.seen for as long as mavend ran. The set is
|
||||
// bounded now, and the bound must not cost the dedupe an item still on the
|
||||
// front page — only ids far older than any page fall out.
|
||||
func TestSeenIDsBounded(t *testing.T) {
|
||||
var s seenIDs
|
||||
for i := 0; i < maxSeenPerFeed*3; i++ {
|
||||
if !s.add(fmt.Sprintf("item-%d", i)) {
|
||||
t.Fatalf("item-%d read as already seen", i)
|
||||
}
|
||||
if len(s.ids) > maxSeenPerFeed || len(s.order) > maxSeenPerFeed {
|
||||
t.Fatalf("after %d inserts: ids=%d order=%d, cap is %d",
|
||||
i+1, len(s.ids), len(s.order), maxSeenPerFeed)
|
||||
}
|
||||
}
|
||||
// The newest insert is still deduped; the oldest was evicted.
|
||||
last := fmt.Sprintf("item-%d", maxSeenPerFeed*3-1)
|
||||
if s.add(last) {
|
||||
t.Fatalf("%s read as new, so the most recent id was dropped", last)
|
||||
}
|
||||
if !s.add("item-0") {
|
||||
t.Fatal("item-0 survived, so nothing was evicted")
|
||||
}
|
||||
}
|
||||
|
||||
+135
-11
@@ -68,6 +68,13 @@ func (m *MemoryStore) Insert(ctx context.Context, id string, vec []float32, meta
|
||||
// Rows under memory.NonRecallPrefix are excluded in SQL. They are speaker
|
||||
// voiceprints sharing this table, and note recall must not rank them; see that
|
||||
// constant for why the previous arrangement only appeared to do this.
|
||||
//
|
||||
// Every row is still scored, because a full scan is what picks the winners.
|
||||
// What the scan does NOT do is pay for a row it is about to discard: the score
|
||||
// is read straight off the stored bytes without materializing a []float32, and
|
||||
// the meta blob is copied and unmarshalled only for a row that has entered the
|
||||
// topK. Losers cost one dot product and nothing else. Ranking is unchanged —
|
||||
// same scores, same order, same ties.
|
||||
func (m *MemoryStore) Search(ctx context.Context, vec []float32, topK int) ([]memory.Result, error) {
|
||||
if topK <= 0 {
|
||||
topK = 10
|
||||
@@ -80,30 +87,127 @@ func (m *MemoryStore) Search(ctx context.Context, vec []float32, topK int) ([]me
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []memory.Result
|
||||
// sql.RawBytes hands us the driver's own buffer, valid only until the next
|
||||
// Next(). Nothing here outlives the row except what topK.offer copies on a
|
||||
// survivor, so the three columns cost no allocation per row.
|
||||
var id, blob, metaJSON sql.RawBytes
|
||||
top := newTopK(topK)
|
||||
for rows.Next() {
|
||||
var id, metaJSON string
|
||||
var blob []byte
|
||||
if err := rows.Scan(&id, &blob, &metaJSON); err != nil {
|
||||
return nil, fmt.Errorf("memory: row: %w", err)
|
||||
}
|
||||
meta := map[string]string{}
|
||||
if err := json.Unmarshal([]byte(metaJSON), &meta); err != nil {
|
||||
return nil, fmt.Errorf("memory: unmarshal meta for %q: %w", id, err)
|
||||
}
|
||||
out = append(out, memory.Result{ID: id, Score: dot(vec, decodeVec(blob)), Meta: meta})
|
||||
top.offer(dotBlob(vec, blob), id, metaJSON)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("memory: rows: %w", err)
|
||||
}
|
||||
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Score > out[j].Score })
|
||||
if topK < len(out) {
|
||||
out = out[:topK]
|
||||
survivors := top.sorted()
|
||||
out := make([]memory.Result, 0, len(survivors))
|
||||
for _, c := range survivors {
|
||||
meta := map[string]string{}
|
||||
if err := json.Unmarshal(c.meta, &meta); err != nil {
|
||||
return nil, fmt.Errorf("memory: unmarshal meta for %q: %w", c.id, err)
|
||||
}
|
||||
out = append(out, memory.Result{ID: c.id, Score: c.score, Meta: meta})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// candidate is one row that is currently in the topK: its score, its id, and
|
||||
// its meta blob copied out of the driver's buffer. The copy is the price of
|
||||
// surviving, and only survivors pay it.
|
||||
type candidate struct {
|
||||
score float64
|
||||
id string
|
||||
meta []byte
|
||||
}
|
||||
|
||||
// topK keeps the k highest-scoring candidates seen so far as a min-heap, so the
|
||||
// weakest survivor is always heap[0] and one comparison decides whether a new
|
||||
// row is worth copying. k is 10 in practice, so the heap is tiny and the whole
|
||||
// structure fits in cache.
|
||||
//
|
||||
// It is a plain slice with hand-written sift operations rather than
|
||||
// container/heap, because that interface boxes every element into an `any` on
|
||||
// Push and costs an allocation per surviving row.
|
||||
type topK struct {
|
||||
k int
|
||||
heap []candidate
|
||||
}
|
||||
|
||||
func newTopK(k int) *topK {
|
||||
return &topK{k: k, heap: make([]candidate, 0, k)}
|
||||
}
|
||||
|
||||
// offer admits a row if it beats the weakest survivor, or if the heap is not
|
||||
// full yet. id and meta are the driver's buffers and are copied here, never
|
||||
// retained.
|
||||
//
|
||||
// A row that only ties the weakest survivor does not displace it, so among
|
||||
// equal scores the earliest k rows are kept. The full sort this replaced used
|
||||
// sort.Slice, which is not stable, so it broke such a tie arbitrarily. That is
|
||||
// the ONE observable difference between the two, and it is deliberate:
|
||||
// deterministic beats arbitrary.
|
||||
//
|
||||
// It is not academic. Under the real embedder an exact tie means duplicate
|
||||
// vectors and nothing in the recall eval moved (V-643). Under the hash
|
||||
// embedder the eval's deterministic floor uses, ties are everywhere — it is
|
||||
// bag-of-words, so every note sharing no word with the query scores exactly 0
|
||||
// — and recall@3 on that run moved 74.1% to 81.5% purely because the zeros now
|
||||
// come out in a fixed order. Neither number measures retrieval. recall@1 and
|
||||
// false recall, which the eval actually asserts, are unchanged on both runs.
|
||||
func (t *topK) offer(score float64, id, meta []byte) {
|
||||
if t.k == 0 {
|
||||
return
|
||||
}
|
||||
if len(t.heap) < t.k {
|
||||
t.heap = append(t.heap, candidate{score: score, id: string(id), meta: append([]byte(nil), meta...)})
|
||||
t.up(len(t.heap) - 1)
|
||||
return
|
||||
}
|
||||
if score <= t.heap[0].score {
|
||||
return
|
||||
}
|
||||
t.heap[0] = candidate{score: score, id: string(id), meta: append([]byte(nil), meta...)}
|
||||
t.down(0)
|
||||
}
|
||||
|
||||
func (t *topK) up(i int) {
|
||||
for i > 0 {
|
||||
parent := (i - 1) / 2
|
||||
if t.heap[parent].score <= t.heap[i].score {
|
||||
return
|
||||
}
|
||||
t.heap[parent], t.heap[i] = t.heap[i], t.heap[parent]
|
||||
i = parent
|
||||
}
|
||||
}
|
||||
|
||||
func (t *topK) down(i int) {
|
||||
for {
|
||||
l, r, small := 2*i+1, 2*i+2, i
|
||||
if l < len(t.heap) && t.heap[l].score < t.heap[small].score {
|
||||
small = l
|
||||
}
|
||||
if r < len(t.heap) && t.heap[r].score < t.heap[small].score {
|
||||
small = r
|
||||
}
|
||||
if small == i {
|
||||
return
|
||||
}
|
||||
t.heap[small], t.heap[i] = t.heap[i], t.heap[small]
|
||||
i = small
|
||||
}
|
||||
}
|
||||
|
||||
// sorted drains the heap into descending score order — what Search returns.
|
||||
func (t *topK) sorted() []candidate {
|
||||
out := t.heap
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].score > out[j].score })
|
||||
return out
|
||||
}
|
||||
|
||||
// ByPrefix returns every row whose id starts with prefix, vectors included.
|
||||
//
|
||||
// This is not a similarity query and deliberately does not score anything:
|
||||
@@ -240,6 +344,26 @@ func decodeVec(b []byte) []float32 {
|
||||
return v
|
||||
}
|
||||
|
||||
// dotBlob is dot against a vector still in its stored encoding, so scoring a
|
||||
// row the query is about to discard does not allocate the []float32 that
|
||||
// decodeVec would build. Same arithmetic, same order of operations, so it
|
||||
// returns bit-identical scores to dot(a, decodeVec(b)).
|
||||
//
|
||||
// A blob whose length isn't a multiple of 4 is truncated to the whole-element
|
||||
// prefix, matching decodeVec, and a length mismatch is 0, matching dot.
|
||||
func dotBlob(a []float32, b []byte) float64 {
|
||||
n := len(b) / 4
|
||||
if len(a) != n || n == 0 {
|
||||
return 0
|
||||
}
|
||||
var sum float64
|
||||
for i := 0; i < n; i++ {
|
||||
f := math.Float32frombits(binary.LittleEndian.Uint32(b[4*i:]))
|
||||
sum += float64(a[i]) * float64(f)
|
||||
}
|
||||
return sum
|
||||
}
|
||||
|
||||
// dot is the cosine similarity for L2-normalized vectors (mismatched lengths ⇒
|
||||
// 0, matching internal/memory's cosine).
|
||||
func dot(a, b []float32) float64 {
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math"
|
||||
"math/rand"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// benchDim is the resident embedder's width (multilingual-e5-small, 384), so
|
||||
// the per-row decode cost the benchmark measures is the real one.
|
||||
const benchDim = 384
|
||||
|
||||
// seedMemVectors fills a fresh store with n L2-normalized rows carrying a meta
|
||||
// blob the size recall actually stores — the note text plus its type — because
|
||||
// the cost this benchmark exists to measure is unmarshalling that blob for
|
||||
// every row when only topK survivors need it.
|
||||
func seedMemVectors(tb testing.TB, n int) *MemoryStore {
|
||||
tb.Helper()
|
||||
path := filepath.Join(tb.TempDir(), "mem_bench.db")
|
||||
st, err := Open(context.Background(), path)
|
||||
if err != nil {
|
||||
tb.Fatalf("Open: %v", err)
|
||||
}
|
||||
tb.Cleanup(func() { _ = st.Close() })
|
||||
m := st.VectorMemory()
|
||||
|
||||
rng := rand.New(rand.NewSource(1))
|
||||
ctx := context.Background()
|
||||
for i := 0; i < n; i++ {
|
||||
if err := m.Insert(ctx, fmt.Sprintf("note:%d", i), randUnitVec(rng, benchDim), map[string]string{
|
||||
"type": "note",
|
||||
"text": fmt.Sprintf("заметка номер %d о том, что надо не забыть сделать на неделе", i),
|
||||
}); err != nil {
|
||||
tb.Fatalf("Insert %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func randUnitVec(rng *rand.Rand, dim int) []float32 {
|
||||
v := make([]float32, dim)
|
||||
var norm float64
|
||||
for i := range v {
|
||||
f := rng.NormFloat64()
|
||||
v[i] = float32(f)
|
||||
norm += f * f
|
||||
}
|
||||
norm = math.Sqrt(norm)
|
||||
for i := range v {
|
||||
v[i] = float32(float64(v[i]) / norm)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// BenchmarkMemoryStoreSearch measures one recall query against a store of n
|
||||
// rows. Row counts bracket the documented scale: 1000 is a plausible today,
|
||||
// 10000 is the "thousands, not millions" ceiling the type doc claims a full
|
||||
// scan is fine at.
|
||||
func BenchmarkMemoryStoreSearch(b *testing.B) {
|
||||
for _, n := range []int{1000, 10000} {
|
||||
b.Run(fmt.Sprintf("rows=%d", n), func(b *testing.B) {
|
||||
m := seedMemVectors(b, n)
|
||||
q := randUnitVec(rand.New(rand.NewSource(2)), benchDim)
|
||||
ctx := context.Background()
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if _, err := m.Search(ctx, q, 10); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"sort"
|
||||
"testing"
|
||||
|
||||
"github.com/kami/maven/internal/memory"
|
||||
)
|
||||
|
||||
// naiveSearch is the implementation Search replaced: score every row into a
|
||||
// slice, sort the whole slice, truncate. It stays in the test file as the
|
||||
// reference the bounded-heap version is judged against, because "recall must
|
||||
// not change" is a claim about output, not about the code that produces it.
|
||||
func naiveSearch(t *testing.T, m *MemoryStore, vec []float32, topK int) []memory.Result {
|
||||
t.Helper()
|
||||
rows, err := m.db.QueryContext(context.Background(),
|
||||
`SELECT id, vec FROM memory_vectors WHERE id NOT LIKE ? ESCAPE '\'`,
|
||||
escapeLike(memory.NonRecallPrefix)+"%")
|
||||
if err != nil {
|
||||
t.Fatalf("naive scan: %v", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []memory.Result
|
||||
for rows.Next() {
|
||||
var id string
|
||||
var blob []byte
|
||||
if err := rows.Scan(&id, &blob); err != nil {
|
||||
t.Fatalf("naive row: %v", err)
|
||||
}
|
||||
out = append(out, memory.Result{ID: id, Score: dot(vec, decodeVec(blob))})
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
t.Fatalf("naive rows: %v", err)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Score > out[j].Score })
|
||||
if topK < len(out) {
|
||||
out = out[:topK]
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestMemoryStoreSearchMatchesNaive is the constraint on V-643: the bounded
|
||||
// heap must return exactly what a full scan and sort returned. Distinct random
|
||||
// vectors, so no two scores tie and the ranking is total — a mismatch here is
|
||||
// arithmetic or heap logic, not a tie-break difference.
|
||||
func TestMemoryStoreSearchMatchesNaive(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
m := newMemTestStore(t).VectorMemory()
|
||||
|
||||
rng := rand.New(rand.NewSource(7))
|
||||
const rows, dim = 500, 64
|
||||
for i := 0; i < rows; i++ {
|
||||
if err := m.Insert(ctx, fmt.Sprintf("n%d", i), randUnitVec(rng, dim), map[string]string{
|
||||
"text": fmt.Sprintf("note %d", i),
|
||||
}); err != nil {
|
||||
t.Fatalf("Insert %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
for _, topK := range []int{1, 3, 10, 50, rows, rows + 100} {
|
||||
q := randUnitVec(rng, dim)
|
||||
got, err := m.Search(ctx, q, topK)
|
||||
if err != nil {
|
||||
t.Fatalf("Search topK=%d: %v", topK, err)
|
||||
}
|
||||
want := naiveSearch(t, m, q, topK)
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("topK=%d: got %d results, naive returned %d", topK, len(got), len(want))
|
||||
}
|
||||
for i := range want {
|
||||
if got[i].ID != want[i].ID {
|
||||
t.Errorf("topK=%d rank %d: got %q, naive says %q", topK, i, got[i].ID, want[i].ID)
|
||||
}
|
||||
if got[i].Score != want[i].Score {
|
||||
t.Errorf("topK=%d rank %d (%s): score %v, naive says %v",
|
||||
topK, i, got[i].ID, got[i].Score, want[i].Score)
|
||||
}
|
||||
}
|
||||
if len(got) > 0 && got[0].Meta["text"] == "" {
|
||||
t.Errorf("topK=%d: survivor %s has no meta — it was never unmarshalled", topK, got[0].ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestDotBlobMatchesDot pins the claim in dotBlob's doc comment: reading the
|
||||
// vector out of its stored bytes is bit-identical to decoding it first. Scores
|
||||
// feed a gate with a 0.008 margin, so "close enough" is not the bar.
|
||||
func TestDotBlobMatchesDot(t *testing.T) {
|
||||
rng := rand.New(rand.NewSource(11))
|
||||
for i := 0; i < 200; i++ {
|
||||
a := randUnitVec(rng, 384)
|
||||
b := randUnitVec(rng, 384)
|
||||
if got, want := dotBlob(a, encodeVec(b)), dot(a, b); got != want {
|
||||
t.Fatalf("dotBlob = %v, dot = %v", got, want)
|
||||
}
|
||||
}
|
||||
// Length mismatch is 0 in both, and so is an empty vector.
|
||||
if got := dotBlob([]float32{1, 0}, encodeVec([]float32{1, 0, 0})); got != 0 {
|
||||
t.Errorf("mismatched lengths scored %v, want 0", got)
|
||||
}
|
||||
if got := dotBlob(nil, nil); got != 0 {
|
||||
t.Errorf("empty scored %v, want 0", got)
|
||||
}
|
||||
}
|
||||
@@ -295,6 +295,27 @@ func (f *Fetcher) checkURL(u *url.URL) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// pruneHostsAbove is when pruneLocked bothers to walk the map. Below it the
|
||||
// walk costs more than the entries do, and `crawl.on_demand` means the host set
|
||||
// is whatever he names out loud, so it grows slowly.
|
||||
const pruneHostsAbove = 64
|
||||
|
||||
// pruneLocked drops hosts whose last dial is further back than HostInterval.
|
||||
// Such an entry cannot delay anything — waitTurn would let the next request
|
||||
// through immediately — so keeping it only holds memory for the life of the
|
||||
// process. Caller holds f.mu.
|
||||
func (f *Fetcher) pruneLocked(now time.Time) {
|
||||
if len(f.last) <= pruneHostsAbove {
|
||||
return
|
||||
}
|
||||
cutoff := now.Add(-f.cfg.HostInterval)
|
||||
for h, at := range f.last {
|
||||
if at.Before(cutoff) {
|
||||
delete(f.last, h)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// waitTurn blocks until this host's rate-limit interval has elapsed. It holds
|
||||
// no lock while sleeping, so two hosts never wait on each other.
|
||||
func (f *Fetcher) waitTurn(ctx context.Context, host string) error {
|
||||
@@ -304,6 +325,7 @@ func (f *Fetcher) waitTurn(ctx context.Context, host string) error {
|
||||
earliest := f.last[host].Add(f.cfg.HostInterval)
|
||||
if !now.Before(earliest) {
|
||||
f.last[host] = now
|
||||
f.pruneLocked(now)
|
||||
f.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
@@ -318,3 +319,37 @@ func TestPostObeysDenylist(t *testing.T) {
|
||||
t.Fatalf("error = %v, want ErrBlocked", err)
|
||||
}
|
||||
}
|
||||
|
||||
// f.last used to hold one entry per host ever dialed, for the life of the
|
||||
// process. A host whose last dial is older than HostInterval cannot delay
|
||||
// anything, so it is dropped once the map is worth walking.
|
||||
func TestHostRateMapIsPruned(t *testing.T) {
|
||||
f := New(Config{HostInterval: time.Minute, AllowPrivate: true})
|
||||
stale := time.Now().Add(-time.Hour)
|
||||
for i := 0; i < pruneHostsAbove*2; i++ {
|
||||
f.last[fmt.Sprintf("h%d.example", i)] = stale
|
||||
}
|
||||
|
||||
// One real turn is what triggers the sweep.
|
||||
if err := f.waitTurn(context.Background(), "fresh.example"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(f.last) != 1 {
|
||||
t.Fatalf("len(f.last) = %d after the sweep, want 1 (only the host just dialed)", len(f.last))
|
||||
}
|
||||
if _, ok := f.last["fresh.example"]; !ok {
|
||||
t.Fatal("the host just dialed was pruned, so its own rate limit is lost")
|
||||
}
|
||||
|
||||
// A host inside the interval is kept: pruning must not hand out a free turn.
|
||||
f.last["recent.example"] = time.Now()
|
||||
for i := 0; i < pruneHostsAbove*2; i++ {
|
||||
f.last[fmt.Sprintf("g%d.example", i)] = stale
|
||||
}
|
||||
if err := f.waitTurn(context.Background(), "other.example"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok := f.last["recent.example"]; !ok {
|
||||
t.Fatal("a host dialed inside HostInterval was pruned")
|
||||
}
|
||||
}
|
||||
|
||||
Executable
+143
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env bash
|
||||
# audit.sh — the repo inventory, in one command.
|
||||
#
|
||||
# Every section here was reconstructed by hand, from scratch, in session after
|
||||
# session: 93 grep sweeps across the four longest ones before a single edit was
|
||||
# made. The answers move slowly and the sweeps did not, so they are written down
|
||||
# once here instead.
|
||||
#
|
||||
# It prints and never writes. A committed inventory file goes stale silently and
|
||||
# then lies; a report you regenerate cannot.
|
||||
#
|
||||
# Read-only. Safe to run at any point, including mid-conflict.
|
||||
#
|
||||
# make audit # everything
|
||||
# make audit SECTION=todo # one section: loc, todo, stubs, docs, tests, gaps
|
||||
|
||||
set -uo pipefail
|
||||
cd "$(dirname "$0")/.." || exit 1
|
||||
|
||||
SECTION="${SECTION:-all}"
|
||||
want() { [ "$SECTION" = all ] || [ "$SECTION" = "$1" ]; }
|
||||
rule() { printf '\n=== %s %s\n' "$1" "$(printf '%.0s=' $(seq 1 $((66 - ${#1}))))"; }
|
||||
|
||||
# git grep over tracked files only. deps/ and models/ are gitignored and huge,
|
||||
# and a plain grep -r walks into both.
|
||||
g() { git grep -nI "$@" 2>/dev/null; }
|
||||
|
||||
printf 'Maven repo inventory @ %s (%s)\n' \
|
||||
"$(git rev-parse --short HEAD 2>/dev/null || echo '?')" \
|
||||
"$(git log -1 --format=%cs 2>/dev/null || echo '?')"
|
||||
|
||||
# --- loc -------------------------------------------------------------------
|
||||
# Non-test Go lines per package. Size is the cheapest proxy for "where does the
|
||||
# complexity actually sit", and it is the first thing every audit asked for.
|
||||
if want loc; then
|
||||
rule "PACKAGES BY LOC (non-test)"
|
||||
for d in $(find ./cmd ./internal ./pkg -maxdepth 2 -type d 2>/dev/null | sort); do
|
||||
files=$(find "$d" -maxdepth 1 -name '*.go' ! -name '*_test.go' 2>/dev/null)
|
||||
[ -z "$files" ] && continue
|
||||
n=$(printf '%s\n' "$files" | wc -l)
|
||||
l=$(printf '%s\0' $files | xargs -0 cat 2>/dev/null | wc -l)
|
||||
printf '%7d %3d files %s\n' "$l" "$n" "$d"
|
||||
done | sort -rn
|
||||
fi
|
||||
|
||||
# --- todo ------------------------------------------------------------------
|
||||
if want todo; then
|
||||
rule "TODO / FIXME / XXX / HACK / BUG (non-test)"
|
||||
g -E '(^|[^a-zA-Z])(TODO|FIXME|XXX|HACK|BUG:)' -- 'cmd/**/*.go' 'internal/**/*.go' 'pkg/**/*.go' \
|
||||
| grep -v '_test\.go:' | sed 's/^/ /' || echo " none"
|
||||
fi
|
||||
|
||||
# --- stubs -----------------------------------------------------------------
|
||||
# Code only, never doc comments. "not wired" is this repo's design vocabulary
|
||||
# for a nil dependency and appears in ~30 comments that describe working code,
|
||||
# so searching prose here reports the architecture back as a gap. Likewise
|
||||
# internal/ipc/unimplemented.go is skipped whole: the file IS the deliberate
|
||||
# Unimplemented*Server pattern, not 60 missing methods. "placeholder" is not a
|
||||
# term here either -- it names real identifiers (SQL placeholders,
|
||||
# Deck.RequirePlaceholder, tokenPlaceholder) and matched 16 working lines.
|
||||
if want stubs; then
|
||||
rule "STUBS / NOT IMPLEMENTED (code, non-test)"
|
||||
g -iE 'not (yet )?implemented|unimplemented|пока не умею|panic\("TODO' \
|
||||
-- 'cmd/**/*.go' 'internal/**/*.go' 'pkg/**/*.go' \
|
||||
| grep -v '_test\.go:' \
|
||||
| grep -v '^internal/ipc/unimplemented\.go:' \
|
||||
| grep -vE '^[^:]+:[0-9]+:[[:space:]]*//' \
|
||||
| sed 's/^/ /' || echo " none"
|
||||
|
||||
# CoreAPI stub parity. A stub missing from unimplemented.go breaks the build
|
||||
# via `var _ CoreAPI = UnimplementedCoreAPI{}`. A stub left behind after its
|
||||
# method leaves an interface compiles forever and is caught by nothing, so it
|
||||
# is counted here until V-652 turns it into a test.
|
||||
printf '\n CoreAPI stub parity:\n'
|
||||
python3 - <<'PY' 2>/dev/null | sed 's/^/ /' || echo " (skipped: python3 unavailable)"
|
||||
import re
|
||||
src = open('internal/ipc/coreapi.go').read()
|
||||
decl = set()
|
||||
for m in re.finditer(r'type (\w+API) interface \{(.*?)\n\}', src, re.S):
|
||||
decl |= set(re.findall(r'^\t([A-Z]\w*)\(', m.group(2), re.M))
|
||||
stub = set(re.findall(r'func \(UnimplementedCoreAPI\) (\w+)\(',
|
||||
open('internal/ipc/unimplemented.go').read()))
|
||||
print(f"{len(decl)} declared, {len(stub)} stubbed")
|
||||
for name in sorted(stub - decl):
|
||||
print(f"STALE {name} (stubbed, on no interface)")
|
||||
for name in sorted(decl - stub):
|
||||
print(f"MISSING {name} (declared, no stub)")
|
||||
PY
|
||||
fi
|
||||
|
||||
# --- docs ------------------------------------------------------------------
|
||||
# Living tier only. docs/evals/ are dated measurements that are never edited
|
||||
# after the day, and docs/archive/ is dead by definition, so neither can be
|
||||
# stale. Age is against the recorded date, not against HEAD: every commit moves
|
||||
# HEAD, so a sha comparison would mark the whole tier stale every day.
|
||||
if want docs; then
|
||||
rule "LIVING DOCS — Last verified"
|
||||
today=$(date +%s)
|
||||
for f in docs/*.md; do
|
||||
[ -e "$f" ] || continue
|
||||
line=$(grep -m1 -o 'Last verified: *[0-9-]\{8,10\}[^ ]*\( *@ *[0-9a-f]\{7,\}\)\?' "$f" 2>/dev/null)
|
||||
if [ -z "$line" ]; then
|
||||
printf ' %-44s %s\n' "$(basename "$f")" "MISSING"
|
||||
continue
|
||||
fi
|
||||
d=$(printf '%s' "$line" | grep -o '[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}' | head -1)
|
||||
age=""
|
||||
if [ -n "$d" ] && when=$(date -d "$d" +%s 2>/dev/null); then
|
||||
days=$(( (today - when) / 86400 ))
|
||||
age="${days}d"
|
||||
[ "$days" -gt 30 ] && age="${days}d <-- STALE"
|
||||
fi
|
||||
printf ' %-44s %-34s %s\n' "$(basename "$f")" "${line#Last verified: }" "$age"
|
||||
done
|
||||
fi
|
||||
|
||||
# --- tests -----------------------------------------------------------------
|
||||
if want tests; then
|
||||
rule "TEST SHAPE"
|
||||
printf ' benchmarks : %s\n' "$(g -c 'func Benchmark' -- '**/*_test.go' | awk -F: '{s+=$2} END{print s+0}')"
|
||||
printf ' fuzz : %s\n' "$(g -c 'func Fuzz' -- '**/*_test.go' | awk -F: '{s+=$2} END{print s+0}')"
|
||||
printf ' table t.Run: %s\n' "$(g -c 't.Run(' -- '**/*_test.go' | awk -F: '{s+=$2} END{print s+0}')"
|
||||
printf ' golden files: %s\n' "$(g -l 'golden' -- '**/*_test.go' | wc -l)"
|
||||
fi
|
||||
|
||||
# --- gaps ------------------------------------------------------------------
|
||||
# A package with production code and no test file at all. Not a verdict — some
|
||||
# are pure wiring — but it is the list worth looking at before adding more.
|
||||
if want gaps; then
|
||||
rule "PACKAGES WITH NO TEST FILE"
|
||||
found=0
|
||||
for d in $(find ./cmd ./internal ./pkg -maxdepth 2 -type d 2>/dev/null | sort); do
|
||||
ls "$d"/*.go >/dev/null 2>&1 || continue
|
||||
ls "$d"/*_test.go >/dev/null 2>&1 && continue
|
||||
n=$(ls "$d"/*.go 2>/dev/null | wc -l)
|
||||
l=$(cat "$d"/*.go 2>/dev/null | wc -l)
|
||||
printf ' %6d lines %2d files %s\n' "$l" "$n" "$d"
|
||||
found=1
|
||||
done
|
||||
[ "$found" = 0 ] && echo " none"
|
||||
fi
|
||||
|
||||
exit 0
|
||||
Reference in New Issue
Block a user