Compare commits

...

8 Commits

Author SHA1 Message Date
claude beb093aebb Run one test and audit the repo without retyping either (V-653)
Two commands replace work that 66 sessions of transcripts show being
redone by hand.

`make t` replaces the CGO preamble, pasted 391 times across past
sessions and documented in CLAUDE.md as the way to do it. It also sets
MAVEN_ONNX_LIB, which that recipe did not: the four TestONNX*
measurements self-skip without it and the run still prints "ok", so
every targeted eval done the old way reported the hash ratchet while
reading as a real embedder score. -race keeps it honest against `make
test`, -count=1 keeps a stale cache from passing as a result.

`make audit` replaces the inventory sweep. The four longest sessions
spent 93 greps rebuilding it before their first edit. Runs in 0.75s.

Its stub search is narrower than the sweeps were, on purpose. "not
wired" is this repo's word for a nil dependency and matched ~30
comments describing working code; "placeholder" names real identifiers
and matched 16 more; internal/ipc/unimplemented.go is the deliberate
Unimplemented*Server pattern, not 60 gaps. A gap report that reports
the architecture back at you is one nobody reads twice.
2026-08-07 03:13:08 +04:00
claude b1b326018f Merge pull request 'NEEDS-KAMI: telegram is the only reach, and it depends on a socks relay that has failed before' (#196) from task/649-needs-kami-telegram-is-the-only-reach-an into master 2026-08-07 00:50:34 +02:00
claude 08889cad88 Give the box a second reach (V-649)
Telegram was the only way off this box, and it is not a direct path: it
needs api.telegram.org, a socks relay on the host and a matching ufw rule.
Each of those three has failed once, and when they do a sev4 nudge has
nowhere to go. ntfy shares none of them.

The spare is the smaller half of it. The routing table already sends
sev3-away nudges and away reminders to ntfy and to nothing else, so with no
block configured those two routes hit a nil sink in DispatchNudge and
DispatchReminder and are skipped — no log line, no delivery_attempts row.
An away reminder is worse than dropped: out stays empty, so MarkReminder
never runs and it re-fires every tick without ever being delivered.

Owner's call, 07-08-2026: ntfy.kvmx.ru, topic maven.

The sink now takes a bearer token, which is what that server wants and what
it could not do before. ntfy scopes a token to one topic and to write-only,
so a popped sink can push to the maven topic and cannot read it back. Basic
auth stays for a server with no tokens; configuring both is refused rather
than resolved by guessing.

Config keys got json tags. docs/operations.md has documented this block as
base_url/topic since before it existed, and the untagged struct would only
have answered to BaseURL/Topic — the documented config would have parsed
into an empty one.

The token is a ${NTFY_TOKEN} expansion from the gitignored
deploy/telegram.env, beside the telegram secrets. TestDeployConfigLoads now
fails if the block goes missing, because deleting it is how you turn the
reach off and the two silent routes are what that costs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 02:16:18 +04:00
claude a4630b9314 Merge pull request 'MemoryStore.Search decodes and unmarshals every row before keeping topK' (#195) from task/643-memorystore-search-decodes-and-unmarshal into master 2026-08-07 00:09:50 +02:00
claude 39d44bb384 Close a Vikunja task with done, and nothing else (V-641)
Owner's call, 07-08-2026. A completion summary written into the
description on the way out is lost anyway, and the durable record is the
commit messages and the merged PR.

Written during the V-641 session and left uncommitted; it rides this
branch rather than being dropped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 01:48:19 +04:00
claude 65ee0f9c61 Score every row, pay for only the ten that survive (V-643)
Search decoded the vector blob into a []float32 and JSON-unmarshalled the
meta map for every row, then sorted all N and threw away everything past
topK. Meta only ever matters for a survivor, and the sort answered a
question a bounded heap answers cheaper.

The scan still visits every row — that is what picks the winners. What it
no longer does is allocate for a row it is about to discard. dotBlob reads
the vector out of its stored bytes, so scoring costs nothing; a row is
copied and its meta unmarshalled only once it has entered the topK.

At 10000 rows and topK 10: 70.6ms to 26.8ms, 58MB to 17.5MB, 240k allocs
to 60k.

Recall is unchanged where it is measured. recall+onnx scores 22/32 with
recall@1 70.4% and recall@3 85.2%, identical to before.
TestMemoryStoreSearchMatchesNaive pins the ranking against the full-sort
implementation it replaced, and TestDotBlobMatchesDot pins bit-identical
scores, which the 0.008 gate margin demands.

One behaviour did move: ties. sort.Slice is not stable, so equal scores
were ordered arbitrarily; the heap now keeps the earliest. Under the real
embedder an exact tie is a duplicate vector and nothing moved. Under the
hash embedder the eval's floor uses, everything ties at 0 and that run's
recall@3 went 74.1% to 81.5% — a number that measures tie order, not
retrieval. recall@1 and false recall, the two the eval asserts, are
unchanged on both runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 01:48:05 +04:00
claude 76938e206d Put a number on the recall scan before changing it (V-643)
MemoryStore.Search is on the per-turn recall path and had no benchmark, so
any claim about its cost was an argument rather than a measurement.

Seeds a store with rows the shape recall actually stores — 384-wide
vectors, the resident embedder's width, and a meta blob carrying the note
text — at 1000 and 10000 rows. 10000 is the ceiling the type doc claims a
full scan is fine at.

Measured as it stands: 5.3ms and 24k allocs at 1000 rows, 70.6ms and 240k
allocs at 10000.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 01:48:05 +04:00
claude 0b3d81ecbf Merge pull request 'Two maps grow for the process lifetime with no eviction' (#194) from task/641-two-maps-grow-for-the-process-lifetime-w into master 2026-08-06 23:33:23 +02:00
12 changed files with 650 additions and 34 deletions
+22 -8
View File
@@ -58,16 +58,27 @@ make build-web # single daemon (pure-Go ones: web/waked/poll/caldav build w
make test # go test -race across ./internal/... ./cmd/... with CGO env set
```
Run a single test (must carry the CGO env for packages that touch STT/TTS/voice):
Run one package or one test with `make t`. **Do not hand-write the CGO preamble.**
Past sessions pasted it about 390 times. That is where the shell-quoting failures
came from. This box runs zsh, so an unquoted `-run Test*` or `--include=*.go`
dies on "no matches found" before `go` is ever reached.
```sh
CGO_CFLAGS="-I$(pwd)/deps/include -I$(pwd)/deps/whisper.cpp/ggml/include" \
CGO_LDFLAGS="-L$(pwd)/deps/lib -Wl,-rpath,$(pwd)/deps/lib" \
LD_LIBRARY_PATH="$(pwd)/deps/lib" \
deps/go/go/bin/go test -run TestName ./internal/router/
make t PKG=./internal/router/
make t PKG=./cmd/mavend/ RUN=TestSimulator
make t PKG=./internal/router/eval/ RUN='TestONNX' V=1 # V=1 for -v, RACE=0 to drop -race
```
Pure-Go packages (`router`, `memory`, `mavweb`, …) run under a plain `go test ./pkg/`.
`t` carries `-race`, so a green `make t` cannot turn red under `make test`. It carries
`-count=1`, so a cached PASS from before your edit is never mistaken for a result.
It also sets `MAVEN_ONNX_LIB`, which the hand-written recipe did not. The four
`TestONNX*` measurements self-skip when that variable is unset. The run still prints
`ok`. So every targeted eval done the old way reported the hash ratchet while reading
as a real embedder score.
Pure-Go packages (`router`, `memory`, `mavweb`, …) also run under a plain `go test ./pkg/`,
but `make t` works everywhere and is one thing to remember.
## The daemons (`cmd/`)
@@ -460,8 +471,11 @@ start of a session rather than one lookup per first use:
ToolSearch("select:mcp__vikunja__list_tasks,mcp__vikunja__get_task_details,mcp__vikunja__create_task,mcp__vikunja__update_task")
```
`update_task` carrying a `description` resets `done` to false, so closing a task with a
write-up takes two calls: the description, then `done: true`.
**Close a finished task with `done: true` and nothing else** (owner's call, 07-08-2026).
Do not write a completion summary into the description on the way out. It is lost anyway,
and the durable record is the commit messages and the merged PR. Note that `update_task`
carrying a `description` resets `done` to false, which is why a write-up ever took two
calls.
## Session workflow
+40 -1
View File
@@ -16,7 +16,7 @@ PIPER_BIN := $(shell pwd)/deps/piper/piper
PIPER_MODEL := $(shell pwd)/models/tts/ru_RU-irina-medium.onnx
PIPER_ESPEAK := $(shell pwd)/deps/piper/espeak-ng-data
.PHONY: simulate stt-fixtures test-stt-golden all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go deps-sentinel tidy eval-router eval-reach eval-recall eval-phrasing eval-models build-gpud
.PHONY: t audit simulate stt-fixtures test-stt-golden all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go deps-sentinel tidy eval-router eval-reach eval-recall eval-phrasing eval-models build-gpud
all: build
@@ -128,6 +128,35 @@ test: fmt-check vet
CGO_CFLAGS="$(CGO_CFLAGS)" CGO_LDFLAGS="$(CGO_LDFLAGS)" LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
$(GO) test -race -coverprofile=coverage.out ./internal/... ./cmd/...
# t — run ONE package or ONE test with the toolchain env already wired. This is
# the iteration target; `test` is the gate. Reach for it instead of pasting the
# CGO_CFLAGS/CGO_LDFLAGS/LD_LIBRARY_PATH preamble by hand, which is how it was
# done ~390 times across past sessions and is where the shell-quoting failures
# came from -- the interactive shell here is zsh, and an unquoted `-run Test*`
# or `--include=*.go` dies on "no matches found" before go ever starts.
#
# make t # whole tree (same scope as `test`)
# make t PKG=./internal/router/
# make t PKG=./cmd/mavend/ RUN=TestSimulator
# make t PKG=./internal/router/eval/ RUN='TestONNX' V=1
# make t PKG=./internal/store/ RACE=0 # drop -race when iterating hot
#
# -race is on by default so a green `make t` cannot turn red under `make test`.
# -count=1 because a cached PASS from before your edit is worse than no answer.
# MAVEN_ONNX_LIB is set for the same reason: the four TestONNX* measurements
# self-skip when it is unset, so a targeted eval run would otherwise report the
# deterministic hash ratchet and look like it scored the real embedder.
PKG ?= ./internal/... ./cmd/...
RUN ?=
V ?=
RACE ?= 1
t:
CGO_CFLAGS="$(CGO_CFLAGS)" CGO_LDFLAGS="$(CGO_LDFLAGS)" LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" \
$(GO) test $(if $(V),-v,) $(if $(filter-out 0,$(RACE)),-race,) -count=1 \
$(if $(RUN),-run '$(RUN)',) $(PKG)
# eval-router — score the held-out RU routing fixture (internal/router/eval).
# Verbose so the report tables land in the terminal. MAVEN_ONNX_LIB points the
# prod-representative baseline at the vendored runtime; override it or set it
@@ -189,6 +218,16 @@ eval-models:
# scores the fixtures against ggml-small and self-skips when the model is
# absent, and TestGoldenFixturesAreCanonical, which checks the committed audio
# and the manifest with no model at all.
# audit — the repo inventory: LOC per package, open TODOs, real stubs, living-doc
# staleness, test shape, packages with no test. Read-only, prints, writes nothing.
# Run it instead of rebuilding the same greps by hand; past sessions spent 93 of
# them on this before their first edit. SECTION=loc|todo|stubs|docs|tests|gaps
# narrows it.
SECTION ?= all
audit:
@SECTION="$(SECTION)" ./scripts/audit.sh
stt-fixtures:
./scripts/gen-stt-fixtures.sh
+19
View File
@@ -25,6 +25,25 @@
"llm_nudges": false
},
"//ntfy": [
"The second reach (V-649). Until 07-08-2026 telegram was the only one, and",
"telegram needs api.telegram.org, the socks relay below and a matching ufw",
"rule — three things in series that have each failed once, and when they do",
"a sev4 nudge has nowhere to go. ntfy shares none of them: it is reached",
"directly, no relay.",
"It is not only a spare. The routing table sends sev3-away and away",
"reminders here and NOWHERE else, so with this block absent those two",
"routes hit a nil sink and vanish without a log or an outbox row.",
"The credential is an ntfy access token, scoped write-only to this one",
"topic, so a popped sink can push to it and cannot read it back. Set it in",
"deploy/telegram.env beside the telegram secrets; that file is gitignored."
],
"ntfy": {
"base_url": "https://ntfy.kvmx.ru",
"topic": "maven",
"token": "${NTFY_TOKEN}"
},
"telegram": {
"bot_token": "${TELEGRAM_BOT_TOKEN}",
"chat_id": "${TELEGRAM_CHAT_ID}",
+8 -1
View File
@@ -1,5 +1,12 @@
# Telegram bot token and chat ID for mavend's away-channel reach.
# Secrets for mavend's away-channel reaches. The file is still called
# telegram.env because compose names it that; it holds both reaches now.
# Copy this file to deploy/telegram.env and fill in real values.
# deploy/telegram.env is gitignored — never commit the real secrets.
TELEGRAM_BOT_TOKEN=
TELEGRAM_CHAT_ID=
# ntfy access token for the `maven` topic, the second reach (V-649). Mint it on
# the ntfy server with write access to that topic and nothing else:
# ntfy token add --expires=never maven
# Read access is not needed — mavend publishes and never subscribes.
NTFY_TOKEN=
+12 -2
View File
@@ -1,6 +1,6 @@
# Start Commands
*Last verified: 2026-08-02 @ 7079a24. Living doc: correct it in place, do not append.*
*Last verified: 2026-08-07 @ a4630b9. Living doc: correct it in place, do not append.*
All commands assume `ROOT=/home/kami/apps/Maven` and the local Go toolchain at `$ROOT/deps/go/go/bin/go`.
@@ -44,7 +44,8 @@ Config path: `~/.config/maven/mavend.json`. Full example with all options.
"repeat_interval": "5m",
"ntfy": {
"base_url": "https://ntfy.kvmx.ru",
"topic": "maven"
"topic": "maven",
"token": "${NTFY_TOKEN}"
},
"phraser": {
"model_path": "/mnt/hdd1/llms/Qwen3-Maven-1.7B-Q8_0.gguf",
@@ -66,6 +67,15 @@ Config path: `~/.config/maven/mavend.json`. Full example with all options.
Omit the `embedder` block entirely to use the deterministic HashEmbedder floor (no ML, no ONNX runtime dependency). Useful for testing or low-resource setups.
`${NTFY_TOKEN}` and the `${TELEGRAM_*}` vars are expanded from `deploy/telegram.env`, which is gitignored. Copy `deploy/telegram.env.example` and fill it in. Mint a scoped token rather than reusing an admin one. It needs write access to the `maven` topic and nothing else:
```sh
ntfy access maven maven write-only
ntfy token add --expires=never maven
```
Deleting the `ntfy` block turns the reach off, and that is not a no-op. The routing table sends sev3-away nudges and away reminders to ntfy and nowhere else. With no sink wired they hit a nil and vanish, leaving no log line and no `delivery_attempts` row (V-649).
## mavsttd — STT worker (optional, remote whisper.cpp)
Requires `LD_LIBRARY_PATH` to include deps/lib (for libwhisper.so, libggml-vulkan.so).
+13
View File
@@ -45,4 +45,17 @@ func TestDeployConfigLoads(t *testing.T) {
if cfg.Voice.RouterThreshold <= 0 {
t.Error("router threshold did not get its default")
}
// The second reach (V-649). Deleting this block is how you turn ntfy off,
// so its absence has to be loud: sev3-away nudges and away reminders route
// to ntfy and to nothing else, and a nil sink drops them with no log and no
// outbox row. The token is a ${VAR} that CI cannot resolve, so this checks
// the wiring and not the credential.
if cfg.Ntfy == nil {
t.Fatal("deploy config has no ntfy block — sev3-away and away reminders " +
"would have nowhere to land, and would vanish silently rather than fail")
}
if cfg.Ntfy.BaseURL == "" || cfg.Ntfy.Topic == "" {
t.Errorf("ntfy block is incomplete: base_url=%q topic=%q", cfg.Ntfy.BaseURL, cfg.Ntfy.Topic)
}
}
+43 -11
View File
@@ -7,11 +7,17 @@
// the relay). the dispatcher already strips detail off away sendables; the
// sink uses the same helper so it can't leak the body on its own either.
//
// ntfy runs locally (docker, 127.0.0.1:8085, deny-all auth). maven publishes
// with a dedicated user (write-only to maven-* topics) — the credential is a
// delivery-config secret, not a db key; a popped ntfy sink can push spam to
// your phone, nothing else. matches the module key-isolation invariant: the
// sink never holds the sqlcipher key.
// ntfy is a self-hosted server with deny-all auth — ntfy.kvmx.ru as of
// 07-08-2026, reached directly, not through the socks relay telegram needs.
// maven publishes with a write-only token scoped to its own topic; the
// credential is a delivery-config secret, not a db key. a popped ntfy sink
// can push spam to that one topic, nothing else — it cannot read the topic
// back and it never holds the sqlcipher key.
//
// this is the second reach, and the reason there is one is that telegram was
// the only one (V-649). telegram needs api.telegram.org, a socks relay on the
// host and a matching ufw rule, three things in series that have each broken
// once. ntfy shares none of them.
package ntfysink
import (
@@ -31,11 +37,29 @@ import (
// the credential lives in the daemon's config (or a systemd credential),
// never in the binary.
type Config struct {
BaseURL string // e.g. http://127.0.0.1:8085 (no trailing path)
Topic string // e.g. maven (all maven notifications land here)
Username string // basic auth; empty = anonymous (won't work with deny-all)
Password string // basic auth
Timeout time.Duration // per-request; 0 = DefaultTimeout
// BaseURL — the ntfy server, no trailing path. Required.
BaseURL string `json:"base_url"`
// Topic — where maven publishes. Required. All maven notifications land
// on this one topic; severity rides the Priority header, not the topic.
Topic string `json:"topic"`
// Token — an ntfy access token, sent as a bearer. This is the preferred
// credential: ntfy scopes a token to a topic and to write-only, so a
// popped sink can push to this one topic and cannot read it back or
// touch another. Revoking it does not disturb a password anyone else
// uses. Mutually exclusive with Username.
Token string `json:"token,omitempty"`
// Username, Password — basic auth, for a server that has no tokens.
// Empty username means no credential is sent at all, which a deny-all
// server rejects.
Username string `json:"username,omitempty"`
Password string `json:"password,omitempty"`
// Timeout — per-request; 0 = DefaultTimeout. A dead server must not hang
// the tick loop.
Timeout time.Duration `json:"-"`
}
const DefaultTimeout = 10 * time.Second
@@ -59,6 +83,12 @@ func New(cfg Config) (*Sink, error) {
if cfg.Topic == "" {
return nil, fmt.Errorf("ntfysink: Topic is required")
}
// Refuse rather than pick. Two credentials configured means someone
// intended one of them, and guessing which would send the other nowhere
// and leave a working config that is not the one they wrote.
if cfg.Token != "" && cfg.Username != "" {
return nil, fmt.Errorf("ntfysink: set Token or Username, not both")
}
to := cfg.Timeout
if to == 0 {
to = DefaultTimeout
@@ -84,7 +114,9 @@ func (s *Sink) Send(ctx context.Context, d delivery.Sendable) error {
}
req.Header.Set("Title", "maven")
req.Header.Set("Priority", priorityFor(d).String())
if s.cfg.Username != "" {
if s.cfg.Token != "" {
req.Header.Set("Authorization", "Bearer "+s.cfg.Token)
} else if s.cfg.Username != "" {
req.SetBasicAuth(s.cfg.Username, s.cfg.Password)
}
@@ -224,6 +224,37 @@ func TestSendNoAuthWhenUsernameEmpty(t *testing.T) {
}
}
// TestSendSetsBearerToken — the deployed credential (V-649) is an ntfy access
// token scoped write-only to the maven topic, not a password. A token sent as
// basic auth is rejected by ntfy, so the header shape is the whole test.
func TestSendSetsBearerToken(t *testing.T) {
rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler())
defer srv.Close()
sink, _ := New(Config{BaseURL: srv.URL, Topic: "maven", Token: "tk_secret"})
if err := sink.Send(context.Background(), nudgeSendable(loop.Sev3, "down")); err != nil {
t.Fatalf("Send: %v", err)
}
_, _, _, auth, _, _ := rs.snapshot()
if auth != "Bearer tk_secret" {
t.Fatalf("auth: want 'Bearer tk_secret', got %q", auth)
}
}
// TestNewRejectsBothCredentials — configuring a token and a username means one
// of them was meant and the other is a leftover. Picking either would leave a
// server that authenticates against a credential nobody wrote down.
func TestNewRejectsBothCredentials(t *testing.T) {
_, err := New(Config{BaseURL: "http://x", Topic: "maven", Token: "tk_x", Username: "maven"})
if err == nil {
t.Fatal("New accepted both a token and a username")
}
if !strings.Contains(err.Error(), "not both") {
t.Errorf("error does not say which to fix: %v", err)
}
}
func TestSendTitleIsMaven(t *testing.T) {
rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler())
+135 -11
View File
@@ -68,6 +68,13 @@ func (m *MemoryStore) Insert(ctx context.Context, id string, vec []float32, meta
// Rows under memory.NonRecallPrefix are excluded in SQL. They are speaker
// voiceprints sharing this table, and note recall must not rank them; see that
// constant for why the previous arrangement only appeared to do this.
//
// Every row is still scored, because a full scan is what picks the winners.
// What the scan does NOT do is pay for a row it is about to discard: the score
// is read straight off the stored bytes without materializing a []float32, and
// the meta blob is copied and unmarshalled only for a row that has entered the
// topK. Losers cost one dot product and nothing else. Ranking is unchanged —
// same scores, same order, same ties.
func (m *MemoryStore) Search(ctx context.Context, vec []float32, topK int) ([]memory.Result, error) {
if topK <= 0 {
topK = 10
@@ -80,30 +87,127 @@ func (m *MemoryStore) Search(ctx context.Context, vec []float32, topK int) ([]me
}
defer rows.Close()
var out []memory.Result
// sql.RawBytes hands us the driver's own buffer, valid only until the next
// Next(). Nothing here outlives the row except what topK.offer copies on a
// survivor, so the three columns cost no allocation per row.
var id, blob, metaJSON sql.RawBytes
top := newTopK(topK)
for rows.Next() {
var id, metaJSON string
var blob []byte
if err := rows.Scan(&id, &blob, &metaJSON); err != nil {
return nil, fmt.Errorf("memory: row: %w", err)
}
meta := map[string]string{}
if err := json.Unmarshal([]byte(metaJSON), &meta); err != nil {
return nil, fmt.Errorf("memory: unmarshal meta for %q: %w", id, err)
}
out = append(out, memory.Result{ID: id, Score: dot(vec, decodeVec(blob)), Meta: meta})
top.offer(dotBlob(vec, blob), id, metaJSON)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("memory: rows: %w", err)
}
sort.Slice(out, func(i, j int) bool { return out[i].Score > out[j].Score })
if topK < len(out) {
out = out[:topK]
survivors := top.sorted()
out := make([]memory.Result, 0, len(survivors))
for _, c := range survivors {
meta := map[string]string{}
if err := json.Unmarshal(c.meta, &meta); err != nil {
return nil, fmt.Errorf("memory: unmarshal meta for %q: %w", c.id, err)
}
out = append(out, memory.Result{ID: c.id, Score: c.score, Meta: meta})
}
return out, nil
}
// candidate is one row that is currently in the topK: its score, its id, and
// its meta blob copied out of the driver's buffer. The copy is the price of
// surviving, and only survivors pay it.
type candidate struct {
score float64
id string
meta []byte
}
// topK keeps the k highest-scoring candidates seen so far as a min-heap, so the
// weakest survivor is always heap[0] and one comparison decides whether a new
// row is worth copying. k is 10 in practice, so the heap is tiny and the whole
// structure fits in cache.
//
// It is a plain slice with hand-written sift operations rather than
// container/heap, because that interface boxes every element into an `any` on
// Push and costs an allocation per surviving row.
type topK struct {
k int
heap []candidate
}
func newTopK(k int) *topK {
return &topK{k: k, heap: make([]candidate, 0, k)}
}
// offer admits a row if it beats the weakest survivor, or if the heap is not
// full yet. id and meta are the driver's buffers and are copied here, never
// retained.
//
// A row that only ties the weakest survivor does not displace it, so among
// equal scores the earliest k rows are kept. The full sort this replaced used
// sort.Slice, which is not stable, so it broke such a tie arbitrarily. That is
// the ONE observable difference between the two, and it is deliberate:
// deterministic beats arbitrary.
//
// It is not academic. Under the real embedder an exact tie means duplicate
// vectors and nothing in the recall eval moved (V-643). Under the hash
// embedder the eval's deterministic floor uses, ties are everywhere — it is
// bag-of-words, so every note sharing no word with the query scores exactly 0
// — and recall@3 on that run moved 74.1% to 81.5% purely because the zeros now
// come out in a fixed order. Neither number measures retrieval. recall@1 and
// false recall, which the eval actually asserts, are unchanged on both runs.
func (t *topK) offer(score float64, id, meta []byte) {
if t.k == 0 {
return
}
if len(t.heap) < t.k {
t.heap = append(t.heap, candidate{score: score, id: string(id), meta: append([]byte(nil), meta...)})
t.up(len(t.heap) - 1)
return
}
if score <= t.heap[0].score {
return
}
t.heap[0] = candidate{score: score, id: string(id), meta: append([]byte(nil), meta...)}
t.down(0)
}
func (t *topK) up(i int) {
for i > 0 {
parent := (i - 1) / 2
if t.heap[parent].score <= t.heap[i].score {
return
}
t.heap[parent], t.heap[i] = t.heap[i], t.heap[parent]
i = parent
}
}
func (t *topK) down(i int) {
for {
l, r, small := 2*i+1, 2*i+2, i
if l < len(t.heap) && t.heap[l].score < t.heap[small].score {
small = l
}
if r < len(t.heap) && t.heap[r].score < t.heap[small].score {
small = r
}
if small == i {
return
}
t.heap[small], t.heap[i] = t.heap[i], t.heap[small]
i = small
}
}
// sorted drains the heap into descending score order — what Search returns.
func (t *topK) sorted() []candidate {
out := t.heap
sort.Slice(out, func(i, j int) bool { return out[i].score > out[j].score })
return out
}
// ByPrefix returns every row whose id starts with prefix, vectors included.
//
// This is not a similarity query and deliberately does not score anything:
@@ -240,6 +344,26 @@ func decodeVec(b []byte) []float32 {
return v
}
// dotBlob is dot against a vector still in its stored encoding, so scoring a
// row the query is about to discard does not allocate the []float32 that
// decodeVec would build. Same arithmetic, same order of operations, so it
// returns bit-identical scores to dot(a, decodeVec(b)).
//
// A blob whose length isn't a multiple of 4 is truncated to the whole-element
// prefix, matching decodeVec, and a length mismatch is 0, matching dot.
func dotBlob(a []float32, b []byte) float64 {
n := len(b) / 4
if len(a) != n || n == 0 {
return 0
}
var sum float64
for i := 0; i < n; i++ {
f := math.Float32frombits(binary.LittleEndian.Uint32(b[4*i:]))
sum += float64(a[i]) * float64(f)
}
return sum
}
// dot is the cosine similarity for L2-normalized vectors (mismatched lengths ⇒
// 0, matching internal/memory's cosine).
func dot(a, b []float32) float64 {
+77
View File
@@ -0,0 +1,77 @@
package store
import (
"context"
"fmt"
"math"
"math/rand"
"path/filepath"
"testing"
)
// benchDim is the resident embedder's width (multilingual-e5-small, 384), so
// the per-row decode cost the benchmark measures is the real one.
const benchDim = 384
// seedMemVectors fills a fresh store with n L2-normalized rows carrying a meta
// blob the size recall actually stores — the note text plus its type — because
// the cost this benchmark exists to measure is unmarshalling that blob for
// every row when only topK survivors need it.
func seedMemVectors(tb testing.TB, n int) *MemoryStore {
tb.Helper()
path := filepath.Join(tb.TempDir(), "mem_bench.db")
st, err := Open(context.Background(), path)
if err != nil {
tb.Fatalf("Open: %v", err)
}
tb.Cleanup(func() { _ = st.Close() })
m := st.VectorMemory()
rng := rand.New(rand.NewSource(1))
ctx := context.Background()
for i := 0; i < n; i++ {
if err := m.Insert(ctx, fmt.Sprintf("note:%d", i), randUnitVec(rng, benchDim), map[string]string{
"type": "note",
"text": fmt.Sprintf("заметка номер %d о том, что надо не забыть сделать на неделе", i),
}); err != nil {
tb.Fatalf("Insert %d: %v", i, err)
}
}
return m
}
func randUnitVec(rng *rand.Rand, dim int) []float32 {
v := make([]float32, dim)
var norm float64
for i := range v {
f := rng.NormFloat64()
v[i] = float32(f)
norm += f * f
}
norm = math.Sqrt(norm)
for i := range v {
v[i] = float32(float64(v[i]) / norm)
}
return v
}
// BenchmarkMemoryStoreSearch measures one recall query against a store of n
// rows. Row counts bracket the documented scale: 1000 is a plausible today,
// 10000 is the "thousands, not millions" ceiling the type doc claims a full
// scan is fine at.
func BenchmarkMemoryStoreSearch(b *testing.B) {
for _, n := range []int{1000, 10000} {
b.Run(fmt.Sprintf("rows=%d", n), func(b *testing.B) {
m := seedMemVectors(b, n)
q := randUnitVec(rand.New(rand.NewSource(2)), benchDim)
ctx := context.Background()
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := m.Search(ctx, q, 10); err != nil {
b.Fatal(err)
}
}
})
}
}
+107
View File
@@ -0,0 +1,107 @@
package store
import (
"context"
"fmt"
"math/rand"
"sort"
"testing"
"github.com/kami/maven/internal/memory"
)
// naiveSearch is the implementation Search replaced: score every row into a
// slice, sort the whole slice, truncate. It stays in the test file as the
// reference the bounded-heap version is judged against, because "recall must
// not change" is a claim about output, not about the code that produces it.
func naiveSearch(t *testing.T, m *MemoryStore, vec []float32, topK int) []memory.Result {
t.Helper()
rows, err := m.db.QueryContext(context.Background(),
`SELECT id, vec FROM memory_vectors WHERE id NOT LIKE ? ESCAPE '\'`,
escapeLike(memory.NonRecallPrefix)+"%")
if err != nil {
t.Fatalf("naive scan: %v", err)
}
defer rows.Close()
var out []memory.Result
for rows.Next() {
var id string
var blob []byte
if err := rows.Scan(&id, &blob); err != nil {
t.Fatalf("naive row: %v", err)
}
out = append(out, memory.Result{ID: id, Score: dot(vec, decodeVec(blob))})
}
if err := rows.Err(); err != nil {
t.Fatalf("naive rows: %v", err)
}
sort.Slice(out, func(i, j int) bool { return out[i].Score > out[j].Score })
if topK < len(out) {
out = out[:topK]
}
return out
}
// TestMemoryStoreSearchMatchesNaive is the constraint on V-643: the bounded
// heap must return exactly what a full scan and sort returned. Distinct random
// vectors, so no two scores tie and the ranking is total — a mismatch here is
// arithmetic or heap logic, not a tie-break difference.
func TestMemoryStoreSearchMatchesNaive(t *testing.T) {
ctx := context.Background()
m := newMemTestStore(t).VectorMemory()
rng := rand.New(rand.NewSource(7))
const rows, dim = 500, 64
for i := 0; i < rows; i++ {
if err := m.Insert(ctx, fmt.Sprintf("n%d", i), randUnitVec(rng, dim), map[string]string{
"text": fmt.Sprintf("note %d", i),
}); err != nil {
t.Fatalf("Insert %d: %v", i, err)
}
}
for _, topK := range []int{1, 3, 10, 50, rows, rows + 100} {
q := randUnitVec(rng, dim)
got, err := m.Search(ctx, q, topK)
if err != nil {
t.Fatalf("Search topK=%d: %v", topK, err)
}
want := naiveSearch(t, m, q, topK)
if len(got) != len(want) {
t.Fatalf("topK=%d: got %d results, naive returned %d", topK, len(got), len(want))
}
for i := range want {
if got[i].ID != want[i].ID {
t.Errorf("topK=%d rank %d: got %q, naive says %q", topK, i, got[i].ID, want[i].ID)
}
if got[i].Score != want[i].Score {
t.Errorf("topK=%d rank %d (%s): score %v, naive says %v",
topK, i, got[i].ID, got[i].Score, want[i].Score)
}
}
if len(got) > 0 && got[0].Meta["text"] == "" {
t.Errorf("topK=%d: survivor %s has no meta — it was never unmarshalled", topK, got[0].ID)
}
}
}
// TestDotBlobMatchesDot pins the claim in dotBlob's doc comment: reading the
// vector out of its stored bytes is bit-identical to decoding it first. Scores
// feed a gate with a 0.008 margin, so "close enough" is not the bar.
func TestDotBlobMatchesDot(t *testing.T) {
rng := rand.New(rand.NewSource(11))
for i := 0; i < 200; i++ {
a := randUnitVec(rng, 384)
b := randUnitVec(rng, 384)
if got, want := dotBlob(a, encodeVec(b)), dot(a, b); got != want {
t.Fatalf("dotBlob = %v, dot = %v", got, want)
}
}
// Length mismatch is 0 in both, and so is an empty vector.
if got := dotBlob([]float32{1, 0}, encodeVec([]float32{1, 0, 0})); got != 0 {
t.Errorf("mismatched lengths scored %v, want 0", got)
}
if got := dotBlob(nil, nil); got != 0 {
t.Errorf("empty scored %v, want 0", got)
}
}
+143
View File
@@ -0,0 +1,143 @@
#!/usr/bin/env bash
# audit.sh — the repo inventory, in one command.
#
# Every section here was reconstructed by hand, from scratch, in session after
# session: 93 grep sweeps across the four longest ones before a single edit was
# made. The answers move slowly and the sweeps did not, so they are written down
# once here instead.
#
# It prints and never writes. A committed inventory file goes stale silently and
# then lies; a report you regenerate cannot.
#
# Read-only. Safe to run at any point, including mid-conflict.
#
# make audit # everything
# make audit SECTION=todo # one section: loc, todo, stubs, docs, tests, gaps
set -uo pipefail
cd "$(dirname "$0")/.." || exit 1
SECTION="${SECTION:-all}"
want() { [ "$SECTION" = all ] || [ "$SECTION" = "$1" ]; }
rule() { printf '\n=== %s %s\n' "$1" "$(printf '%.0s=' $(seq 1 $((66 - ${#1}))))"; }
# git grep over tracked files only. deps/ and models/ are gitignored and huge,
# and a plain grep -r walks into both.
g() { git grep -nI "$@" 2>/dev/null; }
printf 'Maven repo inventory @ %s (%s)\n' \
"$(git rev-parse --short HEAD 2>/dev/null || echo '?')" \
"$(git log -1 --format=%cs 2>/dev/null || echo '?')"
# --- loc -------------------------------------------------------------------
# Non-test Go lines per package. Size is the cheapest proxy for "where does the
# complexity actually sit", and it is the first thing every audit asked for.
if want loc; then
rule "PACKAGES BY LOC (non-test)"
for d in $(find ./cmd ./internal ./pkg -maxdepth 2 -type d 2>/dev/null | sort); do
files=$(find "$d" -maxdepth 1 -name '*.go' ! -name '*_test.go' 2>/dev/null)
[ -z "$files" ] && continue
n=$(printf '%s\n' "$files" | wc -l)
l=$(printf '%s\0' $files | xargs -0 cat 2>/dev/null | wc -l)
printf '%7d %3d files %s\n' "$l" "$n" "$d"
done | sort -rn
fi
# --- todo ------------------------------------------------------------------
if want todo; then
rule "TODO / FIXME / XXX / HACK / BUG (non-test)"
g -E '(^|[^a-zA-Z])(TODO|FIXME|XXX|HACK|BUG:)' -- 'cmd/**/*.go' 'internal/**/*.go' 'pkg/**/*.go' \
| grep -v '_test\.go:' | sed 's/^/ /' || echo " none"
fi
# --- stubs -----------------------------------------------------------------
# Code only, never doc comments. "not wired" is this repo's design vocabulary
# for a nil dependency and appears in ~30 comments that describe working code,
# so searching prose here reports the architecture back as a gap. Likewise
# internal/ipc/unimplemented.go is skipped whole: the file IS the deliberate
# Unimplemented*Server pattern, not 60 missing methods. "placeholder" is not a
# term here either -- it names real identifiers (SQL placeholders,
# Deck.RequirePlaceholder, tokenPlaceholder) and matched 16 working lines.
if want stubs; then
rule "STUBS / NOT IMPLEMENTED (code, non-test)"
g -iE 'not (yet )?implemented|unimplemented|пока не умею|panic\("TODO' \
-- 'cmd/**/*.go' 'internal/**/*.go' 'pkg/**/*.go' \
| grep -v '_test\.go:' \
| grep -v '^internal/ipc/unimplemented\.go:' \
| grep -vE '^[^:]+:[0-9]+:[[:space:]]*//' \
| sed 's/^/ /' || echo " none"
# CoreAPI stub parity. A stub missing from unimplemented.go breaks the build
# via `var _ CoreAPI = UnimplementedCoreAPI{}`. A stub left behind after its
# method leaves an interface compiles forever and is caught by nothing, so it
# is counted here until V-652 turns it into a test.
printf '\n CoreAPI stub parity:\n'
python3 - <<'PY' 2>/dev/null | sed 's/^/ /' || echo " (skipped: python3 unavailable)"
import re
src = open('internal/ipc/coreapi.go').read()
decl = set()
for m in re.finditer(r'type (\w+API) interface \{(.*?)\n\}', src, re.S):
decl |= set(re.findall(r'^\t([A-Z]\w*)\(', m.group(2), re.M))
stub = set(re.findall(r'func \(UnimplementedCoreAPI\) (\w+)\(',
open('internal/ipc/unimplemented.go').read()))
print(f"{len(decl)} declared, {len(stub)} stubbed")
for name in sorted(stub - decl):
print(f"STALE {name} (stubbed, on no interface)")
for name in sorted(decl - stub):
print(f"MISSING {name} (declared, no stub)")
PY
fi
# --- docs ------------------------------------------------------------------
# Living tier only. docs/evals/ are dated measurements that are never edited
# after the day, and docs/archive/ is dead by definition, so neither can be
# stale. Age is against the recorded date, not against HEAD: every commit moves
# HEAD, so a sha comparison would mark the whole tier stale every day.
if want docs; then
rule "LIVING DOCS — Last verified"
today=$(date +%s)
for f in docs/*.md; do
[ -e "$f" ] || continue
line=$(grep -m1 -o 'Last verified: *[0-9-]\{8,10\}[^ ]*\( *@ *[0-9a-f]\{7,\}\)\?' "$f" 2>/dev/null)
if [ -z "$line" ]; then
printf ' %-44s %s\n' "$(basename "$f")" "MISSING"
continue
fi
d=$(printf '%s' "$line" | grep -o '[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}' | head -1)
age=""
if [ -n "$d" ] && when=$(date -d "$d" +%s 2>/dev/null); then
days=$(( (today - when) / 86400 ))
age="${days}d"
[ "$days" -gt 30 ] && age="${days}d <-- STALE"
fi
printf ' %-44s %-34s %s\n' "$(basename "$f")" "${line#Last verified: }" "$age"
done
fi
# --- tests -----------------------------------------------------------------
if want tests; then
rule "TEST SHAPE"
printf ' benchmarks : %s\n' "$(g -c 'func Benchmark' -- '**/*_test.go' | awk -F: '{s+=$2} END{print s+0}')"
printf ' fuzz : %s\n' "$(g -c 'func Fuzz' -- '**/*_test.go' | awk -F: '{s+=$2} END{print s+0}')"
printf ' table t.Run: %s\n' "$(g -c 't.Run(' -- '**/*_test.go' | awk -F: '{s+=$2} END{print s+0}')"
printf ' golden files: %s\n' "$(g -l 'golden' -- '**/*_test.go' | wc -l)"
fi
# --- gaps ------------------------------------------------------------------
# A package with production code and no test file at all. Not a verdict — some
# are pure wiring — but it is the list worth looking at before adding more.
if want gaps; then
rule "PACKAGES WITH NO TEST FILE"
found=0
for d in $(find ./cmd ./internal ./pkg -maxdepth 2 -type d 2>/dev/null | sort); do
ls "$d"/*.go >/dev/null 2>&1 || continue
ls "$d"/*_test.go >/dev/null 2>&1 && continue
n=$(ls "$d"/*.go 2>/dev/null | wc -l)
l=$(cat "$d"/*.go 2>/dev/null | wc -l)
printf ' %6d lines %2d files %s\n' "$l" "$n" "$d"
found=1
done
[ "$found" = 0 ] && echo " none"
fi
exit 0