Compare commits

...

12 Commits

Author SHA1 Message Date
claude 08889cad88 Give the box a second reach (V-649)
Telegram was the only way off this box, and it is not a direct path: it
needs api.telegram.org, a socks relay on the host and a matching ufw rule.
Each of those three has failed once, and when they do a sev4 nudge has
nowhere to go. ntfy shares none of them.

The spare is the smaller half of it. The routing table already sends
sev3-away nudges and away reminders to ntfy and to nothing else, so with no
block configured those two routes hit a nil sink in DispatchNudge and
DispatchReminder and are skipped — no log line, no delivery_attempts row.
An away reminder is worse than dropped: out stays empty, so MarkReminder
never runs and it re-fires every tick without ever being delivered.

Owner's call, 07-08-2026: ntfy.kvmx.ru, topic maven.

The sink now takes a bearer token, which is what that server wants and what
it could not do before. ntfy scopes a token to one topic and to write-only,
so a popped sink can push to the maven topic and cannot read it back. Basic
auth stays for a server with no tokens; configuring both is refused rather
than resolved by guessing.

Config keys got json tags. docs/operations.md has documented this block as
base_url/topic since before it existed, and the untagged struct would only
have answered to BaseURL/Topic — the documented config would have parsed
into an empty one.

The token is a ${NTFY_TOKEN} expansion from the gitignored
deploy/telegram.env, beside the telegram secrets. TestDeployConfigLoads now
fails if the block goes missing, because deleting it is how you turn the
reach off and the two silent routes are what that costs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 02:16:18 +04:00
claude a4630b9314 Merge pull request 'MemoryStore.Search decodes and unmarshals every row before keeping topK' (#195) from task/643-memorystore-search-decodes-and-unmarshal into master 2026-08-07 00:09:50 +02:00
claude 39d44bb384 Close a Vikunja task with done, and nothing else (V-641)
Owner's call, 07-08-2026. A completion summary written into the
description on the way out is lost anyway, and the durable record is the
commit messages and the merged PR.

Written during the V-641 session and left uncommitted; it rides this
branch rather than being dropped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 01:48:19 +04:00
claude 65ee0f9c61 Score every row, pay for only the ten that survive (V-643)
Search decoded the vector blob into a []float32 and JSON-unmarshalled the
meta map for every row, then sorted all N and threw away everything past
topK. Meta only ever matters for a survivor, and the sort answered a
question a bounded heap answers cheaper.

The scan still visits every row — that is what picks the winners. What it
no longer does is allocate for a row it is about to discard. dotBlob reads
the vector out of its stored bytes, so scoring costs nothing; a row is
copied and its meta unmarshalled only once it has entered the topK.

At 10000 rows and topK 10: 70.6ms to 26.8ms, 58MB to 17.5MB, 240k allocs
to 60k.

Recall is unchanged where it is measured. recall+onnx scores 22/32 with
recall@1 70.4% and recall@3 85.2%, identical to before.
TestMemoryStoreSearchMatchesNaive pins the ranking against the full-sort
implementation it replaced, and TestDotBlobMatchesDot pins bit-identical
scores, which the 0.008 gate margin demands.

One behaviour did move: ties. sort.Slice is not stable, so equal scores
were ordered arbitrarily; the heap now keeps the earliest. Under the real
embedder an exact tie is a duplicate vector and nothing moved. Under the
hash embedder the eval's floor uses, everything ties at 0 and that run's
recall@3 went 74.1% to 81.5% — a number that measures tie order, not
retrieval. recall@1 and false recall, the two the eval asserts, are
unchanged on both runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 01:48:05 +04:00
claude 76938e206d Put a number on the recall scan before changing it (V-643)
MemoryStore.Search is on the per-turn recall path and had no benchmark, so
any claim about its cost was an argument rather than a measurement.

Seeds a store with rows the shape recall actually stores — 384-wide
vectors, the resident embedder's width, and a meta blob carrying the note
text — at 1000 and 10000 rows. 10000 is the ceiling the type doc claims a
full scan is fine at.

Measured as it stands: 5.3ms and 24k allocs at 1000 rows, 70.6ms and 240k
allocs at 10000.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMNNEkYx1mZFtHNrFk7uqb
2026-08-07 01:48:05 +04:00
claude 0b3d81ecbf Merge pull request 'Two maps grow for the process lifetime with no eviction' (#194) from task/641-two-maps-grow-for-the-process-lifetime-w into master 2026-08-06 23:33:23 +02:00
claude 4be6852b94 Drop host rate-limit entries that can no longer delay anything (V-641)
webfetch.Fetcher.last held one entry per distinct host the crawler ever
dialed, never pruned. Bounded in practice by how many hosts get crawled, but
crawl.on_demand is true in deploy, so the host set is whatever he names out
loud.

An entry older than HostInterval cannot delay a request — waitTurn would let
the next one straight through — so it is dropped. The sweep runs on write and
only once the map passes 64 entries, below which walking it costs more than
the entries do.

Rate limiting is unchanged: a host dialed inside the interval is kept, which
the test asserts, because pruning one would hand out a free turn.
2026-08-07 01:32:26 +04:00
claude f7b76c572f Bound the undated-item set per feed (V-641)
rss.Poller.seen held every undated item ever seen, one entry per id, for as
long as mavend ran. fresh() added and nothing removed. A feed that ships items
with no <pubDate> grew it forever.

seenIDs is the same set with a bound: the map answers the lookup, a slice
remembers insertion order, and the oldest id falls out past 512. The cap has
to stay above any one feed's front page or an item still listed there would be
written a second time, and a few hundred covers the largest page anyone
publishes. The set only ever had to span one poll window plus the resync
guard, not all of history.

Dedupe behaviour is unchanged. The comment at fresh() explains why the set
does not survive a restart; it never bounded it within one run.
2026-08-07 01:32:15 +04:00
claude 05ddc5c92e Merge pull request 'mavcaldav is built, documented as running, and deployed nowhere' (#193) from task/644-mavcaldav-is-built-documented-as-running into master 2026-08-06 23:22:18 +02:00
claude b55e68f98d Say in compose that the calendar is off, and why (V-644)
mavcaldav was built, in `make build`, listed in CLAUDE.md's daemon table, and
deployed nowhere. Not commented out the way mavmaild is, which at least
records the decision and the enable steps. Built and mentioned nowhere is the
worst of the three states, so this writes the decision down.

The box has no CalDAV account, so the block stays commented. It names what the
absence costs, because both costs are invisible from the daemon table. Agenda
questions route correctly and answer from nothing: stage 0 sends "что у меня
сегодня" to IntentQuery (V-498) and the calendar query source then reads facts
nobody writes. And loop.State.CalendarBusy is fed by those same facts, so the
gate's "do not nag mid-meeting" is permanently false.

CLAUDE.md said the absence was an oversight. It is a decision now.
2026-08-07 01:19:44 +04:00
claude beaa24754c Read the CalDAV password from a file, not from argv (V-644)
mavcaldav took -pass and -render-pass as flag values, so enabling it would
have put his calendar password in `ps` inside the container, in the compose
file, and in shell history. mavpoll and mavmaild both read their secret from
a file for exactly that reason.

readSecret reads once at start, trims, and refuses an empty or missing file.
An empty file is a deployment mistake, not a password, and basic auth would
otherwise send "" and collect a 401 every poll. A rotated password means a
restart, which is cheaper than re-reading the credential every five minutes.

Nothing called the old flags: no compose service, no systemd unit, no test.
So they are replaced rather than kept beside the new ones.
2026-08-07 01:19:33 +04:00
kami aed8cac439 Merge pull request 'The store caps sqlite at one connection under WAL, so every read queues behind every write' (#192) from task/642-the-store-caps-sqlite-at-one-connection into master 2026-08-06 23:04:39 +02:00
17 changed files with 677 additions and 44 deletions
+14 -5
View File
@@ -101,9 +101,15 @@ protocol; the config in `deploy/mavend.json` (with `${VAR}` env expansion from g
Count against compose, not against the table. Four of the nine daemons are absent, and each
absence has a different reason.
`mavmaild` is commented out in compose, with the reason written beside it: it needs a mail
account and this box has none. `mavcaldav` appears nowhere at all, and unlike the other
three that is an oversight rather than a decision (V-644).
`mavmaild` and `mavcaldav` are commented out in compose, each with the reason written
beside it: the first needs a mail account, the second a CalDAV account, and this box has
neither. `mavcaldav` used to appear nowhere at all, which was an oversight; it became a
recorded decision on 07-08-2026 (V-644). Two things ride on that absence and the block
names them. Agenda questions route to `IntentQuery` at stage 0 (V-498) and the `calendar`
query source then reads a table nobody writes. And `loop.State.CalendarBusy` is fed by the
same facts, so the gate's "do not nag mid-meeting" is permanently false. Its password is
read from a file (`-pass-file`, and `-render-pass-file` for the render collection), never
taken as a flag value, which is the rule `mavpoll` and `mavmaild` follow too.
**`mavwaked` and `mavenclient` are absent by decision, not oversight** (Vikunja #463,
`docs/plans/17-where-the-voice-loop-runs.md`).
@@ -454,8 +460,11 @@ start of a session rather than one lookup per first use:
ToolSearch("select:mcp__vikunja__list_tasks,mcp__vikunja__get_task_details,mcp__vikunja__create_task,mcp__vikunja__update_task")
```
`update_task` carrying a `description` resets `done` to false, so closing a task with a
write-up takes two calls: the description, then `done: true`.
**Close a finished task with `done: true` and nothing else** (owner's call, 07-08-2026).
Do not write a completion summary into the description on the way out. It is lost anyway,
and the durable record is the commit messages and the merged PR. Note that `update_task`
carrying a `description` resets `done` to false, which is why a write-up ever took two
calls.
## Session workflow
+35 -8
View File
@@ -50,10 +50,10 @@ func run(args []string) error {
socket := fs.String("socket", "", "core IPC socket path (required)")
url := fs.String("url", "", "CalDAV calendar URL, e.g. http://localhost:5232/kami/personal (required)")
user := fs.String("user", "", "CalDAV basic-auth username (required)")
pass := fs.String("pass", "", "CalDAV basic-auth password (required)")
passFile := fs.String("pass-file", "", "file holding the CalDAV basic-auth password (required — never passed as a flag value)")
renderURL := fs.String("render-url", "", "CalDAV collection maven publishes her own reminders to; empty disables rendering")
renderUser := fs.String("render-user", "", "basic-auth username for -render-url (defaults to -user)")
renderPass := fs.String("render-pass", "", "basic-auth password for -render-url (defaults to -pass)")
renderPassFile := fs.String("render-pass-file", "", "file holding the password for -render-url (defaults to -pass-file)")
renderDur := fs.Duration("render-duration", calendar.DefaultReminderDuration, "how long a rendered reminder occupies")
interval := fs.Duration("interval", 5*time.Minute, "poll cadence")
timeout := fs.Duration("timeout", 10*time.Second, "per-request HTTP timeout")
@@ -63,13 +63,22 @@ func run(args []string) error {
if *socket == "" {
return fmt.Errorf("-socket is required")
}
if *url == "" || *user == "" || *pass == "" {
return fmt.Errorf("-url, -user, -pass are required")
if *url == "" || *user == "" || *passFile == "" {
return fmt.Errorf("-url, -user, -pass-file are required")
}
if err := checkRenderTarget([]string{*url}, *renderURL); err != nil {
return err
}
// The password is read from a file, never taken as a flag value: an argv
// secret is visible in `ps` to every user on the box and lands in the compose
// file and the shell history. Same rule mavmaild and mavpoll follow. Read
// once at start, so a rotated password means a restart.
pass, err := readSecret(*passFile)
if err != nil {
return err
}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
@@ -85,17 +94,20 @@ func run(args []string) error {
http: hc,
url: strings.TrimRight(*url, "/"),
user: *user,
pass: *pass,
pass: pass,
}
var rend *renderer
if *renderURL != "" {
ru, rp := *renderUser, *renderPass
ru, rp := *renderUser, pass
if ru == "" {
ru = *user
}
if rp == "" {
rp = *pass
if *renderPassFile != "" {
rp, err = readSecret(*renderPassFile)
if err != nil {
return err
}
}
rend = newRenderer(core, hc, *renderURL, ru, rp, *renderDur)
log.Printf("mavcaldav: rendering reminders to %s", *renderURL)
@@ -131,6 +143,21 @@ func run(args []string) error {
// It takes the whole read set, not one URL. The guarantee in the package
// comment is about every calendar maven reads, and a second read target added
// later must not quietly fall outside the check.
// readSecret reads one credential from a file and refuses an empty one. An
// empty file is a deployment mistake, not a password, and CalDAV basic auth
// would send it and get a 401 every poll.
func readSecret(path string) (string, error) {
raw, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("read password file: %w", err)
}
secret := strings.TrimSpace(string(raw))
if secret == "" {
return "", fmt.Errorf("password file %s is empty", path)
}
return secret, nil
}
func checkRenderTarget(readURLs []string, renderURL string) error {
if renderURL == "" {
return nil
+26
View File
@@ -5,12 +5,38 @@ import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"time"
"github.com/kami/maven/internal/ipc"
)
// The password comes from a file so it never reaches argv. An empty or missing
// file must fail at start rather than authenticate as "" against his calendar.
func TestReadSecret(t *testing.T) {
dir := t.TempDir()
good := filepath.Join(dir, "ok")
if err := os.WriteFile(good, []byte(" hunter2\n"), 0o600); err != nil {
t.Fatal(err)
}
if got, err := readSecret(good); err != nil || got != "hunter2" {
t.Fatalf("readSecret(good) = %q, %v; want \"hunter2\", nil", got, err)
}
empty := filepath.Join(dir, "empty")
if err := os.WriteFile(empty, []byte("\n \n"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := readSecret(empty); err == nil {
t.Fatal("readSecret(empty) = nil error, want refusal")
}
if _, err := readSecret(filepath.Join(dir, "absent")); err == nil {
t.Fatal("readSecret(absent) = nil error, want refusal")
}
}
type fakeCore struct {
ipc.UnimplementedCoreAPI
facts map[string]ipc.Fact // composite key "key|source" → Fact
+19
View File
@@ -25,6 +25,25 @@
"llm_nudges": false
},
"//ntfy": [
"The second reach (V-649). Until 07-08-2026 telegram was the only one, and",
"telegram needs api.telegram.org, the socks relay below and a matching ufw",
"rule — three things in series that have each failed once, and when they do",
"a sev4 nudge has nowhere to go. ntfy shares none of them: it is reached",
"directly, no relay.",
"It is not only a spare. The routing table sends sev3-away and away",
"reminders here and NOWHERE else, so with this block absent those two",
"routes hit a nil sink and vanish without a log or an outbox row.",
"The credential is an ntfy access token, scoped write-only to this one",
"topic, so a popped sink can push to it and cannot read it back. Set it in",
"deploy/telegram.env beside the telegram secrets; that file is gitignored."
],
"ntfy": {
"base_url": "https://ntfy.kvmx.ru",
"topic": "maven",
"token": "${NTFY_TOKEN}"
},
"telegram": {
"bot_token": "${TELEGRAM_BOT_TOKEN}",
"chat_id": "${TELEGRAM_CHAT_ID}",
+8 -1
View File
@@ -1,5 +1,12 @@
# Telegram bot token and chat ID for mavend's away-channel reach.
# Secrets for mavend's away-channel reaches. The file is still called
# telegram.env because compose names it that; it holds both reaches now.
# Copy this file to deploy/telegram.env and fill in real values.
# deploy/telegram.env is gitignored — never commit the real secrets.
TELEGRAM_BOT_TOKEN=
TELEGRAM_CHAT_ID=
# ntfy access token for the `maven` topic, the second reach (V-649). Mint it on
# the ntfy server with write access to that topic and nothing else:
# ntfy token add --expires=never maven
# Read access is not needed — mavend publishes and never subscribes.
NTFY_TOKEN=
+38
View File
@@ -157,6 +157,44 @@ services:
# - maildata:/var/lib/mavmaild
# - ./deploy/imap.password:/run/secrets/imap.password:ro
# The calendar reader (Vikunja #644) is OFF and commented out: it needs a
# CalDAV account, and there is none on this box. It was built, listed in
# `make build`, and deployed nowhere, which is the worst of the three states —
# this block records the decision instead.
#
# What its absence costs, so the cost is visible from here:
# - Agenda questions route correctly and answer from nothing. Stage 0 sends
# "что у меня сегодня" to IntentQuery (V-498) and the `calendar` query
# source reads facts(kind=env, source=caldav:*) that nobody writes.
# - The nudge gate loses a suppressor. loop.State.CalendarBusy is fed by
# those same facts, so "do not nag mid-meeting" is permanently false.
#
# Core never sees the CalDAV password: the reader polls the collection itself
# and hands core one fact per event over WriteFact. Nothing here can create a
# reminder, so a misread event cannot fire.
#
# The password is read from a FILE, so it never appears in `ps`, in this file,
# or in shell history — the same rule mavpoll and mavmaild follow.
#
# To enable: write the password to deploy/caldav.password (0600, gitignored),
# point -url at the collection, and uncomment this service. No mavend.json
# block is needed — events arrive over IPC as facts. -render-url is optional
# and OFF here: it publishes Maven's own reminders back as events, and it must
# not name the collection -url reads, or the poller reads its own writes back
# in (checkRenderTarget refuses that). It takes -render-pass-file, and falls
# back to this password when that is not given.
# mavcaldav:
# <<: *image
# command: ["mavcaldav", "-socket", "/run/maven/mavend.sock",
# "-url", "http://localhost:5232/kami/personal",
# "-user", "kami",
# "-pass-file", "/run/secrets/caldav.password",
# "-interval", "5m"]
# depends_on: [mavend]
# volumes:
# - sockets:/run/maven
# - ./deploy/caldav.password:/run/secrets/caldav.password:ro
volumes:
dbdata:
sockets:
+12 -2
View File
@@ -1,6 +1,6 @@
# Start Commands
*Last verified: 2026-08-02 @ 7079a24. Living doc: correct it in place, do not append.*
*Last verified: 2026-08-07 @ a4630b9. Living doc: correct it in place, do not append.*
All commands assume `ROOT=/home/kami/apps/Maven` and the local Go toolchain at `$ROOT/deps/go/go/bin/go`.
@@ -44,7 +44,8 @@ Config path: `~/.config/maven/mavend.json`. Full example with all options.
"repeat_interval": "5m",
"ntfy": {
"base_url": "https://ntfy.kvmx.ru",
"topic": "maven"
"topic": "maven",
"token": "${NTFY_TOKEN}"
},
"phraser": {
"model_path": "/mnt/hdd1/llms/Qwen3-Maven-1.7B-Q8_0.gguf",
@@ -66,6 +67,15 @@ Config path: `~/.config/maven/mavend.json`. Full example with all options.
Omit the `embedder` block entirely to use the deterministic HashEmbedder floor (no ML, no ONNX runtime dependency). Useful for testing or low-resource setups.
`${NTFY_TOKEN}` and the `${TELEGRAM_*}` vars are expanded from `deploy/telegram.env`, which is gitignored. Copy `deploy/telegram.env.example` and fill it in. Mint a scoped token rather than reusing an admin one. It needs write access to the `maven` topic and nothing else:
```sh
ntfy access maven maven write-only
ntfy token add --expires=never maven
```
Deleting the `ntfy` block turns the reach off, and that is not a no-op. The routing table sends sev3-away nudges and away reminders to ntfy and nowhere else. With no sink wired they hit a nil and vanish, leaving no log line and no `delivery_attempts` row (V-649).
## mavsttd — STT worker (optional, remote whisper.cpp)
Requires `LD_LIBRARY_PATH` to include deps/lib (for libwhisper.so, libggml-vulkan.so).
+13
View File
@@ -45,4 +45,17 @@ func TestDeployConfigLoads(t *testing.T) {
if cfg.Voice.RouterThreshold <= 0 {
t.Error("router threshold did not get its default")
}
// The second reach (V-649). Deleting this block is how you turn ntfy off,
// so its absence has to be loud: sev3-away nudges and away reminders route
// to ntfy and to nothing else, and a nil sink drops them with no log and no
// outbox row. The token is a ${VAR} that CI cannot resolve, so this checks
// the wiring and not the credential.
if cfg.Ntfy == nil {
t.Fatal("deploy config has no ntfy block — sev3-away and away reminders " +
"would have nowhere to land, and would vanish silently rather than fail")
}
if cfg.Ntfy.BaseURL == "" || cfg.Ntfy.Topic == "" {
t.Errorf("ntfy block is incomplete: base_url=%q topic=%q", cfg.Ntfy.BaseURL, cfg.Ntfy.Topic)
}
}
+43 -11
View File
@@ -7,11 +7,17 @@
// the relay). the dispatcher already strips detail off away sendables; the
// sink uses the same helper so it can't leak the body on its own either.
//
// ntfy runs locally (docker, 127.0.0.1:8085, deny-all auth). maven publishes
// with a dedicated user (write-only to maven-* topics) — the credential is a
// delivery-config secret, not a db key; a popped ntfy sink can push spam to
// your phone, nothing else. matches the module key-isolation invariant: the
// sink never holds the sqlcipher key.
// ntfy is a self-hosted server with deny-all auth — ntfy.kvmx.ru as of
// 07-08-2026, reached directly, not through the socks relay telegram needs.
// maven publishes with a write-only token scoped to its own topic; the
// credential is a delivery-config secret, not a db key. a popped ntfy sink
// can push spam to that one topic, nothing else — it cannot read the topic
// back and it never holds the sqlcipher key.
//
// this is the second reach, and the reason there is one is that telegram was
// the only one (V-649). telegram needs api.telegram.org, a socks relay on the
// host and a matching ufw rule, three things in series that have each broken
// once. ntfy shares none of them.
package ntfysink
import (
@@ -31,11 +37,29 @@ import (
// the credential lives in the daemon's config (or a systemd credential),
// never in the binary.
type Config struct {
BaseURL string // e.g. http://127.0.0.1:8085 (no trailing path)
Topic string // e.g. maven (all maven notifications land here)
Username string // basic auth; empty = anonymous (won't work with deny-all)
Password string // basic auth
Timeout time.Duration // per-request; 0 = DefaultTimeout
// BaseURL — the ntfy server, no trailing path. Required.
BaseURL string `json:"base_url"`
// Topic — where maven publishes. Required. All maven notifications land
// on this one topic; severity rides the Priority header, not the topic.
Topic string `json:"topic"`
// Token — an ntfy access token, sent as a bearer. This is the preferred
// credential: ntfy scopes a token to a topic and to write-only, so a
// popped sink can push to this one topic and cannot read it back or
// touch another. Revoking it does not disturb a password anyone else
// uses. Mutually exclusive with Username.
Token string `json:"token,omitempty"`
// Username, Password — basic auth, for a server that has no tokens.
// Empty username means no credential is sent at all, which a deny-all
// server rejects.
Username string `json:"username,omitempty"`
Password string `json:"password,omitempty"`
// Timeout — per-request; 0 = DefaultTimeout. A dead server must not hang
// the tick loop.
Timeout time.Duration `json:"-"`
}
const DefaultTimeout = 10 * time.Second
@@ -59,6 +83,12 @@ func New(cfg Config) (*Sink, error) {
if cfg.Topic == "" {
return nil, fmt.Errorf("ntfysink: Topic is required")
}
// Refuse rather than pick. Two credentials configured means someone
// intended one of them, and guessing which would send the other nowhere
// and leave a working config that is not the one they wrote.
if cfg.Token != "" && cfg.Username != "" {
return nil, fmt.Errorf("ntfysink: set Token or Username, not both")
}
to := cfg.Timeout
if to == 0 {
to = DefaultTimeout
@@ -84,7 +114,9 @@ func (s *Sink) Send(ctx context.Context, d delivery.Sendable) error {
}
req.Header.Set("Title", "maven")
req.Header.Set("Priority", priorityFor(d).String())
if s.cfg.Username != "" {
if s.cfg.Token != "" {
req.Header.Set("Authorization", "Bearer "+s.cfg.Token)
} else if s.cfg.Username != "" {
req.SetBasicAuth(s.cfg.Username, s.cfg.Password)
}
@@ -224,6 +224,37 @@ func TestSendNoAuthWhenUsernameEmpty(t *testing.T) {
}
}
// TestSendSetsBearerToken — the deployed credential (V-649) is an ntfy access
// token scoped write-only to the maven topic, not a password. A token sent as
// basic auth is rejected by ntfy, so the header shape is the whole test.
func TestSendSetsBearerToken(t *testing.T) {
rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler())
defer srv.Close()
sink, _ := New(Config{BaseURL: srv.URL, Topic: "maven", Token: "tk_secret"})
if err := sink.Send(context.Background(), nudgeSendable(loop.Sev3, "down")); err != nil {
t.Fatalf("Send: %v", err)
}
_, _, _, auth, _, _ := rs.snapshot()
if auth != "Bearer tk_secret" {
t.Fatalf("auth: want 'Bearer tk_secret', got %q", auth)
}
}
// TestNewRejectsBothCredentials — configuring a token and a username means one
// of them was meant and the other is a leftover. Picking either would leave a
// server that authenticates against a credential nobody wrote down.
func TestNewRejectsBothCredentials(t *testing.T) {
_, err := New(Config{BaseURL: "http://x", Topic: "maven", Token: "tk_x", Username: "maven"})
if err == nil {
t.Fatal("New accepted both a token and a username")
}
if !strings.Contains(err.Error(), "not both") {
t.Errorf("error does not say which to fix: %v", err)
}
}
func TestSendTitleIsMaven(t *testing.T) {
rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler())
+37 -6
View File
@@ -89,8 +89,8 @@ type Poller struct {
ranker Ranker
cfg Config
nextDue map[string]time.Time
seen map[string]map[string]bool // feed → item ID, for items with no date
polled map[string]bool // feed → polled at least once in THIS process
seen map[string]*seenIDs // feed → item IDs, for items with no date
polled map[string]bool // feed → polled at least once in THIS process
}
// NewPoller wires a poller. Returns nil when there is nothing to poll — a
@@ -122,7 +122,7 @@ func NewPoller(feeds []FeedConfig, fetch Fetcher, notes Notes, marks Marks, embe
feeds: valid, fetch: fetch, notes: notes, marks: marks,
embed: embed, ranker: ranker, cfg: cfg,
nextDue: map[string]time.Time{},
seen: map[string]map[string]bool{},
seen: map[string]*seenIDs{},
polled: map[string]bool{},
}
}
@@ -283,6 +283,38 @@ func (p *Poller) mark(ctx context.Context, feed string, now time.Time) (time.Tim
return at, true
}
// maxSeenPerFeed bounds the undated-item set. It has to stay comfortably above
// any one feed's front page, or an item still listed there would fall out of the
// set and be written a second time. A few hundred entries covers the largest
// page anyone publishes, and the set only has to span one poll window plus the
// resync guard, not all of history.
const maxSeenPerFeed = 512
// seenIDs is a bounded insertion-ordered set. The map answers the lookup, the
// slice remembers what to drop first, so an undated feed cannot grow the poller
// for as long as mavend runs.
type seenIDs struct {
ids map[string]bool
order []string
}
// add records id and reports whether it was new.
func (s *seenIDs) add(id string) bool {
if s.ids == nil {
s.ids = make(map[string]bool, maxSeenPerFeed)
}
if s.ids[id] {
return false
}
s.ids[id] = true
s.order = append(s.order, id)
if len(s.order) > maxSeenPerFeed {
delete(s.ids, s.order[0])
s.order = s.order[1:]
}
return true
}
// fresh — two dedup rules, because feeds are inconsistent about dates. A dated
// item must be newer than the mark; an undated one is kept once per process by
// ID.
@@ -308,12 +340,11 @@ func (p *Poller) fresh(f FeedConfig, it Item, mark, now time.Time, resync bool)
id = it.Title
}
if p.seen[f.Name] == nil {
p.seen[f.Name] = map[string]bool{}
p.seen[f.Name] = &seenIDs{}
}
if p.seen[f.Name][id] {
if !p.seen[f.Name].add(id) {
return false
}
p.seen[f.Name][id] = true
return !resync
}
+25
View File
@@ -3,6 +3,7 @@ package rss
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
@@ -208,3 +209,27 @@ func TestNoFeedsMeansNoPoller(t *testing.T) {
t.Fatal("a feed with no name or url is not a configuration")
}
}
// An undated feed used to grow p.seen for as long as mavend ran. The set is
// bounded now, and the bound must not cost the dedupe an item still on the
// front page — only ids far older than any page fall out.
func TestSeenIDsBounded(t *testing.T) {
var s seenIDs
for i := 0; i < maxSeenPerFeed*3; i++ {
if !s.add(fmt.Sprintf("item-%d", i)) {
t.Fatalf("item-%d read as already seen", i)
}
if len(s.ids) > maxSeenPerFeed || len(s.order) > maxSeenPerFeed {
t.Fatalf("after %d inserts: ids=%d order=%d, cap is %d",
i+1, len(s.ids), len(s.order), maxSeenPerFeed)
}
}
// The newest insert is still deduped; the oldest was evicted.
last := fmt.Sprintf("item-%d", maxSeenPerFeed*3-1)
if s.add(last) {
t.Fatalf("%s read as new, so the most recent id was dropped", last)
}
if !s.add("item-0") {
t.Fatal("item-0 survived, so nothing was evicted")
}
}
+135 -11
View File
@@ -68,6 +68,13 @@ func (m *MemoryStore) Insert(ctx context.Context, id string, vec []float32, meta
// Rows under memory.NonRecallPrefix are excluded in SQL. They are speaker
// voiceprints sharing this table, and note recall must not rank them; see that
// constant for why the previous arrangement only appeared to do this.
//
// Every row is still scored, because a full scan is what picks the winners.
// What the scan does NOT do is pay for a row it is about to discard: the score
// is read straight off the stored bytes without materializing a []float32, and
// the meta blob is copied and unmarshalled only for a row that has entered the
// topK. Losers cost one dot product and nothing else. Ranking is unchanged —
// same scores, same order, same ties.
func (m *MemoryStore) Search(ctx context.Context, vec []float32, topK int) ([]memory.Result, error) {
if topK <= 0 {
topK = 10
@@ -80,30 +87,127 @@ func (m *MemoryStore) Search(ctx context.Context, vec []float32, topK int) ([]me
}
defer rows.Close()
var out []memory.Result
// sql.RawBytes hands us the driver's own buffer, valid only until the next
// Next(). Nothing here outlives the row except what topK.offer copies on a
// survivor, so the three columns cost no allocation per row.
var id, blob, metaJSON sql.RawBytes
top := newTopK(topK)
for rows.Next() {
var id, metaJSON string
var blob []byte
if err := rows.Scan(&id, &blob, &metaJSON); err != nil {
return nil, fmt.Errorf("memory: row: %w", err)
}
meta := map[string]string{}
if err := json.Unmarshal([]byte(metaJSON), &meta); err != nil {
return nil, fmt.Errorf("memory: unmarshal meta for %q: %w", id, err)
}
out = append(out, memory.Result{ID: id, Score: dot(vec, decodeVec(blob)), Meta: meta})
top.offer(dotBlob(vec, blob), id, metaJSON)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("memory: rows: %w", err)
}
sort.Slice(out, func(i, j int) bool { return out[i].Score > out[j].Score })
if topK < len(out) {
out = out[:topK]
survivors := top.sorted()
out := make([]memory.Result, 0, len(survivors))
for _, c := range survivors {
meta := map[string]string{}
if err := json.Unmarshal(c.meta, &meta); err != nil {
return nil, fmt.Errorf("memory: unmarshal meta for %q: %w", c.id, err)
}
out = append(out, memory.Result{ID: c.id, Score: c.score, Meta: meta})
}
return out, nil
}
// candidate is one row that is currently in the topK: its score, its id, and
// its meta blob copied out of the driver's buffer. The copy is the price of
// surviving, and only survivors pay it.
type candidate struct {
score float64
id string
meta []byte
}
// topK keeps the k highest-scoring candidates seen so far as a min-heap, so the
// weakest survivor is always heap[0] and one comparison decides whether a new
// row is worth copying. k is 10 in practice, so the heap is tiny and the whole
// structure fits in cache.
//
// It is a plain slice with hand-written sift operations rather than
// container/heap, because that interface boxes every element into an `any` on
// Push and costs an allocation per surviving row.
type topK struct {
k int
heap []candidate
}
func newTopK(k int) *topK {
return &topK{k: k, heap: make([]candidate, 0, k)}
}
// offer admits a row if it beats the weakest survivor, or if the heap is not
// full yet. id and meta are the driver's buffers and are copied here, never
// retained.
//
// A row that only ties the weakest survivor does not displace it, so among
// equal scores the earliest k rows are kept. The full sort this replaced used
// sort.Slice, which is not stable, so it broke such a tie arbitrarily. That is
// the ONE observable difference between the two, and it is deliberate:
// deterministic beats arbitrary.
//
// It is not academic. Under the real embedder an exact tie means duplicate
// vectors and nothing in the recall eval moved (V-643). Under the hash
// embedder the eval's deterministic floor uses, ties are everywhere — it is
// bag-of-words, so every note sharing no word with the query scores exactly 0
// — and recall@3 on that run moved 74.1% to 81.5% purely because the zeros now
// come out in a fixed order. Neither number measures retrieval. recall@1 and
// false recall, which the eval actually asserts, are unchanged on both runs.
func (t *topK) offer(score float64, id, meta []byte) {
if t.k == 0 {
return
}
if len(t.heap) < t.k {
t.heap = append(t.heap, candidate{score: score, id: string(id), meta: append([]byte(nil), meta...)})
t.up(len(t.heap) - 1)
return
}
if score <= t.heap[0].score {
return
}
t.heap[0] = candidate{score: score, id: string(id), meta: append([]byte(nil), meta...)}
t.down(0)
}
func (t *topK) up(i int) {
for i > 0 {
parent := (i - 1) / 2
if t.heap[parent].score <= t.heap[i].score {
return
}
t.heap[parent], t.heap[i] = t.heap[i], t.heap[parent]
i = parent
}
}
func (t *topK) down(i int) {
for {
l, r, small := 2*i+1, 2*i+2, i
if l < len(t.heap) && t.heap[l].score < t.heap[small].score {
small = l
}
if r < len(t.heap) && t.heap[r].score < t.heap[small].score {
small = r
}
if small == i {
return
}
t.heap[small], t.heap[i] = t.heap[i], t.heap[small]
i = small
}
}
// sorted drains the heap into descending score order — what Search returns.
func (t *topK) sorted() []candidate {
out := t.heap
sort.Slice(out, func(i, j int) bool { return out[i].score > out[j].score })
return out
}
// ByPrefix returns every row whose id starts with prefix, vectors included.
//
// This is not a similarity query and deliberately does not score anything:
@@ -240,6 +344,26 @@ func decodeVec(b []byte) []float32 {
return v
}
// dotBlob is dot against a vector still in its stored encoding, so scoring a
// row the query is about to discard does not allocate the []float32 that
// decodeVec would build. Same arithmetic, same order of operations, so it
// returns bit-identical scores to dot(a, decodeVec(b)).
//
// A blob whose length isn't a multiple of 4 is truncated to the whole-element
// prefix, matching decodeVec, and a length mismatch is 0, matching dot.
func dotBlob(a []float32, b []byte) float64 {
n := len(b) / 4
if len(a) != n || n == 0 {
return 0
}
var sum float64
for i := 0; i < n; i++ {
f := math.Float32frombits(binary.LittleEndian.Uint32(b[4*i:]))
sum += float64(a[i]) * float64(f)
}
return sum
}
// dot is the cosine similarity for L2-normalized vectors (mismatched lengths ⇒
// 0, matching internal/memory's cosine).
func dot(a, b []float32) float64 {
+77
View File
@@ -0,0 +1,77 @@
package store
import (
"context"
"fmt"
"math"
"math/rand"
"path/filepath"
"testing"
)
// benchDim is the resident embedder's width (multilingual-e5-small, 384), so
// the per-row decode cost the benchmark measures is the real one.
const benchDim = 384
// seedMemVectors fills a fresh store with n L2-normalized rows carrying a meta
// blob the size recall actually stores — the note text plus its type — because
// the cost this benchmark exists to measure is unmarshalling that blob for
// every row when only topK survivors need it.
func seedMemVectors(tb testing.TB, n int) *MemoryStore {
tb.Helper()
path := filepath.Join(tb.TempDir(), "mem_bench.db")
st, err := Open(context.Background(), path)
if err != nil {
tb.Fatalf("Open: %v", err)
}
tb.Cleanup(func() { _ = st.Close() })
m := st.VectorMemory()
rng := rand.New(rand.NewSource(1))
ctx := context.Background()
for i := 0; i < n; i++ {
if err := m.Insert(ctx, fmt.Sprintf("note:%d", i), randUnitVec(rng, benchDim), map[string]string{
"type": "note",
"text": fmt.Sprintf("заметка номер %d о том, что надо не забыть сделать на неделе", i),
}); err != nil {
tb.Fatalf("Insert %d: %v", i, err)
}
}
return m
}
func randUnitVec(rng *rand.Rand, dim int) []float32 {
v := make([]float32, dim)
var norm float64
for i := range v {
f := rng.NormFloat64()
v[i] = float32(f)
norm += f * f
}
norm = math.Sqrt(norm)
for i := range v {
v[i] = float32(float64(v[i]) / norm)
}
return v
}
// BenchmarkMemoryStoreSearch measures one recall query against a store of n
// rows. Row counts bracket the documented scale: 1000 is a plausible today,
// 10000 is the "thousands, not millions" ceiling the type doc claims a full
// scan is fine at.
func BenchmarkMemoryStoreSearch(b *testing.B) {
for _, n := range []int{1000, 10000} {
b.Run(fmt.Sprintf("rows=%d", n), func(b *testing.B) {
m := seedMemVectors(b, n)
q := randUnitVec(rand.New(rand.NewSource(2)), benchDim)
ctx := context.Background()
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := m.Search(ctx, q, 10); err != nil {
b.Fatal(err)
}
}
})
}
}
+107
View File
@@ -0,0 +1,107 @@
package store
import (
"context"
"fmt"
"math/rand"
"sort"
"testing"
"github.com/kami/maven/internal/memory"
)
// naiveSearch is the implementation Search replaced: score every row into a
// slice, sort the whole slice, truncate. It stays in the test file as the
// reference the bounded-heap version is judged against, because "recall must
// not change" is a claim about output, not about the code that produces it.
func naiveSearch(t *testing.T, m *MemoryStore, vec []float32, topK int) []memory.Result {
t.Helper()
rows, err := m.db.QueryContext(context.Background(),
`SELECT id, vec FROM memory_vectors WHERE id NOT LIKE ? ESCAPE '\'`,
escapeLike(memory.NonRecallPrefix)+"%")
if err != nil {
t.Fatalf("naive scan: %v", err)
}
defer rows.Close()
var out []memory.Result
for rows.Next() {
var id string
var blob []byte
if err := rows.Scan(&id, &blob); err != nil {
t.Fatalf("naive row: %v", err)
}
out = append(out, memory.Result{ID: id, Score: dot(vec, decodeVec(blob))})
}
if err := rows.Err(); err != nil {
t.Fatalf("naive rows: %v", err)
}
sort.Slice(out, func(i, j int) bool { return out[i].Score > out[j].Score })
if topK < len(out) {
out = out[:topK]
}
return out
}
// TestMemoryStoreSearchMatchesNaive is the constraint on V-643: the bounded
// heap must return exactly what a full scan and sort returned. Distinct random
// vectors, so no two scores tie and the ranking is total — a mismatch here is
// arithmetic or heap logic, not a tie-break difference.
func TestMemoryStoreSearchMatchesNaive(t *testing.T) {
ctx := context.Background()
m := newMemTestStore(t).VectorMemory()
rng := rand.New(rand.NewSource(7))
const rows, dim = 500, 64
for i := 0; i < rows; i++ {
if err := m.Insert(ctx, fmt.Sprintf("n%d", i), randUnitVec(rng, dim), map[string]string{
"text": fmt.Sprintf("note %d", i),
}); err != nil {
t.Fatalf("Insert %d: %v", i, err)
}
}
for _, topK := range []int{1, 3, 10, 50, rows, rows + 100} {
q := randUnitVec(rng, dim)
got, err := m.Search(ctx, q, topK)
if err != nil {
t.Fatalf("Search topK=%d: %v", topK, err)
}
want := naiveSearch(t, m, q, topK)
if len(got) != len(want) {
t.Fatalf("topK=%d: got %d results, naive returned %d", topK, len(got), len(want))
}
for i := range want {
if got[i].ID != want[i].ID {
t.Errorf("topK=%d rank %d: got %q, naive says %q", topK, i, got[i].ID, want[i].ID)
}
if got[i].Score != want[i].Score {
t.Errorf("topK=%d rank %d (%s): score %v, naive says %v",
topK, i, got[i].ID, got[i].Score, want[i].Score)
}
}
if len(got) > 0 && got[0].Meta["text"] == "" {
t.Errorf("topK=%d: survivor %s has no meta — it was never unmarshalled", topK, got[0].ID)
}
}
}
// TestDotBlobMatchesDot pins the claim in dotBlob's doc comment: reading the
// vector out of its stored bytes is bit-identical to decoding it first. Scores
// feed a gate with a 0.008 margin, so "close enough" is not the bar.
func TestDotBlobMatchesDot(t *testing.T) {
rng := rand.New(rand.NewSource(11))
for i := 0; i < 200; i++ {
a := randUnitVec(rng, 384)
b := randUnitVec(rng, 384)
if got, want := dotBlob(a, encodeVec(b)), dot(a, b); got != want {
t.Fatalf("dotBlob = %v, dot = %v", got, want)
}
}
// Length mismatch is 0 in both, and so is an empty vector.
if got := dotBlob([]float32{1, 0}, encodeVec([]float32{1, 0, 0})); got != 0 {
t.Errorf("mismatched lengths scored %v, want 0", got)
}
if got := dotBlob(nil, nil); got != 0 {
t.Errorf("empty scored %v, want 0", got)
}
}
+22
View File
@@ -295,6 +295,27 @@ func (f *Fetcher) checkURL(u *url.URL) error {
return nil
}
// pruneHostsAbove is when pruneLocked bothers to walk the map. Below it the
// walk costs more than the entries do, and `crawl.on_demand` means the host set
// is whatever he names out loud, so it grows slowly.
const pruneHostsAbove = 64
// pruneLocked drops hosts whose last dial is further back than HostInterval.
// Such an entry cannot delay anything — waitTurn would let the next request
// through immediately — so keeping it only holds memory for the life of the
// process. Caller holds f.mu.
func (f *Fetcher) pruneLocked(now time.Time) {
if len(f.last) <= pruneHostsAbove {
return
}
cutoff := now.Add(-f.cfg.HostInterval)
for h, at := range f.last {
if at.Before(cutoff) {
delete(f.last, h)
}
}
}
// waitTurn blocks until this host's rate-limit interval has elapsed. It holds
// no lock while sleeping, so two hosts never wait on each other.
func (f *Fetcher) waitTurn(ctx context.Context, host string) error {
@@ -304,6 +325,7 @@ func (f *Fetcher) waitTurn(ctx context.Context, host string) error {
earliest := f.last[host].Add(f.cfg.HostInterval)
if !now.Before(earliest) {
f.last[host] = now
f.pruneLocked(now)
f.mu.Unlock()
return nil
}
+35
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net"
"net/http"
@@ -318,3 +319,37 @@ func TestPostObeysDenylist(t *testing.T) {
t.Fatalf("error = %v, want ErrBlocked", err)
}
}
// f.last used to hold one entry per host ever dialed, for the life of the
// process. A host whose last dial is older than HostInterval cannot delay
// anything, so it is dropped once the map is worth walking.
func TestHostRateMapIsPruned(t *testing.T) {
f := New(Config{HostInterval: time.Minute, AllowPrivate: true})
stale := time.Now().Add(-time.Hour)
for i := 0; i < pruneHostsAbove*2; i++ {
f.last[fmt.Sprintf("h%d.example", i)] = stale
}
// One real turn is what triggers the sweep.
if err := f.waitTurn(context.Background(), "fresh.example"); err != nil {
t.Fatal(err)
}
if len(f.last) != 1 {
t.Fatalf("len(f.last) = %d after the sweep, want 1 (only the host just dialed)", len(f.last))
}
if _, ok := f.last["fresh.example"]; !ok {
t.Fatal("the host just dialed was pruned, so its own rate limit is lost")
}
// A host inside the interval is kept: pruning must not hand out a free turn.
f.last["recent.example"] = time.Now()
for i := 0; i < pruneHostsAbove*2; i++ {
f.last[fmt.Sprintf("g%d.example", i)] = stale
}
if err := f.waitTurn(context.Background(), "other.example"); err != nil {
t.Fatal(err)
}
if _, ok := f.last["recent.example"]; !ok {
t.Fatal("a host dialed inside HostInterval was pruned")
}
}