Add background memory evaluation, off unless configured #55

2026-07-31T23:46:10+02:00

claude commented

2026-07-31 23:46:10 +02:00

Ships the part of docs/plans/03-memory-evaluation.md that is real, local and testable now. Maven reads back her own recent memory on a slow ticker, asks the resident model what it notices, and records the confident answers as notes.

What changed

internal/memeval (new package). Evaluator.Evaluate(ctx, now) reads RecentFacts / RecentNotes / RecentNudges, prompts the resident model under a GBNF grammar bounded to three {observation, confidence, suggested_action} objects, drops anything under min_confidence, deduplicates against what earlier evaluations wrote, and records the rest as notes with source infer:memory-eval.
cmd/mavend/memoryeval.go — the driver: its own goroutine on its own ticker, wired in main.go beside the fact-enrichment worker.
internal/config — new memory_eval block (interval, max_items, min_confidence).

Not internal/memory/eval.go as the plan says: internal/store imports internal/memory for the vector-store backend, and an evaluator has to read store.Fact/Note/Nudge, which would close the import cycle. Hence a sibling package.

Visibility comes free — /dash already renders notes with their source, so evaluation output shows up with no UI change.

Off unless configured

No memory_eval block ⇒ the goroutine does not exist. deploy/mavend.json is unchanged, so this branch changes nothing about a running deployment until someone adds the block. No LLM phraser also means no loop: there is no template fallback, because a "memory evaluation" assembled from string templates is a fixed sentence pretending to be an observation.

What it deliberately cannot do

This is the feature most likely to turn Maven into a nag — an hourly loop with an LLM in it and permission to talk — so the restraint is structural, not conventional:

It cannot speak. No dispatcher reference, no channel, no nudge. An observation is a thought she wrote down; he reads it on /dash when he wants to. Wiring observations to delivery.Dispatcher is a separate decision with its own opt-in and is not in this branch.
It cannot act. suggested_action is recorded inside the note text and interpreted by nobody. No reminder, routine or fact is created.
It says nothing about an empty store. No memory ⇒ no LLM call, so no observations are invented out of two facts.
Its own notes are not input, and are written with a nil embedding so they stay out of the RAG recall pool. Feeding generated text back into the pool it came from is how a small model starts citing its own guesses as evidence.

Deferred (stated at the bottom of the plan doc too)

Plan step 6, the /eval IPC method and an evaluation-history view. /dash covers reading the output; a trace surface is worth building once there is real output, and should show the prompt.
RecentEvents as an input — action/object events already drive pattern proposals (#43).
Output quality is unmeasured. There is no fixture for "did she notice something true". The tests cover the machinery, not the observations. The wording and the 0.7 min_confidence default are unvalidated until someone reads a week of real output.

Verified

make build, make fmt-check and make test (go test -race) clean. internal/memeval/eval_test.go covers: empty store never reaches the model, confidence floor, dedupe across three runs including a whitespace variant, own notes excluded from input, empty array is not an error, LLM failure writes nothing, and prose-wrapped / over-long replies parse and are capped.

Vikunja #248

Ships the part of `docs/plans/03-memory-evaluation.md` that is real, local and testable now. Maven reads back her own recent memory on a slow ticker, asks the resident model what it notices, and records the confident answers as notes. ## What changed - **`internal/memeval`** (new package). `Evaluator.Evaluate(ctx, now)` reads `RecentFacts` / `RecentNotes` / `RecentNudges`, prompts the resident model under a GBNF grammar bounded to three `{observation, confidence, suggested_action}` objects, drops anything under `min_confidence`, deduplicates against what earlier evaluations wrote, and records the rest as notes with source `infer:memory-eval`. - **`cmd/mavend/memoryeval.go`** — the driver: its own goroutine on its own ticker, wired in `main.go` beside the fact-enrichment worker. - **`internal/config`** — new `memory_eval` block (`interval`, `max_items`, `min_confidence`). Not `internal/memory/eval.go` as the plan says: `internal/store` imports `internal/memory` for the vector-store backend, and an evaluator has to read `store.Fact`/`Note`/`Nudge`, which would close the import cycle. Hence a sibling package. Visibility comes free — `/dash` already renders notes with their source, so evaluation output shows up with no UI change. ## Off unless configured No `memory_eval` block ⇒ the goroutine does not exist. `deploy/mavend.json` is unchanged, so this branch changes nothing about a running deployment until someone adds the block. No LLM phraser also means no loop: there is no template fallback, because a "memory evaluation" assembled from string templates is a fixed sentence pretending to be an observation. ## What it deliberately cannot do This is the feature most likely to turn Maven into a nag — an hourly loop with an LLM in it and permission to talk — so the restraint is structural, not conventional: - **It cannot speak.** No dispatcher reference, no channel, no nudge. An observation is a thought she wrote down; he reads it on `/dash` when he wants to. Wiring observations to `delivery.Dispatcher` is a separate decision with its own opt-in and is not in this branch. - **It cannot act.** `suggested_action` is recorded inside the note text and interpreted by nobody. No reminder, routine or fact is created. - **It says nothing about an empty store.** No memory ⇒ no LLM call, so no observations are invented out of two facts. - **Its own notes are not input**, and are written with a nil embedding so they stay out of the RAG recall pool. Feeding generated text back into the pool it came from is how a small model starts citing its own guesses as evidence. ## Deferred (stated at the bottom of the plan doc too) - Plan step 6, the `/eval` IPC method and an evaluation-history view. `/dash` covers reading the output; a trace surface is worth building once there is real output, and should show the prompt. - `RecentEvents` as an input — action/object events already drive pattern proposals (#43). - **Output quality is unmeasured.** There is no fixture for "did she notice something true". The tests cover the machinery, not the observations. The wording and the 0.7 `min_confidence` default are unvalidated until someone reads a week of real output. ## Verified `make build`, `make fmt-check` and `make test` (`go test -race`) clean. `internal/memeval/eval_test.go` covers: empty store never reaches the model, confidence floor, dedupe across three runs including a whitespace variant, own notes excluded from input, empty array is not an error, LLM failure writes nothing, and prose-wrapped / over-long replies parse and are capped. Vikunja #248

claude added 1 commit 2026-07-31 23:46:10 +02:00

Add background memory evaluation, off unless configured (#248 ) dc7c72a3d7

Ships the real, local, testable part of the memory-evaluation plan
(docs/plans/03-memory-evaluation.md): Maven reads back her own recent
memory on a slow ticker, asks the resident model what it notices, and
records the confident answers as notes.

internal/memeval — not internal/memory/eval.go as the plan says, because
internal/store imports internal/memory for the vector backend and an
evaluator has to read store.Fact/Note/Nudge, which would close the
cycle. Evaluate() gathers RecentFacts/RecentNotes/RecentNudges, prompts
under a GBNF grammar bounded to three {observation, confidence,
suggested_action} objects, drops anything under min_confidence,
deduplicates against what earlier runs wrote, and writes the rest as
notes with source infer:memory-eval. /dash already renders notes with
their source, so the output is visible with no UI change.

cmd/mavend/memoryeval.go drives it on its own goroutine and ticker, not
on the 60s tick: an evaluation is a multi-second round-trip on the same
llama-server that answers voice turns, and it runs hourly at most. The
memory_eval config block is absent by default and absence means the
goroutine does not exist. No llama-server phraser also means no loop —
there is no template fallback, because a "memory evaluation" assembled
from templates is a fixed sentence pretending to be an observation.

What it deliberately cannot do, since this is the feature most likely to
turn Maven into a nag:

  - It cannot speak. No dispatcher reference, no channel, no nudge. An
    observation is a thought she wrote down and he reads on /dash.
    Announcing them is a separate decision with its own opt-in.
  - It cannot act. suggested_action is recorded as text and interpreted
    by nobody — no reminder, routine or fact is created from it.
  - It says nothing about an empty store: no memory means no LLM call,
    so there are no observations invented out of two facts.
  - Its own notes are excluded from the next evaluation's input, and are
    written with a nil embedding so they stay out of the recall pool.

The plan's remaining items (dispatching observations, an /eval IPC
method and trace view, RecentEvents) and the fact that output quality is
entirely unmeasured are written up at the bottom of the plan doc.

claude reviewed 2026-08-01 11:24:00 +02:00

claude left a comment

The restraint is the best part of this. No dispatcher reference at all, so the loop cannot reach a channel by accident. suggested_action recorded and interpreted by nobody. Nil embeddings, so generated text stays out of the RAG pool it came from. Empty store means no LLM call, because a 1.7B asked to find a pattern will always find one. The status section in docs/plans/03-memory-evaluation.md naming the plan steps skipped on purpose is worth as much as the code.

Three things.

1. The dedupe window is a note count, not an eval-note count. recordedTexts reads RecentNotes(ctx, 200) and keeps only the infer:memory-eval rows. Once 200 ordinary notes are newer than an observation, that observation falls out of the window. The next evaluation is then free to write the same sentence again. That is the exact failure the function exists to prevent, arriving quietly after a few months of normal use. A source-filtered read would make the window mean what the comment says.

2. Her own notes eat the input budget. snapshot asks for MaxItems notes, then discards the EvalNoteSource ones. After a few weeks of hourly evaluation, most of the 30 most recent notes are hers, so the model sees a handful of real ones. The own counter is computed and never read, which suggests this was noticed and left. Either fetch past the discards, or log own so the shrinking window is visible.

3. A five-minute timeout on the shared llama-server. The comment says nobody is waiting on the answer. True of the evaluation. Not true of the voice turn that arrives while it runs. There is one resident model and one server, so a long evaluation is a long stall in front of whoever speaks next. The hourly cadence makes the collision rare rather than impossible. Either shorten the timeout to something a voice turn can absorb, or skip the evaluation when a turn ran recently.

Smaller: formatNote appends " [action]", and recordedTexts strips it with LastIndex(text, " ["). An observation whose own text ends in a bracketed clause loses part of itself before hashing. It only affects the dedupe key, never the stored note, so this is cosmetic.

The restraint is the best part of this. No dispatcher reference at all, so the loop cannot reach a channel by accident. `suggested_action` recorded and interpreted by nobody. Nil embeddings, so generated text stays out of the RAG pool it came from. Empty store means no LLM call, because a 1.7B asked to find a pattern will always find one. The status section in `docs/plans/03-memory-evaluation.md` naming the plan steps skipped on purpose is worth as much as the code. Three things. **1. The dedupe window is a note count, not an eval-note count.** `recordedTexts` reads `RecentNotes(ctx, 200)` and keeps only the `infer:memory-eval` rows. Once 200 ordinary notes are newer than an observation, that observation falls out of the window. The next evaluation is then free to write the same sentence again. That is the exact failure the function exists to prevent, arriving quietly after a few months of normal use. A source-filtered read would make the window mean what the comment says. **2. Her own notes eat the input budget.** `snapshot` asks for `MaxItems` notes, then discards the `EvalNoteSource` ones. After a few weeks of hourly evaluation, most of the 30 most recent notes are hers, so the model sees a handful of real ones. The `own` counter is computed and never read, which suggests this was noticed and left. Either fetch past the discards, or log `own` so the shrinking window is visible. **3. A five-minute timeout on the shared llama-server.** The comment says nobody is waiting on the answer. True of the evaluation. Not true of the voice turn that arrives while it runs. There is one resident model and one server, so a long evaluation is a long stall in front of whoever speaks next. The hourly cadence makes the collision rare rather than impossible. Either shorten the timeout to something a voice turn can absorb, or skip the evaluation when a turn ran recently. Smaller: `formatNote` appends `" [action]"`, and `recordedTexts` strips it with `LastIndex(text, " [")`. An observation whose own text ends in a bracketed clause loses part of itself before hashing. It only affects the dedupe key, never the stored note, so this is cosmetic.

kami referenced this issue from a commit

2026-08-01 14:43:47 +02:00

memeval: scope the evaluator's note windows by source

kami closed this pull request

2026-08-01 14:51:41 +02:00

kami commented

2026-08-01 14:52:33 +02:00

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#55