kami
dc7c72a3d7
Ships the real, local, testable part of the memory-evaluation plan
(docs/plans/03-memory-evaluation.md): Maven reads back her own recent
memory on a slow ticker, asks the resident model what it notices, and
records the confident answers as notes.
internal/memeval — not internal/memory/eval.go as the plan says, because
internal/store imports internal/memory for the vector backend and an
evaluator has to read store.Fact/Note/Nudge, which would close the
cycle. Evaluate() gathers RecentFacts/RecentNotes/RecentNudges, prompts
under a GBNF grammar bounded to three {observation, confidence,
suggested_action} objects, drops anything under min_confidence,
deduplicates against what earlier runs wrote, and writes the rest as
notes with source infer:memory-eval. /dash already renders notes with
their source, so the output is visible with no UI change.
cmd/mavend/memoryeval.go drives it on its own goroutine and ticker, not
on the 60s tick: an evaluation is a multi-second round-trip on the same
llama-server that answers voice turns, and it runs hourly at most. The
memory_eval config block is absent by default and absence means the
goroutine does not exist. No llama-server phraser also means no loop —
there is no template fallback, because a "memory evaluation" assembled
from templates is a fixed sentence pretending to be an observation.
What it deliberately cannot do, since this is the feature most likely to
turn Maven into a nag:
- It cannot speak. No dispatcher reference, no channel, no nudge. An
observation is a thought she wrote down and he reads on /dash.
Announcing them is a separate decision with its own opt-in.
- It cannot act. suggested_action is recorded as text and interpreted
by nobody — no reminder, routine or fact is created from it.
- It says nothing about an empty store: no memory means no LLM call,
so there are no observations invented out of two facts.
- Its own notes are excluded from the next evaluation's input, and are
written with a nil embedding so they stay out of the recall pool.
The plan's remaining items (dispatching observations, an /eval IPC
method and trace view, RecentEvents) and the fact that output quality is
entirely unmeasured are written up at the bottom of the plan doc.
71 lines
4.6 KiB
Markdown
71 lines
4.6 KiB
Markdown
# Plan: Background Memory Evaluation & Idea Generation
|
|
|
|
**Goal:** Maven periodically reviews her own memory stores (facts, notes, events, nudges), evaluates coherence and gaps, and generates proactive proposals — new routines, configuration tweaks, observations she can share with the user.
|
|
|
|
**Done when:**
|
|
- `internal/memory/eval.go` — periodic evaluation loop runs on a slow cadence (1h)
|
|
- Evaluation reads `RecentFacts`, `RecentNotes`, `RecentNudges`, `RecentEvents` via `store.Store` or `ipc.CoreAPI`
|
|
- LLM summarizes state, detects anomalies (e.g. "you haven't recorded a meal in 3 days — is your routine broken?"), proposes new care rules
|
|
- Generated proposals are written as notes (kind `note`, source `infer:memory-eval`) and/or trigger nudges through the dispatcher
|
|
- Evaluation trace visible on `/history` page in mavweb
|
|
|
|
**Scope:**
|
|
- New `internal/memory/eval.go` — evaluator struct calling `internal/llm.Client` with a summarization prompt
|
|
- Reuses `internal/delivery.Dispatcher` for surfacing insights as care nudges (sev1)
|
|
- Reuses `internal/store` for reading memory state and writing evaluation notes
|
|
- Daemon wiring: new evaluation goroutine in `cmd/mavend/main.go`
|
|
- Config: `memory_eval_interval` in `config.Config` (default 1h, 0 to disable)
|
|
|
|
**Steps:**
|
|
1. Create `internal/memory/eval.go` — `Evaluator` struct holding `*store.Store`, `*llm.Client`, `*delivery.Dispatcher`
|
|
2. Implement `Evaluate(ctx)` — reads last N facts, notes, nudges, events, builds a prompt summarizing patterns, anomalies, gaps
|
|
3. LLM call returns structured observations: `{"observation":"...","confidence":0.8,"suggested_action":"remind|propose|notify"}`
|
|
4. High-confidence observations written as notes (`source:infer:memory-eval`) or dispatched as care nudges (sev1) through `dispatcher.DispatchNudge`
|
|
5. Wire evaluator goroutine in `cmd/mavend/main.go` — separate ticker, not on the main tick loop
|
|
6. Add `/eval` API method to `ipc.CoreAPI` (or reuse `Chat` with system context) so mavweb can show evaluation history
|
|
7. Add `memory_eval` block to `deploy/mavend.json`
|
|
8. Test with synthetic store state — verify observations match expected patterns
|
|
|
|
---
|
|
|
|
## Status 2026-08-01 — foundation shipped (Vikunja #248)
|
|
|
|
**Shipped:** `internal/memeval` (not `internal/memory/eval.go` — `internal/store`
|
|
imports `internal/memory` for the vector backend, so an evaluator that reads
|
|
`store.Fact` there would close an import cycle). `Evaluator.Evaluate` reads
|
|
`RecentFacts` / `RecentNotes` / `RecentNudges`, prompts the resident model under
|
|
a GBNF grammar for at most three `{observation, confidence, suggested_action}`
|
|
objects, drops anything under `min_confidence`, deduplicates against what earlier
|
|
evaluations wrote, and records the rest as notes with source `infer:memory-eval`.
|
|
Driver: `cmd/mavend/memoryeval.go`, its own goroutine on its own ticker. Config:
|
|
the `memory_eval` block — **absent ⇒ the loop does not run**. Visibility: `/dash`
|
|
already renders notes with their source, so evaluation output is visible with no
|
|
UI change.
|
|
|
|
**Deliberately not shipped — this is policy, not an unfinished edge:**
|
|
|
|
- *Dispatching observations as care nudges (plan step 4).* An hourly LLM loop
|
|
with permission to speak is a machine for generating interruptions, and the
|
|
content is model-generated text about his own life. The evaluator has no
|
|
dispatcher reference at all, so it cannot reach a channel by accident. Wiring
|
|
it to `delivery.Dispatcher` is a separate decision with its own opt-in.
|
|
- *Acting on `suggested_action`.* It is recorded inside the note text and
|
|
interpreted by nobody. No reminder, routine or fact is created.
|
|
- *Writing observation embeddings.* Notes are written with a nil embedding, so
|
|
they stay out of the RAG recall pool. Feeding generated text back into the pool
|
|
it came from is how a small model starts citing its own guesses as evidence.
|
|
|
|
**Deferred, wants a decision or another capability:**
|
|
|
|
- *Plan step 6, the `/eval` IPC method and an evaluation-history view.* `/dash`
|
|
covers reading the output; a dedicated trace surface is worth building once
|
|
there is real output to look at, and it should probably show the prompt too.
|
|
- *`RecentEvents`.* The plan lists it; the evaluator reads facts, notes and
|
|
nudges. Detected action/object events already drive pattern proposals (#43), and
|
|
duplicating them here would mostly re-derive that.
|
|
- *Output quality is unmeasured.* There is no fixture for "did she notice
|
|
something true". The tests cover the machinery — empty store, confidence floor,
|
|
dedupe, own-notes exclusion, error handling — not the observations. Until
|
|
someone reads a week of real output on `/dash`, treat the wording and the
|
|
`min_confidence` default as unvalidated.
|