Compare commits

..

5 Commits

Author SHA1 Message Date
kami 0b3b8d0a9e Test the clarify round-trip end to end at the daemon level
Covers: a reminder with no time is asked about and completes on the answer; the
same for a fact; an answer past the TTL falls through as a fresh utterance; a
second unclear answer drops the request with no second question; a clarified act
off the allowlist neither runs nor gets enabled; a clarified destructive act
still parks a confirm; noise keeps the canned reply. No model, no network.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:31:07 +04:00
kami fe0e654ab1 Ask the question, then act on the answer
On a clarify decision with one identifiable gap she now asks instead of saying
"не поняла", and parks the request. The next utterance is parsed as the answer
with the router's own extractor and the completed decision runs through
applyAction like any other — so a clarified act still needs the allowlist and
still hits the destructive confirm gate. An answer that does not fill the gap
drops the request; she never asks twice. Also pulls the session-store block
that HandlePushToTalk and handleText both had into rememberTurn, since the
clarify path needed a third copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:31:07 +04:00
kami a54ebac0cb Work out which slot is missing and phrase one short question
A table per intent (reminder needs a time, fact needs a key, act needs a fn)
plus one fixed Russian question per slot. Templates, not model output: a 0.8B
would wander and a question that rewords itself is harder to answer. Note,
query, chat and system get no question — for those a clarify decision keeps
the canned reply rather than inventing a question for noise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:31:07 +04:00
kami 39d83a33e8 Add the pending-question data layer for slot clarification
PendingQuestion plus ClarifyStore: same shape, locking and expiry as SessionStore. Answer fills only the missing slots and never overwrites a filled one. No wiring yet — TODOs mark the daemon hooks.
Reviewer: MaxAttempts is 1 on purpose (Maven asks once, she is not a nag).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:13:48 +04:00
kami 925ce223a0 Add a Value slot to dialogue.Slots
router.Slots already carries the fact payload; the dialogue copy did not, so a clarifying answer had nowhere to put it. InheritSlots carries it like Key.
Reviewer: check the new inherit block does not overwrite a filled value.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:13:40 +04:00
12 changed files with 872 additions and 1316 deletions
+1 -8
View File
@@ -16,7 +16,7 @@ PIPER_BIN := $(shell pwd)/deps/piper/piper
PIPER_MODEL := $(shell pwd)/models/tts/ru_RU-irina-medium.onnx
PIPER_ESPEAK := $(shell pwd)/deps/piper/espeak-ng-data
.PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test run-stt run-tts run-web download-embedder deps-go eval-router eval-recall
.PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test run-stt run-tts run-web download-embedder deps-go eval-router
all: build
@@ -83,13 +83,6 @@ MAVEN_ONNX_LIB ?= $(shell pwd)/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxrunt
eval-router:
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/router/eval/
# eval-recall — score the held-out note-recall fixture (internal/memory/recalleval).
# Answers "can she find the note again when it matters": recall@1, recall@3,
# false recall and the query_min_score sweep. Same MAVEN_ONNX_LIB deal as
# eval-router; without it only the deterministic hash ratchet runs.
eval-recall:
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
run-stt: build-stt
LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
./mavsttd -socket /tmp/maven/stt.sock -model $(WHISPER_MODEL)
-99
View File
@@ -1,99 +0,0 @@
# Note recall evaluation — 31-07-2026
The operator's goal is that Maven "memorize/note things … and know more about me/world". This
measures whether the note/recall path delivers that.
- Fixture + scorer: `internal/memory/recalleval/` (`ru_recall_v1.json`, 30 cases)
- Reproduce: `make eval-recall` — hash ratchet always, ONNX when `deps/` is present
- Commit: `43470ab` (harness)
Each case inserts its own 3 notes **plus 12 shared filler notes** into a fresh store, embeds the
query, takes the top 3 — the read path `cmd/mavend/voice.go` runs for `IntentQuery`. Filler is
load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable
cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5
that must recall **nothing**. `TestFixtureIsParaphrased` fails the build if a query shares over half
its words with its note; equal-score ties count as ties, not recall.
## Results
| | recall+hash (CI ratchet) | recall+onnx (deployed) |
|---|---|---|
| **recall@1** | 36.0% (9/25) | **60.0% (15/25)** |
| recall@3 | 76.0% (19/25) | 80.0% (20/25) |
| **answered after the 0.55 gate** | **0.0% (0/25)** | **48.0% (12/25)** |
| wrong note on top / tie on top | 9 / 7 | 10 / 0 |
| ranked first, then silenced by the gate | 9 | 3 |
| **false recall** | 0/5 | **1/5 (20%)** |
| top-1 score when right, min / median | n/a | 0.559 / 0.678 |
| top-1 when it must stay silent, median / max | 0.000 / 0.144 | 0.470 / **0.567** |
| RU / EN / `hard` cases passed | 4/24 / 1/6 / 0/11 | 13/24 / 3/6 / 2/11 |
| latency p50 / p95 / max | 49µs / 70µs | 59ms / 148ms / 194ms |
Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so
CI has a deterministic ratchet with no model files.
## Findings
### 1. Real recall is 48%, not 60%
The right note ranks first 60% of the time, but the daemon only *says* it 48% of the time — three
more cases rank first and are then silenced by `voice.go:776`'s `queryMinScore`. **Roughly one
useful question in two gets "не знаю".** This is not a working memory yet.
### 2. The gate cannot separate a real recall from a false one — the distributions overlap
Right-note top-1 scores start at **0.559**. Must-stay-silent top-1 scores reach **0.567**. No
threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25
answered, 1/5 false; **0.55 (default) → 12/25, 1/5**; **0.60 → 10/25, 0/5**; 0.70 → 5/25, 0/5. What
the data says about `DefaultQueryMinScore` (`internal/config/config.go:392`): **0.55 is
slightly too loose** — it admits one confident wrong answer ("как зовут сестру моего коллеги"
recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60
silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means
the threshold is the wrong dial anyway (finding 3).
### 3. Filler notes outrank the right answer — the model scores similarity, not relevance
`models/embedder/` is **paraphrase-multilingual-MiniLM-L12-v2** (`Makefile:119`), a *symmetric*
paraphrase model. It scores "do these sentences look alike", not "does this passage answer this
question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего
кончилось место" and "откуда берётся токен бота" both return `выучил пару аккордов на гитаре`
(0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note
not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning
problem — an asymmetric retrieval model (`multilingual-e5-small`, with `query:` / `passage:`
prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately:
`deploy/mavend.json:39` loads a 470MB fp32 `model.onnx` while `make download-embedder` fetches
`model_quantized.onnx` — not the same file.
`hard` cases score **2/11**: every one is a query where the operator did not reuse his own words.
That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises.
### 4. The memory-store recall branch is dead for notes
`voice.go:776` only reaches `h.memStore.Search` when the notes-RAG top score is already below
`queryMinScore`, and `bestRecall` (`cmd/mavend/recall.go:19`) then applies the **same** gate to the
same vector. A note is indexed in both places with the same embedding, so if it failed the gate in
`QueryNotes` it fails again here — the branch can only ever return a **fact**. Its comment calls it
"additive"; for notes it is not.
### 5. Ranking has no recency or type signal, and the store is not the bottleneck
`internal/store/notes.go:67` sorts by cosine and uses `ts` only to break an exact float tie, which
never happens; `kind` never enters the ranking. Meanwhile `TestPersistentStoreScoresTheSame` scores
sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — both full-scan cosine
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
not the problem to solve.
## Next steps — ordered by value-to-risk; nothing here is a decision
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
change plus a prefix in `onnxembedder.go`, re-measurable in one command.
2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
`query_min_score` picked against today's embedder describes a model on its way out.
3. **Replace the absolute-score gate with a margin gate** (`top1 top2 > δ`) — as the routing eval
concluded, absolute cosine cannot see a flat distribution.
4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
gate it separately, or restrict it to facts and say so.
5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
6. **Grow the fixture from real misses.** 30 cases can rank two embedders, not trust 4 points.
7. **Re-measure end to end.** Recall is gated twice — the utterance must first route to `query`,
which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.
+180
View File
@@ -0,0 +1,180 @@
package main
import (
"context"
"log"
"time"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/router"
)
// clarifyTTL — how long a parked question stays answerable. Same 90s as the
// confirm gate, for the same reason: an answer is a same-breath gesture, and a
// stale question must not eat an unrelated later utterance.
const clarifyTTL = 90 * time.Second
// wantedSlots — what each intent needs before she can act on it. First entry is
// the one she asks about; the rest are only used to decide act-vs-drop.
//
// Intents not listed here are never worth a question: note and query act on the
// raw utterance, chat and system have nothing to fill in. For those a clarify
// decision keeps the canned "не поняла" reply — inventing a question for noise
// is worse than admitting she missed it.
var wantedSlots = map[router.Intent][]dialogue.Slot{
router.IntentReminder: {dialogue.SlotTime},
router.IntentFact: {dialogue.SlotKey},
router.IntentAct: {dialogue.SlotFn},
}
// clarifyQuestions — one short question per missing slot.
//
// These are fixed templates, not model output. The resident model is a 0.8B; it
// would wander, and a question whose wording changes every time is harder to
// answer than a blunt one that always reads the same. They are infinitive
// questions, so there is no gender agreement to get wrong; the feminine
// self-reference lives in the reply she gives when she drops the request.
var clarifyQuestions = map[dialogue.Slot]string{
dialogue.SlotTime: "На когда напомнить?",
dialogue.SlotKey: "Что записать?",
dialogue.SlotFn: "Что сделать?",
}
// clarifyDropped — she asked once, the answer still did not fill the gap, so
// the request is gone. Said plainly, once, with no second question.
const clarifyDropped = "Не разобрала — скажи целиком, пожалуйста."
// missingFor returns the slots a decision still needs, most important first.
// Empty ⇒ there is nothing identifiable to ask about.
func missingFor(dec router.Decision) []dialogue.Slot {
return dialogue.StillMissing(wantedSlots[dec.Intent], toDialogueSlots(dec.Slots))
}
// clarifyQuestion picks the one question to ask for a clarify decision. Returns
// ("", false) when she has no idea what is missing.
//
// One question about one thing: if two slots are missing she asks about the
// first and lets the rest go. Two questions in a row is an interrogation.
func clarifyQuestion(dec router.Decision) (dialogue.Slot, string, bool) {
missing := missingFor(dec)
if len(missing) == 0 {
return "", "", false
}
q, ok := clarifyQuestions[missing[0]]
if !ok {
return "", "", false
}
return missing[0], q, true
}
// askClarify parks the request and returns the question to ask instead of the
// canned "не поняла". Returns ("", false) when there is nothing to ask about, so
// the caller falls back to the canned reply.
func (h *reactiveHandler) askClarify(dec router.Decision) (string, bool) {
if h.clarifyStore == nil {
return "", false
}
slot, question, ok := clarifyQuestion(dec)
if !ok {
return "", false
}
h.clarifyStore.Put(voiceDialogueID, &dialogue.PendingQuestion{
Intent: dialogue.Intent(dec.Intent),
Slots: toDialogueSlots(dec.Slots),
Missing: []dialogue.Slot{slot},
Utterance: dec.Utterance,
Asked: h.now(),
TTL: clarifyTTL,
Attempts: 1, // asked once; MaxAttempts is 1, so there is no second ask
})
log.Printf("voice: clarify — asked about %s for intent=%s", slot, dec.Intent)
return question, true
}
// resolveClarifyAnswer reads an utterance as the answer to a parked question.
// Returns ("", false) when no live question is parked (or it expired), so the
// caller routes the utterance normally as a fresh request. Sibling of
// resolveConfirm and checked in the same place.
//
// The answer is parsed with the same extractor the router uses, for the intent
// she parked — no second parser. If it still does not fill the gap the request
// is dropped: she does not ask again.
func (h *reactiveHandler) resolveClarifyAnswer(ctx context.Context, text string) (string, bool) {
if h.clarifyStore == nil {
return "", false
}
q := h.clarifyStore.Get(voiceDialogueID, h.now())
if q == nil {
return "", false
}
// One shot either way: the question is consumed whether or not the answer
// works, so a failed answer can't leave the question armed.
h.clarifyStore.Delete(voiceDialogueID)
intent := router.Intent(q.Intent)
answer := h.extractor.Extract(ctx, intent, text, h.now())
merged := q.Answer(text, toDialogueSlots(answer))
if len(dialogue.StillMissing(q.Missing, merged)) > 0 {
log.Printf("voice: clarify — answer %q did not fill %v, dropping", text, q.Missing)
return clarifyDropped, true
}
// Rebuild the decision as if it had routed cleanly, then run it down the
// normal path. Clarify is deliberately false and the intent is unchanged:
// filling in an argument never grants authority, so the completed decision
// still meets the allowlist and the destructive-act confirm gate in
// applyAction exactly like any other decision.
dec := router.Decision{
Utterance: q.Utterance,
Stage: 2,
Intent: intent,
Slots: applyDialogueSlots(answer, merged),
}
return h.finishClarified(ctx, dec), true
}
// finishClarified runs a completed decision through the same steps a freshly
// routed one takes: remember the turn, act, then phrase.
func (h *reactiveHandler) finishClarified(ctx context.Context, dec router.Decision) string {
if h.dialogueSessions != nil {
now := h.now()
prev := h.dialogueSessions.Get(voiceDialogueID, now)
dec = followUpMerge(prev, dec, now)
h.rememberTurn(prev, dec, now)
}
reply := h.applyAction(ctx, dec)
if reply == "" {
reply = h.replier.Reply(dec)
}
return reply
}
// rememberTurn stores this turn as the dialogue session the next follow-up
// inherits from, carrying up to 4 prior turns of history for anaphora. Capped so
// one long conversation can't grow the session unboundedly.
func (h *reactiveHandler) rememberTurn(prev *dialogue.Session, dec router.Decision, now time.Time) {
var history []dialogue.Turn
if prev != nil {
history = append(history, dialogue.Turn{
Intent: prev.Intent,
Slots: prev.Slots,
Text: prev.Slots.Text,
})
maxHist := len(prev.History)
if maxHist > 3 {
maxHist = 3
}
history = append(history, prev.History[:maxHist]...)
}
ttl := time.Duration(0) // use the store default (2 min)
if dec.Intent == router.IntentChat {
ttl = 15 * time.Minute // conversational turns should last longer
}
h.dialogueSessions.Put(voiceDialogueID, &dialogue.Session{
Intent: dialogue.Intent(dec.Intent),
Slots: toDialogueSlots(dec.Slots),
Timestamp: now,
TTL: ttl,
History: history,
})
}
+238
View File
@@ -0,0 +1,238 @@
package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
"github.com/kami/maven/internal/tool"
"github.com/kami/maven/internal/voice"
)
// newClarifyHandler builds a handler with the clarify path wired and no model:
// stub date parser, the real fact parser, and a matcher over whatever tools the
// test enabled. `now` is fixed so TTL behaviour is testable.
func newClarifyHandler(t *testing.T) (*reactiveHandler, *store.Store, *time.Time) {
t.Helper()
st := newTestStore(t)
api := ipc.NewStoreAPI(st)
now := time.Date(2026, 7, 31, 9, 0, 0, 0, time.UTC)
matcher := tool.NewMatcher(api)
h := &reactiveHandler{
api: api,
dataStore: st,
tools: tool.NewExecutor(api, 2*time.Second),
matcher: matcher,
replier: voice.NewStubReplier(),
now: func() time.Time { return now },
dialogueSessions: dialogue.NewSessionStore(2 * time.Minute),
clarifyStore: dialogue.NewClarifyStore(clarifyTTL),
extractor: router.Extractor{
Time: router.StubDateTimeParser{},
Acts: matcher,
Facts: router.DefaultFactParser{},
},
}
return h, st, &now
}
func clarifyDec(intent router.Intent, slots router.Slots, utterance string) router.Decision {
return router.Decision{Utterance: utterance, Stage: 3, Intent: intent, Slots: slots, Clarify: true}
}
// TestClarifyQuestionForMissingSlot pins which question goes with which gap, and
// which intents get no question at all.
func TestClarifyQuestionForMissingSlot(t *testing.T) {
cases := []struct {
name string
dec router.Decision
want string
asked bool
}{
{"reminder without a time", clarifyDec(router.IntentReminder, router.Slots{Text: "напомни позвонить маме"}, "напомни позвонить маме"), "На когда напомнить?", true},
{"fact without a key", clarifyDec(router.IntentFact, router.Slots{Text: "запиши"}, "запиши"), "Что записать?", true},
{"act without a fn", clarifyDec(router.IntentAct, router.Slots{Text: "сделай это"}, "сделай это"), "Что сделать?", true},
{"reminder that already has a time", clarifyDec(router.IntentReminder, router.Slots{HasTime: true}, "напомни в 11"), "", false},
{"chat is never worth a question", clarifyDec(router.IntentChat, router.Slots{Text: "мгм"}, "мгм"), "", false},
{"query is never worth a question", clarifyDec(router.IntentQuery, router.Slots{Text: "а"}, "а"), "", false},
}
for _, tc := range cases {
_, got, asked := clarifyQuestion(tc.dec)
if asked != tc.asked || got != tc.want {
t.Errorf("%s: got (%q, %v), want (%q, %v)", tc.name, got, asked, tc.want, tc.asked)
}
}
}
// TestClarifyReminderCompletesOnAnswer is the whole point of the feature: she
// asks for the missing time and the answer creates the reminder.
func TestClarifyReminderCompletesOnAnswer(t *testing.T) {
ctx := context.Background()
h, st, _ := newClarifyHandler(t)
question, asked := h.askClarify(clarifyDec(router.IntentReminder, router.Slots{Text: "напомни позвонить маме"}, "напомни позвонить маме"))
if !asked || question != "На когда напомнить?" {
t.Fatalf("expected the time question, got %q asked=%v", question, asked)
}
reply, handled := h.resolveClarifyAnswer(ctx, "в 11:00")
if !handled {
t.Fatal("the answer to an open question must be consumed as an answer")
}
if reply == clarifyDropped {
t.Fatalf("a good answer must not drop the request: %q", reply)
}
reminders, err := st.DueReminders(ctx, h.now().Add(48*time.Hour))
if err != nil || len(reminders) != 1 {
t.Fatalf("clarified reminder was not created: reminders=%v err=%v", reminders, err)
}
if !strings.Contains(reminders[0].Payload, "маме") {
t.Fatalf("the reminder lost the original request: %q", reminders[0].Payload)
}
if h.clarifyStore.Get(voiceDialogueID, h.now()) != nil {
t.Fatal("the question must be cleared once answered")
}
}
// TestClarifyFactCompletesOnAnswer — the fact path, where the answer carries
// both the key and the value.
func TestClarifyFactCompletesOnAnswer(t *testing.T) {
ctx := context.Background()
h, st, _ := newClarifyHandler(t)
if _, asked := h.askClarify(clarifyDec(router.IntentFact, router.Slots{Text: "запиши"}, "запиши")); !asked {
t.Fatal("a fact with no key should be asked about")
}
if reply, handled := h.resolveClarifyAnswer(ctx, "пил воду"); !handled || reply == clarifyDropped {
t.Fatalf("answer should complete the fact, handled=%v reply=%q", handled, reply)
}
if fact, err := st.LatestFact(ctx, "water"); err != nil || fact.Key != "water" {
t.Fatalf("clarified fact was not written: fact=%+v err=%v", fact, err)
}
}
// TestClarifyAnswerAfterTTLIsANewRequest — a late answer is not an answer.
func TestClarifyAnswerAfterTTLIsANewRequest(t *testing.T) {
ctx := context.Background()
h, st, now := newClarifyHandler(t)
if _, asked := h.askClarify(clarifyDec(router.IntentReminder, router.Slots{Text: "напомни"}, "напомни")); !asked {
t.Fatal("expected a question")
}
*now = now.Add(clarifyTTL + time.Second)
if reply, handled := h.resolveClarifyAnswer(ctx, "в 11:00"); handled {
t.Fatalf("an answer past the TTL must fall through to normal routing, got %q", reply)
}
if reminders, err := st.DueReminders(ctx, now.Add(48*time.Hour)); err != nil || len(reminders) != 0 {
t.Fatalf("expired question must not create anything: reminders=%v err=%v", reminders, err)
}
}
// TestClarifyUnclearAnswerDropsWithoutAskingAgain — MaxAttempts is 1.
func TestClarifyUnclearAnswerDropsWithoutAskingAgain(t *testing.T) {
ctx := context.Background()
h, st, _ := newClarifyHandler(t)
if _, asked := h.askClarify(clarifyDec(router.IntentReminder, router.Slots{Text: "напомни"}, "напомни")); !asked {
t.Fatal("expected a question")
}
reply, handled := h.resolveClarifyAnswer(ctx, "ну не знаю")
if !handled || reply != clarifyDropped {
t.Fatalf("an unclear answer should drop the request, handled=%v reply=%q", handled, reply)
}
if strings.Contains(reply, "?") {
t.Fatalf("she must not ask a second question: %q", reply)
}
if h.clarifyStore.Get(voiceDialogueID, h.now()) != nil {
t.Fatal("a dropped request must leave no armed question")
}
if reminders, err := st.DueReminders(ctx, h.now().Add(48*time.Hour)); err != nil || len(reminders) != 0 {
t.Fatalf("a dropped request must not create anything: reminders=%v err=%v", reminders, err)
}
}
// TestClarifiedActOffAllowlistIsStillRefused — clarification fills in an
// argument, it never grants authority.
func TestClarifiedActOffAllowlistIsStillRefused(t *testing.T) {
ctx := context.Background()
h, st, _ := newClarifyHandler(t)
marker := filepath.Join(t.TempDir(), "not-allowed-ran")
if _, asked := h.askClarify(clarifyDec(router.IntentAct, router.Slots{Text: "сделай это"}, "сделай это")); !asked {
t.Fatal("an act with no fn should be asked about")
}
reply, handled := h.resolveClarifyAnswer(ctx, "rm "+marker)
if !handled {
t.Fatal("the answer should be consumed")
}
if strings.Contains(reply, "готово") {
t.Fatalf("an act that is not on the allowlist must not report success: %q", reply)
}
if _, err := os.Stat(marker); !os.IsNotExist(err) {
t.Fatalf("a clarified act off the allowlist ran anyway: %v", err)
}
if tools, err := st.ListTools(ctx, "enabled"); err != nil || len(tools) != 0 {
t.Fatalf("clarify must not enable a tool: tools=%+v err=%v", tools, err)
}
}
// TestClarifiedDestructiveActStillNeedsConfirm — the confirm gate survives the
// clarify path.
func TestClarifiedDestructiveActStillNeedsConfirm(t *testing.T) {
ctx := context.Background()
h, st, _ := newClarifyHandler(t)
marker := filepath.Join(t.TempDir(), "destructive-ran")
if err := st.EnableTool(ctx, "delete_backups", []string{"touch", marker}, true, "test", h.now()); err != nil {
t.Fatal(err)
}
if _, asked := h.askClarify(clarifyDec(router.IntentAct, router.Slots{Text: "сделай это"}, "сделай это")); !asked {
t.Fatal("expected a question")
}
reply, handled := h.resolveClarifyAnswer(ctx, "delete_backups")
if !handled {
t.Fatal("the answer should be consumed")
}
if !strings.Contains(reply, "да") || h.pending == nil {
t.Fatalf("a clarified destructive act must still park a confirm: reply=%q pending=%+v", reply, h.pending)
}
if _, err := os.Stat(marker); !os.IsNotExist(err) {
t.Fatalf("a clarified destructive act ran before confirmation: %v", err)
}
}
// TestNoQuestionWhenNothingIsMissing — noise keeps the canned reply, so she
// never invents a question for nothing.
func TestNoQuestionWhenNothingIsMissing(t *testing.T) {
h, _, _ := newClarifyHandler(t)
for _, dec := range []router.Decision{
clarifyDec(router.IntentChat, router.Slots{Text: "эм"}, "эм"),
clarifyDec(router.IntentQuery, router.Slots{Text: "ммм"}, "ммм"),
clarifyDec(router.IntentNote, router.Slots{Text: "..."}, "..."),
} {
if question, asked := h.askClarify(dec); asked {
t.Fatalf("intent %s should keep the canned reply, got %q", dec.Intent, question)
}
}
if h.clarifyStore.Get(voiceDialogueID, h.now()) != nil {
t.Fatal("noise must not park a question")
}
}
// TestNoPendingQuestionFallsThrough — with nothing parked, an utterance routes
// normally.
func TestNoPendingQuestionFallsThrough(t *testing.T) {
h, _, _ := newClarifyHandler(t)
if reply, handled := h.resolveClarifyAnswer(context.Background(), "напомни в 11:00"); handled {
t.Fatalf("no open question ⇒ must not be treated as an answer, got %q", reply)
}
}
+43 -53
View File
@@ -226,6 +226,8 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
// ----- dialogue (multi-turn slot carry-over; 2-min follow-up window) -----
dialogueSessions := dialogue.NewSessionStore(2 * time.Minute)
clarifyStore := dialogue.NewClarifyStore(clarifyTTL)
timeParser := router.NewPythonDateParser()
// ----- replier (LLM-backed when the engine is on, Stub floor otherwise) -----
replier := voice.Replier(voice.NewStubReplier())
@@ -250,8 +252,10 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
memStore: memStore,
dataStore: dataStore,
dialogueSessions: dialogueSessions,
clarifyStore: clarifyStore,
extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}},
queryMinScore: cfg.Voice.QueryMinScore,
timeParser: router.NewPythonDateParser(),
timeParser: timeParser,
ecosystem: eco,
}
@@ -304,6 +308,14 @@ type reactiveHandler struct {
// box → one session slot, keyed voiceDialogueID). nil ⇒ no carry-over.
dialogueSessions *dialogue.SessionStore
// clarifyStore parks the request behind an open question she asked (see
// clarify.go). nil ⇒ she falls back to the canned "не поняла" reply.
clarifyStore *dialogue.ClarifyStore
// extractor parses the answer to an open question, with the same parsers
// the router's own stage-2 uses.
extractor router.Extractor
// pending destructive-act confirmation. A destructive act replies with a
// "выполнить X? да/нет" prompt and parks here; the NEXT utterance is read as
// the y/n answer. ponytail: single slot, single-user box — a second act
@@ -378,6 +390,13 @@ func (h *reactiveHandler) HandlePushToTalk(ctx context.Context, req voice.PushTo
return h.reply(ctx, reply, nil)
}
// 1b2. clarify answer — if she asked a question last turn, this utterance is
// its answer, not a fresh command. After the confirm check: a y/n gate is
// armed by her own prompt and is the narrower claim on the utterance.
if reply, handled := h.resolveClarifyAnswer(ctx, text); handled {
return h.reply(ctx, reply, nil)
}
// 1c. quiet-hours toggle — keyword match, not classifier-dependent.
// "тихий режим" / "quiet on" would route through the classifier
// unreliably (it's a command, not a free-form query), so we match it
@@ -407,34 +426,16 @@ func (h *reactiveHandler) HandlePushToTalk(ctx context.Context, req voice.PushTo
prev := h.dialogueSessions.Get(voiceDialogueID, now)
dec = followUpMerge(prev, dec, now)
if !dec.Clarify {
// Build history: carry over up to 4 prior turns for cross-intent
// reference. The most recent prior turn is prepended to history.
var history []dialogue.Turn
if prev != nil {
history = append(history, dialogue.Turn{
Intent: prev.Intent,
Slots: prev.Slots,
Text: prev.Slots.Text, // the prior turn's utterance
})
// Cap history depth so one long conversation can't grow
// the session unboundedly.
maxHist := len(prev.History)
if maxHist > 3 {
maxHist = 3
}
history = append(history, prev.History[:maxHist]...)
}
ttl := time.Duration(0) // use default (2 min)
if dec.Intent == router.IntentChat {
ttl = 15 * time.Minute // conversational turns should last longer
}
h.dialogueSessions.Put(voiceDialogueID, &dialogue.Session{
Intent: dialogue.Intent(dec.Intent),
Slots: toDialogueSlots(dec.Slots),
Timestamp: now,
TTL: ttl,
History: history,
})
h.rememberTurn(prev, dec, now)
}
}
// 2c. clarify — she is not sure. If one named thing is missing, ask about it
// and park the request (clarify.go); otherwise the replier's canned reply
// stands.
if dec.Clarify {
if question, asked := h.askClarify(dec); asked {
return h.reply(ctx, question, nil)
}
}
@@ -465,6 +466,11 @@ func (h *reactiveHandler) handleText(ctx context.Context, text string) string {
return reply
}
// 1b2. clarify answer — same check as HandlePushToTalk.
if reply, handled := h.resolveClarifyAnswer(ctx, text); handled {
return reply
}
// 2. router — classify the utterance.
dec, err := h.router.Route(ctx, text, h.now())
if err != nil {
@@ -482,30 +488,14 @@ func (h *reactiveHandler) handleText(ctx context.Context, text string) string {
prev := h.dialogueSessions.Get(voiceDialogueID, now)
dec = followUpMerge(prev, dec, now)
if !dec.Clarify {
var history []dialogue.Turn
if prev != nil {
history = append(history, dialogue.Turn{
Intent: prev.Intent,
Slots: prev.Slots,
Text: prev.Slots.Text,
})
maxHist := len(prev.History)
if maxHist > 3 {
maxHist = 3
}
history = append(history, prev.History[:maxHist]...)
}
ttl := time.Duration(0)
if dec.Intent == router.IntentChat {
ttl = 15 * time.Minute
}
h.dialogueSessions.Put(voiceDialogueID, &dialogue.Session{
Intent: dialogue.Intent(dec.Intent),
Slots: toDialogueSlots(dec.Slots),
Timestamp: now,
TTL: ttl,
History: history,
})
h.rememberTurn(prev, dec, now)
}
}
// 2c. clarify — same as HandlePushToTalk: ask about the one missing thing.
if dec.Clarify {
if question, asked := h.askClarify(dec); asked {
return question
}
}
+171
View File
@@ -0,0 +1,171 @@
package dialogue
import (
"sync"
"time"
)
// Slot names one field of Slots. Named type, not a free string, so a missing
// slot cannot be misspelled — the question phrasing switches on these.
type Slot string
const (
SlotTime Slot = "time" // Slots.Time / HasTime
SlotKey Slot = "key" // Slots.Key / HasKey
SlotValue Slot = "value" // Slots.Value (paired with Key)
SlotFn Slot = "fn" // Slots.Fn / HasFn
SlotText Slot = "text" // Slots.Text
)
// MaxAttempts is 1 because Maven is not a nag (DESIGN.md § Non-goals). She asks
// one clarifying question. If the answer still leaves the slot empty she drops
// the request instead of asking again.
const MaxAttempts = 1
// PendingQuestion is what Maven holds while she waits for an answer to an open
// question. Unlike the yes/no confirms in cmd/mavend/voice.go, the answer here
// is free text that fills a missing slot rather than a verdict.
type PendingQuestion struct {
Intent Intent // what the router already guessed
Slots Slots // what it already filled
Missing []Slot // what is still empty, in the order to ask about
Utterance string // the user's original raw words
Asked time.Time
TTL time.Duration
Attempts int // questions already asked; capped by MaxAttempts
}
func (q *PendingQuestion) IsExpired(now time.Time) bool {
return now.After(q.Asked.Add(q.TTL))
}
// CanAsk reports whether Maven may ask another question about this request.
func (q *PendingQuestion) CanAsk() bool {
return q.Attempts < MaxAttempts
}
// TODO: the daemon will phrase the question text from Missing (one short ru
// question per Slot, feminine self-reference) and speak it here.
// ClarifyStore holds the parked questions. Same shape and locking as
// SessionStore: keyed by dialogue id, expired entries dropped on read.
type ClarifyStore struct {
mu sync.RWMutex
questions map[string]*PendingQuestion
defaultTTL time.Duration
}
func NewClarifyStore(defaultTTL time.Duration) *ClarifyStore {
if defaultTTL <= 0 {
// Short, like confirmTTL in voice.go: a clarifying question is a
// same-breath gesture, a stale one should not eat a later utterance.
defaultTTL = 90 * time.Second
}
return &ClarifyStore{
questions: make(map[string]*PendingQuestion),
defaultTTL: defaultTTL,
}
}
// TODO: the daemon will Put a question here when Decision.Clarify fires, in
// place of the flat "не разобрала" reply (cmd/mavend/voice.go).
func (s *ClarifyStore) Put(id string, q *PendingQuestion) {
if q.TTL <= 0 {
q.TTL = s.defaultTTL
}
s.mu.Lock()
s.questions[id] = q
s.mu.Unlock()
}
// TODO: the daemon will Get on the next turn, parse that turn into Slots, call
// Answer, and Delete — the open-question twin of resolveConfirm.
func (s *ClarifyStore) Get(id string, now time.Time) *PendingQuestion {
s.mu.RLock()
q, ok := s.questions[id]
s.mu.RUnlock()
if !ok {
return nil
}
if q.IsExpired(now) {
s.Delete(id)
return nil
}
return q
}
func (s *ClarifyStore) Delete(id string) {
s.mu.Lock()
delete(s.questions, id)
s.mu.Unlock()
}
// Answer merges the slots parsed from the user's answer into the parked ones.
// Only the slots listed in Missing are filled, and an already filled slot is
// never overwritten — the answer completes the original request, it does not
// restate it. Parsing the answer text into `answer` is the caller's job; this
// package must stay free of internal/router.
func (q *PendingQuestion) Answer(text string, answer Slots) Slots {
out := q.Slots
for _, slot := range q.Missing {
switch slot {
case SlotTime:
if !out.HasTime && answer.HasTime {
out.Time = answer.Time
out.HasTime = true
}
case SlotKey:
if !out.HasKey && answer.HasKey {
out.Key = answer.Key
out.HasKey = true
}
case SlotValue:
if out.Value == "" && answer.Value != "" {
out.Value = answer.Value
}
case SlotFn:
if !out.HasFn && answer.HasFn {
out.Fn = answer.Fn
out.HasFn = true
if len(out.Args) == 0 {
out.Args = append([]string(nil), answer.Args...)
}
}
case SlotText:
if out.Text == "" {
if answer.Text != "" {
out.Text = answer.Text
} else {
// No parse for a text slot — the raw answer IS the text.
out.Text = text
}
}
}
}
return out
}
// StillMissing lists the slots that are empty in s, out of the ones asked for.
// The caller uses it to decide between acting and dropping the request.
func StillMissing(want []Slot, s Slots) []Slot {
var out []Slot
for _, slot := range want {
empty := false
switch slot {
case SlotTime:
empty = !s.HasTime
case SlotKey:
empty = !s.HasKey
case SlotValue:
empty = s.Value == ""
case SlotFn:
empty = !s.HasFn
case SlotText:
empty = s.Text == ""
}
if empty {
out = append(out, slot)
}
}
return out
}
+225
View File
@@ -0,0 +1,225 @@
package dialogue
import (
"testing"
"time"
)
var base = time.Date(2026, 7, 31, 12, 0, 0, 0, time.UTC)
func TestPendingQuestionIsExpired(t *testing.T) {
cases := []struct {
name string
ttl time.Duration
now time.Time
want bool
}{
{"fresh", time.Minute, base.Add(10 * time.Second), false},
{"exactly at ttl", time.Minute, base.Add(time.Minute), false},
{"past ttl", time.Minute, base.Add(2 * time.Minute), true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
q := &PendingQuestion{Asked: base, TTL: tc.ttl}
if got := q.IsExpired(tc.now); got != tc.want {
t.Fatalf("IsExpired = %v, want %v", got, tc.want)
}
})
}
}
func TestClarifyStoreGetPutDelete(t *testing.T) {
s := NewClarifyStore(time.Minute)
if got := s.Get("voice", base); got != nil {
t.Fatalf("empty store returned %+v", got)
}
q := &PendingQuestion{Intent: IntentReminder, Missing: []Slot{SlotTime}, Asked: base}
s.Put("voice", q)
if q.TTL != time.Minute {
t.Fatalf("Put did not apply the default TTL, got %v", q.TTL)
}
if got := s.Get("voice", base.Add(time.Second)); got != q {
t.Fatalf("Get returned %+v, want the parked question", got)
}
// Expired questions are dropped on read, not returned.
if got := s.Get("voice", base.Add(2*time.Minute)); got != nil {
t.Fatalf("expired Get returned %+v", got)
}
if got := s.Get("voice", base); got != nil {
t.Fatalf("expired question was not deleted: %+v", got)
}
s.Put("voice", &PendingQuestion{Asked: base, TTL: time.Hour})
s.Delete("voice")
if got := s.Get("voice", base); got != nil {
t.Fatalf("Delete left %+v", got)
}
}
func TestNewClarifyStoreDefaultTTL(t *testing.T) {
s := NewClarifyStore(0)
q := &PendingQuestion{Asked: base}
s.Put("voice", q)
if q.TTL != 90*time.Second {
t.Fatalf("TTL = %v, want 90s", q.TTL)
}
}
func TestAnswerFillsOnlyMissingSlots(t *testing.T) {
answerTime := base.Add(3 * time.Hour)
other := base.Add(9 * time.Hour)
cases := []struct {
name string
parked Slots
missing []Slot
text string
answer Slots
want Slots
}{
{
name: "fills the missing time",
parked: Slots{Text: "напомни позвонить"},
missing: []Slot{SlotTime},
text: "в три",
answer: Slots{Time: answerTime, HasTime: true},
want: Slots{Text: "напомни позвонить", Time: answerTime, HasTime: true},
},
{
name: "does not overwrite a filled time",
parked: Slots{Time: other, HasTime: true},
missing: []Slot{SlotTime},
text: "в три",
answer: Slots{Time: answerTime, HasTime: true},
want: Slots{Time: other, HasTime: true},
},
{
name: "ignores slots that were not missing",
parked: Slots{Key: "water", HasKey: true},
missing: []Slot{SlotValue},
text: "два литра",
answer: Slots{Key: "sleep", HasKey: true, Value: "2l"},
want: Slots{Key: "water", HasKey: true, Value: "2l"},
},
{
name: "fills key when empty",
parked: Slots{},
missing: []Slot{SlotKey, SlotValue},
text: "воды",
answer: Slots{Key: "water", HasKey: true, Value: `"drank"`},
want: Slots{Key: "water", HasKey: true, Value: `"drank"`},
},
{
name: "fills fn and its args",
parked: Slots{},
missing: []Slot{SlotFn},
text: "перезапусти nginx",
answer: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
want: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
},
{
name: "keeps existing args when fn was already known",
parked: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
missing: []Slot{SlotFn},
text: "останови postgres",
answer: Slots{Fn: "stop", Args: []string{"postgres"}, HasFn: true},
want: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
},
{
name: "raw answer becomes the text when nothing was parsed",
parked: Slots{},
missing: []Slot{SlotText},
text: "купить хлеб",
answer: Slots{},
want: Slots{Text: "купить хлеб"},
},
{
name: "parsed text wins over the raw answer",
parked: Slots{},
missing: []Slot{SlotText},
text: "запиши купить хлеб",
answer: Slots{Text: "купить хлеб"},
want: Slots{Text: "купить хлеб"},
},
{
name: "empty answer leaves the slot missing",
parked: Slots{Text: "напомни"},
missing: []Slot{SlotTime},
text: "не знаю",
answer: Slots{},
want: Slots{Text: "напомни"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
q := &PendingQuestion{Slots: tc.parked, Missing: tc.missing, Asked: base}
got := q.Answer(tc.text, tc.answer)
if got.Time != tc.want.Time || got.HasTime != tc.want.HasTime ||
got.Key != tc.want.Key || got.HasKey != tc.want.HasKey ||
got.Value != tc.want.Value || got.Text != tc.want.Text ||
got.Fn != tc.want.Fn || got.HasFn != tc.want.HasFn {
t.Fatalf("Answer = %+v, want %+v", got, tc.want)
}
if len(got.Args) != len(tc.want.Args) {
t.Fatalf("Args = %v, want %v", got.Args, tc.want.Args)
}
for i := range got.Args {
if got.Args[i] != tc.want.Args[i] {
t.Fatalf("Args = %v, want %v", got.Args, tc.want.Args)
}
}
})
}
}
func TestCanAskCapsAtOneQuestion(t *testing.T) {
if MaxAttempts != 1 {
t.Fatalf("MaxAttempts = %d, want 1 (Maven asks once, she is not a nag)", MaxAttempts)
}
q := &PendingQuestion{Asked: base}
if !q.CanAsk() {
t.Fatal("a fresh question should be askable")
}
q.Attempts = MaxAttempts
if q.CanAsk() {
t.Fatal("the question should not be asked twice")
}
}
func TestStillMissing(t *testing.T) {
want := []Slot{SlotTime, SlotKey, SlotValue, SlotFn, SlotText}
cases := []struct {
name string
slots Slots
want []Slot
}{
{"all empty", Slots{}, want},
{
name: "all filled",
slots: Slots{Time: base, HasTime: true, Key: "water", HasKey: true, Value: "1l", Fn: "restart", HasFn: true, Text: "t"},
want: nil,
},
{
name: "only value left",
slots: Slots{Time: base, HasTime: true, Key: "water", HasKey: true, Fn: "restart", HasFn: true, Text: "t"},
want: []Slot{SlotValue},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := StillMissing(want, tc.slots)
if len(got) != len(tc.want) {
t.Fatalf("StillMissing = %v, want %v", got, tc.want)
}
for i := range got {
if got[i] != tc.want[i] {
t.Fatalf("StillMissing = %v, want %v", got, tc.want)
}
}
})
}
}
+4
View File
@@ -21,6 +21,7 @@ type Slots struct {
Time time.Time
HasTime bool
Key string
Value string // payload for a fact key, mirrors router.Slots.Value
HasKey bool
Text string
Fn string
@@ -104,6 +105,9 @@ func InheritSlots(prev, cur Slots) Slots {
out.Key = prev.Key
out.HasKey = true
}
if out.Value == "" && prev.Value != "" {
out.Value = prev.Value
}
if out.Text == "" && prev.Text != "" {
out.Text = prev.Text
}
+10
View File
@@ -113,4 +113,14 @@ func TestInheritSlots(t *testing.T) {
if inherited6.Text != "какая погода в москве" {
t.Error("should inherit text when current is empty")
}
prevValue := Slots{Key: "water", HasKey: true, Value: `"drank"`}
inherited7 := InheritSlots(prevValue, Slots{})
if inherited7.Value != `"drank"` {
t.Error("should inherit value when current is empty")
}
kept := InheritSlots(prevValue, Slots{Value: "2l"})
if kept.Value != "2l" {
t.Error("should keep current value")
}
}
-472
View File
@@ -1,472 +0,0 @@
// Package recalleval is the held-out contract for note recall: can Maven find
// the right note again when the user asks for it weeks later?
//
// Why it sits beside internal/memory rather than inside it: the thing under
// test is a whole path, not one function — an embedder (internal/router), a
// vector store (internal/memory or internal/store) and the confidence gate the
// daemon applies on top (cmd/mavend/recall.go's bestRecall, config's
// query_min_score). A _test.go file inside internal/memory could not reach the
// persistent store without an import cycle, and testdata is not reachable from
// another package's working directory — so the fixture is embedded here and the
// scorer takes the store as a factory. Same layout and same reasons as
// internal/router/eval.
//
// The fixture is HELD OUT the same way the routing fixture is: a query never
// repeats its note's wording verbatim beyond ordinary shared vocabulary, and
// TestFixtureIsParaphrased enforces a floor on how little the two overlap.
// Scoring recall on a query that is a copy of the note measures string
// matching, not recall.
package recalleval
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/kami/maven/internal/memory"
"github.com/kami/maven/internal/router"
)
//go:embed ru_recall_v1.json
var fixtureJSON []byte
// SchemaVersion — the version this package understands. The loader refuses any
// other version rather than misreading a fixture and reporting a number.
const SchemaVersion = 1
// StoredNote — one thing the user said once, as it lands in the semantic store.
// Kind is "note" or "fact"; both share the vector index (see
// cmd/mavend/recall.go), so a fact can legitimately win a recall.
type StoredNote struct {
ID string `json:"id"`
Text string `json:"text"`
Kind string `json:"kind"`
}
// Case — a small set of notes, one query, and the note that must come back
// first. Want is empty exactly when the query should recall NOTHING: that lane
// measures false recall, which is the direction the spec cares about ("a
// confident wrong fact is worse than a known gap").
type Case struct {
ID string `json:"id"`
Lang string `json:"lang"`
Notes []StoredNote `json:"notes"`
Query string `json:"query"`
Want string `json:"want"`
Tags []string `json:"tags"`
Note string `json:"note"`
}
// Answerable reports whether the case expects a recall at all.
func (c Case) Answerable() bool { return c.Want != "" }
// Fixture — the versioned envelope, same shape as the routing fixture.
//
// Filler is inserted into EVERY case's store on top of that case's own notes.
// Without it a case with three notes scores recall@3 = 100% by construction,
// which measures nothing. A real store holds months of unrelated notes, and the
// wanted note has to beat all of them.
type Fixture struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
Notes []string `json:"notes"`
Filler []StoredNote `json:"filler"`
Cases []Case `json:"cases"`
}
// Load returns the embedded fixture.
func Load() (Fixture, error) {
var f Fixture
if err := json.Unmarshal(fixtureJSON, &f); err != nil {
return Fixture{}, fmt.Errorf("parse fixture: %w", err)
}
if f.SchemaVersion != SchemaVersion {
return Fixture{}, fmt.Errorf("fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
}
if len(f.Cases) == 0 {
return Fixture{}, fmt.Errorf("fixture has no cases")
}
return f, nil
}
// NewStore builds an empty store for one case, plus a function to release it.
// A factory rather than a store because every case needs a clean index — notes
// from case A must not be visible to case B's query.
type NewStore func() (memory.Store, func(), error)
// InMemory is the NewStore for memory.InMemoryStore — the fallback the daemon
// uses when there is no database (cmd/mavend/voice.go:224).
func InMemory() (memory.Store, func(), error) {
return memory.NewInMemoryStore(), func() {}, nil
}
// Cache wraps an embedder so repeated text is embedded once. The gate sweep
// scores the same fixture at nine thresholds, and every case re-inserts the
// filler notes — without this the ONNX run spends minutes re-embedding
// identical strings. Latency numbers come from the uncached run.
func Cache(inner router.Embedder) router.Embedder {
return &cachingEmbedder{inner: inner, seen: map[string][]float32{}}
}
type cachingEmbedder struct {
inner router.Embedder
seen map[string][]float32
}
func (c *cachingEmbedder) Dim() int { return c.inner.Dim() }
func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner
func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
if v, ok := c.seen[text]; ok {
return v, nil
}
v, err := c.inner.Embed(ctx, text)
if err != nil {
return nil, err
}
c.seen[text] = v
return v, nil
}
// Outcome — one scored case.
type Outcome struct {
Case Case
Hits []memory.Result
Err error
// Latency is the read path only: embed the query, then Search. Insert time
// is excluded because it happens once, weeks earlier.
Latency time.Duration
// Rank1/Rank3 — the wanted note came back first / in the top three,
// ignoring the confidence gate. Ranking is the store's job.
Rank1 bool
Rank3 bool
// Recalled — what the daemon would actually say back: the top hit's text
// when it clears the gate. Mirrors bestRecall in cmd/mavend/recall.go.
Recalled string
// Pass — the wanted note was recalled AND survived the gate; or, for a
// no-answer case, nothing was recalled.
Pass bool
// Tied — the wanted note is on top but shares its score with the next hit,
// so the sort decided it, not the embedder. Counted apart from a real hit.
Tied bool
TopID string
TopScor float64
Reasons []string
}
// Report — the aggregate. Rank and gate are kept apart on purpose: a note that
// ranks first but is silenced by query_min_score is a threshold problem, and a
// note that never ranks first is an embedder problem. Those are different fixes.
type Report struct {
Name string
MinScore float64
Total int
Answerable int
Rank1 int
Rank3 int
// Gated — ranked first but the score was under MinScore, so the daemon
// stays silent and answers "не знаю".
Gated int
// WrongTop — a different note outranked the right one.
WrongTop int
// Tied — the right note was on top only because of sort order. Not credited
// as recall; tracked because it is a distinct failure (the embedder scored
// the query and the note the same as everything else).
Tied int
// NoAnswer / FalseRecall — the cases that must recall nothing, and how many
// of them the daemon would answer anyway.
NoAnswer int
FalseRecall int
Errors int
Passed int
Outcomes []Outcome
ByTag map[string]TagStat
ByLang map[string]TagStat
// CorrectTop / NoAnswerTop — sorted top-1 scores for the answerable cases
// where the right note ranked first, and for the no-answer cases. The gap
// between these two distributions is what a defensible query_min_score
// would have to sit inside; if they overlap, no threshold separates them.
CorrectTop []float64
NoAnswerTop []float64
P50, P95, Max time.Duration
}
// TagStat — passed/total for one slice of the fixture.
type TagStat struct{ Passed, Total int }
// Recall1 — fraction of answerable cases whose wanted note ranked first.
func (r Report) Recall1() float64 { return ratio(r.Rank1, r.Answerable) }
// Recall3 — same, in the top three. The daemon asks for 3 (voice.go), so this
// is the ceiling a better gate or a reranker could reach.
func (r Report) Recall3() float64 { return ratio(r.Rank3, r.Answerable) }
// Answered — fraction of answerable cases the daemon would actually answer
// correctly, gate included. This is the number the operator experiences.
func (r Report) Answered() float64 { return ratio(r.Rank1-r.Gated, r.Answerable) }
// FalseRecallRate — fraction of the no-answer cases the daemon answers anyway.
func (r Report) FalseRecallRate() float64 { return ratio(r.FalseRecall, r.NoAnswer) }
func ratio(n, d int) float64 {
if d == 0 {
return 0
}
return float64(n) / float64(d)
}
// Score runs every case against a fresh store and aggregates. It never fails
// the run on an embed or search error: an erroring case scores as a miss and is
// counted in Errors, because "the embedder was down" and "the embedder was
// wrong" are different numbers.
func Score(ctx context.Context, name string, emb router.Embedder, newStore NewStore, minScore float64, f Fixture) (Report, error) {
rep := Report{
Name: name,
MinScore: minScore,
Total: len(f.Cases),
ByTag: map[string]TagStat{},
ByLang: map[string]TagStat{},
}
lat := make([]time.Duration, 0, len(f.Cases))
for _, c := range f.Cases {
if c.Answerable() {
rep.Answerable++
} else {
rep.NoAnswer++
}
o, err := scoreCase(ctx, emb, newStore, minScore, c, f.Filler)
if err != nil {
return Report{}, err
}
lat = append(lat, o.Latency)
switch {
case o.Err != nil:
rep.Errors++
case c.Answerable():
if o.Rank1 {
rep.Rank1++
}
if o.Rank3 {
rep.Rank3++
}
if o.Rank1 && o.Recalled == "" {
rep.Gated++
}
if o.Tied {
rep.Tied++
} else if !o.Rank1 {
rep.WrongTop++
}
if o.Rank1 && o.Recalled != "" {
rep.CorrectTop = append(rep.CorrectTop, o.TopScor)
}
default:
if o.Recalled != "" {
rep.FalseRecall++
}
rep.NoAnswerTop = append(rep.NoAnswerTop, o.TopScor)
}
if o.Pass {
rep.Passed++
}
bump(rep.ByLang, c.Lang, o.Pass)
for _, tag := range c.Tags {
bump(rep.ByTag, tag, o.Pass)
}
rep.Outcomes = append(rep.Outcomes, o)
}
sort.Float64s(rep.CorrectTop)
sort.Float64s(rep.NoAnswerTop)
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
if len(lat) > 0 {
rep.Max = lat[len(lat)-1]
}
return rep, nil
}
// scoreCase inserts the case's notes into a fresh store, then runs the read
// path the daemon runs. The returned error is fatal (the harness is broken);
// an embedder or store failure on the query lands in Outcome.Err instead.
func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minScore float64, c Case, filler []StoredNote) (Outcome, error) {
st, release, err := newStore()
if err != nil {
return Outcome{}, fmt.Errorf("%s: new store: %w", c.ID, err)
}
defer release()
all := append(append([]StoredNote(nil), c.Notes...), filler...)
for _, n := range all {
vec, err := emb.Embed(ctx, n.Text)
if err != nil {
return Outcome{}, fmt.Errorf("%s: embed note %s: %w", c.ID, n.ID, err)
}
meta := map[string]string{"text": n.Text, "type": n.Kind}
if err := st.Insert(ctx, n.ID, vec, meta); err != nil {
return Outcome{}, fmt.Errorf("%s: insert %s: %w", c.ID, n.ID, err)
}
}
o := Outcome{Case: c}
start := time.Now()
qvec, err := emb.Embed(ctx, c.Query)
if err != nil {
o.Latency = time.Since(start)
o.Err = err
o.Reasons = []string{fmt.Sprintf("embed query: %v", err)}
return o, nil
}
hits, err := st.Search(ctx, qvec, 3)
o.Latency = time.Since(start)
if err != nil {
o.Err = err
o.Reasons = []string{fmt.Sprintf("search: %v", err)}
return o, nil
}
o.Hits = hits
if len(hits) > 0 {
o.TopID, o.TopScor = hits[0].ID, hits[0].Score
o.Recalled = bestRecall(hits, minScore)
}
for i, h := range hits {
if h.ID != c.Want {
continue
}
o.Rank3 = true
// A tie is not a hit. With a lexical embedder several notes score
// exactly 0 against a paraphrased query, and whichever one the sort
// happens to leave on top would otherwise be credited as recall.
if i == 0 && (len(hits) < 2 || hits[0].Score > hits[1].Score) {
o.Rank1 = true
}
if i == 0 && !o.Rank1 {
o.Tied = true
}
}
switch {
case !c.Answerable():
if o.Recalled != "" {
o.Reasons = append(o.Reasons, fmt.Sprintf("false recall: %q at %.3f, want silence", o.TopID, o.TopScor))
}
case o.Tied:
o.Reasons = append(o.Reasons, fmt.Sprintf("tie at %.3f — the right note is on top only by sort order", o.TopScor))
case !o.Rank1:
o.Reasons = append(o.Reasons, fmt.Sprintf("top hit %q (%.3f), want %q%s", o.TopID, o.TopScor, c.Want, rankNote(o.Rank3)))
case o.Recalled == "":
o.Reasons = append(o.Reasons, fmt.Sprintf("right note ranked first but scored %.3f < gate %.2f — daemon says \"не знаю\"", o.TopScor, minScore))
}
o.Pass = len(o.Reasons) == 0
return o, nil
}
func rankNote(inTop3 bool) string {
if inTop3 {
return " (wanted note is in the top 3)"
}
return " (wanted note is not in the top 3)"
}
// bestRecall mirrors cmd/mavend/recall.go — the gate the daemon actually
// applies to a memory hit. Duplicated rather than imported because package main
// is not importable; recalleval_test.go asserts the two agree in behaviour.
func bestRecall(results []memory.Result, min float64) string {
if len(results) == 0 || results[0].Score < min {
return ""
}
return results[0].Meta["text"]
}
func bump(m map[string]TagStat, key string, pass bool) {
if key == "" {
return
}
s := m[key]
s.Total++
if pass {
s.Passed++
}
m[key] = s
}
// percentile — nearest-rank on a pre-sorted slice. No interpolation: with ~30
// samples an interpolated p95 invents a latency no query actually took.
func percentile(sorted []time.Duration, p float64) time.Duration {
if len(sorted) == 0 {
return 0
}
i := int(p * float64(len(sorted)))
if i >= len(sorted) {
i = len(sorted) - 1
}
return sorted[i]
}
// String renders the report in the routing eval's style — headline first, then
// the slices that name where the path is weak.
func (r Report) String() string {
var b strings.Builder
fmt.Fprintf(&b, "%s: %d/%d cases pass (gate %.2f)\n", r.Name, r.Passed, r.Total, r.MinScore)
fmt.Fprintf(&b, " recall@1 %.1f%% (%d/%d) recall@3 %.1f%% (%d/%d) answered after gate %.1f%% (%d/%d)\n",
100*r.Recall1(), r.Rank1, r.Answerable,
100*r.Recall3(), r.Rank3, r.Answerable,
100*r.Answered(), r.Rank1-r.Gated, r.Answerable)
fmt.Fprintf(&b, " wrong note on top: %d | tie on top (sort order, not recall): %d | silenced by gate: %d | errors: %d\n",
r.WrongTop, r.Tied, r.Gated, r.Errors)
fmt.Fprintf(&b, " false recall %.1f%% (%d/%d must-be-silent cases answered anyway)\n",
100*r.FalseRecallRate(), r.FalseRecall, r.NoAnswer)
fmt.Fprintf(&b, " top-1 score, right note first: %s\n", spread(r.CorrectTop))
fmt.Fprintf(&b, " top-1 score, must be silent: %s\n", spread(r.NoAnswerTop))
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
fmt.Fprintf(&b, " by lang: %s\n", renderStats(r.ByLang))
fmt.Fprintf(&b, " by tag: %s\n", renderStats(r.ByTag))
return b.String()
}
// Failures — per-case detail, sorted by ID so two runs diff cleanly.
func (r Report) Failures() string {
var b strings.Builder
out := append([]Outcome(nil), r.Outcomes...)
sort.Slice(out, func(i, j int) bool { return out[i].Case.ID < out[j].Case.ID })
for _, o := range out {
if o.Pass {
continue
}
fmt.Fprintf(&b, " %s %q: %s\n", o.Case.ID, o.Case.Query, strings.Join(o.Reasons, "; "))
}
return b.String()
}
// spread — min / median / max of a sorted score list. Three numbers is enough
// to see whether two distributions overlap, which is the only question a
// threshold can answer.
func spread(sorted []float64) string {
if len(sorted) == 0 {
return "n/a"
}
return fmt.Sprintf("min %.3f median %.3f max %.3f (n=%d)",
sorted[0], sorted[len(sorted)/2], sorted[len(sorted)-1], len(sorted))
}
func renderStats(m map[string]TagStat) string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
parts := make([]string, 0, len(keys))
for _, k := range keys {
s := m[k]
parts = append(parts, fmt.Sprintf("%s %d/%d", k, s.Passed, s.Total))
}
return strings.Join(parts, " ")
}
@@ -1,292 +0,0 @@
package recalleval
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"unicode"
"github.com/kami/maven/internal/config"
"github.com/kami/maven/internal/memory"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/store"
)
// dim 1024 for the hash embedder: it is bag-of-words, so a narrower space
// collides tokens between unrelated notes and would measure the hash.
const hashDim = 1024
func TestLoadFixture(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if len(f.Cases) < 25 {
t.Errorf("%d cases, want >= 25", len(f.Cases))
}
// Filler is what stops recall@3 being free: three case notes and a top-3
// search would put the wanted note in the top 3 every time.
if len(f.Filler) < 10 {
t.Errorf("%d filler notes, want >= 10", len(f.Filler))
}
seen := map[string]bool{}
silent, en := 0, 0
for _, c := range f.Cases {
if c.ID == "" || seen[c.ID] {
t.Errorf("case %q: empty or duplicate id", c.ID)
}
seen[c.ID] = true
if c.Lang != "ru" && c.Lang != "en" {
t.Errorf("%s: lang %q, want ru|en", c.ID, c.Lang)
}
if c.Lang == "en" {
en++
}
if strings.TrimSpace(c.Query) == "" {
t.Errorf("%s: empty query", c.ID)
}
// Fewer than three notes and a wrong answer has nowhere to come from,
// so recall@1 would be near-free.
if len(c.Notes) < 3 {
t.Errorf("%s: %d notes, want >= 3", c.ID, len(c.Notes))
}
ids := map[string]bool{}
for _, n := range c.Notes {
if n.ID == "" || ids[n.ID] {
t.Errorf("%s: note %q empty or duplicate id", c.ID, n.ID)
}
ids[n.ID] = true
if strings.TrimSpace(n.Text) == "" {
t.Errorf("%s: note %q empty text", c.ID, n.ID)
}
if n.Kind != "note" && n.Kind != "fact" {
t.Errorf("%s: note %q kind %q, want note|fact", c.ID, n.ID, n.Kind)
}
}
if !c.Answerable() {
silent++
continue
}
if !ids[c.Want] {
t.Errorf("%s: want %q is not one of the case's notes", c.ID, c.Want)
}
}
// Both lanes need enough cases that a rate means something.
if silent < 5 {
t.Errorf("%d must-be-silent cases, want >= 5", silent)
}
if en < 5 {
t.Errorf("%d English cases, want >= 5", en)
}
}
// TestFixtureIsParaphrased — the fixture's claim to measuring recall at all. If
// a query repeats its note's words, cosine over a bag-of-words embedder gets it
// for free and the score says nothing about semantic recall. Half the query's
// words is the line: some shared vocabulary is natural ("nginx", "чай"), a copy
// is the failure.
func TestFixtureIsParaphrased(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
for _, c := range f.Cases {
if !c.Answerable() {
continue
}
var want string
for _, n := range c.Notes {
if n.ID == c.Want {
want = n.Text
}
}
q := words(c.Query)
if len(q) == 0 {
continue
}
inNote := map[string]bool{}
for _, w := range words(want) {
inNote[w] = true
}
shared := 0
for _, w := range q {
if inNote[w] {
shared++
}
}
if frac := float64(shared) / float64(len(q)); frac > 0.5 {
t.Errorf("%s: query shares %.0f%% of its words with the note — not a paraphrase\n query: %q\n note: %q",
c.ID, 100*frac, c.Query, want)
}
}
}
// words — lowercased words of two runes or more, matching how the hash
// embedder tokenizes.
func words(s string) []string {
var out []string
for _, w := range strings.FieldsFunc(strings.ToLower(s), func(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
}) {
if len([]rune(w)) > 1 {
out = append(out, w)
}
}
return out
}
// TestBestRecallMatchesDaemon — the harness duplicates bestRecall from
// cmd/mavend/recall.go (package main is not importable). This pins the copy to
// the original's three rules: no hits, below the gate, or no text ⇒ silence.
func TestBestRecallMatchesDaemon(t *testing.T) {
if got := bestRecall(nil, 0.55); got != "" {
t.Errorf("no hits: got %q, want silence", got)
}
low := []memory.Result{{ID: "a", Score: 0.4, Meta: map[string]string{"text": "чай"}}}
if got := bestRecall(low, 0.55); got != "" {
t.Errorf("below gate: got %q, want silence", got)
}
noText := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{}}}
if got := bestRecall(noText, 0.55); got != "" {
t.Errorf("no text: got %q, want silence", got)
}
ok := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{"text": "чай"}}}
if got := bestRecall(ok, 0.55); got != "чай" {
t.Errorf("above gate: got %q, want %q", got, "чай")
}
}
// TestHashRecallBaseline — the CI ratchet. HashEmbedder, so it needs no model
// files and is byte-for-byte reproducible.
//
// It is a floor, not a target. The hash embedder is lexical, so most of this
// fixture is unwinnable for it by construction; the number worth moving is
// TestONNXRecall's. Never compare a hash-embedder number to an ONNX one.
func TestHashRecallBaseline(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
rep, err := Score(context.Background(), "recall+hash", router.NewHashEmbedder(hashDim), InMemory,
config.DefaultQueryMinScore, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String() + rep.Failures())
t.Log("\ngate sweep:\n" + sweep(t, router.NewHashEmbedder(hashDim), f))
// 0.32 sits under the observed 0.360 recall@1.
const floorRecall1 = 0.32
if rep.Recall1() < floorRecall1 {
t.Errorf("recall@1 %.3f below ratchet %.2f — note recall regressed", rep.Recall1(), floorRecall1)
}
// The dangerous direction, asserted tightly and separately: answering from
// the wrong note is worse than a gap. Observed 0 under the hash floor.
if rep.FalseRecall > 1 {
t.Errorf("%d false recalls, want <= 1:\n%s", rep.FalseRecall, rep.Failures())
}
}
// TestPersistentStoreScoresTheSame — the deployed store is sqlite-backed
// (store.MemoryStore via st.VectorMemory()), not the in-memory fallback. Its
// Search is a separate implementation of the same cosine scan, so it gets its
// own run: a divergence here would mean recall quality depends on whether a
// database was configured.
func TestPersistentStoreScoresTheSame(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
emb := router.NewHashEmbedder(hashDim)
inMem, err := Score(context.Background(), "recall+hash+memory", emb, InMemory, config.DefaultQueryMinScore, f)
if err != nil {
t.Fatalf("Score in-memory: %v", err)
}
persistent, err := Score(context.Background(), "recall+hash+sqlite", emb, sqliteStores(t), config.DefaultQueryMinScore, f)
if err != nil {
t.Fatalf("Score sqlite: %v", err)
}
t.Log("\n" + persistent.String())
if persistent.Rank1 != inMem.Rank1 || persistent.FalseRecall != inMem.FalseRecall {
t.Errorf("sqlite recall@1 %d/%d fr %d, in-memory %d/%d fr %d — the two backends disagree",
persistent.Rank1, persistent.Answerable, persistent.FalseRecall,
inMem.Rank1, inMem.Answerable, inMem.FalseRecall)
}
}
// sqliteStores returns a NewStore that hands each case its own plaintext
// database file, so cases stay isolated the way they are with InMemory.
func sqliteStores(t *testing.T) NewStore {
t.Helper()
dir := t.TempDir()
n := 0
return func() (memory.Store, func(), error) {
n++
st, err := store.Open(context.Background(), filepath.Join(dir, fmt.Sprintf("recall-%d.db", n)))
if err != nil {
return nil, nil, err
}
return st.VectorMemory(), func() { _ = st.Close() }, nil
}
}
// TestONNXRecall — the number that matters: the multilingual embedder homesrv
// actually runs. Opt-in via MAVEN_ONNX_LIB because deps/ is gitignored, exactly
// like TestONNXBaseline in internal/router/eval. `make eval-recall` points it at
// the vendored runtime.
//
// Reports rather than asserts. The gate sweep is the point: it prints
// answered-vs-false-recall at a range of query_min_score values, so the right
// threshold is read off data instead of guessed.
func TestONNXRecall(t *testing.T) {
lib := os.Getenv("MAVEN_ONNX_LIB")
if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
}
model := filepath.Join("../../..", "models/embedder/model.onnx")
tok := filepath.Join("../../..", "models/embedder/tokenizer.json")
for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err)
}
}
emb, err := router.NewONNXEmbedder(model, tok, lib)
if err != nil {
t.Skipf("onnx embedder unavailable: %v", err)
}
defer emb.Close()
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
rep, err := Score(context.Background(), "recall+onnx", emb, InMemory, config.DefaultQueryMinScore, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String() + rep.Failures())
// Cached for the sweep only: the headline run above must pay the real
// embedder cost so its latency numbers mean something.
t.Log("\ngate sweep:\n" + sweep(t, Cache(emb), f))
}
// sweep scores the fixture at a range of gates and renders one line each. Two
// columns matter: how many real questions get answered, and how many made-up
// ones get answered anyway. A gate is only defensible if some value keeps the
// first high and the second at zero.
func sweep(t *testing.T, emb router.Embedder, f Fixture) string {
t.Helper()
var b strings.Builder
for _, gate := range []float64{0.0, 0.30, 0.40, 0.50, 0.55, 0.60, 0.70, 0.80, 0.90} {
rep, err := Score(context.Background(), "sweep", emb, InMemory, gate, f)
if err != nil {
t.Fatalf("sweep at %.2f: %v", gate, err)
}
fmt.Fprintf(&b, " gate %.2f: answered %d/%d (%.0f%%) false recall %d/%d\n",
gate, rep.Rank1-rep.Gated, rep.Answerable, 100*rep.Answered(), rep.FalseRecall, rep.NoAnswer)
}
return b.String()
}
@@ -1,392 +0,0 @@
{
"schema_version": 1,
"name": "ru_recall_v1",
"notes": [
"Held-out note-recall fixture. Each case is a fresh semantic store: insert every note, embed the query, take the top 3 — the same read path cmd/mavend/voice.go runs for IntentQuery.",
"Queries paraphrase their note on purpose. A query that repeats the note's words measures string matching, not recall. TestFixtureIsParaphrased enforces a ceiling on word overlap.",
"want:\"\" means the query must recall NOTHING. Those cases measure false recall — the direction the spec calls out (a confident wrong fact is worse than a known gap).",
"The distractor tag marks cases where a second note is plausible and only one is right. The hard tag marks cases with little or no shared vocabulary.",
"Content is written for this operator: his preferences, his homelab, things he said once and would expect Maven to remember weeks later."
],
"filler": [
{"id": "f1", "text": "в субботу ходил в баню", "kind": "note"},
{"id": "f2", "text": "купил новые кроссовки сорок третьего размера", "kind": "note"},
{"id": "f3", "text": "сериал закончился на третьем сезоне", "kind": "note"},
{"id": "f4", "text": "сосед сверху делает ремонт", "kind": "note"},
{"id": "f5", "text": "билеты в театр брал заранее", "kind": "note"},
{"id": "f6", "text": "выучил пару аккордов на гитаре", "kind": "note"},
{"id": "f7", "text": "записался к стоматологу", "kind": "note"},
{"id": "f8", "text": "поменял лампочку в коридоре", "kind": "note"},
{"id": "f9", "text": "погулял вдоль реки", "kind": "note"},
{"id": "f10", "text": "the balcony door sticks in winter", "kind": "note"},
{"id": "f11", "text": "the neighbour's dog barks at cyclists", "kind": "note"},
{"id": "f12", "text": "i finished the book about volcanoes", "kind": "note"}
],
"cases": [
{
"id": "ru-pref-001",
"lang": "ru",
"tags": ["preference", "paraphrase"],
"query": "какой кофе мне наливать",
"want": "n1",
"notes": [
{"id": "n1", "text": "я пью кофе без сахара", "kind": "note"},
{"id": "n2", "text": "по утрам бегаю в парке", "kind": "note"},
{"id": "n3", "text": "не люблю громкую музыку", "kind": "note"}
]
},
{
"id": "ru-pref-002",
"lang": "ru",
"tags": ["preference", "homelab", "paraphrase", "hard"],
"query": "когда запускать резервное копирование",
"want": "n1",
"note": "The DESIGN.md preference-seam example, phrased as the operator would ask it later.",
"notes": [
{"id": "n1", "text": "бэкапы лучше делать ночью в три часа", "kind": "note"},
{"id": "n2", "text": "обновления ставлю по субботам", "kind": "note"},
{"id": "n3", "text": "логи храню месяц", "kind": "note"}
]
},
{
"id": "ru-home-003",
"lang": "ru",
"tags": ["homelab", "paraphrase"],
"query": "что помогло от мерцания монитора",
"want": "n1",
"notes": [
{"id": "n1", "text": "мерцание экрана прошло после обновления драйвера amdgpu", "kind": "note"},
{"id": "n2", "text": "вентилятор шумит на полной нагрузке", "kind": "note"},
{"id": "n3", "text": "поставил новый ssd в ноутбук", "kind": "note"}
]
},
{
"id": "ru-home-004",
"lang": "ru",
"tags": ["homelab", "distractor", "hard"],
"query": "адрес домашнего сервера",
"want": "n2",
"note": "Two notes carry an IP. Only one is the server.",
"notes": [
{"id": "n1", "text": "роутер живёт на 192.168.1.1", "kind": "note"},
{"id": "n2", "text": "домашний сервер на 192.168.1.104", "kind": "note"},
{"id": "n3", "text": "принтер подключен по usb", "kind": "note"}
]
},
{
"id": "ru-home-005",
"lang": "ru",
"tags": ["homelab"],
"query": "где искать настройки nginx",
"want": "n1",
"notes": [
{"id": "n1", "text": "конфиг nginx лежит в /etc/nginx/sites-enabled", "kind": "note"},
{"id": "n2", "text": "сертификаты обновляет certbot по расписанию", "kind": "note"},
{"id": "n3", "text": "порт 8080 занят вебкой", "kind": "note"}
]
},
{
"id": "ru-pref-006",
"lang": "ru",
"tags": ["preference", "distractor", "hard"],
"query": "что мне нельзя есть",
"want": "n1",
"note": "The Friday-meat note is a plausible second answer but it is a habit, not a restriction.",
"notes": [
{"id": "n1", "text": "у меня аллергия на орехи", "kind": "note"},
{"id": "n2", "text": "не ем мясо по пятницам", "kind": "note"},
{"id": "n3", "text": "люблю острую еду", "kind": "note"}
]
},
{
"id": "ru-pers-007",
"lang": "ru",
"tags": ["distractor", "paraphrase"],
"query": "когда мамин праздник",
"want": "n1",
"notes": [
{"id": "n1", "text": "день рождения мамы четырнадцатого марта", "kind": "note"},
{"id": "n2", "text": "у брата день рождения в июле", "kind": "note"},
{"id": "n3", "text": "годовщина в сентябре", "kind": "note"}
]
},
{
"id": "ru-home-008",
"lang": "ru",
"tags": ["homelab", "hard", "paraphrase"],
"query": "чем ускоряется языковая модель",
"want": "n1",
"notes": [
{"id": "n1", "text": "модель крутится на встройке через vulkan", "kind": "note"},
{"id": "n2", "text": "whisper работает на процессоре", "kind": "note"},
{"id": "n3", "text": "голос у piper русский", "kind": "note"}
]
},
{
"id": "ru-pref-009",
"lang": "ru",
"tags": ["preference", "distractor"],
"query": "во сколько я обычно засыпаю",
"want": "n1",
"notes": [
{"id": "n1", "text": "ложусь спать около часа ночи", "kind": "note"},
{"id": "n2", "text": "встаю в семь утра", "kind": "note"},
{"id": "n3", "text": "днём не сплю", "kind": "note"}
]
},
{
"id": "ru-home-010",
"lang": "ru",
"tags": ["homelab", "paraphrase"],
"query": "где у меня хранятся пароли",
"want": "n1",
"notes": [
{"id": "n1", "text": "пароли держу в keepassxc", "kind": "note"},
{"id": "n2", "text": "двухфакторку сделал через totp", "kind": "note"},
{"id": "n3", "text": "ssh ключи лежат на юбикее", "kind": "note"}
]
},
{
"id": "ru-home-011",
"lang": "ru",
"tags": ["homelab", "hard", "paraphrase"],
"query": "из-за чего кончилось место",
"want": "n1",
"notes": [
{"id": "n1", "text": "диск забился логами докера в июне", "kind": "note"},
{"id": "n2", "text": "рейд собрал из двух дисков", "kind": "note"},
{"id": "n3", "text": "бэкап на внешний диск раз в неделю", "kind": "note"}
]
},
{
"id": "ru-pref-012",
"lang": "ru",
"tags": ["preference", "distractor"],
"query": "какой чай мне нравится",
"want": "n1",
"notes": [
{"id": "n1", "text": "чай пью только зелёный", "kind": "note"},
{"id": "n2", "text": "кофе пью без сахара", "kind": "note"},
{"id": "n3", "text": "воду пью из фильтра", "kind": "note"}
]
},
{
"id": "ru-silent-013",
"lang": "ru",
"tags": ["silent"],
"query": "какая погода будет в пятницу",
"want": "",
"notes": [
{"id": "n1", "text": "роутер живёт на 192.168.1.1", "kind": "note"},
{"id": "n2", "text": "бэкапы лучше делать ночью", "kind": "note"},
{"id": "n3", "text": "у меня аллергия на орехи", "kind": "note"}
]
},
{
"id": "ru-silent-014",
"lang": "ru",
"tags": ["silent"],
"query": "как зовут сестру моего коллеги",
"want": "",
"notes": [
{"id": "n1", "text": "конфиг nginx лежит в /etc/nginx/sites-enabled", "kind": "note"},
{"id": "n2", "text": "порт 8080 занят вебкой", "kind": "note"},
{"id": "n3", "text": "сертификаты обновляет certbot", "kind": "note"}
]
},
{
"id": "ru-silent-015",
"lang": "ru",
"tags": ["silent"],
"query": "сколько я заплатил за машину",
"want": "",
"notes": [
{"id": "n1", "text": "чай пью только зелёный", "kind": "note"},
{"id": "n2", "text": "ложусь спать около часа ночи", "kind": "note"},
{"id": "n3", "text": "не люблю громкую музыку", "kind": "note"}
]
},
{
"id": "ru-home-016",
"lang": "ru",
"tags": ["homelab", "paraphrase"],
"query": "откуда берётся токен бота",
"want": "n1",
"notes": [
{"id": "n1", "text": "токен телеграма лежит в deploy/telegram.env", "kind": "note"},
{"id": "n2", "text": "вебхуки не использую, только long-poll", "kind": "note"},
{"id": "n3", "text": "уведомления приходят в личку", "kind": "note"}
]
},
{
"id": "ru-hard-017",
"lang": "ru",
"tags": ["hard", "paraphrase", "homelab"],
"query": "как я восстановил конфиги",
"want": "n1",
"note": "No shared word between query and note beyond none at all. This is the case a lexical embedder cannot win.",
"notes": [
{"id": "n1", "text": "после переустановки системы вернул все настройки из git", "kind": "note"},
{"id": "n2", "text": "разделы на диске резал вручную", "kind": "note"},
{"id": "n3", "text": "загрузчик поставил заново", "kind": "note"}
]
},
{
"id": "ru-dist-018",
"lang": "ru",
"tags": ["distractor"],
"query": "чем кормить кота",
"want": "n1",
"notes": [
{"id": "n1", "text": "кот ест только сухой корм", "kind": "note"},
{"id": "n2", "text": "собаке даю мясо", "kind": "note"},
{"id": "n3", "text": "рыбок кормлю раз в день", "kind": "note"}
]
},
{
"id": "ru-home-019",
"lang": "ru",
"tags": ["homelab", "hard", "paraphrase"],
"query": "как контейнер получает доступ к видеокарте",
"want": "n1",
"notes": [
{"id": "n1", "text": "docker compose пробрасывает /dev/dri внутрь", "kind": "note"},
{"id": "n2", "text": "контейнеры рестартуют сами", "kind": "note"},
{"id": "n3", "text": "образы чищу вручную", "kind": "note"}
]
},
{
"id": "ru-pref-020",
"lang": "ru",
"tags": ["preference", "distractor"],
"query": "когда мне нельзя звонить",
"want": "n1",
"notes": [
{"id": "n1", "text": "не звони мне после десяти вечера", "kind": "note"},
{"id": "n2", "text": "утром не трогай меня до кофе", "kind": "note"},
{"id": "n3", "text": "по выходным не работаю", "kind": "note"}
]
},
{
"id": "en-pref-021",
"lang": "en",
"tags": ["preference", "paraphrase"],
"query": "which colour scheme do i like",
"want": "n1",
"notes": [
{"id": "n1", "text": "i prefer dark theme everywhere", "kind": "note"},
{"id": "n2", "text": "font size 14 is fine", "kind": "note"},
{"id": "n3", "text": "i use vim keybindings", "kind": "note"}
]
},
{
"id": "en-home-022",
"lang": "en",
"tags": ["homelab", "distractor"],
"query": "where is the big disk mounted",
"want": "n1",
"notes": [
{"id": "n1", "text": "the nas drive is mounted at /mnt/hdd1", "kind": "note"},
{"id": "n2", "text": "models live on the ssd", "kind": "note"},
{"id": "n3", "text": "backups go to the nas nightly", "kind": "note"}
]
},
{
"id": "en-silent-023",
"lang": "en",
"tags": ["silent"],
"query": "what is my bank account number",
"want": "",
"notes": [
{"id": "n1", "text": "the nas drive is mounted at /mnt/hdd1", "kind": "note"},
{"id": "n2", "text": "i prefer dark theme everywhere", "kind": "note"},
{"id": "n3", "text": "the router runs openwrt", "kind": "note"}
]
},
{
"id": "en-hard-024",
"lang": "en",
"tags": ["hard", "paraphrase"],
"query": "what fixed the screen problem",
"want": "n1",
"notes": [
{"id": "n1", "text": "the flicker went away once i swapped the display cable", "kind": "note"},
{"id": "n2", "text": "the laptop fan is loud", "kind": "note"},
{"id": "n3", "text": "the second monitor is 1440p", "kind": "note"}
]
},
{
"id": "en-pref-025",
"lang": "en",
"tags": ["preference", "hard"],
"query": "should i be offered wine",
"want": "n1",
"notes": [
{"id": "n1", "text": "i do not drink alcohol", "kind": "note"},
{"id": "n2", "text": "i skip breakfast", "kind": "note"},
{"id": "n3", "text": "i like spicy food", "kind": "note"}
]
},
{
"id": "ru-home-026",
"lang": "ru",
"tags": ["homelab", "paraphrase"],
"query": "какая модель распознавания речи мне подходит",
"want": "n1",
"notes": [
{"id": "n1", "text": "whisper модель small хватает для русского", "kind": "note"},
{"id": "n2", "text": "голос ирина звучит лучше остальных", "kind": "note"},
{"id": "n3", "text": "слово активации маven", "kind": "note"}
]
},
{
"id": "ru-fact-027",
"lang": "ru",
"tags": ["distractor", "hard", "paraphrase"],
"query": "когда я последний раз обслуживал машину",
"want": "n1",
"note": "A fact, not a note — both share the vector index, so a fact can win a recall.",
"notes": [
{"id": "n1", "text": "последний раз менял масло в мае", "kind": "fact"},
{"id": "n2", "text": "шины поменял осенью", "kind": "fact"},
{"id": "n3", "text": "страховка до декабря", "kind": "fact"}
]
},
{
"id": "ru-pref-028",
"lang": "ru",
"tags": ["preference", "hard", "paraphrase"],
"query": "как мне присылать оповещения",
"want": "n1",
"notes": [
{"id": "n1", "text": "терпеть не могу уведомления со звуком", "kind": "note"},
{"id": "n2", "text": "вибрацию оставь включённой", "kind": "note"},
{"id": "n3", "text": "письма читаю вечером", "kind": "note"}
]
},
{
"id": "ru-silent-029",
"lang": "ru",
"tags": ["silent"],
"query": "во сколько отходит поезд",
"want": "",
"notes": [
{"id": "n1", "text": "кот ест только сухой корм", "kind": "note"},
{"id": "n2", "text": "люблю острую еду", "kind": "note"},
{"id": "n3", "text": "пароли держу в keepassxc", "kind": "note"}
]
},
{
"id": "en-home-030",
"lang": "en",
"tags": ["homelab", "paraphrase"],
"query": "what firmware is on the router",
"want": "n1",
"notes": [
{"id": "n1", "text": "the router runs openwrt", "kind": "note"},
{"id": "n2", "text": "wifi channel is 6", "kind": "note"},
{"id": "n3", "text": "the guest network is off", "kind": "note"}
]
}
]
}