Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0b3b8d0a9e | |||
| fe0e654ab1 | |||
| a54ebac0cb | |||
| 39d83a33e8 | |||
| 925ce223a0 |
@@ -16,7 +16,7 @@ PIPER_BIN := $(shell pwd)/deps/piper/piper
|
||||
PIPER_MODEL := $(shell pwd)/models/tts/ru_RU-irina-medium.onnx
|
||||
PIPER_ESPEAK := $(shell pwd)/deps/piper/espeak-ng-data
|
||||
|
||||
.PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test run-stt run-tts run-web download-embedder deps-go eval-router eval-recall
|
||||
.PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test run-stt run-tts run-web download-embedder deps-go eval-router
|
||||
|
||||
all: build
|
||||
|
||||
@@ -83,13 +83,6 @@ MAVEN_ONNX_LIB ?= $(shell pwd)/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxrunt
|
||||
eval-router:
|
||||
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/router/eval/
|
||||
|
||||
# eval-recall — score the held-out note-recall fixture (internal/memory/recalleval).
|
||||
# Answers "can she find the note again when it matters": recall@1, recall@3,
|
||||
# false recall and the query_min_score sweep. Same MAVEN_ONNX_LIB deal as
|
||||
# eval-router; without it only the deterministic hash ratchet runs.
|
||||
eval-recall:
|
||||
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
|
||||
|
||||
run-stt: build-stt
|
||||
LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
|
||||
./mavsttd -socket /tmp/maven/stt.sock -model $(WHISPER_MODEL)
|
||||
|
||||
@@ -1,99 +0,0 @@
|
||||
# Note recall evaluation — 31-07-2026
|
||||
|
||||
The operator's goal is that Maven "memorize/note things … and know more about me/world". This
|
||||
measures whether the note/recall path delivers that.
|
||||
|
||||
- Fixture + scorer: `internal/memory/recalleval/` (`ru_recall_v1.json`, 30 cases)
|
||||
- Reproduce: `make eval-recall` — hash ratchet always, ONNX when `deps/` is present
|
||||
- Commit: `43470ab` (harness)
|
||||
|
||||
Each case inserts its own 3 notes **plus 12 shared filler notes** into a fresh store, embeds the
|
||||
query, takes the top 3 — the read path `cmd/mavend/voice.go` runs for `IntentQuery`. Filler is
|
||||
load-bearing: with 3 notes and a top-3 search, recall@3 is 100% by construction. 25 answerable
|
||||
cases (paraphrased queries, homelab and preference content, 9 with a plausible second note) and 5
|
||||
that must recall **nothing**. `TestFixtureIsParaphrased` fails the build if a query shares over half
|
||||
its words with its note; equal-score ties count as ties, not recall.
|
||||
|
||||
## Results
|
||||
|
||||
| | recall+hash (CI ratchet) | recall+onnx (deployed) |
|
||||
|---|---|---|
|
||||
| **recall@1** | 36.0% (9/25) | **60.0% (15/25)** |
|
||||
| recall@3 | 76.0% (19/25) | 80.0% (20/25) |
|
||||
| **answered after the 0.55 gate** | **0.0% (0/25)** | **48.0% (12/25)** |
|
||||
| wrong note on top / tie on top | 9 / 7 | 10 / 0 |
|
||||
| ranked first, then silenced by the gate | 9 | 3 |
|
||||
| **false recall** | 0/5 | **1/5 (20%)** |
|
||||
| top-1 score when right, min / median | n/a | 0.559 / 0.678 |
|
||||
| top-1 when it must stay silent, median / max | 0.000 / 0.144 | 0.470 / **0.567** |
|
||||
| RU / EN / `hard` cases passed | 4/24 / 1/6 / 0/11 | 13/24 / 3/6 / 2/11 |
|
||||
| latency p50 / p95 / max | 49µs / 70µs | 59ms / 148ms / 194ms |
|
||||
|
||||
Never compare a hash-embedder number to an ONNX one — the hash floor is lexical and exists only so
|
||||
CI has a deterministic ratchet with no model files.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Real recall is 48%, not 60%
|
||||
|
||||
The right note ranks first 60% of the time, but the daemon only *says* it 48% of the time — three
|
||||
more cases rank first and are then silenced by `voice.go:776`'s `queryMinScore`. **Roughly one
|
||||
useful question in two gets "не знаю".** This is not a working memory yet.
|
||||
|
||||
### 2. The gate cannot separate a real recall from a false one — the distributions overlap
|
||||
|
||||
Right-note top-1 scores start at **0.559**. Must-stay-silent top-1 scores reach **0.567**. No
|
||||
threshold keeps every real recall and rejects every false one. From the sweep: gate 0.50 → 13/25
|
||||
answered, 1/5 false; **0.55 (default) → 12/25, 1/5**; **0.60 → 10/25, 0/5**; 0.70 → 5/25, 0/5. What
|
||||
the data says about `DefaultQueryMinScore` (`internal/config/config.go:392`): **0.55 is
|
||||
slightly too loose** — it admits one confident wrong answer ("как зовут сестру моего коллеги"
|
||||
recalls "выучил пару аккордов на гитаре" at 0.567), which the spec ranks as worse than a gap. 0.60
|
||||
silences all five and costs 8 points of real recall. Left alone as instructed; the overlap means
|
||||
the threshold is the wrong dial anyway (finding 3).
|
||||
|
||||
### 3. Filler notes outrank the right answer — the model scores similarity, not relevance
|
||||
|
||||
`models/embedder/` is **paraphrase-multilingual-MiniLM-L12-v2** (`Makefile:119`), a *symmetric*
|
||||
paraphrase model. It scores "do these sentences look alike", not "does this passage answer this
|
||||
question", so question-shaped queries drift to whatever note is stylistically closest. "из-за чего
|
||||
кончилось место" and "откуда берётся токен бота" both return `выучил пару аккордов на гитаре`
|
||||
(0.730, 0.729); "как я восстановил конфиги" returns a bootloader note at 0.703 with the right note
|
||||
not even in the top 3. An unrelated guitar note beating a homelab note at 0.73 is not a tuning
|
||||
problem — an asymmetric retrieval model (`multilingual-e5-small`, with `query:` / `passage:`
|
||||
prefixes) is the targeted fix, and it would move findings 1 and 2 together. Separately:
|
||||
`deploy/mavend.json:39` loads a 470MB fp32 `model.onnx` while `make download-embedder` fetches
|
||||
`model_quantized.onnx` — not the same file.
|
||||
|
||||
`hard` cases score **2/11**: every one is a query where the operator did not reuse his own words.
|
||||
That is the normal case weeks later, and exactly what DESIGN.md's "recall when relevant" promises.
|
||||
|
||||
### 4. The memory-store recall branch is dead for notes
|
||||
|
||||
`voice.go:776` only reaches `h.memStore.Search` when the notes-RAG top score is already below
|
||||
`queryMinScore`, and `bestRecall` (`cmd/mavend/recall.go:19`) then applies the **same** gate to the
|
||||
same vector. A note is indexed in both places with the same embedding, so if it failed the gate in
|
||||
`QueryNotes` it fails again here — the branch can only ever return a **fact**. Its comment calls it
|
||||
"additive"; for notes it is not.
|
||||
|
||||
### 5. Ranking has no recency or type signal, and the store is not the bottleneck
|
||||
|
||||
`internal/store/notes.go:67` sorts by cosine and uses `ts` only to break an exact float tie, which
|
||||
never happens; `kind` never enters the ranking. Meanwhile `TestPersistentStoreScoresTheSame` scores
|
||||
sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — both full-scan cosine
|
||||
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
|
||||
not the problem to solve.
|
||||
|
||||
## Next steps — ordered by value-to-risk; nothing here is a decision
|
||||
|
||||
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
|
||||
change plus a prefix in `onnxembedder.go`, re-measurable in one command.
|
||||
2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
|
||||
`query_min_score` picked against today's embedder describes a model on its way out.
|
||||
3. **Replace the absolute-score gate with a margin gate** (`top1 − top2 > δ`) — as the routing eval
|
||||
concluded, absolute cosine cannot see a flat distribution.
|
||||
4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
|
||||
gate it separately, or restrict it to facts and say so.
|
||||
5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
|
||||
6. **Grow the fixture from real misses.** 30 cases can rank two embedders, not trust 4 points.
|
||||
7. **Re-measure end to end.** Recall is gated twice — the utterance must first route to `query`,
|
||||
which the routing eval puts at ~50%. The product is ~24%, and that is what he experiences.
|
||||
@@ -0,0 +1,180 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/dialogue"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
// clarifyTTL — how long a parked question stays answerable. Same 90s as the
|
||||
// confirm gate, for the same reason: an answer is a same-breath gesture, and a
|
||||
// stale question must not eat an unrelated later utterance.
|
||||
const clarifyTTL = 90 * time.Second
|
||||
|
||||
// wantedSlots — what each intent needs before she can act on it. First entry is
|
||||
// the one she asks about; the rest are only used to decide act-vs-drop.
|
||||
//
|
||||
// Intents not listed here are never worth a question: note and query act on the
|
||||
// raw utterance, chat and system have nothing to fill in. For those a clarify
|
||||
// decision keeps the canned "не поняла" reply — inventing a question for noise
|
||||
// is worse than admitting she missed it.
|
||||
var wantedSlots = map[router.Intent][]dialogue.Slot{
|
||||
router.IntentReminder: {dialogue.SlotTime},
|
||||
router.IntentFact: {dialogue.SlotKey},
|
||||
router.IntentAct: {dialogue.SlotFn},
|
||||
}
|
||||
|
||||
// clarifyQuestions — one short question per missing slot.
|
||||
//
|
||||
// These are fixed templates, not model output. The resident model is a 0.8B; it
|
||||
// would wander, and a question whose wording changes every time is harder to
|
||||
// answer than a blunt one that always reads the same. They are infinitive
|
||||
// questions, so there is no gender agreement to get wrong; the feminine
|
||||
// self-reference lives in the reply she gives when she drops the request.
|
||||
var clarifyQuestions = map[dialogue.Slot]string{
|
||||
dialogue.SlotTime: "На когда напомнить?",
|
||||
dialogue.SlotKey: "Что записать?",
|
||||
dialogue.SlotFn: "Что сделать?",
|
||||
}
|
||||
|
||||
// clarifyDropped — she asked once, the answer still did not fill the gap, so
|
||||
// the request is gone. Said plainly, once, with no second question.
|
||||
const clarifyDropped = "Не разобрала — скажи целиком, пожалуйста."
|
||||
|
||||
// missingFor returns the slots a decision still needs, most important first.
|
||||
// Empty ⇒ there is nothing identifiable to ask about.
|
||||
func missingFor(dec router.Decision) []dialogue.Slot {
|
||||
return dialogue.StillMissing(wantedSlots[dec.Intent], toDialogueSlots(dec.Slots))
|
||||
}
|
||||
|
||||
// clarifyQuestion picks the one question to ask for a clarify decision. Returns
|
||||
// ("", false) when she has no idea what is missing.
|
||||
//
|
||||
// One question about one thing: if two slots are missing she asks about the
|
||||
// first and lets the rest go. Two questions in a row is an interrogation.
|
||||
func clarifyQuestion(dec router.Decision) (dialogue.Slot, string, bool) {
|
||||
missing := missingFor(dec)
|
||||
if len(missing) == 0 {
|
||||
return "", "", false
|
||||
}
|
||||
q, ok := clarifyQuestions[missing[0]]
|
||||
if !ok {
|
||||
return "", "", false
|
||||
}
|
||||
return missing[0], q, true
|
||||
}
|
||||
|
||||
// askClarify parks the request and returns the question to ask instead of the
|
||||
// canned "не поняла". Returns ("", false) when there is nothing to ask about, so
|
||||
// the caller falls back to the canned reply.
|
||||
func (h *reactiveHandler) askClarify(dec router.Decision) (string, bool) {
|
||||
if h.clarifyStore == nil {
|
||||
return "", false
|
||||
}
|
||||
slot, question, ok := clarifyQuestion(dec)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
h.clarifyStore.Put(voiceDialogueID, &dialogue.PendingQuestion{
|
||||
Intent: dialogue.Intent(dec.Intent),
|
||||
Slots: toDialogueSlots(dec.Slots),
|
||||
Missing: []dialogue.Slot{slot},
|
||||
Utterance: dec.Utterance,
|
||||
Asked: h.now(),
|
||||
TTL: clarifyTTL,
|
||||
Attempts: 1, // asked once; MaxAttempts is 1, so there is no second ask
|
||||
})
|
||||
log.Printf("voice: clarify — asked about %s for intent=%s", slot, dec.Intent)
|
||||
return question, true
|
||||
}
|
||||
|
||||
// resolveClarifyAnswer reads an utterance as the answer to a parked question.
|
||||
// Returns ("", false) when no live question is parked (or it expired), so the
|
||||
// caller routes the utterance normally as a fresh request. Sibling of
|
||||
// resolveConfirm and checked in the same place.
|
||||
//
|
||||
// The answer is parsed with the same extractor the router uses, for the intent
|
||||
// she parked — no second parser. If it still does not fill the gap the request
|
||||
// is dropped: she does not ask again.
|
||||
func (h *reactiveHandler) resolveClarifyAnswer(ctx context.Context, text string) (string, bool) {
|
||||
if h.clarifyStore == nil {
|
||||
return "", false
|
||||
}
|
||||
q := h.clarifyStore.Get(voiceDialogueID, h.now())
|
||||
if q == nil {
|
||||
return "", false
|
||||
}
|
||||
// One shot either way: the question is consumed whether or not the answer
|
||||
// works, so a failed answer can't leave the question armed.
|
||||
h.clarifyStore.Delete(voiceDialogueID)
|
||||
|
||||
intent := router.Intent(q.Intent)
|
||||
answer := h.extractor.Extract(ctx, intent, text, h.now())
|
||||
merged := q.Answer(text, toDialogueSlots(answer))
|
||||
if len(dialogue.StillMissing(q.Missing, merged)) > 0 {
|
||||
log.Printf("voice: clarify — answer %q did not fill %v, dropping", text, q.Missing)
|
||||
return clarifyDropped, true
|
||||
}
|
||||
|
||||
// Rebuild the decision as if it had routed cleanly, then run it down the
|
||||
// normal path. Clarify is deliberately false and the intent is unchanged:
|
||||
// filling in an argument never grants authority, so the completed decision
|
||||
// still meets the allowlist and the destructive-act confirm gate in
|
||||
// applyAction exactly like any other decision.
|
||||
dec := router.Decision{
|
||||
Utterance: q.Utterance,
|
||||
Stage: 2,
|
||||
Intent: intent,
|
||||
Slots: applyDialogueSlots(answer, merged),
|
||||
}
|
||||
return h.finishClarified(ctx, dec), true
|
||||
}
|
||||
|
||||
// finishClarified runs a completed decision through the same steps a freshly
|
||||
// routed one takes: remember the turn, act, then phrase.
|
||||
func (h *reactiveHandler) finishClarified(ctx context.Context, dec router.Decision) string {
|
||||
if h.dialogueSessions != nil {
|
||||
now := h.now()
|
||||
prev := h.dialogueSessions.Get(voiceDialogueID, now)
|
||||
dec = followUpMerge(prev, dec, now)
|
||||
h.rememberTurn(prev, dec, now)
|
||||
}
|
||||
reply := h.applyAction(ctx, dec)
|
||||
if reply == "" {
|
||||
reply = h.replier.Reply(dec)
|
||||
}
|
||||
return reply
|
||||
}
|
||||
|
||||
// rememberTurn stores this turn as the dialogue session the next follow-up
|
||||
// inherits from, carrying up to 4 prior turns of history for anaphora. Capped so
|
||||
// one long conversation can't grow the session unboundedly.
|
||||
func (h *reactiveHandler) rememberTurn(prev *dialogue.Session, dec router.Decision, now time.Time) {
|
||||
var history []dialogue.Turn
|
||||
if prev != nil {
|
||||
history = append(history, dialogue.Turn{
|
||||
Intent: prev.Intent,
|
||||
Slots: prev.Slots,
|
||||
Text: prev.Slots.Text,
|
||||
})
|
||||
maxHist := len(prev.History)
|
||||
if maxHist > 3 {
|
||||
maxHist = 3
|
||||
}
|
||||
history = append(history, prev.History[:maxHist]...)
|
||||
}
|
||||
ttl := time.Duration(0) // use the store default (2 min)
|
||||
if dec.Intent == router.IntentChat {
|
||||
ttl = 15 * time.Minute // conversational turns should last longer
|
||||
}
|
||||
h.dialogueSessions.Put(voiceDialogueID, &dialogue.Session{
|
||||
Intent: dialogue.Intent(dec.Intent),
|
||||
Slots: toDialogueSlots(dec.Slots),
|
||||
Timestamp: now,
|
||||
TTL: ttl,
|
||||
History: history,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/dialogue"
|
||||
"github.com/kami/maven/internal/ipc"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
"github.com/kami/maven/internal/tool"
|
||||
"github.com/kami/maven/internal/voice"
|
||||
)
|
||||
|
||||
// newClarifyHandler builds a handler with the clarify path wired and no model:
|
||||
// stub date parser, the real fact parser, and a matcher over whatever tools the
|
||||
// test enabled. `now` is fixed so TTL behaviour is testable.
|
||||
func newClarifyHandler(t *testing.T) (*reactiveHandler, *store.Store, *time.Time) {
|
||||
t.Helper()
|
||||
st := newTestStore(t)
|
||||
api := ipc.NewStoreAPI(st)
|
||||
now := time.Date(2026, 7, 31, 9, 0, 0, 0, time.UTC)
|
||||
matcher := tool.NewMatcher(api)
|
||||
h := &reactiveHandler{
|
||||
api: api,
|
||||
dataStore: st,
|
||||
tools: tool.NewExecutor(api, 2*time.Second),
|
||||
matcher: matcher,
|
||||
replier: voice.NewStubReplier(),
|
||||
now: func() time.Time { return now },
|
||||
dialogueSessions: dialogue.NewSessionStore(2 * time.Minute),
|
||||
clarifyStore: dialogue.NewClarifyStore(clarifyTTL),
|
||||
extractor: router.Extractor{
|
||||
Time: router.StubDateTimeParser{},
|
||||
Acts: matcher,
|
||||
Facts: router.DefaultFactParser{},
|
||||
},
|
||||
}
|
||||
return h, st, &now
|
||||
}
|
||||
|
||||
func clarifyDec(intent router.Intent, slots router.Slots, utterance string) router.Decision {
|
||||
return router.Decision{Utterance: utterance, Stage: 3, Intent: intent, Slots: slots, Clarify: true}
|
||||
}
|
||||
|
||||
// TestClarifyQuestionForMissingSlot pins which question goes with which gap, and
|
||||
// which intents get no question at all.
|
||||
func TestClarifyQuestionForMissingSlot(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
dec router.Decision
|
||||
want string
|
||||
asked bool
|
||||
}{
|
||||
{"reminder without a time", clarifyDec(router.IntentReminder, router.Slots{Text: "напомни позвонить маме"}, "напомни позвонить маме"), "На когда напомнить?", true},
|
||||
{"fact without a key", clarifyDec(router.IntentFact, router.Slots{Text: "запиши"}, "запиши"), "Что записать?", true},
|
||||
{"act without a fn", clarifyDec(router.IntentAct, router.Slots{Text: "сделай это"}, "сделай это"), "Что сделать?", true},
|
||||
{"reminder that already has a time", clarifyDec(router.IntentReminder, router.Slots{HasTime: true}, "напомни в 11"), "", false},
|
||||
{"chat is never worth a question", clarifyDec(router.IntentChat, router.Slots{Text: "мгм"}, "мгм"), "", false},
|
||||
{"query is never worth a question", clarifyDec(router.IntentQuery, router.Slots{Text: "а"}, "а"), "", false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
_, got, asked := clarifyQuestion(tc.dec)
|
||||
if asked != tc.asked || got != tc.want {
|
||||
t.Errorf("%s: got (%q, %v), want (%q, %v)", tc.name, got, asked, tc.want, tc.asked)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestClarifyReminderCompletesOnAnswer is the whole point of the feature: she
|
||||
// asks for the missing time and the answer creates the reminder.
|
||||
func TestClarifyReminderCompletesOnAnswer(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, st, _ := newClarifyHandler(t)
|
||||
|
||||
question, asked := h.askClarify(clarifyDec(router.IntentReminder, router.Slots{Text: "напомни позвонить маме"}, "напомни позвонить маме"))
|
||||
if !asked || question != "На когда напомнить?" {
|
||||
t.Fatalf("expected the time question, got %q asked=%v", question, asked)
|
||||
}
|
||||
|
||||
reply, handled := h.resolveClarifyAnswer(ctx, "в 11:00")
|
||||
if !handled {
|
||||
t.Fatal("the answer to an open question must be consumed as an answer")
|
||||
}
|
||||
if reply == clarifyDropped {
|
||||
t.Fatalf("a good answer must not drop the request: %q", reply)
|
||||
}
|
||||
|
||||
reminders, err := st.DueReminders(ctx, h.now().Add(48*time.Hour))
|
||||
if err != nil || len(reminders) != 1 {
|
||||
t.Fatalf("clarified reminder was not created: reminders=%v err=%v", reminders, err)
|
||||
}
|
||||
if !strings.Contains(reminders[0].Payload, "маме") {
|
||||
t.Fatalf("the reminder lost the original request: %q", reminders[0].Payload)
|
||||
}
|
||||
if h.clarifyStore.Get(voiceDialogueID, h.now()) != nil {
|
||||
t.Fatal("the question must be cleared once answered")
|
||||
}
|
||||
}
|
||||
|
||||
// TestClarifyFactCompletesOnAnswer — the fact path, where the answer carries
|
||||
// both the key and the value.
|
||||
func TestClarifyFactCompletesOnAnswer(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, st, _ := newClarifyHandler(t)
|
||||
|
||||
if _, asked := h.askClarify(clarifyDec(router.IntentFact, router.Slots{Text: "запиши"}, "запиши")); !asked {
|
||||
t.Fatal("a fact with no key should be asked about")
|
||||
}
|
||||
if reply, handled := h.resolveClarifyAnswer(ctx, "пил воду"); !handled || reply == clarifyDropped {
|
||||
t.Fatalf("answer should complete the fact, handled=%v reply=%q", handled, reply)
|
||||
}
|
||||
if fact, err := st.LatestFact(ctx, "water"); err != nil || fact.Key != "water" {
|
||||
t.Fatalf("clarified fact was not written: fact=%+v err=%v", fact, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestClarifyAnswerAfterTTLIsANewRequest — a late answer is not an answer.
|
||||
func TestClarifyAnswerAfterTTLIsANewRequest(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, st, now := newClarifyHandler(t)
|
||||
|
||||
if _, asked := h.askClarify(clarifyDec(router.IntentReminder, router.Slots{Text: "напомни"}, "напомни")); !asked {
|
||||
t.Fatal("expected a question")
|
||||
}
|
||||
*now = now.Add(clarifyTTL + time.Second)
|
||||
|
||||
if reply, handled := h.resolveClarifyAnswer(ctx, "в 11:00"); handled {
|
||||
t.Fatalf("an answer past the TTL must fall through to normal routing, got %q", reply)
|
||||
}
|
||||
if reminders, err := st.DueReminders(ctx, now.Add(48*time.Hour)); err != nil || len(reminders) != 0 {
|
||||
t.Fatalf("expired question must not create anything: reminders=%v err=%v", reminders, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestClarifyUnclearAnswerDropsWithoutAskingAgain — MaxAttempts is 1.
|
||||
func TestClarifyUnclearAnswerDropsWithoutAskingAgain(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, st, _ := newClarifyHandler(t)
|
||||
|
||||
if _, asked := h.askClarify(clarifyDec(router.IntentReminder, router.Slots{Text: "напомни"}, "напомни")); !asked {
|
||||
t.Fatal("expected a question")
|
||||
}
|
||||
reply, handled := h.resolveClarifyAnswer(ctx, "ну не знаю")
|
||||
if !handled || reply != clarifyDropped {
|
||||
t.Fatalf("an unclear answer should drop the request, handled=%v reply=%q", handled, reply)
|
||||
}
|
||||
if strings.Contains(reply, "?") {
|
||||
t.Fatalf("she must not ask a second question: %q", reply)
|
||||
}
|
||||
if h.clarifyStore.Get(voiceDialogueID, h.now()) != nil {
|
||||
t.Fatal("a dropped request must leave no armed question")
|
||||
}
|
||||
if reminders, err := st.DueReminders(ctx, h.now().Add(48*time.Hour)); err != nil || len(reminders) != 0 {
|
||||
t.Fatalf("a dropped request must not create anything: reminders=%v err=%v", reminders, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestClarifiedActOffAllowlistIsStillRefused — clarification fills in an
|
||||
// argument, it never grants authority.
|
||||
func TestClarifiedActOffAllowlistIsStillRefused(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, st, _ := newClarifyHandler(t)
|
||||
marker := filepath.Join(t.TempDir(), "not-allowed-ran")
|
||||
|
||||
if _, asked := h.askClarify(clarifyDec(router.IntentAct, router.Slots{Text: "сделай это"}, "сделай это")); !asked {
|
||||
t.Fatal("an act with no fn should be asked about")
|
||||
}
|
||||
reply, handled := h.resolveClarifyAnswer(ctx, "rm "+marker)
|
||||
if !handled {
|
||||
t.Fatal("the answer should be consumed")
|
||||
}
|
||||
if strings.Contains(reply, "готово") {
|
||||
t.Fatalf("an act that is not on the allowlist must not report success: %q", reply)
|
||||
}
|
||||
if _, err := os.Stat(marker); !os.IsNotExist(err) {
|
||||
t.Fatalf("a clarified act off the allowlist ran anyway: %v", err)
|
||||
}
|
||||
if tools, err := st.ListTools(ctx, "enabled"); err != nil || len(tools) != 0 {
|
||||
t.Fatalf("clarify must not enable a tool: tools=%+v err=%v", tools, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestClarifiedDestructiveActStillNeedsConfirm — the confirm gate survives the
|
||||
// clarify path.
|
||||
func TestClarifiedDestructiveActStillNeedsConfirm(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
h, st, _ := newClarifyHandler(t)
|
||||
marker := filepath.Join(t.TempDir(), "destructive-ran")
|
||||
if err := st.EnableTool(ctx, "delete_backups", []string{"touch", marker}, true, "test", h.now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if _, asked := h.askClarify(clarifyDec(router.IntentAct, router.Slots{Text: "сделай это"}, "сделай это")); !asked {
|
||||
t.Fatal("expected a question")
|
||||
}
|
||||
reply, handled := h.resolveClarifyAnswer(ctx, "delete_backups")
|
||||
if !handled {
|
||||
t.Fatal("the answer should be consumed")
|
||||
}
|
||||
if !strings.Contains(reply, "да") || h.pending == nil {
|
||||
t.Fatalf("a clarified destructive act must still park a confirm: reply=%q pending=%+v", reply, h.pending)
|
||||
}
|
||||
if _, err := os.Stat(marker); !os.IsNotExist(err) {
|
||||
t.Fatalf("a clarified destructive act ran before confirmation: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNoQuestionWhenNothingIsMissing — noise keeps the canned reply, so she
|
||||
// never invents a question for nothing.
|
||||
func TestNoQuestionWhenNothingIsMissing(t *testing.T) {
|
||||
h, _, _ := newClarifyHandler(t)
|
||||
for _, dec := range []router.Decision{
|
||||
clarifyDec(router.IntentChat, router.Slots{Text: "эм"}, "эм"),
|
||||
clarifyDec(router.IntentQuery, router.Slots{Text: "ммм"}, "ммм"),
|
||||
clarifyDec(router.IntentNote, router.Slots{Text: "..."}, "..."),
|
||||
} {
|
||||
if question, asked := h.askClarify(dec); asked {
|
||||
t.Fatalf("intent %s should keep the canned reply, got %q", dec.Intent, question)
|
||||
}
|
||||
}
|
||||
if h.clarifyStore.Get(voiceDialogueID, h.now()) != nil {
|
||||
t.Fatal("noise must not park a question")
|
||||
}
|
||||
}
|
||||
|
||||
// TestNoPendingQuestionFallsThrough — with nothing parked, an utterance routes
|
||||
// normally.
|
||||
func TestNoPendingQuestionFallsThrough(t *testing.T) {
|
||||
h, _, _ := newClarifyHandler(t)
|
||||
if reply, handled := h.resolveClarifyAnswer(context.Background(), "напомни в 11:00"); handled {
|
||||
t.Fatalf("no open question ⇒ must not be treated as an answer, got %q", reply)
|
||||
}
|
||||
}
|
||||
+43
-53
@@ -226,6 +226,8 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
|
||||
|
||||
// ----- dialogue (multi-turn slot carry-over; 2-min follow-up window) -----
|
||||
dialogueSessions := dialogue.NewSessionStore(2 * time.Minute)
|
||||
clarifyStore := dialogue.NewClarifyStore(clarifyTTL)
|
||||
timeParser := router.NewPythonDateParser()
|
||||
|
||||
// ----- replier (LLM-backed when the engine is on, Stub floor otherwise) -----
|
||||
replier := voice.Replier(voice.NewStubReplier())
|
||||
@@ -250,8 +252,10 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
|
||||
memStore: memStore,
|
||||
dataStore: dataStore,
|
||||
dialogueSessions: dialogueSessions,
|
||||
clarifyStore: clarifyStore,
|
||||
extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}},
|
||||
queryMinScore: cfg.Voice.QueryMinScore,
|
||||
timeParser: router.NewPythonDateParser(),
|
||||
timeParser: timeParser,
|
||||
ecosystem: eco,
|
||||
}
|
||||
|
||||
@@ -304,6 +308,14 @@ type reactiveHandler struct {
|
||||
// box → one session slot, keyed voiceDialogueID). nil ⇒ no carry-over.
|
||||
dialogueSessions *dialogue.SessionStore
|
||||
|
||||
// clarifyStore parks the request behind an open question she asked (see
|
||||
// clarify.go). nil ⇒ she falls back to the canned "не поняла" reply.
|
||||
clarifyStore *dialogue.ClarifyStore
|
||||
|
||||
// extractor parses the answer to an open question, with the same parsers
|
||||
// the router's own stage-2 uses.
|
||||
extractor router.Extractor
|
||||
|
||||
// pending destructive-act confirmation. A destructive act replies with a
|
||||
// "выполнить X? да/нет" prompt and parks here; the NEXT utterance is read as
|
||||
// the y/n answer. ponytail: single slot, single-user box — a second act
|
||||
@@ -378,6 +390,13 @@ func (h *reactiveHandler) HandlePushToTalk(ctx context.Context, req voice.PushTo
|
||||
return h.reply(ctx, reply, nil)
|
||||
}
|
||||
|
||||
// 1b2. clarify answer — if she asked a question last turn, this utterance is
|
||||
// its answer, not a fresh command. After the confirm check: a y/n gate is
|
||||
// armed by her own prompt and is the narrower claim on the utterance.
|
||||
if reply, handled := h.resolveClarifyAnswer(ctx, text); handled {
|
||||
return h.reply(ctx, reply, nil)
|
||||
}
|
||||
|
||||
// 1c. quiet-hours toggle — keyword match, not classifier-dependent.
|
||||
// "тихий режим" / "quiet on" would route through the classifier
|
||||
// unreliably (it's a command, not a free-form query), so we match it
|
||||
@@ -407,34 +426,16 @@ func (h *reactiveHandler) HandlePushToTalk(ctx context.Context, req voice.PushTo
|
||||
prev := h.dialogueSessions.Get(voiceDialogueID, now)
|
||||
dec = followUpMerge(prev, dec, now)
|
||||
if !dec.Clarify {
|
||||
// Build history: carry over up to 4 prior turns for cross-intent
|
||||
// reference. The most recent prior turn is prepended to history.
|
||||
var history []dialogue.Turn
|
||||
if prev != nil {
|
||||
history = append(history, dialogue.Turn{
|
||||
Intent: prev.Intent,
|
||||
Slots: prev.Slots,
|
||||
Text: prev.Slots.Text, // the prior turn's utterance
|
||||
})
|
||||
// Cap history depth so one long conversation can't grow
|
||||
// the session unboundedly.
|
||||
maxHist := len(prev.History)
|
||||
if maxHist > 3 {
|
||||
maxHist = 3
|
||||
}
|
||||
history = append(history, prev.History[:maxHist]...)
|
||||
}
|
||||
ttl := time.Duration(0) // use default (2 min)
|
||||
if dec.Intent == router.IntentChat {
|
||||
ttl = 15 * time.Minute // conversational turns should last longer
|
||||
}
|
||||
h.dialogueSessions.Put(voiceDialogueID, &dialogue.Session{
|
||||
Intent: dialogue.Intent(dec.Intent),
|
||||
Slots: toDialogueSlots(dec.Slots),
|
||||
Timestamp: now,
|
||||
TTL: ttl,
|
||||
History: history,
|
||||
})
|
||||
h.rememberTurn(prev, dec, now)
|
||||
}
|
||||
}
|
||||
|
||||
// 2c. clarify — she is not sure. If one named thing is missing, ask about it
|
||||
// and park the request (clarify.go); otherwise the replier's canned reply
|
||||
// stands.
|
||||
if dec.Clarify {
|
||||
if question, asked := h.askClarify(dec); asked {
|
||||
return h.reply(ctx, question, nil)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -465,6 +466,11 @@ func (h *reactiveHandler) handleText(ctx context.Context, text string) string {
|
||||
return reply
|
||||
}
|
||||
|
||||
// 1b2. clarify answer — same check as HandlePushToTalk.
|
||||
if reply, handled := h.resolveClarifyAnswer(ctx, text); handled {
|
||||
return reply
|
||||
}
|
||||
|
||||
// 2. router — classify the utterance.
|
||||
dec, err := h.router.Route(ctx, text, h.now())
|
||||
if err != nil {
|
||||
@@ -482,30 +488,14 @@ func (h *reactiveHandler) handleText(ctx context.Context, text string) string {
|
||||
prev := h.dialogueSessions.Get(voiceDialogueID, now)
|
||||
dec = followUpMerge(prev, dec, now)
|
||||
if !dec.Clarify {
|
||||
var history []dialogue.Turn
|
||||
if prev != nil {
|
||||
history = append(history, dialogue.Turn{
|
||||
Intent: prev.Intent,
|
||||
Slots: prev.Slots,
|
||||
Text: prev.Slots.Text,
|
||||
})
|
||||
maxHist := len(prev.History)
|
||||
if maxHist > 3 {
|
||||
maxHist = 3
|
||||
}
|
||||
history = append(history, prev.History[:maxHist]...)
|
||||
}
|
||||
ttl := time.Duration(0)
|
||||
if dec.Intent == router.IntentChat {
|
||||
ttl = 15 * time.Minute
|
||||
}
|
||||
h.dialogueSessions.Put(voiceDialogueID, &dialogue.Session{
|
||||
Intent: dialogue.Intent(dec.Intent),
|
||||
Slots: toDialogueSlots(dec.Slots),
|
||||
Timestamp: now,
|
||||
TTL: ttl,
|
||||
History: history,
|
||||
})
|
||||
h.rememberTurn(prev, dec, now)
|
||||
}
|
||||
}
|
||||
|
||||
// 2c. clarify — same as HandlePushToTalk: ask about the one missing thing.
|
||||
if dec.Clarify {
|
||||
if question, asked := h.askClarify(dec); asked {
|
||||
return question
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
package dialogue
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Slot names one field of Slots. Named type, not a free string, so a missing
|
||||
// slot cannot be misspelled — the question phrasing switches on these.
|
||||
type Slot string
|
||||
|
||||
const (
|
||||
SlotTime Slot = "time" // Slots.Time / HasTime
|
||||
SlotKey Slot = "key" // Slots.Key / HasKey
|
||||
SlotValue Slot = "value" // Slots.Value (paired with Key)
|
||||
SlotFn Slot = "fn" // Slots.Fn / HasFn
|
||||
SlotText Slot = "text" // Slots.Text
|
||||
)
|
||||
|
||||
// MaxAttempts is 1 because Maven is not a nag (DESIGN.md § Non-goals). She asks
|
||||
// one clarifying question. If the answer still leaves the slot empty she drops
|
||||
// the request instead of asking again.
|
||||
const MaxAttempts = 1
|
||||
|
||||
// PendingQuestion is what Maven holds while she waits for an answer to an open
|
||||
// question. Unlike the yes/no confirms in cmd/mavend/voice.go, the answer here
|
||||
// is free text that fills a missing slot rather than a verdict.
|
||||
type PendingQuestion struct {
|
||||
Intent Intent // what the router already guessed
|
||||
Slots Slots // what it already filled
|
||||
Missing []Slot // what is still empty, in the order to ask about
|
||||
Utterance string // the user's original raw words
|
||||
Asked time.Time
|
||||
TTL time.Duration
|
||||
Attempts int // questions already asked; capped by MaxAttempts
|
||||
}
|
||||
|
||||
func (q *PendingQuestion) IsExpired(now time.Time) bool {
|
||||
return now.After(q.Asked.Add(q.TTL))
|
||||
}
|
||||
|
||||
// CanAsk reports whether Maven may ask another question about this request.
|
||||
func (q *PendingQuestion) CanAsk() bool {
|
||||
return q.Attempts < MaxAttempts
|
||||
}
|
||||
|
||||
// TODO: the daemon will phrase the question text from Missing (one short ru
|
||||
// question per Slot, feminine self-reference) and speak it here.
|
||||
|
||||
// ClarifyStore holds the parked questions. Same shape and locking as
|
||||
// SessionStore: keyed by dialogue id, expired entries dropped on read.
|
||||
type ClarifyStore struct {
|
||||
mu sync.RWMutex
|
||||
questions map[string]*PendingQuestion
|
||||
defaultTTL time.Duration
|
||||
}
|
||||
|
||||
func NewClarifyStore(defaultTTL time.Duration) *ClarifyStore {
|
||||
if defaultTTL <= 0 {
|
||||
// Short, like confirmTTL in voice.go: a clarifying question is a
|
||||
// same-breath gesture, a stale one should not eat a later utterance.
|
||||
defaultTTL = 90 * time.Second
|
||||
}
|
||||
return &ClarifyStore{
|
||||
questions: make(map[string]*PendingQuestion),
|
||||
defaultTTL: defaultTTL,
|
||||
}
|
||||
}
|
||||
|
||||
// TODO: the daemon will Put a question here when Decision.Clarify fires, in
|
||||
// place of the flat "не разобрала" reply (cmd/mavend/voice.go).
|
||||
func (s *ClarifyStore) Put(id string, q *PendingQuestion) {
|
||||
if q.TTL <= 0 {
|
||||
q.TTL = s.defaultTTL
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.questions[id] = q
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// TODO: the daemon will Get on the next turn, parse that turn into Slots, call
|
||||
// Answer, and Delete — the open-question twin of resolveConfirm.
|
||||
func (s *ClarifyStore) Get(id string, now time.Time) *PendingQuestion {
|
||||
s.mu.RLock()
|
||||
q, ok := s.questions[id]
|
||||
s.mu.RUnlock()
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
if q.IsExpired(now) {
|
||||
s.Delete(id)
|
||||
return nil
|
||||
}
|
||||
return q
|
||||
}
|
||||
|
||||
func (s *ClarifyStore) Delete(id string) {
|
||||
s.mu.Lock()
|
||||
delete(s.questions, id)
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// Answer merges the slots parsed from the user's answer into the parked ones.
|
||||
// Only the slots listed in Missing are filled, and an already filled slot is
|
||||
// never overwritten — the answer completes the original request, it does not
|
||||
// restate it. Parsing the answer text into `answer` is the caller's job; this
|
||||
// package must stay free of internal/router.
|
||||
func (q *PendingQuestion) Answer(text string, answer Slots) Slots {
|
||||
out := q.Slots
|
||||
for _, slot := range q.Missing {
|
||||
switch slot {
|
||||
case SlotTime:
|
||||
if !out.HasTime && answer.HasTime {
|
||||
out.Time = answer.Time
|
||||
out.HasTime = true
|
||||
}
|
||||
case SlotKey:
|
||||
if !out.HasKey && answer.HasKey {
|
||||
out.Key = answer.Key
|
||||
out.HasKey = true
|
||||
}
|
||||
case SlotValue:
|
||||
if out.Value == "" && answer.Value != "" {
|
||||
out.Value = answer.Value
|
||||
}
|
||||
case SlotFn:
|
||||
if !out.HasFn && answer.HasFn {
|
||||
out.Fn = answer.Fn
|
||||
out.HasFn = true
|
||||
if len(out.Args) == 0 {
|
||||
out.Args = append([]string(nil), answer.Args...)
|
||||
}
|
||||
}
|
||||
case SlotText:
|
||||
if out.Text == "" {
|
||||
if answer.Text != "" {
|
||||
out.Text = answer.Text
|
||||
} else {
|
||||
// No parse for a text slot — the raw answer IS the text.
|
||||
out.Text = text
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// StillMissing lists the slots that are empty in s, out of the ones asked for.
|
||||
// The caller uses it to decide between acting and dropping the request.
|
||||
func StillMissing(want []Slot, s Slots) []Slot {
|
||||
var out []Slot
|
||||
for _, slot := range want {
|
||||
empty := false
|
||||
switch slot {
|
||||
case SlotTime:
|
||||
empty = !s.HasTime
|
||||
case SlotKey:
|
||||
empty = !s.HasKey
|
||||
case SlotValue:
|
||||
empty = s.Value == ""
|
||||
case SlotFn:
|
||||
empty = !s.HasFn
|
||||
case SlotText:
|
||||
empty = s.Text == ""
|
||||
}
|
||||
if empty {
|
||||
out = append(out, slot)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,225 @@
|
||||
package dialogue
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
var base = time.Date(2026, 7, 31, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
func TestPendingQuestionIsExpired(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
ttl time.Duration
|
||||
now time.Time
|
||||
want bool
|
||||
}{
|
||||
{"fresh", time.Minute, base.Add(10 * time.Second), false},
|
||||
{"exactly at ttl", time.Minute, base.Add(time.Minute), false},
|
||||
{"past ttl", time.Minute, base.Add(2 * time.Minute), true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
q := &PendingQuestion{Asked: base, TTL: tc.ttl}
|
||||
if got := q.IsExpired(tc.now); got != tc.want {
|
||||
t.Fatalf("IsExpired = %v, want %v", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClarifyStoreGetPutDelete(t *testing.T) {
|
||||
s := NewClarifyStore(time.Minute)
|
||||
|
||||
if got := s.Get("voice", base); got != nil {
|
||||
t.Fatalf("empty store returned %+v", got)
|
||||
}
|
||||
|
||||
q := &PendingQuestion{Intent: IntentReminder, Missing: []Slot{SlotTime}, Asked: base}
|
||||
s.Put("voice", q)
|
||||
if q.TTL != time.Minute {
|
||||
t.Fatalf("Put did not apply the default TTL, got %v", q.TTL)
|
||||
}
|
||||
if got := s.Get("voice", base.Add(time.Second)); got != q {
|
||||
t.Fatalf("Get returned %+v, want the parked question", got)
|
||||
}
|
||||
|
||||
// Expired questions are dropped on read, not returned.
|
||||
if got := s.Get("voice", base.Add(2*time.Minute)); got != nil {
|
||||
t.Fatalf("expired Get returned %+v", got)
|
||||
}
|
||||
if got := s.Get("voice", base); got != nil {
|
||||
t.Fatalf("expired question was not deleted: %+v", got)
|
||||
}
|
||||
|
||||
s.Put("voice", &PendingQuestion{Asked: base, TTL: time.Hour})
|
||||
s.Delete("voice")
|
||||
if got := s.Get("voice", base); got != nil {
|
||||
t.Fatalf("Delete left %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewClarifyStoreDefaultTTL(t *testing.T) {
|
||||
s := NewClarifyStore(0)
|
||||
q := &PendingQuestion{Asked: base}
|
||||
s.Put("voice", q)
|
||||
if q.TTL != 90*time.Second {
|
||||
t.Fatalf("TTL = %v, want 90s", q.TTL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnswerFillsOnlyMissingSlots(t *testing.T) {
|
||||
answerTime := base.Add(3 * time.Hour)
|
||||
other := base.Add(9 * time.Hour)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
parked Slots
|
||||
missing []Slot
|
||||
text string
|
||||
answer Slots
|
||||
want Slots
|
||||
}{
|
||||
{
|
||||
name: "fills the missing time",
|
||||
parked: Slots{Text: "напомни позвонить"},
|
||||
missing: []Slot{SlotTime},
|
||||
text: "в три",
|
||||
answer: Slots{Time: answerTime, HasTime: true},
|
||||
want: Slots{Text: "напомни позвонить", Time: answerTime, HasTime: true},
|
||||
},
|
||||
{
|
||||
name: "does not overwrite a filled time",
|
||||
parked: Slots{Time: other, HasTime: true},
|
||||
missing: []Slot{SlotTime},
|
||||
text: "в три",
|
||||
answer: Slots{Time: answerTime, HasTime: true},
|
||||
want: Slots{Time: other, HasTime: true},
|
||||
},
|
||||
{
|
||||
name: "ignores slots that were not missing",
|
||||
parked: Slots{Key: "water", HasKey: true},
|
||||
missing: []Slot{SlotValue},
|
||||
text: "два литра",
|
||||
answer: Slots{Key: "sleep", HasKey: true, Value: "2l"},
|
||||
want: Slots{Key: "water", HasKey: true, Value: "2l"},
|
||||
},
|
||||
{
|
||||
name: "fills key when empty",
|
||||
parked: Slots{},
|
||||
missing: []Slot{SlotKey, SlotValue},
|
||||
text: "воды",
|
||||
answer: Slots{Key: "water", HasKey: true, Value: `"drank"`},
|
||||
want: Slots{Key: "water", HasKey: true, Value: `"drank"`},
|
||||
},
|
||||
{
|
||||
name: "fills fn and its args",
|
||||
parked: Slots{},
|
||||
missing: []Slot{SlotFn},
|
||||
text: "перезапусти nginx",
|
||||
answer: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
want: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
},
|
||||
{
|
||||
name: "keeps existing args when fn was already known",
|
||||
parked: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
missing: []Slot{SlotFn},
|
||||
text: "останови postgres",
|
||||
answer: Slots{Fn: "stop", Args: []string{"postgres"}, HasFn: true},
|
||||
want: Slots{Fn: "restart", Args: []string{"nginx"}, HasFn: true},
|
||||
},
|
||||
{
|
||||
name: "raw answer becomes the text when nothing was parsed",
|
||||
parked: Slots{},
|
||||
missing: []Slot{SlotText},
|
||||
text: "купить хлеб",
|
||||
answer: Slots{},
|
||||
want: Slots{Text: "купить хлеб"},
|
||||
},
|
||||
{
|
||||
name: "parsed text wins over the raw answer",
|
||||
parked: Slots{},
|
||||
missing: []Slot{SlotText},
|
||||
text: "запиши купить хлеб",
|
||||
answer: Slots{Text: "купить хлеб"},
|
||||
want: Slots{Text: "купить хлеб"},
|
||||
},
|
||||
{
|
||||
name: "empty answer leaves the slot missing",
|
||||
parked: Slots{Text: "напомни"},
|
||||
missing: []Slot{SlotTime},
|
||||
text: "не знаю",
|
||||
answer: Slots{},
|
||||
want: Slots{Text: "напомни"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
q := &PendingQuestion{Slots: tc.parked, Missing: tc.missing, Asked: base}
|
||||
got := q.Answer(tc.text, tc.answer)
|
||||
if got.Time != tc.want.Time || got.HasTime != tc.want.HasTime ||
|
||||
got.Key != tc.want.Key || got.HasKey != tc.want.HasKey ||
|
||||
got.Value != tc.want.Value || got.Text != tc.want.Text ||
|
||||
got.Fn != tc.want.Fn || got.HasFn != tc.want.HasFn {
|
||||
t.Fatalf("Answer = %+v, want %+v", got, tc.want)
|
||||
}
|
||||
if len(got.Args) != len(tc.want.Args) {
|
||||
t.Fatalf("Args = %v, want %v", got.Args, tc.want.Args)
|
||||
}
|
||||
for i := range got.Args {
|
||||
if got.Args[i] != tc.want.Args[i] {
|
||||
t.Fatalf("Args = %v, want %v", got.Args, tc.want.Args)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCanAskCapsAtOneQuestion(t *testing.T) {
|
||||
if MaxAttempts != 1 {
|
||||
t.Fatalf("MaxAttempts = %d, want 1 (Maven asks once, she is not a nag)", MaxAttempts)
|
||||
}
|
||||
q := &PendingQuestion{Asked: base}
|
||||
if !q.CanAsk() {
|
||||
t.Fatal("a fresh question should be askable")
|
||||
}
|
||||
q.Attempts = MaxAttempts
|
||||
if q.CanAsk() {
|
||||
t.Fatal("the question should not be asked twice")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStillMissing(t *testing.T) {
|
||||
want := []Slot{SlotTime, SlotKey, SlotValue, SlotFn, SlotText}
|
||||
cases := []struct {
|
||||
name string
|
||||
slots Slots
|
||||
want []Slot
|
||||
}{
|
||||
{"all empty", Slots{}, want},
|
||||
{
|
||||
name: "all filled",
|
||||
slots: Slots{Time: base, HasTime: true, Key: "water", HasKey: true, Value: "1l", Fn: "restart", HasFn: true, Text: "t"},
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "only value left",
|
||||
slots: Slots{Time: base, HasTime: true, Key: "water", HasKey: true, Fn: "restart", HasFn: true, Text: "t"},
|
||||
want: []Slot{SlotValue},
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := StillMissing(want, tc.slots)
|
||||
if len(got) != len(tc.want) {
|
||||
t.Fatalf("StillMissing = %v, want %v", got, tc.want)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tc.want[i] {
|
||||
t.Fatalf("StillMissing = %v, want %v", got, tc.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -21,6 +21,7 @@ type Slots struct {
|
||||
Time time.Time
|
||||
HasTime bool
|
||||
Key string
|
||||
Value string // payload for a fact key, mirrors router.Slots.Value
|
||||
HasKey bool
|
||||
Text string
|
||||
Fn string
|
||||
@@ -104,6 +105,9 @@ func InheritSlots(prev, cur Slots) Slots {
|
||||
out.Key = prev.Key
|
||||
out.HasKey = true
|
||||
}
|
||||
if out.Value == "" && prev.Value != "" {
|
||||
out.Value = prev.Value
|
||||
}
|
||||
if out.Text == "" && prev.Text != "" {
|
||||
out.Text = prev.Text
|
||||
}
|
||||
|
||||
@@ -113,4 +113,14 @@ func TestInheritSlots(t *testing.T) {
|
||||
if inherited6.Text != "какая погода в москве" {
|
||||
t.Error("should inherit text when current is empty")
|
||||
}
|
||||
|
||||
prevValue := Slots{Key: "water", HasKey: true, Value: `"drank"`}
|
||||
inherited7 := InheritSlots(prevValue, Slots{})
|
||||
if inherited7.Value != `"drank"` {
|
||||
t.Error("should inherit value when current is empty")
|
||||
}
|
||||
kept := InheritSlots(prevValue, Slots{Value: "2l"})
|
||||
if kept.Value != "2l" {
|
||||
t.Error("should keep current value")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,472 +0,0 @@
|
||||
// Package recalleval is the held-out contract for note recall: can Maven find
|
||||
// the right note again when the user asks for it weeks later?
|
||||
//
|
||||
// Why it sits beside internal/memory rather than inside it: the thing under
|
||||
// test is a whole path, not one function — an embedder (internal/router), a
|
||||
// vector store (internal/memory or internal/store) and the confidence gate the
|
||||
// daemon applies on top (cmd/mavend/recall.go's bestRecall, config's
|
||||
// query_min_score). A _test.go file inside internal/memory could not reach the
|
||||
// persistent store without an import cycle, and testdata is not reachable from
|
||||
// another package's working directory — so the fixture is embedded here and the
|
||||
// scorer takes the store as a factory. Same layout and same reasons as
|
||||
// internal/router/eval.
|
||||
//
|
||||
// The fixture is HELD OUT the same way the routing fixture is: a query never
|
||||
// repeats its note's wording verbatim beyond ordinary shared vocabulary, and
|
||||
// TestFixtureIsParaphrased enforces a floor on how little the two overlap.
|
||||
// Scoring recall on a query that is a copy of the note measures string
|
||||
// matching, not recall.
|
||||
package recalleval
|
||||
|
||||
import (
|
||||
"context"
|
||||
_ "embed"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/memory"
|
||||
"github.com/kami/maven/internal/router"
|
||||
)
|
||||
|
||||
//go:embed ru_recall_v1.json
|
||||
var fixtureJSON []byte
|
||||
|
||||
// SchemaVersion — the version this package understands. The loader refuses any
|
||||
// other version rather than misreading a fixture and reporting a number.
|
||||
const SchemaVersion = 1
|
||||
|
||||
// StoredNote — one thing the user said once, as it lands in the semantic store.
|
||||
// Kind is "note" or "fact"; both share the vector index (see
|
||||
// cmd/mavend/recall.go), so a fact can legitimately win a recall.
|
||||
type StoredNote struct {
|
||||
ID string `json:"id"`
|
||||
Text string `json:"text"`
|
||||
Kind string `json:"kind"`
|
||||
}
|
||||
|
||||
// Case — a small set of notes, one query, and the note that must come back
|
||||
// first. Want is empty exactly when the query should recall NOTHING: that lane
|
||||
// measures false recall, which is the direction the spec cares about ("a
|
||||
// confident wrong fact is worse than a known gap").
|
||||
type Case struct {
|
||||
ID string `json:"id"`
|
||||
Lang string `json:"lang"`
|
||||
Notes []StoredNote `json:"notes"`
|
||||
Query string `json:"query"`
|
||||
Want string `json:"want"`
|
||||
Tags []string `json:"tags"`
|
||||
Note string `json:"note"`
|
||||
}
|
||||
|
||||
// Answerable reports whether the case expects a recall at all.
|
||||
func (c Case) Answerable() bool { return c.Want != "" }
|
||||
|
||||
// Fixture — the versioned envelope, same shape as the routing fixture.
|
||||
//
|
||||
// Filler is inserted into EVERY case's store on top of that case's own notes.
|
||||
// Without it a case with three notes scores recall@3 = 100% by construction,
|
||||
// which measures nothing. A real store holds months of unrelated notes, and the
|
||||
// wanted note has to beat all of them.
|
||||
type Fixture struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
Name string `json:"name"`
|
||||
Notes []string `json:"notes"`
|
||||
Filler []StoredNote `json:"filler"`
|
||||
Cases []Case `json:"cases"`
|
||||
}
|
||||
|
||||
// Load returns the embedded fixture.
|
||||
func Load() (Fixture, error) {
|
||||
var f Fixture
|
||||
if err := json.Unmarshal(fixtureJSON, &f); err != nil {
|
||||
return Fixture{}, fmt.Errorf("parse fixture: %w", err)
|
||||
}
|
||||
if f.SchemaVersion != SchemaVersion {
|
||||
return Fixture{}, fmt.Errorf("fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
|
||||
}
|
||||
if len(f.Cases) == 0 {
|
||||
return Fixture{}, fmt.Errorf("fixture has no cases")
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// NewStore builds an empty store for one case, plus a function to release it.
|
||||
// A factory rather than a store because every case needs a clean index — notes
|
||||
// from case A must not be visible to case B's query.
|
||||
type NewStore func() (memory.Store, func(), error)
|
||||
|
||||
// InMemory is the NewStore for memory.InMemoryStore — the fallback the daemon
|
||||
// uses when there is no database (cmd/mavend/voice.go:224).
|
||||
func InMemory() (memory.Store, func(), error) {
|
||||
return memory.NewInMemoryStore(), func() {}, nil
|
||||
}
|
||||
|
||||
// Cache wraps an embedder so repeated text is embedded once. The gate sweep
|
||||
// scores the same fixture at nine thresholds, and every case re-inserts the
|
||||
// filler notes — without this the ONNX run spends minutes re-embedding
|
||||
// identical strings. Latency numbers come from the uncached run.
|
||||
func Cache(inner router.Embedder) router.Embedder {
|
||||
return &cachingEmbedder{inner: inner, seen: map[string][]float32{}}
|
||||
}
|
||||
|
||||
type cachingEmbedder struct {
|
||||
inner router.Embedder
|
||||
seen map[string][]float32
|
||||
}
|
||||
|
||||
func (c *cachingEmbedder) Dim() int { return c.inner.Dim() }
|
||||
func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner
|
||||
|
||||
func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
|
||||
if v, ok := c.seen[text]; ok {
|
||||
return v, nil
|
||||
}
|
||||
v, err := c.inner.Embed(ctx, text)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.seen[text] = v
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// Outcome — one scored case.
|
||||
type Outcome struct {
|
||||
Case Case
|
||||
Hits []memory.Result
|
||||
Err error
|
||||
// Latency is the read path only: embed the query, then Search. Insert time
|
||||
// is excluded because it happens once, weeks earlier.
|
||||
Latency time.Duration
|
||||
// Rank1/Rank3 — the wanted note came back first / in the top three,
|
||||
// ignoring the confidence gate. Ranking is the store's job.
|
||||
Rank1 bool
|
||||
Rank3 bool
|
||||
// Recalled — what the daemon would actually say back: the top hit's text
|
||||
// when it clears the gate. Mirrors bestRecall in cmd/mavend/recall.go.
|
||||
Recalled string
|
||||
// Pass — the wanted note was recalled AND survived the gate; or, for a
|
||||
// no-answer case, nothing was recalled.
|
||||
Pass bool
|
||||
// Tied — the wanted note is on top but shares its score with the next hit,
|
||||
// so the sort decided it, not the embedder. Counted apart from a real hit.
|
||||
Tied bool
|
||||
TopID string
|
||||
TopScor float64
|
||||
Reasons []string
|
||||
}
|
||||
|
||||
// Report — the aggregate. Rank and gate are kept apart on purpose: a note that
|
||||
// ranks first but is silenced by query_min_score is a threshold problem, and a
|
||||
// note that never ranks first is an embedder problem. Those are different fixes.
|
||||
type Report struct {
|
||||
Name string
|
||||
MinScore float64
|
||||
Total int
|
||||
Answerable int
|
||||
Rank1 int
|
||||
Rank3 int
|
||||
// Gated — ranked first but the score was under MinScore, so the daemon
|
||||
// stays silent and answers "не знаю".
|
||||
Gated int
|
||||
// WrongTop — a different note outranked the right one.
|
||||
WrongTop int
|
||||
// Tied — the right note was on top only because of sort order. Not credited
|
||||
// as recall; tracked because it is a distinct failure (the embedder scored
|
||||
// the query and the note the same as everything else).
|
||||
Tied int
|
||||
// NoAnswer / FalseRecall — the cases that must recall nothing, and how many
|
||||
// of them the daemon would answer anyway.
|
||||
NoAnswer int
|
||||
FalseRecall int
|
||||
Errors int
|
||||
Passed int
|
||||
Outcomes []Outcome
|
||||
ByTag map[string]TagStat
|
||||
ByLang map[string]TagStat
|
||||
// CorrectTop / NoAnswerTop — sorted top-1 scores for the answerable cases
|
||||
// where the right note ranked first, and for the no-answer cases. The gap
|
||||
// between these two distributions is what a defensible query_min_score
|
||||
// would have to sit inside; if they overlap, no threshold separates them.
|
||||
CorrectTop []float64
|
||||
NoAnswerTop []float64
|
||||
P50, P95, Max time.Duration
|
||||
}
|
||||
|
||||
// TagStat — passed/total for one slice of the fixture.
|
||||
type TagStat struct{ Passed, Total int }
|
||||
|
||||
// Recall1 — fraction of answerable cases whose wanted note ranked first.
|
||||
func (r Report) Recall1() float64 { return ratio(r.Rank1, r.Answerable) }
|
||||
|
||||
// Recall3 — same, in the top three. The daemon asks for 3 (voice.go), so this
|
||||
// is the ceiling a better gate or a reranker could reach.
|
||||
func (r Report) Recall3() float64 { return ratio(r.Rank3, r.Answerable) }
|
||||
|
||||
// Answered — fraction of answerable cases the daemon would actually answer
|
||||
// correctly, gate included. This is the number the operator experiences.
|
||||
func (r Report) Answered() float64 { return ratio(r.Rank1-r.Gated, r.Answerable) }
|
||||
|
||||
// FalseRecallRate — fraction of the no-answer cases the daemon answers anyway.
|
||||
func (r Report) FalseRecallRate() float64 { return ratio(r.FalseRecall, r.NoAnswer) }
|
||||
|
||||
func ratio(n, d int) float64 {
|
||||
if d == 0 {
|
||||
return 0
|
||||
}
|
||||
return float64(n) / float64(d)
|
||||
}
|
||||
|
||||
// Score runs every case against a fresh store and aggregates. It never fails
|
||||
// the run on an embed or search error: an erroring case scores as a miss and is
|
||||
// counted in Errors, because "the embedder was down" and "the embedder was
|
||||
// wrong" are different numbers.
|
||||
func Score(ctx context.Context, name string, emb router.Embedder, newStore NewStore, minScore float64, f Fixture) (Report, error) {
|
||||
rep := Report{
|
||||
Name: name,
|
||||
MinScore: minScore,
|
||||
Total: len(f.Cases),
|
||||
ByTag: map[string]TagStat{},
|
||||
ByLang: map[string]TagStat{},
|
||||
}
|
||||
lat := make([]time.Duration, 0, len(f.Cases))
|
||||
|
||||
for _, c := range f.Cases {
|
||||
if c.Answerable() {
|
||||
rep.Answerable++
|
||||
} else {
|
||||
rep.NoAnswer++
|
||||
}
|
||||
o, err := scoreCase(ctx, emb, newStore, minScore, c, f.Filler)
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
lat = append(lat, o.Latency)
|
||||
|
||||
switch {
|
||||
case o.Err != nil:
|
||||
rep.Errors++
|
||||
case c.Answerable():
|
||||
if o.Rank1 {
|
||||
rep.Rank1++
|
||||
}
|
||||
if o.Rank3 {
|
||||
rep.Rank3++
|
||||
}
|
||||
if o.Rank1 && o.Recalled == "" {
|
||||
rep.Gated++
|
||||
}
|
||||
if o.Tied {
|
||||
rep.Tied++
|
||||
} else if !o.Rank1 {
|
||||
rep.WrongTop++
|
||||
}
|
||||
if o.Rank1 && o.Recalled != "" {
|
||||
rep.CorrectTop = append(rep.CorrectTop, o.TopScor)
|
||||
}
|
||||
default:
|
||||
if o.Recalled != "" {
|
||||
rep.FalseRecall++
|
||||
}
|
||||
rep.NoAnswerTop = append(rep.NoAnswerTop, o.TopScor)
|
||||
}
|
||||
|
||||
if o.Pass {
|
||||
rep.Passed++
|
||||
}
|
||||
bump(rep.ByLang, c.Lang, o.Pass)
|
||||
for _, tag := range c.Tags {
|
||||
bump(rep.ByTag, tag, o.Pass)
|
||||
}
|
||||
rep.Outcomes = append(rep.Outcomes, o)
|
||||
}
|
||||
|
||||
sort.Float64s(rep.CorrectTop)
|
||||
sort.Float64s(rep.NoAnswerTop)
|
||||
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
|
||||
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
|
||||
if len(lat) > 0 {
|
||||
rep.Max = lat[len(lat)-1]
|
||||
}
|
||||
return rep, nil
|
||||
}
|
||||
|
||||
// scoreCase inserts the case's notes into a fresh store, then runs the read
|
||||
// path the daemon runs. The returned error is fatal (the harness is broken);
|
||||
// an embedder or store failure on the query lands in Outcome.Err instead.
|
||||
func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minScore float64, c Case, filler []StoredNote) (Outcome, error) {
|
||||
st, release, err := newStore()
|
||||
if err != nil {
|
||||
return Outcome{}, fmt.Errorf("%s: new store: %w", c.ID, err)
|
||||
}
|
||||
defer release()
|
||||
|
||||
all := append(append([]StoredNote(nil), c.Notes...), filler...)
|
||||
for _, n := range all {
|
||||
vec, err := emb.Embed(ctx, n.Text)
|
||||
if err != nil {
|
||||
return Outcome{}, fmt.Errorf("%s: embed note %s: %w", c.ID, n.ID, err)
|
||||
}
|
||||
meta := map[string]string{"text": n.Text, "type": n.Kind}
|
||||
if err := st.Insert(ctx, n.ID, vec, meta); err != nil {
|
||||
return Outcome{}, fmt.Errorf("%s: insert %s: %w", c.ID, n.ID, err)
|
||||
}
|
||||
}
|
||||
|
||||
o := Outcome{Case: c}
|
||||
start := time.Now()
|
||||
qvec, err := emb.Embed(ctx, c.Query)
|
||||
if err != nil {
|
||||
o.Latency = time.Since(start)
|
||||
o.Err = err
|
||||
o.Reasons = []string{fmt.Sprintf("embed query: %v", err)}
|
||||
return o, nil
|
||||
}
|
||||
hits, err := st.Search(ctx, qvec, 3)
|
||||
o.Latency = time.Since(start)
|
||||
if err != nil {
|
||||
o.Err = err
|
||||
o.Reasons = []string{fmt.Sprintf("search: %v", err)}
|
||||
return o, nil
|
||||
}
|
||||
o.Hits = hits
|
||||
|
||||
if len(hits) > 0 {
|
||||
o.TopID, o.TopScor = hits[0].ID, hits[0].Score
|
||||
o.Recalled = bestRecall(hits, minScore)
|
||||
}
|
||||
for i, h := range hits {
|
||||
if h.ID != c.Want {
|
||||
continue
|
||||
}
|
||||
o.Rank3 = true
|
||||
// A tie is not a hit. With a lexical embedder several notes score
|
||||
// exactly 0 against a paraphrased query, and whichever one the sort
|
||||
// happens to leave on top would otherwise be credited as recall.
|
||||
if i == 0 && (len(hits) < 2 || hits[0].Score > hits[1].Score) {
|
||||
o.Rank1 = true
|
||||
}
|
||||
if i == 0 && !o.Rank1 {
|
||||
o.Tied = true
|
||||
}
|
||||
}
|
||||
|
||||
switch {
|
||||
case !c.Answerable():
|
||||
if o.Recalled != "" {
|
||||
o.Reasons = append(o.Reasons, fmt.Sprintf("false recall: %q at %.3f, want silence", o.TopID, o.TopScor))
|
||||
}
|
||||
case o.Tied:
|
||||
o.Reasons = append(o.Reasons, fmt.Sprintf("tie at %.3f — the right note is on top only by sort order", o.TopScor))
|
||||
case !o.Rank1:
|
||||
o.Reasons = append(o.Reasons, fmt.Sprintf("top hit %q (%.3f), want %q%s", o.TopID, o.TopScor, c.Want, rankNote(o.Rank3)))
|
||||
case o.Recalled == "":
|
||||
o.Reasons = append(o.Reasons, fmt.Sprintf("right note ranked first but scored %.3f < gate %.2f — daemon says \"не знаю\"", o.TopScor, minScore))
|
||||
}
|
||||
o.Pass = len(o.Reasons) == 0
|
||||
return o, nil
|
||||
}
|
||||
|
||||
func rankNote(inTop3 bool) string {
|
||||
if inTop3 {
|
||||
return " (wanted note is in the top 3)"
|
||||
}
|
||||
return " (wanted note is not in the top 3)"
|
||||
}
|
||||
|
||||
// bestRecall mirrors cmd/mavend/recall.go — the gate the daemon actually
|
||||
// applies to a memory hit. Duplicated rather than imported because package main
|
||||
// is not importable; recalleval_test.go asserts the two agree in behaviour.
|
||||
func bestRecall(results []memory.Result, min float64) string {
|
||||
if len(results) == 0 || results[0].Score < min {
|
||||
return ""
|
||||
}
|
||||
return results[0].Meta["text"]
|
||||
}
|
||||
|
||||
func bump(m map[string]TagStat, key string, pass bool) {
|
||||
if key == "" {
|
||||
return
|
||||
}
|
||||
s := m[key]
|
||||
s.Total++
|
||||
if pass {
|
||||
s.Passed++
|
||||
}
|
||||
m[key] = s
|
||||
}
|
||||
|
||||
// percentile — nearest-rank on a pre-sorted slice. No interpolation: with ~30
|
||||
// samples an interpolated p95 invents a latency no query actually took.
|
||||
func percentile(sorted []time.Duration, p float64) time.Duration {
|
||||
if len(sorted) == 0 {
|
||||
return 0
|
||||
}
|
||||
i := int(p * float64(len(sorted)))
|
||||
if i >= len(sorted) {
|
||||
i = len(sorted) - 1
|
||||
}
|
||||
return sorted[i]
|
||||
}
|
||||
|
||||
// String renders the report in the routing eval's style — headline first, then
|
||||
// the slices that name where the path is weak.
|
||||
func (r Report) String() string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "%s: %d/%d cases pass (gate %.2f)\n", r.Name, r.Passed, r.Total, r.MinScore)
|
||||
fmt.Fprintf(&b, " recall@1 %.1f%% (%d/%d) recall@3 %.1f%% (%d/%d) answered after gate %.1f%% (%d/%d)\n",
|
||||
100*r.Recall1(), r.Rank1, r.Answerable,
|
||||
100*r.Recall3(), r.Rank3, r.Answerable,
|
||||
100*r.Answered(), r.Rank1-r.Gated, r.Answerable)
|
||||
fmt.Fprintf(&b, " wrong note on top: %d | tie on top (sort order, not recall): %d | silenced by gate: %d | errors: %d\n",
|
||||
r.WrongTop, r.Tied, r.Gated, r.Errors)
|
||||
fmt.Fprintf(&b, " false recall %.1f%% (%d/%d must-be-silent cases answered anyway)\n",
|
||||
100*r.FalseRecallRate(), r.FalseRecall, r.NoAnswer)
|
||||
fmt.Fprintf(&b, " top-1 score, right note first: %s\n", spread(r.CorrectTop))
|
||||
fmt.Fprintf(&b, " top-1 score, must be silent: %s\n", spread(r.NoAnswerTop))
|
||||
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
|
||||
fmt.Fprintf(&b, " by lang: %s\n", renderStats(r.ByLang))
|
||||
fmt.Fprintf(&b, " by tag: %s\n", renderStats(r.ByTag))
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Failures — per-case detail, sorted by ID so two runs diff cleanly.
|
||||
func (r Report) Failures() string {
|
||||
var b strings.Builder
|
||||
out := append([]Outcome(nil), r.Outcomes...)
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Case.ID < out[j].Case.ID })
|
||||
for _, o := range out {
|
||||
if o.Pass {
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(&b, " %s %q: %s\n", o.Case.ID, o.Case.Query, strings.Join(o.Reasons, "; "))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// spread — min / median / max of a sorted score list. Three numbers is enough
|
||||
// to see whether two distributions overlap, which is the only question a
|
||||
// threshold can answer.
|
||||
func spread(sorted []float64) string {
|
||||
if len(sorted) == 0 {
|
||||
return "n/a"
|
||||
}
|
||||
return fmt.Sprintf("min %.3f median %.3f max %.3f (n=%d)",
|
||||
sorted[0], sorted[len(sorted)/2], sorted[len(sorted)-1], len(sorted))
|
||||
}
|
||||
|
||||
func renderStats(m map[string]TagStat) string {
|
||||
keys := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
parts := make([]string, 0, len(keys))
|
||||
for _, k := range keys {
|
||||
s := m[k]
|
||||
parts = append(parts, fmt.Sprintf("%s %d/%d", k, s.Passed, s.Total))
|
||||
}
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
@@ -1,292 +0,0 @@
|
||||
package recalleval
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode"
|
||||
|
||||
"github.com/kami/maven/internal/config"
|
||||
"github.com/kami/maven/internal/memory"
|
||||
"github.com/kami/maven/internal/router"
|
||||
"github.com/kami/maven/internal/store"
|
||||
)
|
||||
|
||||
// dim 1024 for the hash embedder: it is bag-of-words, so a narrower space
|
||||
// collides tokens between unrelated notes and would measure the hash.
|
||||
const hashDim = 1024
|
||||
|
||||
func TestLoadFixture(t *testing.T) {
|
||||
f, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
if len(f.Cases) < 25 {
|
||||
t.Errorf("%d cases, want >= 25", len(f.Cases))
|
||||
}
|
||||
// Filler is what stops recall@3 being free: three case notes and a top-3
|
||||
// search would put the wanted note in the top 3 every time.
|
||||
if len(f.Filler) < 10 {
|
||||
t.Errorf("%d filler notes, want >= 10", len(f.Filler))
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
silent, en := 0, 0
|
||||
for _, c := range f.Cases {
|
||||
if c.ID == "" || seen[c.ID] {
|
||||
t.Errorf("case %q: empty or duplicate id", c.ID)
|
||||
}
|
||||
seen[c.ID] = true
|
||||
if c.Lang != "ru" && c.Lang != "en" {
|
||||
t.Errorf("%s: lang %q, want ru|en", c.ID, c.Lang)
|
||||
}
|
||||
if c.Lang == "en" {
|
||||
en++
|
||||
}
|
||||
if strings.TrimSpace(c.Query) == "" {
|
||||
t.Errorf("%s: empty query", c.ID)
|
||||
}
|
||||
// Fewer than three notes and a wrong answer has nowhere to come from,
|
||||
// so recall@1 would be near-free.
|
||||
if len(c.Notes) < 3 {
|
||||
t.Errorf("%s: %d notes, want >= 3", c.ID, len(c.Notes))
|
||||
}
|
||||
ids := map[string]bool{}
|
||||
for _, n := range c.Notes {
|
||||
if n.ID == "" || ids[n.ID] {
|
||||
t.Errorf("%s: note %q empty or duplicate id", c.ID, n.ID)
|
||||
}
|
||||
ids[n.ID] = true
|
||||
if strings.TrimSpace(n.Text) == "" {
|
||||
t.Errorf("%s: note %q empty text", c.ID, n.ID)
|
||||
}
|
||||
if n.Kind != "note" && n.Kind != "fact" {
|
||||
t.Errorf("%s: note %q kind %q, want note|fact", c.ID, n.ID, n.Kind)
|
||||
}
|
||||
}
|
||||
if !c.Answerable() {
|
||||
silent++
|
||||
continue
|
||||
}
|
||||
if !ids[c.Want] {
|
||||
t.Errorf("%s: want %q is not one of the case's notes", c.ID, c.Want)
|
||||
}
|
||||
}
|
||||
// Both lanes need enough cases that a rate means something.
|
||||
if silent < 5 {
|
||||
t.Errorf("%d must-be-silent cases, want >= 5", silent)
|
||||
}
|
||||
if en < 5 {
|
||||
t.Errorf("%d English cases, want >= 5", en)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFixtureIsParaphrased — the fixture's claim to measuring recall at all. If
|
||||
// a query repeats its note's words, cosine over a bag-of-words embedder gets it
|
||||
// for free and the score says nothing about semantic recall. Half the query's
|
||||
// words is the line: some shared vocabulary is natural ("nginx", "чай"), a copy
|
||||
// is the failure.
|
||||
func TestFixtureIsParaphrased(t *testing.T) {
|
||||
f, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
for _, c := range f.Cases {
|
||||
if !c.Answerable() {
|
||||
continue
|
||||
}
|
||||
var want string
|
||||
for _, n := range c.Notes {
|
||||
if n.ID == c.Want {
|
||||
want = n.Text
|
||||
}
|
||||
}
|
||||
q := words(c.Query)
|
||||
if len(q) == 0 {
|
||||
continue
|
||||
}
|
||||
inNote := map[string]bool{}
|
||||
for _, w := range words(want) {
|
||||
inNote[w] = true
|
||||
}
|
||||
shared := 0
|
||||
for _, w := range q {
|
||||
if inNote[w] {
|
||||
shared++
|
||||
}
|
||||
}
|
||||
if frac := float64(shared) / float64(len(q)); frac > 0.5 {
|
||||
t.Errorf("%s: query shares %.0f%% of its words with the note — not a paraphrase\n query: %q\n note: %q",
|
||||
c.ID, 100*frac, c.Query, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// words — lowercased words of two runes or more, matching how the hash
|
||||
// embedder tokenizes.
|
||||
func words(s string) []string {
|
||||
var out []string
|
||||
for _, w := range strings.FieldsFunc(strings.ToLower(s), func(r rune) bool {
|
||||
return !unicode.IsLetter(r) && !unicode.IsDigit(r)
|
||||
}) {
|
||||
if len([]rune(w)) > 1 {
|
||||
out = append(out, w)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestBestRecallMatchesDaemon — the harness duplicates bestRecall from
|
||||
// cmd/mavend/recall.go (package main is not importable). This pins the copy to
|
||||
// the original's three rules: no hits, below the gate, or no text ⇒ silence.
|
||||
func TestBestRecallMatchesDaemon(t *testing.T) {
|
||||
if got := bestRecall(nil, 0.55); got != "" {
|
||||
t.Errorf("no hits: got %q, want silence", got)
|
||||
}
|
||||
low := []memory.Result{{ID: "a", Score: 0.4, Meta: map[string]string{"text": "чай"}}}
|
||||
if got := bestRecall(low, 0.55); got != "" {
|
||||
t.Errorf("below gate: got %q, want silence", got)
|
||||
}
|
||||
noText := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{}}}
|
||||
if got := bestRecall(noText, 0.55); got != "" {
|
||||
t.Errorf("no text: got %q, want silence", got)
|
||||
}
|
||||
ok := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{"text": "чай"}}}
|
||||
if got := bestRecall(ok, 0.55); got != "чай" {
|
||||
t.Errorf("above gate: got %q, want %q", got, "чай")
|
||||
}
|
||||
}
|
||||
|
||||
// TestHashRecallBaseline — the CI ratchet. HashEmbedder, so it needs no model
|
||||
// files and is byte-for-byte reproducible.
|
||||
//
|
||||
// It is a floor, not a target. The hash embedder is lexical, so most of this
|
||||
// fixture is unwinnable for it by construction; the number worth moving is
|
||||
// TestONNXRecall's. Never compare a hash-embedder number to an ONNX one.
|
||||
func TestHashRecallBaseline(t *testing.T) {
|
||||
f, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
rep, err := Score(context.Background(), "recall+hash", router.NewHashEmbedder(hashDim), InMemory,
|
||||
config.DefaultQueryMinScore, f)
|
||||
if err != nil {
|
||||
t.Fatalf("Score: %v", err)
|
||||
}
|
||||
t.Log("\n" + rep.String() + rep.Failures())
|
||||
t.Log("\ngate sweep:\n" + sweep(t, router.NewHashEmbedder(hashDim), f))
|
||||
|
||||
// 0.32 sits under the observed 0.360 recall@1.
|
||||
const floorRecall1 = 0.32
|
||||
if rep.Recall1() < floorRecall1 {
|
||||
t.Errorf("recall@1 %.3f below ratchet %.2f — note recall regressed", rep.Recall1(), floorRecall1)
|
||||
}
|
||||
// The dangerous direction, asserted tightly and separately: answering from
|
||||
// the wrong note is worse than a gap. Observed 0 under the hash floor.
|
||||
if rep.FalseRecall > 1 {
|
||||
t.Errorf("%d false recalls, want <= 1:\n%s", rep.FalseRecall, rep.Failures())
|
||||
}
|
||||
}
|
||||
|
||||
// TestPersistentStoreScoresTheSame — the deployed store is sqlite-backed
|
||||
// (store.MemoryStore via st.VectorMemory()), not the in-memory fallback. Its
|
||||
// Search is a separate implementation of the same cosine scan, so it gets its
|
||||
// own run: a divergence here would mean recall quality depends on whether a
|
||||
// database was configured.
|
||||
func TestPersistentStoreScoresTheSame(t *testing.T) {
|
||||
f, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
emb := router.NewHashEmbedder(hashDim)
|
||||
inMem, err := Score(context.Background(), "recall+hash+memory", emb, InMemory, config.DefaultQueryMinScore, f)
|
||||
if err != nil {
|
||||
t.Fatalf("Score in-memory: %v", err)
|
||||
}
|
||||
persistent, err := Score(context.Background(), "recall+hash+sqlite", emb, sqliteStores(t), config.DefaultQueryMinScore, f)
|
||||
if err != nil {
|
||||
t.Fatalf("Score sqlite: %v", err)
|
||||
}
|
||||
t.Log("\n" + persistent.String())
|
||||
if persistent.Rank1 != inMem.Rank1 || persistent.FalseRecall != inMem.FalseRecall {
|
||||
t.Errorf("sqlite recall@1 %d/%d fr %d, in-memory %d/%d fr %d — the two backends disagree",
|
||||
persistent.Rank1, persistent.Answerable, persistent.FalseRecall,
|
||||
inMem.Rank1, inMem.Answerable, inMem.FalseRecall)
|
||||
}
|
||||
}
|
||||
|
||||
// sqliteStores returns a NewStore that hands each case its own plaintext
|
||||
// database file, so cases stay isolated the way they are with InMemory.
|
||||
func sqliteStores(t *testing.T) NewStore {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
n := 0
|
||||
return func() (memory.Store, func(), error) {
|
||||
n++
|
||||
st, err := store.Open(context.Background(), filepath.Join(dir, fmt.Sprintf("recall-%d.db", n)))
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return st.VectorMemory(), func() { _ = st.Close() }, nil
|
||||
}
|
||||
}
|
||||
|
||||
// TestONNXRecall — the number that matters: the multilingual embedder homesrv
|
||||
// actually runs. Opt-in via MAVEN_ONNX_LIB because deps/ is gitignored, exactly
|
||||
// like TestONNXBaseline in internal/router/eval. `make eval-recall` points it at
|
||||
// the vendored runtime.
|
||||
//
|
||||
// Reports rather than asserts. The gate sweep is the point: it prints
|
||||
// answered-vs-false-recall at a range of query_min_score values, so the right
|
||||
// threshold is read off data instead of guessed.
|
||||
func TestONNXRecall(t *testing.T) {
|
||||
lib := os.Getenv("MAVEN_ONNX_LIB")
|
||||
if lib == "" {
|
||||
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
|
||||
}
|
||||
model := filepath.Join("../../..", "models/embedder/model.onnx")
|
||||
tok := filepath.Join("../../..", "models/embedder/tokenizer.json")
|
||||
for _, p := range []string{lib, model, tok} {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
t.Skipf("missing %s: %v", p, err)
|
||||
}
|
||||
}
|
||||
emb, err := router.NewONNXEmbedder(model, tok, lib)
|
||||
if err != nil {
|
||||
t.Skipf("onnx embedder unavailable: %v", err)
|
||||
}
|
||||
defer emb.Close()
|
||||
|
||||
f, err := Load()
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
rep, err := Score(context.Background(), "recall+onnx", emb, InMemory, config.DefaultQueryMinScore, f)
|
||||
if err != nil {
|
||||
t.Fatalf("Score: %v", err)
|
||||
}
|
||||
t.Log("\n" + rep.String() + rep.Failures())
|
||||
// Cached for the sweep only: the headline run above must pay the real
|
||||
// embedder cost so its latency numbers mean something.
|
||||
t.Log("\ngate sweep:\n" + sweep(t, Cache(emb), f))
|
||||
}
|
||||
|
||||
// sweep scores the fixture at a range of gates and renders one line each. Two
|
||||
// columns matter: how many real questions get answered, and how many made-up
|
||||
// ones get answered anyway. A gate is only defensible if some value keeps the
|
||||
// first high and the second at zero.
|
||||
func sweep(t *testing.T, emb router.Embedder, f Fixture) string {
|
||||
t.Helper()
|
||||
var b strings.Builder
|
||||
for _, gate := range []float64{0.0, 0.30, 0.40, 0.50, 0.55, 0.60, 0.70, 0.80, 0.90} {
|
||||
rep, err := Score(context.Background(), "sweep", emb, InMemory, gate, f)
|
||||
if err != nil {
|
||||
t.Fatalf("sweep at %.2f: %v", gate, err)
|
||||
}
|
||||
fmt.Fprintf(&b, " gate %.2f: answered %d/%d (%.0f%%) false recall %d/%d\n",
|
||||
gate, rep.Rank1-rep.Gated, rep.Answerable, 100*rep.Answered(), rep.FalseRecall, rep.NoAnswer)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
@@ -1,392 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"name": "ru_recall_v1",
|
||||
"notes": [
|
||||
"Held-out note-recall fixture. Each case is a fresh semantic store: insert every note, embed the query, take the top 3 — the same read path cmd/mavend/voice.go runs for IntentQuery.",
|
||||
"Queries paraphrase their note on purpose. A query that repeats the note's words measures string matching, not recall. TestFixtureIsParaphrased enforces a ceiling on word overlap.",
|
||||
"want:\"\" means the query must recall NOTHING. Those cases measure false recall — the direction the spec calls out (a confident wrong fact is worse than a known gap).",
|
||||
"The distractor tag marks cases where a second note is plausible and only one is right. The hard tag marks cases with little or no shared vocabulary.",
|
||||
"Content is written for this operator: his preferences, his homelab, things he said once and would expect Maven to remember weeks later."
|
||||
],
|
||||
"filler": [
|
||||
{"id": "f1", "text": "в субботу ходил в баню", "kind": "note"},
|
||||
{"id": "f2", "text": "купил новые кроссовки сорок третьего размера", "kind": "note"},
|
||||
{"id": "f3", "text": "сериал закончился на третьем сезоне", "kind": "note"},
|
||||
{"id": "f4", "text": "сосед сверху делает ремонт", "kind": "note"},
|
||||
{"id": "f5", "text": "билеты в театр брал заранее", "kind": "note"},
|
||||
{"id": "f6", "text": "выучил пару аккордов на гитаре", "kind": "note"},
|
||||
{"id": "f7", "text": "записался к стоматологу", "kind": "note"},
|
||||
{"id": "f8", "text": "поменял лампочку в коридоре", "kind": "note"},
|
||||
{"id": "f9", "text": "погулял вдоль реки", "kind": "note"},
|
||||
{"id": "f10", "text": "the balcony door sticks in winter", "kind": "note"},
|
||||
{"id": "f11", "text": "the neighbour's dog barks at cyclists", "kind": "note"},
|
||||
{"id": "f12", "text": "i finished the book about volcanoes", "kind": "note"}
|
||||
],
|
||||
"cases": [
|
||||
{
|
||||
"id": "ru-pref-001",
|
||||
"lang": "ru",
|
||||
"tags": ["preference", "paraphrase"],
|
||||
"query": "какой кофе мне наливать",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "я пью кофе без сахара", "kind": "note"},
|
||||
{"id": "n2", "text": "по утрам бегаю в парке", "kind": "note"},
|
||||
{"id": "n3", "text": "не люблю громкую музыку", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-pref-002",
|
||||
"lang": "ru",
|
||||
"tags": ["preference", "homelab", "paraphrase", "hard"],
|
||||
"query": "когда запускать резервное копирование",
|
||||
"want": "n1",
|
||||
"note": "The DESIGN.md preference-seam example, phrased as the operator would ask it later.",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "бэкапы лучше делать ночью в три часа", "kind": "note"},
|
||||
{"id": "n2", "text": "обновления ставлю по субботам", "kind": "note"},
|
||||
{"id": "n3", "text": "логи храню месяц", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-003",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "paraphrase"],
|
||||
"query": "что помогло от мерцания монитора",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "мерцание экрана прошло после обновления драйвера amdgpu", "kind": "note"},
|
||||
{"id": "n2", "text": "вентилятор шумит на полной нагрузке", "kind": "note"},
|
||||
{"id": "n3", "text": "поставил новый ssd в ноутбук", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-004",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "distractor", "hard"],
|
||||
"query": "адрес домашнего сервера",
|
||||
"want": "n2",
|
||||
"note": "Two notes carry an IP. Only one is the server.",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "роутер живёт на 192.168.1.1", "kind": "note"},
|
||||
{"id": "n2", "text": "домашний сервер на 192.168.1.104", "kind": "note"},
|
||||
{"id": "n3", "text": "принтер подключен по usb", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-005",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab"],
|
||||
"query": "где искать настройки nginx",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "конфиг nginx лежит в /etc/nginx/sites-enabled", "kind": "note"},
|
||||
{"id": "n2", "text": "сертификаты обновляет certbot по расписанию", "kind": "note"},
|
||||
{"id": "n3", "text": "порт 8080 занят вебкой", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-pref-006",
|
||||
"lang": "ru",
|
||||
"tags": ["preference", "distractor", "hard"],
|
||||
"query": "что мне нельзя есть",
|
||||
"want": "n1",
|
||||
"note": "The Friday-meat note is a plausible second answer but it is a habit, not a restriction.",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "у меня аллергия на орехи", "kind": "note"},
|
||||
{"id": "n2", "text": "не ем мясо по пятницам", "kind": "note"},
|
||||
{"id": "n3", "text": "люблю острую еду", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-pers-007",
|
||||
"lang": "ru",
|
||||
"tags": ["distractor", "paraphrase"],
|
||||
"query": "когда мамин праздник",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "день рождения мамы четырнадцатого марта", "kind": "note"},
|
||||
{"id": "n2", "text": "у брата день рождения в июле", "kind": "note"},
|
||||
{"id": "n3", "text": "годовщина в сентябре", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-008",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "hard", "paraphrase"],
|
||||
"query": "чем ускоряется языковая модель",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "модель крутится на встройке через vulkan", "kind": "note"},
|
||||
{"id": "n2", "text": "whisper работает на процессоре", "kind": "note"},
|
||||
{"id": "n3", "text": "голос у piper русский", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-pref-009",
|
||||
"lang": "ru",
|
||||
"tags": ["preference", "distractor"],
|
||||
"query": "во сколько я обычно засыпаю",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "ложусь спать около часа ночи", "kind": "note"},
|
||||
{"id": "n2", "text": "встаю в семь утра", "kind": "note"},
|
||||
{"id": "n3", "text": "днём не сплю", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-010",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "paraphrase"],
|
||||
"query": "где у меня хранятся пароли",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "пароли держу в keepassxc", "kind": "note"},
|
||||
{"id": "n2", "text": "двухфакторку сделал через totp", "kind": "note"},
|
||||
{"id": "n3", "text": "ssh ключи лежат на юбикее", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-011",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "hard", "paraphrase"],
|
||||
"query": "из-за чего кончилось место",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "диск забился логами докера в июне", "kind": "note"},
|
||||
{"id": "n2", "text": "рейд собрал из двух дисков", "kind": "note"},
|
||||
{"id": "n3", "text": "бэкап на внешний диск раз в неделю", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-pref-012",
|
||||
"lang": "ru",
|
||||
"tags": ["preference", "distractor"],
|
||||
"query": "какой чай мне нравится",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "чай пью только зелёный", "kind": "note"},
|
||||
{"id": "n2", "text": "кофе пью без сахара", "kind": "note"},
|
||||
{"id": "n3", "text": "воду пью из фильтра", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-silent-013",
|
||||
"lang": "ru",
|
||||
"tags": ["silent"],
|
||||
"query": "какая погода будет в пятницу",
|
||||
"want": "",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "роутер живёт на 192.168.1.1", "kind": "note"},
|
||||
{"id": "n2", "text": "бэкапы лучше делать ночью", "kind": "note"},
|
||||
{"id": "n3", "text": "у меня аллергия на орехи", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-silent-014",
|
||||
"lang": "ru",
|
||||
"tags": ["silent"],
|
||||
"query": "как зовут сестру моего коллеги",
|
||||
"want": "",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "конфиг nginx лежит в /etc/nginx/sites-enabled", "kind": "note"},
|
||||
{"id": "n2", "text": "порт 8080 занят вебкой", "kind": "note"},
|
||||
{"id": "n3", "text": "сертификаты обновляет certbot", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-silent-015",
|
||||
"lang": "ru",
|
||||
"tags": ["silent"],
|
||||
"query": "сколько я заплатил за машину",
|
||||
"want": "",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "чай пью только зелёный", "kind": "note"},
|
||||
{"id": "n2", "text": "ложусь спать около часа ночи", "kind": "note"},
|
||||
{"id": "n3", "text": "не люблю громкую музыку", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-016",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "paraphrase"],
|
||||
"query": "откуда берётся токен бота",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "токен телеграма лежит в deploy/telegram.env", "kind": "note"},
|
||||
{"id": "n2", "text": "вебхуки не использую, только long-poll", "kind": "note"},
|
||||
{"id": "n3", "text": "уведомления приходят в личку", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-hard-017",
|
||||
"lang": "ru",
|
||||
"tags": ["hard", "paraphrase", "homelab"],
|
||||
"query": "как я восстановил конфиги",
|
||||
"want": "n1",
|
||||
"note": "No shared word between query and note beyond none at all. This is the case a lexical embedder cannot win.",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "после переустановки системы вернул все настройки из git", "kind": "note"},
|
||||
{"id": "n2", "text": "разделы на диске резал вручную", "kind": "note"},
|
||||
{"id": "n3", "text": "загрузчик поставил заново", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-dist-018",
|
||||
"lang": "ru",
|
||||
"tags": ["distractor"],
|
||||
"query": "чем кормить кота",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "кот ест только сухой корм", "kind": "note"},
|
||||
{"id": "n2", "text": "собаке даю мясо", "kind": "note"},
|
||||
{"id": "n3", "text": "рыбок кормлю раз в день", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-019",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "hard", "paraphrase"],
|
||||
"query": "как контейнер получает доступ к видеокарте",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "docker compose пробрасывает /dev/dri внутрь", "kind": "note"},
|
||||
{"id": "n2", "text": "контейнеры рестартуют сами", "kind": "note"},
|
||||
{"id": "n3", "text": "образы чищу вручную", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-pref-020",
|
||||
"lang": "ru",
|
||||
"tags": ["preference", "distractor"],
|
||||
"query": "когда мне нельзя звонить",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "не звони мне после десяти вечера", "kind": "note"},
|
||||
{"id": "n2", "text": "утром не трогай меня до кофе", "kind": "note"},
|
||||
{"id": "n3", "text": "по выходным не работаю", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "en-pref-021",
|
||||
"lang": "en",
|
||||
"tags": ["preference", "paraphrase"],
|
||||
"query": "which colour scheme do i like",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "i prefer dark theme everywhere", "kind": "note"},
|
||||
{"id": "n2", "text": "font size 14 is fine", "kind": "note"},
|
||||
{"id": "n3", "text": "i use vim keybindings", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "en-home-022",
|
||||
"lang": "en",
|
||||
"tags": ["homelab", "distractor"],
|
||||
"query": "where is the big disk mounted",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "the nas drive is mounted at /mnt/hdd1", "kind": "note"},
|
||||
{"id": "n2", "text": "models live on the ssd", "kind": "note"},
|
||||
{"id": "n3", "text": "backups go to the nas nightly", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "en-silent-023",
|
||||
"lang": "en",
|
||||
"tags": ["silent"],
|
||||
"query": "what is my bank account number",
|
||||
"want": "",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "the nas drive is mounted at /mnt/hdd1", "kind": "note"},
|
||||
{"id": "n2", "text": "i prefer dark theme everywhere", "kind": "note"},
|
||||
{"id": "n3", "text": "the router runs openwrt", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "en-hard-024",
|
||||
"lang": "en",
|
||||
"tags": ["hard", "paraphrase"],
|
||||
"query": "what fixed the screen problem",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "the flicker went away once i swapped the display cable", "kind": "note"},
|
||||
{"id": "n2", "text": "the laptop fan is loud", "kind": "note"},
|
||||
{"id": "n3", "text": "the second monitor is 1440p", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "en-pref-025",
|
||||
"lang": "en",
|
||||
"tags": ["preference", "hard"],
|
||||
"query": "should i be offered wine",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "i do not drink alcohol", "kind": "note"},
|
||||
{"id": "n2", "text": "i skip breakfast", "kind": "note"},
|
||||
{"id": "n3", "text": "i like spicy food", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-home-026",
|
||||
"lang": "ru",
|
||||
"tags": ["homelab", "paraphrase"],
|
||||
"query": "какая модель распознавания речи мне подходит",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "whisper модель small хватает для русского", "kind": "note"},
|
||||
{"id": "n2", "text": "голос ирина звучит лучше остальных", "kind": "note"},
|
||||
{"id": "n3", "text": "слово активации маven", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-fact-027",
|
||||
"lang": "ru",
|
||||
"tags": ["distractor", "hard", "paraphrase"],
|
||||
"query": "когда я последний раз обслуживал машину",
|
||||
"want": "n1",
|
||||
"note": "A fact, not a note — both share the vector index, so a fact can win a recall.",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "последний раз менял масло в мае", "kind": "fact"},
|
||||
{"id": "n2", "text": "шины поменял осенью", "kind": "fact"},
|
||||
{"id": "n3", "text": "страховка до декабря", "kind": "fact"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-pref-028",
|
||||
"lang": "ru",
|
||||
"tags": ["preference", "hard", "paraphrase"],
|
||||
"query": "как мне присылать оповещения",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "терпеть не могу уведомления со звуком", "kind": "note"},
|
||||
{"id": "n2", "text": "вибрацию оставь включённой", "kind": "note"},
|
||||
{"id": "n3", "text": "письма читаю вечером", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ru-silent-029",
|
||||
"lang": "ru",
|
||||
"tags": ["silent"],
|
||||
"query": "во сколько отходит поезд",
|
||||
"want": "",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "кот ест только сухой корм", "kind": "note"},
|
||||
{"id": "n2", "text": "люблю острую еду", "kind": "note"},
|
||||
{"id": "n3", "text": "пароли держу в keepassxc", "kind": "note"}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "en-home-030",
|
||||
"lang": "en",
|
||||
"tags": ["homelab", "paraphrase"],
|
||||
"query": "what firmware is on the router",
|
||||
"want": "n1",
|
||||
"notes": [
|
||||
{"id": "n1", "text": "the router runs openwrt", "kind": "note"},
|
||||
{"id": "n2", "text": "wifi channel is 6", "kind": "note"},
|
||||
{"id": "n3", "text": "the guest network is off", "kind": "note"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user