Compare commits

...

7 Commits

Author SHA1 Message Date
kami 34521c30b8 Merge branch 'worktree-agent-ad5da57e47b822152' into overnight-jul31 2026-07-31 11:39:35 +04:00
kami 1d48755d12 Record the recall numbers after the embedder swap
recall@1 60% to 72%, answered 48% to 72%, latency 3x better. But false
recall went 1/5 to 5/5: e5 packs every score into a narrow high band, so
the 0.55 gate now admits everything. Left the gate alone as instructed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:18 +04:00
kami f6d5a2a7a4 Swap the embedder to multilingual-e5-small (Vikunja #371, #372)
The old model was a symmetric paraphrase model, so it scored "do these
look alike" instead of "does this note answer this question". Also fixes
the file mismatch: the Makefile, the deploy config and both evals now all
name the same quantized file, and the quantized one is what gets measured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:18 +04:00
kami 751c2a705f Embed a question and a stored note differently (Vikunja #371)
Note recall is asymmetric: a short question goes in, a longer note comes
out. Adds EmbedQuery/EmbedPassage helpers and the e5 prefixes, and points
the note/fact write path at the passage side and the query path at the
query side. Reviewers: the three call sites in voice.go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:07 +04:00
kami 5d5b0cfd49 Merge branch 'worktree-agent-a0ab2a9b7439296e3' into overnight-jul31 2026-07-31 11:37:53 +04:00
kami bd16ca69e5 Let the LLM router answer "unknown" when it cannot route
Chose an 8th enum value over a confidence number: the model already picks
one enum token, so it costs nothing in the grammar, while a score from a
0.8B model would be uncalibrated noise. A refusal returns "no decision"
with no error, which is the fall-through the caller already uses for a
bad parse, so the classifier and its clarify gate take the turn.

Reviewers: the prompt's counter-examples matter most — a small model will
over-use any easy escape hatch. The training workspace copy of the prompt
still needs the same edit (Vikunja #362).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:35:37 +04:00
kami 0ed386eca6 Re-measure the router on a quiet box and record the numbers
The earlier before/after was taken while another eval shared
llama-server. This run had the box to itself.

Intent accuracy 61.8% llm-only, 63.2% cascade, 67.1% with thinking off.
The prompt fix holds. note→fact shows up here too, so it is real.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:34:57 +04:00
15 changed files with 360 additions and 34 deletions
+8 -5
View File
@@ -43,14 +43,17 @@ notes. Without it, the floor `HashEmbedder` is used — deterministic but weak
(Russian recall rarely clears the confidence gate, many commands fall to (Russian recall rarely clears the confidence gate, many commands fall to
"clarify"). "clarify").
**Download the embedder** (ONNX, ~90 MB): **Download the embedder** (ONNX, ~120 MB):
```sh ```sh
make download-embedder make download-embedder
``` ```
This fetches `paraphrase-multilingual-MiniLM-L12-v2` (384-dim, 12-layer, This fetches `multilingual-e5-small` (384-dim, 12-layer, Russian and English)
supports 50+ languages including Russian) to `models/embedder/`. to `models/embedder/multilingual-e5-small/`. It is an asymmetric retrieval
model: the code puts `query: ` in front of a question and `passage: ` in front
of a stored note, which is how e5 was trained. The quantized file is the one
that is downloaded, deployed and measured.
**Also need ONNX Runtime** (`libonnxruntime.so`): **Also need ONNX Runtime** (`libonnxruntime.so`):
@@ -64,8 +67,8 @@ sudo cp onnxruntime-linux-x64-1.15.1/lib/libonnxruntime.so* /usr/local/lib/
```json ```json
"voice": { "voice": {
"embedder": { "embedder": {
"model_path": "models/embedder/model_quantized.onnx", "model_path": "models/embedder/multilingual-e5-small/model_quantized.onnx",
"tokenizer_path": "models/embedder/tokenizer.json", "tokenizer_path": "models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/usr/local/lib/libonnxruntime.so" "lib_path": "/usr/local/lib/libonnxruntime.so"
} }
} }
+7 -3
View File
@@ -152,9 +152,13 @@ deps-piper:
-o /tmp/piper.tar.gz -o /tmp/piper.tar.gz
tar -xzf /tmp/piper.tar.gz -C deps/ tar -xzf /tmp/piper.tar.gz -C deps/
EMBEDDER_DIR := $(shell pwd)/models/embedder # multilingual-e5-small: an asymmetric retrieval model. It is trained to match
EMBEDDER_MODEL_URL := https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/onnx/model_quantized.onnx # a short question against a longer passage, which is what note recall is.
EMBEDDER_TOKENIZER_URL := https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/tokenizer.json # The quantized file is the one we download, deploy and measure — see
# RECALL-EVAL-31-07-2026.md.
EMBEDDER_DIR := $(shell pwd)/models/embedder/multilingual-e5-small
EMBEDDER_MODEL_URL := https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/onnx/model_quantized.onnx
EMBEDDER_TOKENIZER_URL := https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/tokenizer.json
download-embedder: download-embedder:
mkdir -p $(EMBEDDER_DIR) mkdir -p $(EMBEDDER_DIR)
+60
View File
@@ -83,6 +83,66 @@ sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — bot
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is (`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
not the problem to solve. not the problem to solve.
## Re-measured after the embedder swap — 31-07-2026, later the same day
Changed: `models/embedder/` is now **multilingual-e5-small** (quantized, 118MB), with `query: ` in
front of a question and `passage: ` in front of a stored note (Vikunja #371). `deploy/mavend.json`
and `make download-embedder` now name the same file, and it is the quantized one — that is what the
column below measures (Vikunja #372). Everything else is unchanged: same fixture, same store, same
0.55 gate. The old column is the baseline and is left as it was.
| | recall+onnx, MiniLM (baseline) | recall+onnx, e5-small (new) |
|---|---|---|
| **recall@1** | 60.0% (15/25) | **72.0% (18/25)** |
| recall@3 | 80.0% (20/25) | 84.0% (21/25) |
| **answered after the 0.55 gate** | 48.0% (12/25) | **72.0% (18/25)** |
| wrong note on top / tie on top | 10 / 0 | 7 / 0 |
| ranked first, then silenced by the gate | 3 | 0 |
| **false recall** | 1/5 (20%) | **5/5 (100%)** |
| top-1 score when right, min / median | 0.559 / 0.678 | 0.791 / 0.857 |
| top-1 when it must stay silent, median / max | 0.470 / 0.567 | 0.815 / 0.835 |
| RU / EN / `hard` cases passed | 13/24 / 3/6 / 2/11 | 14/24 / 4/6 / 5/11 |
| latency p50 / p95 / max | 59ms / 148ms / 194ms | 18ms / 37ms / 49ms |
### What moved
Ranking got better and got faster. Half the previously-unwinnable `hard` cases now pass (2/11 →
5/11), the guitar note no longer beats the docker-logs note, and the gate stops silencing notes that
already ranked first. The quantized e5 is also ~3x quicker than the fp32 MiniLM it replaces.
### What got worse: the gate is now a no-op
e5 packs every cosine into a narrow high band. Right-note scores start at 0.791; must-stay-silent
scores reach 0.835. **The distributions still overlap, and now they overlap above the gate**, so
0.55 admits everything and false recall goes from 1/5 to 5/5. The sweep:
```
gate 0.500.70: answered 18/25 (72%) false recall 5/5
gate 0.80: answered 17/25 (68%) false recall 4/5
gate 0.90: answered 0/25 ( 0%) false recall 0/5
```
There is no value that keeps real recall and rejects made-up questions — same conclusion as before,
now with a wider band and no room at all. `query_min_score` was left at 0.55 as instructed. **The
recommendation is to leave it there and stop tuning it**: any number under ~0.79 is a no-op and
anything above starts cutting real recall long before it stops the false ones. The fix is a margin
gate (`top1 top2 > δ`), next-steps item 3, which is now the top item.
### The prefixes did not do the work
A control run with both prefixes set to the empty string scored the **same** recall@1 (72%), a
slightly better recall@3 (88%) and the same 5/5 false recall. So on this fixture the gain comes from
the model, not from the `query:` / `passage:` split. The prefixes are kept because they are how e5
was trained and the split is the right shape for the read path, but they are not worth defending on
this evidence — a bigger fixture may say otherwise.
### Stored vectors from the old model are now junk
Cosine between a MiniLM vector and an e5 vector means nothing. Every row already in `notes` and in
the vector memory table was written by the old model, so after this deploy they will score as noise
against a new query. A live database needs every note and fact re-embedded before recall works at
all. Filed as its own task.
## Next steps — ordered by value-to-risk; nothing here is a decision ## Next steps — ordered by value-to-risk; nothing here is a decision
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config 1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
+35
View File
@@ -40,6 +40,41 @@ model → classifier as failure floor.
Never compare a hash-embedder run to an ONNX one. Never compare a hash-embedder run to an ONNX one.
## Re-measured after the prompt fix
The table above is the **baseline at commit `46259b4`**, kept as-is. The prompt fix (query
tested before fact, plus `repeat_penalty` and a bounded grammar string) was then measured on
an otherwise idle box — no other eval sharing llama-server, so these latencies are real
rather than contention.
| | llm-only (0.8B) | cascade+llm (0.8B) | llm-only, thinking off |
|---|---|---|---|
| **intent-only accuracy** | 48.7% → **61.8%** | 50.0% → **63.2%** | **67.1%** |
| full accuracy (intent+slots+gate) | 23.7% → **38.2%** | 32.9% → **47.4%** | **42.1%** |
| route errors | 2 → **0** | 0 → 0 | **0** |
| p50 / p95 latency | **1.08s / 1.55s** | **1.04s / 1.53s** | **0.93s / 1.41s** |
Three things this run settles:
1. **The prompt fix holds.** An earlier contended run reported 60.5% / 36.8% for llm-only;
the quiet run gives 61.8% / 38.2%. Close enough to call the gain real, and the earlier
run's 4-5s latency figures were contention, not the model.
2. **`query→fact` fell from ×15 to ×7**, and both unparseable replies are gone. Zero route
errors in every LLM configuration.
3. **`note→fact ×4` is real, not noise.** It shows up in the quiet run too. The agent that
wrote the prompt fix suspected its own change might have caused it by pulling assertive
`запиши что…` phrasings toward fact, and that suspicion stands — all five `ru-note-*`
cases now land on fact. Tracked as Vikunja #375.
**Thinking off is the best configuration measured so far**, on both accuracy and latency
(Vikunja #376). That is worth understanding before flipping: routing is a short
classification into a fixed enum with grammar-constrained output, so there is little to
reason about, and the thinking trace mostly gives a small model room to talk itself out of
the right answer. Phrasing is a different job and needs measuring separately.
Still `6 / 6` missed clarify — the router has no way to say "I don't know" (Vikunja #359).
That is unchanged by anything here.
## Findings ## Findings
### 1. The resident model does route better — 50.0% vs 36.8% ### 1. The resident model does route better — 50.0% vs 36.8%
+3 -3
View File
@@ -559,7 +559,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
// fail the fact write). Facts aren't in the notes table, so this is the // fail the fact write). Facts aren't in the notes table, so this is the
// only recall path for them — "когда я пил воду?" reads back from here. // only recall path for them — "когда я пил воду?" reads back from here.
if h.memStore != nil { if h.memStore != nil {
if vec, err := h.embedder.Embed(ctx, dec.Utterance); err != nil { if vec, err := router.EmbedPassage(ctx, h.embedder, dec.Utterance); err != nil {
log.Printf("voice: embed fact for memory: %v", err) log.Printf("voice: embed fact for memory: %v", err)
} else if err := h.memStore.Insert(ctx, "fact:"+dec.Slots.Key+":"+strconv.FormatInt(now.Unix(), 10), vec, map[string]string{ } else if err := h.memStore.Insert(ctx, "fact:"+dec.Slots.Key+":"+strconv.FormatInt(now.Unix(), 10), vec, map[string]string{
"source": "voice", "source": "voice",
@@ -673,7 +673,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
// embed the note text with the same model the classifier uses, persist // embed the note text with the same model the classifier uses, persist
// via CoreAPI (source=tap:voice). Semantic recall lives in `notes`, not // via CoreAPI (source=tap:voice). Semantic recall lives in `notes`, not
// facts — no predicate reads it (spec's two-memory split). // facts — no predicate reads it (spec's two-memory split).
vec, err := h.embedder.Embed(ctx, dec.Utterance) vec, err := router.EmbedPassage(ctx, h.embedder, dec.Utterance)
if err != nil { if err != nil {
log.Printf("voice: embed note: %v", err) log.Printf("voice: embed note: %v", err)
return "не получилось сохранить заметку." return "не получилось сохранить заметку."
@@ -750,7 +750,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
return fmt.Sprintf("в %s сейчас %.0f градусов, %s.", w.Location, w.Temperature, w.Condition) return fmt.Sprintf("в %s сейчас %.0f градусов, %s.", w.Location, w.Temperature, w.Condition)
} }
vec, err := h.embedder.Embed(ctx, dec.Utterance) vec, err := router.EmbedQuery(ctx, h.embedder, dec.Utterance)
if err != nil { if err != nil {
log.Printf("voice: embed query: %v", err) log.Printf("voice: embed query: %v", err)
return "не получилось найти ответ." return "не получилось найти ответ."
+2 -2
View File
@@ -36,8 +36,8 @@
"stt": { "socket": "/run/maven/stt.sock", "lang": "ru" }, "stt": { "socket": "/run/maven/stt.sock", "lang": "ru" },
"tts": { "socket": "/run/maven/tts.sock", "lang": "ru" }, "tts": { "socket": "/run/maven/tts.sock", "lang": "ru" },
"embedder": { "embedder": {
"model_path": "/opt/maven/models/embedder/model.onnx", "model_path": "/opt/maven/models/embedder/multilingual-e5-small/model_quantized.onnx",
"tokenizer_path": "/opt/maven/models/embedder/tokenizer.json", "tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/opt/maven/lib/libonnxruntime.so" "lib_path": "/opt/maven/lib/libonnxruntime.so"
}, },
"llm_router": false, "llm_router": false,
+7 -7
View File
@@ -262,13 +262,13 @@ type VoiceConfig struct {
// the model gets 50.0% of intents right against the classifier's 36.8%, but // the model gets 50.0% of intents right against the classifier's 36.8%, but
// it costs about 800ms per turn instead of 30ms. // it costs about 800ms per turn instead of 30ms.
// //
// TODO: the default stays false until two things land. // TODO: the default stays false until this lands.
// 1. The LLM router cannot refuse. LLMRouter.Route hardcodes // Extractor.Extract never runs on an LLM decision, so acts arrive with no
// Confidence: 1.0, so the stage-3 clarify gate never fires and an // Fn and reminders with no Time. Turning this on today makes routing more
// unclear utterance becomes a confident wrong action (Vikunja #359). // accurate and less safe.
// 2. Extractor.Extract never runs on an LLM decision, so acts arrive with //
// no Fn and reminders with no Time. // The router can now refuse: it answers "unknown" when it cannot route, and
// Turning this on today makes routing more accurate and less safe. // the turn drops to the classifier and its clarify gate (Vikunja #359).
LLMRouter bool `json:"llm_router,omitempty"` LLMRouter bool `json:"llm_router,omitempty"`
// QueryMinScore — the note-recall confidence gate. Top cosine below this // QueryMinScore — the note-recall confidence gate. Top cosine below this
+27 -5
View File
@@ -117,18 +117,40 @@ type cachingEmbedder struct {
seen map[string][]float32 seen map[string][]float32
} }
var _ router.AsymmetricEmbedder = (*cachingEmbedder)(nil)
func (c *cachingEmbedder) Dim() int { return c.inner.Dim() } func (c *cachingEmbedder) Dim() int { return c.inner.Dim() }
func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner
func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) { func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
if v, ok := c.seen[text]; ok { return c.cached(ctx, "embed:"+text, func() ([]float32, error) {
return c.inner.Embed(ctx, text)
})
}
// The two sides of an asymmetric embedder give different vectors for the same
// string, so the cache key has to say which side asked.
func (c *cachingEmbedder) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
return c.cached(ctx, "query:"+text, func() ([]float32, error) {
return router.EmbedQuery(ctx, c.inner, text)
})
}
func (c *cachingEmbedder) EmbedPassage(ctx context.Context, text string) ([]float32, error) {
return c.cached(ctx, "passage:"+text, func() ([]float32, error) {
return router.EmbedPassage(ctx, c.inner, text)
})
}
func (c *cachingEmbedder) cached(_ context.Context, key string, embed func() ([]float32, error)) ([]float32, error) {
if v, ok := c.seen[key]; ok {
return v, nil return v, nil
} }
v, err := c.inner.Embed(ctx, text) v, err := embed()
if err != nil { if err != nil {
return nil, err return nil, err
} }
c.seen[text] = v c.seen[key] = v
return v, nil return v, nil
} }
@@ -305,7 +327,7 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
all := append(append([]StoredNote(nil), c.Notes...), filler...) all := append(append([]StoredNote(nil), c.Notes...), filler...)
for _, n := range all { for _, n := range all {
vec, err := emb.Embed(ctx, n.Text) vec, err := router.EmbedPassage(ctx, emb, n.Text)
if err != nil { if err != nil {
return Outcome{}, fmt.Errorf("%s: embed note %s: %w", c.ID, n.ID, err) return Outcome{}, fmt.Errorf("%s: embed note %s: %w", c.ID, n.ID, err)
} }
@@ -317,7 +339,7 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
o := Outcome{Case: c} o := Outcome{Case: c}
start := time.Now() start := time.Now()
qvec, err := emb.Embed(ctx, c.Query) qvec, err := router.EmbedQuery(ctx, emb, c.Query)
if err != nil { if err != nil {
o.Latency = time.Since(start) o.Latency = time.Since(start)
o.Err = err o.Err = err
@@ -246,8 +246,8 @@ func TestONNXRecall(t *testing.T) {
if lib == "" { if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing") t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
} }
model := filepath.Join("../../..", "models/embedder/model.onnx") model := filepath.Join("../../..", "models/embedder/multilingual-e5-small/model_quantized.onnx")
tok := filepath.Join("../../..", "models/embedder/tokenizer.json") tok := filepath.Join("../../..", "models/embedder/multilingual-e5-small/tokenizer.json")
for _, p := range []string{lib, model, tok} { for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil { if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err) t.Skipf("missing %s: %v", p, err)
+31
View File
@@ -19,6 +19,37 @@ type Embedder interface {
Close() error Close() error
} }
// AsymmetricEmbedder — an embedder that wants to know whether a text is a
// search query or a stored passage. Recall is asymmetric: a short question
// goes in, a longer note comes out. The e5 family is trained for exactly that
// and needs the side written into the text ("query: " / "passage: ").
//
// Optional on purpose: HashEmbedder has no such notion, so callers go through
// EmbedQuery and EmbedPassage below, which fall back to plain Embed.
type AsymmetricEmbedder interface {
Embedder
EmbedQuery(ctx context.Context, text string) ([]float32, error)
EmbedPassage(ctx context.Context, text string) ([]float32, error)
}
// EmbedQuery embeds text that is being searched WITH — a question.
func EmbedQuery(ctx context.Context, e Embedder, text string) ([]float32, error) {
if a, ok := e.(AsymmetricEmbedder); ok {
return a.EmbedQuery(ctx, text)
}
return e.Embed(ctx, text)
}
// EmbedPassage embeds text that is being searched FOR — a note or a fact on
// its way into the store. Store and lookup must use these two calls, not one
// of them twice, or the asymmetry buys nothing.
func EmbedPassage(ctx context.Context, e Embedder, text string) ([]float32, error) {
if a, ok := e.(AsymmetricEmbedder); ok {
return a.EmbedPassage(ctx, text)
}
return e.Embed(ctx, text)
}
// HashEmbedder — a deterministic bag-of-words embedder used for tests and as a // HashEmbedder — a deterministic bag-of-words embedder used for tests and as a
// non-zero default floor. NOT semantically meaningful across languages; the // non-zero default floor. NOT semantically meaningful across languages; the
// real classifier swaps in the multilingual ONNX model wholesale. // real classifier swaps in the multilingual ONNX model wholesale.
+57
View File
@@ -37,3 +37,60 @@ func TestHashEmbedderCyrillic(t *testing.T) {
t.Fatalf("cosine(shared)=%.3f not > cosine(disjoint)=%.3f", cosine(a, b), cosine(a, c)) t.Fatalf("cosine(shared)=%.3f not > cosine(disjoint)=%.3f", cosine(a, b), cosine(a, c))
} }
} }
// recordingEmbedder — an asymmetric embedder that only remembers which side
// was asked for. Enough to pin the dispatch; real vectors need the model.
type recordingEmbedder struct{ calls []string }
func (r *recordingEmbedder) Dim() int { return 2 }
func (r *recordingEmbedder) Close() error { return nil }
func (r *recordingEmbedder) Embed(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "embed")
return []float32{1, 0}, nil
}
func (r *recordingEmbedder) EmbedQuery(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "query")
return []float32{1, 0}, nil
}
func (r *recordingEmbedder) EmbedPassage(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "passage")
return []float32{0, 1}, nil
}
// TestEmbedQueryAndPassageSplit — a question and a stored note must not take
// the same path. If both ended up on the same call the asymmetric model buys
// nothing, which is the whole reason for the swap.
func TestEmbedQueryAndPassageSplit(t *testing.T) {
rec := &recordingEmbedder{}
if _, err := EmbedQuery(context.Background(), rec, "где логи?"); err != nil {
t.Fatalf("EmbedQuery: %v", err)
}
if _, err := EmbedPassage(context.Background(), rec, "логи в /var/log"); err != nil {
t.Fatalf("EmbedPassage: %v", err)
}
if len(rec.calls) != 2 || rec.calls[0] != "query" || rec.calls[1] != "passage" {
t.Errorf("calls %v, want [query passage]", rec.calls)
}
}
// TestEmbedFallsBackToPlainEmbed — HashEmbedder has no sides, so both helpers
// must still work and give the same vector.
func TestEmbedFallsBackToPlainEmbed(t *testing.T) {
h := NewHashEmbedder(64)
q, err := EmbedQuery(context.Background(), h, "text")
if err != nil {
t.Fatalf("EmbedQuery: %v", err)
}
p, err := EmbedPassage(context.Background(), h, "text")
if err != nil {
t.Fatalf("EmbedPassage: %v", err)
}
for i := range q {
if q[i] != p[i] {
t.Fatalf("hash embedder gave two different vectors for the same text")
}
}
}
+2 -2
View File
@@ -185,8 +185,8 @@ func TestONNXBaseline(t *testing.T) {
if lib == "" { if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing") t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
} }
model := filepath.Join("../../..", "models/embedder/model.onnx") model := filepath.Join("../../..", "models/embedder/multilingual-e5-small/model_quantized.onnx")
tok := filepath.Join("../../..", "models/embedder/tokenizer.json") tok := filepath.Join("../../..", "models/embedder/multilingual-e5-small/tokenizer.json")
for _, p := range []string{lib, model, tok} { for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil { if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err) t.Skipf("missing %s: %v", p, err)
+33 -3
View File
@@ -29,7 +29,7 @@ func NewLLMRouter(c Completer) *LLMRouter { return &LLMRouter{c: c} }
const routeGrammar = ` const routeGrammar = `
root ::= "[" ws action ("," ws action)* ws "]" root ::= "[" ws action ("," ws action)* ws "]"
action ::= "{" ws "\"intent\"" ws ":" ws intent ("," ws field)* ws "}" action ::= "{" ws "\"intent\"" ws ":" ws intent ("," ws field)* ws "}"
intent ::= "\"fact\"" | "\"reminder\"" | "\"note\"" | "\"query\"" | "\"act\"" | "\"chat\"" | "\"system\"" intent ::= "\"fact\"" | "\"reminder\"" | "\"note\"" | "\"query\"" | "\"act\"" | "\"chat\"" | "\"system\"" | "\"unknown\""
field ::= key ws ":" ws string field ::= key ws ":" ws string
key ::= "\"key\"" | "\"value\"" | "\"text\"" | "\"verb\"" key ::= "\"key\"" | "\"value\"" | "\"text\"" | "\"verb\""
string ::= "\"" ([^"\\] | "\\" .){0,120} "\"" string ::= "\"" ([^"\\] | "\\" .){0,120} "\""
@@ -41,12 +41,17 @@ ws ::= [ \t\n]*
// question naming a fact key ("сколько воды я выпил с утра") matched the fact // question naming a fact key ("сколько воды я выпил с утра") matched the fact
// rule first and was stored as an assertion — 15 of 76 fixture cases. // rule first and was stored as an assertion — 15 of 76 fixture cases.
// //
// Changed again 31-07-2026: added the "unknown" escape hatch so the model can
// admit it cannot route (Vikunja #359).
//
// The training workspace keeps its own copy of this prompt for relabelling, and // The training workspace keeps its own copy of this prompt for relabelling, and
// `llm/check_prompt_parity.py` there compares the two. That copy is in another // `llm/check_prompt_parity.py` there compares the two. That copy is in another
// repo and was not touched, so parity will fail until it gets the same edit. // repo and was not touched, so parity will fail until it gets the same edits —
// both the rule reorder and the "unknown" wording (Vikunja #362).
const routeSystem = `Классифицируй ровно одно сообщение пользователя. Верни ОДИН JSON-массив действий. const routeSystem = `Классифицируй ровно одно сообщение пользователя. Верни ОДИН JSON-массив действий.
Ровно одно намерение: fact, reminder, note, query, act, chat, system. Ровно одно намерение: fact, reminder, note, query, act, chat, system.
Есть восьмое значение unknown — только для случаев, когда просьбу невозможно понять.
Классифицируй по цели пользователя. Порядок решения: Классифицируй по цели пользователя. Порядок решения:
1. Хочет напоминание в будущем → reminder 1. Хочет напоминание в будущем → reminder
@@ -56,12 +61,14 @@ const routeSystem = `Классифицируй ровно одно сообще
5. Утверждает: сообщает или обновляет текущее состояние/событие → fact 5. Утверждает: сообщает или обновляет текущее состояние/событие → fact
6. Просит выполнить работу → act 6. Просит выполнить работу → act
7. Про ассистента, настройки или память → system 7. Про ассистента, настройки или память → system
8. Иначе → chat 8. Реплика — обрывок или указание на неназванное («это», «то», «потом»), и без него непонятно, что именно нужно сделать → unknown
9. Иначе → chat
Различия: Различия:
- note — сохранить информацию, без напоминания. text = суть. - note — сохранить информацию, без напоминания. text = суть.
- reminder — уведомить позже. text = что напомнить. - reminder — уведомить позже. text = что напомнить.
- fact — неявное обновление: пользователь сообщает, что что-то в мире изменилось (текущее/изменённое состояние, случившееся событие). key/value. - fact — неявное обновление: пользователь сообщает, что что-то в мире изменилось (текущее/изменённое состояние, случившееся событие). key/value.
- unknown — редкий случай. Ставь его, только если в самой реплике нет ни предмета, ни действия. Короткая, простая или незнакомая тема — это не причина для unknown: приветствие и болтовня — это chat, вопрос на любую тему — это query, просьба сделать что-то названное — это act.
- query против fact — решает форма реплики, а не тема. Вопрос о состоянии — это query, даже если названо то же самое, что бывает в fact. Только утверждение — это fact. - query против fact — решает форма реплики, а не тема. Вопрос о состоянии — это query, даже если названо то же самое, что бывает в fact. Только утверждение — это fact.
Примеры: Примеры:
@@ -76,6 +83,13 @@ const routeSystem = `Классифицируй ровно одно сообще
"напиши письмо" → {"intent":"act","verb":"написать письмо"} "напиши письмо" → {"intent":"act","verb":"написать письмо"}
"очисти память" → {"intent":"system"} "очисти память" → {"intent":"system"}
"привет" → {"intent":"chat","text":"привет"} "привет" → {"intent":"chat","text":"привет"}
"сделай это" → {"intent":"unknown"}
"ну это" → {"intent":"unknown"}
"потом" → {"intent":"unknown"}
Но не путай — здесь unknown не нужен:
"сделай кофе" → {"intent":"act","verb":"сделать кофе"}
"что такое кватернион?" → {"intent":"query","text":"что такое кватернион"}
"ага" → {"intent":"chat","text":"ага"}
Ответ — JSON-массив: по одному объекту на каждую просьбу. Обычно один. Если в реплике несколько просьб — по объекту на каждую. "напомни купить молоко, и запиши что кофе кончился" → [{"intent":"reminder","text":"купить молоко"},{"intent":"note","text":"кофе кончился"}]. Только JSON, без пояснений.` Ответ — JSON-массив: по одному объекту на каждую просьбу. Обычно один. Если в реплике несколько просьб — по объекту на каждую. "напомни купить молоко, и запиши что кофе кончился" → [{"intent":"reminder","text":"купить молоко"},{"intent":"note","text":"кофе кончился"}]. Только JSON, без пояснений.`
@@ -84,6 +98,11 @@ const routeSystem = `Классифицируй ровно одно сообще
// the loop without hurting short slot values. // the loop without hurting short slot values.
const routeRepeatPenalty = 1.15 const routeRepeatPenalty = 1.15
// routeIntentUnknown — the model's way of saying "I could not route this".
// It is a wire value only: it never becomes a router.Intent, it just makes
// Route return ok=false so the caller drops to the classifier cascade.
const routeIntentUnknown = "unknown"
type routeAction struct { type routeAction struct {
Intent string `json:"intent"` Intent string `json:"intent"`
Key string `json:"key"` Key string `json:"key"`
@@ -92,6 +111,10 @@ type routeAction struct {
Verb string `json:"verb"` Verb string `json:"verb"`
} }
// Route asks the model for one decision. The bool is false when there is no
// decision to use: either the model failed (err set) or it refused with the
// "unknown" intent (err nil). Both mean the same thing to the caller — use the
// classifier instead.
func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time) (Decision, bool, error) { func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time) (Decision, bool, error) {
raw, err := lr.c.Complete(ctx, llm.Req{ raw, err := lr.c.Complete(ctx, llm.Req{
System: routeSystem, System: routeSystem,
@@ -115,6 +138,13 @@ func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time)
// with the engine turn-on (Router.Route → []Decision, both voice.go handlers // with the engine turn-on (Router.Route → []Decision, both voice.go handlers
// loop). Until then only the first ask is honored. // loop). Until then only the first ask is honored.
a := acts[0] a := acts[0]
// The model refused. Report "no decision" without an error, which is the
// same fall-through the caller already uses for a parse failure — the
// classifier cascade gets the turn and its own confidence gate decides
// whether to ask. Better a slower second opinion than a confident guess.
if a.Intent == routeIntentUnknown {
return Decision{}, false, nil
}
d := Decision{Utterance: utterance, Stage: 1, Confidence: 1.0} d := Decision{Utterance: utterance, Stage: 1, Confidence: 1.0}
switch Intent(a.Intent) { switch Intent(a.Intent) {
case IntentFact: case IntentFact:
+58 -1
View File
@@ -100,8 +100,10 @@ func TestLLMRouterReminderMapping(t *testing.T) {
} }
} }
// An intent name that is not in the contract at all (as opposed to "unknown",
// which is a real refusal) still defaults to chat.
func TestLLMRouterChatFallback(t *testing.T) { func TestLLMRouterChatFallback(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`}) lr := NewLLMRouter(mockLLM{out: `{"intent":"banana"}`})
d, ok, err := lr.Route(context.Background(), "как дела?", time.Now()) d, ok, err := lr.Route(context.Background(), "как дела?", time.Now())
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("ok=%v err=%v", ok, err) t.Fatalf("ok=%v err=%v", ok, err)
@@ -111,6 +113,61 @@ func TestLLMRouterChatFallback(t *testing.T) {
} }
} }
// The model must be able to say "I could not route this".
func TestRouteGrammarAllowsUnknown(t *testing.T) {
if !strings.Contains(routeGrammar, `"\"unknown\""`) {
t.Fatal("grammar cannot express a refusal")
}
}
// If the prompt does not tell the model when to refuse, it never will.
func TestRoutePromptExplainsUnknown(t *testing.T) {
if !strings.Contains(routeSystem, "unknown") {
t.Fatal("prompt never mentions the unknown intent")
}
if !strings.Contains(routeSystem, `"сделай это" → {"intent":"unknown"}`) {
t.Fatal("prompt lost its worked refusal example")
}
// A refusal-only router is useless, so the prompt must also show cases that
// look ambiguous but are not.
if !strings.Contains(routeSystem, "здесь unknown не нужен") {
t.Fatal("prompt lost its counter-examples")
}
}
// A refusal is not an error. It reports "no decision" so the cascade moves on.
func TestLLMRouterUnknownRefuses(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`})
_, ok, err := lr.Route(context.Background(), "сделай это", time.Now())
if ok {
t.Fatal("a refusal must not produce a usable decision")
}
if err != nil {
t.Fatalf("a refusal is not an error, got %v", err)
}
}
// The whole point of the refusal: the turn keeps going on the classifier, the
// same way it does when the model returns garbage.
func TestRouterFallsBackWhenLLMRefuses(t *testing.T) {
c := NewClassifier(NewHashEmbedder(1024))
seedClassifier(t, c)
r := New(Config{
Classifier: c,
Extractor: Extractor{Time: StubDateTimeParser{}, Facts: DefaultFactParser{}},
Threshold: 0.4,
LLM: NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`}),
})
d, err := r.Route(context.Background(), "напомни позвонить маме", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
// Stage 1 is the LLM's own answer; the classifier lands on stage 2 or 3.
if d.Stage < 2 {
t.Fatalf("want the classifier to decide, got stage %d (%+v)", d.Stage, d)
}
}
func TestLLMRouterLLMError(t *testing.T) { func TestLLMRouterLLMError(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: "", err: fmt.Errorf("llm down")}) lr := NewLLMRouter(mockLLM{out: "", err: fmt.Errorf("llm down")})
_, ok, err := lr.Route(context.Background(), "x", time.Now()) _, ok, err := lr.Route(context.Background(), "x", time.Now())
+28 -1
View File
@@ -12,6 +12,15 @@ import (
"golang.org/x/text/unicode/norm" "golang.org/x/text/unicode/norm"
) )
// The deployed model is multilingual-e5-small. e5 was trained with these two
// words glued to the front of every text, and it scores badly without them —
// they are part of the model, not a style choice. Swapping back to a symmetric
// paraphrase model means dropping them again.
const (
queryPrefix = "query: "
passagePrefix = "passage: "
)
const ( const (
padTokenID = 1 padTokenID = 1
unkTokenID = 3 unkTokenID = 3
@@ -54,7 +63,25 @@ func NewONNXEmbedder(modelPath, tokenizerPath, libPath string) (*onnxEmbedder, e
func (e *onnxEmbedder) Dim() int { return embedDim } func (e *onnxEmbedder) Dim() int { return embedDim }
// Embed treats the text as a query. The classifier compares one short
// utterance to another short seed phrase, so both sides get the same prefix
// and the comparison stays fair. The recall path must call EmbedQuery and
// EmbedPassage instead.
func (e *onnxEmbedder) Embed(ctx context.Context, text string) ([]float32, error) { func (e *onnxEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, queryPrefix+text)
}
// EmbedQuery — the question the user just asked.
func (e *onnxEmbedder) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, queryPrefix+text)
}
// EmbedPassage — a note or fact being stored, or re-scored at lookup time.
func (e *onnxEmbedder) EmbedPassage(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, passagePrefix+text)
}
func (e *onnxEmbedder) embed(ctx context.Context, text string) ([]float32, error) {
inputIDs, attentionMask, _ := e.tokenizer.Encode(text) inputIDs, attentionMask, _ := e.tokenizer.Encode(text)
inputShape := ort.NewShape(1, int64(maxLength)) inputShape := ort.NewShape(1, int64(maxLength))
@@ -313,4 +340,4 @@ func preTokenize(text string) []string {
return out return out
} }
var _ Embedder = (*onnxEmbedder)(nil) var _ AsymmetricEmbedder = (*onnxEmbedder)(nil)