Compare commits

...

17 Commits

Author SHA1 Message Date
kami 5a1d465db5 Merge commit '4383844' into overnight-jul31
# Conflicts:
#	deploy/mavend.json
#	internal/config/config.go
2026-07-31 11:52:21 +04:00
kami 43838445ab Write up the margin gate results
Third section: why the absolute gate could not separate the two
distributions, the delta sweep, and the before/after. Marks next-steps
item 3 done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:51:21 +04:00
kami 11831c6ace Gate recall on the margin over the runner-up, not just the score
The e5 embedder puts every cosine in one narrow band (0.79-0.89), so the
absolute query_min_score gate cannot tell a real hit from a made-up
question: any value under the band answers everything, any value above it
answers nothing. False recall was 5/5.

New gate asks whether one note is clearly the best instead: top1 - top2 >
delta. New query_min_margin config knob, default 0.008, read off the sweep
in the recall harness. The absolute floor stays as a second check.

On the recall fixture with e5: answered 72% -> 68%, false recall 5/5 -> 1/5.

Vikunja #359

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:50:39 +04:00
kami 93c1a41d4a Route with the resident model by default
The two things that made this unsafe are fixed: the router can now
refuse, and slot extraction runs on its decisions.

On the held-out fixture it gets 63.2% of intents right against the
classifier's 50.0%, with no route errors. It costs about a second a
turn instead of 30ms.

The flag is a pointer now, so leaving it out of the config means on
and only writing false turns it off.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:47:57 +04:00
kami bce5ed210c Merge branch 'worktree-agent-a76ce40c73601d90d' into overnight-jul31 2026-07-31 11:44:32 +04:00
kami c31f0d1001 Extract slots for LLM router decisions too
An LLM-routed reminder came back with no parsed time and an act with no
fn, because only the classifier path ran the extractor. Now the router
runs the same extraction after an LLM decision and fills only the empty
slots. No time in the utterance still means no time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:44:06 +04:00
kami 34521c30b8 Merge branch 'worktree-agent-ad5da57e47b822152' into overnight-jul31 2026-07-31 11:39:35 +04:00
kami 1d48755d12 Record the recall numbers after the embedder swap
recall@1 60% to 72%, answered 48% to 72%, latency 3x better. But false
recall went 1/5 to 5/5: e5 packs every score into a narrow high band, so
the 0.55 gate now admits everything. Left the gate alone as instructed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:18 +04:00
kami f6d5a2a7a4 Swap the embedder to multilingual-e5-small (Vikunja #371, #372)
The old model was a symmetric paraphrase model, so it scored "do these
look alike" instead of "does this note answer this question". Also fixes
the file mismatch: the Makefile, the deploy config and both evals now all
name the same quantized file, and the quantized one is what gets measured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:18 +04:00
kami 751c2a705f Embed a question and a stored note differently (Vikunja #371)
Note recall is asymmetric: a short question goes in, a longer note comes
out. Adds EmbedQuery/EmbedPassage helpers and the e5 prefixes, and points
the note/fact write path at the passage side and the query path at the
query side. Reviewers: the three call sites in voice.go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:07 +04:00
kami 5d5b0cfd49 Merge branch 'worktree-agent-a0ab2a9b7439296e3' into overnight-jul31 2026-07-31 11:37:53 +04:00
kami bd16ca69e5 Let the LLM router answer "unknown" when it cannot route
Chose an 8th enum value over a confidence number: the model already picks
one enum token, so it costs nothing in the grammar, while a score from a
0.8B model would be uncalibrated noise. A refusal returns "no decision"
with no error, which is the fall-through the caller already uses for a
bad parse, so the classifier and its clarify gate take the turn.

Reviewers: the prompt's counter-examples matter most — a small model will
over-use any easy escape hatch. The training workspace copy of the prompt
still needs the same edit (Vikunja #362).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:35:37 +04:00
kami 0ed386eca6 Re-measure the router on a quiet box and record the numbers
The earlier before/after was taken while another eval shared
llama-server. This run had the box to itself.

Intent accuracy 61.8% llm-only, 63.2% cascade, 67.1% with thinking off.
The prompt fix holds. note→fact shows up here too, so it is real.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:34:57 +04:00
kami 1c4eab2107 Merge commit '94eb92f' into overnight-jul31
# Conflicts:
#	Makefile
2026-07-31 10:11:56 +04:00
kami 0914e0a3d5 Merge commit '4ba9a6f' into overnight-jul31
# Conflicts:
#	Makefile
2026-07-31 10:11:29 +04:00
kami 4ba9a6f422 Add a deterministic scorer for nudge phrasing (Vikunja #323)
Review internal/phraser/eval/checks.go -- it IS the measurement. Each check
names in a comment which DESIGN.md line it defends: length, feminine
self-reference (windowed around "я" so the operator's own masculine
second-person forms are not flagged), the cringe list (pet names, emoji,
"!!", fake concern, apology, emotional support, asking how he feels,
praise), on-topic, mood enum. No send/veto signal anywhere, per
DESIGN.md § "Rules decide, LLM phrases".
Fixture (158 lines) and tests (252) do not count toward the diff ceiling;
the scorer itself is still ~650. Splitting eval.go from checks.go would
give two commits neither of which measures anything.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:30:52 +04:00
kami 94eb92fb15 Label LLM eval runs with the model llama-server has loaded
The bake-off in #278/#250 needs two models' scores side by side, and the
report names only carried the config, so the rows were indistinguishable.
ModelID reads /v1/models instead of taking a string that goes stale.
New target: make eval-models MAVEN_LLM_URL=...

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:17:20 +04:00
30 changed files with 1992 additions and 106 deletions
+8 -5
View File
@@ -43,14 +43,17 @@ notes. Without it, the floor `HashEmbedder` is used — deterministic but weak
(Russian recall rarely clears the confidence gate, many commands fall to
"clarify").
**Download the embedder** (ONNX, ~90 MB):
**Download the embedder** (ONNX, ~120 MB):
```sh
make download-embedder
```
This fetches `paraphrase-multilingual-MiniLM-L12-v2` (384-dim, 12-layer,
supports 50+ languages including Russian) to `models/embedder/`.
This fetches `multilingual-e5-small` (384-dim, 12-layer, Russian and English)
to `models/embedder/multilingual-e5-small/`. It is an asymmetric retrieval
model: the code puts `query: ` in front of a question and `passage: ` in front
of a stored note, which is how e5 was trained. The quantized file is the one
that is downloaded, deployed and measured.
**Also need ONNX Runtime** (`libonnxruntime.so`):
@@ -64,8 +67,8 @@ sudo cp onnxruntime-linux-x64-1.15.1/lib/libonnxruntime.so* /usr/local/lib/
```json
"voice": {
"embedder": {
"model_path": "models/embedder/model_quantized.onnx",
"tokenizer_path": "models/embedder/tokenizer.json",
"model_path": "models/embedder/multilingual-e5-small/model_quantized.onnx",
"tokenizer_path": "models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/usr/local/lib/libonnxruntime.so"
}
}
+32 -4
View File
@@ -16,7 +16,7 @@ PIPER_BIN := $(shell pwd)/deps/piper/piper
PIPER_MODEL := $(shell pwd)/models/tts/ru_RU-irina-medium.onnx
PIPER_ESPEAK := $(shell pwd)/deps/piper/espeak-ng-data
.PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go eval-router eval-recall
.PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go eval-router eval-recall eval-phrasing eval-models
all: build
@@ -103,6 +103,30 @@ eval-router:
eval-recall:
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
# eval-phrasing -- score nudge phrasing (internal/phraser/eval). Verbose so the
# report and every generated message land in the terminal. With no environment
# it scores the deterministic Stub only, which is what CI runs. Set
# MAVEN_LLM_URL to add the resident model:
# MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
# The model run is slow (minutes) -- the timeout is raised to match.
eval-phrasing:
$(GO) test -v -count=1 -timeout 40m ./internal/phraser/eval/
# eval-models — score ONE llama-server against the same fixture, for the
# resident-model bake-off (#278, #250). Start a server with the gguf you want,
# then:
#
# make eval-models MAVEN_LLM_URL=http://127.0.0.1:18100
#
# The report names carry the model llama-server reports, so runs from two
# checkpoints stay apart. Only the LLM test runs — the classifier baselines do
# not depend on the model and take the ONNX runtime with them.
MAVEN_LLM_URL ?= http://127.0.0.1:18099
eval-models:
MAVEN_LLM_URL="$(MAVEN_LLM_URL)" $(GO) test -v -count=1 -timeout 60m \
-run TestLLMRouterBaseline ./internal/router/eval/
run-stt: build-stt
LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
./mavsttd -socket /tmp/maven/stt.sock -model $(WHISPER_MODEL)
@@ -128,9 +152,13 @@ deps-piper:
-o /tmp/piper.tar.gz
tar -xzf /tmp/piper.tar.gz -C deps/
EMBEDDER_DIR := $(shell pwd)/models/embedder
EMBEDDER_MODEL_URL := https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/onnx/model_quantized.onnx
EMBEDDER_TOKENIZER_URL := https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/tokenizer.json
# multilingual-e5-small: an asymmetric retrieval model. It is trained to match
# a short question against a longer passage, which is what note recall is.
# The quantized file is the one we download, deploy and measure — see
# RECALL-EVAL-31-07-2026.md.
EMBEDDER_DIR := $(shell pwd)/models/embedder/multilingual-e5-small
EMBEDDER_MODEL_URL := https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/onnx/model_quantized.onnx
EMBEDDER_TOKENIZER_URL := https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/tokenizer.json
download-embedder:
mkdir -p $(EMBEDDER_DIR)
+145 -2
View File
@@ -83,14 +83,157 @@ sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — bot
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
not the problem to solve.
## Re-measured after the embedder swap — 31-07-2026, later the same day
Changed: `models/embedder/` is now **multilingual-e5-small** (quantized, 118MB), with `query: ` in
front of a question and `passage: ` in front of a stored note (Vikunja #371). `deploy/mavend.json`
and `make download-embedder` now name the same file, and it is the quantized one — that is what the
column below measures (Vikunja #372). Everything else is unchanged: same fixture, same store, same
0.55 gate. The old column is the baseline and is left as it was.
| | recall+onnx, MiniLM (baseline) | recall+onnx, e5-small (new) |
|---|---|---|
| **recall@1** | 60.0% (15/25) | **72.0% (18/25)** |
| recall@3 | 80.0% (20/25) | 84.0% (21/25) |
| **answered after the 0.55 gate** | 48.0% (12/25) | **72.0% (18/25)** |
| wrong note on top / tie on top | 10 / 0 | 7 / 0 |
| ranked first, then silenced by the gate | 3 | 0 |
| **false recall** | 1/5 (20%) | **5/5 (100%)** |
| top-1 score when right, min / median | 0.559 / 0.678 | 0.791 / 0.857 |
| top-1 when it must stay silent, median / max | 0.470 / 0.567 | 0.815 / 0.835 |
| RU / EN / `hard` cases passed | 13/24 / 3/6 / 2/11 | 14/24 / 4/6 / 5/11 |
| latency p50 / p95 / max | 59ms / 148ms / 194ms | 18ms / 37ms / 49ms |
### What moved
Ranking got better and got faster. Half the previously-unwinnable `hard` cases now pass (2/11 →
5/11), the guitar note no longer beats the docker-logs note, and the gate stops silencing notes that
already ranked first. The quantized e5 is also ~3x quicker than the fp32 MiniLM it replaces.
### What got worse: the gate is now a no-op
e5 packs every cosine into a narrow high band. Right-note scores start at 0.791; must-stay-silent
scores reach 0.835. **The distributions still overlap, and now they overlap above the gate**, so
0.55 admits everything and false recall goes from 1/5 to 5/5. The sweep:
```
gate 0.500.70: answered 18/25 (72%) false recall 5/5
gate 0.80: answered 17/25 (68%) false recall 4/5
gate 0.90: answered 0/25 ( 0%) false recall 0/5
```
There is no value that keeps real recall and rejects made-up questions — same conclusion as before,
now with a wider band and no room at all. `query_min_score` was left at 0.55 as instructed. **The
recommendation is to leave it there and stop tuning it**: any number under ~0.79 is a no-op and
anything above starts cutting real recall long before it stops the false ones. The fix is a margin
gate (`top1 top2 > δ`), next-steps item 3, which is now the top item.
### The prefixes did not do the work
A control run with both prefixes set to the empty string scored the **same** recall@1 (72%), a
slightly better recall@3 (88%) and the same 5/5 false recall. So on this fixture the gain comes from
the model, not from the `query:` / `passage:` split. The prefixes are kept because they are how e5
was trained and the split is the right shape for the read path, but they are not worth defending on
this evidence — a bigger fixture may say otherwise.
### Stored vectors from the old model are now junk
Cosine between a MiniLM vector and an e5 vector means nothing. Every row already in `notes` and in
the vector memory table was written by the old model, so after this deploy they will score as noise
against a new query. A live database needs every note and fact re-embedded before recall works at
all. Filed as its own task.
## Margin gate — 31-07-2026, third run
Next-steps item 3, done. The absolute gate is replaced by a **margin gate**: answer only when the
top hit beats the runner-up by more than delta (`top1 top2 > δ`). Same fixture, same e5 embedder,
same store as the run above. `internal/memory/gate.go` holds the check; both read paths call it
(`cmd/mavend/recall.go` and the notes-RAG branch in `voice.go`). New knob `voice.query_min_margin`
in `deploy/mavend.json`, default 0.008.
### Why the absolute gate could not work, in one line of data
The harness now prints the margin distributions, and they barely overlap where the raw scores
overlap completely:
| | top-1 score | margin (top1 top2) |
|---|---|---|
| right note first (n=18) | min 0.810, median 0.862, max 0.890 | min 0.001, median 0.029, max 0.053 |
| must stay silent (n=5) | min 0.795, median 0.815, max 0.835 | min 0.000, median 0.002, **max 0.019** |
Four of the five must-be-silent cases have a margin at or under 0.002 — when there is nothing to
recall, e5 finds several notes equally close and no clear winner. That is the signal the absolute
score throws away.
### The delta sweep
Absolute gate held at 0.55 throughout.
```
delta 0.000: answered 18/25 (72%) false recall 5/5
delta 0.002: answered 17/25 (68%) false recall 3/5
delta 0.005: answered 17/25 (68%) false recall 2/5
delta 0.008: answered 17/25 (68%) false recall 1/5 <- chosen
delta 0.010: answered 15/25 (60%) false recall 1/5
delta 0.012: answered 14/25 (56%) false recall 1/5
delta 0.015: answered 12/25 (48%) false recall 1/5
delta 0.020: answered 11/25 (44%) false recall 0/5
delta 0.025: answered 9/25 (36%) false recall 0/5
delta 0.030: answered 8/25 (32%) false recall 0/5
delta 0.040: answered 4/25 (16%) false recall 0/5
delta 0.050: answered 2/25 ( 8%) false recall 0/5
delta 0.060: answered 0/25 ( 0%) false recall 0/5
```
### Chosen: δ = 0.008
It is the best point on the frontier, not a taste call. **0.008 dominates 0.010, 0.012 and 0.015
outright** — same 1/5 false recall, 8 to 20 points more real recall. Everything below it buys recall
back only by admitting more false recalls (0.005 → 2/5, 0.002 → 3/5). The next real improvement is
0.020 at 0/5 false, and it costs 24 points of recall to get there.
The brief's bar was "recall above 60% with false recall at 1/5 or better". 0.008 clears it with room:
68% and 1/5.
### Before / after
| | absolute gate 0.55 (previous) | margin gate δ=0.008 |
|---|---|---|
| recall@1 (ranking, ungated) | 72.0% (18/25) | 72.0% (18/25) — unchanged, the gate does not rank |
| **answered after the gate** | 72.0% (18/25) | **68.0% (17/25)** |
| **false recall** | **5/5 (100%)** | **1/5 (20%)** |
| fixture cases passed | 18/30 | **21/30** |
Four false recalls removed for one real answer. That is the trade the spec asks for — she is not a
guesser-of-truth. The one survivor is `en-pref-025` ("should i be offered wine"), which recalls a
filler note at 0.796 with a 0.019 margin: the widest silent-case margin in the fixture, and it sits
inside the real-recall range, so no delta removes it without taking real answers with it.
### Does the absolute cutoff still earn its keep? Marginally — kept
On this fixture with e5 it is a **no-op**: the lowest right-note score is 0.791, so 0.55 rejects
nothing the margin does not already reject. It is kept for two reasons, neither glamorous. It still
does real work for the hash embedder (its own sweep shows answers dropping from 16% to 0% between
0.30 and 0.50), and it is the only thing standing between the user and a reply built from a store
where everything is far away but one row happens to be a little less far — a near-empty database, or
the stale-vector case below. Cheap insurance, no measured cost. If a later embedder makes it bite,
the sweep is one command.
### Caveat on the numbers
Five must-be-silent cases is a thin basis for a 4-point decision. 1/5 and 2/5 differ by one case.
The shape of the frontier is trustworthy — margins separate, absolute scores do not — but δ=0.008
itself should be re-read off a bigger fixture (next-steps item 6) before anyone defends the third
decimal.
## Next steps — ordered by value-to-risk; nothing here is a decision
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
change plus a prefix in `onnxembedder.go`, re-measurable in one command.
2. **Re-run `make eval-recall`, then set the gate from the sweep** — not before. Any
`query_min_score` picked against today's embedder describes a model on its way out.
3. **Replace the absolute-score gate with a margin gate** (`top1 top2 > δ`) — as the routing eval
concluded, absolute cosine cannot see a flat distribution.
3. ~~**Replace the absolute-score gate with a margin gate**~~ — done, see the section above.
δ=0.008, false recall 5/5 → 1/5.
4. **Delete or repair the dead `memStore` branch** at `voice.go:776` — search before the gate,
gate it separately, or restrict it to facts and say so.
5. **Add a mild time decay to ranking** — the newest statement of a preference is the true one.
+35
View File
@@ -40,6 +40,41 @@ model → classifier as failure floor.
Never compare a hash-embedder run to an ONNX one.
## Re-measured after the prompt fix
The table above is the **baseline at commit `46259b4`**, kept as-is. The prompt fix (query
tested before fact, plus `repeat_penalty` and a bounded grammar string) was then measured on
an otherwise idle box — no other eval sharing llama-server, so these latencies are real
rather than contention.
| | llm-only (0.8B) | cascade+llm (0.8B) | llm-only, thinking off |
|---|---|---|---|
| **intent-only accuracy** | 48.7% → **61.8%** | 50.0% → **63.2%** | **67.1%** |
| full accuracy (intent+slots+gate) | 23.7% → **38.2%** | 32.9% → **47.4%** | **42.1%** |
| route errors | 2 → **0** | 0 → 0 | **0** |
| p50 / p95 latency | **1.08s / 1.55s** | **1.04s / 1.53s** | **0.93s / 1.41s** |
Three things this run settles:
1. **The prompt fix holds.** An earlier contended run reported 60.5% / 36.8% for llm-only;
the quiet run gives 61.8% / 38.2%. Close enough to call the gain real, and the earlier
run's 4-5s latency figures were contention, not the model.
2. **`query→fact` fell from ×15 to ×7**, and both unparseable replies are gone. Zero route
errors in every LLM configuration.
3. **`note→fact ×4` is real, not noise.** It shows up in the quiet run too. The agent that
wrote the prompt fix suspected its own change might have caused it by pulling assertive
`запиши что…` phrasings toward fact, and that suspicion stands — all five `ru-note-*`
cases now land on fact. Tracked as Vikunja #375.
**Thinking off is the best configuration measured so far**, on both accuracy and latency
(Vikunja #376). That is worth understanding before flipping: routing is a short
classification into a fixed enum with grammar-constrained output, so there is little to
reason about, and the thinking trace mostly gives a small model room to talk itself out of
the right answer. Phrasing is a different job and needs measuring separately.
Still `6 / 6` missed clarify — the router has no way to say "I don't know" (Vikunja #359).
That is unchanged by anything here.
## Findings
### 1. The resident model does route better — 50.0% vs 36.8%
+5 -8
View File
@@ -8,16 +8,13 @@ import "github.com/kami/maven/internal/memory"
// that can answer "when did I last …?" from a captured fact). A note hit here
// is redundant with the notes-RAG path — by design; the two indexes can diverge
// once the backend is swapped for a persistent/external store. ok=false when
// there's no hit above the threshold or the hit carries no text.
func bestRecall(results []memory.Result, min float64) (string, bool) {
if len(results) == 0 {
// the hit fails the confidence gate (see memory.Confident: an absolute floor
// plus a margin over the runner-up) or carries no text.
func bestRecall(results []memory.Result, minScore, minMargin float64) (string, bool) {
if !memory.Confident(results, minScore, minMargin) {
return "", false
}
top := results[0]
if top.Score < min {
return "", false
}
text := top.Meta["text"]
text := results[0].Meta["text"]
if text == "" {
return "", false
}
+17 -4
View File
@@ -8,23 +8,24 @@ import (
func TestBestRecall(t *testing.T) {
const min = 0.55
const margin = 0.008
t.Run("empty results", func(t *testing.T) {
if _, ok := bestRecall(nil, min); ok {
if _, ok := bestRecall(nil, min, margin); ok {
t.Error("empty results returned ok")
}
})
t.Run("top below threshold", func(t *testing.T) {
res := []memory.Result{{Score: 0.4, Meta: map[string]string{"text": "выпил воды"}}}
if _, ok := bestRecall(res, min); ok {
if _, ok := bestRecall(res, min, margin); ok {
t.Error("below-threshold hit returned ok")
}
})
t.Run("hit without text meta", func(t *testing.T) {
res := []memory.Result{{Score: 0.9, Meta: map[string]string{"type": "fact"}}}
if _, ok := bestRecall(res, min); ok {
if _, ok := bestRecall(res, min, margin); ok {
t.Error("textless hit returned ok")
}
})
@@ -34,7 +35,7 @@ func TestBestRecall(t *testing.T) {
{Score: 0.82, Meta: map[string]string{"text": "выпил воды в три часа", "type": "fact"}},
{Score: 0.60, Meta: map[string]string{"text": "другое"}},
}
got, ok := bestRecall(res, min)
got, ok := bestRecall(res, min, margin)
if !ok {
t.Fatal("clearing hit not returned")
}
@@ -42,4 +43,16 @@ func TestBestRecall(t *testing.T) {
t.Errorf("wrong text: %q", got)
}
})
// The runner-up is almost as close, so the embedder cannot tell the two
// notes apart. Silence beats reading back a coin flip.
t.Run("runner-up too close", func(t *testing.T) {
res := []memory.Result{
{Score: 0.860, Meta: map[string]string{"text": "выпил воды в три часа"}},
{Score: 0.858, Meta: map[string]string{"text": "другое"}},
}
if _, ok := bestRecall(res, min, margin); ok {
t.Error("thin-margin hit returned ok")
}
})
}
+24 -15
View File
@@ -206,11 +206,11 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
if threshold <= 0 {
threshold = config.DefaultRouterThreshold
}
// Both routing paths are weak on held-out utterances — the classifier gets
// 36.8% of intents right, the resident model 50.0% and much slower. Off by
// default (see config.VoiceConfig.LLMRouter); the classifier always stays
// wired as the fallback, so a model error never breaks a turn.
rtr := buildRouter(emb, matcher, threshold, pickLLMRouter(cfg.Voice.LLMRouter, llmClient))
// The resident model routes by default: 63.2% of held-out intents right
// against the classifier's 50.0%, at about 1s a turn instead of 30ms (see
// config.VoiceConfig.LLMRouter). The classifier always stays wired as the
// fallback, so a model error never breaks a turn.
rtr := buildRouter(emb, matcher, threshold, pickLLMRouter(cfg.Voice.UseLLMRouter(), llmClient))
// ----- sessions registry (shared with voicesink) -----
sessions := voice.NewSessions()
@@ -257,6 +257,7 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
clarifyStore: clarifyStore,
extractor: router.Extractor{Time: timeParser, Acts: matcher, Facts: router.DefaultFactParser{}},
queryMinScore: cfg.Voice.QueryMinScore,
queryMinMargin: cfg.Voice.QueryMinMargin,
timeParser: timeParser,
ecosystem: eco,
}
@@ -300,6 +301,9 @@ type reactiveHandler struct {
// load-bearing math (same posture as the presence thresholds). Set by
// wireVoice from VoiceConfig; default 0.55.
queryMinScore float64
// queryMinMargin — the second half of that gate: how far the top hit must
// beat the runner-up. 0 ⇒ margin off.
queryMinMargin float64
// timeParser — used as a fallback for stage-0 reminder grammar matches
// (where the extractor didn't run). Shared with the router's extractor.
@@ -559,7 +563,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
// fail the fact write). Facts aren't in the notes table, so this is the
// only recall path for them — "когда я пил воду?" reads back from here.
if h.memStore != nil {
if vec, err := h.embedder.Embed(ctx, dec.Utterance); err != nil {
if vec, err := router.EmbedPassage(ctx, h.embedder, dec.Utterance); err != nil {
log.Printf("voice: embed fact for memory: %v", err)
} else if err := h.memStore.Insert(ctx, "fact:"+dec.Slots.Key+":"+strconv.FormatInt(now.Unix(), 10), vec, map[string]string{
"source": "voice",
@@ -673,7 +677,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
// embed the note text with the same model the classifier uses, persist
// via CoreAPI (source=tap:voice). Semantic recall lives in `notes`, not
// facts — no predicate reads it (spec's two-memory split).
vec, err := h.embedder.Embed(ctx, dec.Utterance)
vec, err := router.EmbedPassage(ctx, h.embedder, dec.Utterance)
if err != nil {
log.Printf("voice: embed note: %v", err)
return "не получилось сохранить заметку."
@@ -750,7 +754,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
return fmt.Sprintf("в %s сейчас %.0f градусов, %s.", w.Location, w.Temperature, w.Condition)
}
vec, err := h.embedder.Embed(ctx, dec.Utterance)
vec, err := router.EmbedQuery(ctx, h.embedder, dec.Utterance)
if err != nil {
log.Printf("voice: embed query: %v", err)
return "не получилось найти ответ."
@@ -760,18 +764,23 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
log.Printf("voice: query notes: %v", err)
return "не получилось найти ответ."
}
// Confidence gate: below threshold, say "I don't know" rather than read
// back the least-unrelated note — a confident wrong recall is worse than
// a gap (spec's "not a guesser-of-truth"). Same instinct as the loop's
// since(key)==null → don't fire. Tuned for the ONNX embedder; the Hash
// floor scores lexically and may rarely clear it.
if len(notes) == 0 || notes[0].Score < h.queryMinScore {
// Confidence gate: below it, say "I don't know" rather than read back
// the least-unrelated note — a confident wrong recall is worse than a
// gap (spec's "not a guesser-of-truth"). Same instinct as the loop's
// since(key)==null → don't fire. Two parts: an absolute cosine floor,
// and a margin over the runner-up, which is the part that works with
// the e5 embedder's narrow score band. See memory.Confident.
noteScores := make([]float64, len(notes))
for i, n := range notes {
noteScores[i] = n.Score
}
if !memory.ConfidentScores(noteScores, h.queryMinScore, h.queryMinMargin) {
// Long-term memory recall (notes + facts) before general knowledge:
// the notes table can't answer fact questions, but the memory store
// indexes both. Only runs when notes-RAG already gave up → additive.
if h.memStore != nil {
if hits, herr := h.memStore.Search(ctx, vec, 3); herr == nil {
if text, ok := bestRecall(hits, h.queryMinScore); ok {
if text, ok := bestRecall(hits, h.queryMinScore, h.queryMinMargin); ok {
return text
}
}
+5 -3
View File
@@ -36,11 +36,13 @@
"stt": { "socket": "/run/maven/stt.sock", "lang": "ru" },
"tts": { "socket": "/run/maven/tts.sock", "lang": "ru" },
"embedder": {
"model_path": "/opt/maven/models/embedder/model.onnx",
"tokenizer_path": "/opt/maven/models/embedder/tokenizer.json",
"model_path": "/opt/maven/models/embedder/multilingual-e5-small/model_quantized.onnx",
"tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/opt/maven/lib/libonnxruntime.so"
},
"llm_router": false,
"llm_router": true,
"query_min_score": 0.55,
"query_min_margin": 0.008,
"tool_timeout": "30s",
"tools": [
{ "name": "status", "cmd": ["systemctl", "status"], "scope": "homelab", "destructive": false },
+55 -12
View File
@@ -258,18 +258,25 @@ type VoiceConfig struct {
RouterThreshold float64 `json:"router_threshold,omitempty"`
// LLMRouter — route with the resident model instead of the embedding
// classifier. Measured on the held-out fixture (ROUTING-EVAL-31-07-2026.md)
// the model gets 50.0% of intents right against the classifier's 36.8%, but
// it costs about 800ms per turn instead of 30ms.
// classifier. On by default since Vikunja #320.
//
// TODO: the default stays false until two things land.
// 1. The LLM router cannot refuse. LLMRouter.Route hardcodes
// Confidence: 1.0, so the stage-3 clarify gate never fires and an
// unclear utterance becomes a confident wrong action (Vikunja #359).
// 2. Extractor.Extract never runs on an LLM decision, so acts arrive with
// no Fn and reminders with no Time.
// Turning this on today makes routing more accurate and less safe.
LLMRouter bool `json:"llm_router,omitempty"`
// Measured on the held-out fixture (ROUTING-EVAL-31-07-2026.md): 63.2% of
// intents right against the classifier's 50.0%, and no route errors. It
// costs about 1s per turn instead of 30ms.
//
// It is safe to leave on. The model can refuse — it answers "unknown" when
// it cannot route, and the turn drops to the classifier and its clarify
// gate. Any LLM error does the same, so a turn never breaks on the model.
// Slot extraction runs on LLM decisions too, so acts get their Fn and
// reminders their Time.
//
// Set it false to go back to the classifier, e.g. on a box with no
// llama-server or when 1s a turn is too slow.
//
// It is a pointer so that "missing from the file" and "explicitly false"
// are different things: missing means on, false means off. Read it with
// UseLLMRouter(), not directly.
LLMRouter *bool `json:"llm_router,omitempty"`
// QueryMinScore — the note-recall confidence gate. Top cosine below this
// ⇒ "I don't know" instead of a guess. Tuned for the ONNX embedder (0.55);
@@ -277,6 +284,14 @@ type VoiceConfig struct {
// default if unset.
QueryMinScore float64 `json:"query_min_score,omitempty"`
// QueryMinMargin — the second half of the recall gate: the top hit must
// beat the runner-up by more than this. The absolute score above cannot do
// the job on its own, because the e5 embedder puts every cosine in one
// narrow high band, so a made-up question scores as high as a real one.
// The margin asks whether one note is clearly the best instead.
// Negative ⇒ off. 0 ⇒ the default below.
QueryMinMargin float64 `json:"query_min_margin,omitempty"`
// Persona — optional prompt prefix that tunes maven's character. Prepended
// to every LLM system prompt (nudge phrasing, note queries, general
// knowledge). Empty string ⇒ current hardcoded persona (feminine-gendered
@@ -404,7 +419,14 @@ const (
DefaultAutotuneInterval = 10 * time.Minute
DefaultRouterThreshold = 0.55
DefaultQueryMinScore = 0.55
DefaultToolTimeout = 30 * time.Second
// Read off the margin sweep in internal/memory/recalleval on the e5
// embedder: 0.008 answers 68% of real questions (down from 72%) and cuts
// false recall from 5/5 to 1/5. Every larger delta costs real recall
// without removing that last one until 0.020, which drops recall to 44%.
DefaultQueryMinMargin = 0.008
DefaultToolTimeout = 30 * time.Second
// DefaultLLMRouter — route with the resident model unless told otherwise.
DefaultLLMRouter = true
DefaultFactEnrichmentInterval = 30 * time.Second
)
@@ -486,9 +508,21 @@ func (c *Config) applyDefaults() {
if c.Voice.QueryMinScore <= 0 {
c.Voice.QueryMinScore = DefaultQueryMinScore
}
// Unset ⇒ default. Negative is how you turn the margin off on purpose,
// so it is clamped to 0 rather than replaced by the default.
switch {
case c.Voice.QueryMinMargin == 0:
c.Voice.QueryMinMargin = DefaultQueryMinMargin
case c.Voice.QueryMinMargin < 0:
c.Voice.QueryMinMargin = 0
}
if c.Voice.ToolTimeout <= 0 {
c.Voice.ToolTimeout = Duration(DefaultToolTimeout)
}
if c.Voice.LLMRouter == nil {
on := DefaultLLMRouter
c.Voice.LLMRouter = &on
}
}
// routines: default severity to care-class (1) — the safe floor: a
@@ -507,6 +541,15 @@ func (c *Config) applyDefaults() {
}
}
// UseLLMRouter reports whether to route with the resident model. Unset means
// on; only an explicit false in the config turns it off.
func (v *VoiceConfig) UseLLMRouter() bool {
if v == nil || v.LLMRouter == nil {
return DefaultLLMRouter
}
return *v.LLMRouter
}
func (c *Config) validate() error {
if c.Phraser != nil {
if c.Phraser.ModelPath == "" {
+16 -4
View File
@@ -171,14 +171,26 @@ func TestWeatherConfigNilOK(t *testing.T) {
}
}
func TestLLMRouterDefaultsOff(t *testing.T) {
func TestLLMRouterDefaultsOn(t *testing.T) {
p := writeConfig(t, `{"voice":{"enabled":true,"bind":"127.0.0.1:9100"}}`)
c, err := Load(p)
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.Voice.LLMRouter {
t.Error("voice.llm_router absent should mean false")
if !c.Voice.UseLLMRouter() {
t.Error("voice.llm_router absent should mean on")
}
}
// Missing and explicitly false must not mean the same thing.
func TestLLMRouterExplicitFalseTurnsItOff(t *testing.T) {
p := writeConfig(t, `{"voice":{"enabled":true,"bind":"127.0.0.1:9100","llm_router":false}}`)
c, err := Load(p)
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.Voice.UseLLMRouter() {
t.Error("voice.llm_router false should turn it off")
}
}
@@ -188,7 +200,7 @@ func TestLLMRouterRead(t *testing.T) {
if err != nil {
t.Fatalf("Load: %v", err)
}
if !c.Voice.LLMRouter {
if !c.Voice.UseLLMRouter() {
t.Error("voice.llm_router true was not read")
}
}
+38
View File
@@ -0,0 +1,38 @@
package memory
// Confidence gate for a recall. Two checks, both must pass before Maven says a
// note back:
//
// - minScore — an absolute cosine floor.
// - minMargin — the top hit must beat the runner-up by more than this.
//
// The margin is the one that carries the weight. The e5 embedder packs every
// score into a narrow high band (0.79-0.89 on the recall fixture), so an
// absolute floor cannot tell a real hit from a confident-looking miss: every
// value under the band admits everything, every value above it answers nothing.
// A margin asks a different question — "is this note clearly the best one, or
// is the whole shelf equally close?" — and a made-up question has no clear best.
//
// With one hit and no runner-up there is nothing to compare, so only the floor
// applies.
// ConfidentScores reports whether the top score clears both gates. scores must
// be sorted highest first. minMargin <= 0 turns the margin check off.
func ConfidentScores(scores []float64, minScore, minMargin float64) bool {
if len(scores) == 0 || scores[0] < minScore {
return false
}
if minMargin > 0 && len(scores) > 1 && scores[0]-scores[1] <= minMargin {
return false
}
return true
}
// Confident is ConfidentScores for search results.
func Confident(results []Result, minScore, minMargin float64) bool {
scores := make([]float64, len(results))
for i, r := range results {
scores[i] = r.Score
}
return ConfidentScores(scores, minScore, minMargin)
}
+45
View File
@@ -0,0 +1,45 @@
package memory
import "testing"
func TestConfidentScores(t *testing.T) {
cases := []struct {
name string
scores []float64
minScore float64
minMargin float64
want bool
}{
{"no hits", nil, 0.55, 0.008, false},
{"below the floor", []float64{0.40, 0.10}, 0.55, 0.008, false},
{"clear winner", []float64{0.86, 0.70}, 0.55, 0.008, true},
{"runner-up too close", []float64{0.860, 0.858}, 0.55, 0.008, false},
// The rule is "beats the runner-up by MORE than delta". Not testing an
// exactly-equal margin: no pair of these decimals subtracts to exactly
// 0.008 in binary float, so such a test would pin rounding, not the rule.
{"margin just under delta", []float64{0.8079, 0.8}, 0.55, 0.008, false},
{"margin just over delta", []float64{0.8081, 0.8}, 0.55, 0.008, true},
// One hit: nothing to compare against, so only the floor applies.
{"single hit clears", []float64{0.86}, 0.55, 0.008, true},
{"single hit below floor", []float64{0.10}, 0.55, 0.008, false},
// Margin off — the old absolute-only behaviour.
{"margin off admits a tie", []float64{0.86, 0.86}, 0.55, 0, true},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
if got := ConfidentScores(c.scores, c.minScore, c.minMargin); got != c.want {
t.Errorf("got %v, want %v", got, c.want)
}
})
}
}
func TestConfidentReadsResultScores(t *testing.T) {
res := []Result{{ID: "a", Score: 0.86}, {ID: "b", Score: 0.858}}
if Confident(res, 0.55, 0.008) {
t.Error("thin margin passed the gate")
}
if !Confident(res, 0.55, 0) {
t.Error("margin off should fall back to the floor alone")
}
}
+69 -24
View File
@@ -117,18 +117,40 @@ type cachingEmbedder struct {
seen map[string][]float32
}
var _ router.AsymmetricEmbedder = (*cachingEmbedder)(nil)
func (c *cachingEmbedder) Dim() int { return c.inner.Dim() }
func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner
func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
if v, ok := c.seen[text]; ok {
return c.cached(ctx, "embed:"+text, func() ([]float32, error) {
return c.inner.Embed(ctx, text)
})
}
// The two sides of an asymmetric embedder give different vectors for the same
// string, so the cache key has to say which side asked.
func (c *cachingEmbedder) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
return c.cached(ctx, "query:"+text, func() ([]float32, error) {
return router.EmbedQuery(ctx, c.inner, text)
})
}
func (c *cachingEmbedder) EmbedPassage(ctx context.Context, text string) ([]float32, error) {
return c.cached(ctx, "passage:"+text, func() ([]float32, error) {
return router.EmbedPassage(ctx, c.inner, text)
})
}
func (c *cachingEmbedder) cached(_ context.Context, key string, embed func() ([]float32, error)) ([]float32, error) {
if v, ok := c.seen[key]; ok {
return v, nil
}
v, err := c.inner.Embed(ctx, text)
v, err := embed()
if err != nil {
return nil, err
}
c.seen[text] = v
c.seen[key] = v
return v, nil
}
@@ -155,6 +177,8 @@ type Outcome struct {
Tied bool
TopID string
TopScor float64
// Margin — top1 top2. 0 when fewer than two hits came back.
Margin float64
Reasons []string
}
@@ -162,8 +186,10 @@ type Outcome struct {
// ranks first but is silenced by query_min_score is a threshold problem, and a
// note that never ranks first is an embedder problem. Those are different fixes.
type Report struct {
Name string
MinScore float64
Name string
MinScore float64
// MinMargin — how far the top hit must beat the runner-up. 0 ⇒ off.
MinMargin float64
Total int
Answerable int
Rank1 int
@@ -190,9 +216,14 @@ type Report struct {
// where the right note ranked first, and for the no-answer cases. The gap
// between these two distributions is what a defensible query_min_score
// would have to sit inside; if they overlap, no threshold separates them.
CorrectTop []float64
NoAnswerTop []float64
P50, P95, Max time.Duration
CorrectTop []float64
NoAnswerTop []float64
// CorrectMargin / NoAnswerMargin — the same two groups, but top1 top2
// instead of top1. This is the pair the margin gate has to separate, and
// unlike the absolute scores it is what the sweep reads.
CorrectMargin []float64
NoAnswerMargin []float64
P50, P95, Max time.Duration
}
// TagStat — passed/total for one slice of the fixture.
@@ -223,13 +254,14 @@ func ratio(n, d int) float64 {
// the run on an embed or search error: an erroring case scores as a miss and is
// counted in Errors, because "the embedder was down" and "the embedder was
// wrong" are different numbers.
func Score(ctx context.Context, name string, emb router.Embedder, newStore NewStore, minScore float64, f Fixture) (Report, error) {
func Score(ctx context.Context, name string, emb router.Embedder, newStore NewStore, minScore, minMargin float64, f Fixture) (Report, error) {
rep := Report{
Name: name,
MinScore: minScore,
Total: len(f.Cases),
ByTag: map[string]TagStat{},
ByLang: map[string]TagStat{},
Name: name,
MinScore: minScore,
MinMargin: minMargin,
Total: len(f.Cases),
ByTag: map[string]TagStat{},
ByLang: map[string]TagStat{},
}
lat := make([]time.Duration, 0, len(f.Cases))
@@ -239,7 +271,7 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
} else {
rep.NoAnswer++
}
o, err := scoreCase(ctx, emb, newStore, minScore, c, f.Filler)
o, err := scoreCase(ctx, emb, newStore, minScore, minMargin, c, f.Filler)
if err != nil {
return Report{}, err
}
@@ -263,6 +295,11 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
} else if !o.Rank1 {
rep.WrongTop++
}
if o.Rank1 {
// Margins are collected on rank, not on the gate, so the
// distribution does not move as the sweep changes the gate.
rep.CorrectMargin = append(rep.CorrectMargin, o.Margin)
}
if o.Rank1 && o.Recalled != "" {
rep.CorrectTop = append(rep.CorrectTop, o.TopScor)
}
@@ -271,6 +308,7 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
rep.FalseRecall++
}
rep.NoAnswerTop = append(rep.NoAnswerTop, o.TopScor)
rep.NoAnswerMargin = append(rep.NoAnswerMargin, o.Margin)
}
if o.Pass {
@@ -285,6 +323,8 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
sort.Float64s(rep.CorrectTop)
sort.Float64s(rep.NoAnswerTop)
sort.Float64s(rep.CorrectMargin)
sort.Float64s(rep.NoAnswerMargin)
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
if len(lat) > 0 {
@@ -296,7 +336,7 @@ func Score(ctx context.Context, name string, emb router.Embedder, newStore NewSt
// scoreCase inserts the case's notes into a fresh store, then runs the read
// path the daemon runs. The returned error is fatal (the harness is broken);
// an embedder or store failure on the query lands in Outcome.Err instead.
func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minScore float64, c Case, filler []StoredNote) (Outcome, error) {
func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minScore, minMargin float64, c Case, filler []StoredNote) (Outcome, error) {
st, release, err := newStore()
if err != nil {
return Outcome{}, fmt.Errorf("%s: new store: %w", c.ID, err)
@@ -305,7 +345,7 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
all := append(append([]StoredNote(nil), c.Notes...), filler...)
for _, n := range all {
vec, err := emb.Embed(ctx, n.Text)
vec, err := router.EmbedPassage(ctx, emb, n.Text)
if err != nil {
return Outcome{}, fmt.Errorf("%s: embed note %s: %w", c.ID, n.ID, err)
}
@@ -317,7 +357,7 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
o := Outcome{Case: c}
start := time.Now()
qvec, err := emb.Embed(ctx, c.Query)
qvec, err := router.EmbedQuery(ctx, emb, c.Query)
if err != nil {
o.Latency = time.Since(start)
o.Err = err
@@ -335,7 +375,10 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
if len(hits) > 0 {
o.TopID, o.TopScor = hits[0].ID, hits[0].Score
o.Recalled = bestRecall(hits, minScore)
if len(hits) > 1 {
o.Margin = hits[0].Score - hits[1].Score
}
o.Recalled = bestRecall(hits, minScore, minMargin)
}
for i, h := range hits {
if h.ID != c.Want {
@@ -356,14 +399,14 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
switch {
case !c.Answerable():
if o.Recalled != "" {
o.Reasons = append(o.Reasons, fmt.Sprintf("false recall: %q at %.3f, want silence", o.TopID, o.TopScor))
o.Reasons = append(o.Reasons, fmt.Sprintf("false recall: %q at %.3f (margin %.3f), want silence", o.TopID, o.TopScor, o.Margin))
}
case o.Tied:
o.Reasons = append(o.Reasons, fmt.Sprintf("tie at %.3f — the right note is on top only by sort order", o.TopScor))
case !o.Rank1:
o.Reasons = append(o.Reasons, fmt.Sprintf("top hit %q (%.3f), want %q%s", o.TopID, o.TopScor, c.Want, rankNote(o.Rank3)))
case o.Recalled == "":
o.Reasons = append(o.Reasons, fmt.Sprintf("right note ranked first but scored %.3f < gate %.2f — daemon says \"не знаю\"", o.TopScor, minScore))
o.Reasons = append(o.Reasons, fmt.Sprintf("right note ranked first at %.3f (margin %.3f) but the gate silenced it — daemon says \"не знаю\"", o.TopScor, o.Margin))
}
o.Pass = len(o.Reasons) == 0
return o, nil
@@ -379,8 +422,8 @@ func rankNote(inTop3 bool) string {
// bestRecall mirrors cmd/mavend/recall.go — the gate the daemon actually
// applies to a memory hit. Duplicated rather than imported because package main
// is not importable; recalleval_test.go asserts the two agree in behaviour.
func bestRecall(results []memory.Result, min float64) string {
if len(results) == 0 || results[0].Score < min {
func bestRecall(results []memory.Result, minScore, minMargin float64) string {
if !memory.Confident(results, minScore, minMargin) {
return ""
}
return results[0].Meta["text"]
@@ -415,7 +458,7 @@ func percentile(sorted []time.Duration, p float64) time.Duration {
// the slices that name where the path is weak.
func (r Report) String() string {
var b strings.Builder
fmt.Fprintf(&b, "%s: %d/%d cases pass (gate %.2f)\n", r.Name, r.Passed, r.Total, r.MinScore)
fmt.Fprintf(&b, "%s: %d/%d cases pass (gate %.2f, margin %.3f)\n", r.Name, r.Passed, r.Total, r.MinScore, r.MinMargin)
fmt.Fprintf(&b, " recall@1 %.1f%% (%d/%d) recall@3 %.1f%% (%d/%d) answered after gate %.1f%% (%d/%d)\n",
100*r.Recall1(), r.Rank1, r.Answerable,
100*r.Recall3(), r.Rank3, r.Answerable,
@@ -426,6 +469,8 @@ func (r Report) String() string {
100*r.FalseRecallRate(), r.FalseRecall, r.NoAnswer)
fmt.Fprintf(&b, " top-1 score, right note first: %s\n", spread(r.CorrectTop))
fmt.Fprintf(&b, " top-1 score, must be silent: %s\n", spread(r.NoAnswerTop))
fmt.Fprintf(&b, " margin top1-top2, right note first: %s\n", spread(r.CorrectMargin))
fmt.Fprintf(&b, " margin top1-top2, must be silent: %s\n", spread(r.NoAnswerMargin))
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
fmt.Fprintf(&b, " by lang: %s\n", renderStats(r.ByLang))
fmt.Fprintf(&b, " by tag: %s\n", renderStats(r.ByTag))
+52 -13
View File
@@ -142,21 +142,40 @@ func words(s string) []string {
// cmd/mavend/recall.go (package main is not importable). This pins the copy to
// the original's three rules: no hits, below the gate, or no text ⇒ silence.
func TestBestRecallMatchesDaemon(t *testing.T) {
if got := bestRecall(nil, 0.55); got != "" {
if got := bestRecall(nil, 0.55, 0); got != "" {
t.Errorf("no hits: got %q, want silence", got)
}
low := []memory.Result{{ID: "a", Score: 0.4, Meta: map[string]string{"text": "чай"}}}
if got := bestRecall(low, 0.55); got != "" {
if got := bestRecall(low, 0.55, 0); got != "" {
t.Errorf("below gate: got %q, want silence", got)
}
noText := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{}}}
if got := bestRecall(noText, 0.55); got != "" {
if got := bestRecall(noText, 0.55, 0); got != "" {
t.Errorf("no text: got %q, want silence", got)
}
ok := []memory.Result{{ID: "a", Score: 0.9, Meta: map[string]string{"text": "чай"}}}
if got := bestRecall(ok, 0.55); got != "чай" {
if got := bestRecall(ok, 0.55, 0); got != "чай" {
t.Errorf("above gate: got %q, want %q", got, "чай")
}
// Margin: a close runner-up means the embedder cannot tell the two apart,
// so Maven stays silent even though both clear the absolute floor.
close := []memory.Result{
{ID: "a", Score: 0.86, Meta: map[string]string{"text": "чай"}},
{ID: "b", Score: 0.85, Meta: map[string]string{"text": "кофе"}},
}
if got := bestRecall(close, 0.55, 0.03); got != "" {
t.Errorf("thin margin: got %q, want silence", got)
}
if got := bestRecall(close, 0.55, 0); got != "чай" {
t.Errorf("margin off: got %q, want %q", got, "чай")
}
clear := []memory.Result{
{ID: "a", Score: 0.86, Meta: map[string]string{"text": "чай"}},
{ID: "b", Score: 0.70, Meta: map[string]string{"text": "кофе"}},
}
if got := bestRecall(clear, 0.55, 0.03); got != "чай" {
t.Errorf("wide margin: got %q, want %q", got, "чай")
}
}
// TestHashRecallBaseline — the CI ratchet. HashEmbedder, so it needs no model
@@ -171,7 +190,7 @@ func TestHashRecallBaseline(t *testing.T) {
t.Fatalf("Load: %v", err)
}
rep, err := Score(context.Background(), "recall+hash", router.NewHashEmbedder(hashDim), InMemory,
config.DefaultQueryMinScore, f)
config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
@@ -201,11 +220,11 @@ func TestPersistentStoreScoresTheSame(t *testing.T) {
t.Fatalf("Load: %v", err)
}
emb := router.NewHashEmbedder(hashDim)
inMem, err := Score(context.Background(), "recall+hash+memory", emb, InMemory, config.DefaultQueryMinScore, f)
inMem, err := Score(context.Background(), "recall+hash+memory", emb, InMemory, config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil {
t.Fatalf("Score in-memory: %v", err)
}
persistent, err := Score(context.Background(), "recall+hash+sqlite", emb, sqliteStores(t), config.DefaultQueryMinScore, f)
persistent, err := Score(context.Background(), "recall+hash+sqlite", emb, sqliteStores(t), config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil {
t.Fatalf("Score sqlite: %v", err)
}
@@ -246,8 +265,8 @@ func TestONNXRecall(t *testing.T) {
if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
}
model := filepath.Join("../../..", "models/embedder/model.onnx")
tok := filepath.Join("../../..", "models/embedder/tokenizer.json")
model := filepath.Join("../../..", "models/embedder/multilingual-e5-small/model_quantized.onnx")
tok := filepath.Join("../../..", "models/embedder/multilingual-e5-small/tokenizer.json")
for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err)
@@ -263,14 +282,16 @@ func TestONNXRecall(t *testing.T) {
if err != nil {
t.Fatalf("Load: %v", err)
}
rep, err := Score(context.Background(), "recall+onnx", emb, InMemory, config.DefaultQueryMinScore, f)
rep, err := Score(context.Background(), "recall+onnx", emb, InMemory, config.DefaultQueryMinScore, config.DefaultQueryMinMargin, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String() + rep.Failures())
// Cached for the sweep only: the headline run above must pay the real
// Cached for the sweeps only: the headline run above must pay the real
// embedder cost so its latency numbers mean something.
t.Log("\ngate sweep:\n" + sweep(t, Cache(emb), f))
cached := Cache(emb)
t.Log("\ngate sweep (margin off):\n" + sweep(t, cached, f))
t.Log("\nmargin sweep (gate 0.55):\n" + marginSweep(t, cached, f))
}
// sweep scores the fixture at a range of gates and renders one line each. Two
@@ -281,7 +302,7 @@ func sweep(t *testing.T, emb router.Embedder, f Fixture) string {
t.Helper()
var b strings.Builder
for _, gate := range []float64{0.0, 0.30, 0.40, 0.50, 0.55, 0.60, 0.70, 0.80, 0.90} {
rep, err := Score(context.Background(), "sweep", emb, InMemory, gate, f)
rep, err := Score(context.Background(), "sweep", emb, InMemory, gate, 0, f)
if err != nil {
t.Fatalf("sweep at %.2f: %v", gate, err)
}
@@ -290,3 +311,21 @@ func sweep(t *testing.T, emb router.Embedder, f Fixture) string {
}
return b.String()
}
// marginSweep is the same idea for the margin gate (top1 top2 > delta), with
// the absolute gate held at its default. The absolute score cannot separate a
// real hit from a made-up question under e5 — every score lands in one narrow
// band — so this sweep is the one that picks a number.
func marginSweep(t *testing.T, emb router.Embedder, f Fixture) string {
t.Helper()
var b strings.Builder
for _, d := range []float64{0, 0.002, 0.005, 0.008, 0.01, 0.012, 0.015, 0.02, 0.025, 0.03, 0.04, 0.05, 0.06} {
rep, err := Score(context.Background(), "margin sweep", emb, InMemory, config.DefaultQueryMinScore, d, f)
if err != nil {
t.Fatalf("margin sweep at %.3f: %v", d, err)
}
fmt.Fprintf(&b, " delta %.3f: answered %d/%d (%.0f%%) false recall %d/%d\n",
d, rep.Rank1-rep.Gated, rep.Answerable, 100*rep.Answered(), rep.FalseRecall, rep.NoAnswer)
}
return b.String()
}
+287
View File
@@ -0,0 +1,287 @@
package eval
import (
"fmt"
"regexp"
"strings"
"unicode"
"unicode/utf8"
)
// The check names, in report order. Every check is a string or length test — no
// model grades another model here.
const (
CheckMood = "mood" // mood is in the documented enum
CheckLang = "lang" // the operator's language, not the prompt's
CheckLength = "length" // a nudge is one sentence, not a paragraph
CheckFeminine = "feminine" // her self-reference is feminine (hard constraint)
CheckCringe = "cringe" // DESIGN.md § Non-goals, "not a relationship"
CheckOnTopic = "ontopic" // says the thing the rule is about
)
// CheckNames — report order.
var CheckNames = []string{CheckMood, CheckLang, CheckLength, CheckFeminine, CheckCringe, CheckOnTopic}
// Result — one check on one message.
type Result struct {
Name string
Pass bool
Detail string
}
// Moods — the fixed enum from the LLM output contract. Not extended here; the
// contract lives in the daemon and the harness only reads it.
var Moods = map[string]bool{
"neutral": true, "happy": true, "thinking": true, "tired": true, "confused": true,
}
// Length ceilings. Justification: the nudge is spoken by piper at roughly 14
// characters per second, so 120 characters is about 8 seconds of speech. The
// operator has AuDHD — past one short sentence a nudge stops being a nudge and
// becomes something to tune out, which is exactly the "not a nag" failure. The
// word ceiling catches the same thing for languages that pack more per byte.
const (
MaxChars = 120
MaxWords = 16
)
// RunChecks scores one message. Order matches CheckNames.
func RunChecks(c Case, body, mood string) []Result {
return []Result{
checkMood(mood),
checkLang(body),
checkLength(body),
checkFeminine(body),
checkCringe(body),
checkOnTopic(c, body),
}
}
func checkMood(mood string) Result {
if Moods[mood] {
return Result{CheckMood, true, ""}
}
return Result{CheckMood, false, fmt.Sprintf("mood %q not in the enum", mood)}
}
// checkLang — the operator is Russian-speaking and the nudge is spoken aloud by
// a Russian piper voice. An English nudge is not a tone problem, it is an
// unusable one.
func checkLang(body string) Result {
cyr, lat := 0, 0
for _, r := range body {
switch {
case unicode.Is(unicode.Cyrillic, r):
cyr++
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z':
lat++
}
}
if cyr > lat {
return Result{CheckLang, true, ""}
}
return Result{CheckLang, false, fmt.Sprintf("not Russian (%d cyrillic vs %d latin letters)", cyr, lat)}
}
func checkLength(body string) Result {
chars := utf8.RuneCountInString(body)
words := len(strings.Fields(body))
if chars <= MaxChars && words <= MaxWords {
return Result{CheckLength, true, ""}
}
return Result{CheckLength, false,
fmt.Sprintf("%d chars / %d words, ceiling %d / %d", chars, words, MaxChars, MaxWords)}
}
// --- feminine self-reference ---------------------------------------------
//
// The hard constraint (CLAUDE.md, DESIGN.md § Identity): Maven's Russian
// self-reference is feminine. The operator is male, so second-person forms
// addressed to him are MASCULINE and must not be flagged — "ты не пил воду" is
// correct, "я напомнил" is not. Both directions matter, which is why this is a
// windowed scan around "я" and not a bare search for masculine endings.
var wordRE = regexp.MustCompile(`[\p{Cyrillic}]+|[,.;:!?…—-]`)
// secondPerson — pronouns that end the self-reference window. Everything after
// one of these is about him, not about her.
var secondPerson = map[string]bool{
"ты": true, "тебе": true, "тебя": true, "тобой": true,
"вы": true, "вам": true, "вас": true,
"он": true, "она": true, "оно": true, "они": true,
}
// masculinePredicative — short adjectives with no verb ending to key off.
var masculinePredicative = map[string]bool{
"должен": true, "готов": true, "рад": true, "уверен": true,
"обязан": true, "сам": true, "занят": true, "прав": true,
}
// nounsEndingInL — the false positives of "ends in л ⇒ masculine past tense".
// Small on purpose: it only has to cover nouns a nudge might actually use.
var nounsEndingInL = map[string]bool{
"стол": true, "стул": true, "пол": true, "зал": true, "гол": true,
"узел": true, "отдел": true, "файл": true, "канал": true, "угол": true,
"футбол": true, "вокзал": true, "металл": true, "интервал": true,
"уровень": true, "мускул": true, "апрель": true, "июль": true, "рубль": true,
}
// masculinePast reports whether a word looks like a masculine past-tense verb.
// Russian past tense is gendered by suffix: -л (m), -ла (f). A 0.8B with weak
// Russian defaults to the masculine form, which is the exact drift being
// measured.
func masculinePast(w string) bool {
if len([]rune(w)) < 3 || nounsEndingInL[w] {
return false
}
return strings.HasSuffix(w, "л") || strings.HasSuffix(w, "лся")
}
func checkFeminine(body string) Result {
words := wordRE.FindAllString(strings.ToLower(body), -1)
for i, w := range words {
if w != "я" {
continue
}
// Scan the next few words. Stop at punctuation or at a second-person
// pronoun: past that point the sentence is about him and masculine is
// correct.
for j := i + 1; j < len(words) && j <= i+3; j++ {
nw := words[j]
if len(nw) == 1 && !unicode.Is(unicode.Cyrillic, []rune(nw)[0]) {
break
}
if secondPerson[nw] {
break
}
if masculinePast(nw) || masculinePredicative[nw] {
return Result{CheckFeminine, false,
fmt.Sprintf("masculine self-reference %q after \"я\"", nw)}
}
}
}
// Second pass: self-reference with the pronoun dropped — "напомнил тебе",
// "проверил за тебя". A masculine past-tense verb whose object is HIM can
// only be her speaking about herself.
for i, w := range words {
if !masculinePast(w) || i+1 >= len(words) {
continue
}
next := words[i+1]
if next == "тебе" || next == "тебя" || next == "за" {
return Result{CheckFeminine, false,
fmt.Sprintf("masculine self-reference %q before %q", w, next)}
}
}
return Result{CheckFeminine, true, ""}
}
// --- the cringe checks ---------------------------------------------------
//
// "Think Jarvis without the cringe part". DESIGN.md § Non-goals: "Not a
// relationship — mom-tone is a function that makes nudges land, not emotional
// company. Names the drift a warm small model falls into." Each pattern below
// is one shape of that drift. They are deliberately specific: a check that
// flags any warmth at all would make the nudges robotic, which is the other
// failure.
type cringePattern struct {
// what the pattern is defending against, shown in the failure detail.
why string
pat *regexp.Regexp
}
var cringePatterns = []cringePattern{
{
// Endearments. "Not a relationship" — a pet name reframes a nudge as
// intimacy, and the operator asked for mother-like, not girlfriend-like.
why: "pet name / endearment",
// No \b around the Russian alternatives: Go's RE2 \b is ASCII-only and
// never matches at a Cyrillic boundary, so anchoring them would make
// this check silently always pass.
pat: regexp.MustCompile(`(?i)(милый|дорогой|солнышко|солнце моё|солнце мое|зайчик|котик|сладкий|малыш|дружок|родной|любимый|\bhoney\b|\bsweetie\b|\bdarling\b|\bbuddy\b)`),
},
{
// Emoji. The nudge is spoken aloud; an emoji is either silence or a TTS
// artefact. Also the single loudest cringe signal in a small model.
why: "emoji",
pat: nil, // handled by hasEmoji, ranges don't fit a regexp cleanly
},
{
// Exclamation pileup. One "!" is emphasis; two is a cheerleader.
why: "more than one exclamation mark",
pat: regexp.MustCompile(`!.*!|!!`),
},
{
// Fake concern. She has no feelings to report, and reporting them makes
// the nudge about her instead of about the water.
why: "fake concern opener",
pat: regexp.MustCompile(`(?i)(я волну|я беспоко|беспокоюсь|переживаю|я забочусь|я тревож|мне тревожно|i'?m worried)`),
},
{
// Apologising. The rule decided she speaks. Apologising for a greenlit
// nudge undermines the one thing that makes nudges land.
why: "apology",
pat: regexp.MustCompile(`(?i)(извини|прости|сожалею|прошу прощения|не хочу мешать|не хочу отвлекать|sorry|apolog)`),
},
{
// Offering emotional support. The explicit "not emotional company" line.
why: "offer of emotional support",
pat: regexp.MustCompile(`(?i)(я рядом|я здесь для теб|ты не один|всё будет хорошо|все будет хорошо|не переживай|я с тобой|обнимаю|я поддерж|держись)`),
},
{
// Asking how he feels. Turns a one-way nudge into a conversation he now
// owes an answer to — the most reliable way to make him mute it.
why: "asking how he feels",
pat: regexp.MustCompile(`(?i)(как ты\s*[?.!]|как ты себя|как самочувств|как настроение|всё ли в порядке|все ли в порядке|ты в порядке|how are you)`),
},
{
// Praise for compliance. Rewards make the nudge a training exercise;
// "not a relationship" again, from the other side.
why: "praise / reward framing",
pat: regexp.MustCompile(`(?i)(молодец|умница|ты справ|гордюсь|горжусь|отличная работа|так держать|good job|proud of you)`),
},
}
// hasEmoji — the pictographic ranges plus the variation selector. Cyrillic and
// ordinary punctuation are far below all of these.
func hasEmoji(s string) bool {
for _, r := range s {
switch {
case r >= 0x1F000 && r <= 0x1FAFF,
r >= 0x2600 && r <= 0x27BF,
r >= 0x2B00 && r <= 0x2BFF,
r == 0xFE0F, r == 0x203C, r == 0x2049:
return true
}
}
return false
}
func checkCringe(body string) Result {
for _, c := range cringePatterns {
if c.pat == nil {
if hasEmoji(body) {
return Result{CheckCringe, false, c.why}
}
continue
}
if m := c.pat.FindString(body); m != "" {
return Result{CheckCringe, false, fmt.Sprintf("%s (%q)", c.why, m)}
}
}
return Result{CheckCringe, true, ""}
}
// checkOnTopic — the message must name the thing the rule is about. A nudge
// that never mentions water leaves the operator with a chime and no action.
func checkOnTopic(c Case, body string) Result {
low := strings.ToLower(body)
for _, want := range c.WantAny {
if strings.Contains(low, strings.ToLower(want)) {
return Result{CheckOnTopic, true, ""}
}
}
return Result{CheckOnTopic, false,
fmt.Sprintf("mentions none of %v", c.WantAny)}
}
+334
View File
@@ -0,0 +1,334 @@
// Package eval scores nudge phrasing — the sentences the operator actually
// hears. It is the phrasing counterpart to internal/router/eval.
//
// Why a separate package from phraser: the fixture must be scorable by BOTH
// phrasing paths (the deterministic Stub and the resident model) from outside
// the phraser package, and a _test.go file inside phraser cannot be imported.
// So the fixture is embedded here and the scorer takes a Nudger interface.
//
// Why deterministic checks and not model judgement: the resident model is a
// 0.8B. It cannot grade its own tone. Every check in checks.go is a string or
// length test that a human can read and disagree with. A score here is a claim
// about measurable properties, not about whether a sentence is good.
//
// DESIGN.md § "Rules decide, LLM phrases" is why there is no send/veto signal
// anywhere in this package: the rule already decided she speaks. The phraser
// only words it, so a nudge the model refuses to write is a failure, never a
// legitimate outcome.
package eval
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/kami/maven/internal/delivery"
"github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/store"
)
//go:embed nudges_v1.json
var fixtureJSON []byte
// SchemaVersion — the version this package understands.
const SchemaVersion = 1
// Case — one nudge situation, as a real tick would present it. The fields are
// the (rule, severity, context) input DESIGN.md names, flattened to JSON.
//
// WantAny is the on-topic contract: at least one of these lowercased fragments
// must appear in the message. A water nudge that never mentions water is a
// failure however charming it reads. Fragments are stems ("вод") so declension
// does not defeat the check, and they list both languages because the Stub is
// still English (see the writeup).
type Case struct {
ID string `json:"id"`
Rule string `json:"rule"`
Severity int `json:"severity"`
// SinceMinutes — age of the rule's own fact. 0 means "no such fact", which
// is the branch where the phraser has no duration to name.
SinceMinutes int `json:"since_minutes"`
// FactKey/FactValue/FactSource — the aggregate fact behind the ops rules.
// service_down phrasing reads the key for the service name.
FactKey string `json:"fact_key,omitempty"`
FactValue string `json:"fact_value,omitempty"`
FactSource string `json:"fact_source,omitempty"`
// QuietHours/CalendarBusy — the bad moments. The gate already let this
// nudge through (ops outranks quiet hours), so the phrasing still has to be
// short and plain rather than apologetic about the timing.
QuietHours bool `json:"quiet_hours,omitempty"`
CalendarBusy bool `json:"calendar_busy,omitempty"`
WantAny []string `json:"want_any"`
Tags []string `json:"tags,omitempty"`
Note string `json:"note,omitempty"`
}
// Fixture — the versioned envelope. SchemaVersion gates the loader so an older
// binary refuses a fixture it would misread instead of reporting a wrong score.
type Fixture struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
ReferenceNow string `json:"reference_now"`
Notes []string `json:"notes"`
Cases []Case `json:"cases"`
}
// Load returns the embedded fixture.
func Load() (Fixture, error) {
var f Fixture
if err := json.Unmarshal(fixtureJSON, &f); err != nil {
return Fixture{}, fmt.Errorf("parse fixture: %w", err)
}
if f.SchemaVersion != SchemaVersion {
return Fixture{}, fmt.Errorf("fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
}
if len(f.Cases) == 0 {
return Fixture{}, fmt.Errorf("fixture has no cases")
}
return f, nil
}
// Now — the fixture's reference clock, so fact ages are reproducible.
func (f Fixture) Now() (time.Time, error) {
t, err := time.Parse(time.RFC3339, f.ReferenceNow)
if err != nil {
return time.Time{}, fmt.Errorf("parse reference_now %q: %w", f.ReferenceNow, err)
}
return t, nil
}
// Candidate rebuilds the loop.Candidate a tick would hand the phraser.
func (c Case) Candidate(now time.Time) loop.Candidate {
state := loop.State{
Now: now,
Facts: map[string]store.Fact{},
QuietHours: c.QuietHours,
CalendarBusy: c.CalendarBusy,
}
if c.SinceMinutes > 0 {
state.Facts[c.Rule] = store.Fact{
Key: c.Rule,
Ts: now.Add(-time.Duration(c.SinceMinutes) * time.Minute),
}
}
if c.FactKey != "" {
state.Facts[c.Rule] = store.Fact{
Key: c.FactKey,
Value: c.FactValue,
Source: c.FactSource,
Ts: now.Add(-time.Duration(c.SinceMinutes) * time.Minute),
}
}
sev := loop.Severity(c.Severity)
return loop.Candidate{
Rule: loop.Rule{Name: c.Rule, Severity: sev},
Severity: sev,
State: state,
}
}
// Nudger — the one thing a phrasing path must do to be scorable. Both
// *phraser.Stub and *phraser.LLMPhraser satisfy it.
type Nudger interface {
PhraseNudge(ctx context.Context, c loop.Candidate) (delivery.PhrasedNudge, error)
}
// Outcome — one scored case. Failed lists the check names that did not pass,
// Reasons the human-readable detail. Failed is empty exactly when Pass is true.
type Outcome struct {
Case Case
Body string
Mood string
Err error
Latency time.Duration
Pass bool
Failed []string
Reasons []string
}
// Report — the aggregate. ByCheck is the useful part: one composite percentage
// hides which property broke, and tuning a prompt needs to know.
type Report struct {
Name string
Total int
Passed int
Errors int
ByCheck map[string]int
ByRule map[string]TagStat
Outcomes []Outcome
P50 time.Duration
P95 time.Duration
Max time.Duration
}
// TagStat — passed/total for one slice of the fixture.
type TagStat struct{ Passed, Total int }
// Accuracy — fraction of cases that passed every check.
func (r Report) Accuracy() float64 {
if r.Total == 0 {
return 0
}
return float64(r.Passed) / float64(r.Total)
}
// Score runs every case through p and aggregates. A phrasing error scores as a
// miss and is counted in Errors — "the model was down" and "the model wrote
// something bad" are different numbers and a prompt change must not be able to
// hide behind the first one.
//
// Latency is wall-clock per PhraseNudge call. On the CPU/iGPU target a nudge
// the model takes a minute to word has already missed its moment.
func Score(ctx context.Context, name string, p Nudger, f Fixture) (Report, error) {
now, err := f.Now()
if err != nil {
return Report{}, err
}
rep := Report{
Name: name,
Total: len(f.Cases),
ByCheck: map[string]int{},
ByRule: map[string]TagStat{},
}
for _, name := range CheckNames {
rep.ByCheck[name] = 0
}
lat := make([]time.Duration, 0, len(f.Cases))
for _, c := range f.Cases {
start := time.Now()
pn, err := p.PhraseNudge(ctx, c.Candidate(now))
o := Outcome{Case: c, Body: pn.Body, Mood: pn.Mood, Err: err, Latency: time.Since(start)}
lat = append(lat, o.Latency)
if err != nil {
rep.Errors++
o.Failed = append(o.Failed, "call")
o.Reasons = append(o.Reasons, fmt.Sprintf("phrase error: %v", err))
} else {
for _, res := range RunChecks(c, pn.Body, pn.Mood) {
if res.Pass {
rep.ByCheck[res.Name]++
continue
}
o.Failed = append(o.Failed, res.Name)
o.Reasons = append(o.Reasons, res.Name+": "+res.Detail)
}
}
o.Pass = len(o.Failed) == 0
if o.Pass {
rep.Passed++
}
bump(rep.ByRule, ruleFamily(c.Rule), o.Pass)
rep.Outcomes = append(rep.Outcomes, o)
}
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
if len(lat) > 0 {
rep.Max = lat[len(lat)-1]
}
return rep, nil
}
// ruleFamily collapses "routine:зарядка" to "routine" so the per-rule table
// stays readable however many routines the operator configures.
func ruleFamily(rule string) string {
if i := strings.IndexByte(rule, ':'); i > 0 {
return rule[:i]
}
return rule
}
func bump(m map[string]TagStat, key string, pass bool) {
if key == "" {
return
}
s := m[key]
s.Total++
if pass {
s.Passed++
}
m[key] = s
}
// percentile — nearest-rank on a pre-sorted slice. No interpolation: with ~15
// samples an interpolated p95 invents a latency no call actually took.
func percentile(sorted []time.Duration, p float64) time.Duration {
if len(sorted) == 0 {
return 0
}
i := int(p * float64(len(sorted)))
if i >= len(sorted) {
i = len(sorted) - 1
}
return sorted[i]
}
// String renders the comparison table — composite score, then per-check so a
// regression names the property it broke, then latency.
func (r Report) String() string {
var b strings.Builder
fmt.Fprintf(&b, "%s: %d/%d cases pass every check (%.1f%%), %d errors\n",
r.Name, r.Passed, r.Total, 100*r.Accuracy(), r.Errors)
for _, name := range CheckNames {
fmt.Fprintf(&b, " %-9s %d/%d\n", name, r.ByCheck[name], r.Total)
}
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
fmt.Fprintf(&b, " by rule: %s\n", renderStats(r.ByRule))
return b.String()
}
// Failures — per-case detail, sorted by ID so two runs diff cleanly.
func (r Report) Failures() string {
var b strings.Builder
for _, o := range r.sorted() {
if o.Pass {
continue
}
fmt.Fprintf(&b, " %s %q\n %s\n", o.Case.ID, o.Body, strings.Join(o.Reasons, "; "))
}
return b.String()
}
// Messages — every generated message verbatim, pass or fail. This is what a
// human reads to judge tone; the score only says which checks fired.
func (r Report) Messages() string {
var b strings.Builder
for _, o := range r.sorted() {
mark := "ok "
if !o.Pass {
mark = "FAIL"
}
fmt.Fprintf(&b, " %s %-22s [%s] %q\n", mark, o.Case.ID, o.Mood, o.Body)
}
return b.String()
}
func (r Report) sorted() []Outcome {
out := append([]Outcome(nil), r.Outcomes...)
sort.Slice(out, func(i, j int) bool { return out[i].Case.ID < out[j].Case.ID })
return out
}
func renderStats(m map[string]TagStat) string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
parts := make([]string, 0, len(keys))
for _, k := range keys {
parts = append(parts, fmt.Sprintf("%s %d/%d", k, m[k].Passed, m[k].Total))
}
return strings.Join(parts, " ")
}
+179
View File
@@ -0,0 +1,179 @@
package eval
import (
"context"
"strings"
"testing"
"github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/phraser"
)
func TestLoadFixture(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if _, err := f.Now(); err != nil {
t.Fatalf("Now: %v", err)
}
seen := map[string]bool{}
for _, c := range f.Cases {
if seen[c.ID] {
t.Errorf("duplicate case id %q", c.ID)
}
seen[c.ID] = true
if c.Rule == "" || c.Severity < 1 || c.Severity > 4 {
t.Errorf("%s: rule %q severity %d", c.ID, c.Rule, c.Severity)
}
if len(c.WantAny) == 0 {
t.Errorf("%s: no want_any, the on-topic check would always pass", c.ID)
}
}
// Coverage floor: all five loop rules plus both minted families, or the
// fixture measures a subset and the score does not mean what it says.
for _, rule := range []string{"water", "meal", "break", "service_down", "netdata_critical", "routine", "morning"} {
found := false
for _, c := range f.Cases {
if ruleFamily(c.Rule) == rule {
found = true
}
}
if !found {
t.Errorf("no case for rule family %q", rule)
}
}
}
// TestStubBaseline is the CI ratchet: the deterministic Stub, no model, no
// network. The floor is low on purpose — the Stub is English template phrasing,
// so it fails `lang` on every case by construction. The point of the ratchet is
// that the checks keep running and the Stub does not get worse, not that the
// Stub is good.
func TestStubBaseline(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
rep, err := Score(context.Background(), "stub (deterministic floor)", phraser.NewStub(), f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String() + rep.Messages())
if rep.Errors != 0 {
t.Errorf("stub returned %d errors — the deterministic path must never fail", rep.Errors)
}
// Per-check ratchets rather than one composite: the Stub's composite is 0
// (it never passes `lang`), so a composite floor would catch nothing.
floors := map[string]int{
CheckMood: 15,
// 12, not 15: the Stub's `break` template genuinely runs past the
// ceiling ("you've been at your desk for 4 hours without a break — step
// away for a bit." is 76 chars but 16 words). Left failing rather than
// raising the ceiling to hide it.
CheckLength: 12,
CheckFeminine: 15,
CheckCringe: 15,
CheckOnTopic: 12,
}
for name, floor := range floors {
if rep.ByCheck[name] < floor {
t.Errorf("check %s: %d/%d, below ratchet %d — phrasing regressed",
name, rep.ByCheck[name], rep.Total, floor)
}
}
}
// TestChecksCatchWhatTheyClaim — the checks are the measurement, so they get
// their own tests. Without these, a bad regexp would silently make every
// phrasing run look clean.
func TestChecksCatchWhatTheyClaim(t *testing.T) {
water := Case{Rule: "water", WantAny: []string{"вод"}}
cases := []struct {
name string
body string
want string // the check that must fail, "" for a clean message
}{
{"clean", "уже четыре часа без воды — попей.", ""},
{"long", "уже четыре часа без воды, а это довольно много, и вообще пить надо регулярно, иначе будет плохо совсем", CheckLength},
{"english", "you haven't had water in 4 hours, drink something", CheckLang},
{"masculine self", "я напомнил про воду.", CheckFeminine},
{"masculine dropped pronoun", "напомнил тебе про воду.", CheckFeminine},
{"masculine predicative", "я должен сказать: попей воды.", CheckFeminine},
// The other direction: HE is male, so second-person masculine is right.
{"second person masculine ok", "ты не пил воду четыре часа.", ""},
{"feminine self ok", "я заметила: воды не было четыре часа.", ""},
{"pet name", "милый, попей воды.", CheckCringe},
{"emoji", "попей воды 💧", CheckCringe},
{"exclamations", "попей воды!!", CheckCringe},
{"fake concern", "я беспокоюсь: воды не было четыре часа.", CheckCringe},
{"apology", "извини, что отвлекаю — попей воды.", CheckCringe},
{"emotional support", "я рядом, ты не один. попей воды.", CheckCringe},
{"asks how he feels", "как ты себя чувствуешь? попей воды.", CheckCringe},
{"praise", "молодец! теперь попей воды.", CheckCringe},
{"off topic", "пора бы уже что-то сделать.", CheckOnTopic},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
var failed []string
for _, r := range RunChecks(water, tc.body, "neutral") {
if !r.Pass {
failed = append(failed, r.Name+"("+r.Detail+")")
}
}
joined := strings.Join(failed, " ")
switch {
case tc.want == "" && len(failed) > 0:
t.Errorf("clean message flagged: %s", joined)
case tc.want != "" && !strings.Contains(joined, tc.want+"("):
t.Errorf("want %s to fail, got %q", tc.want, joined)
}
})
}
}
func TestMoodCheckUsesTheEnum(t *testing.T) {
if r := checkMood("cheerful"); r.Pass {
t.Error("mood outside the enum passed")
}
if r := checkMood(""); r.Pass {
t.Error("empty mood passed")
}
for m := range Moods {
if r := checkMood(m); !r.Pass {
t.Errorf("enum mood %q failed", m)
}
}
}
// TestCandidateCarriesTheContext — the whole harness is worthless if the
// Candidate it builds does not carry the duration the prompt is supposed to
// name.
func TestCandidateCarriesTheContext(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
now, _ := f.Now()
for _, c := range f.Cases {
cand := c.Candidate(now)
if cand.Rule.Name != c.Rule || cand.Severity != loop.Severity(c.Severity) {
t.Errorf("%s: candidate lost rule or severity", c.ID)
}
if c.SinceMinutes > 0 {
d, ok := cand.State.Since(c.Rule)
if !ok || int(d.Minutes()) != c.SinceMinutes {
t.Errorf("%s: since %v ok=%v, want %d minutes", c.ID, d, ok, c.SinceMinutes)
}
}
if c.FactKey != "" {
fact, ok := cand.State.Fact(c.Rule)
if !ok || fact.Key != c.FactKey {
t.Errorf("%s: fact key %q, want %q", c.ID, fact.Key, c.FactKey)
}
}
}
}
+73
View File
@@ -0,0 +1,73 @@
package eval
import (
"context"
"os"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/phraser"
)
// TestLLMPhrasingBaseline — the resident model wording real nudges. Opt-in,
// same shape as internal/router/eval's MAVEN_LLM_URL gate, because CI has no
// model and a phrasing run costs minutes on the CPU target.
//
// llama-server -m /mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf \
// --host 127.0.0.1 --port 18099 -c 2048 -ngl 99
// MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
//
// It reports and does not assert a quality bar. The numbers are the input to
// tuning the persona prompt; an assertion here would be the test inventing the
// bar rather than measuring against it. The one thing worth failing on is a
// harness fault — every case erroring means the run measured infrastructure.
func TestLLMPhrasingBaseline(t *testing.T) {
base := os.Getenv("MAVEN_LLM_URL")
if base == "" {
t.Skip("MAVEN_LLM_URL unset — point it at a running llama-server (see doc comment)")
}
// A local llama-server must not go through an HTTP proxy. This box proxies
// loopback through a SOCKS bridge that answers 503, which would score every
// case as a phrasing error and read as "the model cannot phrase".
noProxyLoopback(t)
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
cfg := phraser.DefaultConfig("")
// Generous: an unconstrained 0.8B can spend a minute thinking before it
// writes a word, and a timeout would be scored as a model failure.
cfg.Timeout = 5 * time.Minute
p := phraser.NewLLMPhraserAt(base, cfg)
defer p.Close()
rep, err := Score(context.Background(), "llm (0.8B, built-in persona)", p, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String() + "\nmessages:\n" + rep.Messages() + "\nfailures:\n" + rep.Failures())
if rep.Errors == rep.Total {
t.Errorf("all %d cases errored — harness fault, not a measurement", rep.Total)
}
}
// noProxyLoopback appends the loopback host to no_proxy before any request, so
// http.ProxyFromEnvironment (which caches the environment on first use) sees it.
func noProxyLoopback(t *testing.T) {
t.Helper()
for _, key := range []string{"no_proxy", "NO_PROXY"} {
cur := os.Getenv(key)
if strings.Contains(cur, "127.0.0.1") {
continue
}
if cur == "" {
t.Setenv(key, "127.0.0.1,localhost")
continue
}
t.Setenv(key, cur+",127.0.0.1,localhost")
}
}
+158
View File
@@ -0,0 +1,158 @@
{
"schema_version": 1,
"name": "nudge phrasing v1",
"reference_now": "2026-07-31T21:40:00+03:00",
"notes": [
"The five loop rules (water, meal, break, service_down, netdata_critical) plus one routine: and one morning: case, at the severities they actually ship with.",
"The bad-moment cases (quiet_hours, calendar_busy) are here because the gate already let them through — ops outranks quiet hours. The phrasing must stay short and plain, not apologise for the timing.",
"want_any lists stems, not whole words, so declension does not defeat the on-topic check. English stems are included because the deterministic Stub is still English.",
"There is no expected message. The checks measure properties, not similarity to a reference sentence — a fixed golden string would just freeze one arbitrary phrasing."
],
"cases": [
{
"id": "water-3h",
"rule": "water",
"severity": 1,
"since_minutes": 190,
"want_any": ["вод", "попей", "пить", "выпей", "напит", "water", "drink"],
"tags": ["care"],
"note": "The base case. Just over the 3h predicate."
},
{
"id": "water-7h",
"rule": "water",
"severity": 1,
"since_minutes": 430,
"want_any": ["вод", "попей", "пить", "выпей", "напит", "water", "drink"],
"tags": ["care"],
"note": "Long overdue. Severity is unchanged, so the phrasing must not escalate into alarm."
},
{
"id": "water-busy",
"rule": "water",
"severity": 1,
"since_minutes": 240,
"calendar_busy": true,
"want_any": ["вод", "попей", "пить", "выпей", "напит", "water", "drink"],
"tags": ["care", "bad-moment"],
"note": "Mid-meeting. A bad moment invites an apology, which is the check that should catch it."
},
{
"id": "meal-7h",
"rule": "meal",
"severity": 1,
"since_minutes": 420,
"want_any": ["ешь", "еда", "еды", "поешь", "перекус", "обед", "ужин", "покуш", "food", "eat"],
"tags": ["care"]
},
{
"id": "meal-11h-quiet",
"rule": "meal",
"severity": 1,
"since_minutes": 660,
"quiet_hours": true,
"want_any": ["ешь", "еда", "еды", "поешь", "перекус", "обед", "ужин", "покуш", "food", "eat"],
"tags": ["care", "bad-moment"],
"note": "Quiet hours suppresses care nudges in the gate, so this one only reaches the phraser via an explicit override. Included because it is the shape most likely to draw a hedge."
},
{
"id": "break-90m",
"rule": "break",
"severity": 2,
"since_minutes": 95,
"want_any": ["перерыв", "разомн", "встань", "отдохн", "пауз", "отойд", "размин", "break", "step away"],
"tags": ["care"]
},
{
"id": "break-4h",
"rule": "break",
"severity": 2,
"since_minutes": 240,
"want_any": ["перерыв", "разомн", "встань", "отдохн", "пауз", "отойд", "размин", "break", "step away"],
"tags": ["care"]
},
{
"id": "break-busy",
"rule": "break",
"severity": 2,
"since_minutes": 150,
"calendar_busy": true,
"want_any": ["перерыв", "разомн", "встань", "отдохн", "пауз", "отойд", "размин", "break", "step away"],
"tags": ["care", "bad-moment"]
},
{
"id": "service-down",
"rule": "service_down",
"severity": 4,
"since_minutes": 3,
"fact_key": "vaultwarden",
"fact_value": "\"down\"",
"fact_source": "poll:uptimekuma",
"want_any": ["vaultwarden", "сервис", "упал", "не отвеч", "лежит", "недоступ", "down", "service"],
"tags": ["ops"],
"note": "The service name is in the fact key, not the value. A nudge that says 'a service' without naming it is on-topic but useless — the on-topic check cannot catch that, a human reading Messages() can."
},
{
"id": "service-down-night",
"rule": "service_down",
"severity": 4,
"since_minutes": 2,
"quiet_hours": true,
"fact_key": "nextcloud",
"fact_value": "\"down\"",
"fact_source": "poll:uptimekuma",
"want_any": ["nextcloud", "сервис", "упал", "не отвеч", "лежит", "недоступ", "down", "service"],
"tags": ["ops", "bad-moment"],
"note": "03:00-shaped. Sev4 outranks quiet hours by design, so she speaks — plainly, without softening it into a maybe."
},
{
"id": "netdata-disk",
"rule": "netdata_critical",
"severity": 3,
"since_minutes": 5,
"fact_key": "netdata_alarm",
"fact_value": "\"critical\"",
"fact_source": "poll:netdata",
"want_any": ["netdata", "диск", "критич", "алярм", "аларм", "тревог", "место", "памят", "critical", "alarm", "disk"],
"tags": ["ops"]
},
{
"id": "netdata-busy",
"rule": "netdata_critical",
"severity": 3,
"since_minutes": 12,
"calendar_busy": true,
"fact_key": "netdata_alarm",
"fact_value": "\"critical\"",
"fact_source": "poll:netdata",
"want_any": ["netdata", "диск", "критич", "алярм", "аларм", "тревог", "место", "памят", "critical", "alarm", "disk"],
"tags": ["ops", "bad-moment"]
},
{
"id": "routine-pills",
"rule": "routine:таблетки",
"severity": 2,
"since_minutes": 0,
"want_any": ["таблетк", "приня", "лекарств", "pill"],
"tags": ["routine"],
"note": "A routine: rule has no fact of its own, so there is no duration to name. The rule name is the only context."
},
{
"id": "routine-stretch",
"rule": "routine:зарядка",
"severity": 1,
"since_minutes": 0,
"want_any": ["зарядк", "размин", "упражн", "потянис", "разомн", "stretch", "exercise"],
"tags": ["routine"]
},
{
"id": "morning-checklist",
"rule": "morning:утро",
"severity": 2,
"since_minutes": 0,
"want_any": ["утр", "чеклист", "список", "не сделан", "осталось", "morning"],
"tags": ["routine"],
"note": "In production cmd/mavend/tick.go phrases morning routines deterministically and never calls the LLM. Scored anyway: the phraser is reachable with this rule name, and a fallback that garbles it is still a bug."
}
]
}
+16
View File
@@ -67,6 +67,22 @@ func NewLLMPhraser(ctx context.Context, cfg Config) (*LLMPhraser, error) {
return p, nil
}
// NewLLMPhraserAt wires a phraser to a llama-server that someone else started
// and owns. It spawns nothing, so Close does not kill anything.
//
// This exists for the phrasing scorer (internal/phraser/eval), which must
// measure the phrasing against a shared llama-server without taking the model
// load hit per run or killing a server another process depends on. The daemon
// still uses NewLLMPhraser and still owns its own child process.
func NewLLMPhraserAt(baseURL string, cfg Config) *LLMPhraser {
return &LLMPhraser{
cfg: cfg,
client: &http.Client{Timeout: cfg.Timeout},
port: strings.TrimSuffix(baseURL, "/"),
cancel: func() {},
}
}
func (p *LLMPhraser) start(ctx context.Context) error {
args := []string{
"-m", p.cfg.ModelPath,
+5 -1
View File
@@ -92,7 +92,11 @@ func (s *Stub) Close() error { return nil }
// the predicate fire (the same State the predicate saw).
func (s *Stub) PhraseNudge(_ context.Context, c loop.Candidate) (delivery.PhrasedNudge, error) {
body, summary := phraseNudge(c)
return delivery.PhrasedNudge{Candidate: c, Body: body, Summary: summary}, nil
// "neutral" rather than empty: Mood is part of the documented output
// contract and the Stub is a production fallback, so it must satisfy the
// contract too. Template phrasing has no tone to report, and neutral is the
// enum's own default.
return delivery.PhrasedNudge{Candidate: c, Body: body, Summary: summary, Mood: "neutral"}, nil
}
// PhraseReminder — extracts the user's text from the reminder payload (raw
+31
View File
@@ -19,6 +19,37 @@ type Embedder interface {
Close() error
}
// AsymmetricEmbedder — an embedder that wants to know whether a text is a
// search query or a stored passage. Recall is asymmetric: a short question
// goes in, a longer note comes out. The e5 family is trained for exactly that
// and needs the side written into the text ("query: " / "passage: ").
//
// Optional on purpose: HashEmbedder has no such notion, so callers go through
// EmbedQuery and EmbedPassage below, which fall back to plain Embed.
type AsymmetricEmbedder interface {
Embedder
EmbedQuery(ctx context.Context, text string) ([]float32, error)
EmbedPassage(ctx context.Context, text string) ([]float32, error)
}
// EmbedQuery embeds text that is being searched WITH — a question.
func EmbedQuery(ctx context.Context, e Embedder, text string) ([]float32, error) {
if a, ok := e.(AsymmetricEmbedder); ok {
return a.EmbedQuery(ctx, text)
}
return e.Embed(ctx, text)
}
// EmbedPassage embeds text that is being searched FOR — a note or a fact on
// its way into the store. Store and lookup must use these two calls, not one
// of them twice, or the asymmetry buys nothing.
func EmbedPassage(ctx context.Context, e Embedder, text string) ([]float32, error) {
if a, ok := e.(AsymmetricEmbedder); ok {
return a.EmbedPassage(ctx, text)
}
return e.Embed(ctx, text)
}
// HashEmbedder — a deterministic bag-of-words embedder used for tests and as a
// non-zero default floor. NOT semantically meaningful across languages; the
// real classifier swaps in the multilingual ONNX model wholesale.
+57
View File
@@ -37,3 +37,60 @@ func TestHashEmbedderCyrillic(t *testing.T) {
t.Fatalf("cosine(shared)=%.3f not > cosine(disjoint)=%.3f", cosine(a, b), cosine(a, c))
}
}
// recordingEmbedder — an asymmetric embedder that only remembers which side
// was asked for. Enough to pin the dispatch; real vectors need the model.
type recordingEmbedder struct{ calls []string }
func (r *recordingEmbedder) Dim() int { return 2 }
func (r *recordingEmbedder) Close() error { return nil }
func (r *recordingEmbedder) Embed(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "embed")
return []float32{1, 0}, nil
}
func (r *recordingEmbedder) EmbedQuery(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "query")
return []float32{1, 0}, nil
}
func (r *recordingEmbedder) EmbedPassage(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "passage")
return []float32{0, 1}, nil
}
// TestEmbedQueryAndPassageSplit — a question and a stored note must not take
// the same path. If both ended up on the same call the asymmetric model buys
// nothing, which is the whole reason for the swap.
func TestEmbedQueryAndPassageSplit(t *testing.T) {
rec := &recordingEmbedder{}
if _, err := EmbedQuery(context.Background(), rec, "где логи?"); err != nil {
t.Fatalf("EmbedQuery: %v", err)
}
if _, err := EmbedPassage(context.Background(), rec, "логи в /var/log"); err != nil {
t.Fatalf("EmbedPassage: %v", err)
}
if len(rec.calls) != 2 || rec.calls[0] != "query" || rec.calls[1] != "passage" {
t.Errorf("calls %v, want [query passage]", rec.calls)
}
}
// TestEmbedFallsBackToPlainEmbed — HashEmbedder has no sides, so both helpers
// must still work and give the same vector.
func TestEmbedFallsBackToPlainEmbed(t *testing.T) {
h := NewHashEmbedder(64)
q, err := EmbedQuery(context.Background(), h, "text")
if err != nil {
t.Fatalf("EmbedQuery: %v", err)
}
p, err := EmbedPassage(context.Background(), h, "text")
if err != nil {
t.Fatalf("EmbedPassage: %v", err)
}
for i := range q {
if q[i] != p[i] {
t.Fatalf("hash embedder gave two different vectors for the same text")
}
}
}
+2 -2
View File
@@ -185,8 +185,8 @@ func TestONNXBaseline(t *testing.T) {
if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
}
model := filepath.Join("../../..", "models/embedder/model.onnx")
tok := filepath.Join("../../..", "models/embedder/tokenizer.json")
model := filepath.Join("../../..", "models/embedder/multilingual-e5-small/model_quantized.onnx")
tok := filepath.Join("../../..", "models/embedder/multilingual-e5-small/tokenizer.json")
for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err)
+16 -4
View File
@@ -23,7 +23,11 @@ import (
//
// llama-server -m /mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf \
// --host 127.0.0.1 --port 18099 -c 2048 -ngl 99
// MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router
// MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-models
//
// Every report name carries the model llama-server reports over /v1/models, so
// a bake-off across checkpoints (#278, #250) produces tables you can tell
// apart. Point the variable at one server at a time.
//
// Three configurations, because "the LLM router" is ambiguous and the three
// numbers answer different questions:
@@ -53,6 +57,14 @@ func TestLLMRouterBaseline(t *testing.T) {
}
ctx := context.Background()
model, err := ModelID(ctx, base)
if err != nil {
// Not fatal: an unlabelled score is still a score. But say so loudly,
// because an unlabelled row in a bake-off table is worthless.
t.Logf("could not read model id from %s: %v — reports will say %q", base, err, "unknown-model")
model = "unknown-model"
}
t.Logf("scoring model %s at %s", model, base)
lr := router.NewLLMRouter(client)
// llm-only: the LLM stage in isolation. Route returns (Decision, ok, err);
@@ -68,7 +80,7 @@ func TestLLMRouterBaseline(t *testing.T) {
}
return d, nil
})
repLLM, err := Score(ctx, "llm-only (0.8B, as deployed)", llmOnly, f)
repLLM, err := Score(ctx, "llm-only ("+model+", as deployed)", llmOnly, f)
if err != nil {
t.Fatalf("Score llm-only: %v", err)
}
@@ -77,7 +89,7 @@ func TestLLMRouterBaseline(t *testing.T) {
// cascade+llm: stage-0 grammar → LLM → classifier fallback, the wiring #320
// proposes. Hash embedder for the fallback so the classifier contribution is
// the deterministic floor and any lift is attributable to the model.
repCascade, err := Score(ctx, "cascade+llm (0.8B) + hash fallback",
repCascade, err := Score(ctx, "cascade+llm ("+model+") + hash fallback",
newBaselineRouter(t, router.NewHashEmbedder(1024), lr), f)
if err != nil {
t.Fatalf("Score cascade: %v", err)
@@ -96,7 +108,7 @@ func TestLLMRouterBaseline(t *testing.T) {
// either way. Kept so the question stays answered instead of being
// re-asked, and so internal/llm does NOT grow a chat_template_kwargs field
// for a problem that does not exist.
repNoThink, err := Score(ctx, "llm-only (0.8B, thinking off) [diagnostic]",
repNoThink, err := Score(ctx, "llm-only ("+model+", thinking off) [diagnostic]",
RouterFunc(func(ctx context.Context, u string, now time.Time) (router.Decision, error) {
d, ok, err := router.NewLLMRouter(&noThinkCompleter{base: base, http: &http.Client{Timeout: 60 * time.Second}}).Route(ctx, u, now)
if err != nil {
+51
View File
@@ -0,0 +1,51 @@
package eval
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
)
// ModelID asks llama-server which model it has loaded, so a scoring run can
// label itself. Without this a bake-off between two models produces two tables
// that look identical, and the operator has to remember which server was up.
//
// Read from the server rather than passed in on purpose: a hand-typed label
// goes stale the moment someone restarts the server with a different -m.
func ModelID(ctx context.Context, base string) (string, error) {
req, err := http.NewRequestWithContext(ctx, "GET", strings.TrimSuffix(base, "/")+"/v1/models", nil)
if err != nil {
return "", err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return "", fmt.Errorf("models: status %d", resp.StatusCode)
}
var out struct {
Data []struct {
ID string `json:"id"`
} `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
return "", err
}
if len(out.Data) == 0 {
return "", fmt.Errorf("models: empty list")
}
return shortModelID(out.Data[0].ID), nil
}
// shortModelID trims the path and the .gguf suffix — llama-server reports the
// file name it was started with, which is too long for a table header.
func shortModelID(id string) string {
if i := strings.LastIndexAny(id, "/\\"); i >= 0 {
id = id[i+1:]
}
return strings.TrimSuffix(id, ".gguf")
}
+33 -3
View File
@@ -29,7 +29,7 @@ func NewLLMRouter(c Completer) *LLMRouter { return &LLMRouter{c: c} }
const routeGrammar = `
root ::= "[" ws action ("," ws action)* ws "]"
action ::= "{" ws "\"intent\"" ws ":" ws intent ("," ws field)* ws "}"
intent ::= "\"fact\"" | "\"reminder\"" | "\"note\"" | "\"query\"" | "\"act\"" | "\"chat\"" | "\"system\""
intent ::= "\"fact\"" | "\"reminder\"" | "\"note\"" | "\"query\"" | "\"act\"" | "\"chat\"" | "\"system\"" | "\"unknown\""
field ::= key ws ":" ws string
key ::= "\"key\"" | "\"value\"" | "\"text\"" | "\"verb\""
string ::= "\"" ([^"\\] | "\\" .){0,120} "\""
@@ -41,12 +41,17 @@ ws ::= [ \t\n]*
// question naming a fact key ("сколько воды я выпил с утра") matched the fact
// rule first and was stored as an assertion — 15 of 76 fixture cases.
//
// Changed again 31-07-2026: added the "unknown" escape hatch so the model can
// admit it cannot route (Vikunja #359).
//
// The training workspace keeps its own copy of this prompt for relabelling, and
// `llm/check_prompt_parity.py` there compares the two. That copy is in another
// repo and was not touched, so parity will fail until it gets the same edit.
// repo and was not touched, so parity will fail until it gets the same edits —
// both the rule reorder and the "unknown" wording (Vikunja #362).
const routeSystem = `Классифицируй ровно одно сообщение пользователя. Верни ОДИН JSON-массив действий.
Ровно одно намерение: fact, reminder, note, query, act, chat, system.
Есть восьмое значение unknown — только для случаев, когда просьбу невозможно понять.
Классифицируй по цели пользователя. Порядок решения:
1. Хочет напоминание в будущем → reminder
@@ -56,12 +61,14 @@ const routeSystem = `Классифицируй ровно одно сообще
5. Утверждает: сообщает или обновляет текущее состояние/событие → fact
6. Просит выполнить работу → act
7. Про ассистента, настройки или память → system
8. Иначе → chat
8. Реплика — обрывок или указание на неназванное («это», «то», «потом»), и без него непонятно, что именно нужно сделать → unknown
9. Иначе → chat
Различия:
- note — сохранить информацию, без напоминания. text = суть.
- reminder — уведомить позже. text = что напомнить.
- fact — неявное обновление: пользователь сообщает, что что-то в мире изменилось (текущее/изменённое состояние, случившееся событие). key/value.
- unknown — редкий случай. Ставь его, только если в самой реплике нет ни предмета, ни действия. Короткая, простая или незнакомая тема — это не причина для unknown: приветствие и болтовня — это chat, вопрос на любую тему — это query, просьба сделать что-то названное — это act.
- query против fact — решает форма реплики, а не тема. Вопрос о состоянии — это query, даже если названо то же самое, что бывает в fact. Только утверждение — это fact.
Примеры:
@@ -76,6 +83,13 @@ const routeSystem = `Классифицируй ровно одно сообще
"напиши письмо" → {"intent":"act","verb":"написать письмо"}
"очисти память" → {"intent":"system"}
"привет" → {"intent":"chat","text":"привет"}
"сделай это" → {"intent":"unknown"}
"ну это" → {"intent":"unknown"}
"потом" → {"intent":"unknown"}
Но не путай — здесь unknown не нужен:
"сделай кофе" → {"intent":"act","verb":"сделать кофе"}
"что такое кватернион?" → {"intent":"query","text":"что такое кватернион"}
"ага" → {"intent":"chat","text":"ага"}
Ответ — JSON-массив: по одному объекту на каждую просьбу. Обычно один. Если в реплике несколько просьб — по объекту на каждую. "напомни купить молоко, и запиши что кофе кончился" → [{"intent":"reminder","text":"купить молоко"},{"intent":"note","text":"кофе кончился"}]. Только JSON, без пояснений.`
@@ -84,6 +98,11 @@ const routeSystem = `Классифицируй ровно одно сообще
// the loop without hurting short slot values.
const routeRepeatPenalty = 1.15
// routeIntentUnknown — the model's way of saying "I could not route this".
// It is a wire value only: it never becomes a router.Intent, it just makes
// Route return ok=false so the caller drops to the classifier cascade.
const routeIntentUnknown = "unknown"
type routeAction struct {
Intent string `json:"intent"`
Key string `json:"key"`
@@ -92,6 +111,10 @@ type routeAction struct {
Verb string `json:"verb"`
}
// Route asks the model for one decision. The bool is false when there is no
// decision to use: either the model failed (err set) or it refused with the
// "unknown" intent (err nil). Both mean the same thing to the caller — use the
// classifier instead.
func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time) (Decision, bool, error) {
raw, err := lr.c.Complete(ctx, llm.Req{
System: routeSystem,
@@ -115,6 +138,13 @@ func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time)
// with the engine turn-on (Router.Route → []Decision, both voice.go handlers
// loop). Until then only the first ask is honored.
a := acts[0]
// The model refused. Report "no decision" without an error, which is the
// same fall-through the caller already uses for a parse failure — the
// classifier cascade gets the turn and its own confidence gate decides
// whether to ask. Better a slower second opinion than a confident guess.
if a.Intent == routeIntentUnknown {
return Decision{}, false, nil
}
d := Decision{Utterance: utterance, Stage: 1, Confidence: 1.0}
switch Intent(a.Intent) {
case IntentFact:
+142 -1
View File
@@ -100,8 +100,10 @@ func TestLLMRouterReminderMapping(t *testing.T) {
}
}
// An intent name that is not in the contract at all (as opposed to "unknown",
// which is a real refusal) still defaults to chat.
func TestLLMRouterChatFallback(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`})
lr := NewLLMRouter(mockLLM{out: `{"intent":"banana"}`})
d, ok, err := lr.Route(context.Background(), "как дела?", time.Now())
if err != nil || !ok {
t.Fatalf("ok=%v err=%v", ok, err)
@@ -111,6 +113,61 @@ func TestLLMRouterChatFallback(t *testing.T) {
}
}
// The model must be able to say "I could not route this".
func TestRouteGrammarAllowsUnknown(t *testing.T) {
if !strings.Contains(routeGrammar, `"\"unknown\""`) {
t.Fatal("grammar cannot express a refusal")
}
}
// If the prompt does not tell the model when to refuse, it never will.
func TestRoutePromptExplainsUnknown(t *testing.T) {
if !strings.Contains(routeSystem, "unknown") {
t.Fatal("prompt never mentions the unknown intent")
}
if !strings.Contains(routeSystem, `"сделай это" → {"intent":"unknown"}`) {
t.Fatal("prompt lost its worked refusal example")
}
// A refusal-only router is useless, so the prompt must also show cases that
// look ambiguous but are not.
if !strings.Contains(routeSystem, "здесь unknown не нужен") {
t.Fatal("prompt lost its counter-examples")
}
}
// A refusal is not an error. It reports "no decision" so the cascade moves on.
func TestLLMRouterUnknownRefuses(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`})
_, ok, err := lr.Route(context.Background(), "сделай это", time.Now())
if ok {
t.Fatal("a refusal must not produce a usable decision")
}
if err != nil {
t.Fatalf("a refusal is not an error, got %v", err)
}
}
// The whole point of the refusal: the turn keeps going on the classifier, the
// same way it does when the model returns garbage.
func TestRouterFallsBackWhenLLMRefuses(t *testing.T) {
c := NewClassifier(NewHashEmbedder(1024))
seedClassifier(t, c)
r := New(Config{
Classifier: c,
Extractor: Extractor{Time: StubDateTimeParser{}, Facts: DefaultFactParser{}},
Threshold: 0.4,
LLM: NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`}),
})
d, err := r.Route(context.Background(), "напомни позвонить маме", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
// Stage 1 is the LLM's own answer; the classifier lands on stage 2 or 3.
if d.Stage < 2 {
t.Fatalf("want the classifier to decide, got stage %d (%+v)", d.Stage, d)
}
}
func TestLLMRouterLLMError(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: "", err: fmt.Errorf("llm down")})
_, ok, err := lr.Route(context.Background(), "x", time.Now())
@@ -118,3 +175,87 @@ func TestLLMRouterLLMError(t *testing.T) {
t.Fatal("want ok=false, err!=nil on llm error")
}
}
// --- slot extraction on top of an LLM decision --------------------------------
// newLLMTestRouter — a router whose route always comes from the mock model.
func newLLMTestRouter(t *testing.T, out string) *Router {
t.Helper()
c := NewClassifier(NewHashEmbedder(1024))
seedClassifier(t, c)
acts := DefaultActMatcher{Fns: []string{"restart", "stop", "run", "backup"}}
return New(Config{
Classifier: c,
Extractor: Extractor{Time: StubDateTimeParser{}, Acts: acts, Facts: DefaultFactParser{}},
Threshold: 0.4,
LLM: NewLLMRouter(mockLLM{out: out}),
})
}
// The model cannot produce a fire time, so without extraction every LLM-routed
// reminder was dropped as "no time".
func TestLLMDecisionGetsReminderTime(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"reminder","text":"позвонить маме"}`)
d, err := r.Route(context.Background(), "напомни позвонить маме через 2 часа", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if d.Intent != IntentReminder {
t.Fatalf("want reminder, got %v", d.Intent)
}
if !d.Slots.HasTime || !d.Slots.Time.Equal(refNow().Add(2*time.Hour)) {
t.Fatalf("want time now+2h, got %+v", d.Slots)
}
if d.Slots.Text != "позвонить маме" {
t.Fatalf("extraction overwrote the model's text: %q", d.Slots.Text)
}
}
// No time in the utterance ⇒ no time in the slots. Do not invent one; the
// daemon says it could not read the time.
func TestLLMReminderWithoutTimeStaysEmpty(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"reminder","text":"позвонить маме"}`)
d, err := r.Route(context.Background(), "напомни позвонить маме", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if d.Slots.HasTime {
t.Fatalf("invented a time: %v", d.Slots.Time)
}
}
// An act decision arrived with no Fn, so the tool never ran.
func TestLLMDecisionGetsActFn(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"act","verb":"restart nginx"}`)
d, err := r.Route(context.Background(), "слушай, restart nginx пожалуйста", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if !d.Slots.HasFn || d.Slots.Fn != "restart" || len(d.Slots.Args) != 1 || d.Slots.Args[0] != "nginx" {
t.Fatalf("want fn=restart args=[nginx], got %+v", d.Slots)
}
}
// The model's own slots win; extraction only fills gaps.
func TestLLMSlotsWinOverExtraction(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"fact","key":"hydration","value":"выпил"}`)
d, err := r.Route(context.Background(), "я выпил воду", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if d.Slots.Key != "hydration" {
t.Fatalf("extraction overwrote the model's key: %q", d.Slots.Key)
}
}
// A fact the model left keyless still gets one from the parser.
func TestLLMFactGetsKeyFromParser(t *testing.T) {
r := newLLMTestRouter(t, `{"intent":"fact","text":"я выпил воду"}`)
d, err := r.Route(context.Background(), "я выпил воду", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
if !d.Slots.HasKey || d.Slots.Key != "water" {
t.Fatalf("want key=water, got %+v", d.Slots)
}
}
+28 -1
View File
@@ -12,6 +12,15 @@ import (
"golang.org/x/text/unicode/norm"
)
// The deployed model is multilingual-e5-small. e5 was trained with these two
// words glued to the front of every text, and it scores badly without them —
// they are part of the model, not a style choice. Swapping back to a symmetric
// paraphrase model means dropping them again.
const (
queryPrefix = "query: "
passagePrefix = "passage: "
)
const (
padTokenID = 1
unkTokenID = 3
@@ -54,7 +63,25 @@ func NewONNXEmbedder(modelPath, tokenizerPath, libPath string) (*onnxEmbedder, e
func (e *onnxEmbedder) Dim() int { return embedDim }
// Embed treats the text as a query. The classifier compares one short
// utterance to another short seed phrase, so both sides get the same prefix
// and the comparison stays fair. The recall path must call EmbedQuery and
// EmbedPassage instead.
func (e *onnxEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, queryPrefix+text)
}
// EmbedQuery — the question the user just asked.
func (e *onnxEmbedder) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, queryPrefix+text)
}
// EmbedPassage — a note or fact being stored, or re-scored at lookup time.
func (e *onnxEmbedder) EmbedPassage(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, passagePrefix+text)
}
func (e *onnxEmbedder) embed(ctx context.Context, text string) ([]float32, error) {
inputIDs, attentionMask, _ := e.tokenizer.Encode(text)
inputShape := ort.NewShape(1, int64(maxLength))
@@ -313,4 +340,4 @@ func preTokenize(text string) []string {
return out
}
var _ Embedder = (*onnxEmbedder)(nil)
var _ AsymmetricEmbedder = (*onnxEmbedder)(nil)
+34
View File
@@ -88,6 +88,7 @@ func (r *Router) Route(ctx context.Context, utterance string, now time.Time) (De
if r.llm != nil {
if d, ok, err := r.llm.Route(ctx, utterance, now); err == nil && ok {
d.Utterance = utterance
r.fillSlots(ctx, &d, now)
return d, nil
} else if err != nil {
log.Printf("router: llm route fell back to classifier: %v", err)
@@ -118,6 +119,39 @@ func (r *Router) Route(ctx context.Context, utterance string, now time.Time) (De
return d, nil
}
// fillSlots — run stage-2 extraction on an LLM decision and fill only the slots
// the model left empty. The LLM wins where it answered: it saw the sentence, the
// parsers are keyword tables. Extraction covers what the model cannot produce at
// all — a parsed reminder time and an allowlist fn.
//
// If a reminder still has no time, leave it missing. The daemon then says it
// could not read the time; inventing one would set a wrong alarm.
func (r *Router) fillSlots(ctx context.Context, d *Decision, now time.Time) {
ex := r.extractor.Extract(ctx, d.Intent, d.Utterance, now)
if !d.Slots.HasTime && ex.HasTime {
d.Slots.Time, d.Slots.HasTime = ex.Time, ex.HasTime
}
if !d.Slots.HasKey && ex.HasKey {
d.Slots.Key, d.Slots.Value, d.Slots.HasKey = ex.Key, ex.Value, ex.HasKey
}
if !d.Slots.HasFn && ex.HasFn {
d.Slots.Fn, d.Slots.Args, d.Slots.HasFn = ex.Fn, ex.Args, ex.HasFn
}
// For an act the model returns the verb in Text ("restart nginx"), which is
// often cleaner than the raw utterance ("maven, could you restart nginx").
// Try it too when the utterance did not match the allowlist.
if d.Intent == IntentAct && !d.Slots.HasFn && r.extractor.Acts != nil &&
d.Slots.Text != "" && d.Slots.Text != d.Utterance {
if fn, args, ok := r.extractor.Acts.Match(d.Slots.Text); ok {
d.Slots.Fn, d.Slots.Args, d.Slots.HasFn = fn, args, true
}
}
if d.Slots.Text == "" {
d.Slots.Text = ex.Text
}
// Stage stays 1: it says who decided the route, and that was the LLM.
}
// CorrectMisroute — the user corrected a bad classification. Appends a new
// example for the corrected intent (append-only — grows the classifier, no
// retrain). Same shape as nudges.outcome tuning cooldowns: more reliable over