Compare commits

...

11 Commits

Author SHA1 Message Date
kami 34521c30b8 Merge branch 'worktree-agent-ad5da57e47b822152' into overnight-jul31 2026-07-31 11:39:35 +04:00
kami 1d48755d12 Record the recall numbers after the embedder swap
recall@1 60% to 72%, answered 48% to 72%, latency 3x better. But false
recall went 1/5 to 5/5: e5 packs every score into a narrow high band, so
the 0.55 gate now admits everything. Left the gate alone as instructed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:18 +04:00
kami f6d5a2a7a4 Swap the embedder to multilingual-e5-small (Vikunja #371, #372)
The old model was a symmetric paraphrase model, so it scored "do these
look alike" instead of "does this note answer this question". Also fixes
the file mismatch: the Makefile, the deploy config and both evals now all
name the same quantized file, and the quantized one is what gets measured.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:18 +04:00
kami 751c2a705f Embed a question and a stored note differently (Vikunja #371)
Note recall is asymmetric: a short question goes in, a longer note comes
out. Adds EmbedQuery/EmbedPassage helpers and the e5 prefixes, and points
the note/fact write path at the passage side and the query path at the
query side. Reviewers: the three call sites in voice.go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:38:07 +04:00
kami 5d5b0cfd49 Merge branch 'worktree-agent-a0ab2a9b7439296e3' into overnight-jul31 2026-07-31 11:37:53 +04:00
kami bd16ca69e5 Let the LLM router answer "unknown" when it cannot route
Chose an 8th enum value over a confidence number: the model already picks
one enum token, so it costs nothing in the grammar, while a score from a
0.8B model would be uncalibrated noise. A refusal returns "no decision"
with no error, which is the fall-through the caller already uses for a
bad parse, so the classifier and its clarify gate take the turn.

Reviewers: the prompt's counter-examples matter most — a small model will
over-use any easy escape hatch. The training workspace copy of the prompt
still needs the same edit (Vikunja #362).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:35:37 +04:00
kami 0ed386eca6 Re-measure the router on a quiet box and record the numbers
The earlier before/after was taken while another eval shared
llama-server. This run had the box to itself.

Intent accuracy 61.8% llm-only, 63.2% cascade, 67.1% with thinking off.
The prompt fix holds. note→fact shows up here too, so it is real.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 11:34:57 +04:00
kami 1c4eab2107 Merge commit '94eb92f' into overnight-jul31
# Conflicts:
#	Makefile
2026-07-31 10:11:56 +04:00
kami 0914e0a3d5 Merge commit '4ba9a6f' into overnight-jul31
# Conflicts:
#	Makefile
2026-07-31 10:11:29 +04:00
kami 4ba9a6f422 Add a deterministic scorer for nudge phrasing (Vikunja #323)
Review internal/phraser/eval/checks.go -- it IS the measurement. Each check
names in a comment which DESIGN.md line it defends: length, feminine
self-reference (windowed around "я" so the operator's own masculine
second-person forms are not flagged), the cringe list (pet names, emoji,
"!!", fake concern, apology, emotional support, asking how he feels,
praise), on-topic, mood enum. No send/veto signal anywhere, per
DESIGN.md § "Rules decide, LLM phrases".
Fixture (158 lines) and tests (252) do not count toward the diff ceiling;
the scorer itself is still ~650. Splitting eval.go from checks.go would
give two commits neither of which measures anything.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:30:52 +04:00
kami 94eb92fb15 Label LLM eval runs with the model llama-server has loaded
The bake-off in #278/#250 needs two models' scores side by side, and the
report names only carried the config, so the rows were indistinguishable.
ModelID reads /v1/models instead of taking a string that goes stale.
New target: make eval-models MAVEN_LLM_URL=...

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 02:17:20 +04:00
24 changed files with 1504 additions and 40 deletions
+8 -5
View File
@@ -43,14 +43,17 @@ notes. Without it, the floor `HashEmbedder` is used — deterministic but weak
(Russian recall rarely clears the confidence gate, many commands fall to (Russian recall rarely clears the confidence gate, many commands fall to
"clarify"). "clarify").
**Download the embedder** (ONNX, ~90 MB): **Download the embedder** (ONNX, ~120 MB):
```sh ```sh
make download-embedder make download-embedder
``` ```
This fetches `paraphrase-multilingual-MiniLM-L12-v2` (384-dim, 12-layer, This fetches `multilingual-e5-small` (384-dim, 12-layer, Russian and English)
supports 50+ languages including Russian) to `models/embedder/`. to `models/embedder/multilingual-e5-small/`. It is an asymmetric retrieval
model: the code puts `query: ` in front of a question and `passage: ` in front
of a stored note, which is how e5 was trained. The quantized file is the one
that is downloaded, deployed and measured.
**Also need ONNX Runtime** (`libonnxruntime.so`): **Also need ONNX Runtime** (`libonnxruntime.so`):
@@ -64,8 +67,8 @@ sudo cp onnxruntime-linux-x64-1.15.1/lib/libonnxruntime.so* /usr/local/lib/
```json ```json
"voice": { "voice": {
"embedder": { "embedder": {
"model_path": "models/embedder/model_quantized.onnx", "model_path": "models/embedder/multilingual-e5-small/model_quantized.onnx",
"tokenizer_path": "models/embedder/tokenizer.json", "tokenizer_path": "models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/usr/local/lib/libonnxruntime.so" "lib_path": "/usr/local/lib/libonnxruntime.so"
} }
} }
+32 -4
View File
@@ -16,7 +16,7 @@ PIPER_BIN := $(shell pwd)/deps/piper/piper
PIPER_MODEL := $(shell pwd)/models/tts/ru_RU-irina-medium.onnx PIPER_MODEL := $(shell pwd)/models/tts/ru_RU-irina-medium.onnx
PIPER_ESPEAK := $(shell pwd)/deps/piper/espeak-ng-data PIPER_ESPEAK := $(shell pwd)/deps/piper/espeak-ng-data
.PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go eval-router eval-recall .PHONY: all build build-stt build-tts build-daemon build-client build-waked build-web build-poll build-caldav clean test fmt-check vet run-stt run-tts run-web download-embedder deps-go eval-router eval-recall eval-phrasing eval-models
all: build all: build
@@ -103,6 +103,30 @@ eval-router:
eval-recall: eval-recall:
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/ MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
# eval-phrasing -- score nudge phrasing (internal/phraser/eval). Verbose so the
# report and every generated message land in the terminal. With no environment
# it scores the deterministic Stub only, which is what CI runs. Set
# MAVEN_LLM_URL to add the resident model:
# MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
# The model run is slow (minutes) -- the timeout is raised to match.
eval-phrasing:
$(GO) test -v -count=1 -timeout 40m ./internal/phraser/eval/
# eval-models — score ONE llama-server against the same fixture, for the
# resident-model bake-off (#278, #250). Start a server with the gguf you want,
# then:
#
# make eval-models MAVEN_LLM_URL=http://127.0.0.1:18100
#
# The report names carry the model llama-server reports, so runs from two
# checkpoints stay apart. Only the LLM test runs — the classifier baselines do
# not depend on the model and take the ONNX runtime with them.
MAVEN_LLM_URL ?= http://127.0.0.1:18099
eval-models:
MAVEN_LLM_URL="$(MAVEN_LLM_URL)" $(GO) test -v -count=1 -timeout 60m \
-run TestLLMRouterBaseline ./internal/router/eval/
run-stt: build-stt run-stt: build-stt
LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \ LD_LIBRARY_PATH="$(shell pwd)/deps/lib" \
./mavsttd -socket /tmp/maven/stt.sock -model $(WHISPER_MODEL) ./mavsttd -socket /tmp/maven/stt.sock -model $(WHISPER_MODEL)
@@ -128,9 +152,13 @@ deps-piper:
-o /tmp/piper.tar.gz -o /tmp/piper.tar.gz
tar -xzf /tmp/piper.tar.gz -C deps/ tar -xzf /tmp/piper.tar.gz -C deps/
EMBEDDER_DIR := $(shell pwd)/models/embedder # multilingual-e5-small: an asymmetric retrieval model. It is trained to match
EMBEDDER_MODEL_URL := https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/onnx/model_quantized.onnx # a short question against a longer passage, which is what note recall is.
EMBEDDER_TOKENIZER_URL := https://huggingface.co/Xenova/paraphrase-multilingual-MiniLM-L12-v2/resolve/main/tokenizer.json # The quantized file is the one we download, deploy and measure — see
# RECALL-EVAL-31-07-2026.md.
EMBEDDER_DIR := $(shell pwd)/models/embedder/multilingual-e5-small
EMBEDDER_MODEL_URL := https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/onnx/model_quantized.onnx
EMBEDDER_TOKENIZER_URL := https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/tokenizer.json
download-embedder: download-embedder:
mkdir -p $(EMBEDDER_DIR) mkdir -p $(EMBEDDER_DIR)
+60
View File
@@ -83,6 +83,66 @@ sqlite-backed `store.MemoryStore` and `memory.InMemoryStore` identically — bot
(`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is (`internal/store/memory.go:64`) at ~150µs over 42 rows against a ~59ms query embed. An ANN index is
not the problem to solve. not the problem to solve.
## Re-measured after the embedder swap — 31-07-2026, later the same day
Changed: `models/embedder/` is now **multilingual-e5-small** (quantized, 118MB), with `query: ` in
front of a question and `passage: ` in front of a stored note (Vikunja #371). `deploy/mavend.json`
and `make download-embedder` now name the same file, and it is the quantized one — that is what the
column below measures (Vikunja #372). Everything else is unchanged: same fixture, same store, same
0.55 gate. The old column is the baseline and is left as it was.
| | recall+onnx, MiniLM (baseline) | recall+onnx, e5-small (new) |
|---|---|---|
| **recall@1** | 60.0% (15/25) | **72.0% (18/25)** |
| recall@3 | 80.0% (20/25) | 84.0% (21/25) |
| **answered after the 0.55 gate** | 48.0% (12/25) | **72.0% (18/25)** |
| wrong note on top / tie on top | 10 / 0 | 7 / 0 |
| ranked first, then silenced by the gate | 3 | 0 |
| **false recall** | 1/5 (20%) | **5/5 (100%)** |
| top-1 score when right, min / median | 0.559 / 0.678 | 0.791 / 0.857 |
| top-1 when it must stay silent, median / max | 0.470 / 0.567 | 0.815 / 0.835 |
| RU / EN / `hard` cases passed | 13/24 / 3/6 / 2/11 | 14/24 / 4/6 / 5/11 |
| latency p50 / p95 / max | 59ms / 148ms / 194ms | 18ms / 37ms / 49ms |
### What moved
Ranking got better and got faster. Half the previously-unwinnable `hard` cases now pass (2/11 →
5/11), the guitar note no longer beats the docker-logs note, and the gate stops silencing notes that
already ranked first. The quantized e5 is also ~3x quicker than the fp32 MiniLM it replaces.
### What got worse: the gate is now a no-op
e5 packs every cosine into a narrow high band. Right-note scores start at 0.791; must-stay-silent
scores reach 0.835. **The distributions still overlap, and now they overlap above the gate**, so
0.55 admits everything and false recall goes from 1/5 to 5/5. The sweep:
```
gate 0.500.70: answered 18/25 (72%) false recall 5/5
gate 0.80: answered 17/25 (68%) false recall 4/5
gate 0.90: answered 0/25 ( 0%) false recall 0/5
```
There is no value that keeps real recall and rejects made-up questions — same conclusion as before,
now with a wider band and no room at all. `query_min_score` was left at 0.55 as instructed. **The
recommendation is to leave it there and stop tuning it**: any number under ~0.79 is a no-op and
anything above starts cutting real recall long before it stops the false ones. The fix is a margin
gate (`top1 top2 > δ`), next-steps item 3, which is now the top item.
### The prefixes did not do the work
A control run with both prefixes set to the empty string scored the **same** recall@1 (72%), a
slightly better recall@3 (88%) and the same 5/5 false recall. So on this fixture the gain comes from
the model, not from the `query:` / `passage:` split. The prefixes are kept because they are how e5
was trained and the split is the right shape for the read path, but they are not worth defending on
this evidence — a bigger fixture may say otherwise.
### Stored vectors from the old model are now junk
Cosine between a MiniLM vector and an e5 vector means nothing. Every row already in `notes` and in
the vector memory table was written by the old model, so after this deploy they will score as noise
against a new query. A live database needs every note and fact re-embedded before recall works at
all. Filed as its own task.
## Next steps — ordered by value-to-risk; nothing here is a decision ## Next steps — ordered by value-to-risk; nothing here is a decision
1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config 1. **Swap the embedder to `multilingual-e5-small` with `query:`/`passage:` prefixes.** One config
+35
View File
@@ -40,6 +40,41 @@ model → classifier as failure floor.
Never compare a hash-embedder run to an ONNX one. Never compare a hash-embedder run to an ONNX one.
## Re-measured after the prompt fix
The table above is the **baseline at commit `46259b4`**, kept as-is. The prompt fix (query
tested before fact, plus `repeat_penalty` and a bounded grammar string) was then measured on
an otherwise idle box — no other eval sharing llama-server, so these latencies are real
rather than contention.
| | llm-only (0.8B) | cascade+llm (0.8B) | llm-only, thinking off |
|---|---|---|---|
| **intent-only accuracy** | 48.7% → **61.8%** | 50.0% → **63.2%** | **67.1%** |
| full accuracy (intent+slots+gate) | 23.7% → **38.2%** | 32.9% → **47.4%** | **42.1%** |
| route errors | 2 → **0** | 0 → 0 | **0** |
| p50 / p95 latency | **1.08s / 1.55s** | **1.04s / 1.53s** | **0.93s / 1.41s** |
Three things this run settles:
1. **The prompt fix holds.** An earlier contended run reported 60.5% / 36.8% for llm-only;
the quiet run gives 61.8% / 38.2%. Close enough to call the gain real, and the earlier
run's 4-5s latency figures were contention, not the model.
2. **`query→fact` fell from ×15 to ×7**, and both unparseable replies are gone. Zero route
errors in every LLM configuration.
3. **`note→fact ×4` is real, not noise.** It shows up in the quiet run too. The agent that
wrote the prompt fix suspected its own change might have caused it by pulling assertive
`запиши что…` phrasings toward fact, and that suspicion stands — all five `ru-note-*`
cases now land on fact. Tracked as Vikunja #375.
**Thinking off is the best configuration measured so far**, on both accuracy and latency
(Vikunja #376). That is worth understanding before flipping: routing is a short
classification into a fixed enum with grammar-constrained output, so there is little to
reason about, and the thinking trace mostly gives a small model room to talk itself out of
the right answer. Phrasing is a different job and needs measuring separately.
Still `6 / 6` missed clarify — the router has no way to say "I don't know" (Vikunja #359).
That is unchanged by anything here.
## Findings ## Findings
### 1. The resident model does route better — 50.0% vs 36.8% ### 1. The resident model does route better — 50.0% vs 36.8%
+3 -3
View File
@@ -559,7 +559,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
// fail the fact write). Facts aren't in the notes table, so this is the // fail the fact write). Facts aren't in the notes table, so this is the
// only recall path for them — "когда я пил воду?" reads back from here. // only recall path for them — "когда я пил воду?" reads back from here.
if h.memStore != nil { if h.memStore != nil {
if vec, err := h.embedder.Embed(ctx, dec.Utterance); err != nil { if vec, err := router.EmbedPassage(ctx, h.embedder, dec.Utterance); err != nil {
log.Printf("voice: embed fact for memory: %v", err) log.Printf("voice: embed fact for memory: %v", err)
} else if err := h.memStore.Insert(ctx, "fact:"+dec.Slots.Key+":"+strconv.FormatInt(now.Unix(), 10), vec, map[string]string{ } else if err := h.memStore.Insert(ctx, "fact:"+dec.Slots.Key+":"+strconv.FormatInt(now.Unix(), 10), vec, map[string]string{
"source": "voice", "source": "voice",
@@ -673,7 +673,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
// embed the note text with the same model the classifier uses, persist // embed the note text with the same model the classifier uses, persist
// via CoreAPI (source=tap:voice). Semantic recall lives in `notes`, not // via CoreAPI (source=tap:voice). Semantic recall lives in `notes`, not
// facts — no predicate reads it (spec's two-memory split). // facts — no predicate reads it (spec's two-memory split).
vec, err := h.embedder.Embed(ctx, dec.Utterance) vec, err := router.EmbedPassage(ctx, h.embedder, dec.Utterance)
if err != nil { if err != nil {
log.Printf("voice: embed note: %v", err) log.Printf("voice: embed note: %v", err)
return "не получилось сохранить заметку." return "не получилось сохранить заметку."
@@ -750,7 +750,7 @@ func (h *reactiveHandler) applyAction(ctx context.Context, dec router.Decision)
return fmt.Sprintf("в %s сейчас %.0f градусов, %s.", w.Location, w.Temperature, w.Condition) return fmt.Sprintf("в %s сейчас %.0f градусов, %s.", w.Location, w.Temperature, w.Condition)
} }
vec, err := h.embedder.Embed(ctx, dec.Utterance) vec, err := router.EmbedQuery(ctx, h.embedder, dec.Utterance)
if err != nil { if err != nil {
log.Printf("voice: embed query: %v", err) log.Printf("voice: embed query: %v", err)
return "не получилось найти ответ." return "не получилось найти ответ."
+2 -2
View File
@@ -36,8 +36,8 @@
"stt": { "socket": "/run/maven/stt.sock", "lang": "ru" }, "stt": { "socket": "/run/maven/stt.sock", "lang": "ru" },
"tts": { "socket": "/run/maven/tts.sock", "lang": "ru" }, "tts": { "socket": "/run/maven/tts.sock", "lang": "ru" },
"embedder": { "embedder": {
"model_path": "/opt/maven/models/embedder/model.onnx", "model_path": "/opt/maven/models/embedder/multilingual-e5-small/model_quantized.onnx",
"tokenizer_path": "/opt/maven/models/embedder/tokenizer.json", "tokenizer_path": "/opt/maven/models/embedder/multilingual-e5-small/tokenizer.json",
"lib_path": "/opt/maven/lib/libonnxruntime.so" "lib_path": "/opt/maven/lib/libonnxruntime.so"
}, },
"llm_router": false, "llm_router": false,
+7 -7
View File
@@ -262,13 +262,13 @@ type VoiceConfig struct {
// the model gets 50.0% of intents right against the classifier's 36.8%, but // the model gets 50.0% of intents right against the classifier's 36.8%, but
// it costs about 800ms per turn instead of 30ms. // it costs about 800ms per turn instead of 30ms.
// //
// TODO: the default stays false until two things land. // TODO: the default stays false until this lands.
// 1. The LLM router cannot refuse. LLMRouter.Route hardcodes // Extractor.Extract never runs on an LLM decision, so acts arrive with no
// Confidence: 1.0, so the stage-3 clarify gate never fires and an // Fn and reminders with no Time. Turning this on today makes routing more
// unclear utterance becomes a confident wrong action (Vikunja #359). // accurate and less safe.
// 2. Extractor.Extract never runs on an LLM decision, so acts arrive with //
// no Fn and reminders with no Time. // The router can now refuse: it answers "unknown" when it cannot route, and
// Turning this on today makes routing more accurate and less safe. // the turn drops to the classifier and its clarify gate (Vikunja #359).
LLMRouter bool `json:"llm_router,omitempty"` LLMRouter bool `json:"llm_router,omitempty"`
// QueryMinScore — the note-recall confidence gate. Top cosine below this // QueryMinScore — the note-recall confidence gate. Top cosine below this
+27 -5
View File
@@ -117,18 +117,40 @@ type cachingEmbedder struct {
seen map[string][]float32 seen map[string][]float32
} }
var _ router.AsymmetricEmbedder = (*cachingEmbedder)(nil)
func (c *cachingEmbedder) Dim() int { return c.inner.Dim() } func (c *cachingEmbedder) Dim() int { return c.inner.Dim() }
func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner func (c *cachingEmbedder) Close() error { return nil } // the caller owns inner
func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) { func (c *cachingEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
if v, ok := c.seen[text]; ok { return c.cached(ctx, "embed:"+text, func() ([]float32, error) {
return c.inner.Embed(ctx, text)
})
}
// The two sides of an asymmetric embedder give different vectors for the same
// string, so the cache key has to say which side asked.
func (c *cachingEmbedder) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
return c.cached(ctx, "query:"+text, func() ([]float32, error) {
return router.EmbedQuery(ctx, c.inner, text)
})
}
func (c *cachingEmbedder) EmbedPassage(ctx context.Context, text string) ([]float32, error) {
return c.cached(ctx, "passage:"+text, func() ([]float32, error) {
return router.EmbedPassage(ctx, c.inner, text)
})
}
func (c *cachingEmbedder) cached(_ context.Context, key string, embed func() ([]float32, error)) ([]float32, error) {
if v, ok := c.seen[key]; ok {
return v, nil return v, nil
} }
v, err := c.inner.Embed(ctx, text) v, err := embed()
if err != nil { if err != nil {
return nil, err return nil, err
} }
c.seen[text] = v c.seen[key] = v
return v, nil return v, nil
} }
@@ -305,7 +327,7 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
all := append(append([]StoredNote(nil), c.Notes...), filler...) all := append(append([]StoredNote(nil), c.Notes...), filler...)
for _, n := range all { for _, n := range all {
vec, err := emb.Embed(ctx, n.Text) vec, err := router.EmbedPassage(ctx, emb, n.Text)
if err != nil { if err != nil {
return Outcome{}, fmt.Errorf("%s: embed note %s: %w", c.ID, n.ID, err) return Outcome{}, fmt.Errorf("%s: embed note %s: %w", c.ID, n.ID, err)
} }
@@ -317,7 +339,7 @@ func scoreCase(ctx context.Context, emb router.Embedder, newStore NewStore, minS
o := Outcome{Case: c} o := Outcome{Case: c}
start := time.Now() start := time.Now()
qvec, err := emb.Embed(ctx, c.Query) qvec, err := router.EmbedQuery(ctx, emb, c.Query)
if err != nil { if err != nil {
o.Latency = time.Since(start) o.Latency = time.Since(start)
o.Err = err o.Err = err
@@ -246,8 +246,8 @@ func TestONNXRecall(t *testing.T) {
if lib == "" { if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing") t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
} }
model := filepath.Join("../../..", "models/embedder/model.onnx") model := filepath.Join("../../..", "models/embedder/multilingual-e5-small/model_quantized.onnx")
tok := filepath.Join("../../..", "models/embedder/tokenizer.json") tok := filepath.Join("../../..", "models/embedder/multilingual-e5-small/tokenizer.json")
for _, p := range []string{lib, model, tok} { for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil { if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err) t.Skipf("missing %s: %v", p, err)
+287
View File
@@ -0,0 +1,287 @@
package eval
import (
"fmt"
"regexp"
"strings"
"unicode"
"unicode/utf8"
)
// The check names, in report order. Every check is a string or length test — no
// model grades another model here.
const (
CheckMood = "mood" // mood is in the documented enum
CheckLang = "lang" // the operator's language, not the prompt's
CheckLength = "length" // a nudge is one sentence, not a paragraph
CheckFeminine = "feminine" // her self-reference is feminine (hard constraint)
CheckCringe = "cringe" // DESIGN.md § Non-goals, "not a relationship"
CheckOnTopic = "ontopic" // says the thing the rule is about
)
// CheckNames — report order.
var CheckNames = []string{CheckMood, CheckLang, CheckLength, CheckFeminine, CheckCringe, CheckOnTopic}
// Result — one check on one message.
type Result struct {
Name string
Pass bool
Detail string
}
// Moods — the fixed enum from the LLM output contract. Not extended here; the
// contract lives in the daemon and the harness only reads it.
var Moods = map[string]bool{
"neutral": true, "happy": true, "thinking": true, "tired": true, "confused": true,
}
// Length ceilings. Justification: the nudge is spoken by piper at roughly 14
// characters per second, so 120 characters is about 8 seconds of speech. The
// operator has AuDHD — past one short sentence a nudge stops being a nudge and
// becomes something to tune out, which is exactly the "not a nag" failure. The
// word ceiling catches the same thing for languages that pack more per byte.
const (
MaxChars = 120
MaxWords = 16
)
// RunChecks scores one message. Order matches CheckNames.
func RunChecks(c Case, body, mood string) []Result {
return []Result{
checkMood(mood),
checkLang(body),
checkLength(body),
checkFeminine(body),
checkCringe(body),
checkOnTopic(c, body),
}
}
func checkMood(mood string) Result {
if Moods[mood] {
return Result{CheckMood, true, ""}
}
return Result{CheckMood, false, fmt.Sprintf("mood %q not in the enum", mood)}
}
// checkLang — the operator is Russian-speaking and the nudge is spoken aloud by
// a Russian piper voice. An English nudge is not a tone problem, it is an
// unusable one.
func checkLang(body string) Result {
cyr, lat := 0, 0
for _, r := range body {
switch {
case unicode.Is(unicode.Cyrillic, r):
cyr++
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z':
lat++
}
}
if cyr > lat {
return Result{CheckLang, true, ""}
}
return Result{CheckLang, false, fmt.Sprintf("not Russian (%d cyrillic vs %d latin letters)", cyr, lat)}
}
func checkLength(body string) Result {
chars := utf8.RuneCountInString(body)
words := len(strings.Fields(body))
if chars <= MaxChars && words <= MaxWords {
return Result{CheckLength, true, ""}
}
return Result{CheckLength, false,
fmt.Sprintf("%d chars / %d words, ceiling %d / %d", chars, words, MaxChars, MaxWords)}
}
// --- feminine self-reference ---------------------------------------------
//
// The hard constraint (CLAUDE.md, DESIGN.md § Identity): Maven's Russian
// self-reference is feminine. The operator is male, so second-person forms
// addressed to him are MASCULINE and must not be flagged — "ты не пил воду" is
// correct, "я напомнил" is not. Both directions matter, which is why this is a
// windowed scan around "я" and not a bare search for masculine endings.
var wordRE = regexp.MustCompile(`[\p{Cyrillic}]+|[,.;:!?…—-]`)
// secondPerson — pronouns that end the self-reference window. Everything after
// one of these is about him, not about her.
var secondPerson = map[string]bool{
"ты": true, "тебе": true, "тебя": true, "тобой": true,
"вы": true, "вам": true, "вас": true,
"он": true, "она": true, "оно": true, "они": true,
}
// masculinePredicative — short adjectives with no verb ending to key off.
var masculinePredicative = map[string]bool{
"должен": true, "готов": true, "рад": true, "уверен": true,
"обязан": true, "сам": true, "занят": true, "прав": true,
}
// nounsEndingInL — the false positives of "ends in л ⇒ masculine past tense".
// Small on purpose: it only has to cover nouns a nudge might actually use.
var nounsEndingInL = map[string]bool{
"стол": true, "стул": true, "пол": true, "зал": true, "гол": true,
"узел": true, "отдел": true, "файл": true, "канал": true, "угол": true,
"футбол": true, "вокзал": true, "металл": true, "интервал": true,
"уровень": true, "мускул": true, "апрель": true, "июль": true, "рубль": true,
}
// masculinePast reports whether a word looks like a masculine past-tense verb.
// Russian past tense is gendered by suffix: -л (m), -ла (f). A 0.8B with weak
// Russian defaults to the masculine form, which is the exact drift being
// measured.
func masculinePast(w string) bool {
if len([]rune(w)) < 3 || nounsEndingInL[w] {
return false
}
return strings.HasSuffix(w, "л") || strings.HasSuffix(w, "лся")
}
func checkFeminine(body string) Result {
words := wordRE.FindAllString(strings.ToLower(body), -1)
for i, w := range words {
if w != "я" {
continue
}
// Scan the next few words. Stop at punctuation or at a second-person
// pronoun: past that point the sentence is about him and masculine is
// correct.
for j := i + 1; j < len(words) && j <= i+3; j++ {
nw := words[j]
if len(nw) == 1 && !unicode.Is(unicode.Cyrillic, []rune(nw)[0]) {
break
}
if secondPerson[nw] {
break
}
if masculinePast(nw) || masculinePredicative[nw] {
return Result{CheckFeminine, false,
fmt.Sprintf("masculine self-reference %q after \"я\"", nw)}
}
}
}
// Second pass: self-reference with the pronoun dropped — "напомнил тебе",
// "проверил за тебя". A masculine past-tense verb whose object is HIM can
// only be her speaking about herself.
for i, w := range words {
if !masculinePast(w) || i+1 >= len(words) {
continue
}
next := words[i+1]
if next == "тебе" || next == "тебя" || next == "за" {
return Result{CheckFeminine, false,
fmt.Sprintf("masculine self-reference %q before %q", w, next)}
}
}
return Result{CheckFeminine, true, ""}
}
// --- the cringe checks ---------------------------------------------------
//
// "Think Jarvis without the cringe part". DESIGN.md § Non-goals: "Not a
// relationship — mom-tone is a function that makes nudges land, not emotional
// company. Names the drift a warm small model falls into." Each pattern below
// is one shape of that drift. They are deliberately specific: a check that
// flags any warmth at all would make the nudges robotic, which is the other
// failure.
type cringePattern struct {
// what the pattern is defending against, shown in the failure detail.
why string
pat *regexp.Regexp
}
var cringePatterns = []cringePattern{
{
// Endearments. "Not a relationship" — a pet name reframes a nudge as
// intimacy, and the operator asked for mother-like, not girlfriend-like.
why: "pet name / endearment",
// No \b around the Russian alternatives: Go's RE2 \b is ASCII-only and
// never matches at a Cyrillic boundary, so anchoring them would make
// this check silently always pass.
pat: regexp.MustCompile(`(?i)(милый|дорогой|солнышко|солнце моё|солнце мое|зайчик|котик|сладкий|малыш|дружок|родной|любимый|\bhoney\b|\bsweetie\b|\bdarling\b|\bbuddy\b)`),
},
{
// Emoji. The nudge is spoken aloud; an emoji is either silence or a TTS
// artefact. Also the single loudest cringe signal in a small model.
why: "emoji",
pat: nil, // handled by hasEmoji, ranges don't fit a regexp cleanly
},
{
// Exclamation pileup. One "!" is emphasis; two is a cheerleader.
why: "more than one exclamation mark",
pat: regexp.MustCompile(`!.*!|!!`),
},
{
// Fake concern. She has no feelings to report, and reporting them makes
// the nudge about her instead of about the water.
why: "fake concern opener",
pat: regexp.MustCompile(`(?i)(я волну|я беспоко|беспокоюсь|переживаю|я забочусь|я тревож|мне тревожно|i'?m worried)`),
},
{
// Apologising. The rule decided she speaks. Apologising for a greenlit
// nudge undermines the one thing that makes nudges land.
why: "apology",
pat: regexp.MustCompile(`(?i)(извини|прости|сожалею|прошу прощения|не хочу мешать|не хочу отвлекать|sorry|apolog)`),
},
{
// Offering emotional support. The explicit "not emotional company" line.
why: "offer of emotional support",
pat: regexp.MustCompile(`(?i)(я рядом|я здесь для теб|ты не один|всё будет хорошо|все будет хорошо|не переживай|я с тобой|обнимаю|я поддерж|держись)`),
},
{
// Asking how he feels. Turns a one-way nudge into a conversation he now
// owes an answer to — the most reliable way to make him mute it.
why: "asking how he feels",
pat: regexp.MustCompile(`(?i)(как ты\s*[?.!]|как ты себя|как самочувств|как настроение|всё ли в порядке|все ли в порядке|ты в порядке|how are you)`),
},
{
// Praise for compliance. Rewards make the nudge a training exercise;
// "not a relationship" again, from the other side.
why: "praise / reward framing",
pat: regexp.MustCompile(`(?i)(молодец|умница|ты справ|гордюсь|горжусь|отличная работа|так держать|good job|proud of you)`),
},
}
// hasEmoji — the pictographic ranges plus the variation selector. Cyrillic and
// ordinary punctuation are far below all of these.
func hasEmoji(s string) bool {
for _, r := range s {
switch {
case r >= 0x1F000 && r <= 0x1FAFF,
r >= 0x2600 && r <= 0x27BF,
r >= 0x2B00 && r <= 0x2BFF,
r == 0xFE0F, r == 0x203C, r == 0x2049:
return true
}
}
return false
}
func checkCringe(body string) Result {
for _, c := range cringePatterns {
if c.pat == nil {
if hasEmoji(body) {
return Result{CheckCringe, false, c.why}
}
continue
}
if m := c.pat.FindString(body); m != "" {
return Result{CheckCringe, false, fmt.Sprintf("%s (%q)", c.why, m)}
}
}
return Result{CheckCringe, true, ""}
}
// checkOnTopic — the message must name the thing the rule is about. A nudge
// that never mentions water leaves the operator with a chime and no action.
func checkOnTopic(c Case, body string) Result {
low := strings.ToLower(body)
for _, want := range c.WantAny {
if strings.Contains(low, strings.ToLower(want)) {
return Result{CheckOnTopic, true, ""}
}
}
return Result{CheckOnTopic, false,
fmt.Sprintf("mentions none of %v", c.WantAny)}
}
+334
View File
@@ -0,0 +1,334 @@
// Package eval scores nudge phrasing — the sentences the operator actually
// hears. It is the phrasing counterpart to internal/router/eval.
//
// Why a separate package from phraser: the fixture must be scorable by BOTH
// phrasing paths (the deterministic Stub and the resident model) from outside
// the phraser package, and a _test.go file inside phraser cannot be imported.
// So the fixture is embedded here and the scorer takes a Nudger interface.
//
// Why deterministic checks and not model judgement: the resident model is a
// 0.8B. It cannot grade its own tone. Every check in checks.go is a string or
// length test that a human can read and disagree with. A score here is a claim
// about measurable properties, not about whether a sentence is good.
//
// DESIGN.md § "Rules decide, LLM phrases" is why there is no send/veto signal
// anywhere in this package: the rule already decided she speaks. The phraser
// only words it, so a nudge the model refuses to write is a failure, never a
// legitimate outcome.
package eval
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/kami/maven/internal/delivery"
"github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/store"
)
//go:embed nudges_v1.json
var fixtureJSON []byte
// SchemaVersion — the version this package understands.
const SchemaVersion = 1
// Case — one nudge situation, as a real tick would present it. The fields are
// the (rule, severity, context) input DESIGN.md names, flattened to JSON.
//
// WantAny is the on-topic contract: at least one of these lowercased fragments
// must appear in the message. A water nudge that never mentions water is a
// failure however charming it reads. Fragments are stems ("вод") so declension
// does not defeat the check, and they list both languages because the Stub is
// still English (see the writeup).
type Case struct {
ID string `json:"id"`
Rule string `json:"rule"`
Severity int `json:"severity"`
// SinceMinutes — age of the rule's own fact. 0 means "no such fact", which
// is the branch where the phraser has no duration to name.
SinceMinutes int `json:"since_minutes"`
// FactKey/FactValue/FactSource — the aggregate fact behind the ops rules.
// service_down phrasing reads the key for the service name.
FactKey string `json:"fact_key,omitempty"`
FactValue string `json:"fact_value,omitempty"`
FactSource string `json:"fact_source,omitempty"`
// QuietHours/CalendarBusy — the bad moments. The gate already let this
// nudge through (ops outranks quiet hours), so the phrasing still has to be
// short and plain rather than apologetic about the timing.
QuietHours bool `json:"quiet_hours,omitempty"`
CalendarBusy bool `json:"calendar_busy,omitempty"`
WantAny []string `json:"want_any"`
Tags []string `json:"tags,omitempty"`
Note string `json:"note,omitempty"`
}
// Fixture — the versioned envelope. SchemaVersion gates the loader so an older
// binary refuses a fixture it would misread instead of reporting a wrong score.
type Fixture struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
ReferenceNow string `json:"reference_now"`
Notes []string `json:"notes"`
Cases []Case `json:"cases"`
}
// Load returns the embedded fixture.
func Load() (Fixture, error) {
var f Fixture
if err := json.Unmarshal(fixtureJSON, &f); err != nil {
return Fixture{}, fmt.Errorf("parse fixture: %w", err)
}
if f.SchemaVersion != SchemaVersion {
return Fixture{}, fmt.Errorf("fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
}
if len(f.Cases) == 0 {
return Fixture{}, fmt.Errorf("fixture has no cases")
}
return f, nil
}
// Now — the fixture's reference clock, so fact ages are reproducible.
func (f Fixture) Now() (time.Time, error) {
t, err := time.Parse(time.RFC3339, f.ReferenceNow)
if err != nil {
return time.Time{}, fmt.Errorf("parse reference_now %q: %w", f.ReferenceNow, err)
}
return t, nil
}
// Candidate rebuilds the loop.Candidate a tick would hand the phraser.
func (c Case) Candidate(now time.Time) loop.Candidate {
state := loop.State{
Now: now,
Facts: map[string]store.Fact{},
QuietHours: c.QuietHours,
CalendarBusy: c.CalendarBusy,
}
if c.SinceMinutes > 0 {
state.Facts[c.Rule] = store.Fact{
Key: c.Rule,
Ts: now.Add(-time.Duration(c.SinceMinutes) * time.Minute),
}
}
if c.FactKey != "" {
state.Facts[c.Rule] = store.Fact{
Key: c.FactKey,
Value: c.FactValue,
Source: c.FactSource,
Ts: now.Add(-time.Duration(c.SinceMinutes) * time.Minute),
}
}
sev := loop.Severity(c.Severity)
return loop.Candidate{
Rule: loop.Rule{Name: c.Rule, Severity: sev},
Severity: sev,
State: state,
}
}
// Nudger — the one thing a phrasing path must do to be scorable. Both
// *phraser.Stub and *phraser.LLMPhraser satisfy it.
type Nudger interface {
PhraseNudge(ctx context.Context, c loop.Candidate) (delivery.PhrasedNudge, error)
}
// Outcome — one scored case. Failed lists the check names that did not pass,
// Reasons the human-readable detail. Failed is empty exactly when Pass is true.
type Outcome struct {
Case Case
Body string
Mood string
Err error
Latency time.Duration
Pass bool
Failed []string
Reasons []string
}
// Report — the aggregate. ByCheck is the useful part: one composite percentage
// hides which property broke, and tuning a prompt needs to know.
type Report struct {
Name string
Total int
Passed int
Errors int
ByCheck map[string]int
ByRule map[string]TagStat
Outcomes []Outcome
P50 time.Duration
P95 time.Duration
Max time.Duration
}
// TagStat — passed/total for one slice of the fixture.
type TagStat struct{ Passed, Total int }
// Accuracy — fraction of cases that passed every check.
func (r Report) Accuracy() float64 {
if r.Total == 0 {
return 0
}
return float64(r.Passed) / float64(r.Total)
}
// Score runs every case through p and aggregates. A phrasing error scores as a
// miss and is counted in Errors — "the model was down" and "the model wrote
// something bad" are different numbers and a prompt change must not be able to
// hide behind the first one.
//
// Latency is wall-clock per PhraseNudge call. On the CPU/iGPU target a nudge
// the model takes a minute to word has already missed its moment.
func Score(ctx context.Context, name string, p Nudger, f Fixture) (Report, error) {
now, err := f.Now()
if err != nil {
return Report{}, err
}
rep := Report{
Name: name,
Total: len(f.Cases),
ByCheck: map[string]int{},
ByRule: map[string]TagStat{},
}
for _, name := range CheckNames {
rep.ByCheck[name] = 0
}
lat := make([]time.Duration, 0, len(f.Cases))
for _, c := range f.Cases {
start := time.Now()
pn, err := p.PhraseNudge(ctx, c.Candidate(now))
o := Outcome{Case: c, Body: pn.Body, Mood: pn.Mood, Err: err, Latency: time.Since(start)}
lat = append(lat, o.Latency)
if err != nil {
rep.Errors++
o.Failed = append(o.Failed, "call")
o.Reasons = append(o.Reasons, fmt.Sprintf("phrase error: %v", err))
} else {
for _, res := range RunChecks(c, pn.Body, pn.Mood) {
if res.Pass {
rep.ByCheck[res.Name]++
continue
}
o.Failed = append(o.Failed, res.Name)
o.Reasons = append(o.Reasons, res.Name+": "+res.Detail)
}
}
o.Pass = len(o.Failed) == 0
if o.Pass {
rep.Passed++
}
bump(rep.ByRule, ruleFamily(c.Rule), o.Pass)
rep.Outcomes = append(rep.Outcomes, o)
}
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
if len(lat) > 0 {
rep.Max = lat[len(lat)-1]
}
return rep, nil
}
// ruleFamily collapses "routine:зарядка" to "routine" so the per-rule table
// stays readable however many routines the operator configures.
func ruleFamily(rule string) string {
if i := strings.IndexByte(rule, ':'); i > 0 {
return rule[:i]
}
return rule
}
func bump(m map[string]TagStat, key string, pass bool) {
if key == "" {
return
}
s := m[key]
s.Total++
if pass {
s.Passed++
}
m[key] = s
}
// percentile — nearest-rank on a pre-sorted slice. No interpolation: with ~15
// samples an interpolated p95 invents a latency no call actually took.
func percentile(sorted []time.Duration, p float64) time.Duration {
if len(sorted) == 0 {
return 0
}
i := int(p * float64(len(sorted)))
if i >= len(sorted) {
i = len(sorted) - 1
}
return sorted[i]
}
// String renders the comparison table — composite score, then per-check so a
// regression names the property it broke, then latency.
func (r Report) String() string {
var b strings.Builder
fmt.Fprintf(&b, "%s: %d/%d cases pass every check (%.1f%%), %d errors\n",
r.Name, r.Passed, r.Total, 100*r.Accuracy(), r.Errors)
for _, name := range CheckNames {
fmt.Fprintf(&b, " %-9s %d/%d\n", name, r.ByCheck[name], r.Total)
}
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
fmt.Fprintf(&b, " by rule: %s\n", renderStats(r.ByRule))
return b.String()
}
// Failures — per-case detail, sorted by ID so two runs diff cleanly.
func (r Report) Failures() string {
var b strings.Builder
for _, o := range r.sorted() {
if o.Pass {
continue
}
fmt.Fprintf(&b, " %s %q\n %s\n", o.Case.ID, o.Body, strings.Join(o.Reasons, "; "))
}
return b.String()
}
// Messages — every generated message verbatim, pass or fail. This is what a
// human reads to judge tone; the score only says which checks fired.
func (r Report) Messages() string {
var b strings.Builder
for _, o := range r.sorted() {
mark := "ok "
if !o.Pass {
mark = "FAIL"
}
fmt.Fprintf(&b, " %s %-22s [%s] %q\n", mark, o.Case.ID, o.Mood, o.Body)
}
return b.String()
}
func (r Report) sorted() []Outcome {
out := append([]Outcome(nil), r.Outcomes...)
sort.Slice(out, func(i, j int) bool { return out[i].Case.ID < out[j].Case.ID })
return out
}
func renderStats(m map[string]TagStat) string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
parts := make([]string, 0, len(keys))
for _, k := range keys {
parts = append(parts, fmt.Sprintf("%s %d/%d", k, m[k].Passed, m[k].Total))
}
return strings.Join(parts, " ")
}
+179
View File
@@ -0,0 +1,179 @@
package eval
import (
"context"
"strings"
"testing"
"github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/phraser"
)
func TestLoadFixture(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
if _, err := f.Now(); err != nil {
t.Fatalf("Now: %v", err)
}
seen := map[string]bool{}
for _, c := range f.Cases {
if seen[c.ID] {
t.Errorf("duplicate case id %q", c.ID)
}
seen[c.ID] = true
if c.Rule == "" || c.Severity < 1 || c.Severity > 4 {
t.Errorf("%s: rule %q severity %d", c.ID, c.Rule, c.Severity)
}
if len(c.WantAny) == 0 {
t.Errorf("%s: no want_any, the on-topic check would always pass", c.ID)
}
}
// Coverage floor: all five loop rules plus both minted families, or the
// fixture measures a subset and the score does not mean what it says.
for _, rule := range []string{"water", "meal", "break", "service_down", "netdata_critical", "routine", "morning"} {
found := false
for _, c := range f.Cases {
if ruleFamily(c.Rule) == rule {
found = true
}
}
if !found {
t.Errorf("no case for rule family %q", rule)
}
}
}
// TestStubBaseline is the CI ratchet: the deterministic Stub, no model, no
// network. The floor is low on purpose — the Stub is English template phrasing,
// so it fails `lang` on every case by construction. The point of the ratchet is
// that the checks keep running and the Stub does not get worse, not that the
// Stub is good.
func TestStubBaseline(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
rep, err := Score(context.Background(), "stub (deterministic floor)", phraser.NewStub(), f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String() + rep.Messages())
if rep.Errors != 0 {
t.Errorf("stub returned %d errors — the deterministic path must never fail", rep.Errors)
}
// Per-check ratchets rather than one composite: the Stub's composite is 0
// (it never passes `lang`), so a composite floor would catch nothing.
floors := map[string]int{
CheckMood: 15,
// 12, not 15: the Stub's `break` template genuinely runs past the
// ceiling ("you've been at your desk for 4 hours without a break — step
// away for a bit." is 76 chars but 16 words). Left failing rather than
// raising the ceiling to hide it.
CheckLength: 12,
CheckFeminine: 15,
CheckCringe: 15,
CheckOnTopic: 12,
}
for name, floor := range floors {
if rep.ByCheck[name] < floor {
t.Errorf("check %s: %d/%d, below ratchet %d — phrasing regressed",
name, rep.ByCheck[name], rep.Total, floor)
}
}
}
// TestChecksCatchWhatTheyClaim — the checks are the measurement, so they get
// their own tests. Without these, a bad regexp would silently make every
// phrasing run look clean.
func TestChecksCatchWhatTheyClaim(t *testing.T) {
water := Case{Rule: "water", WantAny: []string{"вод"}}
cases := []struct {
name string
body string
want string // the check that must fail, "" for a clean message
}{
{"clean", "уже четыре часа без воды — попей.", ""},
{"long", "уже четыре часа без воды, а это довольно много, и вообще пить надо регулярно, иначе будет плохо совсем", CheckLength},
{"english", "you haven't had water in 4 hours, drink something", CheckLang},
{"masculine self", "я напомнил про воду.", CheckFeminine},
{"masculine dropped pronoun", "напомнил тебе про воду.", CheckFeminine},
{"masculine predicative", "я должен сказать: попей воды.", CheckFeminine},
// The other direction: HE is male, so second-person masculine is right.
{"second person masculine ok", "ты не пил воду четыре часа.", ""},
{"feminine self ok", "я заметила: воды не было четыре часа.", ""},
{"pet name", "милый, попей воды.", CheckCringe},
{"emoji", "попей воды 💧", CheckCringe},
{"exclamations", "попей воды!!", CheckCringe},
{"fake concern", "я беспокоюсь: воды не было четыре часа.", CheckCringe},
{"apology", "извини, что отвлекаю — попей воды.", CheckCringe},
{"emotional support", "я рядом, ты не один. попей воды.", CheckCringe},
{"asks how he feels", "как ты себя чувствуешь? попей воды.", CheckCringe},
{"praise", "молодец! теперь попей воды.", CheckCringe},
{"off topic", "пора бы уже что-то сделать.", CheckOnTopic},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
var failed []string
for _, r := range RunChecks(water, tc.body, "neutral") {
if !r.Pass {
failed = append(failed, r.Name+"("+r.Detail+")")
}
}
joined := strings.Join(failed, " ")
switch {
case tc.want == "" && len(failed) > 0:
t.Errorf("clean message flagged: %s", joined)
case tc.want != "" && !strings.Contains(joined, tc.want+"("):
t.Errorf("want %s to fail, got %q", tc.want, joined)
}
})
}
}
func TestMoodCheckUsesTheEnum(t *testing.T) {
if r := checkMood("cheerful"); r.Pass {
t.Error("mood outside the enum passed")
}
if r := checkMood(""); r.Pass {
t.Error("empty mood passed")
}
for m := range Moods {
if r := checkMood(m); !r.Pass {
t.Errorf("enum mood %q failed", m)
}
}
}
// TestCandidateCarriesTheContext — the whole harness is worthless if the
// Candidate it builds does not carry the duration the prompt is supposed to
// name.
func TestCandidateCarriesTheContext(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
now, _ := f.Now()
for _, c := range f.Cases {
cand := c.Candidate(now)
if cand.Rule.Name != c.Rule || cand.Severity != loop.Severity(c.Severity) {
t.Errorf("%s: candidate lost rule or severity", c.ID)
}
if c.SinceMinutes > 0 {
d, ok := cand.State.Since(c.Rule)
if !ok || int(d.Minutes()) != c.SinceMinutes {
t.Errorf("%s: since %v ok=%v, want %d minutes", c.ID, d, ok, c.SinceMinutes)
}
}
if c.FactKey != "" {
fact, ok := cand.State.Fact(c.Rule)
if !ok || fact.Key != c.FactKey {
t.Errorf("%s: fact key %q, want %q", c.ID, fact.Key, c.FactKey)
}
}
}
}
+73
View File
@@ -0,0 +1,73 @@
package eval
import (
"context"
"os"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/phraser"
)
// TestLLMPhrasingBaseline — the resident model wording real nudges. Opt-in,
// same shape as internal/router/eval's MAVEN_LLM_URL gate, because CI has no
// model and a phrasing run costs minutes on the CPU target.
//
// llama-server -m /mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf \
// --host 127.0.0.1 --port 18099 -c 2048 -ngl 99
// MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
//
// It reports and does not assert a quality bar. The numbers are the input to
// tuning the persona prompt; an assertion here would be the test inventing the
// bar rather than measuring against it. The one thing worth failing on is a
// harness fault — every case erroring means the run measured infrastructure.
func TestLLMPhrasingBaseline(t *testing.T) {
base := os.Getenv("MAVEN_LLM_URL")
if base == "" {
t.Skip("MAVEN_LLM_URL unset — point it at a running llama-server (see doc comment)")
}
// A local llama-server must not go through an HTTP proxy. This box proxies
// loopback through a SOCKS bridge that answers 503, which would score every
// case as a phrasing error and read as "the model cannot phrase".
noProxyLoopback(t)
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
cfg := phraser.DefaultConfig("")
// Generous: an unconstrained 0.8B can spend a minute thinking before it
// writes a word, and a timeout would be scored as a model failure.
cfg.Timeout = 5 * time.Minute
p := phraser.NewLLMPhraserAt(base, cfg)
defer p.Close()
rep, err := Score(context.Background(), "llm (0.8B, built-in persona)", p, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String() + "\nmessages:\n" + rep.Messages() + "\nfailures:\n" + rep.Failures())
if rep.Errors == rep.Total {
t.Errorf("all %d cases errored — harness fault, not a measurement", rep.Total)
}
}
// noProxyLoopback appends the loopback host to no_proxy before any request, so
// http.ProxyFromEnvironment (which caches the environment on first use) sees it.
func noProxyLoopback(t *testing.T) {
t.Helper()
for _, key := range []string{"no_proxy", "NO_PROXY"} {
cur := os.Getenv(key)
if strings.Contains(cur, "127.0.0.1") {
continue
}
if cur == "" {
t.Setenv(key, "127.0.0.1,localhost")
continue
}
t.Setenv(key, cur+",127.0.0.1,localhost")
}
}
+158
View File
@@ -0,0 +1,158 @@
{
"schema_version": 1,
"name": "nudge phrasing v1",
"reference_now": "2026-07-31T21:40:00+03:00",
"notes": [
"The five loop rules (water, meal, break, service_down, netdata_critical) plus one routine: and one morning: case, at the severities they actually ship with.",
"The bad-moment cases (quiet_hours, calendar_busy) are here because the gate already let them through — ops outranks quiet hours. The phrasing must stay short and plain, not apologise for the timing.",
"want_any lists stems, not whole words, so declension does not defeat the on-topic check. English stems are included because the deterministic Stub is still English.",
"There is no expected message. The checks measure properties, not similarity to a reference sentence — a fixed golden string would just freeze one arbitrary phrasing."
],
"cases": [
{
"id": "water-3h",
"rule": "water",
"severity": 1,
"since_minutes": 190,
"want_any": ["вод", "попей", "пить", "выпей", "напит", "water", "drink"],
"tags": ["care"],
"note": "The base case. Just over the 3h predicate."
},
{
"id": "water-7h",
"rule": "water",
"severity": 1,
"since_minutes": 430,
"want_any": ["вод", "попей", "пить", "выпей", "напит", "water", "drink"],
"tags": ["care"],
"note": "Long overdue. Severity is unchanged, so the phrasing must not escalate into alarm."
},
{
"id": "water-busy",
"rule": "water",
"severity": 1,
"since_minutes": 240,
"calendar_busy": true,
"want_any": ["вод", "попей", "пить", "выпей", "напит", "water", "drink"],
"tags": ["care", "bad-moment"],
"note": "Mid-meeting. A bad moment invites an apology, which is the check that should catch it."
},
{
"id": "meal-7h",
"rule": "meal",
"severity": 1,
"since_minutes": 420,
"want_any": ["ешь", "еда", "еды", "поешь", "перекус", "обед", "ужин", "покуш", "food", "eat"],
"tags": ["care"]
},
{
"id": "meal-11h-quiet",
"rule": "meal",
"severity": 1,
"since_minutes": 660,
"quiet_hours": true,
"want_any": ["ешь", "еда", "еды", "поешь", "перекус", "обед", "ужин", "покуш", "food", "eat"],
"tags": ["care", "bad-moment"],
"note": "Quiet hours suppresses care nudges in the gate, so this one only reaches the phraser via an explicit override. Included because it is the shape most likely to draw a hedge."
},
{
"id": "break-90m",
"rule": "break",
"severity": 2,
"since_minutes": 95,
"want_any": ["перерыв", "разомн", "встань", "отдохн", "пауз", "отойд", "размин", "break", "step away"],
"tags": ["care"]
},
{
"id": "break-4h",
"rule": "break",
"severity": 2,
"since_minutes": 240,
"want_any": ["перерыв", "разомн", "встань", "отдохн", "пауз", "отойд", "размин", "break", "step away"],
"tags": ["care"]
},
{
"id": "break-busy",
"rule": "break",
"severity": 2,
"since_minutes": 150,
"calendar_busy": true,
"want_any": ["перерыв", "разомн", "встань", "отдохн", "пауз", "отойд", "размин", "break", "step away"],
"tags": ["care", "bad-moment"]
},
{
"id": "service-down",
"rule": "service_down",
"severity": 4,
"since_minutes": 3,
"fact_key": "vaultwarden",
"fact_value": "\"down\"",
"fact_source": "poll:uptimekuma",
"want_any": ["vaultwarden", "сервис", "упал", "не отвеч", "лежит", "недоступ", "down", "service"],
"tags": ["ops"],
"note": "The service name is in the fact key, not the value. A nudge that says 'a service' without naming it is on-topic but useless — the on-topic check cannot catch that, a human reading Messages() can."
},
{
"id": "service-down-night",
"rule": "service_down",
"severity": 4,
"since_minutes": 2,
"quiet_hours": true,
"fact_key": "nextcloud",
"fact_value": "\"down\"",
"fact_source": "poll:uptimekuma",
"want_any": ["nextcloud", "сервис", "упал", "не отвеч", "лежит", "недоступ", "down", "service"],
"tags": ["ops", "bad-moment"],
"note": "03:00-shaped. Sev4 outranks quiet hours by design, so she speaks — plainly, without softening it into a maybe."
},
{
"id": "netdata-disk",
"rule": "netdata_critical",
"severity": 3,
"since_minutes": 5,
"fact_key": "netdata_alarm",
"fact_value": "\"critical\"",
"fact_source": "poll:netdata",
"want_any": ["netdata", "диск", "критич", "алярм", "аларм", "тревог", "место", "памят", "critical", "alarm", "disk"],
"tags": ["ops"]
},
{
"id": "netdata-busy",
"rule": "netdata_critical",
"severity": 3,
"since_minutes": 12,
"calendar_busy": true,
"fact_key": "netdata_alarm",
"fact_value": "\"critical\"",
"fact_source": "poll:netdata",
"want_any": ["netdata", "диск", "критич", "алярм", "аларм", "тревог", "место", "памят", "critical", "alarm", "disk"],
"tags": ["ops", "bad-moment"]
},
{
"id": "routine-pills",
"rule": "routine:таблетки",
"severity": 2,
"since_minutes": 0,
"want_any": ["таблетк", "приня", "лекарств", "pill"],
"tags": ["routine"],
"note": "A routine: rule has no fact of its own, so there is no duration to name. The rule name is the only context."
},
{
"id": "routine-stretch",
"rule": "routine:зарядка",
"severity": 1,
"since_minutes": 0,
"want_any": ["зарядк", "размин", "упражн", "потянис", "разомн", "stretch", "exercise"],
"tags": ["routine"]
},
{
"id": "morning-checklist",
"rule": "morning:утро",
"severity": 2,
"since_minutes": 0,
"want_any": ["утр", "чеклист", "список", "не сделан", "осталось", "morning"],
"tags": ["routine"],
"note": "In production cmd/mavend/tick.go phrases morning routines deterministically and never calls the LLM. Scored anyway: the phraser is reachable with this rule name, and a fallback that garbles it is still a bug."
}
]
}
+16
View File
@@ -67,6 +67,22 @@ func NewLLMPhraser(ctx context.Context, cfg Config) (*LLMPhraser, error) {
return p, nil return p, nil
} }
// NewLLMPhraserAt wires a phraser to a llama-server that someone else started
// and owns. It spawns nothing, so Close does not kill anything.
//
// This exists for the phrasing scorer (internal/phraser/eval), which must
// measure the phrasing against a shared llama-server without taking the model
// load hit per run or killing a server another process depends on. The daemon
// still uses NewLLMPhraser and still owns its own child process.
func NewLLMPhraserAt(baseURL string, cfg Config) *LLMPhraser {
return &LLMPhraser{
cfg: cfg,
client: &http.Client{Timeout: cfg.Timeout},
port: strings.TrimSuffix(baseURL, "/"),
cancel: func() {},
}
}
func (p *LLMPhraser) start(ctx context.Context) error { func (p *LLMPhraser) start(ctx context.Context) error {
args := []string{ args := []string{
"-m", p.cfg.ModelPath, "-m", p.cfg.ModelPath,
+5 -1
View File
@@ -92,7 +92,11 @@ func (s *Stub) Close() error { return nil }
// the predicate fire (the same State the predicate saw). // the predicate fire (the same State the predicate saw).
func (s *Stub) PhraseNudge(_ context.Context, c loop.Candidate) (delivery.PhrasedNudge, error) { func (s *Stub) PhraseNudge(_ context.Context, c loop.Candidate) (delivery.PhrasedNudge, error) {
body, summary := phraseNudge(c) body, summary := phraseNudge(c)
return delivery.PhrasedNudge{Candidate: c, Body: body, Summary: summary}, nil // "neutral" rather than empty: Mood is part of the documented output
// contract and the Stub is a production fallback, so it must satisfy the
// contract too. Template phrasing has no tone to report, and neutral is the
// enum's own default.
return delivery.PhrasedNudge{Candidate: c, Body: body, Summary: summary, Mood: "neutral"}, nil
} }
// PhraseReminder — extracts the user's text from the reminder payload (raw // PhraseReminder — extracts the user's text from the reminder payload (raw
+31
View File
@@ -19,6 +19,37 @@ type Embedder interface {
Close() error Close() error
} }
// AsymmetricEmbedder — an embedder that wants to know whether a text is a
// search query or a stored passage. Recall is asymmetric: a short question
// goes in, a longer note comes out. The e5 family is trained for exactly that
// and needs the side written into the text ("query: " / "passage: ").
//
// Optional on purpose: HashEmbedder has no such notion, so callers go through
// EmbedQuery and EmbedPassage below, which fall back to plain Embed.
type AsymmetricEmbedder interface {
Embedder
EmbedQuery(ctx context.Context, text string) ([]float32, error)
EmbedPassage(ctx context.Context, text string) ([]float32, error)
}
// EmbedQuery embeds text that is being searched WITH — a question.
func EmbedQuery(ctx context.Context, e Embedder, text string) ([]float32, error) {
if a, ok := e.(AsymmetricEmbedder); ok {
return a.EmbedQuery(ctx, text)
}
return e.Embed(ctx, text)
}
// EmbedPassage embeds text that is being searched FOR — a note or a fact on
// its way into the store. Store and lookup must use these two calls, not one
// of them twice, or the asymmetry buys nothing.
func EmbedPassage(ctx context.Context, e Embedder, text string) ([]float32, error) {
if a, ok := e.(AsymmetricEmbedder); ok {
return a.EmbedPassage(ctx, text)
}
return e.Embed(ctx, text)
}
// HashEmbedder — a deterministic bag-of-words embedder used for tests and as a // HashEmbedder — a deterministic bag-of-words embedder used for tests and as a
// non-zero default floor. NOT semantically meaningful across languages; the // non-zero default floor. NOT semantically meaningful across languages; the
// real classifier swaps in the multilingual ONNX model wholesale. // real classifier swaps in the multilingual ONNX model wholesale.
+57
View File
@@ -37,3 +37,60 @@ func TestHashEmbedderCyrillic(t *testing.T) {
t.Fatalf("cosine(shared)=%.3f not > cosine(disjoint)=%.3f", cosine(a, b), cosine(a, c)) t.Fatalf("cosine(shared)=%.3f not > cosine(disjoint)=%.3f", cosine(a, b), cosine(a, c))
} }
} }
// recordingEmbedder — an asymmetric embedder that only remembers which side
// was asked for. Enough to pin the dispatch; real vectors need the model.
type recordingEmbedder struct{ calls []string }
func (r *recordingEmbedder) Dim() int { return 2 }
func (r *recordingEmbedder) Close() error { return nil }
func (r *recordingEmbedder) Embed(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "embed")
return []float32{1, 0}, nil
}
func (r *recordingEmbedder) EmbedQuery(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "query")
return []float32{1, 0}, nil
}
func (r *recordingEmbedder) EmbedPassage(_ context.Context, _ string) ([]float32, error) {
r.calls = append(r.calls, "passage")
return []float32{0, 1}, nil
}
// TestEmbedQueryAndPassageSplit — a question and a stored note must not take
// the same path. If both ended up on the same call the asymmetric model buys
// nothing, which is the whole reason for the swap.
func TestEmbedQueryAndPassageSplit(t *testing.T) {
rec := &recordingEmbedder{}
if _, err := EmbedQuery(context.Background(), rec, "где логи?"); err != nil {
t.Fatalf("EmbedQuery: %v", err)
}
if _, err := EmbedPassage(context.Background(), rec, "логи в /var/log"); err != nil {
t.Fatalf("EmbedPassage: %v", err)
}
if len(rec.calls) != 2 || rec.calls[0] != "query" || rec.calls[1] != "passage" {
t.Errorf("calls %v, want [query passage]", rec.calls)
}
}
// TestEmbedFallsBackToPlainEmbed — HashEmbedder has no sides, so both helpers
// must still work and give the same vector.
func TestEmbedFallsBackToPlainEmbed(t *testing.T) {
h := NewHashEmbedder(64)
q, err := EmbedQuery(context.Background(), h, "text")
if err != nil {
t.Fatalf("EmbedQuery: %v", err)
}
p, err := EmbedPassage(context.Background(), h, "text")
if err != nil {
t.Fatalf("EmbedPassage: %v", err)
}
for i := range q {
if q[i] != p[i] {
t.Fatalf("hash embedder gave two different vectors for the same text")
}
}
}
+2 -2
View File
@@ -185,8 +185,8 @@ func TestONNXBaseline(t *testing.T) {
if lib == "" { if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing") t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
} }
model := filepath.Join("../../..", "models/embedder/model.onnx") model := filepath.Join("../../..", "models/embedder/multilingual-e5-small/model_quantized.onnx")
tok := filepath.Join("../../..", "models/embedder/tokenizer.json") tok := filepath.Join("../../..", "models/embedder/multilingual-e5-small/tokenizer.json")
for _, p := range []string{lib, model, tok} { for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil { if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err) t.Skipf("missing %s: %v", p, err)
+16 -4
View File
@@ -23,7 +23,11 @@ import (
// //
// llama-server -m /mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf \ // llama-server -m /mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf \
// --host 127.0.0.1 --port 18099 -c 2048 -ngl 99 // --host 127.0.0.1 --port 18099 -c 2048 -ngl 99
// MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-router // MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-models
//
// Every report name carries the model llama-server reports over /v1/models, so
// a bake-off across checkpoints (#278, #250) produces tables you can tell
// apart. Point the variable at one server at a time.
// //
// Three configurations, because "the LLM router" is ambiguous and the three // Three configurations, because "the LLM router" is ambiguous and the three
// numbers answer different questions: // numbers answer different questions:
@@ -53,6 +57,14 @@ func TestLLMRouterBaseline(t *testing.T) {
} }
ctx := context.Background() ctx := context.Background()
model, err := ModelID(ctx, base)
if err != nil {
// Not fatal: an unlabelled score is still a score. But say so loudly,
// because an unlabelled row in a bake-off table is worthless.
t.Logf("could not read model id from %s: %v — reports will say %q", base, err, "unknown-model")
model = "unknown-model"
}
t.Logf("scoring model %s at %s", model, base)
lr := router.NewLLMRouter(client) lr := router.NewLLMRouter(client)
// llm-only: the LLM stage in isolation. Route returns (Decision, ok, err); // llm-only: the LLM stage in isolation. Route returns (Decision, ok, err);
@@ -68,7 +80,7 @@ func TestLLMRouterBaseline(t *testing.T) {
} }
return d, nil return d, nil
}) })
repLLM, err := Score(ctx, "llm-only (0.8B, as deployed)", llmOnly, f) repLLM, err := Score(ctx, "llm-only ("+model+", as deployed)", llmOnly, f)
if err != nil { if err != nil {
t.Fatalf("Score llm-only: %v", err) t.Fatalf("Score llm-only: %v", err)
} }
@@ -77,7 +89,7 @@ func TestLLMRouterBaseline(t *testing.T) {
// cascade+llm: stage-0 grammar → LLM → classifier fallback, the wiring #320 // cascade+llm: stage-0 grammar → LLM → classifier fallback, the wiring #320
// proposes. Hash embedder for the fallback so the classifier contribution is // proposes. Hash embedder for the fallback so the classifier contribution is
// the deterministic floor and any lift is attributable to the model. // the deterministic floor and any lift is attributable to the model.
repCascade, err := Score(ctx, "cascade+llm (0.8B) + hash fallback", repCascade, err := Score(ctx, "cascade+llm ("+model+") + hash fallback",
newBaselineRouter(t, router.NewHashEmbedder(1024), lr), f) newBaselineRouter(t, router.NewHashEmbedder(1024), lr), f)
if err != nil { if err != nil {
t.Fatalf("Score cascade: %v", err) t.Fatalf("Score cascade: %v", err)
@@ -96,7 +108,7 @@ func TestLLMRouterBaseline(t *testing.T) {
// either way. Kept so the question stays answered instead of being // either way. Kept so the question stays answered instead of being
// re-asked, and so internal/llm does NOT grow a chat_template_kwargs field // re-asked, and so internal/llm does NOT grow a chat_template_kwargs field
// for a problem that does not exist. // for a problem that does not exist.
repNoThink, err := Score(ctx, "llm-only (0.8B, thinking off) [diagnostic]", repNoThink, err := Score(ctx, "llm-only ("+model+", thinking off) [diagnostic]",
RouterFunc(func(ctx context.Context, u string, now time.Time) (router.Decision, error) { RouterFunc(func(ctx context.Context, u string, now time.Time) (router.Decision, error) {
d, ok, err := router.NewLLMRouter(&noThinkCompleter{base: base, http: &http.Client{Timeout: 60 * time.Second}}).Route(ctx, u, now) d, ok, err := router.NewLLMRouter(&noThinkCompleter{base: base, http: &http.Client{Timeout: 60 * time.Second}}).Route(ctx, u, now)
if err != nil { if err != nil {
+51
View File
@@ -0,0 +1,51 @@
package eval
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
)
// ModelID asks llama-server which model it has loaded, so a scoring run can
// label itself. Without this a bake-off between two models produces two tables
// that look identical, and the operator has to remember which server was up.
//
// Read from the server rather than passed in on purpose: a hand-typed label
// goes stale the moment someone restarts the server with a different -m.
func ModelID(ctx context.Context, base string) (string, error) {
req, err := http.NewRequestWithContext(ctx, "GET", strings.TrimSuffix(base, "/")+"/v1/models", nil)
if err != nil {
return "", err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return "", fmt.Errorf("models: status %d", resp.StatusCode)
}
var out struct {
Data []struct {
ID string `json:"id"`
} `json:"data"`
}
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
return "", err
}
if len(out.Data) == 0 {
return "", fmt.Errorf("models: empty list")
}
return shortModelID(out.Data[0].ID), nil
}
// shortModelID trims the path and the .gguf suffix — llama-server reports the
// file name it was started with, which is too long for a table header.
func shortModelID(id string) string {
if i := strings.LastIndexAny(id, "/\\"); i >= 0 {
id = id[i+1:]
}
return strings.TrimSuffix(id, ".gguf")
}
+33 -3
View File
@@ -29,7 +29,7 @@ func NewLLMRouter(c Completer) *LLMRouter { return &LLMRouter{c: c} }
const routeGrammar = ` const routeGrammar = `
root ::= "[" ws action ("," ws action)* ws "]" root ::= "[" ws action ("," ws action)* ws "]"
action ::= "{" ws "\"intent\"" ws ":" ws intent ("," ws field)* ws "}" action ::= "{" ws "\"intent\"" ws ":" ws intent ("," ws field)* ws "}"
intent ::= "\"fact\"" | "\"reminder\"" | "\"note\"" | "\"query\"" | "\"act\"" | "\"chat\"" | "\"system\"" intent ::= "\"fact\"" | "\"reminder\"" | "\"note\"" | "\"query\"" | "\"act\"" | "\"chat\"" | "\"system\"" | "\"unknown\""
field ::= key ws ":" ws string field ::= key ws ":" ws string
key ::= "\"key\"" | "\"value\"" | "\"text\"" | "\"verb\"" key ::= "\"key\"" | "\"value\"" | "\"text\"" | "\"verb\""
string ::= "\"" ([^"\\] | "\\" .){0,120} "\"" string ::= "\"" ([^"\\] | "\\" .){0,120} "\""
@@ -41,12 +41,17 @@ ws ::= [ \t\n]*
// question naming a fact key ("сколько воды я выпил с утра") matched the fact // question naming a fact key ("сколько воды я выпил с утра") matched the fact
// rule first and was stored as an assertion — 15 of 76 fixture cases. // rule first and was stored as an assertion — 15 of 76 fixture cases.
// //
// Changed again 31-07-2026: added the "unknown" escape hatch so the model can
// admit it cannot route (Vikunja #359).
//
// The training workspace keeps its own copy of this prompt for relabelling, and // The training workspace keeps its own copy of this prompt for relabelling, and
// `llm/check_prompt_parity.py` there compares the two. That copy is in another // `llm/check_prompt_parity.py` there compares the two. That copy is in another
// repo and was not touched, so parity will fail until it gets the same edit. // repo and was not touched, so parity will fail until it gets the same edits —
// both the rule reorder and the "unknown" wording (Vikunja #362).
const routeSystem = `Классифицируй ровно одно сообщение пользователя. Верни ОДИН JSON-массив действий. const routeSystem = `Классифицируй ровно одно сообщение пользователя. Верни ОДИН JSON-массив действий.
Ровно одно намерение: fact, reminder, note, query, act, chat, system. Ровно одно намерение: fact, reminder, note, query, act, chat, system.
Есть восьмое значение unknown — только для случаев, когда просьбу невозможно понять.
Классифицируй по цели пользователя. Порядок решения: Классифицируй по цели пользователя. Порядок решения:
1. Хочет напоминание в будущем → reminder 1. Хочет напоминание в будущем → reminder
@@ -56,12 +61,14 @@ const routeSystem = `Классифицируй ровно одно сообще
5. Утверждает: сообщает или обновляет текущее состояние/событие → fact 5. Утверждает: сообщает или обновляет текущее состояние/событие → fact
6. Просит выполнить работу → act 6. Просит выполнить работу → act
7. Про ассистента, настройки или память → system 7. Про ассистента, настройки или память → system
8. Иначе → chat 8. Реплика — обрывок или указание на неназванное («это», «то», «потом»), и без него непонятно, что именно нужно сделать → unknown
9. Иначе → chat
Различия: Различия:
- note — сохранить информацию, без напоминания. text = суть. - note — сохранить информацию, без напоминания. text = суть.
- reminder — уведомить позже. text = что напомнить. - reminder — уведомить позже. text = что напомнить.
- fact — неявное обновление: пользователь сообщает, что что-то в мире изменилось (текущее/изменённое состояние, случившееся событие). key/value. - fact — неявное обновление: пользователь сообщает, что что-то в мире изменилось (текущее/изменённое состояние, случившееся событие). key/value.
- unknown — редкий случай. Ставь его, только если в самой реплике нет ни предмета, ни действия. Короткая, простая или незнакомая тема — это не причина для unknown: приветствие и болтовня — это chat, вопрос на любую тему — это query, просьба сделать что-то названное — это act.
- query против fact — решает форма реплики, а не тема. Вопрос о состоянии — это query, даже если названо то же самое, что бывает в fact. Только утверждение — это fact. - query против fact — решает форма реплики, а не тема. Вопрос о состоянии — это query, даже если названо то же самое, что бывает в fact. Только утверждение — это fact.
Примеры: Примеры:
@@ -76,6 +83,13 @@ const routeSystem = `Классифицируй ровно одно сообще
"напиши письмо" → {"intent":"act","verb":"написать письмо"} "напиши письмо" → {"intent":"act","verb":"написать письмо"}
"очисти память" → {"intent":"system"} "очисти память" → {"intent":"system"}
"привет" → {"intent":"chat","text":"привет"} "привет" → {"intent":"chat","text":"привет"}
"сделай это" → {"intent":"unknown"}
"ну это" → {"intent":"unknown"}
"потом" → {"intent":"unknown"}
Но не путай — здесь unknown не нужен:
"сделай кофе" → {"intent":"act","verb":"сделать кофе"}
"что такое кватернион?" → {"intent":"query","text":"что такое кватернион"}
"ага" → {"intent":"chat","text":"ага"}
Ответ — JSON-массив: по одному объекту на каждую просьбу. Обычно один. Если в реплике несколько просьб — по объекту на каждую. "напомни купить молоко, и запиши что кофе кончился" → [{"intent":"reminder","text":"купить молоко"},{"intent":"note","text":"кофе кончился"}]. Только JSON, без пояснений.` Ответ — JSON-массив: по одному объекту на каждую просьбу. Обычно один. Если в реплике несколько просьб — по объекту на каждую. "напомни купить молоко, и запиши что кофе кончился" → [{"intent":"reminder","text":"купить молоко"},{"intent":"note","text":"кофе кончился"}]. Только JSON, без пояснений.`
@@ -84,6 +98,11 @@ const routeSystem = `Классифицируй ровно одно сообще
// the loop without hurting short slot values. // the loop without hurting short slot values.
const routeRepeatPenalty = 1.15 const routeRepeatPenalty = 1.15
// routeIntentUnknown — the model's way of saying "I could not route this".
// It is a wire value only: it never becomes a router.Intent, it just makes
// Route return ok=false so the caller drops to the classifier cascade.
const routeIntentUnknown = "unknown"
type routeAction struct { type routeAction struct {
Intent string `json:"intent"` Intent string `json:"intent"`
Key string `json:"key"` Key string `json:"key"`
@@ -92,6 +111,10 @@ type routeAction struct {
Verb string `json:"verb"` Verb string `json:"verb"`
} }
// Route asks the model for one decision. The bool is false when there is no
// decision to use: either the model failed (err set) or it refused with the
// "unknown" intent (err nil). Both mean the same thing to the caller — use the
// classifier instead.
func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time) (Decision, bool, error) { func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time) (Decision, bool, error) {
raw, err := lr.c.Complete(ctx, llm.Req{ raw, err := lr.c.Complete(ctx, llm.Req{
System: routeSystem, System: routeSystem,
@@ -115,6 +138,13 @@ func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time)
// with the engine turn-on (Router.Route → []Decision, both voice.go handlers // with the engine turn-on (Router.Route → []Decision, both voice.go handlers
// loop). Until then only the first ask is honored. // loop). Until then only the first ask is honored.
a := acts[0] a := acts[0]
// The model refused. Report "no decision" without an error, which is the
// same fall-through the caller already uses for a parse failure — the
// classifier cascade gets the turn and its own confidence gate decides
// whether to ask. Better a slower second opinion than a confident guess.
if a.Intent == routeIntentUnknown {
return Decision{}, false, nil
}
d := Decision{Utterance: utterance, Stage: 1, Confidence: 1.0} d := Decision{Utterance: utterance, Stage: 1, Confidence: 1.0}
switch Intent(a.Intent) { switch Intent(a.Intent) {
case IntentFact: case IntentFact:
+58 -1
View File
@@ -100,8 +100,10 @@ func TestLLMRouterReminderMapping(t *testing.T) {
} }
} }
// An intent name that is not in the contract at all (as opposed to "unknown",
// which is a real refusal) still defaults to chat.
func TestLLMRouterChatFallback(t *testing.T) { func TestLLMRouterChatFallback(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`}) lr := NewLLMRouter(mockLLM{out: `{"intent":"banana"}`})
d, ok, err := lr.Route(context.Background(), "как дела?", time.Now()) d, ok, err := lr.Route(context.Background(), "как дела?", time.Now())
if err != nil || !ok { if err != nil || !ok {
t.Fatalf("ok=%v err=%v", ok, err) t.Fatalf("ok=%v err=%v", ok, err)
@@ -111,6 +113,61 @@ func TestLLMRouterChatFallback(t *testing.T) {
} }
} }
// The model must be able to say "I could not route this".
func TestRouteGrammarAllowsUnknown(t *testing.T) {
if !strings.Contains(routeGrammar, `"\"unknown\""`) {
t.Fatal("grammar cannot express a refusal")
}
}
// If the prompt does not tell the model when to refuse, it never will.
func TestRoutePromptExplainsUnknown(t *testing.T) {
if !strings.Contains(routeSystem, "unknown") {
t.Fatal("prompt never mentions the unknown intent")
}
if !strings.Contains(routeSystem, `"сделай это" → {"intent":"unknown"}`) {
t.Fatal("prompt lost its worked refusal example")
}
// A refusal-only router is useless, so the prompt must also show cases that
// look ambiguous but are not.
if !strings.Contains(routeSystem, "здесь unknown не нужен") {
t.Fatal("prompt lost its counter-examples")
}
}
// A refusal is not an error. It reports "no decision" so the cascade moves on.
func TestLLMRouterUnknownRefuses(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`})
_, ok, err := lr.Route(context.Background(), "сделай это", time.Now())
if ok {
t.Fatal("a refusal must not produce a usable decision")
}
if err != nil {
t.Fatalf("a refusal is not an error, got %v", err)
}
}
// The whole point of the refusal: the turn keeps going on the classifier, the
// same way it does when the model returns garbage.
func TestRouterFallsBackWhenLLMRefuses(t *testing.T) {
c := NewClassifier(NewHashEmbedder(1024))
seedClassifier(t, c)
r := New(Config{
Classifier: c,
Extractor: Extractor{Time: StubDateTimeParser{}, Facts: DefaultFactParser{}},
Threshold: 0.4,
LLM: NewLLMRouter(mockLLM{out: `{"intent":"unknown"}`}),
})
d, err := r.Route(context.Background(), "напомни позвонить маме", refNow())
if err != nil {
t.Fatalf("route: %v", err)
}
// Stage 1 is the LLM's own answer; the classifier lands on stage 2 or 3.
if d.Stage < 2 {
t.Fatalf("want the classifier to decide, got stage %d (%+v)", d.Stage, d)
}
}
func TestLLMRouterLLMError(t *testing.T) { func TestLLMRouterLLMError(t *testing.T) {
lr := NewLLMRouter(mockLLM{out: "", err: fmt.Errorf("llm down")}) lr := NewLLMRouter(mockLLM{out: "", err: fmt.Errorf("llm down")})
_, ok, err := lr.Route(context.Background(), "x", time.Now()) _, ok, err := lr.Route(context.Background(), "x", time.Now())
+28 -1
View File
@@ -12,6 +12,15 @@ import (
"golang.org/x/text/unicode/norm" "golang.org/x/text/unicode/norm"
) )
// The deployed model is multilingual-e5-small. e5 was trained with these two
// words glued to the front of every text, and it scores badly without them —
// they are part of the model, not a style choice. Swapping back to a symmetric
// paraphrase model means dropping them again.
const (
queryPrefix = "query: "
passagePrefix = "passage: "
)
const ( const (
padTokenID = 1 padTokenID = 1
unkTokenID = 3 unkTokenID = 3
@@ -54,7 +63,25 @@ func NewONNXEmbedder(modelPath, tokenizerPath, libPath string) (*onnxEmbedder, e
func (e *onnxEmbedder) Dim() int { return embedDim } func (e *onnxEmbedder) Dim() int { return embedDim }
// Embed treats the text as a query. The classifier compares one short
// utterance to another short seed phrase, so both sides get the same prefix
// and the comparison stays fair. The recall path must call EmbedQuery and
// EmbedPassage instead.
func (e *onnxEmbedder) Embed(ctx context.Context, text string) ([]float32, error) { func (e *onnxEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, queryPrefix+text)
}
// EmbedQuery — the question the user just asked.
func (e *onnxEmbedder) EmbedQuery(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, queryPrefix+text)
}
// EmbedPassage — a note or fact being stored, or re-scored at lookup time.
func (e *onnxEmbedder) EmbedPassage(ctx context.Context, text string) ([]float32, error) {
return e.embed(ctx, passagePrefix+text)
}
func (e *onnxEmbedder) embed(ctx context.Context, text string) ([]float32, error) {
inputIDs, attentionMask, _ := e.tokenizer.Encode(text) inputIDs, attentionMask, _ := e.tokenizer.Encode(text)
inputShape := ort.NewShape(1, int64(maxLength)) inputShape := ort.NewShape(1, int64(maxLength))
@@ -313,4 +340,4 @@ func preTokenize(text string) []string {
return out return out
} }
var _ Embedder = (*onnxEmbedder)(nil) var _ AsymmetricEmbedder = (*onnxEmbedder)(nil)