Compare commits

...

4 Commits

Author SHA1 Message Date
kami 0ca5748699 Stop the docs claiming the LLM router is off
CLAUDE.md's routing section said "llmrouter is wired nil" and called the
classifier cascade the committed default. That stopped being true when the
integration merge landed: voice.go:214 wires pickLLMRouter, DefaultLLMRouter is
on, and deploy/mavend.json sets llm_router true. It is the first thing anyone
reads before touching the router, so it was pointing the next reader at a
wiring job that is already done.

Rewritten to say the LLM router is the default, the classifier is the failure
floor and must not be deleted, and what the two actually measure — 36.8% at
p50 31ms against 67.5%/72.7% at p50 ~2.7s, a trade accepted on purpose. Names
the one thing still open on that path: Confidence is hardcoded 1.0 in
llmrouter.go, so the LLM never asks for clarification (#359).

Also in CLAUDE.md: the persona line pointed at a memory file that does not
exist, so the actual rule was nowhere in the repo. Written out instead —
feminine self-reference, informal singular address, pet names forbidden but his
name allowed — plus the three eval checks that enforce it.

MODEL-BAKEOFF: three claims had gone stale within hours of being written. There
IS a make eval-models target now; the routing numbers ARE the production path,
not a bench artifact waiting on a wiring change; and the truncated 293 MB gguf
is deleted. Struck through rather than removed, since the caveats are part of
how the evening read at the time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 22:37:15 +04:00
kami b43bb265b5 Say it the way she would actually say it (#48) 2026-07-31 20:17:27 +02:00
kami b9a24334ea Say it the way she'd actually say it
Wording fixes from the review of the clarify + phrasing PRs.

- "На когда напомнить?" → "Когда?". After she has just been asked something,
  the long form is the phrasing of a form field, not of a person.
- A reminder now wants a subject as well as a time. "напомни в 11" had a time
  and nothing to say at 11, and she asked nothing at all — she now asks
  "О чём напомнить?". Subject first, since a reminder with no subject is not
  worth setting.
- The expiry notice is five phrasings picked at random instead of one fixed
  sentence. It is the line he hears every time he walks off mid-request, so it
  is the line that repeats most.
- The nudge prompt's ban on "обращения" is now "ласковые обращения". It was
  meant to forbid "милый"/"дорогой", not his name — "Ками, ноутбук на трёх
  процентах" is how she talks, and the eval's cringe check already only flags
  pet names.
- The nudge example no longer claims she plugged the laptop in. She has no
  hands and no smart plug; an example where she acts teaches the model to
  invent actions Maven never took.
- replySystem: "тепло" → "спокойно и без официальных формулировок". A one-word
  mood instruction a 1.7B can't act on, replaced with the behaviour meant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 21:52:12 +04:00
kami c97aebf55a Merge tonight's work: all 46 reviewed PRs as one verified branch
135 commits. make build produces all 8 binaries; make test exits 0 across 38 packages with no failures, no data races, gofmt and vet clean.

See PR #47 for what had to be fixed to make it build as a unit.
2026-07-31 19:41:52 +02:00
7 changed files with 115 additions and 30 deletions
+24 -9
View File
@@ -69,19 +69,30 @@ protocol; the config in `deploy/mavend.json` (with `${VAR}` env expansion from g
## Routing — read this before touching the router
`internal/router/` has TWO layered engines and the committed default is an **interim
stopgap, not the intended design** (see memory `routing-architecture-target`):
`internal/router/` has TWO layered engines. **The LLM router is now the default and it is
on in deploy** — this section used to say it was wired `nil`, which stopped being true on
2026-07-31.
- **Target (REARCH.md):** LLM-as-router. One resident Qwen3-1.7B (`llmrouter.go`) emits
GBNF-constrained structured JSON, and the SAME model phrases replies. Embedder is demoted
from a routing gate to a RAG hint.
- **Current stopgap:** `llmrouter` is wired `nil` (around `voice.go`), so the
`classifier.go` + `embedder.go` nearest-neighbour cascade actually runs. It routes by
similarity to frozen seed phrases — the known cause of weak RU query handling.
- **LLM router (the intended design, REARCH.md):** the resident Qwen3-1.7B (`llmrouter.go`)
emits GBNF-constrained structured JSON, and the SAME model phrases replies. Embedder is
demoted from a routing gate to a RAG hint. Wired at `voice.go:214` via
`pickLLMRouter(cfg.Voice.UseLLMRouter(), llmClient)`; the flag is `voice.llm_router`
(`config.go`), `DefaultLLMRouter` is **on**, and `deploy/mavend.json` sets it `true`.
- **Classifier cascade (the failure floor, not dead code):** `classifier.go` +
`embedder.go` nearest-neighbour over frozen seed phrases. It runs when the LLM router is
off, when there is no llama-server to talk to (`pickLLMRouter` logs that and degrades),
and on any per-turn LLM error. Do not delete it — routing by seed similarity is the known
cause of weak RU query handling, but a turn must never break on the model.
Cascade order: `stage0.go` exact-match fast-path → LLM router (when non-nil) → classifier
fallback. Any LLM error falls through to the classifier so a turn never breaks on the model.
Measured on the 77-case RU fixture (`MODEL-BAKEOFF-31-07-2026.md`): the classifier scores
36.8% full accuracy at p50 31ms; Qwen3-1.7B scores 67.5% intent-only / 72.7% through the
cascade at p50 ≈2.7s. Accuracy roughly doubled, latency is ~90× worse, and that trade was
accepted deliberately. Still open: `Confidence: 1.0` is hardcoded in `llmrouter.go`, so the
LLM path never asks for clarification (6/6 refusal cases missed) — Vikunja #359.
## LLM output contract
All phrasing paths emit `{"response":"...","mood":"..."}` (parsed in `replier_llm.go` and
@@ -94,7 +105,11 @@ workspace enforces that the Go and relabelling prompts remain identical.
## Non-goals (hard constraints)
Not a nag, not autonomous. Maven's persona is **feminine** — Russian
self-reference must use feminine forms (the user is male; see memory `maven-persona-gender`).
self-reference must use feminine forms `рада`, not `рад`; `поняла`, not `понял`. The owner
is male and is addressed informally: "ты", singular, never "вы"/"ваш" and never "он"/"его"
(she talks TO him, not about him). Pet names ("милый", "дорогой") are forbidden; his name
("Ками") is not. The eval enforces this: `CheckAddress`, `CheckFeminine` and `CheckCringe` in
`internal/phraser/eval/checks.go`, scored by `make eval-phrasing`.
**"Never phones home" is DEPRECATED** (owner's call, 2026-07-31). It used to be a hard
constraint and it is not one any more: a 0.8B — and a 1.7B — does not know enough to answer
+12 -5
View File
@@ -19,7 +19,10 @@ Settles Vikunja **#278 / #250**.
- Same fixture and scorer as `ROUTING-EVAL-31-07-2026.md`: `internal/router/eval/`
(`ru_routing_v1.json`, 76 held-out cases).
- Reproduce: `MAVEN_LLM_URL=http://127.0.0.1:<port> make eval-router`
(`TestLLMRouterBaseline`). Note: there is no `make eval-models` target.
(`TestLLMRouterBaseline`). (This line used to say there is no `make eval-models` target.
There is one now — start a server with the gguf you want, then
`make eval-models MAVEN_LLM_URL=http://127.0.0.1:<port>`. It runs only the LLM test, since
the classifier baselines do not depend on the model.)
- All three models served by the same `llama-server` flags — `-c 2048 -ngl 99 -t 6`, only
`-m` and `--port` differ. One server at a time on an otherwise idle box, so latencies are
real and not contention.
@@ -207,10 +210,14 @@ swapped again when the CPT lands.
- Routing is one run per model, not three. The gaps between families are far larger
than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3
vs 67.5) is not safe to call on one run.
- The routing numbers only reach production once the LLM router is wired on. It is
still `nil`.
- `/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download
in the wrong directory. The good 1.13 GB copy is in `qwen3/`. Delete the stray one.
- ~~The routing numbers only reach production once the LLM router is wired on. It is
still `nil`.~~ **Resolved the same evening:** the LLM router is wired at `voice.go:214`
behind `voice.llm_router`, the default is on, and `deploy/mavend.json` sets it `true`.
These numbers are the production path now, so the p50 ≈2.7s is a real per-turn cost and
not a bench artifact.
- ~~`/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download
in the wrong directory.~~ **Deleted 2026-07-31.** The good 1.13 GB copy in `qwen3/` is
what `deploy/mavend.json` loads.
- Harness: `scratchpad/bakeoff.sh`, one server at a time, health-checked before each
run, `/v1/models` recorded per run. Never run two LLM consumers at once — see the
contamination note in `TALK-EVAL-31-07-2026.md`.
+57 -6
View File
@@ -3,6 +3,8 @@ package main
import (
"context"
"log"
"math/rand"
"strings"
"time"
"github.com/kami/maven/internal/dialogue"
@@ -21,8 +23,12 @@ const clarifyTTL = 90 * time.Second
// raw utterance, chat and system have nothing to fill in. For those a clarify
// decision keeps the canned "не поняла" reply — inventing a question for noise
// is worse than admitting she missed it.
// A reminder wants BOTH what to remind about and when. Subject first: "напомни
// в 11" has a time and nothing to say at 11, and a reminder with no subject is
// not worth setting. Order here is the order she asks in — she still only asks
// about the first one missing.
var wantedSlots = map[router.Intent][]dialogue.Slot{
router.IntentReminder: {dialogue.SlotTime},
router.IntentReminder: {dialogue.SlotText, dialogue.SlotTime},
router.IntentFact: {dialogue.SlotKey},
router.IntentAct: {dialogue.SlotFn},
}
@@ -35,7 +41,8 @@ var wantedSlots = map[router.Intent][]dialogue.Slot{
// questions, so there is no gender agreement to get wrong; the feminine
// self-reference lives in the reply she gives when she drops the request.
var clarifyQuestions = map[dialogue.Slot]string{
dialogue.SlotTime: "На когда напомнить?",
dialogue.SlotTime: "Когда?",
dialogue.SlotText: "О чём напомнить?",
dialogue.SlotKey: "Что записать?",
dialogue.SlotFn: "Что сделать?",
}
@@ -45,11 +52,55 @@ var clarifyQuestions = map[dialogue.Slot]string{
// landed. Feminine self-reference ("поняла"), as everywhere.
const clarifyGaveUp = "Прости, я не поняла. Скажи, пожалуйста, по-другому."
// clarifyExpired — his answer came after the TTL, so the parked request is
// already gone. Same tone as clarifyGaveUp, different reason: too much time
// clarifyExpiredVariants — his answer came after the TTL, so the parked request
// is already gone. Same tone as clarifyGaveUp, different reason: too much time
// passed, not "I did not understand". Feminine self-reference ("ждала",
// "отпустила"); he is addressed with a plain imperative.
const clarifyExpired = "Прости, я слишком долго ждала ответа и отпустила прошлую просьбу. Если она ещё нужна, скажи заново."
//
// Five phrasings, not one. This is the line he hears whenever he walks off
// mid-request, so it is the line that repeats most — and the same sentence every
// time is what makes a house assistant sound like a kiosk. They all carry the
// same two facts (the old request is gone; say it again if it still matters),
// because the wording may vary and the meaning may not.
//
// Fixed templates rather than model output, for the same reason as
// clarifyQuestions: this text has to be right every time, and it is not worth a
// generation to say something this small.
var clarifyExpiredVariants = []string{
"Прости, я слишком долго ждала ответа и отпустила прошлую просьбу. Если она ещё нужна, скажи заново.",
"Кажется, прошлая просьба уже не важна — я её отпустила. Если я ошибаюсь, повтори.",
"Ты как-то резко замолчал, и я не стала ждать дальше. Если та просьба ещё нужна, скажи заново.",
"Я не дождалась ответа и убрала прошлую просьбу. Повтори, если она всё ещё нужна.",
"Столько времени прошло, что я отпустила прошлую просьбу. Скажи заново, если она в силе.",
}
// clarifyExpiredLine picks one of them at random.
func clarifyExpiredLine() string {
return clarifyExpiredVariants[rand.Intn(len(clarifyExpiredVariants))]
}
// isClarifyExpired reports whether s opens with any of the expiry lines. The
// notice is glued in front of this turn's reply (see withNotice), so a caller
// checking for it has to match a prefix, not the whole string.
func isClarifyExpired(s string) bool {
for _, v := range clarifyExpiredVariants {
if strings.HasPrefix(s, v) {
return true
}
}
return false
}
// trimClarifyExpired strips a leading expiry notice, leaving this turn's actual
// reply. "" ⇒ the notice was the whole thing.
func trimClarifyExpired(s string) string {
for _, v := range clarifyExpiredVariants {
if strings.HasPrefix(s, v) {
return strings.TrimSpace(strings.TrimPrefix(s, v))
}
}
return strings.TrimSpace(s)
}
// clarifyExpiredNotice returns that line when a parked question had just timed
// out, and "" when nothing was parked. Call it right after
@@ -63,7 +114,7 @@ func (h *reactiveHandler) clarifyExpiredNotice() string {
return ""
}
log.Printf("voice: clarify — parked question expired, telling him and routing the words fresh")
return clarifyExpired
return clarifyExpiredLine()
}
// withNotice glues the expiry notice in front of this turn's reply. One turn
+10 -7
View File
@@ -56,10 +56,13 @@ func TestClarifyQuestionForMissingSlot(t *testing.T) {
want string
asked bool
}{
{"reminder without a time", clarifyDec(router.IntentReminder, router.Slots{Text: "напомни позвонить маме"}, "напомни позвонить маме"), "На когда напомнить?", true},
{"reminder without a time", clarifyDec(router.IntentReminder, router.Slots{Text: "напомни позвонить маме"}, "напомни позвонить маме"), "Когда?", true},
{"fact without a key", clarifyDec(router.IntentFact, router.Slots{Text: "запиши"}, "запиши"), "Что записать?", true},
{"act without a fn", clarifyDec(router.IntentAct, router.Slots{Text: "сделай это"}, "сделай это"), "Что сделать?", true},
{"reminder that already has a time", clarifyDec(router.IntentReminder, router.Slots{HasTime: true}, "напомни в 11"), "", false},
// A time with nothing to say at that time is still half a reminder, so
// the subject is what she asks about — not silence.
{"reminder that has a time but no subject", clarifyDec(router.IntentReminder, router.Slots{HasTime: true}, "напомни в 11"), "О чём напомнить?", true},
{"reminder that has both", clarifyDec(router.IntentReminder, router.Slots{Text: "позвонить маме", HasTime: true}, "напомни в 11 позвонить маме"), "", false},
{"chat is never worth a question", clarifyDec(router.IntentChat, router.Slots{Text: "мгм"}, "мгм"), "", false},
{"query is never worth a question", clarifyDec(router.IntentQuery, router.Slots{Text: "а"}, "а"), "", false},
}
@@ -78,7 +81,7 @@ func TestClarifyReminderCompletesOnAnswer(t *testing.T) {
h, st, _ := newClarifyHandler(t)
question, asked := h.askClarify(clarifyDec(router.IntentReminder, router.Slots{Text: "напомни позвонить маме"}, "напомни позвонить маме"))
if !asked || question != "На когда напомнить?" {
if !asked || question != "Когда?" {
t.Fatalf("expected the time question, got %q asked=%v", question, asked)
}
@@ -152,7 +155,7 @@ func TestClarifyAsksThreeTimesThenSaysSo(t *testing.T) {
if !handled {
t.Fatalf("answer %d must be consumed as an answer", i)
}
if reply != "На когда напомнить?" {
if reply != "Когда?" {
t.Fatalf("attempt %d should ask again, got %q", i, reply)
}
if h.clarifyStore.Get(voiceDialogueID, h.now()) == nil {
@@ -304,17 +307,17 @@ func TestClarifyExpiryIsAnnouncedAndWordsStillRoute(t *testing.T) {
*now = now.Add(clarifyTTL + time.Second)
reply := h.handleText(ctx, "как дела")
if !strings.HasPrefix(reply, clarifyExpired) {
if !isClarifyExpired(reply) {
t.Fatalf("expired question must be announced first, got %q", reply)
}
if strings.TrimSpace(strings.TrimPrefix(reply, clarifyExpired)) == "" {
if trimClarifyExpired(reply) == "" {
t.Fatalf("the new words must still be answered, got only the notice: %q", reply)
}
if h.clarifyStore.Get(voiceDialogueID, h.now()) != nil {
t.Fatal("the expired question must be gone")
}
// The notice is said once, not on every later utterance.
if reply := h.handleText(ctx, "как дела"); strings.Contains(reply, clarifyExpired) {
if reply := h.handleText(ctx, "как дела"); isClarifyExpired(reply) {
t.Fatalf("notice repeated on a later turn: %q", reply)
}
}
+1 -1
View File
@@ -34,7 +34,7 @@ func newLLMReplier(c completer, block func() string) *llmReplier {
return &llmReplier{c: c, stub: voice.NewStubReplier(), block: block}
}
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Владелец — мужчина, говоришь с ним на "ты", в единственном числе; никогда не "вы"/"ваш" и не "он"/"его". Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Владелец — мужчина, говоришь с ним на "ты", в единственном числе; никогда не "вы"/"ваш" и не "он"/"его". Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), по-русски, спокойно и без официальных формулировок. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"}
Никогда не пиши "..." в поле response.`
+11 -2
View File
@@ -551,12 +551,21 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma
// Russian only, feminine self-reference, second person masculine (the owner is
// a man). She talks TO him, informally, singular — never "вы", never "он".
// One short sentence — the nudge is spoken aloud.
//
// What the ban on обращения forbids is pet names ("дорогой", "милый"), not his
// name: "Ками, ноутбук на трёх процентах" is exactly how she talks, and the
// unqualified word read as forbidding that too. Hence "ласковые обращения".
//
// The examples also never claim a physical act. She has no hands and no smart
// plug — she can tell him the battery is at three percent, she cannot put the
// laptop on charge. An example that says she did teaches the model to invent
// actions Maven never took, which is worse than a missing nudge.
const nudgeSystem = `Ты — Maven, домашняя ассистентка. О себе говоришь в женском роде ("я проверила", "я записала"). Владелец — мужчина, обращайся к нему в мужском роде ("ты пил", "ты забыл").
Говоришь с ним на "ты", в единственном числе ("выпей", "встань"). Никогда не "вы"/"вас"/"ваш" и никогда "он"/"его" — ты говоришь ему, а не о нём.
Пиши ОДНО короткое напоминание по-русски: не больше 120 символов и не больше 16 слов. Только по делу.
Запрещено: обращения ("дорогой", "милый"), эмодзи, извинения ("прости", "извини"), вопросы о самочувствии, похвала, больше одного восклицательного знака, английские слова кроме имён сервисов.
Запрещено: ласковые обращения ("дорогой", "милый"), эмодзи, извинения ("прости", "извини"), вопросы о самочувствии, похвала, больше одного восклицательного знака, английские слова кроме имён сервисов.
Отвечай ТОЛЬКО одним объектом JSON с полями "response" и "mood".
"response" — сам текст напоминания.
@@ -564,7 +573,7 @@ const nudgeSystem = `Ты — Maven, домашняя ассистентка. О
Так выглядит правильный ответ по форме. Темы здесь посторонние — их в запросе не будет:
{"response": "Стиральная машина закончила. Развесь бельё.", "mood": "neutral"}
{"response": "Ноутбук на трёх процентах. Я поставила его на зарядку.", "mood": "confused"}
{"response": "Ками, ноутбук на трёх процентах. Поставь его на зарядку.", "mood": "confused"}
Это примеры ФОРМЫ, а не темы. Пиши только про ту ситуацию, которую тебе дали в запросе. Не копируй примеры и никогда не пиши "..." в поле response.`
Executable
BIN
View File
Binary file not shown.