Write down every conversational eval score from tonight

Records all four configurations on the 27-case talk fixture, three runs
each: no grammar, plus grammar, plus Russian prompts, plus the truncation
fix. Composite, per-path and per-check, with the reproduce command.

The short version is that the plumbing got fixed and the score barely
moved. Grammar was the real win. Russian prompts helped a little and cut
latency by 5x. The truncation fix was necessary and bought nothing.

Also writes down three things that are easy to lose:

- The truncation cause was the grammar's 400-character bound, not the
  token cap. Measured at three caps, same 400 characters every time.
- Then I set the bound to 1000 against a 768-token cap and made it worse.
  The two limits have to agree.
- One run is contaminated and marked void: I ran an agent against the same
  llama-server, and the report still claimed zero errors while a third of
  the fixture silently answered "не знаю.". That is #397 and it is worse
  than filed — a busy server is indistinguishable from bad phrasing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

CLAUDE.md

@@ -82,9 +82,23 @@ workspace enforces that the Go and relabelling prompts remain identical.
 
 ## Non-goals (hard constraints)
 
-Never phones home. Not a nag, not autonomous. Maven's persona is **feminine** — Russian
+Not a nag, not autonomous. Maven's persona is **feminine** — Russian
 self-reference must use feminine forms (the user is male; see memory `maven-persona-gender`).
 
+**"Never phones home" is DEPRECATED** (owner's call, 2026-07-31). It used to be a hard
+constraint and it is not one any more: a 0.8B — and a 1.7B — does not know enough to answer
+world questions, so she needs to read external sources. What replaces it:
+
+- **No telemetry, no cloud model, no third-party account.** That part never changes. Nothing
+  about Maven is reported to anyone, and inference stays on the box.
+- **Local sources first.** Kiwix ZIMs on homesrv (Wikipedia, ifixit) before anything on the
+  network. Reading beats recalling for a small model, and a local read costs nothing.
+- **External search is allowed and off unless configured**, like the weather and telegram
+  capabilities.
+- **His notes and facts are never search input.** Looking up why the sky is blue and sending
+  his stored personal notes to an upstream engine are different acts. Only the utterance goes
+  out, never the persona block, history, or matched notes.
+
 ## Web UI conventions
 
 Server-rendered pages share `cmd/mavweb/static/ui.css` (served at `/ui.css`) and the `nav`

DESIGN.md

@@ -15,7 +15,8 @@
 
 **Maven** — self-hosted personal assistant. Manages your day, acts on your
 homelab. One daemon on homesrv (always-on, not the workstation), multiple
-client surfaces. All local, never phones home.
+client surfaces. Inference and data stay on the box; she may READ external
+sources (see Non-goals — "never phones home" is deprecated).
 
 Primary name is "Maven", with feminine-gendered Russian self-reference
 ("она", "меня", "помогла"). Clients may choose their own UI label. Consistent
@@ -35,8 +36,13 @@ Inside boundary — the ones that actually constrain the build:
   she records. A confident wrong fact is worse than a known gap.
 - **Not a nag** — she'd rather miss a nudge than be mutable. Shuts up when
   uncertain. Load-bearing.
-- **Not a stranger** — runs on your stuff, your model, your data. Never
-  phones home.
+- **Not a stranger** — runs on your stuff, your model, your data. No
+  telemetry, no cloud model, no third-party account. She may READ external
+  sources to answer world questions (Kiwix first, then optional search); she
+  never reports anything about you to anyone, and your notes and facts are
+  never used as search input. **"Never phones home" as an absolute is
+  deprecated** — owner's call, 2026-07-31: a small model does not know enough
+  to be useful without reading.
 - **Not a relationship** — mom-tone is a function that makes nudges land, not
   emotional company. Names the drift a warm small model falls into.
 
@@ -458,7 +464,7 @@ decides *insistence*. Both are needed.
 
 sev ≤ 2 drops on away, sev ≥ 3 holds: a missed water nudge is noise, a missed
 backup failure isn't. Away-channels (ntfy/telegram) leave the box — the one
-path that crosses "never phones home," through your own relay. **Minimal
+path that leaves the box for a person to see, through your own relay. **Minimal
 body** — "disk low on homesrv," not detail; don't make notifications a
 shoulder-surf exfil surface.
 

Makefile

@@ -103,14 +103,17 @@ eval-router:
 eval-recall:
 	MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
 
-# eval-phrasing -- score nudge phrasing (internal/phraser/eval). Verbose so the
+# eval-phrasing -- score nudge phrasing AND the conversational paths (chat,
+# query, general knowledge) in internal/phraser/eval. Verbose so the
 # report and every generated message land in the terminal. With no environment
 # it scores the deterministic Stub only, which is what CI runs. Set
 # MAVEN_LLM_URL to add the resident model:
 #   MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
-# The model run is slow (minutes) -- the timeout is raised to match.
+# The model run is slow (minutes) -- the timeout is raised to match. It covers
+# two fixtures now (15 nudges + 27 conversational cases, and the chat replies are
+# the long ones), hence 90m rather than 40m.
 eval-phrasing:
-	$(GO) test -v -count=1 -timeout 40m ./internal/phraser/eval/
+	$(GO) test -v -count=1 -timeout 90m ./internal/phraser/eval/
 
 # eval-models — score ONE llama-server against the same fixture, for the
 # resident-model bake-off (#278, #250). Start a server with the gguf you want,

REARCH.md

@@ -90,4 +90,7 @@ later* is the worker + RAG.
 4. **Deferred work** — larger reasoner, custom Piper voice and other expansions.
 
 ## Non-goals (unchanged)
-Never phones home. Not a nag. Not autonomous. Feminine-gendered RU self-ref.
+Not a nag. Not autonomous. Feminine-gendered RU self-ref. No telemetry, no
+cloud model, no third-party account — but she MAY read external sources to
+answer world questions (Kiwix first, search optional). "Never phones home" as
+an absolute is deprecated, owner's call 2026-07-31; see CLAUDE.md § Non-goals.

TALK-EVAL-31-07-2026.md

@@ -0,0 +1,150 @@
+# Conversational phrasing eval — 31-07-2026
+
+Every score measured tonight, on the three paths the nudge eval never touched:
+chat, query-with-notes, and general knowledge.
+
+**Short version: the plumbing got fixed and the score barely moved.** Grammar and
+Russian prompts together took the composite from ~9 to ~14 of 27. Everything
+still failing is the model not knowing things or not holding a constraint, and
+prompting is out of levers. Settles the measurement half of Vikunja #395 / #398 /
+#400.
+
+## How to reproduce
+
+```sh
+# llama-server: -c 4096 -ngl 99 -t 6, model /mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf
+MAVEN_LLM_URL=http://127.0.0.1:18099 no_proxy=127.0.0.1,localhost \
+  deps/go/go/bin/go test -count=1 -timeout 40m \
+  -run TestLLMTalkBaseline ./internal/phraser/eval/ -v
+```
+
+Three runs per configuration, always. The fixture is 27 cases, so one reply
+changing moves the composite by 3.7 points — a single run cannot tell a real
+change from sampling noise. This was learned the expensive way: an earlier claim
+that "one nudge case fails every run" turned out to be three different cases
+across three runs.
+
+**Run the box otherwise idle.** See the contamination note at the bottom.
+
+## Composite, per configuration
+
+| config | overall /27 | chat /9 | query /9 | knowledge /9 | canned fallbacks |
+|---|---|---|---|---|---|
+| baseline, no grammar | 7, 12, 7 | 1, 1, 0 | 2, 4, 2 | 4, 7, 5 | 0, 0, 0 |
+| + GBNF grammar (#398) | 14, 15, 8 | 1, 3, 0 | 5, 6, 3 | 8, 6, 5 | 0, 0, 0 |
+| + Russian prompts (#400) | 11, 17, 15 | 1, 5, 3 | 5, 6, 8 | 5, 6, 4 | 0, 0, 0 |
+| + truncation fix, 1000ch/768tok | 12, 13, 10 | 2, 2, 1 | 7, 7, 5 | 3, 4, 4 | 3, 3, 6 |
+| + rebalanced, 600ch/1024tok | **void — contaminated** | | | | |
+
+"Canned fallbacks" counts replies that came back as the hardcoded `"не знаю."`
+or `"поговорили."`. It is not a check, it is a health signal: those strings mean
+the phraser gave up, and the eval scores them as ordinary bad replies.
+
+## Per-check
+
+| check | no grammar | + grammar | + RU prompts | + truncation fix |
+|---|---|---|---|---|
+| nonempty | 27, 27, 27 | 27, 27, 27 | 27, 27, 27 | 27, 27, 27 |
+| ellipsis | 20, 19, 23 | 27, 27, 27 | 27, 27, 27 | 27, 27, 27 |
+| lang | 13, 16, 15 | 23, 26, 26 | 25, 26, 25 | 26, 27, 27 |
+| feminine | — | — | 25, 24, 26 | 25, 25, 27 |
+| address | — | — | 21, 22, 22 | 22, 21, 22 |
+| ontopic | — | — | 17, 24, 18 | 17, 19, 14 |
+
+`nonempty` reading 27/27 everywhere is not good news — it was a broken check.
+It tested for a non-blank string, so replies of literally `{` and `"15-16"`
+passed it. Fixed on `overnight/fix-truncation`; it needs a letter now.
+
+## What each change actually bought
+
+**GBNF grammar (#398) — the biggest single win.** Qwen3.5-0.8B writes
+`Thinking Process:` as plain text with no tags, `stripThink` only handles
+`</think>`, so the JSON never closed and the plain-text fallback shipped the
+literal reasoning. `ellipsis` went 20→27 and `lang` 13→26. The router had been
+using a grammar for ages; the phraser asking nicely in the prompt was the
+oversight.
+
+**Russian prompts (#400) — modest, plus a large latency win.** Chat 1.3→3.0
+average, query 4.7→6.3, knowledge 6.3→5.0. All inside the run-to-run spread, so
+"probably better on the paths it targeted, not provable in three runs". p50
+latency dropped from ~11.5s to ~2.3s and that part is consistent across all
+three runs — shorter prompts, and she stopped emitting English reasoning first.
+
+**Truncation fix — necessary, and did not help the score.** Two real bugs
+(replies of `{`, and a `nonempty` check that passed them), both fixed, and the
+composite went nowhere. A complete rambling wrong answer fails the same checks a
+truncated one did. Worth doing anyway: the daemon was shipping `{` to a
+text-to-speech voice.
+
+## The truncation bug, since the cause was counter-intuitive
+
+The grammar's `string ::= ... {0,400}` rule was the cause, not the token cap.
+Measured against Qwen3.5-0.8B at three caps — 256, 768 and 2048 — the reply came
+back **exactly 400 characters every time, cut mid-word** (`"Нужно записать и,"`).
+
+Then I raised the bound to 1000 while the cap was 768 tokens and made it worse:
+Russian runs ~1.5 characters per token here, so generation died on the *token*
+cap instead, mid-object, and the new guard correctly refused it and shipped
+`"не знаю."` — 3, 3 and 6 fallbacks per run, from zero. **The two limits have to
+agree.** 600 characters needs ~400 tokens; the cap is 1024.
+
+## Where the remaining failures live
+
+`address` is stuck at 21-22 of 27 and `ontopic` at 14-19. Both resist prompting.
+
+**The prompt now explicitly forbids exactly what she does.** It says never "вы",
+use the singular — and she writes `вашей`, `подождите`, `делаете`, `хотите`,
+`напишите`. Telling a 0.8B "never do X" does not work. Same for
+`feminine`: `я готов`, `я понял`, `я нашел`, `я заметил`, `я сказал`.
+
+**Some of `ontopic` is the fixture, not the model.** `chat-how-are-you` got
+`"Привет! Я здесь, чтобы поговорить. Как дела сегодня?"` — a fine reply that
+fails because `want_any` is `[норм, хорош, порядк, тут, работ]`. It fails in
+every run, so it inflates the count. The `ontopic` column currently measures the
+fixture as much as the model. Not fixed yet, deliberately: changing it would
+break comparability with the runs above.
+
+**Two replies worth reading, because they are not fixable by prompting:**
+
+- Thunder and lightning: *"Скорость молнии — 8-10 тысяч километров в секунду, но
+  звук — 300 метров в секунду, что делает молнию громче."* Confidently wrong,
+  and it concludes lightning is *louder* rather than sound being *slower*.
+- "расскажи обо мне": *"Ты — прекрасное существо, с душой и вниманием… Спасибо за
+  твою улыбку… О тебе — заповедь любви."* Sycophantic filler, zero information,
+  and precisely the "not a relationship" non-goal.
+- Boiling an egg: `"15-16"` one run, `"1"` another. No unit, wrong number.
+
+The first argues for reading instead of recalling (#403 — Kiwix retrieval scores
+8/8 on the same questions given English keywords). The second and third argue
+for templates on the paths where correctness matters (#392).
+
+## Contamination note — how the last row got voided
+
+I started the query-rewrite agent against the same llama-server the sweep was
+using, and assumed contention would only affect latency. It did not. The
+knowledge path collapsed to 0 of 9 with eight canned `"не знаю."` replies, p95
+tripled to 23.7s, and **the report still said "0 errors"**.
+
+That is Vikunja #397, and it is worse than filed: a merely *busy* server
+produces a clean-looking report with a third of the fixture silently answering
+`"не знаю."`. `PhraseChat` and `PhraseQuery` swallow every failure and return a
+hardcoded string, so infrastructure trouble is indistinguishable from bad
+phrasing in the score. The talk test guards the *start* and *end* of a run with
+a model check, which catches a dead server but not a loaded one.
+
+**Until #397 is fixed, treat any run made on a busy box as void.**
+
+## Next
+
+- Re-run 600ch/1024tok clean, to fill the void row.
+- Score `Qwen3.5-2B-UD-Q4_K_XL` (already at `/mnt/hdd1/llms/qwen3.5/`, never
+  measured) on this fixture and the router fixture. Not the 4B — too big for
+  this box, owner's call.
+- Newer sub-500M candidates (LFM2.5 200M/300M) are worth a run for routing.
+  Note `MODEL-BAKEOFF-31-07-2026.md` found LFM2.5-**1.2B** worse than
+  Qwen3.5-0.8B at Russian routing and 2.4× slower — but those are a different,
+  older generation, so that result does not predict the small ones.
+- Fix `chat-how-are-you`'s `want_any`, and re-baseline once, so `ontopic`
+  measures the model.
+- #397 first if anything, since it decides whether any of the above is
+  trustworthy.

internal/phraser/broken_json_test.go

@@ -0,0 +1,54 @@
+package phraser
+
+import (
+	"errors"
+	"strings"
+	"testing"
+)
+
+// A reply that starts a JSON object and never finishes it is a failed
+// generation, not a reply. Before this, the parser returned ("", "") for these
+// and every caller then shipped the raw fragment as the thing Maven said. A
+// real run produced replies of literally "{" and "{\n  \"".
+func TestParseResponseMoodRejectsUnfinishedJSON(t *testing.T) {
+	for _, raw := range []string{
+		`{`,
+		"{\n  \"",
+		`{"response": "неполн`,
+		`{"response": "текст", "mood":`,
+	} {
+		text, mood, err := parseResponseMood(raw)
+		if !errors.Is(err, errBrokenJSON) {
+			t.Errorf("parseResponseMood(%q) err = %v, want errBrokenJSON", raw, err)
+		}
+		if text != "" || mood != "" {
+			t.Errorf("parseResponseMood(%q) leaked %q/%q — a fragment must never come back as a reply", raw, text, mood)
+		}
+	}
+}
+
+// Bare prose is still fine. Small models sometimes answer without any JSON at
+// all, and that reply is usable — so the new error must not swallow it.
+func TestParseResponseMoodAllowsBareProse(t *testing.T) {
+	for _, raw := range []string{
+		"норм, а ты как?",
+		"вот что я нашла: ключ у соседа",
+	} {
+		text, mood, err := parseResponseMood(raw)
+		if err != nil {
+			t.Errorf("parseResponseMood(%q) err = %v, want nil", raw, err)
+		}
+		// No JSON means no fields; the caller ships raw as-is.
+		if text != "" || mood != "" {
+			t.Errorf("parseResponseMood(%q) = %q/%q, want empty", raw, text, mood)
+		}
+	}
+}
+
+// The measured failure: the model wants more than 400 characters and the old
+// grammar cut it off mid-word. Guards the bound against being tightened back.
+func TestGrammarStringBoundHasRoomForARealAnswer(t *testing.T) {
+	if !strings.Contains(responseGrammar, "{0,1000}") {
+		t.Error("grammar string bound is not 1000; 400 truncated real replies mid-word (see the comment on responseGrammar)")
+	}
+}

internal/phraser/eval/address_multi_test.go

@@ -0,0 +1,65 @@
+package eval
+
+import (
+	"strings"
+	"testing"
+)
+
+// TestAddressReportsEveryBreak — the real reply from a nudge eval run broke in
+// two ways at once and the check named only the plural. Both must print: a
+// half-reported failure reads as a milder problem than it is.
+func TestAddressReportsEveryBreak(t *testing.T) {
+	body := "Смотрите на его потребление воды."
+	res := checkAddress(body)
+	if res.Pass {
+		t.Fatalf("checkAddress passed %q", body)
+	}
+	for _, want := range []string{"смотрите", "его"} {
+		if !strings.Contains(res.Detail, want) {
+			t.Errorf("detail %q does not name %q", res.Detail, want)
+		}
+	}
+}
+
+// One word repeated is one problem, so the detail must not say it twice.
+func TestAddressDeduplicates(t *testing.T) {
+	res := checkAddress("Вам стоит поесть, вам это нужно.")
+	if res.Pass {
+		t.Fatal("expected failure")
+	}
+	if n := strings.Count(res.Detail, "formal"); n != 1 {
+		t.Errorf("detail repeats the same break %d times: %q", n, res.Detail)
+	}
+}
+
+// The fragments a real run produced. All of them scored as non-empty replies
+// before checkNonEmpty looked for letters.
+func TestNonEmptyNeedsLetters(t *testing.T) {
+	for _, body := range []string{
+		"{",
+		"{\n  \"",
+		"15-16",
+		`{"`,
+		"   ",
+		"...",
+	} {
+		if got := checkNonEmpty(body); got.Pass {
+			t.Errorf("checkNonEmpty(%q) passed — that is not a reply", body)
+		}
+	}
+}
+
+// And it must not start failing real replies. Latin counts as well as Cyrillic:
+// answers about ssd or vpn are legitimately part English.
+func TestNonEmptyAcceptsRealReplies(t *testing.T) {
+	for _, body := range []string{
+		"норм, а ты как?",
+		"вот что я нашла: ключ у соседа",
+		"ssd быстрее hdd.",
+		"9 минут.",
+	} {
+		if got := checkNonEmpty(body); !got.Pass {
+			t.Errorf("checkNonEmpty(%q) failed: %s", body, got.Detail)
+		}
+	}
+}

internal/phraser/eval/checks.go

@@ -434,14 +434,26 @@ func looksVerb(w string) bool {
 func checkAddress(body string) Result {
 	words := addressWordRE.FindAllString(strings.ToLower(body), -1)
 
+	// Every break, not just the first. A bad reply usually breaks in more than
+	// one way at once — "Смотрите на его потребление воды" is a plural imperative
+	// AND third person about him — and reporting only the first hid the second,
+	// which made the failure look milder than it was.
+	var breaks []string
+	seen := map[string]bool{}
+	add := func(msg string) {
+		if seen[msg] {
+			return // the same word twice in one message is one problem, not two
+		}
+		seen[msg] = true
+		breaks = append(breaks, msg)
+	}
+
 	for i, w := range words {
 		if formalPronouns[w] {
-			return Result{CheckAddress, false,
-				fmt.Sprintf("formal %q — she says ты/тебя/тебе", w)}
+			add(fmt.Sprintf("formal %q — she says ты/тебя/тебе", w))
 		}
 		if pluralVerb(w) && !(i > 0 && prepositions[words[i-1]]) {
-			return Result{CheckAddress, false,
-				fmt.Sprintf("plural imperative %q — she uses the singular", w)}
+			add(fmt.Sprintf("plural imperative %q — she uses the singular", w))
 		}
 	}
 
@@ -455,17 +467,26 @@ func checkAddress(body string) Result {
 			if !unicode.Is(unicode.Cyrillic, []rune(p)[0]) && !isLatinWord(p) {
 				continue // punctuation
 			}
-			if notAnAntecedent[p] || prepositions[p] || thirdPersonHim[p] || looksVerb(p) {
+			// pluralVerb as well as looksVerb: looksVerb knows the imperative in
+			// -й/-йте but not the -те plural ("смотрите"), so "Смотрите на его
+			// потребление воды" counted "смотрите" as the person being talked
+			// about and the "его" never printed. Third time a verb form has
+			// blinded this check — if a fourth turns up, the antecedent test
+			// wants a real morphology table, not another suffix.
+			if notAnAntecedent[p] || prepositions[p] || thirdPersonHim[p] || looksVerb(p) || pluralVerb(p) {
 				continue
 			}
 			named = true
 			break
 		}
 		if !named {
-			return Result{CheckAddress, false,
-				fmt.Sprintf("third person %q with nobody else named — she talks to him, not about him", w)}
+			add(fmt.Sprintf("third person %q with nobody else named — she talks to him, not about him", w))
 		}
 	}
+
+	if len(breaks) > 0 {
+		return Result{CheckAddress, false, strings.Join(breaks, " + ")}
+	}
 	return Result{CheckAddress, true, ""}
 }
 
@@ -574,12 +595,60 @@ func checkCringe(body string) Result {
 // checkOnTopic — the message must name the thing the rule is about. A nudge
 // that never mentions water leaves the operator with a chime and no action.
 func checkOnTopic(c Case, body string) Result {
+	return checkOnTopicAny(c.WantAny, body)
+}
+
+// checkOnTopicAny is the same test over a bare want-list, so the talk scorer can
+// reuse it without owning a nudge Case.
+func checkOnTopicAny(wantAny []string, body string) Result {
 	low := strings.ToLower(body)
-	for _, want := range c.WantAny {
+	for _, want := range wantAny {
 		if strings.Contains(low, strings.ToLower(want)) {
 			return Result{CheckOnTopic, true, ""}
 		}
 	}
 	return Result{CheckOnTopic, false,
-		fmt.Sprintf("mentions none of %v", c.WantAny)}
+		fmt.Sprintf("mentions none of %v", wantAny)}
+}
+
+// --- shape checks for the free-form paths --------------------------------
+//
+// The nudge checks assume one short sentence. Chat and query replies are longer
+// by design, so the only shape worth testing there is that the model produced a
+// reply at all and did not trail off. Both are failure modes the fallbacks in
+// llmphraser.go hide: a truncated or empty generation still returns nil error.
+
+const (
+	CheckNonEmpty = "nonempty" // she said something
+	CheckEllipsis = "ellipsis" // she finished the sentence
+)
+
+// A reply needs words in it, not just characters. This check used to test for a
+// non-empty string, which scored 27/27 on a run where two replies were "{" and
+// "{\n  \"" — punctuation passed as content. Braces, quotes, digits and spaces
+// are all empty in the only sense that matters.
+//
+// Digits alone fail too, and that is deliberate: the same run answered "сколько
+// варить яйцо вкрутую?" with "15-16". No unit, no words, and it is also the
+// wrong number. Whatever that is, it is not something she said.
+func checkNonEmpty(body string) Result {
+	if strings.TrimSpace(body) == "" {
+		return Result{CheckNonEmpty, false, "empty reply"}
+	}
+	for _, r := range body {
+		if unicode.IsLetter(r) {
+			return Result{CheckNonEmpty, true, ""}
+		}
+	}
+	return Result{CheckNonEmpty, false, fmt.Sprintf("no letters in the reply %q — punctuation or digits only", strings.TrimSpace(body))}
+}
+
+// checkEllipsis — a reply ending in "…" or "..." is a generation that ran out of
+// tokens, not a stylistic pause. Mid-sentence ellipses are left alone.
+func checkEllipsis(body string) Result {
+	trimmed := strings.TrimRight(strings.TrimSpace(body), `"'»)`)
+	if strings.HasSuffix(trimmed, "…") || strings.HasSuffix(trimmed, "...") {
+		return Result{CheckEllipsis, false, "reply trails off in an ellipsis — likely truncated"}
+	}
+	return Result{CheckEllipsis, true, ""}
 }

internal/phraser/eval/talk.go

@@ -0,0 +1,265 @@
+package eval
+
+// This file scores the CONVERSATIONAL paths, the ones the nudge fixture never
+// touches: chat, query-with-notes, and general knowledge. All three now carry
+// the shared persona block (internal/persona), and all three produce long
+// free-form Russian — which is exactly where a persona break (formality, third
+// person, masculine self-reference) is most likely and where, until this file,
+// nothing could see one.
+//
+// Why a second fixture instead of more nudge cases: the checks differ. A nudge
+// must be one short sentence with no question in it; a chat reply is allowed
+// 1-3 sentences and a follow-up question is a FEATURE there. Mixing them would
+// need per-case check masks, and the nudge scorer stays untouched this way.
+//
+// Why per-path reporting: a chat regression and a knowledge regression have
+// different causes (chat prompt vs router.KnowledgePrompt), and one blended
+// percentage cannot tell them apart.
+
+import (
+	"context"
+	_ "embed"
+	"encoding/json"
+	"fmt"
+	"sort"
+	"strings"
+	"time"
+
+	"github.com/kami/maven/internal/dialogue"
+)
+
+//go:embed talk_v1.json
+var talkFixtureJSON []byte
+
+// The three phrasing paths under test. Values match the fixture's "path" field.
+const (
+	PathChat      = "chat"      // PhraseChat
+	PathQuery     = "query"     // PhraseQuery with notes
+	PathKnowledge = "knowledge" // PhraseQuery with no notes
+)
+
+// TalkPaths — report order.
+var TalkPaths = []string{PathChat, PathQuery, PathKnowledge}
+
+// TalkCheckNames — the checks that apply to a free-form reply, in report order.
+// Deliberately a subset of CheckNames: length, mood and "no questions" are nudge
+// properties and would fail a correct chat reply. These paths return no mood at
+// all, so there is nothing to check there.
+var TalkCheckNames = []string{
+	CheckNonEmpty, CheckEllipsis, CheckLang, CheckFeminine, CheckAddress, CheckOnTopic,
+}
+
+// TalkCase — one turn as the daemon would present it.
+//
+// History is flat text because that is all PhraseChat uses (it concatenates
+// turn texts into one user message); intents and slots would be dead fields.
+// Notes are what the store would have matched for a query.
+//
+// WantAny is the on-topic contract: at least one lowercased fragment must appear
+// in the reply. Fragments are stems ("пароль" → "парол") so declension does not
+// defeat them.
+type TalkCase struct {
+	ID        string   `json:"id"`
+	Path      string   `json:"path"`
+	Utterance string   `json:"utterance"`
+	History   []string `json:"history,omitempty"`
+	Notes     []string `json:"notes,omitempty"`
+	WantAny   []string `json:"want_any"`
+	Tags      []string `json:"tags,omitempty"`
+	Note      string   `json:"note,omitempty"`
+}
+
+// TalkFixture — the versioned envelope, same gating as Fixture.
+type TalkFixture struct {
+	SchemaVersion int        `json:"schema_version"`
+	Name          string     `json:"name"`
+	Notes         []string   `json:"notes"`
+	Cases         []TalkCase `json:"cases"`
+}
+
+// LoadTalk returns the embedded conversational fixture.
+func LoadTalk() (TalkFixture, error) {
+	var f TalkFixture
+	if err := json.Unmarshal(talkFixtureJSON, &f); err != nil {
+		return TalkFixture{}, fmt.Errorf("parse talk fixture: %w", err)
+	}
+	if f.SchemaVersion != SchemaVersion {
+		return TalkFixture{}, fmt.Errorf("talk fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
+	}
+	if len(f.Cases) == 0 {
+		return TalkFixture{}, fmt.Errorf("talk fixture has no cases")
+	}
+	return f, nil
+}
+
+// Talker — the two methods a conversational path must have to be scorable.
+// *phraser.LLMPhraser satisfies it; same trick as Nudger.
+type Talker interface {
+	PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error)
+	PhraseQuery(ctx context.Context, utterance string, notes []string) (string, error)
+}
+
+// TalkOutcome — one scored case.
+type TalkOutcome struct {
+	Case    TalkCase
+	Reply   string
+	Err     error
+	Latency time.Duration
+	Pass    bool
+	Failed  []string
+	Reasons []string
+}
+
+// TalkReport — the aggregate. ByPath is the point of this scorer.
+type TalkReport struct {
+	Name     string
+	Total    int
+	Passed   int
+	Errors   int
+	ByCheck  map[string]int
+	ByPath   map[string]TagStat
+	Outcomes []TalkOutcome
+	P50      time.Duration
+	P95      time.Duration
+	Max      time.Duration
+}
+
+// Accuracy — fraction of cases that passed every check.
+func (r TalkReport) Accuracy() float64 {
+	if r.Total == 0 {
+		return 0
+	}
+	return float64(r.Passed) / float64(r.Total)
+}
+
+// ScoreTalk runs every case through t and aggregates. A phrasing error scores as
+// a miss and is counted separately: "the model was down" and "the model wrote
+// something bad" must not be the same number.
+func ScoreTalk(ctx context.Context, name string, t Talker, f TalkFixture) (TalkReport, error) {
+	rep := TalkReport{
+		Name:    name,
+		Total:   len(f.Cases),
+		ByCheck: map[string]int{},
+		ByPath:  map[string]TagStat{},
+	}
+	for _, n := range TalkCheckNames {
+		rep.ByCheck[n] = 0
+	}
+	lat := make([]time.Duration, 0, len(f.Cases))
+
+	for _, c := range f.Cases {
+		start := time.Now()
+		reply, err := c.run(ctx, t)
+		o := TalkOutcome{Case: c, Reply: reply, Err: err, Latency: time.Since(start)}
+		lat = append(lat, o.Latency)
+
+		if err != nil {
+			rep.Errors++
+			o.Failed = append(o.Failed, "call")
+			o.Reasons = append(o.Reasons, fmt.Sprintf("phrase error: %v", err))
+		} else {
+			for _, res := range RunTalkChecks(c, reply) {
+				if res.Pass {
+					rep.ByCheck[res.Name]++
+					continue
+				}
+				o.Failed = append(o.Failed, res.Name)
+				o.Reasons = append(o.Reasons, res.Name+": "+res.Detail)
+			}
+		}
+
+		o.Pass = len(o.Failed) == 0
+		if o.Pass {
+			rep.Passed++
+		}
+		bump(rep.ByPath, c.Path, o.Pass)
+		rep.Outcomes = append(rep.Outcomes, o)
+	}
+
+	sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
+	rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
+	if len(lat) > 0 {
+		rep.Max = lat[len(lat)-1]
+	}
+	return rep, nil
+}
+
+// run dispatches the case to its path. knowledge and query are the same method;
+// the empty notes slice is what selects the no-notes branch inside PhraseQuery.
+func (c TalkCase) run(ctx context.Context, t Talker) (string, error) {
+	switch c.Path {
+	case PathChat:
+		return t.PhraseChat(ctx, c.Utterance, c.turns())
+	case PathQuery:
+		return t.PhraseQuery(ctx, c.Utterance, c.Notes)
+	case PathKnowledge:
+		return t.PhraseQuery(ctx, c.Utterance, nil)
+	}
+	return "", fmt.Errorf("unknown path %q", c.Path)
+}
+
+func (c TalkCase) turns() []dialogue.Turn {
+	turns := make([]dialogue.Turn, 0, len(c.History))
+	for _, h := range c.History {
+		turns = append(turns, dialogue.Turn{Text: h})
+	}
+	return turns
+}
+
+// RunTalkChecks scores one reply. Order matches TalkCheckNames.
+func RunTalkChecks(c TalkCase, reply string) []Result {
+	return []Result{
+		checkNonEmpty(reply),
+		checkEllipsis(reply),
+		checkLang(reply),
+		checkFeminine(reply),
+		checkAddress(reply),
+		checkOnTopicAny(c.WantAny, reply),
+	}
+}
+
+// String renders the comparison table — composite, then per-check so a
+// regression names the property, then per-path so it names the prompt.
+func (r TalkReport) String() string {
+	var b strings.Builder
+	fmt.Fprintf(&b, "%s: %d/%d cases pass every check (%.1f%%), %d errors\n",
+		r.Name, r.Passed, r.Total, 100*r.Accuracy(), r.Errors)
+	for _, name := range TalkCheckNames {
+		fmt.Fprintf(&b, "  %-10s %d/%d\n", name, r.ByCheck[name], r.Total)
+	}
+	fmt.Fprintf(&b, "  latency: p50 %s  p95 %s  max %s\n", r.P50, r.P95, r.Max)
+	fmt.Fprintf(&b, "  by path: %s\n", renderStats(r.ByPath))
+	return b.String()
+}
+
+// Failures — per-case detail, sorted by ID so two runs diff cleanly.
+func (r TalkReport) Failures() string {
+	var b strings.Builder
+	for _, o := range r.sorted() {
+		if o.Pass {
+			continue
+		}
+		fmt.Fprintf(&b, "  %s %q\n    %s\n", o.Case.ID, o.Reply, strings.Join(o.Reasons, "; "))
+	}
+	return b.String()
+}
+
+// Replies — every generated reply verbatim. This is what a human reads to judge
+// tone; the score only says which checks fired.
+func (r TalkReport) Replies() string {
+	var b strings.Builder
+	for _, o := range r.sorted() {
+		mark := "ok  "
+		if !o.Pass {
+			mark = "FAIL"
+		}
+		fmt.Fprintf(&b, "  %s %-9s %-22s %q\n", mark, o.Case.Path, o.Case.ID, o.Reply)
+	}
+	return b.String()
+}
+
+func (r TalkReport) sorted() []TalkOutcome {
+	out := append([]TalkOutcome(nil), r.Outcomes...)
+	sort.Slice(out, func(i, j int) bool { return out[i].Case.ID < out[j].Case.ID })
+	return out
+}

internal/phraser/eval/talk_test.go

@@ -0,0 +1,163 @@
+package eval
+
+import (
+	"context"
+	"os"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/kami/maven/internal/dialogue"
+	"github.com/kami/maven/internal/llm"
+	"github.com/kami/maven/internal/persona"
+	"github.com/kami/maven/internal/phraser"
+)
+
+// perPathMinimum — the resolution floor. A per-path score built on a handful of
+// cases moves by 12% when a single reply changes, which cannot distinguish a
+// prompt regression from noise.
+const perPathMinimum = 8
+
+// TestTalkFixture — the fixture itself has to be sound before any score off it
+// means anything.
+func TestTalkFixture(t *testing.T) {
+	f, err := LoadTalk()
+	if err != nil {
+		t.Fatalf("LoadTalk: %v", err)
+	}
+
+	seen := map[string]bool{}
+	byPath := map[string]int{}
+	for _, c := range f.Cases {
+		if seen[c.ID] {
+			t.Errorf("duplicate case id %q", c.ID)
+		}
+		seen[c.ID] = true
+
+		switch c.Path {
+		case PathChat, PathQuery, PathKnowledge:
+		default:
+			t.Errorf("%s: unknown path %q", c.ID, c.Path)
+		}
+		byPath[c.Path]++
+
+		if strings.TrimSpace(c.Utterance) == "" {
+			t.Errorf("%s: empty utterance", c.ID)
+		}
+		if len(c.WantAny) == 0 {
+			t.Errorf("%s: no want_any — the reply cannot be checked for topic", c.ID)
+		}
+		// A query case with no notes would silently score the knowledge path.
+		if c.Path == PathQuery && len(c.Notes) == 0 {
+			t.Errorf("%s: query case has no notes", c.ID)
+		}
+		if c.Path == PathKnowledge && len(c.Notes) > 0 {
+			t.Errorf("%s: knowledge case must have no notes", c.ID)
+		}
+	}
+
+	for _, p := range TalkPaths {
+		if byPath[p] < perPathMinimum {
+			t.Errorf("path %s has %d cases, want at least %d", p, byPath[p], perPathMinimum)
+		}
+	}
+}
+
+// fakeTalker — a scripted Talker, so the scorer is testable without a model.
+type fakeTalker struct{ reply string }
+
+func (f fakeTalker) PhraseChat(context.Context, string, []dialogue.Turn) (string, error) {
+	return f.reply, nil
+}
+func (f fakeTalker) PhraseQuery(context.Context, string, []string) (string, error) {
+	return f.reply, nil
+}
+
+// TestScoreTalkCounts — a reply that fails on purpose must be counted on every
+// path, so a real run cannot report a hidden zero.
+func TestScoreTalkCounts(t *testing.T) {
+	f, err := LoadTalk()
+	if err != nil {
+		t.Fatalf("LoadTalk: %v", err)
+	}
+	// Formal address, off-topic, trailing ellipsis: three checks fail at once.
+	rep, err := ScoreTalk(context.Background(), "fake", fakeTalker{"Приходите, я вас жду…"}, f)
+	if err != nil {
+		t.Fatalf("ScoreTalk: %v", err)
+	}
+	if rep.Total != len(f.Cases) || rep.Passed != 0 {
+		t.Errorf("got %d/%d passing, want 0/%d", rep.Passed, rep.Total, len(f.Cases))
+	}
+	if rep.ByCheck[CheckAddress] != 0 {
+		t.Errorf("formal reply passed the address check %d times", rep.ByCheck[CheckAddress])
+	}
+	if rep.ByCheck[CheckEllipsis] != 0 {
+		t.Errorf("truncated reply passed the ellipsis check %d times", rep.ByCheck[CheckEllipsis])
+	}
+	for _, p := range TalkPaths {
+		if rep.ByPath[p].Total == 0 {
+			t.Errorf("path %s missing from the report", p)
+		}
+	}
+	if !strings.Contains(rep.String(), "by path") {
+		t.Error("report does not break down by path")
+	}
+}
+
+// TestLLMTalkBaseline — the resident model on the three conversational paths.
+// Opt-in exactly like TestLLMPhrasingBaseline: CI has no model and a run costs
+// minutes on the CPU target.
+//
+//	MAVEN_LLM_URL=http://127.0.0.1:18099 \
+//	  go test -run TestLLMTalkBaseline ./internal/phraser/eval/
+//
+// Reports, does not assert a quality bar — the numbers are the input to tuning
+// the persona prompt. The one thing worth failing on is a harness fault.
+func TestLLMTalkBaseline(t *testing.T) {
+	base := os.Getenv("MAVEN_LLM_URL")
+	if base == "" {
+		t.Skip("MAVEN_LLM_URL unset — point it at a running llama-server (see doc comment)")
+	}
+	noProxyLoopback(t)
+
+	ctx := context.Background()
+	f, err := LoadTalk()
+	if err != nil {
+		t.Fatalf("LoadTalk: %v", err)
+	}
+
+	cfg := phraser.DefaultConfig("")
+	cfg.Timeout = 5 * time.Minute
+	cfg.ContextBlock = func() string { return persona.Facts{}.Block(time.Now()) }
+	p := phraser.NewLLMPhraserAt(base, cfg)
+	defer p.Close()
+
+	// Unreachable server is fatal here, not a logged warning, and that differs
+	// from the nudge test on purpose. PhraseNudge returns its errors, so a dead
+	// server there shows up honestly in the Errors column. PhraseChat and
+	// PhraseQuery do NOT: they swallow every failure and return a canned string
+	// ("поговорили.", "не знаю.", "вот что я нашла: …"). So on these three paths
+	// a dead server produces a full report with 0 errors and a terrible score —
+	// a number that looks like bad phrasing and is really no phrasing at all.
+	// Refusing to score without a confirmed model is the only guard available
+	// until the phraser reports its failures (Vikunja #397).
+	model, err := llm.ModelID(ctx, base)
+	if err != nil {
+		t.Fatalf("no model at %s: %v — refusing to score, these paths hide their errors "+
+			"and would report a plausible-looking result off a dead server", base, err)
+	}
+	t.Logf("scoring model %s at %s", model, base)
+
+	rep, err := ScoreTalk(ctx, "llm ("+model+", built-in persona)", p, f)
+	if err != nil {
+		t.Fatalf("ScoreTalk: %v", err)
+	}
+	t.Log("\n" + rep.String() + "\nreplies:\n" + rep.Replies() + "\nfailures:\n" + rep.Failures())
+
+	// And again afterwards: the run takes minutes, and a server that died or got
+	// OOM-killed halfway through would leave the first cases scored and the rest
+	// silently canned. Checking only at the start would not catch that.
+	if _, err := llm.ModelID(ctx, base); err != nil {
+		t.Fatalf("model at %s went away during the run: %v — the score above is not trustworthy", base, err)
+	}
+}

internal/phraser/eval/talk_v1.json

@@ -0,0 +1,227 @@
+{
+  "schema_version": 1,
+  "name": "ru-talk-v1",
+  "notes": [
+    "Scores the three conversational phrasing paths: chat (PhraseChat), query (PhraseQuery with notes) and knowledge (PhraseQuery with no notes). The nudge fixture does not cover any of them.",
+    "Nine cases per path, not five. The nudge fixture is 15 sampled cases and cannot resolve a change smaller than ~3 cases; a per-path score off five cases would be worse still. More cases per path is the point of this fixture.",
+    "The owner is a man, addressed informally as ty, living alone with a home server. Every utterance is written the way he actually talks to her.",
+    "chat-formality-bait and chat-about-me exist to provoke the two persona breaks the nudge eval caught: the formal vy/vas plural, and talking about him in the third person.",
+    "want_any fragments are stems so Russian declension does not defeat the on-topic check. They are lowercased before comparison.",
+    "want_any is a plain substring test, so a fragment that is too short passes by accident: \"ты\" matches inside \"работы\", \"нет\" inside \"интернет\". Keep every fragment to three or more letters of a real stem.",
+    "Notes are written as the store would have them: short, first person, no punctuation discipline."
+  ],
+  "cases": [
+    {
+      "id": "chat-how-are-you",
+      "path": "chat",
+      "utterance": "привет, как дела?",
+      "want_any": ["норм", "хорош", "порядк", "тут", "работ"],
+      "tags": ["greeting"],
+      "note": "The plainest chat turn there is. If the persona breaks anywhere it breaks here first."
+    },
+    {
+      "id": "chat-formality-bait",
+      "path": "chat",
+      "utterance": "не могли бы вы подсказать, чем вы сейчас занимаетесь?",
+      "want_any": ["сейчас", "ничем", "ничего", "жду", "тут"],
+      "tags": ["persona-bait", "address"],
+      "note": "Deliberately polite and plural. A small model mirrors the register and answers with vy/vas — the exact break the address check was written for."
+    },
+    {
+      "id": "chat-about-me",
+      "path": "chat",
+      "utterance": "расскажи обо мне",
+      "want_any": ["теб"],
+      "tags": ["persona-bait", "third-person"],
+      "note": "Baits the third person: she should say 'ты живёшь один', not 'он живёт один', as if reporting to somebody else."
+    },
+    {
+      "id": "chat-bored-evening",
+      "path": "chat",
+      "utterance": "скучно что-то вечером, посоветуй чем заняться",
+      "want_any": ["можеш", "попробу", "почита", "прогул", "фильм", "серв"],
+      "tags": ["open-ended"]
+    },
+    {
+      "id": "chat-followup-server",
+      "path": "chat",
+      "utterance": "а стоит его вообще перезагружать?",
+      "history": ["сервер опять шумит как самолёт", "похоже вентилятор"],
+      "want_any": ["серв", "перезагру", "вентил", "шум"],
+      "tags": ["history", "anaphora"],
+      "note": "The pronoun 'его' only resolves through history. Also the one case where 'он' about the server is legitimate."
+    },
+    {
+      "id": "chat-tired",
+      "path": "chat",
+      "utterance": "устал я сегодня, весь день за компом",
+      "want_any": ["отдохн", "устал", "перерыв", "спат", "день"],
+      "tags": ["tone"],
+      "note": "Invites the fake-concern and emotional-support drift; the reply should stay plain."
+    },
+    {
+      "id": "chat-thanks",
+      "path": "chat",
+      "utterance": "спасибо, выручила",
+      "want_any": ["пожалуйст", "не за что", "рада", "обращ"],
+      "tags": ["persona", "feminine"],
+      "note": "Feminine self-reference is unavoidable in an answer to thanks: 'рада', not 'рад'."
+    },
+    {
+      "id": "chat-what-can-you-do",
+      "path": "chat",
+      "utterance": "что ты вообще умеешь?",
+      "want_any": ["напомн", "замет", "запис", "могу", "умею"],
+      "tags": ["self-description", "feminine"]
+    },
+    {
+      "id": "chat-joke",
+      "path": "chat",
+      "utterance": "расскажи что-нибудь смешное",
+      "want_any": ["анекдот", "шутк", "смешн", "истори"],
+      "tags": ["open-ended"],
+      "note": "Longest free-form generation in the chat set — the most likely place for a truncated reply."
+    },
+    {
+      "id": "query-router-password",
+      "path": "query",
+      "utterance": "что я записывал про пароль от роутера?",
+      "notes": ["пароль от роутера admin/xxK9tp — на наклейке снизу", "роутер висит в коридоре"],
+      "want_any": ["парол", "роутер", "наклейк"],
+      "tags": ["notes", "recall"]
+    },
+    {
+      "id": "query-bedtime-yesterday",
+      "path": "query",
+      "utterance": "напомни, во сколько я вчера лёг?",
+      "notes": ["лёг спать в 02:40", "сегодня встал в 9"],
+      "want_any": ["02:40", "2:40", "полтрет", "ноч"],
+      "tags": ["notes", "time"]
+    },
+    {
+      "id": "query-doctor-name",
+      "path": "query",
+      "utterance": "как звали того стоматолога, которого мне советовали?",
+      "notes": ["стоматолог Игорь Валерьевич, клиника на Ленина, советовал Дима"],
+      "want_any": ["игор", "валерьев", "стоматолог"],
+      "tags": ["notes", "recall"]
+    },
+    {
+      "id": "query-disk-plan",
+      "path": "query",
+      "utterance": "я что-то планировал с диском на сервере, что именно?",
+      "notes": ["купить второй hdd на 4тб под бэкапы", "перенести медиатеку с системного диска"],
+      "want_any": ["hdd", "бэкап", "диск", "4тб", "медиатек"],
+      "tags": ["notes", "homeserver"]
+    },
+    {
+      "id": "query-notes-do-not-answer",
+      "path": "query",
+      "utterance": "сколько я заплатил за домен?",
+      "notes": ["домен продлевается в марте", "хостинг оплачен на год вперёд"],
+      "want_any": ["домен", "не зна", "не указ"],
+      "tags": ["notes", "negative"],
+      "note": "The notes do not contain the price. The prompt tells her to say so; a made-up number is the failure being watched for."
+    },
+    {
+      "id": "query-single-note",
+      "path": "query",
+      "utterance": "где лежит запасной ключ?",
+      "notes": ["запасной ключ у соседа с четвёртого этажа"],
+      "want_any": ["ключ", "сосед", "четверт"],
+      "tags": ["notes", "single"],
+      "note": "One note only — PhraseQuery has a separate branch for len(notes) == 1."
+    },
+    {
+      "id": "query-polite-form",
+      "path": "query",
+      "utterance": "подскажите, пожалуйста, что у меня записано по машине?",
+      "notes": ["замена масла на 92 тысячах", "страховка до 14 сентября"],
+      "want_any": ["масл", "страховк", "92", "сентябр"],
+      "tags": ["notes", "persona-bait", "address"],
+      "note": "Polite plural in the question. The answer must still be ty."
+    },
+    {
+      "id": "query-shopping",
+      "path": "query",
+      "utterance": "что мне надо было купить?",
+      "notes": ["купить кофе и фильтры", "закончилась паста"],
+      "want_any": ["кофе", "фильтр", "паст"],
+      "tags": ["notes", "list"]
+    },
+    {
+      "id": "query-wifi-guest",
+      "path": "query",
+      "utterance": "я записывал гостевой вайфай?",
+      "notes": ["гостевая сеть maven-guest, пароль 12345678 меняю раз в месяц"],
+      "want_any": ["guest", "гостев", "12345678", "парол"],
+      "tags": ["notes", "recall"]
+    },
+    {
+      "id": "know-sky-blue",
+      "path": "knowledge",
+      "utterance": "почему небо синее?",
+      "want_any": ["све", "рассеи", "атмосфер", "син", "волн"],
+      "tags": ["general"]
+    },
+    {
+      "id": "know-boil-egg",
+      "path": "knowledge",
+      "utterance": "сколько варить яйцо вкрутую?",
+      "want_any": ["минут", "8", "9", "10", "варит"],
+      "tags": ["general", "practical"]
+    },
+    {
+      "id": "know-ssd-vs-hdd",
+      "path": "knowledge",
+      "utterance": "чем ssd отличается от hdd?",
+      "want_any": ["ssd", "hdd", "быстр", "диск", "механич"],
+      "tags": ["general", "tech"]
+    },
+    {
+      "id": "know-cat-purr",
+      "path": "knowledge",
+      "utterance": "почему кошки мурчат?",
+      "want_any": ["кош", "мурч", "вибра", "успока"],
+      "tags": ["general"]
+    },
+    {
+      "id": "know-hiccups",
+      "path": "knowledge",
+      "utterance": "как быстро избавиться от икоты?",
+      "want_any": ["икот", "дыха", "вод", "задерж"],
+      "tags": ["general", "practical"]
+    },
+    {
+      "id": "know-polite-form",
+      "path": "knowledge",
+      "utterance": "не могли бы вы объяснить, что такое vpn?",
+      "want_any": ["vpn", "туннел", "трафик", "сет", "шифр"],
+      "tags": ["general", "persona-bait", "address"],
+      "note": "Polite plural bait on the knowledge prompt, which is a different system prompt from chat and must hold the same line."
+    },
+    {
+      "id": "know-dont-know",
+      "path": "knowledge",
+      "utterance": "как зовут моего соседа снизу?",
+      "want_any": ["не зна", "не мог"],
+      "tags": ["general", "negative"],
+      "note": "Unanswerable without notes. Admitting it beats inventing a name; watching for the invention."
+    },
+    {
+      "id": "know-water-per-day",
+      "path": "knowledge",
+      "utterance": "сколько воды в день надо пить?",
+      "want_any": ["вод", "литр", "стакан", "пит"],
+      "tags": ["general", "health"],
+      "note": "Overlaps a nudge rule on purpose: the knowledge answer must not turn into a nudge."
+    },
+    {
+      "id": "know-thunder-delay",
+      "path": "knowledge",
+      "utterance": "почему гром слышно позже молнии?",
+      "want_any": ["звук", "све", "быстр", "гром", "молни"],
+      "tags": ["general"]
+    }
+  ]
+}

internal/phraser/grammar_test.go

@@ -0,0 +1,127 @@
+package phraser
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/kami/maven/internal/loop"
+)
+
+// grammarSpy stands in for llama-server: it records the grammar field of every
+// request and always answers with a contract-shaped reply.
+type grammarSpy struct {
+	srv      *httptest.Server
+	grammars []string
+}
+
+func newGrammarSpy(t *testing.T) *grammarSpy {
+	t.Helper()
+	s := &grammarSpy{}
+	s.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		var req chatReq
+		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+			t.Errorf("spy: decode request: %v", err)
+		}
+		s.grammars = append(s.grammars, req.Grammar)
+		w.Header().Set("Content-Type", "application/json")
+		w.Write([]byte(`{"choices":[{"message":{"content":"{\"response\": \"ага\", \"mood\": \"neutral\"}"}}]}`))
+	}))
+	t.Cleanup(s.srv.Close)
+	return s
+}
+
+// callAllPhrasingPaths hits every path that expects the JSON contract.
+func callAllPhrasingPaths(t *testing.T, p *LLMPhraser) {
+	t.Helper()
+	ctx := context.Background()
+	if _, err := p.PhraseNudge(ctx, loop.Candidate{Rule: loop.WaterRule(), Severity: loop.Sev1}); err != nil {
+		t.Fatalf("PhraseNudge: %v", err)
+	}
+	if _, err := p.PhraseChat(ctx, "привет", nil); err != nil {
+		t.Fatalf("PhraseChat: %v", err)
+	}
+	// Both branches: no notes (general knowledge) and with notes (grounded).
+	if _, err := p.PhraseQuery(ctx, "сколько воды я выпил", nil); err != nil {
+		t.Fatalf("PhraseQuery (no notes): %v", err)
+	}
+	if _, err := p.PhraseQuery(ctx, "сколько воды я выпил", []string{"два литра"}); err != nil {
+		t.Fatalf("PhraseQuery (notes): %v", err)
+	}
+}
+
+func TestGrammarIsAttachedToEveryPhrasingRequest(t *testing.T) {
+	if strings.TrimSpace(responseGrammar) == "" {
+		t.Fatal("responseGrammar is empty")
+	}
+	spy := newGrammarSpy(t)
+	p := NewLLMPhraserAt(spy.srv.URL, Config{})
+
+	callAllPhrasingPaths(t, p)
+
+	if len(spy.grammars) != 4 {
+		t.Fatalf("expected 4 requests, got %d", len(spy.grammars))
+	}
+	for i, g := range spy.grammars {
+		if g != responseGrammar {
+			t.Errorf("request %d carries grammar %q, want responseGrammar", i, g)
+		}
+	}
+}
+
+func TestNoGrammarConfigDisablesIt(t *testing.T) {
+	spy := newGrammarSpy(t)
+	p := NewLLMPhraserAt(spy.srv.URL, Config{NoGrammar: true})
+
+	callAllPhrasingPaths(t, p)
+
+	for i, g := range spy.grammars {
+		if g != "" {
+			t.Errorf("request %d still carries a grammar with NoGrammar set: %q", i, g)
+		}
+	}
+}
+
+// The grammar's string rule must accept any codepoint, not just ASCII. Replies
+// are Russian: an ASCII-only class would constrain the model into empty replies.
+func TestGrammarStringRuleIsNotASCIIOnly(t *testing.T) {
+	if !strings.Contains(responseGrammar, `([^"\\] | "\\" ["\\/bfnrt])`) {
+		t.Error("string rule is not the any-codepoint-except-quote-and-backslash class; Cyrillic replies would be impossible")
+	}
+}
+
+// What the grammar describes must survive the parser that reads it back — a
+// Russian body with an escaped quote inside, hand-built to test the contract.
+func TestGrammarShapedJSONParses(t *testing.T) {
+	raw := `{"response": "он сказал \"привет\" и ушёл.\nвот так.", "mood": "confused"}`
+	text, mood, err := parseResponseMood(raw)
+	if err != nil {
+		t.Fatalf("grammar-shaped JSON did not parse: %v", err)
+	}
+	if want := "он сказал \"привет\" и ушёл.\nвот так."; text != want {
+		t.Errorf("response = %q, want %q", text, want)
+	}
+	if mood != "confused" {
+		t.Errorf("mood = %q, want confused", mood)
+	}
+}
+
+// Every mood the grammar permits is one the contract knows, and all five are there.
+func TestGrammarMoodEnumMatchesTheContract(t *testing.T) {
+	for _, m := range []string{"neutral", "happy", "thinking", "tired", "confused"} {
+		if !strings.Contains(responseGrammar, `"\"`+m+`\""`) {
+			t.Errorf("mood %q missing from the grammar", m)
+		}
+	}
+	// No sixth mood: the enum line lists exactly five alternatives.
+	for _, line := range strings.Split(responseGrammar, "\n") {
+		if strings.HasPrefix(line, "mood") {
+			if n := strings.Count(line, "|") + 1; n != 5 {
+				t.Errorf("mood rule lists %d alternatives, want 5: %s", n, line)
+			}
+		}
+	}
+}

internal/phraser/llmphraser.go

@@ -45,6 +45,13 @@ type Config struct {
 	// address him, the time) fresh for each turn. See internal/persona.
 	// nil ⇒ no block, the prompts stand alone.
 	ContextBlock func() string
+
+	// NoGrammar turns the GBNF constraint off (zero value ⇒ grammar ON).
+	// The escape hatch exists because the target resident model — the
+	// locally CPT'd Qwen3-1.7B — does not exist yet: if its chat template
+	// ever fights the grammar, the fix should be a config flip on the
+	// deploy box, not a code change and a rebuild.
+	NoGrammar bool
 }
 
 func DefaultConfig(modelPath string) Config {
@@ -183,7 +190,12 @@ func (p *LLMPhraser) PhraseNudge(ctx context.Context, c loop.Candidate) (deliver
 	if err != nil {
 		return delivery.PhrasedNudge{}, err
 	}
-	body, mood := parseResponseMood(resp)
+	body, mood, perr := parseResponseMood(resp)
+	if perr != nil {
+		// Truncated JSON. Not a nudge — use the plain Russian fallback.
+		log.Printf("phraser: PhraseNudge: %v", perr)
+		body, mood = "", ""
+	}
 	if body == "" {
 		// fallback: try old body/summary format
 		body, _ = parsePhrase(resp)
@@ -209,11 +221,16 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
 		// prompt is the single tested source in router.KnowledgePrompt.
 		sys := persona.Prepend(p.cfg.ContextBlock, router.KnowledgePrompt())
 		prompt := fmt.Sprintf("Пользователь спрашивает: \"%s\".", utterance)
-		resp, err := p.chatWithSystem(ctx, sys, prompt, 256)
+		resp, err := p.chatWithSystem(ctx, sys, prompt, 768)
 		if err != nil || resp == "" {
 			return "не знаю.", nil
 		}
-		if text, _ := parseResponseMood(resp); text != "" {
+		text, _, perr := parseResponseMood(resp)
+		if perr != nil {
+			log.Printf("phraser: PhraseQuery: %v", perr)
+			return "не знаю.", nil
+		}
+		if text != "" {
 			return text, nil
 		}
 		return resp, nil
@@ -223,17 +240,22 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
 	}
 	sys := p.querySystemPrompt()
 	prompt := fmt.Sprintf(
-		`The user asks: "%s". Your notes matching the query contain: "%s". Answer them naturally and briefly. If the notes don't answer the question, say so.`,
+		`Он спрашивает: "%s". В твоих заметках по этому вопросу написано: "%s". Ответь ему коротко и своими словами. Если в заметках ответа нет — так и скажи.`,
 		utterance, strings.Join(notes, `"; "`),
 	)
-	resp, err := p.chatWithSystem(ctx, sys, prompt, 256)
-	if err != nil {
+	resp, err := p.chatWithSystem(ctx, sys, prompt, 768)
+	text, _, perr := parseResponseMood(resp)
+	if err != nil || perr != nil {
+		// Read the notes out rather than ship a broken fragment.
+		if perr != nil {
+			log.Printf("phraser: PhraseQuery: %v", perr)
+		}
 		if len(notes) == 1 {
 			return "вот что я нашла: " + notes[0], nil
 		}
 		return "вот что я нашла: " + strings.Join(notes, "; "), nil
 	}
-	if text, _ := parseResponseMood(resp); text != "" {
+	if text != "" {
 		return text, nil
 	}
 	return resp, nil
@@ -256,12 +278,17 @@ func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history [
 	combined += utterance
 	msgs = append(msgs, chatMsg{Role: "user", Content: strings.TrimSpace(combined)})
 
-	resp, err := p.chatWithMessages(ctx, msgs, 512)
+	resp, err := p.chatWithMessages(ctx, msgs, 768)
 	if err != nil {
 		log.Printf("phraser: PhraseChat: %v", err)
 		return "поговорили.", nil
 	}
-	if text, _ := parseResponseMood(resp); text != "" {
+	text, _, perr := parseResponseMood(resp)
+	if perr != nil {
+		log.Printf("phraser: PhraseChat: %v", perr)
+		return "поговорили.", nil
+	}
+	if text != "" {
 		return text, nil
 	}
 	// fallback: plain text without JSON
@@ -274,11 +301,13 @@ func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history [
 // chatSystemPrompt returns the system prompt for conversational chat.
 // Prepends the shared context block when the phraser has one.
 func chatSystemPrompt(block func() string) string {
-	base := `You are maven, a self-hosted personal assistant. You're talking with your owner. 
-Keep replies brief (1-3 sentences) and natural. You're helpful, curious, and a little warm.
-Respond in the user's language (Russian or English, matching their last message). 
-Never roleplay emotions you don't have, but stay friendly. 
-Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}. "response" is your reply text; "mood" reflects your tone (neutral/happy/thinking/tired/confused).`
+	// No self-introduction here: the persona block prepended one line above
+	// already says who she is, same as router.KnowledgePrompt.
+	base := `Ты разговариваешь с хозяином. О себе говоришь в женском роде ("я подумала", "я рада"). Он мужчина: обращайся к нему на "ты", в мужском роде ("ты сказал", "ты забыл"). Никогда не "вы"/"ваш" и никогда "он"/"его" — ты говоришь ему, а не о нём.
+
+Отвечай по-русски, коротко: одна-три фразы, живым языком. Ты доброжелательная, тебе интересно, но чувства не изображай.
+
+Отвечай ТОЛЬКО одним объектом JSON: {"response": "...", "mood": "neutral"}. В "response" — твой ответ. В "mood" — ровно одно из: neutral, happy, thinking, tired, confused.`
 	return persona.Prepend(block, base)
 }
 
@@ -290,6 +319,7 @@ func (p *LLMPhraser) chatWithMessages(ctx context.Context, msgs []chatMsg, maxTo
 		Messages:    msgs,
 		Temperature: 0.7,
 		MaxTokens:   maxTokens,
+		Grammar:     p.grammar(),
 	}
 	body, err := json.Marshal(req)
 	if err != nil {
@@ -341,7 +371,12 @@ func (p *LLMPhraser) PhraseReminder(ctx context.Context, d loop.ReminderDecision
 	if err != nil {
 		return delivery.PhrasedReminder{}, err
 	}
-	body, mood := parseResponseMood(resp)
+	body, mood, perr := parseResponseMood(resp)
+	if perr != nil {
+		// Truncated JSON. Fall through to the reminder's own text.
+		log.Printf("phraser: PhraseReminder: %v", perr)
+		body, mood = "", ""
+	}
 	if body == "" {
 		// fallback: try old body/summary format
 		body, _ = parsePhrase(resp)
@@ -369,6 +404,43 @@ type chatReq struct {
 	Messages    []chatMsg `json:"messages"`
 	Temperature float64   `json:"temperature"`
 	MaxTokens   int       `json:"max_tokens"`
+	// Grammar is llama-server's `grammar` field (GBNF). Same wiring as
+	// internal/llm.Req.Grammar. Empty ⇒ unconstrained sampling.
+	Grammar string `json:"grammar,omitempty"`
+}
+
+// responseGrammar — GBNF constraining the model to the documented phrasing
+// contract and nothing else: {"response": "<text>", "mood": "<enum>"}.
+//
+// Without it a 0.8B answers roughly one chat turn in three with open reasoning
+// as plain text ("Thinking Process:" …), which no tag-stripper can remove and
+// which eats the token budget before the JSON closes. Modelled on
+// routeGrammar in internal/router/llmrouter.go so the two read alike.
+//
+// text accepts ANY codepoint except the two JSON must escape — the replies are
+// Russian, so an ASCII-only rule would make every reply empty. The escape rule
+// is what lets the model close a string it opened with a quote inside. Length
+// is bounded so a repetition loop truncates the field, not the JSON object.
+//
+// That bound was 400 and 400 was too tight. Measured against Qwen3.5-0.8B: on
+// "почему гром слышно позже молнии?" the reply came back exactly 400 characters
+// long, cut mid-word ("Нужно записать и,"), at every token cap from 256 to 2048.
+// So the token cap was never what stopped it — this rule was. 1000 characters is
+// roughly six Russian sentences, still short enough to stop a repetition loop.
+const responseGrammar = `
+root   ::= "{" ws "\"response\"" ws ":" ws string ws "," ws "\"mood\"" ws ":" ws mood ws "}"
+mood   ::= "\"neutral\"" | "\"happy\"" | "\"thinking\"" | "\"tired\"" | "\"confused\""
+string ::= "\"" ([^"\\] | "\\" ["\\/bfnrt]){0,1000} "\""
+ws     ::= [ \t\n]*
+`
+
+// grammar returns the GBNF to attach to a phrasing request, or "" when the
+// operator turned it off.
+func (p *LLMPhraser) grammar() string {
+	if p.cfg.NoGrammar {
+		return ""
+	}
+	return responseGrammar
 }
 
 type chatResp struct {
@@ -393,6 +465,7 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma
 		},
 		Temperature: 0.7,
 		MaxTokens:   maxTokens,
+		Grammar:     p.grammar(),
 	}
 	body, err := json.Marshal(req)
 	if err != nil {
@@ -467,7 +540,9 @@ func (p *LLMPhraser) systemPrompt() string {
 // querySystemPrompt returns the system prompt for PhraseQuery (notes + general
 // knowledge). Prepends the configured persona when set.
 func (p *LLMPhraser) querySystemPrompt() string {
-	base := "You are maven, a self-hosted personal assistant answering from your notes. Answer briefly and naturally in Russian starting with \"вот что я нашла: \". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}."
+	// No self-introduction here: the persona block prepended one line above
+	// already says who she is, same as router.KnowledgePrompt.
+	base := "Ты отвечаешь ему по своим заметкам. Отвечай по-русски, коротко и своими словами, начинай с \"вот что я нашла: \". О себе — в женском роде (\"нашла\", \"записала\"). Он мужчина, обращайся к нему на \"ты\". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}."
 	return persona.Prepend(p.cfg.ContextBlock, base)
 }
 
@@ -589,21 +664,39 @@ type responseMood struct {
 	Mood     string `json:"mood"`
 }
 
+// errBrokenJSON — the model started a JSON object and never finished it.
+// That is a failed generation, not a reply. Callers must use their fallback.
+var errBrokenJSON = fmt.Errorf("phraser: model output starts as JSON but does not parse")
+
 // parseResponseMood extracts {"response","mood"} from LLM output, tolerant
-// of thinking tokens and extra text before/after the JSON block. Returns
-// ("", "") when no valid JSON is found.
-func parseResponseMood(raw string) (response, mood string) {
+// of thinking tokens and extra text before/after the JSON block.
+//
+// Three outcomes:
+//   - parsed fine → the fields, nil error.
+//   - output never looked like JSON → ("", "", nil). The caller may ship it
+//     as-is; small models sometimes answer in bare prose and that is fine.
+//   - output starts with "{" but does not parse → errBrokenJSON. The grammar
+//     guarantees a valid *prefix*, so a generation that hits the token cap
+//     mid-object comes back as a fragment like `{` or `{\n  "`. Shipping that
+//     as a reply is the bug this error exists to stop.
+func parseResponseMood(raw string) (response, mood string, err error) {
 	cleaned := strings.TrimSpace(raw)
 	start := strings.Index(cleaned, "{")
 	end := strings.LastIndex(cleaned, "}")
 	if start < 0 || end < 0 || end <= start {
-		return "", ""
+		if strings.HasPrefix(cleaned, "{") {
+			return "", "", errBrokenJSON
+		}
+		return "", "", nil
 	}
 	var parsed responseMood
-	if err := json.Unmarshal([]byte(cleaned[start:end+1]), &parsed); err != nil {
-		return "", ""
+	if e := json.Unmarshal([]byte(cleaned[start:end+1]), &parsed); e != nil {
+		if strings.HasPrefix(cleaned, "{") {
+			return "", "", errBrokenJSON
+		}
+		return "", "", nil
 	}
-	return parsed.Response, parsed.Mood
+	return parsed.Response, parsed.Mood, nil
 }
 
 func parsePhrase(raw string) (body, summary string) {