Compare commits

...

22 Commits

Author SHA1 Message Date
kami 6b67e6f3c2 Word nudges from templates by default, model optional
DEPRECATION, flagged not asked: LLM-phrased nudges are no longer the default.
LLMPhraser.PhraseNudge now returns a hand-written Russian template. The model
still phrases chat, queries and reminders — only nudges moved.

Why: measured over many runs, Qwen3.5-0.8B wrote formal "вы" and plural
imperatives, used masculine self-reference, and invented facts and units
(90-95 seconds to boil an egg). A nudge is five words of known content, so
generation buys nothing and risks the persona every time. Templates score
15/15 on the nudge fixture, the model 11-13/15.

Nothing is deleted: the prompt, the fallbacks and the whole LLM nudge path
stay. Set phraser.llm_nudges = true in deploy/mavend.json to get them back.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 18:21:52 +04:00
kami 13e5170e9e Hand-written Russian nudge templates plus a picker
Nudge wording as data instead of generation. The wording lives in
internal/phraser/nudges_ru_v1.json (embedded), about 10 variants per rule:
water, meal, break, service_down, netdata_critical, routine:, morning:, plus
a contentless default. That JSON is long because it is data — the owner can
edit any line of Russian without touching Go.

The picker:
- random, but never the same variant twice in a row for the same rule
- deterministic when seeded (math/rand with an injectable source)
- fills {since} / {service} / {what} from the candidate, and skips any variant
  whose value is missing, so no raw placeholder can reach the piper voice
- {since} is spelled out in words ("полтора часа", "семь часов"), because
  "3 ч" is wrong in a Russian voice

Scores 15/15 on the existing nudge fixture, on every seed swept. Nothing is
wired yet — that is the next commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 18:18:55 +04:00
kami aa8f5b2ee2 Make the nonempty check look for actual words
It scored 27/27 on a run where two replies were "{" and "{\n  \"". It only
tested that the string was not blank, so punctuation counted as content and
the worst replies of the run passed the first check.

Now a reply needs at least one letter, Cyrillic or Latin. Latin counts
because answers about ssd or vpn are legitimately part English.

Digits alone fail too. The same run answered "сколько варить яйцо
вкрутую?" with "15-16" — no unit, no words, and the wrong number as well.
That is not something she said.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:57:39 +04:00
kami d7cdcb63bd Stop shipping half-written JSON as a reply
Two bugs, one symptom. A run of the talk eval produced replies that were
literally "{" and "{\n  \"" — those strings went out as things Maven said.

First bug: the parser could not tell "the model answered in plain prose"
from "the model started a JSON object and got cut off". Both came back as
empty, and every caller then shipped the raw text. Now an unfinished object
returns an error and each caller uses its own fallback instead. Bare prose
with no JSON in it still passes through, because small models do sometimes
answer that way and the reply is fine.

Second bug, and the actual cause: the grammar capped the response field at
400 characters. I measured it against Qwen3.5-0.8B at three different token
caps — 256, 768 and 2048 — and the reply came back exactly 400 characters
every time, cut mid-word. So the token limit was never what stopped it.
The bound is 1000 now, about six Russian sentences, still low enough to cut
off a repetition loop.

Token caps go from 256 to 768 on the chat and query paths so 1000
characters of Russian actually fits. The nudge path keeps its own cap; a
nudge is meant to be one sentence.

Note: cmd/mavend/replier_llm.go has its own copy of this parser with the
same bug. Left alone here so this commit stays small — that duplicate is
Vikunja #396.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:56:55 +04:00
kami c7dadc97d9 Write the chat and notes prompts in Russian
The reply has to be Russian, but two of the phrasing prompts told her
what to do in English. Both are Russian now, in the same style as the
nudge prompt that already works better.

Also dropped the "you are maven, a self-hosted personal assistant"
line from both. The persona block right above it already says who she
is, so it was said twice.

The JSON part is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:38:36 +04:00
kami c9d88c152e Drop "never phones home" as a hard rule
The owner's call, 2026-07-31: a 0.8B model does not know enough about the
world to be useful without reading something. So she may now read external
sources to answer world questions.

What replaces the old rule, in all three docs:

- No telemetry, no cloud model, no third-party account. Unchanged.
- Local first: the Kiwix ZIMs on the box before anything on the network.
- External search is allowed but off unless configured, same as weather
  and telegram.
- His notes and facts are never search input. Only the utterance goes out
  — never the persona block, the history, or matched notes.

Docs only, no code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:34:53 +04:00
kami 1890ff5d5d Constrain the phrasing output with a GBNF grammar
The 0.8B answered about one chat turn in three with open reasoning as plain text, so no JSON ever closed and the fallback shipped "Thinking Process:" to the user. A grammar makes that output impossible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:18:06 +04:00
kami 0110e9bc8c Report every address break, and stop -те verbs blinding the check
From a real reply in a nudge eval run: "Смотрите на его потребление
воды" is a plural imperative AND third person about him. Only the plural
printed.

Two separate faults. The check returned on its first hit, so the second
break stayed invisible and the failure read as milder than it was; it now
joins them. And "его" was not detected at all — looksVerb knows the
-й/-йте imperative but not the -те plural, so "смотрите" counted as the
person being talked about, which is what an antecedent means here.
pluralVerb already knows that form, so the antecedent test uses it too.

Third time a verb form has blinded this check. A fourth means it wants a
morphology table rather than another suffix.
2026-07-31 16:52:16 +04:00
kami 50ca8c8b5a Score the chat, query and knowledge phrasing paths (#395)
The phrasing fixture was 15 nudge cases, so every prompt change we
measured only told us about nudges. But the shared context block sits in
front of five prompts, and three of them — chat, note query, general
knowledge — had no scorer at all. Those are the long free-form replies,
where a persona break is most likely and where nothing could see one.

27 cases, nine per path. Nine rather than five because the nudge fixture
already cannot resolve a change smaller than about three cases, and a
per-path score off five would be worse.

Reuses the persona checks instead of copying them. Length, mood and
"no questions" are left out on purpose: these paths return no mood, and
a follow-up question is a feature in chat, not a fault.

The run refuses to score unless the model answers before and after it.
PhraseChat and PhraseQuery swallow model errors and return a canned
string, so without that guard a dead server produces a full report with
zero errors and a bad score — which reads as bad phrasing rather than as
nothing measured. Vikunja #397 is the real fix.
2026-07-31 16:51:52 +04:00
kami de09471421 Merge the shared prompt context block 2026-07-31 16:07:35 +04:00
kami d65c16a567 Don't tell her she can't talk
The block listed what she can do and ended with "nothing else". It sits
in front of the chat and general-knowledge prompts too, so that told her
to refuse the exact thing those prompts are for. Talking is now first in
the list, and the closing line limits ACTIONS rather than everything.

Also dropped the self-introduction from the knowledge prompt. It said
"Мавена, персональный ассистент" — a different name and a masculine
noun, right after the block says she is Maven and feminine. Identity
lives in the block now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 16:07:35 +04:00
kami 062d4252ef Tell her what she can actually do
The context block now lists her real capabilities: reminders, notes and
facts (write and recall), and the calendar — all three are code paths in
mavend today. Weather, telegram and shell acts are listed only when the
config actually has them, because offering something she cannot do is
worse than staying quiet about it.

Also drops the pronouns from the optional name/city line. The block's
own "ты" is Maven, so "тебя зовут" read as her name and "его" would have
shown her the third-person form she must never use about him. They are
plain labels now.

Vikunja #394.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 15:57:58 +04:00
kami 2c27e2ce1f Give every prompt one shared context block
The "address him as ты" rule had only reached two of the five system
prompts. Instead of pasting it into the other three (five copies drift —
that is how this happened), there is now one block, in internal/persona,
prepended to all five: nudges, action replies, chat, note queries and
general knowledge.

The block says who he is and how to address him (a man, always "ты",
never "вы", never "он" about him; Maven stays feminine), plus the
current local date and time. It is rendered fresh each turn because the
time changes, and it is correct with an empty config — the address and
gender rules are defaults in code. Config only adds optional facts:
owner_name, city, and the existing free-text `persona` string, which is
now the static half of the block.

Russian even in front of the English prompts: the rules are Russian
grammar, so they read best stated in Russian, and there is one copy.

Vikunja #394.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 15:55:30 +04:00
kami ccc5cba2a3 Merge the example-led prompt finding 2026-07-31 15:41:01 +04:00
kami 89d83c0b11 Record the example-led nudge prompt experiment (#393) — it made things worse
Tried rewriting the nudge prompt to lead with five on-topic examples instead
of rules. Three eval runs each side: before 12/13/14 of 15, after 11/12/11.
The loss is all in the address check — formal "вы" and plural imperatives came
back once the "говоришь на ты" rule stopped being its own sentence, and the
on-topic examples leaked their wording into the wrong cases.

Prompt reverted. Only the finding is committed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 15:40:18 +04:00
kami a97f554802 Merge the informal address prompt rule 2026-07-31 14:54:20 +04:00
kami f4de2fc5e1 Don't let a verb count as the person being talked about
The third-person check asks whether anyone else was named before "он".
A nudge is mostly verbs, and they were counted as possible people, so
"попробуй встать и отдохнуть — у него есть перерыв" passed. Infinitives
and imperatives now join past tense as words that cannot be a person.

A plain noun before the pronoun still blinds it. That needs a parser,
and the comment says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:54:20 +04:00
kami eef5d4da4f Tell the phraser to speak to him informally, singular
The prompts stated the feminine self-reference rule but never said whom she is
speaking to, so the model produced formal plural ("Жду вас") and talked about
him in third person ("Он не ел 11 дней"). Adds the address rule right next to
the feminine one, in the nudge prompt and the confirmation prompt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:52:23 +04:00
kami 09f1696fce Merge the eval label and kill script fixes 2026-07-31 14:32:45 +04:00
kami 80f7322294 Don't fail when docker confirms nothing is running
"Nothing on the host" meant two different things and the script treated
them the same. If docker answers and names no running containers, Maven
really is down and the script should say so and exit 0. Only when docker
cannot be asked is the answer unknown, and that is the case that must
fail loudly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:32:45 +04:00
kami 02e8786695 Stop the containers instead of claiming success (#380)
docker-compose.yml has no 'pid: host', so each container has its own PID
namespace and pkill on the host matches nothing inside them. The script
then printed "All services gracefully stopped" while mavend, its
llama-server and the rest were still running.

Now it checks for running compose containers first and stops them with
docker compose. If it cannot ask docker and finds nothing to kill on the
host, or anything survives the kill, it says so and exits non-zero
instead of claiming success. The bare-metal path is unchanged apart from
verifying the SIGKILL actually worked, and no longer risks killing the
shell it was launched from.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:30:51 +04:00
kami a788ca3915 Label eval runs with the model the server actually loaded (#379)
The phrasing eval printed "llm (0.8B, ...)" no matter which gguf
llama-server had loaded, so two runs of two different models came out
named the same and were easy to mix up when comparing.

It now asks llama-server over /v1/models, same as the router eval
already did. The helper moved to internal/llm so both share it, and it
now errors instead of returning a blank name when the id field is
missing — an unreachable server gets labelled "unknown-model", never a
plausible-looking guess.

Both eval paths stay opt-in behind MAVEN_LLM_URL; no server needed for
go test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:25:28 +04:00
35 changed files with 2435 additions and 113 deletions
+15 -1
View File
@@ -82,9 +82,23 @@ workspace enforces that the Go and relabelling prompts remain identical.
## Non-goals (hard constraints)
Never phones home. Not a nag, not autonomous. Maven's persona is **feminine** — Russian
Not a nag, not autonomous. Maven's persona is **feminine** — Russian
self-reference must use feminine forms (the user is male; see memory `maven-persona-gender`).
**"Never phones home" is DEPRECATED** (owner's call, 2026-07-31). It used to be a hard
constraint and it is not one any more: a 0.8B — and a 1.7B — does not know enough to answer
world questions, so she needs to read external sources. What replaces it:
- **No telemetry, no cloud model, no third-party account.** That part never changes. Nothing
about Maven is reported to anyone, and inference stays on the box.
- **Local sources first.** Kiwix ZIMs on homesrv (Wikipedia, ifixit) before anything on the
network. Reading beats recalling for a small model, and a local read costs nothing.
- **External search is allowed and off unless configured**, like the weather and telegram
capabilities.
- **His notes and facts are never search input.** Looking up why the sky is blue and sending
his stored personal notes to an upstream engine are different acts. Only the utterance goes
out, never the persona block, history, or matched notes.
## Web UI conventions
Server-rendered pages share `cmd/mavweb/static/ui.css` (served at `/ui.css`) and the `nav`
+10 -4
View File
@@ -15,7 +15,8 @@
**Maven** — self-hosted personal assistant. Manages your day, acts on your
homelab. One daemon on homesrv (always-on, not the workstation), multiple
client surfaces. All local, never phones home.
client surfaces. Inference and data stay on the box; she may READ external
sources (see Non-goals — "never phones home" is deprecated).
Primary name is "Maven", with feminine-gendered Russian self-reference
("она", "меня", "помогла"). Clients may choose their own UI label. Consistent
@@ -35,8 +36,13 @@ Inside boundary — the ones that actually constrain the build:
she records. A confident wrong fact is worse than a known gap.
- **Not a nag** — she'd rather miss a nudge than be mutable. Shuts up when
uncertain. Load-bearing.
- **Not a stranger** — runs on your stuff, your model, your data. Never
phones home.
- **Not a stranger** — runs on your stuff, your model, your data. No
telemetry, no cloud model, no third-party account. She may READ external
sources to answer world questions (Kiwix first, then optional search); she
never reports anything about you to anyone, and your notes and facts are
never used as search input. **"Never phones home" as an absolute is
deprecated** — owner's call, 2026-07-31: a small model does not know enough
to be useful without reading.
- **Not a relationship** — mom-tone is a function that makes nudges land, not
emotional company. Names the drift a warm small model falls into.
@@ -458,7 +464,7 @@ decides *insistence*. Both are needed.
sev ≤ 2 drops on away, sev ≥ 3 holds: a missed water nudge is noise, a missed
backup failure isn't. Away-channels (ntfy/telegram) leave the box — the one
path that crosses "never phones home," through your own relay. **Minimal
path that leaves the box for a person to see, through your own relay. **Minimal
body** — "disk low on homesrv," not detail; don't make notifications a
shoulder-surf exfil surface.
+6 -3
View File
@@ -103,14 +103,17 @@ eval-router:
eval-recall:
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
# eval-phrasing -- score nudge phrasing (internal/phraser/eval). Verbose so the
# eval-phrasing -- score nudge phrasing AND the conversational paths (chat,
# query, general knowledge) in internal/phraser/eval. Verbose so the
# report and every generated message land in the terminal. With no environment
# it scores the deterministic Stub only, which is what CI runs. Set
# MAVEN_LLM_URL to add the resident model:
# MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
# The model run is slow (minutes) -- the timeout is raised to match.
# The model run is slow (minutes) -- the timeout is raised to match. It covers
# two fixtures now (15 nudges + 27 conversational cases, and the chat replies are
# the long ones), hence 90m rather than 40m.
eval-phrasing:
$(GO) test -v -count=1 -timeout 40m ./internal/phraser/eval/
$(GO) test -v -count=1 -timeout 90m ./internal/phraser/eval/
# eval-models — score ONE llama-server against the same fixture, for the
# resident-model bake-off (#278, #250). Start a server with the gguf you want,
+29
View File
@@ -103,6 +103,35 @@ but a large part of the jump is that failure now degrades into Russian instead o
The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge
that never says food.
## Tried and reverted: an example-led nudge prompt (#393)
The idea was that a 0.8B copies examples better than it follows rules, so the nudge prompt
was rewritten to lead with five on-topic examples (water, break, pills, morning, service) and
the prose rules were compressed to pay for the tokens: 1190 chars down to 986.
It measured **worse**, three runs each side, same llama-server, same fixture:
| run | before | after |
|---|---|---|
| 1 | 12/15 (address 14) | 11/15 (address 13) |
| 2 | 13/15 (address 15) | 12/15 (address 15) |
| 3 | 14/15 (address 15) | 11/15 (address 12) |
`feminine` and `hisgender` were 15/15 on all six runs, so they measure nothing here. The
regression is all in `address`: 44/45 before, 40/45 after. Formal "вы"/"ваше" and plural
imperatives came back, and so did `"..."`.
Two likely causes, both about the same thing — **examples do not carry a prohibition**. The
old prompt spent a whole sentence on «говоришь на "ты", в единственном числе»; the new one
demoted that to one item in a long "никогда" list, and the model stopped obeying it. And
making the examples on-topic let their *wording* leak: a break case came back as
«Вы давно не пили воду. Выпей стакан.» — the water example, verbatim, in the wrong slot.
That is exactly the failure the laundry/laptop examples were chosen to avoid.
Change reverted. What survives is the measurement: a rule the model must obey needs its own
sentence, and examples must stay off-topic. Also note the before side alone spans 1214 of
15 — this fixture cannot resolve anything smaller than about three cases.
## Broken, found, not fixed
1. ~~**`checkFeminine` only catches half the constraint.**~~ **Fixed** (#381). It scanned for
+4 -1
View File
@@ -90,4 +90,7 @@ later* is the worker + RAG.
4. **Deferred work** — larger reasoner, custom Piper voice and other expansions.
## Non-goals (unchanged)
Never phones home. Not a nag. Not autonomous. Feminine-gendered RU self-ref.
Not a nag. Not autonomous. Feminine-gendered RU self-ref. No telemetry, no
cloud model, no third-party account — but she MAY read external sources to
answer world questions (Kiwix first, search optional). "Never phones home" as
an absolute is deprecated, owner's call 2026-07-31; see CLAUDE.md § Non-goals.
+41 -21
View File
@@ -58,6 +58,7 @@ import (
"github.com/kami/maven/internal/delivery/telegramsink"
"github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/store"
"github.com/kami/maven/internal/webauthn"
@@ -271,13 +272,14 @@ func run(args []string) error {
phr = phraser.NewStub()
if cfg.Phraser != nil {
pc := phraser.Config{
ModelPath: cfg.Phraser.ModelPath,
BinPath: cfg.Phraser.BinPath,
Listen: cfg.Phraser.Listen,
NGpuLayers: cfg.Phraser.NGpuLayers,
NCtx: cfg.Phraser.NCtx,
Timeout: time.Duration(cfg.Phraser.Timeout),
Persona: personaFromCfg(cfg),
ModelPath: cfg.Phraser.ModelPath,
BinPath: cfg.Phraser.BinPath,
Listen: cfg.Phraser.Listen,
NGpuLayers: cfg.Phraser.NGpuLayers,
NCtx: cfg.Phraser.NCtx,
Timeout: time.Duration(cfg.Phraser.Timeout),
LLMNudges: cfg.Phraser.LLMNudges,
ContextBlock: contextBlockFn(cfg, time.Now),
}
if pc.BinPath == "" {
pc.BinPath = "llama-server"
@@ -441,13 +443,14 @@ func run(args []string) error {
phr = phraser.NewStub()
if cfg.Phraser != nil {
pc := phraser.Config{
ModelPath: cfg.Phraser.ModelPath,
BinPath: cfg.Phraser.BinPath,
Listen: cfg.Phraser.Listen,
NGpuLayers: cfg.Phraser.NGpuLayers,
NCtx: cfg.Phraser.NCtx,
Timeout: time.Duration(cfg.Phraser.Timeout),
Persona: personaFromCfg(cfg),
ModelPath: cfg.Phraser.ModelPath,
BinPath: cfg.Phraser.BinPath,
Listen: cfg.Phraser.Listen,
NGpuLayers: cfg.Phraser.NGpuLayers,
NCtx: cfg.Phraser.NCtx,
Timeout: time.Duration(cfg.Phraser.Timeout),
LLMNudges: cfg.Phraser.LLMNudges,
ContextBlock: contextBlockFn(cfg, time.Now),
}
if pc.BinPath == "" {
pc.BinPath = "llama-server"
@@ -601,12 +604,29 @@ func run(args []string) error {
return nil
}
// personaFromCfg extracts the voice persona from the config, or returns ""
// when voice isn't configured. Used to pass a character prompt into the
// LLM phraser without requiring voice to be enabled.
func personaFromCfg(cfg *config.Config) string {
if cfg.Voice != nil {
return cfg.Voice.Persona
// personaFacts reads the optional, deployment-specific facts (his name, his
// city, the free-text persona string) out of the config. Everything here may
// be empty — the context block is correct without any of it.
func personaFacts(cfg *config.Config) persona.Facts {
f := persona.Facts{
// Telegram lives outside the voice block, so it counts either way.
Telegram: cfg.Telegram != nil && cfg.Telegram.BotToken != "" && cfg.Telegram.ChatID != "",
}
return ""
if cfg.Voice == nil {
return f
}
f.OwnerName = cfg.Voice.OwnerName
f.City = cfg.Voice.City
f.Static = cfg.Voice.Persona
// Same test wireVoice uses to pick the real provider over the stub.
f.Weather = cfg.Voice.Weather != nil && cfg.Voice.Weather.Provider == "open-meteo"
f.Tools = len(cfg.Voice.Tools) > 0
return f
}
// contextBlockFn returns the per-turn renderer of the shared context block.
// Per turn, not once at startup, because the block states the current time.
func contextBlockFn(cfg *config.Config, now func() time.Time) func() string {
f := personaFacts(cfg)
return func() string { return f.Block(now()) }
}
+9 -4
View File
@@ -7,6 +7,7 @@ import (
"time"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/voice"
)
@@ -23,13 +24,17 @@ type completer interface {
type llmReplier struct {
c completer
stub *voice.StubReplier
// block renders the shared context block per turn (who he is, the time).
// nil ⇒ the prompt stands alone.
block func() string
}
func newLLMReplier(c completer) *llmReplier {
return &llmReplier{c: c, stub: voice.NewStubReplier()}
func newLLMReplier(c completer, block func() string) *llmReplier {
return &llmReplier{c: c, stub: voice.NewStubReplier(), block: block}
}
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Владелец — мужчина, говоришь с ним на "ты", в единственном числе; никогда не "вы"/"ваш" и не "он"/"его". Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"}
Никогда не пиши "..." в поле response.`
@@ -40,7 +45,7 @@ func (r *llmReplier) Reply(d router.Decision) string {
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
out, err := r.c.Complete(ctx, llm.Req{
System: replySystem,
System: persona.Prepend(r.block, replySystem),
User: replyContext(d),
MaxTokens: 512,
RepeatPenalty: 1.3,
+5 -5
View File
@@ -17,7 +17,7 @@ type mockCompleter struct {
func (m mockCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return m.out, m.err }
func TestLLMReplierReturnsLLMReply(t *testing.T) {
r := newLLMReplier(mockCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`})
r := newLLMReplier(mockCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}, nil)
got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
if got != "записала, кофе закончился" {
t.Errorf("got %q, want %q", got, "записала, кофе закончился")
@@ -25,7 +25,7 @@ func TestLLMReplierReturnsLLMReply(t *testing.T) {
}
func TestLLMReplierFallsBackToPlainText(t *testing.T) {
r := newLLMReplier(mockCompleter{out: "записала, кофе закончился"})
r := newLLMReplier(mockCompleter{out: "записала, кофе закончился"}, nil)
got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
if got != "записала, кофе закончился" {
t.Errorf("got %q, want %q", got, "записала, кофе закончился")
@@ -33,7 +33,7 @@ func TestLLMReplierFallsBackToPlainText(t *testing.T) {
}
func TestLLMReplierFallsBackToStubOnError(t *testing.T) {
r := newLLMReplier(mockCompleter{err: errTestLLMDown})
r := newLLMReplier(mockCompleter{err: errTestLLMDown}, nil)
noteDec := router.Decision{Intent: router.IntentNote}
got := r.Reply(noteDec)
want := voice.NewStubReplier().Reply(noteDec)
@@ -43,7 +43,7 @@ func TestLLMReplierFallsBackToStubOnError(t *testing.T) {
}
func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) {
r := newLLMReplier(mockCompleter{out: ""})
r := newLLMReplier(mockCompleter{out: ""}, nil)
noteDec := router.Decision{Intent: router.IntentNote}
got := r.Reply(noteDec)
want := voice.NewStubReplier().Reply(noteDec)
@@ -53,7 +53,7 @@ func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) {
}
func TestLLMReplierClarifyUsesStub(t *testing.T) {
r := newLLMReplier(mockCompleter{out: "я всё поняла"})
r := newLLMReplier(mockCompleter{out: "я всё поняла"}, nil)
clarifyDec := router.Decision{Clarify: true}
got := r.Reply(clarifyDec)
want := voice.NewStubReplier().Reply(clarifyDec)
+1 -1
View File
@@ -246,7 +246,7 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
// ----- replier (LLM-backed when the engine is on, Stub floor otherwise) -----
replier := voice.Replier(voice.NewStubReplier())
if llmClient != nil {
replier = newLLMReplier(llmClient)
replier = newLLMReplier(llmClient, contextBlockFn(cfg, time.Now))
}
// ----- the handler (the reactive path; closes over stt / tts / router / coreAPI / memory) -----
+2 -1
View File
@@ -10,7 +10,8 @@
"bin_path": "llama-server",
"n_gpu_layers": 99,
"n_ctx": 2048,
"timeout": "60s"
"timeout": "60s",
"llm_nudges": false
},
"telegram": {
+13
View File
@@ -302,6 +302,13 @@ type VoiceConfig struct {
// Russian self-reference). Example: "Be formal and answer in English only."
Persona string `json:"persona,omitempty"`
// OwnerName / City — optional facts about the owner, added to the shared
// context block (internal/persona). Empty is fine: the block still states
// who he is grammatically (a man, addressed as "ты") and the current time.
// Nothing about correct behaviour may depend on these being filled in.
OwnerName string `json:"owner_name,omitempty"`
City string `json:"city,omitempty"`
// Weather — the weather provider config. nil ⇒ the daemon wires
// the stub provider (returns ErrNotConfigured — "погода не настроена").
// Set provider to "open-meteo" to use the keyless Open-Meteo API.
@@ -362,6 +369,12 @@ type PhraserConfig struct {
NGpuLayers int `json:"n_gpu_layers,omitempty"`
NCtx int `json:"n_ctx,omitempty"`
Timeout Duration `json:"timeout,omitempty"`
// LLMNudges — let the model word nudges again. Off by default: nudges are
// worded from hand-written Russian templates now (the model broke the
// persona and invented units). Chat, query and reminder phrasing always go
// through the model regardless. See phraser.Config.LLMNudges.
LLMNudges bool `json:"llm_nudges,omitempty"`
}
// EmbedderConfig — paths for the ONNX multilingual embedder. The daemon
+21
View File
@@ -35,6 +35,27 @@ func TestLoadDefaults(t *testing.T) {
}
}
// Nudges come from templates unless the config says otherwise.
func TestPhraserLLMNudgesDefaultsOff(t *testing.T) {
p := writeConfig(t, `{"phraser":{"model_path":"/tmp/m.gguf"}}`)
c, err := Load(p)
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.Phraser.LLMNudges {
t.Error("llm_nudges defaults on; templates must be the default")
}
p = writeConfig(t, `{"phraser":{"model_path":"/tmp/m.gguf","llm_nudges":true}}`)
c, err = Load(p)
if err != nil {
t.Fatalf("Load: %v", err)
}
if !c.Phraser.LLMNudges {
t.Error("llm_nudges:true did not parse")
}
}
func TestLoadDurationsParse(t *testing.T) {
p := writeConfig(t, `{"tick_interval":"90s","repeat_interval":"10m"}`)
c, err := Load(p)
@@ -1,4 +1,4 @@
package eval
package llm
import (
"context"
@@ -6,8 +6,20 @@ import (
"fmt"
"net/http"
"strings"
"time"
)
// UnknownModel is the label to print when the server would not say what it has
// loaded. Deliberately ugly: an honest "unknown" is fine, a plausible-looking
// but wrong model name is the bug this whole file exists to prevent.
const UnknownModel = "unknown-model"
// llama-server is local, so never send this through a proxy: this box's
// http_proxy answers 503 for loopback, which would look like "server won't say
// which model it has" when the server is right there and fine.
// A Transport with no Proxy set bypasses http_proxy entirely.
var modelHTTP = &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{}}
// ModelID asks llama-server which model it has loaded, so a scoring run can
// label itself. Without this a bake-off between two models produces two tables
// that look identical, and the operator has to remember which server was up.
@@ -19,7 +31,7 @@ func ModelID(ctx context.Context, base string) (string, error) {
if err != nil {
return "", err
}
resp, err := http.DefaultClient.Do(req)
resp, err := modelHTTP.Do(req)
if err != nil {
return "", err
}
@@ -38,14 +50,21 @@ func ModelID(ctx context.Context, base string) (string, error) {
if len(out.Data) == 0 {
return "", fmt.Errorf("models: empty list")
}
return shortModelID(out.Data[0].ID), nil
short := shortModelID(out.Data[0].ID)
if short == "" {
// Server answered but the id field was missing or blank. Say so
// instead of handing back an empty label that reads as a real name.
return "", fmt.Errorf("models: no id in response")
}
return short, nil
}
// shortModelID trims the path and the .gguf suffix — llama-server reports the
// file name it was started with, which is too long for a table header.
func shortModelID(id string) string {
id = strings.TrimSpace(id)
if i := strings.LastIndexAny(id, "/\\"); i >= 0 {
id = id[i+1:]
}
return strings.TrimSuffix(id, ".gguf")
return strings.TrimSpace(strings.TrimSuffix(id, ".gguf"))
}
+64
View File
@@ -0,0 +1,64 @@
package llm
import (
"context"
"net/http"
"net/http/httptest"
"testing"
)
// The point of these tests: a wrong-but-plausible model label is the bug, so
// every path that cannot learn the real name must return an error instead of a
// guess. No llama-server needed — a stub server stands in.
func TestModelID(t *testing.T) {
cases := []struct {
name string
body string
code int
want string // "" ⇒ expect an error
}{
{"full path", `{"data":[{"id":"/mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf"}]}`, 200, "Qwen3.5-0.8B.Q4_K_M"},
{"bare name", `{"data":[{"id":"LFM2.5-1.2B"}]}`, 200, "LFM2.5-1.2B"},
{"empty list", `{"data":[]}`, 200, ""},
{"id missing", `{"data":[{}]}`, 200, ""},
{"id blank", `{"data":[{"id":" "}]}`, 200, ""},
{"server error", `nope`, 500, ""},
{"not json", `<html>`, 200, ""},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/models" {
t.Errorf("asked for %s, want /v1/models", r.URL.Path)
}
w.WriteHeader(c.code)
_, _ = w.Write([]byte(c.body))
}))
defer srv.Close()
got, err := ModelID(context.Background(), srv.URL+"/")
if c.want == "" {
if err == nil {
t.Fatalf("want an error, got label %q", got)
}
return
}
if err != nil {
t.Fatalf("ModelID: %v", err)
}
if got != c.want {
t.Errorf("got %q, want %q", got, c.want)
}
})
}
}
func TestModelIDUnreachable(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
url := srv.URL
srv.Close() // nothing listening now
if got, err := ModelID(context.Background(), url); err == nil {
t.Fatalf("want an error from a dead server, got label %q", got)
}
}
+138
View File
@@ -0,0 +1,138 @@
// Package persona builds the one shared context block that goes in front of
// every LLM system prompt: who the owner is, how to address him, and what
// time it is right now.
//
// Why one block and not a line pasted into each prompt: there are five
// prompts (nudges, action replies, chat, note queries, general knowledge) and
// the "address him as ты" rule had only reached two of them. Five copies drift.
// One block cannot.
//
// The rules here are defaults in code, not config. Maven is feminine and the
// owner is a man addressed informally — that is a hard constraint of the
// product, so it must hold with an empty config file. Config only ADDS
// optional facts (his name, his city).
package persona
import (
"fmt"
"strings"
"time"
)
// Facts — the optional, deployment-specific half of the block. All fields may
// be empty; the block is still correct and useful without them.
type Facts struct {
OwnerName string // his name, e.g. "Ками"
City string // where he is, e.g. "Москва"
Static string // the free-text `persona` config string, appended verbatim
// The two config-gated capabilities. They are listed only when this
// deployment actually has them, because a capability she names and cannot
// do is worse than one she never mentions.
Weather bool // an open-meteo provider is configured
Telegram bool // a telegram bot token + chat id are configured
Tools bool // at least one shell act is on the allowlist
}
var ruWeekdays = [...]string{"воскресенье", "понедельник", "вторник", "среда", "четверг", "пятница", "суббота"}
var ruMonths = [...]string{
"января", "февраля", "марта", "апреля", "мая", "июня",
"июля", "августа", "сентября", "октября", "ноября", "декабря",
}
// Block renders the context block for one turn. Russian even in front of the
// English prompts: the rules it states are Russian grammar (ты/тебя, feminine
// verbs), and a Russian rule reads best stated in Russian.
//
// Keep it short. It ships on every turn to a 0.8B on laptop CPU, so every
// line here is latency.
func (f Facts) Block(now time.Time) string {
var b strings.Builder
b.WriteString("Ты — Maven, домашняя ассистентка. О себе говоришь в женском роде: \"я записала\", \"я проверила\".\n")
// The address form gets its own line. It is the thing that kept getting
// lost when it was buried in prose.
b.WriteString("ОБРАЩЕНИЕ: владелец — мужчина, всегда на \"ты\" (ты, тебя, тебе, твой) и в единственном числе (\"выпей\", \"посмотри\"). Никогда \"вы\"/\"вас\"/\"ваш\". Никогда \"он\"/\"его\" о нём — ты говоришь ему, а не о нём. Глаголы о нём — в мужском роде (\"ты забыл\").\n")
if who := f.who(); who != "" {
b.WriteString(who + "\n")
}
b.WriteString(fmt.Sprintf("Сейчас: %s, %d %s %d, %02d:%02d (местное время).\n",
ruWeekdays[int(now.Weekday())], now.Day(), ruMonths[int(now.Month())-1], now.Year(),
now.Hour(), now.Minute()))
b.WriteString("Умеешь: " + strings.Join(f.can(), "; ") +
". Других ДЕЙСТВИЙ не умеешь — если просят такое, скажи прямо.\n")
if s := strings.TrimSpace(f.Static); s != "" {
b.WriteString(s + "\n")
}
return b.String()
}
// can lists what she can really do. Every entry here is a code path that
// exists in the daemon today:
// - reminders: IntentReminder → CoreAPI.CreateReminder, fired by the tick.
// - notes and facts: IntentNote/IntentFact write, IntentQuery reads them back.
// - calendar: IntentQuery answers "что у меня сегодня" from CalendarEvents.
// - weather / telegram / shell acts: only when configured (see Facts).
//
// Nothing speculative goes in this list. A capability she offers and cannot
// perform is worse than one she never mentions.
func (f Facts) can() []string {
c := []string{
// Talking comes first, and the closing line says "действий" rather than
// "ничего", because this same block sits in front of the chat and
// general-knowledge prompts. A flat "you can do nothing else" would
// tell her to refuse the exact thing those two prompts are for.
"разговаривать и отвечать на вопросы",
"ставить напоминания",
"записывать заметки и факты и отвечать по ним",
"смотреть календарь",
}
if f.Weather {
c = append(c, "говорить погоду")
}
if f.Telegram {
c = append(c, "писать в телеграм")
}
if f.Tools {
c = append(c, "запускать разрешённые команды на сервере")
}
return c
}
// who renders the optional name/city line, or "" when neither is configured.
//
// Written as labels ("Имя владельца: ..."), not as a sentence with pronouns:
// the block's own "ты" is Maven, so "тебя зовут" would read as her name and
// "его" would model the third-person form she must never use about him.
func (f Facts) who() string {
name := strings.TrimSpace(f.OwnerName)
city := strings.TrimSpace(f.City)
switch {
case name != "" && city != "":
return "Имя владельца: " + name + ". Город: " + city + "."
case name != "":
return "Имя владельца: " + name + "."
case city != "":
return "Город: " + city + "."
}
return ""
}
// Prepend puts the block in front of a system prompt. Nil-safe: a nil renderer
// (tests, the stub paths) returns the prompt untouched.
func Prepend(block func() string, prompt string) string {
if block == nil {
return prompt
}
s := strings.TrimSpace(block())
if s == "" {
return prompt
}
return s + "\n\n" + prompt
}
+72
View File
@@ -0,0 +1,72 @@
package persona
import (
"strings"
"testing"
"time"
)
var ref = time.Date(2026, 7, 31, 14, 5, 0, 0, time.UTC)
// The block must be correct with an empty config: the address form and the
// gender rules are hard constraints, not preferences.
func TestBlockWorksWithZeroConfig(t *testing.T) {
b := Facts{}.Block(ref)
for _, want := range []string{"женском роде", "ОБРАЩЕНИЕ", "\"ты\"", "31 июля 2026", "пятница", "14:05"} {
if !strings.Contains(b, want) {
t.Errorf("block missing %q:\n%s", want, b)
}
}
}
func TestBlockAddsOptionalFacts(t *testing.T) {
b := Facts{OwnerName: "Ками", City: "Москва", Static: "Будь краткой."}.Block(ref)
for _, want := range []string{"Ками", "Москва", "Будь краткой."} {
if !strings.Contains(b, want) {
t.Errorf("block missing %q:\n%s", want, b)
}
}
}
// The time changes between turns, so two renders must differ.
func TestBlockRendersTimePerTurn(t *testing.T) {
a := Facts{}.Block(ref)
c := Facts{}.Block(ref.Add(time.Hour))
if a == c {
t.Errorf("block did not change with the clock:\n%s", a)
}
}
// She may only offer what this deployment actually has.
func TestCapabilitiesAreConfigGated(t *testing.T) {
bare := Facts{}.Block(ref)
for _, want := range []string{"напоминания", "заметки", "календарь"} {
if !strings.Contains(bare, want) {
t.Errorf("block missing always-on capability %q:\n%s", want, bare)
}
}
for _, unwanted := range []string{"погоду", "телеграм", "команды"} {
if strings.Contains(bare, unwanted) {
t.Errorf("block offers unconfigured %q:\n%s", unwanted, bare)
}
}
full := Facts{Weather: true, Telegram: true, Tools: true}.Block(ref)
for _, want := range []string{"погоду", "телеграм", "команды"} {
if !strings.Contains(full, want) {
t.Errorf("block missing configured capability %q:\n%s", want, full)
}
}
}
func TestPrependNilIsSafe(t *testing.T) {
if got := Prepend(nil, "PROMPT"); got != "PROMPT" {
t.Errorf("Prepend(nil) = %q", got)
}
if got := Prepend(func() string { return " " }, "PROMPT"); got != "PROMPT" {
t.Errorf("Prepend(blank) = %q", got)
}
if got := Prepend(func() string { return "CTX" }, "PROMPT"); got != "CTX\n\nPROMPT" {
t.Errorf("Prepend = %q", got)
}
}
+54
View File
@@ -0,0 +1,54 @@
package phraser
import (
"errors"
"strings"
"testing"
)
// A reply that starts a JSON object and never finishes it is a failed
// generation, not a reply. Before this, the parser returned ("", "") for these
// and every caller then shipped the raw fragment as the thing Maven said. A
// real run produced replies of literally "{" and "{\n \"".
func TestParseResponseMoodRejectsUnfinishedJSON(t *testing.T) {
for _, raw := range []string{
`{`,
"{\n \"",
`{"response": "неполн`,
`{"response": "текст", "mood":`,
} {
text, mood, err := parseResponseMood(raw)
if !errors.Is(err, errBrokenJSON) {
t.Errorf("parseResponseMood(%q) err = %v, want errBrokenJSON", raw, err)
}
if text != "" || mood != "" {
t.Errorf("parseResponseMood(%q) leaked %q/%q — a fragment must never come back as a reply", raw, text, mood)
}
}
}
// Bare prose is still fine. Small models sometimes answer without any JSON at
// all, and that reply is usable — so the new error must not swallow it.
func TestParseResponseMoodAllowsBareProse(t *testing.T) {
for _, raw := range []string{
"норм, а ты как?",
"вот что я нашла: ключ у соседа",
} {
text, mood, err := parseResponseMood(raw)
if err != nil {
t.Errorf("parseResponseMood(%q) err = %v, want nil", raw, err)
}
// No JSON means no fields; the caller ships raw as-is.
if text != "" || mood != "" {
t.Errorf("parseResponseMood(%q) = %q/%q, want empty", raw, text, mood)
}
}
}
// The measured failure: the model wants more than 400 characters and the old
// grammar cut it off mid-word. Guards the bound against being tightened back.
func TestGrammarStringBoundHasRoomForARealAnswer(t *testing.T) {
if !strings.Contains(responseGrammar, "{0,1000}") {
t.Error("grammar string bound is not 1000; 400 truncated real replies mid-word (see the comment on responseGrammar)")
}
}
+33
View File
@@ -0,0 +1,33 @@
package phraser
import (
"strings"
"testing"
)
// Every phrasing prompt must carry the shared context block. This is the
// regression guard for the bug that started this: the "ты" rule reached only
// two of the five prompts because each prompt had its own copy of the rules.
func TestEveryPromptCarriesTheContextBlock(t *testing.T) {
block := func() string { return "CTXBLOCK" }
p := &LLMPhraser{cfg: Config{ContextBlock: block}}
prompts := map[string]string{
"nudge": p.systemPrompt(),
"query": p.querySystemPrompt(),
"chat": chatSystemPrompt(block),
}
for name, got := range prompts {
if !strings.HasPrefix(got, "CTXBLOCK\n\n") {
t.Errorf("%s prompt does not start with the context block:\n%s", name, got)
}
}
}
// Without a block the prompts are unchanged — the stub and test paths pass nil.
func TestPromptsWithoutBlockAreUnchanged(t *testing.T) {
p := &LLMPhraser{}
if p.systemPrompt() != nudgeSystem {
t.Errorf("nudge prompt changed with no block set")
}
}
@@ -0,0 +1,65 @@
package eval
import (
"strings"
"testing"
)
// TestAddressReportsEveryBreak — the real reply from a nudge eval run broke in
// two ways at once and the check named only the plural. Both must print: a
// half-reported failure reads as a milder problem than it is.
func TestAddressReportsEveryBreak(t *testing.T) {
body := "Смотрите на его потребление воды."
res := checkAddress(body)
if res.Pass {
t.Fatalf("checkAddress passed %q", body)
}
for _, want := range []string{"смотрите", "его"} {
if !strings.Contains(res.Detail, want) {
t.Errorf("detail %q does not name %q", res.Detail, want)
}
}
}
// One word repeated is one problem, so the detail must not say it twice.
func TestAddressDeduplicates(t *testing.T) {
res := checkAddress("Вам стоит поесть, вам это нужно.")
if res.Pass {
t.Fatal("expected failure")
}
if n := strings.Count(res.Detail, "formal"); n != 1 {
t.Errorf("detail repeats the same break %d times: %q", n, res.Detail)
}
}
// The fragments a real run produced. All of them scored as non-empty replies
// before checkNonEmpty looked for letters.
func TestNonEmptyNeedsLetters(t *testing.T) {
for _, body := range []string{
"{",
"{\n \"",
"15-16",
`{"`,
" ",
"...",
} {
if got := checkNonEmpty(body); got.Pass {
t.Errorf("checkNonEmpty(%q) passed — that is not a reply", body)
}
}
}
// And it must not start failing real replies. Latin counts as well as Cyrillic:
// answers about ssd or vpn are legitimately part English.
func TestNonEmptyAcceptsRealReplies(t *testing.T) {
for _, body := range []string{
"норм, а ты как?",
"вот что я нашла: ключ у соседа",
"ssd быстрее hdd.",
"9 минут.",
} {
if got := checkNonEmpty(body); !got.Pass {
t.Errorf("checkNonEmpty(%q) failed: %s", body, got.Detail)
}
}
}
@@ -24,3 +24,19 @@ func TestAddressTimeWordDoesNotBlind(t *testing.T) {
}
}
}
// TestAddressVerbIsNotAnAntecedent — a nudge is mostly verbs, and a verb is
// never who "он" refers to. This exact string passed the check before.
func TestAddressVerbIsNotAnAntecedent(t *testing.T) {
s := "попробуй встать и отдохнуть — у него есть перерыв"
if r := checkAddress(s); r.Pass {
t.Errorf("checkAddress(%q) passed, want a third-person failure", s)
}
// Still missed, and this is the documented hole: "выпей воды, он не пил" has
// a real noun ("воды") before the pronoun, so the scan believes somebody
// else was named. Telling that apart needs a parser, not a suffix rule.
// A named third party still wins over the verbs around it.
if r := checkAddress("сервис упал, он не отвечает"); !r.Pass {
t.Errorf("checkAddress on a real third party failed: %s", r.Detail)
}
}
+99 -16
View File
@@ -340,7 +340,9 @@ func prevWord(words []string, i int) string {
// - it only looks BACKWARD. "Он не отвечает, сервис упал" names the subject
// after the pronoun and is flagged wrongly.
// - any noun earlier in the message counts as an antecedent, even when it is
// not one ("после обеда он не ел" reads as legitimate and is missed). The
// not one ("после обеда он не ел", "выпей воды, он не пил" — both missed).
// Verbs and time words no longer count, which covers the usual nudge, but a
// plain noun before the pronoun still blinds it. The
// common time words are stoplisted so the usual nudge opening does not
// blind it, but a message with any other noun in front still slips through.
// This is the check's real hole; widening it further would start flagging
@@ -407,27 +409,51 @@ var notAnAntecedent = map[string]bool{
"твой": true, "твоя": true, "твоё": true, "твое": true, "твои": true, "твою": true,
}
// looksPastVerb — a past-tense verb needs a subject of its own, so it is not an
// antecedent either. Keeps "сервис упал, он не отвечает" working off "сервис".
func looksPastVerb(w string) bool {
if len([]rune(w)) < 3 {
// looksVerb — a verb is never the thing "он" refers to, so it must not count as
// an antecedent. Past tense keeps "сервис упал, он не отвечает" working off
// "сервис"; the infinitive and imperative endings are here because a nudge is
// mostly made of them ("попробуй встать и отдохнуть — у него есть перерыв"
// slipped through with "попробуй" taken for the person being talked about).
func looksVerb(w string) bool {
r := []rune(w)
if len(r) < 3 {
return false
}
return strings.HasSuffix(w, "л") || strings.HasSuffix(w, "ла") ||
strings.HasSuffix(w, о") || strings.HasSuffix(w, "ли")
for _, suf := range []string{
"л", а", "ло", "ли", // past tense
"ть", "ться", "ти", "чь", // infinitive
"й", "йся", "йте", // imperative
} {
if strings.HasSuffix(w, suf) {
return true
}
}
return false
}
func checkAddress(body string) Result {
words := addressWordRE.FindAllString(strings.ToLower(body), -1)
// Every break, not just the first. A bad reply usually breaks in more than
// one way at once — "Смотрите на его потребление воды" is a plural imperative
// AND third person about him — and reporting only the first hid the second,
// which made the failure look milder than it was.
var breaks []string
seen := map[string]bool{}
add := func(msg string) {
if seen[msg] {
return // the same word twice in one message is one problem, not two
}
seen[msg] = true
breaks = append(breaks, msg)
}
for i, w := range words {
if formalPronouns[w] {
return Result{CheckAddress, false,
fmt.Sprintf("formal %q — she says ты/тебя/тебе", w)}
add(fmt.Sprintf("formal %q — she says ты/тебя/тебе", w))
}
if pluralVerb(w) && !(i > 0 && prepositions[words[i-1]]) {
return Result{CheckAddress, false,
fmt.Sprintf("plural imperative %q — she uses the singular", w)}
add(fmt.Sprintf("plural imperative %q — she uses the singular", w))
}
}
@@ -441,17 +467,26 @@ func checkAddress(body string) Result {
if !unicode.Is(unicode.Cyrillic, []rune(p)[0]) && !isLatinWord(p) {
continue // punctuation
}
if notAnAntecedent[p] || prepositions[p] || thirdPersonHim[p] || looksPastVerb(p) {
// pluralVerb as well as looksVerb: looksVerb knows the imperative in
// -й/-йте but not the -те plural ("смотрите"), so "Смотрите на его
// потребление воды" counted "смотрите" as the person being talked
// about and the "его" never printed. Third time a verb form has
// blinded this check — if a fourth turns up, the antecedent test
// wants a real morphology table, not another suffix.
if notAnAntecedent[p] || prepositions[p] || thirdPersonHim[p] || looksVerb(p) || pluralVerb(p) {
continue
}
named = true
break
}
if !named {
return Result{CheckAddress, false,
fmt.Sprintf("third person %q with nobody else named — she talks to him, not about him", w)}
add(fmt.Sprintf("third person %q with nobody else named — she talks to him, not about him", w))
}
}
if len(breaks) > 0 {
return Result{CheckAddress, false, strings.Join(breaks, " + ")}
}
return Result{CheckAddress, true, ""}
}
@@ -560,12 +595,60 @@ func checkCringe(body string) Result {
// checkOnTopic — the message must name the thing the rule is about. A nudge
// that never mentions water leaves the operator with a chime and no action.
func checkOnTopic(c Case, body string) Result {
return checkOnTopicAny(c.WantAny, body)
}
// checkOnTopicAny is the same test over a bare want-list, so the talk scorer can
// reuse it without owning a nudge Case.
func checkOnTopicAny(wantAny []string, body string) Result {
low := strings.ToLower(body)
for _, want := range c.WantAny {
for _, want := range wantAny {
if strings.Contains(low, strings.ToLower(want)) {
return Result{CheckOnTopic, true, ""}
}
}
return Result{CheckOnTopic, false,
fmt.Sprintf("mentions none of %v", c.WantAny)}
fmt.Sprintf("mentions none of %v", wantAny)}
}
// --- shape checks for the free-form paths --------------------------------
//
// The nudge checks assume one short sentence. Chat and query replies are longer
// by design, so the only shape worth testing there is that the model produced a
// reply at all and did not trail off. Both are failure modes the fallbacks in
// llmphraser.go hide: a truncated or empty generation still returns nil error.
const (
CheckNonEmpty = "nonempty" // she said something
CheckEllipsis = "ellipsis" // she finished the sentence
)
// A reply needs words in it, not just characters. This check used to test for a
// non-empty string, which scored 27/27 on a run where two replies were "{" and
// "{\n \"" — punctuation passed as content. Braces, quotes, digits and spaces
// are all empty in the only sense that matters.
//
// Digits alone fail too, and that is deliberate: the same run answered "сколько
// варить яйцо вкрутую?" with "15-16". No unit, no words, and it is also the
// wrong number. Whatever that is, it is not something she said.
func checkNonEmpty(body string) Result {
if strings.TrimSpace(body) == "" {
return Result{CheckNonEmpty, false, "empty reply"}
}
for _, r := range body {
if unicode.IsLetter(r) {
return Result{CheckNonEmpty, true, ""}
}
}
return Result{CheckNonEmpty, false, fmt.Sprintf("no letters in the reply %q — punctuation or digits only", strings.TrimSpace(body))}
}
// checkEllipsis — a reply ending in "…" or "..." is a generation that ran out of
// tokens, not a stylistic pause. Mid-sentence ellipses are left alone.
func checkEllipsis(body string) Result {
trimmed := strings.TrimRight(strings.TrimSpace(body), `"'»)`)
if strings.HasSuffix(trimmed, "…") || strings.HasSuffix(trimmed, "...") {
return Result{CheckEllipsis, false, "reply trails off in an ellipsis — likely truncated"}
}
return Result{CheckEllipsis, true, ""}
}
+19 -1
View File
@@ -7,6 +7,8 @@ import (
"testing"
"time"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser"
)
@@ -32,6 +34,7 @@ func TestLLMPhrasingBaseline(t *testing.T) {
// case as a phrasing error and read as "the model cannot phrase".
noProxyLoopback(t)
ctx := context.Background()
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
@@ -41,10 +44,25 @@ func TestLLMPhrasingBaseline(t *testing.T) {
// Generous: an unconstrained 0.8B can spend a minute thinking before it
// writes a word, and a timeout would be scored as a model failure.
cfg.Timeout = 5 * time.Minute
// The same shared context block the daemon prepends (internal/persona),
// with an empty config — that is the deployment we actually ship.
cfg.ContextBlock = func() string { return persona.Facts{}.Block(time.Now()) }
p := phraser.NewLLMPhraserAt(base, cfg)
defer p.Close()
rep, err := Score(context.Background(), "llm (0.8B, built-in persona)", p, f)
// Label the run with whatever gguf the server actually has loaded. It used
// to say "0.8B" no matter what, so two runs of two different models came
// out named the same and were easy to mix up when comparing.
model, err := llm.ModelID(ctx, base)
if err != nil {
// An unlabelled score is still a score, but say so loudly — a made-up
// name in a bake-off table is worse than no name.
t.Logf("could not read model id from %s: %v — report will say %q", base, err, llm.UnknownModel)
model = llm.UnknownModel
}
t.Logf("scoring model %s at %s", model, base)
rep, err := Score(ctx, "llm ("+model+", built-in persona)", p, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
+265
View File
@@ -0,0 +1,265 @@
package eval
// This file scores the CONVERSATIONAL paths, the ones the nudge fixture never
// touches: chat, query-with-notes, and general knowledge. All three now carry
// the shared persona block (internal/persona), and all three produce long
// free-form Russian — which is exactly where a persona break (formality, third
// person, masculine self-reference) is most likely and where, until this file,
// nothing could see one.
//
// Why a second fixture instead of more nudge cases: the checks differ. A nudge
// must be one short sentence with no question in it; a chat reply is allowed
// 1-3 sentences and a follow-up question is a FEATURE there. Mixing them would
// need per-case check masks, and the nudge scorer stays untouched this way.
//
// Why per-path reporting: a chat regression and a knowledge regression have
// different causes (chat prompt vs router.KnowledgePrompt), and one blended
// percentage cannot tell them apart.
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/kami/maven/internal/dialogue"
)
//go:embed talk_v1.json
var talkFixtureJSON []byte
// The three phrasing paths under test. Values match the fixture's "path" field.
const (
PathChat = "chat" // PhraseChat
PathQuery = "query" // PhraseQuery with notes
PathKnowledge = "knowledge" // PhraseQuery with no notes
)
// TalkPaths — report order.
var TalkPaths = []string{PathChat, PathQuery, PathKnowledge}
// TalkCheckNames — the checks that apply to a free-form reply, in report order.
// Deliberately a subset of CheckNames: length, mood and "no questions" are nudge
// properties and would fail a correct chat reply. These paths return no mood at
// all, so there is nothing to check there.
var TalkCheckNames = []string{
CheckNonEmpty, CheckEllipsis, CheckLang, CheckFeminine, CheckAddress, CheckOnTopic,
}
// TalkCase — one turn as the daemon would present it.
//
// History is flat text because that is all PhraseChat uses (it concatenates
// turn texts into one user message); intents and slots would be dead fields.
// Notes are what the store would have matched for a query.
//
// WantAny is the on-topic contract: at least one lowercased fragment must appear
// in the reply. Fragments are stems ("пароль" → "парол") so declension does not
// defeat them.
type TalkCase struct {
ID string `json:"id"`
Path string `json:"path"`
Utterance string `json:"utterance"`
History []string `json:"history,omitempty"`
Notes []string `json:"notes,omitempty"`
WantAny []string `json:"want_any"`
Tags []string `json:"tags,omitempty"`
Note string `json:"note,omitempty"`
}
// TalkFixture — the versioned envelope, same gating as Fixture.
type TalkFixture struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
Notes []string `json:"notes"`
Cases []TalkCase `json:"cases"`
}
// LoadTalk returns the embedded conversational fixture.
func LoadTalk() (TalkFixture, error) {
var f TalkFixture
if err := json.Unmarshal(talkFixtureJSON, &f); err != nil {
return TalkFixture{}, fmt.Errorf("parse talk fixture: %w", err)
}
if f.SchemaVersion != SchemaVersion {
return TalkFixture{}, fmt.Errorf("talk fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
}
if len(f.Cases) == 0 {
return TalkFixture{}, fmt.Errorf("talk fixture has no cases")
}
return f, nil
}
// Talker — the two methods a conversational path must have to be scorable.
// *phraser.LLMPhraser satisfies it; same trick as Nudger.
type Talker interface {
PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error)
PhraseQuery(ctx context.Context, utterance string, notes []string) (string, error)
}
// TalkOutcome — one scored case.
type TalkOutcome struct {
Case TalkCase
Reply string
Err error
Latency time.Duration
Pass bool
Failed []string
Reasons []string
}
// TalkReport — the aggregate. ByPath is the point of this scorer.
type TalkReport struct {
Name string
Total int
Passed int
Errors int
ByCheck map[string]int
ByPath map[string]TagStat
Outcomes []TalkOutcome
P50 time.Duration
P95 time.Duration
Max time.Duration
}
// Accuracy — fraction of cases that passed every check.
func (r TalkReport) Accuracy() float64 {
if r.Total == 0 {
return 0
}
return float64(r.Passed) / float64(r.Total)
}
// ScoreTalk runs every case through t and aggregates. A phrasing error scores as
// a miss and is counted separately: "the model was down" and "the model wrote
// something bad" must not be the same number.
func ScoreTalk(ctx context.Context, name string, t Talker, f TalkFixture) (TalkReport, error) {
rep := TalkReport{
Name: name,
Total: len(f.Cases),
ByCheck: map[string]int{},
ByPath: map[string]TagStat{},
}
for _, n := range TalkCheckNames {
rep.ByCheck[n] = 0
}
lat := make([]time.Duration, 0, len(f.Cases))
for _, c := range f.Cases {
start := time.Now()
reply, err := c.run(ctx, t)
o := TalkOutcome{Case: c, Reply: reply, Err: err, Latency: time.Since(start)}
lat = append(lat, o.Latency)
if err != nil {
rep.Errors++
o.Failed = append(o.Failed, "call")
o.Reasons = append(o.Reasons, fmt.Sprintf("phrase error: %v", err))
} else {
for _, res := range RunTalkChecks(c, reply) {
if res.Pass {
rep.ByCheck[res.Name]++
continue
}
o.Failed = append(o.Failed, res.Name)
o.Reasons = append(o.Reasons, res.Name+": "+res.Detail)
}
}
o.Pass = len(o.Failed) == 0
if o.Pass {
rep.Passed++
}
bump(rep.ByPath, c.Path, o.Pass)
rep.Outcomes = append(rep.Outcomes, o)
}
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
if len(lat) > 0 {
rep.Max = lat[len(lat)-1]
}
return rep, nil
}
// run dispatches the case to its path. knowledge and query are the same method;
// the empty notes slice is what selects the no-notes branch inside PhraseQuery.
func (c TalkCase) run(ctx context.Context, t Talker) (string, error) {
switch c.Path {
case PathChat:
return t.PhraseChat(ctx, c.Utterance, c.turns())
case PathQuery:
return t.PhraseQuery(ctx, c.Utterance, c.Notes)
case PathKnowledge:
return t.PhraseQuery(ctx, c.Utterance, nil)
}
return "", fmt.Errorf("unknown path %q", c.Path)
}
func (c TalkCase) turns() []dialogue.Turn {
turns := make([]dialogue.Turn, 0, len(c.History))
for _, h := range c.History {
turns = append(turns, dialogue.Turn{Text: h})
}
return turns
}
// RunTalkChecks scores one reply. Order matches TalkCheckNames.
func RunTalkChecks(c TalkCase, reply string) []Result {
return []Result{
checkNonEmpty(reply),
checkEllipsis(reply),
checkLang(reply),
checkFeminine(reply),
checkAddress(reply),
checkOnTopicAny(c.WantAny, reply),
}
}
// String renders the comparison table — composite, then per-check so a
// regression names the property, then per-path so it names the prompt.
func (r TalkReport) String() string {
var b strings.Builder
fmt.Fprintf(&b, "%s: %d/%d cases pass every check (%.1f%%), %d errors\n",
r.Name, r.Passed, r.Total, 100*r.Accuracy(), r.Errors)
for _, name := range TalkCheckNames {
fmt.Fprintf(&b, " %-10s %d/%d\n", name, r.ByCheck[name], r.Total)
}
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
fmt.Fprintf(&b, " by path: %s\n", renderStats(r.ByPath))
return b.String()
}
// Failures — per-case detail, sorted by ID so two runs diff cleanly.
func (r TalkReport) Failures() string {
var b strings.Builder
for _, o := range r.sorted() {
if o.Pass {
continue
}
fmt.Fprintf(&b, " %s %q\n %s\n", o.Case.ID, o.Reply, strings.Join(o.Reasons, "; "))
}
return b.String()
}
// Replies — every generated reply verbatim. This is what a human reads to judge
// tone; the score only says which checks fired.
func (r TalkReport) Replies() string {
var b strings.Builder
for _, o := range r.sorted() {
mark := "ok "
if !o.Pass {
mark = "FAIL"
}
fmt.Fprintf(&b, " %s %-9s %-22s %q\n", mark, o.Case.Path, o.Case.ID, o.Reply)
}
return b.String()
}
func (r TalkReport) sorted() []TalkOutcome {
out := append([]TalkOutcome(nil), r.Outcomes...)
sort.Slice(out, func(i, j int) bool { return out[i].Case.ID < out[j].Case.ID })
return out
}
+163
View File
@@ -0,0 +1,163 @@
package eval
import (
"context"
"os"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser"
)
// perPathMinimum — the resolution floor. A per-path score built on a handful of
// cases moves by 12% when a single reply changes, which cannot distinguish a
// prompt regression from noise.
const perPathMinimum = 8
// TestTalkFixture — the fixture itself has to be sound before any score off it
// means anything.
func TestTalkFixture(t *testing.T) {
f, err := LoadTalk()
if err != nil {
t.Fatalf("LoadTalk: %v", err)
}
seen := map[string]bool{}
byPath := map[string]int{}
for _, c := range f.Cases {
if seen[c.ID] {
t.Errorf("duplicate case id %q", c.ID)
}
seen[c.ID] = true
switch c.Path {
case PathChat, PathQuery, PathKnowledge:
default:
t.Errorf("%s: unknown path %q", c.ID, c.Path)
}
byPath[c.Path]++
if strings.TrimSpace(c.Utterance) == "" {
t.Errorf("%s: empty utterance", c.ID)
}
if len(c.WantAny) == 0 {
t.Errorf("%s: no want_any — the reply cannot be checked for topic", c.ID)
}
// A query case with no notes would silently score the knowledge path.
if c.Path == PathQuery && len(c.Notes) == 0 {
t.Errorf("%s: query case has no notes", c.ID)
}
if c.Path == PathKnowledge && len(c.Notes) > 0 {
t.Errorf("%s: knowledge case must have no notes", c.ID)
}
}
for _, p := range TalkPaths {
if byPath[p] < perPathMinimum {
t.Errorf("path %s has %d cases, want at least %d", p, byPath[p], perPathMinimum)
}
}
}
// fakeTalker — a scripted Talker, so the scorer is testable without a model.
type fakeTalker struct{ reply string }
func (f fakeTalker) PhraseChat(context.Context, string, []dialogue.Turn) (string, error) {
return f.reply, nil
}
func (f fakeTalker) PhraseQuery(context.Context, string, []string) (string, error) {
return f.reply, nil
}
// TestScoreTalkCounts — a reply that fails on purpose must be counted on every
// path, so a real run cannot report a hidden zero.
func TestScoreTalkCounts(t *testing.T) {
f, err := LoadTalk()
if err != nil {
t.Fatalf("LoadTalk: %v", err)
}
// Formal address, off-topic, trailing ellipsis: three checks fail at once.
rep, err := ScoreTalk(context.Background(), "fake", fakeTalker{"Приходите, я вас жду…"}, f)
if err != nil {
t.Fatalf("ScoreTalk: %v", err)
}
if rep.Total != len(f.Cases) || rep.Passed != 0 {
t.Errorf("got %d/%d passing, want 0/%d", rep.Passed, rep.Total, len(f.Cases))
}
if rep.ByCheck[CheckAddress] != 0 {
t.Errorf("formal reply passed the address check %d times", rep.ByCheck[CheckAddress])
}
if rep.ByCheck[CheckEllipsis] != 0 {
t.Errorf("truncated reply passed the ellipsis check %d times", rep.ByCheck[CheckEllipsis])
}
for _, p := range TalkPaths {
if rep.ByPath[p].Total == 0 {
t.Errorf("path %s missing from the report", p)
}
}
if !strings.Contains(rep.String(), "by path") {
t.Error("report does not break down by path")
}
}
// TestLLMTalkBaseline — the resident model on the three conversational paths.
// Opt-in exactly like TestLLMPhrasingBaseline: CI has no model and a run costs
// minutes on the CPU target.
//
// MAVEN_LLM_URL=http://127.0.0.1:18099 \
// go test -run TestLLMTalkBaseline ./internal/phraser/eval/
//
// Reports, does not assert a quality bar — the numbers are the input to tuning
// the persona prompt. The one thing worth failing on is a harness fault.
func TestLLMTalkBaseline(t *testing.T) {
base := os.Getenv("MAVEN_LLM_URL")
if base == "" {
t.Skip("MAVEN_LLM_URL unset — point it at a running llama-server (see doc comment)")
}
noProxyLoopback(t)
ctx := context.Background()
f, err := LoadTalk()
if err != nil {
t.Fatalf("LoadTalk: %v", err)
}
cfg := phraser.DefaultConfig("")
cfg.Timeout = 5 * time.Minute
cfg.ContextBlock = func() string { return persona.Facts{}.Block(time.Now()) }
p := phraser.NewLLMPhraserAt(base, cfg)
defer p.Close()
// Unreachable server is fatal here, not a logged warning, and that differs
// from the nudge test on purpose. PhraseNudge returns its errors, so a dead
// server there shows up honestly in the Errors column. PhraseChat and
// PhraseQuery do NOT: they swallow every failure and return a canned string
// ("поговорили.", "не знаю.", "вот что я нашла: …"). So on these three paths
// a dead server produces a full report with 0 errors and a terrible score —
// a number that looks like bad phrasing and is really no phrasing at all.
// Refusing to score without a confirmed model is the only guard available
// until the phraser reports its failures (Vikunja #397).
model, err := llm.ModelID(ctx, base)
if err != nil {
t.Fatalf("no model at %s: %v — refusing to score, these paths hide their errors "+
"and would report a plausible-looking result off a dead server", base, err)
}
t.Logf("scoring model %s at %s", model, base)
rep, err := ScoreTalk(ctx, "llm ("+model+", built-in persona)", p, f)
if err != nil {
t.Fatalf("ScoreTalk: %v", err)
}
t.Log("\n" + rep.String() + "\nreplies:\n" + rep.Replies() + "\nfailures:\n" + rep.Failures())
// And again afterwards: the run takes minutes, and a server that died or got
// OOM-killed halfway through would leave the first cases scored and the rest
// silently canned. Checking only at the start would not catch that.
if _, err := llm.ModelID(ctx, base); err != nil {
t.Fatalf("model at %s went away during the run: %v — the score above is not trustworthy", base, err)
}
}
+227
View File
@@ -0,0 +1,227 @@
{
"schema_version": 1,
"name": "ru-talk-v1",
"notes": [
"Scores the three conversational phrasing paths: chat (PhraseChat), query (PhraseQuery with notes) and knowledge (PhraseQuery with no notes). The nudge fixture does not cover any of them.",
"Nine cases per path, not five. The nudge fixture is 15 sampled cases and cannot resolve a change smaller than ~3 cases; a per-path score off five cases would be worse still. More cases per path is the point of this fixture.",
"The owner is a man, addressed informally as ty, living alone with a home server. Every utterance is written the way he actually talks to her.",
"chat-formality-bait and chat-about-me exist to provoke the two persona breaks the nudge eval caught: the formal vy/vas plural, and talking about him in the third person.",
"want_any fragments are stems so Russian declension does not defeat the on-topic check. They are lowercased before comparison.",
"want_any is a plain substring test, so a fragment that is too short passes by accident: \"ты\" matches inside \"работы\", \"нет\" inside \"интернет\". Keep every fragment to three or more letters of a real stem.",
"Notes are written as the store would have them: short, first person, no punctuation discipline."
],
"cases": [
{
"id": "chat-how-are-you",
"path": "chat",
"utterance": "привет, как дела?",
"want_any": ["норм", "хорош", "порядк", "тут", "работ"],
"tags": ["greeting"],
"note": "The plainest chat turn there is. If the persona breaks anywhere it breaks here first."
},
{
"id": "chat-formality-bait",
"path": "chat",
"utterance": "не могли бы вы подсказать, чем вы сейчас занимаетесь?",
"want_any": ["сейчас", "ничем", "ничего", "жду", "тут"],
"tags": ["persona-bait", "address"],
"note": "Deliberately polite and plural. A small model mirrors the register and answers with vy/vas — the exact break the address check was written for."
},
{
"id": "chat-about-me",
"path": "chat",
"utterance": "расскажи обо мне",
"want_any": ["теб"],
"tags": ["persona-bait", "third-person"],
"note": "Baits the third person: she should say 'ты живёшь один', not 'он живёт один', as if reporting to somebody else."
},
{
"id": "chat-bored-evening",
"path": "chat",
"utterance": "скучно что-то вечером, посоветуй чем заняться",
"want_any": ["можеш", "попробу", "почита", "прогул", "фильм", "серв"],
"tags": ["open-ended"]
},
{
"id": "chat-followup-server",
"path": "chat",
"utterance": "а стоит его вообще перезагружать?",
"history": ["сервер опять шумит как самолёт", "похоже вентилятор"],
"want_any": ["серв", "перезагру", "вентил", "шум"],
"tags": ["history", "anaphora"],
"note": "The pronoun 'его' only resolves through history. Also the one case where 'он' about the server is legitimate."
},
{
"id": "chat-tired",
"path": "chat",
"utterance": "устал я сегодня, весь день за компом",
"want_any": ["отдохн", "устал", "перерыв", "спат", "день"],
"tags": ["tone"],
"note": "Invites the fake-concern and emotional-support drift; the reply should stay plain."
},
{
"id": "chat-thanks",
"path": "chat",
"utterance": "спасибо, выручила",
"want_any": ["пожалуйст", "не за что", "рада", "обращ"],
"tags": ["persona", "feminine"],
"note": "Feminine self-reference is unavoidable in an answer to thanks: 'рада', not 'рад'."
},
{
"id": "chat-what-can-you-do",
"path": "chat",
"utterance": "что ты вообще умеешь?",
"want_any": ["напомн", "замет", "запис", "могу", "умею"],
"tags": ["self-description", "feminine"]
},
{
"id": "chat-joke",
"path": "chat",
"utterance": "расскажи что-нибудь смешное",
"want_any": ["анекдот", "шутк", "смешн", "истори"],
"tags": ["open-ended"],
"note": "Longest free-form generation in the chat set — the most likely place for a truncated reply."
},
{
"id": "query-router-password",
"path": "query",
"utterance": "что я записывал про пароль от роутера?",
"notes": ["пароль от роутера admin/xxK9tp — на наклейке снизу", "роутер висит в коридоре"],
"want_any": ["парол", "роутер", "наклейк"],
"tags": ["notes", "recall"]
},
{
"id": "query-bedtime-yesterday",
"path": "query",
"utterance": "напомни, во сколько я вчера лёг?",
"notes": ["лёг спать в 02:40", "сегодня встал в 9"],
"want_any": ["02:40", "2:40", "полтрет", "ноч"],
"tags": ["notes", "time"]
},
{
"id": "query-doctor-name",
"path": "query",
"utterance": "как звали того стоматолога, которого мне советовали?",
"notes": ["стоматолог Игорь Валерьевич, клиника на Ленина, советовал Дима"],
"want_any": ["игор", "валерьев", "стоматолог"],
"tags": ["notes", "recall"]
},
{
"id": "query-disk-plan",
"path": "query",
"utterance": "я что-то планировал с диском на сервере, что именно?",
"notes": ["купить второй hdd на 4тб под бэкапы", "перенести медиатеку с системного диска"],
"want_any": ["hdd", "бэкап", "диск", "4тб", "медиатек"],
"tags": ["notes", "homeserver"]
},
{
"id": "query-notes-do-not-answer",
"path": "query",
"utterance": "сколько я заплатил за домен?",
"notes": ["домен продлевается в марте", "хостинг оплачен на год вперёд"],
"want_any": ["домен", "не зна", "не указ"],
"tags": ["notes", "negative"],
"note": "The notes do not contain the price. The prompt tells her to say so; a made-up number is the failure being watched for."
},
{
"id": "query-single-note",
"path": "query",
"utterance": "где лежит запасной ключ?",
"notes": ["запасной ключ у соседа с четвёртого этажа"],
"want_any": ["ключ", "сосед", "четверт"],
"tags": ["notes", "single"],
"note": "One note only — PhraseQuery has a separate branch for len(notes) == 1."
},
{
"id": "query-polite-form",
"path": "query",
"utterance": "подскажите, пожалуйста, что у меня записано по машине?",
"notes": ["замена масла на 92 тысячах", "страховка до 14 сентября"],
"want_any": ["масл", "страховк", "92", "сентябр"],
"tags": ["notes", "persona-bait", "address"],
"note": "Polite plural in the question. The answer must still be ty."
},
{
"id": "query-shopping",
"path": "query",
"utterance": "что мне надо было купить?",
"notes": ["купить кофе и фильтры", "закончилась паста"],
"want_any": ["кофе", "фильтр", "паст"],
"tags": ["notes", "list"]
},
{
"id": "query-wifi-guest",
"path": "query",
"utterance": "я записывал гостевой вайфай?",
"notes": ["гостевая сеть maven-guest, пароль 12345678 меняю раз в месяц"],
"want_any": ["guest", "гостев", "12345678", "парол"],
"tags": ["notes", "recall"]
},
{
"id": "know-sky-blue",
"path": "knowledge",
"utterance": "почему небо синее?",
"want_any": ["све", "рассеи", "атмосфер", "син", "волн"],
"tags": ["general"]
},
{
"id": "know-boil-egg",
"path": "knowledge",
"utterance": "сколько варить яйцо вкрутую?",
"want_any": ["минут", "8", "9", "10", "варит"],
"tags": ["general", "practical"]
},
{
"id": "know-ssd-vs-hdd",
"path": "knowledge",
"utterance": "чем ssd отличается от hdd?",
"want_any": ["ssd", "hdd", "быстр", "диск", "механич"],
"tags": ["general", "tech"]
},
{
"id": "know-cat-purr",
"path": "knowledge",
"utterance": "почему кошки мурчат?",
"want_any": ["кош", "мурч", "вибра", "успока"],
"tags": ["general"]
},
{
"id": "know-hiccups",
"path": "knowledge",
"utterance": "как быстро избавиться от икоты?",
"want_any": ["икот", "дыха", "вод", "задерж"],
"tags": ["general", "practical"]
},
{
"id": "know-polite-form",
"path": "knowledge",
"utterance": "не могли бы вы объяснить, что такое vpn?",
"want_any": ["vpn", "туннел", "трафик", "сет", "шифр"],
"tags": ["general", "persona-bait", "address"],
"note": "Polite plural bait on the knowledge prompt, which is a different system prompt from chat and must hold the same line."
},
{
"id": "know-dont-know",
"path": "knowledge",
"utterance": "как зовут моего соседа снизу?",
"want_any": ["не зна", "не мог"],
"tags": ["general", "negative"],
"note": "Unanswerable without notes. Admitting it beats inventing a name; watching for the invention."
},
{
"id": "know-water-per-day",
"path": "knowledge",
"utterance": "сколько воды в день надо пить?",
"want_any": ["вод", "литр", "стакан", "пит"],
"tags": ["general", "health"],
"note": "Overlaps a nudge rule on purpose: the knowledge answer must not turn into a nudge."
},
{
"id": "know-thunder-delay",
"path": "knowledge",
"utterance": "почему гром слышно позже молнии?",
"want_any": ["звук", "све", "быстр", "гром", "молни"],
"tags": ["general"]
}
]
}
+58
View File
@@ -0,0 +1,58 @@
package eval
import (
"context"
"math/rand"
"testing"
"github.com/kami/maven/internal/phraser"
)
// TestTemplateNudges scores the hand-written Russian templates on the same
// fixture the model is scored on. No model, no network — it runs in milliseconds.
//
// The bar is every case, not most of them: the templates are hand-written, so a
// failure is a bug in one line of Russian, not model variance.
func TestTemplateNudges(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
// Fixed seed: the score must not depend on which variant came up.
nt, err := phraser.NewNudgeTemplates(rand.NewSource(20260731))
if err != nil {
t.Fatalf("NewNudgeTemplates: %v", err)
}
rep, err := Score(context.Background(), "ru templates", nt, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
t.Log("\n" + rep.String())
t.Log("\n" + rep.Messages())
if rep.Passed != rep.Total {
t.Errorf("templates scored %d/%d, want every case:\n%s",
rep.Passed, rep.Total, rep.Failures())
}
}
// TestTemplateNudgesEverySeed — one seed passing could be luck. Every variant of
// every rule has to pass every check, so sweep seeds until each has been used.
func TestTemplateNudgesEverySeed(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
for seed := int64(0); seed < 60; seed++ {
nt, err := phraser.NewNudgeTemplates(rand.NewSource(seed))
if err != nil {
t.Fatalf("NewNudgeTemplates: %v", err)
}
rep, err := Score(context.Background(), "ru templates", nt, f)
if err != nil {
t.Fatalf("Score: %v", err)
}
if rep.Passed != rep.Total {
t.Errorf("seed %d: %d/%d\n%s", seed, rep.Passed, rep.Total, rep.Failures())
}
}
}
+129
View File
@@ -0,0 +1,129 @@
package phraser
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/kami/maven/internal/loop"
)
// grammarSpy stands in for llama-server: it records the grammar field of every
// request and always answers with a contract-shaped reply.
type grammarSpy struct {
srv *httptest.Server
grammars []string
}
func newGrammarSpy(t *testing.T) *grammarSpy {
t.Helper()
s := &grammarSpy{}
s.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var req chatReq
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
t.Errorf("spy: decode request: %v", err)
}
s.grammars = append(s.grammars, req.Grammar)
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(`{"choices":[{"message":{"content":"{\"response\": \"ага\", \"mood\": \"neutral\"}"}}]}`))
}))
t.Cleanup(s.srv.Close)
return s
}
// callAllPhrasingPaths hits every path that expects the JSON contract.
// LLMNudges must be set on the phraser under test: nudges come from templates
// by default and never reach the model at all.
func callAllPhrasingPaths(t *testing.T, p *LLMPhraser) {
t.Helper()
ctx := context.Background()
if _, err := p.PhraseNudge(ctx, loop.Candidate{Rule: loop.WaterRule(), Severity: loop.Sev1}); err != nil {
t.Fatalf("PhraseNudge: %v", err)
}
if _, err := p.PhraseChat(ctx, "привет", nil); err != nil {
t.Fatalf("PhraseChat: %v", err)
}
// Both branches: no notes (general knowledge) and with notes (grounded).
if _, err := p.PhraseQuery(ctx, "сколько воды я выпил", nil); err != nil {
t.Fatalf("PhraseQuery (no notes): %v", err)
}
if _, err := p.PhraseQuery(ctx, "сколько воды я выпил", []string{"два литра"}); err != nil {
t.Fatalf("PhraseQuery (notes): %v", err)
}
}
func TestGrammarIsAttachedToEveryPhrasingRequest(t *testing.T) {
if strings.TrimSpace(responseGrammar) == "" {
t.Fatal("responseGrammar is empty")
}
spy := newGrammarSpy(t)
p := NewLLMPhraserAt(spy.srv.URL, Config{LLMNudges: true})
callAllPhrasingPaths(t, p)
if len(spy.grammars) != 4 {
t.Fatalf("expected 4 requests, got %d", len(spy.grammars))
}
for i, g := range spy.grammars {
if g != responseGrammar {
t.Errorf("request %d carries grammar %q, want responseGrammar", i, g)
}
}
}
func TestNoGrammarConfigDisablesIt(t *testing.T) {
spy := newGrammarSpy(t)
p := NewLLMPhraserAt(spy.srv.URL, Config{NoGrammar: true, LLMNudges: true})
callAllPhrasingPaths(t, p)
for i, g := range spy.grammars {
if g != "" {
t.Errorf("request %d still carries a grammar with NoGrammar set: %q", i, g)
}
}
}
// The grammar's string rule must accept any codepoint, not just ASCII. Replies
// are Russian: an ASCII-only class would constrain the model into empty replies.
func TestGrammarStringRuleIsNotASCIIOnly(t *testing.T) {
if !strings.Contains(responseGrammar, `([^"\\] | "\\" ["\\/bfnrt])`) {
t.Error("string rule is not the any-codepoint-except-quote-and-backslash class; Cyrillic replies would be impossible")
}
}
// What the grammar describes must survive the parser that reads it back — a
// Russian body with an escaped quote inside, hand-built to test the contract.
func TestGrammarShapedJSONParses(t *testing.T) {
raw := `{"response": "он сказал \"привет\" и ушёл.\nвот так.", "mood": "confused"}`
text, mood, err := parseResponseMood(raw)
if err != nil {
t.Fatalf("grammar-shaped JSON did not parse: %v", err)
}
if want := "он сказал \"привет\" и ушёл.\nвот так."; text != want {
t.Errorf("response = %q, want %q", text, want)
}
if mood != "confused" {
t.Errorf("mood = %q, want confused", mood)
}
}
// Every mood the grammar permits is one the contract knows, and all five are there.
func TestGrammarMoodEnumMatchesTheContract(t *testing.T) {
for _, m := range []string{"neutral", "happy", "thinking", "tired", "confused"} {
if !strings.Contains(responseGrammar, `"\"`+m+`\""`) {
t.Errorf("mood %q missing from the grammar", m)
}
}
// No sixth mood: the enum line lists exactly five alternatives.
for _, line := range strings.Split(responseGrammar, "\n") {
if strings.HasPrefix(line, "mood") {
if n := strings.Count(line, "|") + 1; n != 5 {
t.Errorf("mood rule lists %d alternatives, want 5: %s", n, line)
}
}
}
}
+167 -42
View File
@@ -18,6 +18,7 @@ import (
"github.com/kami/maven/internal/delivery"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/router"
)
@@ -30,6 +31,10 @@ type LLMPhraser struct {
cmd *exec.Cmd
cancel context.CancelFunc
wg sync.WaitGroup
// tmpl — the hand-written Russian nudges. Default path for nudges; see
// Config.LLMNudges. nil only if the template file failed to load.
tmpl *NudgeTemplates
}
type Config struct {
@@ -39,7 +44,31 @@ type Config struct {
NGpuLayers int
NCtx int
Timeout time.Duration
Persona string // optional prompt prefix tuning maven's character
// ContextBlock renders the shared context block (who he is, how to
// address him, the time) fresh for each turn. See internal/persona.
// nil ⇒ no block, the prompts stand alone.
ContextBlock func() string
// LLMNudges puts the model back in charge of nudge wording.
//
// Off by default, and that is a deliberate deprecation of LLM-phrased
// nudges: hand-written templates (nudges_ru_v1.json) word every nudge now.
// A nudge has nothing to be creative about, and measured over many runs the
// 0.8B broke the persona (formal "вы", plural imperatives, masculine
// self-reference) and invented facts and units. Templates score 15/15 on the
// nudge fixture, the model 11-13/15.
//
// The LLM path is kept, not deleted: flip this on to get it back. Chat,
// query and reminder phrasing are untouched and still go through the model.
LLMNudges bool
// NoGrammar turns the GBNF constraint off (zero value ⇒ grammar ON).
// The escape hatch exists because the target resident model — the
// locally CPT'd Qwen3-1.7B — does not exist yet: if its chat template
// ever fights the grammar, the fix should be a config flip on the
// deploy box, not a code change and a rebuild.
NoGrammar bool
}
func DefaultConfig(modelPath string) Config {
@@ -59,6 +88,7 @@ func NewLLMPhraser(ctx context.Context, cfg Config) (*LLMPhraser, error) {
cfg: cfg,
client: &http.Client{Timeout: cfg.Timeout},
cancel: cancel,
tmpl: loadNudgeTemplates(),
}
if err := p.start(ctx); err != nil {
cancel()
@@ -80,9 +110,22 @@ func NewLLMPhraserAt(baseURL string, cfg Config) *LLMPhraser {
client: &http.Client{Timeout: cfg.Timeout},
port: strings.TrimSuffix(baseURL, "/"),
cancel: func() {},
tmpl: loadNudgeTemplates(),
}
}
// loadNudgeTemplates loads the Russian nudge templates. A broken template file
// must not stop the daemon booting, so a failure logs and leaves the LLM path
// in charge of nudges.
func loadNudgeTemplates() *NudgeTemplates {
nt, err := NewNudgeTemplates(nil)
if err != nil {
log.Printf("phraser: nudge templates unavailable, using the model: %v", err)
return nil
}
return nt
}
func (p *LLMPhraser) start(ctx context.Context) error {
args := []string{
"-m", p.cfg.ModelPath,
@@ -173,12 +216,21 @@ func (p *LLMPhraser) Close() error {
}
func (p *LLMPhraser) PhraseNudge(ctx context.Context, c loop.Candidate) (delivery.PhrasedNudge, error) {
// Templates first — see Config.LLMNudges for why this is the default.
if !p.cfg.LLMNudges && p.tmpl != nil {
return p.tmpl.PhraseNudge(ctx, c)
}
prompt := buildNudgePrompt(c)
resp, err := p.chat(ctx, prompt)
if err != nil {
return delivery.PhrasedNudge{}, err
}
body, mood := parseResponseMood(resp)
body, mood, perr := parseResponseMood(resp)
if perr != nil {
// Truncated JSON. Not a nudge — use the plain Russian fallback.
log.Printf("phraser: PhraseNudge: %v", perr)
body, mood = "", ""
}
if body == "" {
// fallback: try old body/summary format
body, _ = parsePhrase(resp)
@@ -202,13 +254,18 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
if len(notes) == 0 {
// General knowledge — no notes to ground the answer. The system
// prompt is the single tested source in router.KnowledgePrompt.
sys := router.KnowledgePrompt()
sys := persona.Prepend(p.cfg.ContextBlock, router.KnowledgePrompt())
prompt := fmt.Sprintf("Пользователь спрашивает: \"%s\".", utterance)
resp, err := p.chatWithSystem(ctx, sys, prompt, 256)
resp, err := p.chatWithSystem(ctx, sys, prompt, 768)
if err != nil || resp == "" {
return "не знаю.", nil
}
if text, _ := parseResponseMood(resp); text != "" {
text, _, perr := parseResponseMood(resp)
if perr != nil {
log.Printf("phraser: PhraseQuery: %v", perr)
return "не знаю.", nil
}
if text != "" {
return text, nil
}
return resp, nil
@@ -218,17 +275,22 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
}
sys := p.querySystemPrompt()
prompt := fmt.Sprintf(
`The user asks: "%s". Your notes matching the query contain: "%s". Answer them naturally and briefly. If the notes don't answer the question, say so.`,
`Он спрашивает: "%s". В твоих заметках по этому вопросу написано: "%s". Ответь ему коротко и своими словами. Если в заметках ответа нет — так и скажи.`,
utterance, strings.Join(notes, `"; "`),
)
resp, err := p.chatWithSystem(ctx, sys, prompt, 256)
if err != nil {
resp, err := p.chatWithSystem(ctx, sys, prompt, 768)
text, _, perr := parseResponseMood(resp)
if err != nil || perr != nil {
// Read the notes out rather than ship a broken fragment.
if perr != nil {
log.Printf("phraser: PhraseQuery: %v", perr)
}
if len(notes) == 1 {
return "вот что я нашла: " + notes[0], nil
}
return "вот что я нашла: " + strings.Join(notes, "; "), nil
}
if text, _ := parseResponseMood(resp); text != "" {
if text != "" {
return text, nil
}
return resp, nil
@@ -238,7 +300,7 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
// message array from dialogue history + the current user utterance. Falls back
// to a simple greeting on any LLM error — better to say something than nothing.
func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error) {
sys := chatSystemPrompt(p.cfg.Persona)
sys := chatSystemPrompt(p.cfg.ContextBlock)
msgs := []chatMsg{
{Role: "system", Content: sys},
}
@@ -251,12 +313,17 @@ func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history [
combined += utterance
msgs = append(msgs, chatMsg{Role: "user", Content: strings.TrimSpace(combined)})
resp, err := p.chatWithMessages(ctx, msgs, 512)
resp, err := p.chatWithMessages(ctx, msgs, 768)
if err != nil {
log.Printf("phraser: PhraseChat: %v", err)
return "поговорили.", nil
}
if text, _ := parseResponseMood(resp); text != "" {
text, _, perr := parseResponseMood(resp)
if perr != nil {
log.Printf("phraser: PhraseChat: %v", perr)
return "поговорили.", nil
}
if text != "" {
return text, nil
}
// fallback: plain text without JSON
@@ -267,17 +334,16 @@ func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history [
}
// chatSystemPrompt returns the system prompt for conversational chat.
// Prepends the configured persona when set.
func chatSystemPrompt(persona string) string {
base := `You are maven, a self-hosted personal assistant. You're talking with your owner.
Keep replies brief (1-3 sentences) and natural. You're helpful, curious, and a little warm.
Respond in the user's language (Russian or English, matching their last message).
Never roleplay emotions you don't have, but stay friendly.
Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}. "response" is your reply text; "mood" reflects your tone (neutral/happy/thinking/tired/confused).`
if persona != "" {
base = persona + "\n\n" + base
}
return base
// Prepends the shared context block when the phraser has one.
func chatSystemPrompt(block func() string) string {
// No self-introduction here: the persona block prepended one line above
// already says who she is, same as router.KnowledgePrompt.
base := `Ты разговариваешь с хозяином. О себе говоришь в женском роде ("я подумала", "я рада"). Он мужчина: обращайся к нему на "ты", в мужском роде ("ты сказал", "ты забыл"). Никогда не "вы"/"ваш" и никогда "он"/"его" ты говоришь ему, а не о нём.
Отвечай по-русски, коротко: одна-три фразы, живым языком. Ты доброжелательная, тебе интересно, но чувства не изображай.
Отвечай ТОЛЬКО одним объектом JSON: {"response": "...", "mood": "neutral"}. В "response" твой ответ. В "mood" ровно одно из: neutral, happy, thinking, tired, confused.`
return persona.Prepend(block, base)
}
// chatWithMessages sends a full message array (system + history + current) to
@@ -288,6 +354,7 @@ func (p *LLMPhraser) chatWithMessages(ctx context.Context, msgs []chatMsg, maxTo
Messages: msgs,
Temperature: 0.7,
MaxTokens: maxTokens,
Grammar: p.grammar(),
}
body, err := json.Marshal(req)
if err != nil {
@@ -339,7 +406,12 @@ func (p *LLMPhraser) PhraseReminder(ctx context.Context, d loop.ReminderDecision
if err != nil {
return delivery.PhrasedReminder{}, err
}
body, mood := parseResponseMood(resp)
body, mood, perr := parseResponseMood(resp)
if perr != nil {
// Truncated JSON. Fall through to the reminder's own text.
log.Printf("phraser: PhraseReminder: %v", perr)
body, mood = "", ""
}
if body == "" {
// fallback: try old body/summary format
body, _ = parsePhrase(resp)
@@ -367,6 +439,43 @@ type chatReq struct {
Messages []chatMsg `json:"messages"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
// Grammar is llama-server's `grammar` field (GBNF). Same wiring as
// internal/llm.Req.Grammar. Empty ⇒ unconstrained sampling.
Grammar string `json:"grammar,omitempty"`
}
// responseGrammar — GBNF constraining the model to the documented phrasing
// contract and nothing else: {"response": "<text>", "mood": "<enum>"}.
//
// Without it a 0.8B answers roughly one chat turn in three with open reasoning
// as plain text ("Thinking Process:" …), which no tag-stripper can remove and
// which eats the token budget before the JSON closes. Modelled on
// routeGrammar in internal/router/llmrouter.go so the two read alike.
//
// text accepts ANY codepoint except the two JSON must escape — the replies are
// Russian, so an ASCII-only rule would make every reply empty. The escape rule
// is what lets the model close a string it opened with a quote inside. Length
// is bounded so a repetition loop truncates the field, not the JSON object.
//
// That bound was 400 and 400 was too tight. Measured against Qwen3.5-0.8B: on
// "почему гром слышно позже молнии?" the reply came back exactly 400 characters
// long, cut mid-word ("Нужно записать и,"), at every token cap from 256 to 2048.
// So the token cap was never what stopped it — this rule was. 1000 characters is
// roughly six Russian sentences, still short enough to stop a repetition loop.
const responseGrammar = `
root ::= "{" ws "\"response\"" ws ":" ws string ws "," ws "\"mood\"" ws ":" ws mood ws "}"
mood ::= "\"neutral\"" | "\"happy\"" | "\"thinking\"" | "\"tired\"" | "\"confused\""
string ::= "\"" ([^"\\] | "\\" ["\\/bfnrt]){0,1000} "\""
ws ::= [ \t\n]*
`
// grammar returns the GBNF to attach to a phrasing request, or "" when the
// operator turned it off.
func (p *LLMPhraser) grammar() string {
if p.cfg.NoGrammar {
return ""
}
return responseGrammar
}
type chatResp struct {
@@ -391,6 +500,7 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma
},
Temperature: 0.7,
MaxTokens: maxTokens,
Grammar: p.grammar(),
}
body, err := json.Marshal(req)
if err != nil {
@@ -439,8 +549,10 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma
// as "..." before this. See PHRASING-EVAL-31-07-2026.md.
//
// Russian only, feminine self-reference, second person masculine (the owner is
// a man). One short sentence — the nudge is spoken aloud.
// a man). She talks TO him, informally, singular — never "вы", never "он".
// One short sentence — the nudge is spoken aloud.
const nudgeSystem = `Ты Maven, домашняя ассистентка. О себе говоришь в женском роде ("я проверила", "я записала"). Владелец мужчина, обращайся к нему в мужском роде ("ты пил", "ты забыл").
Говоришь с ним на "ты", в единственном числе ("выпей", "встань"). Никогда не "вы"/"вас"/"ваш" и никогда "он"/"его" ты говоришь ему, а не о нём.
Пиши ОДНО короткое напоминание по-русски: не больше 120 символов и не больше 16 слов. Только по делу.
@@ -457,21 +569,16 @@ const nudgeSystem = `Ты — Maven, домашняя ассистентка. О
Это примеры ФОРМЫ, а не темы. Пиши только про ту ситуацию, которую тебе дали в запросе. Не копируй примеры и никогда не пиши "..." в поле response.`
func (p *LLMPhraser) systemPrompt() string {
base := nudgeSystem
if p.cfg.Persona != "" {
base = p.cfg.Persona + "\n\n" + base
}
return base
return persona.Prepend(p.cfg.ContextBlock, nudgeSystem)
}
// querySystemPrompt returns the system prompt for PhraseQuery (notes + general
// knowledge). Prepends the configured persona when set.
func (p *LLMPhraser) querySystemPrompt() string {
base := "You are maven, a self-hosted personal assistant answering from your notes. Answer briefly and naturally in Russian starting with \"вот что я нашла: \". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}."
if p.cfg.Persona != "" {
base = p.cfg.Persona + "\n\n" + base
}
return base
// No self-introduction here: the persona block prepended one line above
// already says who she is, same as router.KnowledgePrompt.
base := "Ты отвечаешь ему по своим заметкам. Отвечай по-русски, коротко и своими словами, начинай с \"вот что я нашла: \". О себе — в женском роде (\"нашла\", \"записала\"). Он мужчина, обращайся к нему на \"ты\". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}."
return persona.Prepend(p.cfg.ContextBlock, base)
}
// ruleTopics — Russian gloss for each built-in rule name. The rule names are
@@ -592,21 +699,39 @@ type responseMood struct {
Mood string `json:"mood"`
}
// errBrokenJSON — the model started a JSON object and never finished it.
// That is a failed generation, not a reply. Callers must use their fallback.
var errBrokenJSON = fmt.Errorf("phraser: model output starts as JSON but does not parse")
// parseResponseMood extracts {"response","mood"} from LLM output, tolerant
// of thinking tokens and extra text before/after the JSON block. Returns
// ("", "") when no valid JSON is found.
func parseResponseMood(raw string) (response, mood string) {
// of thinking tokens and extra text before/after the JSON block.
//
// Three outcomes:
// - parsed fine → the fields, nil error.
// - output never looked like JSON → ("", "", nil). The caller may ship it
// as-is; small models sometimes answer in bare prose and that is fine.
// - output starts with "{" but does not parse → errBrokenJSON. The grammar
// guarantees a valid *prefix*, so a generation that hits the token cap
// mid-object comes back as a fragment like `{` or `{\n "`. Shipping that
// as a reply is the bug this error exists to stop.
func parseResponseMood(raw string) (response, mood string, err error) {
cleaned := strings.TrimSpace(raw)
start := strings.Index(cleaned, "{")
end := strings.LastIndex(cleaned, "}")
if start < 0 || end < 0 || end <= start {
return "", ""
if strings.HasPrefix(cleaned, "{") {
return "", "", errBrokenJSON
}
return "", "", nil
}
var parsed responseMood
if err := json.Unmarshal([]byte(cleaned[start:end+1]), &parsed); err != nil {
return "", ""
if e := json.Unmarshal([]byte(cleaned[start:end+1]), &parsed); e != nil {
if strings.HasPrefix(cleaned, "{") {
return "", "", errBrokenJSON
}
return "", "", nil
}
return parsed.Response, parsed.Mood
return parsed.Response, parsed.Mood, nil
}
func parsePhrase(raw string) (body, summary string) {
+261
View File
@@ -0,0 +1,261 @@
package phraser
// Hand-written Russian nudges instead of generated ones.
//
// Why: on a nudge there is nothing to be creative about. Measured over many
// runs, Qwen3.5-0.8B breaks the persona (formal "вы", plural imperatives,
// masculine self-reference) and invents facts and units — it once told him to
// boil an egg for "90-95 секунд". A nudge is five words of known content, so
// wording it with a model buys nothing and risks the persona every time.
//
// The wording lives in nudges_ru_v1.json so it can be edited without touching
// Go. This file only picks one and fills in the values.
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"math/rand"
"regexp"
"strings"
"sync"
"time"
"unicode"
"github.com/kami/maven/internal/delivery"
"github.com/kami/maven/internal/loop"
)
//go:embed nudges_ru_v1.json
var nudgeTemplateJSON []byte
// NudgeTemplateSchemaVersion — the version this code understands.
const NudgeTemplateSchemaVersion = 1
type nudgeRuleSet struct {
Mood string `json:"mood"`
Variants []string `json:"variants"`
}
type nudgeTemplateFile struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
Notes []string `json:"notes"`
Rules map[string]nudgeRuleSet `json:"rules"`
}
// NudgeTemplates picks a hand-written Russian nudge for a candidate.
//
// Safe for concurrent use. Random, but never the same variant twice in a row
// for the same rule — being nagged with identical words is what makes a nudge
// easy to tune out.
type NudgeTemplates struct {
mu sync.Mutex
rnd *rand.Rand
last map[string]string // rule family -> the text used last time
file nudgeTemplateFile
}
// NewNudgeTemplates loads the embedded template file. Pass a source to make the
// picking reproducible in tests; nil means seed from the clock.
func NewNudgeTemplates(src rand.Source) (*NudgeTemplates, error) {
var f nudgeTemplateFile
if err := json.Unmarshal(nudgeTemplateJSON, &f); err != nil {
return nil, fmt.Errorf("nudge templates: parse: %w", err)
}
if f.SchemaVersion != NudgeTemplateSchemaVersion {
return nil, fmt.Errorf("nudge templates: schema_version %d, want %d",
f.SchemaVersion, NudgeTemplateSchemaVersion)
}
if len(f.Rules) == 0 {
return nil, fmt.Errorf("nudge templates: no rules")
}
if src == nil {
src = rand.NewSource(time.Now().UnixNano())
}
return &NudgeTemplates{
rnd: rand.New(src),
last: map[string]string{},
file: f,
}, nil
}
// PhraseNudge implements the nudge half of the Phraser interface, so the
// templates can be scored by the same harness as the model.
func (t *NudgeTemplates) PhraseNudge(_ context.Context, c loop.Candidate) (delivery.PhrasedNudge, error) {
body, mood := t.Nudge(c)
return delivery.PhrasedNudge{Candidate: c, Body: body, Summary: body, Mood: mood}, nil
}
// Nudge returns the text and the mood for one candidate. Never fails: if no
// template fits it uses the plain per-rule fallback.
func (t *NudgeTemplates) Nudge(c loop.Candidate) (body, mood string) {
rule := c.Rule.Name
family := t.family(rule)
set, ok := t.file.Rules[family]
if !ok {
return fallbackNudge(c), "neutral"
}
vals := nudgeValues(c)
// Only variants whose placeholders all have a value.
usable := make([]string, 0, len(set.Variants))
for _, v := range set.Variants {
if text, ok := fillTemplate(v, vals); ok {
usable = append(usable, text)
}
}
if len(usable) == 0 {
return fallbackNudge(c), "neutral"
}
mood = set.Mood
if mood == "" {
mood = "neutral"
}
return t.pick(family, usable), mood
}
// pick chooses at random, skipping whatever this rule said last time.
func (t *NudgeTemplates) pick(family string, usable []string) string {
t.mu.Lock()
defer t.mu.Unlock()
choices := usable
if len(usable) > 1 {
choices = make([]string, 0, len(usable))
for _, v := range usable {
if v != t.last[family] {
choices = append(choices, v)
}
}
if len(choices) == 0 { // every variant equals the last one
choices = usable
}
}
got := choices[t.rnd.Intn(len(choices))]
t.last[family] = got
return got
}
// family maps a rule name to a block in the template file: an exact match
// first, then the prefix of "routine:зарядка" / "morning:утро", then "default".
func (t *NudgeTemplates) family(rule string) string {
if _, ok := t.file.Rules[rule]; ok {
return rule
}
if i := strings.IndexByte(rule, ':'); i > 0 {
if _, ok := t.file.Rules[rule[:i]]; ok {
return rule[:i]
}
}
return "default"
}
// placeholderRE — the {name} slots a template may use.
var placeholderRE = regexp.MustCompile(`\{([a-z]+)\}`)
// nudgeValues collects what this candidate can fill in. A key missing here
// means every template needing it is skipped, so nothing half-filled is ever
// spoken.
func nudgeValues(c loop.Candidate) map[string]string {
vals := map[string]string{}
rule := c.Rule.Name
// {since} — only at hour scale. Below an hour the phrase would be minutes,
// and none of the templates read well with "сорок минут".
if d, ok := c.State.Since(rule); ok && d >= time.Hour {
if s := ruSinceWords(d); s != "" {
vals["since"] = s
}
}
// {service} — the aggregate fact's key carries the service name.
if f, ok := c.State.Fact(rule); ok && f.Key != "" && f.Key != rule {
vals["service"] = f.Key
}
// {what} — the Russian suffix of "routine:таблетки" / "morning:утро".
if i := strings.IndexByte(rule, ':'); i > 0 && i+1 < len(rule) {
vals["what"] = rule[i+1:]
}
return vals
}
// fillTemplate substitutes the placeholders. Returns false when a value is
// missing, so a raw "{since}" can never reach the text-to-speech voice.
func fillTemplate(tmpl string, vals map[string]string) (string, bool) {
missing := false
out := placeholderRE.ReplaceAllStringFunc(tmpl, func(m string) string {
name := m[1 : len(m)-1]
v, ok := vals[name]
if !ok || v == "" {
missing = true
return m
}
return v
})
if missing || strings.ContainsAny(out, "{}%") {
return "", false
}
return capitalizeFirst(out), true
}
// capitalizeFirst — a placeholder can start the sentence, and "полтора часа без
// перерыва" should be spoken as a sentence, not a fragment.
func capitalizeFirst(s string) string {
for i, r := range s {
return string(unicode.ToUpper(r)) + s[i+len(string(r)):]
}
return s
}
// hourWords — hours spelled out. "3 ч" is fine on a screen and wrong in a
// Russian voice, so the number goes out as words.
var hourWords = []string{
"ноль", "один", "два", "три", "четыре", "пять", "шесть", "семь", "восемь",
"девять", "десять", "одиннадцать", "двенадцать", "тринадцать",
"четырнадцать", "пятнадцать", "шестнадцать", "семнадцать", "восемнадцать",
"девятнадцать", "двадцать", "двадцать один", "двадцать два", "двадцать три",
}
// hourPlural — час / часа / часов by Russian counting rules.
func hourPlural(h int) string {
if h%100 >= 11 && h%100 <= 14 {
return "часов"
}
switch h % 10 {
case 1:
return "час"
case 2, 3, 4:
return "часа"
default:
return "часов"
}
}
// ruSinceWords — "полтора часа", "два с половиной часа", "семь часов".
// Empty string means "do not say it" (under an hour, or over a day).
func ruSinceWords(d time.Duration) string {
if d < time.Hour {
return ""
}
h := int(d.Hours())
m := int(d.Minutes()) % 60
if m >= 45 {
h++
m = 0
}
if h >= len(hourWords) {
return "больше суток"
}
if h == 1 {
if m >= 15 {
return "полтора часа"
}
return "час"
}
if m >= 15 {
return hourWords[h] + " с половиной часа"
}
return hourWords[h] + " " + hourPlural(h)
}
+202
View File
@@ -0,0 +1,202 @@
package phraser
import (
"context"
"math/rand"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/store"
)
// cand builds a candidate the way a tick would.
func cand(rule string, sinceMin int, factKey string) loop.Candidate {
now := time.Date(2026, 7, 31, 21, 40, 0, 0, time.UTC)
st := loop.State{Now: now, Facts: map[string]store.Fact{}}
if sinceMin > 0 || factKey != "" {
key := rule
if factKey != "" {
key = factKey
}
st.Facts[rule] = store.Fact{Key: key, Ts: now.Add(-time.Duration(sinceMin) * time.Minute)}
}
return loop.Candidate{Rule: loop.Rule{Name: rule, Severity: loop.Sev1}, Severity: loop.Sev1, State: st}
}
func newTestTemplates(t *testing.T, seed int64) *NudgeTemplates {
t.Helper()
nt, err := NewNudgeTemplates(rand.NewSource(seed))
if err != nil {
t.Fatalf("NewNudgeTemplates: %v", err)
}
return nt
}
func TestNudgeTemplatesLoad(t *testing.T) {
nt := newTestTemplates(t, 1)
for _, rule := range []string{"water", "meal", "break", "service_down", "netdata_critical", "routine", "morning", "default"} {
set, ok := nt.file.Rules[rule]
if !ok {
t.Errorf("no templates for %q", rule)
continue
}
if len(set.Variants) < 5 {
t.Errorf("%s: only %d variants", rule, len(set.Variants))
}
// Every rule needs one variant that needs no value, or a candidate
// without context has nothing to say. routine and morning are exempt:
// they always carry a name and must always say it.
plain := 0
seen := map[string]bool{}
for _, v := range set.Variants {
if !placeholderRE.MatchString(v) {
plain++
}
if seen[v] {
t.Errorf("%s: duplicate variant %q", rule, v)
}
seen[v] = true
}
if plain == 0 && rule != "routine" && rule != "morning" {
t.Errorf("%s: every variant needs a placeholder value", rule)
}
}
}
// The whole point of the picker: never the same words twice in a row.
func TestNudgeNoImmediateRepeat(t *testing.T) {
nt := newTestTemplates(t, 7)
prev := ""
for i := 0; i < 200; i++ {
body, _ := nt.Nudge(cand("water", 200, ""))
if body == prev {
t.Fatalf("repeat at %d: %q", i, body)
}
prev = body
}
}
// Same seed, same sequence — otherwise the fixture score would drift run to run.
func TestNudgeDeterministicWithSeed(t *testing.T) {
var runs [2][]string
for r := range runs {
nt := newTestTemplates(t, 42)
for i := 0; i < 20; i++ {
body, _ := nt.Nudge(cand("break", 100, ""))
runs[r] = append(runs[r], body)
}
}
for i := range runs[0] {
if runs[0][i] != runs[1][i] {
t.Fatalf("run %d differs: %q vs %q", i, runs[0][i], runs[1][i])
}
}
}
// A variant is only used when its value exists, and nothing half-filled ships.
func TestNudgeNoLeftoverPlaceholders(t *testing.T) {
nt := newTestTemplates(t, 3)
cases := []loop.Candidate{
cand("water", 0, ""), // no duration
cand("water", 30, ""), // under an hour
cand("water", 200, ""), // hours
cand("service_down", 3, "vaultwarden"),
cand("service_down", 3, ""), // no service name
cand("routine:таблетки", 0, ""),
cand("morning:утро", 0, ""),
cand("unknown_rule", 0, ""),
}
for _, c := range cases {
for i := 0; i < 40; i++ {
body, mood := nt.Nudge(c)
if body == "" {
t.Fatalf("%s: empty body", c.Rule.Name)
}
if strings.ContainsAny(body, "{}%") {
t.Fatalf("%s: unfilled template %q", c.Rule.Name, body)
}
if mood != "neutral" {
t.Fatalf("%s: mood %q", c.Rule.Name, mood)
}
}
}
}
// The routine name must actually land in the text.
func TestNudgeSubstitutesWhat(t *testing.T) {
nt := newTestTemplates(t, 11)
for i := 0; i < 40; i++ {
body, _ := nt.Nudge(cand("routine:таблетки", 0, ""))
if !strings.Contains(strings.ToLower(body), "таблетки") {
t.Fatalf("routine text lost the name: %q", body)
}
}
}
func TestRuSinceWords(t *testing.T) {
cases := []struct {
min int
want string
}{
{30, ""},
{60, "час"},
{95, "полтора часа"},
{150, "два с половиной часа"},
{190, "три часа"},
{240, "четыре часа"},
{430, "семь часов"},
{660, "одиннадцать часов"},
{60 * 30, "больше суток"},
}
for _, c := range cases {
got := ruSinceWords(time.Duration(c.min) * time.Minute)
if got != c.want {
t.Errorf("%d min: got %q want %q", c.min, got, c.want)
}
}
}
// Templates are the default: a nudge must not reach the model at all.
func TestLLMPhraserUsesTemplatesByDefault(t *testing.T) {
spy := newGrammarSpy(t)
p := NewLLMPhraserAt(spy.srv.URL, Config{})
pn, err := p.PhraseNudge(context.Background(), cand("water", 200, ""))
if err != nil {
t.Fatalf("PhraseNudge: %v", err)
}
if len(spy.grammars) != 0 {
t.Errorf("nudge hit the model %d times, want 0", len(spy.grammars))
}
if !strings.Contains(strings.ToLower(pn.Body), "вод") {
t.Errorf("nudge is not the water template: %q", pn.Body)
}
}
// ...and the flag brings the model back.
func TestLLMNudgesFlagRestoresTheModel(t *testing.T) {
spy := newGrammarSpy(t)
p := NewLLMPhraserAt(spy.srv.URL, Config{LLMNudges: true})
pn, err := p.PhraseNudge(context.Background(), cand("water", 200, ""))
if err != nil {
t.Fatalf("PhraseNudge: %v", err)
}
if len(spy.grammars) != 1 {
t.Fatalf("nudge hit the model %d times, want 1", len(spy.grammars))
}
if pn.Body != "ага" {
t.Errorf("body = %q, want the model's reply", pn.Body)
}
}
func TestNudgeTemplatesPhraseNudge(t *testing.T) {
nt := newTestTemplates(t, 5)
pn, err := nt.PhraseNudge(context.Background(), cand("water", 200, ""))
if err != nil {
t.Fatalf("PhraseNudge: %v", err)
}
if pn.Body == "" || pn.Summary != pn.Body || pn.Mood != "neutral" {
t.Fatalf("bad nudge: %+v", pn)
}
}
+129
View File
@@ -0,0 +1,129 @@
{
"schema_version": 1,
"name": "russian nudge templates v1",
"notes": [
"Hand-written Russian nudges. Edit the wording here, no Go changes needed.",
"Rules: she is feminine about herself, he is a man addressed as ты. Never вы/вас/ваш, never plural imperatives (выпейте), never он/его about him.",
"One short sentence. No questions, no emoji, no pet names, no emotional support.",
"Placeholders: {since} how long it has been (only used when it is at least an hour), {service} the service name, {what} the routine name. A variant whose placeholder has no value is skipped, so every rule needs at least one variant with no placeholder. The exception is routine and morning: those only exist for rules like routine:таблетки that always carry a name, and a routine nudge that drops the name is useless.",
"mood must be one of: neutral, happy, thinking, tired, confused."
],
"rules": {
"water": {
"mood": "neutral",
"variants": [
"Ты не пил воду {since} — выпей стакан.",
"Пора выпить воды.",
"Стакан воды не помешает.",
"Воду ты не пил уже {since}.",
"Напоминаю про воду.",
"Сходи за водой, дела подождут.",
"Сделай глоток воды, пока помнишь.",
"Между делом выпей воды.",
"Вода — простое дело: выпей стакан.",
"Отвлекись на стакан воды."
]
},
"meal": {
"mood": "neutral",
"variants": [
"Ты не ел {since} — поешь.",
"Пора поесть, сделай перекус.",
"Еда важнее ещё одного часа за столом.",
"Без еды уже {since}, поешь.",
"Напоминаю про еду — поешь.",
"Возьми перерыв на обед.",
"Сделай себе перекус, это пять минут.",
"Поешь, потом вернёшься к работе.",
"Поешь нормально, а не на ходу.",
"Еды не было {since} — разогрей что-нибудь."
]
},
"break": {
"mood": "neutral",
"variants": [
"Ты за столом {since} — встань и разомнись.",
"Пора сделать перерыв.",
"Встань на пять минут.",
"{since} без перерыва — отойди от экрана.",
"Напоминаю про перерыв.",
"Разомни спину, потом продолжишь.",
"Короткая пауза не сорвёт дела.",
"Отойди от компьютера на минуту.",
"Сидишь без перерыва {since}.",
"Встань, пройдись, вернись."
]
},
"service_down": {
"mood": "neutral",
"variants": [
"Сервис {service} не отвечает.",
"{service} упал — сервис не отвечает.",
"{service} не отвечает, сервис нужно поднимать.",
"Сервис {service} недоступен.",
"Проверь {service}: сервис не отвечает.",
"Сервис перестал отвечать.",
"Сервис {service} лежит, нужно смотреть.",
"{service} не отвечает уже {since}.",
"Мониторинг сообщает: {service} лежит.",
"Сервис {service} не отвечает, посмотри логи."
]
},
"netdata_critical": {
"mood": "neutral",
"variants": [
"Netdata: критический алярм, проверь диск.",
"Критический алярм в netdata — посмотри диск.",
"Netdata поднял тревогу по диску.",
"Проверь диск: netdata ругается.",
"Алярм от netdata, критический.",
"Netdata: критический уровень, дело в диске.",
"Диск требует внимания — критический алярм в netdata.",
"Критический алярм: проверь место на диске.",
"Netdata сообщает о критической проблеме с диском.",
"Открой netdata: там критический алярм по диску."
]
},
"routine": {
"mood": "neutral",
"variants": [
"По распорядку: {what}.",
"Пора — {what}.",
"Напоминаю: {what}.",
"В списке на сейчас: {what}.",
"{what} — сейчас самое время.",
"Не пропусти: {what}.",
"{what}: пора сделать.",
"Сейчас по плану {what}.",
"Твой распорядок: {what}.",
"{what} — по распорядку сейчас."
]
},
"morning": {
"mood": "neutral",
"variants": [
"{what} — пора начать день.",
"{what}: пройди утренний список.",
"Начни {what} со списка.",
"{what}. Осталось пройти чеклист.",
"Утренний список ещё не пройден: {what}.",
"{what}: первый пункт списка за тобой.",
"{what} идёт, а список стоит.",
"{what}: не забудь про утренние дела.",
"По утреннему чеклисту ещё есть дела: {what}.",
"{what} — утренний список дел ещё ждёт."
]
},
"default": {
"mood": "neutral",
"variants": [
"Напоминаю: есть дело.",
"Пора вернуться к отложенному делу.",
"Одно дело ждёт тебя.",
"Напоминаю про дело из списка.",
"В списке осталось дело.",
"Дело всё ещё не сделано."
]
}
}
}
+3 -3
View File
@@ -61,12 +61,12 @@ func TestLLMRouterBaseline(t *testing.T) {
}
ctx := context.Background()
model, err := ModelID(ctx, base)
model, err := llm.ModelID(ctx, base)
if err != nil {
// Not fatal: an unlabelled score is still a score. But say so loudly,
// because an unlabelled row in a bake-off table is worthless.
t.Logf("could not read model id from %s: %v — reports will say %q", base, err, "unknown-model")
model = "unknown-model"
t.Logf("could not read model id from %s: %v — reports will say %q", base, err, llm.UnknownModel)
model = llm.UnknownModel
}
t.Logf("scoring model %s at %s", model, base)
lr := router.NewLLMRouter(client)
+4 -1
View File
@@ -3,5 +3,8 @@ package router
// KnowledgePrompt returns the system prompt for general knowledge questions
// that the phraser uses when no notes match the query.
func KnowledgePrompt() string {
return `Ты — Мавена, персональный ассистент. Ответь кратко из своих знаний. Если не знаешь — скажи "не знаю". Не выдумывай. Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}.`
// No self-introduction here: the shared persona block already says who she
// is, and this line used to disagree with it — a different name ("Мавена")
// and a masculine noun ("ассистент") in front of a feminine persona.
return `Ответь кратко из своих знаний. Если не знаешь — скажи "не знаю". Не выдумывай. Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}.`
}
+9 -1
View File
@@ -11,10 +11,18 @@ func TestKnowledgePrompt(t *testing.T) {
t.Fatal("KnowledgePrompt returned empty string")
}
// Must contain key instructions
checks := []string{"Мавена", "не знаю", "не выдумывай"}
checks := []string{"не знаю", "не выдумывай"}
for _, c := range checks {
if !strings.Contains(strings.ToLower(prompt), strings.ToLower(c)) {
t.Errorf("KnowledgePrompt should mention %q", c)
}
}
// Who she is comes from the shared persona block now. This prompt used to
// say it too, with a different name and a masculine noun, which is the
// drift the block exists to stop.
for _, w := range []string{"Мавена", "ассистент"} {
if strings.Contains(prompt, w) {
t.Errorf("KnowledgePrompt should not introduce her (%q) — the persona block does", w)
}
}
}
+79 -4
View File
@@ -1,8 +1,10 @@
#!/usr/bin/env bash
# Unified script to stop all Maven services.
# Usage: ./kill-maven.sh
# - Graceful SIGTERM is attempted first.
# - If any process lingers, force with SIGKILL.
# - Docker deploy: `docker compose stop` (see why below).
# - Bare-metal / dev run: graceful SIGTERM first, SIGKILL if anything lingers.
# Exits non-zero if it cannot confirm everything is stopped. It must never say
# "stopped" unless it checked.
set -euo pipefail
@@ -24,6 +26,73 @@ else
LLM='llama-server.*\.gguf'
fi
COMPOSE_FILE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/docker-compose.yml"
# --- containerised deploy ------------------------------------------------
# docker-compose.yml does not set `pid: host`, so each container has its own
# PID namespace: pkill on the host sees nothing inside them. This script used
# to print "all stopped" while every daemon was still happily running. Stop the
# containers through compose instead — that actually reaches them.
#
# running_containers prints the ids of the project's running containers, or
# nothing. Empty output plus a non-zero return means "could not ask docker",
# which is different from "nothing is running" and is handled below.
running_containers() {
docker compose -f "$COMPOSE_FILE" ps -q --status running 2>/dev/null
}
DOCKER_OK=0
CONTAINERS=""
if command -v docker >/dev/null 2>&1 && [ -f "$COMPOSE_FILE" ]; then
if CONTAINERS="$(running_containers)"; then
DOCKER_OK=1
fi
fi
if [ "$DOCKER_OK" = 1 ] && [ -n "$CONTAINERS" ]; then
echo "--- Maven is running in containers: stopping via docker compose ---"
if ! docker compose -f "$COMPOSE_FILE" stop; then
echo "ERROR: 'docker compose stop' failed. Containers may still be running." >&2
exit 1
fi
echo "--- Verifying containers are gone ---"
LEFT="$(running_containers || true)"
if [ -n "$LEFT" ]; then
echo "ERROR: containers still running after stop:" >&2
docker compose -f "$COMPOSE_FILE" ps >&2 || true
exit 1
fi
echo "All containers stopped."
exit 0
fi
# --- bare-metal / dev run -----------------------------------------------
# pgrep -f matches whole command lines, so a shell that merely mentions
# "mavend" (this script's own parent, for one) shows up. Drop ourselves and our
# parent, otherwise the SIGKILL sweep can take out the terminal you ran this in.
host_pids() {
pgrep -f "$PAT|$LLM" | grep -v -e "^$$\$" -e "^$PPID\$" | paste -sd, - || true
}
HOST_PIDS=$(host_pids)
if [ -z "$HOST_PIDS" ]; then
# Nothing on the host. Whether that means "already down" depends on whether
# we managed to ask docker, and the two must not read the same.
if [ "$DOCKER_OK" = 1 ]; then
# Docker answered and named no running containers, and there is nothing
# on the host either. That is a real answer: Maven is already stopped.
echo "Nothing to stop: no Maven processes and no running containers."
exit 0
fi
# We could not ask docker, so Maven may be alive in a container we cannot
# see. Saying "stopped" here is the exact false success this script had.
echo "ERROR: no Maven processes on this host, and docker could not be asked." >&2
echo " If this is the container deploy it may still be running:" >&2
echo " docker compose -f $COMPOSE_FILE stop" >&2
echo " Nothing was stopped. Check by hand before assuming Maven is down." >&2
exit 1
fi
echo "--- Sending graceful SIGTERM to Maven services ---"
pkill -TERM -f "$PAT" || true
# mavend's Pdeathsig SIGKILLs its llama-server on exit, but sweep strays too
@@ -32,12 +101,18 @@ pkill -TERM -f "$LLM" || true
echo "--- Verifying processes are gone ---"
sleep 1
PIDS=$(pgrep -d ',' -f "$PAT|$LLM") || PIDS=""
PIDS=$(host_pids)
if [ -n "$PIDS" ]; then
echo "Warning: some processes still alive. PIDs: $PIDS"
echo "--- Force killing with SIGKILL ---"
echo "$PIDS" | tr ',' '\n' | xargs -r kill -9
sleep 1
LEFT=$(host_pids)
if [ -n "$LEFT" ]; then
echo "ERROR: still alive after SIGKILL. PIDs: $LEFT" >&2
exit 1
fi
echo "Done (SIGKILL)."
else
echo "All services gracefully stopped."
fi
fi