Compare commits

...

28 Commits

Author SHA1 Message Date
kami 0b90952e55 Write down every conversational eval score from tonight
Records all four configurations on the 27-case talk fixture, three runs
each: no grammar, plus grammar, plus Russian prompts, plus the truncation
fix. Composite, per-path and per-check, with the reproduce command.

The short version is that the plumbing got fixed and the score barely
moved. Grammar was the real win. Russian prompts helped a little and cut
latency by 5x. The truncation fix was necessary and bought nothing.

Also writes down three things that are easy to lose:

- The truncation cause was the grammar's 400-character bound, not the
  token cap. Measured at three caps, same 400 characters every time.
- Then I set the bound to 1000 against a 768-token cap and made it worse.
  The two limits have to agree.
- One run is contaminated and marked void: I ran an agent against the same
  llama-server, and the report still claimed zero errors while a third of
  the fixture silently answered "не знаю.". That is #397 and it is worse
  than filed — a busy server is indistinguishable from bad phrasing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 18:18:19 +04:00
kami aa8f5b2ee2 Make the nonempty check look for actual words
It scored 27/27 on a run where two replies were "{" and "{\n  \"". It only
tested that the string was not blank, so punctuation counted as content and
the worst replies of the run passed the first check.

Now a reply needs at least one letter, Cyrillic or Latin. Latin counts
because answers about ssd or vpn are legitimately part English.

Digits alone fail too. The same run answered "сколько варить яйцо
вкрутую?" with "15-16" — no unit, no words, and the wrong number as well.
That is not something she said.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:57:39 +04:00
kami d7cdcb63bd Stop shipping half-written JSON as a reply
Two bugs, one symptom. A run of the talk eval produced replies that were
literally "{" and "{\n  \"" — those strings went out as things Maven said.

First bug: the parser could not tell "the model answered in plain prose"
from "the model started a JSON object and got cut off". Both came back as
empty, and every caller then shipped the raw text. Now an unfinished object
returns an error and each caller uses its own fallback instead. Bare prose
with no JSON in it still passes through, because small models do sometimes
answer that way and the reply is fine.

Second bug, and the actual cause: the grammar capped the response field at
400 characters. I measured it against Qwen3.5-0.8B at three different token
caps — 256, 768 and 2048 — and the reply came back exactly 400 characters
every time, cut mid-word. So the token limit was never what stopped it.
The bound is 1000 now, about six Russian sentences, still low enough to cut
off a repetition loop.

Token caps go from 256 to 768 on the chat and query paths so 1000
characters of Russian actually fits. The nudge path keeps its own cap; a
nudge is meant to be one sentence.

Note: cmd/mavend/replier_llm.go has its own copy of this parser with the
same bug. Left alone here so this commit stays small — that duplicate is
Vikunja #396.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:56:55 +04:00
kami c7dadc97d9 Write the chat and notes prompts in Russian
The reply has to be Russian, but two of the phrasing prompts told her
what to do in English. Both are Russian now, in the same style as the
nudge prompt that already works better.

Also dropped the "you are maven, a self-hosted personal assistant"
line from both. The persona block right above it already says who she
is, so it was said twice.

The JSON part is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:38:36 +04:00
kami c9d88c152e Drop "never phones home" as a hard rule
The owner's call, 2026-07-31: a 0.8B model does not know enough about the
world to be useful without reading something. So she may now read external
sources to answer world questions.

What replaces the old rule, in all three docs:

- No telemetry, no cloud model, no third-party account. Unchanged.
- Local first: the Kiwix ZIMs on the box before anything on the network.
- External search is allowed but off unless configured, same as weather
  and telegram.
- His notes and facts are never search input. Only the utterance goes out
  — never the persona block, the history, or matched notes.

Docs only, no code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:34:53 +04:00
kami 1890ff5d5d Constrain the phrasing output with a GBNF grammar
The 0.8B answered about one chat turn in three with open reasoning as plain text, so no JSON ever closed and the fallback shipped "Thinking Process:" to the user. A grammar makes that output impossible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 17:18:06 +04:00
kami 0110e9bc8c Report every address break, and stop -те verbs blinding the check
From a real reply in a nudge eval run: "Смотрите на его потребление
воды" is a plural imperative AND third person about him. Only the plural
printed.

Two separate faults. The check returned on its first hit, so the second
break stayed invisible and the failure read as milder than it was; it now
joins them. And "его" was not detected at all — looksVerb knows the
-й/-йте imperative but not the -те plural, so "смотрите" counted as the
person being talked about, which is what an antecedent means here.
pluralVerb already knows that form, so the antecedent test uses it too.

Third time a verb form has blinded this check. A fourth means it wants a
morphology table rather than another suffix.
2026-07-31 16:52:16 +04:00
kami 50ca8c8b5a Score the chat, query and knowledge phrasing paths (#395)
The phrasing fixture was 15 nudge cases, so every prompt change we
measured only told us about nudges. But the shared context block sits in
front of five prompts, and three of them — chat, note query, general
knowledge — had no scorer at all. Those are the long free-form replies,
where a persona break is most likely and where nothing could see one.

27 cases, nine per path. Nine rather than five because the nudge fixture
already cannot resolve a change smaller than about three cases, and a
per-path score off five would be worse.

Reuses the persona checks instead of copying them. Length, mood and
"no questions" are left out on purpose: these paths return no mood, and
a follow-up question is a feature in chat, not a fault.

The run refuses to score unless the model answers before and after it.
PhraseChat and PhraseQuery swallow model errors and return a canned
string, so without that guard a dead server produces a full report with
zero errors and a bad score — which reads as bad phrasing rather than as
nothing measured. Vikunja #397 is the real fix.
2026-07-31 16:51:52 +04:00
kami de09471421 Merge the shared prompt context block 2026-07-31 16:07:35 +04:00
kami d65c16a567 Don't tell her she can't talk
The block listed what she can do and ended with "nothing else". It sits
in front of the chat and general-knowledge prompts too, so that told her
to refuse the exact thing those prompts are for. Talking is now first in
the list, and the closing line limits ACTIONS rather than everything.

Also dropped the self-introduction from the knowledge prompt. It said
"Мавена, персональный ассистент" — a different name and a masculine
noun, right after the block says she is Maven and feminine. Identity
lives in the block now.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 16:07:35 +04:00
kami 062d4252ef Tell her what she can actually do
The context block now lists her real capabilities: reminders, notes and
facts (write and recall), and the calendar — all three are code paths in
mavend today. Weather, telegram and shell acts are listed only when the
config actually has them, because offering something she cannot do is
worse than staying quiet about it.

Also drops the pronouns from the optional name/city line. The block's
own "ты" is Maven, so "тебя зовут" read as her name and "его" would have
shown her the third-person form she must never use about him. They are
plain labels now.

Vikunja #394.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 15:57:58 +04:00
kami 2c27e2ce1f Give every prompt one shared context block
The "address him as ты" rule had only reached two of the five system
prompts. Instead of pasting it into the other three (five copies drift —
that is how this happened), there is now one block, in internal/persona,
prepended to all five: nudges, action replies, chat, note queries and
general knowledge.

The block says who he is and how to address him (a man, always "ты",
never "вы", never "он" about him; Maven stays feminine), plus the
current local date and time. It is rendered fresh each turn because the
time changes, and it is correct with an empty config — the address and
gender rules are defaults in code. Config only adds optional facts:
owner_name, city, and the existing free-text `persona` string, which is
now the static half of the block.

Russian even in front of the English prompts: the rules are Russian
grammar, so they read best stated in Russian, and there is one copy.

Vikunja #394.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 15:55:30 +04:00
kami ccc5cba2a3 Merge the example-led prompt finding 2026-07-31 15:41:01 +04:00
kami 89d83c0b11 Record the example-led nudge prompt experiment (#393) — it made things worse
Tried rewriting the nudge prompt to lead with five on-topic examples instead
of rules. Three eval runs each side: before 12/13/14 of 15, after 11/12/11.
The loss is all in the address check — formal "вы" and plural imperatives came
back once the "говоришь на ты" rule stopped being its own sentence, and the
on-topic examples leaked their wording into the wrong cases.

Prompt reverted. Only the finding is committed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 15:40:18 +04:00
kami a97f554802 Merge the informal address prompt rule 2026-07-31 14:54:20 +04:00
kami f4de2fc5e1 Don't let a verb count as the person being talked about
The third-person check asks whether anyone else was named before "он".
A nudge is mostly verbs, and they were counted as possible people, so
"попробуй встать и отдохнуть — у него есть перерыв" passed. Infinitives
and imperatives now join past tense as words that cannot be a person.

A plain noun before the pronoun still blinds it. That needs a parser,
and the comment says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:54:20 +04:00
kami eef5d4da4f Tell the phraser to speak to him informally, singular
The prompts stated the feminine self-reference rule but never said whom she is
speaking to, so the model produced formal plural ("Жду вас") and talked about
him in third person ("Он не ел 11 дней"). Adds the address rule right next to
the feminine one, in the nudge prompt and the confirmation prompt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:52:23 +04:00
kami 09f1696fce Merge the eval label and kill script fixes 2026-07-31 14:32:45 +04:00
kami 80f7322294 Don't fail when docker confirms nothing is running
"Nothing on the host" meant two different things and the script treated
them the same. If docker answers and names no running containers, Maven
really is down and the script should say so and exit 0. Only when docker
cannot be asked is the answer unknown, and that is the case that must
fail loudly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:32:45 +04:00
kami fa5aebfbe4 Merge the delivery boundary fixes 2026-07-31 14:30:54 +04:00
kami 59cec63da1 List the columns in the table rebuild
The migration copied rows with SELECT *, which matches columns by
position. It is correct today, but if the old table's order ever
differed it would shuffle every row instead of failing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:30:54 +04:00
kami 02e8786695 Stop the containers instead of claiming success (#380)
docker-compose.yml has no 'pid: host', so each container has its own PID
namespace and pkill on the host matches nothing inside them. The script
then printed "All services gracefully stopped" while mavend, its
llama-server and the rest were still running.

Now it checks for running compose containers first and stops them with
docker compose. If it cannot ask docker and finds nothing to kill on the
host, or anything survives the kill, it says so and exits non-zero
instead of claiming success. The bare-metal path is unchanged apart from
verifying the SIGKILL actually worked, and no longer risks killing the
shell it was launched from.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:30:51 +04:00
kami 0272dc9d89 Record a suppressed care nudge instead of dropping it silently (#370)
Dropping a sev1-2 care nudge while you're away is right and still happens.
But it was a bare `continue`: no row, no log, so "she dropped it", "the gate
suppressed it" and "the rule never fired" all looked identical afterwards.

Adds a 'dropped' delivery status (migration #12 widens the CHECK constraint;
sqlite can't do that in place, so the table is rebuilt) and records the drop
as one delivery_attempts row plus a log line.

No nudges row for a drop: that table feeds the ignored_rate signal, and a
nudge nobody could see must not count as ignored.

TestVoiceNoSessionFallthroughLeavesOutboxTrail expected exactly one row for
sev1-2 when voice had no session. It now expects the voice failure plus the
drop, which is the point of the change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:27:47 +04:00
kami 2ad7635501 Merge the address-form eval check 2026-07-31 14:27:16 +04:00
kami 9949b309b1 Don't let a time word blind the third-person check
The check asks whether anyone else was named before "он". Time words
were not stoplisted, so "сегодня он не ел" read "сегодня" as the person
being talked about and passed — which is the recorded break with a word
in front of it, and nudges open with those words constantly.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:27:08 +04:00
kami a788ca3915 Label eval runs with the model the server actually loaded (#379)
The phrasing eval printed "llm (0.8B, ...)" no matter which gguf
llama-server had loaded, so two runs of two different models came out
named the same and were easy to mix up when comparing.

It now asks llama-server over /v1/models, same as the router eval
already did. The helper moved to internal/llm so both share it, and it
now errors instead of returning a blank name when the id field is
missing — an unreachable server gets labelled "unknown-model", never a
plausible-looking guess.

Both eval paths stay opt-in behind MAVEN_LLM_URL; no server needed for
go test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:25:28 +04:00
kami 62d47d28ac Add an eval check for formal and third-person address (#384)
The phrasing run produced two persona breaks that scored clean:
"Приходите… Жду вас" (formal plural) and "Он не ел 11 дней" (talks
about him instead of to him). She is feminine, he is male, and she
speaks to him informally, one to one.

The new `address` check flags the "вы" family, plural imperative
endings, and a third-person "он" with no other subject named earlier in
the message. Like `hisgender` it is a keyword/suffix heuristic, not a
parser, and it prints the word it tripped on so a false alarm is easy to
dismiss. Limits are written out in the comment.

Both recorded strings are pinned as unit tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:25:23 +04:00
kami e9ff2c4912 Never send the full nudge body off-box (#368)
The away sinks fell back to the whole Body when Summary was empty. ntfy and
telegram leave the box, and the 0.8B phraser drops fields regularly, so that
fallback could push full detail off the machine.

The dispatcher already strips detail from away sendables. This exports that
one rule as delivery.AwayMessage and has both sinks use it, so a sink can't
leak the body on its own either: empty Summary means a generic line plus the
rule name, never the body.

The two sink tests named TestSendFallsBackToBodyWhenSummaryEmpty asserted the
old, wrong behaviour, so they are rewritten to assert the generic line.
TestSendRejectsEmptyMessage is likewise replaced: an away message can no
longer be empty, so the sink has nothing left to reject.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
2026-07-31 14:23:54 +04:00
41 changed files with 2222 additions and 159 deletions
+15 -1
View File
@@ -82,9 +82,23 @@ workspace enforces that the Go and relabelling prompts remain identical.
## Non-goals (hard constraints) ## Non-goals (hard constraints)
Never phones home. Not a nag, not autonomous. Maven's persona is **feminine** — Russian Not a nag, not autonomous. Maven's persona is **feminine** — Russian
self-reference must use feminine forms (the user is male; see memory `maven-persona-gender`). self-reference must use feminine forms (the user is male; see memory `maven-persona-gender`).
**"Never phones home" is DEPRECATED** (owner's call, 2026-07-31). It used to be a hard
constraint and it is not one any more: a 0.8B — and a 1.7B — does not know enough to answer
world questions, so she needs to read external sources. What replaces it:
- **No telemetry, no cloud model, no third-party account.** That part never changes. Nothing
about Maven is reported to anyone, and inference stays on the box.
- **Local sources first.** Kiwix ZIMs on homesrv (Wikipedia, ifixit) before anything on the
network. Reading beats recalling for a small model, and a local read costs nothing.
- **External search is allowed and off unless configured**, like the weather and telegram
capabilities.
- **His notes and facts are never search input.** Looking up why the sky is blue and sending
his stored personal notes to an upstream engine are different acts. Only the utterance goes
out, never the persona block, history, or matched notes.
## Web UI conventions ## Web UI conventions
Server-rendered pages share `cmd/mavweb/static/ui.css` (served at `/ui.css`) and the `nav` Server-rendered pages share `cmd/mavweb/static/ui.css` (served at `/ui.css`) and the `nav`
+10 -4
View File
@@ -15,7 +15,8 @@
**Maven** — self-hosted personal assistant. Manages your day, acts on your **Maven** — self-hosted personal assistant. Manages your day, acts on your
homelab. One daemon on homesrv (always-on, not the workstation), multiple homelab. One daemon on homesrv (always-on, not the workstation), multiple
client surfaces. All local, never phones home. client surfaces. Inference and data stay on the box; she may READ external
sources (see Non-goals — "never phones home" is deprecated).
Primary name is "Maven", with feminine-gendered Russian self-reference Primary name is "Maven", with feminine-gendered Russian self-reference
("она", "меня", "помогла"). Clients may choose their own UI label. Consistent ("она", "меня", "помогла"). Clients may choose their own UI label. Consistent
@@ -35,8 +36,13 @@ Inside boundary — the ones that actually constrain the build:
she records. A confident wrong fact is worse than a known gap. she records. A confident wrong fact is worse than a known gap.
- **Not a nag** — she'd rather miss a nudge than be mutable. Shuts up when - **Not a nag** — she'd rather miss a nudge than be mutable. Shuts up when
uncertain. Load-bearing. uncertain. Load-bearing.
- **Not a stranger** — runs on your stuff, your model, your data. Never - **Not a stranger** — runs on your stuff, your model, your data. No
phones home. telemetry, no cloud model, no third-party account. She may READ external
sources to answer world questions (Kiwix first, then optional search); she
never reports anything about you to anyone, and your notes and facts are
never used as search input. **"Never phones home" as an absolute is
deprecated** — owner's call, 2026-07-31: a small model does not know enough
to be useful without reading.
- **Not a relationship** — mom-tone is a function that makes nudges land, not - **Not a relationship** — mom-tone is a function that makes nudges land, not
emotional company. Names the drift a warm small model falls into. emotional company. Names the drift a warm small model falls into.
@@ -458,7 +464,7 @@ decides *insistence*. Both are needed.
sev ≤ 2 drops on away, sev ≥ 3 holds: a missed water nudge is noise, a missed sev ≤ 2 drops on away, sev ≥ 3 holds: a missed water nudge is noise, a missed
backup failure isn't. Away-channels (ntfy/telegram) leave the box — the one backup failure isn't. Away-channels (ntfy/telegram) leave the box — the one
path that crosses "never phones home," through your own relay. **Minimal path that leaves the box for a person to see, through your own relay. **Minimal
body** — "disk low on homesrv," not detail; don't make notifications a body** — "disk low on homesrv," not detail; don't make notifications a
shoulder-surf exfil surface. shoulder-surf exfil surface.
+6 -3
View File
@@ -103,14 +103,17 @@ eval-router:
eval-recall: eval-recall:
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/ MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
# eval-phrasing -- score nudge phrasing (internal/phraser/eval). Verbose so the # eval-phrasing -- score nudge phrasing AND the conversational paths (chat,
# query, general knowledge) in internal/phraser/eval. Verbose so the
# report and every generated message land in the terminal. With no environment # report and every generated message land in the terminal. With no environment
# it scores the deterministic Stub only, which is what CI runs. Set # it scores the deterministic Stub only, which is what CI runs. Set
# MAVEN_LLM_URL to add the resident model: # MAVEN_LLM_URL to add the resident model:
# MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing # MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
# The model run is slow (minutes) -- the timeout is raised to match. # The model run is slow (minutes) -- the timeout is raised to match. It covers
# two fixtures now (15 nudges + 27 conversational cases, and the chat replies are
# the long ones), hence 90m rather than 40m.
eval-phrasing: eval-phrasing:
$(GO) test -v -count=1 -timeout 40m ./internal/phraser/eval/ $(GO) test -v -count=1 -timeout 90m ./internal/phraser/eval/
# eval-models — score ONE llama-server against the same fixture, for the # eval-models — score ONE llama-server against the same fixture, for the
# resident-model bake-off (#278, #250). Start a server with the gguf you want, # resident-model bake-off (#278, #250). Start a server with the gguf you want,
+29
View File
@@ -103,6 +103,35 @@ but a large part of the jump is that failure now degrades into Russian instead o
The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge The two remaining failures: one `"..."` recurrence (`routine-stretch`) and one meal nudge
that never says food. that never says food.
## Tried and reverted: an example-led nudge prompt (#393)
The idea was that a 0.8B copies examples better than it follows rules, so the nudge prompt
was rewritten to lead with five on-topic examples (water, break, pills, morning, service) and
the prose rules were compressed to pay for the tokens: 1190 chars down to 986.
It measured **worse**, three runs each side, same llama-server, same fixture:
| run | before | after |
|---|---|---|
| 1 | 12/15 (address 14) | 11/15 (address 13) |
| 2 | 13/15 (address 15) | 12/15 (address 15) |
| 3 | 14/15 (address 15) | 11/15 (address 12) |
`feminine` and `hisgender` were 15/15 on all six runs, so they measure nothing here. The
regression is all in `address`: 44/45 before, 40/45 after. Formal "вы"/"ваше" and plural
imperatives came back, and so did `"..."`.
Two likely causes, both about the same thing — **examples do not carry a prohibition**. The
old prompt spent a whole sentence on «говоришь на "ты", в единственном числе»; the new one
demoted that to one item in a long "никогда" list, and the model stopped obeying it. And
making the examples on-topic let their *wording* leak: a break case came back as
«Вы давно не пили воду. Выпей стакан.» — the water example, verbatim, in the wrong slot.
That is exactly the failure the laundry/laptop examples were chosen to avoid.
Change reverted. What survives is the measurement: a rule the model must obey needs its own
sentence, and examples must stay off-topic. Also note the before side alone spans 1214 of
15 — this fixture cannot resolve anything smaller than about three cases.
## Broken, found, not fixed ## Broken, found, not fixed
1. ~~**`checkFeminine` only catches half the constraint.**~~ **Fixed** (#381). It scanned for 1. ~~**`checkFeminine` only catches half the constraint.**~~ **Fixed** (#381). It scanned for
+4 -1
View File
@@ -90,4 +90,7 @@ later* is the worker + RAG.
4. **Deferred work** — larger reasoner, custom Piper voice and other expansions. 4. **Deferred work** — larger reasoner, custom Piper voice and other expansions.
## Non-goals (unchanged) ## Non-goals (unchanged)
Never phones home. Not a nag. Not autonomous. Feminine-gendered RU self-ref. Not a nag. Not autonomous. Feminine-gendered RU self-ref. No telemetry, no
cloud model, no third-party account — but she MAY read external sources to
answer world questions (Kiwix first, search optional). "Never phones home" as
an absolute is deprecated, owner's call 2026-07-31; see CLAUDE.md § Non-goals.
+150
View File
@@ -0,0 +1,150 @@
# Conversational phrasing eval — 31-07-2026
Every score measured tonight, on the three paths the nudge eval never touched:
chat, query-with-notes, and general knowledge.
**Short version: the plumbing got fixed and the score barely moved.** Grammar and
Russian prompts together took the composite from ~9 to ~14 of 27. Everything
still failing is the model not knowing things or not holding a constraint, and
prompting is out of levers. Settles the measurement half of Vikunja #395 / #398 /
#400.
## How to reproduce
```sh
# llama-server: -c 4096 -ngl 99 -t 6, model /mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf
MAVEN_LLM_URL=http://127.0.0.1:18099 no_proxy=127.0.0.1,localhost \
deps/go/go/bin/go test -count=1 -timeout 40m \
-run TestLLMTalkBaseline ./internal/phraser/eval/ -v
```
Three runs per configuration, always. The fixture is 27 cases, so one reply
changing moves the composite by 3.7 points — a single run cannot tell a real
change from sampling noise. This was learned the expensive way: an earlier claim
that "one nudge case fails every run" turned out to be three different cases
across three runs.
**Run the box otherwise idle.** See the contamination note at the bottom.
## Composite, per configuration
| config | overall /27 | chat /9 | query /9 | knowledge /9 | canned fallbacks |
|---|---|---|---|---|---|
| baseline, no grammar | 7, 12, 7 | 1, 1, 0 | 2, 4, 2 | 4, 7, 5 | 0, 0, 0 |
| + GBNF grammar (#398) | 14, 15, 8 | 1, 3, 0 | 5, 6, 3 | 8, 6, 5 | 0, 0, 0 |
| + Russian prompts (#400) | 11, 17, 15 | 1, 5, 3 | 5, 6, 8 | 5, 6, 4 | 0, 0, 0 |
| + truncation fix, 1000ch/768tok | 12, 13, 10 | 2, 2, 1 | 7, 7, 5 | 3, 4, 4 | 3, 3, 6 |
| + rebalanced, 600ch/1024tok | **void — contaminated** | | | | |
"Canned fallbacks" counts replies that came back as the hardcoded `"не знаю."`
or `"поговорили."`. It is not a check, it is a health signal: those strings mean
the phraser gave up, and the eval scores them as ordinary bad replies.
## Per-check
| check | no grammar | + grammar | + RU prompts | + truncation fix |
|---|---|---|---|---|
| nonempty | 27, 27, 27 | 27, 27, 27 | 27, 27, 27 | 27, 27, 27 |
| ellipsis | 20, 19, 23 | 27, 27, 27 | 27, 27, 27 | 27, 27, 27 |
| lang | 13, 16, 15 | 23, 26, 26 | 25, 26, 25 | 26, 27, 27 |
| feminine | — | — | 25, 24, 26 | 25, 25, 27 |
| address | — | — | 21, 22, 22 | 22, 21, 22 |
| ontopic | — | — | 17, 24, 18 | 17, 19, 14 |
`nonempty` reading 27/27 everywhere is not good news — it was a broken check.
It tested for a non-blank string, so replies of literally `{` and `"15-16"`
passed it. Fixed on `overnight/fix-truncation`; it needs a letter now.
## What each change actually bought
**GBNF grammar (#398) — the biggest single win.** Qwen3.5-0.8B writes
`Thinking Process:` as plain text with no tags, `stripThink` only handles
`</think>`, so the JSON never closed and the plain-text fallback shipped the
literal reasoning. `ellipsis` went 20→27 and `lang` 13→26. The router had been
using a grammar for ages; the phraser asking nicely in the prompt was the
oversight.
**Russian prompts (#400) — modest, plus a large latency win.** Chat 1.3→3.0
average, query 4.7→6.3, knowledge 6.3→5.0. All inside the run-to-run spread, so
"probably better on the paths it targeted, not provable in three runs". p50
latency dropped from ~11.5s to ~2.3s and that part is consistent across all
three runs — shorter prompts, and she stopped emitting English reasoning first.
**Truncation fix — necessary, and did not help the score.** Two real bugs
(replies of `{`, and a `nonempty` check that passed them), both fixed, and the
composite went nowhere. A complete rambling wrong answer fails the same checks a
truncated one did. Worth doing anyway: the daemon was shipping `{` to a
text-to-speech voice.
## The truncation bug, since the cause was counter-intuitive
The grammar's `string ::= ... {0,400}` rule was the cause, not the token cap.
Measured against Qwen3.5-0.8B at three caps — 256, 768 and 2048 — the reply came
back **exactly 400 characters every time, cut mid-word** (`"Нужно записать и,"`).
Then I raised the bound to 1000 while the cap was 768 tokens and made it worse:
Russian runs ~1.5 characters per token here, so generation died on the *token*
cap instead, mid-object, and the new guard correctly refused it and shipped
`"не знаю."` — 3, 3 and 6 fallbacks per run, from zero. **The two limits have to
agree.** 600 characters needs ~400 tokens; the cap is 1024.
## Where the remaining failures live
`address` is stuck at 21-22 of 27 and `ontopic` at 14-19. Both resist prompting.
**The prompt now explicitly forbids exactly what she does.** It says never "вы",
use the singular — and she writes `вашей`, `подождите`, `делаете`, `хотите`,
`напишите`. Telling a 0.8B "never do X" does not work. Same for
`feminine`: `я готов`, `я понял`, `я нашел`, `я заметил`, `я сказал`.
**Some of `ontopic` is the fixture, not the model.** `chat-how-are-you` got
`"Привет! Я здесь, чтобы поговорить. Как дела сегодня?"` — a fine reply that
fails because `want_any` is `[норм, хорош, порядк, тут, работ]`. It fails in
every run, so it inflates the count. The `ontopic` column currently measures the
fixture as much as the model. Not fixed yet, deliberately: changing it would
break comparability with the runs above.
**Two replies worth reading, because they are not fixable by prompting:**
- Thunder and lightning: *"Скорость молнии — 8-10 тысяч километров в секунду, но
звук — 300 метров в секунду, что делает молнию громче."* Confidently wrong,
and it concludes lightning is *louder* rather than sound being *slower*.
- "расскажи обо мне": *"Ты — прекрасное существо, с душой и вниманием… Спасибо за
твою улыбку… О тебе — заповедь любви."* Sycophantic filler, zero information,
and precisely the "not a relationship" non-goal.
- Boiling an egg: `"15-16"` one run, `"1"` another. No unit, wrong number.
The first argues for reading instead of recalling (#403 — Kiwix retrieval scores
8/8 on the same questions given English keywords). The second and third argue
for templates on the paths where correctness matters (#392).
## Contamination note — how the last row got voided
I started the query-rewrite agent against the same llama-server the sweep was
using, and assumed contention would only affect latency. It did not. The
knowledge path collapsed to 0 of 9 with eight canned `"не знаю."` replies, p95
tripled to 23.7s, and **the report still said "0 errors"**.
That is Vikunja #397, and it is worse than filed: a merely *busy* server
produces a clean-looking report with a third of the fixture silently answering
`"не знаю."`. `PhraseChat` and `PhraseQuery` swallow every failure and return a
hardcoded string, so infrastructure trouble is indistinguishable from bad
phrasing in the score. The talk test guards the *start* and *end* of a run with
a model check, which catches a dead server but not a loaded one.
**Until #397 is fixed, treat any run made on a busy box as void.**
## Next
- Re-run 600ch/1024tok clean, to fill the void row.
- Score `Qwen3.5-2B-UD-Q4_K_XL` (already at `/mnt/hdd1/llms/qwen3.5/`, never
measured) on this fixture and the router fixture. Not the 4B — too big for
this box, owner's call.
- Newer sub-500M candidates (LFM2.5 200M/300M) are worth a run for routing.
Note `MODEL-BAKEOFF-31-07-2026.md` found LFM2.5-**1.2B** worse than
Qwen3.5-0.8B at Russian routing and 2.4× slower — but those are a different,
older generation, so that result does not predict the small ones.
- Fix `chat-how-are-you`'s `want_any`, and re-baseline once, so `ontopic`
measures the model.
- #397 first if anything, since it decides whether any of the above is
trustworthy.
+39 -21
View File
@@ -58,6 +58,7 @@ import (
"github.com/kami/maven/internal/delivery/telegramsink" "github.com/kami/maven/internal/delivery/telegramsink"
"github.com/kami/maven/internal/ipc" "github.com/kami/maven/internal/ipc"
"github.com/kami/maven/internal/loop" "github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser" "github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/store" "github.com/kami/maven/internal/store"
"github.com/kami/maven/internal/webauthn" "github.com/kami/maven/internal/webauthn"
@@ -271,13 +272,13 @@ func run(args []string) error {
phr = phraser.NewStub() phr = phraser.NewStub()
if cfg.Phraser != nil { if cfg.Phraser != nil {
pc := phraser.Config{ pc := phraser.Config{
ModelPath: cfg.Phraser.ModelPath, ModelPath: cfg.Phraser.ModelPath,
BinPath: cfg.Phraser.BinPath, BinPath: cfg.Phraser.BinPath,
Listen: cfg.Phraser.Listen, Listen: cfg.Phraser.Listen,
NGpuLayers: cfg.Phraser.NGpuLayers, NGpuLayers: cfg.Phraser.NGpuLayers,
NCtx: cfg.Phraser.NCtx, NCtx: cfg.Phraser.NCtx,
Timeout: time.Duration(cfg.Phraser.Timeout), Timeout: time.Duration(cfg.Phraser.Timeout),
Persona: personaFromCfg(cfg), ContextBlock: contextBlockFn(cfg, time.Now),
} }
if pc.BinPath == "" { if pc.BinPath == "" {
pc.BinPath = "llama-server" pc.BinPath = "llama-server"
@@ -441,13 +442,13 @@ func run(args []string) error {
phr = phraser.NewStub() phr = phraser.NewStub()
if cfg.Phraser != nil { if cfg.Phraser != nil {
pc := phraser.Config{ pc := phraser.Config{
ModelPath: cfg.Phraser.ModelPath, ModelPath: cfg.Phraser.ModelPath,
BinPath: cfg.Phraser.BinPath, BinPath: cfg.Phraser.BinPath,
Listen: cfg.Phraser.Listen, Listen: cfg.Phraser.Listen,
NGpuLayers: cfg.Phraser.NGpuLayers, NGpuLayers: cfg.Phraser.NGpuLayers,
NCtx: cfg.Phraser.NCtx, NCtx: cfg.Phraser.NCtx,
Timeout: time.Duration(cfg.Phraser.Timeout), Timeout: time.Duration(cfg.Phraser.Timeout),
Persona: personaFromCfg(cfg), ContextBlock: contextBlockFn(cfg, time.Now),
} }
if pc.BinPath == "" { if pc.BinPath == "" {
pc.BinPath = "llama-server" pc.BinPath = "llama-server"
@@ -601,12 +602,29 @@ func run(args []string) error {
return nil return nil
} }
// personaFromCfg extracts the voice persona from the config, or returns "" // personaFacts reads the optional, deployment-specific facts (his name, his
// when voice isn't configured. Used to pass a character prompt into the // city, the free-text persona string) out of the config. Everything here may
// LLM phraser without requiring voice to be enabled. // be empty — the context block is correct without any of it.
func personaFromCfg(cfg *config.Config) string { func personaFacts(cfg *config.Config) persona.Facts {
if cfg.Voice != nil { f := persona.Facts{
return cfg.Voice.Persona // Telegram lives outside the voice block, so it counts either way.
Telegram: cfg.Telegram != nil && cfg.Telegram.BotToken != "" && cfg.Telegram.ChatID != "",
} }
return "" if cfg.Voice == nil {
return f
}
f.OwnerName = cfg.Voice.OwnerName
f.City = cfg.Voice.City
f.Static = cfg.Voice.Persona
// Same test wireVoice uses to pick the real provider over the stub.
f.Weather = cfg.Voice.Weather != nil && cfg.Voice.Weather.Provider == "open-meteo"
f.Tools = len(cfg.Voice.Tools) > 0
return f
}
// contextBlockFn returns the per-turn renderer of the shared context block.
// Per turn, not once at startup, because the block states the current time.
func contextBlockFn(cfg *config.Config, now func() time.Time) func() string {
f := personaFacts(cfg)
return func() string { return f.Block(now()) }
} }
+9 -4
View File
@@ -7,6 +7,7 @@ import (
"time" "time"
"github.com/kami/maven/internal/llm" "github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/router" "github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/voice" "github.com/kami/maven/internal/voice"
) )
@@ -23,13 +24,17 @@ type completer interface {
type llmReplier struct { type llmReplier struct {
c completer c completer
stub *voice.StubReplier stub *voice.StubReplier
// block renders the shared context block per turn (who he is, the time).
// nil ⇒ the prompt stands alone.
block func() string
} }
func newLLMReplier(c completer) *llmReplier { func newLLMReplier(c completer, block func() string) *llmReplier {
return &llmReplier{c: c, stub: voice.NewStubReplier()} return &llmReplier{c: c, stub: voice.NewStubReplier(), block: block}
} }
const replySystem = `Ты Maven, домашняя ассистентка (о себе в женском роде). Подтверди действие РОВНО ОДНИМ коротким предложением (120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused). const replySystem = `Ты Maven, домашняя ассистентка (о себе в женском роде). Владелец мужчина, говоришь с ним на "ты", в единственном числе; никогда не "вы"/"ваш" и не "он"/"его". Подтверди действие РОВНО ОДНИМ коротким предложением (120 символов), тепло и по-русски. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"} Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"}
Никогда не пиши "..." в поле response.` Никогда не пиши "..." в поле response.`
@@ -40,7 +45,7 @@ func (r *llmReplier) Reply(d router.Decision) string {
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel() defer cancel()
out, err := r.c.Complete(ctx, llm.Req{ out, err := r.c.Complete(ctx, llm.Req{
System: replySystem, System: persona.Prepend(r.block, replySystem),
User: replyContext(d), User: replyContext(d),
MaxTokens: 512, MaxTokens: 512,
RepeatPenalty: 1.3, RepeatPenalty: 1.3,
+5 -5
View File
@@ -17,7 +17,7 @@ type mockCompleter struct {
func (m mockCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return m.out, m.err } func (m mockCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return m.out, m.err }
func TestLLMReplierReturnsLLMReply(t *testing.T) { func TestLLMReplierReturnsLLMReply(t *testing.T) {
r := newLLMReplier(mockCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}) r := newLLMReplier(mockCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}, nil)
got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}}) got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
if got != "записала, кофе закончился" { if got != "записала, кофе закончился" {
t.Errorf("got %q, want %q", got, "записала, кофе закончился") t.Errorf("got %q, want %q", got, "записала, кофе закончился")
@@ -25,7 +25,7 @@ func TestLLMReplierReturnsLLMReply(t *testing.T) {
} }
func TestLLMReplierFallsBackToPlainText(t *testing.T) { func TestLLMReplierFallsBackToPlainText(t *testing.T) {
r := newLLMReplier(mockCompleter{out: "записала, кофе закончился"}) r := newLLMReplier(mockCompleter{out: "записала, кофе закончился"}, nil)
got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}}) got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
if got != "записала, кофе закончился" { if got != "записала, кофе закончился" {
t.Errorf("got %q, want %q", got, "записала, кофе закончился") t.Errorf("got %q, want %q", got, "записала, кофе закончился")
@@ -33,7 +33,7 @@ func TestLLMReplierFallsBackToPlainText(t *testing.T) {
} }
func TestLLMReplierFallsBackToStubOnError(t *testing.T) { func TestLLMReplierFallsBackToStubOnError(t *testing.T) {
r := newLLMReplier(mockCompleter{err: errTestLLMDown}) r := newLLMReplier(mockCompleter{err: errTestLLMDown}, nil)
noteDec := router.Decision{Intent: router.IntentNote} noteDec := router.Decision{Intent: router.IntentNote}
got := r.Reply(noteDec) got := r.Reply(noteDec)
want := voice.NewStubReplier().Reply(noteDec) want := voice.NewStubReplier().Reply(noteDec)
@@ -43,7 +43,7 @@ func TestLLMReplierFallsBackToStubOnError(t *testing.T) {
} }
func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) { func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) {
r := newLLMReplier(mockCompleter{out: ""}) r := newLLMReplier(mockCompleter{out: ""}, nil)
noteDec := router.Decision{Intent: router.IntentNote} noteDec := router.Decision{Intent: router.IntentNote}
got := r.Reply(noteDec) got := r.Reply(noteDec)
want := voice.NewStubReplier().Reply(noteDec) want := voice.NewStubReplier().Reply(noteDec)
@@ -53,7 +53,7 @@ func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) {
} }
func TestLLMReplierClarifyUsesStub(t *testing.T) { func TestLLMReplierClarifyUsesStub(t *testing.T) {
r := newLLMReplier(mockCompleter{out: "я всё поняла"}) r := newLLMReplier(mockCompleter{out: "я всё поняла"}, nil)
clarifyDec := router.Decision{Clarify: true} clarifyDec := router.Decision{Clarify: true}
got := r.Reply(clarifyDec) got := r.Reply(clarifyDec)
want := voice.NewStubReplier().Reply(clarifyDec) want := voice.NewStubReplier().Reply(clarifyDec)
+1 -1
View File
@@ -246,7 +246,7 @@ func wireVoice(cfg *config.Config, coreAPI ipc.CoreAPI, phr phraser.Phraser, mem
// ----- replier (LLM-backed when the engine is on, Stub floor otherwise) ----- // ----- replier (LLM-backed when the engine is on, Stub floor otherwise) -----
replier := voice.Replier(voice.NewStubReplier()) replier := voice.Replier(voice.NewStubReplier())
if llmClient != nil { if llmClient != nil {
replier = newLLMReplier(llmClient) replier = newLLMReplier(llmClient, contextBlockFn(cfg, time.Now))
} }
// ----- the handler (the reactive path; closes over stt / tts / router / coreAPI / memory) ----- // ----- the handler (the reactive path; closes over stt / tts / router / coreAPI / memory) -----
+7
View File
@@ -302,6 +302,13 @@ type VoiceConfig struct {
// Russian self-reference). Example: "Be formal and answer in English only." // Russian self-reference). Example: "Be formal and answer in English only."
Persona string `json:"persona,omitempty"` Persona string `json:"persona,omitempty"`
// OwnerName / City — optional facts about the owner, added to the shared
// context block (internal/persona). Empty is fine: the block still states
// who he is grammatically (a man, addressed as "ты") and the current time.
// Nothing about correct behaviour may depend on these being filled in.
OwnerName string `json:"owner_name,omitempty"`
City string `json:"city,omitempty"`
// Weather — the weather provider config. nil ⇒ the daemon wires // Weather — the weather provider config. nil ⇒ the daemon wires
// the stub provider (returns ErrNotConfigured — "погода не настроена"). // the stub provider (returns ErrNotConfigured — "погода не настроена").
// Set provider to "open-meteo" to use the keyless Open-Meteo API. // Set provider to "open-meteo" to use the keyless Open-Meteo API.
+21 -3
View File
@@ -137,9 +137,10 @@ func NewDispatcher(cfg Config) *Dispatcher {
// picks for (severity, presence), sends via the matching sink, and records // picks for (severity, presence), sends via the matching sink, and records
// one nudge row per successful send. returns the dispatches (one per channel). // one nudge row per successful send. returns the dispatches (one per channel).
// //
// a Drop channel = no send, no record (the nudge was suppressed by routing, // a Drop channel = no send (the nudge was suppressed by routing, not by a
// not by a failure — "a missed water nudge is noise"). a nil sink = channel // failure — "a missed water nudge is noise"), but it does leave a 'dropped'
// not wired, skip silently. a send error stops the dispatch and returns what // outbox row so the suppression is visible. a nil sink = channel not wired,
// skip silently. a send error stops the dispatch and returns what
// got through — the daemon decides whether to retry. // got through — the daemon decides whether to retry.
func (d *Dispatcher) DispatchNudge(ctx context.Context, pn PhrasedNudge, now time.Time) ([]Dispatch, error) { func (d *Dispatcher) DispatchNudge(ctx context.Context, pn PhrasedNudge, now time.Time) ([]Dispatch, error) {
c := pn.Candidate c := pn.Candidate
@@ -148,6 +149,16 @@ func (d *Dispatcher) DispatchNudge(ctx context.Context, pn PhrasedNudge, now tim
for i := 0; i < len(channels); i++ { for i := 0; i < len(channels); i++ {
ch := channels[i] ch := channels[i]
if ch == ChannelDrop { if ch == ChannelDrop {
// the routing table suppressed this nudge on purpose (a care nudge
// while you're away is noise). that stays — but it must not be
// invisible, or "she dropped it" and "the rule never fired" look
// the same afterwards. no nudges row: that table feeds the
// ignored_rate signal, and a nudge nobody could see must not
// count as ignored.
id := d.beginOutbox(ctx, "nudge", c.Rule.Name, 0, ch, pn.Summary, now)
d.completeOutbox(ctx, id, store.DeliveryDropped, now)
log.Printf("dispatcher: dropped %s (sev%d, presence=%s) — routing table suppressed it",
c.Rule.Name, c.Severity, c.State.Presence)
continue continue
} }
s := Sendable{ s := Sendable{
@@ -396,6 +407,13 @@ func messageForChannel(s Sendable) string {
if !isAway(s.Channel) { if !isAway(s.Channel) {
return s.Body return s.Body
} }
return AwayMessage(s)
}
// AwayMessage — the only text an off-box channel may ever carry. Exported so
// the away sinks share this one rule instead of each inventing a fallback: the
// summary if we have one, otherwise a fixed generic line. Never the body.
func AwayMessage(s Sendable) string {
if s.Summary != "" { if s.Summary != "" {
return s.Summary return s.Summary
} }
+6 -4
View File
@@ -37,10 +37,12 @@ func TestVoiceNoSessionFallthroughLeavesOutboxTrail(t *testing.T) {
[]string{"voice", "ntfy"}, []string{store.DeliveryFailed, store.DeliverySent}}, []string{"voice", "ntfy"}, []string{store.DeliveryFailed, store.DeliverySent}},
{"sev4 falls through to telegram", loop.Sev4, {"sev4 falls through to telegram", loop.Sev4,
[]string{"voice", "telegram"}, []string{store.DeliveryFailed, store.DeliverySent}}, []string{"voice", "telegram"}, []string{store.DeliveryFailed, store.DeliverySent}},
{"sev1 does not fall through", loop.Sev1, // care severities still don't reach an away channel; since #370 the
[]string{"voice"}, []string{store.DeliveryFailed}}, // drop itself is a visible row instead of nothing.
{"sev2 does not fall through", loop.Sev2, {"sev1 drops instead of falling through", loop.Sev1,
[]string{"voice"}, []string{store.DeliveryFailed}}, []string{"voice", "drop"}, []string{store.DeliveryFailed, store.DeliveryDropped}},
{"sev2 drops instead of falling through", loop.Sev2,
[]string{"voice", "drop"}, []string{store.DeliveryFailed, store.DeliveryDropped}},
} }
for _, c := range cases { for _, c := range cases {
t.Run(c.name, func(t *testing.T) { t.Run(c.name, func(t *testing.T) {
+9 -13
View File
@@ -2,10 +2,10 @@
// //
// ntfy is the away-channel for sev3 (ops soft) nudges, sev4 (ops hard) // ntfy is the away-channel for sev3 (ops soft) nudges, sev4 (ops hard)
// nudges when present (alongside voice), and reminders when away. the // nudges when present (alongside voice), and reminders when away. the
// message body is the Sendable's Summary — the minimal-body rule from the // message body is delivery.AwayMessage — the minimal-body rule from the
// spec ("disk low on homesrv," not detail; no shoulder-surf exfil through // spec ("disk low on homesrv," not detail; no shoulder-surf exfil through
// the relay). voice gets Body; away channels get Summary, enforced at the // the relay). the dispatcher already strips detail off away sendables; the
// sink so a phraser bug can't exfil. // sink uses the same helper so it can't leak the body on its own either.
// //
// ntfy runs locally (docker, 127.0.0.1:8085, deny-all auth). maven publishes // ntfy runs locally (docker, 127.0.0.1:8085, deny-all auth). maven publishes
// with a dedicated user (write-only to maven-* topics) — the credential is a // with a dedicated user (write-only to maven-* topics) — the credential is a
@@ -69,18 +69,14 @@ func New(cfg Config) (*Sink, error) {
}, nil }, nil
} }
// Send publishes one notification to ntfy. the body is the Sendable's Summary // Send publishes one notification to ntfy. the body is the minimal away
// (minimal body); Title is "maven" (consistent sender identity on the lock // message (never the full body); Title is "maven" (consistent sender identity
// screen — the content is in the body). Priority maps from severity/kind so // on the lock screen — the content is in the body). Priority maps from severity/kind so
// the phone client can ring differently for an alarm vs a soft ops nudge. // the phone client can ring differently for an alarm vs a soft ops nudge.
func (s *Sink) Send(ctx context.Context, d delivery.Sendable) error { func (s *Sink) Send(ctx context.Context, d delivery.Sendable) error {
body := d.Summary // never fall back to d.Body: ntfy leaves the box, so an empty summary gets
if body == "" { // a generic line instead of the full detail.
body = d.Body // terse full message beats no message body := delivery.AwayMessage(d)
}
if body == "" {
return fmt.Errorf("ntfysink: empty message for %s", d.Channel)
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.topicURL(), strings.NewReader(body)) req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.topicURL(), strings.NewReader(body))
if err != nil { if err != nil {
+16 -9
View File
@@ -147,9 +147,9 @@ func TestSendBodyIsSummaryNotFullBody(t *testing.T) {
} }
} }
func TestSendFallsBackToBodyWhenSummaryEmpty(t *testing.T) { func TestSendNeverSendsTheBodyWhenSummaryEmpty(t *testing.T) {
// a terse full message is better than no message; the phraser should // #368: this used to fall back to the full body. ntfy leaves the box, so
// produce a summary for away-bound severities, but don't silently drop. // an empty summary gets a fixed generic line plus the rule name instead.
rs := newRecordingServer(t, 200, "") rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler()) srv := httptest.NewServer(rs.handler())
defer srv.Close() defer srv.Close()
@@ -160,12 +160,15 @@ func TestSendFallsBackToBodyWhenSummaryEmpty(t *testing.T) {
t.Fatalf("Send: %v", err) t.Fatalf("Send: %v", err)
} }
_, _, body, _, _, _ := rs.snapshot() _, _, body, _, _, _ := rs.snapshot()
if body != s.Body { want := delivery.GenericAwayMessage + ": service_down"
t.Fatalf("fallback body: want %q, got %q", s.Body, body) if body != want {
t.Fatalf("body: want %q, got %q", want, body)
} }
} }
func TestSendRejectsEmptyMessage(t *testing.T) { func TestSendNeverSendsAnEmptyMessage(t *testing.T) {
// with nothing at all to say we still send the generic line — an away
// channel can never carry detail, but it also never goes out blank.
rs := newRecordingServer(t, 200, "") rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler()) srv := httptest.NewServer(rs.handler())
defer srv.Close() defer srv.Close()
@@ -173,9 +176,13 @@ func TestSendRejectsEmptyMessage(t *testing.T) {
sink, _ := New(Config{BaseURL: srv.URL, Topic: "maven"}) sink, _ := New(Config{BaseURL: srv.URL, Topic: "maven"})
s := nudgeSendable(loop.Sev3, "") s := nudgeSendable(loop.Sev3, "")
s.Body = "" s.Body = ""
err := sink.Send(context.Background(), s) s.RuleName = ""
if err == nil { if err := sink.Send(context.Background(), s); err != nil {
t.Fatal("want error for empty message") t.Fatalf("Send: %v", err)
}
_, _, body, _, _, _ := rs.snapshot()
if body != delivery.GenericAwayMessage {
t.Fatalf("body: want %q, got %q", delivery.GenericAwayMessage, body)
} }
} }
+2 -3
View File
@@ -208,10 +208,9 @@ func TestAwayChannelsGetMinimalBody(t *testing.T) {
// TestCareAwayDropIsRecorded — DESIGN.md's drop is a decision ("a missed water // TestCareAwayDropIsRecorded — DESIGN.md's drop is a decision ("a missed water
// nudge is noise, a missed backup failure isn't"), so it should be visible // nudge is noise, a missed backup failure isn't"), so it should be visible
// rather than vanish. Today drop is a bare `continue`: no nudge row, no outbox // rather than vanish. Today drop is a bare `continue`: no nudge row, no outbox
// attempt, no log — nothing an operator can see afterwards. // attempt, no log — nothing an operator can see afterwards. now it leaves a
// 'dropped' outbox row.
func TestCareAwayDropIsRecorded(t *testing.T) { func TestCareAwayDropIsRecorded(t *testing.T) {
t.Skip("not implemented: dispatcher.go:149-151 skips a Drop channel with no record; there is no 'dropped' outcome in store/delivery.go:16-21")
ob := &fakeOutbox{} ob := &fakeOutbox{}
d := NewDispatcher(Config{Voice: &fakeSink{}, Nudges: &fakeNudgeRecorder{}, Outbox: ob}) d := NewDispatcher(Config{Voice: &fakeSink{}, Nudges: &fakeNudgeRecorder{}, Outbox: ob})
+12 -16
View File
@@ -2,11 +2,12 @@
// //
// telegram is the away-channel for sev4 (ops hard) nudges — "disk-fire alarm // telegram is the away-channel for sev4 (ops hard) nudges — "disk-fire alarm
// at 2am routes to telegram, repeat til ack." the message body is the // at 2am routes to telegram, repeat til ack." the message body is the
// Sendable's Summary — the minimal-body rule from the spec ("disk low on // delivery.AwayMessage — the minimal-body rule from the spec ("disk low on
// homesrv," not detail; no shoulder-surf exfil through the relay). voice gets // homesrv," not detail; no shoulder-surf exfil through the relay). the
// Body; away channels get Summary, enforced at the sink so a phraser bug can't // dispatcher already strips detail off away sendables; the sink uses the same
// exfil. additionally, protect_content=true is passed on every send so the // helper so it can't leak the body on its own either. additionally,
// message can't be forwarded out of the chat — locks the minimal body further. // protect_content=true is passed on every send so the message can't be
// forwarded out of the chat — locks the minimal body further.
// //
// telegram's bot API is region-restricted for this homesrv — direct egress to // telegram's bot API is region-restricted for this homesrv — direct egress to
// api.telegram.org is unreliable. the spec's "away channels leave the box — // api.telegram.org is unreliable. the spec's "away channels leave the box —
@@ -140,18 +141,13 @@ type telegramResp struct {
} }
// Send publishes one message to the configured telegram chat. the body is the // Send publishes one message to the configured telegram chat. the body is the
// Sendable's Summary (minimal body); empty Summary falls back to Body (terse // minimal away message (never the full body). protect_content=true so even
// full message beats no message). protect_content=true so a phraser bug (Body // that can't be forwarded onward by the user or a chat observer — locks the
// leaking detail through Summary) can't be forwarded onward by the user or a // minimal-body rule at the channel's own last mile.
// chat observer — locks the minimal-body rule at the channel's own last mile.
func (s *Sink) Send(ctx context.Context, d delivery.Sendable) error { func (s *Sink) Send(ctx context.Context, d delivery.Sendable) error {
body := d.Summary // never fall back to d.Body: telegram leaves the box, so an empty summary
if body == "" { // gets a generic line instead of the full detail.
body = d.Body body := delivery.AwayMessage(d)
}
if body == "" {
return fmt.Errorf("telegramsink: empty message for %s", d.Channel)
}
payload := sendMessageReq{ payload := sendMessageReq{
ChatID: s.cfg.ChatID, ChatID: s.cfg.ChatID,
@@ -173,9 +173,9 @@ func TestSendBodyIsSummaryNotFullBody(t *testing.T) {
} }
} }
func TestSendFallsBackToBodyWhenSummaryEmpty(t *testing.T) { func TestSendNeverSendsTheBodyWhenSummaryEmpty(t *testing.T) {
// terse full message beats none; the phraser should produce a summary for // #368: this used to fall back to the full body. telegram leaves the box,
// away-bound severities, but don't silently drop. // so an empty summary gets a fixed generic line plus the rule name.
rs := newRecordingServer(t, 200, "") rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler()) srv := httptest.NewServer(rs.handler())
defer srv.Close() defer srv.Close()
@@ -188,12 +188,14 @@ func TestSendFallsBackToBodyWhenSummaryEmpty(t *testing.T) {
_, _, body, _, _ := rs.snapshot() _, _, body, _, _ := rs.snapshot()
var req sendMessageReq var req sendMessageReq
_ = json.Unmarshal([]byte(body), &req) _ = json.Unmarshal([]byte(body), &req)
if req.Text != s.Body { want := delivery.GenericAwayMessage + ": service_down"
t.Fatalf("fallback text: want %q, got %q", s.Body, req.Text) if req.Text != want {
t.Fatalf("text: want %q, got %q", want, req.Text)
} }
} }
func TestSendRejectsEmptyMessage(t *testing.T) { func TestSendNeverSendsAnEmptyMessage(t *testing.T) {
// with nothing at all to say we still send the generic line.
rs := newRecordingServer(t, 200, "") rs := newRecordingServer(t, 200, "")
srv := httptest.NewServer(rs.handler()) srv := httptest.NewServer(rs.handler())
defer srv.Close() defer srv.Close()
@@ -201,9 +203,15 @@ func TestSendRejectsEmptyMessage(t *testing.T) {
sink, _ := New(sinkCfg(srv.URL)) sink, _ := New(sinkCfg(srv.URL))
s := nudgeSendable(loop.Sev4, "") s := nudgeSendable(loop.Sev4, "")
s.Body = "" s.Body = ""
err := sink.Send(context.Background(), s) s.RuleName = ""
if err == nil { if err := sink.Send(context.Background(), s); err != nil {
t.Fatal("want error for empty message") t.Fatalf("Send: %v", err)
}
_, _, body, _, _ := rs.snapshot()
var req sendMessageReq
_ = json.Unmarshal([]byte(body), &req)
if req.Text != delivery.GenericAwayMessage {
t.Fatalf("text: want %q, got %q", delivery.GenericAwayMessage, req.Text)
} }
} }
@@ -1,4 +1,4 @@
package eval package llm
import ( import (
"context" "context"
@@ -6,8 +6,20 @@ import (
"fmt" "fmt"
"net/http" "net/http"
"strings" "strings"
"time"
) )
// UnknownModel is the label to print when the server would not say what it has
// loaded. Deliberately ugly: an honest "unknown" is fine, a plausible-looking
// but wrong model name is the bug this whole file exists to prevent.
const UnknownModel = "unknown-model"
// llama-server is local, so never send this through a proxy: this box's
// http_proxy answers 503 for loopback, which would look like "server won't say
// which model it has" when the server is right there and fine.
// A Transport with no Proxy set bypasses http_proxy entirely.
var modelHTTP = &http.Client{Timeout: 10 * time.Second, Transport: &http.Transport{}}
// ModelID asks llama-server which model it has loaded, so a scoring run can // ModelID asks llama-server which model it has loaded, so a scoring run can
// label itself. Without this a bake-off between two models produces two tables // label itself. Without this a bake-off between two models produces two tables
// that look identical, and the operator has to remember which server was up. // that look identical, and the operator has to remember which server was up.
@@ -19,7 +31,7 @@ func ModelID(ctx context.Context, base string) (string, error) {
if err != nil { if err != nil {
return "", err return "", err
} }
resp, err := http.DefaultClient.Do(req) resp, err := modelHTTP.Do(req)
if err != nil { if err != nil {
return "", err return "", err
} }
@@ -38,14 +50,21 @@ func ModelID(ctx context.Context, base string) (string, error) {
if len(out.Data) == 0 { if len(out.Data) == 0 {
return "", fmt.Errorf("models: empty list") return "", fmt.Errorf("models: empty list")
} }
return shortModelID(out.Data[0].ID), nil short := shortModelID(out.Data[0].ID)
if short == "" {
// Server answered but the id field was missing or blank. Say so
// instead of handing back an empty label that reads as a real name.
return "", fmt.Errorf("models: no id in response")
}
return short, nil
} }
// shortModelID trims the path and the .gguf suffix — llama-server reports the // shortModelID trims the path and the .gguf suffix — llama-server reports the
// file name it was started with, which is too long for a table header. // file name it was started with, which is too long for a table header.
func shortModelID(id string) string { func shortModelID(id string) string {
id = strings.TrimSpace(id)
if i := strings.LastIndexAny(id, "/\\"); i >= 0 { if i := strings.LastIndexAny(id, "/\\"); i >= 0 {
id = id[i+1:] id = id[i+1:]
} }
return strings.TrimSuffix(id, ".gguf") return strings.TrimSpace(strings.TrimSuffix(id, ".gguf"))
} }
+64
View File
@@ -0,0 +1,64 @@
package llm
import (
"context"
"net/http"
"net/http/httptest"
"testing"
)
// The point of these tests: a wrong-but-plausible model label is the bug, so
// every path that cannot learn the real name must return an error instead of a
// guess. No llama-server needed — a stub server stands in.
func TestModelID(t *testing.T) {
cases := []struct {
name string
body string
code int
want string // "" ⇒ expect an error
}{
{"full path", `{"data":[{"id":"/mnt/hdd1/llms/qwen3.5/Qwen3.5-0.8B.Q4_K_M.gguf"}]}`, 200, "Qwen3.5-0.8B.Q4_K_M"},
{"bare name", `{"data":[{"id":"LFM2.5-1.2B"}]}`, 200, "LFM2.5-1.2B"},
{"empty list", `{"data":[]}`, 200, ""},
{"id missing", `{"data":[{}]}`, 200, ""},
{"id blank", `{"data":[{"id":" "}]}`, 200, ""},
{"server error", `nope`, 500, ""},
{"not json", `<html>`, 200, ""},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/models" {
t.Errorf("asked for %s, want /v1/models", r.URL.Path)
}
w.WriteHeader(c.code)
_, _ = w.Write([]byte(c.body))
}))
defer srv.Close()
got, err := ModelID(context.Background(), srv.URL+"/")
if c.want == "" {
if err == nil {
t.Fatalf("want an error, got label %q", got)
}
return
}
if err != nil {
t.Fatalf("ModelID: %v", err)
}
if got != c.want {
t.Errorf("got %q, want %q", got, c.want)
}
})
}
}
func TestModelIDUnreachable(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {}))
url := srv.URL
srv.Close() // nothing listening now
if got, err := ModelID(context.Background(), url); err == nil {
t.Fatalf("want an error from a dead server, got label %q", got)
}
}
+138
View File
@@ -0,0 +1,138 @@
// Package persona builds the one shared context block that goes in front of
// every LLM system prompt: who the owner is, how to address him, and what
// time it is right now.
//
// Why one block and not a line pasted into each prompt: there are five
// prompts (nudges, action replies, chat, note queries, general knowledge) and
// the "address him as ты" rule had only reached two of them. Five copies drift.
// One block cannot.
//
// The rules here are defaults in code, not config. Maven is feminine and the
// owner is a man addressed informally — that is a hard constraint of the
// product, so it must hold with an empty config file. Config only ADDS
// optional facts (his name, his city).
package persona
import (
"fmt"
"strings"
"time"
)
// Facts — the optional, deployment-specific half of the block. All fields may
// be empty; the block is still correct and useful without them.
type Facts struct {
OwnerName string // his name, e.g. "Ками"
City string // where he is, e.g. "Москва"
Static string // the free-text `persona` config string, appended verbatim
// The two config-gated capabilities. They are listed only when this
// deployment actually has them, because a capability she names and cannot
// do is worse than one she never mentions.
Weather bool // an open-meteo provider is configured
Telegram bool // a telegram bot token + chat id are configured
Tools bool // at least one shell act is on the allowlist
}
var ruWeekdays = [...]string{"воскресенье", "понедельник", "вторник", "среда", "четверг", "пятница", "суббота"}
var ruMonths = [...]string{
"января", "февраля", "марта", "апреля", "мая", "июня",
"июля", "августа", "сентября", "октября", "ноября", "декабря",
}
// Block renders the context block for one turn. Russian even in front of the
// English prompts: the rules it states are Russian grammar (ты/тебя, feminine
// verbs), and a Russian rule reads best stated in Russian.
//
// Keep it short. It ships on every turn to a 0.8B on laptop CPU, so every
// line here is latency.
func (f Facts) Block(now time.Time) string {
var b strings.Builder
b.WriteString("Ты — Maven, домашняя ассистентка. О себе говоришь в женском роде: \"я записала\", \"я проверила\".\n")
// The address form gets its own line. It is the thing that kept getting
// lost when it was buried in prose.
b.WriteString("ОБРАЩЕНИЕ: владелец — мужчина, всегда на \"ты\" (ты, тебя, тебе, твой) и в единственном числе (\"выпей\", \"посмотри\"). Никогда \"вы\"/\"вас\"/\"ваш\". Никогда \"он\"/\"его\" о нём — ты говоришь ему, а не о нём. Глаголы о нём — в мужском роде (\"ты забыл\").\n")
if who := f.who(); who != "" {
b.WriteString(who + "\n")
}
b.WriteString(fmt.Sprintf("Сейчас: %s, %d %s %d, %02d:%02d (местное время).\n",
ruWeekdays[int(now.Weekday())], now.Day(), ruMonths[int(now.Month())-1], now.Year(),
now.Hour(), now.Minute()))
b.WriteString("Умеешь: " + strings.Join(f.can(), "; ") +
". Других ДЕЙСТВИЙ не умеешь — если просят такое, скажи прямо.\n")
if s := strings.TrimSpace(f.Static); s != "" {
b.WriteString(s + "\n")
}
return b.String()
}
// can lists what she can really do. Every entry here is a code path that
// exists in the daemon today:
// - reminders: IntentReminder → CoreAPI.CreateReminder, fired by the tick.
// - notes and facts: IntentNote/IntentFact write, IntentQuery reads them back.
// - calendar: IntentQuery answers "что у меня сегодня" from CalendarEvents.
// - weather / telegram / shell acts: only when configured (see Facts).
//
// Nothing speculative goes in this list. A capability she offers and cannot
// perform is worse than one she never mentions.
func (f Facts) can() []string {
c := []string{
// Talking comes first, and the closing line says "действий" rather than
// "ничего", because this same block sits in front of the chat and
// general-knowledge prompts. A flat "you can do nothing else" would
// tell her to refuse the exact thing those two prompts are for.
"разговаривать и отвечать на вопросы",
"ставить напоминания",
"записывать заметки и факты и отвечать по ним",
"смотреть календарь",
}
if f.Weather {
c = append(c, "говорить погоду")
}
if f.Telegram {
c = append(c, "писать в телеграм")
}
if f.Tools {
c = append(c, "запускать разрешённые команды на сервере")
}
return c
}
// who renders the optional name/city line, or "" when neither is configured.
//
// Written as labels ("Имя владельца: ..."), not as a sentence with pronouns:
// the block's own "ты" is Maven, so "тебя зовут" would read as her name and
// "его" would model the third-person form she must never use about him.
func (f Facts) who() string {
name := strings.TrimSpace(f.OwnerName)
city := strings.TrimSpace(f.City)
switch {
case name != "" && city != "":
return "Имя владельца: " + name + ". Город: " + city + "."
case name != "":
return "Имя владельца: " + name + "."
case city != "":
return "Город: " + city + "."
}
return ""
}
// Prepend puts the block in front of a system prompt. Nil-safe: a nil renderer
// (tests, the stub paths) returns the prompt untouched.
func Prepend(block func() string, prompt string) string {
if block == nil {
return prompt
}
s := strings.TrimSpace(block())
if s == "" {
return prompt
}
return s + "\n\n" + prompt
}
+72
View File
@@ -0,0 +1,72 @@
package persona
import (
"strings"
"testing"
"time"
)
var ref = time.Date(2026, 7, 31, 14, 5, 0, 0, time.UTC)
// The block must be correct with an empty config: the address form and the
// gender rules are hard constraints, not preferences.
func TestBlockWorksWithZeroConfig(t *testing.T) {
b := Facts{}.Block(ref)
for _, want := range []string{"женском роде", "ОБРАЩЕНИЕ", "\"ты\"", "31 июля 2026", "пятница", "14:05"} {
if !strings.Contains(b, want) {
t.Errorf("block missing %q:\n%s", want, b)
}
}
}
func TestBlockAddsOptionalFacts(t *testing.T) {
b := Facts{OwnerName: "Ками", City: "Москва", Static: "Будь краткой."}.Block(ref)
for _, want := range []string{"Ками", "Москва", "Будь краткой."} {
if !strings.Contains(b, want) {
t.Errorf("block missing %q:\n%s", want, b)
}
}
}
// The time changes between turns, so two renders must differ.
func TestBlockRendersTimePerTurn(t *testing.T) {
a := Facts{}.Block(ref)
c := Facts{}.Block(ref.Add(time.Hour))
if a == c {
t.Errorf("block did not change with the clock:\n%s", a)
}
}
// She may only offer what this deployment actually has.
func TestCapabilitiesAreConfigGated(t *testing.T) {
bare := Facts{}.Block(ref)
for _, want := range []string{"напоминания", "заметки", "календарь"} {
if !strings.Contains(bare, want) {
t.Errorf("block missing always-on capability %q:\n%s", want, bare)
}
}
for _, unwanted := range []string{"погоду", "телеграм", "команды"} {
if strings.Contains(bare, unwanted) {
t.Errorf("block offers unconfigured %q:\n%s", unwanted, bare)
}
}
full := Facts{Weather: true, Telegram: true, Tools: true}.Block(ref)
for _, want := range []string{"погоду", "телеграм", "команды"} {
if !strings.Contains(full, want) {
t.Errorf("block missing configured capability %q:\n%s", want, full)
}
}
}
func TestPrependNilIsSafe(t *testing.T) {
if got := Prepend(nil, "PROMPT"); got != "PROMPT" {
t.Errorf("Prepend(nil) = %q", got)
}
if got := Prepend(func() string { return " " }, "PROMPT"); got != "PROMPT" {
t.Errorf("Prepend(blank) = %q", got)
}
if got := Prepend(func() string { return "CTX" }, "PROMPT"); got != "CTX\n\nPROMPT" {
t.Errorf("Prepend = %q", got)
}
}
+54
View File
@@ -0,0 +1,54 @@
package phraser
import (
"errors"
"strings"
"testing"
)
// A reply that starts a JSON object and never finishes it is a failed
// generation, not a reply. Before this, the parser returned ("", "") for these
// and every caller then shipped the raw fragment as the thing Maven said. A
// real run produced replies of literally "{" and "{\n \"".
func TestParseResponseMoodRejectsUnfinishedJSON(t *testing.T) {
for _, raw := range []string{
`{`,
"{\n \"",
`{"response": "неполн`,
`{"response": "текст", "mood":`,
} {
text, mood, err := parseResponseMood(raw)
if !errors.Is(err, errBrokenJSON) {
t.Errorf("parseResponseMood(%q) err = %v, want errBrokenJSON", raw, err)
}
if text != "" || mood != "" {
t.Errorf("parseResponseMood(%q) leaked %q/%q — a fragment must never come back as a reply", raw, text, mood)
}
}
}
// Bare prose is still fine. Small models sometimes answer without any JSON at
// all, and that reply is usable — so the new error must not swallow it.
func TestParseResponseMoodAllowsBareProse(t *testing.T) {
for _, raw := range []string{
"норм, а ты как?",
"вот что я нашла: ключ у соседа",
} {
text, mood, err := parseResponseMood(raw)
if err != nil {
t.Errorf("parseResponseMood(%q) err = %v, want nil", raw, err)
}
// No JSON means no fields; the caller ships raw as-is.
if text != "" || mood != "" {
t.Errorf("parseResponseMood(%q) = %q/%q, want empty", raw, text, mood)
}
}
}
// The measured failure: the model wants more than 400 characters and the old
// grammar cut it off mid-word. Guards the bound against being tightened back.
func TestGrammarStringBoundHasRoomForARealAnswer(t *testing.T) {
if !strings.Contains(responseGrammar, "{0,1000}") {
t.Error("grammar string bound is not 1000; 400 truncated real replies mid-word (see the comment on responseGrammar)")
}
}
+33
View File
@@ -0,0 +1,33 @@
package phraser
import (
"strings"
"testing"
)
// Every phrasing prompt must carry the shared context block. This is the
// regression guard for the bug that started this: the "ты" rule reached only
// two of the five prompts because each prompt had its own copy of the rules.
func TestEveryPromptCarriesTheContextBlock(t *testing.T) {
block := func() string { return "CTXBLOCK" }
p := &LLMPhraser{cfg: Config{ContextBlock: block}}
prompts := map[string]string{
"nudge": p.systemPrompt(),
"query": p.querySystemPrompt(),
"chat": chatSystemPrompt(block),
}
for name, got := range prompts {
if !strings.HasPrefix(got, "CTXBLOCK\n\n") {
t.Errorf("%s prompt does not start with the context block:\n%s", name, got)
}
}
}
// Without a block the prompts are unchanged — the stub and test paths pass nil.
func TestPromptsWithoutBlockAreUnchanged(t *testing.T) {
p := &LLMPhraser{}
if p.systemPrompt() != nudgeSystem {
t.Errorf("nudge prompt changed with no block set")
}
}
@@ -0,0 +1,65 @@
package eval
import (
"strings"
"testing"
)
// TestAddressReportsEveryBreak — the real reply from a nudge eval run broke in
// two ways at once and the check named only the plural. Both must print: a
// half-reported failure reads as a milder problem than it is.
func TestAddressReportsEveryBreak(t *testing.T) {
body := "Смотрите на его потребление воды."
res := checkAddress(body)
if res.Pass {
t.Fatalf("checkAddress passed %q", body)
}
for _, want := range []string{"смотрите", "его"} {
if !strings.Contains(res.Detail, want) {
t.Errorf("detail %q does not name %q", res.Detail, want)
}
}
}
// One word repeated is one problem, so the detail must not say it twice.
func TestAddressDeduplicates(t *testing.T) {
res := checkAddress("Вам стоит поесть, вам это нужно.")
if res.Pass {
t.Fatal("expected failure")
}
if n := strings.Count(res.Detail, "formal"); n != 1 {
t.Errorf("detail repeats the same break %d times: %q", n, res.Detail)
}
}
// The fragments a real run produced. All of them scored as non-empty replies
// before checkNonEmpty looked for letters.
func TestNonEmptyNeedsLetters(t *testing.T) {
for _, body := range []string{
"{",
"{\n \"",
"15-16",
`{"`,
" ",
"...",
} {
if got := checkNonEmpty(body); got.Pass {
t.Errorf("checkNonEmpty(%q) passed — that is not a reply", body)
}
}
}
// And it must not start failing real replies. Latin counts as well as Cyrillic:
// answers about ssd or vpn are legitimately part English.
func TestNonEmptyAcceptsRealReplies(t *testing.T) {
for _, body := range []string{
"норм, а ты как?",
"вот что я нашла: ключ у соседа",
"ssd быстрее hdd.",
"9 минут.",
} {
if got := checkNonEmpty(body); !got.Pass {
t.Errorf("checkNonEmpty(%q) failed: %s", body, got.Detail)
}
}
}
@@ -0,0 +1,42 @@
package eval
import "testing"
func TestAddressTimeWordDoesNotBlind(t *testing.T) {
// A nudge that opens with a time word must still be caught. Without the
// time words in the stoplist, "сегодня" was read as the third party.
for _, s := range []string{
"сегодня он не ел 11 дней",
"вчера он не пил воду",
"опять он забыл про таблетки",
} {
if r := checkAddress(s); r.Pass {
t.Errorf("checkAddress(%q) passed, want a third-person failure", s)
}
}
// Still must not fire when a third party really is named.
for _, s := range []string{
"сегодня сервис упал, он не отвечает",
"ты не пил воду четыре часа",
} {
if r := checkAddress(s); !r.Pass {
t.Errorf("checkAddress(%q) failed: %s", s, r.Detail)
}
}
}
// TestAddressVerbIsNotAnAntecedent — a nudge is mostly verbs, and a verb is
// never who "он" refers to. This exact string passed the check before.
func TestAddressVerbIsNotAnAntecedent(t *testing.T) {
s := "попробуй встать и отдохнуть — у него есть перерыв"
if r := checkAddress(s); r.Pass {
t.Errorf("checkAddress(%q) passed, want a third-person failure", s)
}
// Still missed, and this is the documented hole: "выпей воды, он не пил" has
// a real noun ("воды") before the pronoun, so the scan believes somebody
// else was named. Telling that apart needs a parser, not a suffix rule.
// A named third party still wins over the verbs around it.
if r := checkAddress("сервис упал, он не отвечает"); !r.Pass {
t.Errorf("checkAddress on a real third party failed: %s", r.Detail)
}
}
+244 -3
View File
@@ -21,10 +21,14 @@ const (
// CheckHisGender — the other half of the persona rule: SHE is feminine, HE // CheckHisGender — the other half of the persona rule: SHE is feminine, HE
// is male. "ты давно не отдыхала" addresses the operator as a woman. // is male. "ты давно не отдыхала" addresses the operator as a woman.
CheckHisGender = "hisgender" CheckHisGender = "hisgender"
// CheckAddress — she talks TO him, informally, one to one. Not "вы", not
// "он". See the comment block above checkAddress.
CheckAddress = "address"
) )
// CheckNames — report order. // CheckNames — report order.
var CheckNames = []string{CheckMood, CheckLang, CheckLength, CheckFeminine, CheckHisGender, CheckCringe, CheckOnTopic} var CheckNames = []string{CheckMood, CheckLang, CheckLength, CheckFeminine, CheckHisGender, CheckAddress, CheckCringe, CheckOnTopic}
// Result — one check on one message. // Result — one check on one message.
type Result struct { type Result struct {
@@ -57,6 +61,7 @@ func RunChecks(c Case, body, mood string) []Result {
checkLength(body), checkLength(body),
checkFeminine(body), checkFeminine(body),
checkHisGender(body), checkHisGender(body),
checkAddress(body),
checkCringe(body), checkCringe(body),
checkOnTopic(c, body), checkOnTopic(c, body),
} }
@@ -302,6 +307,194 @@ func prevWord(words []string, i int) string {
return "" return ""
} }
// --- how she addresses him ------------------------------------------------
//
// Persona hard constraint: Maven speaks TO him, informally, one to one. The
// phrasing eval produced two breaks of it, and both scored clean:
//
// - "Приходите… Жду вас" — the formal plural. Correct is ты/тебя/тебе and a
// singular imperative ("приходи", "жду тебя").
// - "Он не ел 11 дней" — she talks ABOUT him, in the third person, as if
// reporting to somebody else. Correct is "ты не ел 11 дней".
//
// Like checkHisGender this is a keyword + suffix heuristic, NOT a parser. Every
// hit prints the word it tripped on, so a false alarm is obvious at a glance and
// can be dismissed.
//
// Part 1, formal address. Two signals:
// - the "вы" pronoun family, matched as whole words, so there is nothing to
// exclude — "вы" and "вас" are never anything else.
// - a plural verb ending: -ите/-ете/-йте/-ьте ("приходите", "выпейте",
// "не забудьте", "хотите"). Nouns in the prepositional case share those
// endings ("в интернете", "в свете"), so a word right after a preposition is
// skipped. That is the whole exclusion list, on purpose: a bigger one would
// start swallowing real imperatives.
//
// Part 2, third person. "он" is perfectly fine when the message really is about
// somebody or something else ("сервис упал, он не отвечает"). The way to tell
// them apart: a legitimate third person has an ANTECEDENT — the thing it refers
// to was named earlier in the message. So "он" is only flagged when nothing
// before it in the message could be that thing.
//
// Where this gives up, plainly:
// - it only looks BACKWARD. "Он не отвечает, сервис упал" names the subject
// after the pronoun and is flagged wrongly.
// - any noun earlier in the message counts as an antecedent, even when it is
// not one ("после обеда он не ел", "выпей воды, он не пил" — both missed).
// Verbs and time words no longer count, which covers the usual nudge, but a
// plain noun before the pronoun still blinds it. The
// common time words are stoplisted so the usual nudge opening does not
// blind it, but a message with any other noun in front still slips through.
// This is the check's real hole; widening it further would start flagging
// legitimate third-party messages, so it stops here.
// - a message that opens with "ты" and only later slips into "он" is missed,
// because "ты" itself is skipped but the words around it are not.
// - formal address outside these endings (short adjectives, "вашими" style
// forms not listed) is missed.
// addressWordRE also takes Latin words, because "him"/"he" is the same break in
// English.
var addressWordRE = regexp.MustCompile(`[\p{Cyrillic}]+|[a-zA-Z]+|[,.;:!?…—-]`)
// formalPronouns — the "вы" family. Whole-word match, so no false hits.
var formalPronouns = map[string]bool{
"вы": true, "вас": true, "вам": true, "вами": true,
"ваш": true, "ваша": true, "ваше": true, "ваши": true,
"вашего": true, "вашей": true, "вашему": true, "вашим": true,
"вашими": true, "вашу": true,
}
// prepositions — used twice: to skip prepositional-case nouns that look like
// plural verbs, and as words that cannot be what "он" refers to.
var prepositions = map[string]bool{
"в": true, "во": true, "на": true, "о": true, "об": true, "обо": true,
"при": true, "по": true, "за": true, "из": true, "с": true, "со": true,
"к": true, "ко": true, "до": true, "от": true, "у": true, "над": true,
"под": true, "про": true, "без": true, "для": true, "через": true,
}
// pluralVerb reports whether a word looks like a plural/formal verb form:
// "приходите", "выпейте", "забудьте", "хотите".
func pluralVerb(w string) bool {
if len([]rune(w)) < 5 {
return false
}
return strings.HasSuffix(w, "ите") || strings.HasSuffix(w, "ете") ||
strings.HasSuffix(w, "йте") || strings.HasSuffix(w, "ьте")
}
// thirdPersonHim — pronouns that would be talking about him instead of to him.
var thirdPersonHim = map[string]bool{
"он": true, "его": true, "ему": true, "него": true, "нему": true, "ним": true,
"he": true, "him": true, "his": true,
}
// notAnAntecedent — words that cannot be the thing "он" refers to: pronouns,
// particles, conjunctions, adverbs of time. If only these come before "он", the
// message never named a third party and "он" is him.
var notAnAntecedent = map[string]bool{
"не": true, "ни": true, "и": true, "а": true, "но": true, "да": true,
"же": true, "бы": true, "ли": true, "вот": true, "уже": true,
"ещё": true, "еще": true, "тоже": true, "там": true, "тут": true,
"здесь": true, "это": true, "что": true, "как": true, "когда": true,
"чтобы": true, "потому": true, "сейчас": true, "потом": true,
// Time words. A nudge almost always opens with one ("сегодня он не ел"),
// and without them the very next word is read as the person being talked
// about, so the check misses the exact break it was written for.
"сегодня": true, "вчера": true, "завтра": true, "послезавтра": true,
"утром": true, "днём": true, "днем": true, "вечером": true, "ночью": true,
"опять": true, "снова": true, "весь": true, "всю": true, "целый": true,
"я": true, "мне": true, "меня": true, "мной": true, "мы": true, "нас": true,
"ты": true, "тебя": true, "тебе": true, "тобой": true,
"твой": true, "твоя": true, "твоё": true, "твое": true, "твои": true, "твою": true,
}
// looksVerb — a verb is never the thing "он" refers to, so it must not count as
// an antecedent. Past tense keeps "сервис упал, он не отвечает" working off
// "сервис"; the infinitive and imperative endings are here because a nudge is
// mostly made of them ("попробуй встать и отдохнуть — у него есть перерыв"
// slipped through with "попробуй" taken for the person being talked about).
func looksVerb(w string) bool {
r := []rune(w)
if len(r) < 3 {
return false
}
for _, suf := range []string{
"л", "ла", "ло", "ли", // past tense
"ть", "ться", "ти", "чь", // infinitive
"й", "йся", "йте", // imperative
} {
if strings.HasSuffix(w, suf) {
return true
}
}
return false
}
func checkAddress(body string) Result {
words := addressWordRE.FindAllString(strings.ToLower(body), -1)
// Every break, not just the first. A bad reply usually breaks in more than
// one way at once — "Смотрите на его потребление воды" is a plural imperative
// AND third person about him — and reporting only the first hid the second,
// which made the failure look milder than it was.
var breaks []string
seen := map[string]bool{}
add := func(msg string) {
if seen[msg] {
return // the same word twice in one message is one problem, not two
}
seen[msg] = true
breaks = append(breaks, msg)
}
for i, w := range words {
if formalPronouns[w] {
add(fmt.Sprintf("formal %q — she says ты/тебя/тебе", w))
}
if pluralVerb(w) && !(i > 0 && prepositions[words[i-1]]) {
add(fmt.Sprintf("plural imperative %q — she uses the singular", w))
}
}
for i, w := range words {
if !thirdPersonHim[w] {
continue
}
named := false
for j := 0; j < i; j++ {
p := words[j]
if !unicode.Is(unicode.Cyrillic, []rune(p)[0]) && !isLatinWord(p) {
continue // punctuation
}
// pluralVerb as well as looksVerb: looksVerb knows the imperative in
// -й/-йте but not the -те plural ("смотрите"), so "Смотрите на его
// потребление воды" counted "смотрите" as the person being talked
// about and the "его" never printed. Third time a verb form has
// blinded this check — if a fourth turns up, the antecedent test
// wants a real morphology table, not another suffix.
if notAnAntecedent[p] || prepositions[p] || thirdPersonHim[p] || looksVerb(p) || pluralVerb(p) {
continue
}
named = true
break
}
if !named {
add(fmt.Sprintf("third person %q with nobody else named — she talks to him, not about him", w))
}
}
if len(breaks) > 0 {
return Result{CheckAddress, false, strings.Join(breaks, " + ")}
}
return Result{CheckAddress, true, ""}
}
func isLatinWord(w string) bool {
r := []rune(w)[0]
return (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z')
}
// --- the cringe checks --------------------------------------------------- // --- the cringe checks ---------------------------------------------------
// //
// "Think Jarvis without the cringe part". DESIGN.md § Non-goals: "Not a // "Think Jarvis without the cringe part". DESIGN.md § Non-goals: "Not a
@@ -402,12 +595,60 @@ func checkCringe(body string) Result {
// checkOnTopic — the message must name the thing the rule is about. A nudge // checkOnTopic — the message must name the thing the rule is about. A nudge
// that never mentions water leaves the operator with a chime and no action. // that never mentions water leaves the operator with a chime and no action.
func checkOnTopic(c Case, body string) Result { func checkOnTopic(c Case, body string) Result {
return checkOnTopicAny(c.WantAny, body)
}
// checkOnTopicAny is the same test over a bare want-list, so the talk scorer can
// reuse it without owning a nudge Case.
func checkOnTopicAny(wantAny []string, body string) Result {
low := strings.ToLower(body) low := strings.ToLower(body)
for _, want := range c.WantAny { for _, want := range wantAny {
if strings.Contains(low, strings.ToLower(want)) { if strings.Contains(low, strings.ToLower(want)) {
return Result{CheckOnTopic, true, ""} return Result{CheckOnTopic, true, ""}
} }
} }
return Result{CheckOnTopic, false, return Result{CheckOnTopic, false,
fmt.Sprintf("mentions none of %v", c.WantAny)} fmt.Sprintf("mentions none of %v", wantAny)}
}
// --- shape checks for the free-form paths --------------------------------
//
// The nudge checks assume one short sentence. Chat and query replies are longer
// by design, so the only shape worth testing there is that the model produced a
// reply at all and did not trail off. Both are failure modes the fallbacks in
// llmphraser.go hide: a truncated or empty generation still returns nil error.
const (
CheckNonEmpty = "nonempty" // she said something
CheckEllipsis = "ellipsis" // she finished the sentence
)
// A reply needs words in it, not just characters. This check used to test for a
// non-empty string, which scored 27/27 on a run where two replies were "{" and
// "{\n \"" — punctuation passed as content. Braces, quotes, digits and spaces
// are all empty in the only sense that matters.
//
// Digits alone fail too, and that is deliberate: the same run answered "сколько
// варить яйцо вкрутую?" with "15-16". No unit, no words, and it is also the
// wrong number. Whatever that is, it is not something she said.
func checkNonEmpty(body string) Result {
if strings.TrimSpace(body) == "" {
return Result{CheckNonEmpty, false, "empty reply"}
}
for _, r := range body {
if unicode.IsLetter(r) {
return Result{CheckNonEmpty, true, ""}
}
}
return Result{CheckNonEmpty, false, fmt.Sprintf("no letters in the reply %q — punctuation or digits only", strings.TrimSpace(body))}
}
// checkEllipsis — a reply ending in "…" or "..." is a generation that ran out of
// tokens, not a stylistic pause. Mid-sentence ellipses are left alone.
func checkEllipsis(body string) Result {
trimmed := strings.TrimRight(strings.TrimSpace(body), `"'»)`)
if strings.HasSuffix(trimmed, "…") || strings.HasSuffix(trimmed, "...") {
return Result{CheckEllipsis, false, "reply trails off in an ellipsis — likely truncated"}
}
return Result{CheckEllipsis, true, ""}
} }
+35
View File
@@ -75,6 +75,7 @@ func TestStubBaseline(t *testing.T) {
CheckLength: 12, CheckLength: 12,
CheckFeminine: 15, CheckFeminine: 15,
CheckHisGender: 15, CheckHisGender: 15,
CheckAddress: 15,
CheckCringe: 15, CheckCringe: 15,
CheckOnTopic: 12, CheckOnTopic: 12,
} }
@@ -123,6 +124,10 @@ func TestChecksCatchWhatTheyClaim(t *testing.T) {
{"asks how he feels", "как ты себя чувствуешь? попей воды.", CheckCringe}, {"asks how he feels", "как ты себя чувствуешь? попей воды.", CheckCringe},
{"praise", "молодец! теперь попей воды.", CheckCringe}, {"praise", "молодец! теперь попей воды.", CheckCringe},
{"off topic", "пора бы уже что-то сделать.", CheckOnTopic}, {"off topic", "пора бы уже что-то сделать.", CheckOnTopic},
// The two recorded persona breaks from the phrasing eval run. Pinned as
// unit tests because an eval run is sampled and may not reproduce them.
{"formal plural", "Приходите… Жду вас", CheckAddress},
{"third person about him", "Он не ел 11 дней", CheckAddress},
} }
for _, tc := range cases { for _, tc := range cases {
@@ -144,6 +149,36 @@ func TestChecksCatchWhatTheyClaim(t *testing.T) {
} }
} }
// TestAddressCheck — the address check on its own, so the messages that must NOT
// trip it can be written without also having to satisfy the on-topic check.
func TestAddressCheck(t *testing.T) {
bad := []string{
"Приходите… Жду вас", // the recorded formal-plural break
"Он не ел 11 дней", // the recorded third-person break
"Выпейте воды, пожалуйста.", // plural imperative on its own
"Ваш обед был давно.", // formal possessive
}
for _, body := range bad {
if r := checkAddress(body); r.Pass {
t.Errorf("persona break not caught: %q", body)
} else {
t.Logf("%q -> %s", body, r.Detail)
}
}
good := []string{
"ты не пил воду четыре часа — попей.", // correct informal address
"сервис netdata упал, он не отвечает.", // legitimately about a third party
"я заметила, что зарядка была утром.", // no address at all
"в интернете опять тихо, всё работает.", // "интернете" is a noun, not an imperative
}
for _, body := range good {
if r := checkAddress(body); !r.Pass {
t.Errorf("clean message flagged: %q -> %s", body, r.Detail)
}
}
}
func TestMoodCheckUsesTheEnum(t *testing.T) { func TestMoodCheckUsesTheEnum(t *testing.T) {
if r := checkMood("cheerful"); r.Pass { if r := checkMood("cheerful"); r.Pass {
t.Error("mood outside the enum passed") t.Error("mood outside the enum passed")
+19 -1
View File
@@ -7,6 +7,8 @@ import (
"testing" "testing"
"time" "time"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser" "github.com/kami/maven/internal/phraser"
) )
@@ -32,6 +34,7 @@ func TestLLMPhrasingBaseline(t *testing.T) {
// case as a phrasing error and read as "the model cannot phrase". // case as a phrasing error and read as "the model cannot phrase".
noProxyLoopback(t) noProxyLoopback(t)
ctx := context.Background()
f, err := Load() f, err := Load()
if err != nil { if err != nil {
t.Fatalf("Load: %v", err) t.Fatalf("Load: %v", err)
@@ -41,10 +44,25 @@ func TestLLMPhrasingBaseline(t *testing.T) {
// Generous: an unconstrained 0.8B can spend a minute thinking before it // Generous: an unconstrained 0.8B can spend a minute thinking before it
// writes a word, and a timeout would be scored as a model failure. // writes a word, and a timeout would be scored as a model failure.
cfg.Timeout = 5 * time.Minute cfg.Timeout = 5 * time.Minute
// The same shared context block the daemon prepends (internal/persona),
// with an empty config — that is the deployment we actually ship.
cfg.ContextBlock = func() string { return persona.Facts{}.Block(time.Now()) }
p := phraser.NewLLMPhraserAt(base, cfg) p := phraser.NewLLMPhraserAt(base, cfg)
defer p.Close() defer p.Close()
rep, err := Score(context.Background(), "llm (0.8B, built-in persona)", p, f) // Label the run with whatever gguf the server actually has loaded. It used
// to say "0.8B" no matter what, so two runs of two different models came
// out named the same and were easy to mix up when comparing.
model, err := llm.ModelID(ctx, base)
if err != nil {
// An unlabelled score is still a score, but say so loudly — a made-up
// name in a bake-off table is worse than no name.
t.Logf("could not read model id from %s: %v — report will say %q", base, err, llm.UnknownModel)
model = llm.UnknownModel
}
t.Logf("scoring model %s at %s", model, base)
rep, err := Score(ctx, "llm ("+model+", built-in persona)", p, f)
if err != nil { if err != nil {
t.Fatalf("Score: %v", err) t.Fatalf("Score: %v", err)
} }
+265
View File
@@ -0,0 +1,265 @@
package eval
// This file scores the CONVERSATIONAL paths, the ones the nudge fixture never
// touches: chat, query-with-notes, and general knowledge. All three now carry
// the shared persona block (internal/persona), and all three produce long
// free-form Russian — which is exactly where a persona break (formality, third
// person, masculine self-reference) is most likely and where, until this file,
// nothing could see one.
//
// Why a second fixture instead of more nudge cases: the checks differ. A nudge
// must be one short sentence with no question in it; a chat reply is allowed
// 1-3 sentences and a follow-up question is a FEATURE there. Mixing them would
// need per-case check masks, and the nudge scorer stays untouched this way.
//
// Why per-path reporting: a chat regression and a knowledge regression have
// different causes (chat prompt vs router.KnowledgePrompt), and one blended
// percentage cannot tell them apart.
import (
"context"
_ "embed"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/kami/maven/internal/dialogue"
)
//go:embed talk_v1.json
var talkFixtureJSON []byte
// The three phrasing paths under test. Values match the fixture's "path" field.
const (
PathChat = "chat" // PhraseChat
PathQuery = "query" // PhraseQuery with notes
PathKnowledge = "knowledge" // PhraseQuery with no notes
)
// TalkPaths — report order.
var TalkPaths = []string{PathChat, PathQuery, PathKnowledge}
// TalkCheckNames — the checks that apply to a free-form reply, in report order.
// Deliberately a subset of CheckNames: length, mood and "no questions" are nudge
// properties and would fail a correct chat reply. These paths return no mood at
// all, so there is nothing to check there.
var TalkCheckNames = []string{
CheckNonEmpty, CheckEllipsis, CheckLang, CheckFeminine, CheckAddress, CheckOnTopic,
}
// TalkCase — one turn as the daemon would present it.
//
// History is flat text because that is all PhraseChat uses (it concatenates
// turn texts into one user message); intents and slots would be dead fields.
// Notes are what the store would have matched for a query.
//
// WantAny is the on-topic contract: at least one lowercased fragment must appear
// in the reply. Fragments are stems ("пароль" → "парол") so declension does not
// defeat them.
type TalkCase struct {
ID string `json:"id"`
Path string `json:"path"`
Utterance string `json:"utterance"`
History []string `json:"history,omitempty"`
Notes []string `json:"notes,omitempty"`
WantAny []string `json:"want_any"`
Tags []string `json:"tags,omitempty"`
Note string `json:"note,omitempty"`
}
// TalkFixture — the versioned envelope, same gating as Fixture.
type TalkFixture struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
Notes []string `json:"notes"`
Cases []TalkCase `json:"cases"`
}
// LoadTalk returns the embedded conversational fixture.
func LoadTalk() (TalkFixture, error) {
var f TalkFixture
if err := json.Unmarshal(talkFixtureJSON, &f); err != nil {
return TalkFixture{}, fmt.Errorf("parse talk fixture: %w", err)
}
if f.SchemaVersion != SchemaVersion {
return TalkFixture{}, fmt.Errorf("talk fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
}
if len(f.Cases) == 0 {
return TalkFixture{}, fmt.Errorf("talk fixture has no cases")
}
return f, nil
}
// Talker — the two methods a conversational path must have to be scorable.
// *phraser.LLMPhraser satisfies it; same trick as Nudger.
type Talker interface {
PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error)
PhraseQuery(ctx context.Context, utterance string, notes []string) (string, error)
}
// TalkOutcome — one scored case.
type TalkOutcome struct {
Case TalkCase
Reply string
Err error
Latency time.Duration
Pass bool
Failed []string
Reasons []string
}
// TalkReport — the aggregate. ByPath is the point of this scorer.
type TalkReport struct {
Name string
Total int
Passed int
Errors int
ByCheck map[string]int
ByPath map[string]TagStat
Outcomes []TalkOutcome
P50 time.Duration
P95 time.Duration
Max time.Duration
}
// Accuracy — fraction of cases that passed every check.
func (r TalkReport) Accuracy() float64 {
if r.Total == 0 {
return 0
}
return float64(r.Passed) / float64(r.Total)
}
// ScoreTalk runs every case through t and aggregates. A phrasing error scores as
// a miss and is counted separately: "the model was down" and "the model wrote
// something bad" must not be the same number.
func ScoreTalk(ctx context.Context, name string, t Talker, f TalkFixture) (TalkReport, error) {
rep := TalkReport{
Name: name,
Total: len(f.Cases),
ByCheck: map[string]int{},
ByPath: map[string]TagStat{},
}
for _, n := range TalkCheckNames {
rep.ByCheck[n] = 0
}
lat := make([]time.Duration, 0, len(f.Cases))
for _, c := range f.Cases {
start := time.Now()
reply, err := c.run(ctx, t)
o := TalkOutcome{Case: c, Reply: reply, Err: err, Latency: time.Since(start)}
lat = append(lat, o.Latency)
if err != nil {
rep.Errors++
o.Failed = append(o.Failed, "call")
o.Reasons = append(o.Reasons, fmt.Sprintf("phrase error: %v", err))
} else {
for _, res := range RunTalkChecks(c, reply) {
if res.Pass {
rep.ByCheck[res.Name]++
continue
}
o.Failed = append(o.Failed, res.Name)
o.Reasons = append(o.Reasons, res.Name+": "+res.Detail)
}
}
o.Pass = len(o.Failed) == 0
if o.Pass {
rep.Passed++
}
bump(rep.ByPath, c.Path, o.Pass)
rep.Outcomes = append(rep.Outcomes, o)
}
sort.Slice(lat, func(i, j int) bool { return lat[i] < lat[j] })
rep.P50, rep.P95 = percentile(lat, 0.50), percentile(lat, 0.95)
if len(lat) > 0 {
rep.Max = lat[len(lat)-1]
}
return rep, nil
}
// run dispatches the case to its path. knowledge and query are the same method;
// the empty notes slice is what selects the no-notes branch inside PhraseQuery.
func (c TalkCase) run(ctx context.Context, t Talker) (string, error) {
switch c.Path {
case PathChat:
return t.PhraseChat(ctx, c.Utterance, c.turns())
case PathQuery:
return t.PhraseQuery(ctx, c.Utterance, c.Notes)
case PathKnowledge:
return t.PhraseQuery(ctx, c.Utterance, nil)
}
return "", fmt.Errorf("unknown path %q", c.Path)
}
func (c TalkCase) turns() []dialogue.Turn {
turns := make([]dialogue.Turn, 0, len(c.History))
for _, h := range c.History {
turns = append(turns, dialogue.Turn{Text: h})
}
return turns
}
// RunTalkChecks scores one reply. Order matches TalkCheckNames.
func RunTalkChecks(c TalkCase, reply string) []Result {
return []Result{
checkNonEmpty(reply),
checkEllipsis(reply),
checkLang(reply),
checkFeminine(reply),
checkAddress(reply),
checkOnTopicAny(c.WantAny, reply),
}
}
// String renders the comparison table — composite, then per-check so a
// regression names the property, then per-path so it names the prompt.
func (r TalkReport) String() string {
var b strings.Builder
fmt.Fprintf(&b, "%s: %d/%d cases pass every check (%.1f%%), %d errors\n",
r.Name, r.Passed, r.Total, 100*r.Accuracy(), r.Errors)
for _, name := range TalkCheckNames {
fmt.Fprintf(&b, " %-10s %d/%d\n", name, r.ByCheck[name], r.Total)
}
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
fmt.Fprintf(&b, " by path: %s\n", renderStats(r.ByPath))
return b.String()
}
// Failures — per-case detail, sorted by ID so two runs diff cleanly.
func (r TalkReport) Failures() string {
var b strings.Builder
for _, o := range r.sorted() {
if o.Pass {
continue
}
fmt.Fprintf(&b, " %s %q\n %s\n", o.Case.ID, o.Reply, strings.Join(o.Reasons, "; "))
}
return b.String()
}
// Replies — every generated reply verbatim. This is what a human reads to judge
// tone; the score only says which checks fired.
func (r TalkReport) Replies() string {
var b strings.Builder
for _, o := range r.sorted() {
mark := "ok "
if !o.Pass {
mark = "FAIL"
}
fmt.Fprintf(&b, " %s %-9s %-22s %q\n", mark, o.Case.Path, o.Case.ID, o.Reply)
}
return b.String()
}
func (r TalkReport) sorted() []TalkOutcome {
out := append([]TalkOutcome(nil), r.Outcomes...)
sort.Slice(out, func(i, j int) bool { return out[i].Case.ID < out[j].Case.ID })
return out
}
+163
View File
@@ -0,0 +1,163 @@
package eval
import (
"context"
"os"
"strings"
"testing"
"time"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser"
)
// perPathMinimum — the resolution floor. A per-path score built on a handful of
// cases moves by 12% when a single reply changes, which cannot distinguish a
// prompt regression from noise.
const perPathMinimum = 8
// TestTalkFixture — the fixture itself has to be sound before any score off it
// means anything.
func TestTalkFixture(t *testing.T) {
f, err := LoadTalk()
if err != nil {
t.Fatalf("LoadTalk: %v", err)
}
seen := map[string]bool{}
byPath := map[string]int{}
for _, c := range f.Cases {
if seen[c.ID] {
t.Errorf("duplicate case id %q", c.ID)
}
seen[c.ID] = true
switch c.Path {
case PathChat, PathQuery, PathKnowledge:
default:
t.Errorf("%s: unknown path %q", c.ID, c.Path)
}
byPath[c.Path]++
if strings.TrimSpace(c.Utterance) == "" {
t.Errorf("%s: empty utterance", c.ID)
}
if len(c.WantAny) == 0 {
t.Errorf("%s: no want_any — the reply cannot be checked for topic", c.ID)
}
// A query case with no notes would silently score the knowledge path.
if c.Path == PathQuery && len(c.Notes) == 0 {
t.Errorf("%s: query case has no notes", c.ID)
}
if c.Path == PathKnowledge && len(c.Notes) > 0 {
t.Errorf("%s: knowledge case must have no notes", c.ID)
}
}
for _, p := range TalkPaths {
if byPath[p] < perPathMinimum {
t.Errorf("path %s has %d cases, want at least %d", p, byPath[p], perPathMinimum)
}
}
}
// fakeTalker — a scripted Talker, so the scorer is testable without a model.
type fakeTalker struct{ reply string }
func (f fakeTalker) PhraseChat(context.Context, string, []dialogue.Turn) (string, error) {
return f.reply, nil
}
func (f fakeTalker) PhraseQuery(context.Context, string, []string) (string, error) {
return f.reply, nil
}
// TestScoreTalkCounts — a reply that fails on purpose must be counted on every
// path, so a real run cannot report a hidden zero.
func TestScoreTalkCounts(t *testing.T) {
f, err := LoadTalk()
if err != nil {
t.Fatalf("LoadTalk: %v", err)
}
// Formal address, off-topic, trailing ellipsis: three checks fail at once.
rep, err := ScoreTalk(context.Background(), "fake", fakeTalker{"Приходите, я вас жду…"}, f)
if err != nil {
t.Fatalf("ScoreTalk: %v", err)
}
if rep.Total != len(f.Cases) || rep.Passed != 0 {
t.Errorf("got %d/%d passing, want 0/%d", rep.Passed, rep.Total, len(f.Cases))
}
if rep.ByCheck[CheckAddress] != 0 {
t.Errorf("formal reply passed the address check %d times", rep.ByCheck[CheckAddress])
}
if rep.ByCheck[CheckEllipsis] != 0 {
t.Errorf("truncated reply passed the ellipsis check %d times", rep.ByCheck[CheckEllipsis])
}
for _, p := range TalkPaths {
if rep.ByPath[p].Total == 0 {
t.Errorf("path %s missing from the report", p)
}
}
if !strings.Contains(rep.String(), "by path") {
t.Error("report does not break down by path")
}
}
// TestLLMTalkBaseline — the resident model on the three conversational paths.
// Opt-in exactly like TestLLMPhrasingBaseline: CI has no model and a run costs
// minutes on the CPU target.
//
// MAVEN_LLM_URL=http://127.0.0.1:18099 \
// go test -run TestLLMTalkBaseline ./internal/phraser/eval/
//
// Reports, does not assert a quality bar — the numbers are the input to tuning
// the persona prompt. The one thing worth failing on is a harness fault.
func TestLLMTalkBaseline(t *testing.T) {
base := os.Getenv("MAVEN_LLM_URL")
if base == "" {
t.Skip("MAVEN_LLM_URL unset — point it at a running llama-server (see doc comment)")
}
noProxyLoopback(t)
ctx := context.Background()
f, err := LoadTalk()
if err != nil {
t.Fatalf("LoadTalk: %v", err)
}
cfg := phraser.DefaultConfig("")
cfg.Timeout = 5 * time.Minute
cfg.ContextBlock = func() string { return persona.Facts{}.Block(time.Now()) }
p := phraser.NewLLMPhraserAt(base, cfg)
defer p.Close()
// Unreachable server is fatal here, not a logged warning, and that differs
// from the nudge test on purpose. PhraseNudge returns its errors, so a dead
// server there shows up honestly in the Errors column. PhraseChat and
// PhraseQuery do NOT: they swallow every failure and return a canned string
// ("поговорили.", "не знаю.", "вот что я нашла: …"). So on these three paths
// a dead server produces a full report with 0 errors and a terrible score —
// a number that looks like bad phrasing and is really no phrasing at all.
// Refusing to score without a confirmed model is the only guard available
// until the phraser reports its failures (Vikunja #397).
model, err := llm.ModelID(ctx, base)
if err != nil {
t.Fatalf("no model at %s: %v — refusing to score, these paths hide their errors "+
"and would report a plausible-looking result off a dead server", base, err)
}
t.Logf("scoring model %s at %s", model, base)
rep, err := ScoreTalk(ctx, "llm ("+model+", built-in persona)", p, f)
if err != nil {
t.Fatalf("ScoreTalk: %v", err)
}
t.Log("\n" + rep.String() + "\nreplies:\n" + rep.Replies() + "\nfailures:\n" + rep.Failures())
// And again afterwards: the run takes minutes, and a server that died or got
// OOM-killed halfway through would leave the first cases scored and the rest
// silently canned. Checking only at the start would not catch that.
if _, err := llm.ModelID(ctx, base); err != nil {
t.Fatalf("model at %s went away during the run: %v — the score above is not trustworthy", base, err)
}
}
+227
View File
@@ -0,0 +1,227 @@
{
"schema_version": 1,
"name": "ru-talk-v1",
"notes": [
"Scores the three conversational phrasing paths: chat (PhraseChat), query (PhraseQuery with notes) and knowledge (PhraseQuery with no notes). The nudge fixture does not cover any of them.",
"Nine cases per path, not five. The nudge fixture is 15 sampled cases and cannot resolve a change smaller than ~3 cases; a per-path score off five cases would be worse still. More cases per path is the point of this fixture.",
"The owner is a man, addressed informally as ty, living alone with a home server. Every utterance is written the way he actually talks to her.",
"chat-formality-bait and chat-about-me exist to provoke the two persona breaks the nudge eval caught: the formal vy/vas plural, and talking about him in the third person.",
"want_any fragments are stems so Russian declension does not defeat the on-topic check. They are lowercased before comparison.",
"want_any is a plain substring test, so a fragment that is too short passes by accident: \"ты\" matches inside \"работы\", \"нет\" inside \"интернет\". Keep every fragment to three or more letters of a real stem.",
"Notes are written as the store would have them: short, first person, no punctuation discipline."
],
"cases": [
{
"id": "chat-how-are-you",
"path": "chat",
"utterance": "привет, как дела?",
"want_any": ["норм", "хорош", "порядк", "тут", "работ"],
"tags": ["greeting"],
"note": "The plainest chat turn there is. If the persona breaks anywhere it breaks here first."
},
{
"id": "chat-formality-bait",
"path": "chat",
"utterance": "не могли бы вы подсказать, чем вы сейчас занимаетесь?",
"want_any": ["сейчас", "ничем", "ничего", "жду", "тут"],
"tags": ["persona-bait", "address"],
"note": "Deliberately polite and plural. A small model mirrors the register and answers with vy/vas — the exact break the address check was written for."
},
{
"id": "chat-about-me",
"path": "chat",
"utterance": "расскажи обо мне",
"want_any": ["теб"],
"tags": ["persona-bait", "third-person"],
"note": "Baits the third person: she should say 'ты живёшь один', not 'он живёт один', as if reporting to somebody else."
},
{
"id": "chat-bored-evening",
"path": "chat",
"utterance": "скучно что-то вечером, посоветуй чем заняться",
"want_any": ["можеш", "попробу", "почита", "прогул", "фильм", "серв"],
"tags": ["open-ended"]
},
{
"id": "chat-followup-server",
"path": "chat",
"utterance": "а стоит его вообще перезагружать?",
"history": ["сервер опять шумит как самолёт", "похоже вентилятор"],
"want_any": ["серв", "перезагру", "вентил", "шум"],
"tags": ["history", "anaphora"],
"note": "The pronoun 'его' only resolves through history. Also the one case where 'он' about the server is legitimate."
},
{
"id": "chat-tired",
"path": "chat",
"utterance": "устал я сегодня, весь день за компом",
"want_any": ["отдохн", "устал", "перерыв", "спат", "день"],
"tags": ["tone"],
"note": "Invites the fake-concern and emotional-support drift; the reply should stay plain."
},
{
"id": "chat-thanks",
"path": "chat",
"utterance": "спасибо, выручила",
"want_any": ["пожалуйст", "не за что", "рада", "обращ"],
"tags": ["persona", "feminine"],
"note": "Feminine self-reference is unavoidable in an answer to thanks: 'рада', not 'рад'."
},
{
"id": "chat-what-can-you-do",
"path": "chat",
"utterance": "что ты вообще умеешь?",
"want_any": ["напомн", "замет", "запис", "могу", "умею"],
"tags": ["self-description", "feminine"]
},
{
"id": "chat-joke",
"path": "chat",
"utterance": "расскажи что-нибудь смешное",
"want_any": ["анекдот", "шутк", "смешн", "истори"],
"tags": ["open-ended"],
"note": "Longest free-form generation in the chat set — the most likely place for a truncated reply."
},
{
"id": "query-router-password",
"path": "query",
"utterance": "что я записывал про пароль от роутера?",
"notes": ["пароль от роутера admin/xxK9tp — на наклейке снизу", "роутер висит в коридоре"],
"want_any": ["парол", "роутер", "наклейк"],
"tags": ["notes", "recall"]
},
{
"id": "query-bedtime-yesterday",
"path": "query",
"utterance": "напомни, во сколько я вчера лёг?",
"notes": ["лёг спать в 02:40", "сегодня встал в 9"],
"want_any": ["02:40", "2:40", "полтрет", "ноч"],
"tags": ["notes", "time"]
},
{
"id": "query-doctor-name",
"path": "query",
"utterance": "как звали того стоматолога, которого мне советовали?",
"notes": ["стоматолог Игорь Валерьевич, клиника на Ленина, советовал Дима"],
"want_any": ["игор", "валерьев", "стоматолог"],
"tags": ["notes", "recall"]
},
{
"id": "query-disk-plan",
"path": "query",
"utterance": "я что-то планировал с диском на сервере, что именно?",
"notes": ["купить второй hdd на 4тб под бэкапы", "перенести медиатеку с системного диска"],
"want_any": ["hdd", "бэкап", "диск", "4тб", "медиатек"],
"tags": ["notes", "homeserver"]
},
{
"id": "query-notes-do-not-answer",
"path": "query",
"utterance": "сколько я заплатил за домен?",
"notes": ["домен продлевается в марте", "хостинг оплачен на год вперёд"],
"want_any": ["домен", "не зна", "не указ"],
"tags": ["notes", "negative"],
"note": "The notes do not contain the price. The prompt tells her to say so; a made-up number is the failure being watched for."
},
{
"id": "query-single-note",
"path": "query",
"utterance": "где лежит запасной ключ?",
"notes": ["запасной ключ у соседа с четвёртого этажа"],
"want_any": ["ключ", "сосед", "четверт"],
"tags": ["notes", "single"],
"note": "One note only — PhraseQuery has a separate branch for len(notes) == 1."
},
{
"id": "query-polite-form",
"path": "query",
"utterance": "подскажите, пожалуйста, что у меня записано по машине?",
"notes": ["замена масла на 92 тысячах", "страховка до 14 сентября"],
"want_any": ["масл", "страховк", "92", "сентябр"],
"tags": ["notes", "persona-bait", "address"],
"note": "Polite plural in the question. The answer must still be ty."
},
{
"id": "query-shopping",
"path": "query",
"utterance": "что мне надо было купить?",
"notes": ["купить кофе и фильтры", "закончилась паста"],
"want_any": ["кофе", "фильтр", "паст"],
"tags": ["notes", "list"]
},
{
"id": "query-wifi-guest",
"path": "query",
"utterance": "я записывал гостевой вайфай?",
"notes": ["гостевая сеть maven-guest, пароль 12345678 меняю раз в месяц"],
"want_any": ["guest", "гостев", "12345678", "парол"],
"tags": ["notes", "recall"]
},
{
"id": "know-sky-blue",
"path": "knowledge",
"utterance": "почему небо синее?",
"want_any": ["све", "рассеи", "атмосфер", "син", "волн"],
"tags": ["general"]
},
{
"id": "know-boil-egg",
"path": "knowledge",
"utterance": "сколько варить яйцо вкрутую?",
"want_any": ["минут", "8", "9", "10", "варит"],
"tags": ["general", "practical"]
},
{
"id": "know-ssd-vs-hdd",
"path": "knowledge",
"utterance": "чем ssd отличается от hdd?",
"want_any": ["ssd", "hdd", "быстр", "диск", "механич"],
"tags": ["general", "tech"]
},
{
"id": "know-cat-purr",
"path": "knowledge",
"utterance": "почему кошки мурчат?",
"want_any": ["кош", "мурч", "вибра", "успока"],
"tags": ["general"]
},
{
"id": "know-hiccups",
"path": "knowledge",
"utterance": "как быстро избавиться от икоты?",
"want_any": ["икот", "дыха", "вод", "задерж"],
"tags": ["general", "practical"]
},
{
"id": "know-polite-form",
"path": "knowledge",
"utterance": "не могли бы вы объяснить, что такое vpn?",
"want_any": ["vpn", "туннел", "трафик", "сет", "шифр"],
"tags": ["general", "persona-bait", "address"],
"note": "Polite plural bait on the knowledge prompt, which is a different system prompt from chat and must hold the same line."
},
{
"id": "know-dont-know",
"path": "knowledge",
"utterance": "как зовут моего соседа снизу?",
"want_any": ["не зна", "не мог"],
"tags": ["general", "negative"],
"note": "Unanswerable without notes. Admitting it beats inventing a name; watching for the invention."
},
{
"id": "know-water-per-day",
"path": "knowledge",
"utterance": "сколько воды в день надо пить?",
"want_any": ["вод", "литр", "стакан", "пит"],
"tags": ["general", "health"],
"note": "Overlaps a nudge rule on purpose: the knowledge answer must not turn into a nudge."
},
{
"id": "know-thunder-delay",
"path": "knowledge",
"utterance": "почему гром слышно позже молнии?",
"want_any": ["звук", "све", "быстр", "гром", "молни"],
"tags": ["general"]
}
]
}
+127
View File
@@ -0,0 +1,127 @@
package phraser
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/kami/maven/internal/loop"
)
// grammarSpy stands in for llama-server: it records the grammar field of every
// request and always answers with a contract-shaped reply.
type grammarSpy struct {
srv *httptest.Server
grammars []string
}
func newGrammarSpy(t *testing.T) *grammarSpy {
t.Helper()
s := &grammarSpy{}
s.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var req chatReq
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
t.Errorf("spy: decode request: %v", err)
}
s.grammars = append(s.grammars, req.Grammar)
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(`{"choices":[{"message":{"content":"{\"response\": \"ага\", \"mood\": \"neutral\"}"}}]}`))
}))
t.Cleanup(s.srv.Close)
return s
}
// callAllPhrasingPaths hits every path that expects the JSON contract.
func callAllPhrasingPaths(t *testing.T, p *LLMPhraser) {
t.Helper()
ctx := context.Background()
if _, err := p.PhraseNudge(ctx, loop.Candidate{Rule: loop.WaterRule(), Severity: loop.Sev1}); err != nil {
t.Fatalf("PhraseNudge: %v", err)
}
if _, err := p.PhraseChat(ctx, "привет", nil); err != nil {
t.Fatalf("PhraseChat: %v", err)
}
// Both branches: no notes (general knowledge) and with notes (grounded).
if _, err := p.PhraseQuery(ctx, "сколько воды я выпил", nil); err != nil {
t.Fatalf("PhraseQuery (no notes): %v", err)
}
if _, err := p.PhraseQuery(ctx, "сколько воды я выпил", []string{"два литра"}); err != nil {
t.Fatalf("PhraseQuery (notes): %v", err)
}
}
func TestGrammarIsAttachedToEveryPhrasingRequest(t *testing.T) {
if strings.TrimSpace(responseGrammar) == "" {
t.Fatal("responseGrammar is empty")
}
spy := newGrammarSpy(t)
p := NewLLMPhraserAt(spy.srv.URL, Config{})
callAllPhrasingPaths(t, p)
if len(spy.grammars) != 4 {
t.Fatalf("expected 4 requests, got %d", len(spy.grammars))
}
for i, g := range spy.grammars {
if g != responseGrammar {
t.Errorf("request %d carries grammar %q, want responseGrammar", i, g)
}
}
}
func TestNoGrammarConfigDisablesIt(t *testing.T) {
spy := newGrammarSpy(t)
p := NewLLMPhraserAt(spy.srv.URL, Config{NoGrammar: true})
callAllPhrasingPaths(t, p)
for i, g := range spy.grammars {
if g != "" {
t.Errorf("request %d still carries a grammar with NoGrammar set: %q", i, g)
}
}
}
// The grammar's string rule must accept any codepoint, not just ASCII. Replies
// are Russian: an ASCII-only class would constrain the model into empty replies.
func TestGrammarStringRuleIsNotASCIIOnly(t *testing.T) {
if !strings.Contains(responseGrammar, `([^"\\] | "\\" ["\\/bfnrt])`) {
t.Error("string rule is not the any-codepoint-except-quote-and-backslash class; Cyrillic replies would be impossible")
}
}
// What the grammar describes must survive the parser that reads it back — a
// Russian body with an escaped quote inside, hand-built to test the contract.
func TestGrammarShapedJSONParses(t *testing.T) {
raw := `{"response": "он сказал \"привет\" и ушёл.\nвот так.", "mood": "confused"}`
text, mood, err := parseResponseMood(raw)
if err != nil {
t.Fatalf("grammar-shaped JSON did not parse: %v", err)
}
if want := "он сказал \"привет\" и ушёл.\nвот так."; text != want {
t.Errorf("response = %q, want %q", text, want)
}
if mood != "confused" {
t.Errorf("mood = %q, want confused", mood)
}
}
// Every mood the grammar permits is one the contract knows, and all five are there.
func TestGrammarMoodEnumMatchesTheContract(t *testing.T) {
for _, m := range []string{"neutral", "happy", "thinking", "tired", "confused"} {
if !strings.Contains(responseGrammar, `"\"`+m+`\""`) {
t.Errorf("mood %q missing from the grammar", m)
}
}
// No sixth mood: the enum line lists exactly five alternatives.
for _, line := range strings.Split(responseGrammar, "\n") {
if strings.HasPrefix(line, "mood") {
if n := strings.Count(line, "|") + 1; n != 5 {
t.Errorf("mood rule lists %d alternatives, want 5: %s", n, line)
}
}
}
}
+132 -42
View File
@@ -18,6 +18,7 @@ import (
"github.com/kami/maven/internal/delivery" "github.com/kami/maven/internal/delivery"
"github.com/kami/maven/internal/dialogue" "github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/loop" "github.com/kami/maven/internal/loop"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/router" "github.com/kami/maven/internal/router"
) )
@@ -39,7 +40,18 @@ type Config struct {
NGpuLayers int NGpuLayers int
NCtx int NCtx int
Timeout time.Duration Timeout time.Duration
Persona string // optional prompt prefix tuning maven's character
// ContextBlock renders the shared context block (who he is, how to
// address him, the time) fresh for each turn. See internal/persona.
// nil ⇒ no block, the prompts stand alone.
ContextBlock func() string
// NoGrammar turns the GBNF constraint off (zero value ⇒ grammar ON).
// The escape hatch exists because the target resident model — the
// locally CPT'd Qwen3-1.7B — does not exist yet: if its chat template
// ever fights the grammar, the fix should be a config flip on the
// deploy box, not a code change and a rebuild.
NoGrammar bool
} }
func DefaultConfig(modelPath string) Config { func DefaultConfig(modelPath string) Config {
@@ -178,7 +190,12 @@ func (p *LLMPhraser) PhraseNudge(ctx context.Context, c loop.Candidate) (deliver
if err != nil { if err != nil {
return delivery.PhrasedNudge{}, err return delivery.PhrasedNudge{}, err
} }
body, mood := parseResponseMood(resp) body, mood, perr := parseResponseMood(resp)
if perr != nil {
// Truncated JSON. Not a nudge — use the plain Russian fallback.
log.Printf("phraser: PhraseNudge: %v", perr)
body, mood = "", ""
}
if body == "" { if body == "" {
// fallback: try old body/summary format // fallback: try old body/summary format
body, _ = parsePhrase(resp) body, _ = parsePhrase(resp)
@@ -202,13 +219,18 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
if len(notes) == 0 { if len(notes) == 0 {
// General knowledge — no notes to ground the answer. The system // General knowledge — no notes to ground the answer. The system
// prompt is the single tested source in router.KnowledgePrompt. // prompt is the single tested source in router.KnowledgePrompt.
sys := router.KnowledgePrompt() sys := persona.Prepend(p.cfg.ContextBlock, router.KnowledgePrompt())
prompt := fmt.Sprintf("Пользователь спрашивает: \"%s\".", utterance) prompt := fmt.Sprintf("Пользователь спрашивает: \"%s\".", utterance)
resp, err := p.chatWithSystem(ctx, sys, prompt, 256) resp, err := p.chatWithSystem(ctx, sys, prompt, 768)
if err != nil || resp == "" { if err != nil || resp == "" {
return "не знаю.", nil return "не знаю.", nil
} }
if text, _ := parseResponseMood(resp); text != "" { text, _, perr := parseResponseMood(resp)
if perr != nil {
log.Printf("phraser: PhraseQuery: %v", perr)
return "не знаю.", nil
}
if text != "" {
return text, nil return text, nil
} }
return resp, nil return resp, nil
@@ -218,17 +240,22 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
} }
sys := p.querySystemPrompt() sys := p.querySystemPrompt()
prompt := fmt.Sprintf( prompt := fmt.Sprintf(
`The user asks: "%s". Your notes matching the query contain: "%s". Answer them naturally and briefly. If the notes don't answer the question, say so.`, `Он спрашивает: "%s". В твоих заметках по этому вопросу написано: "%s". Ответь ему коротко и своими словами. Если в заметках ответа нет — так и скажи.`,
utterance, strings.Join(notes, `"; "`), utterance, strings.Join(notes, `"; "`),
) )
resp, err := p.chatWithSystem(ctx, sys, prompt, 256) resp, err := p.chatWithSystem(ctx, sys, prompt, 768)
if err != nil { text, _, perr := parseResponseMood(resp)
if err != nil || perr != nil {
// Read the notes out rather than ship a broken fragment.
if perr != nil {
log.Printf("phraser: PhraseQuery: %v", perr)
}
if len(notes) == 1 { if len(notes) == 1 {
return "вот что я нашла: " + notes[0], nil return "вот что я нашла: " + notes[0], nil
} }
return "вот что я нашла: " + strings.Join(notes, "; "), nil return "вот что я нашла: " + strings.Join(notes, "; "), nil
} }
if text, _ := parseResponseMood(resp); text != "" { if text != "" {
return text, nil return text, nil
} }
return resp, nil return resp, nil
@@ -238,7 +265,7 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
// message array from dialogue history + the current user utterance. Falls back // message array from dialogue history + the current user utterance. Falls back
// to a simple greeting on any LLM error — better to say something than nothing. // to a simple greeting on any LLM error — better to say something than nothing.
func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error) { func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error) {
sys := chatSystemPrompt(p.cfg.Persona) sys := chatSystemPrompt(p.cfg.ContextBlock)
msgs := []chatMsg{ msgs := []chatMsg{
{Role: "system", Content: sys}, {Role: "system", Content: sys},
} }
@@ -251,12 +278,17 @@ func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history [
combined += utterance combined += utterance
msgs = append(msgs, chatMsg{Role: "user", Content: strings.TrimSpace(combined)}) msgs = append(msgs, chatMsg{Role: "user", Content: strings.TrimSpace(combined)})
resp, err := p.chatWithMessages(ctx, msgs, 512) resp, err := p.chatWithMessages(ctx, msgs, 768)
if err != nil { if err != nil {
log.Printf("phraser: PhraseChat: %v", err) log.Printf("phraser: PhraseChat: %v", err)
return "поговорили.", nil return "поговорили.", nil
} }
if text, _ := parseResponseMood(resp); text != "" { text, _, perr := parseResponseMood(resp)
if perr != nil {
log.Printf("phraser: PhraseChat: %v", perr)
return "поговорили.", nil
}
if text != "" {
return text, nil return text, nil
} }
// fallback: plain text without JSON // fallback: plain text without JSON
@@ -267,17 +299,16 @@ func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history [
} }
// chatSystemPrompt returns the system prompt for conversational chat. // chatSystemPrompt returns the system prompt for conversational chat.
// Prepends the configured persona when set. // Prepends the shared context block when the phraser has one.
func chatSystemPrompt(persona string) string { func chatSystemPrompt(block func() string) string {
base := `You are maven, a self-hosted personal assistant. You're talking with your owner. // No self-introduction here: the persona block prepended one line above
Keep replies brief (1-3 sentences) and natural. You're helpful, curious, and a little warm. // already says who she is, same as router.KnowledgePrompt.
Respond in the user's language (Russian or English, matching their last message). base := `Ты разговариваешь с хозяином. О себе говоришь в женском роде ("я подумала", "я рада"). Он мужчина: обращайся к нему на "ты", в мужском роде ("ты сказал", "ты забыл"). Никогда не "вы"/"ваш" и никогда "он"/"его" ты говоришь ему, а не о нём.
Never roleplay emotions you don't have, but stay friendly.
Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}. "response" is your reply text; "mood" reflects your tone (neutral/happy/thinking/tired/confused).` Отвечай по-русски, коротко: одна-три фразы, живым языком. Ты доброжелательная, тебе интересно, но чувства не изображай.
if persona != "" {
base = persona + "\n\n" + base Отвечай ТОЛЬКО одним объектом JSON: {"response": "...", "mood": "neutral"}. В "response" твой ответ. В "mood" ровно одно из: neutral, happy, thinking, tired, confused.`
} return persona.Prepend(block, base)
return base
} }
// chatWithMessages sends a full message array (system + history + current) to // chatWithMessages sends a full message array (system + history + current) to
@@ -288,6 +319,7 @@ func (p *LLMPhraser) chatWithMessages(ctx context.Context, msgs []chatMsg, maxTo
Messages: msgs, Messages: msgs,
Temperature: 0.7, Temperature: 0.7,
MaxTokens: maxTokens, MaxTokens: maxTokens,
Grammar: p.grammar(),
} }
body, err := json.Marshal(req) body, err := json.Marshal(req)
if err != nil { if err != nil {
@@ -339,7 +371,12 @@ func (p *LLMPhraser) PhraseReminder(ctx context.Context, d loop.ReminderDecision
if err != nil { if err != nil {
return delivery.PhrasedReminder{}, err return delivery.PhrasedReminder{}, err
} }
body, mood := parseResponseMood(resp) body, mood, perr := parseResponseMood(resp)
if perr != nil {
// Truncated JSON. Fall through to the reminder's own text.
log.Printf("phraser: PhraseReminder: %v", perr)
body, mood = "", ""
}
if body == "" { if body == "" {
// fallback: try old body/summary format // fallback: try old body/summary format
body, _ = parsePhrase(resp) body, _ = parsePhrase(resp)
@@ -367,6 +404,43 @@ type chatReq struct {
Messages []chatMsg `json:"messages"` Messages []chatMsg `json:"messages"`
Temperature float64 `json:"temperature"` Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"` MaxTokens int `json:"max_tokens"`
// Grammar is llama-server's `grammar` field (GBNF). Same wiring as
// internal/llm.Req.Grammar. Empty ⇒ unconstrained sampling.
Grammar string `json:"grammar,omitempty"`
}
// responseGrammar — GBNF constraining the model to the documented phrasing
// contract and nothing else: {"response": "<text>", "mood": "<enum>"}.
//
// Without it a 0.8B answers roughly one chat turn in three with open reasoning
// as plain text ("Thinking Process:" …), which no tag-stripper can remove and
// which eats the token budget before the JSON closes. Modelled on
// routeGrammar in internal/router/llmrouter.go so the two read alike.
//
// text accepts ANY codepoint except the two JSON must escape — the replies are
// Russian, so an ASCII-only rule would make every reply empty. The escape rule
// is what lets the model close a string it opened with a quote inside. Length
// is bounded so a repetition loop truncates the field, not the JSON object.
//
// That bound was 400 and 400 was too tight. Measured against Qwen3.5-0.8B: on
// "почему гром слышно позже молнии?" the reply came back exactly 400 characters
// long, cut mid-word ("Нужно записать и,"), at every token cap from 256 to 2048.
// So the token cap was never what stopped it — this rule was. 1000 characters is
// roughly six Russian sentences, still short enough to stop a repetition loop.
const responseGrammar = `
root ::= "{" ws "\"response\"" ws ":" ws string ws "," ws "\"mood\"" ws ":" ws mood ws "}"
mood ::= "\"neutral\"" | "\"happy\"" | "\"thinking\"" | "\"tired\"" | "\"confused\""
string ::= "\"" ([^"\\] | "\\" ["\\/bfnrt]){0,1000} "\""
ws ::= [ \t\n]*
`
// grammar returns the GBNF to attach to a phrasing request, or "" when the
// operator turned it off.
func (p *LLMPhraser) grammar() string {
if p.cfg.NoGrammar {
return ""
}
return responseGrammar
} }
type chatResp struct { type chatResp struct {
@@ -391,6 +465,7 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma
}, },
Temperature: 0.7, Temperature: 0.7,
MaxTokens: maxTokens, MaxTokens: maxTokens,
Grammar: p.grammar(),
} }
body, err := json.Marshal(req) body, err := json.Marshal(req)
if err != nil { if err != nil {
@@ -439,8 +514,10 @@ func (p *LLMPhraser) chatWithSystem(ctx context.Context, system, user string, ma
// as "..." before this. See PHRASING-EVAL-31-07-2026.md. // as "..." before this. See PHRASING-EVAL-31-07-2026.md.
// //
// Russian only, feminine self-reference, second person masculine (the owner is // Russian only, feminine self-reference, second person masculine (the owner is
// a man). One short sentence — the nudge is spoken aloud. // a man). She talks TO him, informally, singular — never "вы", never "он".
// One short sentence — the nudge is spoken aloud.
const nudgeSystem = `Ты Maven, домашняя ассистентка. О себе говоришь в женском роде ("я проверила", "я записала"). Владелец мужчина, обращайся к нему в мужском роде ("ты пил", "ты забыл"). const nudgeSystem = `Ты Maven, домашняя ассистентка. О себе говоришь в женском роде ("я проверила", "я записала"). Владелец мужчина, обращайся к нему в мужском роде ("ты пил", "ты забыл").
Говоришь с ним на "ты", в единственном числе ("выпей", "встань"). Никогда не "вы"/"вас"/"ваш" и никогда "он"/"его" ты говоришь ему, а не о нём.
Пиши ОДНО короткое напоминание по-русски: не больше 120 символов и не больше 16 слов. Только по делу. Пиши ОДНО короткое напоминание по-русски: не больше 120 символов и не больше 16 слов. Только по делу.
@@ -457,21 +534,16 @@ const nudgeSystem = `Ты — Maven, домашняя ассистентка. О
Это примеры ФОРМЫ, а не темы. Пиши только про ту ситуацию, которую тебе дали в запросе. Не копируй примеры и никогда не пиши "..." в поле response.` Это примеры ФОРМЫ, а не темы. Пиши только про ту ситуацию, которую тебе дали в запросе. Не копируй примеры и никогда не пиши "..." в поле response.`
func (p *LLMPhraser) systemPrompt() string { func (p *LLMPhraser) systemPrompt() string {
base := nudgeSystem return persona.Prepend(p.cfg.ContextBlock, nudgeSystem)
if p.cfg.Persona != "" {
base = p.cfg.Persona + "\n\n" + base
}
return base
} }
// querySystemPrompt returns the system prompt for PhraseQuery (notes + general // querySystemPrompt returns the system prompt for PhraseQuery (notes + general
// knowledge). Prepends the configured persona when set. // knowledge). Prepends the configured persona when set.
func (p *LLMPhraser) querySystemPrompt() string { func (p *LLMPhraser) querySystemPrompt() string {
base := "You are maven, a self-hosted personal assistant answering from your notes. Answer briefly and naturally in Russian starting with \"вот что я нашла: \". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}." // No self-introduction here: the persona block prepended one line above
if p.cfg.Persona != "" { // already says who she is, same as router.KnowledgePrompt.
base = p.cfg.Persona + "\n\n" + base base := "Ты отвечаешь ему по своим заметкам. Отвечай по-русски, коротко и своими словами, начинай с \"вот что я нашла: \". О себе — в женском роде (\"нашла\", \"записала\"). Он мужчина, обращайся к нему на \"ты\". Respond ONLY with valid JSON: {\"response\": \"...\", \"mood\": \"neutral\"}."
} return persona.Prepend(p.cfg.ContextBlock, base)
return base
} }
// ruleTopics — Russian gloss for each built-in rule name. The rule names are // ruleTopics — Russian gloss for each built-in rule name. The rule names are
@@ -592,21 +664,39 @@ type responseMood struct {
Mood string `json:"mood"` Mood string `json:"mood"`
} }
// errBrokenJSON — the model started a JSON object and never finished it.
// That is a failed generation, not a reply. Callers must use their fallback.
var errBrokenJSON = fmt.Errorf("phraser: model output starts as JSON but does not parse")
// parseResponseMood extracts {"response","mood"} from LLM output, tolerant // parseResponseMood extracts {"response","mood"} from LLM output, tolerant
// of thinking tokens and extra text before/after the JSON block. Returns // of thinking tokens and extra text before/after the JSON block.
// ("", "") when no valid JSON is found. //
func parseResponseMood(raw string) (response, mood string) { // Three outcomes:
// - parsed fine → the fields, nil error.
// - output never looked like JSON → ("", "", nil). The caller may ship it
// as-is; small models sometimes answer in bare prose and that is fine.
// - output starts with "{" but does not parse → errBrokenJSON. The grammar
// guarantees a valid *prefix*, so a generation that hits the token cap
// mid-object comes back as a fragment like `{` or `{\n "`. Shipping that
// as a reply is the bug this error exists to stop.
func parseResponseMood(raw string) (response, mood string, err error) {
cleaned := strings.TrimSpace(raw) cleaned := strings.TrimSpace(raw)
start := strings.Index(cleaned, "{") start := strings.Index(cleaned, "{")
end := strings.LastIndex(cleaned, "}") end := strings.LastIndex(cleaned, "}")
if start < 0 || end < 0 || end <= start { if start < 0 || end < 0 || end <= start {
return "", "" if strings.HasPrefix(cleaned, "{") {
return "", "", errBrokenJSON
}
return "", "", nil
} }
var parsed responseMood var parsed responseMood
if err := json.Unmarshal([]byte(cleaned[start:end+1]), &parsed); err != nil { if e := json.Unmarshal([]byte(cleaned[start:end+1]), &parsed); e != nil {
return "", "" if strings.HasPrefix(cleaned, "{") {
return "", "", errBrokenJSON
}
return "", "", nil
} }
return parsed.Response, parsed.Mood return parsed.Response, parsed.Mood, nil
} }
func parsePhrase(raw string) (body, summary string) { func parsePhrase(raw string) (body, summary string) {
+3 -3
View File
@@ -61,12 +61,12 @@ func TestLLMRouterBaseline(t *testing.T) {
} }
ctx := context.Background() ctx := context.Background()
model, err := ModelID(ctx, base) model, err := llm.ModelID(ctx, base)
if err != nil { if err != nil {
// Not fatal: an unlabelled score is still a score. But say so loudly, // Not fatal: an unlabelled score is still a score. But say so loudly,
// because an unlabelled row in a bake-off table is worthless. // because an unlabelled row in a bake-off table is worthless.
t.Logf("could not read model id from %s: %v — reports will say %q", base, err, "unknown-model") t.Logf("could not read model id from %s: %v — reports will say %q", base, err, llm.UnknownModel)
model = "unknown-model" model = llm.UnknownModel
} }
t.Logf("scoring model %s at %s", model, base) t.Logf("scoring model %s at %s", model, base)
lr := router.NewLLMRouter(client) lr := router.NewLLMRouter(client)
+4 -1
View File
@@ -3,5 +3,8 @@ package router
// KnowledgePrompt returns the system prompt for general knowledge questions // KnowledgePrompt returns the system prompt for general knowledge questions
// that the phraser uses when no notes match the query. // that the phraser uses when no notes match the query.
func KnowledgePrompt() string { func KnowledgePrompt() string {
return `Ты — Мавена, персональный ассистент. Ответь кратко из своих знаний. Если не знаешь — скажи "не знаю". Не выдумывай. Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}.` // No self-introduction here: the shared persona block already says who she
// is, and this line used to disagree with it — a different name ("Мавена")
// and a masculine noun ("ассистент") in front of a feminine persona.
return `Ответь кратко из своих знаний. Если не знаешь — скажи "не знаю". Не выдумывай. Respond ONLY with valid JSON: {"response": "...", "mood": "neutral"}.`
} }
+9 -1
View File
@@ -11,10 +11,18 @@ func TestKnowledgePrompt(t *testing.T) {
t.Fatal("KnowledgePrompt returned empty string") t.Fatal("KnowledgePrompt returned empty string")
} }
// Must contain key instructions // Must contain key instructions
checks := []string{"Мавена", "не знаю", "не выдумывай"} checks := []string{"не знаю", "не выдумывай"}
for _, c := range checks { for _, c := range checks {
if !strings.Contains(strings.ToLower(prompt), strings.ToLower(c)) { if !strings.Contains(strings.ToLower(prompt), strings.ToLower(c)) {
t.Errorf("KnowledgePrompt should mention %q", c) t.Errorf("KnowledgePrompt should mention %q", c)
} }
} }
// Who she is comes from the shared persona block now. This prompt used to
// say it too, with a different name and a masculine noun, which is the
// drift the block exists to stop.
for _, w := range []string{"Мавена", "ассистент"} {
if strings.Contains(prompt, w) {
t.Errorf("KnowledgePrompt should not introduce her (%q) — the persona block does", w)
}
}
} }
+8 -3
View File
@@ -13,11 +13,15 @@ import (
// unknown = a pending row found stale at startup: the process that started it // unknown = a pending row found stale at startup: the process that started it
// is gone, and the send may or may not have reached the external channel. // is gone, and the send may or may not have reached the external channel.
// Never auto-resolved into sent or failed — that would be guessing. // Never auto-resolved into sent or failed — that would be guessing.
// dropped = the routing table deliberately suppressed this one (a care nudge
// while you're away). Nothing was sent and nothing went wrong; the row exists
// so "she dropped it" and "the rule never fired" don't look the same later.
const ( const (
DeliveryPending = "pending" DeliveryPending = "pending"
DeliverySent = "sent" DeliverySent = "sent"
DeliveryFailed = "failed" DeliveryFailed = "failed"
DeliveryUnknown = "unknown" DeliveryUnknown = "unknown"
DeliveryDropped = "dropped"
) )
// BeginDeliveryAttempt durably records intent to send BEFORE the external // BeginDeliveryAttempt durably records intent to send BEFORE the external
@@ -43,10 +47,11 @@ func (s *Store) BeginDeliveryAttempt(ctx context.Context, kind, rule string, rem
} }
// CompleteDeliveryAttempt records the sink's outcome for a prior // CompleteDeliveryAttempt records the sink's outcome for a prior
// BeginDeliveryAttempt. status is "sent" or "failed" — never "pending" or // BeginDeliveryAttempt. status is "sent", "failed" or "dropped" — never
// "unknown" (those are set only by Begin and reconciliation respectively). // "pending" or "unknown" (those are set only by Begin and reconciliation
// respectively).
func (s *Store) CompleteDeliveryAttempt(ctx context.Context, id int64, status string, now time.Time) error { func (s *Store) CompleteDeliveryAttempt(ctx context.Context, id int64, status string, now time.Time) error {
if status != DeliverySent && status != DeliveryFailed { if status != DeliverySent && status != DeliveryFailed && status != DeliveryDropped {
return fmt.Errorf("store: invalid delivery completion status %q", status) return fmt.Errorf("store: invalid delivery completion status %q", status)
} }
_, err := s.db.ExecContext(ctx, _, err := s.db.ExecContext(ctx,
+34
View File
@@ -0,0 +1,34 @@
package store
import (
"context"
"testing"
"time"
)
// TestDroppedDeliveryAttemptRoundTrips — Vikunja #370. A suppressed nudge is
// recorded as 'dropped'. The status column has a CHECK constraint, so this
// only works if migration #12 widened it; a fake outbox in a unit test would
// not catch that.
func TestDroppedDeliveryAttemptRoundTrips(t *testing.T) {
s := newTestStore(t)
ctx := context.Background()
now := time.Now()
id, err := s.BeginDeliveryAttempt(ctx, "nudge", "water", 0, "drop", "abc123", now)
if err != nil {
t.Fatalf("BeginDeliveryAttempt: %v", err)
}
if err := s.CompleteDeliveryAttempt(ctx, id, DeliveryDropped, now); err != nil {
t.Fatalf("CompleteDeliveryAttempt: %v", err)
}
var status string
err = s.db.QueryRowContext(ctx, `SELECT status FROM delivery_attempts WHERE id = ?`, id).Scan(&status)
if err != nil {
t.Fatalf("read back: %v", err)
}
if status != DeliveryDropped {
t.Fatalf("status: want %q, got %q", DeliveryDropped, status)
}
}
+24
View File
@@ -88,6 +88,30 @@ ALTER TABLE reminders ADD COLUMN next_fire_ts INTEGER;`, // #2
key TEXT PRIMARY KEY, key TEXT PRIMARY KEY,
value TEXT NOT NULL value TEXT NOT NULL
);`, // #11 — small key/value table for facts about the DB itself; first key is embedder_id (Vikunja #378) );`, // #11 — small key/value table for facts about the DB itself; first key is embedder_id (Vikunja #378)
// #12 — a suppressed nudge gets a 'dropped' row (Vikunja #370). sqlite
// can't widen a CHECK constraint in place, so the table is rebuilt; the
// index goes with the old table and is recreated. The columns are listed
// out rather than `SELECT *` — copying by position would silently shuffle
// every row if the old table's column order ever differed from this one.
`CREATE TABLE delivery_attempts_v12 (
id INTEGER PRIMARY KEY AUTOINCREMENT,
kind TEXT NOT NULL CHECK (kind IN ('nudge','reminder')),
rule TEXT NOT NULL DEFAULT '',
reminder_id INTEGER NOT NULL DEFAULT 0,
channel TEXT NOT NULL,
body_hash TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending','sent','failed','unknown','dropped')),
created_ts INTEGER NOT NULL,
completed_ts INTEGER
);
INSERT INTO delivery_attempts_v12
(id, kind, rule, reminder_id, channel, body_hash, status, created_ts, completed_ts)
SELECT id, kind, rule, reminder_id, channel, body_hash, status, created_ts, completed_ts
FROM delivery_attempts;
DROP TABLE delivery_attempts;
ALTER TABLE delivery_attempts_v12 RENAME TO delivery_attempts;
CREATE INDEX IF NOT EXISTS idx_delivery_attempts_status ON delivery_attempts (status);`,
} }
// migrate applies every migration with a number greater than the DB's current // migrate applies every migration with a number greater than the DB's current
+78 -3
View File
@@ -1,8 +1,10 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Unified script to stop all Maven services. # Unified script to stop all Maven services.
# Usage: ./kill-maven.sh # Usage: ./kill-maven.sh
# - Graceful SIGTERM is attempted first. # - Docker deploy: `docker compose stop` (see why below).
# - If any process lingers, force with SIGKILL. # - Bare-metal / dev run: graceful SIGTERM first, SIGKILL if anything lingers.
# Exits non-zero if it cannot confirm everything is stopped. It must never say
# "stopped" unless it checked.
set -euo pipefail set -euo pipefail
@@ -24,6 +26,73 @@ else
LLM='llama-server.*\.gguf' LLM='llama-server.*\.gguf'
fi fi
COMPOSE_FILE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/docker-compose.yml"
# --- containerised deploy ------------------------------------------------
# docker-compose.yml does not set `pid: host`, so each container has its own
# PID namespace: pkill on the host sees nothing inside them. This script used
# to print "all stopped" while every daemon was still happily running. Stop the
# containers through compose instead — that actually reaches them.
#
# running_containers prints the ids of the project's running containers, or
# nothing. Empty output plus a non-zero return means "could not ask docker",
# which is different from "nothing is running" and is handled below.
running_containers() {
docker compose -f "$COMPOSE_FILE" ps -q --status running 2>/dev/null
}
DOCKER_OK=0
CONTAINERS=""
if command -v docker >/dev/null 2>&1 && [ -f "$COMPOSE_FILE" ]; then
if CONTAINERS="$(running_containers)"; then
DOCKER_OK=1
fi
fi
if [ "$DOCKER_OK" = 1 ] && [ -n "$CONTAINERS" ]; then
echo "--- Maven is running in containers: stopping via docker compose ---"
if ! docker compose -f "$COMPOSE_FILE" stop; then
echo "ERROR: 'docker compose stop' failed. Containers may still be running." >&2
exit 1
fi
echo "--- Verifying containers are gone ---"
LEFT="$(running_containers || true)"
if [ -n "$LEFT" ]; then
echo "ERROR: containers still running after stop:" >&2
docker compose -f "$COMPOSE_FILE" ps >&2 || true
exit 1
fi
echo "All containers stopped."
exit 0
fi
# --- bare-metal / dev run -----------------------------------------------
# pgrep -f matches whole command lines, so a shell that merely mentions
# "mavend" (this script's own parent, for one) shows up. Drop ourselves and our
# parent, otherwise the SIGKILL sweep can take out the terminal you ran this in.
host_pids() {
pgrep -f "$PAT|$LLM" | grep -v -e "^$$\$" -e "^$PPID\$" | paste -sd, - || true
}
HOST_PIDS=$(host_pids)
if [ -z "$HOST_PIDS" ]; then
# Nothing on the host. Whether that means "already down" depends on whether
# we managed to ask docker, and the two must not read the same.
if [ "$DOCKER_OK" = 1 ]; then
# Docker answered and named no running containers, and there is nothing
# on the host either. That is a real answer: Maven is already stopped.
echo "Nothing to stop: no Maven processes and no running containers."
exit 0
fi
# We could not ask docker, so Maven may be alive in a container we cannot
# see. Saying "stopped" here is the exact false success this script had.
echo "ERROR: no Maven processes on this host, and docker could not be asked." >&2
echo " If this is the container deploy it may still be running:" >&2
echo " docker compose -f $COMPOSE_FILE stop" >&2
echo " Nothing was stopped. Check by hand before assuming Maven is down." >&2
exit 1
fi
echo "--- Sending graceful SIGTERM to Maven services ---" echo "--- Sending graceful SIGTERM to Maven services ---"
pkill -TERM -f "$PAT" || true pkill -TERM -f "$PAT" || true
# mavend's Pdeathsig SIGKILLs its llama-server on exit, but sweep strays too # mavend's Pdeathsig SIGKILLs its llama-server on exit, but sweep strays too
@@ -32,11 +101,17 @@ pkill -TERM -f "$LLM" || true
echo "--- Verifying processes are gone ---" echo "--- Verifying processes are gone ---"
sleep 1 sleep 1
PIDS=$(pgrep -d ',' -f "$PAT|$LLM") || PIDS="" PIDS=$(host_pids)
if [ -n "$PIDS" ]; then if [ -n "$PIDS" ]; then
echo "Warning: some processes still alive. PIDs: $PIDS" echo "Warning: some processes still alive. PIDs: $PIDS"
echo "--- Force killing with SIGKILL ---" echo "--- Force killing with SIGKILL ---"
echo "$PIDS" | tr ',' '\n' | xargs -r kill -9 echo "$PIDS" | tr ',' '\n' | xargs -r kill -9
sleep 1
LEFT=$(host_pids)
if [ -n "$LEFT" ]; then
echo "ERROR: still alive after SIGKILL. PIDs: $LEFT" >&2
exit 1
fi
echo "Done (SIGKILL)." echo "Done (SIGKILL)."
else else
echo "All services gracefully stopped." echo "All services gracefully stopped."