Compare commits

...

11 Commits

Author SHA1 Message Date
claude 865623ef3e phraser, mavend: read the fallbacks from the file (V-501)
The accessors are functions now, so the call sites that compared against one
literal compare against the entry instead: IsUnknownFallback and
IsSourcesFallback in the daemon tests, the entry key in the phraser tests. A
reworded variant no longer breaks a Go test.

The eval scores every variant on the persona checks the nudges already pass.
2026-08-04 01:19:41 +04:00
claude 4fdce3ca2c phraser: put the phrasing fallbacks in a versioned json (V-501)
Four lines he hears out loud lived as string literals in three Go files, so
rewording one meant a rebuild. They move to fallbacks_ru_v1.json on the shape
nudges_ru_v1.json already uses: embedded, schema-versioned, several variants,
never the same one twice running.

The gap phrase is marked fixed, because it names one specific missing model and
must not drift into a general "I do not know". Every accessor falls back to the
literal it replaced, including on a nil receiver: these strings exist because
something already failed, so a broken template file must not take her last
words away.
2026-08-04 01:19:41 +04:00
claude c47881106e phraser: say "даже не знаю, что сказать" when there is nothing to say (V-397)
Review of #108: "поговорили." reads as a summary of a conversation that did
not happen. One exported constant now, so the Stub, the LLMPhraser fallback
and the daemon all say the same thing.

internal/voice/replier.go keeps its own copy — that is the separate replier
seam, not this one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-04 00:50:46 +04:00
claude 9a70f7378b phraser: move errEmptyResponse next to its only caller (V-397)
It sat in world.go, which is about the workstation model; it is a phrasing
error and belongs in llmphraser.go. Also trims the PhraseQuery doc.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-04 00:47:31 +04:00
claude b18f608594 mavend, eval: use the phrasing errors the phraser now returns (V-397)
Call sites take the fallback text and log the error instead of treating a
canned string as success. phraseSource drops the text entirely — its callers
hold the passage and read it back better than "вот что я нашла: <passage>".

The talk scorer's before-and-after model probe (the #395 workaround) goes;
the run now fails only when every case errored, which is the honest
"nothing was measured" condition. TalkFixture gets its own schema version so
the two fixtures can be versioned apart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-04 00:41:16 +04:00
claude d1f8a734c5 phraser: report the failure next to the fallback (V-397)
PhraseChat and PhraseQuery returned canned text with a nil error, so a dead
or OOM-killed server was indistinguishable from bad phrasing — "не знаю." is
also a legitimate answer.

Both now return the fallback text AND the error. The daemon keeps using the
text, so the turn still survives; a measuring caller counts a real failure.
An empty response is its own error: the model is up and said nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-04 00:41:16 +04:00
kami 71041029e2 Merge pull request 'The reply path can't be tested — llmReplier is stuck in package main' (#107) from task/396-the-reply-path-can-t-be-tested-llmreplie into master
Reviewed-on: #107
2026-08-03 22:35:56 +02:00
claude 35018226ef eval: score the reply path, the fourth phrasing path (V-396)
Nine reply cases and a fourth column in the talk report. The reply path is a
separate object from the phraser in the daemon, so Pair joins a Talker and a
Confirmer for a run that covers everything Maven says.

Cases carry intent/key/value because the replier is phrased from the decision the
router resolved, not from the raw utterance. Three of them are baits the other
paths cannot produce: a masculine verb about himself that she must not copy onto
herself, a polite plural input that must still come back на ты, and an unresolved
note that invites a question a confirmation is not allowed to ask.

Not scored against a model here — this box has no llama-server, and the baseline
test is opt-in on MAVEN_LLM_URL.
2026-08-04 00:33:41 +04:00
claude 8833a9c76b mavend: keep only the stub floor in llmReplier (V-396)
The prompt, the call and the output parsing now live in internal/phraser. What is
left here is the one thing the daemon adds: a clarify, a model error and an
unusable generation all answer from voice.StubReplier, so a turn never breaks on
the model. The duplicated stripThink and parseResponseMood copies are gone;
capture.go uses phraser.StripThink.
2026-08-04 00:33:30 +04:00
claude 6c07409452 phraser: add Replier, the reply path lifted out of package main (V-396)
llmReplier lived in cmd/mavend, so the confirmation he hears after every fact,
note and reminder was the one phrasing path nothing could import or score.

Replier owns the prompt, the call and the parsing, and returns its errors instead
of hiding them — a dead model shows up as an error rather than as bad phrasing.
It has no stub fallback of its own; the daemon keeps that. StripThink is exported
for the daemon's own model callers.
2026-08-04 00:33:30 +04:00
kami 1c2541f7d6 Merge pull request 'llama-server holds 7.9GB RSS for a 1.1GB model, and its startup log goes nowhere' (#105) from task/496-recall-a-cross-language-question-loses-i into master
Reviewed-on: #105
2026-08-03 22:04:29 +02:00
21 changed files with 934 additions and 215 deletions
+6 -1
View File
@@ -40,6 +40,7 @@ import (
"context"
"log"
"github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/router"
)
@@ -58,10 +59,14 @@ func (h *reactiveHandler) actionChat(ctx context.Context, dec router.Decision) s
// Conversational: build history from dialogue session (prior user turns)
// and let the LLM respond from general knowledge + context.
history := h.chatHistory()
// The phraser hands back its own fallback text alongside the error, so the
// turn survives a dead server and the failure still reaches the log.
reply, err := h.phraser.PhraseChat(ctx, dec.Utterance, history)
if err != nil {
log.Printf("voice: chat: %v", err)
return "поговорили."
}
if reply == "" {
return phraser.ChatFallback()
}
return reply
}
+8 -2
View File
@@ -445,7 +445,13 @@ func (h *reactiveHandler) queryMemory(ctx context.Context, t *queryTurn) (string
// A note is phrased in Maven's voice; a fact is read back as it was
// stored.
if hit.Meta["type"] == "note" {
if reply, perr := h.phraser.PhraseQuery(ctx, t.dec.Utterance, []string{text}); perr == nil && reply != "" {
reply, perr := h.phraser.PhraseQuery(ctx, t.dec.Utterance, []string{text})
switch {
case perr != nil:
// Reading the note back verbatim beats the phraser's own fallback,
// which only wraps the same text in "вот что я нашла:".
log.Printf("voice: recall phrase: %v", perr)
case reply != "":
return reply, true
}
}
@@ -792,7 +798,7 @@ func (h *reactiveHandler) queryGeneral(ctx context.Context, t *queryTurn) (strin
reply, err := h.phraseWorld(ctx, t.dec.Utterance, nil)
if errors.Is(err, phraser.ErrNoWorldModel) {
log.Printf("voice: %q needs the world model and it is not available", t.dec.Utterance)
return worldGap, true
return worldGap(), true
}
if err != nil || reply == "" {
return "не знаю.", true
+1 -1
View File
@@ -100,7 +100,7 @@ func (l llmCompleter) Complete(ctx context.Context, system, user string) (string
// grammar, or a llama-server too old to honour one, gets the plain text it used
// to get rather than an empty meeting summary.
func unwrapSummary(raw string) string {
s := stripThink(strings.TrimSpace(raw))
s := phraser.StripThink(strings.TrimSpace(raw))
start := strings.Index(s, "{")
end := strings.LastIndex(s, "}")
if start < 0 || end <= start {
+3 -3
View File
@@ -121,8 +121,8 @@ func TestQueryRecallNoteCanWin(t *testing.T) {
{text: "выучил пару аккордов", score: 0.50, kind: "note"},
})
reply := askQuery(t, h, q)
if want := "вот что я нашла: молоко стоит в холодильнике"; reply != want {
t.Errorf("reply %q, want %q", reply, want)
if !phraser.IsSourcesFallback(reply, "молоко стоит в холодильнике") {
t.Errorf("reply %q, want the note read back", reply)
}
// One text, the winning memory's — the answer came from the memory
// pass, not from handing the phraser every note in the table.
@@ -151,7 +151,7 @@ func TestQueryRecallNoteCanWin(t *testing.T) {
{text: "молоко стоит в холодильнике", score: 0.860, kind: "note"},
{text: "молоко закончилось", score: 0.858, kind: "note"},
})
if reply := askQuery(t, h, q); reply != "не знаю." {
if reply := askQuery(t, h, q); !phraser.IsUnknownFallback(reply) {
t.Errorf("reply %q, want silence", reply)
}
})
+11 -100
View File
@@ -2,122 +2,33 @@ package main
import (
"context"
"encoding/json"
"strings"
"time"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/voice"
)
// completer is the LLM seam for the replier (subset of router.Completer).
// *llm.Client satisfies it.
type completer interface {
Complete(ctx context.Context, r llm.Req) (string, error)
}
// llmReplier phrases reactive confirmations with the resident model
// (Qwen3-1.7B). Stub is the
// floor on any error (offline-safe). Maven speaks as "she", feminine RU.
// llmReplier is the daemon-side wiring around phraser.Replier: it owns the
// deterministic floor, and nothing else. The phrasing itself, the prompt and the
// output parsing live in internal/phraser so the eval can score them (#396).
type llmReplier struct {
c completer
p *phraser.Replier
stub *voice.StubReplier
// block renders the shared context block per turn (who he is, the time).
// nil ⇒ the prompt stands alone.
block func() string
}
func newLLMReplier(c completer, block func() string) *llmReplier {
return &llmReplier{c: c, stub: voice.NewStubReplier(), block: block}
func newLLMReplier(c phraser.Completer, block func() string) *llmReplier {
return &llmReplier{p: phraser.NewReplier(c, block), stub: voice.NewStubReplier()}
}
const replySystem = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Владелец — мужчина, говоришь с ним на "ты", в единственном числе; никогда не "вы"/"ваш" и не "он"/"его". Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), по-русски, спокойно и без официальных формулировок. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"}
Никогда не пиши "..." в поле response.`
// Reply never fails: a clarify, a model error and an unusable generation all
// answer from the stub, which is what keeps a turn from breaking on the model.
func (r *llmReplier) Reply(d router.Decision) string {
if d.Clarify {
return r.stub.Reply(d)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
out, err := r.c.Complete(ctx, llm.Req{
System: persona.Prepend(r.block, replySystem),
User: replyContext(d),
Grammar: phraser.ResponseGrammar,
MaxTokens: 512,
RepeatPenalty: 1.3,
})
if err != nil {
out, err := r.p.PhraseReply(context.Background(), d)
if err != nil || out == "" {
return r.stub.Reply(d)
}
out = stripThink(out)
if response, _ := parseResponseMood(out); response != "" {
return response
}
// fallback: try plain-text parsing
if out = firstSentence(out); out != "" {
return out
}
return r.stub.Reply(d)
}
// firstSentence trims the model's output to a single clean confirmation: first
// line, first sentence, whitespace-normalized — the last-line defense against a
// small model that rambles past the first period despite the prompt + stop.
// stripThink removes the <think> block that Thinking-variant models emit.
func stripThink(s string) string {
if i := strings.LastIndex(s, "</think>"); i >= 0 {
s = strings.TrimSpace(s[i+8:])
}
return s
}
func firstSentence(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexByte(s, '\n'); i >= 0 {
s = s[:i]
}
// keep up to and including the first sentence-ending punctuation.
if i := strings.IndexAny(s, ".!?"); i >= 0 {
s = s[:i+1]
}
return strings.TrimSpace(s)
}
// parseResponseMood extracts {"response","mood"} from LLM output, tolerant
// of thinking tokens and extra text before/after the JSON block.
func parseResponseMood(raw string) (response, mood string) {
cleaned := strings.TrimSpace(raw)
start := strings.Index(cleaned, "{")
end := strings.LastIndex(cleaned, "}")
if start < 0 || end < 0 || end <= start {
return "", ""
}
var parsed struct {
Response string `json:"response"`
Mood string `json:"mood"`
}
if err := json.Unmarshal([]byte(cleaned[start:end+1]), &parsed); err != nil {
return "", ""
}
return parsed.Response, parsed.Mood
}
// replyContext renders the decision into a compact RU description for the model.
func replyContext(d router.Decision) string {
switch d.Intent {
case router.IntentFact:
return "записала факт: " + d.Slots.Key + " " + d.Slots.Value
case router.IntentNote:
return "сохранила заметку: " + d.Slots.Text
case router.IntentReminder:
return "поставила напоминание: " + d.Slots.Text
default:
return string(d.Intent) + ": " + d.Slots.Text
}
return out
}
+20 -50
View File
@@ -5,28 +5,22 @@ import (
"testing"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/router"
"github.com/kami/maven/internal/voice"
)
type mockCompleter struct {
// The phrasing itself is tested in internal/phraser. What is left here is the
// only thing the daemon adds: the stub floor, on the three ways a reply can
// fail to arrive.
type stubCompleter struct {
out string
err error
}
func (m mockCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return m.out, m.err }
func (s stubCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return s.out, s.err }
func TestLLMReplierReturnsLLMReply(t *testing.T) {
r := newLLMReplier(mockCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}, nil)
got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
if got != "записала, кофе закончился" {
t.Errorf("got %q, want %q", got, "записала, кофе закончился")
}
}
func TestLLMReplierFallsBackToPlainText(t *testing.T) {
r := newLLMReplier(mockCompleter{out: "записала, кофе закончился"}, nil)
func TestLLMReplierPassesTheModelReplyThrough(t *testing.T) {
r := newLLMReplier(stubCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}, nil)
got := r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
if got != "записала, кофе закончился" {
t.Errorf("got %q, want %q", got, "записала, кофе закончился")
@@ -34,54 +28,30 @@ func TestLLMReplierFallsBackToPlainText(t *testing.T) {
}
func TestLLMReplierFallsBackToStubOnError(t *testing.T) {
r := newLLMReplier(mockCompleter{err: errTestLLMDown}, nil)
noteDec := router.Decision{Intent: router.IntentNote}
got := r.Reply(noteDec)
want := voice.NewStubReplier().Reply(noteDec)
if got != want {
t.Errorf("on llm error: got %q, want stub %q", got, want)
}
r := newLLMReplier(stubCompleter{err: errReplierTest}, nil)
assertStub(t, r, router.Decision{Intent: router.IntentNote}, "llm error")
}
func TestLLMReplierFallsBackToStubOnEmpty(t *testing.T) {
r := newLLMReplier(mockCompleter{out: ""}, nil)
noteDec := router.Decision{Intent: router.IntentNote}
got := r.Reply(noteDec)
want := voice.NewStubReplier().Reply(noteDec)
if got != want {
t.Errorf("on empty llm: got %q, want stub %q", got, want)
}
r := newLLMReplier(stubCompleter{out: ""}, nil)
assertStub(t, r, router.Decision{Intent: router.IntentNote}, "empty llm")
}
func TestLLMReplierClarifyUsesStub(t *testing.T) {
r := newLLMReplier(mockCompleter{out: "я всё поняла"}, nil)
clarifyDec := router.Decision{Clarify: true}
got := r.Reply(clarifyDec)
want := voice.NewStubReplier().Reply(clarifyDec)
r := newLLMReplier(stubCompleter{out: "я всё поняла"}, nil)
assertStub(t, r, router.Decision{Clarify: true}, "clarify")
}
func assertStub(t *testing.T, r *llmReplier, d router.Decision, what string) {
t.Helper()
got, want := r.Reply(d), voice.NewStubReplier().Reply(d)
if got != want {
t.Errorf("on clarify: got %q, want stub %q", got, want)
t.Errorf("on %s: got %q, want stub %q", what, got, want)
}
}
var errTestLLMDown = errTest("llm down")
var errReplierTest = errTest("llm down")
type errTest string
func (e errTest) Error() string { return string(e) }
// grammarRecorder captures the request so the grammar can be asserted on.
type grammarRecorder struct{ req llm.Req }
func (g *grammarRecorder) Complete(_ context.Context, r llm.Req) (string, error) {
g.req = r
return `{"response":"записала","mood":"neutral"}`, nil
}
func TestLLMReplierCarriesTheResponseGrammar(t *testing.T) {
rec := &grammarRecorder{}
r := newLLMReplier(rec, nil)
r.Reply(router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}})
if rec.req.Grammar != phraser.ResponseGrammar {
t.Errorf("grammar = %q, want phraser.ResponseGrammar", rec.req.Grammar)
}
}
+9 -1
View File
@@ -23,7 +23,11 @@ type worldPhraser interface {
// question about his meeting came back as a swimming competition in Nottingham.
// Naming the gap is the rule CLAUDE.md already applies to a sibling service
// being down.
const worldGap = "сейчас не могу ответить — большая модель недоступна, а придумывать не хочу."
//
// The wording lives in fallbacks_ru_v1.json and is fixed there, not picked from
// variants: this sentence names one specific gap and must not drift into a
// general "I don't know".
func worldGap() string { return phraser.WorldGap() }
// phraseWorld asks the world model, or reports the gap.
//
@@ -54,7 +58,11 @@ func (h *reactiveHandler) phraseSource(ctx context.Context, name, utterance stri
log.Printf("voice: %s: no world model, reading the source back instead", name)
return ""
case err != nil:
// The resident phraser answers this call with its fallback text and the
// error together. Drop the text: these callers hold the passage itself
// and read it back better than "вот что я нашла: <passage>" does.
log.Printf("voice: %s: phrase: %v", name, err)
return ""
}
return reply
}
+6 -6
View File
@@ -35,7 +35,7 @@ func TestQueryGeneralNamesTheGap(t *testing.T) {
if !ok {
t.Fatal("queryGeneral passed on the last source in the chain")
}
if reply != worldGap {
if reply != worldGap() {
t.Fatalf("reply = %q, want the named gap", reply)
}
if g.worldCalls != 1 {
@@ -51,7 +51,7 @@ func TestQueryGeneralWithoutAWorldModelIsUnchanged(t *testing.T) {
if !ok {
t.Fatal("queryGeneral passed on the last source in the chain")
}
if reply != "не знаю." {
if !phraser.IsUnknownFallback(reply) {
t.Fatalf("reply = %q, want the Stub's answer", reply)
}
}
@@ -61,12 +61,12 @@ func TestQueryGeneralWithoutAWorldModelIsUnchanged(t *testing.T) {
// English in it.
func TestWorldGapIsInPersona(t *testing.T) {
for _, bad := range []string{"вы", "ваш", "рад ", "дорогой", "милый"} {
if strings.Contains(worldGap, bad) {
t.Errorf("the gap phrase contains %q: %s", bad, worldGap)
if strings.Contains(worldGap(), bad) {
t.Errorf("the gap phrase contains %q: %s", bad, worldGap())
}
}
if strings.ContainsAny(worldGap, "abcdefghijklmnopqrstuvwxyz") {
t.Errorf("the gap phrase has Latin letters in it: %s", worldGap)
if strings.ContainsAny(worldGap(), "abcdefghijklmnopqrstuvwxyz") {
t.Errorf("the gap phrase has Latin letters in it: %s", worldGap())
}
}
+40
View File
@@ -0,0 +1,40 @@
package eval
import (
"math/rand"
"strings"
"testing"
"github.com/kami/maven/internal/phraser"
)
// TestFallbackPersona scores every line in fallbacks_ru_v1.json on the persona
// checks the nudges are already held to. These lines are heard out loud, and
// they live in a JSON file now, so a reworded variant that says "рад" or "вы"
// would otherwise reach him with nothing between it and the speaker.
//
// Only the persona checks run. Mood and topic belong to a nudge, and these are
// not nudges: they are what she says when there is no answer.
func TestFallbackPersona(t *testing.T) {
fb, err := phraser.LoadFallbacks(rand.NewSource(20260804))
if err != nil {
t.Fatalf("LoadFallbacks: %v", err)
}
persona := map[string]bool{
CheckLang: true, CheckFeminine: true, CheckHisGender: true,
CheckAddress: true, CheckCringe: true, CheckLength: true,
}
variants := fb.Variants()
if len(variants) == 0 {
t.Fatal("no variants — the file loaded empty")
}
for _, v := range variants {
// {sources} stands for his own notes and never carries persona of its own.
body := strings.ReplaceAll(v, "{sources}", "два литра")
for _, r := range RunChecks(Case{}, body, "neutral") {
if persona[r.Name] && !r.Pass {
t.Errorf("%q fails %s: %s", v, r.Name, r.Detail)
}
}
}
}
+55 -6
View File
@@ -26,20 +26,22 @@ import (
"time"
"github.com/kami/maven/internal/dialogue"
"github.com/kami/maven/internal/router"
)
//go:embed talk_v1.json
var talkFixtureJSON []byte
// The three phrasing paths under test. Values match the fixture's "path" field.
// The phrasing paths under test. Values match the fixture's "path" field.
const (
PathChat = "chat" // PhraseChat
PathQuery = "query" // PhraseQuery with notes
PathKnowledge = "knowledge" // PhraseQuery with no notes
PathReply = "reply" // PhraseReply, the reactive confirmation
)
// TalkPaths — report order.
var TalkPaths = []string{PathChat, PathQuery, PathKnowledge}
var TalkPaths = []string{PathChat, PathQuery, PathKnowledge, PathReply}
// TalkCheckNames — the checks that apply to a free-form reply, in report order.
// Deliberately a subset of CheckNames: length, mood and "no questions" are nudge
@@ -58,17 +60,30 @@ var TalkCheckNames = []string{
// WantAny is the on-topic contract: at least one lowercased fragment must appear
// in the reply. Fragments are stems ("пароль" → "парол") so declension does not
// defeat them.
//
// Intent, Key and Value carry the reply path's decision: that path is phrased
// from what the router already resolved, not from the raw utterance. Utterance
// stays filled anyway, because it is what a human reads in the report.
type TalkCase struct {
ID string `json:"id"`
Path string `json:"path"`
Utterance string `json:"utterance"`
History []string `json:"history,omitempty"`
Notes []string `json:"notes,omitempty"`
Intent string `json:"intent,omitempty"`
Key string `json:"key,omitempty"`
Value string `json:"value,omitempty"`
WantAny []string `json:"want_any"`
Tags []string `json:"tags,omitempty"`
Note string `json:"note,omitempty"`
}
// TalkSchemaVersion — the version this loader understands. Separate from the
// nudge fixture's SchemaVersion: the two fixtures have different shapes and
// change on different days, and one shared constant would force a bump on the
// fixture that did not move.
const TalkSchemaVersion = 1
// TalkFixture — the versioned envelope, same gating as Fixture.
type TalkFixture struct {
SchemaVersion int `json:"schema_version"`
@@ -83,8 +98,8 @@ func LoadTalk() (TalkFixture, error) {
if err := json.Unmarshal(talkFixtureJSON, &f); err != nil {
return TalkFixture{}, fmt.Errorf("parse talk fixture: %w", err)
}
if f.SchemaVersion != SchemaVersion {
return TalkFixture{}, fmt.Errorf("talk fixture schema_version %d, want %d", f.SchemaVersion, SchemaVersion)
if f.SchemaVersion != TalkSchemaVersion {
return TalkFixture{}, fmt.Errorf("talk fixture schema_version %d, want %d", f.SchemaVersion, TalkSchemaVersion)
}
if len(f.Cases) == 0 {
return TalkFixture{}, fmt.Errorf("talk fixture has no cases")
@@ -92,13 +107,27 @@ func LoadTalk() (TalkFixture, error) {
return f, nil
}
// Talker — the two methods a conversational path must have to be scorable.
// *phraser.LLMPhraser satisfies it; same trick as Nudger.
// Talker — the methods a conversational path must have to be scorable.
// *phraser.LLMPhraser satisfies the first two; *phraser.Replier satisfies the
// third, so a run that scores all four paths passes a Pair.
type Talker interface {
PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error)
PhraseQuery(ctx context.Context, utterance string, notes []string) (string, error)
}
// Confirmer — the reply path. *phraser.Replier satisfies it.
type Confirmer interface {
PhraseReply(ctx context.Context, d router.Decision) (string, error)
}
// Pair joins the two objects the daemon wires separately — the phraser and the
// replier — so one ScoreTalk call covers every path Maven speaks through. A bare
// Talker still works; its reply cases score as errors, which is honest.
type Pair struct {
Talker
Confirmer
}
// TalkOutcome — one scored case.
type TalkOutcome struct {
Case TalkCase
@@ -194,10 +223,30 @@ func (c TalkCase) run(ctx context.Context, t Talker) (string, error) {
return t.PhraseQuery(ctx, c.Utterance, c.Notes)
case PathKnowledge:
return t.PhraseQuery(ctx, c.Utterance, nil)
case PathReply:
conf, ok := t.(Confirmer)
if !ok {
return "", fmt.Errorf("target cannot phrase replies — pass a Pair")
}
return conf.PhraseReply(ctx, c.decision())
}
return "", fmt.Errorf("unknown path %q", c.Path)
}
// decision rebuilds what the router would have handed the replier. Text is the
// utterance for a note or a reminder, which is what the router puts there.
func (c TalkCase) decision() router.Decision {
return router.Decision{
Intent: router.Intent(c.Intent),
Slots: router.Slots{
Key: c.Key,
Value: c.Value,
Text: c.Utterance,
HasKey: c.Key != "",
},
}
}
func (c TalkCase) turns() []dialogue.Turn {
turns := make([]dialogue.Turn, 0, len(c.History))
for _, h := range c.History {
+28 -18
View File
@@ -11,6 +11,7 @@ import (
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/phraser"
"github.com/kami/maven/internal/router"
)
// perPathMinimum — the resolution floor. A per-path score built on a handful of
@@ -36,6 +37,10 @@ func TestTalkFixture(t *testing.T) {
switch c.Path {
case PathChat, PathQuery, PathKnowledge:
case PathReply:
if c.Intent == "" {
t.Errorf("%s: reply case has no intent — the replier is phrased from the decision", c.ID)
}
default:
t.Errorf("%s: unknown path %q", c.ID, c.Path)
}
@@ -69,10 +74,15 @@ type fakeTalker struct{ reply string }
func (f fakeTalker) PhraseChat(context.Context, string, []dialogue.Turn) (string, error) {
return f.reply, nil
}
func (f fakeTalker) PhraseQuery(context.Context, string, []string) (string, error) {
return f.reply, nil
}
func (f fakeTalker) PhraseReply(context.Context, router.Decision) (string, error) {
return f.reply, nil
}
// TestScoreTalkCounts — a reply that fails on purpose must be counted on every
// path, so a real run cannot report a hidden zero.
func TestScoreTalkCounts(t *testing.T) {
@@ -104,7 +114,7 @@ func TestScoreTalkCounts(t *testing.T) {
}
}
// TestLLMTalkBaseline — the resident model on the three conversational paths.
// TestLLMTalkBaseline — the resident model on all four phrasing paths.
// Opt-in exactly like TestLLMPhrasingBaseline: CI has no model and a run costs
// minutes on the CPU target.
//
@@ -132,32 +142,32 @@ func TestLLMTalkBaseline(t *testing.T) {
p := phraser.NewLLMPhraserAt(base, cfg)
defer p.Close()
// Unreachable server is fatal here, not a logged warning, and that differs
// from the nudge test on purpose. PhraseNudge returns its errors, so a dead
// server there shows up honestly in the Errors column. PhraseChat and
// PhraseQuery do NOT: they swallow every failure and return a canned string
// ("поговорили.", "не знаю.", "вот что я нашла: …"). So on these three paths
// a dead server produces a full report with 0 errors and a terrible score —
// a number that looks like bad phrasing and is really no phrasing at all.
// Refusing to score without a confirmed model is the only guard available
// until the phraser reports its failures (Vikunja #397).
// The model id names the run in the report. Since Vikunja #397 every path
// returns its errors, so a server that dies mid-run shows up in the Errors
// column instead of scoring as bad phrasing — the before-and-after probe that
// used to stand in for that is gone.
model, err := llm.ModelID(ctx, base)
if err != nil {
t.Fatalf("no model at %s: %v — refusing to score, these paths hide their errors "+
"and would report a plausible-looking result off a dead server", base, err)
t.Fatalf("no model at %s: %v", base, err)
}
t.Logf("scoring model %s at %s", model, base)
rep, err := ScoreTalk(ctx, "llm ("+model+", built-in persona)", p, f)
// The reply path is a separate object in the daemon too: the phraser owns its
// own llama-server, the replier is handed an llm.Client. Pair scores both.
block := func() string { return persona.Facts{}.Block(time.Now()) }
target := Pair{Talker: p, Confirmer: phraser.NewReplier(llm.New(base, cfg.Timeout), block)}
rep, err := ScoreTalk(ctx, "llm ("+model+", built-in persona)", target, f)
if err != nil {
t.Fatalf("ScoreTalk: %v", err)
}
t.Log("\n" + rep.String() + "\nreplies:\n" + rep.Replies() + "\nfailures:\n" + rep.Failures())
// And again afterwards: the run takes minutes, and a server that died or got
// OOM-killed halfway through would leave the first cases scored and the rest
// silently canned. Checking only at the start would not catch that.
if _, err := llm.ModelID(ctx, base); err != nil {
t.Fatalf("model at %s went away during the run: %v — the score above is not trustworthy", base, err)
// A run where nothing was phrased is not a low score, it is no measurement.
if rep.Errors == rep.Total {
t.Fatalf("every case errored — nothing was measured, the score above is not a phrasing result")
}
if rep.Errors > 0 {
t.Logf("%d/%d cases errored — those are model failures, not phrasing failures", rep.Errors, rep.Total)
}
}
+84
View File
@@ -222,6 +222,90 @@
"utterance": "почему гром слышно позже молнии?",
"want_any": ["звук", "све", "быстр", "гром", "молни"],
"tags": ["general"]
},
{
"id": "reply-fact-coffee",
"path": "reply",
"intent": "fact",
"key": "кофе",
"value": "закончился",
"utterance": "кофе закончился",
"want_any": ["коф"],
"tags": ["fact"],
"note": "The plainest confirmation there is, and the sentence he hears most often."
},
{
"id": "reply-fact-weight",
"path": "reply",
"intent": "fact",
"key": "вес",
"value": "82",
"utterance": "мой вес 82",
"want_any": ["вес", "82"],
"tags": ["fact", "number"],
"note": "A number must survive into the confirmation; a paraphrase that drops it is useless."
},
{
"id": "reply-fact-pill",
"path": "reply",
"intent": "fact",
"key": "таблетки",
"value": "выпил",
"utterance": "таблетки выпил",
"want_any": ["таблетк"],
"tags": ["fact", "feminine"],
"note": "He says 'выпил', masculine and about himself. She must not copy the form onto herself."
},
{
"id": "reply-note-router",
"path": "reply",
"intent": "note",
"utterance": "роутер перезагружается сам по ночам",
"want_any": ["роутер"],
"tags": ["note"]
},
{
"id": "reply-note-long",
"path": "reply",
"intent": "note",
"utterance": "если диск снова отвалится, посмотреть кабель, а не контроллер, в прошлый раз был кабель",
"want_any": ["диск", "кабел"],
"tags": ["note", "length"],
"note": "A long note baits a long confirmation. One sentence is the contract."
},
{
"id": "reply-reminder-evening",
"path": "reply",
"intent": "reminder",
"utterance": "напомни вечером полить цветы",
"want_any": ["цвет", "полит", "вечер"],
"tags": ["reminder"]
},
{
"id": "reply-reminder-tomorrow",
"path": "reply",
"intent": "reminder",
"utterance": "напомни завтра позвонить в поликлинику",
"want_any": ["поликлиник", "позвон", "звон"],
"tags": ["reminder"]
},
{
"id": "reply-formality-bait",
"path": "reply",
"intent": "note",
"utterance": "запишите пожалуйста что счётчики я сдал",
"want_any": ["счётчик", "счетчик"],
"tags": ["note", "persona-bait", "address"],
"note": "Polite plural in the input. The confirmation must still be на ты."
},
{
"id": "reply-question-bait",
"path": "reply",
"intent": "note",
"utterance": "надо купить фильтр для воды, не помню какой",
"want_any": ["фильтр"],
"tags": ["note", "no-question"],
"note": "An unresolved note invites her to ask which filter. A confirmation does not ask."
}
]
}
+89
View File
@@ -0,0 +1,89 @@
package phraser
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// isFallback — the text she says is picked from that entry's variants, so a test
// pins the entry rather than the wording. Pinning one line would make editing
// fallbacks_ru_v1.json break Go tests, which is the coupling this file removed.
func isFallback(t *testing.T, key, sources, got string) bool {
t.Helper()
e, ok := DefaultFallbacks().file.Entries[key]
if !ok {
t.Fatalf("no fallback entry %q", key)
}
for _, v := range e.Variants {
if strings.ReplaceAll(v, "{sources}", sources) == got {
return true
}
}
return false
}
// A dead server must be distinguishable from bad phrasing. Both PhraseChat and
// PhraseQuery keep the turn alive with canned text — and every one of those
// lines is also a legitimate reply, so the text alone cannot say which happened.
// The error is the only signal, and before Vikunja #397 it was dropped: the talk
// scorer reported a full run with zero errors off a server that answered nothing.
func TestPhrasingReportsTheFailureWithTheFallback(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "model not loaded", http.StatusServiceUnavailable)
}))
t.Cleanup(srv.Close)
p := NewLLMPhraserAt(srv.URL, Config{})
cases := []struct {
name string
call func() (string, error)
key string
sources string
}{
{"chat", func() (string, error) {
return p.PhraseChat(context.Background(), "как дела", nil)
}, fbChat, ""},
{"knowledge", func() (string, error) {
return p.PhraseQuery(context.Background(), "кто написал войну и мир", nil)
}, fbQueryUnknown, ""},
{"evidence", func() (string, error) {
return p.PhraseQuery(context.Background(), "сколько воды я выпил", []string{"два литра"})
}, fbQuerySources, "два литра"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
got, err := c.call()
if err == nil {
t.Fatalf("no error from a dead server; the scorer would count this as bad phrasing")
}
if !isFallback(t, c.key, c.sources, got) {
t.Errorf("fallback text = %q, want a %q variant — the daemon still has to say something", got, c.key)
}
})
}
}
// An empty answer is a failure too: the server is up and produced no tokens,
// which is not an answer and must not score as one.
func TestEmptyKnowledgeAnswerIsAnError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(`{"choices":[{"message":{"content":""}}]}`))
}))
t.Cleanup(srv.Close)
p := NewLLMPhraserAt(srv.URL, Config{})
got, err := p.PhraseQuery(context.Background(), "кто написал войну и мир", nil)
if err == nil {
t.Fatal("an empty response scored as an answer")
}
if !isFallback(t, fbQueryUnknown, "", got) {
t.Errorf("fallback text = %q, want a %q variant", got, fbQueryUnknown)
}
if !strings.Contains(err.Error(), "empty") {
t.Errorf("error = %v; want it to name the empty response", err)
}
}
+238
View File
@@ -0,0 +1,238 @@
package phraser
// The phrasing fallbacks — what she says when the model gave her nothing usable.
//
// They were four string literals spread across phraser.go, llmphraser.go and
// cmd/mavend/worldmodel.go. Every one of them is a line he hears out loud, so
// rewording one was a Go edit, a rebuild and a redeploy for what is product copy.
// This is the same shape nudges_ru_v1.json already uses for nudges: embedded,
// schema-versioned, several variants, and never the same variant twice running.
//
// The floor under the floor is deliberate. These strings exist because something
// already failed, so a broken template file must not be able to take the last
// words she has: every accessor falls back to the literal it replaced.
import (
_ "embed"
"encoding/json"
"fmt"
"log"
"math/rand"
"strings"
"sync"
"time"
)
//go:embed fallbacks_ru_v1.json
var fallbackJSON []byte
// FallbackSchemaVersion — the version this code understands. Its own constant,
// not shared with the nudge templates or the eval fixtures: two files that change
// on different days cannot be versioned by one number (Vikunja #397).
const FallbackSchemaVersion = 1
// The entry keys. Every one of them is read by a method below, so a typo in the
// file is caught at load rather than at the moment she needs the words.
const (
fbChat = "chat"
fbQueryUnknown = "query_unknown"
fbQuerySources = "query_sources"
fbWorldGap = "world_gap"
)
// fbKeys — every key the code requires the file to define.
var fbKeys = []string{fbChat, fbQueryUnknown, fbQuerySources, fbWorldGap}
// hardFloor — the literal each key falls back to when the file is unusable.
// These are the exact strings that lived in Go before this file existed.
var hardFloor = map[string]string{
fbChat: "даже не знаю, что сказать.",
fbQueryUnknown: "не знаю.",
fbQuerySources: "вот что я нашла: {sources}",
fbWorldGap: "сейчас не могу ответить — большая модель недоступна, а придумывать не хочу.",
}
type fallbackEntry struct {
// Fixed — one variant, never picked between. For wording that must not drift
// from turn to turn, like the gap phrase that names an unavailable model.
Fixed bool `json:"fixed"`
Variants []string `json:"variants"`
}
type fallbackFile struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
Notes []string `json:"notes"`
Entries map[string]fallbackEntry `json:"entries"`
}
// Fallbacks picks a hand-written Russian fallback line.
//
// Safe for concurrent use. Never the same variant twice in a row for the same
// entry: hearing the identical words every time a request fails is how a failure
// stops registering as one.
type Fallbacks struct {
mu sync.Mutex
rnd *rand.Rand
last map[string]string
file fallbackFile
}
// LoadFallbacks reads the embedded file. Pass a source to make the picking
// reproducible in tests; nil seeds from the clock.
func LoadFallbacks(src rand.Source) (*Fallbacks, error) {
var f fallbackFile
if err := json.Unmarshal(fallbackJSON, &f); err != nil {
return nil, fmt.Errorf("fallbacks: parse: %w", err)
}
if f.SchemaVersion != FallbackSchemaVersion {
return nil, fmt.Errorf("fallbacks: schema_version %d, want %d",
f.SchemaVersion, FallbackSchemaVersion)
}
for _, k := range fbKeys {
e, ok := f.Entries[k]
if !ok || len(e.Variants) == 0 {
return nil, fmt.Errorf("fallbacks: entry %q is missing or empty", k)
}
if e.Fixed && len(e.Variants) != 1 {
return nil, fmt.Errorf("fallbacks: entry %q is fixed but has %d variants", k, len(e.Variants))
}
}
// query_sources is the one entry whose whole job is to read something back,
// so a variant without the placeholder would silently drop the sources.
for _, v := range f.Entries[fbQuerySources].Variants {
if !strings.Contains(v, "{sources}") {
return nil, fmt.Errorf("fallbacks: %q variant %q does not use {sources}", fbQuerySources, v)
}
}
if src == nil {
src = rand.NewSource(time.Now().UnixNano())
}
return &Fallbacks{rnd: rand.New(src), last: map[string]string{}, file: f}, nil
}
// text returns one variant for key, with {sources} filled in. A nil receiver is
// the unloadable-file case and answers from hardFloor, so the caller never has
// to check whether the templates loaded.
func (f *Fallbacks) text(key, sources string) string {
tmpl := hardFloor[key]
if f != nil {
if e, ok := f.file.Entries[key]; ok && len(e.Variants) > 0 {
tmpl = f.pick(key, e)
}
}
return strings.ReplaceAll(tmpl, "{sources}", sources)
}
// pick chooses at random, skipping whatever this entry said last time.
func (f *Fallbacks) pick(key string, e fallbackEntry) string {
f.mu.Lock()
defer f.mu.Unlock()
choices := e.Variants
if len(choices) > 1 {
fresh := make([]string, 0, len(choices))
for _, v := range choices {
if v != f.last[key] {
fresh = append(fresh, v)
}
}
if len(fresh) > 0 {
choices = fresh
}
}
got := choices[f.rnd.Intn(len(choices))]
f.last[key] = got
return got
}
// Chat — nothing usable came back on the chat path.
func (f *Fallbacks) Chat() string { return f.text(fbChat, "") }
// Unknown — a question she cannot answer and will not guess at.
func (f *Fallbacks) Unknown() string { return f.text(fbQueryUnknown, "") }
// FromSources — read back what she was handed, because phrasing it failed.
func (f *Fallbacks) FromSources(sources string) string {
return f.text(fbQuerySources, sources)
}
// WorldGap — the world model is the one configured to answer and it is not
// answering. Fixed wording: it names a specific gap, and a variant set here
// would let "the big model is asleep" drift into "I don't know".
func (f *Fallbacks) WorldGap() string { return f.text(fbWorldGap, "") }
// Variants returns every line the file can produce, for the persona scorer.
// Order is stable so a failure names the same variant twice running.
func (f *Fallbacks) Variants() []string {
var out []string
for _, k := range fbKeys {
out = append(out, f.file.Entries[k].Variants...)
}
return out
}
// The process-wide instance. Package-level because these lines are needed on
// paths that have no phraser to hand — cmd/mavend names the world gap without
// one — and because a template file that is embedded and validated at load has
// nothing per-instance to configure.
var (
fallbackOnce sync.Once
fallbacks *Fallbacks
)
// DefaultFallbacks returns the shared instance, loading it on first use. A
// broken file logs once and leaves a nil *Fallbacks, which still answers from
// hardFloor — a daemon must not fail to boot over its own copy deck.
func DefaultFallbacks() *Fallbacks {
fallbackOnce.Do(func() {
fb, err := LoadFallbacks(nil)
if err != nil {
log.Printf("phraser: fallbacks unavailable, using the built-in lines: %v", err)
return
}
fallbacks = fb
})
return fallbacks
}
// ChatFallback — what she says when the chat path produced nothing.
func ChatFallback() string { return DefaultFallbacks().Chat() }
// UnknownFallback — what she says when she has no answer and will not invent one.
func UnknownFallback() string { return DefaultFallbacks().Unknown() }
// SourcesFallback — read the sources back rather than ship a broken fragment.
func SourcesFallback(sources string) string { return DefaultFallbacks().FromSources(sources) }
// WorldGap — what he hears when the world model is configured and unreachable.
func WorldGap() string { return DefaultFallbacks().WorldGap() }
// matches reports whether text is a line the given entry could have produced.
// A caller that has to recognise a fallback cannot compare against one literal
// any more, because the entry picks between variants.
func (f *Fallbacks) matches(key, sources, text string) bool {
if strings.ReplaceAll(hardFloor[key], "{sources}", sources) == text {
return true
}
if f == nil {
return false
}
for _, v := range f.file.Entries[key].Variants {
if strings.ReplaceAll(v, "{sources}", sources) == text {
return true
}
}
return false
}
// IsUnknownFallback reports whether text is one of her "I do not know" lines.
// The daemon tests read it to tell an answer from a shrug.
func IsUnknownFallback(text string) bool {
return DefaultFallbacks().matches(fbQueryUnknown, "", text)
}
// IsSourcesFallback reports whether text is sources read back verbatim.
func IsSourcesFallback(text, sources string) bool {
return DefaultFallbacks().matches(fbQuerySources, sources, text)
}
+42
View File
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"name": "russian phrasing fallbacks v1",
"notes": [
"What she says when the model gave her nothing usable. Edit the wording here, no Go changes needed.",
"Rules: she is feminine about herself, he is a man addressed as ты. Never вы/вас/ваш, never plural imperatives, never он/его about him. No pet names.",
"These are heard after a failure, so they stay short and admit the gap. None of them may claim knowledge she does not have.",
"Placeholders: {sources} the notes or passages she was handed. A variant whose placeholder has no value is skipped, so every entry needs at least one variant with no placeholder — except query_sources, which exists only to read sources back.",
"fixed: true means exactly one variant and no picking. Used where the wording is load-bearing and must not drift between turns."
],
"entries": {
"chat": {
"variants": [
"даже не знаю, что сказать.",
"не могу найти слов.",
"мысль ускользнула, повтори?",
"у меня сейчас пусто в голове."
]
},
"query_unknown": {
"variants": [
"не знаю.",
"не знаю, честно.",
"тут я пас.",
"не скажу, не знаю."
]
},
"query_sources": {
"variants": [
"вот что я нашла: {sources}",
"нашла вот это: {sources}",
"есть только это: {sources}"
]
},
"world_gap": {
"fixed": true,
"variants": [
"сейчас не могу ответить — большая модель недоступна, а придумывать не хочу."
]
}
}
}
+57
View File
@@ -0,0 +1,57 @@
package phraser
import (
"math/rand"
"strings"
"testing"
)
// The embedded file must load, or the daemon speaks from hardFloor and nobody
// finds out until he hears the wrong words.
func TestFallbacksLoad(t *testing.T) {
fb, err := LoadFallbacks(rand.NewSource(1))
if err != nil {
t.Fatalf("LoadFallbacks: %v", err)
}
if got := fb.FromSources("два литра"); !strings.Contains(got, "два литра") {
t.Errorf("FromSources = %q, want the sources in it", got)
}
if fb.WorldGap() != hardFloor[fbWorldGap] {
t.Errorf("WorldGap = %q, want the fixed wording %q", fb.WorldGap(), hardFloor[fbWorldGap])
}
}
// A broken or missing file must not take her last words away: every accessor
// answers from the literal it replaced.
func TestNilFallbacksAnswerFromTheHardFloor(t *testing.T) {
var fb *Fallbacks
if got := fb.Chat(); got != hardFloor[fbChat] {
t.Errorf("Chat = %q, want %q", got, hardFloor[fbChat])
}
if got := fb.Unknown(); got != hardFloor[fbQueryUnknown] {
t.Errorf("Unknown = %q, want %q", got, hardFloor[fbQueryUnknown])
}
if got := fb.FromSources("два литра"); got != "вот что я нашла: два литра" {
t.Errorf("FromSources = %q", got)
}
if got := fb.WorldGap(); got != hardFloor[fbWorldGap] {
t.Errorf("WorldGap = %q", got)
}
}
// Hearing the identical words every time a request fails is how a failure stops
// registering as one.
func TestFallbacksDoNotRepeat(t *testing.T) {
fb, err := LoadFallbacks(rand.NewSource(7))
if err != nil {
t.Fatalf("LoadFallbacks: %v", err)
}
prev := fb.Chat()
for i := 0; i < 20; i++ {
got := fb.Chat()
if got == prev {
t.Fatalf("chat repeated %q on turn %d", got, i)
}
prev = got
}
}
+27 -18
View File
@@ -5,6 +5,7 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
@@ -26,6 +27,11 @@ import (
var listenRE = regexp.MustCompile(`listening on (https?://\S+)`)
// errEmptyResponse — the server answered and said nothing. Separate from a
// transport failure: the model is up and produced no tokens, which is still not
// an answer and must not score as one.
var errEmptyResponse = errors.New("phraser: empty response from the model")
type LLMPhraser struct {
cfg Config
client *http.Client
@@ -428,8 +434,11 @@ func (p *LLMPhraser) PhraseNudge(ctx context.Context, c loop.Candidate) (deliver
}
// PhraseQuery prompts the LLM with the user's utterance and matching notes to
// compose a natural answer. Falls back to "вот что я нашла: <notes>" on any
// LLM error — better to give the raw data than silence.
// compose a natural answer. On any LLM error it returns the fallback text —
// "вот что я нашла: <notes>", or "не знаю." with no notes — and the error
// together. The daemon uses the text and keeps the turn alive; a caller that is
// measuring counts the failure. Until Vikunja #397 the error was dropped, so a
// dead server scored as bad phrasing.
func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []string) (string, error) {
// Blank sources are no sources. A caller that hands over one empty string —
// a page that fetched to nothing, a snippet trimmed away — used to take the
@@ -439,13 +448,15 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
if len(notes) == 0 {
sys, prompt := p.knowledgePrompt(utterance)
resp, err := p.chatWithSystem(ctx, sys, prompt, 768)
if err != nil || resp == "" {
return "не знаю.", nil
if err != nil {
return UnknownFallback(), fmt.Errorf("phrase query (knowledge): %w", err)
}
if resp == "" {
return UnknownFallback(), errEmptyResponse
}
text, _, perr := parseResponseMood(resp)
if perr != nil {
log.Printf("phraser: PhraseQuery: %v", perr)
return "не знаю.", nil
return UnknownFallback(), fmt.Errorf("phrase query (knowledge): %w", perr)
}
if text != "" {
return text, nil
@@ -457,13 +468,12 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
text, _, perr := parseResponseMood(resp)
if err != nil || perr != nil {
// Read the notes out rather than ship a broken fragment.
if perr != nil {
log.Printf("phraser: PhraseQuery: %v", perr)
cause := err
if cause == nil {
cause = perr
}
if len(notes) == 1 {
return "вот что я нашла: " + notes[0], nil
}
return "вот что я нашла: " + strings.Join(notes, "; "), nil
return SourcesFallback(strings.Join(notes, "; ")),
fmt.Errorf("phrase query (evidence): %w", cause)
}
if text != "" {
return text, nil
@@ -472,8 +482,9 @@ func (p *LLMPhraser) PhraseQuery(ctx context.Context, utterance string, notes []
}
// PhraseChat uses the LLM to respond conversationally, building a multi-turn
// message array from dialogue history + the current user utterance. Falls back
// to a simple greeting on any LLM error — better to say something than nothing.
// message array from dialogue history + the current user utterance. On any LLM
// error it returns both ChatFallback and the error, on the same rule as
// PhraseQuery: the fallback keeps the turn alive, the error stays visible.
func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history []dialogue.Turn) (string, error) {
sys := chatSystemPrompt(p.cfg.ContextBlock)
msgs := []chatMsg{
@@ -490,13 +501,11 @@ func (p *LLMPhraser) PhraseChat(ctx context.Context, utterance string, history [
resp, err := p.chatWithMessages(ctx, msgs, 768)
if err != nil {
log.Printf("phraser: PhraseChat: %v", err)
return "поговорили.", nil
return ChatFallback(), fmt.Errorf("phrase chat: %w", err)
}
text, _, perr := parseResponseMood(resp)
if perr != nil {
log.Printf("phraser: PhraseChat: %v", perr)
return "поговорили.", nil
return ChatFallback(), fmt.Errorf("phrase chat: %w", perr)
}
if text != "" {
return text, nil
+3 -6
View File
@@ -70,18 +70,15 @@ func NewStub() *Stub { return &Stub{} }
// prompted response from the model. The history parameter is accepted but
// ignored at the stub level (the production impl uses it for multi-turn).
func (s *Stub) PhraseChat(_ context.Context, _ string, _ []dialogue.Turn) (string, error) {
return "поговорили.", nil
return ChatFallback(), nil
}
// PhraseQuery returns a deterministic summary of the best matching notes.
func (s *Stub) PhraseQuery(_ context.Context, _ string, notes []string) (string, error) {
if len(notes) == 0 {
return "не знаю.", nil
return UnknownFallback(), nil
}
if len(notes) == 1 {
return "вот что я нашла: " + notes[0], nil
}
return "вот что я нашла: " + strings.Join(notes, "; "), nil
return SourcesFallback(strings.Join(notes, "; ")), nil
}
// Close implements Phraser.Close (no-op for the stub).
+112
View File
@@ -0,0 +1,112 @@
// phraser/replier.go — reactive reply phrasing, the confirmation he hears
// after every fact, note and reminder.
//
// It lived in cmd/mavend as package main until Vikunja #396, which meant the
// most frequently heard sentence Maven says was the one path the phrasing eval
// could not import, let alone score. Nothing here talks to the daemon: the
// caller supplies the completer and the context block, and cmd/mavend keeps the
// stub fallback so a model error still answers.
package phraser
import (
"context"
"strings"
"time"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/persona"
"github.com/kami/maven/internal/router"
)
// Completer is the model seam for the replier, a subset of router.Completer.
// *llm.Client satisfies it.
type Completer interface {
Complete(ctx context.Context, r llm.Req) (string, error)
}
// replyTimeout bounds one reply. Generous because the resident model on the CPU
// floor is slow and the caller has a deterministic fallback anyway.
const replyTimeout = 60 * time.Second
// ReplySystemPrompt — the reactive confirmation contract: one short Russian
// sentence, feminine self-reference, informal address, no question.
const ReplySystemPrompt = `Ты — Maven, домашняя ассистентка (о себе — в женском роде). Владелец — мужчина, говоришь с ним на "ты", в единственном числе; никогда не "вы"/"ваш" и не "он"/"его". Подтверди действие РОВНО ОДНИМ коротким предложением (≤120 символов), по-русски, спокойно и без официальных формулировок. Не задавай вопросов, не повторяй слова, не добавляй ничего после точки. Отвечай ТОЛЬКО одним объектом JSON с полями "response" (текст) и "mood" (ровно одно из: neutral, happy, thinking, tired, confused).
Пример: {"response": "Записала, что ты выпил стакан воды.", "mood": "neutral"}
Никогда не пиши "..." в поле response.`
// Replier phrases reactive confirmations with the resident model. It has no
// fallback of its own: an error is returned, and the daemon answers from the
// deterministic stub. That is also what makes it scorable — a dead server shows
// up as an error rather than as bad phrasing.
type Replier struct {
c Completer
// block renders the shared context block per turn (who he is, the time).
// nil ⇒ the prompt stands alone.
block func() string
}
// NewReplier builds a replier over c. block may be nil.
func NewReplier(c Completer, block func() string) *Replier {
return &Replier{c: c, block: block}
}
// PhraseReply returns the confirmation for one decision. An empty string with a
// nil error means the model produced nothing usable, which the caller must
// treat exactly like an error.
func (r *Replier) PhraseReply(ctx context.Context, d router.Decision) (string, error) {
ctx, cancel := context.WithTimeout(ctx, replyTimeout)
defer cancel()
out, err := r.c.Complete(ctx, llm.Req{
System: persona.Prepend(r.block, ReplySystemPrompt),
User: replyContext(d),
Grammar: ResponseGrammar,
MaxTokens: 512,
RepeatPenalty: 1.3,
})
if err != nil {
return "", err
}
out = stripThink(out)
if response, _, perr := parseResponseMood(out); perr != nil {
return "", perr
} else if response != "" {
return response, nil
}
// fallback: the model answered in bare prose, which is fine here.
return firstSentence(out), nil
}
// firstSentence trims the model's output to a single clean confirmation: first
// line, first sentence, whitespace-normalized — the last-line defense against a
// small model that rambles past the first period despite the prompt + stop.
func firstSentence(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexByte(s, '\n'); i >= 0 {
s = s[:i]
}
// keep up to and including the first sentence-ending punctuation.
if i := strings.IndexAny(s, ".!?"); i >= 0 {
s = s[:i+1]
}
return strings.TrimSpace(s)
}
// replyContext renders the decision into a compact RU description for the model.
func replyContext(d router.Decision) string {
switch d.Intent {
case router.IntentFact:
return "записала факт: " + d.Slots.Key + " " + d.Slots.Value
case router.IntentNote:
return "сохранила заметку: " + d.Slots.Text
case router.IntentReminder:
return "поставила напоминание: " + d.Slots.Text
default:
return string(d.Intent) + ": " + d.Slots.Text
}
}
// StripThink removes the <think> block a Thinking-variant model emits before its
// answer. Exported for the daemon's own model callers, which parse output that
// never passes through a phraser method.
func StripThink(s string) string { return stripThink(s) }
+90
View File
@@ -0,0 +1,90 @@
package phraser
import (
"context"
"testing"
"github.com/kami/maven/internal/llm"
"github.com/kami/maven/internal/router"
)
type mockCompleter struct {
out string
err error
}
func (m mockCompleter) Complete(_ context.Context, _ llm.Req) (string, error) { return m.out, m.err }
func TestReplierReturnsLLMReply(t *testing.T) {
r := NewReplier(mockCompleter{out: `{"response":"записала, кофе закончился","mood":"neutral"}`}, nil)
got, err := r.PhraseReply(context.Background(), noteDecision())
if err != nil || got != "записала, кофе закончился" {
t.Errorf("got %q, %v, want %q, nil", got, err, "записала, кофе закончился")
}
}
func TestReplierFallsBackToPlainText(t *testing.T) {
r := NewReplier(mockCompleter{out: "записала, кофе закончился"}, nil)
got, err := r.PhraseReply(context.Background(), noteDecision())
if err != nil || got != "записала, кофе закончился" {
t.Errorf("got %q, %v, want %q, nil", got, err, "записала, кофе закончился")
}
}
func TestReplierReportsTheModelError(t *testing.T) {
r := NewReplier(mockCompleter{err: errTestLLMDown}, nil)
got, err := r.PhraseReply(context.Background(), noteDecision())
if err == nil {
t.Errorf("got %q, nil error — a dead model must be reported, not phrased around", got)
}
}
// A fragment the grammar left half-open is a failed generation. It must come
// back as an error so the daemon reaches its stub, not as a reply.
func TestReplierRejectsBrokenJSON(t *testing.T) {
r := NewReplier(mockCompleter{out: `{"response":"запис`}, nil)
got, err := r.PhraseReply(context.Background(), noteDecision())
if err == nil || got != "" {
t.Errorf("got %q, %v, want empty and an error", got, err)
}
}
func TestReplierEmptyOutputIsEmpty(t *testing.T) {
r := NewReplier(mockCompleter{out: ""}, nil)
got, err := r.PhraseReply(context.Background(), noteDecision())
if err != nil || got != "" {
t.Errorf("got %q, %v, want empty and no error", got, err)
}
}
// grammarRecorder captures the request so the grammar can be asserted on.
type grammarRecorder struct{ req llm.Req }
func (g *grammarRecorder) Complete(_ context.Context, r llm.Req) (string, error) {
g.req = r
return `{"response":"записала","mood":"neutral"}`, nil
}
func TestReplierCarriesTheResponseGrammar(t *testing.T) {
rec := &grammarRecorder{}
r := NewReplier(rec, nil)
if _, err := r.PhraseReply(context.Background(), noteDecision()); err != nil {
t.Fatalf("PhraseReply: %v", err)
}
if rec.req.Grammar != ResponseGrammar {
t.Errorf("grammar = %q, want ResponseGrammar", rec.req.Grammar)
}
if rec.req.System != ReplySystemPrompt {
t.Errorf("system prompt = %q, want ReplySystemPrompt", rec.req.System)
}
}
func noteDecision() router.Decision {
return router.Decision{Intent: router.IntentNote, Slots: router.Slots{Text: "кофе закончился"}}
}
var errTestLLMDown = errTest("llm down")
type errTest string
func (e errTest) Error() string { return string(e) }
+5 -3
View File
@@ -215,10 +215,12 @@ func TestSwap_RollbackFailureLeavesNoBackendAndDegrades(t *testing.T) {
if _, _, aerr := p.acquire(); !errors.Is(aerr, ErrNoBackend) {
t.Errorf("acquire error = %v; want ErrNoBackend", aerr)
}
// Phrasing degrades to its fallback instead of failing the turn.
// Phrasing degrades to its fallback instead of failing the turn, and since
// Vikunja #397 it reports the error next to that fallback so a measuring
// caller can tell "no model" from "bad phrasing".
got, err := p.PhraseChat(context.Background(), "привет", nil)
if err != nil {
t.Fatalf("PhraseChat after a total failure returned an error: %v", err)
if !errors.Is(err, ErrNoBackend) {
t.Errorf("PhraseChat error = %v; want ErrNoBackend alongside the fallback", err)
}
if got == "" {
t.Error("PhraseChat returned empty; the fallback must still say something")