From 3513e508b7e2aefacf8cedde980063b8cc2b5ff4 Mon Sep 17 00:00:00 2001 From: claude Date: Sat, 8 Aug 2026 18:22:00 +0400 Subject: [PATCH 1/2] Give the router prompt a destination to write (V-660) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit V-659 measured the destination at 12/33 on the classifier cascade and named the gap: recall 0/15, because nothing anywhere names it. The model could not help, for a structural reason rather than a capability one. Nothing in routeSystem mentioned a Source and routeGrammar could not emit one, so there was no string for it to write. Same shape as the Praxis reach V-517 measured at 0/12. routeGrammar grows a source rule, closed over router.Sources plus the empty floor. A grammar cannot emit a destination that does not exist, which is the guarantee V-546 wants from a softmax and gets here for free. The prompt lists the twelve in Russian, one line each, and says plainly that "" is a normal answer to give often: two sources that can both answer means the chain walks, and guessing is the failure mode this whole field exists to stop. The read-back goes through ValidSource and runs on IntentQuery alone. The grammar already bounds the enum, but it is a request to a server that may be running another build, and only a query reaches queryWalk. Measured against gemma-4-12b on the workstation, same fixture, cascade with a hash fallback: destination 24/33 (72.7%) against the classifier's 12/33, and intent 81/96 (84.4%) which is where it already was. Recall is the whole move, 0/15 to 14/15. The model alone scores 26/33. Four cases the cascade loses and llm-only wins are calendar. The possessive agenda rules claim them at stage 0 and deliberately name nothing, because "что у меня в списке покупок" matches the same rule and naming the calendar would take the list source off the turn. So stage 0's caution now costs four destination points it did not cost before. That is a real trade and it wants its own argument, not a quiet edit here. The resident Qwen3-1.7B is unmeasured: it binds --port 0 inside the container and no host process can reach it. llm/check_prompt_parity.py in the training workspace compares its copy of routeSystem to this one and will fail until that copy gets the same edit. V-362 covers the catch-up. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN --- internal/router/llmrouter.go | 43 +++++++++++++++++++- internal/router/llmrouter_test.go | 67 +++++++++++++++++++++++++++++++ 2 files changed, 109 insertions(+), 1 deletion(-) diff --git a/internal/router/llmrouter.go b/internal/router/llmrouter.go index 1889efc..d609b3c 100644 --- a/internal/router/llmrouter.go +++ b/internal/router/llmrouter.go @@ -45,12 +45,19 @@ const routeGrammar = ` root ::= "[" ws action ("," ws action)* ws "]" action ::= "{" ws "\"intent\"" ws ":" ws intent ("," ws field)* ws "}" intent ::= "\"fact\"" | "\"reminder\"" | "\"note\"" | "\"query\"" | "\"act\"" | "\"chat\"" | "\"system\"" | "\"unknown\"" -field ::= key ws ":" ws string +field ::= (key ws ":" ws string) | ("\"source\"" ws ":" ws source) key ::= "\"key\"" | "\"value\"" | "\"text\"" | "\"verb\"" +source ::= "\"recall\"" | "\"calendar\"" | "\"tasks\"" | "\"list\"" | "\"money\"" | "\"weather\"" | "\"home\"" | "\"network\"" | "\"feeds\"" | "\"attention\"" | "\"self\"" | "\"world\"" | "\"\"" string ::= "\"" ([^"\\\x00-\x1F] | "\\" ["\\/bfnrt] | "\\u" [0-9a-fA-F]{4}){0,120} "\"" ws ::= [ \t\n]{0,4} ` +// TestRouteGrammarCoversSources holds the source rule above to router.Sources. +// The enum is the point: a grammar cannot emit a destination that does not +// exist, which is the guarantee V-546 wants from a softmax and gets here for +// free. Empty is the thirteenth alternative and it is not an oversight — it is +// the SourceUnknown floor, and the model must be able to decline. + // routeSystem — the router prompt. Changed 31-07-2026: the query test now sits // above the fact test and there is an explicit question test. Before that, a // question naming a fact key ("сколько воды я выпил с утра") matched the fact @@ -121,6 +128,31 @@ const routeSystem = `Классифицируй ровно одно сообще "что такое кватернион?" → {"intent":"query","text":"что такое кватернион"} "ага" → {"intent":"chat","text":"ага"} +Только для query добавь поле source — где лежит ответ: +- recall — его заметки, факты и то, что он раньше говорил +- calendar — встречи и события +- tasks — список задач +- list — списки покупок и другие именованные списки +- money — траты +- weather — погода +- home — свет, устройства, дом +- network — локальная сеть, сервер, диски +- feeds — новостные ленты +- attention — что требует внимания сейчас +- self — вопрос про самого ассистента +- world — всё остальное: определения, счёт, люди, факты о мире + +Пустое значение "" — нормальный ответ и его надо ставить часто. Ставь "", если ответ могут дать сразу два источника или если не уверен: тогда проверяются все по порядку, и это правильно. Никогда не угадывай. + +"сколько воды я выпил с утра" → {"intent":"query","text":"сколько воды я выпил с утра","source":"recall"} +"что я записывал про кота" → {"intent":"query","text":"что я записывал про кота","source":"recall"} +"во сколько у меня встреча" → {"intent":"query","text":"во сколько у меня встреча","source":"calendar"} +"что такое docker?" → {"intent":"query","text":"что такое docker","source":"world"} +"кто такой Линус Торвальдс?" → {"intent":"query","text":"кто такой Линус Торвальдс","source":"world"} +"сколько будет 17 на 23?" → {"intent":"query","text":"сколько будет 17 на 23","source":"world"} +"почему сервер тормозит" → {"intent":"query","text":"почему сервер тормозит","source":""} +"есть новости по бэкапу базы" → {"intent":"query","text":"есть новости по бэкапу базы","source":""} + Ответ — JSON-массив: по одному объекту на каждую просьбу. Обычно один. Если в реплике несколько просьб — по объекту на каждую. "напомни купить молоко, и запиши что кофе кончился" → [{"intent":"reminder","text":"купить молоко"},{"intent":"note","text":"кофе кончился"}]. Только JSON, без пояснений.` // routeRepeatPenalty — the sub-1B model loops one sentence inside the text field @@ -172,6 +204,7 @@ type routeAction struct { Value string `json:"value"` Text string `json:"text"` Verb string `json:"verb"` + Source string `json:"source"` } // Route asks the model for one decision. The bool is false when there is no @@ -240,6 +273,14 @@ func (lr *LLMRouter) Route(ctx context.Context, utterance string, now time.Time) case IntentQuery: d.Intent = IntentQuery d.Slots.Text = firstNonEmpty(a.Text, utterance) + // Through ValidSource, and on query alone. The grammar already bounds + // the enum, but the grammar is a request to a server that may be + // running a different build, and a destination this binary does not + // know would take real query sources off the turn. Anything unknown + // drops to SourceUnknown, which is the floor and costs nothing. + if ValidSource(Source(a.Source)) { + d.Source = Source(a.Source) + } case IntentAct: d.Intent = IntentAct d.Slots.Text = firstNonEmpty(a.Verb, utterance) diff --git a/internal/router/llmrouter_test.go b/internal/router/llmrouter_test.go index 492061f..bd58d27 100644 --- a/internal/router/llmrouter_test.go +++ b/internal/router/llmrouter_test.go @@ -398,3 +398,70 @@ func TestLLMReminderWithSubjectIsNotGated(t *testing.T) { t.Fatalf("a complete reminder was sent back as a question: %+v", d.Slots) } } + +// TestRouteGrammarCoversSources — the grammar enum and router.Sources are two +// hand-written lists of the same twelve destinations, and nothing else notices +// when one grows. A destination missing from the grammar is a destination the +// model is structurally unable to name, which is the exact defect V-517 +// measured for Praxis: not a weak model, an absent string. +func TestRouteGrammarCoversSources(t *testing.T) { + for _, s := range Sources { + if !strings.Contains(routeGrammar, `"\"`+string(s)+`\""`) { + t.Errorf("routeGrammar cannot emit %q — the model can never name it", s) + } + } + // The floor has to be reachable too, or the model is forced to pick one. + if !strings.Contains(routeGrammar, `"\"\""`) { + t.Error(`routeGrammar cannot emit "" — the model cannot decline a destination`) + } + // Count the alternatives on the source rule: an extra one is a destination + // the daemon would drop to SourceUnknown after the model spent tokens on it. + for _, line := range strings.Split(routeGrammar, "\n") { + if !strings.HasPrefix(line, "source ") { + continue + } + if got, want := strings.Count(line, "|")+1, len(Sources)+1; got != want { + t.Errorf("source rule has %d alternatives, want %d (Sources plus the floor)", got, want) + } + } +} + +// The destination is read back only through ValidSource. A model on an older or +// newer build can write a string this binary does not know, and trusting it +// would take real query sources off the turn for a name nothing answers. +func TestLLMUnknownSourceFallsToTheFloor(t *testing.T) { + r := newLLMTestRouter(t, `{"intent":"query","text":"что там с бэкапами","source":"praxis"}`) + d, err := r.Route(context.Background(), "что там с бэкапами", refNow()) + if err != nil { + t.Fatalf("route: %v", err) + } + if d.Source != SourceUnknown { + t.Fatalf("invented destination %q was trusted, want the floor", d.Source) + } +} + +// And a known one survives, or the read-back is just a filter. +func TestLLMNamedSourceSurvives(t *testing.T) { + r := newLLMTestRouter(t, `{"intent":"query","text":"кто такой Линус Торвальдс","source":"world"}`) + d, err := r.Route(context.Background(), "кто такой Линус Торвальдс?", refNow()) + if err != nil { + t.Fatalf("route: %v", err) + } + if d.Source != SourceWorld { + t.Fatalf("source %q, want %q", d.Source, SourceWorld) + } +} + +// A destination on anything but a query is dropped. Only IntentQuery reaches +// queryWalk, so a source elsewhere is a field nobody reads and a claim nobody +// checks. +func TestLLMSourceIsQueryOnly(t *testing.T) { + r := newLLMTestRouter(t, `{"intent":"note","text":"кофе кончился","source":"recall"}`) + d, err := r.Route(context.Background(), "запиши что кофе кончился", refNow()) + if err != nil { + t.Fatalf("route: %v", err) + } + if d.Source != SourceUnknown { + t.Fatalf("a note carried destination %q", d.Source) + } +} From 00f9239ef97adaea1819a5e211861b96104cc249 Mon Sep 17 00:00:00 2001 From: claude Date: Sat, 8 Aug 2026 18:23:22 +0400 Subject: [PATCH 2/2] Record the model arm, and the stage 0 trade it exposed (V-660) Numbers and the argument in docs/evals/2026-08-08-destination-model-arm.md, pointer and the short version in CLAUDE.md. The finding worth carrying is not the 72.7%: it is that stage 0's silence on the possessive agenda rules used to be free and now costs four destination points, because there is finally something downstream that would have named the calendar. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN --- CLAUDE.md | 22 ++++++- .../evals/2026-08-08-destination-model-arm.md | 66 +++++++++++++++++++ 2 files changed, 85 insertions(+), 3 deletions(-) create mode 100644 docs/evals/2026-08-08-destination-model-arm.md diff --git a/CLAUDE.md b/CLAUDE.md index 7262f50..a6c98e8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -433,9 +433,25 @@ not a gap in the labelling. so the fixture scored a grammar set nobody runs. Fixed by V-659, worth 3 points of destination and nothing else. Check that function when adding a grammar. -The model arm is still the follow-up. It lands on V-546. Intent, mood and BIO slot -tags were already three heads on one forward pass of the resident e5-small. -Destination is a fourth head on the same pass. +**The model arm landed the same day** (V-660, +`docs/evals/2026-08-08-destination-model-arm.md`). `routeGrammar` carries a +`source` rule closed over `router.Sources` plus the empty floor, so the model +cannot emit a destination that does not exist. The prompt lists the twelve in +Russian and says `""` is a normal answer to give often. `LLMRouter.Route` reads it +back through `ValidSource` and on `IntentQuery` alone. Against gemma-4-12b on the +workstation the cascade scores destination **24/33 (72.7%)** with intent unmoved +at 84.4%, and **recall goes 0/15 to 14/15**. The resident Qwen3-1.7B is +unmeasured, because it binds `--port 0` inside the container. + +**Stage 0 now costs four destination points.** It did not before. The four cases +the cascade loses and the model alone wins are all calendar. The possessive +agenda rules claim them first and name nothing on purpose. That caution was free +while nothing downstream could name anything either. It is not free now, and the +fix is the owner's call rather than a quiet edit. + +The last arm is V-546. Intent, mood and BIO slot tags were already three heads on +one forward pass of the resident e5-small. Destination is a fourth head on the +same pass, and 72.7% from a 12B teacher is the label source for training it. ## LLM output contract diff --git a/docs/evals/2026-08-08-destination-model-arm.md b/docs/evals/2026-08-08-destination-model-arm.md new file mode 100644 index 0000000..efc69eb --- /dev/null +++ b/docs/evals/2026-08-08-destination-model-arm.md @@ -0,0 +1,66 @@ +# The destination, with a model that can name one + +Measured 2026-08-08 against gemma-4-12b on the workstation, the same 96-case +fixture V-659 built. Covers V-660. + +```sh +no_proxy='*' MAVEN_LLM_URL=http://192.168.1.105:8080 \ + make t PKG=./internal/router/eval/ RUN=TestLLMRouterBaseline V=1 +``` + +## The gap was structural + +V-659 measured the destination at 12/33 on the classifier cascade, with recall +at 0/15. Nothing in `routeSystem` named a `Source` and `routeGrammar` could not +emit one, so the resident model had no string to write. That is the shape V-517 +measured for Praxis reach at 0/12: not a weak model, an absent contract. + +`routeGrammar` now carries a `source` rule closed over `router.Sources` plus the +empty floor. The prompt lists the twelve destinations in Russian and says that +`""` is a normal answer to give often. + +## Result + +| run | intent | destination | +|---|---|---| +| classifier + ONNX (V-659) | 73/96 (76.0%) | 12/33 (36.4%) | +| gemma-4-12b alone | 79/96 intent-only (82.3%) | 26/33 (78.8%) | +| cascade + gemma-4-12b + hash fallback | 81/96 (84.4%) | 24/33 (72.7%) | + +Recall is the move: 0/15 to 14/15. Intent did not shift, which was the +constraint. The prompt is shared, so a destination rule that costs routing +points is not a win. + +The eight llm-only errors are the eight `want_clarify` cases. The model returned +`unknown` on every one, which is correct, and the llm-only harness surfaces a +decline as an error by design. + +## Stage 0 now costs four destination points + +The four cases the cascade loses and the model alone wins are all calendar. The +possessive agenda rules claim them at stage 0 and deliberately name nothing. +"что у меня в списке покупок" matches the same rule. Naming the calendar there +would take the list source off the turn (V-655). + +So a rule written to be careful about the list now blocks a model that would +have named the calendar correctly. Before V-660 that caution was free, because +nothing downstream of stage 0 could name anything either. + +Three ways out, and each costs something. Split the possessive rule so the +calendar-shaped half names its destination. Let a later stage overwrite an empty +destination a grammar left behind, which reverses "a matched value always wins". +Or leave it, on the argument that four points is cheap next to a wrong +destination on a shopping list. This wants the owner's call rather than a quiet +edit. + +## What this does not measure + +The resident Qwen3-1.7B, which is what homesrv runs. It binds `--port 0` inside +the container and no host process can reach it. Scoring it needs a second +llama-server on a fixed port. The workstation is never assumed +up, so the homesrv number is the one that decides whether this ships on by +default. + +The fixture is 33 labelled destinations over twelve values. Recall carries 15 of +them and five destinations carry none at all. A per-destination number below +world, recall, calendar and the floor is not supported by this fixture.