Merge the common unit for claims (#210)

V-565. internal/claim holds Claim{Claimant, Intent, Filled, Consumed,
Unexplained, Band, Veto} and imports nothing from Maven, so the dialogue to
router edge stays impossible. internal/router/claim.go builds one from a
Decision. Additive: nothing in Route calls it and Decision.Confidence is
untouched.

Measured first, on the 91-case fixture. Stage 0 emits 1.0 always and is right
20/20. The classifier cosine spans 0.859 to 0.942 and is right 62% of the
time, with 62% correct below its median and 62% above, so the number carries
no signal about correctness. The top1 to top2 margin is worse: p50 0.009, 68
of 71 cases under 0.02. A calibrated float is not cheaply available from the
classifier, which is what the task's ledger asked to be checked.

So four ordinal bands, highest first: anchored, structural, nearest, vetoed,
with unknown at the bottom so a builder that forgot cannot outrank a measured
claim. Anchored against nearest is 100% against 62% on the same utterances.
Nearest is one band and not a scale because the cosine is flat.

Coverage decides before the band does. That is what fixes Rome: the pending
claimant ate the question while explaining one token of it.

No fixture number moved. TestONNXBaseline is 64/91.
This commit is contained in:
2026-08-06 00:58:34 +04:00
7 changed files with 981 additions and 13 deletions
+135
View File
@@ -0,0 +1,135 @@
# Plan: dialogue arbitration, one channel and many claimants
Umbrella V-558. This file collects the design for its children.
Last verified: 06-08-2026 @ b6305f1
## A common unit for claims on an utterance (V-565)
**Verdict: four ordinal bands, and the band is the tie-break rather than the decision.
Coverage decides first.** The measurement below says no claimant Maven has today can produce
a graded confidence. A float would be an invention either way. What is available is the KIND
of evidence a claimant holds, and there are exactly four kinds.
### What the claimants report today
Measured 06-08-2026 on the 91-case RU fixture (`internal/router/eval`), through the deployed
cascade with the quantized multilingual-e5-small embedder. The harness is
`TestONNXClaimConfidenceDistribution` and `TestStage0Contention` in
`internal/router/eval/claims_test.go`. Correct means the right intent, or a refusal where the
fixture wants one. Slots are excluded, because a slot miss is a parser question and would
blur what the number is being asked to predict.
| Claimant | Values it can emit | Distribution on the fixture | Correct |
|---|---|---|---|
| Stage 0 grammars, 21 of them | `1.0`, always | claimed 20 of 91 cases | 20/20 (100%) |
| Classifier, cosine | continuous in principle | observed range 0.859 to 0.942 over 71 cases | 44/71 (62%) |
| LLM router | `1.0` or `0.3`, nothing between | not run here, no llama-server | see below |
| Query sources, 22 of them | a bool | not routed by the fixture | n/a |
| Stateful four | nothing at all | n/a | n/a |
Four findings, and each one constrains the band set.
**The classifier's cosine carries no signal about correctness.** It scores 62% below the
median and 62% above it. That is 13/21 in 0.8 to 0.9, and 31/50 in 0.9 to 1.0. The spread is
0.083 wide. Every case sits above the 0.55 threshold, so the gate never fires here. A number
flat against correctness, which never crosses its own gate, is not a confidence.
**Nor does the margin between its top two intents.** Top1 minus top2 is min 0.000, p50
0.009, max 0.025. Sixty-eight of the 71 classified cases sit under 0.02 and score 60%. Three
clear 0.02 and score 3/3, which is a sample of three. So the ledger's question is answered:
a calibrated float is NOT cheaply available from the classifier alone. Nearest-centroid over
frozen seeds ranks intents, and the ranking is decided in the third decimal place. It can say
which intent is nearest. It cannot say how near.
**Stage 0 asserts 1.0 by fiat, and on this fixture the fiat is right.** Twenty of twenty.
That is not evidence that a hand-written anchored pattern is always right. It is evidence
that anchored and nearest are different kinds of claim, and must not share a scale. The gap
is 100% against 62% on the same 91 utterances.
**Stage 0 contention is rarer than the list order suggests.** Exactly one case of 91 draws
two grammars. That is `ru-query-019`, where `calendar-query` and `agenda-query` both match,
and `calendar-query` wins because it is earlier in `buildRouter`. Both would route
`IntentQuery`, so the ordering costs nothing there. The finding is not that ordering is
harmless. It is that the fixture barely exercises what V-558 is about. Part of what a claim
object buys is making the contention countable.
**The LLM router emits two values, and one of them is not a confidence.** `llmFullConfidence`
is 1.0 and `llmThinConfidence` is 0.3. `gateLLMDecision` moves a decision to 0.3 through
three named arms. A fact with no key, an act with no allowlisted fn, a reminder with no
subject. Each is a self-veto with a reason, flattened into a number that then loses the
reason. Both values are meaningful only against `config.DefaultRouterThreshold`. 0.3 is below
0.55 and 1.0 is above it, and nothing anywhere reads any other property of either.
### The band set
Four bands, ordinal, highest first. They name the kind of evidence, because that is the one
thing every claimant can report without inventing it.
**`BandAnchored`.** A literal pattern anchored in the utterance matched, and the matched span
is what decides the intent. Stage 0 grammars and query-source matchers. The claimant is
certain about the shape of the sentence. That is not the same as being certain about the
answer. Measured 20/20.
**`BandStructural`.** A claimant read the whole sentence and produced a complete route. Every
slot the intent requires is filled. The LLM router at `llmFullConfidence` sits here, and so
does a stateful claimant holding a pending question. Not anchored, because nothing in the
utterance is pointed at.
**`BandNearest`.** The claim rests only on resemblance to something else. No anchor in the
utterance, no structural check behind it. The classifier. One band rather than a graded
scale, and the measurement is the argument. 62% at both ends of the cosine range, and a
top-two margin that never reaches 0.03.
**`BandVetoed`.** The claimant will take the turn only if nobody else will, and says why it
should not. The three arms of `gateLLMDecision` land here with their reason preserved. A
vetoed claim is still a claim. Maven asking "о чём напомнить?" beats silence.
There is no fifth band, and that is a measurement result rather than a preference. No
claimant in the cascade today can report what a fifth band would carry. V-546 lands a softmax
head whose max probability is a calibrated number. That one gets read as a number, not
squeezed into these four.
### Coverage decides before the band does
The band is the tie-break. The first question is how much of the utterance a claim explains,
and that is `Consumed` against `Unexplained` on the claim object. Two reasons.
It is the fix for the failure that opened V-558. "какая сейчас погода в Риме?" arrived while
a reminder was pending. The pending claimant ate the whole utterance as a time answer while
explaining none of it. Not "погода", not "Риме", not the question mark. A weather claim
explains all of it. Coverage-first arbitration prefers the weather claim without knowing that
a pending reminder is less trustworthy than a grammar. The pending question then survives to
be asked again.
It also keeps the stateful four out of the top slot without special-casing them. They sit at
`BandStructural`, below any anchored claim. That is the whole V-558 complaint about the
highest-priority claimants being the least informed, expressed as one rule.
### The claim object
```go
type Claim struct {
Claimant string // who wants the turn
Intent string // plain string: internal/dialogue must not import internal/router
Filled []string // the slots this claim would fill
Consumed []string // utterance tokens this claim explains
Unexplained []string // the rest, in order
Band Band
Veto string // why this claim should NOT win, empty when there is none
}
```
`Intent` is a plain `string` rather than `router.Intent` on purpose. `internal/dialogue` must
not import `internal/router`, so the claim package must not either, and a shared string costs
one conversion at each edge.
`Unexplained` is carried rather than derived at read time. A claimant can then decline to
explain a span it did match.
### What this task does not do
`router.Decision.Confidence` stays and keeps its float. `r.threshold` and `gateLLMDecision`
read it, and the classifier is the failure floor. A rewire that broke either would trade a
measured floor for an unmeasured design. V-565 lands the type and the builder beside the
existing path. The arbiter that reads claims is V-560.
+192
View File
@@ -0,0 +1,192 @@
// Package claim is the common unit for the many claimants that compete for one
// utterance (V-565, umbrella V-558, design in
// docs/plans/19-dialogue-arbitration.md).
//
// Maven's cascade has roughly ten stage-0 grammars, seven router intents,
// twenty-two query sources and four stateful pre-emptors, and every one of them
// answers "is this mine?" alone. None can answer "is this more mine than
// yours?", because their scores are not comparable: stage 0 asserts 1.0 by
// fiat, the classifier reports a cosine, the LLM router derives one from
// structure. So list order is the whole arbitration.
//
// A Claim carries evidence rather than a verdict. Two things read that evidence
// and neither needs a float:
//
// - specificity — a claim explaining more of the utterance is preferred, and
// that is Consumed against Unexplained;
// - negative constraint — a claimant may veto itself and say why, and that is
// Veto.
//
// Where a number is unavoidable, it is an ordinal Band and not a probability.
// The band set is argued from measurement in the plan doc: the classifier's
// cosine is flat against correctness (62% at both ends of a spread 0.083 wide)
// and its top-two margin is p50 0.009, so no claimant Maven has today can
// produce a graded confidence.
//
// This package deliberately imports nothing from the rest of Maven.
// internal/dialogue must not import internal/router, so a shared unit that
// pulled in router.Intent would smuggle that edge back in. Intent is a plain
// string and the conversion happens at each edge.
package claim
import "strings"
// Band — the kind of evidence behind a claim, ordinal and comparable. Higher
// wins a tie. Four values, because four is what the claimants can report.
type Band int
const (
// BandUnknown — the zero value. A claim that never set a band is a bug in
// its builder, not a weak claim, so it must not silently rank as one.
BandUnknown Band = iota
// BandVetoed — the claimant will take the turn only if nobody else will,
// and Veto says why it should not. The three arms of gateLLMDecision (a
// fact with no key, an act with no allowlisted fn, a reminder with no
// subject) land here. Still a claim: asking "о чём напомнить?" beats
// silence.
BandVetoed
// BandNearest — the claim rests only on resemblance to something else,
// with no anchor in the utterance and no structural check behind it. The
// nearest-centroid classifier. One band and not a graded scale, because
// the cosine measured flat against correctness.
BandNearest
// BandStructural — the claimant read the whole sentence and produced a
// complete route, every slot its intent requires filled. The LLM router at
// full confidence, and a stateful claimant holding a pending question.
// Below BandAnchored on purpose: the four stateful claimants pre-empt
// unconditionally today, and that is the V-558 defect.
BandStructural
// BandAnchored — a literal pattern anchored in the utterance matched, and
// the matched span is what decides the intent. Stage 0 grammars and
// query-source matchers. Certainty about the shape of the sentence, which
// is not certainty about the answer.
BandAnchored
)
// String — the band's name, for a trace line and for a test failure that has to
// say which band it got.
func (b Band) String() string {
switch b {
case BandVetoed:
return "vetoed"
case BandNearest:
return "nearest"
case BandStructural:
return "structural"
case BandAnchored:
return "anchored"
default:
return "unknown"
}
}
// Claim — one claimant's bid for one utterance.
type Claim struct {
// Claimant — who wants the turn. A grammar name, a query source name, a
// stage label. Read by the trace and by a test naming a loser.
Claimant string
// Intent — the route this claim would take. A plain string and not
// router.Intent: see the package comment.
Intent string
// Filled — the slot names this claim would fill ("time", "fn", "key",
// "text"). Names and not values, because arbitration compares shape.
Filled []string
// Consumed — the utterance tokens this claim explains, in the order they
// appear. The numerator of specificity.
Consumed []string
// Unexplained — the tokens this claim does not explain, in order. Carried
// rather than derived, so a claimant may decline a span it did match.
Unexplained []string
// Band — the kind of evidence. The tie-break, after coverage.
Band Band
// Veto — why this claim should NOT win, empty when there is none. A
// non-empty Veto and a Band above BandVetoed is legal: a claim can be
// well-evidenced and still name a reason to prefer somebody else.
Veto string
}
// Coverage — the fraction of the utterance this claim explains, in [0,1]. A
// claim with no tokens either way covers nothing; it is not division by zero
// and it is not a full claim.
func (c Claim) Coverage() float64 {
total := len(c.Consumed) + len(c.Unexplained)
if total == 0 {
return 0
}
return float64(len(c.Consumed)) / float64(total)
}
// Vetoed reports whether the claimant named a reason against itself.
func (c Claim) Vetoed() bool { return c.Veto != "" }
// MoreSpecificThan — the ordering V-560's arbiter will read. Coverage first,
// because that is what fixes the failure this program opened with: a pending
// reminder ate "какая сейчас погода в Риме?" as a time answer while explaining
// none of it. Band only breaks a coverage tie.
//
// Deliberately NOT wired into the cascade by V-565. It is here so the ordering
// is one function with tests on it, rather than a rule restated at each of the
// sites that will eventually call it.
func (c Claim) MoreSpecificThan(other Claim) bool {
cc, oc := c.Coverage(), other.Coverage()
if cc != oc {
return cc > oc
}
return c.Band > other.Band
}
// Tokens — the utterance split for coverage accounting. Whitespace, then
// trailing and leading punctuation, then lowercased.
//
// This is tokenization over the raw string and not a Russian pattern: it
// contains no word list, and its output is a count rather than a fact or a
// route (CLAUDE.md § Russian patterns). Lowercasing is Unicode-aware, so
// Cyrillic folds the same way Latin does.
func Tokens(utterance string) []string {
fields := strings.FieldsFunc(utterance, func(r rune) bool {
return r == ' ' || r == '\t' || r == '\n' || r == '\r'
})
out := make([]string, 0, len(fields))
for _, f := range fields {
t := strings.Trim(strings.ToLower(f), ".,!?;:()\"'«»…-–—")
if t == "" {
continue
}
out = append(out, t)
}
return out
}
// Split partitions the utterance's tokens into the ones a claim explains and
// the rest, preserving order in both. A token is explained when it appears in
// one of the spans the claimant filled (a slot value, a matched substring).
//
// Duplicates are handled by membership and not by count: "напомни напомни
// позвонить" with span "напомни" explains both copies. The alternative is a
// multiset, and no claimant Maven has can say which copy it meant.
func Split(utterance string, spans ...string) (consumed, unexplained []string) {
explained := map[string]bool{}
for _, s := range spans {
for _, t := range Tokens(s) {
explained[t] = true
}
}
for _, t := range Tokens(utterance) {
if explained[t] {
consumed = append(consumed, t)
} else {
unexplained = append(unexplained, t)
}
}
return consumed, unexplained
}
+120
View File
@@ -0,0 +1,120 @@
package claim
import (
"reflect"
"testing"
)
func TestTokensStripsPunctuationAndCase(t *testing.T) {
got := Tokens("Какая сейчас погода в Риме?")
want := []string{"какая", "сейчас", "погода", "в", "риме"}
if !reflect.DeepEqual(got, want) {
t.Errorf("Tokens = %q, want %q", got, want)
}
}
// The dash forms matter: an STT transcript routinely carries "а, да, прости -
// на 9", and a stray dash counted as a token would dilute every coverage score
// in the sentence.
func TestTokensDropsBareDashes(t *testing.T) {
got := Tokens("а, да, прости — на 9")
want := []string{"а", "да", "прости", "на", "9"}
if !reflect.DeepEqual(got, want) {
t.Errorf("Tokens = %q, want %q", got, want)
}
}
func TestSplitPartitionsInOrder(t *testing.T) {
consumed, unexplained := Split("напомни позвонить маме", "позвонить маме")
if want := []string{"позвонить", "маме"}; !reflect.DeepEqual(consumed, want) {
t.Errorf("consumed = %q, want %q", consumed, want)
}
if want := []string{"напомни"}; !reflect.DeepEqual(unexplained, want) {
t.Errorf("unexplained = %q, want %q", unexplained, want)
}
}
func TestCoverageIsZeroWithoutTokens(t *testing.T) {
if got := (Claim{}).Coverage(); got != 0 {
t.Errorf("Coverage of an empty claim = %v, want 0", got)
}
}
func TestCoverageFraction(t *testing.T) {
c := Claim{Consumed: []string{"a", "b", "c"}, Unexplained: []string{"d"}}
if got := c.Coverage(); got != 0.75 {
t.Errorf("Coverage = %v, want 0.75", got)
}
}
// The band order is load-bearing, so it is asserted rather than assumed from
// the iota. BandUnknown must sit at the bottom: a claim whose builder forgot to
// set a band is a bug and must not outrank a measured one.
func TestBandOrder(t *testing.T) {
ordered := []Band{BandUnknown, BandVetoed, BandNearest, BandStructural, BandAnchored}
for i := 1; i < len(ordered); i++ {
if !(ordered[i-1] < ordered[i]) {
t.Errorf("%v is not below %v", ordered[i-1], ordered[i])
}
}
for _, b := range ordered {
if b.String() == "" {
t.Errorf("band %d has no name", b)
}
}
if BandUnknown.String() != "unknown" {
t.Errorf("BandUnknown.String() = %q", BandUnknown.String())
}
}
// The failure V-558 opened with, as an ordering test. A pending reminder eats
// "какая сейчас погода в Риме?" as a time answer and explains none of it; the
// weather source explains all of it. Coverage decides, and the band never gets
// consulted, which is the point: the pending claimant is structural and the
// weather claim is anchored, but even if the bands were equal the weather claim
// wins.
func TestCoverageBeatsBand(t *testing.T) {
utterance := "какая сейчас погода в Риме?"
pendingConsumed, pendingRest := Split(utterance, "сейчас")
pending := Claim{
Claimant: "reminder-followup", Intent: "reminder",
Consumed: pendingConsumed, Unexplained: pendingRest,
Band: BandStructural,
}
weatherConsumed, weatherRest := Split(utterance, utterance)
weather := Claim{
Claimant: "weather", Intent: "query",
Consumed: weatherConsumed, Unexplained: weatherRest,
Band: BandNearest,
}
if !weather.MoreSpecificThan(pending) {
t.Errorf("weather (%v cover) did not beat pending (%v cover)",
weather.Coverage(), pending.Coverage())
}
if pending.MoreSpecificThan(weather) {
t.Error("pending beat weather, so the ordering is not asymmetric")
}
}
// Where coverage ties, the band decides. Two grammars claiming the same
// utterance is the stage-0 contention case (ru-query-019 on the fixture), and
// today list order settles it with nothing recorded.
func TestBandBreaksACoverageTie(t *testing.T) {
anchored := Claim{Claimant: "calendar-query", Consumed: []string{"a"}, Band: BandAnchored}
nearest := Claim{Claimant: "classifier", Consumed: []string{"a"}, Band: BandNearest}
if !anchored.MoreSpecificThan(nearest) {
t.Error("anchored did not beat nearest on equal coverage")
}
if nearest.MoreSpecificThan(anchored) {
t.Error("nearest beat anchored on equal coverage")
}
}
func TestVetoed(t *testing.T) {
if (Claim{}).Vetoed() {
t.Error("a claim with no veto reports itself vetoed")
}
if !(Claim{Veto: "fact with no key"}).Vetoed() {
t.Error("a claim with a veto reason does not report itself vetoed")
}
}
+113
View File
@@ -0,0 +1,113 @@
package router
import "github.com/kami/maven/internal/claim"
// ClaimOf — build a claim.Claim from a Decision (V-565, design in
// docs/plans/19-dialogue-arbitration.md).
//
// Additive and beside the existing path. Decision.Confidence keeps its float
// and keeps working: r.threshold and gateLLMDecision read it, and the
// classifier cascade is the failure floor. Nothing in Route calls this yet.
// The arbiter that reads claims is V-560.
//
// claimant names who produced the decision. The cascade does not record which
// stage-0 grammar matched, so the caller passes what it knows and the builder
// does not guess.
func ClaimOf(claimant string, d Decision) claim.Claim {
consumed, unexplained := claim.Split(d.Utterance, claimSpans(d)...)
return claim.Claim{
Claimant: claimant,
Intent: string(d.Intent),
Filled: filledSlots(d.Slots),
Consumed: consumed,
Unexplained: unexplained,
Band: bandOf(d),
Veto: vetoOf(d),
}
}
// claimSpans — the parts of the utterance the decision says it read. Slot
// values, not the utterance, because coverage is the question of how much of
// the sentence the claim actually explains.
//
// A stage-0 grammar reports whatever its Build put in the slots, which for the
// reminder rule is the text after "напомни" and not the verb itself. That
// under-reports coverage rather than over-reporting it, which is the safe
// direction: a claim that overstates what it explains wins arbitrations it
// should lose.
func claimSpans(d Decision) []string {
spans := []string{d.Slots.Text, d.Slots.Key, d.Slots.Value, d.Slots.Fn}
return append(spans, d.Slots.Args...)
}
// filledSlots — the slot names this decision would fill. Text counts only when
// it differs from the whole utterance: fillSlots backfills the raw utterance
// into Text for a note, a query and a chat turn, so a set Text is not by itself
// evidence that anything was extracted.
func filledSlots(s Slots) []string {
var out []string
if s.HasTime {
out = append(out, "time")
}
if s.HasFn {
out = append(out, "fn")
}
if s.HasKey {
out = append(out, "key")
}
if s.Text != "" {
out = append(out, "text")
}
return out
}
// bandOf — which kind of evidence this decision rests on.
//
// Stage 0 is anchored: a literal pattern matched and its span decided the
// intent. The LLM path (stage 1) is structural: the model read the whole
// sentence, and gateLLMDecision already checked the route for structural
// holes. The classifier (stages 2 and 3) is nearest, and the measurement is why
// it is one band rather than a scale — on the 91-case RU fixture its cosine
// spans 0.859 to 0.942 and scores 62% at both ends.
//
// A decision carrying a veto lands in BandVetoed regardless of who produced it.
// That is the point of the band: a self-vetoed claim should lose to any claim
// that is not, whatever machinery built it.
func bandOf(d Decision) claim.Band {
if vetoOf(d) != "" {
return claim.BandVetoed
}
switch d.Stage {
case 0:
return claim.BandAnchored
case 1:
return claim.BandStructural
default:
return claim.BandNearest
}
}
// vetoOf — why this decision should not win, recovered as a reason rather than
// a number.
//
// gateLLMDecision flattens three named structural holes into
// llmThinConfidence, and the reason is lost at that point: 0.3 tells a reader
// that something was wrong and never which thing. The same three conditions are
// checked here so the claim carries the sentence a trace can print and the
// owner can be told.
//
// Clarify is checked last and is the general case. A decision below threshold
// has already asked to be doubted, whichever path set it.
func vetoOf(d Decision) string {
switch {
case d.Intent == IntentFact && !d.Slots.HasKey:
return "fact with no key: nothing to write, or a confident write under the wrong key"
case d.Intent == IntentAct && !d.Slots.HasFn:
return "act with no allowlisted fn: running an unlisted command or silently doing nothing"
case d.Intent == IntentReminder && !reminderHasSubject(d.Slots.Text):
return "reminder with no subject: it would fire empty at the hour"
case d.Clarify:
return "below the confidence gate"
}
return ""
}
+160
View File
@@ -0,0 +1,160 @@
package router
import (
"context"
"reflect"
"testing"
"time"
"github.com/kami/maven/internal/claim"
)
func TestClaimOfBands(t *testing.T) {
cases := []struct {
name string
dec Decision
want claim.Band
}{
{
name: "stage 0 is anchored",
dec: Decision{
Utterance: "сколько времени", Stage: 0, Intent: IntentSystem,
Confidence: 1.0, Slots: Slots{Text: "сколько времени"},
},
want: claim.BandAnchored,
},
{
name: "the llm path is structural",
dec: Decision{
Utterance: "выпил воды", Stage: 1, Intent: IntentFact,
Confidence: llmFullConfidence,
Slots: Slots{Key: "water", Value: "1", HasKey: true, Text: "выпил воды"},
},
want: claim.BandStructural,
},
{
name: "the classifier is nearest",
dec: Decision{
Utterance: "что нового", Stage: 2, Intent: IntentQuery,
Confidence: 0.91, Slots: Slots{Text: "что нового"},
},
want: claim.BandNearest,
},
{
name: "a structural hole vetoes whoever found it",
dec: Decision{
Utterance: "запиши", Stage: 1, Intent: IntentFact,
Confidence: llmThinConfidence, Slots: Slots{Text: "запиши"},
},
want: claim.BandVetoed,
},
{
name: "clarify vetoes a classifier decision",
dec: Decision{
Utterance: "сделай это", Stage: 3, Intent: IntentNote,
Confidence: 0.2, Clarify: true, Slots: Slots{Text: "сделай это"},
},
want: claim.BandVetoed,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := ClaimOf("test", tc.dec)
if got.Band != tc.want {
t.Errorf("band = %v, want %v (veto %q)", got.Band, tc.want, got.Veto)
}
})
}
}
// The veto has to name the hole. Folding all three arms into llmThinConfidence
// is what lost the reason, and 0.3 tells a reader that something was wrong but
// never which thing.
func TestClaimOfVetoNamesTheHole(t *testing.T) {
cases := []struct {
name string
dec Decision
want string
}{
{"keyless fact", Decision{Intent: IntentFact}, "fact with no key"},
{"act with no fn", Decision{Intent: IntentAct}, "act with no allowlisted fn"},
{"subjectless reminder", Decision{Intent: IntentReminder, Slots: Slots{Text: "напомни"}}, "reminder with no subject"},
{"below the gate", Decision{Intent: IntentQuery, Clarify: true}, "below the confidence gate"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := ClaimOf("test", tc.dec)
if !got.Vetoed() {
t.Fatalf("no veto, want one about %q", tc.want)
}
if len(got.Veto) < len(tc.want) || got.Veto[:len(tc.want)] != tc.want {
t.Errorf("veto = %q, want it to start with %q", got.Veto, tc.want)
}
})
}
}
// A route with every slot filled must NOT be vetoed. The three arms are
// structural holes, not a tax on every decision.
func TestClaimOfCompleteRouteIsNotVetoed(t *testing.T) {
d := Decision{
Utterance: "напомни позвонить маме в семь", Stage: 1, Intent: IntentReminder,
Confidence: llmFullConfidence,
Slots: Slots{Text: "позвонить маме", Time: time.Now(), HasTime: true},
}
c := ClaimOf("llm", d)
if c.Vetoed() {
t.Errorf("complete reminder vetoed: %q", c.Veto)
}
if c.Band != claim.BandStructural {
t.Errorf("band = %v, want structural", c.Band)
}
}
func TestClaimOfCoverageAndFilledSlots(t *testing.T) {
d := Decision{
Utterance: "напомни позвонить маме", Stage: 0, Intent: IntentReminder,
Confidence: 1.0, Slots: Slots{Text: "позвонить маме"},
}
c := ClaimOf("reminder-wakeword", d)
// The grammar captures what follows the verb, so "напомни" itself is
// unexplained. Under-reporting is the safe direction.
if len(c.Consumed) != 2 || len(c.Unexplained) != 1 {
t.Errorf("consumed %q / unexplained %q, want 2 and 1", c.Consumed, c.Unexplained)
}
if got := c.Coverage(); got < 0.66 || got > 0.67 {
t.Errorf("coverage = %v, want about 2/3", got)
}
if c.Intent != string(IntentReminder) {
t.Errorf("intent = %q", c.Intent)
}
if len(c.Filled) != 1 || c.Filled[0] != "text" {
t.Errorf("filled = %q, want [text]", c.Filled)
}
if c.Claimant != "reminder-wakeword" {
t.Errorf("claimant = %q", c.Claimant)
}
}
// The point of V-565's "additive" constraint, asserted rather than trusted:
// building a claim reads a Decision and changes nothing about it, so the
// classifier floor and the two consumers of Confidence are untouched.
func TestClaimOfLeavesTheDecisionAlone(t *testing.T) {
r := New(Config{
Grammars: []Grammar{ReminderGrammar()},
Extractor: Extractor{},
Threshold: 0.55,
})
before, err := r.Route(context.Background(), "напомни полить цветы", time.Now())
if err != nil {
t.Fatalf("Route: %v", err)
}
after := before
_ = ClaimOf("reminder-wakeword", after)
if !reflect.DeepEqual(after, before) {
t.Errorf("ClaimOf mutated the decision: %+v vs %+v", after, before)
}
if before.Confidence != 1.0 {
t.Errorf("stage 0 confidence = %v, want 1.0 — the float still has to work", before.Confidence)
}
}
+230
View File
@@ -0,0 +1,230 @@
package eval
import (
"context"
"fmt"
"os"
"path/filepath"
"sort"
"testing"
"github.com/kami/maven/internal/router"
)
// Package-level note for V-565. The cascade's arbitration is list order, and
// the reason is that no two claimants report a comparable number. These tests
// measure what each claimant actually reports across the 91-case RU fixture,
// so the ordinal band set in docs/plans/19-dialogue-arbitration.md is argued
// from a distribution rather than from taste. They report and never assert:
// a ratchet here would freeze a number nobody has decided to hold yet.
// TestStage0Contention — how often more than one stage-0 grammar matches the
// same utterance. Every one of them reports Confidence 1.0, so where two
// match, list order is the entire decision and nothing in the Decision says a
// second rule wanted the turn.
func TestStage0Contention(t *testing.T) {
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
grammars := baselineGrammars(router.DefaultActMatcher{Fns: actFns})
t.Logf("stage 0: %d grammars over %d cases", len(grammars), len(f.Cases))
matched, contended := 0, 0
pairs := map[string]int{}
for _, c := range f.Cases {
claimants := matchingGrammars(grammars, c.Utterance)
if len(claimants) == 0 {
continue
}
matched++
if len(claimants) < 2 {
continue
}
contended++
t.Logf(" contended %s %q: %v (winner %q by order)", c.ID, c.Utterance, claimants, claimants[0])
for _, loser := range claimants[1:] {
pairs[claimants[0]+" beats "+loser]++
}
}
t.Logf("stage 0 claimed %d/%d cases, %d of those with more than one claimant", matched, len(f.Cases), contended)
for _, k := range sortedKeys(pairs) {
t.Logf(" %s ×%d", k, pairs[k])
}
}
// matchingGrammars — every grammar whose pattern matches AND whose Build
// accepts, in the daemon's order. Route stops at the first; this does not.
func matchingGrammars(grammars []router.Grammar, utterance string) []string {
stripped, hadWake := router.StripWakeToken(utterance)
var out []string
for _, g := range grammars {
m := g.Pattern.FindStringSubmatch(utterance)
if m == nil && hadWake {
m = g.Pattern.FindStringSubmatch(stripped)
}
if m == nil {
continue
}
if _, ok := g.Build(m); !ok {
continue
}
out = append(out, g.Name)
}
return out
}
// TestClaimConfidenceDistributionHash — the confidence each claimant reports,
// on the deterministic hash embedder so it runs anywhere. The ONNX run below
// is the one whose cosines are the deployed numbers.
func TestClaimConfidenceDistributionHash(t *testing.T) {
reportConfidences(t, "hash", router.NewHashEmbedder(1024))
}
// TestONNXClaimConfidenceDistribution — the same measurement on the embedder
// homesrv runs, so the cosine column is the real one. Opt-in via
// MAVEN_ONNX_LIB, same as TestONNXBaseline, and one TestONNX* per process.
func TestONNXClaimConfidenceDistribution(t *testing.T) {
lib := os.Getenv("MAVEN_ONNX_LIB")
if lib == "" {
t.Skip("MAVEN_ONNX_LIB unset — see AGENTS.md § Embedder model for intent routing")
}
model := filepath.Join("../../..", "models/embedder/multilingual-e5-small/model_quantized.onnx")
tok := filepath.Join("../../..", "models/embedder/multilingual-e5-small/tokenizer.json")
for _, p := range []string{lib, model, tok} {
if _, err := os.Stat(p); err != nil {
t.Skipf("missing %s: %v", p, err)
}
}
emb, err := router.NewONNXEmbedder(model, tok, lib)
if err != nil {
t.Skipf("onnx embedder unavailable: %v", err)
}
defer emb.Close()
reportConfidences(t, "onnx", emb)
}
// reportConfidences runs the fixture through the deployed cascade and buckets
// the reported confidence by which layer produced it, then reports how well
// each bucket predicts a correct route. A band is only worth defining if the
// accuracy inside it differs from the accuracy outside it.
func reportConfidences(t *testing.T, name string, emb router.Embedder) {
t.Helper()
f, err := Load()
if err != nil {
t.Fatalf("Load: %v", err)
}
now, err := f.Now()
if err != nil {
t.Fatalf("Now: %v", err)
}
r := newBaselineRouter(t, emb, nil)
cls := newBaselineClassifier(t, emb)
type bucket struct{ n, correct int }
byValue := map[string]*bucket{}
byMargin := map[string]*bucket{}
var cosines, margins []float64
for _, c := range f.Cases {
d, err := r.Route(context.Background(), c.Utterance, now)
if err != nil {
t.Fatalf("%s: %v", c.ID, err)
}
layer := "classifier"
if d.Stage == 0 {
layer = "stage0"
} else {
cosines = append(cosines, d.Confidence)
}
key := fmt.Sprintf("%s conf=%.2f", layer, d.Confidence)
if layer == "classifier" {
key = fmt.Sprintf("%s conf=%.1f..%.1f", layer, floorTo(d.Confidence, 0.1), floorTo(d.Confidence, 0.1)+0.1)
}
b := byValue[key]
if b == nil {
b = &bucket{}
byValue[key] = b
}
ok := routeCorrect(c, d)
b.n++
if ok {
b.correct++
}
// The margin between the classifier's top two intents is the other
// float one could call a confidence. Measured on the same cases, so
// the ledger's "is a calibrated float available cheaply" question
// gets an answer instead of an assumption.
if layer != "classifier" {
continue
}
res, err := cls.Classify(context.Background(), c.Utterance)
if err != nil || len(res) < 2 {
continue
}
margin := res[0].Score - res[1].Score
margins = append(margins, margin)
mk := fmt.Sprintf("margin %.2f..%.2f", floorTo(margin, 0.02), floorTo(margin, 0.02)+0.02)
mb := byMargin[mk]
if mb == nil {
mb = &bucket{}
byMargin[mk] = mb
}
mb.n++
if ok {
mb.correct++
}
}
t.Logf("%s: confidence buckets over %d cases (correct = right intent, or clarified when the fixture wants a refusal)", name, len(f.Cases))
for _, k := range sortedKeys2(byValue) {
b := byValue[k]
t.Logf(" %-32s n=%2d correct=%2d (%.0f%%)", k, b.n, b.correct, 100*float64(b.correct)/float64(b.n))
}
if len(cosines) > 0 {
sort.Float64s(cosines)
t.Logf(" classifier cosine spread: min %.3f p25 %.3f p50 %.3f p75 %.3f max %.3f",
cosines[0], cosines[len(cosines)/4], cosines[len(cosines)/2],
cosines[3*len(cosines)/4], cosines[len(cosines)-1])
}
if len(margins) > 0 {
sort.Float64s(margins)
t.Logf(" classifier top1-top2 margin: min %.3f p50 %.3f max %.3f",
margins[0], margins[len(margins)/2], margins[len(margins)-1])
for _, k := range sortedKeys2(byMargin) {
b := byMargin[k]
t.Logf(" %-32s n=%2d correct=%2d (%.0f%%)", k, b.n, b.correct, 100*float64(b.correct)/float64(b.n))
}
}
}
// routeCorrect — the intent contract only. Slots are a parser question and
// would blur what the confidence number is being asked to predict.
func routeCorrect(c Case, d router.Decision) bool {
if c.WantClarify {
return d.Clarify
}
return d.Intent == c.Intent && !d.Clarify
}
func floorTo(v, step float64) float64 {
return float64(int(v/step)) * step
}
func sortedKeys(m map[string]int) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
func sortedKeys2[T any](m map[string]T) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
+31 -13
View File
@@ -216,6 +216,27 @@ func TestONNXBaseline(t *testing.T) {
func newBaselineRouter(t *testing.T, emb router.Embedder, llmR *router.LLMRouter) *router.Router {
t.Helper()
acts := router.DefaultActMatcher{Fns: actFns}
cls := newBaselineClassifier(t, emb)
return router.New(router.Config{
Grammars: baselineGrammars(acts),
Classifier: cls,
Extractor: router.Extractor{
Time: router.StubDateTimeParser{},
Acts: acts,
Facts: router.DefaultFactParser{},
},
// The deployed gate, not a test-local one: a fixture scored at a looser
// threshold reports an accuracy no real turn would see.
Threshold: config.DefaultRouterThreshold,
LLM: llmR,
})
}
// newBaselineClassifier — the seeded nearest-centroid classifier the cascade
// runs. Split out of newBaselineRouter so the claim measurement can ask it for
// its full ranking, not just the winner the Decision carries.
func newBaselineClassifier(t *testing.T, emb router.Embedder) *router.Classifier {
t.Helper()
cls := router.NewClassifier(emb)
ctx := context.Background()
seeds := seedsWithIntent(t)
@@ -231,6 +252,15 @@ func newBaselineRouter(t *testing.T, emb router.Embedder, llmR *router.LLMRouter
t.Fatalf("seed %q: %v", text, err)
}
}
return cls
}
// baselineGrammars — the stage-0 rule set in the daemon's order (buildRouter in
// cmd/mavend/voicewire.go). Split out of newBaselineRouter so the claim
// measurement can run the same rules one at a time and see which of them
// contend for the same utterance, which the cascade hides by stopping at the
// first match.
func baselineGrammars(acts router.ActMatcher) []router.Grammar {
grammars := router.DefaultGrammars(acts)
grammars = append(grammars, router.SystemTimeDateGrammars()...)
// Same order as buildRouter (voicewire.go). The fixture is only worth
@@ -248,19 +278,7 @@ func newBaselineRouter(t *testing.T, emb router.Embedder, llmR *router.LLMRouter
// "расскажи про X" is a world question the model called a fact, and the
// rule goes last because it matches on the first word alone (Vikunja #498).
grammars = append(grammars, router.NarrativeQueryGrammars()...)
return router.New(router.Config{
Grammars: grammars,
Classifier: cls,
Extractor: router.Extractor{
Time: router.StubDateTimeParser{},
Acts: acts,
Facts: router.DefaultFactParser{},
},
// The deployed gate, not a test-local one: a fixture scored at a looser
// threshold reports an accuracy no real turn would see.
Threshold: config.DefaultRouterThreshold,
LLM: llmR,
})
return grammars
}
// seedOrder — fixed iteration order over the corpus. Not cosmetic: a few