Score the routing fixture with the ONNX embedder (Vikunja #319)
The onnxruntime .so was already vendored at deps/onnxruntime-linux-x64-1.26.0 — nothing to download. make eval-router now defaults MAVEN_ONNX_LIB there, so both baselines run by default and only a fresh clone without deps/ falls back to the hash ratchet alone. Prod-representative result, deployed 0.55 gate: 28/76 (36.8%), RU 25/61, EN 3/15, hard 0/11 → 4/11, p50 31ms / p95 71ms. Versus the hash floor's 13/76 at p50 9µs. The finding is not the accuracy, it's the refusal lane: missed clarifies went 0 → 5 of 6. Better embeddings raise cosine everywhere, so the 0.55 threshold that used to hold ambiguous utterances back stops holding — "сделай это" routes to act at 0.847, "бэкап" to chat at 0.755. The gate was implicitly tuned to the hash floor's low similarities. That is an argument about the threshold, not about the embedder, and it lands before #320 rather than after. Also fixes a fixture-model mismatch: ReminderGrammar deliberately skips the extractor at stage 0 and the daemon's applyAction parses the time downstream (stage0.go says so). Charging the router for that slot made 4 exact-match wins read as misses; they are now counted as SlotsDeferred instead. Hash baseline moves 13/76, ratchet to 0.15. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
This commit is contained in:
@@ -135,6 +135,10 @@ type Report struct {
|
||||
// act is a confident destructive guess.
|
||||
MissedClarify int
|
||||
Errors int
|
||||
// SlotsDeferred — stage-0 hits whose slot the daemon fills downstream
|
||||
// (reminder grammar → applyAction's time parser). Not a miss, but not a
|
||||
// full router-level win either; tracked so the two aren't conflated.
|
||||
SlotsDeferred int
|
||||
Outcomes []Outcome
|
||||
// Confusion counts want→got intent pairs, decided cases only.
|
||||
Confusion map[string]int
|
||||
@@ -218,7 +222,17 @@ func Score(ctx context.Context, name string, r Router, f Fixture) (Report, error
|
||||
o.Reasons = append(o.Reasons, fmt.Sprintf("intent %q, want %q (%.3f)", d.Intent, c.Intent, d.Confidence))
|
||||
}
|
||||
if c.WantTime && !d.Slots.HasTime {
|
||||
o.Reasons = append(o.Reasons, "no time slot, want one")
|
||||
// Stage 0 skips the extractor by design: ReminderGrammar
|
||||
// captures the text after "напомни"/"remind me" and the
|
||||
// daemon's applyAction runs the time parser on it (see
|
||||
// stage0.go). Charging the router for a slot it was never
|
||||
// asked to fill would make an exact-match win look like a
|
||||
// miss — so it is counted, not failed.
|
||||
if d.Stage == 0 {
|
||||
rep.SlotsDeferred++
|
||||
} else {
|
||||
o.Reasons = append(o.Reasons, "no time slot, want one")
|
||||
}
|
||||
}
|
||||
if c.WantFn && !d.Slots.HasFn {
|
||||
o.Reasons = append(o.Reasons, "no fn slot, want an allowlist match")
|
||||
@@ -282,8 +296,8 @@ func (r Report) String() string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "%s: %d/%d cases (%.1f%% full, %.1f%% intent-only)\n",
|
||||
r.Name, r.Passed, r.Total, 100*r.Accuracy(), 100*r.IntentAccuracy())
|
||||
fmt.Fprintf(&b, " clarify: %d false (asked, shouldn't) / %d missed (guessed, shouldn't) | errors: %d\n",
|
||||
r.FalseClarify, r.MissedClarify, r.Errors)
|
||||
fmt.Fprintf(&b, " clarify: %d false (asked, shouldn't) / %d missed (guessed, shouldn't) | errors: %d | slots deferred to daemon: %d\n",
|
||||
r.FalseClarify, r.MissedClarify, r.Errors, r.SlotsDeferred)
|
||||
fmt.Fprintf(&b, " latency: p50 %s p95 %s max %s\n", r.P50, r.P95, r.Max)
|
||||
fmt.Fprintf(&b, " by lang: %s\n", renderStats(r.ByLang))
|
||||
fmt.Fprintf(&b, " by tag: %s\n", renderStats(r.ByTag))
|
||||
|
||||
Reference in New Issue
Block a user