From 0ed386eca61b73491a5bd5b0dec47278a96c4a1c Mon Sep 17 00:00:00 2001 From: kami Date: Fri, 31 Jul 2026 11:34:57 +0400 Subject: [PATCH] Re-measure the router on a quiet box and record the numbers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The earlier before/after was taken while another eval shared llama-server. This run had the box to itself. Intent accuracy 61.8% llm-only, 63.2% cascade, 67.1% with thinking off. The prompt fix holds. note→fact shows up here too, so it is real. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ --- ROUTING-EVAL-31-07-2026.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/ROUTING-EVAL-31-07-2026.md b/ROUTING-EVAL-31-07-2026.md index 7c95565..87d2bb5 100644 --- a/ROUTING-EVAL-31-07-2026.md +++ b/ROUTING-EVAL-31-07-2026.md @@ -40,6 +40,41 @@ model → classifier as failure floor. Never compare a hash-embedder run to an ONNX one. +## Re-measured after the prompt fix + +The table above is the **baseline at commit `46259b4`**, kept as-is. The prompt fix (query +tested before fact, plus `repeat_penalty` and a bounded grammar string) was then measured on +an otherwise idle box — no other eval sharing llama-server, so these latencies are real +rather than contention. + +| | llm-only (0.8B) | cascade+llm (0.8B) | llm-only, thinking off | +|---|---|---|---| +| **intent-only accuracy** | 48.7% → **61.8%** | 50.0% → **63.2%** | **67.1%** | +| full accuracy (intent+slots+gate) | 23.7% → **38.2%** | 32.9% → **47.4%** | **42.1%** | +| route errors | 2 → **0** | 0 → 0 | **0** | +| p50 / p95 latency | **1.08s / 1.55s** | **1.04s / 1.53s** | **0.93s / 1.41s** | + +Three things this run settles: + +1. **The prompt fix holds.** An earlier contended run reported 60.5% / 36.8% for llm-only; + the quiet run gives 61.8% / 38.2%. Close enough to call the gain real, and the earlier + run's 4-5s latency figures were contention, not the model. +2. **`query→fact` fell from ×15 to ×7**, and both unparseable replies are gone. Zero route + errors in every LLM configuration. +3. **`note→fact ×4` is real, not noise.** It shows up in the quiet run too. The agent that + wrote the prompt fix suspected its own change might have caused it by pulling assertive + `запиши что…` phrasings toward fact, and that suspicion stands — all five `ru-note-*` + cases now land on fact. Tracked as Vikunja #375. + +**Thinking off is the best configuration measured so far**, on both accuracy and latency +(Vikunja #376). That is worth understanding before flipping: routing is a short +classification into a fixed enum with grammar-constrained output, so there is little to +reason about, and the thinking trace mostly gives a small model room to talk itself out of +the right answer. Phrasing is a different job and needs measuring separately. + +Still `6 / 6` missed clarify — the router has no way to say "I don't know" (Vikunja #359). +That is unchanged by anything here. + ## Findings ### 1. The resident model does route better — 50.0% vs 36.8%