Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0ed386eca6 |
@@ -40,6 +40,41 @@ model → classifier as failure floor.
|
|||||||
|
|
||||||
Never compare a hash-embedder run to an ONNX one.
|
Never compare a hash-embedder run to an ONNX one.
|
||||||
|
|
||||||
|
## Re-measured after the prompt fix
|
||||||
|
|
||||||
|
The table above is the **baseline at commit `46259b4`**, kept as-is. The prompt fix (query
|
||||||
|
tested before fact, plus `repeat_penalty` and a bounded grammar string) was then measured on
|
||||||
|
an otherwise idle box — no other eval sharing llama-server, so these latencies are real
|
||||||
|
rather than contention.
|
||||||
|
|
||||||
|
| | llm-only (0.8B) | cascade+llm (0.8B) | llm-only, thinking off |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **intent-only accuracy** | 48.7% → **61.8%** | 50.0% → **63.2%** | **67.1%** |
|
||||||
|
| full accuracy (intent+slots+gate) | 23.7% → **38.2%** | 32.9% → **47.4%** | **42.1%** |
|
||||||
|
| route errors | 2 → **0** | 0 → 0 | **0** |
|
||||||
|
| p50 / p95 latency | **1.08s / 1.55s** | **1.04s / 1.53s** | **0.93s / 1.41s** |
|
||||||
|
|
||||||
|
Three things this run settles:
|
||||||
|
|
||||||
|
1. **The prompt fix holds.** An earlier contended run reported 60.5% / 36.8% for llm-only;
|
||||||
|
the quiet run gives 61.8% / 38.2%. Close enough to call the gain real, and the earlier
|
||||||
|
run's 4-5s latency figures were contention, not the model.
|
||||||
|
2. **`query→fact` fell from ×15 to ×7**, and both unparseable replies are gone. Zero route
|
||||||
|
errors in every LLM configuration.
|
||||||
|
3. **`note→fact ×4` is real, not noise.** It shows up in the quiet run too. The agent that
|
||||||
|
wrote the prompt fix suspected its own change might have caused it by pulling assertive
|
||||||
|
`запиши что…` phrasings toward fact, and that suspicion stands — all five `ru-note-*`
|
||||||
|
cases now land on fact. Tracked as Vikunja #375.
|
||||||
|
|
||||||
|
**Thinking off is the best configuration measured so far**, on both accuracy and latency
|
||||||
|
(Vikunja #376). That is worth understanding before flipping: routing is a short
|
||||||
|
classification into a fixed enum with grammar-constrained output, so there is little to
|
||||||
|
reason about, and the thinking trace mostly gives a small model room to talk itself out of
|
||||||
|
the right answer. Phrasing is a different job and needs measuring separately.
|
||||||
|
|
||||||
|
Still `6 / 6` missed clarify — the router has no way to say "I don't know" (Vikunja #359).
|
||||||
|
That is unchanged by anything here.
|
||||||
|
|
||||||
## Findings
|
## Findings
|
||||||
|
|
||||||
### 1. The resident model does route better — 50.0% vs 36.8%
|
### 1. The resident model does route better — 50.0% vs 36.8%
|
||||||
|
|||||||
Reference in New Issue
Block a user