Record the quiet-box router numbers #14

2026-07-31T09:35:26+02:00

claude commented

2026-07-31 09:35:26 +02:00

One commit, doc only.

The prompt-fix PR quoted numbers measured while another eval was sharing llama-server. This is the same fixture re-run with the box to itself, appended to ROUTING-EVAL-31-07-2026.md. The baseline table is untouched — it stays as the before.

	llm-only	cascade+llm	thinking off
intent	48.7% → 61.8%	50.0% → 63.2%	67.1%
intent+slots	23.7% → 38.2%	32.9% → 47.4%	42.1%
p50 latency	1.08s	1.04s	0.93s

Three things it settles:

The gain is real. The contended run said 60.5%; the quiet one says 61.8%. And the scary 4-5s p50 from that run was contention, not the model — it is ~1.0s.
note→fact ×4 is real too. It shows up here as well, so it is not noise. The agent that wrote the prompt fix suspected its own edit caused it by pulling assertive запиши что… phrasings toward fact. That suspicion stands and is worth chasing (#375).
Thinking off is still the best config measured, on accuracy and latency (#376).

Missed clarify is still 6/6 — the router cannot say "I don't know" (#359), which PR 15 fixes.

One commit, doc only. The prompt-fix PR quoted numbers measured while another eval was sharing llama-server. This is the same fixture re-run with the box to itself, appended to `ROUTING-EVAL-31-07-2026.md`. **The baseline table is untouched** — it stays as the before. | | llm-only | cascade+llm | thinking off | |---|---|---|---| | intent | 48.7% → **61.8%** | 50.0% → **63.2%** | **67.1%** | | intent+slots | 23.7% → **38.2%** | 32.9% → **47.4%** | **42.1%** | | p50 latency | **1.08s** | **1.04s** | **0.93s** | Three things it settles: - **The gain is real.** The contended run said 60.5%; the quiet one says 61.8%. And the scary 4-5s p50 from that run was contention, not the model — it is ~1.0s. - **`note→fact ×4` is real too.** It shows up here as well, so it is not noise. The agent that wrote the prompt fix suspected its own edit caused it by pulling assertive `запиши что…` phrasings toward fact. That suspicion stands and is worth chasing (#375). - **Thinking off is still the best config measured**, on accuracy *and* latency (#376). Missed clarify is still 6/6 — the router cannot say "I don't know" (#359), which PR 15 fixes.

claude changed target branch from master to overnight/router-prompt

2026-07-31 10:39:21 +02:00

claude changed target branch from overnight/router-prompt to overnight/eval-rerun-base

2026-07-31 10:42:32 +02:00

claude added 1 commit 2026-07-31 10:42:32 +02:00

Re-measure the router on a quiet box and record the numbers 0ed386eca6

The earlier before/after was taken while another eval shared
llama-server. This run had the box to itself.

Intent accuracy 61.8% llm-only, 63.2% cascade, 67.1% with thinking off.
The prompt fix holds. note→fact shows up here too, so it is real.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

kami commented

2026-07-31 18:46:32 +02:00

is this one even worth reviewing?

kami commented

2026-07-31 20:21:51 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:21:52 +02:00

kami referenced this pull request

2026-07-31 22:34:49 +02:00

jul31 session: five small fixes plus the voice.go/ipc decomposition #50

kami referenced this issue from a commit

2026-08-01 00:33:54 +02:00

Capture tasks, with one intake seam mail can call later (#130)

claude referenced this pull request

2026-08-01 00:34:24 +02:00

Capture tasks, with one intake seam mail can call later (#130) #60

kami referenced this issue from a commit