From cf2af6ce1ec75c537eb80f604040b541f0deff29 Mon Sep 17 00:00:00 2001 From: claude Date: Mon, 7 Sep 2026 17:59:10 +0400 Subject: [PATCH] =?UTF-8?q?router/semantic:=20slice=2019=20report=20?= =?UTF-8?q?=E2=80=94=20correct=20claim=20head=20and=20prose=20to=20the=20s?= =?UTF-8?q?eed-stable=20fail-rate=20cut?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md b/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md index 69c8f3c..727befe 100644 --- a/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md +++ b/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md @@ -1,4 +1,4 @@ -# Slice 19: from-scratch tiny sequence encoders halve the sparse gate's capability-question fail rate, and still land near chance on the leave-generator-out pragmatics split +# Slice 19: from-scratch tiny sequence encoders beat sparse and e5 on the capability-question holdout (seed-stable FA 0.48-0.55 vs 0.667 / ~0.98) yet cap at ~50% accuracy on the leave-generator-out pragmatics split — a pretrained prior is the next step Date: 2026-09-08 · Task: V-726 (slice 19) · Box: homesrv, Ryzen 5 5600U, 13 GB, CPU-only (this is the production machine) · Build: `cmd/semantic-router-experiment/slice19_*.py`, torch 2.14.0+cpu in `/tmp/mvn-exp-venv`, no GPU. @@ -152,7 +152,8 @@ positive — every row here must *not* trip the gate). Threshold 0.5, seed 17: | char_cnn_medium | 123 | 0.976 | 0.024 | Against the sparse gate (0.667 FA → 0.333 accuracy) the leading from-scratch -configs halve the fail rate. Against e5 (1.000 / 0.976) the win is decisive — +configs cut the fail rate by a quarter to a third on seed-stable seeds. Against +e5 (1.000 / 0.976) the win is decisive — the frozen embedder genuinely cannot separate a held-out capability question from its executable sibling.