diff --git a/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md b/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md index 69c8f3c..727befe 100644 --- a/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md +++ b/docs/evals/2026-09-08-tiny-sequence-pragmatics-specialist.md @@ -1,4 +1,4 @@ -# Slice 19: from-scratch tiny sequence encoders halve the sparse gate's capability-question fail rate, and still land near chance on the leave-generator-out pragmatics split +# Slice 19: from-scratch tiny sequence encoders beat sparse and e5 on the capability-question holdout (seed-stable FA 0.48-0.55 vs 0.667 / ~0.98) yet cap at ~50% accuracy on the leave-generator-out pragmatics split — a pretrained prior is the next step Date: 2026-09-08 · Task: V-726 (slice 19) · Box: homesrv, Ryzen 5 5600U, 13 GB, CPU-only (this is the production machine) · Build: `cmd/semantic-router-experiment/slice19_*.py`, torch 2.14.0+cpu in `/tmp/mvn-exp-venv`, no GPU. @@ -152,7 +152,8 @@ positive — every row here must *not* trip the gate). Threshold 0.5, seed 17: | char_cnn_medium | 123 | 0.976 | 0.024 | Against the sparse gate (0.667 FA → 0.333 accuracy) the leading from-scratch -configs halve the fail rate. Against e5 (1.000 / 0.976) the win is decisive — +configs cut the fail rate by a quarter to a third on seed-stable seeds. Against +e5 (1.000 / 0.976) the win is decisive — the frozen embedder genuinely cannot separate a held-out capability question from its executable sibling.