From 6bc71553abbe85eafd6e7fd138643f89bd29b743 Mon Sep 17 00:00:00 2001 From: claude Date: Sat, 8 Aug 2026 20:47:32 +0400 Subject: [PATCH] Say that a transport error is not a wrong answer (V-661) --- docs/evals/2026-08-08-two-weeks.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/evals/2026-08-08-two-weeks.md b/docs/evals/2026-08-08-two-weeks.md index 3ffbb23..bfb9f62 100644 --- a/docs/evals/2026-08-08-two-weeks.md +++ b/docs/evals/2026-08-08-two-weeks.md @@ -30,7 +30,7 @@ present when a re-run reaches day 1. | p50 | 1.5s | 1.6s | | p95 | 8.0s | 7.1s | | max | 12.3s | 33.7s | -| errors | 0 | 0 | +| transport errors | 0 | 0 | | string in the reply | turns | |---|---| @@ -42,6 +42,9 @@ present when a re-run reaches day 1. | `пока не умею` | 5 | | `Когда?` | 3 | +**Zero transport errors is not zero wrong answers.** It counts turns that +failed to return a reply, and none did. Every quality number is below. + Those seven strings appear in 49 of 140 turns. Some turns carry two, because a parked clarify appends to whatever else was said.