kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Say that a transport error is not a wrong answer (V-661)

Browse Source
This commit is contained in:
claude
2026-08-08 20:47:32 +04:00
parent ed1730431c
commit 6bc71553ab
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evals/2026-08-08-two-weeks.md
+4 -1
View File
@@ -30,7 +30,7 @@ present when a re-run reaches day 1.
| p50 | 1.5s | 1.6s | | p50 | 1.5s | 1.6s |
| p95 | 8.0s | 7.1s | | p95 | 8.0s | 7.1s |
| max | 12.3s | 33.7s | | max | 12.3s | 33.7s |
| errors | 0 | 0 | | transport errors | 0 | 0 |
| string in the reply | turns | | string in the reply | turns |
|---|---| |---|---|
@@ -42,6 +42,9 @@ present when a re-run reaches day 1.
| `пока не умею` | 5 | | `пока не умею` | 5 |
| `Когда?` | 3 | | `Когда?` | 3 |
**Zero transport errors is not zero wrong answers.** It counts turns that
failed to return a reply, and none did. Every quality number is below.
Those seven strings appear in 49 of 140 turns. Some turns carry two, because a Those seven strings appear in 49 of 140 turns. Some turns carry two, because a
parked clarify appends to whatever else was said. parked clarify appends to whatever else was said.