diff --git a/docs/evals/2026-08-06-claimant-interleave.md b/docs/evals/2026-08-06-claimant-interleave.md new file mode 100644 index 0000000..aeb550a --- /dev/null +++ b/docs/evals/2026-08-06-claimant-interleave.md @@ -0,0 +1,93 @@ +# Nine claimants interrupting a parked reminder + +Measured 2026-08-06 on homesrv against master at `70b32af`, so after V-560, V-565, V-567 +and V-572. Probed through `POST /api/chat` on mavweb, three turns per scenario, an explicit +`отмена` between scenarios so each starts from an empty pending slot. + +The question this answers: the Rome failure was one claimant eating one question. How many +other claimants do the same, and does routing first fix all of them or only weather? + +## Time forms, no pending state + +V-572 holds for every spoken shape, not just the one it was measured on. + +| said | answered | +|---|---| +| `напомни в 11:00 позвонить маме` | `хорошо, напомню сегодня в 11:00.` | +| `напомни в 9 позвонить маме` | `хорошо, напомню сегодня в 09:00.` | +| `напомни в 9 утра позвонить маме` | `хорошо, напомню сегодня в 09:00.` | +| `напомни завтра в 9 купить молоко` | `хорошо, напомню завтра в 09:00.` | +| `напомни в девять позвонить маме` | `хорошо, напомню сегодня в 09:00.` | +| `напомни через час выпить воды` | `хорошо, напомню сегодня в 02:26.` | + +The spelled-out `девять` works, which is the lexicon's oblique cardinals doing their job. + +## The nine scenarios + +Each opens with a reminder that names no time, so a clarify is parked. Then a different +claimant speaks. + +| | interrupted by | claiming source | flow | verdict | +|---|---|---|---|---| +| A | `какая сейчас погода в Риме?` | weather | dropped, announced | question answered | +| B | `что я говорил про отпуск?` | personal | dropped, announced | question answered | +| C | `какая столица Франции?` | search | dropped, announced | question answered | +| D | `у меня новый ноутбук` | none | kept | **note silently discarded** | +| E | `что у меня сегодня?` | none | completed wrongly | **reminder invented at the clock** | +| F | `что требует внимания?` | none | dropped, announced | question answered | +| G | `напомни завтра в 9 купить молоко` | none | dropped, announced | new reminder set | +| H | reminder said whole, then weather, then a repair | weather | none parked | **repair invents a note** | +| I | `сколько времени?` | none | dropped, announced | question answered | + +Six of nine answer the interrupting utterance. That is what V-560 bought, and it is the +half of Rome that was broken. + +## The three that fail + +**E is the worst of the nine.** + +```text +me: напомни забрать посылку +maven: Когда? +me: что у меня сегодня? +maven: хорошо, напомню сегодня в 01:28. +``` + +The agenda question is never answered and a reminder exists that nobody asked for, at the +current minute. Asked with nothing parked, the same sentence is answered by the calendar. +So stage 0 routes it correctly and the pending claimant runs in front of that route. This +is worse than the pre-V-560 Rome behaviour, which at least failed audibly. Filed as V-577. + +**D drops a note without saying so.** + +```text +me: напомни позвонить врачу +maven: Когда? +me: у меня новый ноутбук +maven: Во сколько напомнить? +``` + +Nothing wrong is written, and nothing right is either. Alone the sentence is stored. Also +V-577. + +**H writes a note nobody dictated.** + +```text +нет, не маме, а папе -> Сохранила заметку о том, что ты поедешь на дачу. +``` + +Reproduced with no pending state at all, twice, with two different inventions. The stored +body is generated rather than captured. Filed as V-576. It is the only defect here that outlives +the turn. A wrong note is indexed and comes back later as recall. + +## What the drop notice looks like now + +Six scenarios answer the question and say `Прошлую просьбу отпускаю.` first. That sentence +is rejected by the owner. V-561 replaces it with suspend and resume for a side query, and +keeps it for a new request (G) and a cancel. + +## Reading of the set + +The pattern behind Rome, V-567 and V-577 is one pattern. The claimant that knows least +about the utterance holds the earliest and strongest trigger. V-560 moved the routing +in front of one resolver. The other roles still decide before the route is read.