Files
Maven/docs/evals/2026-08-08-two-weeks-after-merge.md
T
claude bbefda66e2 Read the source column off the badge, not off the wording (V-662)
The third run of the same 140 turns, with the harness fix in. Sixty-eight
turns name a source.

Two findings the wording could not carry. The unfixed homelab turns are
claimed by weather and by feeds, which the destination fixture predicted.
And agenda questions are claimed by the personal boundary and by Praxis,
not by the calendar: 3 of 6, the same 3 of 6 the destination fixture and
every routing-head seed score.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
2026-08-08 21:31:22 +04:00

5.8 KiB
Raw Blame History

The same fortnight, after V-655 merged

Date: 2026-08-08, a few hours after 2026-08-08-two-weeks.md. Build: f8fa0d1 on master, the five compose services rebuilt and recreated. Same driver, same 140 turns, same reach. This is the diff that baseline was for.

Master now carries V-655, V-659, V-660 and V-661. The change under test is V-655. A query source that decides by seed similarity is marked guesses: true. It is dropped when the cascade names a different destination.

Two things confound the comparison and one of them matters

The workstation was up for the re-run. llama-server on 192.168.1.105 answered a health probe with 200. So routing completed through llm.Pair against gemma-4-12b, which is the arm that names a destination. Its state during the baseline was not recorded. So a difference here may be the merge, or may be the better router, and this run cannot separate them.

The store carried over, as the baseline said it would. Facts written by the first run were present from turn 1 of the second.

Numbers

baseline beb093a after f8fa0d1
turns 140 140
p50 1.5s 1.2s
p95 7.1s 3.0s
max 33.7s 4.2s
transport errors 0 0
turns carrying a failure string 41 38
string in the reply before after
на какое время поставить напоминание 13 13
не нашла у тебя такой записи 8 9
Такую команду я не знаю 8 8
для какого города 6 4
В какой день 6 6
пока не умею 5 1
Когда? 3 3

Read the latency as unattributed. The workstation confound covers all of it.

Defect 2 is the one this was for: four of six fixed

utterance before after
что такое TCP? для какого города? a real definition
сколько будет 17 на 23? для какого города? search, which has no answer
какой у меня любимый язык? kernel headlines не нашла у тебя такой записи
что я сохранил про Сочи? Хорошо, сохраню. answered as a question
какая скорость у меня сейчас? для какого города? для какого города?
хватает ли места под новые бэкапы? kernel headlines kernel headlines

что такое TCP? is the clean win. WorldQueryGrammars names world at stage 0, weather is dropped, and search answers.

сколько будет 17 на 23? moved source and not outcome. Weather no longer claims it. Search cannot do arithmetic, so the reply says the sources have no product of 17 and 23. That is an honest gap where it used to be a wrong question. Arithmetic has no destination in the enum.

что я сохранил про Сочи? was defect 3 and it is gone. The utterance is no longer read as a capture.

The two that did not move are both homelab questions. They are exactly the cluster the destination fixture flagged. SourceRecall, SourceNetwork and SourceAttention overlap on every question about the box. Five of the seven floor cases in that fixture are homelab operations for the same reason. So this is the enum, not the walk.

Defect 1 did not move at all

Twenty-six turns still carry a parked clarify tail, the same count as the baseline. спасибо тебе answers Сейчас 21:12. Это утра или вечера? and спокойной ночи answers хорошо, напомню послезавтра в 10:00.

V-655 was never going to touch this. A parked clarify is dialogue state and not a query source. It remains the single worst thing about talking to her. The week test, the fortnight test and this re-run all report it unchanged.

A gap in the harness, fixed and re-run the same day

ipc.ChatReply.Source came back empty on all 140 turns, in both runs. The driver read the redirect parameter src and cmd/mavweb/chat.go writes s. So every finding above is read off the reply text instead of off the badge.

Fixed in V-662 and the 140 turns were driven a third time. Sixty-eight of them name a source. The rest are not query turns and never reach queryWalk.

source turns
search 27
memory 13
personal 9
weather 5
calendar 3
attention 3
list 2
feeds 2
tasks, money, self, habits 1 each

What the badge shows that the wording did not

The two unfixed homelab turns are now direct evidence.

какая скорость у меня сейчас?        -> weather
хватает ли места под новые бэкапы?   -> feeds

Both are guessing sources claiming a turn about the box, exactly as the destination fixture predicted.

The badge also names a defect the wording hid. Agenda questions are being claimed by the personal boundary and by Praxis, not by the calendar.

во сколько у меня встреча?   -> personal    не нашла у тебя такой записи
когда у меня встреча?        -> attention   у Praxis нет источников
что у меня в понедельник?    -> personal    не нашла у тебя такой записи

Calendar claimed 3 turns of the 6 that asked about the calendar. That is the same 3/6 the destination fixture scores and the same 3/6 every seed of the routing head scores. Three measurements agree. The cause is the one V-660 named. The possessive agenda rules claim these at stage 0 and name no destination, so the walk reaches personal and attention first.

This is the third independent confirmation that the possessive agenda rules should name the calendar. That call is still the owner's.