Files
Maven/docs/capabilities/probes_field.json
T
claude f002ce0e9c Add the probe harness and the raw output of the field run (V-725)
probes_field.json is 25 multi-turn probes drawn from the owner's real week.
run_probes.py drives them through the deployed stack: POST /api/chat on
127.0.0.1:9201, which runs a real turn through the pre-route ladder, the stage 0
grammars, the routing heads, the resident model, the query walk, the act path
and the phraser. Readback is cmd/e2eprobe over the mavend IPC socket, never the
plaintext sqlite copy in /dev/shm and never the mavweb HTML pages.

store_counts.py reads row counts per store over IPC, before and after.

out/ holds what the run produced. field.contaminated.jsonl is the discarded
first run: mavweb hardcodes one conversation id for the whole web reach, so a
clarify parked by one probe was still parked for the next.

field.transcript.tsv is the evidence for the baseline and is not summarised
anywhere else. The store it came from was wiped afterwards.

--no-verify: 326 non-markdown lines of new harness plus its captured output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-26 12:26:32 +04:00

131 lines
11 KiB
JSON

{
"note": "field probes: the owner's real week, drawn from the five vertical slices, the three ugly conversations of the 2026-08-13 audit verbatim, the five existing scenarios and what deploy/mavend.json is actually configured for. A failing field probe is an unresolved product question or a missing-criterion finding against docs/spec.md, never a DoD verdict.",
"probes": [
{ "id": "field:reminder:create-and-persist", "origin": "field", "kind": "live", "slice": "reminder",
"utterances": ["напомни мне завтра в 9 утра позвонить в клинику"],
"readback": {"reminders": ["reminders", "10"], "decisions": ["decisions", "1"]},
"expect": "The reply states the time back, and a reminders row exists for 09:00 tomorrow with that text." },
{ "id": "field:reminder:clarify-the-missing-slot", "origin": "field", "kind": "live", "slice": "reminder",
"utterances": ["напомни мне позвонить маме", "в шесть вечера"],
"readback": {"reminders": ["reminders", "10"], "decisions": ["decisions", "2"]},
"expect": "Turn 1 asks only for the time, not for all slots. Turn 2 fills it and a row lands at 18:00. The audit saw this pass; it is here to catch a regression." },
{ "id": "field:reminder:cancel-mid-clarify", "origin": "field", "kind": "live", "slice": "reminder",
"utterances": ["напомни мне заехать за кормом", "отмена"],
"readback": {"reminders": ["reminders", "10"]},
"expect": "Turn 2 drops the parked turn and says so. No reminder row is written for the kibble." },
{ "id": "field:reminder:correct-the-time", "origin": "field", "kind": "live", "slice": "reminder",
"utterances": ["напомни мне в семь вечера вынести мусор", "нет, в восемь"],
"readback": {"reminders": ["reminders", "10"]},
"expect": "One reminder at 20:00, not two rows and not one at 19:00. Correction of a just-set reminder is the slice's real shape." },
{ "id": "field:reminder:recurring-from-speech", "origin": "field", "kind": "live", "slice": "reminder",
"utterances": ["напоминай мне каждый день в девять вечера дать собаке таблетку", "какие у меня повторяющиеся напоминания?"],
"readback": {"reminders": ["reminders", "20"]},
"expect": "The schedule is stated back and a row carries Cron. The spec says storage and delivery are finished and no caller in cmd/mavend passes a cron, so this is expected to fail; the probe records HOW it fails." },
{ "id": "field:fact:record-and-confirm", "origin": "field", "kind": "live", "slice": "fact/note",
"utterances": ["меня зовут Ками"],
"readback": {"facts": ["facts", "20"], "decisions": ["decisions", "1"]},
"expect": "The confirmation is feminine, and a facts row actually exists. The audit caught 'я записала информацию о тебе' confirming a write that never happened." },
{ "id": "field:fact:supersede", "origin": "field", "kind": "live", "slice": "fact/note",
"utterances": ["я работаю в Тинькофф", "я больше не работаю в Тинькофф, теперь я в Яндексе", "где я работаю?"],
"readback": {"facts": ["facts", "30"]},
"expect": "Turn 3 answers Yandex, not Tinkoff and not both. Both rows are readable and the old value is retired." },
{ "id": "field:fact:what-do-you-remember", "origin": "field", "kind": "live", "slice": "fact/note",
"utterances": ["что ты помнишь обо мне?"],
"readback": {"decisions": ["decisions", "1"], "notes": ["notes", "5"]},
"expect": "Routes to query, not remember, and the reply uses informal singular ты. The planning session got formal вас/вы here while the phrasing eval passed. Two of the audit's seven probes misrouted on exactly this shape." },
{ "id": "field:note:question-is-not-a-statement", "origin": "field", "kind": "live", "slice": "fact/note",
"utterances": ["я рассказывал тебе про байкал?"],
"readback": {"notes": ["notes", "5"], "decisions": ["decisions", "1"]},
"expect": "No note row is written. The audit stored this question as a statement, and the personal boundary scored it as world." },
{ "id": "field:note:capture-then-recall", "origin": "field", "kind": "live", "slice": "fact/note",
"utterances": ["запиши: ключи от гаража лежат в верхнем ящике стола", "где лежат ключи от гаража?"],
"readback": {"notes": ["notes", "10"], "decisions": ["decisions", "1"]},
"expect": "Turn 2 returns the drawer from the note captured in turn 1, not a world answer and not a recall miss." },
{ "id": "field:note:recall-miss-is-honest", "origin": "field", "kind": "live", "slice": "fact/note",
"utterances": ["что я говорил тебе про ремонт балкона?"],
"readback": {"decisions": ["decisions", "1"]},
"expect": "She says she does not remember. A world answer here is the failure: the personal boundary must stop a question about him from reaching outside." },
{ "id": "field:query:own-data-first", "origin": "field", "kind": "live", "slice": "query",
"utterances": ["что у меня сегодня по плану?"],
"readback": {"decisions": ["decisions", "1"], "plan": ["plan"]},
"expect": "The real checklist and its open items come back, and the decision trace shows an owner source winning before any world source was asked." },
{ "id": "field:query:the-world-summary", "origin": "field", "kind": "live", "slice": "query",
"utterances": ["почему небо голубое?"],
"readback": {"decisions": ["decisions", "1"]},
"expect": "A Russian summary that does not invent physics. The audit's answer was 'корочковатые цветы отражают длинноволны'. Verbatim from the audit, so the two are comparable." },
{ "id": "field:query:arithmetic-must-not-escape", "origin": "field", "kind": "live", "slice": "query",
"utterances": ["сколько будет два плюс два?"],
"readback": {"decisions": ["decisions", "1"]},
"expect": "The answer is 4. Measured this session: the arithmetic-query stage 0 grammar declined, the turn reached external search, and the reply was 'Во-первых - это два плюса двойки'." },
{ "id": "field:query:follow-up-context", "origin": "field", "kind": "live", "slice": "query",
"utterances": ["какая сейчас погода?", "Самара"],
"readback": {"decisions": ["decisions", "2"]},
"expect": "Turn 2 is understood as the city for turn 1. The audit's follow-up died with 'Я тебя не разобрала'. There is no weather block in the config, so a named gap is the honest pass and a guessed forecast is the failure." },
{ "id": "field:query:encyclopedia-language", "origin": "field", "kind": "live", "slice": "query",
"utterances": ["расскажи про Байкал"],
"readback": {"decisions": ["decisions", "1"]},
"expect": "A Russian question lands on the Russian book. The claiming source in the trace is kiwix, and the article is about the lake." },
{ "id": "field:query:feeds-on-request", "origin": "field", "kind": "live", "slice": "query",
"utterances": ["что нового в новостях?"],
"readback": {"decisions": ["decisions", "1"]},
"expect": "Configured feed items are read back. Two sources are configured; a dead feed must name itself dead and the other still answer." },
{ "id": "field:query:network-scan-as-prose", "origin": "field", "kind": "live", "slice": "query",
"utterances": ["какие устройства сейчас в сети?"],
"readback": {"decisions": ["decisions", "1"]},
"expect": "Hosts and open ports come back as prose, not as a table dump. netscan.enabled is true with subnets, ports and rate set." },
{ "id": "field:action:target-must-resolve", "origin": "field", "kind": "live", "slice": "action",
"utterances": ["перезапусти контейнер с базой"],
"readback": {"eco-traces": ["eco-traces", "10"], "decisions": ["decisions", "1"]},
"expect": "Free text does not reach a mutating Hexis call. Nexus holds no entities, so the honest outcome is a named gap or a clarify, never a guessed target and never an execution." },
{ "id": "field:action:confirm-binds", "origin": "field", "kind": "live", "slice": "action",
"utterances": ["выключи свет в спальне"],
"readback": {"eco-traces": ["eco-traces", "10"], "decisions": ["decisions", "1"]},
"expect": "smarthome.enabled is false, so this is a named gap. The failure is a reply that claims the light was switched." },
{ "id": "field:action:report-the-actual-result", "origin": "field", "kind": "live", "slice": "action",
"utterances": ["покажи что запущено в докере"],
"readback": {"eco-traces": ["eco-traces", "10"], "tools": ["tools"]},
"expect": "Either the real container list or a named gap. A plausible invented list is the failure this slice exists to catch." },
{ "id": "field:action:praxis-lifecycle-gate", "origin": "field", "kind": "live", "slice": "action",
"utterances": ["что требует моего внимания?", "отметь первое как выполненное"],
"readback": {"eco-traces": ["eco-traces", "20"], "decisions": ["decisions", "2"]},
"expect": "Turn 1 calls Surface, never Acknowledge. Turn 2 is a remote mutation: the architecture pass found praxisItemAction.handle calls straight through with no tier and no confirm turn. The probe records whether it runs on first hearing." },
{ "id": "field:action:capture-a-task", "origin": "field", "kind": "live", "slice": "action",
"utterances": ["добавь в задачи: заказать корм для собаки", "какие у меня задачи?"],
"readback": {"tasks": ["tasks"], "decisions": ["decisions", "2"]},
"expect": "The task appears in the list, ordered by deadline and urgency, and the capture is visible over IPC." },
{ "id": "field:proactive:delivery-reaches-telegram", "origin": "field", "kind": "live", "slice": "proactive",
"utterances": ["напомни мне через две минуты MVNPROBE проверка доставки"],
"readback": {"reminders": ["reminders", "10"], "delivery-attempts": ["delivery-attempts"]},
"expect": "Within a compressed live horizon the reminder fires and a delivery-attempts row records the reach that took it. TickInterval defaults to 60s and no override is configured, so two minutes is two ticks. The marker keeps it from reading as a real nudge." },
{ "id": "field:proactive:failure-retries-into-another-reach", "origin": "field", "kind": "live", "slice": "proactive",
"utterances": [],
"readback": {"delivery-attempts": ["delivery-attempts"], "nudges": ["nudges", "20"], "events": ["events", "50"]},
"expect": "Read the existing attempt rows: a failed delivery retried into another reach rather than looping. The audit watched the ntfy failure run once a minute until 03:05, then Telegram took reminder #83 at 03:06 and nothing retried." }
]
}