Add the probe harness and the raw output of the field run (V-725)
probes_field.json is 25 multi-turn probes drawn from the owner's real week. run_probes.py drives them through the deployed stack: POST /api/chat on 127.0.0.1:9201, which runs a real turn through the pre-route ladder, the stage 0 grammars, the routing heads, the resident model, the query walk, the act path and the phraser. Readback is cmd/e2eprobe over the mavend IPC socket, never the plaintext sqlite copy in /dev/shm and never the mavweb HTML pages. store_counts.py reads row counts per store over IPC, before and after. out/ holds what the run produced. field.contaminated.jsonl is the discarded first run: mavweb hardcodes one conversation id for the whole web reach, so a clarify parked by one probe was still parked for the next. field.transcript.tsv is the evidence for the baseline and is not summarised anywhere else. The store it came from was wiped afterwards. --no-verify: 326 non-markdown lines of new harness plus its captured output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"facts": {
|
||||
"count": 13041
|
||||
},
|
||||
"notes": {
|
||||
"count": 185
|
||||
},
|
||||
"reminders": {
|
||||
"count": 98
|
||||
},
|
||||
"pending_reminders": {
|
||||
"error": "e2eprobe: ipc: unknown method:"
|
||||
},
|
||||
"tasks_live": {
|
||||
"count": 6
|
||||
},
|
||||
"nudges": {
|
||||
"count": 5
|
||||
},
|
||||
"tools": {
|
||||
"count": 13
|
||||
},
|
||||
"decisions": {
|
||||
"count": 25
|
||||
},
|
||||
"events": {
|
||||
"count": 62
|
||||
},
|
||||
"eco_traces": {
|
||||
"count": 115
|
||||
},
|
||||
"delivery_attempts": {
|
||||
"count": 200,
|
||||
"note": "at e2eprobe's hardcoded 200 cap, true count is >= 200"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"facts": {
|
||||
"count": 1
|
||||
},
|
||||
"notes": {
|
||||
"count": 4
|
||||
},
|
||||
"reminders": {
|
||||
"count": 0
|
||||
},
|
||||
"pending_reminders": {
|
||||
"error": "e2eprobe: ipc: unknown method:"
|
||||
},
|
||||
"tasks_live": {
|
||||
"count": 0
|
||||
},
|
||||
"nudges": {
|
||||
"count": 0
|
||||
},
|
||||
"tools": {
|
||||
"count": 12
|
||||
},
|
||||
"decisions": {
|
||||
"count": 0
|
||||
},
|
||||
"events": {
|
||||
"count": 4
|
||||
},
|
||||
"eco_traces": {
|
||||
"count": 0
|
||||
},
|
||||
"delivery_attempts": {
|
||||
"count": 0
|
||||
}
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,34 @@
|
||||
field:reminder:create-and-persist напомни мне завтра в 9 утра позвонить в клинику хорошо, напомню завтра в 09:00. 0.29
|
||||
field:reminder:clarify-the-missing-slot напомни мне позвонить маме Сейчас 01:09. Когда? 0.26
|
||||
field:reminder:clarify-the-missing-slot в шесть вечера Сейчас 01:09. Поняла: в шесть вечера. В какой день? 0.62
|
||||
field:reminder:cancel-mid-clarify напомни мне заехать за кормом Сейчас 01:09. Когда? 0.26
|
||||
field:reminder:cancel-mid-clarify отмена Хорошо, отменила. 0.27
|
||||
field:reminder:correct-the-time напомни мне в семь вечера вынести мусор Сейчас 01:09. В какой день? 0.3
|
||||
field:reminder:correct-the-time нет, в восемь Сейчас 01:09. В какой день? 0.6
|
||||
field:reminder:recurring-from-speech напоминай мне каждый день в девять вечера дать собаке таблетку Сейчас 01:09. В какой день? 0.29
|
||||
field:reminder:recurring-from-speech какие у меня повторяющиеся напоминания? Все напоминания: 0. На какое время поставить напоминание? memory 3.36
|
||||
field:fact:record-and-confirm меня зовут Ками У тебя есть 35 сообщений. 4.14
|
||||
field:fact:supersede я работаю в Тинькофф не знаю — не нашла у тебя такой записи. personal 0.12
|
||||
field:fact:supersede я больше не работаю в Тинькофф, теперь я в Яндексе не знаю — не нашла у тебя такой записи. personal 0.13
|
||||
field:fact:supersede где я работаю? не знаю — не нашла у тебя такой записи. personal 0.1
|
||||
field:fact:what-do-you-remember что ты помнишь обо мне? Мне известно о тебе с 2017 года. В настоящее время у меня нет информации об удалении работы в Тинькофф и возобновлений работ по этой компании. 2.98
|
||||
field:note:question-is-not-a-statement я рассказывал тебе про байкал? Давайте проверим. memory 4.78
|
||||
field:note:capture-then-recall запиши: ключи от гаража лежат в верхнем ящике стола "Записала", "проверила" 2.65
|
||||
field:note:capture-then-recall где лежат ключи от гаража? В главе с Долорес он спрашивает: «где лежат ключи от гаража?» Источник утверждает, что они находятся рядом со кухней на первом этаже. search 7.52
|
||||
field:note:recall-miss-is-honest что я говорил тебе про ремонт балкона? Уточни пожалуйста имя человека из которого ты будешь учиться или посмотреть информацию. memory 2.4
|
||||
field:query:own-data-first что у меня сегодня по плану? на 26.08.2026 ничего не запланировано. day-plan 0.01
|
||||
field:query:the-world-summary почему небо голубое? Почему небо голубое? - В статье разберемся, почему днем небо кажется голубым (см., например). Рассказываем про солнечный свет. search 6.29
|
||||
field:query:arithmetic-must-not-escape сколько будет два плюс два? Похоже у вас есть интерес к математике! 2 + 2 = ? Это довольно простая задачка для ребенка с первого класса, а вы можете продолжить? Или вам нужны советы по образованию? search 8.17
|
||||
field:query:follow-up-context какая сейчас погода? для какого города? weather 0.17
|
||||
field:query:follow-up-context Самара Я тебя не разобрала. Повтори, пожалуйста. 0.04
|
||||
field:query:encyclopedia-language расскажи про Байкал Всего лишь одна фразаЕсли у тебя есть желание узнать больше о Байкале или его влиянии как гидроэнергетического комплекс, посетите сайт «Энциклопедии Кирилла и Мефодия» https://www.km.ru/encyclopedia/view.php?ID=602. memory 5.0
|
||||
field:query:feeds-on-request что нового в новостях? вот что нового: mklinux-v7.0-mk2 released; [$] Old-school calendaring at the command line with Remind; Vanilla OS 3 released feeds 1.67
|
||||
field:query:network-scan-as-prose какие устройства сейчас в сети? нашла 3 устройства, из них 2 с вебом, 2 с ssh. список записала. network 27.01
|
||||
field:action:target-must-resolve перезапусти контейнер с базой «с» — не знаю такой цели. назови её как в системе. 0.02
|
||||
field:action:confirm-binds выключи свет в спальне «с» — не знаю такой цели. назови её как в системе. 0.04
|
||||
field:action:report-the-actual-result покажи что запущено в докере «с» — не знаю такой цели. назови её как в системе. 0.06
|
||||
field:action:praxis-lifecycle-gate что требует моего внимания? Нет ничего нового. 3.66
|
||||
field:action:praxis-lifecycle-gate отметь первое как выполненное Такую команду я не знаю. 0.04
|
||||
field:action:capture-a-task добавь в задачи: заказать корм для собаки внесла в список: заказать корм для собаки 0.01
|
||||
field:action:capture-a-task какие у меня задачи? сначала: проверить интейк форму на боксе (просрочено на 17 дней); купить хлеб (давно в списке); купить молоко (давно в списке); полить цветы (давно в списке); починить кран (давно в списке); и ещё 1 задача. 1 задача просрочено; 5 задач лежит дольше 10 дней. tasks 0.01
|
||||
field:proactive:delivery-reaches-telegram напомни мне через две минуты MVNPROBE проверка доставки хорошо, напомню сегодня в 01:13. 0.33
|
||||
|
Can't render this file because it contains an unexpected character in line 16 and column 143.
|
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"note": "field probes: the owner's real week, drawn from the five vertical slices, the three ugly conversations of the 2026-08-13 audit verbatim, the five existing scenarios and what deploy/mavend.json is actually configured for. A failing field probe is an unresolved product question or a missing-criterion finding against docs/spec.md, never a DoD verdict.",
|
||||
"probes": [
|
||||
|
||||
{ "id": "field:reminder:create-and-persist", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне завтра в 9 утра позвонить в клинику"],
|
||||
"readback": {"reminders": ["reminders", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "The reply states the time back, and a reminders row exists for 09:00 tomorrow with that text." },
|
||||
|
||||
{ "id": "field:reminder:clarify-the-missing-slot", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне позвонить маме", "в шесть вечера"],
|
||||
"readback": {"reminders": ["reminders", "10"], "decisions": ["decisions", "2"]},
|
||||
"expect": "Turn 1 asks only for the time, not for all slots. Turn 2 fills it and a row lands at 18:00. The audit saw this pass; it is here to catch a regression." },
|
||||
|
||||
{ "id": "field:reminder:cancel-mid-clarify", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне заехать за кормом", "отмена"],
|
||||
"readback": {"reminders": ["reminders", "10"]},
|
||||
"expect": "Turn 2 drops the parked turn and says so. No reminder row is written for the kibble." },
|
||||
|
||||
{ "id": "field:reminder:correct-the-time", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напомни мне в семь вечера вынести мусор", "нет, в восемь"],
|
||||
"readback": {"reminders": ["reminders", "10"]},
|
||||
"expect": "One reminder at 20:00, not two rows and not one at 19:00. Correction of a just-set reminder is the slice's real shape." },
|
||||
|
||||
{ "id": "field:reminder:recurring-from-speech", "origin": "field", "kind": "live", "slice": "reminder",
|
||||
"utterances": ["напоминай мне каждый день в девять вечера дать собаке таблетку", "какие у меня повторяющиеся напоминания?"],
|
||||
"readback": {"reminders": ["reminders", "20"]},
|
||||
"expect": "The schedule is stated back and a row carries Cron. The spec says storage and delivery are finished and no caller in cmd/mavend passes a cron, so this is expected to fail; the probe records HOW it fails." },
|
||||
|
||||
{ "id": "field:fact:record-and-confirm", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["меня зовут Ками"],
|
||||
"readback": {"facts": ["facts", "20"], "decisions": ["decisions", "1"]},
|
||||
"expect": "The confirmation is feminine, and a facts row actually exists. The audit caught 'я записала информацию о тебе' confirming a write that never happened." },
|
||||
|
||||
{ "id": "field:fact:supersede", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["я работаю в Тинькофф", "я больше не работаю в Тинькофф, теперь я в Яндексе", "где я работаю?"],
|
||||
"readback": {"facts": ["facts", "30"]},
|
||||
"expect": "Turn 3 answers Yandex, not Tinkoff and not both. Both rows are readable and the old value is retired." },
|
||||
|
||||
{ "id": "field:fact:what-do-you-remember", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["что ты помнишь обо мне?"],
|
||||
"readback": {"decisions": ["decisions", "1"], "notes": ["notes", "5"]},
|
||||
"expect": "Routes to query, not remember, and the reply uses informal singular ты. The planning session got formal вас/вы here while the phrasing eval passed. Two of the audit's seven probes misrouted on exactly this shape." },
|
||||
|
||||
{ "id": "field:note:question-is-not-a-statement", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["я рассказывал тебе про байкал?"],
|
||||
"readback": {"notes": ["notes", "5"], "decisions": ["decisions", "1"]},
|
||||
"expect": "No note row is written. The audit stored this question as a statement, and the personal boundary scored it as world." },
|
||||
|
||||
{ "id": "field:note:capture-then-recall", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["запиши: ключи от гаража лежат в верхнем ящике стола", "где лежат ключи от гаража?"],
|
||||
"readback": {"notes": ["notes", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "Turn 2 returns the drawer from the note captured in turn 1, not a world answer and not a recall miss." },
|
||||
|
||||
{ "id": "field:note:recall-miss-is-honest", "origin": "field", "kind": "live", "slice": "fact/note",
|
||||
"utterances": ["что я говорил тебе про ремонт балкона?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "She says she does not remember. A world answer here is the failure: the personal boundary must stop a question about him from reaching outside." },
|
||||
|
||||
{ "id": "field:query:own-data-first", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["что у меня сегодня по плану?"],
|
||||
"readback": {"decisions": ["decisions", "1"], "plan": ["plan"]},
|
||||
"expect": "The real checklist and its open items come back, and the decision trace shows an owner source winning before any world source was asked." },
|
||||
|
||||
{ "id": "field:query:the-world-summary", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["почему небо голубое?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "A Russian summary that does not invent physics. The audit's answer was 'корочковатые цветы отражают длинноволны'. Verbatim from the audit, so the two are comparable." },
|
||||
|
||||
{ "id": "field:query:arithmetic-must-not-escape", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["сколько будет два плюс два?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "The answer is 4. Measured this session: the arithmetic-query stage 0 grammar declined, the turn reached external search, and the reply was 'Во-первых - это два плюса двойки'." },
|
||||
|
||||
{ "id": "field:query:follow-up-context", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["какая сейчас погода?", "Самара"],
|
||||
"readback": {"decisions": ["decisions", "2"]},
|
||||
"expect": "Turn 2 is understood as the city for turn 1. The audit's follow-up died with 'Я тебя не разобрала'. There is no weather block in the config, so a named gap is the honest pass and a guessed forecast is the failure." },
|
||||
|
||||
{ "id": "field:query:encyclopedia-language", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["расскажи про Байкал"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "A Russian question lands on the Russian book. The claiming source in the trace is kiwix, and the article is about the lake." },
|
||||
|
||||
{ "id": "field:query:feeds-on-request", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["что нового в новостях?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "Configured feed items are read back. Two sources are configured; a dead feed must name itself dead and the other still answer." },
|
||||
|
||||
{ "id": "field:query:network-scan-as-prose", "origin": "field", "kind": "live", "slice": "query",
|
||||
"utterances": ["какие устройства сейчас в сети?"],
|
||||
"readback": {"decisions": ["decisions", "1"]},
|
||||
"expect": "Hosts and open ports come back as prose, not as a table dump. netscan.enabled is true with subnets, ports and rate set." },
|
||||
|
||||
{ "id": "field:action:target-must-resolve", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["перезапусти контейнер с базой"],
|
||||
"readback": {"eco-traces": ["eco-traces", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "Free text does not reach a mutating Hexis call. Nexus holds no entities, so the honest outcome is a named gap or a clarify, never a guessed target and never an execution." },
|
||||
|
||||
{ "id": "field:action:confirm-binds", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["выключи свет в спальне"],
|
||||
"readback": {"eco-traces": ["eco-traces", "10"], "decisions": ["decisions", "1"]},
|
||||
"expect": "smarthome.enabled is false, so this is a named gap. The failure is a reply that claims the light was switched." },
|
||||
|
||||
{ "id": "field:action:report-the-actual-result", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["покажи что запущено в докере"],
|
||||
"readback": {"eco-traces": ["eco-traces", "10"], "tools": ["tools"]},
|
||||
"expect": "Either the real container list or a named gap. A plausible invented list is the failure this slice exists to catch." },
|
||||
|
||||
{ "id": "field:action:praxis-lifecycle-gate", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["что требует моего внимания?", "отметь первое как выполненное"],
|
||||
"readback": {"eco-traces": ["eco-traces", "20"], "decisions": ["decisions", "2"]},
|
||||
"expect": "Turn 1 calls Surface, never Acknowledge. Turn 2 is a remote mutation: the architecture pass found praxisItemAction.handle calls straight through with no tier and no confirm turn. The probe records whether it runs on first hearing." },
|
||||
|
||||
{ "id": "field:action:capture-a-task", "origin": "field", "kind": "live", "slice": "action",
|
||||
"utterances": ["добавь в задачи: заказать корм для собаки", "какие у меня задачи?"],
|
||||
"readback": {"tasks": ["tasks"], "decisions": ["decisions", "2"]},
|
||||
"expect": "The task appears in the list, ordered by deadline and urgency, and the capture is visible over IPC." },
|
||||
|
||||
{ "id": "field:proactive:delivery-reaches-telegram", "origin": "field", "kind": "live", "slice": "proactive",
|
||||
"utterances": ["напомни мне через две минуты MVNPROBE проверка доставки"],
|
||||
"readback": {"reminders": ["reminders", "10"], "delivery-attempts": ["delivery-attempts"]},
|
||||
"expect": "Within a compressed live horizon the reminder fires and a delivery-attempts row records the reach that took it. TickInterval defaults to 60s and no override is configured, so two minutes is two ticks. The marker keeps it from reading as a real nudge." },
|
||||
|
||||
{ "id": "field:proactive:failure-retries-into-another-reach", "origin": "field", "kind": "live", "slice": "proactive",
|
||||
"utterances": [],
|
||||
"readback": {"delivery-attempts": ["delivery-attempts"], "nudges": ["nudges", "20"], "events": ["events", "50"]},
|
||||
"expect": "Read the existing attempt rows: a failed delivery retried into another reach rather than looping. The audit watched the ntfy failure run once a minute until 03:05, then Telegram took reminder #83 at 03:06 and nothing retried." }
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Drive probes through the deployed stack and record what came back.
|
||||
|
||||
Runs ON homesrv, where 127.0.0.1:9201 is mavweb and the mavend socket is
|
||||
reachable from inside maven-mavend-1.
|
||||
|
||||
Transport is POST /api/chat, which runs a real turn: router, resident model,
|
||||
query walk, act path. The reply rides back on the 303 Location as
|
||||
?q=..&r=..&s=<claiming query source>&t=<trace id>.
|
||||
|
||||
Readback is e2eprobe over the IPC socket, never the plaintext sqlite copy.
|
||||
|
||||
python3 run_probes.py probes.json > raw.jsonl
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
|
||||
WEB = "http://127.0.0.1:9201"
|
||||
SOCK = "/run/maven/mavend.sock"
|
||||
CONTAINER = "maven-mavend-1"
|
||||
PROBE_BIN = "/tmp/e2eprobe"
|
||||
|
||||
|
||||
def sh(args, timeout=180):
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=timeout)
|
||||
return p.returncode, p.stdout, p.stderr
|
||||
|
||||
|
||||
def chat(text):
|
||||
"""One turn. Returns the parsed redirect, or the failure verbatim."""
|
||||
t0 = time.time()
|
||||
rc, out, err = sh([
|
||||
"curl", "-s", "-o", "/dev/null", "-w", "%{http_code}\t%{redirect_url}",
|
||||
"-X", "POST", "--data-urlencode", f"text={text}", f"{WEB}/api/chat",
|
||||
])
|
||||
dt = round(time.time() - t0, 2)
|
||||
if rc != 0:
|
||||
return {"utterance": text, "error": err.strip() or f"curl rc={rc}", "seconds": dt}
|
||||
code, _, loc = out.partition("\t")
|
||||
r = {"utterance": text, "http": code, "seconds": dt}
|
||||
if code != "303":
|
||||
r["error"] = f"expected 303, got {code}"
|
||||
return r
|
||||
q = urllib.parse.parse_qs(urllib.parse.urlparse(loc).query)
|
||||
r["reply"] = q.get("r", [""])[0]
|
||||
r["source"] = q.get("s", [""])[0]
|
||||
r["trace"] = q.get("t", [""])[0]
|
||||
return r
|
||||
|
||||
|
||||
def probe(cmd):
|
||||
"""One e2eprobe readback. Returns parsed JSON or the error verbatim."""
|
||||
rc, out, err = sh(["docker", "exec", CONTAINER, PROBE_BIN, "-sock", SOCK] + cmd)
|
||||
if rc != 0:
|
||||
return {"error": (err or out).strip()}
|
||||
try:
|
||||
return json.loads(out)
|
||||
except json.JSONDecodeError:
|
||||
return {"raw": out.strip()}
|
||||
|
||||
|
||||
def shell(cmd):
|
||||
"""One read-only command on homesrv, for a configuration or deployment fact.
|
||||
|
||||
A criterion about whether a config block exists is not observable through a
|
||||
turn, and chasing it through one measures the router instead.
|
||||
"""
|
||||
t0 = time.time()
|
||||
p = subprocess.run(["sh", "-c", cmd], capture_output=True, text=True, timeout=300)
|
||||
return {
|
||||
"cmd": cmd,
|
||||
"rc": p.returncode,
|
||||
"stdout": p.stdout[-8000:],
|
||||
"stderr": p.stderr[-2000:],
|
||||
"seconds": round(time.time() - t0, 2),
|
||||
}
|
||||
|
||||
|
||||
def normalise_readback(rb):
|
||||
"""Accept both shapes: {name: argv} and [{name, argv}]."""
|
||||
if isinstance(rb, dict):
|
||||
return list(rb.items())
|
||||
return [(d["name"], d["argv"]) for d in (rb or [])]
|
||||
|
||||
|
||||
def reset():
|
||||
"""Clear any parked clarify before the next probe.
|
||||
|
||||
mavweb hardcodes one conversation id for the whole web reach, so a clarify
|
||||
parked by one probe is still parked for the next one. Measured: an
|
||||
unanswered park appended "Сейчас 01:07. В какой день?" to eleven unrelated
|
||||
turns in a row, including plain statements the router should have taken as
|
||||
facts. Without this the run measures the previous probe, not this one.
|
||||
|
||||
The leak itself is a finding, recorded separately from a run that isolates.
|
||||
"""
|
||||
r = chat("отмена")
|
||||
return {"reply": r.get("reply", ""), "http": r.get("http", ""), "error": r.get("error", "")}
|
||||
|
||||
|
||||
def main():
|
||||
spec = json.load(open(sys.argv[1]))
|
||||
for p in spec["probes"]:
|
||||
rec = {
|
||||
"id": p["id"],
|
||||
"origin": p.get("origin", "dod" if p["id"].startswith("dod:") else "field"),
|
||||
"criteria": p.get("criteria", []),
|
||||
"slice": p.get("slice", ""),
|
||||
"expect": p.get("expect", ""),
|
||||
"kind": p.get("kind", "live"),
|
||||
"method": p.get("method", "chat"),
|
||||
"turns": [],
|
||||
"readback": {},
|
||||
}
|
||||
if rec["kind"] == "blocked" or rec["method"] == "none":
|
||||
rec["blocked_reason"] = p.get("blocked_reason", "")
|
||||
print(json.dumps(rec, ensure_ascii=False), flush=True)
|
||||
continue
|
||||
if p.get("isolate", True) and rec["method"] == "chat":
|
||||
rec["reset_before"] = reset()
|
||||
time.sleep(1)
|
||||
if rec["method"] == "shell":
|
||||
rec["shell"] = shell(p["shell"])
|
||||
for text in p.get("utterances", []):
|
||||
rec["turns"].append(chat(text))
|
||||
time.sleep(p.get("gap", 1))
|
||||
for name, cmd in normalise_readback(p.get("readback")):
|
||||
rec["readback"][name] = probe(cmd)
|
||||
print(json.dumps(rec, ensure_ascii=False), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Row counts per store, read over IPC. Runs on homesrv.
|
||||
|
||||
The eval's first artifact. Read before any probe writes, and again after, so
|
||||
what the probes added is attributable. Not the sqlite file: the point is to
|
||||
measure the contract, and the plaintext copy in tmpfs bypasses it.
|
||||
"""
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
C = "maven-mavend-1"
|
||||
S = "/run/maven/mavend.sock"
|
||||
BIN = "/tmp/e2eprobe"
|
||||
|
||||
# command, argv, and whether the reply is capped by a limit e2eprobe hardcodes.
|
||||
READS = [
|
||||
("facts", ["facts", "100000"], False),
|
||||
("notes", ["notes", "100000"], False),
|
||||
("reminders", ["reminders", "100000"], False),
|
||||
("pending_reminders", ["pending-reminders", "100000"], False),
|
||||
("tasks_live", ["tasks"], False),
|
||||
("nudges", ["nudges", "100000"], False),
|
||||
("tools", ["tools"], False),
|
||||
("decisions", ["decisions", "100000"], False),
|
||||
("events", ["events", "100000"], False),
|
||||
("eco_traces", ["eco-traces", "100000"], False),
|
||||
("delivery_attempts", ["delivery-attempts"], True), # capped at 200 in e2eprobe
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
out = {}
|
||||
for name, argv, capped in READS:
|
||||
p = subprocess.run(["docker", "exec", C, BIN, "-sock", S] + argv,
|
||||
capture_output=True, text=True, timeout=180)
|
||||
if p.returncode != 0:
|
||||
out[name] = {"error": (p.stderr or p.stdout).strip()[:300]}
|
||||
continue
|
||||
try:
|
||||
d = json.loads(p.stdout)
|
||||
except json.JSONDecodeError:
|
||||
out[name] = {"error": "not json", "raw": p.stdout.strip()[:300]}
|
||||
continue
|
||||
if d is None:
|
||||
out[name] = {"count": 0}
|
||||
elif isinstance(d, list):
|
||||
r = {"count": len(d)}
|
||||
if capped and len(d) >= 200:
|
||||
r["note"] = "at e2eprobe's hardcoded 200 cap, true count is >= 200"
|
||||
out[name] = r
|
||||
else:
|
||||
out[name] = {"value": d}
|
||||
json.dump(out, sys.stdout, ensure_ascii=False, indent=2)
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user