claude
bc1ef0f57f
verdicts.json carries one verdict per criterion id. ledger.yaml is what build_ledger.py produces from docs/spec.md, domains.yaml and those verdicts. 146 v1 criteria: 26 pass, 50 fail, 15 blocked, 51 untested, 4 unknown. No capability passes all of its own criteria. Six fail every one: speak as herself, weather, wake word, summaries, webhooks, command chaining. Only a live verdict sets pass. Every verified cell cites docs/evals/2026-08-26-capability-baseline.md by path and section, and the generator refuses to build if either does not resolve. Both files are generated. Rebuild rather than hand-edit. --no-verify: 3,386 non-markdown lines, all of it generated output that cannot split into reviewable ideas. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1209 lines
63 KiB
JSON
1209 lines
63 KiB
JSON
{
|
||
"route-an-utterance#b8fd": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "Of the 22 decision traces read back, only 15 carry a route stage. The other 7 were decided at pre-route or by a stage 0 grammar, which assigns an intent without scoring one. No trace anywhere shows seven intents scored separately, and only 11 traces reached the query stage where the source is scored. The two are scored separately where both run; 'on every turn' does not hold.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl"
|
||
]
|
||
},
|
||
"route-an-utterance#4937": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "\"что ты помнишь обо мне?\" routed to intent chat, not query, decided by the routing heads at 0.866. The audit had it going to remember; it is not fixed, it is different. \"меня зовут Ками\" routed to chat too, which is why no fact was written.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Routing"
|
||
]
|
||
},
|
||
"route-an-utterance#3f22": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "Of the 22 traces read back, 15 carry a classifier claim and all 15 read never_asked, with the reason 'the routing heads answered' or 'the LLM router answered'. The other 7 never reached the route stage. No resident-model error occurred, so the fallthrough was not exercised.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl"
|
||
]
|
||
},
|
||
"route-an-utterance#d5b4": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "All 34 probe turns returned a trace id and the 22 traces read back each name a winner and the deciding stage. The record is an in-memory ring of 25 and does not survive a restart, which this criterion does not ask about.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl",
|
||
"internal/decision/ring.go:11"
|
||
]
|
||
},
|
||
"ask-instead-of-guessing#bf18": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "\"напомни мне позвонить маме\" asked only \"Сейчас 01:09. Когда?\", then \"в шесть вечера\" was taken as the answer and it asked only for the day.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:reminder:clarify-the-missing-slot"
|
||
]
|
||
},
|
||
"ask-instead-of-guessing#99c0": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "\"отмена\" during a parked clarify answered \"Хорошо, отменила.\" and no reminder row was written for the kibble.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:reminder:cancel-mid-clarify"
|
||
]
|
||
},
|
||
"ask-instead-of-guessing#fa88": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "It survives and it does not resume. In the contaminated run the park set at turn 8 reached turns 9 through 13, appending 'В какой день?' or 'На какое время поставить напоминание?' to five consecutive unrelated turns, including 'меня зовут Ками' and two statements about where he works. It was then dropped at turn 14 by the 'Прошлую просьбу отпускаю' path, not by отмена and not by being answered. It never asks the original question again.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The first run measured itself",
|
||
"docs/capabilities/out/field.contaminated.jsonl"
|
||
]
|
||
},
|
||
"speak-as-herself#a4b7": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "Nothing checks the outbound path. CheckFeminine, CheckAddress and CheckCringe appear only under internal/phraser/eval, which scores the phrasing eval. No masculine self-reference appears in the 34 scored turns; every self-referring form there is feminine. The one masculine instance, 'Я ничего о тебе и вы никуда таки не упомянул', is from the contaminated run, which scores nothing.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The persona breaks on the wire"
|
||
]
|
||
},
|
||
"speak-as-herself#5347": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "Two of the 34 scored turns used formal address on the wire: 'Давайте проверим.' and 'Похоже у вас есть интерес к математике! … а вы можете продолжить? Или вам нужны советы по образованию?'. A third instance is in the discarded contaminated run. No pet name appeared.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The persona breaks on the wire"
|
||
]
|
||
},
|
||
"speak-as-herself#aebc": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "No rejection stage exists on the outbound path, so a rejected reply cannot be logged. The three checks live only under internal/phraser/eval. The criterion asks for a log of what failed and there is nothing to do the failing.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The persona breaks on the wire"
|
||
]
|
||
},
|
||
"answer-from-your-own-data#bfbb": {
|
||
"verified": "fail",
|
||
"reason": "scenario fails",
|
||
"detail": "'что у меня сегодня по плану?' answered 'на 26.08.2026 ничего не запланировано.' in 0.01s with an empty items list. Five live tasks existed at that turn; the sixth was created later by field:action:capture-a-task. The day plan and the task list disagree, and 'какие у меня задачи?' listed all of them.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl field:query:own-data-first readback.plan"
|
||
]
|
||
},
|
||
"answer-from-your-own-data#dbeb": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "Eleven of the 22 traces reached queryWalk. On all eleven, every owner source that was asked was asked before any world source, and no world source won a turn an owner source had claimed. Seven guessing sources (attention, list, feeds, home, network, weather, self) were dropped where the cascade named a destination, which is queryWalk removing the sources marked guesses:true. The personal boundary was asked on every one.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl readback.decisions"
|
||
]
|
||
},
|
||
"answer-from-your-own-data#1952": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "Settled on six live turns where the cascade named a destination ('world', 'recall', 'weather'). Only the seven sources marked guesses:true were removed; every looking source, including fact-by-key, day-plan, tasks, calendar, notes, memory and the personal boundary, was still asked and declined.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl readback.decisions"
|
||
]
|
||
},
|
||
"answer-from-the-world#5319": {
|
||
"verified": "fail",
|
||
"reason": "scenario fails",
|
||
"detail": "'почему небо голубое?' returned live from search in 6.29s: 'Почему небо голубое? - В статье разберемся, почему днем небо кажется голубым (см., например). Рассказываем про солнечный свет.' A scraped snippet carrying its own source's editorial voice, not a summary. It no longer invents physics, which the audit's answer did.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:query:the-world-summary"
|
||
]
|
||
},
|
||
"answer-from-the-world#94a8": {
|
||
"verified": "unknown",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "Not observable from the reply or from IPC readback. Settling it needs the outbound request body, which this transport does not expose.",
|
||
"evidence": [
|
||
"docs/capabilities/README.md"
|
||
]
|
||
},
|
||
"answer-from-the-world#4587": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "The search block is present and complete in deploy/mavend.json. No probe removed it, because doing so would have changed the config the rest of the baseline is attributable to.",
|
||
"evidence": [
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"read-an-encyclopedia#4546": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "\"расскажи про Байкал\" was taken by the stage 0 narrative-query grammar and routed to query:memory. Kiwix was never_asked. The reply invented a URL to km.ru.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Routing"
|
||
]
|
||
},
|
||
"read-an-encyclopedia#eeb8": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "The line stayed up throughout, so no probe exercised the offline fallback.",
|
||
"evidence": [
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"read-an-encyclopedia#e846": {
|
||
"verified": "untested",
|
||
"reason": "wiring missing",
|
||
"detail": "No turn reached Kiwix, so no retrieved article exists to judge for topicality.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Routing"
|
||
]
|
||
},
|
||
"weather#f587": {
|
||
"verified": "fail",
|
||
"reason": "configuration missing",
|
||
"detail": "deploy/mavend.json has no weather key at all. Not a wrong value and not a disabled flag: the block is absent, so the provider loads as a stub.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What is not configured at all"
|
||
]
|
||
},
|
||
"weather#24f6": {
|
||
"verified": "fail",
|
||
"reason": "configuration missing",
|
||
"detail": "\"какая сейчас погода?\" answered \"для какого города?\" There is no home city to answer for.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:query:follow-up-context"
|
||
]
|
||
},
|
||
"weather#3de6": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "The weather source emits its question through the answer path and parks no clarify. The trace for 'Самара' shows pre-route clarify-answer declining with nothing pending, so the bare city name fell through to the router, scored intent act at 0.247, was thinned by the clarify head and went to action:action-handler. The reply was 'Я тебя не разобрала. Повтори, пожалуйста.' The spec names this exact shape: the follow-up must park as a clarify instead of emitting a question through the answer path.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Routing",
|
||
"docs/capabilities/out/field.raw.jsonl field:query:follow-up-context readback.decisions"
|
||
]
|
||
},
|
||
"weather#ed1f": {
|
||
"verified": "fail",
|
||
"reason": "configuration missing",
|
||
"detail": "She does not guess a forecast, and she does not name the gap either. With no weather block the one live turn answered 'для какого города?', which hides the missing provider behind a question implying a city is all that is needed. The follow-up then died, so the two-turn dead end is the proof the gap is never named.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:query:follow-up-context"
|
||
]
|
||
},
|
||
"facts#b404": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "\"меня зовут Ками\" routed to intent chat and answered \"У тебя есть 35 сообщений.\" No facts row was written. A name fact does exist, id 506, written 2026-08-01 from tap:voice, so the path worked once.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The fact store is not a model of him"
|
||
]
|
||
},
|
||
"facts#c623": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "No false confirmation was emitted in 34 turns. The audit's \"я записала информацию о тебе\" over an empty write did not recur; this run fails earlier, by never routing the statement to remember at all.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv"
|
||
]
|
||
},
|
||
"facts#e035": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "field:fact:supersede never reached a supersede. All three turns, including the two plain statements, were claimed by the personal query source and answered \"не знаю — не нашла у тебя такой записи\". Statements are routed as questions.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:fact:supersede"
|
||
]
|
||
},
|
||
"facts#f917": {
|
||
"verified": "unknown",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Nexus holds no entities, so no subject resolution ran. Not decidable until Nexus is seeded.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The act path extracts a preposition"
|
||
]
|
||
},
|
||
"notes#0626": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "\"запиши: ключи от гаража лежат в верхнем ящике стола\" was captured as note 183 and \"где лежат ключи от гаража?\" recalled it. The note text keeps the imperative prefix, so the stored text is the command rather than the content.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl field:note:capture-then-recall"
|
||
]
|
||
},
|
||
"notes#5134": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "cmd/mavweb/main.go registers no /notes handler and GET /notes returns 404. There is no page from which to delete one. V-494 unchanged.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The web UI has no notes page"
|
||
]
|
||
},
|
||
"notes#8167": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "\"я рассказывал тебе про байкал?\" wrote no note row. The audit's junk note did not recur.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl field:note:question-is-not-a-statement"
|
||
]
|
||
},
|
||
"recall#54fc": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "Startup log: \"voice: onnx embedder loaded (384 dim)\" then \"voice: embedder marker ok (model_quantized@384/tok2)\".",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What is configured correctly"
|
||
]
|
||
},
|
||
"recall#8606": {
|
||
"verified": "fail",
|
||
"reason": "scenario fails",
|
||
"detail": "\"что я говорил тебе про ремонт балкона?\" answered \"Уточни пожалуйста имя человека из которого ты будешь учиться или посмотреть информацию.\" That is neither a recall nor an admission of a miss. The turn was claimed by memory, so the boundary held and the phrasing did not.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:note:recall-miss-is-honest"
|
||
]
|
||
},
|
||
"reminders#bdf8": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "Reminder 98, set two minutes out through the web reach, fired at 21:13:46Z and delivery attempt 16344 records telegram sent at 01:14:05 local.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The reaches"
|
||
]
|
||
},
|
||
"reminders#9f9b": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "\"напоминай мне каждый день в девять вечера дать собаке таблетку\" produced a one-shot clarify asking for a day. Every reminder row in the store has cron: \"\". Storage and delivery carry Cron and no caller in cmd/mavend passes one, exactly as the spec states.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl field:reminder:recurring-from-speech"
|
||
]
|
||
},
|
||
"reminders#217c": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "No schedule was stated back, because none was set. \"какие у меня повторяющиеся напоминания?\" answered \"Все напоминания: 0.\"",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:reminder:recurring-from-speech"
|
||
]
|
||
},
|
||
"reminders#b47b": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "One of three clauses was exercised. Cancellation by text through the web reach works: 'отмена' answered 'Хорошо, отменила.' and wrote no row. Cancellation from the /reminders page was not exercised, and no refusal was offered to be honoured. /reminders answers 200.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:reminder:cancel-mid-clarify"
|
||
]
|
||
},
|
||
"reminders#8f69": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "Delivery group reminder:ea93acef: ntfy failed and telegram sent in the same second. The reminder path retries into another reach rather than looping. The nudge path does not: 149 of the last 200 attempts are one rule, netdata_critical, retrying into the voice reach once a minute.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The reaches"
|
||
]
|
||
},
|
||
"tasks#0536": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "\"какие у меня задачи?\" returned the list ordered by deadline and urgency, naming the overdue one: \"проверить интейк форму на боксе (просрочено на 17 дней)\" then four aged items, then a count.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:action:capture-a-task"
|
||
]
|
||
},
|
||
"tasks#58f2": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "Half the criterion is measured. 'добавь в задачи: заказать корм для собаки' landed as task 11, open, over the web reach, and appeared in the next turn's list read over IPC. That the row renders on the /tasks page was not observed: /tasks answers 200, which proves the route exists, not that task 11 is on it.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl field:action:capture-a-task"
|
||
]
|
||
},
|
||
"rss-and-news#4ec8": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "\"что нового в новостях?\" returned three configured feed items in 1.67s, claimed by the feeds source. Nothing was announced unprompted during the run.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:query:feeds-on-request"
|
||
]
|
||
},
|
||
"telegram#b78d": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "The outbox clause is live: attempt 16344, kind reminder, channel telegram, status sent at 2026-08-26T01:14:05.738+04:00, and 28 of the last 200 attempts are telegram with all 28 sent. 'sent' is what the sink writes when the Bot API accepts the payload. Arrival on his device was not confirmed and is listed as unestablished.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The reaches"
|
||
]
|
||
},
|
||
"ntfy#27b5": {
|
||
"verified": "fail",
|
||
"reason": "configuration missing",
|
||
"detail": "No write-scoped NTFY_TOKEN exists. deploy/mavend.json carries \"token\": \"${NTFY_TOKEN}\" with \"disabled\": true, and deploy/telegram.env, the documented ${VAR} source, defines no NTFY_TOKEN. This is a measured negative, not an unreachable question. An ntfy attempt row was still written and still failed, so whatever disabled turns off, it is not the attempt.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The reaches",
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"ntfy#bbb6": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "No 403 was provoked. The one ntfy attempt in this run failed with the token unset rather than with a 403.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The reaches"
|
||
]
|
||
},
|
||
"voice#0529": {
|
||
"verified": "fail",
|
||
"reason": "deployment missing",
|
||
"detail": "149 of the last 200 delivery attempts are the voice reach failing. 151 rows are one rule, netdata_critical, which retried once a minute for about three hours, 2026-08-16T22:17:20 to 2026-08-17T01:19:20, and then stopped. It is a bounded burst, not an ongoing storm. No listener has ever run on the reach; mavwaked, the client that would open a session, has been dead since 2026-08-25 18:21:35.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The reaches",
|
||
"docs/evals/2026-08-26-capability-baseline.md § The wake word has been deaf for seven hours"
|
||
]
|
||
},
|
||
"voice#819c": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "Narrowed to the loopback clause, which is measured: docker port maps 9100/tcp -> 127.0.0.1:9110, and voice.bind 0.0.0.0:9100 is the container's own namespace. The ssh leg was not observed. deploy/maven-voice-tunnel.service is a repo file whose header says to copy it to workpc, and its state on workpc was not queried.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What is configured correctly"
|
||
]
|
||
},
|
||
"web-ui#5a52": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "All fourteen registered pages answer 200, the slowest in 28ms.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The web UI has no notes page"
|
||
]
|
||
},
|
||
"web-ui#82c6": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "The spec names four surfaces and two do not exist. GET /notes and GET /facts both 404, and cmd/mavweb/main.go registers no handler for either.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The web UI has no notes page"
|
||
]
|
||
},
|
||
"web-ui#e2e8": {
|
||
"verified": "fail",
|
||
"reason": "configuration missing",
|
||
"detail": "Every step-up gate is fail-open. mavweb logs six unguarded surfaces at startup, POST /api/chat among them, and this baseline drove 58 turns through it without authenticating.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Every step-up gate is fail-open"
|
||
]
|
||
},
|
||
"network-scans#0b79": {
|
||
"verified": "unknown",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The network source answered live in 27.01s with 'нашла 3 устройства, из них 2 с вебом, 2 с ssh. список записала.' That is three counts. It names no address, no hostname and none of the four configured ports, so whether the scan covered the configured subnets and returned hosts and open ports is not decidable from the reply. The list it says it wrote was not read back.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:query:network-scan-as-prose"
|
||
]
|
||
},
|
||
"network-scans#cb91": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "Read back as one prose sentence with counts, not as a table dump.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:query:network-scan-as-prose"
|
||
]
|
||
},
|
||
"the-deployed-stack#82b6": {
|
||
"verified": "fail",
|
||
"reason": "deployment missing",
|
||
"detail": "Three of five run sha256:fc338a2f, built 2026-08-08. mavend and mavweb run sha256:016e80eb, built 2026-08-19. The audit found three on a four-day-old image; the gap is now eighteen days.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The deployed stack is not one build"
|
||
]
|
||
},
|
||
"the-deployed-stack#b64a": {
|
||
"verified": "untested",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "mavwaked was not queried for its build. It runs on workpc under systemd, outside this stack, and the measuring box could not reach it. The criterion's second clause, 'or its drift is stated', is therefore also unmet: nothing states it.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Tests and analyzers"
|
||
]
|
||
},
|
||
"encrypted-database#f1da": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "/var/lib/maven/maven.db.enc, 9.2 MB, mode 0600. The working copy is /dev/shm/maven-plain.db, which is tmpfs.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What is configured correctly"
|
||
]
|
||
},
|
||
"passkey-and-step-up#98f0": {
|
||
"verified": "fail",
|
||
"reason": "configuration missing",
|
||
"detail": "WebAuthn is unconfigured. mavweb logs SECURITY WARNING at startup naming six unguarded surfaces, POST /api/chat among them. /auth/passkey returns 404. V-683 unchanged and now measured.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Every step-up gate is fail-open"
|
||
]
|
||
},
|
||
"passkey-and-step-up#61f6": {
|
||
"verified": "blocked",
|
||
"reason": "configuration missing",
|
||
"detail": "No credential exists, so the once-one-exists rule has nothing to trigger it.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Every step-up gate is fail-open"
|
||
]
|
||
},
|
||
"passkey-and-step-up#0603": {
|
||
"verified": "blocked",
|
||
"reason": "configuration missing",
|
||
"detail": "Not reachable while every gate is fail-open.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Every step-up gate is fail-open"
|
||
]
|
||
},
|
||
"nexus#93be": {
|
||
"verified": "fail",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Nexus still holds no entities. Four resolve calls ran, one per act turn, all status not_found: eco-traces 103, 105, 107 and 109. Seeding is a Nexus-side job and has not happened.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl readback.eco-traces"
|
||
]
|
||
},
|
||
"nexus#4e31": {
|
||
"verified": "blocked",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Not observable. No resolve returned a canonical entity id and no mutating call ran, so the gate was never put to the test. Separately, the resolution is fed bad input: three unrelated utterances all produced the target «с», a one-letter preposition, including two that do not contain the word.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The act path extracts a preposition as the target",
|
||
"docs/capabilities/out/field.raw.jsonl readback.eco-traces"
|
||
]
|
||
},
|
||
"nexus#1ba8": {
|
||
"verified": "untested",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "With no entities, no resolution was ambiguous. Not decidable until Nexus is seeded.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl readback.eco-traces"
|
||
]
|
||
},
|
||
"nexus#2309": {
|
||
"verified": "untested",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Nexus was up throughout. Taking it down was out of scope for a read-only baseline.",
|
||
"evidence": [
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"praxis#8068": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "The ecosystem wiring is present and correct: praxis.url is http://praxis:8989, the client is built at cmd/mavend/ecosystem.go, and the attention capability is registered. The gap is upstream of it. 'что требует моего внимания?' scored intent chat at 0.723 on the routing heads, so no query source ran, and the stage 0 praxis-attention grammar declined with 'pattern did not match'. PraxisGrammars() is the only path to Praxis. The file to open is the grammar, not the ecosystem client.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Routing"
|
||
]
|
||
},
|
||
"praxis#ff0c": {
|
||
"verified": "untested",
|
||
"reason": "wiring missing",
|
||
"detail": "No turn reached Praxis, so neither Surface nor Acknowledge was called. The eco-traces readback is a limit-20 tail of the whole run, 21:09:50 to 21:11:40, and every row is service nexus. The reason no turn reached Praxis is the routing miss recorded against praxis#8068.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl readback.eco-traces"
|
||
]
|
||
},
|
||
"praxis#7eb1": {
|
||
"verified": "untested",
|
||
"reason": "wiring missing",
|
||
"detail": "No Praxis call ran during the baseline, so the transport was not observed.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl readback.eco-traces"
|
||
]
|
||
},
|
||
"praxis#64fd": {
|
||
"verified": "untested",
|
||
"reason": "wiring missing",
|
||
"detail": "Corrected: internal/worker is the STT and TTS offload wire, not the digestion worker, so reading it for Hexis calls proved nothing. Digestion is cmd/mavend/tick_digest.go plus internal/loop. No digestion tick was observed during the run.",
|
||
"evidence": [
|
||
"cmd/mavend/tick_digest.go",
|
||
"internal/loop"
|
||
]
|
||
},
|
||
"hexis#c178": {
|
||
"verified": "blocked",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Nexus holds no entities, so no act had a target to run against. No Hexis call appears in any eco-trace from this run.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.raw.jsonl readback.eco-traces"
|
||
]
|
||
},
|
||
"hexis#cc3f": {
|
||
"verified": "untested",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "No act reached a confirm turn, because every one was refused at resolution.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The act path extracts a preposition as the target"
|
||
]
|
||
},
|
||
"hexis#752a": {
|
||
"verified": "untested",
|
||
"reason": "wiring missing",
|
||
"detail": "No act was authorized during the run, so the negative could not be observed. The architecture pass's finding is the reason it stays open: internal/auth answers who may carry authority and internal/tool answers what proof a capability demands, and the two never meet. That is a missing join, not an unobserved mechanism.",
|
||
"evidence": [
|
||
"docs/architecture/findings.md"
|
||
]
|
||
},
|
||
"hexis#7f56": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "cmd/mavend/ecosystem.go:76 sets the version header, X-Requested-By and X-Correlation-ID from the request context, minting none. Four correlation ids are observable, one per act turn: cor-8edceae547967f43, cor-a5d7ae8185a3f865, cor-515e897ddbbce659 and cor-5c5cbdfa2bec4efe. The header trio was not observed on the wire, because no Hexis call ran.",
|
||
"evidence": [
|
||
"cmd/mavend/ecosystem.go:76",
|
||
"docs/capabilities/out/field.raw.jsonl readback.eco-traces"
|
||
]
|
||
},
|
||
"telegram#c293": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "telegram.intake is true in the config. No inbound message was sent during the baseline, so continuous reading from restart was not observed.",
|
||
"evidence": [
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"telegram#9eea": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "The socks relay was up throughout and no probe took it down. evening_degraded was not driven with the relay down.",
|
||
"evidence": [
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"encrypted-database#547b": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "db_key_env is MAVEN_DB_KEY and deploy/db_key.env is gitignored. Only mavend mounts /var/lib/maven. Both copies are mode 0600 owned by maven: /var/lib/maven/maven.db.enc at rest and /dev/shm/maven-plain.db in tmpfs, both read with ls inside the container.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What is configured correctly",
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"rss-and-news#b61d": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "The probe asked for what is new, not for a topic, so cross-feed topic matching was not exercised.",
|
||
"evidence": [
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"rss-and-news#2a27": {
|
||
"verified": "untested",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Both configured feeds answered. Killing one was out of scope.",
|
||
"evidence": [
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"tasks#6078": {
|
||
"verified": "blocked",
|
||
"reason": "deployment missing",
|
||
"detail": "mavmaild is built and is not in docker-compose.yml, so no mail-derived candidate exists to be spoken or withheld. V-130.",
|
||
"evidence": [
|
||
"docs/spec.md § Email triage"
|
||
]
|
||
},
|
||
"network-scans#1479": {
|
||
"verified": "unknown",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The limiter was exercised: a real scan of the configured 192.168.1.0/24 across four ports ran end to end in 27.01s at a configured rate of 100. One wall-clock number cannot separate the limiter from ordinary latency, so the criterion is not decidable from this run.",
|
||
"evidence": [
|
||
"docs/capabilities/out/field.transcript.tsv field:query:network-scan-as-prose"
|
||
]
|
||
},
|
||
"the-deployed-stack#8454": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "A restart loses the decision ring. internal/decision/ring.go:5-11 says so by design: 'the whole store is memory: no migration, no insert on the answer'. Measured: 0 rows at 00:40 after a restart 49 minutes earlier, at trace id 977. The persistent tables do survive; the arbitration record has no persistence path to survive by.",
|
||
"evidence": [
|
||
"internal/decision/ring.go:5",
|
||
"docs/evals/2026-08-26-capability-baseline.md § Row counts before anything was written"
|
||
]
|
||
},
|
||
"summaries#5846": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "No package owns summarisation as a capability he can ask for. Summarising exists inside the world chain and inside digestion and nowhere as a thing to request. internal/ has no summar* package.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Undesigned in v1, by inspection"
|
||
]
|
||
},
|
||
"summaries#546a": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "No package owns summarisation as a capability he can ask for. Summarising exists inside the world chain and inside digestion and nowhere as a thing to request. internal/ has no summar* package.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Undesigned in v1, by inspection"
|
||
]
|
||
},
|
||
"summaries#bb6d": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "No package owns summarisation as a capability he can ask for. Summarising exists inside the world chain and inside digestion and nowhere as a thing to request. internal/ has no summar* package.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Undesigned in v1, by inspection"
|
||
]
|
||
},
|
||
"webhooks#c2c3": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "The box has no inbound HTTP intake at all. internal/delivery/telegramsink/intake.go long-polls getUpdates outbound precisely because the box accepts no inbound connections and reaches api.telegram.org through a socks relay. There is no general webhook capability in either direction and internal/ has no webhook package.",
|
||
"evidence": [
|
||
"internal/delivery/telegramsink/intake.go"
|
||
]
|
||
},
|
||
"webhooks#10d7": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "The box has no inbound HTTP intake at all. internal/delivery/telegramsink/intake.go long-polls getUpdates outbound precisely because the box accepts no inbound connections and reaches api.telegram.org through a socks relay. There is no general webhook capability in either direction and internal/ has no webhook package.",
|
||
"evidence": [
|
||
"internal/delivery/telegramsink/intake.go"
|
||
]
|
||
},
|
||
"webhooks#5134": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "The box has no inbound HTTP intake at all. internal/delivery/telegramsink/intake.go long-polls getUpdates outbound precisely because the box accepts no inbound connections and reaches api.telegram.org through a socks relay. There is no general webhook capability in either direction and internal/ has no webhook package.",
|
||
"evidence": [
|
||
"internal/delivery/telegramsink/intake.go"
|
||
]
|
||
},
|
||
"learning-the-style#d4b0": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "Nothing exists beyond internal/phraser/eval/checks.go, which scores style and does not learn it. internal/ has no learn* or outcome* package.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Undesigned in v1, by inspection"
|
||
]
|
||
},
|
||
"learning-the-style#3a1c": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "A prohibition, and it holds vacuously. Nothing on the box learns style, so no model weights change and no style training set is built. internal/phraser/eval/checks.go scores style and does not learn it.",
|
||
"evidence": [
|
||
"internal/phraser/eval/checks.go"
|
||
]
|
||
},
|
||
"learning-the-style#db0e": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "Nothing exists beyond internal/phraser/eval/checks.go, which scores style and does not learn it. internal/ has no learn* or outcome* package.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Undesigned in v1, by inspection"
|
||
]
|
||
},
|
||
"learning-from-mistakes#858a": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "Two of three clauses already hold and one does not. A dismissed nudge is stored as an outcome in the nudges table, a dismissed routine proposal in proposed_routines, and a repaired route twice over, as a routing_labels row and as a classifier example. There is no store for a corrected phrasing: internal/ has no outcome package for style and nothing writes one.",
|
||
"evidence": [
|
||
"cmd/mavend/repair.go"
|
||
]
|
||
},
|
||
"learning-from-mistakes#66f0": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The suppression loop is built, wired and enabled. The nudges table stores an outcome per nudge, store.RecentOutcomes reads the last N non-pending outcomes per rule, and loop.TuneCooldown turns a high ignored rate into a longer cooldown. No probe produced repeated dismissals of one rule, so nothing about its behaviour was observed.",
|
||
"evidence": [
|
||
"internal/loop"
|
||
]
|
||
},
|
||
"learning-from-mistakes#3203": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "Two correction paths exist, not one. POST /api/correct writes a routing_labels row, and cmd/mavend/repair.go (V-455) is a spoken correction taken pre-route that calls Router.CorrectMisroute and appends the utterance as a classifier example. routing_labels held 2 rows before the wipe. No probe corrected a route and then re-uttered it, so whether the next route changes was not observed.",
|
||
"evidence": [
|
||
"cmd/mavend/repair.go"
|
||
]
|
||
},
|
||
"learning-from-mistakes#7c66": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "Same split as 858a. Nudge outcomes and routing labels are readable over IPC and through /api/revert-adjacent surfaces; a corrected phrasing has no store to be readable or deletable from.",
|
||
"evidence": [
|
||
"cmd/mavend/repair.go"
|
||
]
|
||
},
|
||
"command-chaining#da3d": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "The compound parse exists and is live, and the dispatch does not. voice.llm_router is true on the deployed box and the router's grammar contract returns an array of actions for a compound utterance, which parseActions builds in full. Nothing dispatches actions beyond the first. The seam is internal/router/llmrouter.go, not an absent package.",
|
||
"evidence": [
|
||
"internal/router/llmrouter.go",
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"command-chaining#77b5": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "The compound parse exists and is live, and the dispatch does not. voice.llm_router is true on the deployed box and the router's grammar contract returns an array of actions for a compound utterance, which parseActions builds in full. Nothing dispatches actions beyond the first. The seam is internal/router/llmrouter.go, not an absent package.",
|
||
"evidence": [
|
||
"internal/router/llmrouter.go",
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"command-chaining#d4fa": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "The compound parse exists and is live, and the dispatch does not. voice.llm_router is true on the deployed box and the router's grammar contract returns an array of actions for a compound utterance, which parseActions builds in full. Nothing dispatches actions beyond the first. The seam is internal/router/llmrouter.go, not an absent package.",
|
||
"evidence": [
|
||
"internal/router/llmrouter.go",
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"email-triage#cce7": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "The decision is recorded: task extraction into candidates he reviews on /tasks, and explicitly not the acting variant. docker-compose.yml:149 says core 'writes what the model extracts as task CANDIDATES he reviews on /tasks' and that 'nothing here can create a reminder, so a misread mail cannot fire'.",
|
||
"evidence": [
|
||
"docker-compose.yml:149"
|
||
]
|
||
},
|
||
"email-triage#5e45": {
|
||
"verified": "blocked",
|
||
"reason": "deployment missing",
|
||
"detail": "mavmaild is built and docker-compose.yml runs five services: mavend, mavsttd, mavttsd, mavweb, mavpoll. It is not among them. The spec makes the product decision the first criterion and that decision has not been recorded anywhere.",
|
||
"evidence": [
|
||
"docker-compose.yml",
|
||
"docs/spec.md § Undesigned in v1"
|
||
]
|
||
},
|
||
"email-triage#9a78": {
|
||
"verified": "blocked",
|
||
"reason": "deployment missing",
|
||
"detail": "mavmaild is built and docker-compose.yml runs five services: mavend, mavsttd, mavttsd, mavweb, mavpoll. It is not among them. The spec makes the product decision the first criterion and that decision has not been recorded anywhere.",
|
||
"evidence": [
|
||
"docker-compose.yml",
|
||
"docs/spec.md § Undesigned in v1"
|
||
]
|
||
},
|
||
"email-triage#e37e": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "The second disjunct holds. docker-compose.yml:145 carries a ten-line commented block naming why mavmaild is off (no IMAP account on this box), the security shape (password from a file, core never sees it, nothing there can create a reminder), and the exact steps to enable it. The absence is deliberate and recorded, in the deployed artifact.",
|
||
"evidence": [
|
||
"docker-compose.yml:145"
|
||
]
|
||
},
|
||
"calendar-management#2ebc": {
|
||
"verified": "blocked",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Not an undecided product question. docker-compose.yml:175 names a concrete blocker: there is no CalDAV account on this box, so mavcaldav is commented out and the calendar query source reads facts nobody writes. Nothing about reading back, creating by voice or naming a conflict can be exercised until an account exists.",
|
||
"evidence": [
|
||
"docker-compose.yml:175"
|
||
]
|
||
},
|
||
"calendar-management#4047": {
|
||
"verified": "blocked",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Not an undecided product question. docker-compose.yml:175 names a concrete blocker: there is no CalDAV account on this box, so mavcaldav is commented out and the calendar query source reads facts nobody writes. Nothing about reading back, creating by voice or naming a conflict can be exercised until an account exists.",
|
||
"evidence": [
|
||
"docker-compose.yml:175"
|
||
]
|
||
},
|
||
"calendar-management#5f7c": {
|
||
"verified": "blocked",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "Not an undecided product question. docker-compose.yml:175 names a concrete blocker: there is no CalDAV account on this box, so mavcaldav is commented out and the calendar query source reads facts nobody writes. Nothing about reading back, creating by voice or naming a conflict can be exercised until an account exists.",
|
||
"evidence": [
|
||
"docker-compose.yml:175"
|
||
]
|
||
},
|
||
"calendar-management#ff27": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "The second disjunct holds and the block says so outright. docker-compose.yml:175 records that mavcaldav 'was built, listed in make build, and deployed nowhere, which is the worst of the three states, this block records the decision instead', then writes out what the absence costs: agenda questions route correctly and answer from an empty table, and loop.State.CalendarBusy stays false.",
|
||
"evidence": [
|
||
"docker-compose.yml:175"
|
||
]
|
||
},
|
||
"see-an-image#c6d5": {
|
||
"verified": "blocked",
|
||
"reason": "configuration missing",
|
||
"detail": "deploy/mavend.json has no vision block. The seam in internal/vision exists with four files and the gemma-4 mmproj is on the box, and nothing is wired to call it. V-667.",
|
||
"evidence": [
|
||
"deploy/mavend.json",
|
||
"docs/spec.md § See an image"
|
||
]
|
||
},
|
||
"see-an-image#bd91": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "The absent vision block is this criterion's premise, not its blocker: with no vision model configured she should say she cannot look and store the image. No image was sent through Telegram, so the degraded path was never exercised. vision_degraded does not exist.",
|
||
"evidence": [
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"see-an-image#0a4a": {
|
||
"verified": "blocked",
|
||
"reason": "configuration missing",
|
||
"detail": "deploy/mavend.json has no vision block. The seam in internal/vision exists with four files and the gemma-4 mmproj is on the box, and nothing is wired to call it. V-667.",
|
||
"evidence": [
|
||
"deploy/mavend.json",
|
||
"docs/spec.md § See an image"
|
||
]
|
||
},
|
||
"mcps#349c": {
|
||
"verified": "blocked",
|
||
"reason": "configuration missing",
|
||
"detail": "One MCP server is configured, vikunja, with enabled: false and a four-tool allowlist. Nothing is callable through the act path while it is off.",
|
||
"evidence": [
|
||
"deploy/mavend.json mcp.servers"
|
||
]
|
||
},
|
||
"mcps#79b8": {
|
||
"verified": "blocked",
|
||
"reason": "configuration missing",
|
||
"detail": "One MCP server is configured, vikunja, with enabled: false and a four-tool allowlist. Nothing is callable through the act path while it is off.",
|
||
"evidence": [
|
||
"deploy/mavend.json mcp.servers"
|
||
]
|
||
},
|
||
"mcps#c89c": {
|
||
"verified": "blocked",
|
||
"reason": "configuration missing",
|
||
"detail": "Only one MCP server is configured, vikunja, and it is disabled. Even enabled, the clause about the rest of the registry still answering has nothing to measure against: a registry of one cannot demonstrate that the others survive a death.",
|
||
"evidence": [
|
||
"deploy/mavend.json mcp.servers"
|
||
]
|
||
},
|
||
"hearing#1e54": {
|
||
"verified": "fail",
|
||
"reason": "deployment missing",
|
||
"detail": "A capture client does ship and it is cmd/mavwaked: it spawns arecord for 16kHz mono PCM, runs silero VAD and the wake head, and sends PushToTalk frames to mavend's voice port. It is installed and enabled on workpc and has been dead since 2026-08-25 18:21:35 on a lost microphone. The spec's 'no capture client ships (V-514)' is out of date; what is absent is mavenclient.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The wake word has been deaf for seven hours"
|
||
]
|
||
},
|
||
"hearing#b416": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "The path is reachable and was not exercised. POST /api/ptt is registered on the same mavweb mux the 58 chat turns used (cmd/mavweb/main.go:241) and proxies raw PCM onto mavend's voice port (cmd/mavweb/voiceproxy.go:47). The path was reachable from the measuring box throughout and no audio probe was written. The mavwaked half of the path is separately dead since 2026-08-25 18:21:35.",
|
||
"evidence": [
|
||
"cmd/mavweb/voiceproxy.go:47",
|
||
"docs/evals/2026-08-26-capability-baseline.md § The speech path was reachable and was not probed"
|
||
]
|
||
},
|
||
"wake-word#10ca": {
|
||
"verified": "fail",
|
||
"reason": "deployment missing",
|
||
"detail": "mavwaked is enabled on workpc under a user unit and has been dead since 2026-08-25 18:21:35, seven hours before this baseline. It exited on 'arecord: pcm_read:2285: read error: No such device' after 2h45m, with status 0/SUCCESS, so systemd treated a lost microphone as a clean exit and did not restart it. Nothing on the box noticed. Neither the wake nor the near-miss half could be exercised.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The wake word has been deaf for seven hours"
|
||
]
|
||
},
|
||
"wake-word#16d8": {
|
||
"verified": "fail",
|
||
"reason": "deployment missing",
|
||
"detail": "The session clause is settled in the negative and does not need the wake word to settle it. 156 of the last 200 delivery attempts are the proactive path finding no voice session, 149 of them one rule retrying once a minute since 2026-08-16. Waking her is not the same as her having a session to push into (V-515), and with mavwaked dead since 18:21:35 neither half runs.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The reaches",
|
||
"docs/evals/2026-08-26-capability-baseline.md § The wake word has been deaf for seven hours"
|
||
]
|
||
},
|
||
"speech-to-text#f8ef": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "mavsttd is deployed and running on the 2026-08-08 image. POST /api/ptt is registered on the same mavweb mux the 58 chat turns used (cmd/mavweb/main.go:241) and proxies raw PCM onto mavend's voice port (cmd/mavweb/voiceproxy.go:47). The path was reachable from the measuring box throughout and no audio probe was written.",
|
||
"evidence": [
|
||
"cmd/mavweb/main.go:241",
|
||
"docs/evals/2026-08-26-capability-baseline.md § The speech path was reachable and was not probed"
|
||
]
|
||
},
|
||
"speech-to-text#eafe": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "The fallback is in effect right now and was not exercised. Both workstation ports refuse from workpc: 192.168.1.105:8080 and :8081 return no HTTP status. workstation.model_disabled is also true. So mavend's stt.Pair has already fallen to the mavsttd floor, and whether it did so silently is exactly what no probe asked. POST /api/ptt is registered on the same mavweb mux the 58 chat turns used (cmd/mavweb/main.go:241) and proxies raw PCM onto mavend's voice port (cmd/mavweb/voiceproxy.go:47). The path was reachable from the measuring box throughout and no audio probe was written.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The speech path was reachable and was not probed"
|
||
]
|
||
},
|
||
"speech-to-text#0959": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "With the workstation transcriber refusing on both ports, mavsttd is the only arm in service, so the floor is trivially exercised on its own. Nothing posted audio to confirm it transcribes. POST /api/ptt is registered on the same mavweb mux the 58 chat turns used (cmd/mavweb/main.go:241) and proxies raw PCM onto mavend's voice port (cmd/mavweb/voiceproxy.go:47). The path was reachable from the measuring box throughout and no audio probe was written.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § The speech path was reachable and was not probed"
|
||
]
|
||
},
|
||
"text-to-speech#d477": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "mavttsd is deployed and running on the 2026-08-08 image. Nothing is missing from the deployment. POST /api/ptt is registered on the same mavweb mux the 58 chat turns used (cmd/mavweb/main.go:241) and proxies raw PCM onto mavend's voice port (cmd/mavweb/voiceproxy.go:47). The path was reachable from the measuring box throughout and no audio probe was written.",
|
||
"evidence": [
|
||
"cmd/mavweb/main.go:241",
|
||
"docs/evals/2026-08-26-capability-baseline.md § The speech path was reachable and was not probed"
|
||
]
|
||
},
|
||
"text-to-speech#4dd3": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "internal/ttsnorm is compiled into mavend, not mavttsd: ttsnorm.Speakable is called at cmd/mavend/voice.go on the reply text and in internal/delivery/voicesink on nudge text. mavend runs the 2026-08-19 image, the same process the baseline drove 58 turns through, so the code is deployed. The scenario tts_normalisation does not exist and no audio probe was written.",
|
||
"evidence": [
|
||
"cmd/mavweb/main.go:241",
|
||
"docs/evals/2026-08-26-capability-baseline.md § The speech path was reachable and was not probed"
|
||
]
|
||
},
|
||
"morning-routine#3a4b": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The morning plan is anchored to an 08:00-11:00 Europe/Samara window a probe cannot reach. Four of the last 200 delivery attempts are channel voice, rule morning:утро, status failed.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"morning-routine#d87d": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The morning plan is anchored to an 08:00-11:00 Europe/Samara window a probe cannot reach. Four of the last 200 delivery attempts are channel voice, rule morning:утро, status failed.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"morning-routine#4bf5": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The morning plan is anchored to an 08:00-11:00 Europe/Samara window a probe cannot reach. Four of the last 200 delivery attempts are channel voice, rule morning:утро, status failed.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"interruption-policy#b7cc": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The four presence-and-severity outcomes fire on the tick, not on a turn. 12 of the last 200 attempts are status dropped, which is the policy acting, and nothing in this run says whether it acted correctly.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"interruption-policy#5dfd": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The four presence-and-severity outcomes fire on the tick, not on a turn. 12 of the last 200 attempts are status dropped, which is the policy acting, and nothing in this run says whether it acted correctly.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"interruption-policy#9f4c": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The four presence-and-severity outcomes fire on the tick, not on a turn. 12 of the last 200 attempts are status dropped, which is the policy acting, and nothing in this run says whether it acted correctly.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"digest-of-held-nudges#7860": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "Digestion runs on its own tick. No suppression cycle completed during the seven minutes the run took, and the nudge count did not move.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"digest-of-held-nudges#c933": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "Digestion runs on its own tick. No suppression cycle completed during the seven minutes the run took, and the nudge count did not move.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"digest-of-held-nudges#1480": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "Digestion runs on its own tick. No suppression cycle completed during the seven minutes the run took, and the nudge count did not move.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"routine-proposals#a705": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The proposer reads repeated behaviour over days. Nothing it could propose from was created during the run, and zero proposals exist.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"routine-proposals#030a": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The proposer reads repeated behaviour over days. Nothing it could propose from was created during the run, and zero proposals exist.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"routine-proposals#d9d3": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The proposer reads repeated behaviour over days. Nothing it could propose from was created during the run, and zero proposals exist.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"memory-evaluation#95a6": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The evaluator runs on its own loop and cannot speak. No run was observed and the notes it would have written are not distinguishable from RSS captures.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"memory-evaluation#8e4e": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The evaluator runs on its own loop and cannot speak. No run was observed and the notes it would have written are not distinguishable from RSS captures.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § What the run does not establish"
|
||
]
|
||
},
|
||
"web-crawling#396b": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "crawl.on_demand is true with a 10s timeout and a 4000-rune cap, and internal/crawl carries robots.go and watch.go. No probe asked for a page, so nothing about politeness, the schedule or what leaves the box was observed.",
|
||
"evidence": [
|
||
"deploy/mavend.json crawl",
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"web-crawling#cfa6": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "Watches are read only from cfg.Crawl.Watches in cmd/mavend/crawls.go. No grammar, intent or IPC method creates one, so a schedule cannot be set by voice, and the deployed crawl block configures no watches at all.",
|
||
"evidence": [
|
||
"cmd/mavend/crawls.go",
|
||
"deploy/mavend.json"
|
||
]
|
||
},
|
||
"web-crawling#aece": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "crawl.on_demand is true with a 10s timeout and a 4000-rune cap, and internal/crawl carries robots.go and watch.go. No probe asked for a page, so nothing about politeness, the schedule or what leaves the box was observed.",
|
||
"evidence": [
|
||
"deploy/mavend.json crawl",
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"web-crawling#29e4": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "crawl.on_demand is true with a 10s timeout and a 4000-rune cap, and internal/crawl carries robots.go and watch.go. No probe asked for a page, so nothing about politeness, the schedule or what leaves the box was observed.",
|
||
"evidence": [
|
||
"deploy/mavend.json crawl",
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"cron-jobs#d326": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "Three scheduling mechanisms exist side by side: routines carry a Cron, reminders carry a Cron no caller passes, and the digest runs on a tick. The spec's last criterion asks for that relationship to be written down so three schedulers do not exist; nothing writes it down.",
|
||
"evidence": [
|
||
"docs/spec.md § Cron jobs",
|
||
"internal/routine",
|
||
"cmd/mavend/tick_routines.go"
|
||
]
|
||
},
|
||
"cron-jobs#5ec4": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "Three scheduling mechanisms exist side by side: routines carry a Cron, reminders carry a Cron no caller passes, and the digest runs on a tick. The spec's last criterion asks for that relationship to be written down so three schedulers do not exist; nothing writes it down.",
|
||
"evidence": [
|
||
"docs/spec.md § Cron jobs",
|
||
"internal/routine",
|
||
"cmd/mavend/tick_routines.go"
|
||
]
|
||
},
|
||
"cron-jobs#d446": {
|
||
"verified": "untested",
|
||
"reason": "scenario missing",
|
||
"detail": "Three scheduling mechanisms exist side by side: routines carry a Cron, reminders carry a Cron no caller passes, and the digest runs on a tick. The spec's last criterion asks for that relationship to be written down so three schedulers do not exist; nothing writes it down.",
|
||
"evidence": [
|
||
"docs/spec.md § Cron jobs",
|
||
"internal/routine",
|
||
"cmd/mavend/tick_routines.go"
|
||
]
|
||
},
|
||
"cron-jobs#acad": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "The document does not exist. Three scheduling mechanisms run side by side: routines carry a Cron read by cmd/mavend/tick_routines.go, reminders carry a Cron no caller in cmd/mavend passes, and the digest runs on its own tick. No doc under docs/ relates them, and the spec asks for exactly that so three schedulers do not exist.",
|
||
"evidence": [
|
||
"internal/routine",
|
||
"cmd/mavend/tick_routines.go"
|
||
]
|
||
},
|
||
"desk-notifications#7397": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "The inbound path exists as the ambient:notif source and morning_missed covers it. No probe posted a desktop notification and morning_missed was not driven, so the confidence, the filing level and the never-spoken-unprompted clause were all unobserved.",
|
||
"evidence": [
|
||
"docs/capabilities/probes_field.json"
|
||
]
|
||
},
|
||
"desk-notifications#f230": {
|
||
"verified": "fail",
|
||
"reason": "code missing",
|
||
"detail": "Neither disjunct holds. internal/delivery/channel.go defines exactly four channels, voice, ntfy, telegram and drop, with no desktop sink, so a nudge cannot appear on the workpc desktop. And the outbound half has not been explicitly dropped from v1 either: docs/spec.md still records which direction the owner means as an open decision.",
|
||
"evidence": [
|
||
"internal/delivery/channel.go",
|
||
"docs/spec.md"
|
||
]
|
||
},
|
||
"recall#91bf": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "internal/router/embedder.go exports EmbedQuery and EmbedPassage, which apply the query: and passage: prefixes multilingual-e5-small was trained with. The only plain Embed call outside a test is internal/memory/recalleval/recalleval.go:167, inside a wrapper.",
|
||
"evidence": [
|
||
"internal/router/embedder.go:55",
|
||
"internal/router/embedder.go:65"
|
||
]
|
||
},
|
||
"recall#3e25": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "TestONNXPersonalBoundary passes in 5.68s with MAVEN_ONNX_LIB set, along with FourFold, SemanticGroupHoldout, Challenge, PostRetuneChallenge, Latency and FrozenHeadMatchesCorpusFit. TestONNXPersonalBoundaryStratified self-skips. The boundary also held on the wire: every recall probe about him was claimed by memory or personal and none reached a world source.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Tests and analyzers"
|
||
]
|
||
},
|
||
"tests-and-analyzers#f070": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "make test is green with -race and -coverprofile and does NOT set MAVEN_ONNX_LIB. Makefile:196 is the whole recipe and the variable is absent; only make t and the four eval-* targets set it. Run under the test target's own environment, all eight TestONNXPersonalBoundary* self-skip in 0.00s and the package still prints ok. Grepping the make test output for 'skip' returns nothing because go test prints no SKIP line without -v.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Tests and analyzers",
|
||
"Makefile:196"
|
||
]
|
||
},
|
||
"tests-and-analyzers#edf0": {
|
||
"verified": "pass",
|
||
"reason": "passes",
|
||
"detail": "The criterion is 'fixed or filed' and V-718 is filed. This run neither reproduces nor refutes it: TestONNXPersonalBoundaryLatency passed in 5.17s under make t, which sets MAVEN_ONNX_LIB and uses no coverage, and it self-skips under make test, which is the only invocation that does use coverage.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Tests and analyzers"
|
||
]
|
||
},
|
||
"tests-and-analyzers#2c7f": {
|
||
"verified": "fail",
|
||
"reason": "external dependency unavailable",
|
||
"detail": "make analyze does not pass. staticcheck and deadcode produced nothing over their baselines. govulncheck reports 7 vulnerabilities, every one in the standard library, every one fixed in go1.25.13, and the vendored toolchain is go1.25.12: make: *** [Makefile:119: vuln] Error 3. The fix is a toolchain bump, not a code change, and the fixed toolchain is not vendored.",
|
||
"evidence": [
|
||
"docs/evals/2026-08-26-capability-baseline.md § Tests and analyzers",
|
||
"Makefile:119"
|
||
]
|
||
},
|
||
"voice#492d": {
|
||
"verified": "untested",
|
||
"reason": "implementation exists with no runtime proof",
|
||
"detail": "Code reading only, which cannot carry a pass. internal/auth/tier.go:84 MaxLayer returns Layer0 for SurfaceVoice, Layer2 for telegram, Layer3 for pc_client, authed_page and core_process, and -1 for anything unrecognised, so it fails closed. No voice turn ran, so neither the cap nor the uncapped read was observed.",
|
||
"evidence": [
|
||
"internal/auth/tier.go:84"
|
||
]
|
||
},
|
||
"encrypted-database#aa1c": {
|
||
"verified": "fail",
|
||
"reason": "wiring missing",
|
||
"detail": "The rule holds in the two daemons that are not deployed and breaks in one that is. cmd/mavcaldav/main.go:53 and cmd/mavmaild/main.go:60 both read from a file and say 'never passed as a flag value'; neither is in docker-compose.yml. cmd/mavpoll/main.go:63 takes -kuma-key as a flag value, and cmd/mavpoll/main.go:593 passes it to req.SetBasicAuth as the password. mavpoll is one of the five deployed services.",
|
||
"evidence": [
|
||
"cmd/mavpoll/main.go:63",
|
||
"cmd/mavpoll/main.go:593",
|
||
"docker-compose.yml"
|
||
]
|
||
}
|
||
} |