From fdee3de724eefdc9cdf5433f79f7b0c369a35acf Mon Sep 17 00:00:00 2001 From: claude Date: Thu, 13 Aug 2026 01:27:14 +0400 Subject: [PATCH] Index plans and evaluations by lifecycle (V-674) --- docs/CLAUDE.md | 4 +- docs/evals/CLAUDE.md | 131 ++++++++++++++++++ docs/plans/03-memory-evaluation.md | 2 +- docs/plans/07-vision.md | 4 +- docs/plans/08-hearing.md | 6 +- docs/plans/10-speaker-recognition.md | 4 +- docs/plans/13-rss-news-feeds.md | 2 +- docs/plans/14-web-crawler.md | 4 +- docs/plans/15-board-surface.md | 2 +- docs/plans/16-ambient-calendar.md | 10 +- docs/plans/17-where-the-voice-loop-runs.md | 4 +- docs/plans/18-routing-heads-on-e5-small.md | 2 +- docs/plans/21-persisting-the-routing-trace.md | 2 +- docs/plans/CLAUDE.md | 41 ++++++ 14 files changed, 195 insertions(+), 23 deletions(-) create mode 100644 docs/evals/CLAUDE.md create mode 100644 docs/plans/CLAUDE.md diff --git a/docs/CLAUDE.md b/docs/CLAUDE.md index 0113a06..38ca5a7 100644 --- a/docs/CLAUDE.md +++ b/docs/CLAUDE.md @@ -8,9 +8,9 @@ The tier is the path, so staleness is visible from the filename. | path | holds | lifetime | | --- | --- | --- | | `docs/*.md` | living. One file per subsystem: the reasoning, corrected in place. Each carries `Last verified: @ `. | until it is wrong | -| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. | forever | +| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. Indexed in `docs/evals/CLAUDE.md`, which marks each one live or superseded. | forever | | `docs/caveats/` | known limits, one entry per limit, each with a task id and a revisit trigger. Indexed in `docs/caveats/CLAUDE.md`. | until fixed, then deleted | -| `docs/plans/` | the plan for one piece of work, frozen once it starts | until the work lands | +| `docs/plans/` | the plan for one piece of work, frozen once it starts. Indexed in `docs/plans/CLAUDE.md`. | until the work lands | | `docs/archive/` | dead. Read by nobody by default. | forever | ## Rules for this directory diff --git a/docs/evals/CLAUDE.md b/docs/evals/CLAUDE.md new file mode 100644 index 0000000..65e68c8 --- /dev/null +++ b/docs/evals/CLAUDE.md @@ -0,0 +1,131 @@ +# docs/evals/ + +One file per measurement, named for the day it was taken. **Never edited after +that day.** A newer number is a new file, never an edit to an old one. + +Reasoning does not belong here. It belongs in the subsystem's living doc under +`docs/`. An eval holds the setup, the numbers and what they rule out. + +## Rules for this directory + +* Filename is `YYYY-MM-DD-.md`. The date is the day it ran. +* The first line is the claim, not the topic. A reader picks a file from this + index without opening it, so the H1 has to carry the finding. +* Head the file with the date, the task id, the box and the build. +* **A file that replaces an older number says so.** It names the file it + replaces. This index then marks that one superseded. +* A superseded file is not deleted and not edited. It records what was believed + that day. A living doc may still cite it for the run itself. + +## Reading a number out of here + +Check the state column before citing a row. **A number in a living doc must +cite a live file.** The 2026-08-11 classifier baseline exists for that reason. +A pair in `docs/routing.md` went stale unnoticed. Its source predated the +`encodeWord` fix at `feabf9f`. Nothing failed (V-704). + +## Routing: the arms and the cascade + +| measurement | state | +| --- | --- | +| [Routing evaluation](2026-07-31-routing.md) | superseded | +| [Resident model bake-off](2026-07-31-model-bakeoff.md) | live | +| [gemma-4-12b on the workstation, against the resident Qwen3-1.7B](2026-08-02-workstation-gemma4-12b.md) | live | +| [Routing from audio: four paths, one fixture](2026-08-05-audio-in-routing.md) | live | +| [Routing with the resident model, re-measured](2026-08-05-routing-resident-model.md) | live | +| [The routing trajectory, and the number that is missing](2026-08-06-routing-trajectory.md) | live | +| [Gemma as a label function, and what it found in the seeds](2026-08-06-seed-labels-vs-router-prompt.md) | live | +| [Moving the seed files onto the router prompt's boundaries](2026-08-06-seeds-to-prompt-boundary.md) | live | +| [The first destination number](2026-08-08-destination-fixture.md) | superseded | +| [The destination, with a model that can name one](2026-08-08-destination-model-arm.md) | live | +| [The routing heads, running in Go](2026-08-08-routing-heads-in-go.md) | live | +| [Two heads on e5-small, and the first destination the router did not need a model for](2026-08-08-routing-heads-two-head.md) | live | +| [A slot head, and the corpus that did not exist this morning](2026-08-08-slot-head-three-head.md) | live | +| [A clarify head, and a confidence that is not a hardcode](2026-08-08-clarify-head-four-head.md) | live | +| [MASSIVE Russian warm-start for the routing heads](2026-08-08-massive-warm-start.md) | live | +| [gemma-4-E4B against gemma-4-12B on the routing fixture](2026-08-09-e4b-vs-12b-routing.md) | live | +| [The classifier baseline after the tokenizer fix](2026-08-11-classifier-baseline-after-tokenizer-fix.md) | live | + +`docs/routing.md` holds the arm table these feed. Cite from there, not from here. + +## Stage 0, slots and the turn + +| measurement | state | +| --- | --- | +| [Personal boundary, seed scoring vs possession markers](2026-08-03-personal-boundary.md) | live | +| [Half-past and quarter-to hours](2026-08-05-half-past-hours.md) | live | +| [Praxis reach at stage 0](2026-08-05-praxis-reach.md) | live | +| [Alarm verbs reach stage 0](2026-08-06-alarm-verbs-reach-stage-0.md) | live | +| [Russian acts reach tools](2026-08-06-russian-acts-reach-tools.md) | live | +| [The fact parser: closed classes against the substring stems they replaced](2026-08-06-fact-parser.md) | live | +| [The reminder completeness rule, measured on the box](2026-08-06-reminder-completeness-on-the-box.md) | live | +| [Nine claimants interrupting a parked reminder](2026-08-06-claimant-interleave.md) | live | +| [The parked clarify ride, bounded and re-measured](2026-08-08-parked-clarify-ride.md) | live | + +## Ecosystem + +| measurement | state | +| --- | --- | +| [Ecosystem reach, measured](2026-08-04-ecosystem-reach.md) | live | +| [Ecosystem reach with the resident model as router](2026-08-05-reach-llm-router.md) | live | + +## Recall and memory + +| measurement | state | +| --- | --- | +| [Note recall evaluation](2026-07-31-recall.md) | superseded | +| [Recall topic veto, what it costs and what it buys](2026-08-03-recall-topic-veto.md) | live | +| [Note recall after the e5-small swap](2026-08-04-recall-e5-small.md) | live | + +## Phrasing and talk + +| measurement | state | +| --- | --- | +| [Phrasing evaluation](2026-07-31-phrasing.md) | superseded | +| [Conversational phrasing eval](2026-07-31-talk.md) | superseded | +| [Talk fixture against the resident model](2026-08-05-talk-fixture-resident.md) | live | +| [Talk temperature sweep: Qwen3-1.7B, 4 temperatures times 3 runs](2026-08-05-temperature-sweep.md) | live | +| [gemma-4-E4B on the phrasing and talk fixtures](2026-08-09-e4b-phrasing.md) | live | + +## World: search and Kiwix + +| measurement | state | +| --- | --- | +| [Does the ZIM answer when the line is down?](2026-08-05-kiwix-offline-fallback.md) | live | +| [Does SearXNG claim a question it cannot answer?](2026-08-05-search-quality-signals.md) | live | +| [Kiwix answered the wrong question, and the fix was not a relevance gate](2026-08-09-kiwix-topic-retrieval.md) | live | + +## Speech in and out + +| measurement | state | +| --- | --- | +| [CrisperWhisper 2.0 in Russian, measured](2026-08-09-crisperwhisper2-russian-wer.md) | live | +| [silero-vad against the energy threshold in mavwaked](2026-08-09-silero-vad.md) | live | +| [The "Мэйвен" wake word: what it hears and what it invents](2026-08-09-wake-word.md) | live | + +## Runtime and storage + +| measurement | state | +| --- | --- | +| [Where the resident model's 7.9GB of RSS goes](2026-08-03-llama-prompt-cache.md) | live | +| [Does one sqlite connection make reads queue? No](2026-08-07-store-connection-cap.md) | live | + +## Whole-system runs + +Each run pairs a write-up with its raw transcript. The transcript is the +evidence and is not summarised anywhere else. + +| measurement | state | +| --- | --- | +| [A week of talking to Maven](2026-08-07-week-of-usage.md), [transcript](2026-08-07-week-of-usage-transcript.md) | live | +| [Two weeks of talking to Maven, as a baseline to re-run](2026-08-08-two-weeks.md), [transcript](2026-08-08-two-weeks-transcript.md) | live | +| [The same fortnight, after V-655 merged](2026-08-08-two-weeks-after-merge.md), [transcript](2026-08-08-two-weeks-after-merge-transcript.md) | live | + +## Audit + +| measurement | state | +| --- | --- | +| [Repository deep-audit report](2026-08-10-repo-audit.md) | live | + +Its open findings live in `docs/caveats/`, one entry each with a revisit +trigger. Read the index there, not this file, for what is still broken. diff --git a/docs/plans/03-memory-evaluation.md b/docs/plans/03-memory-evaluation.md index 2dfedde..a344e30 100644 --- a/docs/plans/03-memory-evaluation.md +++ b/docs/plans/03-memory-evaluation.md @@ -28,7 +28,7 @@ --- -## Status 2026-08-01 — foundation shipped (Vikunja #248) +## Status 2026-08-01 — foundation shipped (V-248) **Shipped:** `internal/memeval` (not `internal/memory/eval.go` — `internal/store` imports `internal/memory` for the vector backend, so an evaluator that reads diff --git a/docs/plans/07-vision.md b/docs/plans/07-vision.md index 5483a52..4a0ead8 100644 --- a/docs/plans/07-vision.md +++ b/docs/plans/07-vision.md @@ -17,7 +17,7 @@ describing half is **BLOCKED on a model download** — see "What is blocked" bel | IPC method `describe_image` (`AuthRead`) | `internal/ipc/{wire,api,client,server}.go`, `internal/auth/policy.go` | | Daemon wiring + hourly retention prune | `cmd/mavend/vision.go` | -`internal/media` is deliberately shared: hearing (#253) and speaker recognition (#255) have +`internal/media` is deliberately shared: hearing (V-253) and speaker recognition (V-255) have the same intake problem — a blob arrives, gets stored, gets described — and they store their audio in the same place under the same retention. @@ -77,7 +77,7 @@ To unblock, download one pair to `/mnt/hdd1/llms/vision/` (bind-mounted to - `moondream2` — smallest, English-only in practice. Do not bother, per the sub-500M lesson. Then run a second llama-server on 8081 with `--mmproj`, point `vision.endpoint` at it, and -walk the QA steps on Vikunja #252. +walk the QA steps on V-252. ## Config diff --git a/docs/plans/08-hearing.md b/docs/plans/08-hearing.md index 41c30a0..944e296 100644 --- a/docs/plans/08-hearing.md +++ b/docs/plans/08-hearing.md @@ -14,14 +14,14 @@ shipped is the workpc-side microphone agent and the router intent — see "Still |---|---| | Session state machine: start / append / stop / abort / status | `internal/capture/capture.go` | | Map-reduce summarisation against `n_ctx` 4096 | `internal/capture/summarize.go` | -| Audio blobs in the shared store, pruned by `media.retention` | `internal/media` (from #252) | +| Audio blobs in the shared store, pruned by `media.retention` | `internal/media` (from V-252) | | Config block `capture`, off by default | `internal/config/config.go` | | IPC `capture_start` / `capture_append` / `capture_stop` / `capture_status` | `internal/ipc/{wire,api,client,server}.go` | | Authority: the three write methods `AuthWrite`, status `AuthRead` | `internal/auth/policy.go` | | Daemon wiring, note write, STT reuse | `cmd/mavend/capture.go` | The audio lands in the same content-addressed blob store as images, under the same retention -loop, because #252 and #253 have the same intake problem and solving it twice would mean two +loop, because V-252 and V-253 have the same intake problem and solving it twice would mean two directories to remember to prune. ## The refusals, and why @@ -117,5 +117,5 @@ Both absent by default. `capture` alone does nothing without `media`. talking, so the recorder's stop and the speech barge-in must not collide. - **A `/dash` panel** showing a running session, so a recording is visible on a surface and not only in a log line. -- **Speaker attribution** — who said what — is #255 and is blocked on a model; see +- **Speaker attribution** — who said what — is V-255 and is blocked on a model; see `docs/plans/10-speaker-recognition.md`. diff --git a/docs/plans/10-speaker-recognition.md b/docs/plans/10-speaker-recognition.md index 006db41..ec3b21f 100644 --- a/docs/plans/10-speaker-recognition.md +++ b/docs/plans/10-speaker-recognition.md @@ -3,7 +3,7 @@ **Goal:** Maven can tell who is speaking on the voice channel, and tag what she writes with who said it. -**Status (2026-08-01, Vikunja #255):** the enrolment half is shipped. The recognising half is +**Status (2026-08-01, V-255):** the enrolment half is shipped. The recognising half is **BLOCKED on a model download** — there is no speaker-embedding model on this box, and one was not invented to fill the gap. See "Blocked, and on what" below. @@ -125,6 +125,6 @@ lied. - **An enrolment surface.** The three IPC methods exist; no page drives them. Enrolment is step-up, so it belongs on `/dash` behind a passkey, with a per-profile forget button next to each row — that button is the reason `list_speakers` exists. -- **A speaker column on the meeting recorder** (#253). Attributing lines in a transcript is +- **A speaker column on the meeting recorder** (V-253). Attributing lines in a transcript is the obvious pairing, and it is the place where getting attribution wrong is most damaging, so it waits for a real model too. diff --git a/docs/plans/13-rss-news-feeds.md b/docs/plans/13-rss-news-feeds.md index 5a4289e..01a4727 100644 --- a/docs/plans/13-rss-news-feeds.md +++ b/docs/plans/13-rss-news-feeds.md @@ -30,7 +30,7 @@ --- -## Shipped 2026-08-01 (#258) +## Shipped 2026-08-01 (V-258) `internal/webfetch` (the guarded HTTP door: scheme, allow/deny hosts, private-address refusal in the dialer, size cap, redirect cap, per-host rate limit), `internal/rss` diff --git a/docs/plans/14-web-crawler.md b/docs/plans/14-web-crawler.md index a9b22f3..cdc6a45 100644 --- a/docs/plans/14-web-crawler.md +++ b/docs/plans/14-web-crawler.md @@ -29,11 +29,11 @@ 8. Add `crawls` block to `config.Config` and `deploy/mavend.json` 9. Test with a static HTML page — verify extraction matches expected values, verify scheduling fires correctly -## Shipped 2026-08-01 (#259) +## Shipped 2026-08-01 (V-259) Built as `internal/crawl` (pure: robots, extraction, watcher) plus `cmd/mavend/crawls.go` (fetcher, ticker, dedup facts), on top of the guarded -`internal/webfetch` door added with the feed reader (#258). Off unless +`internal/webfetch` door added with the feed reader (V-258). Off unless configured, in two separately-switched halves: `crawl.on_demand` for a URL he names, `crawl.watches` for a scheduled re-read. diff --git a/docs/plans/15-board-surface.md b/docs/plans/15-board-surface.md index 5b49d30..c6e1160 100644 --- a/docs/plans/15-board-surface.md +++ b/docs/plans/15-board-surface.md @@ -1,6 +1,6 @@ # Plan: The work board surface -**The decision Vikunja #431 asked for. Written 04-08-2026.** +**The decision V-431 asked for. Written 04-08-2026.** **Verdict: build it, in a smaller shape than the task imagined.** The board is worth moving out of the file. The intake form belongs on the `/tasks` page, not on the voice diff --git a/docs/plans/16-ambient-calendar.md b/docs/plans/16-ambient-calendar.md index 9478c6a..418a279 100644 --- a/docs/plans/16-ambient-calendar.md +++ b/docs/plans/16-ambient-calendar.md @@ -1,6 +1,6 @@ # Plan: What the ambient calendar path should be -**The decision Vikunja #432 asked for. Written 04-08-2026.** +**The decision V-432 asked for. Written 04-08-2026.** **Verdict: keep the endpoint, change the contract.** The relay app sends structured fields, not a notification blob. The free-text parser stays as the degraded path, because @@ -9,7 +9,7 @@ if the answer to the one open question below is no. ## First, the task's premise is out of date -#432 states as confirmed that every ambient event lands on the day the notification was +V-432 states as confirmed that every ambient event lands on the day the notification was posted, because there is no date parsing at all. That was true when the task was filed and it is not true now. @@ -17,7 +17,7 @@ and it is not true now. "завтра в 15:00" now dates to tomorrow. The same commit added `ambientPastGrace`, which refuses an event landing more than two hours before the notification, on the reasoning that the day was inferred and a stale inference is wrong rather than late. `45a5e37` -(#482, this week) fixed a second dating bug the task did not know about: the wall clock +(V-482, this week) fixed a second dating bug the task did not know about: the wall clock was resolved against the notification's own zone, so every ambient meeting on a non-UTC box landed off by the deploy's UTC offset. @@ -38,7 +38,7 @@ Everything follows from this and nothing in this repo can answer it. A `NotificationListenerService` sees a title and a body. It cannot know a meeting's real start, end or organiser, because those are not in the notification. So if the relay is limited to the notification stream, free-text parsing on this side is not a choice, it is -the only thing available, and #432's suggestion that the phone send structured JSON +the only thing available, and V-432's suggestion that the phone send structured JSON cannot be honoured. If the app may instead read `CalendarContract`, it has the actual event rows, and the @@ -73,7 +73,7 @@ closes this without code, and it is his to give. ## What not to do -Do not add date parsing to the free-text path yet. That is the patch #432 explicitly +Do not add date parsing to the free-text path yet. That is the patch V-432 explicitly refuses to accept as closure, and it is the wrong order: if the relay can send a date, no date parser is needed, and if it cannot, the parser is guessing at a date from text that was never meant to carry one. diff --git a/docs/plans/17-where-the-voice-loop-runs.md b/docs/plans/17-where-the-voice-loop-runs.md index eafbd83..fc602cd 100644 --- a/docs/plans/17-where-the-voice-loop-runs.md +++ b/docs/plans/17-where-the-voice-loop-runs.md @@ -1,6 +1,6 @@ # Plan: Where mavwaked and mavenclient run -**The decision Vikunja #463 asked for. Written 04-08-2026.** +**The decision V-463 asked for. Written 04-08-2026.** **Verdict: not in compose on homesrv. They run on a client machine in the room he is in.** The transport for that already exists and nothing needs building to allow it. What needs @@ -11,7 +11,7 @@ building is a way to check the wake path at all, which is a separate task. `docker-compose.yml` runs mavend, mavsttd, mavttsd, mavweb and mavpoll. `mavwaked` and `mavenclient` appear in no compose file and run as no host process. Both build under `make build`. So the wake word and the voice-activity gate are untested by construction: -QA session 1 step 2 covers push-to-talk from `/dash` only, and #287 (voice session +QA session 1 step 2 covers push-to-talk from `/dash` only, and V-287 (voice session quality) can never be more than half-answered while this holds. ## The reason is not hardware diff --git a/docs/plans/18-routing-heads-on-e5-small.md b/docs/plans/18-routing-heads-on-e5-small.md index 9c3b774..41e569a 100644 --- a/docs/plans/18-routing-heads-on-e5-small.md +++ b/docs/plans/18-routing-heads-on-e5-small.md @@ -1,6 +1,6 @@ # Plan: route with heads on e5-small, not with a generative model -**Owner's call, 05-08-2026. Vikunja #546.** +**Owner's call, 05-08-2026. V-546.** **Verdict: the routing model is the 118M multilingual-e5-small already resident on homesrv.** It gets one classification head per output. No LoRA on a decoder, no 100M model diff --git a/docs/plans/21-persisting-the-routing-trace.md b/docs/plans/21-persisting-the-routing-trace.md index 62a5f0d..bbb9e63 100644 --- a/docs/plans/21-persisting-the-routing-trace.md +++ b/docs/plans/21-persisting-the-routing-trace.md @@ -1,6 +1,6 @@ # Plan: persist the routing trace -**Owner's call, 06-08-2026. Vikunja #629, umbrella #628.** +**Owner's call, 06-08-2026. V-629, umbrella V-628.** **Verdict: the per-turn decision record now persists.** That reverses a written decision, which is the point of this file. It is not an incidental telemetry diff --git a/docs/plans/CLAUDE.md b/docs/plans/CLAUDE.md new file mode 100644 index 0000000..562b614 --- /dev/null +++ b/docs/plans/CLAUDE.md @@ -0,0 +1,41 @@ +# docs/plans/ + +The plan for one piece of work, written before it starts and frozen once it +does. A plan is what was intended. What happened is the commits, the +subsystem's living doc under `docs/`, and the measurement under `docs/evals/`. + +Changes of mind go to the Vikunja task, never back into the plan. The value of +a frozen plan is that it can be compared to the outcome afterwards. + +## Rules for this directory + +* **A plan carries its task id in the `V-` form.** `V-` and not `#`, + because Gitea autolinks `#123` to the wrong tracker. Put it in the first three + lines, so it is visible without reading the plan. +* A plan whose work has landed stays. It is not edited to say "shipped" beyond + a dated status line, and it is never rewritten to match what was built. +* A plan nobody will act on goes to `docs/archive/`, not to the bin. + +## Two naming schemes, on purpose + +`NN-.md` is a Maven feature plan. The number is an append-order id and +carries no priority. A new plan takes the next free number. Six Go source +comments cite these paths, so renaming one is a code change. + +`YYYY-MM-DD-.md` is a training runbook for the model rig on workpc, not +for this repo's daemons. Five exist, all from July 2026, all predating the +numbering. They are dated because a runbook is read as of a day. Renumbering +would put the oldest plans last and imply an order that is not real. Do not do +it. + +## Plans with no task id + +These have never carried one. Do not invent ids for them. Resolve each against +Vikunja project Maven (ID 2) before citing it, or file a task if none exists. + +`01-email-reader.md`, `02-pattern-proposal.md`, `04-self-update.md`, +`05-model-swap.md`, `06-mcp-support.md`, `09-behavioral-memory.md`, +`11-smarthome-integration.md`, `12-bluetooth-network-scan.md`. + +The five training runbooks have no id either. The nearest live task is V-122, +which owns finishing the Qwen3-1.7B CPT run on workpc.