Index plans and evaluations by lifecycle (V-674)

2026-08-13 01:27:14 +04:00
parent f957a3ad13
commit fdee3de724
14 changed files with 195 additions and 23 deletions
@@ -8,9 +8,9 @@ The tier is the path, so staleness is visible from the filename.
 | path | holds | lifetime |
 | --- | --- | --- |
 | `docs/*.md` | living. One file per subsystem: the reasoning, corrected in place. Each carries `Last verified: <date> @ <sha>`. | until it is wrong |
-| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. | forever |
+| `docs/evals/` | dated measurements, one file per measurement. **Never edited after the day.** A newer number is a new file. Indexed in `docs/evals/CLAUDE.md`, which marks each one live or superseded. | forever |
 | `docs/caveats/` | known limits, one entry per limit, each with a task id and a revisit trigger. Indexed in `docs/caveats/CLAUDE.md`. | until fixed, then deleted |
-| `docs/plans/` | the plan for one piece of work, frozen once it starts | until the work lands |
+| `docs/plans/` | the plan for one piece of work, frozen once it starts. Indexed in `docs/plans/CLAUDE.md`. | until the work lands |
 | `docs/archive/` | dead. Read by nobody by default. | forever |
 ## Rules for this directory
@@ -0,0 +1,131 @@
 # docs/evals/
 One file per measurement, named for the day it was taken. **Never edited after
 that day.** A newer number is a new file, never an edit to an old one.
 Reasoning does not belong here. It belongs in the subsystem's living doc under
 `docs/`. An eval holds the setup, the numbers and what they rule out.
 ## Rules for this directory
 * Filename is `YYYY-MM-DD-<what-was-measured>.md`. The date is the day it ran.
 * The first line is the claim, not the topic. A reader picks a file from this
  index without opening it, so the H1 has to carry the finding.
 * Head the file with the date, the task id, the box and the build.
 * **A file that replaces an older number says so.** It names the file it
  replaces. This index then marks that one superseded.
 * A superseded file is not deleted and not edited. It records what was believed
  that day. A living doc may still cite it for the run itself.
 ## Reading a number out of here
 Check the state column before citing a row. **A number in a living doc must
 cite a live file.** The 2026-08-11 classifier baseline exists for that reason.
 A pair in `docs/routing.md` went stale unnoticed. Its source predated the
 `encodeWord` fix at `feabf9f`. Nothing failed (V-704).
 ## Routing: the arms and the cascade
 | measurement | state |
 | --- | --- |
 | [Routing evaluation](2026-07-31-routing.md) | superseded |
 | [Resident model bake-off](2026-07-31-model-bakeoff.md) | live |
 | [gemma-4-12b on the workstation, against the resident Qwen3-1.7B](2026-08-02-workstation-gemma4-12b.md) | live |
 | [Routing from audio: four paths, one fixture](2026-08-05-audio-in-routing.md) | live |
 | [Routing with the resident model, re-measured](2026-08-05-routing-resident-model.md) | live |
 | [The routing trajectory, and the number that is missing](2026-08-06-routing-trajectory.md) | live |
 | [Gemma as a label function, and what it found in the seeds](2026-08-06-seed-labels-vs-router-prompt.md) | live |
 | [Moving the seed files onto the router prompt's boundaries](2026-08-06-seeds-to-prompt-boundary.md) | live |
 | [The first destination number](2026-08-08-destination-fixture.md) | superseded |
 | [The destination, with a model that can name one](2026-08-08-destination-model-arm.md) | live |
 | [The routing heads, running in Go](2026-08-08-routing-heads-in-go.md) | live |
 | [Two heads on e5-small, and the first destination the router did not need a model for](2026-08-08-routing-heads-two-head.md) | live |
 | [A slot head, and the corpus that did not exist this morning](2026-08-08-slot-head-three-head.md) | live |
 | [A clarify head, and a confidence that is not a hardcode](2026-08-08-clarify-head-four-head.md) | live |
 | [MASSIVE Russian warm-start for the routing heads](2026-08-08-massive-warm-start.md) | live |
 | [gemma-4-E4B against gemma-4-12B on the routing fixture](2026-08-09-e4b-vs-12b-routing.md) | live |
 | [The classifier baseline after the tokenizer fix](2026-08-11-classifier-baseline-after-tokenizer-fix.md) | live |
 `docs/routing.md` holds the arm table these feed. Cite from there, not from here.
 ## Stage 0, slots and the turn
 | measurement | state |
 | --- | --- |
 | [Personal boundary, seed scoring vs possession markers](2026-08-03-personal-boundary.md) | live |
 | [Half-past and quarter-to hours](2026-08-05-half-past-hours.md) | live |
 | [Praxis reach at stage 0](2026-08-05-praxis-reach.md) | live |
 | [Alarm verbs reach stage 0](2026-08-06-alarm-verbs-reach-stage-0.md) | live |
 | [Russian acts reach tools](2026-08-06-russian-acts-reach-tools.md) | live |
 | [The fact parser: closed classes against the substring stems they replaced](2026-08-06-fact-parser.md) | live |
 | [The reminder completeness rule, measured on the box](2026-08-06-reminder-completeness-on-the-box.md) | live |
 | [Nine claimants interrupting a parked reminder](2026-08-06-claimant-interleave.md) | live |
 | [The parked clarify ride, bounded and re-measured](2026-08-08-parked-clarify-ride.md) | live |
 ## Ecosystem
 | measurement | state |
 | --- | --- |
 | [Ecosystem reach, measured](2026-08-04-ecosystem-reach.md) | live |
 | [Ecosystem reach with the resident model as router](2026-08-05-reach-llm-router.md) | live |
 ## Recall and memory
 | measurement | state |
 | --- | --- |
 | [Note recall evaluation](2026-07-31-recall.md) | superseded |
 | [Recall topic veto, what it costs and what it buys](2026-08-03-recall-topic-veto.md) | live |
 | [Note recall after the e5-small swap](2026-08-04-recall-e5-small.md) | live |
 ## Phrasing and talk
 | measurement | state |
 | --- | --- |
 | [Phrasing evaluation](2026-07-31-phrasing.md) | superseded |
 | [Conversational phrasing eval](2026-07-31-talk.md) | superseded |
 | [Talk fixture against the resident model](2026-08-05-talk-fixture-resident.md) | live |
 | [Talk temperature sweep: Qwen3-1.7B, 4 temperatures times 3 runs](2026-08-05-temperature-sweep.md) | live |
 | [gemma-4-E4B on the phrasing and talk fixtures](2026-08-09-e4b-phrasing.md) | live |
 ## World: search and Kiwix
 | measurement | state |
 | --- | --- |
 | [Does the ZIM answer when the line is down?](2026-08-05-kiwix-offline-fallback.md) | live |
 | [Does SearXNG claim a question it cannot answer?](2026-08-05-search-quality-signals.md) | live |
 | [Kiwix answered the wrong question, and the fix was not a relevance gate](2026-08-09-kiwix-topic-retrieval.md) | live |
 ## Speech in and out
 | measurement | state |
 | --- | --- |
 | [CrisperWhisper 2.0 in Russian, measured](2026-08-09-crisperwhisper2-russian-wer.md) | live |
 | [silero-vad against the energy threshold in mavwaked](2026-08-09-silero-vad.md) | live |
 | [The "Мэйвен" wake word: what it hears and what it invents](2026-08-09-wake-word.md) | live |
 ## Runtime and storage
 | measurement | state |
 | --- | --- |
 | [Where the resident model's 7.9GB of RSS goes](2026-08-03-llama-prompt-cache.md) | live |
 | [Does one sqlite connection make reads queue? No](2026-08-07-store-connection-cap.md) | live |
 ## Whole-system runs
 Each run pairs a write-up with its raw transcript. The transcript is the
 evidence and is not summarised anywhere else.
 | measurement | state |
 | --- | --- |
 | [A week of talking to Maven](2026-08-07-week-of-usage.md), [transcript](2026-08-07-week-of-usage-transcript.md) | live |
 | [Two weeks of talking to Maven, as a baseline to re-run](2026-08-08-two-weeks.md), [transcript](2026-08-08-two-weeks-transcript.md) | live |
 | [The same fortnight, after V-655 merged](2026-08-08-two-weeks-after-merge.md), [transcript](2026-08-08-two-weeks-after-merge-transcript.md) | live |
 ## Audit
 | measurement | state |
 | --- | --- |
 | [Repository deep-audit report](2026-08-10-repo-audit.md) | live |
 Its open findings live in `docs/caveats/`, one entry each with a revisit
 trigger. Read the index there, not this file, for what is still broken.
@@ -28,7 +28,7 @@
 ---
-## Status 2026-08-01 — foundation shipped (Vikunja #248)
+## Status 2026-08-01 — foundation shipped (V-248)
 **Shipped:** `internal/memeval` (not `internal/memory/eval.go` — `internal/store`
 imports `internal/memory` for the vector backend, so an evaluator that reads
@@ -17,7 +17,7 @@ describing half is **BLOCKED on a model download** — see "What is blocked" bel
 | IPC method `describe_image` (`AuthRead`) | `internal/ipc/{wire,api,client,server}.go`, `internal/auth/policy.go` |
 | Daemon wiring + hourly retention prune | `cmd/mavend/vision.go` |
-`internal/media` is deliberately shared: hearing (#253) and speaker recognition (#255) have
+`internal/media` is deliberately shared: hearing (V-253) and speaker recognition (V-255) have
 the same intake problem — a blob arrives, gets stored, gets described — and they store their
 audio in the same place under the same retention.
@@ -77,7 +77,7 @@ To unblock, download one pair to `/mnt/hdd1/llms/vision/` (bind-mounted to
 - `moondream2` — smallest, English-only in practice. Do not bother, per the sub-500M lesson.
 Then run a second llama-server on 8081 with `--mmproj`, point `vision.endpoint` at it, and
-walk the QA steps on Vikunja #252.
+walk the QA steps on V-252.
 ## Config
@@ -14,14 +14,14 @@ shipped is the workpc-side microphone agent and the router intent — see "Still
 |---|---|
 | Session state machine: start / append / stop / abort / status | `internal/capture/capture.go` |
 | Map-reduce summarisation against `n_ctx` 4096 | `internal/capture/summarize.go` |
-| Audio blobs in the shared store, pruned by `media.retention` | `internal/media` (from #252) |
+| Audio blobs in the shared store, pruned by `media.retention` | `internal/media` (from V-252) |
 | Config block `capture`, off by default | `internal/config/config.go` |
 | IPC `capture_start` / `capture_append` / `capture_stop` / `capture_status` | `internal/ipc/{wire,api,client,server}.go` |
 | Authority: the three write methods `AuthWrite`, status `AuthRead` | `internal/auth/policy.go` |
 | Daemon wiring, note write, STT reuse | `cmd/mavend/capture.go` |
 The audio lands in the same content-addressed blob store as images, under the same retention
-loop, because #252 and #253 have the same intake problem and solving it twice would mean two
+loop, because V-252 and V-253 have the same intake problem and solving it twice would mean two
 directories to remember to prune.
 ## The refusals, and why
@@ -117,5 +117,5 @@ Both absent by default. `capture` alone does nothing without `media`.
  talking, so the recorder's stop and the speech barge-in must not collide.
 - **A `/dash` panel** showing a running session, so a recording is visible on a surface and not
  only in a log line.
- **Speaker attribution** — who said what — is #255 and is blocked on a model; see
+- **Speaker attribution** — who said what — is V-255 and is blocked on a model; see
  `docs/plans/10-speaker-recognition.md`.
@@ -3,7 +3,7 @@
 **Goal:** Maven can tell who is speaking on the voice channel, and tag what she writes with
 who said it.
-**Status (2026-08-01, Vikunja #255):** the enrolment half is shipped. The recognising half is
+**Status (2026-08-01, V-255):** the enrolment half is shipped. The recognising half is
 **BLOCKED on a model download** — there is no speaker-embedding model on this box, and one
 was not invented to fill the gap. See "Blocked, and on what" below.
@@ -125,6 +125,6 @@ lied.
 - **An enrolment surface.** The three IPC methods exist; no page drives them. Enrolment is
  step-up, so it belongs on `/dash` behind a passkey, with a per-profile forget button next to
  each row — that button is the reason `list_speakers` exists.
- **A speaker column on the meeting recorder** (#253). Attributing lines in a transcript is
+- **A speaker column on the meeting recorder** (V-253). Attributing lines in a transcript is
  the obvious pairing, and it is the place where getting attribution wrong is most damaging,
  so it waits for a real model too.
@@ -30,7 +30,7 @@
 ---
-## Shipped 2026-08-01 (#258)
+## Shipped 2026-08-01 (V-258)
 `internal/webfetch` (the guarded HTTP door: scheme, allow/deny hosts, private-address
 refusal in the dialer, size cap, redirect cap, per-host rate limit), `internal/rss`
@@ -29,11 +29,11 @@
 8. Add `crawls` block to `config.Config` and `deploy/mavend.json`
 9. Test with a static HTML page — verify extraction matches expected values, verify scheduling fires correctly
-## Shipped 2026-08-01 (#259)
+## Shipped 2026-08-01 (V-259)
 Built as `internal/crawl` (pure: robots, extraction, watcher) plus
 `cmd/mavend/crawls.go` (fetcher, ticker, dedup facts), on top of the guarded
-`internal/webfetch` door added with the feed reader (#258). Off unless
+`internal/webfetch` door added with the feed reader (V-258). Off unless
 configured, in two separately-switched halves: `crawl.on_demand` for a URL he
 names, `crawl.watches` for a scheduled re-read.
@@ -1,6 +1,6 @@
 # Plan: The work board surface
-**The decision Vikunja #431 asked for. Written 04-08-2026.**
+**The decision V-431 asked for. Written 04-08-2026.**
 **Verdict: build it, in a smaller shape than the task imagined.** The board is worth
 moving out of the file. The intake form belongs on the `/tasks` page, not on the voice
@@ -1,6 +1,6 @@
 # Plan: What the ambient calendar path should be
-**The decision Vikunja #432 asked for. Written 04-08-2026.**
+**The decision V-432 asked for. Written 04-08-2026.**
 **Verdict: keep the endpoint, change the contract.** The relay app sends structured
 fields, not a notification blob. The free-text parser stays as the degraded path, because
@@ -9,7 +9,7 @@ if the answer to the one open question below is no.
 ## First, the task's premise is out of date
-#432 states as confirmed that every ambient event lands on the day the notification was
+V-432 states as confirmed that every ambient event lands on the day the notification was
 posted, because there is no date parsing at all. That was true when the task was filed
 and it is not true now.
@@ -17,7 +17,7 @@ and it is not true now.
 "завтра в 15:00" now dates to tomorrow. The same commit added `ambientPastGrace`, which
 refuses an event landing more than two hours before the notification, on the reasoning
 that the day was inferred and a stale inference is wrong rather than late. `45a5e37`
-(#482, this week) fixed a second dating bug the task did not know about: the wall clock
+(V-482, this week) fixed a second dating bug the task did not know about: the wall clock
 was resolved against the notification's own zone, so every ambient meeting on a non-UTC
 box landed off by the deploy's UTC offset.
@@ -38,7 +38,7 @@ Everything follows from this and nothing in this repo can answer it.
 A `NotificationListenerService` sees a title and a body. It cannot know a meeting's real
 start, end or organiser, because those are not in the notification. So if the relay is
 limited to the notification stream, free-text parsing on this side is not a choice, it is
-the only thing available, and #432's suggestion that the phone send structured JSON
+the only thing available, and V-432's suggestion that the phone send structured JSON
 cannot be honoured.
 If the app may instead read `CalendarContract`, it has the actual event rows, and the
@@ -73,7 +73,7 @@ closes this without code, and it is his to give.
 ## What not to do
-Do not add date parsing to the free-text path yet. That is the patch #432 explicitly
+Do not add date parsing to the free-text path yet. That is the patch V-432 explicitly
 refuses to accept as closure, and it is the wrong order: if the relay can send a date, no
 date parser is needed, and if it cannot, the parser is guessing at a date from text that
 was never meant to carry one.
@@ -1,6 +1,6 @@
 # Plan: Where mavwaked and mavenclient run
-**The decision Vikunja #463 asked for. Written 04-08-2026.**
+**The decision V-463 asked for. Written 04-08-2026.**
 **Verdict: not in compose on homesrv. They run on a client machine in the room he is in.**
 The transport for that already exists and nothing needs building to allow it. What needs
@@ -11,7 +11,7 @@ building is a way to check the wake path at all, which is a separate task.
 `docker-compose.yml` runs mavend, mavsttd, mavttsd, mavweb and mavpoll. `mavwaked` and
 `mavenclient` appear in no compose file and run as no host process. Both build under
 `make build`. So the wake word and the voice-activity gate are untested by construction:
-QA session 1 step 2 covers push-to-talk from `/dash` only, and #287 (voice session
+QA session 1 step 2 covers push-to-talk from `/dash` only, and V-287 (voice session
 quality) can never be more than half-answered while this holds.
 ## The reason is not hardware
@@ -1,6 +1,6 @@
 # Plan: route with heads on e5-small, not with a generative model
-**Owner's call, 05-08-2026. Vikunja #546.**
+**Owner's call, 05-08-2026. V-546.**
 **Verdict: the routing model is the 118M multilingual-e5-small already resident on
 homesrv.** It gets one classification head per output. No LoRA on a decoder, no 100M model
@@ -1,6 +1,6 @@
 # Plan: persist the routing trace
-**Owner's call, 06-08-2026. Vikunja #629, umbrella #628.**
+**Owner's call, 06-08-2026. V-629, umbrella V-628.**
 **Verdict: the per-turn decision record now persists.** That reverses a written decision,
 which is the point of this file. It is not an incidental telemetry
@@ -0,0 +1,41 @@
 # docs/plans/
 The plan for one piece of work, written before it starts and frozen once it
 does. A plan is what was intended. What happened is the commits, the
 subsystem's living doc under `docs/`, and the measurement under `docs/evals/`.
 Changes of mind go to the Vikunja task, never back into the plan. The value of
 a frozen plan is that it can be compared to the outcome afterwards.
 ## Rules for this directory
 * **A plan carries its task id in the `V-<id>` form.** `V-` and not `#`,
  because Gitea autolinks `#123` to the wrong tracker. Put it in the first three
  lines, so it is visible without reading the plan.
 * A plan whose work has landed stays. It is not edited to say "shipped" beyond
  a dated status line, and it is never rewritten to match what was built.
 * A plan nobody will act on goes to `docs/archive/`, not to the bin.
 ## Two naming schemes, on purpose
 `NN-<name>.md` is a Maven feature plan. The number is an append-order id and
 carries no priority. A new plan takes the next free number. Six Go source
 comments cite these paths, so renaming one is a code change.
 `YYYY-MM-DD-<name>.md` is a training runbook for the model rig on workpc, not
 for this repo's daemons. Five exist, all from July 2026, all predating the
 numbering. They are dated because a runbook is read as of a day. Renumbering
 would put the oldest plans last and imply an order that is not real. Do not do
 it.
 ## Plans with no task id
 These have never carried one. Do not invent ids for them. Resolve each against
 Vikunja project Maven (ID 2) before citing it, or file a task if none exists.
 `01-email-reader.md`, `02-pattern-proposal.md`, `04-self-update.md`,
 `05-model-swap.md`, `06-mcp-support.md`, `09-behavioral-memory.md`,
 `11-smarthome-integration.md`, `12-bluetooth-network-scan.md`.
 The five training runbooks have no id either. The nearest live task is V-122,
 which owns finishing the Qwen3-1.7B CPT run on workpc.