From 09f05549a09286516d97dca1940e29ee114e1059 Mon Sep 17 00:00:00 2001 From: kami Date: Sun, 14 Jun 2026 18:41:20 +0400 Subject: [PATCH] docs(backlog): mark event-store health probe shipped, add QA plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flip §A EVENT_STORE probe to pending-live-QA (7a1362c), register the QA gate in §F, and add docs/qa/QA-event-store-health-probe.md (responsive/HEALTHY, no-log-pollution, slow/throw DEGRADED, restore hysteresis, replay independence). --- BACKLOG.md | 7 +++- docs/qa/QA-event-store-health-probe.md | 55 ++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 1 deletion(-) create mode 100644 docs/qa/QA-event-store-health-probe.md diff --git a/BACKLOG.md b/BACKLOG.md index 5ca294b0..ca44c1d3 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -30,7 +30,10 @@ f8fd260) — those are RETRO candidates once live-QA'd. Remaining: `GET /health` + tokens/sec degradation watch (windowed avg of recorded `InferenceCompletedEvent`s, opt-in via `llama_tps_warn_below`). Plugs into the existing `HealthMonitor`. QA gate in §F → `docs/qa/QA-llama-health-probe.md`. -- [ ] **§4 EVENT_STORE health probe** — append + fsync latency heartbeat. +- [~] **§4 EVENT_STORE health probe** — **SHIPPED (pending live-QA)** `7a1362c`. Times a cheap + `lastGlobalSequence()` read as a responsiveness heartbeat (DEGRADED on throw or over + `event_store_latency_warn_ms`); read-only by design (no probe writes into the log). QA gate in §F → + `docs/qa/QA-event-store-health-probe.md`. - [ ] **§4/§3 live health TUI pane** — deferred to follow the metrics cadence (projection + CLI first, then pane). `correx health` CLI already exists. - [ ] **§3 tier-2 stats** — full session-completion summary pane polish (basic pane shipped). @@ -125,6 +128,8 @@ These are SHIPPED in code but prompt-/server-/network-dependent and not yet live - [ ] **LLAMA_SERVER health probe live-QA** — `aaf896d`; needs a real llama-server to ping. Plan: `docs/qa/QA-llama-health-probe.md` (liveness up/down/restore, non-2xx, tps degrade, restart hysteresis, replay independence). +- [ ] **EVENT_STORE health probe live-QA** — `7a1362c`. Plan: `docs/qa/QA-event-store-health-probe.md` + (responsive→HEALTHY, no-log-pollution, slow/throw→DEGRADED, restore hysteresis, replay independence). - [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the *same* stage's output; AMD VRAM/RAM gauge on the real box. - [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection), diff --git a/docs/qa/QA-event-store-health-probe.md b/docs/qa/QA-event-store-health-probe.md new file mode 100644 index 00000000..64fe7b3d --- /dev/null +++ b/docs/qa/QA-event-store-health-probe.md @@ -0,0 +1,55 @@ +# QA Plan: EVENT_STORE health probe — 7a1362c + +Observability spec §4. Drafted from the diff (commit `7a1362c`) per the BACKLOG QA rule. + +**Status:** DRAFT +**Run date / operator:** +**BACKLOG item:** §A "§4 EVENT_STORE health probe — append + fsync latency heartbeat". + +--- + +## Preconditions + +- [ ] **server build/branch:** master @ `7a1362c` (or later) — _rebuild only with the server stopped_ +- [ ] **config:** `[health].enabled = true`; new field `[health].event_store_latency_warn_ms` (default 500). +- [ ] **interval:** `[health].interval_ms` (default 30_000) — how long a status edge takes to surface. +- [ ] **how to read state:** `correx health` / `GET /health/checks`; raw events via `correx events __system__`. +- [ ] **(for the slow-path check)** a way to make a real SQLite read slow — e.g. point the event store at a + path on a slow/contended/network filesystem, or temporarily lower `event_store_latency_warn_ms` to a + value below normal read latency to force a DEGRADED. + +## Acceptance gate (one sentence) + +> The probe is correct **iff** a responsive store reads HEALTHY (latency recorded), an unresponsive or +> slow store emits exactly one `HealthDegradedEvent(subject=EVENT_STORE)` (visible in `correx health` and +> the `__system__` log), recovery emits exactly one `HealthRestoredEvent`, the probe NEVER writes to the +> log itself, and replay re-derives all of it from recorded events. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Server up, `health.enabled=true`, wait one interval | `correx health` shows EVENT_STORE HEALTHY; detail "event store responsive: lastGlobalSequence() in Nms"; `observedValue` = a small latency | | +| 2 | **No log pollution:** note `correx events __system__` event count, let several health intervals pass | The `__system__` event count does NOT grow per tick — only on actual status *edges*. The probe must not append a heartbeat event every tick (Invariant #1) | | +| 3 | Force slow reads (slow FS, or set `event_store_latency_warn_ms` below normal latency), wait one interval | Exactly one `HealthDegradedEvent` subject=EVENT_STORE, metric `read_latency_ms`, detail "lastGlobalSequence() took Nms (warn ≥ …ms)" | | +| 4 | Restore normal latency (or raise the threshold back), wait one interval | Exactly one `HealthRestoredEvent(EVENT_STORE)`; back to HEALTHY. No repeated Degraded/Restored on later ticks (hysteresis = edge-only) | | +| 5 | Simulate store failure (e.g. make the DB unreadable mid-run, if feasible) | `HealthDegradedEvent` with detail "lastGlobalSequence() threw: …"; the monitor tick survives (no crash, next tick still runs) | | +| 6 | Restart the **correx** server while the store is HEALTHY | No spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection) | | +| 7 | `correx replay __system__` | Replay re-derives the same health timeline from recorded `HealthObservation` events; double-read digest stable; no live store-latency re-probe (invariants #8/#9) | | + +Evidence sources: `correx events __system__` (recorded truth + the no-growth check), `correx health` / +`GET /health/checks`, server logs (probe tick + emitted edge events), `correx replay __system__`. + +## Out of scope (NOT covered this pass) + +- LLAMA_SERVER probe (separate QA plan, `docs/qa/QA-llama-health-probe.md`). +- Live health TUI pane (separate BACKLOG item, §A). +- True fsync-latency instrumentation of the append path — this probe is a read-responsiveness heartbeat + by design (writing probe events to time appends would pollute the log); deeper append/fsync metrics, if + ever wanted, belong in MetricsProjection over real append events, not here. + +## Disposition + +- **PASS** → MOVE the §A "EVENT_STORE health probe" entry into `RETRO.md` (cite `7a1362c`, run date, evidence). +- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing + signal), fix, re-run only the failed checks. Set Status: FAILED until green.