docs(backlog): mark event-store health probe shipped, add QA plan
Flip §A EVENT_STORE probe to pending-live-QA (7a1362c), register the QA gate
in §F, and add docs/qa/QA-event-store-health-probe.md (responsive/HEALTHY,
no-log-pollution, slow/throw DEGRADED, restore hysteresis, replay independence).
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# QA Plan: EVENT_STORE health probe — 7a1362c
|
||||
|
||||
Observability spec §4. Drafted from the diff (commit `7a1362c`) per the BACKLOG QA rule.
|
||||
|
||||
**Status:** DRAFT
|
||||
**Run date / operator:**
|
||||
**BACKLOG item:** §A "§4 EVENT_STORE health probe — append + fsync latency heartbeat".
|
||||
|
||||
---
|
||||
|
||||
## Preconditions
|
||||
|
||||
- [ ] **server build/branch:** master @ `7a1362c` (or later) — _rebuild only with the server stopped_
|
||||
- [ ] **config:** `[health].enabled = true`; new field `[health].event_store_latency_warn_ms` (default 500).
|
||||
- [ ] **interval:** `[health].interval_ms` (default 30_000) — how long a status edge takes to surface.
|
||||
- [ ] **how to read state:** `correx health` / `GET /health/checks`; raw events via `correx events __system__`.
|
||||
- [ ] **(for the slow-path check)** a way to make a real SQLite read slow — e.g. point the event store at a
|
||||
path on a slow/contended/network filesystem, or temporarily lower `event_store_latency_warn_ms` to a
|
||||
value below normal read latency to force a DEGRADED.
|
||||
|
||||
## Acceptance gate (one sentence)
|
||||
|
||||
> The probe is correct **iff** a responsive store reads HEALTHY (latency recorded), an unresponsive or
|
||||
> slow store emits exactly one `HealthDegradedEvent(subject=EVENT_STORE)` (visible in `correx health` and
|
||||
> the `__system__` log), recovery emits exactly one `HealthRestoredEvent`, the probe NEVER writes to the
|
||||
> log itself, and replay re-derives all of it from recorded events.
|
||||
|
||||
## Checks
|
||||
|
||||
| # | Action | Expected observable evidence | Result |
|
||||
|---|--------|------------------------------|--------|
|
||||
| 1 | Server up, `health.enabled=true`, wait one interval | `correx health` shows EVENT_STORE HEALTHY; detail "event store responsive: lastGlobalSequence() in Nms"; `observedValue` = a small latency | |
|
||||
| 2 | **No log pollution:** note `correx events __system__` event count, let several health intervals pass | The `__system__` event count does NOT grow per tick — only on actual status *edges*. The probe must not append a heartbeat event every tick (Invariant #1) | |
|
||||
| 3 | Force slow reads (slow FS, or set `event_store_latency_warn_ms` below normal latency), wait one interval | Exactly one `HealthDegradedEvent` subject=EVENT_STORE, metric `read_latency_ms`, detail "lastGlobalSequence() took Nms (warn ≥ …ms)" | |
|
||||
| 4 | Restore normal latency (or raise the threshold back), wait one interval | Exactly one `HealthRestoredEvent(EVENT_STORE)`; back to HEALTHY. No repeated Degraded/Restored on later ticks (hysteresis = edge-only) | |
|
||||
| 5 | Simulate store failure (e.g. make the DB unreadable mid-run, if feasible) | `HealthDegradedEvent` with detail "lastGlobalSequence() threw: …"; the monitor tick survives (no crash, next tick still runs) | |
|
||||
| 6 | Restart the **correx** server while the store is HEALTHY | No spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection) | |
|
||||
| 7 | `correx replay __system__` | Replay re-derives the same health timeline from recorded `HealthObservation` events; double-read digest stable; no live store-latency re-probe (invariants #8/#9) | |
|
||||
|
||||
Evidence sources: `correx events __system__` (recorded truth + the no-growth check), `correx health` /
|
||||
`GET /health/checks`, server logs (probe tick + emitted edge events), `correx replay __system__`.
|
||||
|
||||
## Out of scope (NOT covered this pass)
|
||||
|
||||
- LLAMA_SERVER probe (separate QA plan, `docs/qa/QA-llama-health-probe.md`).
|
||||
- Live health TUI pane (separate BACKLOG item, §A).
|
||||
- True fsync-latency instrumentation of the append path — this probe is a read-responsiveness heartbeat
|
||||
by design (writing probe events to time appends would pollute the log); deeper append/fsync metrics, if
|
||||
ever wanted, belong in MetricsProjection over real append events, not here.
|
||||
|
||||
## Disposition
|
||||
|
||||
- **PASS** → MOVE the §A "EVENT_STORE health probe" entry into `RETRO.md` (cite `7a1362c`, run date, evidence).
|
||||
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing
|
||||
signal), fix, re-run only the failed checks. Set Status: FAILED until green.
|
||||
Reference in New Issue
Block a user