correx/docs/qa/QA-event-store-health-probe.md

QA Plan: EVENT_STORE health probe — 7a1362c

Observability spec §4. Drafted from the diff (commit 7a1362c) per the BACKLOG QA rule.

Status: DRAFT Run date / operator: BACKLOG item: §A "§4 EVENT_STORE health probe — append + fsync latency heartbeat".

---

Preconditions

• server build/branch: master @ 7a1362c (or later) — rebuild only with the server stopped
• config: [health].enabled = true; new field [health].event_store_latency_warn_ms (default 500).
• interval: [health].interval_ms (default 30_000) — how long a status edge takes to surface.
• how to read state: correx health / GET /health/checks; raw events via correx events __system__.
• (for the slow-path check) a way to make a real SQLite read slow — e.g. point the event store at a path on a slow/contended/network filesystem, or temporarily lower event_store_latency_warn_ms to a value below normal read latency to force a DEGRADED.

Acceptance gate (one sentence)

The probe is correct iff a responsive store reads HEALTHY (latency recorded), an unresponsive or slow store emits exactly one HealthDegradedEvent(subject=EVENT_STORE) (visible in correx health and the __system__ log), recovery emits exactly one HealthRestoredEvent, the probe NEVER writes to the log itself, and replay re-derives all of it from recorded events.

Checks

#	Action	Expected observable evidence	Result
1	Server up, health.enabled=true, wait one interval	correx health shows EVENT_STORE HEALTHY; detail "event store responsive: lastGlobalSequence() in Nms"; observedValue = a small latency
2	No log pollution: note correx events __system__ event count, let several health intervals pass	The __system__ event count does NOT grow per tick — only on actual status edges. The probe must not append a heartbeat event every tick (Invariant #1)
3	Force slow reads (slow FS, or set event_store_latency_warn_ms below normal latency), wait one interval	Exactly one HealthDegradedEvent subject=EVENT_STORE, metric read_latency_ms, detail "lastGlobalSequence() took Nms (warn ≥ …ms)"
4	Restore normal latency (or raise the threshold back), wait one interval	Exactly one HealthRestoredEvent(EVENT_STORE); back to HEALTHY. No repeated Degraded/Restored on later ticks (hysteresis = edge-only)
5	Simulate store failure (e.g. make the DB unreadable mid-run, if feasible)	HealthDegradedEvent with detail "lastGlobalSequence() threw: …"; the monitor tick survives (no crash, next tick still runs)
6	Restart the correx server while the store is HEALTHY	No spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection)
7	correx replay __system__	Replay re-derives the same health timeline from recorded HealthObservation events; double-read digest stable; no live store-latency re-probe (invariants #8/#9)

Evidence sources: correx events __system__ (recorded truth + the no-growth check), correx health / GET /health/checks, server logs (probe tick + emitted edge events), correx replay __system__.

Out of scope (NOT covered this pass)

• LLAMA_SERVER probe (separate QA plan, docs/qa/QA-llama-health-probe.md).
• Live health TUI pane (separate BACKLOG item, §A).
• True fsync-latency instrumentation of the append path — this probe is a read-responsiveness heartbeat by design (writing probe events to time appends would pollute the log); deeper append/fsync metrics, if ever wanted, belong in MetricsProjection over real append events, not here.

Disposition

• PASS → MOVE the §A "EVENT_STORE health probe" entry into RETRO.md (cite 7a1362c, run date, evidence).
• FAIL → file each failure as a numbered finding back into BACKLOG.md (action + repro + wrong/missing signal), fix, re-run only the failed checks. Set Status: FAILED until green.