docs(backlog): mark event-store health probe shipped, add QA plan
Flip §A EVENT_STORE probe to pending-live-QA (7a1362c), register the QA gate
in §F, and add docs/qa/QA-event-store-health-probe.md (responsive/HEALTHY,
no-log-pollution, slow/throw DEGRADED, restore hysteresis, replay independence).
This commit is contained in:
+6
-1
@@ -30,7 +30,10 @@ f8fd260) — those are RETRO candidates once live-QA'd. Remaining:
|
|||||||
`GET /health` + tokens/sec degradation watch (windowed avg of recorded `InferenceCompletedEvent`s,
|
`GET /health` + tokens/sec degradation watch (windowed avg of recorded `InferenceCompletedEvent`s,
|
||||||
opt-in via `llama_tps_warn_below`). Plugs into the existing `HealthMonitor`. QA gate in §F →
|
opt-in via `llama_tps_warn_below`). Plugs into the existing `HealthMonitor`. QA gate in §F →
|
||||||
`docs/qa/QA-llama-health-probe.md`.
|
`docs/qa/QA-llama-health-probe.md`.
|
||||||
- [ ] **§4 EVENT_STORE health probe** — append + fsync latency heartbeat.
|
- [~] **§4 EVENT_STORE health probe** — **SHIPPED (pending live-QA)** `7a1362c`. Times a cheap
|
||||||
|
`lastGlobalSequence()` read as a responsiveness heartbeat (DEGRADED on throw or over
|
||||||
|
`event_store_latency_warn_ms`); read-only by design (no probe writes into the log). QA gate in §F →
|
||||||
|
`docs/qa/QA-event-store-health-probe.md`.
|
||||||
- [ ] **§4/§3 live health TUI pane** — deferred to follow the metrics cadence
|
- [ ] **§4/§3 live health TUI pane** — deferred to follow the metrics cadence
|
||||||
(projection + CLI first, then pane). `correx health` CLI already exists.
|
(projection + CLI first, then pane). `correx health` CLI already exists.
|
||||||
- [ ] **§3 tier-2 stats** — full session-completion summary pane polish (basic pane shipped).
|
- [ ] **§3 tier-2 stats** — full session-completion summary pane polish (basic pane shipped).
|
||||||
@@ -125,6 +128,8 @@ These are SHIPPED in code but prompt-/server-/network-dependent and not yet live
|
|||||||
- [ ] **LLAMA_SERVER health probe live-QA** — `aaf896d`; needs a real llama-server to ping.
|
- [ ] **LLAMA_SERVER health probe live-QA** — `aaf896d`; needs a real llama-server to ping.
|
||||||
Plan: `docs/qa/QA-llama-health-probe.md` (liveness up/down/restore, non-2xx, tps degrade,
|
Plan: `docs/qa/QA-llama-health-probe.md` (liveness up/down/restore, non-2xx, tps degrade,
|
||||||
restart hysteresis, replay independence).
|
restart hysteresis, replay independence).
|
||||||
|
- [ ] **EVENT_STORE health probe live-QA** — `7a1362c`. Plan: `docs/qa/QA-event-store-health-probe.md`
|
||||||
|
(responsive→HEALTHY, no-log-pollution, slow/throw→DEGRADED, restore hysteresis, replay independence).
|
||||||
- [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the
|
- [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the
|
||||||
*same* stage's output; AMD VRAM/RAM gauge on the real box.
|
*same* stage's output; AMD VRAM/RAM gauge on the real box.
|
||||||
- [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection),
|
- [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection),
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# QA Plan: EVENT_STORE health probe — 7a1362c
|
||||||
|
|
||||||
|
Observability spec §4. Drafted from the diff (commit `7a1362c`) per the BACKLOG QA rule.
|
||||||
|
|
||||||
|
**Status:** DRAFT
|
||||||
|
**Run date / operator:**
|
||||||
|
**BACKLOG item:** §A "§4 EVENT_STORE health probe — append + fsync latency heartbeat".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Preconditions
|
||||||
|
|
||||||
|
- [ ] **server build/branch:** master @ `7a1362c` (or later) — _rebuild only with the server stopped_
|
||||||
|
- [ ] **config:** `[health].enabled = true`; new field `[health].event_store_latency_warn_ms` (default 500).
|
||||||
|
- [ ] **interval:** `[health].interval_ms` (default 30_000) — how long a status edge takes to surface.
|
||||||
|
- [ ] **how to read state:** `correx health` / `GET /health/checks`; raw events via `correx events __system__`.
|
||||||
|
- [ ] **(for the slow-path check)** a way to make a real SQLite read slow — e.g. point the event store at a
|
||||||
|
path on a slow/contended/network filesystem, or temporarily lower `event_store_latency_warn_ms` to a
|
||||||
|
value below normal read latency to force a DEGRADED.
|
||||||
|
|
||||||
|
## Acceptance gate (one sentence)
|
||||||
|
|
||||||
|
> The probe is correct **iff** a responsive store reads HEALTHY (latency recorded), an unresponsive or
|
||||||
|
> slow store emits exactly one `HealthDegradedEvent(subject=EVENT_STORE)` (visible in `correx health` and
|
||||||
|
> the `__system__` log), recovery emits exactly one `HealthRestoredEvent`, the probe NEVER writes to the
|
||||||
|
> log itself, and replay re-derives all of it from recorded events.
|
||||||
|
|
||||||
|
## Checks
|
||||||
|
|
||||||
|
| # | Action | Expected observable evidence | Result |
|
||||||
|
|---|--------|------------------------------|--------|
|
||||||
|
| 1 | Server up, `health.enabled=true`, wait one interval | `correx health` shows EVENT_STORE HEALTHY; detail "event store responsive: lastGlobalSequence() in Nms"; `observedValue` = a small latency | |
|
||||||
|
| 2 | **No log pollution:** note `correx events __system__` event count, let several health intervals pass | The `__system__` event count does NOT grow per tick — only on actual status *edges*. The probe must not append a heartbeat event every tick (Invariant #1) | |
|
||||||
|
| 3 | Force slow reads (slow FS, or set `event_store_latency_warn_ms` below normal latency), wait one interval | Exactly one `HealthDegradedEvent` subject=EVENT_STORE, metric `read_latency_ms`, detail "lastGlobalSequence() took Nms (warn ≥ …ms)" | |
|
||||||
|
| 4 | Restore normal latency (or raise the threshold back), wait one interval | Exactly one `HealthRestoredEvent(EVENT_STORE)`; back to HEALTHY. No repeated Degraded/Restored on later ticks (hysteresis = edge-only) | |
|
||||||
|
| 5 | Simulate store failure (e.g. make the DB unreadable mid-run, if feasible) | `HealthDegradedEvent` with detail "lastGlobalSequence() threw: …"; the monitor tick survives (no crash, next tick still runs) | |
|
||||||
|
| 6 | Restart the **correx** server while the store is HEALTHY | No spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection) | |
|
||||||
|
| 7 | `correx replay __system__` | Replay re-derives the same health timeline from recorded `HealthObservation` events; double-read digest stable; no live store-latency re-probe (invariants #8/#9) | |
|
||||||
|
|
||||||
|
Evidence sources: `correx events __system__` (recorded truth + the no-growth check), `correx health` /
|
||||||
|
`GET /health/checks`, server logs (probe tick + emitted edge events), `correx replay __system__`.
|
||||||
|
|
||||||
|
## Out of scope (NOT covered this pass)
|
||||||
|
|
||||||
|
- LLAMA_SERVER probe (separate QA plan, `docs/qa/QA-llama-health-probe.md`).
|
||||||
|
- Live health TUI pane (separate BACKLOG item, §A).
|
||||||
|
- True fsync-latency instrumentation of the append path — this probe is a read-responsiveness heartbeat
|
||||||
|
by design (writing probe events to time appends would pollute the log); deeper append/fsync metrics, if
|
||||||
|
ever wanted, belong in MetricsProjection over real append events, not here.
|
||||||
|
|
||||||
|
## Disposition
|
||||||
|
|
||||||
|
- **PASS** → MOVE the §A "EVENT_STORE health probe" entry into `RETRO.md` (cite `7a1362c`, run date, evidence).
|
||||||
|
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing
|
||||||
|
signal), fix, re-run only the failed checks. Set Status: FAILED until green.
|
||||||
Reference in New Issue
Block a user