# QA Plan: LLAMA_SERVER health probe — aaf896d

Observability spec §4. Drafted from the diff (commit `aaf896d`) per the BACKLOG QA rule.

**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §A "§4 LLAMA_SERVER health probe — liveness + tokens/sec degradation trend".

---

## Preconditions

- [ ] **server build/branch:** master @ `aaf896d` (or later) — _rebuild only with the server stopped_
- [ ] **llama-server + model:** a real llama.cpp server reachable at the configured base URL
      (managed `[[models]]` path → `host:port`; static path → first `[[providers]].url` or `CORREX_LLAMA_URL`)
- [ ] **config:** `[health].enabled = true`; for the degradation check set `[health].llama_tps_warn_below`
      (default `0.0` = disabled). New fields: `llama_liveness_timeout_ms` (default 5000), `llama_tps_warn_below` (default 0.0).
- [ ] **interval:** `[health].interval_ms` (default 30_000) — note it so you know how long a status edge takes to surface.
- [ ] **how to read state:** `correx health` CLI / `GET /health/checks`; raw events via `correx events __system__`
      (health events ride the reserved `SessionId("__system__")`).

## Acceptance gate (one sentence)

> The probe is correct **iff** a reachable llama-server reads HEALTHY, an unreachable/erroring one
> emits exactly one `HealthDegradedEvent(subject=LLAMA_SERVER)` (visible in `correx health` and the
> `__system__` log), recovery emits exactly one `HealthRestoredEvent`, and replay re-derives all of
> that from recorded events without touching the network.

## Checks

| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Server + llama up, `health.enabled=true`, wait one interval | `correx health` shows LLAMA_SERVER HEALTHY; **no** `HealthDegradedEvent(LLAMA_SERVER)` in `correx events __system__`; probe detail "llama-server live at <url>" | |
| 2 | Stop llama-server (or point base URL at a dead port), wait one interval | Exactly one `HealthDegradedEvent` with `subject=LLAMA_SERVER`, metric `liveness`, detail "GET <url>/health failed: …"; `correx health` flips to DEGRADED | |
| 3 | Restart llama-server, wait one interval | Exactly one `HealthRestoredEvent(LLAMA_SERVER)`; `correx health` back to HEALTHY. **No** repeated Degraded/Restored on subsequent ticks (hysteresis = edge-only) | |
| 4 | Point base URL at an endpoint returning non-2xx for `/health` | `HealthDegradedEvent` metric `liveness`, detail "returned HTTP <code>", `observedValue` = the status code | |
| 5 | Set `llama_tps_warn_below` above current throughput, run one inference, wait a tick | `HealthDegradedEvent` metric `tokens_per_second`; `observedValue` = avg tps ×100 (fixed-point); detail "avg tokens/sec X below warn threshold Y (window=N)" | |
| 6 | Set `llama_tps_warn_below` below current throughput (or default 0.0), run inferences | **No** `tokens_per_second` DEGRADED; at 0.0 the dimension is fully disabled regardless of throughput | |
| 7 | Restart the **correx** server while llama is HEALTHY | **No** spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection) | |
| 8 | Kill llama-server, then `correx replay __system__` | Replay re-derives the same health timeline from recorded `HealthObservation` events; double-read digest stable; **no** network call to llama (invariants #8/#9) | |

Evidence sources: `correx events __system__` (the recorded truth), `correx health` / `GET /health/checks`,
server logs (look for the probe tick + emitted edge events), `correx replay __system__` for determinism.

## Out of scope (NOT covered this pass)

- EVENT_STORE health probe (separate BACKLOG item, §A).
- Live health TUI pane (separate BACKLOG item, §A).
- Prometheus/Grafana export (spec: do not build).

## Disposition

- **PASS** → MOVE the §A "LLAMA_SERVER health probe" entry into `RETRO.md` (cite `aaf896d`, run date, evidence).
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + exact repro + the
  wrong/missing signal), fix, re-run only the failed checks. Set Status: FAILED until green.