correx/docs/qa/QA-llama-health-probe.md

QA Plan: LLAMA_SERVER health probe — aaf896d

Observability spec §4. Drafted from the diff (commit aaf896d) per the BACKLOG QA rule.

Status: DRAFT Run date / operator: BACKLOG item: §A "§4 LLAMA_SERVER health probe — liveness + tokens/sec degradation trend".

---

Preconditions

• server build/branch: master @ aaf896d (or later) — rebuild only with the server stopped
• llama-server + model: a real llama.cpp server reachable at the configured base URL (managed [[models]] path → host:port; static path → first [[providers]].url or CORREX_LLAMA_URL)
• config: [health].enabled = true; for the degradation check set [health].llama_tps_warn_below (default 0.0 = disabled). New fields: llama_liveness_timeout_ms (default 5000), llama_tps_warn_below (default 0.0).
• interval: [health].interval_ms (default 30_000) — note it so you know how long a status edge takes to surface.
• how to read state: correx health CLI / GET /health/checks; raw events via correx events __system__ (health events ride the reserved SessionId("__system__")).

Acceptance gate (one sentence)

The probe is correct iff a reachable llama-server reads HEALTHY, an unreachable/erroring one emits exactly one HealthDegradedEvent(subject=LLAMA_SERVER) (visible in correx health and the __system__ log), recovery emits exactly one HealthRestoredEvent, and replay re-derives all of that from recorded events without touching the network.

Checks

#	Action	Expected observable evidence	Result
1	Server + llama up, health.enabled=true, wait one interval	correx health shows LLAMA_SERVER HEALTHY; no HealthDegradedEvent(LLAMA_SERVER) in correx events __system__; probe detail "llama-server live at "
2	Stop llama-server (or point base URL at a dead port), wait one interval	Exactly one HealthDegradedEvent with subject=LLAMA_SERVER, metric liveness, detail "GET /health failed: …"; correx health flips to DEGRADED
3	Restart llama-server, wait one interval	Exactly one HealthRestoredEvent(LLAMA_SERVER); correx health back to HEALTHY. No repeated Degraded/Restored on subsequent ticks (hysteresis = edge-only)
4	Point base URL at an endpoint returning non-2xx for /health	HealthDegradedEvent metric liveness, detail "returned HTTP ", observedValue = the status code
5	Set llama_tps_warn_below above current throughput, run one inference, wait a tick	HealthDegradedEvent metric tokens_per_second; observedValue = avg tps ×100 (fixed-point); detail "avg tokens/sec X below warn threshold Y (window=N)"
6	Set llama_tps_warn_below below current throughput (or default 0.0), run inferences	No tokens_per_second DEGRADED; at 0.0 the dimension is fully disabled regardless of throughput
7	Restart the correx server while llama is HEALTHY	No spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection)
8	Kill llama-server, then correx replay __system__	Replay re-derives the same health timeline from recorded HealthObservation events; double-read digest stable; no network call to llama (invariants #8/#9)

Evidence sources: correx events __system__ (the recorded truth), correx health / GET /health/checks, server logs (look for the probe tick + emitted edge events), correx replay __system__ for determinism.

Out of scope (NOT covered this pass)

• EVENT_STORE health probe (separate BACKLOG item, §A).
• Live health TUI pane (separate BACKLOG item, §A).
• Prometheus/Grafana export (spec: do not build).

Disposition

• PASS → MOVE the §A "LLAMA_SERVER health probe" entry into RETRO.md (cite aaf896d, run date, evidence).
• FAIL → file each failure as a numbered finding back into BACKLOG.md (action + exact repro + the wrong/missing signal), fix, re-run only the failed checks. Set Status: FAILED until green.