kami
0957f7c69e
Flip §A LLAMA_SERVER probe to pending-live-QA (aaf896d), register the QA
gate in §F, and add docs/qa/QA-llama-health-probe.md with observable-evidence
checks (liveness up/down/restore, non-2xx, tps degrade, restart hysteresis,
replay independence).
56 lines
4.0 KiB
Markdown
56 lines
4.0 KiB
Markdown
# QA Plan: LLAMA_SERVER health probe — aaf896d
|
||
|
||
Observability spec §4. Drafted from the diff (commit `aaf896d`) per the BACKLOG QA rule.
|
||
|
||
**Status:** DRAFT
|
||
**Run date / operator:**
|
||
**BACKLOG item:** §A "§4 LLAMA_SERVER health probe — liveness + tokens/sec degradation trend".
|
||
|
||
---
|
||
|
||
## Preconditions
|
||
|
||
- [ ] **server build/branch:** master @ `aaf896d` (or later) — _rebuild only with the server stopped_
|
||
- [ ] **llama-server + model:** a real llama.cpp server reachable at the configured base URL
|
||
(managed `[[models]]` path → `host:port`; static path → first `[[providers]].url` or `CORREX_LLAMA_URL`)
|
||
- [ ] **config:** `[health].enabled = true`; for the degradation check set `[health].llama_tps_warn_below`
|
||
(default `0.0` = disabled). New fields: `llama_liveness_timeout_ms` (default 5000), `llama_tps_warn_below` (default 0.0).
|
||
- [ ] **interval:** `[health].interval_ms` (default 30_000) — note it so you know how long a status edge takes to surface.
|
||
- [ ] **how to read state:** `correx health` CLI / `GET /health/checks`; raw events via `correx events __system__`
|
||
(health events ride the reserved `SessionId("__system__")`).
|
||
|
||
## Acceptance gate (one sentence)
|
||
|
||
> The probe is correct **iff** a reachable llama-server reads HEALTHY, an unreachable/erroring one
|
||
> emits exactly one `HealthDegradedEvent(subject=LLAMA_SERVER)` (visible in `correx health` and the
|
||
> `__system__` log), recovery emits exactly one `HealthRestoredEvent`, and replay re-derives all of
|
||
> that from recorded events without touching the network.
|
||
|
||
## Checks
|
||
|
||
| # | Action | Expected observable evidence | Result |
|
||
|---|--------|------------------------------|--------|
|
||
| 1 | Server + llama up, `health.enabled=true`, wait one interval | `correx health` shows LLAMA_SERVER HEALTHY; **no** `HealthDegradedEvent(LLAMA_SERVER)` in `correx events __system__`; probe detail "llama-server live at <url>" | |
|
||
| 2 | Stop llama-server (or point base URL at a dead port), wait one interval | Exactly one `HealthDegradedEvent` with `subject=LLAMA_SERVER`, metric `liveness`, detail "GET <url>/health failed: …"; `correx health` flips to DEGRADED | |
|
||
| 3 | Restart llama-server, wait one interval | Exactly one `HealthRestoredEvent(LLAMA_SERVER)`; `correx health` back to HEALTHY. **No** repeated Degraded/Restored on subsequent ticks (hysteresis = edge-only) | |
|
||
| 4 | Point base URL at an endpoint returning non-2xx for `/health` | `HealthDegradedEvent` metric `liveness`, detail "returned HTTP <code>", `observedValue` = the status code | |
|
||
| 5 | Set `llama_tps_warn_below` above current throughput, run one inference, wait a tick | `HealthDegradedEvent` metric `tokens_per_second`; `observedValue` = avg tps ×100 (fixed-point); detail "avg tokens/sec X below warn threshold Y (window=N)" | |
|
||
| 6 | Set `llama_tps_warn_below` below current throughput (or default 0.0), run inferences | **No** `tokens_per_second` DEGRADED; at 0.0 the dimension is fully disabled regardless of throughput | |
|
||
| 7 | Restart the **correx** server while llama is HEALTHY | **No** spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection) | |
|
||
| 8 | Kill llama-server, then `correx replay __system__` | Replay re-derives the same health timeline from recorded `HealthObservation` events; double-read digest stable; **no** network call to llama (invariants #8/#9) | |
|
||
|
||
Evidence sources: `correx events __system__` (the recorded truth), `correx health` / `GET /health/checks`,
|
||
server logs (look for the probe tick + emitted edge events), `correx replay __system__` for determinism.
|
||
|
||
## Out of scope (NOT covered this pass)
|
||
|
||
- EVENT_STORE health probe (separate BACKLOG item, §A).
|
||
- Live health TUI pane (separate BACKLOG item, §A).
|
||
- Prometheus/Grafana export (spec: do not build).
|
||
|
||
## Disposition
|
||
|
||
- **PASS** → MOVE the §A "LLAMA_SERVER health probe" entry into `RETRO.md` (cite `aaf896d`, run date, evidence).
|
||
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + exact repro + the
|
||
wrong/missing signal), fix, re-run only the failed checks. Set Status: FAILED until green.
|