docs(backlog): mark llama health probe shipped, add QA plan
Flip §A LLAMA_SERVER probe to pending-live-QA (aaf896d), register the QA
gate in §F, and add docs/qa/QA-llama-health-probe.md with observable-evidence
checks (liveness up/down/restore, non-2xx, tps degrade, restart hysteresis,
replay independence).
This commit is contained in:
+7
-2
@@ -26,8 +26,10 @@ against current code before acting (memory is point-in-time).
|
||||
Metrics + CLI + TUI pane and the health backbone are shipped (fe94140, 4f6bfa8,
|
||||
f8fd260) — those are RETRO candidates once live-QA'd. Remaining:
|
||||
|
||||
- [ ] **§4 LLAMA_SERVER health probe** — liveness + tokens/sec degradation trend
|
||||
(session-scoped, needs HTTP). Same `HealthProbe` framework as the shipped disk probe.
|
||||
- [~] **§4 LLAMA_SERVER health probe** — **SHIPPED (pending live-QA)** `aaf896d`. Liveness via
|
||||
`GET /health` + tokens/sec degradation watch (windowed avg of recorded `InferenceCompletedEvent`s,
|
||||
opt-in via `llama_tps_warn_below`). Plugs into the existing `HealthMonitor`. QA gate in §F →
|
||||
`docs/qa/QA-llama-health-probe.md`.
|
||||
- [ ] **§4 EVENT_STORE health probe** — append + fsync latency heartbeat.
|
||||
- [ ] **§4/§3 live health TUI pane** — deferred to follow the metrics cadence
|
||||
(projection + CLI first, then pane). `correx health` CLI already exists.
|
||||
@@ -120,6 +122,9 @@ These are SHIPPED in code but prompt-/server-/network-dependent and not yet live
|
||||
- [ ] **Clarification loop + workflow-propose panel live-QA** — both prompt-dependent (need a
|
||||
real router/analyst model to emit the structured json blocks).
|
||||
- [ ] **Research workflow live-QA** — see §D.
|
||||
- [ ] **LLAMA_SERVER health probe live-QA** — `aaf896d`; needs a real llama-server to ping.
|
||||
Plan: `docs/qa/QA-llama-health-probe.md` (liveness up/down/restore, non-2xx, tps degrade,
|
||||
restart hysteresis, replay independence).
|
||||
- [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the
|
||||
*same* stage's output; AMD VRAM/RAM gauge on the real box.
|
||||
- [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection),
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
# QA Plan: LLAMA_SERVER health probe — aaf896d
|
||||
|
||||
Observability spec §4. Drafted from the diff (commit `aaf896d`) per the BACKLOG QA rule.
|
||||
|
||||
**Status:** DRAFT
|
||||
**Run date / operator:**
|
||||
**BACKLOG item:** §A "§4 LLAMA_SERVER health probe — liveness + tokens/sec degradation trend".
|
||||
|
||||
---
|
||||
|
||||
## Preconditions
|
||||
|
||||
- [ ] **server build/branch:** master @ `aaf896d` (or later) — _rebuild only with the server stopped_
|
||||
- [ ] **llama-server + model:** a real llama.cpp server reachable at the configured base URL
|
||||
(managed `[[models]]` path → `host:port`; static path → first `[[providers]].url` or `CORREX_LLAMA_URL`)
|
||||
- [ ] **config:** `[health].enabled = true`; for the degradation check set `[health].llama_tps_warn_below`
|
||||
(default `0.0` = disabled). New fields: `llama_liveness_timeout_ms` (default 5000), `llama_tps_warn_below` (default 0.0).
|
||||
- [ ] **interval:** `[health].interval_ms` (default 30_000) — note it so you know how long a status edge takes to surface.
|
||||
- [ ] **how to read state:** `correx health` CLI / `GET /health/checks`; raw events via `correx events __system__`
|
||||
(health events ride the reserved `SessionId("__system__")`).
|
||||
|
||||
## Acceptance gate (one sentence)
|
||||
|
||||
> The probe is correct **iff** a reachable llama-server reads HEALTHY, an unreachable/erroring one
|
||||
> emits exactly one `HealthDegradedEvent(subject=LLAMA_SERVER)` (visible in `correx health` and the
|
||||
> `__system__` log), recovery emits exactly one `HealthRestoredEvent`, and replay re-derives all of
|
||||
> that from recorded events without touching the network.
|
||||
|
||||
## Checks
|
||||
|
||||
| # | Action | Expected observable evidence | Result |
|
||||
|---|--------|------------------------------|--------|
|
||||
| 1 | Server + llama up, `health.enabled=true`, wait one interval | `correx health` shows LLAMA_SERVER HEALTHY; **no** `HealthDegradedEvent(LLAMA_SERVER)` in `correx events __system__`; probe detail "llama-server live at <url>" | |
|
||||
| 2 | Stop llama-server (or point base URL at a dead port), wait one interval | Exactly one `HealthDegradedEvent` with `subject=LLAMA_SERVER`, metric `liveness`, detail "GET <url>/health failed: …"; `correx health` flips to DEGRADED | |
|
||||
| 3 | Restart llama-server, wait one interval | Exactly one `HealthRestoredEvent(LLAMA_SERVER)`; `correx health` back to HEALTHY. **No** repeated Degraded/Restored on subsequent ticks (hysteresis = edge-only) | |
|
||||
| 4 | Point base URL at an endpoint returning non-2xx for `/health` | `HealthDegradedEvent` metric `liveness`, detail "returned HTTP <code>", `observedValue` = the status code | |
|
||||
| 5 | Set `llama_tps_warn_below` above current throughput, run one inference, wait a tick | `HealthDegradedEvent` metric `tokens_per_second`; `observedValue` = avg tps ×100 (fixed-point); detail "avg tokens/sec X below warn threshold Y (window=N)" | |
|
||||
| 6 | Set `llama_tps_warn_below` below current throughput (or default 0.0), run inferences | **No** `tokens_per_second` DEGRADED; at 0.0 the dimension is fully disabled regardless of throughput | |
|
||||
| 7 | Restart the **correx** server while llama is HEALTHY | **No** spurious Degraded/Restored re-emission on boot (HealthMonitor seeds status from the projection) | |
|
||||
| 8 | Kill llama-server, then `correx replay __system__` | Replay re-derives the same health timeline from recorded `HealthObservation` events; double-read digest stable; **no** network call to llama (invariants #8/#9) | |
|
||||
|
||||
Evidence sources: `correx events __system__` (the recorded truth), `correx health` / `GET /health/checks`,
|
||||
server logs (look for the probe tick + emitted edge events), `correx replay __system__` for determinism.
|
||||
|
||||
## Out of scope (NOT covered this pass)
|
||||
|
||||
- EVENT_STORE health probe (separate BACKLOG item, §A).
|
||||
- Live health TUI pane (separate BACKLOG item, §A).
|
||||
- Prometheus/Grafana export (spec: do not build).
|
||||
|
||||
## Disposition
|
||||
|
||||
- **PASS** → MOVE the §A "LLAMA_SERVER health probe" entry into `RETRO.md` (cite `aaf896d`, run date, evidence).
|
||||
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + exact repro + the
|
||||
wrong/missing signal), fix, re-run only the failed checks. Set Status: FAILED until green.
|
||||
Reference in New Issue
Block a user