kami
12ff2e97b0
Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks: research egress + batch source-approval (D), architect contradiction (B§4), llama health probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in production (it breaks the planner if the model can't emit a parseable brief_echo). Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/ (sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are set -euo pipefail, syntax-checked, executable. Authored from the repo build/config (server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
42 lines
3.4 KiB
Markdown
42 lines
3.4 KiB
Markdown
# QA Plan: LLAMA_SERVER health probe — `266bbf0` `64a90e7` `9eef936`
|
|
|
|
**Status:** DRAFT
|
|
**Run date / operator:**
|
|
**BACKLOG item:** §A-§4 "LLAMA_SERVER health probe" — probe `266bbf0`, registration `64a90e7`, managed-path coverage `9eef936`.
|
|
|
|
---
|
|
|
|
## Preconditions
|
|
|
|
- [ ] **server build/branch:** `feat/backlog-burndown`.
|
|
- [ ] **llama-server + model:** a real llama-server reachable at the configured URL. The probe registers when **either** a static `[[providers]]` (type `llamacpp`, `url`) **or** a managed `[[models]]` entry exists (managed URL = `[models].host:[models].port`). Verify your config has one.
|
|
- [ ] **external deps:** none.
|
|
- [ ] **config synced:** `[health] enabled = true`; note `[health] interval_ms` (the probe cadence — you wait one interval to see an edge).
|
|
- [ ] **fixtures/seed:** ability to stop/start the llama-server process (managed: it dies with the server; for the kill/restore test use the **static `[[providers]]`** path so you can stop llama-server independently of correx).
|
|
|
|
## Acceptance gate (one sentence)
|
|
|
|
> The `LLAMA_SERVER` subject is monitored only when a model is configured, reports HEALTHY while the server answers `/health`, and emits a **liveness DEGRADED** edge when the server stops — and a restored edge when it returns.
|
|
|
|
## Checks
|
|
|
|
| # | Action | Expected observable evidence | Result |
|
|
|---|--------|------------------------------|--------|
|
|
| 1 | Start correx with a llama-server configured and reachable. Wait one `interval_ms`. | `correx health` (and the health projection) lists `LLAMA_SERVER` as HEALTHY. No spurious DEGRADED. | |
|
|
| 2 | **Stop** the llama-server (static-provider path: kill the external process). Wait one `interval_ms`. | A health **degraded** edge event for subject `LLAMA_SERVER`, `metric = "liveness"`, detail "llama server unreachable" appears in the system-health log (`correx events` on the SYSTEM session / `correx health`). | |
|
|
| 3 | **Restart** the llama-server. Wait one `interval_ms`. | A **restored** edge event for `LLAMA_SERVER`; `correx health` shows HEALTHY again. Edges are recorded once (edge-triggered, not per-tick). | |
|
|
| 4 | Start correx on a box with **no** `[[providers]]` and **no** `[[models]]`. | `LLAMA_SERVER` is **absent** from the health subjects — the probe is not registered (no spurious DEGRADED). Absence is the evidence. | |
|
|
| 5 | `correx replay` the SYSTEM health session. | The degraded/restored edges replay deterministically from the log (the rolling window is live-only per Invariant #8 and does not affect replay). | |
|
|
|
|
Evidence sources: `correx health` CLI, the health projection events, server logs.
|
|
|
|
## Out of scope (explicitly NOT covered this pass)
|
|
|
|
- **tokens/sec degradation** — the `/health` endpoint is reachability-only; `HttpLlamaLivenessClient` never populates `tokensPerSecond`, so the throughput-collapse branch of `LlamaServerHealthProbe` is dormant until a tps telemetry feed exists. Liveness only this pass. (Refiled in BACKLOG §A-§4 remainder.)
|
|
- The live health **TUI pane** (deferred; system-health events are not yet on the TUI wire).
|
|
|
|
## Disposition
|
|
|
|
- **PASS** → move the §A-§4 LLAMA_SERVER bullet to `RETRO.md` (liveness verified) with this run date; keep the tokens/sec remainder open in BACKLOG. Status: PASSED.
|
|
- **FAIL** → numbered findings. A likely miss: the managed `[[models]]` path not registering — confirm `9eef936` is in the build. Status: FAILED until green.
|