docs(qa): live-QA plans + env bring-up for the burndown features

Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks:
research egress + batch source-approval (D), architect contradiction (B§4), llama health
probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the
brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in
production (it breaks the planner if the model can't emit a parseable brief_echo).

Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/
(sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are
set -euo pipefail, syntax-checked, executable. Authored from the repo build/config
(server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 09:14:36 +00:00
parent 9eef936073
commit 12ff2e97b0
8 changed files with 424 additions and 0 deletions
+126
View File
@@ -0,0 +1,126 @@
# Correx live-QA environment — bring-up runbook
The plans in `docs/qa/QA-*.md` cite **observable signals** (events, logs, files, TUI
renders). This runbook stands up the stack that produces them. Run it on a capable box
(JDK 21, Go ≥ 1.24, Docker, a GPU + GGUF model for any model-dependent plan).
Helper scripts live in `scripts/qa/` (see `scripts/qa/README.md`).
> **Low-RAM note:** a machine-local `~/.gradle/gradle.properties` on the dev box caps the
> Gradle/Kotlin heap (`-Xmx768m`, `parallel=false`) for a 1.9 GB machine. It is **not
> committed**. On a QA box with real RAM, delete it (or ignore it) so builds aren't throttled.
---
## Prerequisites
- **JDK 21** (`java -version` → 21).
- **Go ≥ 1.24** — the TUI builds via `GOTOOLCHAIN=auto` (go.mod pins `go 1.24.2`); the
toolchain auto-downloads if your `go` is older.
- **Docker** — only for SearXNG (research-workflow QA).
- **llama-server** binary + at least one **GGUF model** — for any model-dependent plan.
- (optional) an **embedder** model/endpoint — required for the architect-contradiction plan
and any L3-retrieval-dependent behavior (`[router.embedder] backend = "llamacpp"`).
---
## 1. Config
Correx reads `~/.config/correx/config.toml` plus `workflows/`, `prompts/`, `schemas/`
alongside it. Stale copies cause silent bugs (BACKLOG ops note), so **sync from the repo**:
```bash
scripts/qa/sync-config.sh # copies examples/workflows + prompts + docs/schemas
# and seeds config.toml from docs/sample-config.toml IF ABSENT
```
Then edit `~/.config/correx/config.toml` for the plan you're running:
- **Model** — pick ONE:
- *Managed* (`[[models]]` + `[models]`): correx spawns llama-server at `[models].host:port`
(default `127.0.0.1:10000`) and kills it on shutdown. Simplest.
- *Static* (`[[providers]]`, type `llamacpp`, `url`): you run llama-server yourself. **Use
this for the llama-health plan** so you can stop the server independently of correx.
- **`[project] enabled = true`** — required for project memory + the **architect-contradiction**
plan (it queries the `project:<repoRoot>` L3 namespace written at session end).
- **`[router.embedder] backend = "llamacpp"`** (not `noop`) — required wherever embeddings
matter (architect contradiction, L3 retrieval). `noop` makes every distance meaningless.
- **`[tools.research] enabled = true` + `searxng_url`** — for the research/egress plan.
- **`[health] enabled = true`** — for the llama-health plan; note `interval_ms` (probe cadence).
A `[[artifacts]]` entry must exist for every LLM-emitted artifact kind a workflow `produces`
(schemas under `~/.config/correx/schemas/`). `sync-config.sh` copies the schemas; confirm the
`[[artifacts]]` table in `config.toml` references the kinds your workflow uses.
## 2. llama-server
- **Managed:** nothing to start — correx launches it. The health probe pings
`http://<[models].host>:<[models].port>/health`.
- **External/static:** start your llama-server on the `url` host:port; its `/health` must 2xx.
```bash
llama-server -m ~/models/<model>.gguf --host 127.0.0.1 --port 10000
```
## 3. SearXNG (research-workflow QA only)
```bash
scripts/qa/searxng-up.sh # http://localhost:8888, JSON format enabled
# ... run research QA ...
scripts/qa/searxng-down.sh
```
> **Gotcha:** SearXNG ships with only the `html` output format; the research `web_search`
> needs the **JSON API**. `searxng-up.sh` writes a `settings.yml` enabling `formats: [html, json]`
> and mounts it. Without that, `web_search` returns nothing and the workflow stalls.
## 4. Start the server
```bash
./gradlew :apps:server:run # mainClass com.correx.apps.server.MainKt, listens on :8080
# or build a runnable dist once and reuse it:
./gradlew :apps:server:installDist
apps/server/build/install/server/bin/server
```
> **Gotcha:** do **not** rebuild jars while the server JVM is running — lazy classloading
> breaks (`NoClassDefFoundError`). Stop the server first, rebuild, restart.
## 5. Start the TUI
```bash
cd apps/tui-go
GOTOOLCHAIN=auto go build -o correx-tui .
./correx-tui -host localhost -port 8080 # flags default to localhost:8080
```
## 6. Evidence tools (what the plans cite)
| Source | Command / where |
|--------|-----------------|
| Event log (the truth) | `./gradlew :apps:cli:run --args="events <sessionId>"` or `GET /sessions/{id}/events` |
| Deterministic replay | `./gradlew :apps:cli:run --args="replay <sessionId>"` |
| System health | `./gradlew :apps:cli:run --args="health"` |
| Server logs | stdout of `:apps:server:run` — MDC carries `sessionId` |
| Files on disk | e.g. `<workspaceRoot>/.correx/project.toml` (idea-promotion plan) |
| TUI render | the running `correx-tui` |
(Once `:apps:cli:installDist` is built, `apps/cli/build/install/cli/bin/cli events <id>` works too.)
## Teardown
1. Stop the server (Ctrl-C) — a managed `[[models]]` llama-server is killed by the shutdown hook.
2. `scripts/qa/searxng-down.sh` if you started it.
3. External llama-server: stop it yourself.
---
## Per-plan env matrix
| QA plan | model | embedder=llamacpp | SearXNG | `project.enabled` | workspace `.correx/` |
|---------|:-----:|:-----------------:|:-------:|:-----------------:|:--------------------:|
| `QA-research-egress` | yes (tool-calling) | no | **yes** | no | no |
| `QA-architect-contradiction` | yes | **yes** | no | **yes** | repo root, 2 sessions same process |
| `QA-llama-health-probe` | yes (static path to kill) | no | no | no | no |
| `QA-idea-promotion` | router only (or seed event) | no | no | no | **yes** (bound workspace) |
| `QA-reviewer-static-first` | yes | no | no | no | seed `StaticFindingsRecordedEvent` |
| `QA-brief-echo-gate` | yes (the prod candidate) | no | no | no | analyst brief with criteria |
+42
View File
@@ -0,0 +1,42 @@
# QA Plan: architect contradiction-check (display-only) — `4e08510` `eae0a0c`
**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §B-§4 "architect contradiction-check" — checker `eae0a0c`, live server hook `4e08510`.
---
## Preconditions
- [ ] **server build/branch:** `feat/backlog-burndown`.
- [ ] **llama-server + model:** a model that runs the `role_pipeline` (analyst→architect→…) and emits a `design` artifact with a populated `approach` field (see `docs/schemas/design.json`).
- [ ] **external deps:** none beyond the model.
- [ ] **config synced:** `[project] enabled = true` (the checker queries the `project:<repoRoot>` L3 namespace that `ProjectMemoryService.persist` writes); **`[router.embedder] backend = "llamacpp"`** with a real embedder URL — `noop` embeddings make every distance meaningless and the gate never fires. `[router.l3] backend = "in_memory"` is fine for a single run, but **note**: in-memory L3 is process-lifetime, so the two sessions below must run against the **same** server process.
- [ ] **fixtures/seed:** the same workspace/repo root for both sessions (the namespace is `project:<repoRoot>`).
## Acceptance gate (one sentence)
> When a later session's architect `design` decision is semantically near a **prior** session's recorded decision, a `PossibleContradictionFlaggedEvent` is appended — and it never halts the stage.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Run **session A** end-to-end on a repo; architect commits a clear decision (e.g. "use Postgres for persistence"). Let the session **complete** (distillation runs on completion). | Session A completes; `correx events {A}` shows the architect `design` `ArtifactCreatedEvent`. The decision is now distilled into L3 under `project:<repoRoot>` (ProjectMemoryService.persist on completion). | |
| 2 | Run **session B** (same server process, same repo) whose architect decision **reverses** A (e.g. "use SQLite / drop Postgres"). | When B's architect `design` artifact is created, `correx events {B}` contains a `PossibleContradictionFlaggedEvent``decisionSummary` = B's `approach`, `related[]` lists A's decision with a `score ≥ 0.75` and `source` = the `project:<repoRoot>` turnId. | |
| 3 | Confirm non-interference. | Session B's architect stage **still passes** and proceeds to the planner — the flag is display-only (no `WorkflowFailedEvent`, no stage halt around the flag). | |
| 4 | Run **session C** with an architect decision unrelated to A (different domain). | **No** `PossibleContradictionFlaggedEvent` for C (nothing clears the 0.75 threshold). Absence is the evidence. | |
| 5 | Confirm self-exclusion. | The flag in check 2 never lists **B's own** in-flight decision as `related` (checker filters `entry.sessionId != current`). | |
Evidence sources: `correx events {id}`, server logs (the `[Orchestrator]`/checker warn line on a near hit is informative but the **event** is the gate).
## Out of scope (explicitly NOT covered this pass)
- In-session contradiction (two decisions inside one session) — by design the checker filters to PRIOR sessions; not built and not required.
- LLM-judged contradiction — v1 is similarity-only, display-only.
- ADR `alternativesConsidered` shape (BACKLOG marks this deliberately not taken).
## Disposition
- **PASS** → move the §B-§4 entry to `RETRO.md` with this run date + the `PossibleContradictionFlaggedEvent` evidence from check 2. Status: PASSED.
- **FAIL** → file numbered findings in `BACKLOG.md`. Most likely failure = embedder is `noop` or the two sessions ran against different processes (in-memory L3 lost) — fix the env, not the code, then re-run. Status: FAILED until green.
+52
View File
@@ -0,0 +1,52 @@
# QA Plan: brief echo-back gate — ARM-IT plan — `1df7af5`
**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §C-A1 "brief echo-back gate" — gate `1df7af5`. **This plan is the precondition for arming the gate in production.**
---
## Why this plan gates the activation
The gate code (`checkBriefEcho`) is built and unit-tested but **OFF by default** (`briefEcho != "true"`). Arming it is a config/prompt change that is **model-dependent and not safe to flip blind**: once on, the planner stage requires a parseable `brief_echo` block in its produced artifact. If the planner model does **not** emit one, `checkBriefEcho` returns a **retryable Failure** ("must restate the brief as a brief_echo block") — which, on a model that never complies, **exhausts retries and fails the planner stage**, breaking `role_pipeline`. So: prove the model reliably emits a parseable block **here**, on a QA box, before turning it on anywhere real.
## How to arm it (do this on the QA box only)
1. **Prompt** — edit `~/.config/correx/prompts/planner.md` (synced from `examples/workflows/prompts/`) to instruct the model to begin its `impl_plan` artifact with a `brief_echo` object restating the brief, exactly:
```json
{"brief_echo": {"referenced_files": ["..."], "symbols": ["..."], "acceptance_criteria": ["..."]}, "...rest of impl_plan...": ...}
```
(Parsed by `BriefEchoExtractor.fromEcho` — root key `brief_echo`, arrays `referenced_files` / `symbols` / `acceptance_criteria`.)
2. **Workflow metadata** — on the **planner** stage in `~/.config/correx/workflows/role_pipeline.toml`, set stage metadata `briefEcho = "true"` and `briefEchoSource = "analysis"` (the analyst's artifact slot is named `analysis`, not the default `brief`). The original brief's referenced files are pulled by `BriefReferenceExtractor`; optional `acceptance_criteria` / `symbols` arrays on the analysis artifact make the comparison meaningful.
## Preconditions
- [ ] **server build/branch:** `feat/backlog-burndown`.
- [ ] **llama-server + model:** the production candidate model (this plan is specifically testing *that model's* compliance).
- [ ] **config synced + armed** as above.
- [ ] **fixtures/seed:** an analyst `analysis` artifact that names concrete files + acceptance criteria, so divergence is detectable.
## Acceptance gate (one sentence)
> With the gate armed, the planner reliably emits a **parseable** `brief_echo`, the stage **passes** when the echo matches the brief, and **only** diverging echoes produce a `BriefEchoMismatchEvent` + retryable failure — never a "missing block" failure on a compliant run.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Run `role_pipeline` to the planner stage with a faithful brief. | The planner's `impl_plan` artifact contains a parseable `brief_echo`; the stage **passes** (proceeds to implementer). **No** `BriefEchoMismatchEvent`, no "must restate the brief" failure. | |
| 2 | Run it **35 times** across varied briefs. | The model emits a parseable `brief_echo` **every** time (reliability bar). Tally pass rate — anything < 100% means arming will intermittently break the planner. | |
| 3 | Seed a brief with a clear acceptance criterion, then induce the planner to drop it (or hand-edit the echo to omit it). | A `BriefEchoMismatchEvent` with `droppedAcceptanceCriteria` set; stage Failure (retryable) whose message restates the dropped criterion. | |
| 4 | Induce the echo to reference a file **not** in the brief. | `BriefEchoMismatchEvent.inventedFiles` set; failure message names the invented path. | |
| 5 | Temporarily make the planner emit **no** `brief_echo` block. | Failure "must restate the brief as a brief_echo block before planning" — confirming the strict-on-armed behavior (this is the break mode check 2 must rule out for the real model). | |
Evidence sources: `correx events {id}` (`BriefEchoMismatchEvent`), the produced `impl_plan` artifact content, server logs (`[Orchestrator] brief echo diverged`).
## Out of scope
- Symbol grounding precision (deferred; file-path + acceptance-criteria grounding only).
## Disposition
- **PASS (check 2 == 100%)** → it is safe to arm in production; move §C-A1 to `RETRO.md` with the reliability tally. Keep the armed `planner.md` + `role_pipeline.toml` changes (or land them into `examples/` if the repo defaults should ship armed).
- **FAIL (check 2 < 100%, or check 5 fires on a normal run)** → **do NOT arm in production.** File the model's non-compliance as a finding; either improve the prompt and re-run, or leave the gate OFF. Status: FAILED until check 2 is reliably green.
+44
View File
@@ -0,0 +1,44 @@
# QA Plan: idea-board → project-profile promotion — `f107ff5`
**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §E "Idea board → profile promotion".
---
## Preconditions
- [ ] **server build/branch:** `feat/backlog-burndown`.
- [ ] **llama-server + model:** a router model only if you capture the idea via the rubber-duck path (it must emit a ```` ```json {"ideas":[...]} ``` ```` block). To avoid model-dependence you may instead **seed an `IdeaCapturedEvent`** directly (see note below).
- [ ] **external deps:** none.
- [ ] **config synced:** standard.
- [ ] **fixtures/seed:** a session bound to a **workspace** (a `SessionWorkspaceBoundEvent` must exist for the capturing session — promotion writes into `<workspaceRoot>/.correx/project.toml` and no-ops without a bound workspace). Start the TUI/session from inside the target repo dir so the workspace resolves.
## Acceptance gate (one sentence)
> Promoting a captured idea appends its text as a `conventions` entry in `<workspaceRoot>/.correx/project.toml`, records an `IdeaPromotedEvent`, and removes the idea from the board.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Capture an idea (rubber-duck `{"ideas":[...]}`, or seed an `IdeaCapturedEvent`). List the board (TUI `R`/idea overlay, or `ListIdeas` WS). | The idea shows on the board with a stable `ideaId`; `IdeaCapturedEvent` in the log. | |
| 2 | Note the pre-state of `<workspaceRoot>/.correx/project.toml` (may be absent). | File absent or its current `conventions = [...]` recorded. | |
| 3 | Send `PromoteIdea(ideaId)` (TUI promote key, or the WS `PromoteIdea` message). | `<workspaceRoot>/.correx/project.toml` now exists and its `conventions` list contains the idea text **verbatim** (escaped if it had quotes). Re-reading via `ProjectProfileLoader` round-trips it (no corruption). | |
| 4 | Re-read the log and the board. | An `IdeaPromotedEvent(ideaId, sessionId, text, timestampMs)` is appended; the idea **no longer appears** on the board (`activeIdeas` tombstones promoted ids like discards). | |
| 5 | Promote the **same** idea text again from a second idea (dedup). | `withConvention` does not duplicate an identical convention — the list has it once. | |
| 6 | Promote with **no bound workspace** (start a session with an unresolved working dir). | No file write, no `IdeaPromotedEvent` — the handler no-ops and just refreshes the board (safe). | |
> **Seeding without a model:** append an `IdeaCapturedEvent(ideaId, sessionId, text, timestampMs)` for a workspace-bound session via the event store, then drive `PromoteIdea`. This isolates the promotion path from router-prompt behavior.
Evidence sources: the file `<workspaceRoot>/.correx/project.toml` on disk, `correx events {id}`, the TUI board render.
## Out of scope (explicitly NOT covered this pass)
- The rubber-duck capture quality itself (covered by the separate idea-board live-QA, §F).
- Promotion into `about`/`commands` — v1 promotes into `conventions` only.
## Disposition
- **PASS** → move the §E "Idea board → profile promotion" bullet to `RETRO.md` with this run date + the on-disk `.correx/project.toml` evidence. Status: PASSED.
- **FAIL** → numbered findings (e.g. quote-escaping corruption, board not clearing). Status: FAILED until green.
+41
View File
@@ -0,0 +1,41 @@
# QA Plan: LLAMA_SERVER health probe — `266bbf0` `64a90e7` `9eef936`
**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §A-§4 "LLAMA_SERVER health probe" — probe `266bbf0`, registration `64a90e7`, managed-path coverage `9eef936`.
---
## Preconditions
- [ ] **server build/branch:** `feat/backlog-burndown`.
- [ ] **llama-server + model:** a real llama-server reachable at the configured URL. The probe registers when **either** a static `[[providers]]` (type `llamacpp`, `url`) **or** a managed `[[models]]` entry exists (managed URL = `[models].host:[models].port`). Verify your config has one.
- [ ] **external deps:** none.
- [ ] **config synced:** `[health] enabled = true`; note `[health] interval_ms` (the probe cadence — you wait one interval to see an edge).
- [ ] **fixtures/seed:** ability to stop/start the llama-server process (managed: it dies with the server; for the kill/restore test use the **static `[[providers]]`** path so you can stop llama-server independently of correx).
## Acceptance gate (one sentence)
> The `LLAMA_SERVER` subject is monitored only when a model is configured, reports HEALTHY while the server answers `/health`, and emits a **liveness DEGRADED** edge when the server stops — and a restored edge when it returns.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Start correx with a llama-server configured and reachable. Wait one `interval_ms`. | `correx health` (and the health projection) lists `LLAMA_SERVER` as HEALTHY. No spurious DEGRADED. | |
| 2 | **Stop** the llama-server (static-provider path: kill the external process). Wait one `interval_ms`. | A health **degraded** edge event for subject `LLAMA_SERVER`, `metric = "liveness"`, detail "llama server unreachable" appears in the system-health log (`correx events` on the SYSTEM session / `correx health`). | |
| 3 | **Restart** the llama-server. Wait one `interval_ms`. | A **restored** edge event for `LLAMA_SERVER`; `correx health` shows HEALTHY again. Edges are recorded once (edge-triggered, not per-tick). | |
| 4 | Start correx on a box with **no** `[[providers]]` and **no** `[[models]]`. | `LLAMA_SERVER` is **absent** from the health subjects — the probe is not registered (no spurious DEGRADED). Absence is the evidence. | |
| 5 | `correx replay` the SYSTEM health session. | The degraded/restored edges replay deterministically from the log (the rolling window is live-only per Invariant #8 and does not affect replay). | |
Evidence sources: `correx health` CLI, the health projection events, server logs.
## Out of scope (explicitly NOT covered this pass)
- **tokens/sec degradation** — the `/health` endpoint is reachability-only; `HttpLlamaLivenessClient` never populates `tokensPerSecond`, so the throughput-collapse branch of `LlamaServerHealthProbe` is dormant until a tps telemetry feed exists. Liveness only this pass. (Refiled in BACKLOG §A-§4 remainder.)
- The live health **TUI pane** (deferred; system-health events are not yet on the TUI wire).
## Disposition
- **PASS** → move the §A-§4 LLAMA_SERVER bullet to `RETRO.md` (liveness verified) with this run date; keep the tokens/sec remainder open in BACKLOG. Status: PASSED.
- **FAIL** → numbered findings. A likely miss: the managed `[[models]]` path not registering — confirm `9eef936` is in the build. Status: FAILED until green.
+42
View File
@@ -0,0 +1,42 @@
# QA Plan: research egress allowlist + batch source-list approval — `ab7e4be` `4b0024f` `b098d87` `027ff1f`
**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §D — "Dynamic per-session egress allowlist", "§3 batch fetch-approval at source-list level", "Dedicated `SourceFetched` / `LowQualityExtraction` events", plus the §D "Live-QA the full run" gate.
---
## Preconditions
- [ ] **server build/branch:** `feat/backlog-burndown`_rebuild only with the server stopped (lazy classloading breaks otherwise)_
- [ ] **llama-server + model:** a tool-calling-capable model (the research workflow drives `web_search`/`web_fetch` tool calls). Managed `[[models]]` or static `[[providers]]`.
- [ ] **external deps:** **SearXNG** at `[tools.research] searxng_url` with the JSON format enabled (`scripts/qa/searxng-up.sh`; see `docs/qa/ENV.md` §3). Outbound network for the fetched hosts.
- [ ] **config synced:** `~/.config/correx` matches `examples/*` + `docs/schemas/*` (`scripts/qa/sync-config.sh`); `research.toml` present; `[tools.research] enabled = true`.
- [ ] **fixtures/seed:** a research prompt that yields ≥2 distinct source hosts (e.g. "summarize the latest on <topic> citing official docs").
## Acceptance gate (one sentence)
> Approving a source list once clears every subsequent `web_fetch` to those hosts **without a per-URL T2 prompt**, and the fetch quality lands as dedicated `SourceFetched` / `LowQualityExtraction` events — while a host NOT on the list still raises T2.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Start a research session; let it reach a state with candidate source URLs. | `web_search` T1 ran; the candidate URLs appear in the session (search result / dossier event). | |
| 2 | `POST /sessions/{id}/approve-sources` with the candidate URL list (2+ hosts). | Exactly **one** `EgressHostsGrantedEvent` in `correx events {id}``hosts` = the distinct lower-cased hostnames (ports stripped), `reason = "batch source-list approval"`. | |
| 3 | Let the workflow `web_fetch` each approved URL. | Each fetch to an approved host proceeds with **no `ApprovalRequestedEvent` (T2)**`NetworkHostRule` PROCEEDs via the per-session union. A `SourceFetchedEvent` (with `content_sha256` + quality) is recorded per successful fetch. | |
| 4 | Force a low-quality extraction (e.g. a near-empty / boilerplate page among the sources). | A `LowQualityExtractionEvent` is recorded (not just metadata on `ToolExecutionCompletedEvent`). | |
| 5 | Have the agent attempt a `web_fetch` to a host **not** in the approved list. | A T2 `ApprovalRequestedEvent` IS raised for that host (the allowlist is a union, not a blanket open). | |
| 6 | `correx replay {id}`. | Replay reproduces the same allowlist decisions deterministically (the grant + fetch events are the only inputs; no re-fetch). | |
Evidence sources: `correx events {id}` / `GET /sessions/{id}/events`, server logs (MDC sessionId), `correx replay {id}`.
## Out of scope (explicitly NOT covered this pass)
- The standalone §D web-approval client (third client) — separate track, not built.
- Headless-browser / LLM-based extraction (non-responsibilities).
## Disposition
- **PASS** → move the §D egress/batch/source-event bullets to `RETRO.md` with this run date + the cited `EgressHostsGrantedEvent` / `SourceFetchedEvent` evidence. Status: PASSED.
- **FAIL** → file each failure as a numbered finding in `BACKLOG.md` (action + repro + the wrong/missing signal). Status: FAILED until green.
+44
View File
@@ -0,0 +1,44 @@
# QA Plan: reviewer static-first (findings exclusion) — `447fc7a`
**Status:** DRAFT (PARTIAL — see gap)
**Run date / operator:**
**BACKLOG item:** §B-§5 "reviewer static-first INFRA" — runner + event + context-exclusion `447fc7a`.
---
## Gap up front (read first)
The exclusion **filter** is live-wired into reviewer context assembly, and `StaticAnalysisRunner` + `StaticFindingsRecordedEvent` are built and unit-tested — but **no stock workflow stage emits `StaticFindingsRecordedEvent` yet** (the deterministic static-check stage-type seam is an open BACKLOG follow-up). So this pass verifies the **filter behavior by injecting the event**; the end-to-end "a stage runs detekt and the reviewer skips those findings" flow is blocked on that seam.
## Preconditions
- [ ] **server build/branch:** `feat/backlog-burndown`.
- [ ] **llama-server + model:** a model that runs `review_loop` / `role_pipeline` reviewer stage.
- [ ] **external deps:** none.
- [ ] **config synced:** `review_loop.toml` (or role_pipeline) present.
- [ ] **fixtures/seed:** a way to append a `StaticFindingsRecordedEvent` for the session (manual event-store seed), carrying a finding that matches a compile/lint issue the code actually has.
## Acceptance gate (one sentence)
> When a `StaticFindingsRecordedEvent` has already recorded a static finding this session, the reviewer's assembled context **mechanically omits** any feedback entry that merely restates that finding.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Seed a `StaticFindingsRecordedEvent` for the session with a known finding (e.g. an unused-import lint at `Foo.kt:12`). | The event is in `correx events {id}`. | |
| 2 | Drive the implementer→reviewer step so reviewer context is assembled. | The reviewer's input context (server log of the assembled prompt / context pack) **does not** contain a feedback entry restating the seeded static finding — `excludeStaticFindingsFromReview` stripped it. | |
| 3 | Seed a finding that does **not** match any retry-feedback entry. | Non-matching entries are untouched; only restatements of recorded static findings are dropped. | |
| 4 | (When the static-check stage seam lands) run a workflow whose static stage emits the event for real. | The reviewer never re-raises the compiler/detekt issues the deterministic tool already caught. _(Deferred — seam not built.)_ | |
Evidence sources: server logs of the assembled reviewer context / context pack, `correx events {id}`.
## Out of scope (explicitly NOT covered this pass)
- The **producing** static-check stage (open BACKLOG §B-§5 follow-up: "a stage that emits `StaticFindingsRecordedEvent`"). Check 4 stays deferred until that seam exists.
- The reviewer prompt half (shipped earlier, `3467826`).
## Disposition
- **PASS (partial)** → record in `RETRO.md` that the **filter** is live-verified (checks 13) and keep the §B-§5 *producing-stage* follow-up open in BACKLOG. Do not claim the full static-first loop until check 4 passes.
- **FAIL** → numbered findings. Status: FAILED until green.
+33
View File
@@ -0,0 +1,33 @@
# Correx live-QA plans
Each `QA-<feature>.md` is the artifact that authorizes a `BACKLOG.md``RETRO.md` move:
it pins **observable evidence** (events, logs, files, TUI renders) for one shipped feature.
Template: `TEMPLATE.md`. Environment bring-up: **`ENV.md`** (+ `scripts/qa/`).
A feature is **not** "resolved" until its plan passes live. On PASS → move the BACKLOG entry
to RETRO with the run date + cited evidence; on FAIL → refile each miss as a numbered finding.
## Plans (this burndown)
| Plan | Feature | Commits | Needs |
|------|---------|---------|-------|
| `QA-research-egress.md` | per-session egress allowlist + batch source-list approval + dedicated source events | `ab7e4be` `4b0024f` `b098d87` `027ff1f` | model + **SearXNG** |
| `QA-architect-contradiction.md` | architect contradiction-check (display-only) | `4e08510` `eae0a0c` | model + **llamacpp embedder** + `project.enabled`, 2 sessions/1 process |
| `QA-llama-health-probe.md` | LLAMA_SERVER liveness probe | `64a90e7` `9eef936` `266bbf0` | static llama-server you can stop |
| `QA-idea-promotion.md` | idea-board → `.correx/project.toml` promotion | `f107ff5` | bound workspace (router model optional — can seed the event) |
| `QA-reviewer-static-first.md` | reviewer static-findings exclusion (PARTIAL — producing stage not built) | `447fc7a` | seed `StaticFindingsRecordedEvent` |
| `QA-brief-echo-gate.md` | **ARM-IT** plan: brief echo-back gate (off by default; risky to arm blind) | `1df7af5` | prod-candidate model |
## Order of attack (suggested)
1. **`QA-llama-health-probe`** + **`QA-idea-promotion`** — cheapest, least model-dependent (health is liveness; promotion can be seeded). Quick wins.
2. **`QA-research-egress`** — once SearXNG is up.
3. **`QA-architect-contradiction`** — needs a real embedder + two sessions.
4. **`QA-brief-echo-gate`** — run this **before** arming the gate anywhere real; check 2 (100% parseable-echo across runs) is the go/no-go for production.
5. **`QA-reviewer-static-first`** — partial until the static-check stage seam lands.
## Still-open §F live-QA gates (no dedicated plan yet — same env applies)
These predate the burndown and remain in `BACKLOG.md` §F; author a plan from `TEMPLATE.md`
when you tackle one: Epic 15 (events/replay + MDC sessionIds), idea-board capture, clarification
loop + workflow-propose panel, the full research-workflow run, TUI kernel-steering + AMD gauge.