docs(qa): live-QA plans + env bring-up for the burndown features

Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks:
research egress + batch source-approval (D), architect contradiction (B§4), llama health
probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the
brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in
production (it breaks the planner if the model can't emit a parseable brief_echo).

Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/
(sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are
set -euo pipefail, syntax-checked, executable. Authored from the repo build/config
(server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-21 09:14:36 +00:00
parent 9eef936073
commit 12ff2e97b0
8 changed files with 424 additions and 0 deletions
+52
View File
@@ -0,0 +1,52 @@
# QA Plan: brief echo-back gate — ARM-IT plan — `1df7af5`
**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §C-A1 "brief echo-back gate" — gate `1df7af5`. **This plan is the precondition for arming the gate in production.**
---
## Why this plan gates the activation
The gate code (`checkBriefEcho`) is built and unit-tested but **OFF by default** (`briefEcho != "true"`). Arming it is a config/prompt change that is **model-dependent and not safe to flip blind**: once on, the planner stage requires a parseable `brief_echo` block in its produced artifact. If the planner model does **not** emit one, `checkBriefEcho` returns a **retryable Failure** ("must restate the brief as a brief_echo block") — which, on a model that never complies, **exhausts retries and fails the planner stage**, breaking `role_pipeline`. So: prove the model reliably emits a parseable block **here**, on a QA box, before turning it on anywhere real.
## How to arm it (do this on the QA box only)
1. **Prompt** — edit `~/.config/correx/prompts/planner.md` (synced from `examples/workflows/prompts/`) to instruct the model to begin its `impl_plan` artifact with a `brief_echo` object restating the brief, exactly:
```json
{"brief_echo": {"referenced_files": ["..."], "symbols": ["..."], "acceptance_criteria": ["..."]}, "...rest of impl_plan...": ...}
```
(Parsed by `BriefEchoExtractor.fromEcho` — root key `brief_echo`, arrays `referenced_files` / `symbols` / `acceptance_criteria`.)
2. **Workflow metadata** — on the **planner** stage in `~/.config/correx/workflows/role_pipeline.toml`, set stage metadata `briefEcho = "true"` and `briefEchoSource = "analysis"` (the analyst's artifact slot is named `analysis`, not the default `brief`). The original brief's referenced files are pulled by `BriefReferenceExtractor`; optional `acceptance_criteria` / `symbols` arrays on the analysis artifact make the comparison meaningful.
## Preconditions
- [ ] **server build/branch:** `feat/backlog-burndown`.
- [ ] **llama-server + model:** the production candidate model (this plan is specifically testing *that model's* compliance).
- [ ] **config synced + armed** as above.
- [ ] **fixtures/seed:** an analyst `analysis` artifact that names concrete files + acceptance criteria, so divergence is detectable.
## Acceptance gate (one sentence)
> With the gate armed, the planner reliably emits a **parseable** `brief_echo`, the stage **passes** when the echo matches the brief, and **only** diverging echoes produce a `BriefEchoMismatchEvent` + retryable failure — never a "missing block" failure on a compliant run.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Run `role_pipeline` to the planner stage with a faithful brief. | The planner's `impl_plan` artifact contains a parseable `brief_echo`; the stage **passes** (proceeds to implementer). **No** `BriefEchoMismatchEvent`, no "must restate the brief" failure. | |
| 2 | Run it **35 times** across varied briefs. | The model emits a parseable `brief_echo` **every** time (reliability bar). Tally pass rate — anything < 100% means arming will intermittently break the planner. | |
| 3 | Seed a brief with a clear acceptance criterion, then induce the planner to drop it (or hand-edit the echo to omit it). | A `BriefEchoMismatchEvent` with `droppedAcceptanceCriteria` set; stage Failure (retryable) whose message restates the dropped criterion. | |
| 4 | Induce the echo to reference a file **not** in the brief. | `BriefEchoMismatchEvent.inventedFiles` set; failure message names the invented path. | |
| 5 | Temporarily make the planner emit **no** `brief_echo` block. | Failure "must restate the brief as a brief_echo block before planning" — confirming the strict-on-armed behavior (this is the break mode check 2 must rule out for the real model). | |
Evidence sources: `correx events {id}` (`BriefEchoMismatchEvent`), the produced `impl_plan` artifact content, server logs (`[Orchestrator] brief echo diverged`).
## Out of scope
- Symbol grounding precision (deferred; file-path + acceptance-criteria grounding only).
## Disposition
- **PASS (check 2 == 100%)** → it is safe to arm in production; move §C-A1 to `RETRO.md` with the reliability tally. Keep the armed `planner.md` + `role_pipeline.toml` changes (or land them into `examples/` if the repo defaults should ship armed).
- **FAIL (check 2 < 100%, or check 5 fires on a normal run)** → **do NOT arm in production.** File the model's non-compliance as a finding; either improve the prompt and re-run, or leave the gate OFF. Status: FAILED until check 2 is reliably green.