kami
12ff2e97b0
Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks: research egress + batch source-approval (D), architect contradiction (B§4), llama health probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in production (it breaks the planner if the model can't emit a parseable brief_echo). Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/ (sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are set -euo pipefail, syntax-checked, executable. Authored from the repo build/config (server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
53 lines
4.6 KiB
Markdown
53 lines
4.6 KiB
Markdown
# QA Plan: brief echo-back gate — ARM-IT plan — `1df7af5`
|
||
|
||
**Status:** DRAFT
|
||
**Run date / operator:**
|
||
**BACKLOG item:** §C-A1 "brief echo-back gate" — gate `1df7af5`. **This plan is the precondition for arming the gate in production.**
|
||
|
||
---
|
||
|
||
## Why this plan gates the activation
|
||
|
||
The gate code (`checkBriefEcho`) is built and unit-tested but **OFF by default** (`briefEcho != "true"`). Arming it is a config/prompt change that is **model-dependent and not safe to flip blind**: once on, the planner stage requires a parseable `brief_echo` block in its produced artifact. If the planner model does **not** emit one, `checkBriefEcho` returns a **retryable Failure** ("must restate the brief as a brief_echo block") — which, on a model that never complies, **exhausts retries and fails the planner stage**, breaking `role_pipeline`. So: prove the model reliably emits a parseable block **here**, on a QA box, before turning it on anywhere real.
|
||
|
||
## How to arm it (do this on the QA box only)
|
||
|
||
1. **Prompt** — edit `~/.config/correx/prompts/planner.md` (synced from `examples/workflows/prompts/`) to instruct the model to begin its `impl_plan` artifact with a `brief_echo` object restating the brief, exactly:
|
||
```json
|
||
{"brief_echo": {"referenced_files": ["..."], "symbols": ["..."], "acceptance_criteria": ["..."]}, "...rest of impl_plan...": ...}
|
||
```
|
||
(Parsed by `BriefEchoExtractor.fromEcho` — root key `brief_echo`, arrays `referenced_files` / `symbols` / `acceptance_criteria`.)
|
||
2. **Workflow metadata** — on the **planner** stage in `~/.config/correx/workflows/role_pipeline.toml`, set stage metadata `briefEcho = "true"` and `briefEchoSource = "analysis"` (the analyst's artifact slot is named `analysis`, not the default `brief`). The original brief's referenced files are pulled by `BriefReferenceExtractor`; optional `acceptance_criteria` / `symbols` arrays on the analysis artifact make the comparison meaningful.
|
||
|
||
## Preconditions
|
||
|
||
- [ ] **server build/branch:** `feat/backlog-burndown`.
|
||
- [ ] **llama-server + model:** the production candidate model (this plan is specifically testing *that model's* compliance).
|
||
- [ ] **config synced + armed** as above.
|
||
- [ ] **fixtures/seed:** an analyst `analysis` artifact that names concrete files + acceptance criteria, so divergence is detectable.
|
||
|
||
## Acceptance gate (one sentence)
|
||
|
||
> With the gate armed, the planner reliably emits a **parseable** `brief_echo`, the stage **passes** when the echo matches the brief, and **only** diverging echoes produce a `BriefEchoMismatchEvent` + retryable failure — never a "missing block" failure on a compliant run.
|
||
|
||
## Checks
|
||
|
||
| # | Action | Expected observable evidence | Result |
|
||
|---|--------|------------------------------|--------|
|
||
| 1 | Run `role_pipeline` to the planner stage with a faithful brief. | The planner's `impl_plan` artifact contains a parseable `brief_echo`; the stage **passes** (proceeds to implementer). **No** `BriefEchoMismatchEvent`, no "must restate the brief" failure. | |
|
||
| 2 | Run it **3–5 times** across varied briefs. | The model emits a parseable `brief_echo` **every** time (reliability bar). Tally pass rate — anything < 100% means arming will intermittently break the planner. | |
|
||
| 3 | Seed a brief with a clear acceptance criterion, then induce the planner to drop it (or hand-edit the echo to omit it). | A `BriefEchoMismatchEvent` with `droppedAcceptanceCriteria` set; stage Failure (retryable) whose message restates the dropped criterion. | |
|
||
| 4 | Induce the echo to reference a file **not** in the brief. | `BriefEchoMismatchEvent.inventedFiles` set; failure message names the invented path. | |
|
||
| 5 | Temporarily make the planner emit **no** `brief_echo` block. | Failure "must restate the brief as a brief_echo block before planning" — confirming the strict-on-armed behavior (this is the break mode check 2 must rule out for the real model). | |
|
||
|
||
Evidence sources: `correx events {id}` (`BriefEchoMismatchEvent`), the produced `impl_plan` artifact content, server logs (`[Orchestrator] brief echo diverged`).
|
||
|
||
## Out of scope
|
||
|
||
- Symbol grounding precision (deferred; file-path + acceptance-criteria grounding only).
|
||
|
||
## Disposition
|
||
|
||
- **PASS (check 2 == 100%)** → it is safe to arm in production; move §C-A1 to `RETRO.md` with the reliability tally. Keep the armed `planner.md` + `role_pipeline.toml` changes (or land them into `examples/` if the repo defaults should ship armed).
|
||
- **FAIL (check 2 < 100%, or check 5 fires on a normal run)** → **do NOT arm in production.** File the model's non-compliance as a finding; either improve the prompt and re-run, or leave the gate OFF. Status: FAILED until check 2 is reliably green.
|