docs(backlog): mark brief echo-back gate shipped, add QA plan

2026-06-14 23:22:56 +04:00
parent bcc59d2164
commit 0d8df4a130
2 changed files with 68 additions and 4 deletions
@@ -0,0 +1,56 @@
+# QA Plan: brief echo-back gate — bcc59d2
+
+Plan-pipeline-addenda §A1. Drafted from the diff (commit `bcc59d2`) per the BACKLOG QA rule.
+
+**Status:** DRAFT
+**Run date / operator:**
+**BACKLOG item:** §C A1 "brief echo-back gate".
+
+---
+
+## Preconditions
+
+- [ ] **server build/branch:** master @ `bcc59d2` (or later) — _rebuild only with the server stopped_.
+- [ ] **real planner-capable model** running (the echo stage is an inference; needs a live model that can
+      emit the structured restatement — cannot be exercised with a stub).
+- [ ] **runtime install** (like the research workflow): copy `docs/schemas/brief_echo.json` →
+      `~/.config/correx/schemas/`, `examples/workflows/prompts/brief_echo.md` → `~/.config/correx/prompts/`,
+      and add the `brief_echo` `[[artifacts]]` block from `examples/workflows/artifacts.config.toml` to
+      `~/.config/correx/config.toml`. Also copy the updated `role_pipeline.toml`.
+- [ ] launch via the `role_pipeline` workflow.
+
+## Acceptance gate (one sentence)
+
+> The gate is correct **iff** a faithful restatement lets the pipeline proceed to the planner, while a
+> restatement that drops a requirement or invents a file emits `BriefEchoMismatchEvent`, fails the
+> `brief_echo` stage retryably (planner does NOT run), and the same mismatch is recomputed identically on replay.
+
+## Checks
+
+| # | Action | Expected observable evidence | Result |
+|---|--------|------------------------------|--------|
+| 1 | Run role_pipeline on a normal request; let analyst→architect→brief_echo run | `correx events <id>` shows a `brief_echo` stage running + producing a validated `brief_echo` artifact; flow reaches the `planner` stage; NO `BriefEchoMismatchEvent` | |
+| 2 | Inspect the brief_echo artifact (artifact viewer `v` / events) | It contains goal / referenced_files / referenced_symbols / acceptance_criteria reflecting the analyst brief | |
+| 3 | Force a misread: prompt/seed so the echo DROPS a requirement (or use a model run that does) | `BriefEchoMismatchEvent` with the dropped requirement in `uncoveredRequirements`; the `brief_echo` stage FAILS retryably; planner has NOT produced a plan | |
+| 4 | Force a hallucinated file in the echo (reference a path not in the brief) | `BriefEchoMismatchEvent` with that path in `hallucinatedFiles`; stage fails retryably; feedback names the invented file | |
+| 5 | Let the retry happen with the feedback | The re-run echo (corrected) passes → flow proceeds to planner; retry budget respected (≤ max_retries, then surfaces to operator) | |
+| 6 | Confirm symbols are non-blocking | An echo referencing a symbol absent from the brief is recorded in `hallucinatedSymbols` but does NOT by itself fail the stage (isMismatch ignores symbols in v1) | |
+| 7 | `correx replay <id>` on a session that had a mismatch | The mismatch is recomputed identically from the recorded analysis + brief_echo artifacts; double-read digest stable; no environment re-query (pure diff) | |
+| 8 | Non-opted pipeline (any workflow without `brief_echo=true`) | Behaves exactly as before — gate is a no-op; grounding still runs on the analyst stage | |
+
+Evidence sources: `correx events <id>` (stage chain + `BriefEchoMismatchEvent`), the artifact viewer for the
+brief_echo content, `correx replay <id>` for determinism.
+
+## Out of scope (NOT covered this pass)
+
+- `freestyle_planning.toml` wiring (deliberate follow-up — gate is on role_pipeline only).
+- Tuning the COVERAGE_THRESHOLD (0.34) — adjust only if live runs show false mismatches/passes.
+- A1's sibling addenda items A2 (stage checkpointing) / A3 (calibration) — separate BACKLOG items.
+
+## Disposition
+
+- **PASS** → MOVE the §C A1 entry into `RETRO.md` (cite `bcc59d2`, run date, evidence).
+- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing
+  signal), fix, re-run only the failed checks. Set Status: FAILED until green.
+  - If live runs show the deterministic diff is too strict/loose (false mismatches or missed misreads),
+    that's a tuning finding on `COVERAGE_THRESHOLD` / the hallucination heuristic — record the observed case.