docs(backlog): mark brief echo-back gate shipped, add QA plan
This commit is contained in:
+12
-4
@@ -66,11 +66,15 @@ and §5 reviewer prompt/needs wiring (3467826) are shipped — RETRO candidates.
|
||||
|
||||
## C. Plan pipeline addenda — `docs/plans/2026-06-13-plan-pipeline-addenda.md`
|
||||
|
||||
Entire spec is **unbuilt**.
|
||||
A1 is shipped (bcc59d2) — RETRO candidate once live-QA'd. A2/A3 remain.
|
||||
|
||||
- [ ] **A1 brief echo-back gate** — planner restates brief as `BriefEcho`; harness diffs
|
||||
referencedFiles/symbols/acceptanceCriteria vs original; divergence → `BriefEchoMismatchEvent`,
|
||||
halt before any plan generation. (Cheapest failure point.)
|
||||
- [~] **A1 brief echo-back gate** — **SHIPPED (pending live-QA)** `bcc59d2`. New `brief_echo` stage
|
||||
before the planner in `role_pipeline`: the model restates the analyst brief as a structured
|
||||
artifact; a deterministic gate (`checkBriefEcho`, opt-in `brief_echo=true`) diffs the restatement
|
||||
vs the brief — dropped requirements (Jaccard coverage < 0.34) or hallucinated files →
|
||||
`BriefEchoMismatchEvent` + retryable stage failure, so a misread brief never reaches plan
|
||||
generation. Pure replay-safe diff (`BriefEchoDiff`); symbols recorded but non-blocking in v1.
|
||||
QA gate in §F → `docs/qa/QA-brief-echo-gate.md`. `freestyle_planning` wiring is a follow-up.
|
||||
- [ ] **A2 stage-level plan checkpointing** — reconciliation compares produced artifacts vs
|
||||
the confirmed plan's produces-slots per stage; `StageCheckpointPassedEvent` /
|
||||
`StageCheckpointFailedEvent` (halt + surface).
|
||||
@@ -135,6 +139,10 @@ These are SHIPPED in code but prompt-/server-/network-dependent and not yet live
|
||||
(responsive→HEALTHY, no-log-pollution, slow/throw→DEGRADED, restore hysteresis, replay independence).
|
||||
- [ ] **Health TUI pane live-QA** — `fd67a6c`. Plan: `docs/qa/QA-health-tui-pane.md` (`H` opens/fetches,
|
||||
matches `correx health`, degraded loud, disabled fallback, resize, live wire-contract values).
|
||||
- [ ] **Brief echo-back gate live-QA** — `bcc59d2`; needs a real planner-capable model + runtime install
|
||||
(schema/prompt/artifacts block). Plan: `docs/qa/QA-brief-echo-gate.md` (faithful echo→planner runs,
|
||||
dropped requirement→`BriefEchoMismatchEvent`+retryable fail, hallucinated file flagged, symbols
|
||||
non-blocking, replay recomputes identically, non-opted pipeline is a no-op).
|
||||
- [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the
|
||||
*same* stage's output; AMD VRAM/RAM gauge on the real box.
|
||||
- [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection),
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# QA Plan: brief echo-back gate — bcc59d2
|
||||
|
||||
Plan-pipeline-addenda §A1. Drafted from the diff (commit `bcc59d2`) per the BACKLOG QA rule.
|
||||
|
||||
**Status:** DRAFT
|
||||
**Run date / operator:**
|
||||
**BACKLOG item:** §C A1 "brief echo-back gate".
|
||||
|
||||
---
|
||||
|
||||
## Preconditions
|
||||
|
||||
- [ ] **server build/branch:** master @ `bcc59d2` (or later) — _rebuild only with the server stopped_.
|
||||
- [ ] **real planner-capable model** running (the echo stage is an inference; needs a live model that can
|
||||
emit the structured restatement — cannot be exercised with a stub).
|
||||
- [ ] **runtime install** (like the research workflow): copy `docs/schemas/brief_echo.json` →
|
||||
`~/.config/correx/schemas/`, `examples/workflows/prompts/brief_echo.md` → `~/.config/correx/prompts/`,
|
||||
and add the `brief_echo` `[[artifacts]]` block from `examples/workflows/artifacts.config.toml` to
|
||||
`~/.config/correx/config.toml`. Also copy the updated `role_pipeline.toml`.
|
||||
- [ ] launch via the `role_pipeline` workflow.
|
||||
|
||||
## Acceptance gate (one sentence)
|
||||
|
||||
> The gate is correct **iff** a faithful restatement lets the pipeline proceed to the planner, while a
|
||||
> restatement that drops a requirement or invents a file emits `BriefEchoMismatchEvent`, fails the
|
||||
> `brief_echo` stage retryably (planner does NOT run), and the same mismatch is recomputed identically on replay.
|
||||
|
||||
## Checks
|
||||
|
||||
| # | Action | Expected observable evidence | Result |
|
||||
|---|--------|------------------------------|--------|
|
||||
| 1 | Run role_pipeline on a normal request; let analyst→architect→brief_echo run | `correx events <id>` shows a `brief_echo` stage running + producing a validated `brief_echo` artifact; flow reaches the `planner` stage; NO `BriefEchoMismatchEvent` | |
|
||||
| 2 | Inspect the brief_echo artifact (artifact viewer `v` / events) | It contains goal / referenced_files / referenced_symbols / acceptance_criteria reflecting the analyst brief | |
|
||||
| 3 | Force a misread: prompt/seed so the echo DROPS a requirement (or use a model run that does) | `BriefEchoMismatchEvent` with the dropped requirement in `uncoveredRequirements`; the `brief_echo` stage FAILS retryably; planner has NOT produced a plan | |
|
||||
| 4 | Force a hallucinated file in the echo (reference a path not in the brief) | `BriefEchoMismatchEvent` with that path in `hallucinatedFiles`; stage fails retryably; feedback names the invented file | |
|
||||
| 5 | Let the retry happen with the feedback | The re-run echo (corrected) passes → flow proceeds to planner; retry budget respected (≤ max_retries, then surfaces to operator) | |
|
||||
| 6 | Confirm symbols are non-blocking | An echo referencing a symbol absent from the brief is recorded in `hallucinatedSymbols` but does NOT by itself fail the stage (isMismatch ignores symbols in v1) | |
|
||||
| 7 | `correx replay <id>` on a session that had a mismatch | The mismatch is recomputed identically from the recorded analysis + brief_echo artifacts; double-read digest stable; no environment re-query (pure diff) | |
|
||||
| 8 | Non-opted pipeline (any workflow without `brief_echo=true`) | Behaves exactly as before — gate is a no-op; grounding still runs on the analyst stage | |
|
||||
|
||||
Evidence sources: `correx events <id>` (stage chain + `BriefEchoMismatchEvent`), the artifact viewer for the
|
||||
brief_echo content, `correx replay <id>` for determinism.
|
||||
|
||||
## Out of scope (NOT covered this pass)
|
||||
|
||||
- `freestyle_planning.toml` wiring (deliberate follow-up — gate is on role_pipeline only).
|
||||
- Tuning the COVERAGE_THRESHOLD (0.34) — adjust only if live runs show false mismatches/passes.
|
||||
- A1's sibling addenda items A2 (stage checkpointing) / A3 (calibration) — separate BACKLOG items.
|
||||
|
||||
## Disposition
|
||||
|
||||
- **PASS** → MOVE the §C A1 entry into `RETRO.md` (cite `bcc59d2`, run date, evidence).
|
||||
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing
|
||||
signal), fix, re-run only the failed checks. Set Status: FAILED until green.
|
||||
- If live runs show the deterministic diff is too strict/loose (false mismatches or missed misreads),
|
||||
that's a tuning finding on `COVERAGE_THRESHOLD` / the hallucination heuristic — record the observed case.
|
||||
Reference in New Issue
Block a user