From 0d8df4a1307d3a6bf882f39b23b222e65f4dc9c8 Mon Sep 17 00:00:00 2001 From: kami Date: Sun, 14 Jun 2026 23:22:56 +0400 Subject: [PATCH] docs(backlog): mark brief echo-back gate shipped, add QA plan --- BACKLOG.md | 16 +++++++--- docs/qa/QA-brief-echo-gate.md | 56 +++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 4 deletions(-) create mode 100644 docs/qa/QA-brief-echo-gate.md diff --git a/BACKLOG.md b/BACKLOG.md index 814fb16c..2be2f006 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -66,11 +66,15 @@ and §5 reviewer prompt/needs wiring (3467826) are shipped — RETRO candidates. ## C. Plan pipeline addenda — `docs/plans/2026-06-13-plan-pipeline-addenda.md` -Entire spec is **unbuilt**. +A1 is shipped (bcc59d2) — RETRO candidate once live-QA'd. A2/A3 remain. -- [ ] **A1 brief echo-back gate** — planner restates brief as `BriefEcho`; harness diffs - referencedFiles/symbols/acceptanceCriteria vs original; divergence → `BriefEchoMismatchEvent`, - halt before any plan generation. (Cheapest failure point.) +- [~] **A1 brief echo-back gate** — **SHIPPED (pending live-QA)** `bcc59d2`. New `brief_echo` stage + before the planner in `role_pipeline`: the model restates the analyst brief as a structured + artifact; a deterministic gate (`checkBriefEcho`, opt-in `brief_echo=true`) diffs the restatement + vs the brief — dropped requirements (Jaccard coverage < 0.34) or hallucinated files → + `BriefEchoMismatchEvent` + retryable stage failure, so a misread brief never reaches plan + generation. Pure replay-safe diff (`BriefEchoDiff`); symbols recorded but non-blocking in v1. + QA gate in §F → `docs/qa/QA-brief-echo-gate.md`. `freestyle_planning` wiring is a follow-up. - [ ] **A2 stage-level plan checkpointing** — reconciliation compares produced artifacts vs the confirmed plan's produces-slots per stage; `StageCheckpointPassedEvent` / `StageCheckpointFailedEvent` (halt + surface). @@ -135,6 +139,10 @@ These are SHIPPED in code but prompt-/server-/network-dependent and not yet live (responsive→HEALTHY, no-log-pollution, slow/throw→DEGRADED, restore hysteresis, replay independence). - [ ] **Health TUI pane live-QA** — `fd67a6c`. Plan: `docs/qa/QA-health-tui-pane.md` (`H` opens/fetches, matches `correx health`, degraded loud, disabled fallback, resize, live wire-contract values). +- [ ] **Brief echo-back gate live-QA** — `bcc59d2`; needs a real planner-capable model + runtime install + (schema/prompt/artifacts block). Plan: `docs/qa/QA-brief-echo-gate.md` (faithful echo→planner runs, + dropped requirement→`BriefEchoMismatchEvent`+retryable fail, hallucinated file flagged, symbols + non-blocking, replay recomputes identically, non-opted pipeline is a no-op). - [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the *same* stage's output; AMD VRAM/RAM gauge on the real box. - [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection), diff --git a/docs/qa/QA-brief-echo-gate.md b/docs/qa/QA-brief-echo-gate.md new file mode 100644 index 00000000..27991594 --- /dev/null +++ b/docs/qa/QA-brief-echo-gate.md @@ -0,0 +1,56 @@ +# QA Plan: brief echo-back gate — bcc59d2 + +Plan-pipeline-addenda §A1. Drafted from the diff (commit `bcc59d2`) per the BACKLOG QA rule. + +**Status:** DRAFT +**Run date / operator:** +**BACKLOG item:** §C A1 "brief echo-back gate". + +--- + +## Preconditions + +- [ ] **server build/branch:** master @ `bcc59d2` (or later) — _rebuild only with the server stopped_. +- [ ] **real planner-capable model** running (the echo stage is an inference; needs a live model that can + emit the structured restatement — cannot be exercised with a stub). +- [ ] **runtime install** (like the research workflow): copy `docs/schemas/brief_echo.json` → + `~/.config/correx/schemas/`, `examples/workflows/prompts/brief_echo.md` → `~/.config/correx/prompts/`, + and add the `brief_echo` `[[artifacts]]` block from `examples/workflows/artifacts.config.toml` to + `~/.config/correx/config.toml`. Also copy the updated `role_pipeline.toml`. +- [ ] launch via the `role_pipeline` workflow. + +## Acceptance gate (one sentence) + +> The gate is correct **iff** a faithful restatement lets the pipeline proceed to the planner, while a +> restatement that drops a requirement or invents a file emits `BriefEchoMismatchEvent`, fails the +> `brief_echo` stage retryably (planner does NOT run), and the same mismatch is recomputed identically on replay. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Run role_pipeline on a normal request; let analyst→architect→brief_echo run | `correx events ` shows a `brief_echo` stage running + producing a validated `brief_echo` artifact; flow reaches the `planner` stage; NO `BriefEchoMismatchEvent` | | +| 2 | Inspect the brief_echo artifact (artifact viewer `v` / events) | It contains goal / referenced_files / referenced_symbols / acceptance_criteria reflecting the analyst brief | | +| 3 | Force a misread: prompt/seed so the echo DROPS a requirement (or use a model run that does) | `BriefEchoMismatchEvent` with the dropped requirement in `uncoveredRequirements`; the `brief_echo` stage FAILS retryably; planner has NOT produced a plan | | +| 4 | Force a hallucinated file in the echo (reference a path not in the brief) | `BriefEchoMismatchEvent` with that path in `hallucinatedFiles`; stage fails retryably; feedback names the invented file | | +| 5 | Let the retry happen with the feedback | The re-run echo (corrected) passes → flow proceeds to planner; retry budget respected (≤ max_retries, then surfaces to operator) | | +| 6 | Confirm symbols are non-blocking | An echo referencing a symbol absent from the brief is recorded in `hallucinatedSymbols` but does NOT by itself fail the stage (isMismatch ignores symbols in v1) | | +| 7 | `correx replay ` on a session that had a mismatch | The mismatch is recomputed identically from the recorded analysis + brief_echo artifacts; double-read digest stable; no environment re-query (pure diff) | | +| 8 | Non-opted pipeline (any workflow without `brief_echo=true`) | Behaves exactly as before — gate is a no-op; grounding still runs on the analyst stage | | + +Evidence sources: `correx events ` (stage chain + `BriefEchoMismatchEvent`), the artifact viewer for the +brief_echo content, `correx replay ` for determinism. + +## Out of scope (NOT covered this pass) + +- `freestyle_planning.toml` wiring (deliberate follow-up — gate is on role_pipeline only). +- Tuning the COVERAGE_THRESHOLD (0.34) — adjust only if live runs show false mismatches/passes. +- A1's sibling addenda items A2 (stage checkpointing) / A3 (calibration) — separate BACKLOG items. + +## Disposition + +- **PASS** → MOVE the §C A1 entry into `RETRO.md` (cite `bcc59d2`, run date, evidence). +- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing + signal), fix, re-run only the failed checks. Set Status: FAILED until green. + - If live runs show the deterministic diff is too strict/loose (false mismatches or missed misreads), + that's a tuning finding on `COVERAGE_THRESHOLD` / the hallucination heuristic — record the observed case.