docs(backlog): mark brief echo-back gate shipped, add QA plan

This commit is contained in:
2026-06-14 23:22:56 +04:00
parent bcc59d2164
commit 0d8df4a130
2 changed files with 68 additions and 4 deletions
+12 -4
View File
@@ -66,11 +66,15 @@ and §5 reviewer prompt/needs wiring (3467826) are shipped — RETRO candidates.
## C. Plan pipeline addenda — `docs/plans/2026-06-13-plan-pipeline-addenda.md` ## C. Plan pipeline addenda — `docs/plans/2026-06-13-plan-pipeline-addenda.md`
Entire spec is **unbuilt**. A1 is shipped (bcc59d2) — RETRO candidate once live-QA'd. A2/A3 remain.
- [ ] **A1 brief echo-back gate**planner restates brief as `BriefEcho`; harness diffs - [~] **A1 brief echo-back gate****SHIPPED (pending live-QA)** `bcc59d2`. New `brief_echo` stage
referencedFiles/symbols/acceptanceCriteria vs original; divergence → `BriefEchoMismatchEvent`, before the planner in `role_pipeline`: the model restates the analyst brief as a structured
halt before any plan generation. (Cheapest failure point.) artifact; a deterministic gate (`checkBriefEcho`, opt-in `brief_echo=true`) diffs the restatement
vs the brief — dropped requirements (Jaccard coverage < 0.34) or hallucinated files →
`BriefEchoMismatchEvent` + retryable stage failure, so a misread brief never reaches plan
generation. Pure replay-safe diff (`BriefEchoDiff`); symbols recorded but non-blocking in v1.
QA gate in §F → `docs/qa/QA-brief-echo-gate.md`. `freestyle_planning` wiring is a follow-up.
- [ ] **A2 stage-level plan checkpointing** — reconciliation compares produced artifacts vs - [ ] **A2 stage-level plan checkpointing** — reconciliation compares produced artifacts vs
the confirmed plan's produces-slots per stage; `StageCheckpointPassedEvent` / the confirmed plan's produces-slots per stage; `StageCheckpointPassedEvent` /
`StageCheckpointFailedEvent` (halt + surface). `StageCheckpointFailedEvent` (halt + surface).
@@ -135,6 +139,10 @@ These are SHIPPED in code but prompt-/server-/network-dependent and not yet live
(responsive→HEALTHY, no-log-pollution, slow/throw→DEGRADED, restore hysteresis, replay independence). (responsive→HEALTHY, no-log-pollution, slow/throw→DEGRADED, restore hysteresis, replay independence).
- [ ] **Health TUI pane live-QA**`fd67a6c`. Plan: `docs/qa/QA-health-tui-pane.md` (`H` opens/fetches, - [ ] **Health TUI pane live-QA**`fd67a6c`. Plan: `docs/qa/QA-health-tui-pane.md` (`H` opens/fetches,
matches `correx health`, degraded loud, disabled fallback, resize, live wire-contract values). matches `correx health`, degraded loud, disabled fallback, resize, live wire-contract values).
- [ ] **Brief echo-back gate live-QA**`bcc59d2`; needs a real planner-capable model + runtime install
(schema/prompt/artifacts block). Plan: `docs/qa/QA-brief-echo-gate.md` (faithful echo→planner runs,
dropped requirement→`BriefEchoMismatchEvent`+retryable fail, hallucinated file flagged, symbols
non-blocking, replay recomputes identically, non-opted pipeline is a no-op).
- [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the - [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the
*same* stage's output; AMD VRAM/RAM gauge on the real box. *same* stage's output; AMD VRAM/RAM gauge on the real box.
- [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection), - [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection),
+56
View File
@@ -0,0 +1,56 @@
# QA Plan: brief echo-back gate — bcc59d2
Plan-pipeline-addenda §A1. Drafted from the diff (commit `bcc59d2`) per the BACKLOG QA rule.
**Status:** DRAFT
**Run date / operator:**
**BACKLOG item:** §C A1 "brief echo-back gate".
---
## Preconditions
- [ ] **server build/branch:** master @ `bcc59d2` (or later) — _rebuild only with the server stopped_.
- [ ] **real planner-capable model** running (the echo stage is an inference; needs a live model that can
emit the structured restatement — cannot be exercised with a stub).
- [ ] **runtime install** (like the research workflow): copy `docs/schemas/brief_echo.json`
`~/.config/correx/schemas/`, `examples/workflows/prompts/brief_echo.md``~/.config/correx/prompts/`,
and add the `brief_echo` `[[artifacts]]` block from `examples/workflows/artifacts.config.toml` to
`~/.config/correx/config.toml`. Also copy the updated `role_pipeline.toml`.
- [ ] launch via the `role_pipeline` workflow.
## Acceptance gate (one sentence)
> The gate is correct **iff** a faithful restatement lets the pipeline proceed to the planner, while a
> restatement that drops a requirement or invents a file emits `BriefEchoMismatchEvent`, fails the
> `brief_echo` stage retryably (planner does NOT run), and the same mismatch is recomputed identically on replay.
## Checks
| # | Action | Expected observable evidence | Result |
|---|--------|------------------------------|--------|
| 1 | Run role_pipeline on a normal request; let analyst→architect→brief_echo run | `correx events <id>` shows a `brief_echo` stage running + producing a validated `brief_echo` artifact; flow reaches the `planner` stage; NO `BriefEchoMismatchEvent` | |
| 2 | Inspect the brief_echo artifact (artifact viewer `v` / events) | It contains goal / referenced_files / referenced_symbols / acceptance_criteria reflecting the analyst brief | |
| 3 | Force a misread: prompt/seed so the echo DROPS a requirement (or use a model run that does) | `BriefEchoMismatchEvent` with the dropped requirement in `uncoveredRequirements`; the `brief_echo` stage FAILS retryably; planner has NOT produced a plan | |
| 4 | Force a hallucinated file in the echo (reference a path not in the brief) | `BriefEchoMismatchEvent` with that path in `hallucinatedFiles`; stage fails retryably; feedback names the invented file | |
| 5 | Let the retry happen with the feedback | The re-run echo (corrected) passes → flow proceeds to planner; retry budget respected (≤ max_retries, then surfaces to operator) | |
| 6 | Confirm symbols are non-blocking | An echo referencing a symbol absent from the brief is recorded in `hallucinatedSymbols` but does NOT by itself fail the stage (isMismatch ignores symbols in v1) | |
| 7 | `correx replay <id>` on a session that had a mismatch | The mismatch is recomputed identically from the recorded analysis + brief_echo artifacts; double-read digest stable; no environment re-query (pure diff) | |
| 8 | Non-opted pipeline (any workflow without `brief_echo=true`) | Behaves exactly as before — gate is a no-op; grounding still runs on the analyst stage | |
Evidence sources: `correx events <id>` (stage chain + `BriefEchoMismatchEvent`), the artifact viewer for the
brief_echo content, `correx replay <id>` for determinism.
## Out of scope (NOT covered this pass)
- `freestyle_planning.toml` wiring (deliberate follow-up — gate is on role_pipeline only).
- Tuning the COVERAGE_THRESHOLD (0.34) — adjust only if live runs show false mismatches/passes.
- A1's sibling addenda items A2 (stage checkpointing) / A3 (calibration) — separate BACKLOG items.
## Disposition
- **PASS** → MOVE the §C A1 entry into `RETRO.md` (cite `bcc59d2`, run date, evidence).
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing
signal), fix, re-run only the failed checks. Set Status: FAILED until green.
- If live runs show the deterministic diff is too strict/loose (false mismatches or missed misreads),
that's a tuning finding on `COVERAGE_THRESHOLD` / the hallucination heuristic — record the observed case.