correx/docs/qa/QA-brief-echo-gate.md

QA Plan: brief echo-back gate — ARM-IT plan — 1df7af5

Status: DRAFT Run date / operator: BACKLOG item: §C-A1 "brief echo-back gate" — gate 1df7af5. This plan is the precondition for arming the gate in production.

---

Why this plan gates the activation

The gate code (checkBriefEcho) is built and unit-tested but OFF by default (briefEcho != "true"). Arming it is a config/prompt change that is model-dependent and not safe to flip blind: once on, the planner stage requires a parseable brief_echo block in its produced artifact. If the planner model does not emit one, checkBriefEcho returns a retryable Failure ("must restate the brief as a brief_echo block") — which, on a model that never complies, exhausts retries and fails the planner stage, breaking role_pipeline. So: prove the model reliably emits a parseable block here, on a QA box, before turning it on anywhere real.

How to arm it (do this on the QA box only)

1. Prompt — edit ~/.config/correx/prompts/planner.md (synced from examples/workflows/prompts/) to instruct the model to begin its impl_plan artifact with a brief_echo object restating the brief, exactly:

   {"brief_echo": {"referenced_files": ["..."], "symbols": ["..."], "acceptance_criteria": ["..."]}, "...rest of impl_plan...": ...}
   

   (Parsed by BriefEchoExtractor.fromEcho — root key brief_echo, arrays referenced_files / symbols / acceptance_criteria.)
2. Workflow metadata — on the planner stage in ~/.config/correx/workflows/role_pipeline.toml, set stage metadata briefEcho = "true" and briefEchoSource = "analysis" (the analyst's artifact slot is named analysis, not the default brief). The original brief's referenced files are pulled by BriefReferenceExtractor; optional acceptance_criteria / symbols arrays on the analysis artifact make the comparison meaningful.

Preconditions

• server build/branch: feat/backlog-burndown.
• llama-server + model: the production candidate model (this plan is specifically testing that model's compliance).
• config synced + armed as above.
• fixtures/seed: an analyst analysis artifact that names concrete files + acceptance criteria, so divergence is detectable.

Acceptance gate (one sentence)

With the gate armed, the planner reliably emits a parseable brief_echo, the stage passes when the echo matches the brief, and only diverging echoes produce a BriefEchoMismatchEvent + retryable failure — never a "missing block" failure on a compliant run.

Checks

#	Action	Expected observable evidence	Result
1	Run role_pipeline to the planner stage with a faithful brief.	The planner's impl_plan artifact contains a parseable brief_echo; the stage passes (proceeds to implementer). No BriefEchoMismatchEvent, no "must restate the brief" failure.
2	Run it 3–5 times across varied briefs.	The model emits a parseable brief_echo every time (reliability bar). Tally pass rate — anything < 100% means arming will intermittently break the planner.
3	Seed a brief with a clear acceptance criterion, then induce the planner to drop it (or hand-edit the echo to omit it).	A BriefEchoMismatchEvent with droppedAcceptanceCriteria set; stage Failure (retryable) whose message restates the dropped criterion.
4	Induce the echo to reference a file not in the brief.	BriefEchoMismatchEvent.inventedFiles set; failure message names the invented path.
5	Temporarily make the planner emit no brief_echo block.	Failure "must restate the brief as a brief_echo block before planning" — confirming the strict-on-armed behavior (this is the break mode check 2 must rule out for the real model).

Evidence sources: correx events {id} (BriefEchoMismatchEvent), the produced impl_plan artifact content, server logs ([Orchestrator] brief echo diverged).

Out of scope

• Symbol grounding precision (deferred; file-path + acceptance-criteria grounding only).

Disposition

• PASS (check 2 == 100%) → it is safe to arm in production; move §C-A1 to RETRO.md with the reliability tally. Keep the armed planner.md + role_pipeline.toml changes (or land them into examples/ if the repo defaults should ship armed).
• FAIL (check 2 < 100%, or check 5 fires on a normal run) → do NOT arm in production. File the model's non-compliance as a finding; either improve the prompt and re-run, or leave the gate OFF. Status: FAILED until check 2 is reliably green.