# Stage prompt audition results — 2026-07-16

## Decision

Do not claim a material performance benefit for `CURATED_STAGE_OPERATING_GUIDANCE` from this audition. Keep or remove it on clarity and token-cost grounds, not on measured workflow efficiency.

Both variants completed and built successfully in all five controlled samples. Guidance ON had a slightly lower median inference count, but a slightly higher mean because of one severe failure loop. Pairwise outcomes and aggregate error counts do not show a stable causal advantage. The defensible finding is **no demonstrated benefit and no demonstrated final-outcome harm, with high run-to-run variance in both variants**.

The guidance block under test was:

```text
## Operating guidance
Verify the current workspace before declaring completion. Read before modifying existing files.
When the authoritative intent plainly requires project setup, creating its manifest, configuration,
or entry file is in scope; do not spend turns re-deciding that settled boundary. Re-observe facts
instead of assuming them, and use the exact gate/tool feedback to make the smallest effective fix.
```

Guidance OFF removed only this L0 system entry through `CORREX_STAGE_GUIDANCE=off`.

## Controlled batch

- Intent: scaffold a minimal Vite + React + TypeScript frontend in `frontend/` and make `npm run build` pass.
- Scratch workspace: `/tmp/qa-audition-ws`, wiped before every sample.
- Driver: `scripts/prompt-audition.sh` with `BATCH_RUNS=5`, variants `on off`, and `POLL_SECS=1200`.
- Containment gate: every sample required matching `SessionWorkspaceBound.workspaceRoot` and `WorkspaceStateObserved.workspaceRoot`; any emitted `RepoMapComputed.repoRoot` also had to match.
- Completion gate: only a `WorkflowCompleted` or `WorkflowFailed` after `ExecutionPlanLocked` counted.
- Approvals: the unattended driver approved pending T2 gates every 15 seconds.
- Build gate: after the workflow terminal, the driver independently ran `npm run build` in the generated frontend.

### Per-run results

| Pair | Variant | Session | Terminal | Build | Inferences | Approvals | Tools completed | Rejected | Failed | Retries |
|---:|---|---|---|---|---:|---:|---:|---:|---:|---:|
| 1 | ON | `c9d4a30e-b71d-4137-a850-9b6428e30137` | completed | passed | 20 | 10 | 13 | 0 | 1 | 0 |
| 1 | OFF | `c656e3ca-b8dd-43b4-97d9-74add7e05cdc` | completed | passed | 24 | 11 | 16 | 0 | 1 | 1 |
| 2 | ON | `62929ec7-8216-4a53-a300-3fa61ff7e547` | completed | passed | 20 | 13 | 14 | 0 | 0 | 0 |
| 2 | OFF | `1af1364b-9fcb-48b3-bbe8-90110f77f53d` | completed | passed | 21 | 13 | 14 | 0 | 0 | 1 |
| 3 | ON | `310f7220-3157-4567-a495-7be217d2c1df` | completed | passed | 68 | 46 | 33 | 2 | 27 | 0 |
| 3 | OFF | `10c35e0d-8c63-40a8-a948-b9272407ca5f` | completed | passed | 27 | 14 | 20 | 0 | 1 | 0 |
| 4 | ON | `6e3f8809-93df-4e58-9ca5-7f05b3b0500a` | completed | passed | 18 | 12 | 12 | 0 | 0 | 0 |
| 4 | OFF | `362bb82d-71dc-497f-ad73-870db5c33e72` | completed | passed | 21 | 14 | 14 | 0 | 1 | 0 |
| 5 | ON | `84e69b2a-73f8-4052-a05a-00f7ab0f9349` | completed | passed | 44 | 22 | 33 | 0 | 3 | 1 |
| 5 | OFF | `a160a33e-51a5-4e20-98c9-5a635a276fd5` | completed | passed | 67 | 32 | 49 | 2 | 8 | 0 |

### Aggregate results

| Metric | Guidance ON | Guidance OFF | Interpretation |
|---|---:|---:|---|
| Workflow success | 5/5 | 5/5 | no difference |
| Independent build success | 5/5 | 5/5 | no difference |
| Inferences, mean | 34.0 | 32.0 | OFF lower by 2; outlier-sensitive |
| Inferences, median | 20 | 24 | ON lower by 4 |
| Inferences, range | 18–68 | 21–67 | both have large tails |
| Approvals, mean | 20.6 | 16.8 | OFF lower, dominated by ON run 3 |
| Approvals, median | 13 | 14 | effectively tied |
| Tools completed, mean | 21.0 | 22.6 | effectively tied |
| Rejections, total | 2 | 2 | tied |
| Failures, total | 31 | 11 | OFF lower; 27 ON failures came from run 3 |
| Retries, total | 1 | 2 | ON lower by one |

ON used fewer inferences in four of five paired comparisons, but the single ON run-3 failure loop erased that advantage in the mean. OFF then produced its own long-tail result in run 5. With five samples per variant, the distribution is too small and heavy-tailed to attribute these differences to the guidance block.

## ON run 3 failure analysis

The error did not originate in the user intent, analyst artifact, operating guidance, or prior-plan hint. The architect inference generated a locked execution plan whose `scaffold_files` prompt included the nonexistent npm package:

```json
"@types/vite": "^4.0.0"
```

That generated prompt became authoritative downstream context, and `scaffold_files` wrote it into `frontend/package.json`. The structural plan and file gates accepted the manifest because it was valid JSON with the required keys; they do not resolve package names.

`npm install` returned an exact registry 404 for `@types/vite`. The model nevertheless retried npm repeatedly, tried explicit versions and flags, switched to Yarn, probed pnpm and network state, and hit two read-before-write rejections before eventually rewriting the manifest without the invalid package. Stage inference counts were:

| Stage | Inferences |
|---|---:|
| `scaffold_files` | 31 |
| `install_dependencies` | 26 |
| `verify_build` | 6 |
| `analyst` | 2 |
| `discovery` | 2 |
| `architect` | 1 |

The guidance sentence about using exact feedback and making the smallest fix was directly applicable but did not stop the loop. This argues for deterministic defenses rather than stronger prose alone:

- Validate or resolve manifest dependencies before locking or accepting a generated scaffold.
- Bound repeated equivalent failures within a stage.
- Convert an unambiguous package 404 into structured recovery evidence that identifies the offending manifest entry.
- Require a materially different action after repeated identical failures.

## Earlier samples

Earlier exploratory sessions suggested that ON might be cheaper than OFF, followed by an ON repeat that varied substantially. Those samples are excluded from the controlled aggregate because the later audit found a separate `[project].root` leak: tool writes were contained in scratch, but repo-map and project-memory context could still be sourced from the real Correx repository.

The server now rebinds project observation and memory to the authoritative boot workspace, and the batch driver checks the recorded project-observation root before allowing a sample to proceed. Earlier session IDs remain useful only as exploratory history:

- First apparent ON/OFF pair: `742e6330-f4ab-42a4-9466-4c8343452560`, `83ae4e27-ed4b-4d45-ad90-2db177487eaa`.
- Additional ON repeat: `9721ba88-4819-401b-8d68-5d667ecdfe64`.
- Discarded contaminated preflight sample: `8a1a5a3d-7177-4682-afc1-61e179aaa95c`.

Do not combine their metrics with the five-pair table.

## Artifacts and reproduction

Batch artifacts are local and gitignored under:

```text
scripts/output/prompt-audition-20260716-165518/
```

Important files:

- `results.tsv` — authoritative per-run metric table.
- `batch.log` — unattended driver progress and final rendered table.
- `server-NN-on.log`, `server-NN-off.log` — full server/inference/event logs for each sample.
- `build-NN-on.log`, `build-NN-off.log` — independent post-terminal build output.
- `pid` — detached batch PID used during the run; the process has exited.

Reproduce a five-pair detached batch with:

```bash
DETACH=1 BATCH_RUNS=5 POLL_SECS=1200 WORKDIR=/tmp/qa-audition-ws \
  bash scripts/prompt-audition.sh
```

The command prints the timestamped batch directory. A single pair remains the default when `BATCH_RUNS` is omitted. `VARIANTS=on` or `VARIANTS=off` runs only one side.

## Limitations

- No deterministic model seed was pinned, so the audition measures observed stochastic behavior rather than a reproducible seeded benchmark.
- One task fixture cannot establish general guidance value across coding, research, and repair workflows.
- Five samples per variant are enough to expose variance but not enough for a strong statistical claim about heavy-tailed behavior.
- Cross-session “plan shape that worked before” memory remained part of normal runtime context for both variants; only the operating-guidance entry differed.
- Metrics count event-log activity, not token usage or wall-clock duration per stage.

## Recommendation

Treat the current guidance block as performance-neutral and non-protective. If its token cost is material, remove or shorten it unless it has a separate clarity or policy rationale. Prioritize deterministic plan/package validation and bounded recovery loops; those address the observed failure mechanism directly.

If further prompt evidence is desired, run a seeded matrix across multiple task fixtures and compare distributions, not single-run totals. Record token counts and stage durations in the batch summary before the next audition.
