kami/correx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(prompt): trim performance-neutral stage operating-guidance block

Browse Source 
Vikunja #169 audition result. The 5x5 controlled A/B
(docs/qa/QA-stage-prompt-audition-results-2026-07-16.md) found the curated
guidance block performance-neutral: 5/5 build success both variants, cost
differences within heavy-tailed noise. Trimmed to the single sentence the
gates don't already enforce (scaffolding-scope boundary); dropped the
read-before-write / verify-before-complete / use-exact-feedback nudges that
restate gate-enforced behavior. ON run 3 proved prose doesn't stop failure
loops — deterministic defenses do (filed Vikunja #191/#192, unblocked #78).
This commit is contained in:
kami
2026-07-16 18:38:16 +04:00
parent 53d8108189
commit a2b97221d5
3 changed files with 159 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/kernel/src/main/kotlin/com/correx/core/kernel/orchestration/DefaultSessionOrchestrator.kt
+7 -3
View File
@@ -76,11 +76,15 @@ internal val GATE_REQUIRED_CAPABILITY: Map<String, String> = mapOf(
WORKSPACE_PRECONDITION_GATE to "file_write",
)
// ponytail: trimmed to the single non-redundant sentence after the 2026-07-16 audition
// (docs/qa/QA-stage-prompt-audition-results-2026-07-16.md) found the full block performance-neutral —
// the read-before-write / verify-before-complete / use-exact-feedback nudges just restate behavior the
// gates already enforce, and run 3 proved prose doesn't stop failure loops (deterministic defenses do).
// Only the scaffolding-scope boundary encodes a decision no gate makes. Delete outright if a future
// multi-fixture audit still shows no clarity value.
internal const val CURATED_STAGE_OPERATING_GUIDANCE = """## Operating guidance
Verify the current workspace before declaring completion. Read before modifying existing files.
When the authoritative intent plainly requires project setup, creating its manifest, configuration,
or entry file is in scope; do not spend turns re-deciding that settled boundary. Re-observe facts
instead of assuming them, and use the exact gate/tool feedback to make the smallest effective fix."""
or entry file is in scope; do not spend turns re-deciding that settled boundary."""
// Deterministic failure category from the gate id (no LLM — keeps routing off the untrusted path).
internal fun ticketCategory(gate: String): String = when (gate) {
docs/qa/QA-stage-prompt-audition-results-2026-07-16.md
+141
View File
@@ -0,0 +1,141 @@
# Stage prompt audition results — 2026-07-16
## Decision
Do not claim a material performance benefit for `CURATED_STAGE_OPERATING_GUIDANCE` from this audition. Keep or remove it on clarity and token-cost grounds, not on measured workflow efficiency.
Both variants completed and built successfully in all five controlled samples. Guidance ON had a slightly lower median inference count, but a slightly higher mean because of one severe failure loop. Pairwise outcomes and aggregate error counts do not show a stable causal advantage. The defensible finding is **no demonstrated benefit and no demonstrated final-outcome harm, with high run-to-run variance in both variants**.
The guidance block under test was:
```text
## Operating guidance
Verify the current workspace before declaring completion. Read before modifying existing files.
When the authoritative intent plainly requires project setup, creating its manifest, configuration,
or entry file is in scope; do not spend turns re-deciding that settled boundary. Re-observe facts
instead of assuming them, and use the exact gate/tool feedback to make the smallest effective fix.
```
Guidance OFF removed only this L0 system entry through `CORREX_STAGE_GUIDANCE=off`.
## Controlled batch
- Intent: scaffold a minimal Vite + React + TypeScript frontend in `frontend/` and make `npm run build` pass.
- Scratch workspace: `/tmp/qa-audition-ws`, wiped before every sample.
- Driver: `scripts/prompt-audition.sh` with `BATCH_RUNS=5`, variants `on off`, and `POLL_SECS=1200`.
- Containment gate: every sample required matching `SessionWorkspaceBound.workspaceRoot` and `WorkspaceStateObserved.workspaceRoot`; any emitted `RepoMapComputed.repoRoot` also had to match.
- Completion gate: only a `WorkflowCompleted` or `WorkflowFailed` after `ExecutionPlanLocked` counted.
- Approvals: the unattended driver approved pending T2 gates every 15 seconds.
- Build gate: after the workflow terminal, the driver independently ran `npm run build` in the generated frontend.
### Per-run results
| Pair | Variant | Session | Terminal | Build | Inferences | Approvals | Tools completed | Rejected | Failed | Retries |
|---:|---|---|---|---|---:|---:|---:|---:|---:|---:|
| 1 | ON | `c9d4a30e-b71d-4137-a850-9b6428e30137` | completed | passed | 20 | 10 | 13 | 0 | 1 | 0 |
| 1 | OFF | `c656e3ca-b8dd-43b4-97d9-74add7e05cdc` | completed | passed | 24 | 11 | 16 | 0 | 1 | 1 |
| 2 | ON | `62929ec7-8216-4a53-a300-3fa61ff7e547` | completed | passed | 20 | 13 | 14 | 0 | 0 | 0 |
| 2 | OFF | `1af1364b-9fcb-48b3-bbe8-90110f77f53d` | completed | passed | 21 | 13 | 14 | 0 | 0 | 1 |
| 3 | ON | `310f7220-3157-4567-a495-7be217d2c1df` | completed | passed | 68 | 46 | 33 | 2 | 27 | 0 |
| 3 | OFF | `10c35e0d-8c63-40a8-a948-b9272407ca5f` | completed | passed | 27 | 14 | 20 | 0 | 1 | 0 |
| 4 | ON | `6e3f8809-93df-4e58-9ca5-7f05b3b0500a` | completed | passed | 18 | 12 | 12 | 0 | 0 | 0 |
| 4 | OFF | `362bb82d-71dc-497f-ad73-870db5c33e72` | completed | passed | 21 | 14 | 14 | 0 | 1 | 0 |
| 5 | ON | `84e69b2a-73f8-4052-a05a-00f7ab0f9349` | completed | passed | 44 | 22 | 33 | 0 | 3 | 1 |
| 5 | OFF | `a160a33e-51a5-4e20-98c9-5a635a276fd5` | completed | passed | 67 | 32 | 49 | 2 | 8 | 0 |
### Aggregate results
| Metric | Guidance ON | Guidance OFF | Interpretation |
|---|---:|---:|---|
| Workflow success | 5/5 | 5/5 | no difference |
| Independent build success | 5/5 | 5/5 | no difference |
| Inferences, mean | 34.0 | 32.0 | OFF lower by 2; outlier-sensitive |
| Inferences, median | 20 | 24 | ON lower by 4 |
| Inferences, range | 18–68 | 21–67 | both have large tails |
| Approvals, mean | 20.6 | 16.8 | OFF lower, dominated by ON run 3 |
| Approvals, median | 13 | 14 | effectively tied |
| Tools completed, mean | 21.0 | 22.6 | effectively tied |
| Rejections, total | 2 | 2 | tied |
| Failures, total | 31 | 11 | OFF lower; 27 ON failures came from run 3 |
| Retries, total | 1 | 2 | ON lower by one |
ON used fewer inferences in four of five paired comparisons, but the single ON run-3 failure loop erased that advantage in the mean. OFF then produced its own long-tail result in run 5. With five samples per variant, the distribution is too small and heavy-tailed to attribute these differences to the guidance block.
## ON run 3 failure analysis
The error did not originate in the user intent, analyst artifact, operating guidance, or prior-plan hint. The architect inference generated a locked execution plan whose `scaffold_files` prompt included the nonexistent npm package:
```json
"@types/vite": "^4.0.0"
```
That generated prompt became authoritative downstream context, and `scaffold_files` wrote it into `frontend/package.json`. The structural plan and file gates accepted the manifest because it was valid JSON with the required keys; they do not resolve package names.
`npm install` returned an exact registry 404 for `@types/vite`. The model nevertheless retried npm repeatedly, tried explicit versions and flags, switched to Yarn, probed pnpm and network state, and hit two read-before-write rejections before eventually rewriting the manifest without the invalid package. Stage inference counts were:
| Stage | Inferences |
|---|---:|
| `scaffold_files` | 31 |
| `install_dependencies` | 26 |
| `verify_build` | 6 |
| `analyst` | 2 |
| `discovery` | 2 |
| `architect` | 1 |
The guidance sentence about using exact feedback and making the smallest fix was directly applicable but did not stop the loop. This argues for deterministic defenses rather than stronger prose alone:
- Validate or resolve manifest dependencies before locking or accepting a generated scaffold.
- Bound repeated equivalent failures within a stage.
- Convert an unambiguous package 404 into structured recovery evidence that identifies the offending manifest entry.
- Require a materially different action after repeated identical failures.
## Earlier samples
Earlier exploratory sessions suggested that ON might be cheaper than OFF, followed by an ON repeat that varied substantially. Those samples are excluded from the controlled aggregate because the later audit found a separate `[project].root` leak: tool writes were contained in scratch, but repo-map and project-memory context could still be sourced from the real Correx repository.
The server now rebinds project observation and memory to the authoritative boot workspace, and the batch driver checks the recorded project-observation root before allowing a sample to proceed. Earlier session IDs remain useful only as exploratory history:
- First apparent ON/OFF pair: `742e6330-f4ab-42a4-9466-4c8343452560`, `83ae4e27-ed4b-4d45-ad90-2db177487eaa`.
- Additional ON repeat: `9721ba88-4819-401b-8d68-5d667ecdfe64`.
- Discarded contaminated preflight sample: `8a1a5a3d-7177-4682-afc1-61e179aaa95c`.
Do not combine their metrics with the five-pair table.
## Artifacts and reproduction
Batch artifacts are local and gitignored under:
```text
scripts/output/prompt-audition-20260716-165518/
```
Important files:
- `results.tsv` — authoritative per-run metric table.
- `batch.log` — unattended driver progress and final rendered table.
- `server-NN-on.log`, `server-NN-off.log` — full server/inference/event logs for each sample.
- `build-NN-on.log`, `build-NN-off.log` — independent post-terminal build output.
- `pid` — detached batch PID used during the run; the process has exited.
Reproduce a five-pair detached batch with:
```bash
DETACH=1 BATCH_RUNS=5 POLL_SECS=1200 WORKDIR=/tmp/qa-audition-ws \
bash scripts/prompt-audition.sh
```
The command prints the timestamped batch directory. A single pair remains the default when `BATCH_RUNS` is omitted. `VARIANTS=on` or `VARIANTS=off` runs only one side.
## Limitations
- No deterministic model seed was pinned, so the audition measures observed stochastic behavior rather than a reproducible seeded benchmark.
- One task fixture cannot establish general guidance value across coding, research, and repair workflows.
- Five samples per variant are enough to expose variance but not enough for a strong statistical claim about heavy-tailed behavior.
- Cross-session “plan shape that worked before” memory remained part of normal runtime context for both variants; only the operating-guidance entry differed.
- Metrics count event-log activity, not token usage or wall-clock duration per stage.
## Recommendation
Treat the current guidance block as performance-neutral and non-protective. If its token cost is material, remove or shorten it unless it has a separate clarity or policy rationale. Prioritize deterministic plan/package validation and bounded recovery loops; those address the observed failure mechanism directly.
If further prompt evidence is desired, run a seeded matrix across multiple task fixtures and compare distributions, not single-run totals. Record token counts and stage durations in the batch summary before the next audition.
docs/qa/QA-stage-prompt-audition.md
+11 -9
View File
@@ -1,14 +1,16 @@
# QA Plan: Stage prompt audition
**Status:** DRAFT
**Run date / operator:** Not run — requires a fixed local model and seed.
**Status:** COMPLETED — guidance benefit not demonstrated
**Run date / operator:** 2026-07-16 — local 12B audition; unseeded stochastic batch
**BACKLOG item:** Vikunja #169
**Results:** [QA-stage-prompt-audition-results-2026-07-16.md](QA-stage-prompt-audition-results-2026-07-16.md)
## Preconditions
- [ ] Record the model id, model checksum, `context_size`, and sampling config.
- [ ] Use one fixed workspace fixture and deterministic seed per variant.
- [ ] Run `scripts/prompt-audit.sh` and retain its Markdown output with the run artifacts.
- [x] Use one fixed intent and a freshly wiped scratch workspace per variant.
- [x] Retain per-session server logs, independent build logs, and a machine-readable result table.
- [ ] Pin and record a deterministic seed and complete model checksum/config metadata in a future benchmark-grade rerun.
## Acceptance gate
@@ -18,13 +20,13 @@
| # | Action | Expected observable evidence | Result |
|---|---|---|---|
| 1 | Run baseline, trimmed, curated-rules, and scope-reworded variants. | Session event logs identify every stage inference and terminal workflow outcome. | Not run |
| 2 | Compare gate pass rate, rejected/wasted tool rounds, retries, token budget use, and truncation. | Per-variant result table derived from events; same fixture and seed. | Not run |
| 3 | Select the smallest improving variant. | A committed finding names the selected variant and evidence; otherwise no prompt expansion lands. | Not run |
| 1 | Run guidance ON and OFF variants for five samples each. | Session event logs identify every stage inference and terminal workflow outcome. | Pass: 10/10 completed |
| 2 | Compare build success, inference turns, approvals, tool outcomes, and retries. | Per-variant result table derived from events against the same fixture. | Pass: results report committed |
| 3 | Select the smallest improving variant. | A committed finding names the selected variant and evidence; otherwise no prompt expansion lands. | No winner: benefit not demonstrated |
## Current static audit finding
The runtime currently has two stage-facing sources: workflow prompt files and the compact `CURATED_STAGE_OPERATING_GUIDANCE` L0 system entry. Retry feedback remains event-derived and only the latest failure is injected. The static inventory is reproducible with `scripts/prompt-audit.sh`; no performance claim is made until the controlled run above is recorded.
The runtime has two stage-facing sources: workflow prompt files and the compact `CURATED_STAGE_OPERATING_GUIDANCE` L0 system entry. The completed five-pair audition found identical final success and no stable efficiency improvement from guidance. See the linked results report for the controlled data, excluded exploratory sessions, and the architect-generated `@types/vite` failure loop.
## Out of scope