Implement closed-loop workspace follow-ups

2026-07-15 23:48:12 +04:00
parent 3a48ecd24f
commit ed7efb6072
51 changed files with 702 additions and 54 deletions
@@ -0,0 +1,32 @@
+# QA Plan: Stage prompt audition
+
+**Status:** DRAFT
+**Run date / operator:** Not run — requires a fixed local model and seed.
+**BACKLOG item:** Vikunja #169
+
+## Preconditions
+
+- [ ] Record the model id, model checksum, `context_size`, and sampling config.
+- [ ] Use one fixed workspace fixture and deterministic seed per variant.
+- [ ] Run `scripts/prompt-audit.sh` and retain its Markdown output with the run artifacts.
+
+## Acceptance gate
+
+> A curated prompt variant is adopted only if it improves gate pass rate or wasted tool rounds without increasing context truncation on the same model and task fixture.
+
+## Checks
+
+| # | Action | Expected observable evidence | Result |
+|---|---|---|---|
+| 1 | Run baseline, trimmed, curated-rules, and scope-reworded variants. | Session event logs identify every stage inference and terminal workflow outcome. | Not run |
+| 2 | Compare gate pass rate, rejected/wasted tool rounds, retries, token budget use, and truncation. | Per-variant result table derived from events; same fixture and seed. | Not run |
+| 3 | Select the smallest improving variant. | A committed finding names the selected variant and evidence; otherwise no prompt expansion lands. | Not run |
+
+## Current static audit finding
+
+The runtime currently has two stage-facing sources: workflow prompt files and the compact `CURATED_STAGE_OPERATING_GUIDANCE` L0 system entry. Retry feedback remains event-derived and only the latest failure is injected. The static inventory is reproducible with `scripts/prompt-audit.sh`; no performance claim is made until the controlled run above is recorded.
+
+## Out of scope
+
+- A live benchmark result; this repository has no fixed model artifact or benchmark fixture checked in.
+- Replacing deterministic gates with prompt text.