Implement closed-loop workspace follow-ups

This commit is contained in:
2026-07-15 23:48:12 +04:00
parent 3a48ecd24f
commit ed7efb6072
51 changed files with 702 additions and 54 deletions
+1 -1
View File
@@ -12,7 +12,7 @@ Maintained alongside the features they describe. Any agent shipping a significan
- `architecture/` — stable, high-level architectural docs (context layers, event model, replay model, security boundaries).
- `decisions/` — ADRs (adr-NNNN-*.md). Append-only. Never delete or retroactively alter a decided ADR; write a superseding one instead.
- `qa/` — QA run plans (QA-*.md). Each maps to a shipped feature. `TEMPLATE.md` is the canonical shape. `ENV.md` describes the required live environment. `README.md` explains the QA process.
- `qa/` — QA run plans (QA-*.md). Each maps to a shipped feature or an explicitly pending controlled audition. `TEMPLATE.md` is the canonical shape. `ENV.md` describes the required live environment. `README.md` explains the QA process.
- `specs/` — feature specs by date-slug. Inputs for planned or in-flight work.
- `schemas/` — canonical JSON schemas for structured outputs (analysis, brief_echo, design, execution_plan, impl_plan). Shared across the router and validator.
- `epics/`, `modules/`, `diagrams/`, `design/`, `reviews/`, `visual/` — supporting reference material.
+32
View File
@@ -0,0 +1,32 @@
# QA Plan: Stage prompt audition
**Status:** DRAFT
**Run date / operator:** Not run — requires a fixed local model and seed.
**BACKLOG item:** Vikunja #169
## Preconditions
- [ ] Record the model id, model checksum, `context_size`, and sampling config.
- [ ] Use one fixed workspace fixture and deterministic seed per variant.
- [ ] Run `scripts/prompt-audit.sh` and retain its Markdown output with the run artifacts.
## Acceptance gate
> A curated prompt variant is adopted only if it improves gate pass rate or wasted tool rounds without increasing context truncation on the same model and task fixture.
## Checks
| # | Action | Expected observable evidence | Result |
|---|---|---|---|
| 1 | Run baseline, trimmed, curated-rules, and scope-reworded variants. | Session event logs identify every stage inference and terminal workflow outcome. | Not run |
| 2 | Compare gate pass rate, rejected/wasted tool rounds, retries, token budget use, and truncation. | Per-variant result table derived from events; same fixture and seed. | Not run |
| 3 | Select the smallest improving variant. | A committed finding names the selected variant and evidence; otherwise no prompt expansion lands. | Not run |
## Current static audit finding
The runtime currently has two stage-facing sources: workflow prompt files and the compact `CURATED_STAGE_OPERATING_GUIDANCE` L0 system entry. Retry feedback remains event-derived and only the latest failure is injected. The static inventory is reproducible with `scripts/prompt-audit.sh`; no performance claim is made until the controlled run above is recorded.
## Out of scope
- A live benchmark result; this repository has no fixed model artifact or benchmark fixture checked in.
- Replacing deterministic gates with prompt text.
@@ -0,0 +1,12 @@
# Capability accretion
Make run N+1 better through deterministic, replay-safe facts mined from validated prior runs—not model training or mutable hidden memory.
Promote a validated failure-to-fix pair by a reusable class key: gate, evaluator/toolchain kind, and normalized failure fingerprint. Store a reference to the validated fix artifact/diff and the evidence that it passed.
1. Extend the concept rails to retain `classKey`, `fixRef`, confidence, and contradiction state.
2. Deliver high-confidence facts deterministically at plan compilation (contract/gate tightening) or as a bounded stage instruction. L3 remains an instance-recall fallback.
3. Mine positive patterns separately: successful plan shapes, resolved commands, and useful retrievals.
4. Revoke or decay facts when later recorded evidence contradicts them.
Every promotion and delivery is event-derived and recorded; replay never re-mines live state.
@@ -0,0 +1,18 @@
# Closed-loop workspace
## Invariant
After every code-producing stage, the workspace has a recorded verification observation. A passing observation is the only state the next implementation stage may treat as known-good; replay reads the event rather than re-running a command.
## Seams
1. Before locking a plan, ground declared writes, referenced paths, and build prerequisites against the workspace/index. Unknown or absent prerequisites return the plan to the architect.
2. For each write-declaring stage, run the appropriate static/build command and record its result. Failures retry or route through recovery before downstream work continues.
3. Repeated hard precondition observations (for example `REFERENCE_EXISTS`) become a stage failure or bootstrap/clarification decision rather than independent rejected tool calls.
## Delivery order
- Per-write-stage build gates and truthful deferred contract verdicts (#162).
- Consume repeated build-critical `REFERENCE_EXISTS` blocks (#163): three blocks for one manifest/config path now create a recovery-eligible precondition failure.
- Ground plan prerequisites before plan lock; then use the verified observation as the next-stage planning baseline.
- Preserve recovery escalation for failures that still cannot reach a verified increment (#165).
@@ -0,0 +1,15 @@
# Stage prompt audit and audition
## Audit
Inventory each system-role/context block, its origin, token estimate, and outcome evidence. Remove redundant or stale text before adding rules. Inspect retry feedback separately: only the latest failure belongs in the prompt.
## Candidate additions
- Project build manifests, configs, and entry files are normal in-scope setup when required by the authoritative intent.
- Resolve a necessary scope edge from the authoritative intent; do not spend turns re-deciding it.
- Read/observe before writing and verify before declaring completion.
## Audition harness
Run a fixed task and seed through baseline, trimmed, curated-rules, and reworded-scope variants. Record completion, gate pass rate, rejected/wasted tool rounds, retries, token budget use, and context truncation. Choose the smallest variant that improves the 12B results; structural gates remain preferred where a deterministic mechanism exists.