docs(backlog): mark plan-lint Slice 1 shipped, add QA plan
This commit is contained in:
+13
-6
@@ -120,12 +120,15 @@ built analog is `ExecutionPlanLockedEvent` (extend it with a candidate ref; don'
|
||||
|
||||
Slices (spec §, in build order):
|
||||
|
||||
- [ ] **Slice 1 — §5 deterministic lint** *(in progress — see plan)*. Pure Kotlin, zero inference:
|
||||
`PlanLinter` over the compiled `WorkflowGraph` → hard failures (missing file paths, unresolved
|
||||
symbols, unproduced `needs`, produces-kind not in registry, ADR-0000 violations) discard a
|
||||
candidate; soft findings (stage count, fan-out, budget, dup briefs) are scored. Records
|
||||
`PlanLintCompletedEvent` (env observations — repo-map/symbol existence — per invariant #9). Wired
|
||||
into `FreestyleDriver` before lock so it pays off **now** for single-plan freestyle. Standalone.
|
||||
- [~] **Slice 1 — §5 deterministic lint** — **SHIPPED (pending live-QA)** `68136e7`. Pure-Kotlin
|
||||
`PlanLinter` over the compiled `WorkflowGraph`, run in `FreestyleDriver` before lock; records
|
||||
`PlanLintCompletedEvent` (pure function of the recorded plan → replay-safe, no env observation),
|
||||
and a hard failure → `ExecutionPlanRejectedEvent(source="lint")` before the operator is asked or
|
||||
the plan locks. **Hard:** `unproduced_need` (real gap — the freestyle compiler never checked needs),
|
||||
`trap_state` (no path to terminal; uses terminal-reachability so verdict-gated review loops pass).
|
||||
**Soft (non-blocking):** `stage_count`, `fan_out`, `empty_brief`, `duplicate_brief`. QA gate in §F →
|
||||
`docs/qa/QA-plan-lint.md`. **Deferred to Slice 1b:** file-path grounding + symbol resolution (env;
|
||||
symbols need an index, §B §3), token-budget/ADR-bypass checks, a dedicated `:core:planning` module.
|
||||
- [ ] **Slice 2 — §4 multi-candidate round-1 + templates.** `PlanTemplate` library, template-select +
|
||||
slot-fill, GBNF-constrained, N₁=3 high-temp. `PlanCandidateGeneratedEvent`.
|
||||
- [ ] **Slice 3 — §6 critique + §7 aggregate.** One-shot GBNF critic per survivor →
|
||||
@@ -213,6 +216,10 @@ These are SHIPPED in code but prompt-/server-/network-dependent and not yet live
|
||||
gate uncommented in the copied `role_pipeline.toml`). Plan: `docs/qa/QA-static-first-reviewer.md`
|
||||
(clean→reviewer runs, non-clean→`StaticAnalysisCompleted`+retryable fail w/ output fed back,
|
||||
gate-OFF absence check, no-runner no-op WARN, replay reads recorded findings without re-running).
|
||||
- [ ] **Plan-lint (§C★ Slice 1) live-QA** — `68136e7`; logic is unit-covered, so this is confirmatory:
|
||||
needs a freestyle run reaching the plan-lock gate (+ a crafted bad plan to force the reject path).
|
||||
Plan: `docs/qa/QA-plan-lint.md` (unproduced_need/trap_state → `PlanLintCompleted`+`ExecutionPlan-
|
||||
Rejected(source=lint)`+no lock; review-loop NOT trapped; soft findings non-blocking; replay stable).
|
||||
- [ ] **TUI kernel-steering + AMD gauge fresh live-QA** — approve+note actually revising the
|
||||
*same* stage's output; AMD VRAM/RAM gauge on the real box.
|
||||
- [ ] **Unit-tested-only fixes, not live-verified:** `1a7eb05` (verdict-edge rejection),
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# QA Plan: deterministic plan lint — 68136e7
|
||||
|
||||
Plan-generation pipeline epic, Slice 1 (plan-pipeline-spec §5). Drafted from commit `68136e7`.
|
||||
|
||||
**Status:** DRAFT
|
||||
**Run date / operator:**
|
||||
**BACKLOG item:** §C★ "Slice 1 — §5 deterministic lint".
|
||||
|
||||
---
|
||||
|
||||
Note: the lint logic is **deterministic and unit-covered** (`PlanLinterTest` exercises every
|
||||
hard/soft check incl. the review-loop-vs-trap nuance; `FreestyleDriverTest` covers reject-on-hard-fail
|
||||
and record-on-clean). This live pass is **confirmatory** — it verifies the wiring fires in a real
|
||||
freestyle run and the event surfaces in the log.
|
||||
|
||||
## Preconditions
|
||||
|
||||
- [ ] **server build/branch:** master @ `68136e7` (or later) — _rebuild only with the server stopped_.
|
||||
- [ ] **llama-server + model:** a model capable of driving the freestyle planning phase to emit an
|
||||
`execution_plan` (needed to reach `FreestyleDriver.lockAndRun`). The lint itself needs no model.
|
||||
- [ ] **how to force a bad plan:** to exercise the reject path you need a plan that *compiles* but fails
|
||||
lint — i.e. a stage whose `needs` names an artifact no stage produces, or an inescapable loop. Easiest
|
||||
is a crafted `execution_plan` (seed/stub the planning output) rather than hoping the model emits one.
|
||||
- [ ] **fixtures/seed:** a freestyle session (workingDir) that reaches the plan-lock gate.
|
||||
|
||||
## Acceptance gate (one sentence)
|
||||
|
||||
> The lint is correct **iff** a compiled freestyle plan with an unproduced `needs` or an inescapable
|
||||
> (exit-less) loop is rejected with `ExecutionPlanRejectedEvent(source="lint")` — recording a
|
||||
> `PlanLintCompletedEvent` with the matching hard failure, never locking and never executing — while a
|
||||
> clean plan (including one with a legitimate verdict-gated review loop) records a passing
|
||||
> `PlanLintCompletedEvent` and proceeds, and soft findings never block.
|
||||
|
||||
## Checks
|
||||
|
||||
| # | Action | Expected observable evidence | Result |
|
||||
|---|--------|------------------------------|--------|
|
||||
| 1 | `grep PlanLintCompleted core/events/.../serialization/Serialization.kt` | `subclass(PlanLintCompletedEvent::class)` present (unregistered = silent runtime deser failure) | |
|
||||
| 2 | Run a freestyle session to a **clean** plan | `correx events <id>` shows `PlanLintCompleted` with empty `hardFailures`, then `ExecutionPlanLocked`, then execution | |
|
||||
| 3 | Drive freestyle with a plan whose stage `needs` an artifact no stage produces | `PlanLintCompleted` with a `unproduced_need` hard failure naming the stage+artifact; `ExecutionPlanRejected` `source="lint"`; **no** `ExecutionPlanLocked`; no stage executes | |
|
||||
| 4 | Drive a plan with an inescapable loop (cycle with no edge to `done`) | `PlanLintCompleted` with `trap_state` hard failure(s); rejected `source="lint"`; no lock | |
|
||||
| 5 | Drive a plan with a legit verdict-gated review loop (implementer⇄reviewer, reviewer→done) | **No** `trap_state`; `PlanLintCompleted` passes; plan locks and runs (the nuance: cycles-with-exit are fine) | |
|
||||
| 6 | Drive a clean-but-noisy plan (>4 fan-out, or two identical stage briefs) | `PlanLintCompleted` lists `fan_out`/`duplicate_brief` in `softFindings` with `score>0`, **but the plan still locks** (soft = non-blocking) | |
|
||||
| 7 | `correx replay <id>` on a session from check 3/4 | The recorded `PlanLintCompleted` is read back identically; lint is not recomputed against a changed repo (pure function of the recorded plan); double-read digest stable | |
|
||||
|
||||
Evidence sources: `correx events <id>` (the `PlanLintCompleted` + `ExecutionPlanRejected`/`Locked` chain),
|
||||
server logs (the "plan failed lint" WARN with the summary), and `correx replay <id>` for determinism.
|
||||
|
||||
## Out of scope (explicitly NOT covered this pass — later slices)
|
||||
|
||||
- File-path grounding + symbol resolution (Slice 1b — symbols need an index, BACKLOG §B §3).
|
||||
- Multi-candidate generation / templates (Slice 2), critique/aggregate (Slice 3), tournament/selection +
|
||||
the §E comparison UI (Slice 4), preference logging (Slice 5).
|
||||
- Token-budget and ADR-0000 harness-bypass soft/hard checks.
|
||||
- `role_pipeline` `impl_plan` lint (freestyle `execution_plan` only this slice).
|
||||
|
||||
## Disposition
|
||||
|
||||
- **PASS** → mark §C★ Slice 1 done; cite `68136e7` + this plan's run date/evidence. (Keep the epic open;
|
||||
the slice is one step of it — no RETRO move until the epic completes or you choose to retire the slice.)
|
||||
- **FAIL** → file each failure as a numbered finding back into `BACKLOG.md` (action + repro + wrong/missing
|
||||
signal), fix, re-run only the failed checks. Set Status: FAILED until green.
|
||||
Reference in New Issue
Block a user