kami
12ff2e97b0
Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks: research egress + batch source-approval (D), architect contradiction (B§4), llama health probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in production (it breaks the planner if the model can't emit a parseable brief_echo). Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/ (sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are set -euo pipefail, syntax-checked, executable. Authored from the repo build/config (server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
45 lines
3.1 KiB
Markdown
45 lines
3.1 KiB
Markdown
# QA Plan: reviewer static-first (findings exclusion) — `447fc7a`
|
||
|
||
**Status:** DRAFT (PARTIAL — see gap)
|
||
**Run date / operator:**
|
||
**BACKLOG item:** §B-§5 "reviewer static-first INFRA" — runner + event + context-exclusion `447fc7a`.
|
||
|
||
---
|
||
|
||
## Gap up front (read first)
|
||
|
||
The exclusion **filter** is live-wired into reviewer context assembly, and `StaticAnalysisRunner` + `StaticFindingsRecordedEvent` are built and unit-tested — but **no stock workflow stage emits `StaticFindingsRecordedEvent` yet** (the deterministic static-check stage-type seam is an open BACKLOG follow-up). So this pass verifies the **filter behavior by injecting the event**; the end-to-end "a stage runs detekt and the reviewer skips those findings" flow is blocked on that seam.
|
||
|
||
## Preconditions
|
||
|
||
- [ ] **server build/branch:** `feat/backlog-burndown`.
|
||
- [ ] **llama-server + model:** a model that runs `review_loop` / `role_pipeline` reviewer stage.
|
||
- [ ] **external deps:** none.
|
||
- [ ] **config synced:** `review_loop.toml` (or role_pipeline) present.
|
||
- [ ] **fixtures/seed:** a way to append a `StaticFindingsRecordedEvent` for the session (manual event-store seed), carrying a finding that matches a compile/lint issue the code actually has.
|
||
|
||
## Acceptance gate (one sentence)
|
||
|
||
> When a `StaticFindingsRecordedEvent` has already recorded a static finding this session, the reviewer's assembled context **mechanically omits** any feedback entry that merely restates that finding.
|
||
|
||
## Checks
|
||
|
||
| # | Action | Expected observable evidence | Result |
|
||
|---|--------|------------------------------|--------|
|
||
| 1 | Seed a `StaticFindingsRecordedEvent` for the session with a known finding (e.g. an unused-import lint at `Foo.kt:12`). | The event is in `correx events {id}`. | |
|
||
| 2 | Drive the implementer→reviewer step so reviewer context is assembled. | The reviewer's input context (server log of the assembled prompt / context pack) **does not** contain a feedback entry restating the seeded static finding — `excludeStaticFindingsFromReview` stripped it. | |
|
||
| 3 | Seed a finding that does **not** match any retry-feedback entry. | Non-matching entries are untouched; only restatements of recorded static findings are dropped. | |
|
||
| 4 | (When the static-check stage seam lands) run a workflow whose static stage emits the event for real. | The reviewer never re-raises the compiler/detekt issues the deterministic tool already caught. _(Deferred — seam not built.)_ | |
|
||
|
||
Evidence sources: server logs of the assembled reviewer context / context pack, `correx events {id}`.
|
||
|
||
## Out of scope (explicitly NOT covered this pass)
|
||
|
||
- The **producing** static-check stage (open BACKLOG §B-§5 follow-up: "a stage that emits `StaticFindingsRecordedEvent`"). Check 4 stays deferred until that seam exists.
|
||
- The reviewer prompt half (shipped earlier, `3467826`).
|
||
|
||
## Disposition
|
||
|
||
- **PASS (partial)** → record in `RETRO.md` that the **filter** is live-verified (checks 1–3) and keep the §B-§5 *producing-stage* follow-up open in BACKLOG. Do not claim the full static-first loop until check 4 passes.
|
||
- **FAIL** → numbered findings. Status: FAILED until green.
|