From 12ff2e97b0b8cf063bd48baff43f964de95b31a4 Mon Sep 17 00:00:00 2001 From: kami Date: Sun, 21 Jun 2026 09:14:36 +0000 Subject: [PATCH] docs(qa): live-QA plans + env bring-up for the burndown features MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks: research egress + batch source-approval (D), architect contradiction (B§4), llama health probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in production (it breaks the planner if the model can't emit a parseable brief_echo). Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/ (sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are set -euo pipefail, syntax-checked, executable. Authored from the repo build/config (server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml). Co-Authored-By: Claude Opus 4.8 --- docs/qa/ENV.md | 126 ++++++++++++++++++++++++++ docs/qa/QA-architect-contradiction.md | 42 +++++++++ docs/qa/QA-brief-echo-gate.md | 52 +++++++++++ docs/qa/QA-idea-promotion.md | 44 +++++++++ docs/qa/QA-llama-health-probe.md | 41 +++++++++ docs/qa/QA-research-egress.md | 42 +++++++++ docs/qa/QA-reviewer-static-first.md | 44 +++++++++ docs/qa/README.md | 33 +++++++ 8 files changed, 424 insertions(+) create mode 100644 docs/qa/ENV.md create mode 100644 docs/qa/QA-architect-contradiction.md create mode 100644 docs/qa/QA-brief-echo-gate.md create mode 100644 docs/qa/QA-idea-promotion.md create mode 100644 docs/qa/QA-llama-health-probe.md create mode 100644 docs/qa/QA-research-egress.md create mode 100644 docs/qa/QA-reviewer-static-first.md create mode 100644 docs/qa/README.md diff --git a/docs/qa/ENV.md b/docs/qa/ENV.md new file mode 100644 index 00000000..eb87b882 --- /dev/null +++ b/docs/qa/ENV.md @@ -0,0 +1,126 @@ +# Correx live-QA environment — bring-up runbook + +The plans in `docs/qa/QA-*.md` cite **observable signals** (events, logs, files, TUI +renders). This runbook stands up the stack that produces them. Run it on a capable box +(JDK 21, Go ≥ 1.24, Docker, a GPU + GGUF model for any model-dependent plan). + +Helper scripts live in `scripts/qa/` (see `scripts/qa/README.md`). + +> **Low-RAM note:** a machine-local `~/.gradle/gradle.properties` on the dev box caps the +> Gradle/Kotlin heap (`-Xmx768m`, `parallel=false`) for a 1.9 GB machine. It is **not +> committed**. On a QA box with real RAM, delete it (or ignore it) so builds aren't throttled. + +--- + +## Prerequisites + +- **JDK 21** (`java -version` → 21). +- **Go ≥ 1.24** — the TUI builds via `GOTOOLCHAIN=auto` (go.mod pins `go 1.24.2`); the + toolchain auto-downloads if your `go` is older. +- **Docker** — only for SearXNG (research-workflow QA). +- **llama-server** binary + at least one **GGUF model** — for any model-dependent plan. +- (optional) an **embedder** model/endpoint — required for the architect-contradiction plan + and any L3-retrieval-dependent behavior (`[router.embedder] backend = "llamacpp"`). + +--- + +## 1. Config + +Correx reads `~/.config/correx/config.toml` plus `workflows/`, `prompts/`, `schemas/` +alongside it. Stale copies cause silent bugs (BACKLOG ops note), so **sync from the repo**: + +```bash +scripts/qa/sync-config.sh # copies examples/workflows + prompts + docs/schemas + # and seeds config.toml from docs/sample-config.toml IF ABSENT +``` + +Then edit `~/.config/correx/config.toml` for the plan you're running: + +- **Model** — pick ONE: + - *Managed* (`[[models]]` + `[models]`): correx spawns llama-server at `[models].host:port` + (default `127.0.0.1:10000`) and kills it on shutdown. Simplest. + - *Static* (`[[providers]]`, type `llamacpp`, `url`): you run llama-server yourself. **Use + this for the llama-health plan** so you can stop the server independently of correx. +- **`[project] enabled = true`** — required for project memory + the **architect-contradiction** + plan (it queries the `project:` L3 namespace written at session end). +- **`[router.embedder] backend = "llamacpp"`** (not `noop`) — required wherever embeddings + matter (architect contradiction, L3 retrieval). `noop` makes every distance meaningless. +- **`[tools.research] enabled = true` + `searxng_url`** — for the research/egress plan. +- **`[health] enabled = true`** — for the llama-health plan; note `interval_ms` (probe cadence). + +A `[[artifacts]]` entry must exist for every LLM-emitted artifact kind a workflow `produces` +(schemas under `~/.config/correx/schemas/`). `sync-config.sh` copies the schemas; confirm the +`[[artifacts]]` table in `config.toml` references the kinds your workflow uses. + +## 2. llama-server + +- **Managed:** nothing to start — correx launches it. The health probe pings + `http://<[models].host>:<[models].port>/health`. +- **External/static:** start your llama-server on the `url` host:port; its `/health` must 2xx. + ```bash + llama-server -m ~/models/.gguf --host 127.0.0.1 --port 10000 + ``` + +## 3. SearXNG (research-workflow QA only) + +```bash +scripts/qa/searxng-up.sh # http://localhost:8888, JSON format enabled +# ... run research QA ... +scripts/qa/searxng-down.sh +``` + +> **Gotcha:** SearXNG ships with only the `html` output format; the research `web_search` +> needs the **JSON API**. `searxng-up.sh` writes a `settings.yml` enabling `formats: [html, json]` +> and mounts it. Without that, `web_search` returns nothing and the workflow stalls. + +## 4. Start the server + +```bash +./gradlew :apps:server:run # mainClass com.correx.apps.server.MainKt, listens on :8080 +# or build a runnable dist once and reuse it: +./gradlew :apps:server:installDist +apps/server/build/install/server/bin/server +``` + +> **Gotcha:** do **not** rebuild jars while the server JVM is running — lazy classloading +> breaks (`NoClassDefFoundError`). Stop the server first, rebuild, restart. + +## 5. Start the TUI + +```bash +cd apps/tui-go +GOTOOLCHAIN=auto go build -o correx-tui . +./correx-tui -host localhost -port 8080 # flags default to localhost:8080 +``` + +## 6. Evidence tools (what the plans cite) + +| Source | Command / where | +|--------|-----------------| +| Event log (the truth) | `./gradlew :apps:cli:run --args="events "` or `GET /sessions/{id}/events` | +| Deterministic replay | `./gradlew :apps:cli:run --args="replay "` | +| System health | `./gradlew :apps:cli:run --args="health"` | +| Server logs | stdout of `:apps:server:run` — MDC carries `sessionId` | +| Files on disk | e.g. `/.correx/project.toml` (idea-promotion plan) | +| TUI render | the running `correx-tui` | + +(Once `:apps:cli:installDist` is built, `apps/cli/build/install/cli/bin/cli events ` works too.) + +## Teardown + +1. Stop the server (Ctrl-C) — a managed `[[models]]` llama-server is killed by the shutdown hook. +2. `scripts/qa/searxng-down.sh` if you started it. +3. External llama-server: stop it yourself. + +--- + +## Per-plan env matrix + +| QA plan | model | embedder=llamacpp | SearXNG | `project.enabled` | workspace `.correx/` | +|---------|:-----:|:-----------------:|:-------:|:-----------------:|:--------------------:| +| `QA-research-egress` | yes (tool-calling) | no | **yes** | no | no | +| `QA-architect-contradiction` | yes | **yes** | no | **yes** | repo root, 2 sessions same process | +| `QA-llama-health-probe` | yes (static path to kill) | no | no | no | no | +| `QA-idea-promotion` | router only (or seed event) | no | no | no | **yes** (bound workspace) | +| `QA-reviewer-static-first` | yes | no | no | no | seed `StaticFindingsRecordedEvent` | +| `QA-brief-echo-gate` | yes (the prod candidate) | no | no | no | analyst brief with criteria | diff --git a/docs/qa/QA-architect-contradiction.md b/docs/qa/QA-architect-contradiction.md new file mode 100644 index 00000000..81ec854a --- /dev/null +++ b/docs/qa/QA-architect-contradiction.md @@ -0,0 +1,42 @@ +# QA Plan: architect contradiction-check (display-only) — `4e08510` `eae0a0c` + +**Status:** DRAFT +**Run date / operator:** +**BACKLOG item:** §B-§4 "architect contradiction-check" — checker `eae0a0c`, live server hook `4e08510`. + +--- + +## Preconditions + +- [ ] **server build/branch:** `feat/backlog-burndown`. +- [ ] **llama-server + model:** a model that runs the `role_pipeline` (analyst→architect→…) and emits a `design` artifact with a populated `approach` field (see `docs/schemas/design.json`). +- [ ] **external deps:** none beyond the model. +- [ ] **config synced:** `[project] enabled = true` (the checker queries the `project:` L3 namespace that `ProjectMemoryService.persist` writes); **`[router.embedder] backend = "llamacpp"`** with a real embedder URL — `noop` embeddings make every distance meaningless and the gate never fires. `[router.l3] backend = "in_memory"` is fine for a single run, but **note**: in-memory L3 is process-lifetime, so the two sessions below must run against the **same** server process. +- [ ] **fixtures/seed:** the same workspace/repo root for both sessions (the namespace is `project:`). + +## Acceptance gate (one sentence) + +> When a later session's architect `design` decision is semantically near a **prior** session's recorded decision, a `PossibleContradictionFlaggedEvent` is appended — and it never halts the stage. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Run **session A** end-to-end on a repo; architect commits a clear decision (e.g. "use Postgres for persistence"). Let the session **complete** (distillation runs on completion). | Session A completes; `correx events {A}` shows the architect `design` `ArtifactCreatedEvent`. The decision is now distilled into L3 under `project:` (ProjectMemoryService.persist on completion). | | +| 2 | Run **session B** (same server process, same repo) whose architect decision **reverses** A (e.g. "use SQLite / drop Postgres"). | When B's architect `design` artifact is created, `correx events {B}` contains a `PossibleContradictionFlaggedEvent` — `decisionSummary` = B's `approach`, `related[]` lists A's decision with a `score ≥ 0.75` and `source` = the `project:` turnId. | | +| 3 | Confirm non-interference. | Session B's architect stage **still passes** and proceeds to the planner — the flag is display-only (no `WorkflowFailedEvent`, no stage halt around the flag). | | +| 4 | Run **session C** with an architect decision unrelated to A (different domain). | **No** `PossibleContradictionFlaggedEvent` for C (nothing clears the 0.75 threshold). Absence is the evidence. | | +| 5 | Confirm self-exclusion. | The flag in check 2 never lists **B's own** in-flight decision as `related` (checker filters `entry.sessionId != current`). | | + +Evidence sources: `correx events {id}`, server logs (the `[Orchestrator]`/checker warn line on a near hit is informative but the **event** is the gate). + +## Out of scope (explicitly NOT covered this pass) + +- In-session contradiction (two decisions inside one session) — by design the checker filters to PRIOR sessions; not built and not required. +- LLM-judged contradiction — v1 is similarity-only, display-only. +- ADR `alternativesConsidered` shape (BACKLOG marks this deliberately not taken). + +## Disposition + +- **PASS** → move the §B-§4 entry to `RETRO.md` with this run date + the `PossibleContradictionFlaggedEvent` evidence from check 2. Status: PASSED. +- **FAIL** → file numbered findings in `BACKLOG.md`. Most likely failure = embedder is `noop` or the two sessions ran against different processes (in-memory L3 lost) — fix the env, not the code, then re-run. Status: FAILED until green. diff --git a/docs/qa/QA-brief-echo-gate.md b/docs/qa/QA-brief-echo-gate.md new file mode 100644 index 00000000..fe67c5c5 --- /dev/null +++ b/docs/qa/QA-brief-echo-gate.md @@ -0,0 +1,52 @@ +# QA Plan: brief echo-back gate — ARM-IT plan — `1df7af5` + +**Status:** DRAFT +**Run date / operator:** +**BACKLOG item:** §C-A1 "brief echo-back gate" — gate `1df7af5`. **This plan is the precondition for arming the gate in production.** + +--- + +## Why this plan gates the activation + +The gate code (`checkBriefEcho`) is built and unit-tested but **OFF by default** (`briefEcho != "true"`). Arming it is a config/prompt change that is **model-dependent and not safe to flip blind**: once on, the planner stage requires a parseable `brief_echo` block in its produced artifact. If the planner model does **not** emit one, `checkBriefEcho` returns a **retryable Failure** ("must restate the brief as a brief_echo block") — which, on a model that never complies, **exhausts retries and fails the planner stage**, breaking `role_pipeline`. So: prove the model reliably emits a parseable block **here**, on a QA box, before turning it on anywhere real. + +## How to arm it (do this on the QA box only) + +1. **Prompt** — edit `~/.config/correx/prompts/planner.md` (synced from `examples/workflows/prompts/`) to instruct the model to begin its `impl_plan` artifact with a `brief_echo` object restating the brief, exactly: + ```json + {"brief_echo": {"referenced_files": ["..."], "symbols": ["..."], "acceptance_criteria": ["..."]}, "...rest of impl_plan...": ...} + ``` + (Parsed by `BriefEchoExtractor.fromEcho` — root key `brief_echo`, arrays `referenced_files` / `symbols` / `acceptance_criteria`.) +2. **Workflow metadata** — on the **planner** stage in `~/.config/correx/workflows/role_pipeline.toml`, set stage metadata `briefEcho = "true"` and `briefEchoSource = "analysis"` (the analyst's artifact slot is named `analysis`, not the default `brief`). The original brief's referenced files are pulled by `BriefReferenceExtractor`; optional `acceptance_criteria` / `symbols` arrays on the analysis artifact make the comparison meaningful. + +## Preconditions + +- [ ] **server build/branch:** `feat/backlog-burndown`. +- [ ] **llama-server + model:** the production candidate model (this plan is specifically testing *that model's* compliance). +- [ ] **config synced + armed** as above. +- [ ] **fixtures/seed:** an analyst `analysis` artifact that names concrete files + acceptance criteria, so divergence is detectable. + +## Acceptance gate (one sentence) + +> With the gate armed, the planner reliably emits a **parseable** `brief_echo`, the stage **passes** when the echo matches the brief, and **only** diverging echoes produce a `BriefEchoMismatchEvent` + retryable failure — never a "missing block" failure on a compliant run. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Run `role_pipeline` to the planner stage with a faithful brief. | The planner's `impl_plan` artifact contains a parseable `brief_echo`; the stage **passes** (proceeds to implementer). **No** `BriefEchoMismatchEvent`, no "must restate the brief" failure. | | +| 2 | Run it **3–5 times** across varied briefs. | The model emits a parseable `brief_echo` **every** time (reliability bar). Tally pass rate — anything < 100% means arming will intermittently break the planner. | | +| 3 | Seed a brief with a clear acceptance criterion, then induce the planner to drop it (or hand-edit the echo to omit it). | A `BriefEchoMismatchEvent` with `droppedAcceptanceCriteria` set; stage Failure (retryable) whose message restates the dropped criterion. | | +| 4 | Induce the echo to reference a file **not** in the brief. | `BriefEchoMismatchEvent.inventedFiles` set; failure message names the invented path. | | +| 5 | Temporarily make the planner emit **no** `brief_echo` block. | Failure "must restate the brief as a brief_echo block before planning" — confirming the strict-on-armed behavior (this is the break mode check 2 must rule out for the real model). | | + +Evidence sources: `correx events {id}` (`BriefEchoMismatchEvent`), the produced `impl_plan` artifact content, server logs (`[Orchestrator] brief echo diverged`). + +## Out of scope + +- Symbol grounding precision (deferred; file-path + acceptance-criteria grounding only). + +## Disposition + +- **PASS (check 2 == 100%)** → it is safe to arm in production; move §C-A1 to `RETRO.md` with the reliability tally. Keep the armed `planner.md` + `role_pipeline.toml` changes (or land them into `examples/` if the repo defaults should ship armed). +- **FAIL (check 2 < 100%, or check 5 fires on a normal run)** → **do NOT arm in production.** File the model's non-compliance as a finding; either improve the prompt and re-run, or leave the gate OFF. Status: FAILED until check 2 is reliably green. diff --git a/docs/qa/QA-idea-promotion.md b/docs/qa/QA-idea-promotion.md new file mode 100644 index 00000000..fd3f3f2b --- /dev/null +++ b/docs/qa/QA-idea-promotion.md @@ -0,0 +1,44 @@ +# QA Plan: idea-board → project-profile promotion — `f107ff5` + +**Status:** DRAFT +**Run date / operator:** +**BACKLOG item:** §E "Idea board → profile promotion". + +--- + +## Preconditions + +- [ ] **server build/branch:** `feat/backlog-burndown`. +- [ ] **llama-server + model:** a router model only if you capture the idea via the rubber-duck path (it must emit a ```` ```json {"ideas":[...]} ``` ```` block). To avoid model-dependence you may instead **seed an `IdeaCapturedEvent`** directly (see note below). +- [ ] **external deps:** none. +- [ ] **config synced:** standard. +- [ ] **fixtures/seed:** a session bound to a **workspace** (a `SessionWorkspaceBoundEvent` must exist for the capturing session — promotion writes into `/.correx/project.toml` and no-ops without a bound workspace). Start the TUI/session from inside the target repo dir so the workspace resolves. + +## Acceptance gate (one sentence) + +> Promoting a captured idea appends its text as a `conventions` entry in `/.correx/project.toml`, records an `IdeaPromotedEvent`, and removes the idea from the board. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Capture an idea (rubber-duck `{"ideas":[...]}`, or seed an `IdeaCapturedEvent`). List the board (TUI `R`/idea overlay, or `ListIdeas` WS). | The idea shows on the board with a stable `ideaId`; `IdeaCapturedEvent` in the log. | | +| 2 | Note the pre-state of `/.correx/project.toml` (may be absent). | File absent or its current `conventions = [...]` recorded. | | +| 3 | Send `PromoteIdea(ideaId)` (TUI promote key, or the WS `PromoteIdea` message). | `/.correx/project.toml` now exists and its `conventions` list contains the idea text **verbatim** (escaped if it had quotes). Re-reading via `ProjectProfileLoader` round-trips it (no corruption). | | +| 4 | Re-read the log and the board. | An `IdeaPromotedEvent(ideaId, sessionId, text, timestampMs)` is appended; the idea **no longer appears** on the board (`activeIdeas` tombstones promoted ids like discards). | | +| 5 | Promote the **same** idea text again from a second idea (dedup). | `withConvention` does not duplicate an identical convention — the list has it once. | | +| 6 | Promote with **no bound workspace** (start a session with an unresolved working dir). | No file write, no `IdeaPromotedEvent` — the handler no-ops and just refreshes the board (safe). | | + +> **Seeding without a model:** append an `IdeaCapturedEvent(ideaId, sessionId, text, timestampMs)` for a workspace-bound session via the event store, then drive `PromoteIdea`. This isolates the promotion path from router-prompt behavior. + +Evidence sources: the file `/.correx/project.toml` on disk, `correx events {id}`, the TUI board render. + +## Out of scope (explicitly NOT covered this pass) + +- The rubber-duck capture quality itself (covered by the separate idea-board live-QA, §F). +- Promotion into `about`/`commands` — v1 promotes into `conventions` only. + +## Disposition + +- **PASS** → move the §E "Idea board → profile promotion" bullet to `RETRO.md` with this run date + the on-disk `.correx/project.toml` evidence. Status: PASSED. +- **FAIL** → numbered findings (e.g. quote-escaping corruption, board not clearing). Status: FAILED until green. diff --git a/docs/qa/QA-llama-health-probe.md b/docs/qa/QA-llama-health-probe.md new file mode 100644 index 00000000..a17fcd2e --- /dev/null +++ b/docs/qa/QA-llama-health-probe.md @@ -0,0 +1,41 @@ +# QA Plan: LLAMA_SERVER health probe — `266bbf0` `64a90e7` `9eef936` + +**Status:** DRAFT +**Run date / operator:** +**BACKLOG item:** §A-§4 "LLAMA_SERVER health probe" — probe `266bbf0`, registration `64a90e7`, managed-path coverage `9eef936`. + +--- + +## Preconditions + +- [ ] **server build/branch:** `feat/backlog-burndown`. +- [ ] **llama-server + model:** a real llama-server reachable at the configured URL. The probe registers when **either** a static `[[providers]]` (type `llamacpp`, `url`) **or** a managed `[[models]]` entry exists (managed URL = `[models].host:[models].port`). Verify your config has one. +- [ ] **external deps:** none. +- [ ] **config synced:** `[health] enabled = true`; note `[health] interval_ms` (the probe cadence — you wait one interval to see an edge). +- [ ] **fixtures/seed:** ability to stop/start the llama-server process (managed: it dies with the server; for the kill/restore test use the **static `[[providers]]`** path so you can stop llama-server independently of correx). + +## Acceptance gate (one sentence) + +> The `LLAMA_SERVER` subject is monitored only when a model is configured, reports HEALTHY while the server answers `/health`, and emits a **liveness DEGRADED** edge when the server stops — and a restored edge when it returns. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Start correx with a llama-server configured and reachable. Wait one `interval_ms`. | `correx health` (and the health projection) lists `LLAMA_SERVER` as HEALTHY. No spurious DEGRADED. | | +| 2 | **Stop** the llama-server (static-provider path: kill the external process). Wait one `interval_ms`. | A health **degraded** edge event for subject `LLAMA_SERVER`, `metric = "liveness"`, detail "llama server unreachable" appears in the system-health log (`correx events` on the SYSTEM session / `correx health`). | | +| 3 | **Restart** the llama-server. Wait one `interval_ms`. | A **restored** edge event for `LLAMA_SERVER`; `correx health` shows HEALTHY again. Edges are recorded once (edge-triggered, not per-tick). | | +| 4 | Start correx on a box with **no** `[[providers]]` and **no** `[[models]]`. | `LLAMA_SERVER` is **absent** from the health subjects — the probe is not registered (no spurious DEGRADED). Absence is the evidence. | | +| 5 | `correx replay` the SYSTEM health session. | The degraded/restored edges replay deterministically from the log (the rolling window is live-only per Invariant #8 and does not affect replay). | | + +Evidence sources: `correx health` CLI, the health projection events, server logs. + +## Out of scope (explicitly NOT covered this pass) + +- **tokens/sec degradation** — the `/health` endpoint is reachability-only; `HttpLlamaLivenessClient` never populates `tokensPerSecond`, so the throughput-collapse branch of `LlamaServerHealthProbe` is dormant until a tps telemetry feed exists. Liveness only this pass. (Refiled in BACKLOG §A-§4 remainder.) +- The live health **TUI pane** (deferred; system-health events are not yet on the TUI wire). + +## Disposition + +- **PASS** → move the §A-§4 LLAMA_SERVER bullet to `RETRO.md` (liveness verified) with this run date; keep the tokens/sec remainder open in BACKLOG. Status: PASSED. +- **FAIL** → numbered findings. A likely miss: the managed `[[models]]` path not registering — confirm `9eef936` is in the build. Status: FAILED until green. diff --git a/docs/qa/QA-research-egress.md b/docs/qa/QA-research-egress.md new file mode 100644 index 00000000..000932ef --- /dev/null +++ b/docs/qa/QA-research-egress.md @@ -0,0 +1,42 @@ +# QA Plan: research egress allowlist + batch source-list approval — `ab7e4be` `4b0024f` `b098d87` `027ff1f` + +**Status:** DRAFT +**Run date / operator:** +**BACKLOG item:** §D — "Dynamic per-session egress allowlist", "§3 batch fetch-approval at source-list level", "Dedicated `SourceFetched` / `LowQualityExtraction` events", plus the §D "Live-QA the full run" gate. + +--- + +## Preconditions + +- [ ] **server build/branch:** `feat/backlog-burndown` — _rebuild only with the server stopped (lazy classloading breaks otherwise)_ +- [ ] **llama-server + model:** a tool-calling-capable model (the research workflow drives `web_search`/`web_fetch` tool calls). Managed `[[models]]` or static `[[providers]]`. +- [ ] **external deps:** **SearXNG** at `[tools.research] searxng_url` with the JSON format enabled (`scripts/qa/searxng-up.sh`; see `docs/qa/ENV.md` §3). Outbound network for the fetched hosts. +- [ ] **config synced:** `~/.config/correx` matches `examples/*` + `docs/schemas/*` (`scripts/qa/sync-config.sh`); `research.toml` present; `[tools.research] enabled = true`. +- [ ] **fixtures/seed:** a research prompt that yields ≥2 distinct source hosts (e.g. "summarize the latest on citing official docs"). + +## Acceptance gate (one sentence) + +> Approving a source list once clears every subsequent `web_fetch` to those hosts **without a per-URL T2 prompt**, and the fetch quality lands as dedicated `SourceFetched` / `LowQualityExtraction` events — while a host NOT on the list still raises T2. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Start a research session; let it reach a state with candidate source URLs. | `web_search` T1 ran; the candidate URLs appear in the session (search result / dossier event). | | +| 2 | `POST /sessions/{id}/approve-sources` with the candidate URL list (2+ hosts). | Exactly **one** `EgressHostsGrantedEvent` in `correx events {id}` — `hosts` = the distinct lower-cased hostnames (ports stripped), `reason = "batch source-list approval"`. | | +| 3 | Let the workflow `web_fetch` each approved URL. | Each fetch to an approved host proceeds with **no `ApprovalRequestedEvent` (T2)** — `NetworkHostRule` PROCEEDs via the per-session union. A `SourceFetchedEvent` (with `content_sha256` + quality) is recorded per successful fetch. | | +| 4 | Force a low-quality extraction (e.g. a near-empty / boilerplate page among the sources). | A `LowQualityExtractionEvent` is recorded (not just metadata on `ToolExecutionCompletedEvent`). | | +| 5 | Have the agent attempt a `web_fetch` to a host **not** in the approved list. | A T2 `ApprovalRequestedEvent` IS raised for that host (the allowlist is a union, not a blanket open). | | +| 6 | `correx replay {id}`. | Replay reproduces the same allowlist decisions deterministically (the grant + fetch events are the only inputs; no re-fetch). | | + +Evidence sources: `correx events {id}` / `GET /sessions/{id}/events`, server logs (MDC sessionId), `correx replay {id}`. + +## Out of scope (explicitly NOT covered this pass) + +- The standalone §D web-approval client (third client) — separate track, not built. +- Headless-browser / LLM-based extraction (non-responsibilities). + +## Disposition + +- **PASS** → move the §D egress/batch/source-event bullets to `RETRO.md` with this run date + the cited `EgressHostsGrantedEvent` / `SourceFetchedEvent` evidence. Status: PASSED. +- **FAIL** → file each failure as a numbered finding in `BACKLOG.md` (action + repro + the wrong/missing signal). Status: FAILED until green. diff --git a/docs/qa/QA-reviewer-static-first.md b/docs/qa/QA-reviewer-static-first.md new file mode 100644 index 00000000..3602f0a7 --- /dev/null +++ b/docs/qa/QA-reviewer-static-first.md @@ -0,0 +1,44 @@ +# QA Plan: reviewer static-first (findings exclusion) — `447fc7a` + +**Status:** DRAFT (PARTIAL — see gap) +**Run date / operator:** +**BACKLOG item:** §B-§5 "reviewer static-first INFRA" — runner + event + context-exclusion `447fc7a`. + +--- + +## Gap up front (read first) + +The exclusion **filter** is live-wired into reviewer context assembly, and `StaticAnalysisRunner` + `StaticFindingsRecordedEvent` are built and unit-tested — but **no stock workflow stage emits `StaticFindingsRecordedEvent` yet** (the deterministic static-check stage-type seam is an open BACKLOG follow-up). So this pass verifies the **filter behavior by injecting the event**; the end-to-end "a stage runs detekt and the reviewer skips those findings" flow is blocked on that seam. + +## Preconditions + +- [ ] **server build/branch:** `feat/backlog-burndown`. +- [ ] **llama-server + model:** a model that runs `review_loop` / `role_pipeline` reviewer stage. +- [ ] **external deps:** none. +- [ ] **config synced:** `review_loop.toml` (or role_pipeline) present. +- [ ] **fixtures/seed:** a way to append a `StaticFindingsRecordedEvent` for the session (manual event-store seed), carrying a finding that matches a compile/lint issue the code actually has. + +## Acceptance gate (one sentence) + +> When a `StaticFindingsRecordedEvent` has already recorded a static finding this session, the reviewer's assembled context **mechanically omits** any feedback entry that merely restates that finding. + +## Checks + +| # | Action | Expected observable evidence | Result | +|---|--------|------------------------------|--------| +| 1 | Seed a `StaticFindingsRecordedEvent` for the session with a known finding (e.g. an unused-import lint at `Foo.kt:12`). | The event is in `correx events {id}`. | | +| 2 | Drive the implementer→reviewer step so reviewer context is assembled. | The reviewer's input context (server log of the assembled prompt / context pack) **does not** contain a feedback entry restating the seeded static finding — `excludeStaticFindingsFromReview` stripped it. | | +| 3 | Seed a finding that does **not** match any retry-feedback entry. | Non-matching entries are untouched; only restatements of recorded static findings are dropped. | | +| 4 | (When the static-check stage seam lands) run a workflow whose static stage emits the event for real. | The reviewer never re-raises the compiler/detekt issues the deterministic tool already caught. _(Deferred — seam not built.)_ | | + +Evidence sources: server logs of the assembled reviewer context / context pack, `correx events {id}`. + +## Out of scope (explicitly NOT covered this pass) + +- The **producing** static-check stage (open BACKLOG §B-§5 follow-up: "a stage that emits `StaticFindingsRecordedEvent`"). Check 4 stays deferred until that seam exists. +- The reviewer prompt half (shipped earlier, `3467826`). + +## Disposition + +- **PASS (partial)** → record in `RETRO.md` that the **filter** is live-verified (checks 1–3) and keep the §B-§5 *producing-stage* follow-up open in BACKLOG. Do not claim the full static-first loop until check 4 passes. +- **FAIL** → numbered findings. Status: FAILED until green. diff --git a/docs/qa/README.md b/docs/qa/README.md new file mode 100644 index 00000000..00829ff8 --- /dev/null +++ b/docs/qa/README.md @@ -0,0 +1,33 @@ +# Correx live-QA plans + +Each `QA-.md` is the artifact that authorizes a `BACKLOG.md` → `RETRO.md` move: +it pins **observable evidence** (events, logs, files, TUI renders) for one shipped feature. +Template: `TEMPLATE.md`. Environment bring-up: **`ENV.md`** (+ `scripts/qa/`). + +A feature is **not** "resolved" until its plan passes live. On PASS → move the BACKLOG entry +to RETRO with the run date + cited evidence; on FAIL → refile each miss as a numbered finding. + +## Plans (this burndown) + +| Plan | Feature | Commits | Needs | +|------|---------|---------|-------| +| `QA-research-egress.md` | per-session egress allowlist + batch source-list approval + dedicated source events | `ab7e4be` `4b0024f` `b098d87` `027ff1f` | model + **SearXNG** | +| `QA-architect-contradiction.md` | architect contradiction-check (display-only) | `4e08510` `eae0a0c` | model + **llamacpp embedder** + `project.enabled`, 2 sessions/1 process | +| `QA-llama-health-probe.md` | LLAMA_SERVER liveness probe | `64a90e7` `9eef936` `266bbf0` | static llama-server you can stop | +| `QA-idea-promotion.md` | idea-board → `.correx/project.toml` promotion | `f107ff5` | bound workspace (router model optional — can seed the event) | +| `QA-reviewer-static-first.md` | reviewer static-findings exclusion (PARTIAL — producing stage not built) | `447fc7a` | seed `StaticFindingsRecordedEvent` | +| `QA-brief-echo-gate.md` | **ARM-IT** plan: brief echo-back gate (off by default; risky to arm blind) | `1df7af5` | prod-candidate model | + +## Order of attack (suggested) + +1. **`QA-llama-health-probe`** + **`QA-idea-promotion`** — cheapest, least model-dependent (health is liveness; promotion can be seeded). Quick wins. +2. **`QA-research-egress`** — once SearXNG is up. +3. **`QA-architect-contradiction`** — needs a real embedder + two sessions. +4. **`QA-brief-echo-gate`** — run this **before** arming the gate anywhere real; check 2 (100% parseable-echo across runs) is the go/no-go for production. +5. **`QA-reviewer-static-first`** — partial until the static-check stage seam lands. + +## Still-open §F live-QA gates (no dedicated plan yet — same env applies) + +These predate the burndown and remain in `BACKLOG.md` §F; author a plan from `TEMPLATE.md` +when you tackle one: Epic 15 (events/replay + MDC sessionIds), idea-board capture, clarification +loop + workflow-propose panel, the full research-workflow run, TUI kernel-steering + AMD gauge.