kami
12ff2e97b0
Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks: research egress + batch source-approval (D), architect contradiction (B§4), llama health probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in production (it breaks the planner if the model can't emit a parseable brief_echo). Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/ (sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are set -euo pipefail, syntax-checked, executable. Authored from the repo build/config (server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
43 lines
3.7 KiB
Markdown
43 lines
3.7 KiB
Markdown
# QA Plan: architect contradiction-check (display-only) — `4e08510` `eae0a0c`
|
|
|
|
**Status:** DRAFT
|
|
**Run date / operator:**
|
|
**BACKLOG item:** §B-§4 "architect contradiction-check" — checker `eae0a0c`, live server hook `4e08510`.
|
|
|
|
---
|
|
|
|
## Preconditions
|
|
|
|
- [ ] **server build/branch:** `feat/backlog-burndown`.
|
|
- [ ] **llama-server + model:** a model that runs the `role_pipeline` (analyst→architect→…) and emits a `design` artifact with a populated `approach` field (see `docs/schemas/design.json`).
|
|
- [ ] **external deps:** none beyond the model.
|
|
- [ ] **config synced:** `[project] enabled = true` (the checker queries the `project:<repoRoot>` L3 namespace that `ProjectMemoryService.persist` writes); **`[router.embedder] backend = "llamacpp"`** with a real embedder URL — `noop` embeddings make every distance meaningless and the gate never fires. `[router.l3] backend = "in_memory"` is fine for a single run, but **note**: in-memory L3 is process-lifetime, so the two sessions below must run against the **same** server process.
|
|
- [ ] **fixtures/seed:** the same workspace/repo root for both sessions (the namespace is `project:<repoRoot>`).
|
|
|
|
## Acceptance gate (one sentence)
|
|
|
|
> When a later session's architect `design` decision is semantically near a **prior** session's recorded decision, a `PossibleContradictionFlaggedEvent` is appended — and it never halts the stage.
|
|
|
|
## Checks
|
|
|
|
| # | Action | Expected observable evidence | Result |
|
|
|---|--------|------------------------------|--------|
|
|
| 1 | Run **session A** end-to-end on a repo; architect commits a clear decision (e.g. "use Postgres for persistence"). Let the session **complete** (distillation runs on completion). | Session A completes; `correx events {A}` shows the architect `design` `ArtifactCreatedEvent`. The decision is now distilled into L3 under `project:<repoRoot>` (ProjectMemoryService.persist on completion). | |
|
|
| 2 | Run **session B** (same server process, same repo) whose architect decision **reverses** A (e.g. "use SQLite / drop Postgres"). | When B's architect `design` artifact is created, `correx events {B}` contains a `PossibleContradictionFlaggedEvent` — `decisionSummary` = B's `approach`, `related[]` lists A's decision with a `score ≥ 0.75` and `source` = the `project:<repoRoot>` turnId. | |
|
|
| 3 | Confirm non-interference. | Session B's architect stage **still passes** and proceeds to the planner — the flag is display-only (no `WorkflowFailedEvent`, no stage halt around the flag). | |
|
|
| 4 | Run **session C** with an architect decision unrelated to A (different domain). | **No** `PossibleContradictionFlaggedEvent` for C (nothing clears the 0.75 threshold). Absence is the evidence. | |
|
|
| 5 | Confirm self-exclusion. | The flag in check 2 never lists **B's own** in-flight decision as `related` (checker filters `entry.sessionId != current`). | |
|
|
|
|
Evidence sources: `correx events {id}`, server logs (the `[Orchestrator]`/checker warn line on a near hit is informative but the **event** is the gate).
|
|
|
|
## Out of scope (explicitly NOT covered this pass)
|
|
|
|
- In-session contradiction (two decisions inside one session) — by design the checker filters to PRIOR sessions; not built and not required.
|
|
- LLM-judged contradiction — v1 is similarity-only, display-only.
|
|
- ADR `alternativesConsidered` shape (BACKLOG marks this deliberately not taken).
|
|
|
|
## Disposition
|
|
|
|
- **PASS** → move the §B-§4 entry to `RETRO.md` with this run date + the `PossibleContradictionFlaggedEvent` evidence from check 2. Status: PASSED.
|
|
- **FAIL** → file numbered findings in `BACKLOG.md`. Most likely failure = embedder is `noop` or the two sessions ran against different processes (in-memory L3 lost) — fix the env, not the code, then re-run. Status: FAILED until green.
|