12ff2e97b0
Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks: research egress + batch source-approval (D), architect contradiction (B§4), llama health probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in production (it breaks the planner if the model can't emit a parseable brief_echo). Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/ (sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are set -euo pipefail, syntax-checked, executable. Authored from the repo build/config (server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.7 KiB
3.7 KiB
QA Plan: architect contradiction-check (display-only) — 4e08510 eae0a0c
Status: DRAFT
Run date / operator:
BACKLOG item: §B-§4 "architect contradiction-check" — checker eae0a0c, live server hook 4e08510.
Preconditions
- server build/branch:
feat/backlog-burndown. - llama-server + model: a model that runs the
role_pipeline(analyst→architect→…) and emits adesignartifact with a populatedapproachfield (seedocs/schemas/design.json). - external deps: none beyond the model.
- config synced:
[project] enabled = true(the checker queries theproject:<repoRoot>L3 namespace thatProjectMemoryService.persistwrites);[router.embedder] backend = "llamacpp"with a real embedder URL —noopembeddings make every distance meaningless and the gate never fires.[router.l3] backend = "in_memory"is fine for a single run, but note: in-memory L3 is process-lifetime, so the two sessions below must run against the same server process. - fixtures/seed: the same workspace/repo root for both sessions (the namespace is
project:<repoRoot>).
Acceptance gate (one sentence)
When a later session's architect
designdecision is semantically near a prior session's recorded decision, aPossibleContradictionFlaggedEventis appended — and it never halts the stage.
Checks
| # | Action | Expected observable evidence | Result |
|---|---|---|---|
| 1 | Run session A end-to-end on a repo; architect commits a clear decision (e.g. "use Postgres for persistence"). Let the session complete (distillation runs on completion). | Session A completes; correx events {A} shows the architect design ArtifactCreatedEvent. The decision is now distilled into L3 under project:<repoRoot> (ProjectMemoryService.persist on completion). |
|
| 2 | Run session B (same server process, same repo) whose architect decision reverses A (e.g. "use SQLite / drop Postgres"). | When B's architect design artifact is created, correx events {B} contains a PossibleContradictionFlaggedEvent — decisionSummary = B's approach, related[] lists A's decision with a score ≥ 0.75 and source = the project:<repoRoot> turnId. |
|
| 3 | Confirm non-interference. | Session B's architect stage still passes and proceeds to the planner — the flag is display-only (no WorkflowFailedEvent, no stage halt around the flag). |
|
| 4 | Run session C with an architect decision unrelated to A (different domain). | No PossibleContradictionFlaggedEvent for C (nothing clears the 0.75 threshold). Absence is the evidence. |
|
| 5 | Confirm self-exclusion. | The flag in check 2 never lists B's own in-flight decision as related (checker filters entry.sessionId != current). |
Evidence sources: correx events {id}, server logs (the [Orchestrator]/checker warn line on a near hit is informative but the event is the gate).
Out of scope (explicitly NOT covered this pass)
- In-session contradiction (two decisions inside one session) — by design the checker filters to PRIOR sessions; not built and not required.
- LLM-judged contradiction — v1 is similarity-only, display-only.
- ADR
alternativesConsideredshape (BACKLOG marks this deliberately not taken).
Disposition
- PASS → move the §B-§4 entry to
RETRO.mdwith this run date + thePossibleContradictionFlaggedEventevidence from check 2. Status: PASSED. - FAIL → file numbered findings in
BACKLOG.md. Most likely failure = embedder isnoopor the two sessions ran against different processes (in-memory L3 lost) — fix the env, not the code, then re-run. Status: FAILED until green.