kami
12ff2e97b0
Six QA plans (docs/qa/QA-*.md) pinning observable evidence for the shipped tracks: research egress + batch source-approval (D), architect contradiction (B§4), llama health probe (A§4), idea promotion (E), reviewer static-first filter (B§5, partial), and the brief-echo gate ARM-IT plan (C-A1) — the latter is the go/no-go for arming the gate in production (it breaks the planner if the model can't emit a parseable brief_echo). Plus the env: docs/qa/ENV.md runbook + docs/qa/README.md index, and scripts/qa/ (sync-config.sh, searxng-up/down.sh with the JSON-format gotcha, README). Scripts are set -euo pipefail, syntax-checked, executable. Authored from the repo build/config (server :apps:server:run, CLI :apps:cli:run, tui-go GOTOOLCHAIN=auto, sample-config.toml). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
43 lines
3.5 KiB
Markdown
43 lines
3.5 KiB
Markdown
# QA Plan: research egress allowlist + batch source-list approval — `ab7e4be` `4b0024f` `b098d87` `027ff1f`
|
|
|
|
**Status:** DRAFT
|
|
**Run date / operator:**
|
|
**BACKLOG item:** §D — "Dynamic per-session egress allowlist", "§3 batch fetch-approval at source-list level", "Dedicated `SourceFetched` / `LowQualityExtraction` events", plus the §D "Live-QA the full run" gate.
|
|
|
|
---
|
|
|
|
## Preconditions
|
|
|
|
- [ ] **server build/branch:** `feat/backlog-burndown` — _rebuild only with the server stopped (lazy classloading breaks otherwise)_
|
|
- [ ] **llama-server + model:** a tool-calling-capable model (the research workflow drives `web_search`/`web_fetch` tool calls). Managed `[[models]]` or static `[[providers]]`.
|
|
- [ ] **external deps:** **SearXNG** at `[tools.research] searxng_url` with the JSON format enabled (`scripts/qa/searxng-up.sh`; see `docs/qa/ENV.md` §3). Outbound network for the fetched hosts.
|
|
- [ ] **config synced:** `~/.config/correx` matches `examples/*` + `docs/schemas/*` (`scripts/qa/sync-config.sh`); `research.toml` present; `[tools.research] enabled = true`.
|
|
- [ ] **fixtures/seed:** a research prompt that yields ≥2 distinct source hosts (e.g. "summarize the latest on <topic> citing official docs").
|
|
|
|
## Acceptance gate (one sentence)
|
|
|
|
> Approving a source list once clears every subsequent `web_fetch` to those hosts **without a per-URL T2 prompt**, and the fetch quality lands as dedicated `SourceFetched` / `LowQualityExtraction` events — while a host NOT on the list still raises T2.
|
|
|
|
## Checks
|
|
|
|
| # | Action | Expected observable evidence | Result |
|
|
|---|--------|------------------------------|--------|
|
|
| 1 | Start a research session; let it reach a state with candidate source URLs. | `web_search` T1 ran; the candidate URLs appear in the session (search result / dossier event). | |
|
|
| 2 | `POST /sessions/{id}/approve-sources` with the candidate URL list (2+ hosts). | Exactly **one** `EgressHostsGrantedEvent` in `correx events {id}` — `hosts` = the distinct lower-cased hostnames (ports stripped), `reason = "batch source-list approval"`. | |
|
|
| 3 | Let the workflow `web_fetch` each approved URL. | Each fetch to an approved host proceeds with **no `ApprovalRequestedEvent` (T2)** — `NetworkHostRule` PROCEEDs via the per-session union. A `SourceFetchedEvent` (with `content_sha256` + quality) is recorded per successful fetch. | |
|
|
| 4 | Force a low-quality extraction (e.g. a near-empty / boilerplate page among the sources). | A `LowQualityExtractionEvent` is recorded (not just metadata on `ToolExecutionCompletedEvent`). | |
|
|
| 5 | Have the agent attempt a `web_fetch` to a host **not** in the approved list. | A T2 `ApprovalRequestedEvent` IS raised for that host (the allowlist is a union, not a blanket open). | |
|
|
| 6 | `correx replay {id}`. | Replay reproduces the same allowlist decisions deterministically (the grant + fetch events are the only inputs; no re-fetch). | |
|
|
|
|
Evidence sources: `correx events {id}` / `GET /sessions/{id}/events`, server logs (MDC sessionId), `correx replay {id}`.
|
|
|
|
## Out of scope (explicitly NOT covered this pass)
|
|
|
|
- The standalone §D web-approval client (third client) — separate track, not built.
|
|
- Headless-browser / LLM-based extraction (non-responsibilities).
|
|
|
|
## Disposition
|
|
|
|
- **PASS** → move the §D egress/batch/source-event bullets to `RETRO.md` with this run date + the cited `EgressHostsGrantedEvent` / `SourceFetchedEvent` evidence. Status: PASSED.
|
|
- **FAIL** → file each failure as a numbered finding in `BACKLOG.md` (action + repro + the wrong/missing signal). Status: FAILED until green.
|