kami
18cbd34739
Uncommitted work from the session-robustness-and-dox branch sweep
(docs/qa/QA-session-robustness-and-dox.md), verified alongside the
compression/context fixes:
- SandboxedToolExecutor: validate tool args centrally before dispatch. A
malformed/missing-arg call becomes a recoverable ERROR: (surfaced with the
tool's arg schema so the model can correct + retry) instead of stranding
the stage with no artifact. + validation test.
- PlanLinter: seed artifacts (analysis) produced by the planning phase count
as available producers, so a plan stage that `needs` them isn't flagged as
an unproduced-need; H1 unproduced-needs + trap-state checks. + tests.
- DefaultSessionOrchestrator: live-QA robustness fixes (event-tail /
per-stage budget + retry handling).
- workflow prompts/configs: DOX AGENTS.md alignment + freestyle/task/role
prompt tweaks.
- SessionOrchestratorIntegrationTest: coverage for the above.
- FreestylePlanningWorkflowTest: allow list_dir in analyst tools (follows the
list_dir wiring in 968cbfa).
- QA plan doc for the branch sweep.
76 lines
7.1 KiB
Markdown
76 lines
7.1 KiB
Markdown
# QA Plan: session-robustness-and-dox — master..feat/session-robustness-and-dox
|
|
|
|
**Status:** DRAFT
|
|
**Run date / operator:**
|
|
**BACKLOG item:** feat/session-robustness-and-dox full-branch sweep
|
|
|
|
Commits under test:
|
|
```
|
|
c8c2521 fix(kernel,inference): reliable artifact emission for local models
|
|
af9da3c fix(inference): raise llama HTTP request timeout 600s -> 1800s
|
|
5e0c7df feat(server): REST stage-approval route POST /sessions/{id}/approve
|
|
e72051a feat(kernel): emit_artifact tool + actionable validation feedback
|
|
0652d87 fix(inference): salvage truncated/malformed tool-call JSON from content
|
|
4737300 fix(server): anchor static-registry file_read to the workspace
|
|
82674e8 feat(inference): capability-aware routing for tool-heavy stages
|
|
238d353 feat(qa): remote NIM provider + headless-QA robustness
|
|
e0f623a feat(workflow): tasked execution loop replaces role_pipeline
|
|
a00bd4a feat(tools): propose_scope recalibration valve for the execution loop
|
|
3e44c6d feat(kernel): deterministic per-task claim stage + per-task write scope
|
|
c1e4c7b feat(transitions): tasks_ready predicate for execution loop
|
|
d26f20c docs(dox): build out the DOX AGENTS.md hierarchy
|
|
1cb7fec feat(kernel): read-only mode after read-before-write + mandatory task
|
|
201b599 fix(inference): recover tool calls emitted as JSON in message content
|
|
ccfa4b4 feat(tui): composer soft-wrap, alt+backspace, bracketed paste, borders
|
|
8abe7d9 fix(tui): collapse multiline tool summaries so action rows don't stripe
|
|
```
|
|
|
|
---
|
|
|
|
## Preconditions
|
|
|
|
- [x] **server build/branch:** `feat/session-robustness-and-dox`, `:apps:server:run` on :8080 — _rebuild only with the server stopped_
|
|
- [x] **llama-server + model:** router = gemma-4-12B-it-qat-UD-Q4_K_XL :10000 (tool-capable, ~48 tok/s), narrator = LFM2.5-1.2B :10001
|
|
- [ ] **external deps:** none required for the core loop; capability-aware routing (§C8) is only meaningfully exercised if **two** tool-capable providers are configured — otherwise it's a single-candidate no-op. OpenAI-compat/NIM path (238d353) needs a remote endpoint; **out of scope** unless one is wired.
|
|
- [ ] **config synced:** `~/.config/correx/workflows/` must contain the NEW `task_planning.toml` + `prompts/task_planner.md` and the REWRITTEN `role_pipeline.toml` (execution loop). Copy from `examples/workflows/` before running — stale role_pipeline = wrong graph.
|
|
- [ ] **fixtures/seed:** a real workspace with a `.correx/project.toml` and a small, genuinely-actionable request (so `task_decompose` produces >1 task and the loop iterates).
|
|
|
|
## Acceptance gate (one sentence)
|
|
|
|
> The branch is correct **iff** a real request drives the execution loop end-to-end —
|
|
> planner emits a task graph, each task is deterministically claimed, writes are scoped to
|
|
> the claimed task's `affected_paths` (and blocked outside it / in read-only mode), artifacts
|
|
> emit reliably on the local model, and a headless `POST /approve` clears an approval gate —
|
|
> every step evidenced by a recorded event.
|
|
|
|
## Checks
|
|
|
|
| # | Action | Expected observable evidence | Result |
|
|
|---|--------|------------------------------|--------|
|
|
| 1 | Start a `task_planning` session on a real request | `correx events <id>` shows the `planner` stage; a `task_decompose` tool invocation succeeds and task-created events appear; stage cannot complete without it (`require_task_decompose` gate) — try a run where the model skips decompose and confirm the stage is **blocked**, not completed | |
|
|
| 2 | Start a `role_pipeline` (execution loop) session | Stage chain analyst→architect→decomposer→implementer→reviewer in the log; `decomposer-to-implementer` fires on `tasks_ready` (not always_true); loop re-enters implementer via `review-2-more` while tasks remain, exits via `review-3-done` when drained | |
|
|
| 3 | Watch the per-task claim stage (`claimTask=true`) | On each implementer entry, a claim record for the next ready task + its context bundle injected as **L0** (`sourceType=claimedTask`); the loop advances by claim events, not model memory — verify the claimed task id changes across iterations | |
|
|
| 4 | Let the implementer write a file inside the claimed task's `affected_paths` | `FileWriteTool` invocation ASSESSED and allowed; file appears on disk under the workspace | |
|
|
| 5 | Force a write **outside** the claimed scope | `ToolCallAssessedEvent` blocks it (scope violation); the write does not land on disk; feedback surfaced to the stage | |
|
|
| 6 | Trigger read-only mode (a stage that reads before any task is claimed / after read-before-write) | `isReadOnlyMode` denies a subsequent write: `ToolCallAssessedEvent` with the read-only denial; no file mutation event | |
|
|
| 7 | Have the implementer call `propose_scope` with a `paths`/`reason` | Tool records a scope-widen proposal on the task; **operator approval required** (approval gate fires); on approve, paths added to `affected_paths` and the previously-blocked write now lands | |
|
|
| 8 | Confirm artifact emission on the local model (analysis/design/patch) | Each `*_validated` transition fires because the artifact actually emitted — `artifact_validated` events present; no stage stuck retrying because the artifact never materialized (the c8c2521 fix) | |
|
|
| 9 | Feed a truncated/malformed tool-call (or observe a real one from gemma) | Salvage path recovers it: the tool call executes instead of the turn failing; log shows the salvage of JSON-in-content (0652d87 / 201b599) | |
|
|
| 10 | With a session suspended on an approval, `POST /sessions/{id}/approve` `{"decision":"APPROVE"}` (no requestId) | HTTP 200; server resolves the pending requestId itself; the gate clears and the session advances — no WebSocket needed. Also test `REJECT` (session takes reject path) and a bad `decision` → 400, and no-pending → 404 | |
|
|
| 11 | `POST /approve` with `{"decision":"STEER","steeringNote":"..."}` | Treated as APPROVE with the note attached; note recorded on the approval response | |
|
|
| 12 | (If 2 tool-capable providers configured) run a tool-heavy stage | Routing selects the higher tool-capability provider, not just any healthy one; check `/metrics/tool-reliability` reflects observed per-model reliability | |
|
|
| 13 | TUI composer smoke: soft-wrap a long line, alt+backspace, bracketed paste | Composer wraps within borders; alt+backspace deletes a word; pasted multiline text lands as one block; multiline tool summaries render collapsed (no striped action rows) | |
|
|
| 14 | `correx replay <id>` on a completed loop session | Double-read digest matches — the claim/scope/read-only logic is derived from recorded events, replay-stable | |
|
|
|
|
## Out of scope (explicitly NOT covered this pass)
|
|
|
|
- OpenAI-compat / remote NIM provider (238d353) — no remote endpoint wired; unit-tested only.
|
|
- Capability-aware routing real differentiation (§C12) unless a second tool-capable provider is configured; single-provider makes it a no-op.
|
|
- The DOX AGENTS.md hierarchy (d26f20c) — docs only, nothing to exercise at runtime.
|
|
- Layer-2 LLM annotation of tool-intent; the AMD/ROCm perf envelope.
|
|
|
|
## Disposition
|
|
|
|
- **PASS** → move the BACKLOG entry into `RETRO.md` with run date + cited evidence. Set Status: PASSED.
|
|
- **FAIL** → file each miss as a numbered `BACKLOG.md` finding (action + repro + wrong/missing signal), fix, re-run only failed checks. Status: FAILED until green.
|