orchestra/AUDIT.md

# Orchestra audit — handoff first

Audited 2026-07-30 against the working tree, spec, deployed coordinator,
workpc worker, event log, and live herdr (read-only).

**Verdict:** one worker handoff completed, but the system is not safe to run
unattended. It can skip rotation, omit Git state, split ownership, strand a
released agent, or reject a valid completion.

## Evidence

- `go build ./...`, `go vet ./...`, `go test ./...`, and `go test -race ./...`: pass.
- Live B17: release `seq=251`, re-lease `252`, completion `262`; the simple
  probe needed six approvals, logged a `409 lease version conflict`, and
  recorded `consumed:0`.
- Live now: Docker owns the coordinator; the old systemd unit is inactive.
  Workpc runs a dirty `1ca9d64` worker build. No task is active and live herdr
  reports no agents. This does not prove the current working tree.

## P0 — correctness

| ID | Current failure | Required fix |
|---|---|---|
| H1 | **Closed 2026-07-30.** The checkout-owning worker and coordinator turn path now use `RotationStateMachine`. Workers persist harness-native identity (Claude/Codex transcript, OpenCode SQLite session id), apply soft/milestone/thrash/hard-boundary decisions, and record unknown activity/occupancy/boundary as degraded health rather than zero usage. | Verified by `go test -race ./...`; the existing turn-policy coverage now exercises the shared state machine. |
| H2 | **Closed 2026-07-30.** `PrepareRelease` verifies immutable `TASK.md`, checkpoints all repository work except protocol markers, always pushes the per-task project's scratch anchor, verifies it with `ls-remote`, and only then seals the CAS handoff. | `TestScratchCommitCapturesAllGitStatesExceptProtocolMarkers` covers staged, deleted, renamed, untracked, and protocol-marker cases; release uses the configured project remote. |
| H3 | **Closed 2026-07-30.** Worker state persists idempotent release transactions through `prepared → anchor_pushed → event_committed → pickup_validated → predecessor_retired`. Release/pickup endpoints bind transaction, anchor, and lease version; a predecessor remains mapped and is retired only after matching pickup validation. | `TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup` covers transaction propagation and pickup epoch binding; full race suite passes. |
| H4 | **Closed 2026-07-30.** Every new lease carries an opaque durable `lease_epoch`; renew/release/pickup/complete validate the exact harness owner and epoch at the store boundary and federation API. Offline heartbeats retain leases until expiry, new workers require a fresh reachable local-herdr probe, and local/worker ownership loss stops or durably quarantines the old pane before its mapping is dropped. | `TestLeaseEpochFencesStaleOwnerLifecycleWrites`, `TestAvailableRequiresFreshReachableLocalHerdrHealth`, plus the full race suite cover stale re-lease/completion and health admission. |
| H5 | **Closed 2026-07-30.** `Store.Append` validates legal state/owner/epoch transitions, fsyncs the event before applying its projection, and replays projections solely from `events.jsonl` (snapshots are disposable caches). CAS, worker/federation/coordinator state use temp-file + fsync + rename; corrupt worker state aborts startup. The live legacy `/v1/harness/complete` route is retired (410); its retained compatibility handler is fenced if invoked directly. | `TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot`, `TestWorkerRefusesCorruptDurableState`, and `go test -race ./...` pass. |

## P1 — autonomy and recovery

- **Recovery:** **Closed 2026-07-30.** Launch/recovery faults now emit
  `TaskNeedsAttention`, retaining the durable harness owner and lease epoch.
  Renew, release, expiry, and a late reconciled completion accept that same
  fenced lease; worker state advances its expected aggregate version without
  dropping the live session. `TaskBlocked` remains terminal for an explicit
  operator block. `TestNeedsAttentionRetainsFencedLeaseForLateCompletion`
  covers the durable recovery path.
- **Retries:** **Closed 2026-07-30.** Hand-off-less `TaskReleased` is the
  single durable reclaim transition. It projects exponential `attempt`,
  `next_retry_at`, and `failure_class`; router assignment reads those fields,
  so coordinator restarts cannot reset a backoff or retry limit.
  `TestReclaimPersistsAttemptAndBackoffAcrossReopen` covers replay.
- **Launch:** **Closed 2026-07-30.** Workers emit a fenced
  `TaskLaunchAcknowledged` only after a local start/prompt is persisted.
  Typed NACKs immediately reclaim transient unusable capacity, terminally
  block invalid handoffs, and retain uncertain live panes for reconciliation.
- **Completion:** **Closed 2026-07-30.** `.orchestra/done` is explicit
  intent only; the worker also requires native non-busy identity, runs its
  quality gate, verifies immutable `TASK.md`, commits, pushes, and checks
  the remote SHA before it emits completion.
- **Quota:** **Closed 2026-07-30.** Completion receipts contain native
  per-lease deltas plus a known/unknown marker. Five-hour and weekly
  projections are published from the same receipts; any bounded harness
  without fresh known usage fails routing closed.
- **Approvals:** **Closed 2026-07-30.** Projects have a validated audited
  `safe_operations` policy limited to worktree-local read/edit/test/Git.
  Workers inject it into the task prompt; network, secrets, destructive
  actions, and paths outside the worktree remain operator-gated.
- **Observability:** **Closed 2026-07-30.** Task projections now retain
  lifecycle phase, last error, retry time/failure class, lease epoch, pane
  state, and anchor. Release/anchor certification faults enter durable
  `needs_attention` instead of disappearing through retry `continue` paths.

## P2 — performance

- **Closed 2026-07-30.** Each scheduling pass takes one atomic task/lease
  snapshot, batches cached (TTL) reachability probes concurrently, and
  evaluates candidate availability once. It no longer probes candidates or
  scans active tasks once per queued task.
- **Closed 2026-07-30.** Active leases and per-harness, time-ordered quota
  receipts are projection indexes. Availability uses indexed rolling-window
  sums rather than decoding the event log; the task snapshot is a one-time
  disposable compatibility cache rather than a full rewrite on every append.
- **Closed 2026-07-30.** `BenchmarkAssignPending{1K,10K}` and
  `BenchmarkAppend{1K,10K}` report and enforce p95 budgets, with durable
  fsync cost included in their respective paths.

## Delivery order

1. Durable event transitions + lease fencing.
2. Idempotent checkpoint/release/pickup transaction.
3. Worker-local rotation, completion, quota, and typed recovery.
4. Approval policy and performance indexes.
5. Only then: ingestion/UI expansion.

## Release gate

- All build/vet/test/race checks pass.
- Fault-inject every handoff phase, coordinator/worker restart, lost response,
  worker partition/rejoin, stale completion, and corrupt state file.
- Cross-machine tests cover staged/deleted/clean-committed work and prove the
  predecessor remains recoverable until successor pickup validation.
- Live controlled runs pass soft, hard, milestone, thrash, completion, and
  late-recovery paths on each harness without manual intervention for safe
  repository work.
- Coordinator and workers report the same immutable build revision; staged
  worker checksum and Go build revision match before restart.