orchestra/AUDIT.md

# Orchestra audit — handoff first

Audited 2026-07-30 against the working tree, spec, deployed coordinator,
workpc worker, event log, and live herdr (read-only).

**Verdict:** the source-level P0/P1/P2 fixes are in place and the live
coordinator can replay its repaired event log, but the system is not safe to
run unattended until live harness capacity and the worker rollout are restored.

## Evidence

- `go build ./...`, `go vet ./...`, `go test ./...`, and `go test -race ./...`: pass.
- Live B17: release `seq=251`, re-lease `252`, completion `262`; the simple
  probe needed six approvals, logged a `409 lease version conflict`, and
  recorded `consumed:0`.
- Live follow-up: Docker owns the coordinator; the old systemd unit is
  inactive. The pre-v2 repeated-`seq=1` event prefix was migrated with a
  backup-preserving, explicit tool before the current coordinator replayed it.

## Remaining release blockers

- **No live harness capacity.** Homesrv has no reachable herdr. Workpc has no
  listener on any expected herdr port, so no safe live rotation, handoff,
  completion, or late-recovery path can be exercised on Claude, Codex, or
  OpenCode.
- **Worker rollout needs an administrator.** Workpc is still running the old
  dirty `1ca9d64` `/usr/local/bin/orchestra-worker`. The current worker binary
  is staged at `~/orchestra-deploy/orchestra-worker`, but this account cannot
  install it or restart the systemd service. Its project configuration must
  also declare supported projects before routing can admit a task.
- **B17 needs a fresh controlled run.** The historical probe's six approvals,
  one `409 lease version conflict`, and `consumed:0` receipt came from the
  old worker. They cannot be treated as evidence for the current worker until
  a live OpenCode run is repeated; Claude and Codex require their own runs.

## P0 — correctness

| ID | Current failure | Required fix |
|---|---|---|
| H1 | **Closed 2026-07-30.** The checkout-owning worker and coordinator turn path now use `RotationStateMachine`. Workers persist harness-native identity (Claude/Codex transcript, OpenCode SQLite session id), apply soft/milestone/thrash/hard-boundary decisions, and record unknown activity/occupancy/boundary as degraded health rather than zero usage. | Verified by `go test -race ./...`; the existing turn-policy coverage now exercises the shared state machine. |
| H2 | **Closed 2026-07-30.** `PrepareRelease` verifies immutable `TASK.md`, checkpoints all repository work except protocol markers, always pushes the per-task project's scratch anchor, verifies it with `ls-remote`, and only then seals the CAS handoff. | `TestScratchCommitCapturesAllGitStatesExceptProtocolMarkers` covers staged, deleted, renamed, untracked, and protocol-marker cases; release uses the configured project remote. |
| H3 | **Closed 2026-07-30.** Worker state persists idempotent release transactions through `prepared → anchor_pushed → event_committed → pickup_validated → predecessor_retired`. Release/pickup endpoints bind transaction, anchor, and lease version; a predecessor remains mapped and is retired only after matching pickup validation. | `TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup` covers transaction propagation and pickup epoch binding; full race suite passes. |
| H4 | **Closed 2026-07-30.** Every new lease carries an opaque durable `lease_epoch`; renew/release/pickup/complete validate the exact harness owner and epoch at the store boundary and federation API. Offline heartbeats retain leases until expiry, new workers require a fresh reachable local-herdr probe, and local/worker ownership loss stops or durably quarantines the old pane before its mapping is dropped. | `TestLeaseEpochFencesStaleOwnerLifecycleWrites`, `TestAvailableRequiresFreshReachableLocalHerdrHealth`, plus the full race suite cover stale re-lease/completion and health admission. |
| H5 | **Closed 2026-07-30.** `Store.Append` validates legal state/owner/epoch transitions, fsyncs the event before applying its projection, and replays projections solely from `events.jsonl` (snapshots are disposable caches). CAS, worker/federation/coordinator state use temp-file + fsync + rename; corrupt worker state aborts startup. The live legacy `/v1/harness/complete` route is retired (410); its retained compatibility handler is fenced if invoked directly. | `TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot`, `TestWorkerRefusesCorruptDurableState`, and `go test -race ./...` pass. |

## P1 — autonomy and recovery

- **Recovery:** **Closed 2026-07-30.** Launch/recovery faults now emit
  `TaskNeedsAttention`, retaining the durable harness owner and lease epoch.
  Renew, release, expiry, and a late reconciled completion accept that same
  fenced lease; worker state advances its expected aggregate version without
  dropping the live session. `TaskBlocked` remains terminal for an explicit
  operator block. `TestNeedsAttentionRetainsFencedLeaseForLateCompletion`
  covers the durable recovery path.
- **Retries:** **Closed 2026-07-30.** Hand-off-less `TaskReleased` is the
  single durable reclaim transition. It projects exponential `attempt`,
  `next_retry_at`, and `failure_class`; router assignment reads those fields,
  so coordinator restarts cannot reset a backoff or retry limit.
  `TestReclaimPersistsAttemptAndBackoffAcrossReopen` covers replay.
- **Launch:** **Closed 2026-07-30.** Workers emit a fenced
  `TaskLaunchAcknowledged` only after a local start/prompt is persisted.
  Typed NACKs immediately reclaim transient unusable capacity, terminally
  block invalid handoffs, and retain uncertain live panes for reconciliation.
- **Completion:** **Closed 2026-07-30.** `.orchestra/done` is explicit
  intent only; the worker also requires native non-busy identity, runs its
  quality gate, verifies immutable `TASK.md`, commits, pushes, and checks
  the remote SHA before it emits completion.
- **Quota:** **Closed 2026-07-30.** Completion receipts contain native
  per-lease deltas plus a known/unknown marker. Five-hour and weekly
  projections are published from the same receipts; any bounded harness
  without fresh known usage fails routing closed.
- **Approvals:** **Closed 2026-07-30.** Projects have a validated audited
  `safe_operations` policy limited to worktree-local read/edit/test/Git.
  Workers inject it into the task prompt; network, secrets, destructive
  actions, and paths outside the worktree remain operator-gated.
- **Observability:** **Closed 2026-07-30.** Task projections now retain
  lifecycle phase, last error, retry time/failure class, lease epoch, pane
  state, and anchor. Release/anchor certification faults enter durable
  `needs_attention` instead of disappearing through retry `continue` paths.

## P2 — performance

- **Closed 2026-07-30.** Each scheduling pass takes one atomic task/lease
  snapshot, batches cached (TTL) reachability probes concurrently, and
  evaluates candidate availability once. It no longer probes candidates or
  scans active tasks once per queued task.
- **Closed 2026-07-30.** Active leases and per-harness, time-ordered quota
  receipts are projection indexes. Availability uses indexed rolling-window
  sums rather than decoding the event log; the task snapshot is a one-time
  disposable compatibility cache rather than a full rewrite on every append.
- **Closed 2026-07-30.** `BenchmarkAssignPending{1K,10K}` and
  `BenchmarkAppend{1K,10K}` report and enforce p95 budgets, with durable
  fsync cost included in their respective paths.

## Delivery order

1. Durable event transitions + lease fencing.
2. Idempotent checkpoint/release/pickup transaction.
3. Worker-local rotation, completion, quota, and typed recovery.
4. Approval policy and performance indexes.
5. Only then: ingestion/UI expansion.

## Release gate

- **Pass:** build, vet, test, and race checks pass; unit/integration coverage
  includes the defined fault and cross-machine cases.
- **Pending live capacity:** run the controlled soft, hard, milestone, thrash,
  completion, and late-recovery paths on each harness without manual
  intervention for safe repository work.
- **Pending worker installation:** install the staged worker, restart it after
  herdr is restored, and verify its checksum and immutable build revision
  against the coordinator diagnostics.