orchestra/AUDIT.md

Orchestra audit — handoff first

Audited 2026-07-30 against the working tree, spec, deployed coordinator, workpc worker, event log, and live herdr (read-only).

Verdict: one worker handoff completed, but the system is not safe to run unattended. It can skip rotation, omit Git state, split ownership, strand a released agent, or reject a valid completion.

Evidence

• go build ./..., go vet ./..., go test ./..., and go test -race ./...: pass.
• Live B17: release seq=251, re-lease 252, completion 262; the simple probe needed six approvals, logged a 409 lease version conflict, and recorded consumed:0.
• Live now: Docker owns the coordinator; the old systemd unit is inactive. Workpc runs a dirty 1ca9d64 worker build. No task is active and live herdr reports no agents. This does not prove the current working tree.

P0 — correctness

ID	Current failure	Required fix
H1	Closed 2026-07-30. The checkout-owning worker and coordinator turn path now use RotationStateMachine. Workers persist harness-native identity (Claude/Codex transcript, OpenCode SQLite session id), apply soft/milestone/thrash/hard-boundary decisions, and record unknown activity/occupancy/boundary as degraded health rather than zero usage.	Verified by go test -race ./...; the existing turn-policy coverage now exercises the shared state machine.
H2	Closed 2026-07-30. PrepareRelease verifies immutable TASK.md, checkpoints all repository work except protocol markers, always pushes the per-task project's scratch anchor, verifies it with ls-remote, and only then seals the CAS handoff.	TestScratchCommitCapturesAllGitStatesExceptProtocolMarkers covers staged, deleted, renamed, untracked, and protocol-marker cases; release uses the configured project remote.
H3	Closed 2026-07-30. Worker state persists idempotent release transactions through prepared → anchor_pushed → event_committed → pickup_validated → predecessor_retired. Release/pickup endpoints bind transaction, anchor, and lease version; a predecessor remains mapped and is retired only after matching pickup validation.	TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup covers transaction propagation and pickup epoch binding; full race suite passes.
H4	Closed 2026-07-30. Every new lease carries an opaque durable lease_epoch; renew/release/pickup/complete validate the exact harness owner and epoch at the store boundary and federation API. Offline heartbeats retain leases until expiry, new workers require a fresh reachable local-herdr probe, and local/worker ownership loss stops or durably quarantines the old pane before its mapping is dropped.	TestLeaseEpochFencesStaleOwnerLifecycleWrites, TestAvailableRequiresFreshReachableLocalHerdrHealth, plus the full race suite cover stale re-lease/completion and health admission.
H5	Closed 2026-07-30. Store.Append validates legal state/owner/epoch transitions, fsyncs the event before applying its projection, and replays projections solely from events.jsonl (snapshots are disposable caches). CAS, worker/federation/coordinator state use temp-file + fsync + rename; corrupt worker state aborts startup. The live legacy /v1/harness/complete route is retired (410); its retained compatibility handler is fenced if invoked directly.	TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot, TestWorkerRefusesCorruptDurableState, and go test -race ./... pass.

P1 — autonomy and recovery

• Recovery: Closed 2026-07-30. Launch/recovery faults now emit TaskNeedsAttention, retaining the durable harness owner and lease epoch. Renew, release, expiry, and a late reconciled completion accept that same fenced lease; worker state advances its expected aggregate version without dropping the live session. TaskBlocked remains terminal for an explicit operator block. TestNeedsAttentionRetainsFencedLeaseForLateCompletion covers the durable recovery path.
• Retries: Closed 2026-07-30. Hand-off-less TaskReleased is the single durable reclaim transition. It projects exponential attempt, next_retry_at, and failure_class; router assignment reads those fields, so coordinator restarts cannot reset a backoff or retry limit. TestReclaimPersistsAttemptAndBackoffAcrossReopen covers replay.
• Launch: Closed 2026-07-30. Workers emit a fenced TaskLaunchAcknowledged only after a local start/prompt is persisted. Typed NACKs immediately reclaim transient unusable capacity, terminally block invalid handoffs, and retain uncertain live panes for reconciliation.
• Completion: Closed 2026-07-30. .orchestra/done is explicit intent only; the worker also requires native non-busy identity, runs its quality gate, verifies immutable TASK.md, commits, pushes, and checks the remote SHA before it emits completion.
• Quota: Closed 2026-07-30. Completion receipts contain native per-lease deltas plus a known/unknown marker. Five-hour and weekly projections are published from the same receipts; any bounded harness without fresh known usage fails routing closed.
• Approvals: Closed 2026-07-30. Projects have a validated audited safe_operations policy limited to worktree-local read/edit/test/Git. Workers inject it into the task prompt; network, secrets, destructive actions, and paths outside the worktree remain operator-gated.
• Observability: Closed 2026-07-30. Task projections now retain lifecycle phase, last error, retry time/failure class, lease epoch, pane state, and anchor. Release/anchor certification faults enter durable needs_attention instead of disappearing through retry continue paths.

P2 — performance

• Closed 2026-07-30. Each scheduling pass takes one atomic task/lease snapshot, batches cached (TTL) reachability probes concurrently, and evaluates candidate availability once. It no longer probes candidates or scans active tasks once per queued task.
• Closed 2026-07-30. Active leases and per-harness, time-ordered quota receipts are projection indexes. Availability uses indexed rolling-window sums rather than decoding the event log; the task snapshot is a one-time disposable compatibility cache rather than a full rewrite on every append.
• Closed 2026-07-30. BenchmarkAssignPending{1K,10K} and BenchmarkAppend{1K,10K} report and enforce p95 budgets, with durable fsync cost included in their respective paths.

Delivery order

1. Durable event transitions + lease fencing.
2. Idempotent checkpoint/release/pickup transaction.
3. Worker-local rotation, completion, quota, and typed recovery.
4. Approval policy and performance indexes.
5. Only then: ingestion/UI expansion.

Release gate

• All build/vet/test/race checks pass.
• Fault-inject every handoff phase, coordinator/worker restart, lost response, worker partition/rejoin, stale completion, and corrupt state file.
• Cross-machine tests cover staged/deleted/clean-committed work and prove the predecessor remains recoverable until successor pickup validation.
• Live controlled runs pass soft, hard, milestone, thrash, completion, and late-recovery paths on each harness without manual intervention for safe repository work.
• Coordinator and workers report the same immutable build revision; staged worker checksum and Go build revision match before restart.