orchestra/AUDIT.md

Orchestra audit — handoff first

Audited 2026-07-30 against the working tree, spec, deployed coordinator, workpc worker, event log, and live herdr (read-only).

Verdict: one worker handoff completed, but the system is not safe to run unattended. It can skip rotation, omit Git state, split ownership, strand a released agent, or reject a valid completion.

Evidence

• go build ./..., go vet ./..., go test ./..., and go test -race ./...: pass.
• Live B17: release seq=251, re-lease 252, completion 262; the simple probe needed six approvals, logged a 409 lease version conflict, and recorded consumed:0.
• Live now: Docker owns the coordinator; the old systemd unit is inactive. Workpc runs a dirty 1ca9d64 worker build. No task is active and live herdr reports no agents. This does not prove the current working tree.

P0 — correctness

ID	Current failure	Required fix
H1	Closed 2026-07-30. The checkout-owning worker and coordinator turn path now use RotationStateMachine. Workers persist harness-native identity (Claude/Codex transcript, OpenCode SQLite session id), apply soft/milestone/thrash/hard-boundary decisions, and record unknown activity/occupancy/boundary as degraded health rather than zero usage.	Verified by go test -race ./...; the existing turn-policy coverage now exercises the shared state machine.
H2	Closed 2026-07-30. PrepareRelease verifies immutable TASK.md, checkpoints all repository work except protocol markers, always pushes the per-task project's scratch anchor, verifies it with ls-remote, and only then seals the CAS handoff.	TestScratchCommitCapturesAllGitStatesExceptProtocolMarkers covers staged, deleted, renamed, untracked, and protocol-marker cases; release uses the configured project remote.
H3	Closed 2026-07-30. Worker state persists idempotent release transactions through prepared → anchor_pushed → event_committed → pickup_validated → predecessor_retired. Release/pickup endpoints bind transaction, anchor, and lease version; a predecessor remains mapped and is retired only after matching pickup validation.	TestReleaseTransactionSurvivesReLeaseUntilMatchingPickup covers transaction propagation and pickup epoch binding; full race suite passes.
H4	Closed 2026-07-30. Every new lease carries an opaque durable lease_epoch; renew/release/pickup/complete validate the exact harness owner and epoch at the store boundary and federation API. Offline heartbeats retain leases until expiry, new workers require a fresh reachable local-herdr probe, and local/worker ownership loss stops or durably quarantines the old pane before its mapping is dropped.	TestLeaseEpochFencesStaleOwnerLifecycleWrites, TestAvailableRequiresFreshReachableLocalHerdrHealth, plus the full race suite cover stale re-lease/completion and health admission.
H5	Closed 2026-07-30. Store.Append validates legal state/owner/epoch transitions, fsyncs the event before applying its projection, and replays projections solely from events.jsonl (snapshots are disposable caches). CAS, worker/federation/coordinator state use temp-file + fsync + rename; corrupt worker state aborts startup. The live legacy /v1/harness/complete route is retired (410); its retained compatibility handler is fenced if invoked directly.	TestOpenRebuildsOnlyFromLogAndIgnoresCorruptSnapshot, TestWorkerRefusesCorruptDurableState, and go test -race ./... pass.

P1 — autonomy and recovery

• Recovery: TaskBlocked destroys the worker session needed for late completion; aggregate-version changes also stale the lease. Use a separate lease epoch and a recoverable needs_attention state that retains ownership until explicit release, expiry, or reconciled completion.
• Retries: expiry bypasses Router.HandleEvent; attempts/backoff are in-memory and unsynchronised. Project durable attempt, next_retry_at, and failure class; route every reclaim through one transition.
• Launch: repeated start failures hold a lease for up to 30 minutes. Workers must ACK start or NACK with typed evidence; retry transient failures, block invalid handoffs, and immediately free unusable capacity.
• Completion: .orchestra/done is the only worker completion signal. Combine an explicit completion intent with native idle/exit identity, the worker-owned quality gate, verified commit, and verified push.
• Quota: worker sessions do not retain a usage source, so live receipts are zero and quota routing is ineffective. Record per-lease deltas and publish both 5-hour and weekly projections; unknown quota fails closed.
• Approvals: the continuity probe required six manual grants. Add audited per-project policy for safe worktree-local reads, edits, tests, and Git; keep destructive, secret, network, and out-of-worktree actions gated.
• Observability: replace release/rotation continue paths with durable phase, last error, retry time, lease epoch, pane state, and anchor fields.

P2 — performance

• Cache/parallelise health probes; schedule from one task/worker snapshot. Current routing repeatedly scans tasks and probes candidates per queued task.
• Index active leases and quota windows. Do not scan the whole event log per availability check or rewrite the full task snapshot after every event.
• Add 1k/10k-task benchmarks with assignment and append p95 budgets.

Delivery order

1. Durable event transitions + lease fencing.
2. Idempotent checkpoint/release/pickup transaction.
3. Worker-local rotation, completion, quota, and typed recovery.
4. Approval policy and performance indexes.
5. Only then: ingestion/UI expansion.

Release gate

• All build/vet/test/race checks pass.
• Fault-inject every handoff phase, coordinator/worker restart, lost response, worker partition/rejoin, stale completion, and corrupt state file.
• Cross-machine tests cover staged/deleted/clean-committed work and prove the predecessor remains recoverable until successor pickup validation.
• Live controlled runs pass soft, hard, milestone, thrash, completion, and late-recovery paths on each harness without manual intervention for safe repository work.
• Coordinator and workers report the same immutable build revision; staged worker checksum and Go build revision match before restart.