orchestra/orchestra-spec (1).md

# Orchestra — autonomous handoff protocol

Version: 2 (binding target)
Revised: 2026-07-30
Implementation language: Go

`MUST`, `MUST NOT`, `SHOULD`, and `MAY` are normative. Defaults marked `[D]`
are tunable. `AUDIT.md` records implementation gaps; it does not weaken this
contract.

## 0. Objective

Maximize useful unattended runtime across opaque CLI harnesses: Claude Code,
Codex, OpenCode, and future adapters. Orchestra drives processes through
herdr; it does not call model APIs.

Optimize, in order:

1. No lost or duplicated work.
2. One valid owner per task.
3. Recovery without operator reconstruction.
4. Useful autonomous progress.
5. Throughput and resource efficiency.

Cost-per-turn is secondary. UI breadth, provider breadth, and delegation are
secondary until the handoff protocol passes §13.

## 1. Invariants

1. **Append-only truth.** The coordinator event log is canonical. Events are
   never edited or deleted; corrections append compensation events.
2. **Durable before visible.** An event is appended and fsynced before any
   projection changes or success is acknowledged.
3. **Plane-owned lifecycle.** Agents may write code and bounded intent
   artifacts. Only Orchestra emits lifecycle events.
4. **Artifact references.** Events carry content hashes, never embedded
   reports, task specifications, diffs, or handoffs.
5. **Strict input.** Event and artifact schemas reject unknown fields,
   invalid transitions, oversized values, and unbounded instruction text.
6. **Ground-truth pickup.** A successor validates Git, the immutable task
   artifact, and the handoff artifact before starting work.
7. **One owner.** Every lease has an unguessable `lease_id` and monotonically
   increasing `lease_epoch`. Stale owners cannot renew, release, complete, or
   mutate task lifecycle.
8. **Self-fencing workers.** A worker that cannot renew before its local lease
   deadline parks its agent before the lease can be reassigned.
9. **Git crosses machines.** Remote workers exchange repository state only
   through verified Git refs plus CAS artifacts. The coordinator never
   validates a remote checkout.
10. **Split, validate, then close.** A predecessor remains recoverable until
    the successor has validated pickup and started the replacement lease.
11. **Idempotent control.** Every cross-process mutation has an operation ID.
    Retries return the original result; ambiguous UI-changing herdr calls are
    reconciled, not blindly repeated.
12. **Observable refusal.** No lifecycle-critical error is a bare `continue`.
    Phase, error class, retry time, owner, pane, and anchor are queryable.
13. **Liveness is live.** Pane/agent status comes from herdr or process exit,
    not event replay.
14. **No preemption.** Importance orders queued work only; it never interrupts
    a valid running lease.

## 2. Ownership and topology

| Component | Owner | Authority |
|---|---|---|
| Event log, CAS, projections, router | coordinator on homesrv | canonical |
| Git checkout, worktree, herdr socket, pane | worker on that machine | local execution |
| Git remote | homesrv | cross-machine repository transport |
| Browser/TUI/notifications | clients | projections and authorized commands |

### 2.1 Federation

- Remote execution is worker-pull only. A coordinator MUST NOT call a remote
  herdr socket or inspect a remote filesystem.
- One worker represents one configured herdr/harness capacity pool.
- Worker identity and herdr registry identity MUST match.
- Registration declares protocol version, immutable build revision,
  capabilities, concurrency, supported projects, and local health.
- Registration does not imply eligibility. Eligibility requires a recent
  heartbeat and recent local-herdr protocol check.
- Coordinator and worker protocol versions MUST be compatible. Production
  release gates require the same build revision unless a rolling-upgrade
  compatibility test exists.
- Heartbeat loss marks a worker ineligible; it MUST NOT release a lease early.
  Lease expiry or explicit relinquishment is authoritative.

Direct coordinator-to-remote-herdr operation (“Design A”) is forbidden.

### 2.2 Project configuration

Each project declares:

```text
id
machine_affinity[]          # hard by default
worker-local repo
worker-local worktree_root
git remote
quality_gate
safe_action_policy
capabilities
```

Paths are local to the worker that advertises the project. A worker MUST NOT
accept a project lacking a complete local configuration.

Machine affinity is hard `[D]`: an unavailable preferred machine leaves the
task queued instead of running it elsewhere.

## 3. Durable substrate

### 3.1 Event envelope

```json
{
  "schema_version": 3,
  "seq": 123,
  "id": "event-ulid",
  "type": "TaskLeased",
  "task_id": "task-ulid",
  "task_version": 7,
  "at": "RFC3339Nano",
  "surface": "system|web|tui|mcp|maven",
  "actor": "stable-actor-id",
  "operation_id": "idempotency-id",
  "payload": {}
}
```

- The append boundary assigns `seq`, `task_version`, and `at`.
- `seq` is globally monotonic; `task_version` is monotonic per task.
- A producer MAY supply `expected_task_version` for optimistic concurrency.
- Worker ownership is checked with `lease_id` + `lease_epoch`, not task
  version. Unrelated approvals cannot invalidate a lease.
- Reusing `operation_id` with the same request returns the original event.
  Reusing it with different input is rejected.
- Global projections use a named aggregate ID; they do not fake task
  versions.

### 3.2 Append transaction

For each accepted event:

1. Authenticate surface and actor.
2. Strict-decode payload.
3. Validate artifact references.
4. Validate legal state transition.
5. Validate lease owner/epoch when required.
6. Allocate sequence and task version.
7. Append and fsync the event.
8. Apply in-memory projections.
9. Acknowledge the caller.
10. Write snapshots asynchronously and atomically.

Snapshot failure MUST NOT make a durable append appear failed. Restart rebuilds
from the last valid snapshot plus the complete event tail.

### 3.3 CAS

- Address: lowercase SHA-256 of exact bytes.
- Write: temporary file → fsync → atomic rename → directory fsync.
- Existing objects are re-hashed before reuse.
- Read always re-hashes.
- Corruption is a hard, observable error; a corrupt object is never trusted.

### 3.4 Worker state

Worker state is a durable local journal containing:

```text
event cursor
leases and local monotonic deadlines
sessions and exact pane/agent identities
handoff transaction phases
pending coordinator operations
last health/error observations
```

Writes are temporary file → fsync → atomic rename. Missing state starts empty;
corrupt state fails closed and requires reconciliation. It MUST NOT silently
become an empty map.

## 4. Task and lease model

### 4.1 Immutable task

`TaskCreated` references a strict, immutable task artifact rendered as
`TASK.md`:

```text
task_id, source, external_id, project
title, instructions, acceptance[]
capabilities[], quality_gate
```

The artifact hash is `task_ref`. Scheduling metadata—priority, due date,
estimate, parent—may be amended. Instructions, acceptance, and gate are not
amended in place; a scope change supersedes the task with a new task artifact.

### 4.2 Projected states

| State | Meaning | Lease |
|---|---|---|
| `queued` | eligible now or at `next_retry_at` | none |
| `leased` | launching, running, or preparing handoff | required |
| `needs_attention` | recoverable pane awaits approval/operator action | retained when live |
| `completed` | gate, commit, push, and report verified | none |
| `failed` | retry policy exhausted or non-recoverable failure | none |

`needs_attention` is not terminal. The same fenced owner may renew, resume, or
complete it. A task with no recoverable session returns to `queued` through an
explicit retry/release event or becomes `failed`.

Terminal state changes require an explicit compensation event naming the event
being corrected.

A transition to `failed` must include evidence that any live pane was parked or
retired. Terminal state is never used as a substitute for cleanup.

### 4.3 Lease

```json
{
  "lease_id": "unguessable-id",
  "lease_epoch": 4,
  "worker_id": "workpc-opencode",
  "expires_at": "RFC3339Nano",
  "attempt": 2
}
```

- Epoch increases on every new lease.
- Renewal preserves ID and epoch and advances expiry.
- Every worker release, handoff, attention, approval execution, and completion
  includes ID and epoch.
- A worker derives a conservative local monotonic deadline from each lease
  response. It parks the agent before deadline if renewal cannot complete.
- Self-fencing means no process may keep editing after expiry. If an agent is
  still busy inside the safety margin and releasing its binding does not stop
  execution, the worker closes the exact pane while retaining the worktree
  and local recovery journal. Before closing, it best-effort commits current
  local state to an immutable orphan ref.
- A resumed worker reconciles ownership before reading, prompting, or starting
  any persisted session.
- If ownership was lost, the worker never resumes or auto-merges. It pushes
  the orphan ref when connectivity returns and publishes observation evidence
  for operator or explicit recovery.
- Capturing a pane is not sufficient proof for renewal; pane identity, agent
  identity, lease ownership, and local-herdr health must all match.

### 4.4 Retry

Failures are typed:

```text
transient_transport
worker_unhealthy
launch_failed
lease_expired
handoff_invalid
pickup_invalid
quality_gate_failed
git_push_failed
protocol_incompatible
operator_required
```

`attempt`, `failure_class`, `last_failure_ref`, and `next_retry_at` are durable
projection fields. Healthy rotations do not increment attempts. Retry policy
is exponential with bounded jitter `[D]`; the default maximum is three failed
attempts.

## 5. Event contract

| Event | Required payload | Transition / reaction |
|---|---|---|
| `TaskCreated` | `task_ref`, project, source, external ID, scheduling fields | new → queued; dedup `(source, external_id)` |
| `TaskAmended` | scheduling fields only | reproject queued ordering |
| `TaskLeased` | lease, `task_ref`, `handoff_ref?` | queued → leased; worker validates offer |
| `TaskLeaseStarted` | lease token, pane/session evidence | leased → leased; launch ACK |
| `TaskLeaseRenewed` | lease token, new expiry | leased/needs_attention → same |
| `TaskLeaseFailed` | lease token, class, evidence ref, retry time | leased → queued/failed |
| `TaskLeaseExpired` | lease token, attempt, retry time | leased/needs_attention → queued/failed |
| `TaskOrphanCheckpointed` | old lease token, evidence ref, Git ref/SHA | observation only; never restores ownership |
| `TaskNeedsAttention` | lease token, class, evidence ref, pane evidence | leased → needs_attention |
| `TaskResumed` | lease token, resolution event | needs_attention → leased |
| `ApprovalRequested` | lease token, subject ref, bounded options | leased → needs_attention |
| `ApprovalGranted/Denied` | subject ref, actor, capture revision | command result; may resume |
| `TaskHandoffPrepared` | transaction ID, lease token, handoff ref, Git anchor/ref, reason | leased → leased |
| `TaskReleased` | transaction ID, lease token, handoff ref, anchor | leased → queued; predecessor parked |
| `TaskPickupValidated` | new lease token, transaction ID, handoff ref, anchor | leased → leased |
| `TaskHandoffRetired` | transaction ID, predecessor pane result | observation after successor start |
| `TaskCompleted` | lease token, report ref, result SHA/ref, receipt | leased/needs_attention → completed |
| `TaskFailed` | class, attempts, evidence ref | queued/leased/needs_attention → failed |
| `TaskCorrected` | corrected event ID, bounded compensation fields | explicit compensation only |
| `QuotaReported` | worker/harness, window, consumed, observed at | quota projection |

Events not listed here are invalid until this contract is versioned.

Per-turn transcripts, pane liveness, and raw token events are not lifecycle
events.

## 6. Worker launch and pickup

### 6.1 Initial launch

1. Verify current lease token and worker/project eligibility.
2. Fetch/verify `task_ref`; write exact immutable `TASK.md`.
3. Sync the configured base checkout by fast-forward only.
4. Create/open the task worktree and branch.
5. Ask local herdr to open the worktree and start the configured harness.
6. Confirm the exact pane contains the exact attached agent.
7. Persist session identity.
8. Deliver one bounded launch prompt.
9. Emit `TaskLeaseStarted`.

Before step 7, a partial pane creation failure is still reconciled and cleaned
or retained as evidence. A launch failure emits `TaskLeaseFailed`; it does not
hold capacity until TTL by default.

### 6.2 Handoff pickup

Before starting a successor agent:

1. Strict-decode and validate the handoff artifact.
2. Fetch the exact remote handoff ref.
3. Create a worktree at `anchor.git_sha`.
4. Verify `HEAD == anchor.git_sha`.
5. Verify the remote ref resolves to the same SHA.
6. Verify exact `TASK.md` hash equals `task_ref`.
7. Verify transaction ID, task ID, handoff ref, and new lease token.
8. Emit `TaskPickupValidated`.
9. Start/confirm the successor agent.
10. Deliver the bounded bootstrap prompt.
11. Emit `TaskLeaseStarted`.

Any failure emits `TaskLeaseFailed(class=pickup_invalid)` with evidence. The
predecessor remains parked and recoverable.

## 7. Rotation and handoff

Rotation runs on the worker that owns the checkout. Coordinator-local rotation
logic MUST NOT be a separate implementation.

### 7.1 Triggers

| Trigger | Default | Action |
|---|---:|---|
| soft occupancy | 55% | request handoff at a natural boundary |
| hard occupancy | 75% | prepare and rotate at the next verified boundary |
| milestone | successful coherent checkpoint | rotate |
| thrash | repeated failures/edits/calls | rotate with dead ends |
| manual | authorized operator/agent intent | rotate |

Occupancy is current-turn context input, not cumulative token cost. Each
harness adapter binds usage data to the exact session/pane; “newest file
globally” is invalid when identity is ambiguous.

| Harness | Exact source | Occupancy numerator |
|---|---|---|
| Claude Code | attached session transcript, last assistant usage | input + cache read + cache creation |
| Codex | rollout path bound to the attached thread/session | last input + last cached input |
| OpenCode | message/status record bound to the attached session | input + cache read |

Cumulative session totals are receipts, never occupancy.

Unknown occupancy is observable. After two missed samples `[D]`, request a
conservative handoff at the next verified boundary; never interrupt a busy
turn.

### 7.2 Canonical handoff artifact

The agent writes only bounded labelled answers. The worker derives protocol
facts and seals strict JSON:

```json
{
  "schema_version": 2,
  "handoff_id": "id",
  "transaction_id": "id",
  "task_id": "id",
  "lease_id": "id",
  "reason": "threshold|milestone|thrash|manual",
  "rotation_index": 1,
  "anchor": {
    "git_sha": "40-hex",
    "remote_ref": "refs/orchestra/tasks/<task>/<rotation>",
    "task_sha256": "64-hex"
  },
  "next": {
    "action": "one bounded line",
    "why": "one bounded line",
    "command": "optional literal command",
    "files": ["relative/path"]
  },
  "remaining": ["bounded item"],
  "dead_ends": [{"tried": "bounded", "why_failed": "bounded"}],
  "open_questions": ["bounded item"],
  "learned": ["bounded invariant"]
}
```

Goal and done criteria live only in immutable `TASK.md`. Completed work is
derived from Git. The agent does not supply SHA, branch/ref, task hash, lease
identity, completion claims, or a prose report.

Authored handoff fields are treated as bounded data. Orchestra never executes
`next.command` automatically or promotes handoff text into control input.

### 7.3 Checkpoint algorithm

1. Verify lease and exact turn boundary.
2. Obtain and strict-parse bounded handoff answers.
3. Verify immutable `TASK.md`.
4. Remove/exclude transient protocol markers.
5. `git add -A`; staged, unstaged, untracked, renamed, and deleted paths count.
6. Commit to an immutable per-rotation ref if the index differs from `HEAD`.
7. Select current `HEAD` even when the work was already clean and committed.
8. Push exact SHA to `refs/orchestra/tasks/<task>/<rotation>`.
9. Verify `ls-remote` returns that exact SHA.
10. Seal/upload the handoff artifact.
11. Persist local transaction phase `prepared`.
12. Emit idempotent `TaskHandoffPrepared`.

No handoff may reference an object that is only present in a worker-local
object database.

### 7.4 Release transaction

1. After `TaskHandoffPrepared` is durable, persist phase `prepared_acked`.
2. Ask herdr to `pane.release_agent`; retain pane, worktree, report, artifact,
   and transaction state.
3. Reconcile ambiguous herdr results by inspecting the exact pane/agent.
4. Persist phase `predecessor_parked`.
5. Emit idempotent `TaskReleased`.
6. On lost response, pull events by transaction ID and resume the known phase.
7. Successor completes §6.2 and emits `TaskLeaseStarted`.
8. Coordinator queues an idempotent retire command to the predecessor worker.
9. Predecessor closes the exact old pane, cleans protocol markers, persists
   `retired`, and emits `TaskHandoffRetired`.

The predecessor MUST NOT be closed at step 5. It MUST NOT resume work after
release without receiving a new lease epoch.

## 8. Completion

The agent may create an empty, bounded completion-intent marker. The worker
does not trust a narrative completion report.

At a verified idle boundary:

1. Verify lease token and immutable `TASK.md`.
2. If no completion, handoff, or blocker intent exists, issue one bounded
   continuation prompt requiring exactly one of: continue work, completion
   intent, handoff answers, or blocker data.
3. On completion intent, run the configured worker-owned quality gate.
4. Reject a gate that modifies files unless policy explicitly permits it and
   the gate is rerun on the resulting tree.
5. Stage all intended changes, commit if required, and obtain result SHA.
6. Push an immutable result ref.
7. Verify remote ref → exact result SHA.
8. Build a worker-derived report artifact containing gate command/exit,
   result ref/SHA, task hash, timestamps, and pane/session evidence.
9. Record per-lease usage delta.
10. Emit idempotent `TaskCompleted` with the lease token.
11. Only after durable acknowledgement, close the pane and clean the worktree
    according to retention policy.

A late completion from `needs_attention` is valid only for its retained lease
token. A stale or superseded lease can never complete.

## 9. Herdr contract

- Transport is raw JSON-RPC over TCP or Unix socket, not HTTP.
- Every request includes `params`; use `{}` for parameterless methods.
- Protocol is checked with `ping`; protocol 17 is a JSON number.
- Relevant real methods include `worktree.create/open`, `agent.start/get`,
  `agent.prompt`, `pane.get/read`, `pane.send_text/keys`,
  `pane.release_agent`, and `pane.close`.
- `pane.release`, `pane.kill`, `pane.rotation_signal`, and `pane.status` do
  not exist.
- Herdr never creates a handoff or returns a handoff reference.
- `pane.release_agent({pane_id,source,agent})` drops a binding;
  `pane.close({pane_id})` retires the pane.
- UI-changing calls are not retried after an ambiguous post-write transport
  failure. The worker reconciles state first.
- Orchestra never prompts a blocked pane or a visible permission dialog.
- Approval input is bound to an exact pane and capture revision and uses only
  a recognized explicit control.

## 10. Routing, quota, and performance

### 10.1 Scheduling

Router evaluation occurs when:

- a task becomes queued;
- retry time arrives;
- a worker becomes eligible;
- capacity or quota changes.

Candidate order:

1. hard project affinity;
2. supported local project;
3. fresh worker and local-herdr health;
4. capability match;
5. free concurrency;
6. quota headroom;
7. derived importance;
8. creation sequence, then task ID for deterministic ties.

The scheduler uses one immutable task/worker/health snapshot per pass. It MUST
NOT perform network probes or full event-log scans inside the task-candidate
loop.

### 10.2 Quota

Quota and context occupancy are separate:

- occupancy: current session window; drives rotation;
- quota: subscription usage across sessions; drives eligibility.

Maintain indexed 5-hour and 7-day projections per worker/harness. Receipts are
per-lease deltas and sum across rotations. Unknown or stale quota makes a
bounded pool unavailable `[D]`; 80% reported usage is treated as full `[D]`.

### 10.3 Performance requirements

- Active leases, external task keys, worker eligibility, and quota windows
  have indexed projections.
- Event APIs are cursor-paginated and bounded.
- Slow notification/provider consumers cannot block append or scheduling.
- Snapshots are periodic/checkpointed, not rewritten synchronously per event.
- Benchmarks cover 1k and 10k queued tasks and publish append/assignment p50,
  p95, allocation count, and projection rebuild time.
- Default target `[D]`: schedule 10k tasks against 100 worker slots in
  ≤250 ms p95 on the deployment-class coordinator, excluding Git/herdr work.

## 11. Authorization and approvals

Authorization is enforced at the append/command bus:

| Surface | Rights |
|---|---|
| system worker/coordinator | scoped protocol operations |
| web/TUI | full operator control |
| MCP/Maven | read plus approval-gated writes |
| ntfy/telegram | notification only |

Workers are authorized only for their own lease tokens, captures, and command
queue. “System” cannot be selected by an HTTP header.

Approval policy is project-scoped and audited:

- MAY pre-authorize safe reads, edits, tests, formatting, and Git operations
  inside the isolated worktree;
- MUST gate destructive operations, secret access, arbitrary network access,
  privilege changes, and paths outside the worktree;
- has no automatic timeout;
- records requested option, capture revision, actor/policy, worker ACK, and
  resulting lifecycle event.

## 12. Providers, reflection, and operations

- Providers ingest by stable `(source, external_id)` and reflect terminal or
  attention state without becoming task truth.
- Ingestion is idempotent; write-back cannot create an ingestion loop.
- The JSONL provider is the baseline. Gitea/Vikunja adapters use the same port.
- Notifications are asynchronous subscribers with durable cursors.
- The brief is a read-only windowed projection of completions, failures,
  attention, approvals, quota, worker health, and Git refs.
- Mid-day handoff and overnight batch are the same lifecycle. The morning
  brief is the overnight window; workpc pulls verified result refs on wake.
- Parent links project task → epic → roadmap. Sprint is a queued-task
  selection; estimates retain author and confidence.
- Standups are advisory projections and require the same approval path as
  other control changes.
- Shared `AGENTS.md`, `CLAUDE.md`, and vocabulary files are agent-editable
  repository state. Workers hash them at lease time and notify adjacent live
  sessions when the hash changes; pickup always rereads them.
- Deterministic quality-gate residue that is outside the current task creates
  a deduplicated follow-up task instead of relying on an agent promise.
- Health exposes coordinator revision/protocol, worker revision/protocol,
  local-herdr status, active lease/epoch, pane, phase, last error, retry time,
  occupancy source, quota freshness, and event lag.
- Investigative tooling uses read-only herdr calls. Destructive pane operations
  require explicit operator intent outside the protocol owner.

## 13. Release gate

Every production revision MUST pass:

```text
go build ./...
go vet ./...
go test ./...
go test -race ./...
git diff --check
```

Required automated coverage:

1. Strict schema, transition, authorization, and lease-fencing tests.
2. Fuzz/property tests for replay and invalid event/artifact input.
3. Crash injection before/after every append, CAS, worker-state, Git push,
   herdr release, release event, pickup validation, and retire phase.
4. Lost request and lost response tests for every worker mutation.
5. Worker partition, lease self-fence, expiry, reassignment, and stale
   completion tests.
6. Staged, unstaged, untracked, renamed, deleted, and clean-committed handoff
   tests across distinct repositories/machines.
7. Invalid/missing remote ref, task hash, handoff, and pickup tests.
8. Coordinator and worker restart at every handoff phase.
9. Soft, hard, milestone, thrash, manual, completion, attention, approval,
   and late-completion paths.
10. Performance benchmarks from §10.3.

Required controlled live proof on each harness:

```text
lease → start → work → prepare → release → pickup validate
→ successor start → predecessor retire → quality gate → push → complete
```

The proof records event IDs, lease epochs, transaction ID, pane IDs, task hash,
handoff hash, Git refs/SHAs, usage receipt, build revisions, and checksums.
Safe repository work completes without manual approvals under the configured
project policy.

## 14. Migration and defaults

- Existing schema v1/v2 events remain immutable and replay through tolerant
  legacy readers.
- New protocol behavior emits schema v3 events only.
- Migration rebuilds projections; it never rewrites the log.
- Ambiguous legacy blocked/session state is `needs_attention`, never guessed
  active or completed.
- Raw events are retained indefinitely `[D]`.
- No approval timeout `[D]`.
- Hard affinity `[D]`.
- Three failed attempts `[D]`.
- Unknown bounded quota is unavailable `[D]`.
- Unknown occupancy requests a conservative boundary handoff `[D]`.

This specification defines the target. Passing isolated package tests does not
establish conformance; only the live owner path and §13 do.