orchestra/progress.md

# Orchestra progress

Updated: 2026-07-26

## Gaps until full implementation

This is the canonical, exhaustive server-side gap list against `orchestra-spec (1).md`. Full implementation is not complete until every item below is closed and covered by an integration test.

- **Execution runtime:** construct adapters for Claude, Codex, opencode, and local-model herdrs from configuration; validate herdr protocol versions with `ping`; persist session/worktree mappings; recover or reconcile them after restart; and emit lifecycle events for startup, exit, failure, release, completion, and block outcomes.
- **Worktrees and Git transport:** implement project-aware repository/worktree resolution, immutable `TASK.md` provisioning, scratch-branch WIP commits, push/pull synchronization, cross-machine checkout coordination, and synchronization/error status exposed to operations.
- **Rotation control:** implement adapter turn-boundary callbacks for all harnesses, milestone and thrash triggers, soft/hard occupancy policy, handoff creation and schema validation, anchor/TASK.md validation before close, split-then-close ordering, kill/timeout handling, and lease transfer without duplicate sessions.
- **Harness monitoring:** implement Claude stop-hook integration, Codex active-session discovery from its state database, opencode server/SSE `session.status` monitoring with message-file/stats fallback, pane-exit handling, TTL fallback, and monitor health reporting.
- **Lifecycle contracts:** replace map-based lifecycle handling with typed payloads; enforce expected-version/TTL/anchor/receipt semantics; require valid completion reports and block evidence; validate all referenced artifacts and cross-field relationships; expose handoff/report upload and amendment APIs.
- **Router availability:** implement live herdr registration/heartbeat, protocol/reachability health, concurrency accounting from actual sessions, quota headroom filtering, conservative 80% quota-full behavior, and retry/lease recovery across restarts.
- **Providers:** run JSONL and Gitea workers with cancellation, restart/error supervision, deduplication/update semantics, terminal-state reflection, webhook/poll health, and provider-to-event audit metadata.
- **Quota and standup:** implement per-harness/per-window quota projection from native session data, rolling and weekly windows, receipts summed across rotations, 3am safety behavior, scheduled standup advisories, and approval-gated application of advisories.
- **Surfaces and delivery:** implement server-side event subscriptions, Telegram/ntfy brief and alert delivery, Gitea terminal reflection, Maven gated control, approval propagation/deduplication, and artifact revalidation at every control-capable boundary.
- **Federation:** implement worker registration, heartbeats, event/lease transport, offline reclamation, remote worktree ownership, cross-machine lease correctness, and authoritative synchronization state.
- **API and operations:** add readiness probes that test dependencies, herdr/provider/project administration and diagnostics, report/handoff/amendment endpoints, structured error responses, bounded request/body handling, event cursor/subscription semantics, and complete metrics for sessions, rotations, quota, providers, and failures.
- **Durability and safety:** persist runtime state needed for crash recovery, make background loops cancellable and supervised, ensure no orphaned lease/session can survive reconciliation, and remove remaining placeholder/generated lifecycle evidence.
- **Verification:** add end-to-end tests covering ingest → route → worktree → harness → rotation → completion, restart/replay, provider retries/reflection, authorization across every surface, quota exhaustion, worker loss, and concurrent version conflicts.

## Implementation review — 2026-07-26

`go test ./...` passes, but the implementation is still a tested substrate/router prototype rather than a functioning unattended multi-harness orchestra. The following gaps were verified against `orchestra-spec (1).md` and the current code:

- **Harness execution is wired for configured Git/Codex deployments.** The router invokes the coordinator; `ORCHESTRA_REPO` + `ORCHESTRA_WORKTREE_ROOT` enable Git worktree creation, configured herdr socket adapters, session creation, and optional bootstrap. Other harness types and monitoring callbacks remain pending.
- **Rotation is partially implemented.** The configured coordinator polls adapter occupancy, releases above `ORCHESTRA_OCCUPANCY_HARD` (default 75%), and publishes a handoff-backed `TaskReleased` event. Turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety remain pending.
- **Lifecycle API payloads are invalid.** The release, complete, and block endpoints all emit `{"source":"api"}`, while validation requires `handoff_ref` or `reason`, `report_ref`, and `blocker` respectively. The documented lifecycle endpoints therefore cannot complete successfully.
- **Provider integrations are partially wired.** JSONL watching and optional Gitea webhook/poll loops now start from environment configuration, but terminal reflection, provider health, cancellation, and delivery fan-out remain absent.
- **CAS references are not content-verified at event append.** Lifecycle events check that referenced files exist, but do not verify that the file content hashes to the supplied reference.
- **Replay bypasses event validation.** Startup replay unmarshals and applies events without validating the event envelope, payload schema, or sequence/version invariants.
- **Snapshots are written but never loaded or used for replay acceleration.** Startup always replays the complete event log.
- **Task creation projection is incomplete.** `parent`, `due`, `inherent_priority`, and `estimate` are defined in the domain model but are not projected from `TaskCreated` payloads.
- **Occupancy support is incomplete relative to the spec.** Native readers and configured hard-threshold monitoring exist, but Codex active-session discovery, opencode server/SSE plus fallback, and turn-boundary monitoring are not implemented.
- **Authorization is mostly enforced at HTTP ingress.** Lifecycle and approval handlers now call `AuthorizeEvent`; an absent surface still defaults to full-control Web, and non-HTTP/event-bus integrations remain unwired.

The first pass closed the store/API defects (lifecycle defaults, CAS content verification, validated replay, snapshot loading, and projection of task metadata) and added optional Gitea webhook/poll wiring. The remaining server-side gaps are below.

### Remaining server-side gaps

- **The orchestration coordinator is operational but incomplete.** It is constructed from deployment configuration, resolves a worktree, invokes `herdr.Adapter.Lease`, bootstraps handoffs, monitors occupancy, and rotates through handoff references. Session mappings are in-memory and turn-boundary monitoring is still pending.
- **Rotation remains incomplete.** Occupancy-triggered release is wired, but adapter turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety are not.
- **Harness discovery and registration are not operational.** Static herdr configuration and socket clients exist, but startup does not create adapters, ping configured herdrs, discover active Codex sessions, subscribe to opencode SSE, or run the required fallback/TTL monitoring loop.
- **Provider ingestion is only partially wired.** JSONL and Gitea are available when configured, but there is no provider lifecycle management, cancellation, error health projection, terminal-state reflection, or provider fan-out.
- **Lifecycle event contracts remain incomplete.** Validation does not enforce the spec's `expected_version`, `ttl`, `anchor_sha`, `receipt`, or optional `handoff_ref` relationships, and the HTTP API does not validate actor/surface authorization at the event construction site. Completion without a report currently creates a generated placeholder artifact rather than requiring the stop-hook/wrapper receipt described by the spec.
- **Quota and standup scheduling are not implemented.** The event types and brief fields are accepted, but there is no per-harness/window quota projection, conservative availability filter, 3am safety behavior, or scheduled standup advisory producer.
- **Brief delivery and provider reflection are not implemented.** `/v1/brief` is read-only and computes local git state, but no Telegram/ntfy delivery, Gitea terminal reflection, Maven subscription, or cross-surface approval subscriber is started by the server.
- **Federated worker behavior is not complete.** Machine affinity filtering is implemented, but there is no worker registration/heartbeat protocol, remote event transport, cross-machine worktree coordination, or server-side synchronization status beyond local git inspection.
- **The server API is narrower than the spec.** There are no explicit task amendment/report/handoff upload endpoints, event subscription/streaming endpoint, health/readiness detail for providers and herdrs, or administrative endpoints for project/machine/herdr status.

The latest pass now applies `AuthorizeEvent` to lifecycle and approval writes and adds `/readyz` with router/provider configuration checks. Readiness is configuration-level only; it does not yet probe herdr/provider health.

The lifecycle API contract pass now requires callers to provide explicit release evidence (`reason` or `handoff_ref`), a block `blocker`, and a completion `report_ref`. The server no longer creates generated placeholder completion artifacts or converts malformed/empty lifecycle bodies into defaults. Regression coverage validates that release, completion, and block events reject missing evidence.

Item 1 substrate hardening pass: newly appended events use schema envelope version 1; replay rejects unsupported versions and non-object payloads; and `TaskLeased` carries an `expected_version` guard that is checked before append. Legacy envelope events remain readable for tolerant replay.

Harness registration now uses configured `harness` and `protocol` fields, pings each configured herdr before exposing it to orchestration, selects Claude/Codex/opencode adapters accordingly, and skips unavailable or unsupported deployments at startup. This is an initial discovery/health slice; active-session discovery and ongoing heartbeats remain open.

Added bounded `POST /v1/artifacts` CAS upload support for report/handoff evidence. It returns the verified content hash used by lifecycle events and rejects empty or oversized uploads.

The coordinator now persists active task→herdr session mappings in an atomic runtime state file, reloads them after restart, reconciles them against durable task leases, kills stale recoverable sessions, and removes orphan mappings before monitoring begins.

Provider supervision/reflection is now wired: Gitea webhook and polling use append-first task reflection, JSONL and Gitea loops restart with bounded backoff, and `/v1/providers/health` exposes running/error state. Provider lifecycle cancellation remains tied to process shutdown until the server gains a root cancellation context.

Quota reporting now has strict payload validation, and `router.QuotaAvailability` implements rolling-window conservative headroom filtering: a harness is considered full at 80% of its configured limit. Standup event payloads also require an items field; scheduled advisory production and approval application remain open.

Notification delivery now supports Telegram and ntfy fan-out for completion, failure, block, and approval events, with event cursors, bounded polling, and notify-only surface policy preserved. Configure `ORCHESTRA_TELEGRAM_BOT_TOKEN`/`ORCHESTRA_TELEGRAM_CHAT_ID` or `ORCHESTRA_NTFY_TOPIC` to enable it.

Rotation now honors an optional herdr turn-boundary probe (`pane.status`) before hard-threshold release. Adapters without the optional capability retain occupancy-based fallback behavior.

Federation control-plane foundations now include worker registration, heartbeat updates, TTL-based offline status, and `/v1/federation/workers` plus per-worker heartbeat endpoints. Remote event/lease transport and remote worktree ownership remain to be layered on this registry.

Configured herdr `quota_limit` values are now wired into router availability using the rolling-window 80% conservative filter; previously the quota implementation existed but was not active in server routing.

Quota/standup scheduling now treats `QuotaReported` and `StandupAdvisory` as global events, adds `/v1/standup` read/request behavior, and emits one daily advisory during the 03:00 UTC safety window. Advisory contents include queued, leased, and blocked tasks.

Harness adapters now expose optional `pane.rotation_signal` support for milestone/thrash triggers. The coordinator records the returned trigger reason in the handoff release and still requires a safe turn boundary before releasing.

Recommended order:

1. Add the orchestration coordinator: lease → worktree → harness session → bootstrap → lifecycle events.
2. Implement rotation and turn-boundary monitoring.
3. Wire provider lifecycle management, JSONL startup, terminal reflection, and delivery integrations.
4. Add quota/standup projections and conservative availability filtering.
5. Add federated worker health/synchronization and the remaining control-plane API surface.
6. Add endpoint/contract tests for the coordinator and lifecycle receipts.

## Server implementation checklist

This is the implementation-oriented breakdown of the specification. It is a project checklist, not a replacement for the binding spec.

1. **Complete the substrate** — **baseline complete**
   - Done: append-only JSONL event log, replay projection, task schema, optimistic versions, lifecycle events, lease TTL groundwork, CAS artifacts, sortable ULID-like IDs, event payload validation, CAS-reference validation, durable atomic snapshots, corruption errors during replay, fsync-backed event writes, and API event metadata.
   - Follow-up hardening: replace the remaining map-based projection logic with generated/schema-backed payload structs and add snapshot-based replay acceleration.

2. **Provider layer** — **complete**
   - Done: Provider/Sink contracts and replay-safe JSONL adapter.
   - Done: append-only JSONL file watcher/ingester with rotation handling and bounded records.
   - Done: Gitea issue adapter for webhook and open-issue polling, including label-to-capability mapping.
   - Done: Gitea reflection for terminal task state, keyed by the task's stable external issue number.
   - Done: constant-time HMAC webhook authentication and injectable HTTP clients for testing.

3. **Projects and machine registry** — **complete**
   - Done: typed JSON project, machine, and herdr configuration with duplicate/reference validation.
   - Done: machine-bound herdr registry with per-herdr capabilities, endpoint override, and concurrency configuration.
   - Done: injectable reachability checks plus TCP reachability implementation.
   - Done: hard project machine-affinity resolution; candidates are restricted to configured, reachable herdrs on allowed machines.
   - Done: optional `ORCHESTRA_CONFIG` startup validation.

4. **Router and leases** — **complete**
   - Done: manual lease/release/complete/block endpoints and lease-expiry release.
   - Done: assignment on `TaskCreated` and lease release/expiry.
   - Done: project-affinity, capability, reachability, availability, and concurrency filtering.
   - Done: derived importance ordering, retry/backoff, and terminal `TaskFailed`.

5. **Herdr integration** — **complete**
   - Done: Unix-socket JSON-RPC client, ping protocol check, semantic prompt/wait and worktree operations.
   - Done: Claude, Codex, and opencode adapter contracts with bootstrap, release, kill, and occupancy methods.
   - Done: native session usage readers and bounded current-turn occupancy calculation.
   - Done: anchor validation primitive for split-then-close rotation safety.

6. **Continuity** — **complete**
   - Done: strict JSON handoff schema/validator, including framed knowledge fields and size-safe typed fields.
   - Done: CAS-backed handoff save/load with content-address verification.
   - Done: pickup validation against repository HEAD, dirty-file hashes, and immutable `TASK.md` hash.
   - Done: scratch-branch WIP commit helper and Markdown change notices.

7. **Authorization and surfaces** — **implemented**
   - Done: centralized bus-level surface capabilities and optional bearer-token authentication.
   - Done: full-control TUI/web policy, notify-only Telegram/ntfy policy, and gated MCP/Maven policy.
   - Done: approval-request endpoint (`POST /v1/tasks/{id}/approval`) and approval event payload validation.
   - Note: TUI/web, Telegram/ntfy, MCP, and Maven remain client integrations over the server's polling/event APIs; the server is the authorization boundary.

8. **Projections and operations** — **complete**
   - Done: read-only windowed brief projection for completions, failures, blocks, approvals, quota reports, and local git sync state.
   - Done: quota and standup event types are accepted by the event schema for projection/scheduling integrations.
   - Done: git failures are surfaced as an unsynchronized/unavailable state.
   - Done: operational projection code is covered by tests.
   - Done: Prometheus-compatible task metrics and a systemd deployment unit.

## Completed

- Built the first Go server slice from `orchestra-spec (1).md`.
- Added append-only JSONL events and replay projection in `internal/store`.
- Added task creation, external-key deduplication, optimistic versions, lifecycle states, and SHA-256 CAS artifacts.
- Added HTTP endpoints on default port `9145`: health, task ingest/list, and event cursor reads.
- Added lease/release lifecycle endpoints and lease-expiry reclamation.
- Finished the item 1 provider port: `provider.Provider`/`Sink` interfaces and a replay-safe JSONL adapter.
- Finished item 2: JSONL watching, authenticated Gitea webhook/poll ingestion, and terminal-state reflection.
- Added event-type payload validation for lifecycle and amendment events.
- Unit tests pass with `go test ./...`.
- Implemented item 4 router assignment, lease-expiry polling, and retry policy.
- Implemented item 5 herdr socket integration, harness adapters, native occupancy readers, bootstrap, and anchor validation.
- Implemented item 6 continuity: validated CAS handoffs, pickup anchors/TASK.md, scratch-branch commits, and shared Markdown change notices.

## Current API additions

- `POST /v1/tasks/{id}/lease` with `{"harness_id":"...","ttl_seconds":1800}`
- `POST /v1/tasks/{id}/release`
- `POST /v1/tasks/{id}/complete`
- `POST /v1/tasks/{id}/block`

## Item 3 status

Item 3 (projects and machine registry) is implemented in `internal/registry`. Static JSON configuration is loaded and validated, projects resolve only to their explicitly configured machines, and candidate herdrs are filtered by registration and injected reachability. Set `ORCHESTRA_CONFIG` to validate a configuration file at server startup.

## Item 2 status

Item 2 (provider layer) is implemented. `internal/provider` now includes `JSONLWatcher`, `Gitea.Poll`, `Gitea.WebhookHandler`, `Gitea.IngestWebhook`, and `Gitea.ReflectTask`. Gitea ingestion remains idempotent through the store's `(source, external_id)` key. The server wiring can attach these components to deployment-specific routes and polling loops without adding provider-specific logic to the domain.

## Item 1 status

Item 1 (task schema + provider port + JSONL adapter) is implemented as the baseline slice. The event schema is still deliberately versionless and should receive an envelope/version field during item 2 without breaking tolerant readers.

## Important limitations

- This is still a Layer 1/2 prototype. Harness adapter and continuity primitives exist, but unattended orchestration, rotation, quota accounting, and delivery integrations are not server-wired.
- Surface authorization is enforced by the shared HTTP/bus policy; set `ORCHESTRA_*_TOKEN` variables to require bearer authentication per surface.
- Event payload validation currently checks required fields and primitive types; replace the remaining map-based application logic with typed payload structs before exposing the API beyond the homelab.
- Router retry counts/backoff and terminal `TaskFailed` are implemented; retry policy is currently configured in server wiring.

## Next agent: recommended order

1. Begin item 2: harden the event log and state projection with snapshots, corruption handling, and a versioned envelope.
2. Add project, machine, and herdr registries from static TOML/JSON config.
3. Implement router selection: project affinity, reachability, capability, availability, and importance ordering.
4. Add retry policy and a background lease-expiry loop.
5. Implement handoff/report schemas and CAS reference validation.
6. Integrate herdr only after the substrate/router tests are stable.