orchestra/progress.md

# Orchestra progress

Updated: 2026-07-26

## Implementation review — 2026-07-26

`go test ./...` passes, but the implementation is still a tested substrate/router prototype rather than a functioning unattended multi-harness orchestra. The following gaps were verified against `orchestra-spec (1).md` and the current code:

- **Harness execution is wired for configured Git/Codex deployments.** The router invokes the coordinator; `ORCHESTRA_REPO` + `ORCHESTRA_WORKTREE_ROOT` enable Git worktree creation, configured herdr socket adapters, session creation, and optional bootstrap. Other harness types and monitoring callbacks remain pending.
- **Rotation is partially implemented.** The configured coordinator polls adapter occupancy, releases above `ORCHESTRA_OCCUPANCY_HARD` (default 75%), and publishes a handoff-backed `TaskReleased` event. Turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety remain pending.
- **Lifecycle API payloads are invalid.** The release, complete, and block endpoints all emit `{"source":"api"}`, while validation requires `handoff_ref` or `reason`, `report_ref`, and `blocker` respectively. The documented lifecycle endpoints therefore cannot complete successfully.
- **Provider integrations are partially wired.** JSONL watching and optional Gitea webhook/poll loops now start from environment configuration, but terminal reflection, provider health, cancellation, and delivery fan-out remain absent.
- **CAS references are not content-verified at event append.** Lifecycle events check that referenced files exist, but do not verify that the file content hashes to the supplied reference.
- **Replay bypasses event validation.** Startup replay unmarshals and applies events without validating the event envelope, payload schema, or sequence/version invariants.
- **Snapshots are written but never loaded or used for replay acceleration.** Startup always replays the complete event log.
- **Task creation projection is incomplete.** `parent`, `due`, `inherent_priority`, and `estimate` are defined in the domain model but are not projected from `TaskCreated` payloads.
- **Occupancy support is incomplete relative to the spec.** Native readers and configured hard-threshold monitoring exist, but Codex active-session discovery, opencode server/SSE plus fallback, and turn-boundary monitoring are not implemented.
- **Authorization is mostly enforced at HTTP ingress.** Lifecycle and approval handlers now call `AuthorizeEvent`; an absent surface still defaults to full-control Web, and non-HTTP/event-bus integrations remain unwired.

The first pass closed the store/API defects (lifecycle defaults, CAS content verification, validated replay, snapshot loading, and projection of task metadata) and added optional Gitea webhook/poll wiring. The remaining server-side gaps are below.

### Remaining server-side gaps

- **The orchestration coordinator is operational but incomplete.** It is constructed from deployment configuration, resolves a worktree, invokes `herdr.Adapter.Lease`, bootstraps handoffs, monitors occupancy, and rotates through handoff references. Session mappings are in-memory and turn-boundary monitoring is still pending.
- **Rotation remains incomplete.** Occupancy-triggered release is wired, but adapter turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety are not.
- **Harness discovery and registration are not operational.** Static herdr configuration and socket clients exist, but startup does not create adapters, ping configured herdrs, discover active Codex sessions, subscribe to opencode SSE, or run the required fallback/TTL monitoring loop.
- **Provider ingestion is only partially wired.** JSONL and Gitea are available when configured, but there is no provider lifecycle management, cancellation, error health projection, terminal-state reflection, or provider fan-out.
- **Lifecycle event contracts remain incomplete.** Validation does not enforce the spec's `expected_version`, `ttl`, `anchor_sha`, `receipt`, or optional `handoff_ref` relationships, and the HTTP API does not validate actor/surface authorization at the event construction site. Completion without a report currently creates a generated placeholder artifact rather than requiring the stop-hook/wrapper receipt described by the spec.
- **Quota and standup scheduling are not implemented.** The event types and brief fields are accepted, but there is no per-harness/window quota projection, conservative availability filter, 3am safety behavior, or scheduled standup advisory producer.
- **Brief delivery and provider reflection are not implemented.** `/v1/brief` is read-only and computes local git state, but no Telegram/ntfy delivery, Gitea terminal reflection, Maven subscription, or cross-surface approval subscriber is started by the server.
- **Federated worker behavior is not complete.** Machine affinity filtering is implemented, but there is no worker registration/heartbeat protocol, remote event transport, cross-machine worktree coordination, or server-side synchronization status beyond local git inspection.
- **The server API is narrower than the spec.** There are no explicit task amendment/report/handoff upload endpoints, event subscription/streaming endpoint, health/readiness detail for providers and herdrs, or administrative endpoints for project/machine/herdr status.

The latest pass now applies `AuthorizeEvent` to lifecycle and approval writes and adds `/readyz` with router/provider configuration checks. Readiness is configuration-level only; it does not yet probe herdr/provider health.

Recommended order:

1. Add the orchestration coordinator: lease → worktree → harness session → bootstrap → lifecycle events.
2. Implement rotation and turn-boundary monitoring.
3. Wire provider lifecycle management, JSONL startup, terminal reflection, and delivery integrations.
4. Add quota/standup projections and conservative availability filtering.
5. Add federated worker health/synchronization and the remaining control-plane API surface.
6. Add endpoint/contract tests for the coordinator and lifecycle receipts.

## Server implementation checklist

This is the implementation-oriented breakdown of the specification. It is a project checklist, not a replacement for the binding spec.

1. **Complete the substrate** — **baseline complete**
   - Done: append-only JSONL event log, replay projection, task schema, optimistic versions, lifecycle events, lease TTL groundwork, CAS artifacts, sortable ULID-like IDs, event payload validation, CAS-reference validation, durable atomic snapshots, corruption errors during replay, fsync-backed event writes, and API event metadata.
   - Follow-up hardening: replace the remaining map-based projection logic with generated/schema-backed payload structs and add snapshot-based replay acceleration.

2. **Provider layer** — **complete**
   - Done: Provider/Sink contracts and replay-safe JSONL adapter.
   - Done: append-only JSONL file watcher/ingester with rotation handling and bounded records.
   - Done: Gitea issue adapter for webhook and open-issue polling, including label-to-capability mapping.
   - Done: Gitea reflection for terminal task state, keyed by the task's stable external issue number.
   - Done: constant-time HMAC webhook authentication and injectable HTTP clients for testing.

3. **Projects and machine registry** — **complete**
   - Done: typed JSON project, machine, and herdr configuration with duplicate/reference validation.
   - Done: machine-bound herdr registry with per-herdr capabilities, endpoint override, and concurrency configuration.
   - Done: injectable reachability checks plus TCP reachability implementation.
   - Done: hard project machine-affinity resolution; candidates are restricted to configured, reachable herdrs on allowed machines.
   - Done: optional `ORCHESTRA_CONFIG` startup validation.

4. **Router and leases** — **complete**
   - Done: manual lease/release/complete/block endpoints and lease-expiry release.
   - Done: assignment on `TaskCreated` and lease release/expiry.
   - Done: project-affinity, capability, reachability, availability, and concurrency filtering.
   - Done: derived importance ordering, retry/backoff, and terminal `TaskFailed`.

5. **Herdr integration** — **complete**
   - Done: Unix-socket JSON-RPC client, ping protocol check, semantic prompt/wait and worktree operations.
   - Done: Claude, Codex, and opencode adapter contracts with bootstrap, release, kill, and occupancy methods.
   - Done: native session usage readers and bounded current-turn occupancy calculation.
   - Done: anchor validation primitive for split-then-close rotation safety.

6. **Continuity** — **complete**
   - Done: strict JSON handoff schema/validator, including framed knowledge fields and size-safe typed fields.
   - Done: CAS-backed handoff save/load with content-address verification.
   - Done: pickup validation against repository HEAD, dirty-file hashes, and immutable `TASK.md` hash.
   - Done: scratch-branch WIP commit helper and Markdown change notices.

7. **Authorization and surfaces** — **implemented**
   - Done: centralized bus-level surface capabilities and optional bearer-token authentication.
   - Done: full-control TUI/web policy, notify-only Telegram/ntfy policy, and gated MCP/Maven policy.
   - Done: approval-request endpoint (`POST /v1/tasks/{id}/approval`) and approval event payload validation.
   - Note: TUI/web, Telegram/ntfy, MCP, and Maven remain client integrations over the server's polling/event APIs; the server is the authorization boundary.

8. **Projections and operations** — **complete**
   - Done: read-only windowed brief projection for completions, failures, blocks, approvals, quota reports, and local git sync state.
   - Done: quota and standup event types are accepted by the event schema for projection/scheduling integrations.
   - Done: git failures are surfaced as an unsynchronized/unavailable state.
   - Done: operational projection code is covered by tests.
   - Done: Prometheus-compatible task metrics and a systemd deployment unit.

## Completed

- Built the first Go server slice from `orchestra-spec (1).md`.
- Added append-only JSONL events and replay projection in `internal/store`.
- Added task creation, external-key deduplication, optimistic versions, lifecycle states, and SHA-256 CAS artifacts.
- Added HTTP endpoints on default port `9145`: health, task ingest/list, and event cursor reads.
- Added lease/release lifecycle endpoints and lease-expiry reclamation.
- Finished the item 1 provider port: `provider.Provider`/`Sink` interfaces and a replay-safe JSONL adapter.
- Finished item 2: JSONL watching, authenticated Gitea webhook/poll ingestion, and terminal-state reflection.
- Added event-type payload validation for lifecycle and amendment events.
- Unit tests pass with `go test ./...`.
- Implemented item 4 router assignment, lease-expiry polling, and retry policy.
- Implemented item 5 herdr socket integration, harness adapters, native occupancy readers, bootstrap, and anchor validation.
- Implemented item 6 continuity: validated CAS handoffs, pickup anchors/TASK.md, scratch-branch commits, and shared Markdown change notices.

## Current API additions

- `POST /v1/tasks/{id}/lease` with `{"harness_id":"...","ttl_seconds":1800}`
- `POST /v1/tasks/{id}/release`
- `POST /v1/tasks/{id}/complete`
- `POST /v1/tasks/{id}/block`

## Item 3 status

Item 3 (projects and machine registry) is implemented in `internal/registry`. Static JSON configuration is loaded and validated, projects resolve only to their explicitly configured machines, and candidate herdrs are filtered by registration and injected reachability. Set `ORCHESTRA_CONFIG` to validate a configuration file at server startup.

## Item 2 status

Item 2 (provider layer) is implemented. `internal/provider` now includes `JSONLWatcher`, `Gitea.Poll`, `Gitea.WebhookHandler`, `Gitea.IngestWebhook`, and `Gitea.ReflectTask`. Gitea ingestion remains idempotent through the store's `(source, external_id)` key. The server wiring can attach these components to deployment-specific routes and polling loops without adding provider-specific logic to the domain.

## Item 1 status

Item 1 (task schema + provider port + JSONL adapter) is implemented as the baseline slice. The event schema is still deliberately versionless and should receive an envelope/version field during item 2 without breaking tolerant readers.

## Important limitations

- This is still a Layer 1/2 prototype. Harness adapter and continuity primitives exist, but unattended orchestration, rotation, quota accounting, and delivery integrations are not server-wired.
- Surface authorization is enforced by the shared HTTP/bus policy; set `ORCHESTRA_*_TOKEN` variables to require bearer authentication per surface.
- Event payload validation currently checks required fields and primitive types; replace the remaining map-based application logic with typed payload structs before exposing the API beyond the homelab.
- Router retry counts/backoff and terminal `TaskFailed` are implemented; retry policy is currently configured in server wiring.

## Next agent: recommended order

1. Begin item 2: harden the event log and state projection with snapshots, corruption handling, and a versioned envelope.
2. Add project, machine, and herdr registries from static TOML/JSON config.
3. Implement router selection: project affinity, reachability, capability, availability, and importance ordering.
4. Add retry policy and a background lease-expiry loop.
5. Implement handoff/report schemas and CAS reference validation.
6. Integrate herdr only after the substrate/router tests are stable.