# Orchestra progress Updated: 2026-07-27 ## AUDIT.md remediation — in progress Working through `AUDIT.md`'s blocking/secondary defects in order of the "suggested order of attack." Each item below is landed, tested, and committed individually; see the git log for the exact commits. Fixed so far: - **B2** — adapters were looked up by `session.Harness` (the harness kind, e.g. `"claude"`) in `Reconcile`/`expire`/`rotate`, but `AdapterFactory.Herdrs` is keyed by herdr instance id (e.g. `"homesrv-claude"`). Every one of those call sites silently no-opped. Added `Coordinator.adapterFor`, routed all four call sites through it. Regression test registers an adapter under a herdr-id key distinct from the harness kind and asserts rotation fires. - **B1** — `CLIAdapter.Occupancy` called `a.Usage(s.PaneID)`, but the usage readers want a filesystem path to session state, not a herdr pane id. Added `herdr.Session.SessionFile` and per-harness resolution (`ClaudeSessionFile` by newest-mtime under Claude Code's own project directory; codex via the existing `CodexActiveUsage` sqlite discovery; opencode refuses loudly — needs a live session id, not resolvable from the worktree alone). A missing/unreadable session file is now a hard error, surfaced via new `SessionHealth.Occupancy`/`OccupancyError` fields on `GET /v1/tasks/{id}/health`, not a silent zero. **Still needs live verification against a real Claude Code session** (the spec's own acceptance bar for this phase) — not possible from this sandbox. - **B4** — the router counted every `TaskReleased` (including rotation, which *is* a `TaskReleased` carrying a valid `handoff_ref`) against `MaxAttempts`, and double-counted by also incrementing on every subsequent lease. A task that rotated twice hit the default `MaxAttempts=3` and was killed. Now only a release without a `handoff_ref` (expiry/crash) advances the counter. - **B8** — `X-Orchestra-Surface: system` was reachable from an HTTP request header in both `authz.HTTP` and `main.go`'s `surface` closure (the one every handler actually calls). Since no deployment sets `ORCHESTRA_SYSTEM_TOKEN`, this was an unauthenticated full-control bypass reachable from any LAN caller. Both call sites now downgrade `system` to `web` before doing anything else with it. - **S1** — `Brief.From`/`To` and `GitSync.Branch`/`Head`/`Status` all shared one JSON tag each (Go only honors the first `json:"..."` tag on a combined field declaration). `go vet ./...` now passes clean. - **S5** — `Store.Lease`/`ExpireLeases` set `Event.ID` to the task id, so every lease of a task produced colliding event IDs. Now `domain.NewID()`. - **S6** — the ingest dedup path returned `nil` (success) without appending; `main.go` then returned an unrelated event with `201`. Added `domain.ErrDuplicate` and `Store.TaskBySource`; `POST /v1/tasks` now returns the existing task with `200` on a duplicate. Updated every other `Append` caller (Gitea poll/webhook, JSONL ingest) to treat `ErrDuplicate` as expected rather than a failure — without that, Gitea polling would error out of its scan loop on the first already-ingested issue in every batch. - **B3 (partial)** — added `POST /v1/harness/complete`, the first automatic `TaskCompleted` producer (previously only a human calling `/v1/tasks/{id}/complete` could ever complete a task). A Claude Code Stop hook (`deploy/hooks/orchestra-stop.sh`) fires on every turn boundary but only reports completion if the agent has written a `.orchestra-report.md` marker at the worktree root first — an ordinary turn boundary is a no-op, so this doesn't fire completion prematurely. The server reads the transcript locally via `herdr.ClaudeUsage` to build the `receipt` itself (input/cache/output token counts) rather than trusting a self-reported number, and uploads the report body to CAS for `report_ref`. Guarded by an optional `ORCHESTRA_HARNESS_TOKEN` bearer check; the event is appended with `Surface: system` set directly in Go (not derived from a request header — consistent with the B8 fix that system must never be header-controlled). **Not done:** Codex/opencode equivalents (Claude-only for now — Codex would need `CodexActiveUsage`, opencode `OpenCodeUsage`/`OpenCodeStatus`, wired the same way), and the turn-boundary decision endpoint (`continue`/`prepare_handoff`/`rotate_now`/`refuse`) from Phase 2 items 1–2 is still unbuilt — only the completion half of Phase 2 landed. No test added for the new HTTP handler; `cmd/orchestra/main.go` has zero test coverage for any handler (pre-existing gap, everything lives inline in `main()`) so this follows the existing (untested) pattern rather than introducing a one-off test harness. - **B5 (loose end)** — `CLIAdapter.Lease`'s initial prompt used `wait=0`, skipping the inline wait `Bootstrap` already used; the spec (§5.1) requires inline `wait` on `agent.prompt` for bootstrap injection to avoid sending into a half-rendered prompt. Changed to `time.Minute`, matching `Bootstrap`. Small, contained fix — `Release`'s real implementation (needs Phase 4 handoff production) is still outstanding from B5. - **B6 (partial — Phase 4 items 1 and 4)** — nothing wrote a `TASK.md` into a worktree, so pickup validation had nothing to check and never ran anyway. Fixed both halves: `GitWorktrees.Create` now writes and commits an immutable `TASK.md` (`continuity.RenderTaskFile`) into every freshly created worktree, and `Coordinator.Start` now runs `continuity.ValidatePickup` (loading the handoff from CAS, checking anchor SHA + dirty-file hashes + TASK.md hash) before bootstrapping a successor onto a `handoff_ref` — a failure kills the session and emits `TaskBlocked` instead of trusting an unvalidated ref. Covered by `TestGitWorktreesCommitsTaskFile` and `TestStartBlocksOnInvalidPickup` in `internal/orchestrator`. **Not done:** handoff *production* (nothing yet writes a real §6.1 handoff — `Release` still refuses per B5), wiring `ScratchCommit` before release, and the §6.2 bootstrap-prompt rewrite. See AUDIT.md's "B6 — partial fix" section for the full breakdown, including a named caveat: TASK.md hashing is best-effort and untested for the herdr-hosted (`WorktreeCreator`) worktree path. - **B5 (closed)** — `CLIAdapter.Release` previously just refused (no real herdr method existed to call and there was nothing to validate against). Now: reads the agent-authored `.orchestra-handoff.json` from the worktree root, validates it with `continuity.Decode`, cross-checks its anchor SHA against the worktree's real `HeadSHA` (never trusts the agent's self-report outright), uploads it to CAS via `continuity.Save` to mint the `handoff_ref`, and only then calls the real `pane.release_agent({pane_id, source, agent})` to drop herdr's claim — sequenced last so a herdr-side error can't strand an uploaded handoff. Any failure (missing file, invalid schema, anchor mismatch, herdr error) is a refusal, which `rotate` already treats as "retry next tick" rather than stranding the task. `herdr.Claude/ Codex/OpenCode` now take a `continuity.CAS` (main.go passes the existing `*store.Store`). New tests in `internal/herdr/adapter_test.go` cover all four paths against a real git worktree and a fake in-process herdr listener. **Not done:** nothing yet makes the agent actually *write* `.orchestra-handoff.json` (needs a stop-hook convention analogous to `.orchestra-report.md`) — that and the rest of Phase 4 (ScratchCommit before release, §6.2 bootstrap-prompt rewrite, `MarkdownChanges`) remain open. - **Phase 4 items 3, 5, 6** — `CLIAdapter.Release` now re-verifies every `Anchor.Dirty` file hash (previously only the top-level `Anchor.GitSHA` was checked; a file edited after the handoff was written but before release would have gone through unnoticed), then, if there were dirty entries, snapshots them atomically onto a per-task scratch branch (`continuity.ScratchCommit`, made idempotent so a task can rotate more than once) and rewrites the handoff's anchor to that new commit with `Dirty` cleared before uploading — so the successor's pickup check is a single HEAD compare, not N file rehashes. `CLIAdapter.Bootstrap`'s prompt was rewritten to point the agent at `git log`/the scratch branch instead of a vague "read the handoff" instruction, and deliberately avoids claiming a `GET /v1/artifacts/` endpoint, since no such route exists (`/v1/artifacts` is POST-only). `continuity.MarkdownChanges` (§6.3 adjacent-task notice) had zero callers and zero tests despite being listed as implemented in an earlier snapshot — deleted rather than half-wired, per AUDIT.md's explicit "delete and record the deviation" option. New tests: `TestReleaseScratchCommitsDirtyFilesBeforeUpload`, `TestReleaseRefusesOnStaleDirtyFile` (internal/herdr/adapter_test.go). **§6.3 rewired for real, 2026-07-27 (later same day):** the deleted `MarkdownChanges` above was zero-caller dead code, but the underlying spec requirement ("on update, the orchestra injects a notice to agents whose current task is adjacent") wasn't abandoned — rebuilt independently. `continuity.ConventionsHash(root)` hashes whichever of `AGENTS.md`/`CLAUDE.md`/`VOCAB.md` exist at a path; `herdr.Session` gained `ConventionsHash`, snapshotted from the fresh worktree at `Coordinator.Start`; a new `Coordinator.checkConventions`, run every `Monitor` tick, recomputes the hash of the project's *base repo* (via `WorktreeSpec.Spec` — "adjacent" = same project) for every leased session and compares it against that session's stored snapshot. A mismatch calls a new optional `herdr.ConventionsNotifier` capability (`CLIAdapter.NotifyConventionsChanged`, an in-pane `agent.prompt` telling the agent to re-read the docs) and updates the stored hash so the notice fires once per drift, not every tick. Covered by `TestConventionsDriftNotifiesActiveSession` (internal/orchestrator/rotation_test.go): asserts no notification while the base repo is unchanged, then one once it diverges. **Was still open:** Phase 4 item 2 — nothing drove *any* harness to write `.orchestra-handoff.json`, since Release only validated a file whose existence was never solicited. **Closed 2026-07-27:** `rotate()` now checks for the adapter's optional `herdr.HandoffRequester` capability; when `HandoffFile` is missing at the worktree root, it prompts the agent once (`CLIAdapter.RequestHandoff`, mirroring the `.orchestra-report.md`/B3 convention — the plane asks for a handoff, it never invents one) and skips Release that tick, retrying every subsequent tick until the file appears. `herdr.Session.HandoffRequested` avoids re-prompting every tick. Covered by `TestRotationRequestsHandoffBeforeReleasing` (`internal/orchestrator/rotation_test.go`), which asserts Release is never called before the file exists and fires once it does. Codex/opencode still share this same path (no harness-specific gap remains); the only leftover question is whether each harness's own Stop-equivalent hook honors the in-pane prompt to write the file before exiting, which is a live-deployment fact, not something provable from source. - **B7 (post-hoc producer) + Phase 2 turn-decision endpoint** — landed together, since both are new `QuotaReported`/turn-boundary paths off the same completion/turn events. `POST /v1/harness/complete` now appends a `QuotaReported` event (`harness_id` from the closing lease, `consumed` from the same `usage.Numerator()` used for the receipt), so the router's 5h/weekly availability filter and the brief's `quota_consumed` stop evaluating against a permanent zero. New `Coordinator.TurnDecision` (`internal/orchestrator/orchestrator.go`) mirrors `rotate()`'s per-task logic (occupancy → turn-boundary → handoff-file → release) but runs synchronously once per turn instead of waiting for `Monitor`'s ticker, returning one of `continue`/`prepare_handoff`/`rotate_now`/`refuse` via the new `POST /v1/harness/turn`. The Claude Stop hook (`deploy/hooks/orchestra-stop.sh`) now calls this endpoint on every ordinary turn boundary (report marker absent) instead of no-op'ing, and exits 2 on `refuse` to stop the harness from finishing an unsafe turn. Covered by `TestTurnDecision` (`internal/orchestrator/rotation_test.go`): continue-below-threshold, refuse-when-not-at-boundary, and rotate_now-releases-and-emits-a-valid-TaskReleased cases. **Not done:** live per-harness *push* producers (Claude statusline, Codex rollout tail) that would give B7 a second, continuous producer independent of task completion — recorded as a design investigation in AUDIT.md ("Real harness quota sources") but not implemented; Codex/ opencode's own equivalents of the Claude Stop hook (whether their turn-boundary mechanism actually calls `/v1/harness/turn`) also remain unbuilt, same caveat as Phase 2 item 4 already named for `/complete`. - **S4** — `delivery.Fanout.Run` used to `return` on the first sender error, permanently killing the notification goroutine (a single ntfy hiccup meant no notifications for the rest of the process's lifetime, since nothing restarts it). Failed sends now go through an `OnError` hook instead of aborting the loop. Cursor is also persisted now (`SaveCursor` → a `delivery-cursor` file next to `ORCHESTRA_DATA`, loaded on startup), so a restart resumes from the last delivered event instead of re-notifying the entire log from seq 0. `internal/delivery` previously had zero tests; added `TestFanoutContinuesAfterSendError`. Not yet started: Codex/opencode completion producers, S2–S3, S7–S11. See `AUDIT.md` for the full plan. **Phase 0 done (2026-07-27):** this box has live TCP reachability to the real herdr instance at `192.168.1.105:9245` — verified by hand (raw JSON-RPC probes, no `herdr` CLI available locally). Real method list captured in `deploy/herdr-schema.json`. Confirmed `pane.release`/`pane.kill`/ `pane.rotation_signal` are invented, as AUDIT.md's B5 suspected. `pane.kill`→`pane.close` fixed as a drop-in. `pane.rotation_signal`/ `RotationSignal` deleted (no replacement exists). `Release` now refuses loudly instead of calling a nonexistent method — its real implementation needs Phase 4 (handoff production) first, since even the real `pane.release_agent` can't return a `handoff_ref` (herdr doesn't write handoffs, the agent does). See AUDIT.md's new "Phase 0 — done" section for full detail. **Also found: a real task is currently stuck live** — workspace `wA`, task `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode, pane `wA:p1`, blocked — deliberately not touched from this session. ## Current state This is a working Go implementation of `orchestra-spec (1).md`'s Layer 1–3 (substrate, harness/rotation, continuity) plus a first cut of Layer 4 (surfaces). `go build ./...` and `go test ./...` both pass. The codebase is small (~4.6k lines across `internal/{domain,store,provider,registry,router, herdr,orchestrator,continuity,federation,delivery,authz,operations,admin}` and `cmd/orchestra/main.go`). Earlier revisions of this file accumulated a long, self-contradictory chronological log — gaps were listed as open in one section and then claimed closed in a later section, sometimes inaccurately. This revision replaces that log with one audited snapshot. Treat prior git history of this file as session notes, not as ground truth. ### Verified fixed this pass - **Rotation emitted an invalid `TaskReleased` (the previously reported highest-priority defect) — now fixed.** `internal/orchestrator.Coordinator.rotate` built the release payload as `{"handoff_ref","reason"}`, omitting the `anchor_sha` the spec (§4, §6.2) and `domain.ValidatePayload` require whenever `handoff_ref` is present. `store.Append` would reject it, the error was discarded (`if c.Store.Append(e) == nil`), and the lease/session silently never rotated — the coordinator would just retry next tick with no visible failure. Fixed by adding `herdr.HeadSHA(worktree)` and having `rotate` populate `anchor_sha` from the real worktree HEAD before appending; if the anchor can't be read, rotation now correctly skips that tick (leaving the lease intact for TTL/next-tick reclaim) instead of emitting a payload guaranteed to fail validation. Covered by `internal/orchestrator/rotation_test.go` (`TestRotationEmitsValidReleaseWithAnchorSHA`), which drives the real `Coordinator.Monitor` loop against an actual git worktree and asserts the emitted event passes `domain.ValidatePayload` with the correct SHA — the previous end-to-end test masked this bug by manually crafting a replacement `TaskReleased` event after observing the (silently failed) adapter-side release. - **The federation worker release endpoint had the same gap.** The `/v1/federation/workers/{id}/release` handler (cmd/orchestra/main.go) built `TaskReleased` from a request body with only `handoff_ref`, no `anchor_sha`. Since a remote worker is the only party with the actual checkout (§2.1: "validate against the local checkout wherever the harness runs"), the endpoint now requires and forwards a 40-hex-char `anchor_sha` in the request body, rejecting the call with 400 otherwise. ### Multi-repo Gitea ingestion (new) - `provider.Gitea` gained an optional `Project` field and `SourceName()` (`"gitea"` if unset, `"gitea:"` if set) — the namespaced source doubles as the `(source,external_id)` dedup key, so issue #7 in two different repos never collides, and as the reflection dispatch key. - New `provider.MultiGitea{Sources map[string]Gitea}` implements `TaskReflector` by looking up `task.Source` and forwarding to the matching Gitea instance — lets several Gitea repos (one per project) share one `ReflectingSink`. - New `provider.GiteaSourceConfig` + `LoadGiteaConfigs(path)` load a JSON array of `{project,base_url,owner,repo,token,webhook_secret}`. `main.go` reads this from `ORCHESTRA_GITEA_CONFIG` if set; each source gets its own poll supervisor (`gitea:`) and webhook path (`/v1/providers/gitea/webhook/`). - The legacy single-repo env vars (`ORCHESTRA_GITEA_URL/TOKEN/OWNER/REPO/ WEBHOOK_SECRET`) still work unchanged when `ORCHESTRA_GITEA_CONFIG` is unset — same unprefixed webhook path, same `project = ORCHESTRA_GITEA_REPO` tagging, same dedup source `"gitea"` — so existing deployments and already-configured Gitea webhooks need no changes. - Added `internal/provider/gitea_test.go` — previously **there were zero tests exercising the Gitea provider at all** despite progress.md's prior claim of Gitea webhook/poll test coverage; that claim was not accurate. New tests cover source-name namespacing, webhook signature verification/rejection, project tagging, `MultiGitea` dispatch-by-source (via two `httptest.Server`s, asserting only the right one is hit), and `LoadGiteaConfigs` validation/duplicate-project rejection. ### Per-project repos (new) - `registry.Project` gained optional `repo`/`worktree_root` fields. Each project can now resolve its own git checkout rather than every project sharing one global `ORCHESTRA_REPO`/`ORCHESTRA_WORKTREE_ROOT` — matches spec §2.2 ("projects are first-class and extensible... the binding is a field + a config entry, not a schema change"). `main.go` builds a `orchestrator.PerProjectGitWorktrees` from the registry, falling back to the global default for any project that omits these fields, so single-repo deployments are unaffected. Covered by `internal/orchestrator/worktrees_test.go`. ### Closed this pass (were open gaps as of the last snapshot) - **Bus-level authorization.** `authz.AuthorizeEvent` is now enforced inside `store.Append` itself — the single choke point every event passes through (HTTP handlers, router, coordinator/rotation, providers, federation relay) — not just at HTTP handlers. Event schema bumped to v2, which requires every event to declare a `Surface`; a new `authz.System` surface (full control) covers internal emitters (router leases/failures, coordinator releases/blocks, standup advisory/apply). Schema v1 events on disk still replay (tolerant reader). Covered by `internal/store/store_test.go` and `internal/router/router_test.go` additions asserting a non-HTTP append with no/wrong surface is rejected. - **Dual quota windows.** `router.QuotaAvailability` now tracks a 5-hour rolling window and a 7-day weekly window independently per harness (`QuotaWindowLimits{FiveHour, Weekly}`), applying the conservative 80% rule to each separately — a harness over threshold on either window is unavailable. Replaces the old single-`Window` field. Covered by new `router_test.go` cases for weekly-only and 5h-only exhaustion. - **Turn-boundary detection made observable, not silently optional.** Rotation still can't force a harness adapter to implement `TurnBoundary` Face B, but an adapter that fails to answer it now blocks that tick's release (never treats a failed check as "safe to proceed"), and any adapter without the capability — or one whose check errors — increments `MonitorHealth.TurnBoundaryDegraded`, exposed via the coordinator's health endpoint so degraded-safety operation is visible, not silent. - **Cross-machine lease correctness has a real test.** `internal/integration/federation_lease_test.go` (`TestCrossMachineLeaseAnchorAndQuotaArePerHost`) exercises a lease claimed through the federation worker HTTP API, validates the anchor against that worker's own local checkout (not the router's), and asserts quota is accounted per-host. Spec §9 item 8 said "prove on the first federated run" — this is that proof for the primitives that exist today (registration, heartbeat, lease-claim); it does not yet run against two real physical machines. - **Fuzz coverage for lifecycle payload validation.** `internal/domain/fuzz_test.go` adds `FuzzValidatePayload` and `FuzzValidateEvent` covering all event types (including malformed nested `receipt`/`knowledge` shapes) — asserts no panic and always a typed error on adversarial input. ### Believed accurate from prior sessions (spot-checked, not exhaustively re-verified) - Event log: append-only JSONL, versioned envelope (schema v1), snapshot load/replay, CAS with content-hash verification at append. - `domain.ValidatePayload` enforces required fields per event type, including `expected_version`/`ttl` on `TaskLeased`, `anchor_sha` on `TaskReleased` (now correctly emitted, see above), `report_ref`+`receipt` on `TaskCompleted`, and `blocker` on `TaskBlocked`. - Router: project→affinity→machine resolution, capability match, quota availability at conservative 80% threshold, derived-importance ordering, retry-then-`TaskFailed`. - herdr adapters (Claude/Codex/opencode) with native occupancy readers, optional `TurnBoundary`/`RotationSignal`/`PaneExit` capability interfaces, bootstrap/lease/release/kill. - Continuity: strict handoff schema/validation, CAS save/load, pickup validation (HEAD match, dirty-file hashes, immutable `TASK.md` hash), scratch-branch commit/push/pull helpers. - Provider layer: JSONL watcher, Gitea webhook+poll with HMAC auth, idempotent `(source,external_id)` dedup, terminal-state reflection, supervised restart with backoff. - Federation: worker registration, heartbeat/TTL offline detection, event cursor polling/ack, lease claim endpoint. - Authorization: bus-level capability table (notify-only / full / gated) is applied to lifecycle and approval writes via `AuthorizeEvent`. - Delivery: Telegram/ntfy fan-out for completion/failure/block/approval events. - `/readyz`, `/v1/brief`, `/v1/providers/health`, `/v1/standup` exist and return real state (not stubs). ## Known open gaps (named, not silently assumed done) - **Cross-machine lease correctness is proven at the primitive level, not on real hardware.** `TestCrossMachineLeaseAnchorAndQuotaArePerHost` exercises the federation worker HTTP API (registration, lease-claim, anchor validation against the worker's own checkout, per-host quota) inside one test process. Spec §9 item 8 says "prove on the first federated run" — that means an actual homesrv/workpc pair over the real mesh, which this repo cannot exercise by itself. Named here as the one item that needs a live two-machine run to fully close, not more code. - **Turn-boundary Face B still degrades to occupancy-only for adapters that don't implement it**, by design — the spec's Face B is per-harness native session state (Stop hook / rollout tail / SSE), which this repo can only wire against a real running herdr+harness pair. The degradation is now observable (`MonitorHealth.TurnBoundaryDegraded`) and blocks-on-failure rather than silently proceeding, but whether Claude/Codex/opencode's native hooks are wired in a live deployment is a deployment-config fact, not something provable from source alone. Everything else named as open in the previous snapshot (bus-level authorization, dual 5h/weekly quota windows, fuzz coverage of lifecycle payload validation) is now closed — see "Closed this pass" above. Broader areas (provider layer, continuity, router matching, delivery, federation registration) were spot-checked against the code and their tests and matched their described behavior.