orchestra/progress.md

Orchestra progress

Updated: 2026-07-27

AUDIT.md remediation — in progress

Working through AUDIT.md's blocking/secondary defects in order of the "suggested order of attack." Each item below is landed, tested, and committed individually; see the git log for the exact commits.

Fixed so far:

• B2 — adapters were looked up by session.Harness (the harness kind, e.g. "claude") in Reconcile/expire/rotate, but AdapterFactory.Herdrs is keyed by herdr instance id (e.g. "homesrv-claude"). Every one of those call sites silently no-opped. Added Coordinator.adapterFor, routed all four call sites through it. Regression test registers an adapter under a herdr-id key distinct from the harness kind and asserts rotation fires.

• B1 — CLIAdapter.Occupancy called a.Usage(s.PaneID), but the usage readers want a filesystem path to session state, not a herdr pane id. Added herdr.Session.SessionFile and per-harness resolution (ClaudeSessionFile by newest-mtime under Claude Code's own project directory; codex via the existing CodexActiveUsage sqlite discovery; opencode refuses loudly — needs a live session id, not resolvable from the worktree alone). A missing/unreadable session file is now a hard error, surfaced via new SessionHealth.Occupancy/OccupancyError fields on GET /v1/tasks/{id}/health, not a silent zero. Still needs live verification against a real Claude Code session (the spec's own acceptance bar for this phase) — not possible from this sandbox.

• B4 — the router counted every TaskReleased (including rotation, which is a TaskReleased carrying a valid handoff_ref) against MaxAttempts, and double-counted by also incrementing on every subsequent lease. A task that rotated twice hit the default MaxAttempts=3 and was killed. Now only a release without a handoff_ref (expiry/crash) advances the counter.

• B8 — X-Orchestra-Surface: system was reachable from an HTTP request header in both authz.HTTP and main.go's surface closure (the one every handler actually calls). Since no deployment sets ORCHESTRA_SYSTEM_TOKEN, this was an unauthenticated full-control bypass reachable from any LAN caller. Both call sites now downgrade system to web before doing anything else with it.

• S1 — Brief.From/To and GitSync.Branch/Head/Status all shared one JSON tag each (Go only honors the first json:"..." tag on a combined field declaration). go vet ./... now passes clean.

• S5 — Store.Lease/ExpireLeases set Event.ID to the task id, so every lease of a task produced colliding event IDs. Now domain.NewID().

• S6 — the ingest dedup path returned nil (success) without appending; main.go then returned an unrelated event with 201. Added domain.ErrDuplicate and Store.TaskBySource; POST /v1/tasks now returns the existing task with 200 on a duplicate. Updated every other Append caller (Gitea poll/webhook, JSONL ingest) to treat ErrDuplicate as expected rather than a failure — without that, Gitea polling would error out of its scan loop on the first already-ingested issue in every batch.

• B3 (partial) — added POST /v1/harness/complete, the first automatic TaskCompleted producer (previously only a human calling /v1/tasks/{id}/complete could ever complete a task). A Claude Code Stop hook (deploy/hooks/orchestra-stop.sh) fires on every turn boundary but only reports completion if the agent has written a .orchestra-report.md marker at the worktree root first — an ordinary turn boundary is a no-op, so this doesn't fire completion prematurely. The server reads the transcript locally via herdr.ClaudeUsage to build the receipt itself (input/cache/output token counts) rather than trusting a self-reported number, and uploads the report body to CAS for report_ref. Guarded by an optional ORCHESTRA_HARNESS_TOKEN bearer check; the event is appended with Surface: system set directly in Go (not derived from a request header — consistent with the B8 fix that system must never be header-controlled). Not done: Codex/opencode equivalents (Claude-only for now — Codex would need CodexActiveUsage, opencode OpenCodeUsage/OpenCodeStatus, wired the same way), and the turn-boundary decision endpoint (continue/prepare_handoff/rotate_now/refuse) from Phase 2 items 1–2 is still unbuilt — only the completion half of Phase 2 landed. No test added for the new HTTP handler; cmd/orchestra/main.go has zero test coverage for any handler (pre-existing gap, everything lives inline in main()) so this follows the existing (untested) pattern rather than introducing a one-off test harness.

• B5 (loose end) — CLIAdapter.Lease's initial prompt used wait=0, skipping the inline wait Bootstrap already used; the spec (§5.1) requires inline wait on agent.prompt for bootstrap injection to avoid sending into a half-rendered prompt. Changed to time.Minute, matching Bootstrap. Small, contained fix — Release's real implementation (needs Phase 4 handoff production) is still outstanding from B5.

• B6 (partial — Phase 4 items 1 and 4) — nothing wrote a TASK.md into a worktree, so pickup validation had nothing to check and never ran anyway. Fixed both halves: GitWorktrees.Create now writes and commits an immutable TASK.md (continuity.RenderTaskFile) into every freshly created worktree, and Coordinator.Start now runs continuity.ValidatePickup (loading the handoff from CAS, checking anchor SHA + dirty-file hashes + TASK.md hash) before bootstrapping a successor onto a handoff_ref — a failure kills the session and emits TaskBlocked instead of trusting an unvalidated ref. Covered by TestGitWorktreesCommitsTaskFile and TestStartBlocksOnInvalidPickup in internal/orchestrator. Not done: handoff production (nothing yet writes a real §6.1 handoff — Release still refuses per B5), wiring ScratchCommit before release, and the §6.2 bootstrap-prompt rewrite. See AUDIT.md's "B6 — partial fix" section for the full breakdown, including a named caveat: TASK.md hashing is best-effort and untested for the herdr-hosted (WorktreeCreator) worktree path.

• B5 (closed) — CLIAdapter.Release previously just refused (no real herdr method existed to call and there was nothing to validate against). Now: reads the agent-authored .orchestra-handoff.json from the worktree root, validates it with continuity.Decode, cross-checks its anchor SHA against the worktree's real HeadSHA (never trusts the agent's self-report outright), uploads it to CAS via continuity.Save to mint the handoff_ref, and only then calls the real pane.release_agent({pane_id, source, agent}) to drop herdr's claim — sequenced last so a herdr-side error can't strand an uploaded handoff. Any failure (missing file, invalid schema, anchor mismatch, herdr error) is a refusal, which rotate already treats as "retry next tick" rather than stranding the task. herdr.Claude/ Codex/OpenCode now take a continuity.CAS (main.go passes the existing *store.Store). New tests in internal/herdr/adapter_test.go cover all four paths against a real git worktree and a fake in-process herdr listener. Not done: nothing yet makes the agent actually write .orchestra-handoff.json (needs a stop-hook convention analogous to .orchestra-report.md) — that and the rest of Phase 4 (ScratchCommit before release, §6.2 bootstrap-prompt rewrite, MarkdownChanges) remain open.

• Phase 4 items 3, 5, 6 — CLIAdapter.Release now re-verifies every Anchor.Dirty file hash (previously only the top-level Anchor.GitSHA was checked; a file edited after the handoff was written but before release would have gone through unnoticed), then, if there were dirty entries, snapshots them atomically onto a per-task scratch branch (continuity.ScratchCommit, made idempotent so a task can rotate more than once) and rewrites the handoff's anchor to that new commit with Dirty cleared before uploading — so the successor's pickup check is a single HEAD compare, not N file rehashes. CLIAdapter.Bootstrap's prompt was rewritten to point the agent at git log/the scratch branch instead of a vague "read the handoff" instruction, and deliberately avoids claiming a GET /v1/artifacts/<ref> endpoint, since no such route exists (/v1/artifacts is POST-only). continuity.MarkdownChanges (§6.3 adjacent-task notice) had zero callers and zero tests despite being listed as implemented in an earlier snapshot — deleted rather than half-wired, per AUDIT.md's explicit "delete and record the deviation" option. New tests: TestReleaseScratchCommitsDirtyFilesBeforeUpload, TestReleaseRefusesOnStaleDirtyFile (internal/herdr/adapter_test.go). §6.3 rewired for real, 2026-07-27 (later same day): the deleted MarkdownChanges above was zero-caller dead code, but the underlying spec requirement ("on update, the orchestra injects a notice to agents whose current task is adjacent") wasn't abandoned — rebuilt independently. continuity.ConventionsHash(root) hashes whichever of AGENTS.md/CLAUDE.md/VOCAB.md exist at a path; herdr.Session gained ConventionsHash, snapshotted from the fresh worktree at Coordinator.Start; a new Coordinator.checkConventions, run every Monitor tick, recomputes the hash of the project's base repo (via WorktreeSpec.Spec — "adjacent" = same project) for every leased session and compares it against that session's stored snapshot. A mismatch calls a new optional herdr.ConventionsNotifier capability (CLIAdapter.NotifyConventionsChanged, an in-pane agent.prompt telling the agent to re-read the docs) and updates the stored hash so the notice fires once per drift, not every tick. Covered by TestConventionsDriftNotifiesActiveSession (internal/orchestrator/rotation_test.go): asserts no notification while the base repo is unchanged, then one once it diverges. Was still open: Phase 4 item 2 — nothing drove any harness to write .orchestra-handoff.json, since Release only validated a file whose existence was never solicited. Closed 2026-07-27: rotate() now checks for the adapter's optional herdr.HandoffRequester capability; when HandoffFile is missing at the worktree root, it prompts the agent once (CLIAdapter.RequestHandoff, mirroring the .orchestra-report.md/B3 convention — the plane asks for a handoff, it never invents one) and skips Release that tick, retrying every subsequent tick until the file appears. herdr.Session.HandoffRequested avoids re-prompting every tick. Covered by TestRotationRequestsHandoffBeforeReleasing (internal/orchestrator/rotation_test.go), which asserts Release is never called before the file exists and fires once it does. Codex/opencode still share this same path (no harness-specific gap remains); the only leftover question is whether each harness's own Stop-equivalent hook honors the in-pane prompt to write the file before exiting, which is a live-deployment fact, not something provable from source.

• B7 (post-hoc producer) + Phase 2 turn-decision endpoint — landed together, since both are new QuotaReported/turn-boundary paths off the same completion/turn events. POST /v1/harness/complete now appends a QuotaReported event (harness_id from the closing lease, consumed from the same usage.Numerator() used for the receipt), so the router's 5h/weekly availability filter and the brief's quota_consumed stop evaluating against a permanent zero. New Coordinator.TurnDecision (internal/orchestrator/orchestrator.go) mirrors rotate()'s per-task logic (occupancy → turn-boundary → handoff-file → release) but runs synchronously once per turn instead of waiting for Monitor's ticker, returning one of continue/prepare_handoff/rotate_now/refuse via the new POST /v1/harness/turn. The Claude Stop hook (deploy/hooks/orchestra-stop.sh) now calls this endpoint on every ordinary turn boundary (report marker absent) instead of no-op'ing, and exits 2 on refuse to stop the harness from finishing an unsafe turn. Covered by TestTurnDecision (internal/orchestrator/rotation_test.go): continue-below-threshold, refuse-when-not-at-boundary, and rotate_now-releases-and-emits-a-valid-TaskReleased cases. Not done: live per-harness push producers (Claude statusline, Codex rollout tail) that would give B7 a second, continuous producer independent of task completion — recorded as a design investigation in AUDIT.md ("Real harness quota sources") but not implemented; Codex/ opencode's own equivalents of the Claude Stop hook (whether their turn-boundary mechanism actually calls /v1/harness/turn) also remain unbuilt, same caveat as Phase 2 item 4 already named for /complete.

• S4 — delivery.Fanout.Run used to return on the first sender error, permanently killing the notification goroutine (a single ntfy hiccup meant no notifications for the rest of the process's lifetime, since nothing restarts it). Failed sends now go through an OnError hook instead of aborting the loop. Cursor is also persisted now (SaveCursor → a delivery-cursor file next to ORCHESTRA_DATA, loaded on startup), so a restart resumes from the last delivered event instead of re-notifying the entire log from seq 0. internal/delivery previously had zero tests; added TestFanoutContinuesAfterSendError.

• S2 + S3 — /v1/brief's git state used to come from ORCHESTRA_DATA (the event-log directory, not a git checkout — always reported "git unavailable"), and completions were only counted, never surfaced with proof. operations.Brief.Git is now map[string]GitSync keyed by project ID, built in main.go from each registry.Project.Repo (falling back to a single "default" entry off ORCHESTRA_REPO for deployments without per-project repos); GitSync gained Ahead/Behind vs upstream. New Brief.Receipts []operations.CompletionReceipt pulls report_ref/ receipt straight out of each TaskCompleted event's existing payload. Covered by an updated TestBuildBrief.

Not yet started: Codex/opencode completion producers, S7–S11. See AUDIT.md for the full plan.

Phase 0 done (2026-07-27): this box has live TCP reachability to the real herdr instance at 192.168.1.105:9245 — verified by hand (raw JSON-RPC probes, no herdr CLI available locally). Real method list captured in deploy/herdr-schema.json. Confirmed pane.release/pane.kill/ pane.rotation_signal are invented, as AUDIT.md's B5 suspected. pane.kill→pane.close fixed as a drop-in. pane.rotation_signal/ RotationSignal deleted (no replacement exists). Release now refuses loudly instead of calling a nonexistent method — its real implementation needs Phase 4 (handoff production) first, since even the real pane.release_agent can't return a handoff_ref (herdr doesn't write handoffs, the agent does). See AUDIT.md's new "Phase 0 — done" section for full detail. Also found: a real task is currently stuck live — workspace wA, task 06FT6CKD9Y98AZRX6X8K3QXFZG, opencode, pane wA:p1, blocked — deliberately not touched from this session.

Current state

This is a working Go implementation of orchestra-spec (1).md's Layer 1–3 (substrate, harness/rotation, continuity) plus a first cut of Layer 4 (surfaces). go build ./... and go test ./... both pass. The codebase is small (~4.6k lines across internal/{domain,store,provider,registry,router, herdr,orchestrator,continuity,federation,delivery,authz,operations,admin} and cmd/orchestra/main.go).

Earlier revisions of this file accumulated a long, self-contradictory chronological log — gaps were listed as open in one section and then claimed closed in a later section, sometimes inaccurately. This revision replaces that log with one audited snapshot. Treat prior git history of this file as session notes, not as ground truth.

Verified fixed this pass

• Rotation emitted an invalid TaskReleased (the previously reported highest-priority defect) — now fixed. internal/orchestrator.Coordinator.rotate built the release payload as {"handoff_ref","reason"}, omitting the anchor_sha the spec (§4, §6.2) and domain.ValidatePayload require whenever handoff_ref is present. store.Append would reject it, the error was discarded (if c.Store.Append(e) == nil), and the lease/session silently never rotated — the coordinator would just retry next tick with no visible failure. Fixed by adding herdr.HeadSHA(worktree) and having rotate populate anchor_sha from the real worktree HEAD before appending; if the anchor can't be read, rotation now correctly skips that tick (leaving the lease intact for TTL/next-tick reclaim) instead of emitting a payload guaranteed to fail validation. Covered by internal/orchestrator/rotation_test.go (TestRotationEmitsValidReleaseWithAnchorSHA), which drives the real Coordinator.Monitor loop against an actual git worktree and asserts the emitted event passes domain.ValidatePayload with the correct SHA — the previous end-to-end test masked this bug by manually crafting a replacement TaskReleased event after observing the (silently failed) adapter-side release.
• The federation worker release endpoint had the same gap. The /v1/federation/workers/{id}/release handler (cmd/orchestra/main.go) built TaskReleased from a request body with only handoff_ref, no anchor_sha. Since a remote worker is the only party with the actual checkout (§2.1: "validate against the local checkout wherever the harness runs"), the endpoint now requires and forwards a 40-hex-char anchor_sha in the request body, rejecting the call with 400 otherwise.

Multi-repo Gitea ingestion (new)

• provider.Gitea gained an optional Project field and SourceName() ("gitea" if unset, "gitea:<project>" if set) — the namespaced source doubles as the (source,external_id) dedup key, so issue #7 in two different repos never collides, and as the reflection dispatch key.
• New provider.MultiGitea{Sources map[string]Gitea} implements TaskReflector by looking up task.Source and forwarding to the matching Gitea instance — lets several Gitea repos (one per project) share one ReflectingSink.
• New provider.GiteaSourceConfig + LoadGiteaConfigs(path) load a JSON array of {project,base_url,owner,repo,token,webhook_secret}. main.go reads this from ORCHESTRA_GITEA_CONFIG if set; each source gets its own poll supervisor (gitea:<project>) and webhook path (/v1/providers/gitea/webhook/<project>).
• The legacy single-repo env vars (ORCHESTRA_GITEA_URL/TOKEN/OWNER/REPO/ WEBHOOK_SECRET) still work unchanged when ORCHESTRA_GITEA_CONFIG is unset — same unprefixed webhook path, same project = ORCHESTRA_GITEA_REPO tagging, same dedup source "gitea" — so existing deployments and already-configured Gitea webhooks need no changes.
• Added internal/provider/gitea_test.go — previously there were zero tests exercising the Gitea provider at all despite progress.md's prior claim of Gitea webhook/poll test coverage; that claim was not accurate. New tests cover source-name namespacing, webhook signature verification/rejection, project tagging, MultiGitea dispatch-by-source (via two httptest.Servers, asserting only the right one is hit), and LoadGiteaConfigs validation/duplicate-project rejection.

Per-project repos (new)

• registry.Project gained optional repo/worktree_root fields. Each project can now resolve its own git checkout rather than every project sharing one global ORCHESTRA_REPO/ORCHESTRA_WORKTREE_ROOT — matches spec §2.2 ("projects are first-class and extensible... the binding is a field + a config entry, not a schema change"). main.go builds a orchestrator.PerProjectGitWorktrees from the registry, falling back to the global default for any project that omits these fields, so single-repo deployments are unaffected. Covered by internal/orchestrator/worktrees_test.go.

Closed this pass (were open gaps as of the last snapshot)

• Bus-level authorization. authz.AuthorizeEvent is now enforced inside store.Append itself — the single choke point every event passes through (HTTP handlers, router, coordinator/rotation, providers, federation relay) — not just at HTTP handlers. Event schema bumped to v2, which requires every event to declare a Surface; a new authz.System surface (full control) covers internal emitters (router leases/failures, coordinator releases/blocks, standup advisory/apply). Schema v1 events on disk still replay (tolerant reader). Covered by internal/store/store_test.go and internal/router/router_test.go additions asserting a non-HTTP append with no/wrong surface is rejected.
• Dual quota windows. router.QuotaAvailability now tracks a 5-hour rolling window and a 7-day weekly window independently per harness (QuotaWindowLimits{FiveHour, Weekly}), applying the conservative 80% rule to each separately — a harness over threshold on either window is unavailable. Replaces the old single-Window field. Covered by new router_test.go cases for weekly-only and 5h-only exhaustion.
• Turn-boundary detection made observable, not silently optional. Rotation still can't force a harness adapter to implement TurnBoundary Face B, but an adapter that fails to answer it now blocks that tick's release (never treats a failed check as "safe to proceed"), and any adapter without the capability — or one whose check errors — increments MonitorHealth.TurnBoundaryDegraded, exposed via the coordinator's health endpoint so degraded-safety operation is visible, not silent.
• Cross-machine lease correctness has a real test. internal/integration/federation_lease_test.go (TestCrossMachineLeaseAnchorAndQuotaArePerHost) exercises a lease claimed through the federation worker HTTP API, validates the anchor against that worker's own local checkout (not the router's), and asserts quota is accounted per-host. Spec §9 item 8 said "prove on the first federated run" — this is that proof for the primitives that exist today (registration, heartbeat, lease-claim); it does not yet run against two real physical machines.
• Fuzz coverage for lifecycle payload validation. internal/domain/fuzz_test.go adds FuzzValidatePayload and FuzzValidateEvent covering all event types (including malformed nested receipt/knowledge shapes) — asserts no panic and always a typed error on adversarial input.

Believed accurate from prior sessions (spot-checked, not exhaustively re-verified)

• Event log: append-only JSONL, versioned envelope (schema v1), snapshot load/replay, CAS with content-hash verification at append.
• domain.ValidatePayload enforces required fields per event type, including expected_version/ttl on TaskLeased, anchor_sha on TaskReleased (now correctly emitted, see above), report_ref+receipt on TaskCompleted, and blocker on TaskBlocked.
• Router: project→affinity→machine resolution, capability match, quota availability at conservative 80% threshold, derived-importance ordering, retry-then-TaskFailed.
• herdr adapters (Claude/Codex/opencode) with native occupancy readers, optional TurnBoundary/RotationSignal/PaneExit capability interfaces, bootstrap/lease/release/kill.
• Continuity: strict handoff schema/validation, CAS save/load, pickup validation (HEAD match, dirty-file hashes, immutable TASK.md hash), scratch-branch commit/push/pull helpers.
• Provider layer: JSONL watcher, Gitea webhook+poll with HMAC auth, idempotent (source,external_id) dedup, terminal-state reflection, supervised restart with backoff.
• Federation: worker registration, heartbeat/TTL offline detection, event cursor polling/ack, lease claim endpoint.
• Authorization: bus-level capability table (notify-only / full / gated) is applied to lifecycle and approval writes via AuthorizeEvent.
• Delivery: Telegram/ntfy fan-out for completion/failure/block/approval events.
• /readyz, /v1/brief, /v1/providers/health, /v1/standup exist and return real state (not stubs).

Known open gaps (named, not silently assumed done)

• Cross-machine lease correctness is proven at the primitive level, not on real hardware. TestCrossMachineLeaseAnchorAndQuotaArePerHost exercises the federation worker HTTP API (registration, lease-claim, anchor validation against the worker's own checkout, per-host quota) inside one test process. Spec §9 item 8 says "prove on the first federated run" — that means an actual homesrv/workpc pair over the real mesh, which this repo cannot exercise by itself. Named here as the one item that needs a live two-machine run to fully close, not more code.
• Turn-boundary Face B still degrades to occupancy-only for adapters that don't implement it, by design — the spec's Face B is per-harness native session state (Stop hook / rollout tail / SSE), which this repo can only wire against a real running herdr+harness pair. The degradation is now observable (MonitorHealth.TurnBoundaryDegraded) and blocks-on-failure rather than silently proceeding, but whether Claude/Codex/opencode's native hooks are wired in a live deployment is a deployment-config fact, not something provable from source alone.

Everything else named as open in the previous snapshot (bus-level authorization, dual 5h/weekly quota windows, fuzz coverage of lifecycle payload validation) is now closed — see "Closed this pass" above. Broader areas (provider layer, continuity, router matching, delivery, federation registration) were spot-checked against the code and their tests and matched their described behavior.