orchestra/AUDIT.md

Orchestra — spec conformance audit & remediation status

Audited 2026-07-27 against orchestra-spec (1).md at commit 325c684. Method: read every non-test file in internal/ and cmd/, traced each spec section to its call site, and checked whether the live path (main.go → router → coordinator → adapter) actually reaches it. This file is the single merged record — a prior split between AUDIT.md (audit + plan) and progress.md (chronological session log) has been flattened here; treat any older chronological git history of either file as session notes, not ground truth.

Ground truth rule for future sessions: this repo has a documented history of code that looks wired but isn't (packages with tests that pass in isolation while the live call path silently no-ops). Before trusting any "done"/"fixed" claim below, check the actual call site, not just this document.

---

Current verdict (as of 2026-07-28)

The substrate (Layer 1) is solid. Layers 2–4 were originally shaped-but-not- wired; most of the blocking defects below are now closed, verified by reading the current code (not just by trusting this file) and by targeted tests. Two things are not yet proven:

1. A real end-to-end unattended run. The first live attempt (2026-07-28) surfaced B12 (fixed and confirmed live, see below), and re-verifying that fix surfaced B13 (open) — agent.start can return success while silently never starting an agent, under back-to-back leases. One task (wD:p1) did make it all the way to a real attached claude session, so the lease path is provably reachable, but B13 means it's not yet reliable enough to call proven — occupancy/rotation/handoff/completion still haven't been exercised end-to-end against a session that's guaranteed to actually exist.
2. Cross-machine (federation) correctness. Deliberately deferred — see "The federation fork" below.

go build ./..., go vet ./..., and go test ./... all pass.

Spec layer	State
L1 substrate (§3, §4)	Built and correct in the main path.
L2 harness (§5)	Occupancy, rotation, and completion all wired and reachable; B12's readiness race is fixed and confirmed live, but B13 (agent.start silently no-op'ing under back-to-back leases) still blocks reliable live verification of what's downstream.
L3 continuity (§6)	Handoff schema, pickup validation, scratch branches, TASK.md are all wired into the live path (Phase 4 complete).
L4 surfaces (§7)	Brief/standup/delivery real; quota has a post-hoc producer only (no live push feed yet); authz bypass closed.

---

Blocking defect currently open

B13 — agent.start silently no-ops under back-to-back leases (found live 2026-07-28)

Discovered while live-verifying the B12 fix below. B12 itself is confirmed fixed (see that entry), but re-testing it exposed a second, deeper defect that B12's retry logic cannot address.

Three fresh tasks were leased in quick succession against workpc-claude (06FTAKSJZTB73FZQE3QT7XQ1J0/pane wD:p1, 06FTAMFKVEPGPKR0H86C08ZZNG/pane wE:p1, 06FTANAYZPA55BABR2J4MWQ050/pane wF:p1). All three got a worktree.create success and an agent.start call that returned no error. Checked live via pane.get/pane.list (and independently confirmed by the user cd'ing into each worktree by hand):

• wD:p1 (the first, given time alone to settle): a real claude process did eventually attach — agent: claude, agent_status: idle, revision: 2. This is consistent with B12's model: agent.start acknowledges the request but the actual attach is asynchronous, sometimes taking well over a minute.
• wE:p1 and wF:p1 (started moments after wD:p1, while it was presumably still initializing): still no agent attached after 2+ minutes of polling — agent_status: unknown, no agent field, revision never incremented past its creation value. Empty shells, confirmed both via pane.get and by the user directly cding into the worktree and finding no session running.

So agent.start's "success" is not just delayed for these two — it never happened at all, and agent.start never returned an error to tell Orchestra that. One plausible cause: something about launching a second/third claude CLI process while a prior one on the same host/account is still mid-startup (auth handshake, config lock, etc.) silently drops the later request(s) rather than queuing or erroring them. Not confirmed — would need a controlled single-at-a-time repro to isolate from herdr's own internals, which requires spinning up more real panes and wasn't done this pass (see "three orphaned panes" below).

Consequence: unlike B12 (which at least produced an observable TaskBlocked), this failure mode leaves the task leased indefinitely against a pane that will never produce a session, with no error surfaced anywhere — worse than B12 was, because nothing currently distinguishes "still booting, give it more time" from "silently dead, will never start." No fix attempted yet; a client-side retry (B12's approach) cannot fix this because the agent.start call that should have started the process already returned success.

Side effect of this investigation — three live orphaned panes on workpc, left untouched on purpose: wD:p1 (has a real but abandoned claude session, task now stuck in blocked from before B12's fix), wE:p1 and wF:p1 (empty shells, no agent ever attached). Not cleaned up — same standing rule as the pre-existing wA:p1 stuck task: don't call pane.close/pane.release_agent against live panes without asking first.

---

B12 — StartAgent races the pane's shell readiness (found live 2026-07-28) — closed (this specific race; see B13 for a second issue it exposed)

Discovered during the first real end-to-end run against a freshly rebuilt/ redeployed binary (the running service had been on stale pre-audit commit 325c684 the whole time).

Created a fresh task (06FTAH4MCRCAJYY7V75Z4V2YGG, project test-e2e, capability opencode). It reached TaskLeased onto workpc-opencode (worktree created for real at /tmp/test-e2e-worktrees/<task-id> on workpc, confirmed via a live pane.list — pane wB:p1, correct cwd), then 445ms later hit TaskBlocked:

lease: herdr protocol error: agent target pane wB:p1 is not an available shell

internal/herdr/herdr.go:199 (StartAgent) calls agent.start on the pane Worktree just recorded, with no wait or retry after worktree.create returns. herdr hands the pane back before its shell has actually finished initializing, and agent.start refuses it as not-yet-a-shell. None of B1–B11 exercised this — every prior fix assumed a session already existed (occupancy, rotation, handoff); nothing in the remediation plan tested the very first Lease call against a real herdr pane end-to-end. Phase 0's probing used params:{}/missing-field tricks, never a real worktree.create → agent.start sequence timed against actual pane boot.

Reproduced a second time, 2026-07-28, same session: a second fresh task (06FTAHW8GZP0S6DCHEQ6PJXWB0) leased onto a brand-new pane (wC:p1) and hit the identical error on the same sub-second timeline. Two for two — a systematic race, not a flake.

Consequence: every fresh lease that hits this race goes straight to TaskBlocked, and the router never retries a blocked task (internal/router/router.go has no reference to StateBlocked at all, confirmed by grep) — so it sits there permanently until a human intervenes via the approval surface. This is now the single blocking defect for proving the rest of the remediation plan against a real live task.

Fix, iterated live across three rounds, 2026-07-28:

1. First pass: bounded-retry agent.start (10×, 500ms apart) specifically on the "not an available shell" string. Redeployed and re-tested against a real fresh lease — this cleared that exact error, but the very next call in the sequence (Lease's post-agent.start Prompt) then hit a different transient error, "agent ... is not an active named agent" — same underlying readiness race, one call later, worded differently.
2. Second pass: added the identical string-matched retry to Prompt for that specific message. Redeployed, re-tested — this time hit a third distinct wording, "agent target ... not found", on the same call.
3. Given three different error strings for the same race across three live attempts, string-matching was abandoned as unwinnable (herdr's wording for "not ready yet" isn't fixed enough to enumerate). Final shape: both StartAgent's agent.start call and Prompt's agent.prompt call now retry on any error, bounded by wall-clock time (15s, bootRetryDelay of 500ms) rather than attempt count or specific wording — there is no other legitimate reason a call against a pane/worktree orchestra itself just created would fail immediately. StartAgent still special-cases "already" as a non-error (idempotent re-lease).

Confirmed this closes the original race: task 06FTAKSJZTB73FZQE3QT7XQ1J0 (pane wD:p1) leased successfully, and polling pane.get live shows a real agent: claude / agent_status: idle attach — independently confirmed by the user cd-ing into that worktree by hand. go build/go vet/go test ./... all pass throughout.

Not fully clean, however: re-testing this fix by firing two more tasks shortly after the first one surfaced a second, separate defect — agent.start returning success while never actually starting an agent, with no error for the retry logic to even see. That's B13, above — this entry only covers the specific shell/agent-readiness race B12 was originally filed for, which is fixed; B13 is a distinct root cause the same investigation exposed.

---

Closed defects

Each entry is the flattened final state — design + what's verified — not the session-by-session narration. Grouped by original defect ID.

B1 — Occupancy measured from the wrong value (§5.2.1) — closed

CLIAdapter.Occupancy was calling a.Usage(s.PaneID) (a herdr pane id) against readers (ClaudeUsage/CodexUsage/OpenCodeUsage) that want a filesystem path to session state — every call failed and was swallowed by a bare continue. Fixed: herdr.Session.SessionFile is resolved at lease time per harness (Claude: newest-mtime transcript under Claude Code's own project dir; Codex: existing CodexActiveUsage sqlite discovery; opencode: refuses loudly rather than guess, since it needs a live session id not resolvable from the worktree alone). A missing/unreadable session file is now a hard error surfaced via SessionHealth.Occupancy/OccupancyError on GET /v1/tasks/{id}/health, never a silent zero.

Still open: live verification against a real Claude Code session at a known context fill — not possible from this sandbox, blocked further by B13 now that fresh leases don't reliably survive.

B2 — Adapter lookup keyed wrong in 3 of 4 call sites (§5.3, §5.4) — closed

AdapterFactory.Herdrs is keyed by herdr instance id (homesrv-claude); Reconcile/expire/rotate were looking it up by session.Harness (the harness kind, claude) and silently no-op'ing (continue) on every miss — orphaned panes never killed, expired leases never killed their pane, rotation never began. Fixed with a single Coordinator.adapterFor(session) routed through at all four call sites. Regression test registers an adapter under a herdr-id key distinct from the harness kind and asserts rotation still fires.

B3 — Nothing emitted TaskCompleted (§4, §5.2) — closed

Only producer used to be a human calling POST /v1/tasks/{id}/complete. Added POST /v1/harness/complete: a Claude Code Stop hook (deploy/hooks/orchestra-stop.sh) fires every turn boundary but only reports completion once the agent has written a .orchestra-report.md marker at the worktree root (an ordinary turn boundary is a no-op). The server builds the receipt itself from herdr.ClaudeUsage against the transcript (never trusts a self-reported number) and uploads the report body to CAS for report_ref. Gated by optional ORCHESTRA_HARNESS_TOKEN; event appended with Surface: system set directly in Go, consistent with B8. Extended to Codex/opencode via an optional harness field in the request body ("codex" → CodexUsage, "opencode" → OpenCodeUsage).

Still open: no automated test for the HTTP handler (cmd/orchestra/main.go has zero handler test coverage of any kind, pre-existing gap — this follows the existing pattern rather than introducing a one-off harness).

B4 — Router counted rotation as a retry (§5.3, §5.4) — closed

Every TaskReleased (including rotation, which carries a valid handoff_ref) advanced MaxAttempts, and lease also double-counted — a task healthy enough to rotate twice was killed. Now only a release without a handoff_ref (expiry/crash) advances the counter.

B5 — Herdr protocol methods unverified/invented — closed

Confirmed live against 192.168.1.105:9245 (raw JSON-RPC probing, no local herdr CLI available — schema reconstructed from Rust serde's "unknown variant"/"missing field" errors). pane.release, pane.kill, pane.rotation_signal, pane.status do not exist. Real replacements: pane.close({pane_id}) for kill (drop-in); pane.release_agent({pane_id, source, agent}) for release — structurally different, does not return a handoff_ref (herdr never writes handoffs; the agent does, §6.1). RotationSignal interface/method/call site deleted outright (no replacement exists; herdr has no concept of Orchestra rotation).

CLIAdapter.Release now: reads the agent-authored .orchestra-handoff.json, validates with continuity.Decode, cross-checks its anchor SHA against the worktree's real HeadSHA, re-verifies every Anchor.Dirty file hash, scratch-commits any dirty files (continuity.ScratchCommit, idempotent across multiple rotations) and rewrites the handoff's anchor to the new scratch commit before uploading to CAS via continuity.Save (minting the real handoff_ref), and only then calls pane.release_agent — sequenced last so a herdr-side error can't strand an uploaded handoff. Any failure (missing file, invalid schema, anchor mismatch, stale dirty hash, herdr error) is a refusal, which rotate already treats as "retry next tick." CLIAdapter.Lease's bootstrap prompt also fixed to pass time.Minute (not wait=0) for inline-wait, matching Bootstrap and closing the send-into-a-half-rendered-prompt race §5.1 requires guarding against.

Covered by internal/herdr/adapter_test.go against a real git worktree and a fake in-process herdr TCP listener: upload-and-release on a valid handoff, refusal with no handoff file, refusal on anchor mismatch, refusal on stale dirty-file hash, refusal with no CAS configured.

Protocol version confirmed live as a bare JSON number (17), not the string config.jsonc declares — CheckProtocol's raw-bytes fallback happens to compare correctly today; don't "clean up" that code without re-checking this, or it may start doing a real numeric-vs-string comparison and break.

Still open: nothing makes the agent actually write .orchestra-handoff.json unprompted (closed separately, see B6/Phase 4 item 2 below) — that's the piece this entry originally deferred.

B6 — Layer 3 (continuity) was entirely dead code (§6) — closed

Nothing wrote TASK.md, so pickup validation had nothing to check and never ran. Fixed in full:

• GitWorktrees.Create now writes and commits an immutable TASK.md (continuity.RenderTaskFile) into every fresh worktree; continuity.TaskFileHash reads it back for w.TaskFileSHA.
• Coordinator.Start runs continuity.ValidatePickup (anchor SHA + dirty-file hashes + TASK.md hash) before bootstrapping a successor onto a handoff_ref — failure kills the session and emits TaskBlocked instead of trusting an unvalidated ref.
• ScratchCommit wired into Release (see B5) so pickup collapses to a single HEAD compare instead of per-file rehashing.
• CLIAdapter.Bootstrap's prompt rewritten to point the agent at git log/the scratch branch (trust already established by the plane's own ValidatePickup) instead of vague "read the handoff" prose, and deliberately does not claim a GET /v1/artifacts/<ref> endpoint since none exists (/v1/artifacts is POST-only — an earlier draft invented this and was corrected before landing).
• continuity.MarkdownChanges (zero callers, zero tests despite being listed as implemented in an earlier snapshot) was deleted rather than half-wired. §6.3's actual requirement ("notice to agents whose task is adjacent") was rebuilt independently: continuity.ConventionsHash(root) hashes AGENTS.md/CLAUDE.md/VOCAB.md; Coordinator.checkConventions runs every Monitor tick, recomputes the project base repo's hash for every leased session, and on drift calls the optional herdr.ConventionsNotifier capability (in-pane prompt) once per drift.
• Handoff production, the item that most of B6 hinged on: Coordinator.rotate checks the adapter's optional herdr.HandoffRequester capability; if .orchestra-handoff.json is missing, it prompts the agent once (CLIAdapter.RequestHandoff, naming the exact §6.1 JSON shape and explicitly telling the agent not to fabricate SHA/hashes) and skips Release that tick, retrying every subsequent tick — mirrors the .orchestra-report.md/B3 convention exactly. Session.HandoffRequested avoids re-prompting every tick.

Covered by TestGitWorktreesCommitsTaskFile, TestStartBlocksOnInvalidPickup, TestReleaseScratchCommitsDirtyFilesBeforeUpload, TestReleaseRefusesOnStaleDirtyFile, TestConventionsDriftNotifiesActiveSession, TestRotationRequestsHandoffBeforeReleasing (internal/herdr, internal/orchestrator).

Caveat still open: TASK.md hashing is best-effort/untested for the herdr-hosted (WorktreeCreator) worktree path specifically — no adapter or test exercises that path with a real handoff_ref, so pickup validation there runs with an empty taskFileSHA (anchor + dirty-file hashes still checked). Same cross-host caveat as the federation fork, below.

B7 — Quota projection had no producer (§7.2) — closed (post-hoc only)

POST /v1/harness/complete now appends a QuotaReported event (consumed from the same usage.Numerator() used for the receipt), so the router's 5h/weekly filter and the brief's quota_consumed stop evaluating against a permanent zero.

Design gap recorded, not yet built — a live push producer, since a harness that never completes a task cleanly (e.g. the stuck wA pane, see below) currently under-counts its consumption forever:

• Claude: the statusline stdin JSON already carries real .rate_limits.five_hour.used_percentage / .seven_day.used_percentage — confirmed by reading ~/.claude/statusline.sh on this machine. Prefer this over claude.ai/api/.../usage, which needs browser session cookies, not an API key.
• Codex: every token_count event in the rollout carries a rate_limits object with used_percent/window_minutes/resets_at — confirmed directly from a real rollout file. No codex usage subcommand; the rollout tail (or codex app-server, same events live) is the only transport.
• opencode: no first-class quota surface. opencode stats is historical only. Zen's free-tier daily quota arrives as x-ratelimit-* response headers, not a queryable endpoint — would need an opencode plugin to intercept.
• Structural consequence if this is ever built: sumSince's additive model is wrong for a used-percentage feed (need a second Availability reading the latest report per harness, not summing); static quota_limit_5h/weekly become unnecessary for Claude/Codex once real fractions are reported; QuotaWindowLimits{FiveHour,Weekly} is too narrow for Codex's self-describing window_minutes or opencode's daily Zen window.

B8 — Surface: system unauthenticated bypass (§7.1) — closed

X-Orchestra-Surface: system was reachable from any HTTP request header in both authz.HTTP and main.go's surface closure — since no deployment sets ORCHESTRA_SYSTEM_TOKEN, this was an unauthenticated full-control bypass reachable from any LAN caller. Both call sites now downgrade system to web before doing anything else with it. System is only constructible in-process (router leases/failures, coordinator releases/blocks, standup advisory/apply) via the bus-level authz.AuthorizeEvent check now enforced inside store.Append itself.

S1 — duplicate JSON struct tags — closed

Brief.From/To and GitSync.Branch/Head/Status each shared one JSON tag (Go only honors the first json:"..." tag on a combined field declaration), making go vet ./... fail and the brief's git state unparseable. Fixed; go vet ./... passes clean.

S2 + S3 — brief git state and completion receipts — closed

Brief.Git used to read from ORCHESTRA_DATA (the event-log dir, never a git checkout — always "git unavailable") and completions were counted but never surfaced with proof. operations.Brief.Git is now map[string]GitSync keyed by project ID, built from registry.Project.Repo (falling back to a single "default" entry off ORCHESTRA_REPO), with Ahead/Behind vs upstream added. Brief.Receipts []CompletionReceipt pulls report_ref/ receipt straight out of each TaskCompleted event's existing payload.

S4 — notification goroutine died on first send error — closed

delivery.Fanout.Run used to return on the first sender error, permanently killing notifications for the rest of the process's lifetime. Failed sends now go through an OnError hook instead of aborting the loop. Cursor is persisted (SaveCursor → a delivery-cursor file next to ORCHESTRA_DATA), so a restart resumes from the last delivered event instead of re-notifying the entire log from seq 0.

S5 — colliding event IDs on lease — closed

Store.Lease/ExpireLeases set Event.ID to the task id, so every lease of a task produced colliding event IDs. Now uses domain.NewID().

S6 — ingest dedup returned the wrong event with 200 — closed

Dedup path returned nil (success) without appending; the HTTP handler then returned an unrelated event with 201. Added domain.ErrDuplicate and Store.TaskBySource; POST /v1/tasks now returns the existing task with 200 on a duplicate. Every other Append caller (Gitea poll/webhook, JSONL ingest) now treats ErrDuplicate as expected.

S7 — TaskAmended dropped most fields — closed

Only title was ever applied from an amendment payload; due/ description/inherent_priority were accepted, logged, and silently dropped by the projection. Also added the missing Task.Description field entirely (it didn't exist, so TaskCreated dropped it too). Both TaskCreated and TaskAmended now populate all four fields.

S8 — no compensating-event mechanism (§3.1) — closed

Added TaskCorrected: payload requires corrects (the id of the event it repairs) plus at least one change (state, or the existing amend-style fields). Store.Append rejects a corrects that doesn't name a real prior event on the same task; apply applies the changes and clears Lease like every other terminal-state branch. Covered by a mistaken TaskFailed reverted to queued, an unknown-corrects rejection, and both events surviving snapshot+replay.

S9 — duplicate lease-expiry tickers — closed

Coordinator.Monitor's 30s ticker and main.go's own 1s reclaim ticker both called Store.ExpireLeases independently. Previously filed as "harmless — CAS rejects the loser," but the real effect was worse: the coordinator's expire() is the only place that kills the herdr session/pane for an expired lease, and the 1s ticker running 30x more often almost always won the race, leaving the coordinator's own call with nothing left to expire — silently orphaning herdr panes past TTL. Fixed: main.go's ticker skips ExpireLeases entirely when a coordinator is configured (deferring reclaim to the coordinator's loop), keeping only AssignPending as periodic retry.

S10 — federation worker registration had no admission control — closed

Any caller could self-declare a worker id and self-choose a token, and a second caller could silently re-register an existing worker id with a different token, hijacking its identity/capacity. Added Registry.AdmitToken (pre-shared secret via ORCHESTRA_FEDERATION_ADMIT_TOKEN, checked against the registration request's bearer header); same-ID re-registration now requires presenting the existing worker's own token (legitimate restarts still succeed; different-token same-id is rejected as a hijack).

S11 — soft threshold, milestone, thrash, agent-initiated ROTATE (§5.3) — closed

All four of §5.3's rotation triggers now land:

• Soft threshold: Coordinator.Soft (default 0.55, ORCHESTRA_OCCUPANCY_SOFT) — both rotate() and TurnDecision request a handoff once occupancy crosses Soft, before Hard forces one; TurnDecision returns prepare_handoff advisory (task stays leased, no turn boundary required).
• Agent-initiated ROTATE: handoffReason(worktree) reads a written .orchestra-handoff.json; if reason == "manual", both rotate() and TurnDecision skip occupancy and the turn-boundary probe entirely and release immediately — the agent's own handoff is the boundary signal. Shared release-and-certify tail extracted into Coordinator.finishRelease so this path gets the same anchor-safety guarantee as the threshold path.
• Milestone + thrash: needed a transcript/tool-call introspection source this repo didn't have — built new, internal/herdr/activity.go:
  • ToolCall{Name, Kind, Key, Success, IsTest} normalizes one tool/function call across harnesses.
  • ClaudeActivity pairs tool_use/tool_result blocks by tool_use_id from the transcript already opened for occupancy (an unresolved tool_use — mid-turn — is dropped, not reported).
  • CodexActivity — verified against a real live rollout, 2026-07-28, after the original guessed function_call/function_call_output shape turned out not to exist anywhere in ~/.codex/sessions. Real shape: file edits arrive as event_msg/patch_apply_end (has changes + top-level success directly, no pairing needed); shell commands arrive as a freeform response_item/custom_tool_call named "exec" whose input is a JS snippet — codexExecCommand regex-extracts the first embedded cmd:"..."; failure is signaled by the literal output prefix "Script error:" (confirmed for both a JS syntax error and an apply_patch verification failure on this machine's real transcripts). Manually re-ran the rewritten parser against a real multi-hundred-line rollout end-to-end and spot-checked the output by eye, in addition to TestCodexActivityParsesRealRolloutShape / TestCodexActivityMarksScriptErrorAsFailure.
  • OpenCodeActivity refuses outright — opencode's on-disk message storage is only confirmed to carry aggregate token counts, not per-tool-call records; fabricating a parser against an unconfirmed shape would repeat the exact mistake this audit exists to catch.
  • DetectThrash(calls, ThrashConfig) implements all three rules: N consecutive failed test runs (regex-matched build/test commands), the same file edited M times (edit tools only, explicitly excluding Read — caught by an initially-failing test), and an identical tool call repeated K times back-to-back (excluding test re-runs and file reads — also caught by an initially-failing test). Defaults 3/5/4, overridable via Coordinator.Thrash.
  • DetectMilestone(calls) — deliberately narrow: only "the most recent call was a successful git commit." Fuzzier definitions (passing test suite, finished subtask) were not guessed at.
  • CLIAdapter.Activity resolves the session file the same way Occupancy does; ActivityReader is an optional Face-B capability like the others.
  • New ReasonedHandoffRequester/CLIAdapter.RequestHandoffReason names why (thrash's dead ends, milestone reasoning) instead of the generic threshold framing.
  • rotate()/TurnDecision generalize "reason=manual bypasses occupancy" to manual/milestone/thrash alike: checkActivityTriggers runs before falling through to occupancy/soft/hard (thrash takes priority over milestone), and a hit calls requestReasonedHandoff — request only, never release, same as the soft-threshold path.

Covered by internal/herdr/activity_test.go and internal/orchestrator/rotation_test.go's TestActivityTriggersRequestReasonedHandoffWithoutReleasing, TestTurnDecision subtests for soft-threshold and manual-reason, and TestRotationRequestsHandoffBeforeReleasing.

Known remaining limitation: DetectMilestone's single-command extraction only sees the first cmd:"..." in a chained Codex exec script, so a git commit issued later in the same script isn't recognized as the "last call" even though it executed last. Opencode still has no verified per-tool-call source at all.

Also closed this pass (not separately numbered in the original audit)

• Codex/opencode Stop-hook-equivalent scripts. Neither harness has a native Stop hook, so the server-side /v1/harness/turn and /v1/harness/complete (harness dispatch) existed with no caller for either. Added deploy/hooks/orchestra-codex-poll.sh and orchestra-opencode-poll.sh — background poll loops (default 60s, ORCHESTRA_POLL_INTERVAL) that find the newest session-state file (codex: newest rollout-*.jsonl; opencode: newest file under ~/.local/share/opencode/storage/message/), POST to /complete when .orchestra-report.md exists, otherwise POST to /turn and log (not act on) refuse/rotate_now — advisory-only, since there's no real turn boundary to refuse at from outside either harness process without deeper app-server/SSE integration.
• Bus-level authorization. authz.AuthorizeEvent enforced inside store.Append itself (the single choke point every event passes through), not just at HTTP handlers. Event schema bumped to v2, requiring every event to declare a Surface; schema v1 events on disk still replay (tolerant reader).
• Dual quota windows. router.QuotaAvailability tracks a 5-hour rolling window and a 7-day weekly window independently per harness, applying the conservative 80% rule to each separately.
• Turn-boundary detection made observable. An adapter that fails to answer TurnBoundary now blocks that tick's release rather than treating the failure as "safe to proceed," and increments MonitorHealth.TurnBoundaryDegraded.
• Cross-machine lease correctness has a primitive-level test. TestCrossMachineLeaseAnchorAndQuotaArePerHost exercises a lease claimed through the federation worker HTTP API, validates the anchor against that worker's own local checkout, and asserts quota is accounted per-host — proof for the primitives that exist today; does not yet run against two real physical machines (see federation fork, below).
• Fuzz coverage (internal/domain/fuzz_test.go, FuzzValidatePayload/FuzzValidateEvent) for all event types including malformed nested payloads — no panic, always a typed error.
• Multi-repo Gitea ingestion. provider.Gitea gained a Project field and namespaced SourceName(); provider.MultiGitea dispatches by task.Source; LoadGiteaConfigs loads a JSON array of per-project sources. Legacy single-repo env vars still work unchanged. Previously zero tests existed for the Gitea provider at all (an earlier progress.md claim of coverage was inaccurate) — internal/provider/gitea_test.go added.
• Per-project repos. registry.Project gained optional repo/ worktree_root; main.go builds a PerProjectGitWorktrees, falling back to the global default for any project that omits these fields.
• Rotation's TaskReleased payload validity. Coordinator.rotate used to build {"handoff_ref","reason"}, omitting the anchor_sha the spec and domain.ValidatePayload require whenever handoff_ref is present — store.Append would reject it, the error was discarded (if c.Store.Append(e) == nil), and rotation silently never happened, no visible failure. Fixed: herdr.HeadSHA(worktree) populates anchor_sha from the real worktree HEAD before appending; if the anchor can't be read, rotation now skips that tick instead of emitting a payload guaranteed to fail validation. The federation worker release endpoint (/v1/federation/workers/{id}/release) had the identical gap and now requires/forwards a 40-hex-char anchor_sha, rejecting with 400 otherwise.

---

The federation fork — decided 2026-07-27, still the deferred half

Two incompatible federation designs coexist in the tree.

Design A — "drive the remote socket" (currently deployed, clients/herdr-bridge.go): homesrv calls worktree.create/agent.start/ etc. directly on workpc's herdr over TCP as if it were local — meaning anchor validation (git rev-parse HEAD) executes on the wrong machine relative to the actual checkout. Two outcomes if the coordinator's session.Worktree path happens to also exist on homesrv (likely, since every project shares the same directory layout): either HeadSHA errors and rotation silently skips forever, or — the dangerous case — it returns homesrv's HEAD for an unrelated checkout, passing validation while certifying a commit the agent never touched. Same class of bug applies to per-host quota accounting and to cleanupCompleted's git worktree remove, which runs on homesrv for a worktree that lives on workpc.

Design B — "workers pull tasks" (/v1/federation/*): fully built server-side (registration, heartbeat/TTL offline detection, event-cursor polling/ack, lease claim), zero clients — no worker binary exists anywhere in this repo. This is what the spec actually describes (§2.1: "everything crossing a machine boundary is git + a validated artifact, never live state over the wire"), but every endpoint is currently unreachable in the real deployment. Latent, never-surfaced defects in this unused half: offline detection only runs inside Snapshot(), called solely from a GET endpoint — nothing ticks it on its own, so OnOffline (the hook that releases leases held by a vanished worker) only fires if a human hits that endpoint; the registry is in-memory with no persistence, so a restart forgets all workers and cursors.

Decision, unchanged: keep Design A through Phase 5 (single-host concerns — occupancy, Face B, rotation, continuity — are provable on homesrv alone with workpc's herdr as just another pane host); commit to Design B in Phase 6. Two guardrails were meant to land immediately so Design A can't corrupt state in the meantime: (1) refuse to rotate a lease held by a non-local herdr rather than validate against the wrong checkout, since the protocol schema can't run rev-parse where the checkout is; (2) same treatment for cleanupCompleted's worktree removal. Status of these two guardrails is unverified in this pass — re-check internal/orchestrator before assuming they landed; they are not confirmed closed above the way B1–B11/S1–S11 are.

If the worker binary is ever abandoned, delete /v1/federation/* and record the deviation — leaving both designs in place unmarked is explicitly not acceptable; the repo would read as though the spec-conformant path is implemented when nothing can reach it.

---

Live deployment facts (as of 2026-07-28)

• Runs as orchestra.service on homesrv via this machine's own systemd. homesrv has no local herdr running (connection refused on 9245) — only workpc's herdr (192.168.1.105:9245) is live and reachable. main.go only logs herdr connection failures at startup, never successes, so "no log line" for a herdr does not mean it's down.
• Stuck task, deliberately left untouched: workspace wA, task 06FT6CKD9Y98AZRX6X8K3QXFZG, opencode harness, pane wA:p1. From Orchestra's own point of view it is no longer "stuck" — it now reads state: "failed" (retries exhausted, MaxAttempts hit before B4's fix landed). But the underlying herdr pane is still live and agent_status: "blocked", confirmed via a direct agent.get probe — the router gave up and moved on, but nothing ever released or killed the actual agent, confirming the orphaned-pane prediction from B2/B5. Left untouched on purpose: user was asked and chose to leave it rather than have it cleaned up mid-audit. Do not call pane.close/pane.release_agent against it without asking again first.
• First live end-to-end task run (2026-07-28, after rebuilding/redeploying — the running service had been on stale pre-audit commit 325c684) is what surfaced B12 above, since fixed and confirmed live; re-verifying it surfaced B13, still open.
• Three more live orphaned panes from this pass, left untouched: wD:p1 (task 06FTAKSJZTB73FZQE3QT7XQ1J0, worktree /tmp/test-e2e-worktrees/06FTAKSJZTB73FZQE3QT7XQ1J0 — has a real attached claude session but the task itself is stuck blocked from before B12's fix landed), wE:p1 and wF:p1 (tasks 06FTAMFKVEPGPKR0H86C08ZZNG / 06FTANAYZPA55BABR2J4MWQ050 — empty shells, no agent ever attached, see B13). Same rule as wA:p1: don't pane.close/pane.release_agent any of these without asking first.

---

Known open gaps (as of 2026-07-28)

• B13 (above) — agent.start can silently no-op under back-to-back leases, with no error surfaced; blocks reliable live verification of everything downstream (occupancy, rotation, handoff, completion) against a real task, even though B12's readiness race is now fixed.
• Cross-machine lease correctness proven at the primitive level only — needs an actual homesrv/workpc pair over the real mesh; this repo cannot exercise that by itself.
• Turn-boundary Face B degrades to occupancy-only for adapters that don't implement it, by design — degradation is observable (MonitorHealth.TurnBoundaryDegraded) and blocks-on-failure, but whether Claude/Codex/opencode's native hooks are wired in a live deployment is a deployment-config fact, not provable from source alone.
• Quota has no live push producer — only the post-hoc /v1/harness/complete producer exists; see B7's "design consequences" for the (unbuilt) Claude statusline / Codex rollout-tail feeds.
• Federation guardrails' landed-status is unverified — see federation fork section above.
• opencode has no verified per-tool-call activity source and no first-class quota surface (Zen headers only, would need a plugin) — both named, not attempted, in S11/B7.
• DetectMilestone's single-command extraction can miss a Codex commit issued later in a chained exec script.

Everything else audited (provider layer, continuity schema/validation, router matching, delivery, federation registration primitives) was spot-checked against the code and its tests and matched described behavior.