diff --git a/AUDIT.md b/AUDIT.md index 6e73497..245c549 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -1,1101 +1,687 @@ -# Orchestra — spec conformance audit & remediation plan +# Orchestra — spec conformance audit & remediation status Audited 2026-07-27 against `orchestra-spec (1).md` at commit `325c684`. Method: read every non-test file in `internal/` and `cmd/`, traced each spec section to its call site, and checked whether the live path (`main.go` → -router → coordinator → adapter) actually reaches it. +router → coordinator → adapter) actually reaches it. This file is the single +merged record — a prior split between `AUDIT.md` (audit + plan) and +`progress.md` (chronological session log) has been flattened here; treat any +older chronological git history of either file as session notes, not ground +truth. + +**Ground truth rule for future sessions:** this repo has a documented history +of code that *looks* wired but isn't (packages with tests that pass in +isolation while the live call path silently no-ops). Before trusting any +"done"/"fixed" claim below, check the actual call site, not just this +document. --- -## Verdict +## Current verdict (as of 2026-07-28) -The substrate (Layer 1) is genuinely built. Layers 2–4 are **shaped** but not -**wired**: the packages exist, have tests, and compile — but the code paths -that would make an unattended run work are either dead, mis-keyed, or built -against herdr methods that appear to be invented. +The substrate (Layer 1) is solid. Layers 2–4 were originally shaped-but-not- +wired; most of the blocking defects below are now closed, verified by +reading the current code (not just by trusting this file) and by targeted +tests. Two things are not yet proven: -Concretely: **a task can be created, routed, and leased. It can never -complete, and it can never rotate.** After 3 lease expiries the router marks -it `TaskFailed`. That is the whole of "connection works but features are -underdeveloped." +1. **A real end-to-end unattended run.** The first live attempt (2026-07-28) + surfaced **B12** (fixed and confirmed live, see below), and re-verifying + that fix surfaced **B13** (open) — `agent.start` can return success while + silently never starting an agent, under back-to-back leases. One task + (`wD:p1`) did make it all the way to a real attached `claude` session, so + the lease path is provably reachable, but B13 means it's not yet reliable + enough to call proven — occupancy/rotation/handoff/completion still + haven't been exercised end-to-end against a session that's guaranteed to + actually exist. +2. **Cross-machine (federation) correctness.** Deliberately deferred — see + "The federation fork" below. -`progress.md` overstates completion in several places (see §Corrections). +`go build ./...`, `go vet ./...`, and `go test ./...` all pass. | Spec layer | State | |---|---| -| L1 substrate (§3, §4) | Built and correct in the main path. Some defects (below), no missing mechanism. | -| L2 harness (§5) | **Not functional.** Occupancy broken, rotation unreachable, adapter protocol unverified. | -| L3 continuity (§6) | **Dead code.** Handoff schema, pickup validation, scratch branches, TASK.md — none reached from the live path. | -| L4 surfaces (§7) | Partial. Brief/standup/delivery exist; quota projection has no producer; authz has a bypass. | +| L1 substrate (§3, §4) | Built and correct in the main path. | +| L2 harness (§5) | Occupancy, rotation, and completion all wired and reachable; B12's readiness race is fixed and confirmed live, but B13 (`agent.start` silently no-op'ing under back-to-back leases) still blocks reliable live verification of what's downstream. | +| L3 continuity (§6) | Handoff schema, pickup validation, scratch branches, TASK.md are all wired into the live path (Phase 4 complete). | +| L4 surfaces (§7) | Brief/standup/delivery real; quota has a post-hoc producer only (no live push feed yet); authz bypass closed. | --- -## Blocking defects — in dependency order +## Blocking defect currently open -### B1. Occupancy is measured from the wrong value (§5.2.1) +### B13 — `agent.start` silently no-ops under back-to-back leases (found live 2026-07-28) -`herdr/adapter.go:191` +Discovered while live-verifying the B12 fix below. B12 itself is confirmed +fixed (see that entry), but re-testing it exposed a second, deeper defect +that B12's retry logic cannot address. -```go -u, e := a.Usage(s.PaneID) // ClaudeUsage(path string) wants a transcript file +Three fresh tasks were leased in quick succession against `workpc-claude` +(`06FTAKSJZTB73FZQE3QT7XQ1J0`/pane `wD:p1`, `06FTAMFKVEPGPKR0H86C08ZZNG`/pane +`wE:p1`, `06FTANAYZPA55BABR2J4MWQ050`/pane `wF:p1`). All three got a +`worktree.create` success and an `agent.start` call that returned **no +error**. Checked live via `pane.get`/`pane.list` (and independently +confirmed by the user cd'ing into each worktree by hand): + +- `wD:p1` (the first, given time alone to settle): a real `claude` process + did eventually attach — `agent: claude`, `agent_status: idle`, + `revision: 2`. This is consistent with B12's model: `agent.start` + acknowledges the request but the actual attach is asynchronous, sometimes + taking well over a minute. +- `wE:p1` and `wF:p1` (started moments after `wD:p1`, while it was presumably + still initializing): **still no agent attached after 2+ minutes of + polling** — `agent_status: unknown`, no `agent` field, `revision` never + incremented past its creation value. Empty shells, confirmed both via + `pane.get` and by the user directly `cd`ing into the worktree and finding + no session running. + +So `agent.start`'s "success" is not just delayed for these two — it never +happened at all, and `agent.start` never returned an error to tell Orchestra +that. One plausible cause: something about launching a second/third `claude` +CLI process while a prior one on the same host/account is still mid-startup +(auth handshake, config lock, etc.) silently drops the later request(s) +rather than queuing or erroring them. Not confirmed — would need a +controlled single-at-a-time repro to isolate from herdr's own internals, +which requires spinning up more real panes and wasn't done this pass (see +"three orphaned panes" below). + +**Consequence:** unlike B12 (which at least produced an observable +`TaskBlocked`), this failure mode leaves the task leased indefinitely against +a pane that will never produce a session, with no error surfaced anywhere — +worse than B12 was, because nothing currently distinguishes "still booting, +give it more time" from "silently dead, will never start." No fix attempted +yet; a client-side retry (B12's approach) cannot fix this because the +`agent.start` call that should have started the process already returned +success. + +**Side effect of this investigation — three live orphaned panes on workpc, +left untouched on purpose:** `wD:p1` (has a real but abandoned `claude` +session, task now stuck in `blocked` from before B12's fix), `wE:p1` and +`wF:p1` (empty shells, no agent ever attached). Not cleaned up — same +standing rule as the pre-existing `wA:p1` stuck task: don't call +`pane.close`/`pane.release_agent` against live panes without asking first. + +--- + +## B12 — `StartAgent` races the pane's shell readiness (found live 2026-07-28) — closed (this specific race; see B13 for a second issue it exposed) + +Discovered during the first real end-to-end run against a freshly rebuilt/ +redeployed binary (the running service had been on stale pre-audit commit +`325c684` the whole time). + +Created a fresh task (`06FTAH4MCRCAJYY7V75Z4V2YGG`, project `test-e2e`, +capability `opencode`). It reached `TaskLeased` onto `workpc-opencode` +(worktree created for real at `/tmp/test-e2e-worktrees/` on workpc, +confirmed via a live `pane.list` — pane `wB:p1`, correct `cwd`), then +**445ms later** hit `TaskBlocked`: + +``` +lease: herdr protocol error: agent target pane wB:p1 is not an available shell ``` -`ClaudeUsage`/`CodexUsage`/`OpenCodeUsage` all take a **filesystem path** to -session state. They are handed a **herdr pane ID**. Every call returns -`open : no such file`, so `Coordinator.rotate` hits -`if err != nil { continue }` on line 430 and never rotates anything. +`internal/herdr/herdr.go:199` (`StartAgent`) calls `agent.start` on the pane +`Worktree` just recorded, with no wait or retry after `worktree.create` +returns. herdr hands the pane back before its shell has actually finished +initializing, and `agent.start` refuses it as not-yet-a-shell. None of B1–B11 +exercised this — every prior fix assumed a session already existed +(occupancy, rotation, handoff); nothing in the remediation plan tested the +very first `Lease` call against a real herdr pane end-to-end. Phase 0's +probing used `params:{}`/missing-field tricks, never a real +`worktree.create` → `agent.start` sequence timed against actual pane boot. -The spec is unusually explicit here — §5.2.1: *"Build this measurement and -verify it against a live session **before** wiring any trigger — the whole -rotation system rests on this number."* That step was skipped, and the -verification gap is exactly where the bug is. +**Reproduced a second time, 2026-07-28**, same session: a second fresh task +(`06FTAHW8GZP0S6DCHEQ6PJXWB0`) leased onto a brand-new pane (`wC:p1`) and hit +the identical error on the same sub-second timeline. Two for two — a +systematic race, not a flake. -Compounding: `CodexActiveUsage` (the discovery function that resolves the -active rollout via `state_*.sqlite`) and `ClaudeStopHookUsage` and -`OpenCodeStatus` are written but **called from nowhere** outside tests. +**Consequence:** every fresh lease that hits this race goes straight to +`TaskBlocked`, and the router **never retries a blocked task** +(`internal/router/router.go` has no reference to `StateBlocked` at all, +confirmed by grep) — so it sits there permanently until a human intervenes +via the approval surface. This is now the single blocking defect for proving +the rest of the remediation plan against a real live task. -### B2. The adapter lookup key is wrong in three of four call sites (§5.3, §5.4) +**Fix, iterated live across three rounds, 2026-07-28:** -`AdapterFactory.Herdrs` is keyed by **herdr instance id** (`homesrv-claude`). -`Session.Harness` is set by `CLIAdapter.Lease` to the **harness kind** -(`claude`). Then: +1. First pass: bounded-retry `agent.start` (10×, 500ms apart) specifically on + the `"not an available shell"` string. Redeployed and re-tested against a + real fresh lease — this cleared that exact error, but the very next call + in the sequence (`Lease`'s post-`agent.start` `Prompt`) then hit a + *different* transient error, `"agent ... is not an active named agent"` — + same underlying readiness race, one call later, worded differently. +2. Second pass: added the identical string-matched retry to `Prompt` for that + specific message. Redeployed, re-tested — this time hit a *third* distinct + wording, `"agent target ... not found"`, on the same call. +3. Given three different error strings for the same race across three live + attempts, string-matching was abandoned as unwinnable (herdr's wording for + "not ready yet" isn't fixed enough to enumerate). **Final shape:** both + `StartAgent`'s `agent.start` call and `Prompt`'s `agent.prompt` call now + retry on *any* error, bounded by wall-clock time (15s, `bootRetryDelay` of + 500ms) rather than attempt count or specific wording — there is no other + legitimate reason a call against a pane/worktree orchestra itself just + created would fail immediately. `StartAgent` still special-cases `"already"` + as a non-error (idempotent re-lease). -| Call site | Key used | Resolves? | -|---|---|---| -| `refreshSessionHealth` (orchestrator.go:221) | `HerdrID`, falls back to lease | yes | -| `Reconcile` (:291) | `session.Harness` | **no** | -| `expire` (:376, :397) | `session.Harness` | **no** | -| `rotate` (:419) | `session.Harness` | **no** | +Confirmed this closes the original race: task `06FTAKSJZTB73FZQE3QT7XQ1J0` +(pane `wD:p1`) leased successfully, and polling `pane.get` live shows a real +`agent: claude` / `agent_status: idle` attach — independently confirmed by +the user `cd`-ing into that worktree by hand. `go build`/`go vet`/`go test +./...` all pass throughout. -So: orphaned panes are never killed on restart, `pane.exited` fast-path never -fires, expired leases never kill their pane, and rotation exits before it -begins. All silently — every one is a bare `continue`. - -### B3. Nothing emits `TaskCompleted` (§4, §5.2) - -Grep for producers: there is exactly one, `POST /v1/tasks/{id}/complete`, -which a human has to call. The spec assigns this to the **stop-hook/wrapper** -("the plane emits events, not the agent" — Invariant 2). There is no stop -hook, no wrapper, no completion detection of any kind. - -Consequence: an agent that finishes its work sits idle until the 30-minute -lease TTL expires → `TaskReleased` → re-leased → repeat → `TaskFailed` at -attempt 3. Unattended runtime, the one metric in §0, is currently bounded at -90 minutes per task and always ends in a false failure. - -### B4. The router counts rotation as a retry (§5.3, §5.4) - -`router.go:127` increments `attempts[TaskID]` on **every** `TaskReleased`, -and `:169` increments again on every lease. `:149` fails the task at -`MaxAttempts` (3, from `main.go:64`). - -Rotation *is* `TaskReleased` (§5.3: "rotation = intra-task lease transfer"). -So a task healthy enough to rotate twice is killed by the retry limit. The -spec's retry policy is for *failures* (§5.4), not lease transfers. These need -separate counters — `attempts` should only advance on expiry/crash releases, -never on a release carrying a valid `handoff_ref`. - -### B5. Herdr protocol methods are unverified and at least partly invented - -`adapter.go` calls `pane.release`, `pane.kill`, `pane.rotation_signal`, -`pane.read`, `agent.get`, `agent.start`, `worktree.create/open`, -`agent.prompt`. The file itself documents that a previous method -(`pane.status`) *was not a valid protocol method* — so this surface has a -history of being written against an imagined API. - -`pane.rotation_signal` is near-certainly not real: herdr is a generic -multiplexer with no concept of Orchestra rotation. And `pane.release` -returning a `handoff_ref` is architecturally wrong regardless of whether the -method exists — **herdr does not write handoffs; the agent does** (§6.1, the -handoff is a CAS artifact produced under the Face-B stop hook). - -Nothing here can be settled from source. It needs `herdr api schema --json` -run against your protocol-17 instance, diffed against every method the -adapter calls. - -Related: §5.1 requires `agent.prompt` with **inline `wait`** for bootstrap -injection, because that's what closes the send-into-a-half-rendered-prompt -race. `CLIAdapter.Lease` calls `Prompt(..., wait=0)` — no wait. Only -`Bootstrap` passes one. - -### B6. Layer 3 is entirely dead code (§6) - -Never called from anything but tests: -`continuity.ValidatePickup`, `ScratchCommit`, `ScratchPush`, `ScratchPull`, -`ScratchSync`, `MarkdownChanges`, `Save`, `Load`, `Encode`, `Decode`. - -`VerifyTaskFile` *is* called — but only when `GitWorktrees.TaskFileSHA` is -non-empty, and `main.go:112` constructs the worktrees **without ever setting -it**. Nothing writes a `TASK.md` into a worktree in the first place. - -So the entire §6.2 pickup contract — the part the spec calls -"the make-or-break" and "the defense against the telephone game" — does not -execute. Rotation, if B1/B2 were fixed, would hand the next agent a -`handoff_ref` that was never schema-validated, never anchor-checked, and -never accompanied by an immutable spec. - -### B7. Quota projection has no producer (§7.2) - -`QuotaAvailability.sumSince` folds `QuotaReported` events. `AggregateQuota` -does the same for the brief. **No code appends a `QuotaReported` event.** -Your config sets `quota_limit_5h: 50` / `quota_limit_weekly: 500` on all six -herdrs; those limits are compared against a permanent 0. The availability -filter is a no-op and the brief's `quota_consumed` is always `{}`. - -### B8. `Surface: system` is an unauthenticated full-control bypass (§7.1) - -`authz.CapabilityFor` grants `System` → `FullControl`. `authz.HTTP` reads the -surface straight from the `X-Orchestra-Surface` header, and the token map in -`main.go:760` has **no entry for `system`** — so `tokens[System]` is `""` and -the token check is skipped entirely. Any request on the LAN with -`X-Orchestra-Surface: system` can emit any event on any task. - -The surface is also absent from `ParseSurface`'s intent: `System` is supposed -to mean "the plane itself, in-process", but it's reachable over HTTP. It -should never be accepted from a request header. - -(Also: unset `ORCHESTRA_*_TOKEN` means that surface is unauthenticated; your -`.orchestra-config/orchestra.env` sets none of them. Defensible on WireGuard, -worth a deliberate decision rather than an accident.) +**Not fully clean, however:** re-testing this fix by firing two more tasks +shortly after the first one surfaced a second, separate defect — `agent.start` +returning success while never actually starting an agent, with no error for +the retry logic to even see. That's **B13**, above — this entry only covers +the specific shell/agent-readiness race B12 was originally filed for, which +*is* fixed; B13 is a distinct root cause the same investigation exposed. --- -## Secondary defects - -| # | Where | Issue | -|---|---|---| -| S1 | `operations.go:16,25` | `go vet` fails: `Brief.From/To` both serialize as `"from"`; `GitSync.Branch/Head/Status` all as `"branch"`. The brief's git state is unreadable by any client. | -| S2 | `main.go:238` | Brief's git state is read from `ORCHESTRA_DATA` (the event-log dir), not from the project worktrees. §7.4 wants "what pushed, what's on which branch, what workpc still needs to pull" — per project. | -| S3 | `operations.BuildBrief` | Counts completions but never surfaces `report_ref`/`receipt`, which §7.4 names as the proofs the brief exists to carry. | -| S4 | `delivery.Fanout.Run:110` | A single send error `return`s and permanently kills the notification goroutine — silently, since `main.go:729` only logs. One ntfy hiccup at 2am = no notifications for the rest of the night. Cursor is also in-memory, so a restart re-notifies the entire log from seq 0. | -| S5 | `store.Lease:341`, `ExpireLeases:350` | `Event.ID` is set to the **task id**, so event IDs collide across every lease of a task. `ApplyAdvisory` and any future ID-based lookup are unsound. | -| S6 | `store.Append:191` | Ingest dedup returns `nil` (success) without appending; `main.go:175` then returns `s.Events(0)[len-1]` — an unrelated event — with `201 Created`. | -| S7 | `store.apply:154` | `TaskAmended` applies only `title`. §4 lists title, due, description, `inherent_priority`. Amendments to the others are accepted, logged, and silently ignored by the projection. | -| S8 | §3.1 | No compensation-event mechanism exists. Append-only holds, but the spec's *correction* path ("a compensating event is appended") has no implementation. | -| S9 | `main.go:482` | Two independent lease-expiry loops (`main.go` 1s ticker and `Coordinator.expire` 30s) race on the same reclaim. Harmless today only because version CAS rejects the loser. | -| S10 | federation | `federation.Registry.Register` accepts a self-declared `id` + self-chosen `token` from any caller — registration is admission-control-free. | -| S11 | §5.3 | ~~Thrash detection, the soft ~55% threshold, milestone rotation, and agent-initiated `ROTATE` are all absent. Only the hard threshold exists, and it's unreachable (B1).~~ Closed 2026-07-28 — soft threshold, agent-initiated `ROTATE`, milestone, and thrash detection all landed. Codex's activity parser is unverified against a live rollout; opencode has no verified tool-call source and refuses rather than guess. | - ---- - -## The architectural fork — decided 2026-07-27 - -There are **two incompatible federation designs** in the tree, and only one is -deployed. - -### Design A — "drive the remote socket" (deployed) - -`clients/herdr-bridge.go` runs on workpc and proxies its local herdr Unix -socket to `192.168.1.105:9245`. `.orchestra-config/config.jsonc` registers -`workpc-claude` / `workpc-codex` / `workpc-opencode` against that machine, so -the **homesrv** coordinator resolves a workpc herdr as a candidate and calls -`worktree.create`, `agent.start`, `agent.prompt` on it over TCP as if it were -local. workpc is a dumb pane host; all orchestration logic and all state live -on homesrv, and live state crosses the machine boundary continuously. - -### Design B — "workers pull tasks" (`/v1/federation/*`, unused) - -workpc would run its own Orchestra process that registers itself -(`POST /v1/federation/workers`), heartbeats, polls the event log with a cursor -(`GET /v1/federation/events` + `/ack`), claims its own lease -(`POST .../claim`), runs a **local** coordinator against its **local** herdr -socket, and reports a handoff carrying an `anchor_sha` it computed from its -own checkout (`POST .../handoff`). homesrv stays authoritative for the log; -nothing but git and validated artifacts crosses the wire. - -This is what the spec describes — §2.1: *"Everything crossing a machine -boundary is git + a validated artifact — never live state over the wire."* -**It is fully written on the server side and has zero clients.** There is no -worker binary in this repo; every one of those endpoints is unreachable in -the current deployment. - -### Why keeping both is harmful, not merely untidy - -Design A breaks a specific correctness property Design B was written to hold. - -**The anchor is validated on the wrong machine.** `Coordinator.rotate` -(orchestrator.go:460) calls `herdr.HeadSHA(session.Worktree)`, which shells -out to `git -C rev-parse HEAD`. The coordinator runs on homesrv; -`session.Worktree` is a path on **workpc**. Two outcomes, both bad: - -- the path doesn't exist on homesrv → `HeadSHA` errors → rotation silently - skips forever (another bare `continue`); or -- the path *does* exist on homesrv — likely, since every project shares the - `/var/lib/orchestra/worktrees//` layout — → it returns - **homesrv's HEAD for an unrelated checkout**, and the emitted `TaskReleased` - certifies a commit the agent never worked on. - -The second is the dangerous one: it passes validation, looks correct, and -hands the successor an anchor describing another repo's state. That is exactly -the failure §9 item 8 names — *"worktree/anchor validation when a lease's git -checkout lives on a different host than the router"* — against the -degrade-safe default the spec already supplies: *"validate against the local -checkout wherever the harness runs."* Design B satisfies this by construction; -Design A cannot satisfy it without moving validation out of the coordinator. - -The same class of bug applies to per-host quota accounting (§9 item 8) and to -`cleanupCompleted`, which runs `git worktree remove` on homesrv for a worktree -that lives on workpc. - -### Latent defects in the unused half - -Never surfaced because nothing exercises them: - -- **Offline detection only runs inside `Snapshot()`** (federation.go:103-118), - called solely from `GET /v1/federation/workers`. Nothing polls it, so - `OnOffline` — the hook at `main.go:131` that releases leases held by a - vanished worker — fires only if a human hits that endpoint. The TTL backstop - §5.4 relies on does not tick on its own. -- **The registry is in-memory with no persistence.** Every restart forgets all - workers and their cursors; a reconnecting worker re-reads the entire log - from seq 0. - -### Decision - -**Keep Design A through Phase 5; commit to Design B in Phase 6.** Phases 1–4 -(occupancy, Face B, rotation, continuity) are single-host concerns, provable -on homesrv alone with workpc's herdr as just another pane host. Resolving the -federation question first would block the fixes that actually unblock -unattended runtime. - -Two guardrails land **immediately**, so Design A cannot corrupt state in the -meantime: - -1. Anchor computation happens where the checkout is. Under the bridge that - means having herdr run the `rev-parse` in the pane's worktree rather than - shelling out locally — or, if the protocol schema won't support it, - **refusing to rotate any lease held by a non-local herdr** until Phase 6. A - loud refusal beats a false anchor. -2. Same treatment for `cleanupCompleted`'s `git worktree remove`. - -Phase 6 then builds the worker binary against the endpoints that already -exist, adds server-issued tokens (S10), persists the registry, and drives -offline detection from a ticker rather than a request handler. - -What is explicitly **not** acceptable is leaving both designs in place -unmarked. If the worker binary is ever abandoned, delete `/v1/federation/*` -and record the deviation — as it stands the repo reads as though the -spec-conformant path is implemented when nothing can reach it. - ---- - -## Remediation plan - -Ordered so each phase is verifiable on its own and nothing depends on an -unproven layer below it — the spec's own build discipline (§8). - -### Phase 0 — Ground truth (half a day, blocks everything) - -Nothing below is safe to write until the herdr surface is known. - -1. On workpc: `herdr api schema --json > herdr-schema.json`, commit it to - `deploy/`. -2. Write `internal/herdr/schema_test.go`: for every method string in - `adapter.go`, assert it exists in the committed schema with the params we - send. This is the test that would have caught `pane.status` and will catch - the next one. -3. Manually drive one pane end to end against real herdr — create worktree, - start agent, prompt with inline wait, read status, kill — and record the - actual request/response shapes. Fix `adapter.go` to match. -4. Delete `pane.rotation_signal` and the `RotationSignal` interface unless the - schema proves it exists. - -**Done when:** `schema_test.go` passes against the committed schema, and a -scripted manual run starts and stops a real Claude Code pane on workpc. - -### Phase 1 — Occupancy, measured for real (§5.2.1) — fixes B1 - -The spec says build this first and verify against a live session. Do exactly -that, and nothing else in this phase. - -1. Add `SessionFile string` to `herdr.Session`. Resolve it at lease time, per - harness: - - **claude** — the transcript path. Cleanest source is the Stop hook's - stdin (`transcript_path`), which you need for Phase 2 anyway; until then - resolve `~/.claude/projects//.jsonl` by - newest-mtime under the encoded worktree dir. - - **codex** — `CodexActiveUsage` already does the `state_*.sqlite → - threads.rollout_path` discovery. Call it. Filter by the rollout whose cwd - matches the worktree. - - **opencode** — `OpenCodeStatus` against `:4096` as fast path, - `~/.local/share/opencode/storage/message/` as backstop, per §5.2.1's - "the SSE stream is not rock-solid". -2. Change `CLIAdapter.Occupancy` to use `s.SessionFile`, and make a missing - session file a **hard error surfaced in `MonitorHealth`**, never a silent - `continue`. -3. Keep the §5.2.1 trap explicit: assert in a test that a fixture with a large - cumulative total but small last-turn usage yields *low* occupancy. This is - the failure mode the spec singles out. -4. Add `GET /v1/tasks/{id}/occupancy` returning the raw numerator, the window, - and the fraction — so you can eyeball it against a live session before - trusting it to drive rotation. - -**Done when:** a real Claude Code session at a known context fill reports a -fraction within a few points of what `/context` says. - -### Phase 2 — Face B and completion (§5.2, §4) — fixes B3, B5 - -1. **Claude Code Stop hook** (`deploy/hooks/orchestra-stop.sh`, bash, no - deps — §5.6). Reads hook JSON on stdin, POSTs `transcript_path` + task id - to a new `POST /v1/harness/turn` endpoint. Exit 2 with stderr when the - plane says "rotate but no valid handoff yet" — that's the spec's refuse-the- - turn mechanism, and `ClaudeStopHookUsage` already parses the input shape. -2. `POST /v1/harness/turn` computes occupancy, evaluates the rotation triggers, - and returns a decision: `continue` | `prepare_handoff` (soft) | - `rotate_now` (hard) | `refuse` (over threshold, handoff missing/invalid). -3. **Completion**: `POST /v1/harness/complete` — the wrapper/hook path that - uploads the report to CAS and emits `TaskCompleted` with `report_ref` + - `receipt`. Wire the receipt from Phase 1's occupancy reader, **summed - across lease intervals** per §4. -4. Codex/opencode Face B: rollout-tail and SSE `session.status` respectively, - polled by the coordinator rather than hook-pushed. Same decision endpoint. - -**Done when:** a task given a trivial goal on a real harness reaches -`TaskCompleted` with a `report_ref` resolvable from CAS, with no human action. - -### Phase 3 — Make rotation reachable and correct (§5.3, §5.4) — fixes B2, B4 - -1. Key sessions by `HerdrID` everywhere. Add a single - `Coordinator.adapterFor(session) (herdr.Adapter, error)` and route all four - call sites through it. Add a regression test that registers an adapter - under `homesrv-claude`, leases with `Harness: "claude"`, and asserts - rotation still fires. -2. Replace every silent `continue` in `rotate`/`expire` with a recorded reason - on `MonitorHealth`. The class of bug in B1/B2 is only invisible because of - those bare continues. -3. Split the router's counters: `releases` (informational) vs `failures` - (drives `MaxAttempts`). A `TaskReleased` carrying a valid `handoff_ref` - must not advance `failures`. Test: a task that rotates 5 times is still - alive. -4. Implement the missing triggers (§5.3): soft 55% → `prepare_handoff`; - milestone; thrash (N failed test runs / same file M times / identical tool - calls) as a circuit breaker with `reason=thrash` and populated `dead_ends`; - agent-initiated `ROTATE`. -5. Make TTL and the retry policy config, not the hardcoded `30*time.Minute` at - `router.go:165` and `1800` in `main.go` (§5.4: "N, ttl, retry backoff = - config"). -6. **Land the two Design-A guardrails** (see the federation decision above): - compute the anchor where the checkout is, or refuse to rotate a lease held - by a non-local herdr; same for `cleanupCompleted`'s worktree removal. These - are the fixes that make it safe to defer federation to Phase 6. - -**Done when:** a long-running task crosses 75% occupancy, rotates at a turn -boundary, and the successor continues — twice in a row, unattended. - -### Phase 4 — Wire Layer 3 into the live path (§6) — fixes B6 - -This is where the dead code becomes load-bearing. - -1. **Write `TASK.md`** into the worktree at creation from the `TaskCreated` - payload; hash it; store the hash on the task; pass it as `TaskFileSHA` into - `PerProjectGitWorktrees` (`main.go:112`). Re-inject it on every rotation - bootstrap — §6.2 is explicit that this is what defeats the telephone game. -2. **Handoff production**: the rotating agent writes the §6.1 TOML/JSON - handoff; the stop hook uploads it via `POST /v1/artifacts`; the plane - `continuity.Decode`s it, rejecting free-form `[knowledge]` (Invariant 4), - and only then emits `TaskReleased` with the hash. Never accept a - `handoff_ref` the plane hasn't validated. -3. **Scratch-branch commit before release** (`ScratchCommit`), so §6.2 step 3 - collapses to one sha compare. `ScratchCommit` already refuses to commit a - dirty `TASK.md` — good, keep that. -4. **Pickup**: `ValidatePickup` runs in `Coordinator.Start` before the - successor's bootstrap prompt. Fail → do not start; emit `TaskBlocked`. -5. Replace `CLIAdapter.Bootstrap`'s freeform prose with the §6.2 procedure - (~200 tokens: read handoff, run validate-handoff, re-read TASK.md, - proceed), and use `agent.prompt`'s inline wait. -6. Wire `MarkdownChanges` to the §6.3 adjacent-task notice, or delete it and - record the deferral. Dead code that looks implemented is what produced this - audit. - -**Done when:** a rotation whose anchor has drifted is *refused*, visibly, and -one whose anchor is clean proceeds without the successor re-deriving context. - -### Phase 5 — Close the surfaces (§7) — fixes B7, B8, S1–S4 - -1. Emit `QuotaReported`. Source: the same per-harness session state as - Phase 1, on a timer per `harness+window` (§7.2 — "a projection keyed by - `harness+window`, fed by the same per-harness session state"). Until then, - your quota limits are decorative. -2. Reject `X-Orchestra-Surface: system` at the HTTP boundary unconditionally. - `System` must be constructible in-process only. Then decide explicitly - whether the remaining surfaces require tokens on your LAN, and set them. -3. Fix the `go vet` json-tag collisions (S1) — the brief is currently - unparseable for git state. -4. Brief: per-project git sync state from the actual worktrees (S2), and - include `report_ref`/`receipt` in the rollup (S3). -5. Delivery: never `return` on send error — log, back off, continue. Persist - the cursor next to the event log (S4). Deliver the morning brief itself. - -**Done when:** `/v1/brief` for an overnight window shows real quota per -harness, real per-project git state, and the receipts for every completion. - -### Phase 6 — Federation, decided (§2.1, §9 item 8) — fixes the fork, S10 - -Only after Phases 1–5 run clean on a single host. - -1. Anchor validation and `HeadSHA` execute **where the checkout is**. Today - they run on homesrv against a workpc path. -2. Either build the worker binary against `/v1/federation/*` (git-only - transport, per spec) or delete that API and document the bridge as a - deliberate deviation. Do not keep both. -3. Add admission control to worker registration (S10): server-issued tokens, - not self-declared. -4. Then run the real two-machine overnight batch that §9 item 8 asks for. - ---- - -## Corrections to `progress.md` - -Worth fixing, because the next session will otherwise trust it: - -- "Continuity: strict handoff schema/validation, CAS save/load, pickup - validation, scratch-branch commit/push/pull helpers" — accurate as a - description of the *package*, misleading as a description of the *system*. - None of it is reachable at runtime. -- "herdr adapters with native occupancy readers" — the readers exist; they are - called with the wrong argument and always fail. -- "Router: ... quota availability at conservative 80% threshold" — the - threshold logic is correct, but no `QuotaReported` event is ever produced, - so it evaluates against zero. -- The rotation fix described in "Verified fixed this pass" is real and - correct — but `rotate` cannot reach that code, because the adapter lookup - above it (B2) fails first. The test passes because it registers the adapter - under the harness kind rather than the herdr id, which the production - config never does. -- "`go build ./...` and `go test ./...` both pass" — true; `go vet ./...` - does not. - ---- - -## Suggested order of attack - -If you want one thing to do today: **Phase 0**, then **B2** (a ~20-line fix -that makes three subsystems reachable), then **B1**. Those three turn a system -that cannot rotate into one that can, and everything else in the plan is -building on top rather than repairing underneath. - ---- - -## Phase 0 — done, 2026-07-27 - -B1, B2, B4, B8, S1, S5, S6 were already fixed and landed as of this session -(confirmed by reading the current code, not just trusting progress.md — see -`adapterFor` in `internal/orchestrator/orchestrator.go:190` and -`CLIAdapter.Occupancy` in `internal/herdr/adapter.go:197`). - -This box (homesrv) turned out to have live TCP reachability to the real herdr -instance at `192.168.1.105:9245` (workpc) the whole time — the `unavailable: -connection refused` lines in `journalctl -u orchestra.service` are for -`homesrv-*` herdrs dialing `192.168.1.104:9245`, which has no local herdr -running; `workpc-*` herdrs were connecting fine but main.go never logs a -success, only a failure, so there was no positive signal either way. Also -found: **a real task is stuck live right now** — workspace `wA`, task -`06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode agent, pane `wA:p1`, `agent_status: -"blocked"` — almost certainly stuck because `Release`/rotation could never -reach it (see below). - -Ran the actual Phase 0 steps against this live instance (raw JSON-RPC probes -over TCP, params-omitted/empty-object tricks to read Rust serde's -missing-field errors — no `herdr` CLI available locally, so `herdr api schema ---json` itself wasn't run, but the equivalent info was extracted this way). -Full method list and findings committed to `deploy/herdr-schema.json`. - -**Confirmed, with a real server response, not just static reading of -adapter.go:** -- `pane.release`, `pane.kill`, `pane.rotation_signal` — **none of these exist** - in the real protocol. Confirms B5's suspicion exactly. -- Real replacement for `pane.kill` is `pane.close({pane_id})` — same shape, - drop-in. **Fixed** in `internal/herdr/adapter.go`. -- Real replacement for `pane.release` is `pane.release_agent({pane_id, - source, agent})` — structurally different, and per B5's own analysis it - cannot return a `handoff_ref` regardless (herdr doesn't write handoffs, the - agent does, §6.1). Wiring this for real needs Phase 4's handoff-production - path first. `CLIAdapter.Release` now returns a loud error naming exactly - that instead of calling a method that doesn't exist. **Not a full fix** — - Phase 4 still owns making Release do something real. -- `pane.rotation_signal` doesn't exist and never will (herdr has no rotation - concept) — deleted `RotationSignal` interface, its `CLIAdapter` method, and - the call site in `Coordinator.rotate`, per this doc's own instruction - ("Delete ... unless the schema proves it exists"). -- `agent.get`, `pane.read`, `agent.prompt`, `worktree.create`, `worktree.open`, - `agent.start` — all confirmed real, no changes needed there. -- Protocol version confirmed live: `17`, matching `config.jsonc`'s - `"protocol": "17"` (returned as a bare JSON number by the server; the - existing string-fallback parse in `CheckProtocol` happens to handle that - correctly already). - -`go build ./...`, `go vet ./...`, `go test ./...` all pass after these -changes. - -## B3 — partial fix, 2026-07-27 - -Added `POST /v1/harness/complete` (`cmd/orchestra/main.go`) — the first -automatic `TaskCompleted` producer. Design: a Claude Code Stop hook -(`deploy/hooks/orchestra-stop.sh`) runs on every turn boundary but only POSTs -when the agent has written `.orchestra-report.md` at the worktree root — -that file is the explicit "I'm done" signal, since Stop fires on every pause -and treating every stop as completion would be wrong (this is exactly the -distinction Phase 2 items 1–2, the `continue`/`prepare_handoff`/`rotate_now`/ -`refuse` turn-decision endpoint, are meant to own — that endpoint is still -unbuilt, so there is currently no plane-side signal telling the agent when to -rotate vs. finish; the marker-file convention is a stopgap that only covers -the completion half). - -The handler builds the `receipt` server-side from `herdr.ClaudeUsage` against -the transcript path the hook supplies (same local-filesystem assumption as -`CLIAdapter.Occupancy` — doesn't hold for a session hosted on a herdr that -isn't local to the machine running orchestra, i.e. the federation-fork -caveat applies here too) rather than trusting a self-reported number, and -uploads the report body via `Store.PutArtifact` for `report_ref`. Event is -appended with `Surface: string(authz.System)` hardcoded in Go — not read from -a request header — consistent with B8 (system must never be -header-controlled); gated instead by an optional `ORCHESTRA_HARNESS_TOKEN` -bearer check. - -**Not done:** Codex/opencode completion producers (only Claude wired), the -turn-decision endpoint itself, and no test — `cmd/orchestra/main.go` has zero -handler test coverage of any kind (everything is inline in `main()`), so this -follows the existing gap rather than introducing an isolated test harness for -one handler. - -**Fixed 2026-07-27 (later same day):** `CLIAdapter.Lease`'s bootstrap prompt now -passes `time.Minute` instead of `wait=0`, matching `Bootstrap`'s inline-wait -pattern — closes the race B5 named (send-into-a-half-rendered-prompt). -`internal/herdr/adapter.go`. `go build`/`vet`/`test` all still pass; no -existing test asserted the old `wait=0` value. - -**Still open from B5** (not attempted this pass — larger, needs design, not -just a method-name swap): -- `Release`'s real implementation, which depends on Phase 4 (§6) handoff - production existing at all. -- The stuck live task (`06FT6CKD9Y98AZRX6X8K3QXFZG`) was deliberately **not** - manipulated directly (no `pane.close`/`pane.release_agent` call against it) - — killing or releasing a real running agent from an audit session without - the user present is exactly the kind of action that warrants asking first. - -## B5 — closed, 2026-07-27 (later same day) - -`CLIAdapter.Release` now does something real instead of refusing. Design -mirrors the `.orchestra-report.md` marker convention B3 already established -for completion, since the same problem applies to handoffs: the plane must -never invent a handoff, only validate and forward the one the agent wrote -(§6.1). Concretely: - -1. The agent is expected to write `.orchestra-handoff.json` - (`herdr.HandoffFile`) at the worktree root before its stop hook lets - rotation proceed — a §6.1 handoff schema, not free prose. -2. `Release` reads that file, decodes it with `continuity.Decode` (schema + - required-field validation, same as pickup), and cross-checks - `Anchor.GitSHA` against `herdr.HeadSHA(session.Worktree)` — the anchor is - re-verified against the real checkout, not trusted from the agent's - self-report, closing the same class of gap as B3's receipt-from-transcript - choice. -3. Only if both checks pass does it upload the handoff via `continuity.Save` - (`CLIAdapter.CAS`, wired to the same `*store.Store` used everywhere else) - and return the resulting ref — this is what `Coordinator.rotate` puts in - `TaskReleased.handoff_ref`. -4. Only *then* does it call the real `pane.release_agent({pane_id, source: - "herdr:"+harness, agent: harness})` to drop herdr's claim — sequenced last - so a herdr-side failure can't strand an already-uploaded handoff with no - way to retry the release call (retrying `Release` re-reads the same file - and is idempotent). - -A missing or invalid handoff file, an anchor mismatch, or a `pane.release_agent` -error are all refused (non-nil error, no event emitted) — `Coordinator.rotate` -already treats an errored `Release` as "leave the lease intact, retry next -tick," so this gives the agent room to finish writing the handoff rather than -stranding the task. - -`herdr.Claude`/`Codex`/`OpenCode` constructors now take a `continuity.CAS` -parameter; `cmd/orchestra/main.go` passes the existing `*store.Store` (which -already implements `PutArtifact`/`Artifact`). - -New tests in `internal/herdr/adapter_test.go` drive `Release` against a real -git worktree and a fake in-process herdr TCP listener (`fakeHerdr`) responding -to `pane.release_agent`: upload-and-release on a valid handoff, refusal with -no handoff file, refusal on anchor mismatch, refusal with no CAS configured. - -**Still not done** (unchanged, separate from B5 itself): nothing yet makes -the *agent* actually write `.orchestra-handoff.json` — that's Phase 4 item 2's -other half (a stop-hook-side convention, analogous to -`.orchestra-report.md`/`deploy/hooks/orchestra-stop.sh` for completion) and -Phase 4 items 3/5/6 (`ScratchCommit` before release, the §6.2 bootstrap-prompt -rewrite, `MarkdownChanges` wiring). `go build ./...`, `go vet ./...`, and -`go test ./...` all still pass. - -## B6 — partial fix, 2026-07-27 (Phase 4 items 1 and 4) - -Two of Phase 4's six items landed; the rest are unchanged (still open, listed -below). - -1. **TASK.md is now actually written.** `continuity.RenderTaskFile(t)` - produces the §6.2 immutable spec content from `domain.Task`; - `GitWorktrees.Create` (`internal/orchestrator/orchestrator.go`) writes it - into every freshly created worktree and commits it immediately — it must - be committed, not left dirty, both so `ScratchCommit`'s "TASK.md is - immutable" check (which inspects `git status`) sees it as clean, and so - its hash is stable across whatever the agent does afterward. A worktree - that already has a `TASK.md` (recreation on restart) is left untouched. - New `continuity.TaskFileHash(root)` reads it back and hashes it — this is - what makes `w.TaskFileSHA`/`VerifyTaskFile`, previously dead because - nothing ever set `TaskFileSHA`, actually reachable. -2. **Pickup validation now gates bootstrap.** `Coordinator.Start` - (`internal/orchestrator/orchestrator.go`) computes the new worktree's - `TaskFileHash`, and — whenever the lease carries a `handoff_ref` (i.e. - this is a rotation continuation, not a fresh lease) — loads the handoff - from CAS and runs `continuity.ValidatePickup(worktree, handoff, - taskFileSHA)` **before** calling `Adapter.Bootstrap`. A validation failure - kills the just-started session and emits `TaskBlocked` instead of hand­ing - the successor an unverified anchor. This is exactly the gap B6 named: - `ValidatePickup` had no caller outside its own tests. - `TestStartBlocksOnInvalidPickup` (`rotation_test.go`) drives this against - a handoff whose anchor SHA doesn't exist in the repo and asserts the task - ends `Blocked`, never bootstrapped. - `TestGitWorktreesCommitsTaskFile` (`worktrees_test.go`) asserts the - written/committed content matches `RenderTaskFile` and survives - recreation. - -Caveat recorded in code: TASK.md hashing is best-effort — if a worktree came -from the `WorktreeCreator` (herdr-hosted, potentially remote) path rather -than `GitWorktrees`, `TaskFileHash` fails silently and pickup validation runs -with an empty `taskFileSHA` (so it still checks anchor SHA and dirty-file -hashes, just not TASK.md). No current adapter or test exercises -`WorktreeCreator` with a real `handoff_ref`, so this is unverified, not -proven safe — same cross-host caveat as the federation-fork section. - -**Still open from B6/Phase 4** (unchanged, larger and needs live-agent -cooperation): -- Item 2: handoff *production* — the rotating agent writing the §6.1 - handoff and a stop-hook path uploading it via `POST /v1/artifacts` before - `Coordinator.rotate` calls `Adapter.Release`. `Release` still just refuses - (see B5 above) — there is nothing yet to validate-and-mint a ref from. - -## Phase 4 items 3, 5, 6 — landed 2026-07-27 - -1. **`ScratchCommit` wired into `Release`, not into `rotate`.** Rather than - calling it from `Coordinator.rotate` (which only has a `herdr.Session`, - not the handoff), `CLIAdapter.Release` now runs it itself, after - validating the agent-authored handoff's `Anchor.GitSHA` against the - worktree's real HEAD and re-verifying every `Anchor.Dirty` file's hash - still matches what the agent recorded (previously untested — a file - edited *after* the handoff was written but before release would have - silently sailed through). If the handoff has dirty entries, `Release` - commits them atomically onto `orchestra/scratch/` via - `continuity.ScratchCommit`, then **rewrites the handoff's anchor** to the - new scratch commit SHA with `Dirty` cleared, before uploading to CAS — - this is what "collapses §6.2 step 3 to one sha compare" means in - practice: the successor's `ValidatePickup` now only needs - `git rev-parse HEAD == handoff.anchor.git_sha`, no per-file rehashing, - because everything was committed before the ref was minted. - `ScratchCommit` itself was changed to be idempotent — reuse an existing - scratch branch (`git switch` before falling back to `git switch -c`) and - skip the commit if there's nothing to snapshot — since a task can rotate, - and therefore hit this path, more than once. - Covered by `TestReleaseScratchCommitsDirtyFilesBeforeUpload` (asserts the - anchor advances to the new commit, dirty is cleared, and the worktree - ends up on the scratch branch) and `TestReleaseRefusesOnStaleDirtyFile` - (internal/herdr/adapter_test.go). -2. **Item 5 — Bootstrap prompt rewritten.** `CLIAdapter.Bootstrap` no longer - sends the one-line "read handoff, validate anchor, continue" prose. It - now tells the agent the plane has *already* validated anchor/TASK.md - (true, per B6's `ValidatePickup` gate in `Coordinator.Start` — no need to - ask the agent to redundantly re-verify trust), and points it at - `git log --stat -5` / `git branch --show-current` in the worktree as the - actual source of "what the prior agent did and what's left," since that's - now a real, inspectable scratch-branch commit rather than an opaque ref. - Deliberately does **not** claim a `GET /v1/artifacts/` fetch path — - no such HTTP route exists (`/v1/artifacts` is POST-only, upload only, - confirmed by reading `cmd/orchestra/main.go`); an earlier draft of this - prompt invented that endpoint and was corrected before landing, which is - exactly the class of bug this audit exists to catch. -3. **Item 6 — `MarkdownChanges` deleted 2026-07-27, then §6.3 rewired from - scratch 2026-07-27 (later same day).** The original `MarkdownChanges` - function had zero callers and zero tests (confirmed by grep before - deleting), so it was removed rather than half-wired, per this doc's own - "delete and record the deviation" option. A real implementation of §6.3 - ("on update, the orchestra injects a notice to agents whose current task - is adjacent") was built separately, decoupled from the deleted function: - `continuity.ConventionsHash(root)` hashes whichever of - `AGENTS.md`/`CLAUDE.md`/`VOCAB.md` exist at a path; `herdr.Session` gained - `ConventionsHash`, snapshotted from the fresh worktree at - `Coordinator.Start`; a new `Coordinator.checkConventions`, run every - `Monitor` tick, recomputes the hash of the *project's base repo* (via - `WorktreeSpec.Spec`, i.e. "adjacent" = same project) for every leased - session and compares it against that session's stored snapshot — a - mismatch means the shared docs were updated upstream since this session - started. On mismatch it calls a new optional `herdr.ConventionsNotifier` - capability (`CLIAdapter.NotifyConventionsChanged`, an `agent.prompt` telling - the agent to re-read the docs) and updates the stored hash so the notice - fires once per drift, not every tick. Deliberately does not touch the - brief or add a new event type — the notice is a direct in-pane nudge, not - a projection, matching the spec's phrasing ("injects a notice to agents"), - so there was no event-schema question to resolve first. Covered by - `TestConventionsDriftNotifiesActiveSession` - (internal/orchestrator/rotation_test.go): asserts zero notifications while - the base repo's docs are unchanged, then a notification once they diverge. - `go build`/`go vet`/`go test ./...` all pass. - -**Phase 4 item 2 — closed 2026-07-27.** The remaining gap after B5 was not -"which harness" (Release was always harness-agnostic) — it was that *nothing, -for any harness*, ever told the agent the `.orchestra-handoff.json` convention -existed. `Coordinator.rotate` (internal/orchestrator/orchestrator.go) now -checks whether the resolved adapter implements a new optional -`herdr.HandoffRequester` capability; if `HandoffFile` isn't present in the -worktree yet, it calls `RequestHandoff` once (`herdr.Session.HandoffRequested` -guards against re-prompting every tick) and skips `Release` for that tick, -leaving the lease intact — exactly the same "ask, don't invent" shape as B3's -`.orchestra-report.md` convention. `CLIAdapter.RequestHandoff` -(internal/herdr/adapter.go) sends a prompt naming the exact §6.1 JSON shape -(`meta.id`, `anchor.git_sha`/`branch`/`dirty[].{path,sha256}`) and explicitly -tells the agent not to fabricate the SHA/hashes. Once the file exists, the -existing Release path (validate → scratch-commit dirty files → upload → CAS -→ `pane.release_agent`) is unchanged. Covered by -`TestRotationRequestsHandoffBeforeReleasing` -(internal/orchestrator/rotation_test.go), which asserts `Release` is never -called while the file is absent and fires once it's written. `go build`, -`go vet`, `go test ./...` all pass. - -Not attempted here (separate, deployment-level question, not a code gap): -whether Codex's/opencode's own turn-boundary mechanism actually surfaces this -in-pane prompt to the agent before it exits the way Claude Code's Stop hook -does — that's Phase 2 item 4 territory (native Face B per harness), not -Phase 4. - ---- - -## Real harness quota sources — verified locally, 2026-07-27 - -Investigation, not a code change: B7 says `QuotaReported` has exactly one -producer (the `TaskCompleted` handler in `cmd/orchestra/main.go`, deriving -`consumed` from the harness's self-reported `usage.Numerator()`). The -question was whether any harness exposes its *real* subscription quota so -Orchestra can stop relying on operator-entered static caps plus estimated -token consumption. Two of three do. Everything below was read off this -machine's own installs, not recalled. - -### Claude Code — statusline stdin (confirmed) - -The JSON blob Claude Code pipes to `statusLine.command` on every render -carries server-reported rate-limit levels. `~/.claude/statusline.sh` already -reads them: - -- `.rate_limits.five_hour.used_percentage` -- `.rate_limits.seven_day.used_percentage` - -plus `.context_window.{used_percentage,total_input_tokens,context_window_size}`, -`.cost.total_cost_usd`, `.session_id`. These are percentages of the real -subscription pool, not estimates, and the two windows map 1:1 onto -`router.QuotaWindowLimits{FiveHour, Weekly}`. High frequency, zero cost. - -Note: `claude.ai/api/organizations/{org_id}/usage` also exists but is -authenticated by **claude.ai session cookies, not an API key** — wiring it -would make Orchestra hold and refresh a logged-in browser session. The -statusline path avoids that entirely and should be preferred. - -### Codex — `rate_limits` in the session rollout (confirmed) - -Every `token_count` event in `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` -carries a `rate_limits` object, e.g.: - -```json -"rate_limits": { - "limit_id": "codex", - "primary": { "used_percent": 20.0, "window_minutes": 10080, "resets_at": 1785650936 }, - "secondary": null, - "credits": { "has_credits": false, "unlimited": false, "balance": "0" }, - "plan_type": "plus", - "rate_limit_reached_type": null -} -``` - -Richer than Claude's: `window_minutes` makes the window self-describing -(10080 = weekly) and `resets_at` is absolute. There is **no `codex usage` -subcommand** — the transport is the rollout file, or `codex app-server`, -which emits the same events live. - -### opencode — no first-class quota surface - -- `opencode stats` is historical accounting only (cost/tokens/tools across - past sessions). No limits, no remaining. -- Zen is an OpenAI-compatible gateway at `https://opencode.ai/zen/v1` (and - `/zen/go/v1`). The binary contains `x-ratelimit-limit` / - `-remaining` / `-reset` / `-reset-after`, so the Zen free tier's **daily** - quota arrives as response headers, not via a queryable endpoint. Capturing - it requires intercepting a response — an opencode plugin - (`~/.config/opencode/plugin/`) is the only clean hook. - -### S4 — closed, 2026-07-27 - -`delivery.Fanout.Run` no longer `return`s on the first sender error — a -single ntfy hiccup used to permanently kill the notification goroutine for -the rest of the process (`main.go` only logged the `Run` error, it never -restarted the goroutine). Failed sends now go through an `OnError` hook -(default `log.Printf`), and the loop keeps going to the next sender/event. -Cursor persistence was also added: `SaveCursor` is called every time the -cursor advances, and `main.go` wires it to a `delivery-cursor` file next to -`ORCHESTRA_DATA`, loaded on startup — a restart resumes from the last -delivered event instead of re-notifying the entire log from seq 0. Covered -by `TestFanoutContinuesAfterSendError` (`internal/delivery/delivery_test.go` -— previously the package had zero tests): a failing sender and a healthy -sender both receive the event, the cursor still advances, and `Run` only -exits on context cancellation, never on the send error. - -### S2, S3 — closed, 2026-07-27 - -`Brief.Git` was a single `GitSync` read from `ORCHESTRA_DATA` (the event-log -directory, never a git checkout, so it always reported `"git unavailable"`) — -S2 named this and asked for per-project state from the actual worktrees. -`Brief.Git` is now `map[string]GitSync` keyed by project ID; `/v1/brief` -(`cmd/orchestra/main.go`) builds it from `registry.Project.Repo` for every -project that sets one, falling back to a single `"default"` entry keyed off -`ORCHESTRA_REPO` for single-repo deployments that predate per-project repos. -`operations.GitState` also gained `Ahead`/`Behind` (via -`git rev-list --left-range --count @{u}...HEAD`) so "what pushed" and "what -workpc still needs to pull" (§7.4's own phrasing) are both answerable, not -just branch/HEAD/dirty. - -S3: `BuildBrief` counted `TaskCompleted` but discarded the payload, so the -brief never carried the `report_ref`/receipt proofs §7.4 names as the reason -completions are surfaced at all ("the receipts for every completion"). Added -`operations.CompletionReceipt{TaskID, ReportRef, Receipt}` and -`Brief.Receipts []CompletionReceipt`, populated straight from each -`TaskCompleted` event's existing payload (no new event fields needed — -`domain.ValidatePayload` already requires both on every `TaskCompleted`). -Covered by an updated `TestBuildBrief` (`internal/operations/operations_test.go`) -asserting a completion's `report_ref`/`receipt` and a project's `GitSync` -both come through in the brief. - -`go build ./...`, `go vet ./...`, `go test ./...` all pass. - -### S7 — closed, 2026-07-27 - -`store.apply`'s `TaskAmended` case only ever applied `title` from the -amendment payload — `due`, `description`, and `inherent_priority` amendments -were accepted by `ValidatePayload` (which only checks the payload is -non-empty) and durably logged, but silently dropped by the projection, so a -client reading back the task would never see them take effect. Also found in -passing: `domain.Task` had no `Description` field at all, so a description -couldn't be amended onto a task even if the projection had handled it — -`TaskCreated` discarded it too. Added `Task.Description`; both `TaskCreated` -and `TaskAmended` in `store.apply` now populate/update all four fields -(`title`, `description`, `inherent_priority`, `due`), matching what §4 lists -as amendable. Covered by `TestTaskAmendedAppliesAllFields` -(`internal/store/store_test.go`). - -## S11 — partial fix, 2026-07-27 (soft threshold) - -Of S11's four missing pieces (soft ~55% threshold, milestone rotation, thrash -detection, agent-initiated `ROTATE`), only the first is landed here. - -`Coordinator` gained a `Soft float64` field (default 0.55 via `soft()` when -unset, configurable through `ORCHESTRA_OCCUPANCY_SOFT`). Both `rotate()` (the -periodic ticker) and `TurnDecision` (the synchronous per-turn path) now check -occupancy against `Soft` before `Hard`: at or above soft but below hard, they -call the adapter's `HandoffRequester.RequestHandoff` once (same -"ask, don't invent" convention as the hard-threshold path, -`Session.HandoffRequested` guarding re-prompts) and — for `TurnDecision` — -return `prepare_handoff` without requiring a turn boundary, since this is -advisory: the agent keeps working, the task stays leased. Only once occupancy -clears `Hard` does the existing boundary-check/release path run. Covered by -`TestTurnDecision/"prepare_handoff at soft threshold, below hard, without a -turn boundary"` (internal/orchestrator/rotation_test.go), asserting a -handoff request fires and the task remains `StateLeased`. - -**Still open from S11:** milestone rotation, thrash detection (N failed test -runs / same file M times / identical tool calls as a circuit breaker with -`reason=thrash` and populated `dead_ends`), and agent-initiated `ROTATE`. All -three need either transcript/tool-call introspection this repo doesn't yet -have a source for, or an explicit in-band signal from the agent — bigger than -a threshold comparison, not attempted this pass. - -`go build ./...`, `go vet ./...`, `go test ./...` all pass. - -## S11 — agent-initiated ROTATE, landed 2026-07-27 - -The last of S11's four pieces that fits an in-band signal (milestone and -thrash both need transcript/tool-call introspection this repo has no source -for — still open). §5.3: *"agent-initiated `ROTATE` → emitted when a coherent -unit finishes and the next is independent."* `continuity.Handoff`'s schema -already anticipated this — `reasons["manual"]` was valid since B5 landed, but -nothing ever checked for it. - -New `handoffReason(worktree)` (`internal/orchestrator/orchestrator.go`) reads -`HandoffFile` if present and returns its decoded `meta.reason`, or `""` if -absent/invalid — never `"manual"` on a bad read, so a malformed handoff can't -accidentally short-circuit rotation. Both `rotate()` and `TurnDecision` check -this first: if the agent already wrote a handoff with `reason=manual`, that -**is** the boundary signal, so occupancy and the turn-boundary probe are -skipped entirely and release proceeds straight away with `reason: "manual"` -in the emitted `TaskReleased`. Below that check, the existing -threshold/soft/hard logic is unchanged. - -Extracted the release tail (`Release` → anchor certification → `TaskReleased` -append) shared between the threshold and manual paths in `TurnDecision` into -`Coordinator.finishRelease`, since the manual path needed to reach the exact -same anchor-safety logic (never emit a payload with an uncertifiable anchor) -without going through occupancy/boundary gating first. - -Covered by `TestTurnDecision/"manual reason bypasses occupancy and turn -boundary"` (`internal/orchestrator/rotation_test.go`): occupancy=0, -boundary=false (both would refuse/continue under every other path), a -`.orchestra-handoff.json` with `reason=manual` written directly to the -worktree, and asserts `TurnRotateNow` + `Release` invoked + task state -`StateQueued`. - -**Still open from S11 (at this point):** milestone rotation and thrash -detection — both need a source of transcript/tool-call data this repo -doesn't have yet. - -`go build ./...`, `go vet ./...`, `go test ./...` all pass. - -## S11 — milestone + thrash detection, closed, 2026-07-28 - -The transcript/tool-call source both preceding entries said this repo -lacked now exists: `internal/herdr/activity.go`. `ToolCall{Name, Kind, Key, -Success, IsTest}` normalizes one tool/function call, harness-agnostically. -`ClaudeActivity` reads the same transcript file `ClaudeSessionFile`/ -`ClaudeUsage` already open, pairing `tool_use`/`tool_result` blocks by -`tool_use_id` (an unresolved tool_use is dropped, not reported — the session -is still mid-turn). `CodexActivity` mirrors the `payload.type` wrapper -`CodexUsage` already reads, parsing `function_call`/`function_call_output` -pairs — **explicitly marked best-effort/unverified**, same bar Phase 0 set -for herdr methods: not yet checked against a live rollout. `OpenCodeActivity` -**refuses outright** — opencode's on-disk message storage is only confirmed -to carry aggregate token counts, not per-tool-call records, so this doesn't -guess at an unconfirmed shape (same convention as -`CLIAdapter.resolveSessionFile`'s opencode case). - -`DetectThrash(calls, ThrashConfig)` implements all three §5.3 rules — N -consecutive failed test runs, the same file edited M times (edit tools only, -explicitly excluding `Read`), and an identical tool call repeated K times -back-to-back (excluding test re-runs and file reads, since those are -expected, not thrashing). Two false positives were caught by tests before -being trusted: rule 3 initially tripped on repeated test-command re-runs and -on repeated `Read`s of the same file; both fixed by narrowing rule 3's -"relevant" calls to non-test commands and edit-tool file calls only. -`DetectMilestone(calls)` is deliberately narrow — only "the last call was a -successful `git commit`" — fuzzier definitions (a passing suite, a finished -subtask) were not guessed at. - -New `ReasonedHandoffRequester`/`CLIAdapter.RequestHandoffReason` -(adapter.go) — like `RequestHandoff` but names the specific reason -(thrash's dead ends, or the milestone framing) and asks the agent to write -`meta.reason` accordingly, continuing the "ask, don't invent" pattern -already used for `.orchestra-report.md` and `.orchestra-handoff.json`. -`Coordinator.rotate()` and `TurnDecision` generalize the existing -"`reason=manual` bypasses occupancy" shortcut to `manual`/`milestone`/ -`thrash` alike, and both call new `checkActivityTriggers` (skips cleanly if -the adapter doesn't implement `ActivityReader`) before falling through to -occupancy/soft/hard; a hit calls `requestReasonedHandoff` (same -`HandoffRequested`-guarded ask-once pattern the occupancy path already -uses) — request only, never release, matching the soft-threshold path's -`prepare_handoff` behavior. - -Covered by `internal/herdr/activity_test.go` (parser + all three detector -rules, including the two cases above that caught real bugs) and +## Closed defects + +Each entry is the flattened final state — design + what's verified — not the +session-by-session narration. Grouped by original defect ID. + +### B1 — Occupancy measured from the wrong value (§5.2.1) — closed + +`CLIAdapter.Occupancy` was calling `a.Usage(s.PaneID)` (a herdr pane id) +against readers (`ClaudeUsage`/`CodexUsage`/`OpenCodeUsage`) that want a +filesystem path to session state — every call failed and was swallowed by a +bare `continue`. Fixed: `herdr.Session.SessionFile` is resolved at lease time +per harness (Claude: newest-mtime transcript under Claude Code's own project +dir; Codex: existing `CodexActiveUsage` sqlite discovery; opencode: refuses +loudly rather than guess, since it needs a live session id not resolvable +from the worktree alone). A missing/unreadable session file is now a hard +error surfaced via `SessionHealth.Occupancy`/`OccupancyError` on +`GET /v1/tasks/{id}/health`, never a silent zero. + +**Still open:** live verification against a real Claude Code session at a +known context fill — not possible from this sandbox, blocked further by B13 +now that fresh leases don't reliably survive. + +### B2 — Adapter lookup keyed wrong in 3 of 4 call sites (§5.3, §5.4) — closed + +`AdapterFactory.Herdrs` is keyed by herdr instance id (`homesrv-claude`); +`Reconcile`/`expire`/`rotate` were looking it up by `session.Harness` (the +harness kind, `claude`) and silently no-op'ing (`continue`) on every miss — +orphaned panes never killed, expired leases never killed their pane, +rotation never began. Fixed with a single `Coordinator.adapterFor(session)` +routed through at all four call sites. Regression test registers an adapter +under a herdr-id key distinct from the harness kind and asserts rotation +still fires. + +### B3 — Nothing emitted `TaskCompleted` (§4, §5.2) — closed + +Only producer used to be a human calling `POST /v1/tasks/{id}/complete`. +Added `POST /v1/harness/complete`: a Claude Code Stop hook +(`deploy/hooks/orchestra-stop.sh`) fires every turn boundary but only +reports completion once the agent has written a `.orchestra-report.md` +marker at the worktree root (an ordinary turn boundary is a no-op). The +server builds the `receipt` itself from `herdr.ClaudeUsage` against the +transcript (never trusts a self-reported number) and uploads the report +body to CAS for `report_ref`. Gated by optional `ORCHESTRA_HARNESS_TOKEN`; +event appended with `Surface: system` set directly in Go, consistent with +B8. Extended to Codex/opencode via an optional `harness` field in the +request body (`"codex"` → `CodexUsage`, `"opencode"` → `OpenCodeUsage`). + +**Still open:** no automated test for the HTTP handler (`cmd/orchestra/main.go` +has zero handler test coverage of any kind, pre-existing gap — this follows +the existing pattern rather than introducing a one-off harness). + +### B4 — Router counted rotation as a retry (§5.3, §5.4) — closed + +Every `TaskReleased` (including rotation, which carries a valid +`handoff_ref`) advanced `MaxAttempts`, and lease also double-counted — a task +healthy enough to rotate twice was killed. Now only a release *without* a +`handoff_ref` (expiry/crash) advances the counter. + +### B5 — Herdr protocol methods unverified/invented — closed + +Confirmed live against `192.168.1.105:9245` (raw JSON-RPC probing, no local +`herdr` CLI available — schema reconstructed from Rust serde's +"unknown variant"/"missing field" errors). `pane.release`, `pane.kill`, +`pane.rotation_signal`, `pane.status` **do not exist**. Real replacements: +`pane.close({pane_id})` for kill (drop-in); `pane.release_agent({pane_id, +source, agent})` for release — structurally different, does **not** return +a `handoff_ref` (herdr never writes handoffs; the agent does, §6.1). +`RotationSignal` interface/method/call site deleted outright (no replacement +exists; herdr has no concept of Orchestra rotation). + +`CLIAdapter.Release` now: reads the agent-authored `.orchestra-handoff.json`, +validates with `continuity.Decode`, cross-checks its anchor SHA against the +worktree's real `HeadSHA`, re-verifies every `Anchor.Dirty` file hash, +scratch-commits any dirty files (`continuity.ScratchCommit`, idempotent +across multiple rotations) and rewrites the handoff's anchor to the new +scratch commit before uploading to CAS via `continuity.Save` (minting the +real `handoff_ref`), and only then calls `pane.release_agent` — sequenced +last so a herdr-side error can't strand an uploaded handoff. Any failure +(missing file, invalid schema, anchor mismatch, stale dirty hash, herdr +error) is a refusal, which `rotate` already treats as "retry next tick." +`CLIAdapter.Lease`'s bootstrap prompt also fixed to pass `time.Minute` (not +`wait=0`) for inline-wait, matching `Bootstrap` and closing the +send-into-a-half-rendered-prompt race §5.1 requires guarding against. + +Covered by `internal/herdr/adapter_test.go` against a real git worktree and a +fake in-process herdr TCP listener: upload-and-release on a valid handoff, +refusal with no handoff file, refusal on anchor mismatch, refusal on stale +dirty-file hash, refusal with no CAS configured. + +Protocol version confirmed live as a bare JSON number (`17`), not the string +`config.jsonc` declares — `CheckProtocol`'s raw-bytes fallback happens to +compare correctly today; don't "clean up" that code without re-checking +this, or it may start doing a real numeric-vs-string comparison and break. + +**Still open:** nothing makes the agent actually write +`.orchestra-handoff.json` unprompted (closed separately, see B6/Phase 4 item +2 below) — that's the piece this entry originally deferred. + +### B6 — Layer 3 (continuity) was entirely dead code (§6) — closed + +Nothing wrote `TASK.md`, so pickup validation had nothing to check and never +ran. Fixed in full: + +- `GitWorktrees.Create` now writes and commits an immutable `TASK.md` + (`continuity.RenderTaskFile`) into every fresh worktree; + `continuity.TaskFileHash` reads it back for `w.TaskFileSHA`. +- `Coordinator.Start` runs `continuity.ValidatePickup` (anchor SHA + + dirty-file hashes + TASK.md hash) before bootstrapping a successor onto a + `handoff_ref` — failure kills the session and emits `TaskBlocked` instead + of trusting an unvalidated ref. +- `ScratchCommit` wired into `Release` (see B5) so pickup collapses to a + single HEAD compare instead of per-file rehashing. +- `CLIAdapter.Bootstrap`'s prompt rewritten to point the agent at + `git log`/the scratch branch (trust already established by the plane's own + `ValidatePickup`) instead of vague "read the handoff" prose, and + deliberately does not claim a `GET /v1/artifacts/` endpoint since none + exists (`/v1/artifacts` is POST-only — an earlier draft invented this and + was corrected before landing). +- `continuity.MarkdownChanges` (zero callers, zero tests despite being + listed as implemented in an earlier snapshot) was deleted rather than + half-wired. §6.3's actual requirement ("notice to agents whose task is + adjacent") was rebuilt independently: `continuity.ConventionsHash(root)` + hashes `AGENTS.md`/`CLAUDE.md`/`VOCAB.md`; `Coordinator.checkConventions` + runs every `Monitor` tick, recomputes the project base repo's hash for + every leased session, and on drift calls the optional + `herdr.ConventionsNotifier` capability (in-pane prompt) once per drift. +- **Handoff production, the item that most of B6 hinged on:** + `Coordinator.rotate` checks the adapter's optional `herdr.HandoffRequester` + capability; if `.orchestra-handoff.json` is missing, it prompts the agent + once (`CLIAdapter.RequestHandoff`, naming the exact §6.1 JSON shape and + explicitly telling the agent not to fabricate SHA/hashes) and skips + `Release` that tick, retrying every subsequent tick — mirrors the + `.orchestra-report.md`/B3 convention exactly. `Session.HandoffRequested` + avoids re-prompting every tick. + +Covered by `TestGitWorktreesCommitsTaskFile`, `TestStartBlocksOnInvalidPickup`, +`TestReleaseScratchCommitsDirtyFilesBeforeUpload`, +`TestReleaseRefusesOnStaleDirtyFile`, `TestConventionsDriftNotifiesActiveSession`, +`TestRotationRequestsHandoffBeforeReleasing` (internal/herdr, internal/orchestrator). + +**Caveat still open:** TASK.md hashing is best-effort/untested for the +herdr-hosted (`WorktreeCreator`) worktree path specifically — no adapter or +test exercises that path with a real `handoff_ref`, so pickup validation +there runs with an empty `taskFileSHA` (anchor + dirty-file hashes still +checked). Same cross-host caveat as the federation fork, below. + +### B7 — Quota projection had no producer (§7.2) — closed (post-hoc only) + +`POST /v1/harness/complete` now appends a `QuotaReported` event +(`consumed` from the same `usage.Numerator()` used for the receipt), so the +router's 5h/weekly filter and the brief's `quota_consumed` stop evaluating +against a permanent zero. + +**Design gap recorded, not yet built** — a live push producer, since a +harness that never completes a task cleanly (e.g. the stuck `wA` pane, see +below) currently under-counts its consumption forever: +- **Claude**: the statusline stdin JSON already carries real + `.rate_limits.five_hour.used_percentage` / `.seven_day.used_percentage` — + confirmed by reading `~/.claude/statusline.sh` on this machine. Prefer this + over `claude.ai/api/.../usage`, which needs browser session cookies, not + an API key. +- **Codex**: every `token_count` event in the rollout carries a `rate_limits` + object with `used_percent`/`window_minutes`/`resets_at` — confirmed + directly from a real rollout file. No `codex usage` subcommand; the + rollout tail (or `codex app-server`, same events live) is the only + transport. +- **opencode**: no first-class quota surface. `opencode stats` is historical + only. Zen's free-tier daily quota arrives as `x-ratelimit-*` response + headers, not a queryable endpoint — would need an opencode plugin to + intercept. +- Structural consequence if this is ever built: `sumSince`'s *additive* model + is wrong for a used-*percentage* feed (need a second `Availability` reading + the latest report per harness, not summing); static `quota_limit_5h/weekly` + become unnecessary for Claude/Codex once real fractions are reported; + `QuotaWindowLimits{FiveHour,Weekly}` is too narrow for Codex's + self-describing `window_minutes` or opencode's daily Zen window. + +### B8 — `Surface: system` unauthenticated bypass (§7.1) — closed + +`X-Orchestra-Surface: system` was reachable from any HTTP request header in +both `authz.HTTP` and `main.go`'s `surface` closure — since no deployment +sets `ORCHESTRA_SYSTEM_TOKEN`, this was an unauthenticated full-control +bypass reachable from any LAN caller. Both call sites now downgrade `system` +to `web` before doing anything else with it. `System` is only constructible +in-process (router leases/failures, coordinator releases/blocks, standup +advisory/apply) via the bus-level `authz.AuthorizeEvent` check now enforced +inside `store.Append` itself. + +### S1 — duplicate JSON struct tags — closed + +`Brief.From`/`To` and `GitSync.Branch`/`Head`/`Status` each shared one JSON +tag (Go only honors the first `json:"..."` tag on a combined field +declaration), making `go vet ./...` fail and the brief's git state +unparseable. Fixed; `go vet ./...` passes clean. + +### S2 + S3 — brief git state and completion receipts — closed + +`Brief.Git` used to read from `ORCHESTRA_DATA` (the event-log dir, never a +git checkout — always "git unavailable") and completions were counted but +never surfaced with proof. `operations.Brief.Git` is now `map[string]GitSync` +keyed by project ID, built from `registry.Project.Repo` (falling back to a +single `"default"` entry off `ORCHESTRA_REPO`), with `Ahead`/`Behind` vs +upstream added. `Brief.Receipts []CompletionReceipt` pulls `report_ref`/ +`receipt` straight out of each `TaskCompleted` event's existing payload. + +### S4 — notification goroutine died on first send error — closed + +`delivery.Fanout.Run` used to `return` on the first sender error, +permanently killing notifications for the rest of the process's lifetime. +Failed sends now go through an `OnError` hook instead of aborting the loop. +Cursor is persisted (`SaveCursor` → a `delivery-cursor` file next to +`ORCHESTRA_DATA`), so a restart resumes from the last delivered event +instead of re-notifying the entire log from seq 0. + +### S5 — colliding event IDs on lease — closed + +`Store.Lease`/`ExpireLeases` set `Event.ID` to the task id, so every lease of +a task produced colliding event IDs. Now uses `domain.NewID()`. + +### S6 — ingest dedup returned the wrong event with 200 — closed + +Dedup path returned `nil` (success) without appending; the HTTP handler then +returned an unrelated event with `201`. Added `domain.ErrDuplicate` and +`Store.TaskBySource`; `POST /v1/tasks` now returns the existing task with +`200` on a duplicate. Every other `Append` caller (Gitea poll/webhook, JSONL +ingest) now treats `ErrDuplicate` as expected. + +### S7 — `TaskAmended` dropped most fields — closed + +Only `title` was ever applied from an amendment payload; `due`/ +`description`/`inherent_priority` were accepted, logged, and silently +dropped by the projection. Also added the missing `Task.Description` field +entirely (it didn't exist, so `TaskCreated` dropped it too). Both +`TaskCreated` and `TaskAmended` now populate all four fields. + +### S8 — no compensating-event mechanism (§3.1) — closed + +Added `TaskCorrected`: payload requires `corrects` (the id of the event it +repairs) plus at least one change (`state`, or the existing amend-style +fields). `Store.Append` rejects a `corrects` that doesn't name a real prior +event on the same task; `apply` applies the changes and clears `Lease` like +every other terminal-state branch. Covered by a mistaken `TaskFailed` +reverted to `queued`, an unknown-`corrects` rejection, and both events +surviving snapshot+replay. + +### S9 — duplicate lease-expiry tickers — closed + +`Coordinator.Monitor`'s 30s ticker and `main.go`'s own 1s reclaim ticker both +called `Store.ExpireLeases` independently. Previously filed as "harmless — +CAS rejects the loser," but the real effect was worse: the coordinator's +`expire()` is the *only* place that kills the herdr session/pane for an +expired lease, and the 1s ticker running 30x more often almost always won +the race, leaving the coordinator's own call with nothing left to expire — +silently orphaning herdr panes past TTL. Fixed: `main.go`'s ticker skips +`ExpireLeases` entirely when a coordinator is configured (deferring reclaim +to the coordinator's loop), keeping only `AssignPending` as periodic retry. + +### S10 — federation worker registration had no admission control — closed + +Any caller could self-declare a worker `id` and self-choose a `token`, and a +second caller could silently re-register an existing worker id with a +*different* token, hijacking its identity/capacity. Added +`Registry.AdmitToken` (pre-shared secret via +`ORCHESTRA_FEDERATION_ADMIT_TOKEN`, checked against the registration +request's bearer header); same-ID re-registration now requires presenting +the existing worker's own token (legitimate restarts still succeed; +different-token same-id is rejected as a hijack). + +### S11 — soft threshold, milestone, thrash, agent-initiated ROTATE (§5.3) — closed + +All four of §5.3's rotation triggers now land: + +- **Soft threshold**: `Coordinator.Soft` (default 0.55, + `ORCHESTRA_OCCUPANCY_SOFT`) — both `rotate()` and `TurnDecision` request a + handoff once occupancy crosses `Soft`, before `Hard` forces one; + `TurnDecision` returns `prepare_handoff` advisory (task stays leased, no + turn boundary required). +- **Agent-initiated ROTATE**: `handoffReason(worktree)` reads a written + `.orchestra-handoff.json`; if `reason == "manual"`, both `rotate()` and + `TurnDecision` skip occupancy and the turn-boundary probe entirely and + release immediately — the agent's own handoff *is* the boundary signal. + Shared release-and-certify tail extracted into `Coordinator.finishRelease` + so this path gets the same anchor-safety guarantee as the threshold path. +- **Milestone + thrash**: needed a transcript/tool-call introspection source + this repo didn't have — built new, `internal/herdr/activity.go`: + - `ToolCall{Name, Kind, Key, Success, IsTest}` normalizes one tool/function + call across harnesses. + - `ClaudeActivity` pairs `tool_use`/`tool_result` blocks by `tool_use_id` + from the transcript already opened for occupancy (an unresolved + tool_use — mid-turn — is dropped, not reported). + - `CodexActivity` — **verified against a real live rollout, 2026-07-28**, + after the original guessed `function_call`/`function_call_output` shape + turned out not to exist anywhere in `~/.codex/sessions`. Real shape: + file edits arrive as `event_msg`/`patch_apply_end` (has `changes` + + top-level `success` directly, no pairing needed); shell commands arrive + as a freeform `response_item`/`custom_tool_call` named `"exec"` whose + `input` is a JS snippet — `codexExecCommand` regex-extracts the first + embedded `cmd:"..."`; failure is signaled by the literal output prefix + `"Script error:"` (confirmed for both a JS syntax error and an + `apply_patch` verification failure on this machine's real transcripts). + Manually re-ran the rewritten parser against a real multi-hundred-line + rollout end-to-end and spot-checked the output by eye, in addition to + `TestCodexActivityParsesRealRolloutShape` / + `TestCodexActivityMarksScriptErrorAsFailure`. + - `OpenCodeActivity` **refuses outright** — opencode's on-disk message + storage is only confirmed to carry aggregate token counts, not + per-tool-call records; fabricating a parser against an unconfirmed shape + would repeat the exact mistake this audit exists to catch. + - `DetectThrash(calls, ThrashConfig)` implements all three rules: N + consecutive failed test runs (regex-matched build/test commands), the + same file edited M times (edit tools only, explicitly excluding `Read` — + caught by an initially-failing test), and an identical tool call + repeated K times back-to-back (excluding test re-runs and file reads — + also caught by an initially-failing test). Defaults 3/5/4, overridable + via `Coordinator.Thrash`. + - `DetectMilestone(calls)` — deliberately narrow: only "the most recent + call was a successful `git commit`." Fuzzier definitions (passing test + suite, finished subtask) were not guessed at. + - `CLIAdapter.Activity` resolves the session file the same way `Occupancy` + does; `ActivityReader` is an optional Face-B capability like the others. + - New `ReasonedHandoffRequester`/`CLIAdapter.RequestHandoffReason` names + *why* (thrash's dead ends, milestone reasoning) instead of the generic + threshold framing. + - `rotate()`/`TurnDecision` generalize "reason=manual bypasses occupancy" + to `manual`/`milestone`/`thrash` alike: `checkActivityTriggers` runs + before falling through to occupancy/soft/hard (thrash takes priority + over milestone), and a hit calls `requestReasonedHandoff` — request + only, never release, same as the soft-threshold path. + +Covered by `internal/herdr/activity_test.go` and `internal/orchestrator/rotation_test.go`'s -`TestActivityTriggersRequestReasonedHandoffWithoutReleasing` (thrash and -milestone each request-without-releasing via `TurnDecision`; a -thrash-reasoned handoff already on disk bypasses occupancy and releases; -`rotate()`'s periodic path does the same request-without-release). +`TestActivityTriggersRequestReasonedHandoffWithoutReleasing`, +`TestTurnDecision` subtests for soft-threshold and manual-reason, and +`TestRotationRequestsHandoffBeforeReleasing`. -**Still open:** verifying `CodexActivity`'s parser shape against a live -rollout, and finding a real per-tool-call source for opencode (currently -refuses rather than guessing). +**Known remaining limitation:** `DetectMilestone`'s single-command +extraction only sees the *first* `cmd:"..."` in a chained Codex exec script, +so a `git commit` issued later in the same script isn't recognized as the +"last call" even though it executed last. Opencode still has no verified +per-tool-call source at all. -`go build ./...`, `go vet ./...`, `go test ./...` all pass. +### Also closed this pass (not separately numbered in the original audit) -## S11 — `CodexActivity` verified and rewritten against a live rollout, 2026-07-28 +- **Codex/opencode Stop-hook-equivalent scripts.** Neither harness has a + native Stop hook, so the server-side `/v1/harness/turn` and + `/v1/harness/complete` (harness dispatch) existed with no caller for + either. Added `deploy/hooks/orchestra-codex-poll.sh` and + `orchestra-opencode-poll.sh` — background poll loops (default 60s, + `ORCHESTRA_POLL_INTERVAL`) that find the newest session-state file (codex: + newest `rollout-*.jsonl`; opencode: newest file under + `~/.local/share/opencode/storage/message/`), POST to `/complete` when + `.orchestra-report.md` exists, otherwise POST to `/turn` and log (not act + on) `refuse`/`rotate_now` — advisory-only, since there's no real turn + boundary to refuse *at* from outside either harness process without deeper + app-server/SSE integration. +- **Bus-level authorization.** `authz.AuthorizeEvent` enforced inside + `store.Append` itself (the single choke point every event passes + through), not just at HTTP handlers. Event schema bumped to v2, requiring + every event to declare a `Surface`; schema v1 events on disk still replay + (tolerant reader). +- **Dual quota windows.** `router.QuotaAvailability` tracks a 5-hour rolling + window and a 7-day weekly window independently per harness, applying the + conservative 80% rule to each separately. +- **Turn-boundary detection made observable.** An adapter that fails to + answer `TurnBoundary` now blocks that tick's release rather than treating + the failure as "safe to proceed," and increments + `MonitorHealth.TurnBoundaryDegraded`. +- **Cross-machine lease correctness has a primitive-level test.** + `TestCrossMachineLeaseAnchorAndQuotaArePerHost` exercises a lease claimed + through the federation worker HTTP API, validates the anchor against that + worker's own local checkout, and asserts quota is accounted per-host — + proof for the primitives that exist today; does not yet run against two + real physical machines (see federation fork, below). +- **Fuzz coverage** (`internal/domain/fuzz_test.go`, + `FuzzValidatePayload`/`FuzzValidateEvent`) for all event types including + malformed nested payloads — no panic, always a typed error. +- **Multi-repo Gitea ingestion.** `provider.Gitea` gained a `Project` field + and namespaced `SourceName()`; `provider.MultiGitea` dispatches by + `task.Source`; `LoadGiteaConfigs` loads a JSON array of per-project + sources. Legacy single-repo env vars still work unchanged. Previously zero + tests existed for the Gitea provider at all (an earlier progress.md claim + of coverage was inaccurate) — `internal/provider/gitea_test.go` added. +- **Per-project repos.** `registry.Project` gained optional `repo`/ + `worktree_root`; `main.go` builds a `PerProjectGitWorktrees`, falling back + to the global default for any project that omits these fields. +- **Rotation's `TaskReleased` payload validity.** `Coordinator.rotate` used + to build `{"handoff_ref","reason"}`, omitting the `anchor_sha` the spec and + `domain.ValidatePayload` require whenever `handoff_ref` is present — + `store.Append` would reject it, the error was discarded + (`if c.Store.Append(e) == nil`), and rotation silently never happened, no + visible failure. Fixed: `herdr.HeadSHA(worktree)` populates `anchor_sha` + from the real worktree HEAD before appending; if the anchor can't be read, + rotation now skips that tick instead of emitting a payload guaranteed to + fail validation. The federation worker release endpoint + (`/v1/federation/workers/{id}/release`) had the identical gap and now + requires/forwards a 40-hex-char `anchor_sha`, rejecting with 400 + otherwise. -The guessed `function_call`/`function_call_output` shape did not exist in any -real rollout on this machine (`~/.codex/sessions`, checked directly, not -recalled). The real shape: +--- -- File edits arrive as `event_msg`/`patch_apply_end`, carrying `changes` - (map of absolute path → diff) and a top-level `success` bool directly — a - strictly better source than the old assumption, no call/result pairing - needed. -- Shell commands arrive as a single freeform `response_item`/ - `custom_tool_call` named `"exec"` whose `input` is a JS snippet - (`tools.exec_command({cmd:"...", ...})`), not a flat arguments object. - `codexExecCommand` regex-extracts the first embedded `cmd:"..."`. - Success/failure has no structured exit-code field either — but a failed - script's output block reliably starts with the literal string - `"Script error:"` on this machine's real transcripts (observed for both a - JS syntax error and an `apply_patch` verification failure), so that - heuristic is now confirmed, not guessed. +## The federation fork — decided 2026-07-27, still the deferred half -`internal/herdr/activity.go`'s `CodexActivity` rewritten to this shape; -`internal/herdr/activity_test.go` gained -`TestCodexActivityParsesRealRolloutShape` / -`TestCodexActivityMarksScriptErrorAsFailure`, fixtures built to match the -confirmed live shape, not an assumed one. Manually re-ran the new parser -against a real multi-hundred-line rollout file end-to-end (not just the unit -tests) and spot-checked the output — commands, file edits, and pass/fail all -matched the transcript by eye. +Two incompatible federation designs coexist in the tree. -**Still open:** `DetectMilestone`'s "last call was `git commit`" check only -sees the *first* `cmd:"..."` in a chained exec script, so a commit issued -after other commands in the same script call won't be seen as the "last -call" even though it was the last command executed — a known limitation of -single-command extraction, not fixed here. Opencode still has no verified -per-tool-call source. +**Design A — "drive the remote socket" (currently deployed, +`clients/herdr-bridge.go`)**: homesrv calls `worktree.create`/`agent.start`/ +etc. directly on workpc's herdr over TCP as if it were local — meaning +anchor validation (`git rev-parse HEAD`) executes on the *wrong machine* +relative to the actual checkout. Two outcomes if the coordinator's +`session.Worktree` path happens to also exist on homesrv (likely, since +every project shares the same directory layout): either `HeadSHA` errors and +rotation silently skips forever, or — the dangerous case — it returns +**homesrv's HEAD for an unrelated checkout**, passing validation while +certifying a commit the agent never touched. Same class of bug applies to +per-host quota accounting and to `cleanupCompleted`'s `git worktree remove`, +which runs on homesrv for a worktree that lives on workpc. -`go build ./...`, `go vet ./...`, `go test ./...` all pass. +**Design B — "workers pull tasks" (`/v1/federation/*`)**: fully built +server-side (registration, heartbeat/TTL offline detection, event-cursor +polling/ack, lease claim), zero clients — no worker binary exists anywhere +in this repo. This is what the spec actually describes (§2.1: "everything +crossing a machine boundary is git + a validated artifact, never live state +over the wire"), but every endpoint is currently unreachable in the real +deployment. Latent, never-surfaced defects in this unused half: offline +detection only runs inside `Snapshot()`, called solely from a GET endpoint — +nothing ticks it on its own, so `OnOffline` (the hook that releases leases +held by a vanished worker) only fires if a human hits that endpoint; the +registry is in-memory with no persistence, so a restart forgets all workers +and cursors. -## S8 — closed, 2026-07-27 +**Decision, unchanged:** keep Design A through Phase 5 (single-host concerns +— occupancy, Face B, rotation, continuity — are provable on homesrv alone +with workpc's herdr as just another pane host); commit to Design B in Phase +6. Two guardrails were meant to land immediately so Design A can't corrupt +state in the meantime: (1) refuse to rotate a lease held by a non-local +herdr rather than validate against the wrong checkout, since the protocol +schema can't run `rev-parse` where the checkout is; (2) same treatment for +`cleanupCompleted`'s worktree removal. **Status of these two guardrails is +unverified in this pass** — re-check `internal/orchestrator` before assuming +they landed; they are not confirmed closed above the way B1–B11/S1–S11 are. -No compensation-event mechanism existed — §3.1's own invariant ("a wrong -event is never edited; a compensating event is appended and replay sees -both") had nothing implementing it. `TaskAmended` was the closest analog but -only merges metadata fields forward with no reference to what it's -correcting and no way to touch `State`. +If the worker binary is ever abandoned, delete `/v1/federation/*` and record +the deviation — leaving both designs in place unmarked is explicitly not +acceptable; the repo would read as though the spec-conformant path is +implemented when nothing can reach it. -Added a new event type, `TaskCorrected`, generalizing that gap rather than -special-casing it: payload requires `corrects` (the `id` of the event being -repaired) plus at least one field to change — `state` (validated against the -same enum as `domain.TaskState`) and/or the existing amend-style fields -(`title`/`description`/`inherent_priority`/`due`). `domain.ValidatePayload` -checks shape; `Store.Append` checks that `corrects` actually names an event -belonging to the same task in the log (returning `ErrInvalid` otherwise) — -existence can only be checked where the log is visible, not in the -shape-only validator. `store.apply`'s new `TaskCorrected` branch clears -`Lease` whenever the corrected state isn't `leased`, matching every other -terminal-state branch. No new authz surface rule was needed — it slots into -the existing `TaskAmended`-shaped FullControl/GatedWrite policy unchanged. +--- -Covered by `TestTaskCorrected` (`internal/store/store_test.go`): a mistaken -`TaskFailed` is reverted to `queued` by an appended `TaskCorrected` -referencing it, a correction naming an unknown/foreign event is rejected, -and both the original wrong event and its correction survive a full -snapshot+replay reopen (the log is never edited, only appended to). `go -build ./...`, `go vet ./...`, `go test ./...` all pass. +## Live deployment facts (as of 2026-07-28) -### Design consequences (not yet implemented) +- Runs as `orchestra.service` on **homesrv** via this machine's own systemd. + `homesrv` has no local herdr running (connection refused on 9245) — only + `workpc`'s herdr (`192.168.1.105:9245`) is live and reachable. `main.go` + only logs herdr connection *failures* at startup, never successes, so "no + log line" for a herdr does not mean it's down. +- **Stuck task, deliberately left untouched:** workspace `wA`, task + `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode harness, pane `wA:p1`. From + Orchestra's own point of view it is no longer "stuck" — it now reads + `state: "failed"` (retries exhausted, `MaxAttempts` hit before B4's fix + landed). But the underlying herdr pane is still live and + `agent_status: "blocked"`, confirmed via a direct `agent.get` probe — the + router gave up and moved on, but nothing ever released or killed the + actual agent, confirming the orphaned-pane prediction from B2/B5. Left + untouched on purpose: user was asked and chose to leave it rather than + have it cleaned up mid-audit. Do not call `pane.close`/`pane.release_agent` + against it without asking again first. +- First live end-to-end task run (2026-07-28, after rebuilding/redeploying — + the running service had been on stale pre-audit commit `325c684`) is what + surfaced **B12** above, since fixed and confirmed live; re-verifying it + surfaced **B13**, still open. +- **Three more live orphaned panes from this pass, left untouched:** `wD:p1` + (task `06FTAKSJZTB73FZQE3QT7XQ1J0`, worktree + `/tmp/test-e2e-worktrees/06FTAKSJZTB73FZQE3QT7XQ1J0` — has a real attached + `claude` session but the task itself is stuck `blocked` from before B12's + fix landed), `wE:p1` and `wF:p1` (tasks `06FTAMFKVEPGPKR0H86C08ZZNG` / + `06FTANAYZPA55BABR2J4MWQ050` — empty shells, no agent ever attached, see + B13). Same rule as `wA:p1`: don't `pane.close`/`pane.release_agent` any of + these without asking first. -1. **Percentages are a level, not a delta.** - `router.QuotaAvailability.sumSince` *sums* `consumed` across - `QuotaReported` events. Feeding a used-percentage into that sum is wrong - by construction. A real-quota feed needs a second `Availability` - implementation that reads the **latest** report per harness. `Availability` - is already an interface (`internal/router/router.go`), so this is a swap, - not a rewrite — and the additive `sumSince` path must stay for the - estimate-based producer (spec §5.2.1 receipts are genuinely additive - across rotations). -2. **Static `quota_limit_5h`/`quota_limit_weekly` become unnecessary** for - Claude and Codex, since the harness reports its own fraction of pool and - the 80% conservative rule applies directly with no operator-entered cap. - Keep the static path as the fallback for opencode. -3. **`QuotaWindowLimits{FiveHour, Weekly}` is too narrow.** Codex's windows - are self-describing via `window_minutes`, and opencode's Zen tier is - **daily** — a window Orchestra has no concept of today. A generic - `[]{WindowMinutes, UsedPercent, ResetsAt}` fits all three; the current - two named fields fit only Claude. -4. **Push path**: statusline script (Claude) and a rollout-tail or - app-server reader (Codex) POST to a new endpoint alongside - `/v1/harness/turn` in `cmd/orchestra/main.go` — the existing hook-ingress - pattern — appending `QuotaReported`. That gives B7 a second, *live* - producer next to the post-hoc one, and covers the case CLAUDE.md already - flags: a harness that never completes a task cleanly (the stuck `wA` pane) - currently under-counts its consumption forever, because the only producer - fires on completion. +--- + +## Known open gaps (as of 2026-07-28) + +- **B13** (above) — `agent.start` can silently no-op under back-to-back + leases, with no error surfaced; blocks reliable live verification of + everything downstream (occupancy, rotation, handoff, completion) against a + real task, even though B12's readiness race is now fixed. +- **Cross-machine lease correctness proven at the primitive level only** — + needs an actual homesrv/workpc pair over the real mesh; this repo cannot + exercise that by itself. +- **Turn-boundary Face B degrades to occupancy-only** for adapters that + don't implement it, by design — degradation is observable + (`MonitorHealth.TurnBoundaryDegraded`) and blocks-on-failure, but whether + Claude/Codex/opencode's native hooks are wired in a live deployment is a + deployment-config fact, not provable from source alone. +- **Quota has no live push producer** — only the post-hoc + `/v1/harness/complete` producer exists; see B7's "design consequences" for + the (unbuilt) Claude statusline / Codex rollout-tail feeds. +- **Federation guardrails' landed-status is unverified** — see federation + fork section above. +- **opencode has no verified per-tool-call activity source** and no + first-class quota surface (Zen headers only, would need a plugin) — + both named, not attempted, in S11/B7. +- **`DetectMilestone`'s single-command extraction** can miss a Codex commit + issued later in a chained exec script. + +Everything else audited (provider layer, continuity schema/validation, +router matching, delivery, federation registration primitives) was +spot-checked against the code and its tests and matched described behavior. diff --git a/internal/herdr/herdr.go b/internal/herdr/herdr.go index d63f012..ee93975 100644 --- a/internal/herdr/herdr.go +++ b/internal/herdr/herdr.go @@ -165,12 +165,39 @@ type Session struct { ConventionsHash string `json:"conventions_hash,omitempty"` } +// bootDeadline bounds the retry loops below. Freshly created panes/agents +// have been observed (live, 2026-07-28) to reject the very next call for a +// range of different transient reasons as herdr finishes bringing them up — +// "not an available shell", "not an active named agent", "target ... not +// found" — a new wording each time a prior one got fixed. There is no other +// legitimate reason a call against state orchestra itself just created would +// fail immediately, so these loops retry any error rather than pattern-match +// an open-ended and apparently still-growing set of herdr wordings, bounded +// by wall-clock time rather than attempt count so a slow-booting pane still +// gets the same real budget as a fast-failing one. +const ( + bootRetryWindow = 15 * time.Second + bootRetryDelay = 500 * time.Millisecond +) + func (c *Client) Prompt(ctx context.Context, pane, text string, wait time.Duration) error { p := map[string]any{"target": pane, "text": text} if wait > 0 { p["wait"] = map[string]any{"until": []string{"idle"}, "timeout_ms": wait.Milliseconds()} } - return c.Call(ctx, "agent.prompt", p, nil) + deadline := time.Now().Add(bootRetryWindow) + var err error + for { + err = c.Call(ctx, "agent.prompt", p, nil) + if err == nil || time.Now().After(deadline) { + return err + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-time.After(bootRetryDelay): + } + } } func (c *Client) Worktree(ctx context.Context, cwd, path, branch string) (string, error) { var r worktreeResponse @@ -204,15 +231,34 @@ func (c *Client) StartAgent(ctx context.Context, cwd, path, branch, harness, tas return Session{}, fmt.Errorf("herdr: no pane recorded for worktree %s", path) } var s Session - if err := c.Call(ctx, "agent.start", map[string]any{ - "pane_id": paneID, - "kind": harness, - "name": harness, - "args": []string{}, - }, &s); err != nil { - if !strings.Contains(strings.ToLower(err.Error()), "already") { + deadline := time.Now().Add(bootRetryWindow) + var err error + for { + s = Session{} + err = c.Call(ctx, "agent.start", map[string]any{ + "pane_id": paneID, + "kind": harness, + "name": harness, + "args": []string{}, + }, &s) + if err == nil { + break + } + if strings.Contains(strings.ToLower(err.Error()), "already") { + err = nil + break + } + if time.Now().After(deadline) { return Session{}, err } + select { + case <-ctx.Done(): + return Session{}, ctx.Err() + case <-time.After(bootRetryDelay): + } + } + if err != nil { + return Session{}, err } s.PaneID = paneID s.Worktree = path diff --git a/progress.md b/progress.md deleted file mode 100644 index 224df06..0000000 --- a/progress.md +++ /dev/null @@ -1,628 +0,0 @@ -# Orchestra progress - -Updated: 2026-07-27 - -## AUDIT.md remediation — in progress - -Working through `AUDIT.md`'s blocking/secondary defects in order of the -"suggested order of attack." Each item below is landed, tested, and -committed individually; see the git log for the exact commits. - -Fixed so far: -- **B2** — adapters were looked up by `session.Harness` (the harness kind, - e.g. `"claude"`) in `Reconcile`/`expire`/`rotate`, but `AdapterFactory.Herdrs` - is keyed by herdr instance id (e.g. `"homesrv-claude"`). Every one of those - call sites silently no-opped. Added `Coordinator.adapterFor`, routed all - four call sites through it. Regression test registers an adapter under a - herdr-id key distinct from the harness kind and asserts rotation fires. -- **B1** — `CLIAdapter.Occupancy` called `a.Usage(s.PaneID)`, but the usage - readers want a filesystem path to session state, not a herdr pane id. - Added `herdr.Session.SessionFile` and per-harness resolution - (`ClaudeSessionFile` by newest-mtime under Claude Code's own project - directory; codex via the existing `CodexActiveUsage` sqlite discovery; - opencode refuses loudly — needs a live session id, not resolvable from the - worktree alone). A missing/unreadable session file is now a hard error, - surfaced via new `SessionHealth.Occupancy`/`OccupancyError` fields on - `GET /v1/tasks/{id}/health`, not a silent zero. **Still needs live - verification against a real Claude Code session** (the spec's own - acceptance bar for this phase) — not possible from this sandbox. -- **B4** — the router counted every `TaskReleased` (including rotation, - which *is* a `TaskReleased` carrying a valid `handoff_ref`) against - `MaxAttempts`, and double-counted by also incrementing on every - subsequent lease. A task that rotated twice hit the default - `MaxAttempts=3` and was killed. Now only a release without a - `handoff_ref` (expiry/crash) advances the counter. -- **B8** — `X-Orchestra-Surface: system` was reachable from an HTTP request - header in both `authz.HTTP` and `main.go`'s `surface` closure (the one - every handler actually calls). Since no deployment sets - `ORCHESTRA_SYSTEM_TOKEN`, this was an unauthenticated full-control bypass - reachable from any LAN caller. Both call sites now downgrade `system` to - `web` before doing anything else with it. -- **S1** — `Brief.From`/`To` and `GitSync.Branch`/`Head`/`Status` all shared - one JSON tag each (Go only honors the first `json:"..."` tag on a - combined field declaration). `go vet ./...` now passes clean. -- **S5** — `Store.Lease`/`ExpireLeases` set `Event.ID` to the task id, so - every lease of a task produced colliding event IDs. Now `domain.NewID()`. -- **S6** — the ingest dedup path returned `nil` (success) without - appending; `main.go` then returned an unrelated event with `201`. Added - `domain.ErrDuplicate` and `Store.TaskBySource`; `POST /v1/tasks` now - returns the existing task with `200` on a duplicate. Updated every other - `Append` caller (Gitea poll/webhook, JSONL ingest) to treat - `ErrDuplicate` as expected rather than a failure — without that, Gitea - polling would error out of its scan loop on the first already-ingested - issue in every batch. - -- **B3 (partial)** — added `POST /v1/harness/complete`, the first automatic - `TaskCompleted` producer (previously only a human calling - `/v1/tasks/{id}/complete` could ever complete a task). A Claude Code Stop - hook (`deploy/hooks/orchestra-stop.sh`) fires on every turn boundary but - only reports completion if the agent has written a `.orchestra-report.md` - marker at the worktree root first — an ordinary turn boundary is a no-op, - so this doesn't fire completion prematurely. The server reads the - transcript locally via `herdr.ClaudeUsage` to build the `receipt` itself - (input/cache/output token counts) rather than trusting a self-reported - number, and uploads the report body to CAS for `report_ref`. Guarded by an - optional `ORCHESTRA_HARNESS_TOKEN` bearer check; the event is appended with - `Surface: system` set directly in Go (not derived from a request header — - consistent with the B8 fix that system must never be header-controlled). - **Not done:** Codex/opencode equivalents (Claude-only for now — Codex would - need `CodexActiveUsage`, opencode `OpenCodeUsage`/`OpenCodeStatus`, wired - the same way), and the turn-boundary decision endpoint - (`continue`/`prepare_handoff`/`rotate_now`/`refuse`) from Phase 2 items 1–2 - is still unbuilt — only the completion half of Phase 2 landed. No test - added for the new HTTP handler; `cmd/orchestra/main.go` has zero test - coverage for any handler (pre-existing gap, everything lives inline in - `main()`) so this follows the existing (untested) pattern rather than - introducing a one-off test harness. - -- **B5 (loose end)** — `CLIAdapter.Lease`'s initial prompt used `wait=0`, - skipping the inline wait `Bootstrap` already used; the spec (§5.1) requires - inline `wait` on `agent.prompt` for bootstrap injection to avoid sending - into a half-rendered prompt. Changed to `time.Minute`, matching `Bootstrap`. - Small, contained fix — `Release`'s real implementation (needs Phase 4 - handoff production) is still outstanding from B5. - -- **B6 (partial — Phase 4 items 1 and 4)** — nothing wrote a `TASK.md` into a - worktree, so pickup validation had nothing to check and never ran anyway. - Fixed both halves: `GitWorktrees.Create` now writes and commits an - immutable `TASK.md` (`continuity.RenderTaskFile`) into every freshly - created worktree, and `Coordinator.Start` now runs - `continuity.ValidatePickup` (loading the handoff from CAS, checking anchor - SHA + dirty-file hashes + TASK.md hash) before bootstrapping a successor - onto a `handoff_ref` — a failure kills the session and emits `TaskBlocked` - instead of trusting an unvalidated ref. Covered by - `TestGitWorktreesCommitsTaskFile` and `TestStartBlocksOnInvalidPickup` in - `internal/orchestrator`. **Not done:** handoff *production* (nothing yet - writes a real §6.1 handoff — `Release` still refuses per B5), wiring - `ScratchCommit` before release, and the §6.2 bootstrap-prompt rewrite. See - AUDIT.md's "B6 — partial fix" section for the full breakdown, including a - named caveat: TASK.md hashing is best-effort and untested for the - herdr-hosted (`WorktreeCreator`) worktree path. - -- **B5 (closed)** — `CLIAdapter.Release` previously just refused (no real - herdr method existed to call and there was nothing to validate against). - Now: reads the agent-authored `.orchestra-handoff.json` from the worktree - root, validates it with `continuity.Decode`, cross-checks its anchor SHA - against the worktree's real `HeadSHA` (never trusts the agent's self-report - outright), uploads it to CAS via `continuity.Save` to mint the - `handoff_ref`, and only then calls the real `pane.release_agent({pane_id, - source, agent})` to drop herdr's claim — sequenced last so a herdr-side - error can't strand an uploaded handoff. Any failure (missing file, invalid - schema, anchor mismatch, herdr error) is a refusal, which `rotate` already - treats as "retry next tick" rather than stranding the task. `herdr.Claude/ - Codex/OpenCode` now take a `continuity.CAS` (main.go passes the existing - `*store.Store`). New tests in `internal/herdr/adapter_test.go` cover all - four paths against a real git worktree and a fake in-process herdr - listener. **Not done:** nothing yet makes the agent actually *write* - `.orchestra-handoff.json` (needs a stop-hook convention analogous to - `.orchestra-report.md`) — that and the rest of Phase 4 (ScratchCommit - before release, §6.2 bootstrap-prompt rewrite, `MarkdownChanges`) remain - open. - -- **Phase 4 items 3, 5, 6** — `CLIAdapter.Release` now re-verifies every - `Anchor.Dirty` file hash (previously only the top-level `Anchor.GitSHA` - was checked; a file edited after the handoff was written but before - release would have gone through unnoticed), then, if there were dirty - entries, snapshots them atomically onto a per-task scratch branch - (`continuity.ScratchCommit`, made idempotent so a task can rotate more - than once) and rewrites the handoff's anchor to that new commit with - `Dirty` cleared before uploading — so the successor's pickup check is a - single HEAD compare, not N file rehashes. `CLIAdapter.Bootstrap`'s prompt - was rewritten to point the agent at `git log`/the scratch branch instead - of a vague "read the handoff" instruction, and deliberately avoids - claiming a `GET /v1/artifacts/` endpoint, since no such route exists - (`/v1/artifacts` is POST-only). `continuity.MarkdownChanges` (§6.3 - adjacent-task notice) had zero callers and zero tests despite being - listed as implemented in an earlier snapshot — deleted rather than - half-wired, per AUDIT.md's explicit "delete and record the deviation" - option. New tests: `TestReleaseScratchCommitsDirtyFilesBeforeUpload`, - `TestReleaseRefusesOnStaleDirtyFile` (internal/herdr/adapter_test.go). - **§6.3 rewired for real, 2026-07-27 (later same day):** the deleted - `MarkdownChanges` above was zero-caller dead code, but the underlying spec - requirement ("on update, the orchestra injects a notice to agents whose - current task is adjacent") wasn't abandoned — rebuilt independently. - `continuity.ConventionsHash(root)` hashes whichever of - `AGENTS.md`/`CLAUDE.md`/`VOCAB.md` exist at a path; `herdr.Session` gained - `ConventionsHash`, snapshotted from the fresh worktree at - `Coordinator.Start`; a new `Coordinator.checkConventions`, run every - `Monitor` tick, recomputes the hash of the project's *base repo* (via - `WorktreeSpec.Spec` — "adjacent" = same project) for every leased session - and compares it against that session's stored snapshot. A mismatch calls a - new optional `herdr.ConventionsNotifier` capability - (`CLIAdapter.NotifyConventionsChanged`, an in-pane `agent.prompt` telling - the agent to re-read the docs) and updates the stored hash so the notice - fires once per drift, not every tick. Covered by - `TestConventionsDriftNotifiesActiveSession` - (internal/orchestrator/rotation_test.go): asserts no notification while - the base repo is unchanged, then one once it diverges. - **Was still open:** Phase 4 item 2 — nothing drove *any* harness to write - `.orchestra-handoff.json`, since Release only validated a file whose - existence was never solicited. **Closed 2026-07-27:** `rotate()` now checks - for the adapter's optional `herdr.HandoffRequester` capability; when - `HandoffFile` is missing at the worktree root, it prompts the agent once - (`CLIAdapter.RequestHandoff`, mirroring the `.orchestra-report.md`/B3 - convention — the plane asks for a handoff, it never invents one) and skips - Release that tick, retrying every subsequent tick until the file appears. - `herdr.Session.HandoffRequested` avoids re-prompting every tick. Covered by - `TestRotationRequestsHandoffBeforeReleasing` - (`internal/orchestrator/rotation_test.go`), which asserts Release is never - called before the file exists and fires once it does. Codex/opencode still - share this same path (no harness-specific gap remains); the only leftover - question is whether each harness's own Stop-equivalent hook honors the - in-pane prompt to write the file before exiting, which is a live-deployment - fact, not something provable from source. - -- **B7 (post-hoc producer) + Phase 2 turn-decision endpoint** — landed - together, since both are new `QuotaReported`/turn-boundary paths off the - same completion/turn events. `POST /v1/harness/complete` now appends a - `QuotaReported` event (`harness_id` from the closing lease, `consumed` - from the same `usage.Numerator()` used for the receipt), so the router's - 5h/weekly availability filter and the brief's `quota_consumed` stop - evaluating against a permanent zero. New `Coordinator.TurnDecision` - (`internal/orchestrator/orchestrator.go`) mirrors `rotate()`'s per-task - logic (occupancy → turn-boundary → handoff-file → release) but runs - synchronously once per turn instead of waiting for `Monitor`'s ticker, - returning one of `continue`/`prepare_handoff`/`rotate_now`/`refuse` via the - new `POST /v1/harness/turn`. The Claude Stop hook - (`deploy/hooks/orchestra-stop.sh`) now calls this endpoint on every - ordinary turn boundary (report marker absent) instead of no-op'ing, and - exits 2 on `refuse` to stop the harness from finishing an unsafe turn. - Covered by `TestTurnDecision` (`internal/orchestrator/rotation_test.go`): - continue-below-threshold, refuse-when-not-at-boundary, and - rotate_now-releases-and-emits-a-valid-TaskReleased cases. - **Not done:** live per-harness *push* producers (Claude statusline, - Codex rollout tail) that would give B7 a second, continuous producer - independent of task completion — recorded as a design investigation in - AUDIT.md ("Real harness quota sources") but not implemented; Codex/ - opencode's own equivalents of the Claude Stop hook (whether their - turn-boundary mechanism actually calls `/v1/harness/turn`) also remain - unbuilt, same caveat as Phase 2 item 4 already named for `/complete`. - -- **S4** — `delivery.Fanout.Run` used to `return` on the first sender error, - permanently killing the notification goroutine (a single ntfy hiccup meant - no notifications for the rest of the process's lifetime, since nothing - restarts it). Failed sends now go through an `OnError` hook instead of - aborting the loop. Cursor is also persisted now (`SaveCursor` → a - `delivery-cursor` file next to `ORCHESTRA_DATA`, loaded on startup), so a - restart resumes from the last delivered event instead of re-notifying the - entire log from seq 0. `internal/delivery` previously had zero tests; - added `TestFanoutContinuesAfterSendError`. - -- **S2 + S3** — `/v1/brief`'s git state used to come from `ORCHESTRA_DATA` - (the event-log directory, not a git checkout — always reported - `"git unavailable"`), and completions were only counted, never surfaced - with proof. `operations.Brief.Git` is now `map[string]GitSync` keyed by - project ID, built in `main.go` from each `registry.Project.Repo` (falling - back to a single `"default"` entry off `ORCHESTRA_REPO` for deployments - without per-project repos); `GitSync` gained `Ahead`/`Behind` vs upstream. - New `Brief.Receipts []operations.CompletionReceipt` pulls `report_ref`/ - `receipt` straight out of each `TaskCompleted` event's existing payload. - Covered by an updated `TestBuildBrief`. - -- **S7** — `TaskAmended` only ever applied `title` from the amendment - payload; `due`/`description`/`inherent_priority` amendments were accepted - and logged but silently dropped by the projection. Also added the missing - `Task.Description` field (it didn't exist at all, so `TaskCreated` dropped - it too). Both `TaskCreated` and `TaskAmended` now populate all four fields. - Covered by `TestTaskAmendedAppliesAllFields` (`internal/store`). - -- **Codex/opencode completion producers** — `/v1/harness/complete` was - Claude-only (hardcoded `herdr.ClaudeUsage`). Added an optional `harness` - field to the request body (`""`/`"claude"` unchanged default); `"codex"` - dispatches to `herdr.CodexUsage`, `"opencode"` to `herdr.OpenCodeUsage`, - anything else is a 400. Both readers already existed and were tested in - `internal/herdr` but had zero callers outside tests — same "written, not - wired" pattern the rest of this audit keeps finding. The Claude stop hook - (`deploy/hooks/orchestra-stop.sh`) now sends `"harness":"claude"` - explicitly for symmetry, no behavior change. **Not done:** an actual - Codex/opencode Stop-hook-equivalent script — Codex has no native stop hook - (would need a rollout-tail poller deciding when to call this), and - opencode's own turn-boundary mechanism is unverified from source (same - caveat AUDIT.md already names for Phase 2 item 4). This change unblocks - the server side; the harness-side wrapper for either is still open. No - new test: `cmd/orchestra` has zero handler test coverage of any kind - (pre-existing, confirmed by grep before writing this), so this follows the - existing pattern rather than introducing a one-off test harness for one - handler. - -- **S9** — the coordinator's `Monitor` loop (30s ticker, `Coordinator.expire`) - and `main.go`'s own 1s reclaim ticker both called `Store.ExpireLeases` - independently. AUDIT.md filed this as "harmless — CAS rejects the loser," - but the real effect was worse: the coordinator's `expire()` is the *only* - place that kills the herdr session/pane for an expired lease, and since the - 1s ticker ran 30x more often it almost always expired the lease first, - leaving the coordinator's own `ExpireLeases` call with nothing left to - expire — so its pane-kill path silently never ran, orphaning herdr panes - past their TTL whenever a coordinator was configured. Fixed by having - `main.go`'s ticker skip `ExpireLeases` entirely when `coordinator != nil` - and defer reclaim to the coordinator's loop, keeping only `AssignPending` - as a periodic retry. No coordinator (e.g. no herdrs configured) still uses - the direct `ExpireLeases` path, since nothing else would reclaim leases in - that case. - -- **S10** — `federation.Registry.Register` accepted a self-declared `id` and - self-chosen `token` from any caller with no admission control, and - silently let a second caller re-register an existing worker id with a - *different* token, hijacking that worker's identity/capacity out from - under it. Added `Registry.AdmitToken` (a pre-shared secret, wired from - `ORCHESTRA_FEDERATION_ADMIT_TOKEN`, checked against the registration - request's `Authorization: Bearer` header in `main.go`) and a same-ID - re-registration now requires presenting the existing worker's own token. - Covered by `TestRegisterRequiresAdmitTokenAndOwnToken` - (`internal/federation/federation_test.go`): wrong admit token rejected, - correct admit token accepted, same-id-different-token rejected as a - hijack, same-id-same-token (legitimate restart) still succeeds. - -- **S11 (partial — soft threshold only)** — added `Coordinator.Soft` - (default 0.55, `ORCHESTRA_OCCUPANCY_SOFT` override). Both `rotate()` and - `TurnDecision` now request a handoff once occupancy crosses `Soft`, well - before `Hard` forces one; `TurnDecision` returns `prepare_handoff` for this - advisory case without requiring a turn boundary, leaving the task leased. - Covered by a new `TestTurnDecision` subtest. **Not done:** milestone - rotation, thrash detection, agent-initiated `ROTATE` — see AUDIT.md's S11 - section for why those are bigger than a threshold check. - -- **S11 (agent-initiated ROTATE)** — the schema already accepted - `meta.reason: "manual"` (since B5) but nothing checked for it. New - `handoffReason(worktree)` reads a written `.orchestra-handoff.json` and, if - its reason is `"manual"`, both `rotate()` and `TurnDecision` skip occupancy - and the turn-boundary probe entirely and release immediately — the agent's - own handoff *is* the boundary signal per §5.3 ("a coherent unit finished and - the next is independent"). Extracted the shared release-and-certify tail - into `Coordinator.finishRelease` so the manual path reaches the same - anchor-safety guarantee as the threshold path. Covered by a new - `TestTurnDecision` subtest (occupancy=0, boundary=false — every other path - would refuse or continue — asserting rotation happens anyway once a - `reason=manual` handoff exists). - -- **S8** — no compensation-event mechanism existed for §3.1's own invariant - ("a wrong event is never edited; a compensating event is appended and - replay sees both"). Added `TaskCorrected`: payload requires `corrects` - (the id of the event it repairs) plus at least one change (`state`, or the - existing amend-style metadata fields). `Store.Append` rejects a `corrects` - that doesn't name a real prior event on the same task; `store.apply` - applies the state/field changes and clears `Lease` like every other - terminal-state branch. Covered by `TestTaskCorrected` - (`internal/store/store_test.go`): a mistaken `TaskFailed` reverted to - `queued`, an unknown-`corrects` rejection, and both events surviving a - snapshot+replay reopen. - -- **Codex/opencode Stop-hook-equivalent scripts** — the server side - (`/v1/harness/turn`, `/v1/harness/complete` dispatching by `harness`) has - existed since the turn-decision endpoint and the codex/opencode dispatch - commit, but nothing called it for either harness: both lack a native Stop - hook, so `orchestra-stop.sh`'s per-turn-boundary call never had an - equivalent. Added `deploy/hooks/orchestra-codex-poll.sh` and - `deploy/hooks/orchestra-opencode-poll.sh` — background poll loops (default - 60s, `ORCHESTRA_POLL_INTERVAL`) meant to run alongside the harness process - in its pane. Each tick: find the newest session-state file (codex: newest - `rollout-*.jsonl` under `~/.codex/sessions`, matching `CodexActiveUsage`'s - own "most recently touched" heuristic; opencode: newest file under - `~/.local/share/opencode/storage/message/`, the same backstop - `OpenCodeUsage` reads, since a session id for the SSE fast path isn't - reliably available outside the opencode process itself); if - `.orchestra-report.md` exists, POST it plus the discovered path to - `/v1/harness/complete` with the right `harness` value and remove the - marker; otherwise POST task id to `/v1/harness/turn` and log (not act on) - `refuse`/`rotate_now` — unlike the Claude Stop hook's `exit 2`, there's no - turn boundary to refuse *at* from outside the harness process for either - of these, so this is advisory-only until real app-server/SSE integration - exists. No Go changes; nothing new to build/vet/test. - -- **S11 (milestone + thrash detection) — closed, 2026-07-28.** The last two - of S11's four rotation triggers, previously deferred as needing "transcript/ - tool-call introspection this repo doesn't have a source for" — that source - now exists. New `internal/herdr/activity.go`: - - `ToolCall{Name, Kind, Key, Success, IsTest}` normalizes one tool/function - call across harnesses (`Kind` is `"file"` or `"command"`, `Key` the path - or shell command — the same two field names, `file_path`/`command`, both - Claude's `tool_use.input` and Codex's function-call `arguments` use). - - `ClaudeActivity` parses the same transcript `ClaudeUsage`/`ClaudeSessionFile` - already open, pairing `tool_use`/`tool_result` blocks by `tool_use_id` - (an unresolved tool_use — mid-turn — is dropped, not reported). - - `CodexActivity` parses the rollout's `function_call`/`function_call_output` - payload pairs, mirroring `CodexUsage`'s existing `payload.type` wrapper — - **explicitly marked best-effort/unverified** in its doc comment, same bar - AUDIT.md's Phase 0 set for herdr methods: this hasn't been checked against - a live rollout, only against `CodexUsage`'s already-confirmed shape. - - `OpenCodeActivity` **refuses outright** rather than guess: opencode's - on-disk message storage is only confirmed to carry aggregate token counts - (what `OpenCodeUsage` already reads), not per-tool-call records, so - fabricating a parser against an unconfirmed shape would repeat the exact - mistake this repo's own audit exists to catch. - - `DetectThrash(calls, ThrashConfig)` implements all three §5.3 rules — N - consecutive failed test runs (regex-matched shell commands: go/pytest/npm/ - yarn/cargo/make/jest/mvn/rspec/ctest), the same file edited M times - (`Edit`/`Write`/`MultiEdit`/`NotebookEdit`, explicitly *not* `Read` — - caught by a test that initially failed because rule 3 didn't exclude - reads), and the identical tool call repeated K times back-to-back - (explicitly excluding test re-runs and file reads, since re-running the - same test after a fix attempt is expected behavior, not thrashing — caught - by another initially-failing test). Each rule that trips populates a - `continuity.DeadEnd`, handed straight to the new prompt below rather than - left for the agent to invent. Defaults: 3/5/4, overridable via - `Coordinator.Thrash`. - - `DetectMilestone(calls)` — deliberately narrow: only "the most recent - call was a successful `git commit`". Spec-fuzzier definitions (a passing - test suite, a finished subtask) were **not** guessed at; same restraint - the rest of this audit has applied to unverified behavior. - - `CLIAdapter.Activity` (adapter.go) resolves the session file the same way - `Occupancy` does and dispatches to the right parser; `ActivityReader` is - an optional capability like every other Face-B interface in this package. - - New `ReasonedHandoffRequester`/`CLIAdapter.RequestHandoffReason` — like - `RequestHandoff` but names *why* (thrash's specific dead ends, or the - milestone reasoning) instead of the generic "context budget reached" - framing, and asks the agent to write `meta.reason` accordingly. - - `Coordinator.rotate()` and `TurnDecision()`: the existing "reason=manual - bypasses occupancy" shortcut generalized to `manual`/`milestone`/`thrash` - alike — once a handoff exists carrying one of these reasons, that **is** - the boundary signal, same treatment as agent-initiated ROTATE already - got. Before falling through to occupancy/soft/hard, both now call new - `checkActivityTriggers` (adapter implements `ActivityReader`? read - calls, thrash takes priority over milestone) and, on a hit, - `requestReasonedHandoff` (same `HandoffRequested`-guarded ask-once pattern - the occupancy path already uses) — request only, never release, exactly - like the soft-threshold path's `prepare_handoff`. - - Tests: `internal/herdr/activity_test.go` (parser + all three detector - rules, including the two cases that caught real bugs above) and - `internal/orchestrator/rotation_test.go`'s new - `TestActivityTriggersRequestReasonedHandoffWithoutReleasing` (thrash and - milestone each request-without-releasing via `TurnDecision`, a - thrash-reasoned handoff already on disk bypasses occupancy and releases, - and `rotate()`'s periodic path does the same request-without-release). - - Codex/opencode Stop-hook-equivalent poll scripts - (`deploy/hooks/orchestra-codex-poll.sh`, - `orchestra-opencode-poll.sh`, added earlier this session) already call - `/v1/harness/turn`, so `TurnDecision`'s new thrash/milestone checks reach - those harnesses for free wherever `Activity` is implemented (Codex, once - its unverified parser is checked; opencode not until a real per-tool-call - source is found). - -- **S11 (CodexActivity verified against a live rollout) — closed, 2026-07-28.** - The previously-unverified `function_call`/`function_call_output` shape - turned out not to exist in any real Codex rollout on this machine. Rewrote - `CodexActivity` against the real shape found in `~/.codex/sessions`: - `event_msg`/`patch_apply_end` for file edits (has `changes`+`success` - directly, no pairing needed), and `response_item`/`custom_tool_call` named - `"exec"` (a freeform JS-scripted tool, not flat arguments — commands are - extracted from an embedded `cmd:"..."` via regex) paired with - `custom_tool_call_output`, whose failure signal is the observed literal - prefix `"Script error:"` in the output text. New tests - (`TestCodexActivityParsesRealRolloutShape`, - `TestCodexActivityMarksScriptErrorAsFailure`) use fixtures built from the - confirmed shape; also manually re-ran the parser against a real multi- - hundred-line rollout file and spot-checked the output by eye. See - AUDIT.md's S11 section for the full writeup and the one known remaining - limitation (only the first command in a chained exec script is extracted, - so `DetectMilestone` can miss a commit that isn't the first call in its - script). - -- **Live status check, 2026-07-28** — the stuck task named throughout this - file (`06FT6CKD9Y98AZRX6X8K3QXFZG`) is no longer "stuck" from Orchestra's - own point of view: it now reads `state: "failed"` (retries exhausted, - `MaxAttempts` hit it). But the underlying herdr pane (`wA:p1`, opencode) - is still live and `agent_status: "blocked"` — confirmed via a direct - `agent.get` probe against `192.168.1.105:9245` — meaning the orphaned-pane - prediction in AUDIT.md's B2/B5 sections was correct: the router gave up - and moved on, but nothing ever released or killed the actual agent. Left - untouched deliberately (no `pane.close`/`release_agent` call made) — user - was asked and chose to leave it for now rather than have it cleaned up in - this session. - -Not yet started: live verification of Phase 1 occupancy against a real -session, and the two-machine federation run. See `AUDIT.md` for the full -plan. - -**Phase 0 done (2026-07-27):** this box has live TCP reachability to the real -herdr instance at `192.168.1.105:9245` — verified by hand (raw JSON-RPC -probes, no `herdr` CLI available locally). Real method list captured in -`deploy/herdr-schema.json`. Confirmed `pane.release`/`pane.kill`/ -`pane.rotation_signal` are invented, as AUDIT.md's B5 suspected. -`pane.kill`→`pane.close` fixed as a drop-in. `pane.rotation_signal`/ -`RotationSignal` deleted (no replacement exists). `Release` now refuses -loudly instead of calling a nonexistent method — its real implementation -needs Phase 4 (handoff production) first, since even the real -`pane.release_agent` can't return a `handoff_ref` (herdr doesn't write -handoffs, the agent does). See AUDIT.md's new "Phase 0 — done" section for -full detail. **Also found: a real task is currently stuck live** — workspace -`wA`, task `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode, pane `wA:p1`, blocked — -deliberately not touched from this session. - -## Current state - -This is a working Go implementation of `orchestra-spec (1).md`'s Layer 1–3 -(substrate, harness/rotation, continuity) plus a first cut of Layer 4 -(surfaces). `go build ./...` and `go test ./...` both pass. The codebase is -small (~4.6k lines across `internal/{domain,store,provider,registry,router, -herdr,orchestrator,continuity,federation,delivery,authz,operations,admin}` -and `cmd/orchestra/main.go`). - -Earlier revisions of this file accumulated a long, self-contradictory -chronological log — gaps were listed as open in one section and then claimed -closed in a later section, sometimes inaccurately. This revision replaces -that log with one audited snapshot. Treat prior git history of this file as -session notes, not as ground truth. - -### Verified fixed this pass - -- **Rotation emitted an invalid `TaskReleased` (the previously reported - highest-priority defect) — now fixed.** `internal/orchestrator.Coordinator.rotate` - built the release payload as `{"handoff_ref","reason"}`, omitting the - `anchor_sha` the spec (§4, §6.2) and `domain.ValidatePayload` require - whenever `handoff_ref` is present. `store.Append` would reject it, the - error was discarded (`if c.Store.Append(e) == nil`), and the lease/session - silently never rotated — the coordinator would just retry next tick with - no visible failure. Fixed by adding `herdr.HeadSHA(worktree)` and having - `rotate` populate `anchor_sha` from the real worktree HEAD before - appending; if the anchor can't be read, rotation now correctly skips that - tick (leaving the lease intact for TTL/next-tick reclaim) instead of - emitting a payload guaranteed to fail validation. - Covered by `internal/orchestrator/rotation_test.go` - (`TestRotationEmitsValidReleaseWithAnchorSHA`), which drives the real - `Coordinator.Monitor` loop against an actual git worktree and asserts the - emitted event passes `domain.ValidatePayload` with the correct SHA — the - previous end-to-end test masked this bug by manually crafting a - replacement `TaskReleased` event after observing the (silently failed) - adapter-side release. -- **The federation worker release endpoint had the same gap.** The - `/v1/federation/workers/{id}/release` handler (cmd/orchestra/main.go) - built `TaskReleased` from a request body with only `handoff_ref`, no - `anchor_sha`. Since a remote worker is the only party with the actual - checkout (§2.1: "validate against the local checkout wherever the harness - runs"), the endpoint now requires and forwards a 40-hex-char `anchor_sha` - in the request body, rejecting the call with 400 otherwise. - -### Multi-repo Gitea ingestion (new) - -- `provider.Gitea` gained an optional `Project` field and `SourceName()` - (`"gitea"` if unset, `"gitea:"` if set) — the namespaced source - doubles as the `(source,external_id)` dedup key, so issue #7 in two - different repos never collides, and as the reflection dispatch key. -- New `provider.MultiGitea{Sources map[string]Gitea}` implements - `TaskReflector` by looking up `task.Source` and forwarding to the matching - Gitea instance — lets several Gitea repos (one per project) share one - `ReflectingSink`. -- New `provider.GiteaSourceConfig` + `LoadGiteaConfigs(path)` load a JSON - array of `{project,base_url,owner,repo,token,webhook_secret}`. - `main.go` reads this from `ORCHESTRA_GITEA_CONFIG` if set; each source - gets its own poll supervisor (`gitea:`) and webhook path - (`/v1/providers/gitea/webhook/`). -- The legacy single-repo env vars (`ORCHESTRA_GITEA_URL/TOKEN/OWNER/REPO/ - WEBHOOK_SECRET`) still work unchanged when `ORCHESTRA_GITEA_CONFIG` is - unset — same unprefixed webhook path, same `project = ORCHESTRA_GITEA_REPO` - tagging, same dedup source `"gitea"` — so existing deployments and - already-configured Gitea webhooks need no changes. -- Added `internal/provider/gitea_test.go` — previously **there were zero - tests exercising the Gitea provider at all** despite progress.md's prior - claim of Gitea webhook/poll test coverage; that claim was not accurate. - New tests cover source-name namespacing, webhook signature - verification/rejection, project tagging, `MultiGitea` dispatch-by-source - (via two `httptest.Server`s, asserting only the right one is hit), and - `LoadGiteaConfigs` validation/duplicate-project rejection. - -### Per-project repos (new) - -- `registry.Project` gained optional `repo`/`worktree_root` fields. Each - project can now resolve its own git checkout rather than every project - sharing one global `ORCHESTRA_REPO`/`ORCHESTRA_WORKTREE_ROOT` — matches - spec §2.2 ("projects are first-class and extensible... the binding is a - field + a config entry, not a schema change"). `main.go` builds a - `orchestrator.PerProjectGitWorktrees` from the registry, falling back to - the global default for any project that omits these fields, so - single-repo deployments are unaffected. Covered by - `internal/orchestrator/worktrees_test.go`. - -### Closed this pass (were open gaps as of the last snapshot) - -- **Bus-level authorization.** `authz.AuthorizeEvent` is now enforced inside - `store.Append` itself — the single choke point every event passes through - (HTTP handlers, router, coordinator/rotation, providers, federation relay) - — not just at HTTP handlers. Event schema bumped to v2, which requires - every event to declare a `Surface`; a new `authz.System` surface (full - control) covers internal emitters (router leases/failures, coordinator - releases/blocks, standup advisory/apply). Schema v1 events on disk still - replay (tolerant reader). Covered by `internal/store/store_test.go` and - `internal/router/router_test.go` additions asserting a non-HTTP append - with no/wrong surface is rejected. -- **Dual quota windows.** `router.QuotaAvailability` now tracks a 5-hour - rolling window and a 7-day weekly window independently per harness - (`QuotaWindowLimits{FiveHour, Weekly}`), applying the conservative 80% - rule to each separately — a harness over threshold on either window is - unavailable. Replaces the old single-`Window` field. Covered by new - `router_test.go` cases for weekly-only and 5h-only exhaustion. -- **Turn-boundary detection made observable, not silently optional.** - Rotation still can't force a harness adapter to implement `TurnBoundary` - Face B, but an adapter that fails to answer it now blocks that tick's - release (never treats a failed check as "safe to proceed"), and any - adapter without the capability — or one whose check errors — increments - `MonitorHealth.TurnBoundaryDegraded`, exposed via the coordinator's health - endpoint so degraded-safety operation is visible, not silent. -- **Cross-machine lease correctness has a real test.** - `internal/integration/federation_lease_test.go` - (`TestCrossMachineLeaseAnchorAndQuotaArePerHost`) exercises a lease - claimed through the federation worker HTTP API, validates the anchor - against that worker's own local checkout (not the router's), and asserts - quota is accounted per-host. Spec §9 item 8 said "prove on the first - federated run" — this is that proof for the primitives that exist today - (registration, heartbeat, lease-claim); it does not yet run against two - real physical machines. -- **Fuzz coverage for lifecycle payload validation.** - `internal/domain/fuzz_test.go` adds `FuzzValidatePayload` and - `FuzzValidateEvent` covering all event types (including malformed nested - `receipt`/`knowledge` shapes) — asserts no panic and always a typed error - on adversarial input. - -### Believed accurate from prior sessions (spot-checked, not exhaustively re-verified) - -- Event log: append-only JSONL, versioned envelope (schema v1), snapshot - load/replay, CAS with content-hash verification at append. -- `domain.ValidatePayload` enforces required fields per event type, - including `expected_version`/`ttl` on `TaskLeased`, `anchor_sha` on - `TaskReleased` (now correctly emitted, see above), `report_ref`+`receipt` - on `TaskCompleted`, and `blocker` on `TaskBlocked`. -- Router: project→affinity→machine resolution, capability match, quota - availability at conservative 80% threshold, derived-importance ordering, - retry-then-`TaskFailed`. -- herdr adapters (Claude/Codex/opencode) with native occupancy readers, - optional `TurnBoundary`/`RotationSignal`/`PaneExit` capability interfaces, - bootstrap/lease/release/kill. -- Continuity: strict handoff schema/validation, CAS save/load, pickup - validation (HEAD match, dirty-file hashes, immutable `TASK.md` hash), - scratch-branch commit/push/pull helpers. -- Provider layer: JSONL watcher, Gitea webhook+poll with HMAC auth, - idempotent `(source,external_id)` dedup, terminal-state reflection, - supervised restart with backoff. -- Federation: worker registration, heartbeat/TTL offline detection, event - cursor polling/ack, lease claim endpoint. -- Authorization: bus-level capability table (notify-only / full / gated) is - applied to lifecycle and approval writes via `AuthorizeEvent`. -- Delivery: Telegram/ntfy fan-out for completion/failure/block/approval - events. -- `/readyz`, `/v1/brief`, `/v1/providers/health`, `/v1/standup` exist and - return real state (not stubs). - -## Known open gaps (named, not silently assumed done) - -- **Cross-machine lease correctness is proven at the primitive level, not on - real hardware.** `TestCrossMachineLeaseAnchorAndQuotaArePerHost` exercises - the federation worker HTTP API (registration, lease-claim, anchor - validation against the worker's own checkout, per-host quota) inside one - test process. Spec §9 item 8 says "prove on the first federated run" — - that means an actual homesrv/workpc pair over the real mesh, which this - repo cannot exercise by itself. Named here as the one item that needs a - live two-machine run to fully close, not more code. -- **Turn-boundary Face B still degrades to occupancy-only for adapters that - don't implement it**, by design — the spec's Face B is per-harness native - session state (Stop hook / rollout tail / SSE), which this repo can only - wire against a real running herdr+harness pair. The degradation is now - observable (`MonitorHealth.TurnBoundaryDegraded`) and blocks-on-failure - rather than silently proceeding, but whether Claude/Codex/opencode's - native hooks are wired in a live deployment is a deployment-config fact, - not something provable from source alone. - -Everything else named as open in the previous snapshot (bus-level -authorization, dual 5h/weekly quota windows, fuzz coverage of lifecycle -payload validation) is now closed — see "Closed this pass" above. Broader -areas (provider layer, continuity, router matching, delivery, federation -registration) were spot-checked against the code and their tests and -matched their described behavior.