925 lines
53 KiB
Markdown
925 lines
53 KiB
Markdown
# Orchestra — spec conformance audit & remediation status
|
||
|
||
Audited 2026-07-27 against `orchestra-spec (1).md` at commit `325c684`.
|
||
Method: read every non-test file in `internal/` and `cmd/`, traced each spec
|
||
section to its call site, and checked whether the live path (`main.go` →
|
||
router → coordinator → adapter) actually reaches it. This file is the single
|
||
merged record — a prior split between `AUDIT.md` (audit + plan) and
|
||
`progress.md` (chronological session log) has been flattened here; treat any
|
||
older chronological git history of either file as session notes, not ground
|
||
truth.
|
||
|
||
**Ground truth rule for future sessions:** this repo has a documented history
|
||
of code that *looks* wired but isn't (packages with tests that pass in
|
||
isolation while the live call path silently no-ops). Before trusting any
|
||
"done"/"fixed" claim below, check the actual call site, not just this
|
||
document.
|
||
|
||
---
|
||
|
||
## Current verdict (as of 2026-07-28)
|
||
|
||
The substrate (Layer 1) is solid. Layers 2–4 were originally shaped-but-not-
|
||
wired; most of the blocking defects below are now closed, verified by
|
||
reading the current code (not just by trusting this file) and by targeted
|
||
tests. Two things are not yet proven:
|
||
|
||
1. **A real end-to-end unattended run.** The first live attempt (2026-07-28)
|
||
surfaced **B12** (fixed and confirmed live, see below), and re-verifying
|
||
that fix surfaced **B13** (open) — `agent.start` can return success while
|
||
silently never starting an agent, under back-to-back leases. One task
|
||
(`wD:p1`) did make it all the way to a real attached `claude` session, so
|
||
the lease path is provably reachable, but B13 means it's not yet reliable
|
||
enough to call proven — occupancy/rotation/handoff/completion still
|
||
haven't been exercised end-to-end against a session that's guaranteed to
|
||
actually exist.
|
||
2. **Cross-machine (federation) correctness.** Deliberately deferred — see
|
||
"The federation fork" below.
|
||
|
||
`go build ./...`, `go vet ./...`, and `go test ./...` all pass.
|
||
|
||
### Follow-up live verification (2026-07-28)
|
||
|
||
The OpenCode launch path has now been exercised against workpc herdr through
|
||
an isolated `test-e2e` worktree. The verified outcomes are deliberately
|
||
narrow:
|
||
|
||
- Initial prompts use a bounded retry only after an explicit JSON-RPC
|
||
rejection (herdr's short post-start readiness window). Transport timeouts
|
||
and disconnects are never replayed. A pane is inspected first; blocked or
|
||
permission-dialog panes are refused.
|
||
- Herdr accepts task-scoped unique names (`oc-<task-id>`); active sessions
|
||
are visible by that name in `agent.list`. Prompt routing uses that name,
|
||
while pane reads remain pane-scoped.
|
||
- The initial launch message carries the task title and description. This is
|
||
necessary under Design A because homesrv cannot write `TASK.md` inside a
|
||
workpc checkout. A live OpenCode run received the full instruction and,
|
||
after explicit operator approval of two file edits, created the requested
|
||
marker and report without touching other files.
|
||
- The initial launch does **not** wait for the agent to become idle. Waiting
|
||
converted an ordinary long-running first turn into a false `TaskBlocked`;
|
||
a fresh run now remains `TaskLeased` while OpenCode is paused at its normal
|
||
permission boundary.
|
||
|
||
This does not complete the cross-machine continuity design. `clients/herdr-
|
||
bridge.go` is only a byte proxy; it is not a worker. The worker-side process
|
||
described below remains required before a remote checkout can author
|
||
`TASK.md`, derive a canonical handoff anchor, scratch-commit WIP, or safely
|
||
release it. Until then, homesrv must not rotate or clean up a non-local
|
||
worktree.
|
||
|
||
| Spec layer | State |
|
||
|---|---|
|
||
| L1 substrate (§3, §4) | Built and correct in the main path. |
|
||
| L2 harness (§5) | Occupancy, rotation, and completion all wired and reachable; B12's readiness race is fixed and confirmed live, but B13 (`agent.start` silently no-op'ing under back-to-back leases) still blocks reliable live verification of what's downstream. |
|
||
| L3 continuity (§6) | Handoff schema, pickup validation, scratch branches, TASK.md are all wired into the live path (Phase 4 complete). |
|
||
| L4 surfaces (§7) | Brief/standup/delivery real; quota has a post-hoc producer only (no live push feed yet); authz bypass closed. |
|
||
|
||
---
|
||
|
||
## Blocking defect currently open
|
||
|
||
### B14 — `agent.prompt` can duplicate a task prompt after an ambiguous wait timeout (found live 2026-07-28) — open
|
||
|
||
Live Claude E2E task `06FTF8CPH3K3DPN6XQA7G3WRQ8` (pane `wM:p1`) received
|
||
the same initial `Begin Orchestra task ...` prompt twice. Claude completed a
|
||
first turn inspecting the repository and asking for task context, then began
|
||
working on the identical prompt again after the turn stopped. This was
|
||
observed directly in the pane transcript; it is not merely a UI replay.
|
||
|
||
**Root cause:** `CLIAdapter.Lease` calls `Client.Prompt` with
|
||
`wait.until=idle` and a 60-second wait. `Client.Call`, however, applies the
|
||
client's default 10-second connection deadline when the caller context has
|
||
no deadline. Once that local read deadline expires, `Prompt` treats the
|
||
result as any other transient failure and retries `agent.prompt` with the
|
||
same payload for up to `bootRetryWindow`. Herdr may already have accepted
|
||
the first request, so this is an ambiguous outcome, not a safe retry.
|
||
|
||
**Consequence:** an agent can repeat work, waste context/quota, or overwrite
|
||
its own in-progress changes. The initial task prompt is especially exposed
|
||
because it deliberately waits for the agent to become idle.
|
||
|
||
**Additional live consequence — OpenCode permission bypass (2026-07-28):**
|
||
the OpenCode E2E task `06FTF9Z8RP5FM4F9M2SKAQNFZ4` was leased to
|
||
`workpc-opencode` in `/tmp/test-e2e-worktrees/06FTF9Z8RP5FM4F9M2SKAQNFZ4`.
|
||
An operator directly observed OpenCode stop at the interactive permission
|
||
dialog for its initial `git log --oneline -10 && git status --short` command,
|
||
then advance and run that command without operator input. Workpc herdr's
|
||
live log records Orchestra's `agent.prompt` calls at 07:37:09 (request IDs 6
|
||
and 7), 07:37:10 (ID 8), and 07:37:20 (ID 9); the last two only ended when
|
||
Orchestra's 10-second RPC deadline disconnected the client. Herdr labels
|
||
each as `changes_ui=true`. The active OpenCode manifest contains only
|
||
permission-*detection* rules and no auto-approval rule. The task later
|
||
blocked at the next prompt (`~/.claude/RTK.md`) when no further prompt retry
|
||
advanced it.
|
||
|
||
This does not prove which internal herdr action produced the equivalent of
|
||
Enter, but it proves the safety boundary is broken: an Orchestra-originated
|
||
UI-changing `agent.prompt` operation can advance an OpenCode permission
|
||
dialog. Treat this as an authorization bypass, not merely duplicate work.
|
||
|
||
**Required fix:** make the RPC deadline at least the requested herdr wait
|
||
(plus a small transport margin), and do not blindly resend `agent.prompt`
|
||
after a timeout or other ambiguous post-write error. Record an observable
|
||
failure instead, or use an idempotency/acknowledgement mechanism if herdr
|
||
adds one. In addition, never issue `agent.prompt` while `agent_status` is
|
||
`blocked` (or a pane read matches a permission dialog); treat the condition
|
||
as requiring explicit operator approval. Do not run further unattended
|
||
OpenCode E2E tasks until both controls are verified live.
|
||
|
||
### B15 — a coordinator-side block cannot reconcile a later live completion (found live 2026-07-28) — open
|
||
|
||
The same OpenCode E2E task `06FTF9Z8RP5FM4F9M2SKAQNFZ4` was marked
|
||
`TaskBlocked` when the coordinator's `agent.prompt` RPC timed out, despite
|
||
the agent having received the task. The operator later explicitly approved
|
||
the one pending `~/.claude/RTK.md` read through herdr; OpenCode completed the
|
||
verification and became `idle` in pane `wN:p1`. Orchestra still showed
|
||
`state: "blocked"` because `Coordinator.Start` returned through `block()`
|
||
and never persisted a session; its monitor therefore has no live pane to
|
||
observe or reconcile.
|
||
|
||
**Consequence:** the event log can permanently claim a task is blocked when
|
||
the leased harness has actually completed it. This is not merely stale UI:
|
||
blocked tasks have no normal completion hook path and their receipt/quota
|
||
record is lost unless an operator manually corrects the lifecycle.
|
||
|
||
**Required fix:** retain enough launch/session identity after an ambiguous
|
||
lease failure to reconcile `agent.get`/completion evidence, or introduce a
|
||
versioned `TaskCorrected` terminal-state workflow that records the original
|
||
blocker and evidence without pretending it never happened. A live pane must
|
||
not be left outside the coordinator's session map simply because prompt
|
||
delivery timed out.
|
||
|
||
### B16 — hardcoded harness name makes each harness globally single-instance (found live 2026-07-28) — open
|
||
|
||
While starting the real OpenCode healthcheck task
|
||
`06FTFDW22833F1CCB8K43Z8808`, Orchestra created worktree/pane `wP:p1` but
|
||
herdr never attached an agent. A direct retry exposed the actual cause:
|
||
`agent.start` was sent with `name: "opencode"`, which herdr rejected with
|
||
`agent_name_taken` because the completed prior E2E agent in `wN:p1` already
|
||
owned that name. This is not a herdr limit of one OpenCode process per
|
||
machine: retrying the same `kind: "opencode"` in `wP:p1` with the valid,
|
||
unique lowercase name `oc-06ftfdw22833f1cc` immediately started a second
|
||
attached, idle OpenCode session while `wN:p1` remained intact.
|
||
|
||
**Root cause:** `Client.StartAgent` passes the harness kind as both
|
||
`kind` and `name`; every OpenCode lease therefore competes for the same
|
||
global agent-name slot. The same defect applies to Claude and Codex.
|
||
|
||
**Required fix:** keep `kind` as the configured harness, but derive `name`
|
||
from a validated, bounded task/session identifier (lowercase letters,
|
||
digits, `-`, `_`, maximum 32 characters) and persist it in `Session` for
|
||
subsequent lifecycle operations. Add a contract test that starts two same-
|
||
harness sessions in distinct panes and verifies both attach.
|
||
|
||
### B17 — opaque harnesses must not author canonical handoff anchors (found live 2026-07-28) — open
|
||
|
||
The real OpenCode healthcheck run showed that prompting an opaque agent to
|
||
write the full `continuity.Handoff` schema is the wrong ownership boundary.
|
||
The agent wrote JSON that looked plausible but used `anchor.sha` instead of
|
||
`anchor.git_sha`, omitted required top-level fields by nesting them under
|
||
`knowledge`, and could not correctly represent the uncommitted handoff file
|
||
as a dirty-file hash (a file cannot contain the SHA-256 of its own final
|
||
contents). Repeated corrective prompts made the handoff less reliable and
|
||
turned Orchestra's protocol details into agent prompt lore.
|
||
|
||
**Required direction:** prompt a harness exactly once for a small semantic
|
||
handoff/review report (what changed, validation evidence, remaining work,
|
||
review findings). The worker that owns the checkout must then deterministically
|
||
collect `HEAD`, branch, dirty paths and SHA-256 values, construct the canonical
|
||
handoff artifact, and validate it before publication. Canonical handoff files
|
||
are worker-owned protocol state and must be excluded from their own dirty-file
|
||
list. Project-level commands and stable expectations (`build`, `test`, file
|
||
ownership/invariants, etc.) belong in `AGENTS.md`/`CLAUDE.md`; only task-
|
||
specific evidence belongs in the handoff.
|
||
|
||
This cannot be safely implemented in Design A by homesrv: its coordinator
|
||
does not own the workpc checkout. Expand the bridge into a real worker-side
|
||
continuity participant (the deferred Design B direction) so all Git-derived
|
||
handoff facts are gathered and sealed on the machine that hosts the worktree.
|
||
|
||
**Live evidence (OpenCode healthcheck/review run, 2026-07-28):** task
|
||
`06FTFDW22833F1CCB8K43Z8808` initially hit B13/B16: Orchestra created `wP:p1`
|
||
but its hardcoded `name: "opencode"` conflicted with the prior E2E session.
|
||
Starting the same `kind: "opencode"` with unique name `oc-06ftfdw22833f1cc`
|
||
attached successfully without closing the prior pane. The implementation
|
||
agent created executable `scripts/healthcheck.sh`, validated `bash -n`, its
|
||
normal mode, and `--help`, then committed `672c123 Add scripts/healthcheck.sh`.
|
||
It first authored a lookalike handoff (`anchor.sha`, incomplete `meta`, and
|
||
all protocol fields under `knowledge`). A distinct reviewer in a second pane
|
||
(`wP:p2`) independently repeated the checks, noticed the nesting error, and
|
||
rewrote the document. Its claimed schema validation checked only JSON syntax
|
||
and the presence of keys — it did **not** use `continuity.Decode` or the
|
||
actual typed contract: `test` remains an array where `Handoff.Test` is a
|
||
string, and `last_result` remains a string where `Handoff.LastResult` is a
|
||
structured object. This is a concrete proof that a reviewing harness cannot
|
||
be trusted to validate canonical protocol state by visual shape alone.
|
||
|
||
The task is deliberately still `TaskBlocked` from its original failed
|
||
automatic attach even though its manually-started implementation/review panes
|
||
are live, a second live confirmation of B15. The reviewer pane is left open
|
||
for operator inspection; do not close or release it without approval.
|
||
|
||
**Related live observation — unexpected Claude launch explained:** Claude
|
||
pane `wM:p1` did not start spontaneously. At 08:00:41Z the expired lease for
|
||
task `06FTF8CPH3K3DPN6XQA7G3WRQ8` emitted `TaskReleased(reason:
|
||
"lease_expired")`; the router re-leased it at 08:00:42Z, and workpc herdr
|
||
attached Claude at 08:00:43Z. The renewed attempt then hit B14's prompt
|
||
timeout and became blocked. This is expected retry behavior, but illustrates
|
||
why lifecycle retries must be visible to operators.
|
||
|
||
### B13 — `agent.start` silently no-ops under back-to-back leases (found live 2026-07-28) — closed (code fix; not yet re-verified live)
|
||
|
||
Discovered while live-verifying the B12 fix below. B12 itself is confirmed
|
||
fixed (see that entry), but re-testing it exposed a second, deeper defect
|
||
that B12's retry logic cannot address.
|
||
|
||
Three fresh tasks were leased in quick succession against `workpc-claude`
|
||
(`06FTAKSJZTB73FZQE3QT7XQ1J0`/pane `wD:p1`, `06FTAMFKVEPGPKR0H86C08ZZNG`/pane
|
||
`wE:p1`, `06FTANAYZPA55BABR2J4MWQ050`/pane `wF:p1`). All three got a
|
||
`worktree.create` success and an `agent.start` call that returned **no
|
||
error**. Checked live via `pane.get`/`pane.list` (and independently
|
||
confirmed by the user cd'ing into each worktree by hand):
|
||
|
||
- `wD:p1` (the first, given time alone to settle): a real `claude` process
|
||
did eventually attach — `agent: claude`, `agent_status: idle`,
|
||
`revision: 2`. This is consistent with B12's model: `agent.start`
|
||
acknowledges the request but the actual attach is asynchronous, sometimes
|
||
taking well over a minute.
|
||
- `wE:p1` and `wF:p1` (started moments after `wD:p1`, while it was presumably
|
||
still initializing): **still no agent attached after 2+ minutes of
|
||
polling** — `agent_status: unknown`, no `agent` field, `revision` never
|
||
incremented past its creation value. Empty shells, confirmed both via
|
||
`pane.get` and by the user directly `cd`ing into the worktree and finding
|
||
no session running.
|
||
|
||
So `agent.start`'s "success" is not just delayed for these two — it never
|
||
happened at all, and `agent.start` never returned an error to tell Orchestra
|
||
that. One plausible cause: something about launching a second/third `claude`
|
||
CLI process while a prior one on the same host/account is still mid-startup
|
||
(auth handshake, config lock, etc.) silently drops the later request(s)
|
||
rather than queuing or erroring them. Not confirmed — would need a
|
||
controlled single-at-a-time repro to isolate from herdr's own internals,
|
||
which requires spinning up more real panes and wasn't done this pass (see
|
||
"three orphaned panes" below).
|
||
|
||
**Consequence:** unlike B12 (which at least produced an observable
|
||
`TaskBlocked`), this failure mode leaves the task leased indefinitely against
|
||
a pane that will never produce a session, with no error surfaced anywhere —
|
||
worse than B12 was, because nothing currently distinguishes "still booting,
|
||
give it more time" from "silently dead, will never start." A client-side
|
||
retry on `agent.start` itself (B12's approach) cannot fix this, since the
|
||
call that should have started the process already returned success with no
|
||
error to retry on.
|
||
|
||
**Fix:** since `agent.start`'s own return value can't be trusted, `StartAgent`
|
||
(`internal/herdr/herdr.go`) no longer treats its success as the end of the
|
||
story. After `agent.start` returns without error, it polls `pane.get({pane_id})`
|
||
(new `paneAgentAttached` helper) until the pane reports a real attached agent
|
||
(`agent` non-empty and `agent_status` present and not `"unknown"`), bounded by
|
||
a new `agentAttachWindow` (90s, longer than `bootRetryWindow`'s 15s since a
|
||
legitimate attach was observed live taking "well over a minute"). If the
|
||
window elapses with no attach, `StartAgent` now returns an explicit error
|
||
instead of a false success — this is exactly the "recorded/observable
|
||
failure over silent `continue`" pattern this repo's CLAUDE.md calls for, and
|
||
should surface as `TaskBlocked` through the same path B12's fix already
|
||
proved reachable.
|
||
|
||
**Not yet done:** re-verified against a live herdr instance (the schema for
|
||
`pane.get`'s result — `agent`/`agent_status` fields — was inferred from
|
||
prose in this file's own B12/B13 narration of live `pane.get` output, not
|
||
re-confirmed by a fresh probe of `192.168.1.105:9245`; `deploy/herdr-schema.json`
|
||
has no entry for `pane.get`'s result shape). `go build`/`go vet`/`go test
|
||
./...` all pass, but no test exercises `StartAgent`'s new polling loop
|
||
directly (existing `internal/herdr` tests don't call `agent.start`/`pane.get`
|
||
through the fake TCP listener at all). Next session should confirm the
|
||
`pane.get` field names live before trusting this closes B13 operationally,
|
||
and ideally fire a fresh back-to-back-lease test against workpc once that's
|
||
confirmed.
|
||
|
||
**Side effect of this investigation — three live orphaned panes on workpc,
|
||
left untouched on purpose:** `wD:p1` (has a real but abandoned `claude`
|
||
session, task now stuck in `blocked` from before B12's fix), `wE:p1` and
|
||
`wF:p1` (empty shells, no agent ever attached). Not cleaned up — same
|
||
standing rule as the pre-existing `wA:p1` stuck task: don't call
|
||
`pane.close`/`pane.release_agent` against live panes without asking first.
|
||
|
||
---
|
||
|
||
## B12 — `StartAgent` races the pane's shell readiness (found live 2026-07-28) — closed (this specific race; see B13 for a second issue it exposed)
|
||
|
||
Discovered during the first real end-to-end run against a freshly rebuilt/
|
||
redeployed binary (the running service had been on stale pre-audit commit
|
||
`325c684` the whole time).
|
||
|
||
Created a fresh task (`06FTAH4MCRCAJYY7V75Z4V2YGG`, project `test-e2e`,
|
||
capability `opencode`). It reached `TaskLeased` onto `workpc-opencode`
|
||
(worktree created for real at `/tmp/test-e2e-worktrees/<task-id>` on workpc,
|
||
confirmed via a live `pane.list` — pane `wB:p1`, correct `cwd`), then
|
||
**445ms later** hit `TaskBlocked`:
|
||
|
||
```
|
||
lease: herdr protocol error: agent target pane wB:p1 is not an available shell
|
||
```
|
||
|
||
`internal/herdr/herdr.go:199` (`StartAgent`) calls `agent.start` on the pane
|
||
`Worktree` just recorded, with no wait or retry after `worktree.create`
|
||
returns. herdr hands the pane back before its shell has actually finished
|
||
initializing, and `agent.start` refuses it as not-yet-a-shell. None of B1–B11
|
||
exercised this — every prior fix assumed a session already existed
|
||
(occupancy, rotation, handoff); nothing in the remediation plan tested the
|
||
very first `Lease` call against a real herdr pane end-to-end. Phase 0's
|
||
probing used `params:{}`/missing-field tricks, never a real
|
||
`worktree.create` → `agent.start` sequence timed against actual pane boot.
|
||
|
||
**Reproduced a second time, 2026-07-28**, same session: a second fresh task
|
||
(`06FTAHW8GZP0S6DCHEQ6PJXWB0`) leased onto a brand-new pane (`wC:p1`) and hit
|
||
the identical error on the same sub-second timeline. Two for two — a
|
||
systematic race, not a flake.
|
||
|
||
**Consequence:** every fresh lease that hits this race goes straight to
|
||
`TaskBlocked`, and the router **never retries a blocked task**
|
||
(`internal/router/router.go` has no reference to `StateBlocked` at all,
|
||
confirmed by grep) — so it sits there permanently until a human intervenes
|
||
via the approval surface. This is now the single blocking defect for proving
|
||
the rest of the remediation plan against a real live task.
|
||
|
||
**Fix, iterated live across three rounds, 2026-07-28:**
|
||
|
||
1. First pass: bounded-retry `agent.start` (10×, 500ms apart) specifically on
|
||
the `"not an available shell"` string. Redeployed and re-tested against a
|
||
real fresh lease — this cleared that exact error, but the very next call
|
||
in the sequence (`Lease`'s post-`agent.start` `Prompt`) then hit a
|
||
*different* transient error, `"agent ... is not an active named agent"` —
|
||
same underlying readiness race, one call later, worded differently.
|
||
2. Second pass: added the identical string-matched retry to `Prompt` for that
|
||
specific message. Redeployed, re-tested — this time hit a *third* distinct
|
||
wording, `"agent target ... not found"`, on the same call.
|
||
3. Given three different error strings for the same race across three live
|
||
attempts, string-matching was abandoned as unwinnable (herdr's wording for
|
||
"not ready yet" isn't fixed enough to enumerate). **Final shape:** both
|
||
`StartAgent`'s `agent.start` call and `Prompt`'s `agent.prompt` call now
|
||
retry on *any* error, bounded by wall-clock time (15s, `bootRetryDelay` of
|
||
500ms) rather than attempt count or specific wording — there is no other
|
||
legitimate reason a call against a pane/worktree orchestra itself just
|
||
created would fail immediately. `StartAgent` still special-cases `"already"`
|
||
as a non-error (idempotent re-lease).
|
||
|
||
Confirmed this closes the original race: task `06FTAKSJZTB73FZQE3QT7XQ1J0`
|
||
(pane `wD:p1`) leased successfully, and polling `pane.get` live shows a real
|
||
`agent: claude` / `agent_status: idle` attach — independently confirmed by
|
||
the user `cd`-ing into that worktree by hand. `go build`/`go vet`/`go test
|
||
./...` all pass throughout.
|
||
|
||
**Not fully clean, however:** re-testing this fix by firing two more tasks
|
||
shortly after the first one surfaced a second, separate defect — `agent.start`
|
||
returning success while never actually starting an agent, with no error for
|
||
the retry logic to even see. That's **B13**, above — this entry only covers
|
||
the specific shell/agent-readiness race B12 was originally filed for, which
|
||
*is* fixed; B13 is a distinct root cause the same investigation exposed.
|
||
|
||
---
|
||
|
||
## Closed defects
|
||
|
||
Each entry is the flattened final state — design + what's verified — not the
|
||
session-by-session narration. Grouped by original defect ID.
|
||
|
||
### B1 — Occupancy measured from the wrong value (§5.2.1) — closed
|
||
|
||
`CLIAdapter.Occupancy` was calling `a.Usage(s.PaneID)` (a herdr pane id)
|
||
against readers (`ClaudeUsage`/`CodexUsage`/`OpenCodeUsage`) that want a
|
||
filesystem path to session state — every call failed and was swallowed by a
|
||
bare `continue`. Fixed: `herdr.Session.SessionFile` is resolved at lease time
|
||
per harness (Claude: newest-mtime transcript under Claude Code's own project
|
||
dir; Codex: existing `CodexActiveUsage` sqlite discovery; opencode: refuses
|
||
loudly rather than guess, since it needs a live session id not resolvable
|
||
from the worktree alone). A missing/unreadable session file is now a hard
|
||
error surfaced via `SessionHealth.Occupancy`/`OccupancyError` on
|
||
`GET /v1/tasks/{id}/health`, never a silent zero.
|
||
|
||
**Still open:** live verification against a real Claude Code session at a
|
||
known context fill — not possible from this sandbox, blocked further by B13
|
||
now that fresh leases don't reliably survive.
|
||
|
||
### B2 — Adapter lookup keyed wrong in 3 of 4 call sites (§5.3, §5.4) — closed
|
||
|
||
`AdapterFactory.Herdrs` is keyed by herdr instance id (`homesrv-claude`);
|
||
`Reconcile`/`expire`/`rotate` were looking it up by `session.Harness` (the
|
||
harness kind, `claude`) and silently no-op'ing (`continue`) on every miss —
|
||
orphaned panes never killed, expired leases never killed their pane,
|
||
rotation never began. Fixed with a single `Coordinator.adapterFor(session)`
|
||
routed through at all four call sites. Regression test registers an adapter
|
||
under a herdr-id key distinct from the harness kind and asserts rotation
|
||
still fires.
|
||
|
||
### B3 — Nothing emitted `TaskCompleted` (§4, §5.2) — closed
|
||
|
||
Only producer used to be a human calling `POST /v1/tasks/{id}/complete`.
|
||
Added `POST /v1/harness/complete`: a Claude Code Stop hook
|
||
(`deploy/hooks/orchestra-stop.sh`) fires every turn boundary but only
|
||
reports completion once the agent has written a `.orchestra-report.md`
|
||
marker at the worktree root (an ordinary turn boundary is a no-op). The
|
||
server builds the `receipt` itself from `herdr.ClaudeUsage` against the
|
||
transcript (never trusts a self-reported number) and uploads the report
|
||
body to CAS for `report_ref`. Gated by optional `ORCHESTRA_HARNESS_TOKEN`;
|
||
event appended with `Surface: system` set directly in Go, consistent with
|
||
B8. Extended to Codex/opencode via an optional `harness` field in the
|
||
request body (`"codex"` → `CodexUsage`, `"opencode"` → `OpenCodeUsage`).
|
||
|
||
**Still open:** no automated test for the HTTP handler (`cmd/orchestra/main.go`
|
||
has zero handler test coverage of any kind, pre-existing gap — this follows
|
||
the existing pattern rather than introducing a one-off harness).
|
||
|
||
### B4 — Router counted rotation as a retry (§5.3, §5.4) — closed
|
||
|
||
Every `TaskReleased` (including rotation, which carries a valid
|
||
`handoff_ref`) advanced `MaxAttempts`, and lease also double-counted — a task
|
||
healthy enough to rotate twice was killed. Now only a release *without* a
|
||
`handoff_ref` (expiry/crash) advances the counter.
|
||
|
||
### B5 — Herdr protocol methods unverified/invented — closed
|
||
|
||
Confirmed live against `192.168.1.105:9245` (raw JSON-RPC probing, no local
|
||
`herdr` CLI available — schema reconstructed from Rust serde's
|
||
"unknown variant"/"missing field" errors). `pane.release`, `pane.kill`,
|
||
`pane.rotation_signal`, `pane.status` **do not exist**. Real replacements:
|
||
`pane.close({pane_id})` for kill (drop-in); `pane.release_agent({pane_id,
|
||
source, agent})` for release — structurally different, does **not** return
|
||
a `handoff_ref` (herdr never writes handoffs; the agent does, §6.1).
|
||
`RotationSignal` interface/method/call site deleted outright (no replacement
|
||
exists; herdr has no concept of Orchestra rotation).
|
||
|
||
`CLIAdapter.Release` now: reads the agent-authored `.orchestra-handoff.json`,
|
||
validates with `continuity.Decode`, cross-checks its anchor SHA against the
|
||
worktree's real `HeadSHA`, re-verifies every `Anchor.Dirty` file hash,
|
||
scratch-commits any dirty files (`continuity.ScratchCommit`, idempotent
|
||
across multiple rotations) and rewrites the handoff's anchor to the new
|
||
scratch commit before uploading to CAS via `continuity.Save` (minting the
|
||
real `handoff_ref`), and only then calls `pane.release_agent` — sequenced
|
||
last so a herdr-side error can't strand an uploaded handoff. Any failure
|
||
(missing file, invalid schema, anchor mismatch, stale dirty hash, herdr
|
||
error) is a refusal, which `rotate` already treats as "retry next tick."
|
||
`CLIAdapter.Lease`'s bootstrap prompt also fixed to pass `time.Minute` (not
|
||
`wait=0`) for inline-wait, matching `Bootstrap` and closing the
|
||
send-into-a-half-rendered-prompt race §5.1 requires guarding against.
|
||
|
||
Covered by `internal/herdr/adapter_test.go` against a real git worktree and a
|
||
fake in-process herdr TCP listener: upload-and-release on a valid handoff,
|
||
refusal with no handoff file, refusal on anchor mismatch, refusal on stale
|
||
dirty-file hash, refusal with no CAS configured.
|
||
|
||
Protocol version confirmed live as a bare JSON number (`17`), not the string
|
||
`config.jsonc` declares — `CheckProtocol`'s raw-bytes fallback happens to
|
||
compare correctly today; don't "clean up" that code without re-checking
|
||
this, or it may start doing a real numeric-vs-string comparison and break.
|
||
|
||
**Still open:** nothing makes the agent actually write
|
||
`.orchestra-handoff.json` unprompted (closed separately, see B6/Phase 4 item
|
||
2 below) — that's the piece this entry originally deferred.
|
||
|
||
### B6 — Layer 3 (continuity) was entirely dead code (§6) — closed
|
||
|
||
Nothing wrote `TASK.md`, so pickup validation had nothing to check and never
|
||
ran. Fixed in full:
|
||
|
||
- `GitWorktrees.Create` now writes and commits an immutable `TASK.md`
|
||
(`continuity.RenderTaskFile`) into every fresh worktree;
|
||
`continuity.TaskFileHash` reads it back for `w.TaskFileSHA`.
|
||
- `Coordinator.Start` runs `continuity.ValidatePickup` (anchor SHA +
|
||
dirty-file hashes + TASK.md hash) before bootstrapping a successor onto a
|
||
`handoff_ref` — failure kills the session and emits `TaskBlocked` instead
|
||
of trusting an unvalidated ref.
|
||
- `ScratchCommit` wired into `Release` (see B5) so pickup collapses to a
|
||
single HEAD compare instead of per-file rehashing.
|
||
- `CLIAdapter.Bootstrap`'s prompt rewritten to point the agent at
|
||
`git log`/the scratch branch (trust already established by the plane's own
|
||
`ValidatePickup`) instead of vague "read the handoff" prose, and
|
||
deliberately does not claim a `GET /v1/artifacts/<ref>` endpoint since none
|
||
exists (`/v1/artifacts` is POST-only — an earlier draft invented this and
|
||
was corrected before landing).
|
||
- `continuity.MarkdownChanges` (zero callers, zero tests despite being
|
||
listed as implemented in an earlier snapshot) was deleted rather than
|
||
half-wired. §6.3's actual requirement ("notice to agents whose task is
|
||
adjacent") was rebuilt independently: `continuity.ConventionsHash(root)`
|
||
hashes `AGENTS.md`/`CLAUDE.md`/`VOCAB.md`; `Coordinator.checkConventions`
|
||
runs every `Monitor` tick, recomputes the project base repo's hash for
|
||
every leased session, and on drift calls the optional
|
||
`herdr.ConventionsNotifier` capability (in-pane prompt) once per drift.
|
||
- **Handoff production, the item that most of B6 hinged on:**
|
||
`Coordinator.rotate` checks the adapter's optional `herdr.HandoffRequester`
|
||
capability; if `.orchestra-handoff.json` is missing, it prompts the agent
|
||
once (`CLIAdapter.RequestHandoff`, naming the exact §6.1 JSON shape and
|
||
explicitly telling the agent not to fabricate SHA/hashes) and skips
|
||
`Release` that tick, retrying every subsequent tick — mirrors the
|
||
`.orchestra-report.md`/B3 convention exactly. `Session.HandoffRequested`
|
||
avoids re-prompting every tick.
|
||
|
||
Covered by `TestGitWorktreesCommitsTaskFile`, `TestStartBlocksOnInvalidPickup`,
|
||
`TestReleaseScratchCommitsDirtyFilesBeforeUpload`,
|
||
`TestReleaseRefusesOnStaleDirtyFile`, `TestConventionsDriftNotifiesActiveSession`,
|
||
`TestRotationRequestsHandoffBeforeReleasing` (internal/herdr, internal/orchestrator).
|
||
|
||
**Caveat still open:** TASK.md hashing is best-effort/untested for the
|
||
herdr-hosted (`WorktreeCreator`) worktree path specifically — no adapter or
|
||
test exercises that path with a real `handoff_ref`, so pickup validation
|
||
there runs with an empty `taskFileSHA` (anchor + dirty-file hashes still
|
||
checked). Same cross-host caveat as the federation fork, below.
|
||
|
||
### B7 — Quota projection had no producer (§7.2) — closed (post-hoc only)
|
||
|
||
`POST /v1/harness/complete` now appends a `QuotaReported` event
|
||
(`consumed` from the same `usage.Numerator()` used for the receipt), so the
|
||
router's 5h/weekly filter and the brief's `quota_consumed` stop evaluating
|
||
against a permanent zero.
|
||
|
||
**Design gap recorded, not yet built** — a live push producer, since a
|
||
harness that never completes a task cleanly (e.g. the stuck `wA` pane, see
|
||
below) currently under-counts its consumption forever:
|
||
- **Claude**: the statusline stdin JSON already carries real
|
||
`.rate_limits.five_hour.used_percentage` / `.seven_day.used_percentage` —
|
||
confirmed by reading `~/.claude/statusline.sh` on this machine. Prefer this
|
||
over `claude.ai/api/.../usage`, which needs browser session cookies, not
|
||
an API key.
|
||
- **Codex**: every `token_count` event in the rollout carries a `rate_limits`
|
||
object with `used_percent`/`window_minutes`/`resets_at` — confirmed
|
||
directly from a real rollout file. No `codex usage` subcommand; the
|
||
rollout tail (or `codex app-server`, same events live) is the only
|
||
transport.
|
||
- **opencode**: no first-class quota surface. `opencode stats` is historical
|
||
only. Zen's free-tier daily quota arrives as `x-ratelimit-*` response
|
||
headers, not a queryable endpoint — would need an opencode plugin to
|
||
intercept.
|
||
- Structural consequence if this is ever built: `sumSince`'s *additive* model
|
||
is wrong for a used-*percentage* feed (need a second `Availability` reading
|
||
the latest report per harness, not summing); static `quota_limit_5h/weekly`
|
||
become unnecessary for Claude/Codex once real fractions are reported;
|
||
`QuotaWindowLimits{FiveHour,Weekly}` is too narrow for Codex's
|
||
self-describing `window_minutes` or opencode's daily Zen window.
|
||
|
||
### B8 — `Surface: system` unauthenticated bypass (§7.1) — closed
|
||
|
||
`X-Orchestra-Surface: system` was reachable from any HTTP request header in
|
||
both `authz.HTTP` and `main.go`'s `surface` closure — since no deployment
|
||
sets `ORCHESTRA_SYSTEM_TOKEN`, this was an unauthenticated full-control
|
||
bypass reachable from any LAN caller. Both call sites now downgrade `system`
|
||
to `web` before doing anything else with it. `System` is only constructible
|
||
in-process (router leases/failures, coordinator releases/blocks, standup
|
||
advisory/apply) via the bus-level `authz.AuthorizeEvent` check now enforced
|
||
inside `store.Append` itself.
|
||
|
||
### S1 — duplicate JSON struct tags — closed
|
||
|
||
`Brief.From`/`To` and `GitSync.Branch`/`Head`/`Status` each shared one JSON
|
||
tag (Go only honors the first `json:"..."` tag on a combined field
|
||
declaration), making `go vet ./...` fail and the brief's git state
|
||
unparseable. Fixed; `go vet ./...` passes clean.
|
||
|
||
### S2 + S3 — brief git state and completion receipts — closed
|
||
|
||
`Brief.Git` used to read from `ORCHESTRA_DATA` (the event-log dir, never a
|
||
git checkout — always "git unavailable") and completions were counted but
|
||
never surfaced with proof. `operations.Brief.Git` is now `map[string]GitSync`
|
||
keyed by project ID, built from `registry.Project.Repo` (falling back to a
|
||
single `"default"` entry off `ORCHESTRA_REPO`), with `Ahead`/`Behind` vs
|
||
upstream added. `Brief.Receipts []CompletionReceipt` pulls `report_ref`/
|
||
`receipt` straight out of each `TaskCompleted` event's existing payload.
|
||
|
||
### S4 — notification goroutine died on first send error — closed
|
||
|
||
`delivery.Fanout.Run` used to `return` on the first sender error,
|
||
permanently killing notifications for the rest of the process's lifetime.
|
||
Failed sends now go through an `OnError` hook instead of aborting the loop.
|
||
Cursor is persisted (`SaveCursor` → a `delivery-cursor` file next to
|
||
`ORCHESTRA_DATA`), so a restart resumes from the last delivered event
|
||
instead of re-notifying the entire log from seq 0.
|
||
|
||
### S5 — colliding event IDs on lease — closed
|
||
|
||
`Store.Lease`/`ExpireLeases` set `Event.ID` to the task id, so every lease of
|
||
a task produced colliding event IDs. Now uses `domain.NewID()`.
|
||
|
||
### S6 — ingest dedup returned the wrong event with 200 — closed
|
||
|
||
Dedup path returned `nil` (success) without appending; the HTTP handler then
|
||
returned an unrelated event with `201`. Added `domain.ErrDuplicate` and
|
||
`Store.TaskBySource`; `POST /v1/tasks` now returns the existing task with
|
||
`200` on a duplicate. Every other `Append` caller (Gitea poll/webhook, JSONL
|
||
ingest) now treats `ErrDuplicate` as expected.
|
||
|
||
### S7 — `TaskAmended` dropped most fields — closed
|
||
|
||
Only `title` was ever applied from an amendment payload; `due`/
|
||
`description`/`inherent_priority` were accepted, logged, and silently
|
||
dropped by the projection. Also added the missing `Task.Description` field
|
||
entirely (it didn't exist, so `TaskCreated` dropped it too). Both
|
||
`TaskCreated` and `TaskAmended` now populate all four fields.
|
||
|
||
### S8 — no compensating-event mechanism (§3.1) — closed
|
||
|
||
Added `TaskCorrected`: payload requires `corrects` (the id of the event it
|
||
repairs) plus at least one change (`state`, or the existing amend-style
|
||
fields). `Store.Append` rejects a `corrects` that doesn't name a real prior
|
||
event on the same task; `apply` applies the changes and clears `Lease` like
|
||
every other terminal-state branch. Covered by a mistaken `TaskFailed`
|
||
reverted to `queued`, an unknown-`corrects` rejection, and both events
|
||
surviving snapshot+replay.
|
||
|
||
### S9 — duplicate lease-expiry tickers — closed
|
||
|
||
`Coordinator.Monitor`'s 30s ticker and `main.go`'s own 1s reclaim ticker both
|
||
called `Store.ExpireLeases` independently. Previously filed as "harmless —
|
||
CAS rejects the loser," but the real effect was worse: the coordinator's
|
||
`expire()` is the *only* place that kills the herdr session/pane for an
|
||
expired lease, and the 1s ticker running 30x more often almost always won
|
||
the race, leaving the coordinator's own call with nothing left to expire —
|
||
silently orphaning herdr panes past TTL. Fixed: `main.go`'s ticker skips
|
||
`ExpireLeases` entirely when a coordinator is configured (deferring reclaim
|
||
to the coordinator's loop), keeping only `AssignPending` as periodic retry.
|
||
|
||
### S10 — federation worker registration had no admission control — closed
|
||
|
||
Any caller could self-declare a worker `id` and self-choose a `token`, and a
|
||
second caller could silently re-register an existing worker id with a
|
||
*different* token, hijacking its identity/capacity. Added
|
||
`Registry.AdmitToken` (pre-shared secret via
|
||
`ORCHESTRA_FEDERATION_ADMIT_TOKEN`, checked against the registration
|
||
request's bearer header); same-ID re-registration now requires presenting
|
||
the existing worker's own token (legitimate restarts still succeed;
|
||
different-token same-id is rejected as a hijack).
|
||
|
||
### S11 — soft threshold, milestone, thrash, agent-initiated ROTATE (§5.3) — closed
|
||
|
||
All four of §5.3's rotation triggers now land:
|
||
|
||
- **Soft threshold**: `Coordinator.Soft` (default 0.55,
|
||
`ORCHESTRA_OCCUPANCY_SOFT`) — both `rotate()` and `TurnDecision` request a
|
||
handoff once occupancy crosses `Soft`, before `Hard` forces one;
|
||
`TurnDecision` returns `prepare_handoff` advisory (task stays leased, no
|
||
turn boundary required).
|
||
- **Agent-initiated ROTATE**: `handoffReason(worktree)` reads a written
|
||
`.orchestra-handoff.json`; if `reason == "manual"`, both `rotate()` and
|
||
`TurnDecision` skip occupancy and the turn-boundary probe entirely and
|
||
release immediately — the agent's own handoff *is* the boundary signal.
|
||
Shared release-and-certify tail extracted into `Coordinator.finishRelease`
|
||
so this path gets the same anchor-safety guarantee as the threshold path.
|
||
- **Milestone + thrash**: needed a transcript/tool-call introspection source
|
||
this repo didn't have — built new, `internal/herdr/activity.go`:
|
||
- `ToolCall{Name, Kind, Key, Success, IsTest}` normalizes one tool/function
|
||
call across harnesses.
|
||
- `ClaudeActivity` pairs `tool_use`/`tool_result` blocks by `tool_use_id`
|
||
from the transcript already opened for occupancy (an unresolved
|
||
tool_use — mid-turn — is dropped, not reported).
|
||
- `CodexActivity` — **verified against a real live rollout, 2026-07-28**,
|
||
after the original guessed `function_call`/`function_call_output` shape
|
||
turned out not to exist anywhere in `~/.codex/sessions`. Real shape:
|
||
file edits arrive as `event_msg`/`patch_apply_end` (has `changes` +
|
||
top-level `success` directly, no pairing needed); shell commands arrive
|
||
as a freeform `response_item`/`custom_tool_call` named `"exec"` whose
|
||
`input` is a JS snippet — `codexExecCommand` regex-extracts the first
|
||
embedded `cmd:"..."`; failure is signaled by the literal output prefix
|
||
`"Script error:"` (confirmed for both a JS syntax error and an
|
||
`apply_patch` verification failure on this machine's real transcripts).
|
||
Manually re-ran the rewritten parser against a real multi-hundred-line
|
||
rollout end-to-end and spot-checked the output by eye, in addition to
|
||
`TestCodexActivityParsesRealRolloutShape` /
|
||
`TestCodexActivityMarksScriptErrorAsFailure`.
|
||
- `OpenCodeActivity` **refuses outright** — opencode's on-disk message
|
||
storage is only confirmed to carry aggregate token counts, not
|
||
per-tool-call records; fabricating a parser against an unconfirmed shape
|
||
would repeat the exact mistake this audit exists to catch.
|
||
- `DetectThrash(calls, ThrashConfig)` implements all three rules: N
|
||
consecutive failed test runs (regex-matched build/test commands), the
|
||
same file edited M times (edit tools only, explicitly excluding `Read` —
|
||
caught by an initially-failing test), and an identical tool call
|
||
repeated K times back-to-back (excluding test re-runs and file reads —
|
||
also caught by an initially-failing test). Defaults 3/5/4, overridable
|
||
via `Coordinator.Thrash`.
|
||
- `DetectMilestone(calls)` — deliberately narrow: only "the most recent
|
||
call was a successful `git commit`." Fuzzier definitions (passing test
|
||
suite, finished subtask) were not guessed at.
|
||
- `CLIAdapter.Activity` resolves the session file the same way `Occupancy`
|
||
does; `ActivityReader` is an optional Face-B capability like the others.
|
||
- New `ReasonedHandoffRequester`/`CLIAdapter.RequestHandoffReason` names
|
||
*why* (thrash's dead ends, milestone reasoning) instead of the generic
|
||
threshold framing.
|
||
- `rotate()`/`TurnDecision` generalize "reason=manual bypasses occupancy"
|
||
to `manual`/`milestone`/`thrash` alike: `checkActivityTriggers` runs
|
||
before falling through to occupancy/soft/hard (thrash takes priority
|
||
over milestone), and a hit calls `requestReasonedHandoff` — request
|
||
only, never release, same as the soft-threshold path.
|
||
|
||
Covered by `internal/herdr/activity_test.go` and
|
||
`internal/orchestrator/rotation_test.go`'s
|
||
`TestActivityTriggersRequestReasonedHandoffWithoutReleasing`,
|
||
`TestTurnDecision` subtests for soft-threshold and manual-reason, and
|
||
`TestRotationRequestsHandoffBeforeReleasing`.
|
||
|
||
**Known remaining limitation:** `DetectMilestone`'s single-command
|
||
extraction only sees the *first* `cmd:"..."` in a chained Codex exec script,
|
||
so a `git commit` issued later in the same script isn't recognized as the
|
||
"last call" even though it executed last. Opencode still has no verified
|
||
per-tool-call source at all.
|
||
|
||
### Also closed this pass (not separately numbered in the original audit)
|
||
|
||
- **Codex/opencode Stop-hook-equivalent scripts.** Neither harness has a
|
||
native Stop hook, so the server-side `/v1/harness/turn` and
|
||
`/v1/harness/complete` (harness dispatch) existed with no caller for
|
||
either. Added `deploy/hooks/orchestra-codex-poll.sh` and
|
||
`orchestra-opencode-poll.sh` — background poll loops (default 60s,
|
||
`ORCHESTRA_POLL_INTERVAL`) that find the newest session-state file (codex:
|
||
newest `rollout-*.jsonl`; opencode: newest file under
|
||
`~/.local/share/opencode/storage/message/`), POST to `/complete` when
|
||
`.orchestra-report.md` exists, otherwise POST to `/turn` and log (not act
|
||
on) `refuse`/`rotate_now` — advisory-only, since there's no real turn
|
||
boundary to refuse *at* from outside either harness process without deeper
|
||
app-server/SSE integration.
|
||
- **Bus-level authorization.** `authz.AuthorizeEvent` enforced inside
|
||
`store.Append` itself (the single choke point every event passes
|
||
through), not just at HTTP handlers. Event schema bumped to v2, requiring
|
||
every event to declare a `Surface`; schema v1 events on disk still replay
|
||
(tolerant reader).
|
||
- **Dual quota windows.** `router.QuotaAvailability` tracks a 5-hour rolling
|
||
window and a 7-day weekly window independently per harness, applying the
|
||
conservative 80% rule to each separately.
|
||
- **Turn-boundary detection made observable.** An adapter that fails to
|
||
answer `TurnBoundary` now blocks that tick's release rather than treating
|
||
the failure as "safe to proceed," and increments
|
||
`MonitorHealth.TurnBoundaryDegraded`.
|
||
- **Cross-machine lease correctness has a primitive-level test.**
|
||
`TestCrossMachineLeaseAnchorAndQuotaArePerHost` exercises a lease claimed
|
||
through the federation worker HTTP API, validates the anchor against that
|
||
worker's own local checkout, and asserts quota is accounted per-host —
|
||
proof for the primitives that exist today; does not yet run against two
|
||
real physical machines (see federation fork, below).
|
||
- **Fuzz coverage** (`internal/domain/fuzz_test.go`,
|
||
`FuzzValidatePayload`/`FuzzValidateEvent`) for all event types including
|
||
malformed nested payloads — no panic, always a typed error.
|
||
- **Multi-repo Gitea ingestion.** `provider.Gitea` gained a `Project` field
|
||
and namespaced `SourceName()`; `provider.MultiGitea` dispatches by
|
||
`task.Source`; `LoadGiteaConfigs` loads a JSON array of per-project
|
||
sources. Legacy single-repo env vars still work unchanged. Previously zero
|
||
tests existed for the Gitea provider at all (an earlier progress.md claim
|
||
of coverage was inaccurate) — `internal/provider/gitea_test.go` added.
|
||
- **Per-project repos.** `registry.Project` gained optional `repo`/
|
||
`worktree_root`; `main.go` builds a `PerProjectGitWorktrees`, falling back
|
||
to the global default for any project that omits these fields.
|
||
- **Rotation's `TaskReleased` payload validity.** `Coordinator.rotate` used
|
||
to build `{"handoff_ref","reason"}`, omitting the `anchor_sha` the spec and
|
||
`domain.ValidatePayload` require whenever `handoff_ref` is present —
|
||
`store.Append` would reject it, the error was discarded
|
||
(`if c.Store.Append(e) == nil`), and rotation silently never happened, no
|
||
visible failure. Fixed: `herdr.HeadSHA(worktree)` populates `anchor_sha`
|
||
from the real worktree HEAD before appending; if the anchor can't be read,
|
||
rotation now skips that tick instead of emitting a payload guaranteed to
|
||
fail validation. The federation worker release endpoint
|
||
(`/v1/federation/workers/{id}/release`) had the identical gap and now
|
||
requires/forwards a 40-hex-char `anchor_sha`, rejecting with 400
|
||
otherwise.
|
||
|
||
---
|
||
|
||
## The federation fork — decided 2026-07-27, still the deferred half
|
||
|
||
Two incompatible federation designs coexist in the tree.
|
||
|
||
**Design A — "drive the remote socket" (currently deployed,
|
||
`clients/herdr-bridge.go`)**: homesrv calls `worktree.create`/`agent.start`/
|
||
etc. directly on workpc's herdr over TCP as if it were local — meaning
|
||
anchor validation (`git rev-parse HEAD`) executes on the *wrong machine*
|
||
relative to the actual checkout. Two outcomes if the coordinator's
|
||
`session.Worktree` path happens to also exist on homesrv (likely, since
|
||
every project shares the same directory layout): either `HeadSHA` errors and
|
||
rotation silently skips forever, or — the dangerous case — it returns
|
||
**homesrv's HEAD for an unrelated checkout**, passing validation while
|
||
certifying a commit the agent never touched. Same class of bug applies to
|
||
per-host quota accounting and to `cleanupCompleted`'s `git worktree remove`,
|
||
which runs on homesrv for a worktree that lives on workpc.
|
||
|
||
**Design B — "workers pull tasks" (`/v1/federation/*`)**: fully built
|
||
server-side (registration, heartbeat/TTL offline detection, event-cursor
|
||
polling/ack, lease claim), zero clients — no worker binary exists anywhere
|
||
in this repo. This is what the spec actually describes (§2.1: "everything
|
||
crossing a machine boundary is git + a validated artifact, never live state
|
||
over the wire"), but every endpoint is currently unreachable in the real
|
||
deployment. Latent, never-surfaced defects in this unused half: offline
|
||
detection only runs inside `Snapshot()`, called solely from a GET endpoint —
|
||
nothing ticks it on its own, so `OnOffline` (the hook that releases leases
|
||
held by a vanished worker) only fires if a human hits that endpoint; the
|
||
registry is in-memory with no persistence, so a restart forgets all workers
|
||
and cursors.
|
||
|
||
**Decision, unchanged:** keep Design A through Phase 5 (single-host concerns
|
||
— occupancy, Face B, rotation, continuity — are provable on homesrv alone
|
||
with workpc's herdr as just another pane host); commit to Design B in Phase
|
||
6. Two guardrails were meant to land immediately so Design A can't corrupt
|
||
state in the meantime: (1) refuse to rotate a lease held by a non-local
|
||
herdr rather than validate against the wrong checkout, since the protocol
|
||
schema can't run `rev-parse` where the checkout is; (2) same treatment for
|
||
`cleanupCompleted`'s worktree removal. **Status of these two guardrails is
|
||
unverified in this pass** — re-check `internal/orchestrator` before assuming
|
||
they landed; they are not confirmed closed above the way B1–B11/S1–S11 are.
|
||
|
||
If the worker binary is ever abandoned, delete `/v1/federation/*` and record
|
||
the deviation — leaving both designs in place unmarked is explicitly not
|
||
acceptable; the repo would read as though the spec-conformant path is
|
||
implemented when nothing can reach it.
|
||
|
||
---
|
||
|
||
## Live deployment facts (as of 2026-07-28)
|
||
|
||
- Runs as `orchestra.service` on **homesrv** via this machine's own systemd.
|
||
`homesrv` has no local herdr running (connection refused on 9245) — only
|
||
`workpc`'s herdr (`192.168.1.105:9245`) is live and reachable. `main.go`
|
||
only logs herdr connection *failures* at startup, never successes, so "no
|
||
log line" for a herdr does not mean it's down.
|
||
- **Stuck task, deliberately left untouched:** workspace `wA`, task
|
||
`06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode harness, pane `wA:p1`. From
|
||
Orchestra's own point of view it is no longer "stuck" — it now reads
|
||
`state: "failed"` (retries exhausted, `MaxAttempts` hit before B4's fix
|
||
landed). But the underlying herdr pane is still live and
|
||
`agent_status: "blocked"`, confirmed via a direct `agent.get` probe — the
|
||
router gave up and moved on, but nothing ever released or killed the
|
||
actual agent, confirming the orphaned-pane prediction from B2/B5. Left
|
||
untouched on purpose: user was asked and chose to leave it rather than
|
||
have it cleaned up mid-audit. Do not call `pane.close`/`pane.release_agent`
|
||
against it without asking again first.
|
||
- First live end-to-end task run (2026-07-28, after rebuilding/redeploying —
|
||
the running service had been on stale pre-audit commit `325c684`) is what
|
||
surfaced **B12** above, since fixed and confirmed live; re-verifying it
|
||
surfaced **B13**, still open.
|
||
- **Three more live orphaned panes from this pass, left untouched:** `wD:p1`
|
||
(task `06FTAKSJZTB73FZQE3QT7XQ1J0`, worktree
|
||
`/tmp/test-e2e-worktrees/06FTAKSJZTB73FZQE3QT7XQ1J0` — has a real attached
|
||
`claude` session but the task itself is stuck `blocked` from before B12's
|
||
fix landed), `wE:p1` and `wF:p1` (tasks `06FTAMFKVEPGPKR0H86C08ZZNG` /
|
||
`06FTANAYZPA55BABR2J4MWQ050` — empty shells, no agent ever attached, see
|
||
B13). Same rule as `wA:p1`: don't `pane.close`/`pane.release_agent` any of
|
||
these without asking first.
|
||
|
||
---
|
||
|
||
## Known open gaps (as of 2026-07-28)
|
||
|
||
- **B13** (above) — `agent.start` can silently no-op under back-to-back
|
||
leases, with no error surfaced; blocks reliable live verification of
|
||
everything downstream (occupancy, rotation, handoff, completion) against a
|
||
real task, even though B12's readiness race is now fixed.
|
||
- **Cross-machine lease correctness proven at the primitive level only** —
|
||
needs an actual homesrv/workpc pair over the real mesh; this repo cannot
|
||
exercise that by itself.
|
||
- **Turn-boundary Face B degrades to occupancy-only** for adapters that
|
||
don't implement it, by design — degradation is observable
|
||
(`MonitorHealth.TurnBoundaryDegraded`) and blocks-on-failure, but whether
|
||
Claude/Codex/opencode's native hooks are wired in a live deployment is a
|
||
deployment-config fact, not provable from source alone.
|
||
- **Quota has no live push producer** — only the post-hoc
|
||
`/v1/harness/complete` producer exists; see B7's "design consequences" for
|
||
the (unbuilt) Claude statusline / Codex rollout-tail feeds.
|
||
- **Federation guardrails' landed-status is unverified** — see federation
|
||
fork section above.
|
||
- **opencode has no verified per-tool-call activity source** and no
|
||
first-class quota surface (Zen headers only, would need a plugin) —
|
||
both named, not attempted, in S11/B7.
|
||
- **`DetectMilestone`'s single-command extraction** can miss a Codex commit
|
||
issued later in a chained exec script.
|
||
|
||
Everything else audited (provider layer, continuity schema/validation,
|
||
router matching, delivery, federation registration primitives) was
|
||
spot-checked against the code and its tests and matched described behavior.
|
||
|
||
---
|
||
|
||
## Handoff provenance correction (2026-07-28)
|
||
|
||
The previous rotation path had a **primacy inversion**: the harness wrote a
|
||
free-form Markdown “semantic report”, while `canonicalHandoff` fabricated
|
||
the canonical fields around it (`goal`, `done_when`, a circular action and a
|
||
`cat` command) and placed the entire report in one `remaining` element. That
|
||
made the apparent schema a wrapper around prose rather than an authoritative
|
||
handoff.
|
||
|
||
The release path now asks the harness for only six labelled, bounded answers:
|
||
`NEXT`, `WHY`, `REMAINING`, `DEAD ENDS`, `OPEN Q`, and `LEARNED`, each with a
|
||
`NONE` escape. `canonicalHandoff` parses those answers and derives only
|
||
worker-owned facts (Git anchor, dirty state, metadata, and last observed
|
||
harness command). It never invents task intent. `goal` and `done_when` were
|
||
removed from `continuity.Handoff`; pickup must obtain task scope and success
|
||
criteria from immutable `TASK.md`.
|
||
|
||
`continuity.Handoff.Validate` is the shared semantic gate for both release
|
||
and pickup. Strict decoding rejects removed/unknown fields, and validation
|
||
rejects circular handoff actions, commands that point at report/handoff
|
||
files, oversize or Markdown-smuggled authored list items, and malformed dead
|
||
ends. A failed release remains refused and is retried by the existing
|
||
rotation loop; a bad artifact that somehow reaches CAS is also refused at
|
||
pickup. The fuzzy “empty dead ends after a non-trivial later rotation” signal
|
||
is intentionally not yet enforced: no reliable diff-size/rotation-index
|
||
evidence is available at this validation boundary, so inventing a hard rule
|
||
would create false refusals.
|
||
|
||
Verified after the change with `go test ./...`, `go build ./...`, `go vet
|
||
./...`, and `git diff --check`.
|