diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..f1c07f9 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,112 @@ +# Orchestra + +A Go implementation of `orchestra-spec (1).md` — an unattended multi-agent +task orchestrator that leases coding tasks to CLI harnesses (Claude Code, +Codex, opencode) running inside `herdr`-managed panes, rotates them across +context-window limits, and hands off work via a git-anchored continuity +protocol. + +Layout: `internal/{domain,store,provider,registry,router,herdr,orchestrator, +continuity,federation,delivery,authz,operations,admin}` + `cmd/orchestra/main.go`. + +## Ground truth over documentation + +This repo has a documented history of code that *looks* wired but isn't — +packages with tests that pass in isolation while the live call path silently +no-ops (bare `continue` on error, discarded return values). See `AUDIT.md` +for the full audit and `progress.md` for a running log. **Before trusting a +claim in progress.md that something "works" or "is fixed," check the actual +call site** — the file is written by past sessions of this same assistant and +has previously overstated completion. + +The single most reliable way to verify herdr-adapter code is right: don't +read `internal/herdr/adapter.go` and assume the method names are real. Ping +the live herdr instance and check. + +## herdr protocol — verified against a live instance, 2026-07-27 + +- herdr speaks JSON-RPC over a raw TCP (or unix-socket) connection — **not + HTTP**. `internal/herdr/herdr.go`'s `Client.Call` is the only correct way + to talk to it; a bare `curl` to the port returns nothing. +- Request shape: `{"id":"","method":"","params":}`. Herdr's + Rust JSON-RPC decoder requires `params` to be present and rejects a bare + `null` — always send `{}` for parameterless calls (the client does this + automatically). +- Full real method list is committed at `deploy/herdr-schema.json`, captured + live from `192.168.1.105:9245` (the `workpc` herdr) since no local `herdr` + CLI is available in this sandbox — the schema was reconstructed by sending + an unknown method name and reading the `unknown variant ... expected one + of ...` error, then probing each method of interest with `params:{}` / + `params:{pane_id:"nonexistent"}` to read Rust serde's `missing field + ` errors for its param shape. +- **Confirmed invented (do not use, they don't exist):** `pane.release`, + `pane.kill`, `pane.rotation_signal`, `pane.status`. If you see these + anywhere, it's a bug, not a valid call. +- **Real replacements:** `pane.close({pane_id})` for kill; + `pane.release_agent({pane_id, source, agent})` for release (structurally + different — does *not* return a `handoff_ref`, see below). No replacement + exists for `rotation_signal` — herdr has no concept of Orchestra rotation. +- **Architectural point that's easy to get wrong:** herdr never produces a + handoff. The agent writes the handoff artifact (§6.1 of the spec); herdr's + role in "release" is only to drop its own claim on the pane/agent binding. + Any adapter code that expects herdr to hand back a `handoff_ref` is wrong + by construction, independent of whether the method name is right. +- Protocol version is returned as a **JSON number** (`17`), not a string, + even though `config.jsonc` declares `"protocol": "17"` as a string. + `CheckProtocol`'s raw-bytes fallback happens to make this compare correctly + today — don't "clean up" that code without checking this note first, or it + might start doing a real numeric-vs-string comparison and break. + +## Deployment topology (as of 2026-07-27) + +- Runs as `orchestra.service` on **homesrv** (this machine's own systemd — + `journalctl -u orchestra.service` for logs; `sudo` isn't available in this + sandbox environment, but plain `journalctl` without sudo works here). +- Config: `.orchestra-config/orchestra.env` (env vars) + + `.orchestra-config/config.jsonc` (projects/machines/herdrs registry). + `ORCHESTRA_CONFIG=/etc/orchestra/config.jsonc` — the deployed copy, not the + repo's `deploy/config.example.jsonc`. +- Two machines in the registry: `homesrv` (192.168.1.104) and `workpc` + (192.168.1.105), each nominally running 3 herdrs (claude/codex/opencode). + In practice **homesrv has no local herdr running** (connection refused on + 9245) — only workpc's herdr is live and reachable. `main.go` only logs + herdr connection *failures* at startup, never successes, so "no log line" + for a herdr does not mean it's down — check reachability directly. +- There was a real, live, stuck task as of 2026-07-27: workspace `wA`, task + id `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode harness, pane `wA:p1`, + `agent_status: "blocked"`. Likely stuck because rotation/release could + never reach it (B2/B5). Check whether it's still stuck before assuming + fixes here have taken effect operationally — code fixes don't retroactively + unstick an already-orphaned pane; that needs a manual kill/restart once the + release path is trustworthy. + +## The federation fork — read before touching anything cross-machine + +Two incompatible designs coexist. **Design A** ("drive the remote socket", +currently deployed via `clients/herdr-bridge.go`) has homesrv call +`worktree.create`/`agent.start` etc. directly on workpc's herdr over TCP as +if it were local — meaning anchor validation (`git rev-parse HEAD`) run by +the coordinator executes on the *wrong machine* relative to the actual +checkout. **Design B** ("workers pull tasks", `/v1/federation/*`) is fully +built server-side but has zero clients — no worker binary exists. Decision +(AUDIT.md, 2026-07-27): keep Design A through Phase 5, commit to Design B in +Phase 6, with two guardrails landed immediately (refuse to rotate/cleanup a +lease held by a non-local herdr rather than validate against the wrong +checkout). Don't build on top of Design A's cross-machine calls without +reading that section first. + +## Working conventions + +- `go build ./...`, `go vet ./...`, and `go test ./...` must all pass — `go + vet` was broken for a while (duplicate JSON struct tags) and nobody + noticed because only `build`/`test` were being checked. Always run all + three. +- Silent `continue`-on-error is the recurring bug pattern in this codebase + (adapter lookups, rotation, expiry). When touching `internal/orchestrator` + or `internal/herdr`, prefer a recorded/observable failure + (`MonitorHealth` fields) over a bare `continue` — that's literally what + turned B1/B2 invisible for as long as they were. +- Don't invoke destructive herdr calls (`pane.close`, `pane.release_agent`) + against a real pane from an investigative/audit session without asking + first — there is live operator state on the other end (see the stuck-task + note above). diff --git a/AUDIT.md b/AUDIT.md index 6d7fd89..e513994 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -473,3 +473,67 @@ If you want one thing to do today: **Phase 0**, then **B2** (a ~20-line fix that makes three subsystems reachable), then **B1**. Those three turn a system that cannot rotate into one that can, and everything else in the plan is building on top rather than repairing underneath. + +--- + +## Phase 0 — done, 2026-07-27 + +B1, B2, B4, B8, S1, S5, S6 were already fixed and landed as of this session +(confirmed by reading the current code, not just trusting progress.md — see +`adapterFor` in `internal/orchestrator/orchestrator.go:190` and +`CLIAdapter.Occupancy` in `internal/herdr/adapter.go:197`). + +This box (homesrv) turned out to have live TCP reachability to the real herdr +instance at `192.168.1.105:9245` (workpc) the whole time — the `unavailable: +connection refused` lines in `journalctl -u orchestra.service` are for +`homesrv-*` herdrs dialing `192.168.1.104:9245`, which has no local herdr +running; `workpc-*` herdrs were connecting fine but main.go never logs a +success, only a failure, so there was no positive signal either way. Also +found: **a real task is stuck live right now** — workspace `wA`, task +`06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode agent, pane `wA:p1`, `agent_status: +"blocked"` — almost certainly stuck because `Release`/rotation could never +reach it (see below). + +Ran the actual Phase 0 steps against this live instance (raw JSON-RPC probes +over TCP, params-omitted/empty-object tricks to read Rust serde's +missing-field errors — no `herdr` CLI available locally, so `herdr api schema +--json` itself wasn't run, but the equivalent info was extracted this way). +Full method list and findings committed to `deploy/herdr-schema.json`. + +**Confirmed, with a real server response, not just static reading of +adapter.go:** +- `pane.release`, `pane.kill`, `pane.rotation_signal` — **none of these exist** + in the real protocol. Confirms B5's suspicion exactly. +- Real replacement for `pane.kill` is `pane.close({pane_id})` — same shape, + drop-in. **Fixed** in `internal/herdr/adapter.go`. +- Real replacement for `pane.release` is `pane.release_agent({pane_id, + source, agent})` — structurally different, and per B5's own analysis it + cannot return a `handoff_ref` regardless (herdr doesn't write handoffs, the + agent does, §6.1). Wiring this for real needs Phase 4's handoff-production + path first. `CLIAdapter.Release` now returns a loud error naming exactly + that instead of calling a method that doesn't exist. **Not a full fix** — + Phase 4 still owns making Release do something real. +- `pane.rotation_signal` doesn't exist and never will (herdr has no rotation + concept) — deleted `RotationSignal` interface, its `CLIAdapter` method, and + the call site in `Coordinator.rotate`, per this doc's own instruction + ("Delete ... unless the schema proves it exists"). +- `agent.get`, `pane.read`, `agent.prompt`, `worktree.create`, `worktree.open`, + `agent.start` — all confirmed real, no changes needed there. +- Protocol version confirmed live: `17`, matching `config.jsonc`'s + `"protocol": "17"` (returned as a bare JSON number by the server; the + existing string-fallback parse in `CheckProtocol` happens to handle that + correctly already). + +`go build ./...`, `go vet ./...`, `go test ./...` all pass after these +changes. + +**Still open from B5** (not attempted this pass — larger, needs design, not +just a method-name swap): +- `agent.prompt` inline `wait` on `CLIAdapter.Lease` (still `wait=0`, per B5's + note that only `Bootstrap` passes a real wait). +- `Release`'s real implementation, which depends on Phase 4 (§6) handoff + production existing at all. +- The stuck live task (`06FT6CKD9Y98AZRX6X8K3QXFZG`) was deliberately **not** + manipulated directly (no `pane.close`/`pane.release_agent` call against it) + — killing or releasing a real running agent from an audit session without + the user present is exactly the kind of action that warrants asking first. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f1c07f9 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,112 @@ +# Orchestra + +A Go implementation of `orchestra-spec (1).md` — an unattended multi-agent +task orchestrator that leases coding tasks to CLI harnesses (Claude Code, +Codex, opencode) running inside `herdr`-managed panes, rotates them across +context-window limits, and hands off work via a git-anchored continuity +protocol. + +Layout: `internal/{domain,store,provider,registry,router,herdr,orchestrator, +continuity,federation,delivery,authz,operations,admin}` + `cmd/orchestra/main.go`. + +## Ground truth over documentation + +This repo has a documented history of code that *looks* wired but isn't — +packages with tests that pass in isolation while the live call path silently +no-ops (bare `continue` on error, discarded return values). See `AUDIT.md` +for the full audit and `progress.md` for a running log. **Before trusting a +claim in progress.md that something "works" or "is fixed," check the actual +call site** — the file is written by past sessions of this same assistant and +has previously overstated completion. + +The single most reliable way to verify herdr-adapter code is right: don't +read `internal/herdr/adapter.go` and assume the method names are real. Ping +the live herdr instance and check. + +## herdr protocol — verified against a live instance, 2026-07-27 + +- herdr speaks JSON-RPC over a raw TCP (or unix-socket) connection — **not + HTTP**. `internal/herdr/herdr.go`'s `Client.Call` is the only correct way + to talk to it; a bare `curl` to the port returns nothing. +- Request shape: `{"id":"","method":"","params":}`. Herdr's + Rust JSON-RPC decoder requires `params` to be present and rejects a bare + `null` — always send `{}` for parameterless calls (the client does this + automatically). +- Full real method list is committed at `deploy/herdr-schema.json`, captured + live from `192.168.1.105:9245` (the `workpc` herdr) since no local `herdr` + CLI is available in this sandbox — the schema was reconstructed by sending + an unknown method name and reading the `unknown variant ... expected one + of ...` error, then probing each method of interest with `params:{}` / + `params:{pane_id:"nonexistent"}` to read Rust serde's `missing field + ` errors for its param shape. +- **Confirmed invented (do not use, they don't exist):** `pane.release`, + `pane.kill`, `pane.rotation_signal`, `pane.status`. If you see these + anywhere, it's a bug, not a valid call. +- **Real replacements:** `pane.close({pane_id})` for kill; + `pane.release_agent({pane_id, source, agent})` for release (structurally + different — does *not* return a `handoff_ref`, see below). No replacement + exists for `rotation_signal` — herdr has no concept of Orchestra rotation. +- **Architectural point that's easy to get wrong:** herdr never produces a + handoff. The agent writes the handoff artifact (§6.1 of the spec); herdr's + role in "release" is only to drop its own claim on the pane/agent binding. + Any adapter code that expects herdr to hand back a `handoff_ref` is wrong + by construction, independent of whether the method name is right. +- Protocol version is returned as a **JSON number** (`17`), not a string, + even though `config.jsonc` declares `"protocol": "17"` as a string. + `CheckProtocol`'s raw-bytes fallback happens to make this compare correctly + today — don't "clean up" that code without checking this note first, or it + might start doing a real numeric-vs-string comparison and break. + +## Deployment topology (as of 2026-07-27) + +- Runs as `orchestra.service` on **homesrv** (this machine's own systemd — + `journalctl -u orchestra.service` for logs; `sudo` isn't available in this + sandbox environment, but plain `journalctl` without sudo works here). +- Config: `.orchestra-config/orchestra.env` (env vars) + + `.orchestra-config/config.jsonc` (projects/machines/herdrs registry). + `ORCHESTRA_CONFIG=/etc/orchestra/config.jsonc` — the deployed copy, not the + repo's `deploy/config.example.jsonc`. +- Two machines in the registry: `homesrv` (192.168.1.104) and `workpc` + (192.168.1.105), each nominally running 3 herdrs (claude/codex/opencode). + In practice **homesrv has no local herdr running** (connection refused on + 9245) — only workpc's herdr is live and reachable. `main.go` only logs + herdr connection *failures* at startup, never successes, so "no log line" + for a herdr does not mean it's down — check reachability directly. +- There was a real, live, stuck task as of 2026-07-27: workspace `wA`, task + id `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode harness, pane `wA:p1`, + `agent_status: "blocked"`. Likely stuck because rotation/release could + never reach it (B2/B5). Check whether it's still stuck before assuming + fixes here have taken effect operationally — code fixes don't retroactively + unstick an already-orphaned pane; that needs a manual kill/restart once the + release path is trustworthy. + +## The federation fork — read before touching anything cross-machine + +Two incompatible designs coexist. **Design A** ("drive the remote socket", +currently deployed via `clients/herdr-bridge.go`) has homesrv call +`worktree.create`/`agent.start` etc. directly on workpc's herdr over TCP as +if it were local — meaning anchor validation (`git rev-parse HEAD`) run by +the coordinator executes on the *wrong machine* relative to the actual +checkout. **Design B** ("workers pull tasks", `/v1/federation/*`) is fully +built server-side but has zero clients — no worker binary exists. Decision +(AUDIT.md, 2026-07-27): keep Design A through Phase 5, commit to Design B in +Phase 6, with two guardrails landed immediately (refuse to rotate/cleanup a +lease held by a non-local herdr rather than validate against the wrong +checkout). Don't build on top of Design A's cross-machine calls without +reading that section first. + +## Working conventions + +- `go build ./...`, `go vet ./...`, and `go test ./...` must all pass — `go + vet` was broken for a while (duplicate JSON struct tags) and nobody + noticed because only `build`/`test` were being checked. Always run all + three. +- Silent `continue`-on-error is the recurring bug pattern in this codebase + (adapter lookups, rotation, expiry). When touching `internal/orchestrator` + or `internal/herdr`, prefer a recorded/observable failure + (`MonitorHealth` fields) over a bare `continue` — that's literally what + turned B1/B2 invisible for as long as they were. +- Don't invoke destructive herdr calls (`pane.close`, `pane.release_agent`) + against a real pane from an investigative/audit session without asking + first — there is live operator state on the other end (see the stuck-task + note above). diff --git a/deploy/herdr-schema.json b/deploy/herdr-schema.json new file mode 100644 index 0000000..1910a96 --- /dev/null +++ b/deploy/herdr-schema.json @@ -0,0 +1,33 @@ +{ + "_source": "captured live from herdr instance at 192.168.1.105:9245 (workpc), 2026-07-27, via raw ping/schema-discovery probes over the JSON-RPC-over-TCP protocol — see AUDIT.md Phase 0", + "ping_result": {"type": "pong", "version": "0.7.5", "protocol": 17, "capabilities": {"live_handoff": true, "detached_server_daemon": true}}, + "methods": [ + "ping", "server.stop", "server.live_handoff", "server.reload_config", "server.agent_manifests", "server.reload_agent_manifests", + "notification.show", "client.window_title.set", "client.window_title.clear", + "session.snapshot", + "workspace.create", "workspace.list", "workspace.get", "workspace.focus", "workspace.rename", "workspace.move", "workspace.report_metadata", "workspace.close", + "worktree.list", "worktree.create", "worktree.open", "worktree.remove", + "tab.create", "tab.list", "tab.get", "tab.focus", "tab.rename", "tab.move", "tab.close", + "agent.list", "agent.get", "agent.read", "agent.explain", "agent.send_keys", "agent.rename", "agent.view.set", "agent.view.clear", "agent.focus", "agent.start", "agent.prompt", "agent.wait", + "pane.split", "pane.swap", "pane.move", "pane.zoom", "pane.layout", "pane.process_info", + "layout.export", "layout.apply", "layout.set_split_ratio", + "pane.neighbor", "pane.edges", "pane.focus_direction", "pane.resize", "pane.list", "pane.current", "pane.get", "pane.focus", "pane.rename", + "pane.send_text", "pane.send_keys", "pane.send_input", "pane.read", + "pane.graphics.set", "pane.graphics.clear", "pane.graphics.info", "pane.graphics.stream", + "pane.report_agent", "pane.report_agent_session", "pane.report_metadata", "pane.clear_agent_authority", "pane.release_agent", "pane.close", + "popup.close", + "events.subscribe", "events.wait", "pane.wait_for_output", + "integration.install", "integration.uninstall", + "plugin.link", "plugin.list", "plugin.unlink", "plugin.enable", "plugin.disable", "plugin.action.list", "plugin.action.invoke", "plugin.log.list", "plugin.pane.open", "plugin.pane.focus", "plugin.pane.close" + ], + "confirmed_invented_methods_not_in_schema": ["pane.release", "pane.kill", "pane.rotation_signal", "pane.status"], + "confirmed_real_replacements": { + "pane.kill": "pane.close (params: {pane_id})", + "pane.release": "pane.release_agent (params: {pane_id, source, agent} — does NOT return handoff_ref; matches agent_session shape from agent.list: {source, agent, kind, value})", + "pane.rotation_signal": "no equivalent exists; herdr has no concept of Orchestra rotation" + }, + "params_confirmed_by_probing": { + "pane.close": {"pane_id": "string"}, + "pane.release_agent": {"pane_id": "string", "source": "string (e.g. \"herdr:opencode\")", "agent": "string (e.g. \"opencode\")"} + } +} diff --git a/internal/herdr/adapter.go b/internal/herdr/adapter.go index d517a8d..829c3b2 100644 --- a/internal/herdr/adapter.go +++ b/internal/herdr/adapter.go @@ -26,9 +26,6 @@ type WorktreeCreator interface { type TurnBoundary interface { AtTurnBoundary(context.Context, Session) (bool, error) } -type RotationSignal interface { - RotationSignal(context.Context, Session) (string, error) -} type PaneExit interface { PaneExited(context.Context, Session) (bool, error) } @@ -78,15 +75,21 @@ func (a CLIAdapter) Lease(ctx context.Context, task, worktree string) (Session, func (a CLIAdapter) Bootstrap(ctx context.Context, s Session, ref string) error { return a.Client.Prompt(ctx, s.PaneID, fmt.Sprintf("Read handoff %s, validate the anchor and TASK.md, then continue.", ref), time.Minute) } +// Release previously called the invented "pane.release" method expecting a +// handoff_ref back. Neither exists in the real protocol (confirmed against a +// live herdr instance, AUDIT.md Phase 0): the real method is +// pane.release_agent(pane_id, source, agent), which only releases herdr's +// claim on the agent session — it cannot return a handoff_ref, because herdr +// does not write handoffs, the agent does (§6.1). Wiring this correctly needs +// the Phase 4 handoff-production path (agent writes handoff, stop hook +// uploads it to CAS, plane validates and mints the ref) before Release has +// anything real to return. Refusing loudly until then rather than calling a +// method that doesn't exist. func (a CLIAdapter) Release(ctx context.Context, s Session) (string, error) { - var r struct { - Ref string `json:"handoff_ref"` - } - e := a.Client.Call(ctx, "pane.release", s, &r) - return r.Ref, e + return "", fmt.Errorf("adapter: Release not implemented — pane.release is not a real herdr method and handoff production (AUDIT.md Phase 4) is not wired yet") } func (a CLIAdapter) Kill(ctx context.Context, s Session) error { - return a.Client.Call(ctx, "pane.kill", s, nil) + return a.Client.Call(ctx, "pane.close", map[string]any{"pane_id": s.PaneID}, nil) } func (a CLIAdapter) AtTurnBoundary(ctx context.Context, s Session) (bool, error) { status, err := a.AgentStatus(ctx, s) @@ -179,15 +182,6 @@ func statusFromAgentResult(v any) string { var _ = json.RawMessage{} -func (a CLIAdapter) RotationSignal(ctx context.Context, s Session) (string, error) { - var r struct { - Reason string `json:"reason"` - } - if err := a.Client.Call(ctx, "pane.rotation_signal", s, &r); err != nil { - return "", err - } - return r.Reason, nil -} // Occupancy reads the harness's own session state — never the herdr pane id, // which ClaudeUsage/CodexUsage/OpenCodeUsage cannot open (spec §5.2.1: "the // whole rotation system rests on this number"). A session file that cannot diff --git a/internal/orchestrator/orchestrator.go b/internal/orchestrator/orchestrator.go index 36e291c..aab624a 100644 --- a/internal/orchestrator/orchestrator.go +++ b/internal/orchestrator/orchestrator.go @@ -451,13 +451,8 @@ func (c *Coordinator) rotate(ctx context.Context, hard float64) { continue } reason := "threshold" - if signal, ok := a.(herdr.RotationSignal); ok { - if r, signalErr := signal.RotationSignal(ctx, session); signalErr == nil && r != "" { - reason = r - } - } occupancy, err := a.Occupancy(session) - if err != nil || (occupancy < hard && reason == "threshold") { + if err != nil || occupancy < hard { continue } // Face B is treated as required, not best-effort (spec §5.2/§5.3): diff --git a/progress.md b/progress.md index da51595..faa31ae 100644 --- a/progress.md +++ b/progress.md @@ -52,10 +52,24 @@ Fixed so far: polling would error out of its scan loop on the first already-ingested issue in every batch. -Not yet started: B3 (no `TaskCompleted` producer / no Stop hook), B5/B6 -(herdr protocol verification, Layer 3 wiring), B7 (quota projection has no -producer), S2–S4, S7–S11, and the Phase 0 live-herdr verification step that -several of the above still need. See `AUDIT.md` for the full plan. +Not yet started: B3 (no `TaskCompleted` producer / no Stop hook), B6 (Layer 3 +wiring), B7 (quota projection has no producer), S2–S4, S7–S11. See +`AUDIT.md` for the full plan. + +**Phase 0 done (2026-07-27):** this box has live TCP reachability to the real +herdr instance at `192.168.1.105:9245` — verified by hand (raw JSON-RPC +probes, no `herdr` CLI available locally). Real method list captured in +`deploy/herdr-schema.json`. Confirmed `pane.release`/`pane.kill`/ +`pane.rotation_signal` are invented, as AUDIT.md's B5 suspected. +`pane.kill`→`pane.close` fixed as a drop-in. `pane.rotation_signal`/ +`RotationSignal` deleted (no replacement exists). `Release` now refuses +loudly instead of calling a nonexistent method — its real implementation +needs Phase 4 (handoff production) first, since even the real +`pane.release_agent` can't return a `handoff_ref` (herdr doesn't write +handoffs, the agent does). See AUDIT.md's new "Phase 0 — done" section for +full detail. **Also found: a real task is currently stuck live** — workspace +`wA`, task `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode, pane `wA:p1`, blocked — +deliberately not touched from this session. ## Current state