orchestra/AGENTS.md

Orchestra

A Go implementation of orchestra-spec (1).md — an unattended multi-agent task orchestrator that leases coding tasks to CLI harnesses (Claude Code, Codex, opencode) running inside herdr-managed panes, rotates them across context-window limits, and hands off work via a git-anchored continuity protocol.

Layout: internal/{domain,store,provider,registry,router,herdr,orchestrator, continuity,federation,delivery,authz,operations,admin} + cmd/orchestra/main.go.

Ground truth over documentation

This repo has a documented history of code that looks wired but isn't — packages with tests that pass in isolation while the live call path silently no-ops (bare continue on error, discarded return values). See AUDIT.md for the full audit; it is also the running log (there is no separate log file). Before trusting a claim in AUDIT.md that something "works" or "is fixed," check the actual call site — the file is written by past sessions of this same assistant and has previously overstated completion.

The single most reliable way to verify herdr-adapter code is right: don't read internal/herdr/adapter.go and assume the method names are real. Ping the live herdr instance and check.

herdr protocol — verified against a live instance, 2026-07-27

• herdr speaks JSON-RPC over a raw TCP (or unix-socket) connection — not HTTP. internal/herdr/herdr.go's Client.Call is the only correct way to talk to it; a bare curl to the port returns nothing.
• Request shape: {"id":"<n>","method":"<name>","params":<object>}. Herdr's Rust JSON-RPC decoder requires params to be present and rejects a bare null — always send {} for parameterless calls (the client does this automatically).
• Full real method list is committed at deploy/herdr-schema.json, captured live from 192.168.1.105:9245 (the workpc herdr) since no local herdr CLI is available in this sandbox — the schema was reconstructed by sending an unknown method name and reading the unknown variant ... expected one of ... error, then probing each method of interest with params:{} / params:{pane_id:"nonexistent"} to read Rust serde's missing field <x> errors for its param shape.
• Confirmed invented (do not use, they don't exist): pane.release, pane.kill, pane.rotation_signal, pane.status. If you see these anywhere, it's a bug, not a valid call.
• Real replacements: pane.close({pane_id}) for kill; pane.release_agent({pane_id, source, agent}) for release (structurally different — does not return a handoff_ref, see below). No replacement exists for rotation_signal — herdr has no concept of Orchestra rotation.
• Architectural point that's easy to get wrong: herdr never produces a handoff. The agent writes the handoff artifact (§6.1 of the spec); herdr's role in "release" is only to drop its own claim on the pane/agent binding. Any adapter code that expects herdr to hand back a handoff_ref is wrong by construction, independent of whether the method name is right.
• Protocol version is returned as a JSON number (17), not a string, even though config.jsonc declares "protocol": "17" as a string. CheckProtocol's raw-bytes fallback happens to make this compare correctly today — don't "clean up" that code without checking this note first, or it might start doing a real numeric-vs-string comparison and break.

Deployment topology (as of 2026-07-27)

• Runs as orchestra.service on homesrv (this machine's own systemd — journalctl -u orchestra.service for logs; sudo isn't available in this sandbox environment, but plain journalctl without sudo works here).
• Config: .orchestra-config/orchestra.env (env vars) + .orchestra-config/config.jsonc (projects/machines/herdrs registry). ORCHESTRA_CONFIG=/etc/orchestra/config.jsonc — the deployed copy, not the repo's deploy/config.example.jsonc.
• Two machines in the registry: homesrv (192.168.1.104) and workpc (192.168.1.105), each nominally running 3 herdrs (claude/codex/opencode). In practice homesrv has no local herdr running (connection refused on 9245) — only workpc's herdr is live and reachable. main.go only logs herdr connection failures at startup, never successes, so "no log line" for a herdr does not mean it's down — check reachability directly.
• There was a real, live, stuck task as of 2026-07-27: workspace wA, task id 06FT6CKD9Y98AZRX6X8K3QXFZG, opencode harness, pane wA:p1, agent_status: "blocked". Likely stuck because rotation/release could never reach it (B2/B5). Check whether it's still stuck before assuming fixes here have taken effect operationally — code fixes don't retroactively unstick an already-orphaned pane; that needs a manual kill/restart once the release path is trustworthy.

Federation — Design B is the live design (as of 2026-07-30)

Design B ("workers pull tasks", /v1/federation/*) is the live design and has a real client: cmd/orchestra-worker/main.go (~1,131 lines, with tests in cmd/orchestra-worker/main_test.go) is the deployed worker — the workpc OpenCode worker runs it. Build new cross-machine work on Design B.

The Design A guardrail has landed: Coordinator.adapterFor (internal/orchestrator/orchestrator.go, the LocalHerdr check) refuses to resolve an adapter for a session owned by a non-local herdr, returning session %s is owned by non-local herdr %s instead of validating a git anchor (git rev-parse HEAD) against the wrong machine's checkout. Rotation/cleanup therefore no longer act on remote leases.

Design A is retained, not dead. clients/herdr-bridge.go ("drive the remote socket": homesrv calling worktree.create/agent.start directly on workpc's herdr over TCP as if it were local) is kept through Phase 5 per the standing AUDIT.md decision of 2026-07-27; deletion of Design A and clients/ is deferred to the Phase 6 cutover. Do not delete it early, and do not add new cross-machine call paths to it.

Working conventions

• workpc worker deployment target: copy the built worker binary to workpc:~/orchestra-deploy/orchestra-worker (that is, /home/kami/orchestra-deploy/orchestra-worker), not directly to /usr/local/bin. The workpc deployment process installs from this staging path. Verify the remote checksum and Go build revision before restart.
• go build ./..., go vet ./..., and go test ./... must all pass — go vet was broken for a while (duplicate JSON struct tags) and nobody noticed because only build/test were being checked. Always run all three.
• Silent continue-on-error is the recurring bug pattern in this codebase (adapter lookups, rotation, expiry). When touching internal/orchestrator or internal/herdr, prefer a recorded/observable failure (MonitorHealth fields) over a bare continue — that's literally what turned B1/B2 invisible for as long as they were.
• Don't invoke destructive herdr calls (pane.close, pane.release_agent) against a real pane from an investigative/audit session without asking first — there is live operator state on the other end (see the stuck-task note above).