kami
56f5aac582
Acts on the 2026-07-30 senior review (REVIEW.md findings 1, 2, 4, 5, 7).
Docs (finding 1): CLAUDE.md and AGENTS.md both claimed Design B "has zero
clients - no worker binary exists". cmd/orchestra-worker/main.go is the
deployed worker, and the non-local-herdr guardrail has landed in
Coordinator.adapterFor. Both sections rewritten; AUDIT.md gains a matching
federation-status record. The Phase 5 retention / Phase 6 deletion decision
for Design A is preserved, not flattened.
clients/ un-ignored and tracked, including the .service unit and README:
deployed code belongs in version control. Design A is NOT deleted here.
progress.md (finding 2): the file was deleted after 636ed8a, yet CLAUDE.md
instructed every session to cross-check against it. References removed from
CLAUDE.md, AGENTS.md, internal/orchestrator/rotation_test.go (comment only)
and deploy/hooks/orchestra-codex-poll.sh; AUDIT.md now carries the log role.
web/go.mod (finding 4): a module stub ends the parent package graph at the
directory boundary, so go list ./... no longer yields
web/node_modules/flatted/golang/pkg/flatted. A build tag cannot work - the
package is in the package list before tags are evaluated. Local/CI-only
breakage: Dockerfile.api builds ./cmd/orchestra by explicit path and
.dockerignore already excluded node_modules.
orchestra-worker (finding 5): untracked (8.9MB, mode 100755, still on disk);
both binaries now gitignored.
Token compare (finding 7): cmd/orchestra/main.go:139,582 use
subtle.ConstantTimeCompare, matching the authz.go idiom. The token != ""
guard stays first, so an empty configured token still means auth-disabled
rather than auth-bypass. Three further plain != secret compares remain in
internal/federation/federation.go:343,346,368 - tracked, not fixed here.
Also included from the review pass: orchestrator.go records adapter-resolution
failures in SessionHealth.LastError instead of dropping them on a bare
continue, plus an Observed flag so lease-seeded health is not mistaken for a
live reading, with a covering test. GET /v1/tasks/<id>/health now returns a
record with last_error where it previously returned a bare 404.
REVIEW.md's own second pass claimed every checkable fact held up; four did
not. AUDIT.md never contained the false Design B claim (AGENTS.md was the
second copy), the guardrail is at orchestrator.go:312 not :309, the
progress.md site list missed the codex-poll hook, and only orchestra-worker
was tracked. Verified: go build, go vet, go test, and
go list ./... | grep node_modules all clean with every change applied
together. No live herdr or pane was touched; nothing was deployed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GEugbHVYfAXFpTqDYbByEB
124 lines
7.1 KiB
Markdown
124 lines
7.1 KiB
Markdown
# Orchestra
|
|
|
|
A Go implementation of `orchestra-spec (1).md` — an unattended multi-agent
|
|
task orchestrator that leases coding tasks to CLI harnesses (Claude Code,
|
|
Codex, opencode) running inside `herdr`-managed panes, rotates them across
|
|
context-window limits, and hands off work via a git-anchored continuity
|
|
protocol.
|
|
|
|
Layout: `internal/{domain,store,provider,registry,router,herdr,orchestrator,
|
|
continuity,federation,delivery,authz,operations,admin}` + `cmd/orchestra/main.go`.
|
|
|
|
## Ground truth over documentation
|
|
|
|
This repo has a documented history of code that *looks* wired but isn't —
|
|
packages with tests that pass in isolation while the live call path silently
|
|
no-ops (bare `continue` on error, discarded return values). See `AUDIT.md`
|
|
for the full audit; it is also the running log (there is no separate log
|
|
file). **Before trusting a claim in `AUDIT.md` that something "works"
|
|
or "is fixed," check the actual call site** — the file is written by past
|
|
sessions of this same assistant and has previously overstated completion.
|
|
|
|
The single most reliable way to verify herdr-adapter code is right: don't
|
|
read `internal/herdr/adapter.go` and assume the method names are real. Ping
|
|
the live herdr instance and check.
|
|
|
|
## herdr protocol — verified against a live instance, 2026-07-27
|
|
|
|
- herdr speaks JSON-RPC over a raw TCP (or unix-socket) connection — **not
|
|
HTTP**. `internal/herdr/herdr.go`'s `Client.Call` is the only correct way
|
|
to talk to it; a bare `curl` to the port returns nothing.
|
|
- Request shape: `{"id":"<n>","method":"<name>","params":<object>}`. Herdr's
|
|
Rust JSON-RPC decoder requires `params` to be present and rejects a bare
|
|
`null` — always send `{}` for parameterless calls (the client does this
|
|
automatically).
|
|
- Full real method list is committed at `deploy/herdr-schema.json`, captured
|
|
live from `192.168.1.105:9245` (the `workpc` herdr) since no local `herdr`
|
|
CLI is available in this sandbox — the schema was reconstructed by sending
|
|
an unknown method name and reading the `unknown variant ... expected one
|
|
of ...` error, then probing each method of interest with `params:{}` /
|
|
`params:{pane_id:"nonexistent"}` to read Rust serde's `missing field
|
|
<x>` errors for its param shape.
|
|
- **Confirmed invented (do not use, they don't exist):** `pane.release`,
|
|
`pane.kill`, `pane.rotation_signal`, `pane.status`. If you see these
|
|
anywhere, it's a bug, not a valid call.
|
|
- **Real replacements:** `pane.close({pane_id})` for kill;
|
|
`pane.release_agent({pane_id, source, agent})` for release (structurally
|
|
different — does *not* return a `handoff_ref`, see below). No replacement
|
|
exists for `rotation_signal` — herdr has no concept of Orchestra rotation.
|
|
- **Architectural point that's easy to get wrong:** herdr never produces a
|
|
handoff. The agent writes the handoff artifact (§6.1 of the spec); herdr's
|
|
role in "release" is only to drop its own claim on the pane/agent binding.
|
|
Any adapter code that expects herdr to hand back a `handoff_ref` is wrong
|
|
by construction, independent of whether the method name is right.
|
|
- Protocol version is returned as a **JSON number** (`17`), not a string,
|
|
even though `config.jsonc` declares `"protocol": "17"` as a string.
|
|
`CheckProtocol`'s raw-bytes fallback happens to make this compare correctly
|
|
today — don't "clean up" that code without checking this note first, or it
|
|
might start doing a real numeric-vs-string comparison and break.
|
|
|
|
## Deployment topology (as of 2026-07-27)
|
|
|
|
- Runs as `orchestra.service` on **homesrv** (this machine's own systemd —
|
|
`journalctl -u orchestra.service` for logs; `sudo` isn't available in this
|
|
sandbox environment, but plain `journalctl` without sudo works here).
|
|
- Config: `.orchestra-config/orchestra.env` (env vars) +
|
|
`.orchestra-config/config.jsonc` (projects/machines/herdrs registry).
|
|
`ORCHESTRA_CONFIG=/etc/orchestra/config.jsonc` — the deployed copy, not the
|
|
repo's `deploy/config.example.jsonc`.
|
|
- Two machines in the registry: `homesrv` (192.168.1.104) and `workpc`
|
|
(192.168.1.105), each nominally running 3 herdrs (claude/codex/opencode).
|
|
In practice **homesrv has no local herdr running** (connection refused on
|
|
9245) — only workpc's herdr is live and reachable. `main.go` only logs
|
|
herdr connection *failures* at startup, never successes, so "no log line"
|
|
for a herdr does not mean it's down — check reachability directly.
|
|
- There was a real, live, stuck task as of 2026-07-27: workspace `wA`, task
|
|
id `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode harness, pane `wA:p1`,
|
|
`agent_status: "blocked"`. Likely stuck because rotation/release could
|
|
never reach it (B2/B5). Check whether it's still stuck before assuming
|
|
fixes here have taken effect operationally — code fixes don't retroactively
|
|
unstick an already-orphaned pane; that needs a manual kill/restart once the
|
|
release path is trustworthy.
|
|
|
|
## Federation — Design B is the live design (as of 2026-07-30)
|
|
|
|
**Design B** ("workers pull tasks", `/v1/federation/*`) is the live design and
|
|
has a real client: `cmd/orchestra-worker/main.go` (~1,131 lines, with tests in
|
|
`cmd/orchestra-worker/main_test.go`) is the deployed worker — the workpc
|
|
OpenCode worker runs it. Build new cross-machine work on Design B.
|
|
|
|
The **Design A guardrail has landed**: `Coordinator.adapterFor`
|
|
(`internal/orchestrator/orchestrator.go`, the `LocalHerdr` check) refuses to
|
|
resolve an adapter for a session owned by a non-local herdr, returning
|
|
`session %s is owned by non-local herdr %s` instead of validating a git anchor
|
|
(`git rev-parse HEAD`) against the wrong machine's checkout. Rotation/cleanup
|
|
therefore no longer act on remote leases.
|
|
|
|
**Design A is retained, not dead.** `clients/herdr-bridge.go` ("drive the
|
|
remote socket": homesrv calling `worktree.create`/`agent.start` directly on
|
|
workpc's herdr over TCP as if it were local) is kept through Phase 5 per the
|
|
standing AUDIT.md decision of 2026-07-27; deletion of Design A and `clients/`
|
|
is deferred to the Phase 6 cutover. Do not delete it early, and do not add new
|
|
cross-machine call paths to it.
|
|
|
|
## Working conventions
|
|
|
|
- **workpc worker deployment target:** copy the built worker binary to
|
|
`workpc:~/orchestra-deploy/orchestra-worker` (that is,
|
|
`/home/kami/orchestra-deploy/orchestra-worker`), not directly to
|
|
`/usr/local/bin`. The workpc deployment process installs from this staging
|
|
path. Verify the remote checksum and Go build revision before restart.
|
|
- `go build ./...`, `go vet ./...`, and `go test ./...` must all pass — `go
|
|
vet` was broken for a while (duplicate JSON struct tags) and nobody
|
|
noticed because only `build`/`test` were being checked. Always run all
|
|
three.
|
|
- Silent `continue`-on-error is the recurring bug pattern in this codebase
|
|
(adapter lookups, rotation, expiry). When touching `internal/orchestrator`
|
|
or `internal/herdr`, prefer a recorded/observable failure
|
|
(`MonitorHealth` fields) over a bare `continue` — that's literally what
|
|
turned B1/B2 invisible for as long as they were.
|
|
- Don't invoke destructive herdr calls (`pane.close`, `pane.release_agent`)
|
|
against a real pane from an investigative/audit session without asking
|
|
first — there is live operator state on the other end (see the stuck-task
|
|
note above).
|