kami
97a9c65302
Acts on the seven review comments on PR #1. Design A is gone (comment 4). clients/ deleted rather than tracked: with workers carrying cross-machine work the bridge is undeployed, which supersedes the 2026-07-27 "keep through Phase 5" decision. CLAUDE.md, AGENTS.md and AUDIT.md updated from "retained" to "deleted". The harness-hook completion path is gone (comment 10). Investigation of the live OpenCode QA run showed orchestra-worker owns completion end to end: it watches for .orchestra/done, confirms via AgentStatus that the agent is not busy, then posts through /v1/federation/* with both lease epoch and expected version. The hook scripts used a different, older convention (.orchestra-report.md) and posted to /v1/harness/complete, which had already been reduced to a 410 stub - so that path could not have completed a task. Nothing exercised it, because the live run never used it. Deleted: the three deploy/hooks scripts, the 410 route, the unmounted harnessCompletion handler, and its test. That test passed against a handler no mux routed to, which is the exact "looks wired but isn't" pattern CLAUDE.md warns about; the constant-time token compare added to it earlier today goes with it, having never been reachable. /v1/harness/turn is untouched and still live. Retired deployment files (comments 8, 12, 14): deploy/orchestra.service and deploy/redeploy.sh (which sudo-installed to /usr/local/bin and restarted that unit), plus deploy/docker-api-entrypoint.sh. The entrypoint was safe to remove once its premise was checked: env vars reach the container through `env_file: .env` in compose.yaml, not by sourcing /etc/orchestra/orchestra.env - only config.jsonc is bind-mounted there - and Dockerfile.api's line 17 already sets ORCHESTRA_DATA/ORCHESTRA_PORT. Dockerfile.api now execs /app/orchestra directly. orchestra-worker.service is a different, current unit and is kept. deploy/config.example.json deleted as a duplicate (comment 6); the annotated .jsonc is the one registry.go points at, and its header no longer tells the reader to copy the file that just went away. Documentation corrected beyond the deletions: - CLAUDE.md's deployment section claimed the container bind-mounts /etc/orchestra:ro and its entrypoint sources the env file. Both wrong. - AGENTS.md still described a systemd deployment on homesrv as of 2026-07-27. - AUDIT.md's H5 row still described a "retained compatibility handler". - deploy/DEPLOYMENT.md still named redeploy.sh as the deployment path. - deploy/orchestra.env.example still cited EnvironmentFile=. TOKEN_MINIMAL_WORKFLOW_PLAN.md (comment 2) is untouched: it and WEB_UI_PLAN.md were both missed by REVIEW.md's documentation sweep, and reconciling a 534-line forward-looking plan against AUDIT.md is its own task, not a review fixup. Verified: go build ./..., go vet ./..., go test ./... all pass after the deletions, and go list ./... has no node_modules entry. No live herdr or pane was touched; nothing was deployed. The running image still predates this commit until compose is rebuilt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GEugbHVYfAXFpTqDYbByEB
143 lines
8.3 KiB
Markdown
143 lines
8.3 KiB
Markdown
# Orchestra
|
|
|
|
A Go implementation of `orchestra-spec (1).md` — an unattended multi-agent
|
|
task orchestrator that leases coding tasks to CLI harnesses (Claude Code,
|
|
Codex, opencode) running inside `herdr`-managed panes, rotates them across
|
|
context-window limits, and hands off work via a git-anchored continuity
|
|
protocol.
|
|
|
|
Layout: `internal/{domain,store,provider,registry,router,herdr,orchestrator,
|
|
continuity,federation,delivery,authz,operations,admin}` + `cmd/orchestra/main.go`.
|
|
|
|
## Ground truth over documentation
|
|
|
|
This repo has a documented history of code that *looks* wired but isn't —
|
|
packages with tests that pass in isolation while the live call path silently
|
|
no-ops (bare `continue` on error, discarded return values). See `AUDIT.md`
|
|
for the full audit; it is also the running log (there is no separate log
|
|
file). **Before trusting a claim in `AUDIT.md` that something "works"
|
|
or "is fixed," check the actual call site** — the file is written by past
|
|
sessions of this same assistant and has previously overstated completion.
|
|
|
|
The single most reliable way to verify herdr-adapter code is right: don't
|
|
read `internal/herdr/adapter.go` and assume the method names are real. Ping
|
|
the live herdr instance and check.
|
|
|
|
## herdr protocol — verified against a live instance, 2026-07-27
|
|
|
|
- herdr speaks JSON-RPC over a raw TCP (or unix-socket) connection — **not
|
|
HTTP**. `internal/herdr/herdr.go`'s `Client.Call` is the only correct way
|
|
to talk to it; a bare `curl` to the port returns nothing.
|
|
- Request shape: `{"id":"<n>","method":"<name>","params":<object>}`. Herdr's
|
|
Rust JSON-RPC decoder requires `params` to be present and rejects a bare
|
|
`null` — always send `{}` for parameterless calls (the client does this
|
|
automatically).
|
|
- Full real method list is committed at `deploy/herdr-schema.json`, captured
|
|
live from `192.168.1.105:9245` (the `workpc` herdr) since no local `herdr`
|
|
CLI is available in this sandbox — the schema was reconstructed by sending
|
|
an unknown method name and reading the `unknown variant ... expected one
|
|
of ...` error, then probing each method of interest with `params:{}` /
|
|
`params:{pane_id:"nonexistent"}` to read Rust serde's `missing field
|
|
<x>` errors for its param shape.
|
|
- **Confirmed invented (do not use, they don't exist):** `pane.release`,
|
|
`pane.kill`, `pane.rotation_signal`, `pane.status`. If you see these
|
|
anywhere, it's a bug, not a valid call.
|
|
- **Real replacements:** `pane.close({pane_id})` for kill;
|
|
`pane.release_agent({pane_id, source, agent})` for release (structurally
|
|
different — does *not* return a `handoff_ref`, see below). No replacement
|
|
exists for `rotation_signal` — herdr has no concept of Orchestra rotation.
|
|
- **Architectural point that's easy to get wrong:** herdr never produces a
|
|
handoff. The agent writes the handoff artifact (§6.1 of the spec); herdr's
|
|
role in "release" is only to drop its own claim on the pane/agent binding.
|
|
Any adapter code that expects herdr to hand back a `handoff_ref` is wrong
|
|
by construction, independent of whether the method name is right.
|
|
- Protocol version is returned as a **JSON number** (`17`), not a string,
|
|
even though `config.jsonc` declares `"protocol": "17"` as a string.
|
|
`CheckProtocol`'s raw-bytes fallback happens to make this compare correctly
|
|
today — don't "clean up" that code without checking this note first, or it
|
|
might start doing a real numeric-vs-string comparison and break.
|
|
|
|
## Deployment topology (as of 2026-07-31)
|
|
|
|
- **Runs under Docker Compose, not systemd.** `docker compose -f compose.yaml
|
|
-f compose.live.yaml` in `/home/kami/docker-apps/orchestra-web-ui`, building
|
|
both images from this repo: `orchestra-api` (bound `0.0.0.0:9145`, which is
|
|
intentional — ufw restricts the port to one other LAN machine) and
|
|
`orchestra-web-ui` (nginx proxy, `127.0.0.1:19145`). Logs are
|
|
`docker logs orchestra-api`. **Deploying a code change means rebuilding the
|
|
compose images** (`up -d --build`) — the running image can silently predate
|
|
recent commits, so compare its build time against `git log`.
|
|
- `orchestra.service` was the previous deployment; its unit file and
|
|
`redeploy.sh` were deleted from `deploy/` on 2026-07-31. A stale installed
|
|
copy must stay stopped — it binds the same port and data dir as the
|
|
container. `orchestra-worker.service` is a *different*, still-current unit.
|
|
- Config: env vars come from **`.env` in the compose directory** via
|
|
`env_file:`; only `config.jsonc` is bind-mounted into `/etc/orchestra/`.
|
|
There is no container entrypoint script — `Dockerfile.api` execs
|
|
`/app/orchestra` directly. Neither deployed file is the repo's
|
|
`deploy/config.example.jsonc`.
|
|
- Two machines in the registry: `homesrv` (192.168.1.104) and `workpc`
|
|
(192.168.1.105), each nominally running 3 herdrs (claude/codex/opencode).
|
|
In practice **homesrv has no local herdr running** (connection refused on
|
|
9245) — only workpc's herdr is live and reachable. `main.go` only logs
|
|
herdr connection *failures* at startup, never successes, so "no log line"
|
|
for a herdr does not mean it's down — check reachability directly.
|
|
- There was a real, live, stuck task as of 2026-07-27: workspace `wA`, task
|
|
id `06FT6CKD9Y98AZRX6X8K3QXFZG`, opencode harness, pane `wA:p1`,
|
|
`agent_status: "blocked"`. Likely stuck because rotation/release could
|
|
never reach it (B2/B5). Check whether it's still stuck before assuming
|
|
fixes here have taken effect operationally — code fixes don't retroactively
|
|
unstick an already-orphaned pane; that needs a manual kill/restart once the
|
|
release path is trustworthy.
|
|
|
|
## Federation — Design B is the live design (as of 2026-07-30)
|
|
|
|
**Design B** ("workers pull tasks", `/v1/federation/*`) is the live design and
|
|
has a real client: `cmd/orchestra-worker/main.go` (~1,131 lines, with tests in
|
|
`cmd/orchestra-worker/main_test.go`) is the deployed worker — the workpc
|
|
OpenCode worker runs it. Build new cross-machine work on Design B.
|
|
|
|
The **Design A guardrail has landed**: `Coordinator.adapterFor`
|
|
(`internal/orchestrator/orchestrator.go`, the `LocalHerdr` check) refuses to
|
|
resolve an adapter for a session owned by a non-local herdr, returning
|
|
`session %s is owned by non-local herdr %s` instead of validating a git anchor
|
|
(`git rev-parse HEAD`) against the wrong machine's checkout. Rotation/cleanup
|
|
therefore no longer act on remote leases.
|
|
|
|
**Design A is gone (deleted 2026-07-31).** `clients/herdr-bridge.go` ("drive
|
|
the remote socket": homesrv calling `worktree.create`/`agent.start` directly on
|
|
workpc's herdr over TCP as if it were local) was deleted along with the whole
|
|
`clients/` directory — the operator confirmed it is undeployed now that workers
|
|
carry cross-machine work, which superseded the 2026-07-27 "keep through Phase
|
|
5" decision. There is no bridge to preserve; do not reintroduce
|
|
coordinator-side calls to a remote herdr socket.
|
|
|
|
**Completion is worker-owned.** `orchestra-worker` watches for an
|
|
`.orchestra/done` marker, confirms via `AgentStatus` that the agent is no
|
|
longer busy, then posts through `/v1/federation/*` with the lease epoch and
|
|
expected version. The old harness-hook path — `.orchestra-report.md` plus
|
|
`POST /v1/harness/complete` — is **deleted**, endpoint, handler, and
|
|
`deploy/hooks/` scripts alike. `/v1/harness/turn` remains for turn-boundary
|
|
decisions.
|
|
|
|
## Working conventions
|
|
|
|
- **workpc worker deployment target:** copy the built worker binary to
|
|
`workpc:~/orchestra-deploy/orchestra-worker` (that is,
|
|
`/home/kami/orchestra-deploy/orchestra-worker`), not directly to
|
|
`/usr/local/bin`. The workpc deployment process installs from this staging
|
|
path. Verify the remote checksum and Go build revision before restart.
|
|
- `go build ./...`, `go vet ./...`, and `go test ./...` must all pass — `go
|
|
vet` was broken for a while (duplicate JSON struct tags) and nobody
|
|
noticed because only `build`/`test` were being checked. Always run all
|
|
three.
|
|
- Silent `continue`-on-error is the recurring bug pattern in this codebase
|
|
(adapter lookups, rotation, expiry). When touching `internal/orchestrator`
|
|
or `internal/herdr`, prefer a recorded/observable failure
|
|
(`MonitorHealth` fields) over a bare `continue` — that's literally what
|
|
turned B1/B2 invisible for as long as they were.
|
|
- Don't invoke destructive herdr calls (`pane.close`, `pane.release_agent`)
|
|
against a real pane from an investigative/audit session without asking
|
|
first — there is live operator state on the other end (see the stuck-task
|
|
note above).
|