checkpoint: multi-repo Gitea ingestion, per-project repos, rotation anchor_sha fix

Pre-existing uncommitted work found at session start: rotation now emits
anchor_sha on TaskReleased (previously silently dropped by store.Append
validation), multi-repo Gitea provider support, per-project git worktree
roots, and associated test coverage. Committing as a checkpoint before
starting remediation work tracked in AUDIT.md.
This commit is contained in:
kami
2026-07-27 18:15:02 +04:00
parent 325c684eb0
commit ce6f02f9e6
31 changed files with 2717 additions and 320 deletions
+5 -1
View File
@@ -1 +1,5 @@
# fill later
# Private, composed deployment configuration. Install into /etc/orchestra/
# only after all environment-specific values have been filled in.
.orchestra-config/
clients/
/orchestra
+475
View File
@@ -0,0 +1,475 @@
# Orchestra — spec conformance audit & remediation plan
Audited 2026-07-27 against `orchestra-spec (1).md` at commit `325c684`.
Method: read every non-test file in `internal/` and `cmd/`, traced each spec
section to its call site, and checked whether the live path (`main.go`
router → coordinator → adapter) actually reaches it.
---
## Verdict
The substrate (Layer 1) is genuinely built. Layers 24 are **shaped** but not
**wired**: the packages exist, have tests, and compile — but the code paths
that would make an unattended run work are either dead, mis-keyed, or built
against herdr methods that appear to be invented.
Concretely: **a task can be created, routed, and leased. It can never
complete, and it can never rotate.** After 3 lease expiries the router marks
it `TaskFailed`. That is the whole of "connection works but features are
underdeveloped."
`progress.md` overstates completion in several places (see §Corrections).
| Spec layer | State |
|---|---|
| L1 substrate (§3, §4) | Built and correct in the main path. Some defects (below), no missing mechanism. |
| L2 harness (§5) | **Not functional.** Occupancy broken, rotation unreachable, adapter protocol unverified. |
| L3 continuity (§6) | **Dead code.** Handoff schema, pickup validation, scratch branches, TASK.md — none reached from the live path. |
| L4 surfaces (§7) | Partial. Brief/standup/delivery exist; quota projection has no producer; authz has a bypass. |
---
## Blocking defects — in dependency order
### B1. Occupancy is measured from the wrong value (§5.2.1)
`herdr/adapter.go:191`
```go
u, e := a.Usage(s.PaneID) // ClaudeUsage(path string) wants a transcript file
```
`ClaudeUsage`/`CodexUsage`/`OpenCodeUsage` all take a **filesystem path** to
session state. They are handed a **herdr pane ID**. Every call returns
`open <pane-id>: no such file`, so `Coordinator.rotate` hits
`if err != nil { continue }` on line 430 and never rotates anything.
The spec is unusually explicit here — §5.2.1: *"Build this measurement and
verify it against a live session **before** wiring any trigger — the whole
rotation system rests on this number."* That step was skipped, and the
verification gap is exactly where the bug is.
Compounding: `CodexActiveUsage` (the discovery function that resolves the
active rollout via `state_*.sqlite`) and `ClaudeStopHookUsage` and
`OpenCodeStatus` are written but **called from nowhere** outside tests.
### B2. The adapter lookup key is wrong in three of four call sites (§5.3, §5.4)
`AdapterFactory.Herdrs` is keyed by **herdr instance id** (`homesrv-claude`).
`Session.Harness` is set by `CLIAdapter.Lease` to the **harness kind**
(`claude`). Then:
| Call site | Key used | Resolves? |
|---|---|---|
| `refreshSessionHealth` (orchestrator.go:221) | `HerdrID`, falls back to lease | yes |
| `Reconcile` (:291) | `session.Harness` | **no** |
| `expire` (:376, :397) | `session.Harness` | **no** |
| `rotate` (:419) | `session.Harness` | **no** |
So: orphaned panes are never killed on restart, `pane.exited` fast-path never
fires, expired leases never kill their pane, and rotation exits before it
begins. All silently — every one is a bare `continue`.
### B3. Nothing emits `TaskCompleted` (§4, §5.2)
Grep for producers: there is exactly one, `POST /v1/tasks/{id}/complete`,
which a human has to call. The spec assigns this to the **stop-hook/wrapper**
("the plane emits events, not the agent" — Invariant 2). There is no stop
hook, no wrapper, no completion detection of any kind.
Consequence: an agent that finishes its work sits idle until the 30-minute
lease TTL expires → `TaskReleased` → re-leased → repeat → `TaskFailed` at
attempt 3. Unattended runtime, the one metric in §0, is currently bounded at
90 minutes per task and always ends in a false failure.
### B4. The router counts rotation as a retry (§5.3, §5.4)
`router.go:127` increments `attempts[TaskID]` on **every** `TaskReleased`,
and `:169` increments again on every lease. `:149` fails the task at
`MaxAttempts` (3, from `main.go:64`).
Rotation *is* `TaskReleased` (§5.3: "rotation = intra-task lease transfer").
So a task healthy enough to rotate twice is killed by the retry limit. The
spec's retry policy is for *failures* (§5.4), not lease transfers. These need
separate counters — `attempts` should only advance on expiry/crash releases,
never on a release carrying a valid `handoff_ref`.
### B5. Herdr protocol methods are unverified and at least partly invented
`adapter.go` calls `pane.release`, `pane.kill`, `pane.rotation_signal`,
`pane.read`, `agent.get`, `agent.start`, `worktree.create/open`,
`agent.prompt`. The file itself documents that a previous method
(`pane.status`) *was not a valid protocol method* — so this surface has a
history of being written against an imagined API.
`pane.rotation_signal` is near-certainly not real: herdr is a generic
multiplexer with no concept of Orchestra rotation. And `pane.release`
returning a `handoff_ref` is architecturally wrong regardless of whether the
method exists — **herdr does not write handoffs; the agent does** (§6.1, the
handoff is a CAS artifact produced under the Face-B stop hook).
Nothing here can be settled from source. It needs `herdr api schema --json`
run against your protocol-17 instance, diffed against every method the
adapter calls.
Related: §5.1 requires `agent.prompt` with **inline `wait`** for bootstrap
injection, because that's what closes the send-into-a-half-rendered-prompt
race. `CLIAdapter.Lease` calls `Prompt(..., wait=0)` — no wait. Only
`Bootstrap` passes one.
### B6. Layer 3 is entirely dead code (§6)
Never called from anything but tests:
`continuity.ValidatePickup`, `ScratchCommit`, `ScratchPush`, `ScratchPull`,
`ScratchSync`, `MarkdownChanges`, `Save`, `Load`, `Encode`, `Decode`.
`VerifyTaskFile` *is* called — but only when `GitWorktrees.TaskFileSHA` is
non-empty, and `main.go:112` constructs the worktrees **without ever setting
it**. Nothing writes a `TASK.md` into a worktree in the first place.
So the entire §6.2 pickup contract — the part the spec calls
"the make-or-break" and "the defense against the telephone game" — does not
execute. Rotation, if B1/B2 were fixed, would hand the next agent a
`handoff_ref` that was never schema-validated, never anchor-checked, and
never accompanied by an immutable spec.
### B7. Quota projection has no producer (§7.2)
`QuotaAvailability.sumSince` folds `QuotaReported` events. `AggregateQuota`
does the same for the brief. **No code appends a `QuotaReported` event.**
Your config sets `quota_limit_5h: 50` / `quota_limit_weekly: 500` on all six
herdrs; those limits are compared against a permanent 0. The availability
filter is a no-op and the brief's `quota_consumed` is always `{}`.
### B8. `Surface: system` is an unauthenticated full-control bypass (§7.1)
`authz.CapabilityFor` grants `System``FullControl`. `authz.HTTP` reads the
surface straight from the `X-Orchestra-Surface` header, and the token map in
`main.go:760` has **no entry for `system`** — so `tokens[System]` is `""` and
the token check is skipped entirely. Any request on the LAN with
`X-Orchestra-Surface: system` can emit any event on any task.
The surface is also absent from `ParseSurface`'s intent: `System` is supposed
to mean "the plane itself, in-process", but it's reachable over HTTP. It
should never be accepted from a request header.
(Also: unset `ORCHESTRA_*_TOKEN` means that surface is unauthenticated; your
`.orchestra-config/orchestra.env` sets none of them. Defensible on WireGuard,
worth a deliberate decision rather than an accident.)
---
## Secondary defects
| # | Where | Issue |
|---|---|---|
| S1 | `operations.go:16,25` | `go vet` fails: `Brief.From/To` both serialize as `"from"`; `GitSync.Branch/Head/Status` all as `"branch"`. The brief's git state is unreadable by any client. |
| S2 | `main.go:238` | Brief's git state is read from `ORCHESTRA_DATA` (the event-log dir), not from the project worktrees. §7.4 wants "what pushed, what's on which branch, what workpc still needs to pull" — per project. |
| S3 | `operations.BuildBrief` | Counts completions but never surfaces `report_ref`/`receipt`, which §7.4 names as the proofs the brief exists to carry. |
| S4 | `delivery.Fanout.Run:110` | A single send error `return`s and permanently kills the notification goroutine — silently, since `main.go:729` only logs. One ntfy hiccup at 2am = no notifications for the rest of the night. Cursor is also in-memory, so a restart re-notifies the entire log from seq 0. |
| S5 | `store.Lease:341`, `ExpireLeases:350` | `Event.ID` is set to the **task id**, so event IDs collide across every lease of a task. `ApplyAdvisory` and any future ID-based lookup are unsound. |
| S6 | `store.Append:191` | Ingest dedup returns `nil` (success) without appending; `main.go:175` then returns `s.Events(0)[len-1]` — an unrelated event — with `201 Created`. |
| S7 | `store.apply:154` | `TaskAmended` applies only `title`. §4 lists title, due, description, `inherent_priority`. Amendments to the others are accepted, logged, and silently ignored by the projection. |
| S8 | §3.1 | No compensation-event mechanism exists. Append-only holds, but the spec's *correction* path ("a compensating event is appended") has no implementation. |
| S9 | `main.go:482` | Two independent lease-expiry loops (`main.go` 1s ticker and `Coordinator.expire` 30s) race on the same reclaim. Harmless today only because version CAS rejects the loser. |
| S10 | federation | `federation.Registry.Register` accepts a self-declared `id` + self-chosen `token` from any caller — registration is admission-control-free. |
| S11 | §5.3 | Thrash detection, the soft ~55% threshold, milestone rotation, and agent-initiated `ROTATE` are all absent. Only the hard threshold exists, and it's unreachable (B1). |
---
## The architectural fork — decided 2026-07-27
There are **two incompatible federation designs** in the tree, and only one is
deployed.
### Design A — "drive the remote socket" (deployed)
`clients/herdr-bridge.go` runs on workpc and proxies its local herdr Unix
socket to `192.168.1.105:9245`. `.orchestra-config/config.jsonc` registers
`workpc-claude` / `workpc-codex` / `workpc-opencode` against that machine, so
the **homesrv** coordinator resolves a workpc herdr as a candidate and calls
`worktree.create`, `agent.start`, `agent.prompt` on it over TCP as if it were
local. workpc is a dumb pane host; all orchestration logic and all state live
on homesrv, and live state crosses the machine boundary continuously.
### Design B — "workers pull tasks" (`/v1/federation/*`, unused)
workpc would run its own Orchestra process that registers itself
(`POST /v1/federation/workers`), heartbeats, polls the event log with a cursor
(`GET /v1/federation/events` + `/ack`), claims its own lease
(`POST .../claim`), runs a **local** coordinator against its **local** herdr
socket, and reports a handoff carrying an `anchor_sha` it computed from its
own checkout (`POST .../handoff`). homesrv stays authoritative for the log;
nothing but git and validated artifacts crosses the wire.
This is what the spec describes — §2.1: *"Everything crossing a machine
boundary is git + a validated artifact — never live state over the wire."*
**It is fully written on the server side and has zero clients.** There is no
worker binary in this repo; every one of those endpoints is unreachable in
the current deployment.
### Why keeping both is harmful, not merely untidy
Design A breaks a specific correctness property Design B was written to hold.
**The anchor is validated on the wrong machine.** `Coordinator.rotate`
(orchestrator.go:460) calls `herdr.HeadSHA(session.Worktree)`, which shells
out to `git -C <path> rev-parse HEAD`. The coordinator runs on homesrv;
`session.Worktree` is a path on **workpc**. Two outcomes, both bad:
- the path doesn't exist on homesrv → `HeadSHA` errors → rotation silently
skips forever (another bare `continue`); or
- the path *does* exist on homesrv — likely, since every project shares the
`/var/lib/orchestra/worktrees/<project>/<task>` layout — → it returns
**homesrv's HEAD for an unrelated checkout**, and the emitted `TaskReleased`
certifies a commit the agent never worked on.
The second is the dangerous one: it passes validation, looks correct, and
hands the successor an anchor describing another repo's state. That is exactly
the failure §9 item 8 names — *"worktree/anchor validation when a lease's git
checkout lives on a different host than the router"* — against the
degrade-safe default the spec already supplies: *"validate against the local
checkout wherever the harness runs."* Design B satisfies this by construction;
Design A cannot satisfy it without moving validation out of the coordinator.
The same class of bug applies to per-host quota accounting (§9 item 8) and to
`cleanupCompleted`, which runs `git worktree remove` on homesrv for a worktree
that lives on workpc.
### Latent defects in the unused half
Never surfaced because nothing exercises them:
- **Offline detection only runs inside `Snapshot()`** (federation.go:103-118),
called solely from `GET /v1/federation/workers`. Nothing polls it, so
`OnOffline` — the hook at `main.go:131` that releases leases held by a
vanished worker — fires only if a human hits that endpoint. The TTL backstop
§5.4 relies on does not tick on its own.
- **The registry is in-memory with no persistence.** Every restart forgets all
workers and their cursors; a reconnecting worker re-reads the entire log
from seq 0.
### Decision
**Keep Design A through Phase 5; commit to Design B in Phase 6.** Phases 14
(occupancy, Face B, rotation, continuity) are single-host concerns, provable
on homesrv alone with workpc's herdr as just another pane host. Resolving the
federation question first would block the fixes that actually unblock
unattended runtime.
Two guardrails land **immediately**, so Design A cannot corrupt state in the
meantime:
1. Anchor computation happens where the checkout is. Under the bridge that
means having herdr run the `rev-parse` in the pane's worktree rather than
shelling out locally — or, if the protocol schema won't support it,
**refusing to rotate any lease held by a non-local herdr** until Phase 6. A
loud refusal beats a false anchor.
2. Same treatment for `cleanupCompleted`'s `git worktree remove`.
Phase 6 then builds the worker binary against the endpoints that already
exist, adds server-issued tokens (S10), persists the registry, and drives
offline detection from a ticker rather than a request handler.
What is explicitly **not** acceptable is leaving both designs in place
unmarked. If the worker binary is ever abandoned, delete `/v1/federation/*`
and record the deviation — as it stands the repo reads as though the
spec-conformant path is implemented when nothing can reach it.
---
## Remediation plan
Ordered so each phase is verifiable on its own and nothing depends on an
unproven layer below it — the spec's own build discipline (§8).
### Phase 0 — Ground truth (half a day, blocks everything)
Nothing below is safe to write until the herdr surface is known.
1. On workpc: `herdr api schema --json > herdr-schema.json`, commit it to
`deploy/`.
2. Write `internal/herdr/schema_test.go`: for every method string in
`adapter.go`, assert it exists in the committed schema with the params we
send. This is the test that would have caught `pane.status` and will catch
the next one.
3. Manually drive one pane end to end against real herdr — create worktree,
start agent, prompt with inline wait, read status, kill — and record the
actual request/response shapes. Fix `adapter.go` to match.
4. Delete `pane.rotation_signal` and the `RotationSignal` interface unless the
schema proves it exists.
**Done when:** `schema_test.go` passes against the committed schema, and a
scripted manual run starts and stops a real Claude Code pane on workpc.
### Phase 1 — Occupancy, measured for real (§5.2.1) — fixes B1
The spec says build this first and verify against a live session. Do exactly
that, and nothing else in this phase.
1. Add `SessionFile string` to `herdr.Session`. Resolve it at lease time, per
harness:
- **claude** — the transcript path. Cleanest source is the Stop hook's
stdin (`transcript_path`), which you need for Phase 2 anyway; until then
resolve `~/.claude/projects/<enc-worktree-path>/<session>.jsonl` by
newest-mtime under the encoded worktree dir.
- **codex** — `CodexActiveUsage` already does the `state_*.sqlite →
threads.rollout_path` discovery. Call it. Filter by the rollout whose cwd
matches the worktree.
- **opencode** — `OpenCodeStatus` against `:4096` as fast path,
`~/.local/share/opencode/storage/message/` as backstop, per §5.2.1's
"the SSE stream is not rock-solid".
2. Change `CLIAdapter.Occupancy` to use `s.SessionFile`, and make a missing
session file a **hard error surfaced in `MonitorHealth`**, never a silent
`continue`.
3. Keep the §5.2.1 trap explicit: assert in a test that a fixture with a large
cumulative total but small last-turn usage yields *low* occupancy. This is
the failure mode the spec singles out.
4. Add `GET /v1/tasks/{id}/occupancy` returning the raw numerator, the window,
and the fraction — so you can eyeball it against a live session before
trusting it to drive rotation.
**Done when:** a real Claude Code session at a known context fill reports a
fraction within a few points of what `/context` says.
### Phase 2 — Face B and completion (§5.2, §4) — fixes B3, B5
1. **Claude Code Stop hook** (`deploy/hooks/orchestra-stop.sh`, bash, no
deps — §5.6). Reads hook JSON on stdin, POSTs `transcript_path` + task id
to a new `POST /v1/harness/turn` endpoint. Exit 2 with stderr when the
plane says "rotate but no valid handoff yet" — that's the spec's refuse-the-
turn mechanism, and `ClaudeStopHookUsage` already parses the input shape.
2. `POST /v1/harness/turn` computes occupancy, evaluates the rotation triggers,
and returns a decision: `continue` | `prepare_handoff` (soft) |
`rotate_now` (hard) | `refuse` (over threshold, handoff missing/invalid).
3. **Completion**: `POST /v1/harness/complete` — the wrapper/hook path that
uploads the report to CAS and emits `TaskCompleted` with `report_ref` +
`receipt`. Wire the receipt from Phase 1's occupancy reader, **summed
across lease intervals** per §4.
4. Codex/opencode Face B: rollout-tail and SSE `session.status` respectively,
polled by the coordinator rather than hook-pushed. Same decision endpoint.
**Done when:** a task given a trivial goal on a real harness reaches
`TaskCompleted` with a `report_ref` resolvable from CAS, with no human action.
### Phase 3 — Make rotation reachable and correct (§5.3, §5.4) — fixes B2, B4
1. Key sessions by `HerdrID` everywhere. Add a single
`Coordinator.adapterFor(session) (herdr.Adapter, error)` and route all four
call sites through it. Add a regression test that registers an adapter
under `homesrv-claude`, leases with `Harness: "claude"`, and asserts
rotation still fires.
2. Replace every silent `continue` in `rotate`/`expire` with a recorded reason
on `MonitorHealth`. The class of bug in B1/B2 is only invisible because of
those bare continues.
3. Split the router's counters: `releases` (informational) vs `failures`
(drives `MaxAttempts`). A `TaskReleased` carrying a valid `handoff_ref`
must not advance `failures`. Test: a task that rotates 5 times is still
alive.
4. Implement the missing triggers (§5.3): soft 55% → `prepare_handoff`;
milestone; thrash (N failed test runs / same file M times / identical tool
calls) as a circuit breaker with `reason=thrash` and populated `dead_ends`;
agent-initiated `ROTATE`.
5. Make TTL and the retry policy config, not the hardcoded `30*time.Minute` at
`router.go:165` and `1800` in `main.go` (§5.4: "N, ttl, retry backoff =
config").
6. **Land the two Design-A guardrails** (see the federation decision above):
compute the anchor where the checkout is, or refuse to rotate a lease held
by a non-local herdr; same for `cleanupCompleted`'s worktree removal. These
are the fixes that make it safe to defer federation to Phase 6.
**Done when:** a long-running task crosses 75% occupancy, rotates at a turn
boundary, and the successor continues — twice in a row, unattended.
### Phase 4 — Wire Layer 3 into the live path (§6) — fixes B6
This is where the dead code becomes load-bearing.
1. **Write `TASK.md`** into the worktree at creation from the `TaskCreated`
payload; hash it; store the hash on the task; pass it as `TaskFileSHA` into
`PerProjectGitWorktrees` (`main.go:112`). Re-inject it on every rotation
bootstrap — §6.2 is explicit that this is what defeats the telephone game.
2. **Handoff production**: the rotating agent writes the §6.1 TOML/JSON
handoff; the stop hook uploads it via `POST /v1/artifacts`; the plane
`continuity.Decode`s it, rejecting free-form `[knowledge]` (Invariant 4),
and only then emits `TaskReleased` with the hash. Never accept a
`handoff_ref` the plane hasn't validated.
3. **Scratch-branch commit before release** (`ScratchCommit`), so §6.2 step 3
collapses to one sha compare. `ScratchCommit` already refuses to commit a
dirty `TASK.md` — good, keep that.
4. **Pickup**: `ValidatePickup` runs in `Coordinator.Start` before the
successor's bootstrap prompt. Fail → do not start; emit `TaskBlocked`.
5. Replace `CLIAdapter.Bootstrap`'s freeform prose with the §6.2 procedure
(~200 tokens: read handoff, run validate-handoff, re-read TASK.md,
proceed), and use `agent.prompt`'s inline wait.
6. Wire `MarkdownChanges` to the §6.3 adjacent-task notice, or delete it and
record the deferral. Dead code that looks implemented is what produced this
audit.
**Done when:** a rotation whose anchor has drifted is *refused*, visibly, and
one whose anchor is clean proceeds without the successor re-deriving context.
### Phase 5 — Close the surfaces (§7) — fixes B7, B8, S1S4
1. Emit `QuotaReported`. Source: the same per-harness session state as
Phase 1, on a timer per `harness+window` (§7.2 — "a projection keyed by
`harness+window`, fed by the same per-harness session state"). Until then,
your quota limits are decorative.
2. Reject `X-Orchestra-Surface: system` at the HTTP boundary unconditionally.
`System` must be constructible in-process only. Then decide explicitly
whether the remaining surfaces require tokens on your LAN, and set them.
3. Fix the `go vet` json-tag collisions (S1) — the brief is currently
unparseable for git state.
4. Brief: per-project git sync state from the actual worktrees (S2), and
include `report_ref`/`receipt` in the rollup (S3).
5. Delivery: never `return` on send error — log, back off, continue. Persist
the cursor next to the event log (S4). Deliver the morning brief itself.
**Done when:** `/v1/brief` for an overnight window shows real quota per
harness, real per-project git state, and the receipts for every completion.
### Phase 6 — Federation, decided (§2.1, §9 item 8) — fixes the fork, S10
Only after Phases 15 run clean on a single host.
1. Anchor validation and `HeadSHA` execute **where the checkout is**. Today
they run on homesrv against a workpc path.
2. Either build the worker binary against `/v1/federation/*` (git-only
transport, per spec) or delete that API and document the bridge as a
deliberate deviation. Do not keep both.
3. Add admission control to worker registration (S10): server-issued tokens,
not self-declared.
4. Then run the real two-machine overnight batch that §9 item 8 asks for.
---
## Corrections to `progress.md`
Worth fixing, because the next session will otherwise trust it:
- "Continuity: strict handoff schema/validation, CAS save/load, pickup
validation, scratch-branch commit/push/pull helpers" — accurate as a
description of the *package*, misleading as a description of the *system*.
None of it is reachable at runtime.
- "herdr adapters with native occupancy readers" — the readers exist; they are
called with the wrong argument and always fail.
- "Router: ... quota availability at conservative 80% threshold" — the
threshold logic is correct, but no `QuotaReported` event is ever produced,
so it evaluates against zero.
- The rotation fix described in "Verified fixed this pass" is real and
correct — but `rotate` cannot reach that code, because the adapter lookup
above it (B2) fails first. The test passes because it registers the adapter
under the harness kind rather than the herdr id, which the production
config never does.
- "`go build ./...` and `go test ./...` both pass" — true; `go vet ./...`
does not.
---
## Suggested order of attack
If you want one thing to do today: **Phase 0**, then **B2** (a ~20-line fix
that makes three subsystems reachable), then **B1**. Those three turn a system
that cannot rotate into one that can, and everything else in the plan is
building on top rather than repairing underneath.
+166 -34
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"io"
"log"
"net"
"net/http"
"orchestra/internal/admin"
"orchestra/internal/authz"
@@ -26,6 +27,24 @@ import (
)
func id() string { return domain.NewID() }
const defaultHerdrPort = "9245"
func herdrAddress(rr registry.Registry, h registry.Herdr) string {
if h.Address != "" {
return h.Address
}
m, ok := rr.Machine(h.MachineID)
if !ok {
return ""
}
host, _, err := net.SplitHostPort(m.Address)
if err != nil {
return m.Address
}
return net.JoinHostPort(host, defaultHerdrPort)
}
func main() {
dir := os.Getenv("ORCHESTRA_DATA")
if dir == "" {
@@ -37,27 +56,34 @@ func main() {
}
var rr registry.Registry
var rt *router.Router
var coordinator *orchestrator.Coordinator
if config := os.Getenv("ORCHESTRA_CONFIG"); config != "" {
if rr, err = registry.Load(config); err != nil {
log.Fatalf("load orchestra config: %v", err)
}
rt = &router.Router{Store: s, Registry: rr, Reachability: registry.TCPReachability{}, Timeout: time.Second, Retry: router.RetryPolicy{MaxAttempts: 3, Backoff: time.Minute}}
limits := map[string]float64{}
limits := map[string]router.QuotaWindowLimits{}
for _, h := range rr.Herdrs() {
if h.QuotaLimit > 0 {
limits[h.ID] = h.QuotaLimit
w := router.QuotaWindowLimits{FiveHour: h.QuotaLimit5h, Weekly: h.QuotaLimitWeekly}
if w.Weekly <= 0 && h.QuotaLimit > 0 {
// Back-compat: the old single-window field meant weekly.
w.Weekly = h.QuotaLimit
}
if w.FiveHour > 0 || w.Weekly > 0 {
limits[h.ID] = w
}
}
if len(limits) > 0 {
rt.Availability = router.QuotaAvailability{Store: s, Limits: limits, Window: 7 * 24 * time.Hour}
rt.Availability = router.QuotaAvailability{Store: s, Limits: limits}
}
if repo, root := os.Getenv("ORCHESTRA_REPO"), os.Getenv("ORCHESTRA_WORKTREE_ROOT"); repo != "" && root != "" {
adapters := map[string]herdr.Adapter{}
for _, h := range rr.Herdrs() {
if h.Address == "" {
address := herdrAddress(rr, h)
if address == "" {
continue
}
client := herdr.New(h.Address)
client := herdr.New(address)
protocol := h.Protocol
if protocol == "" {
protocol = os.Getenv("ORCHESTRA_HERDR_PROTOCOL")
@@ -77,7 +103,17 @@ func main() {
log.Printf("herdr %s has unsupported harness %q", h.ID, h.Harness)
}
}
coordinator := &orchestrator.Coordinator{Store: s, StatePath: filepath.Join(dir, "runtime-sessions.json"), Worktrees: orchestrator.GitWorktrees{Root: root, Repo: repo}, Adapters: orchestrator.AdapterFactory{Herdrs: adapters}}
projectRepos := map[string]orchestrator.ProjectRepo{}
for _, p := range rr.Projects() {
if p.Repo != "" && p.WorktreeRoot != "" {
projectRepos[p.ID] = orchestrator.ProjectRepo{Repo: p.Repo, WorktreeRoot: p.WorktreeRoot}
}
}
worktrees := orchestrator.PerProjectGitWorktrees{
Projects: projectRepos,
Default: orchestrator.GitWorktrees{Root: root, Repo: repo},
}
coordinator = &orchestrator.Coordinator{Store: s, StatePath: filepath.Join(dir, "runtime-sessions.json"), Worktrees: worktrees, Adapters: orchestrator.AdapterFactory{Herdrs: adapters}}
rt.OnLease = func(e domain.Event) error { return coordinator.Start(context.Background(), e) }
hard := 0.75
if v, parseErr := strconv.ParseFloat(os.Getenv("ORCHESTRA_OCCUPANCY_HARD"), 64); parseErr == nil && v > 0 && v < 1 {
@@ -96,7 +132,7 @@ func main() {
for _, t := range s.Tasks() {
if t.State == domain.StateLeased && t.Lease != nil && t.Lease.HarnessID == w.ID {
p, _ := json.Marshal(map[string]any{"reason": "worker_offline", "harness_id": w.ID})
e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p}
e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
if err := s.Append(e); err == nil && rt != nil {
_, _ = rt.HandleEvent(e)
}
@@ -126,7 +162,7 @@ func main() {
return
}
b, _ := json.Marshal(p)
e := domain.Event{ID: id(), Type: "TaskCreated", TaskID: id(), Version: 1, Payload: b}
e := domain.Event{ID: id(), Type: "TaskCreated", TaskID: id(), Version: 1, Payload: b, Surface: string(surface(r))}
if err := s.Append(e); err != nil {
http.Error(w, err.Error(), 400)
return
@@ -147,6 +183,24 @@ func main() {
}
json.NewEncoder(w).Encode(s.Events(n))
})
mux.HandleFunc("/v1/handoffs", func(w http.ResponseWriter, r *http.Request) {
out := make([]domain.Event, 0)
for _, e := range s.Events(0) {
if e.Type == "TaskReleased" {
out = append(out, e)
}
}
json.NewEncoder(w).Encode(out)
})
mux.HandleFunc("/v1/quotas", func(w http.ResponseWriter, r *http.Request) {
out := make([]domain.Event, 0)
for _, e := range s.Events(0) {
if e.Type == "QuotaReported" {
out = append(out, e)
}
}
json.NewEncoder(w).Encode(out)
})
mux.HandleFunc("/v1/artifacts", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
@@ -223,7 +277,7 @@ func main() {
return
}
b, _ := json.Marshal(map[string]any{"subject_ref": p.AdvisoryID})
e := domain.Event{ID: id(), Type: "ApprovalGranted", TaskID: "system", Version: 0, Payload: b}
e := domain.Event{ID: id(), Type: "ApprovalGranted", TaskID: "system", Version: 0, Payload: b, Surface: string(surface(r))}
if err := s.Append(e); err != nil {
http.Error(w, err.Error(), 400)
return
@@ -256,7 +310,16 @@ func main() {
adminServer := &admin.Server{Store: s, RouterReady: rt != nil, Providers: providerHealth, Probes: map[string]admin.ProbeFunc{
"router": func() (bool, string) { return rt != nil, "configured router" },
"gitea": func() (bool, string) {
return os.Getenv("ORCHESTRA_GITEA_URL") == "" || providerHealth["gitea"] != nil, "configured provider"
configured := os.Getenv("ORCHESTRA_GITEA_URL") != "" || os.Getenv("ORCHESTRA_GITEA_CONFIG") != ""
if !configured {
return true, "configured provider"
}
for name := range providerHealth {
if name == "gitea" || strings.HasPrefix(name, "gitea:") {
return true, "configured provider"
}
}
return false, "configured provider"
},
"jsonl": func() (bool, string) {
return os.Getenv("ORCHESTRA_JSONL") == "" || providerHealth["jsonl"] != nil, "configured provider"
@@ -267,6 +330,26 @@ func main() {
mux.HandleFunc("/v1/admin/diagnostics", adminServer.Diagnostics)
mux.HandleFunc("/v1/tasks/", func(w http.ResponseWriter, r *http.Request) {
parts := strings.Split(strings.Trim(r.URL.Path, "/"), "/")
if r.Method == http.MethodGet && coordinator != nil && len(parts) == 4 {
taskID, view := parts[2], parts[3]
if view == "health" {
if h, ok := coordinator.MonitorHealth().Sessions[taskID]; ok {
json.NewEncoder(w).Encode(h)
return
}
http.Error(w, "session health not found", 404)
return
}
if view == "capture" {
body, err := coordinator.Capture(r.Context(), taskID, r.URL.Query().Get("source"))
if err != nil {
http.Error(w, err.Error(), 404)
return
}
json.NewEncoder(w).Encode(map[string]string{"task_id": taskID, "source": r.URL.Query().Get("source"), "text": body})
return
}
}
if len(parts) < 4 || len(parts) > 5 || r.Method != "POST" {
http.Error(w, "not found", http.StatusNotFound)
return
@@ -300,7 +383,7 @@ func main() {
by = "surface"
}
b, _ := json.Marshal(map[string]any{"subject_ref": taskID, "by": by})
e := domain.Event{ID: id(), Type: typ, TaskID: taskID, Version: t.Version + 1, Payload: b}
e := domain.Event{ID: id(), Type: typ, TaskID: taskID, Version: t.Version + 1, Payload: b, Surface: string(surface(r))}
if err := s.Append(e); err != nil {
http.Error(w, err.Error(), 409)
return
@@ -321,7 +404,7 @@ func main() {
}
b, _ := json.Marshal(map[string]any{"subject_ref": taskID, "options": p.Options})
t, _ := s.Task(taskID)
e := domain.Event{ID: id(), Type: "ApprovalRequested", TaskID: taskID, Version: t.Version + 1, Payload: b}
e := domain.Event{ID: id(), Type: "ApprovalRequested", TaskID: taskID, Version: t.Version + 1, Payload: b, Surface: string(surface(r))}
if err := s.Append(e); err != nil {
http.Error(w, err.Error(), 400)
return
@@ -377,7 +460,7 @@ func main() {
return
}
ePayload, _ := json.Marshal(p)
e = domain.Event{ID: id(), Type: types[action], TaskID: taskID, Version: t.Version + 1, Payload: ePayload}
e = domain.Event{ID: id(), Type: types[action], TaskID: taskID, Version: t.Version + 1, Payload: ePayload, Surface: string(surface(r))}
err = s.Append(e)
default:
http.Error(w, "unknown action", 404)
@@ -416,6 +499,13 @@ func main() {
}
json.NewEncoder(w).Encode(out)
})
mux.HandleFunc("/v1/orchestrator/health", func(w http.ResponseWriter, r *http.Request) {
if coordinator == nil {
http.Error(w, "orchestrator unavailable", http.StatusServiceUnavailable)
return
}
json.NewEncoder(w).Encode(coordinator.MonitorHealth())
})
mux.HandleFunc("/v1/federation/workers", func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet {
json.NewEncoder(w).Encode(workers.Snapshot())
@@ -521,6 +611,7 @@ func main() {
TaskID string `json:"task_id"`
TTLSeconds int `json:"ttl_seconds"`
HandoffRef string `json:"handoff_ref"`
AnchorSHA string `json:"anchor_sha"`
}
if json.NewDecoder(r.Body).Decode(&b) != nil || b.TaskID == "" {
http.Error(w, "invalid lease body", 400)
@@ -551,8 +642,12 @@ func main() {
http.Error(w, "handoff_ref required", 400)
return
}
p, _ := json.Marshal(map[string]any{"handoff_ref": b.HandoffRef, "harness_id": parts[3]})
e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: b.TaskID, Version: t.Version + 1, Payload: p}
if len(b.AnchorSHA) != 40 {
http.Error(w, "anchor_sha required", 400)
return
}
p, _ := json.Marshal(map[string]any{"handoff_ref": b.HandoffRef, "harness_id": parts[3], "anchor_sha": b.AnchorSHA})
e := domain.Event{ID: id(), Type: "TaskReleased", TaskID: b.TaskID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
if err := s.Append(e); err != nil {
http.Error(w, err.Error(), 409)
return
@@ -562,25 +657,57 @@ func main() {
}
json.NewEncoder(w).Encode(e)
})
if base := os.Getenv("ORCHESTRA_GITEA_URL"); base != "" {
g := provider.Gitea{BaseURL: base, Token: os.Getenv("ORCHESTRA_GITEA_TOKEN"), WebhookSecret: os.Getenv("ORCHESTRA_GITEA_WEBHOOK_SECRET"), Owner: os.Getenv("ORCHESTRA_GITEA_OWNER"), Repo: os.Getenv("ORCHESTRA_GITEA_REPO")}
reflecting := provider.ReflectingSink{Sink: s, Tasks: s, Reflector: g}
mux.Handle("/v1/providers/gitea/webhook", g.WebhookHandler(reflecting))
sup := &provider.Supervisor{Name: "gitea", Run: func(ctx context.Context) error {
pollCtx, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
_, err := g.Poll(pollCtx, reflecting)
if err == nil {
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(time.Minute):
}
}
return err
var giteaSources []provider.GiteaSourceConfig
if path := os.Getenv("ORCHESTRA_GITEA_CONFIG"); path != "" {
cfgs, err := provider.LoadGiteaConfigs(path)
if err != nil {
log.Fatalf("load gitea config: %v", err)
}
giteaSources = cfgs
} else if base := os.Getenv("ORCHESTRA_GITEA_URL"); base != "" {
// Legacy single-repo configuration: project defaults to the repo
// name, matching the historical (pre-multi-source) behavior.
giteaSources = []provider.GiteaSourceConfig{{
Project: os.Getenv("ORCHESTRA_GITEA_REPO"), BaseURL: base,
Owner: os.Getenv("ORCHESTRA_GITEA_OWNER"), Repo: os.Getenv("ORCHESTRA_GITEA_REPO"),
Token: os.Getenv("ORCHESTRA_GITEA_TOKEN"), WebhookSecret: os.Getenv("ORCHESTRA_GITEA_WEBHOOK_SECRET"),
}}
sup.Start(context.Background())
providerHealth["gitea"] = sup
}
if len(giteaSources) > 0 {
reflectors := map[string]provider.Gitea{}
for _, c := range giteaSources {
g := provider.Gitea{BaseURL: c.BaseURL, Token: c.Token, WebhookSecret: c.WebhookSecret, Owner: c.Owner, Repo: c.Repo, Project: c.Project}
reflectors[g.SourceName()] = g
}
reflecting := provider.ReflectingSink{Sink: s, Tasks: s, Reflector: provider.MultiGitea{Sources: reflectors}}
for _, c := range giteaSources {
g := provider.Gitea{BaseURL: c.BaseURL, Token: c.Token, WebhookSecret: c.WebhookSecret, Owner: c.Owner, Repo: c.Repo, Project: c.Project}
name := "gitea:" + c.Project
webhookPath := "/v1/providers/gitea/webhook/" + c.Project
if len(giteaSources) == 1 && os.Getenv("ORCHESTRA_GITEA_CONFIG") == "" {
// Preserve the legacy unprefixed webhook path when running
// the single-source (env-var) configuration, so existing
// Gitea webhook configs don't need to be re-pointed.
webhookPath = "/v1/providers/gitea/webhook"
name = "gitea"
}
mux.Handle(webhookPath, g.WebhookHandler(reflecting))
sup := &provider.Supervisor{Name: name, Run: func(ctx context.Context) error {
pollCtx, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
_, err := g.Poll(pollCtx, reflecting)
if err == nil {
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(time.Minute):
}
}
return err
}}
sup.Start(context.Background())
providerHealth[name] = sup
}
}
if path := os.Getenv("ORCHESTRA_JSONL"); path != "" {
sup := &provider.Supervisor{Name: "jsonl", Run: func(ctx context.Context) error {
@@ -620,6 +747,11 @@ func main() {
}
}
}()
if rt != nil {
if _, err := rt.AssignPending(); err != nil {
log.Printf("startup task assignment: %v", err)
}
}
port := os.Getenv("ORCHESTRA_PORT")
if port == "" {
port = "9145"
+66
View File
@@ -0,0 +1,66 @@
{
"projects": [
{
"id": "correx",
"machine_affinity": ["mainframe"],
"repo": "/var/lib/orchestra/repos/correx.git",
"worktree_root": "/var/lib/orchestra/worktrees/correx"
},
{
"id": "maven",
"machine_affinity": ["mainframe", "satellite"]
}
],
"machines": [
{
"id": "mainframe",
"address": "10.0.0.10:9145"
},
{
"id": "satellite",
"address": "10.0.0.11:9145"
}
],
"herdrs": [
{
"id": "mainframe-claude-1",
"machine_id": "mainframe",
"harness": "claude",
"protocol": "1",
"capabilities": ["code", "review"],
"concurrency": 2,
"quota_limit_5h": 50,
"quota_limit_weekly": 500
},
{
"id": "satellite-claude-1",
"machine_id": "satellite",
"address": "10.0.0.11:9245",
"harness": "claude",
"protocol": "1",
"capabilities": ["code"],
"concurrency": 1,
"quota_limit_5h": 20,
"quota_limit_weekly": 200
},
{
"id": "mainframe-codex-1",
"machine_id": "mainframe",
"harness": "codex",
"protocol": "1",
"capabilities": ["code"],
"concurrency": 1,
"quota_limit_weekly": 300
},
{
"id": "satellite-opencode-1",
"machine_id": "satellite",
"address": "10.0.0.11:9345",
"harness": "opencode",
"protocol": "1",
"capabilities": ["code", "review"],
"concurrency": 1,
"quota_limit_weekly": 300
}
]
}
+78
View File
@@ -0,0 +1,78 @@
// Annotated reference for config.example.json (registry.Config, internal/registry/registry.go).
// This file is NOT valid JSON (it has comments) and is not loaded by orchestra —
// it exists purely to document fields. Copy config.example.json, not this file.
{
// Static project topology. One entry per project the fleet routes tasks for.
"projects": [
{
"id": "correx", // Project id; tasks/events are tagged with this.
"machine_affinity": ["mainframe"], // Machine ids (below) this project may run on.
// Required — a project with no affinity can't be routed.
"repo": "/var/lib/orchestra/repos/correx.git", // Optional per-project git repo path.
// Overrides the global ORCHESTRA_REPO default.
"worktree_root": "/var/lib/orchestra/worktrees/correx" // Optional per-project worktree dir.
// Overrides global ORCHESTRA_WORKTREE_ROOT.
},
{
"id": "maven",
"machine_affinity": ["mainframe", "satellite"] // Multiple affinities: routable to either machine.
// repo/worktree_root omitted here: falls back to the deployment's global default.
}
],
// Physical/logical machines in the fleet. herdrs.machine_id below must reference one of these.
"machines": [
{ "id": "mainframe", "address": "10.0.0.10:9145" }, // address: host:port this machine's orchestra API listens on.
{ "id": "satellite", "address": "10.0.0.11:9145" }
],
// Herdrs: individual harness worker slots that execute tasks.
"herdrs": [
{
"id": "mainframe-claude-1", // Unique herdr id.
"machine_id": "mainframe", // Which machine (above) this herdr runs on.
// "address" omitted: falls back to the parent machine's address (used here since
// this herdr's harness listens on the machine's default port).
"harness": "claude", // Harness adapter to use: "claude" | "codex" | "opencode".
"protocol": "1", // Herdr wire protocol version. Falls back to
// ORCHESTRA_HERDR_PROTOCOL if omitted.
"capabilities": ["code", "review"], // Task capability tags this herdr can accept.
"concurrency": 2, // Max simultaneous sessions this herdr will run.
"quota_limit_5h": 50, // Rolling 5-hour usage quota (harness-specific units).
"quota_limit_weekly": 500 // Rolling weekly usage quota.
// "quota_limit" (deprecated): if set without quota_limit_5h, treated as weekly-only,
// to preserve old configs' historical meaning without inventing a 5h cap.
},
{
"id": "satellite-claude-1",
"machine_id": "satellite",
"address": "10.0.0.11:9245", // Explicit override: this herdr listens on a non-default
// port on its machine (e.g. multiple herdrs per machine).
"harness": "claude",
"protocol": "1",
"capabilities": ["code"],
"concurrency": 1,
"quota_limit_5h": 20,
"quota_limit_weekly": 200
},
{
"id": "mainframe-codex-1",
"machine_id": "mainframe",
"harness": "codex", // OpenAI Codex CLI harness adapter.
"protocol": "1",
"capabilities": ["code"],
"concurrency": 1,
"quota_limit_weekly": 300 // codex has no separate 5h window tracked here; weekly only.
},
{
"id": "satellite-opencode-1",
"machine_id": "satellite",
"address": "10.0.0.11:9345",
"harness": "opencode", // OpenCode CLI harness adapter.
"protocol": "1",
"capabilities": ["code", "review"],
"concurrency": 1,
"quota_limit_weekly": 300
}
]
}
+18
View File
@@ -0,0 +1,18 @@
[
{
"project": "correx",
"base_url": "https://gitea.example.internal",
"owner": "kami",
"repo": "correx",
"token": "REPLACE_ME",
"webhook_secret": "REPLACE_ME"
},
{
"project": "maven",
"base_url": "https://gitea.example.internal",
"owner": "kami",
"repo": "maven",
"token": "REPLACE_ME",
"webhook_secret": "REPLACE_ME"
}
]
+69
View File
@@ -0,0 +1,69 @@
# Copy to /etc/orchestra/orchestra.env (chmod 600, owned by the orchestra
# user) and fill in the values you need. Referenced by orchestra.service via
# EnvironmentFile=. Every var below is read directly from os.Getenv in
# cmd/orchestra/main.go and the packages it wires up — grep ORCHESTRA_ in the
# repo if this list ever needs re-deriving.
# --- Core ---
ORCHESTRA_DATA=/var/lib/orchestra/data
ORCHESTRA_PORT=9145
# Static project/machine/herdr topology (registry.Load). Required for
# routing across more than one machine; validated at startup.
ORCHESTRA_CONFIG=/etc/orchestra/config.json
# --- Git worktrees (global default; per-project repo/worktree_root in
# ORCHESTRA_CONFIG overrides this per project — see registry.Project) ---
ORCHESTRA_REPO=/var/lib/orchestra/repo.git
ORCHESTRA_WORKTREE_ROOT=/var/lib/orchestra/worktrees
# Protocol version fallback for herdrs that don't set "protocol" in
# ORCHESTRA_CONFIG. Prefer setting it per-herdr in the config; only use this
# if every herdr on the fleet truly matches.
#ORCHESTRA_HERDR_PROTOCOL=1
# Hard rotation occupancy threshold (0 < x < 1). Default 0.75 if unset/invalid.
ORCHESTRA_OCCUPANCY_HARD=0.75
# --- Providers ---
# Local JSONL task ingestion (baseline adapter).
#ORCHESTRA_JSONL=/var/lib/orchestra/tasks.jsonl
# Gitea issue ingestion + terminal-state reflection.
#
# Multiple repos (one per project) — preferred if you have more than one
# Gitea-backed project. Points at a JSON array of
# {project,base_url,owner,repo,token,webhook_secret}; project is the
# registry project id ingested tasks are tagged with. Each source gets its
# own webhook path: /v1/providers/gitea/webhook/{project}.
#ORCHESTRA_GITEA_CONFIG=/etc/orchestra/gitea.json
#
# Single repo (legacy) — all four required together. Ignored if
# ORCHESTRA_GITEA_CONFIG is set. Webhook path is the unprefixed
# /v1/providers/gitea/webhook. Ingested tasks are tagged with project =
# ORCHESTRA_GITEA_REPO.
#ORCHESTRA_GITEA_URL=https://gitea.example.internal
#ORCHESTRA_GITEA_TOKEN=
#ORCHESTRA_GITEA_OWNER=
#ORCHESTRA_GITEA_REPO=
#ORCHESTRA_GITEA_WEBHOOK_SECRET=
# --- Delivery (notify-only surfaces) ---
# Telegram: both required together.
#ORCHESTRA_TELEGRAM_BOT_TOKEN=
#ORCHESTRA_TELEGRAM_CHAT_ID=
# ntfy: topic required, token/url optional (self-hosted ntfy).
#ORCHESTRA_NTFY_TOPIC=
#ORCHESTRA_NTFY_TOKEN=
#ORCHESTRA_NTFY_URL=https://ntfy.sh
# --- Bus authorization tokens (bearer auth per surface; a surface with no
# token set has no auth requirement — set these once you have real clients) ---
#ORCHESTRA_TUI_TOKEN=
#ORCHESTRA_WEB_TOKEN=
#ORCHESTRA_MCP_TOKEN=
#ORCHESTRA_MAVEN_TOKEN=
# Telegram/ntfy tokens above double as their surface auth tokens
# (ORCHESTRA_TELEGRAM_TOKEN is the inbound bearer token if you also expose an
# endpoint they poll, separate from the bot token used to send messages).
#ORCHESTRA_TELEGRAM_TOKEN=
+7 -3
View File
@@ -6,16 +6,20 @@ Wants=network-online.target
[Service]
Type=simple
User=orchestra
Group=orchestra
WorkingDirectory=/var/lib/orchestra
Environment=ORCHESTRA_DATA=/var/lib/orchestra/data
Environment=ORCHESTRA_PORT=9145
EnvironmentFile=/etc/orchestra/orchestra.env
ExecStart=/usr/local/bin/orchestra
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/orchestra
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/orchestra
# ORCHESTRA_WORKTREE_ROOT / per-project worktree_root paths and
# ORCHESTRA_REPO must live under one of these, or under /var/lib/orchestra —
# add further ReadWritePaths= lines here if you keep repos elsewhere.
[Install]
WantedBy=multi-user.target
+17
View File
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -euo pipefail
repo_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
tmp_bin="$(mktemp)"
trap 'rm -f -- "$tmp_bin"' EXIT
cd "$repo_dir"
echo "Building Orchestra..."
go build -o "$tmp_bin" ./cmd/orchestra
echo "Installing /usr/local/bin/orchestra..."
sudo install -o root -g root -m 0755 "$tmp_bin" /usr/local/bin/orchestra
echo "Restarting orchestra.service..."
sudo systemctl restart orchestra.service
sudo systemctl --no-pager --lines=8 status orchestra.service
+1 -1
View File
@@ -73,7 +73,7 @@ func TestSubscribeEmitsCursorAndEvent(t *testing.T) {
t.Fatal(err)
}
p, _ := json.Marshal(map[string]any{"source": "test", "external_id": "1", "project": "p"})
if err := s.Append(domain.Event{ID: "e", TaskID: "t", Type: "TaskCreated", Version: 1, Payload: p}); err != nil {
if err := s.Append(domain.Event{ID: "e", TaskID: "t", Type: "TaskCreated", Version: 1, Payload: p, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
h := (&Server{Store: s}).Subscribe
+8 -1
View File
@@ -17,6 +17,13 @@ const (
Web Surface = "web"
MCP Surface = "mcp"
Maven Surface = "maven"
// System identifies the plane itself — the router, coordinator, provider
// adapters, and lease-expiry reclaim. Per invariant 2 ("the plane emits
// events, not the agent"), these are the only non-surface emitters and are
// always full control. Every event must carry an explicit Surface; there
// is no unauthenticated default, so an emitter that forgets to declare one
// is rejected at the bus rather than silently treated as trusted.
System Surface = "system"
)
type Capability int
@@ -33,7 +40,7 @@ func CapabilityFor(s Surface) Capability {
switch s {
case Telegram, Ntfy:
return NotifyOnly
case TUI, Web:
case TUI, Web, System:
return FullControl
case MCP, Maven:
return GatedWrite
+14 -1
View File
@@ -17,7 +17,11 @@ var ErrConflict = errors.New("task version conflict")
var ErrNotFound = errors.New("task not found")
var ErrInvalid = errors.New("invalid event")
const CurrentEventSchema = 1
// CurrentEventSchema is 2: schema 2 requires every event to declare its
// authorizing Surface (see ValidateEvent), enforced at the store append
// boundary. Schema 1 events already on disk replay unchanged — tolerant
// reader, not upcast (spec open question #2).
const CurrentEventSchema = 2
type TaskState string
@@ -63,6 +67,12 @@ type Event struct {
Version int `json:"version"`
At time.Time `json:"at"`
Payload json.RawMessage `json:"payload"`
// Surface identifies the bus capability the emitter is authorized under
// (see internal/authz). It is required on every event so authorization is
// enforced once, at the store append boundary, regardless of whether the
// emitter reached the store over HTTP, from the router, from a harness
// adapter, or from a provider.
Surface string `json:"surface"`
}
func Hash(v []byte) string { h := sha256.Sum256(v); return hex.EncodeToString(h[:]) }
@@ -82,6 +92,9 @@ func ValidateEvent(e Event) error {
if e.SchemaVersion > CurrentEventSchema || e.Type == "" || e.TaskID == "" || len(e.Payload) == 0 || len(e.Payload) > 64*1024 {
return ErrInvalid
}
if e.SchemaVersion >= 2 && strings.TrimSpace(e.Surface) == "" {
return fmt.Errorf("%w: surface required", ErrInvalid)
}
allowed := map[string]bool{"TaskCreated": true, "TaskLeased": true, "TaskReleased": true, "TaskCompleted": true, "TaskFailed": true, "TaskBlocked": true, "ApprovalRequested": true, "ApprovalGranted": true, "ApprovalDenied": true, "TaskAmended": true, "QuotaReported": true, "StandupAdvisory": true}
if !allowed[e.Type] {
return fmt.Errorf("%w: unknown type %q", ErrInvalid, e.Type)
+149
View File
@@ -0,0 +1,149 @@
package domain
import (
"encoding/json"
"testing"
)
// eventTypesUnderTest is the full lifecycle vocabulary the spec (§9 item 5)
// requires validators for; a validator that panics or accepts garbage for any
// of these on adversarial input is a defect regardless of whether real
// producers happen to send well-formed payloads.
var eventTypesUnderTest = []string{
"TaskCreated", "TaskLeased", "TaskReleased", "TaskCompleted", "TaskFailed",
"TaskBlocked", "ApprovalRequested", "ApprovalGranted", "ApprovalDenied",
"TaskAmended", "QuotaReported", "StandupAdvisory",
}
// FuzzValidatePayload feeds arbitrary JSON object shapes at every known event
// type's validator and requires it to either return a typed ErrInvalid or
// accept — never panic. The seed corpus below exercises adjacent-to-valid and
// wildly-malformed shapes (wrong types, huge strings, nested structures,
// nulls, NaN-adjacent floats via JSON) for each type.
func FuzzValidatePayload(f *testing.F) {
seeds := []string{
`{}`,
`null`,
`{"source":"jsonl","external_id":"1","project":"p"}`,
`{"source":123,"external_id":null,"project":[]}`,
`{"harness_id":"h1","ttl":60,"expected_version":1}`,
`{"harness_id":"h1","until_ns":1e300,"expected_version":1.5}`,
`{"handoff_ref":"` + fakeHash() + `","anchor_sha":"` + fakeSHA() + `"}`,
`{"handoff_ref":123,"anchor_sha":true}`,
`{"report_ref":"` + fakeHash() + `","receipt":{"harness_id":"h1","consumed":1}}`,
`{"report_ref":"","receipt":{}}`,
`{"reason":"x"}`,
`{"reason":123}`,
`{"blocker":"x","handoff_ref":"` + fakeHash() + `"}`,
`{"blocker":""}`,
`{"amendment":"x"}`,
`{"subject_ref":"x","options":["a","b"]}`,
`{"subject_ref":123,"options":null}`,
`{"harness_id":"h1","consumed":90.5}`,
`{"harness_id":"h1","consumed":-1}`,
`{"harness_id":"h1","consumed":"a lot"}`,
`{"items":["standup line"]}`,
`{"items":null}`,
`{"a":{"b":{"c":{"d":[1,2,3,{"e":"f"}]}}}}`,
`{"x":` + hugeString() + `}`,
}
for _, s := range seeds {
f.Add(s)
}
f.Fuzz(func(t *testing.T, raw string) {
var p map[string]any
if err := json.Unmarshal([]byte(raw), &p); err != nil {
return // not a JSON object; ValidateEvent itself rejects non-objects before reaching ValidatePayload
}
for _, typ := range eventTypesUnderTest {
func() {
defer func() {
if r := recover(); r != nil {
t.Fatalf("ValidatePayload(%q, %s) panicked: %v", typ, raw, r)
}
}()
err := ValidatePayload(typ, p)
if err != nil && err != ErrInvalid {
// must still be a typed validation error, wrapping ErrInvalid
if !isInvalid(err) {
t.Fatalf("ValidatePayload(%q, %s) returned non-typed error: %v", typ, raw, err)
}
}
}()
}
})
}
// FuzzValidateEvent exercises the full envelope path (schema version, surface
// requirement, type allow-list, payload size/parseability) with arbitrary
// type names, surfaces, and payload bytes, proving no combination panics.
func FuzzValidateEvent(f *testing.F) {
f.Add("TaskCreated", "system", 2, []byte(`{"source":"jsonl","external_id":"1","project":"p"}`))
f.Add("", "", 0, []byte(``))
f.Add("Bogus", "system", 2, []byte(`{}`))
f.Add("TaskCreated", "", 2, []byte(`{"source":"jsonl","external_id":"1","project":"p"}`))
f.Add("TaskCreated", "system", 1, []byte(`not json`))
f.Add("QuotaReported", "system", 2, []byte(`null`))
f.Add("TaskCompleted", "system", 99, []byte(`{"report_ref":"x","receipt":{}}`))
f.Fuzz(func(t *testing.T, typ, surface string, schema int, payload []byte) {
defer func() {
if r := recover(); r != nil {
t.Fatalf("ValidateEvent panicked: type=%q surface=%q schema=%d payload=%q: %v", typ, surface, schema, payload, r)
}
}()
e := Event{
SchemaVersion: schema,
Type: typ,
TaskID: "t1",
Payload: payload,
Surface: surface,
}
_ = ValidateEvent(e)
})
}
func isInvalid(err error) bool {
for e := err; e != nil; {
if e == ErrInvalid {
return true
}
u, ok := e.(interface{ Unwrap() error })
if !ok {
return false
}
e = u.Unwrap()
}
return false
}
func fakeHash() string {
b := make([]byte, 32)
for i := range b {
b[i] = byte(i)
}
s := ""
for _, c := range b {
s += string("0123456789abcdef"[c>>4]) + string("0123456789abcdef"[c&0xf])
}
return s
}
func fakeSHA() string {
s := ""
for i := 0; i < 40; i++ {
s += "a"
}
return s
}
func hugeString() string {
b, _ := json.Marshal(make([]byte, 0))
_ = b
s := `"`
for i := 0; i < 5000; i++ {
s += "x"
}
return s + `"`
}
+120 -15
View File
@@ -2,7 +2,9 @@ package herdr
import (
"context"
"encoding/json"
"fmt"
"path/filepath"
"strings"
"time"
)
@@ -15,6 +17,10 @@ type Adapter interface {
Occupancy(Session) (float64, error)
}
type WorktreeCreator interface {
CreateWorktree(context.Context, string, string, string) (string, error)
}
// TurnBoundary is optional so older herdr deployments remain usable. A true
// result means the current harness turn has ended and handoff is safe.
type TurnBoundary interface {
@@ -26,6 +32,18 @@ type RotationSignal interface {
type PaneExit interface {
PaneExited(context.Context, Session) (bool, error)
}
// AgentStatus is a live, non-lifecycle status reported by herdr. Consumers
// must not infer task completion or release from it.
type AgentStatus interface {
AgentStatus(context.Context, Session) (string, error)
}
type AgentBlocker interface {
AgentBlocker(context.Context, Session) (string, error)
}
type PaneCapture interface {
PaneCapture(context.Context, Session, string) (string, error)
}
type CLIAdapter struct {
Client *Client
Harness string
@@ -33,15 +51,29 @@ type CLIAdapter struct {
Usage func(string) (Usage, error)
}
func (a CLIAdapter) CreateWorktree(ctx context.Context, repo, root, taskID string) (string, error) {
path, err := a.Client.Worktree(ctx, repo, filepath.Join(root, taskID), "orchestra/"+taskID)
if err != nil {
return "", err
}
if path == "" {
return "", fmt.Errorf("adapter: herdr returned empty worktree path")
}
return path, nil
}
func (a CLIAdapter) Lease(ctx context.Context, task, worktree string) (Session, error) {
if a.Client == nil {
return Session{}, fmt.Errorf("adapter: client required")
}
var s Session
e := a.Client.Call(ctx, "pane.create", map[string]string{"harness": a.Harness, "task_id": task, "worktree": worktree}, &s)
s.Harness = a.Harness
s.Worktree = worktree
return s, e
s, err := a.Client.StartAgent(ctx, worktree, worktree, "orchestra/"+task, a.Harness, task)
if err != nil {
return Session{}, err
}
if err := a.Client.Prompt(ctx, s.PaneID, fmt.Sprintf("Begin Orchestra task %s. Inspect the repository, understand the task context, and proceed with the requested work.", task), 0); err != nil {
return Session{}, err
}
return s, nil
}
func (a CLIAdapter) Bootstrap(ctx context.Context, s Session, ref string) error {
return a.Client.Prompt(ctx, s.PaneID, fmt.Sprintf("Read handoff %s, validate the anchor and TASK.md, then continue.", ref), time.Minute)
@@ -57,23 +89,96 @@ func (a CLIAdapter) Kill(ctx context.Context, s Session) error {
return a.Client.Call(ctx, "pane.kill", s, nil)
}
func (a CLIAdapter) AtTurnBoundary(ctx context.Context, s Session) (bool, error) {
var r struct {
Status string `json:"status"`
}
if err := a.Client.Call(ctx, "pane.status", s, &r); err != nil {
status, err := a.AgentStatus(ctx, s)
if err != nil {
return false, err
}
return !IsBusy(r.Status), nil
return !IsBusy(status), nil
}
func (a CLIAdapter) PaneExited(ctx context.Context, s Session) (bool, error) {
var r struct {
Status string `json:"status"`
}
if err := a.Client.Call(ctx, "pane.status", s, &r); err != nil {
status, err := a.AgentStatus(ctx, s)
if err != nil {
return false, err
}
return strings.EqualFold(r.Status, "exited") || strings.EqualFold(r.Status, "dead"), nil
return strings.EqualFold(status, "exited") || strings.EqualFold(status, "dead"), nil
}
func (a CLIAdapter) AgentStatus(ctx context.Context, s Session) (string, error) {
// Current herdr protocol exposes agent state through agent.get; older
// Orchestra code used pane.status, which is not a valid protocol method.
var r map[string]any
if err := a.Client.Call(ctx, "agent.get", map[string]any{"target": s.PaneID}, &r); err != nil {
return "", err
}
return statusFromAgentResult(r), nil
}
func (a CLIAdapter) AgentBlocker(ctx context.Context, s Session) (string, error) {
var r struct {
Read struct {
Text string `json:"text"`
} `json:"read"`
}
if err := a.Client.Call(ctx, "pane.read", map[string]any{"pane_id": s.PaneID, "source": "recent"}, &r); err != nil {
return "", err
}
text := strings.TrimSpace(r.Read.Text)
lines := strings.Split(text, "\n")
for i, raw := range lines {
line := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(raw), "┃"))
if !strings.EqualFold(line, "Permission required") && !strings.EqualFold(line, "Approval required") && !strings.HasPrefix(strings.ToLower(line), "waiting for") {
continue
}
for _, next := range lines[i+1:] {
command := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(next), "┃"))
if strings.HasPrefix(command, "$ ") {
return strings.ToLower(line) + ": shell command `" + strings.TrimSpace(strings.TrimPrefix(command, "$ ")) + "`", nil
}
}
return strings.ToLower(line), nil
}
return "", nil
}
func (a CLIAdapter) PaneCapture(ctx context.Context, s Session, source string) (string, error) {
if source == "" {
source = "recent"
}
var r struct {
Read struct {
Text string `json:"text"`
} `json:"read"`
}
if err := a.Client.Call(ctx, "pane.read", map[string]any{"pane_id": s.PaneID, "source": source}, &r); err != nil {
return "", err
}
return r.Read.Text, nil
}
func statusFromAgentResult(v any) string {
if m, ok := v.(map[string]any); ok {
for _, key := range []string{"status", "agent_status", "state"} {
if s, ok := m[key].(string); ok && s != "" {
return s
}
}
for _, child := range m {
if s := statusFromAgentResult(child); s != "" {
return s
}
}
}
if a, ok := v.([]any); ok {
for _, child := range a {
if s := statusFromAgentResult(child); s != "" {
return s
}
}
}
return ""
}
var _ = json.RawMessage{}
func (a CLIAdapter) RotationSignal(ctx context.Context, s Session) (string, error) {
var r struct {
Reason string `json:"reason"`
+113 -14
View File
@@ -13,6 +13,7 @@ import (
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
)
@@ -22,13 +23,15 @@ var ErrProtocol = errors.New("herdr protocol error")
type Request struct {
ID string `json:"id"`
Method string `json:"method"`
Params any `json:"params,omitempty"`
// Herdr's JSON-RPC decoder requires params to be present, including for
// parameterless calls such as ping. Encode nil as an explicit JSON null.
Params any `json:"params"`
}
type Response struct {
ID string `json:"id"`
Result json.RawMessage `json:"result"`
Error *struct {
Code int `json:"code"`
Code string `json:"code"`
Message string `json:"message"`
} `json:"error,omitempty"`
}
@@ -38,14 +41,39 @@ type Client struct {
dial func() (net.Conn, error)
mu sync.Mutex
next uint64
panes map[string]string
}
func New(path string) *Client { return &Client{Path: path, Timeout: 10 * time.Second} }
type WorktreeInfo struct {
Path string `json:"path"`
}
type worktreeResponse struct {
Path string `json:"path"`
Worktree WorktreeInfo `json:"worktree"`
RootPane struct {
PaneID string `json:"pane_id"`
Agent string `json:"agent"`
} `json:"root_pane"`
Workspace struct {
RootPane struct {
PaneID string `json:"pane_id"`
} `json:"root_pane"`
} `json:"workspace"`
}
func New(path string) *Client {
return &Client{Path: path, Timeout: 10 * time.Second, panes: map[string]string{}}
}
func (c *Client) conn() (net.Conn, error) {
if c.dial != nil {
return c.dial()
}
return net.DialTimeout("unix", c.Path, c.Timeout)
network := "unix"
if strings.Contains(c.Path, "://") || (strings.Contains(c.Path, ":") && !strings.HasPrefix(c.Path, "/")) {
network = "tcp"
}
return net.DialTimeout(network, c.Path, c.Timeout)
}
func (c *Client) Call(ctx context.Context, method string, params any, out any) error {
c.mu.Lock()
@@ -62,6 +90,9 @@ func (c *Client) Call(ctx context.Context, method string, params any, out any) e
} else if c.Timeout > 0 {
_ = cn.SetDeadline(time.Now().Add(c.Timeout))
}
if params == nil {
params = map[string]any{}
}
if err = json.NewEncoder(cn).Encode(Request{ID: id, Method: method, Params: params}); err != nil {
return err
}
@@ -79,8 +110,8 @@ func (c *Client) Call(ctx context.Context, method string, params any, out any) e
}
type PingResult struct {
Protocol string `json:"protocol"`
Version string `json:"version"`
Protocol json.RawMessage `json:"protocol"`
Version json.RawMessage `json:"version"`
}
func (c *Client) Ping(ctx context.Context) (PingResult, error) {
@@ -93,8 +124,14 @@ func (c *Client) CheckProtocol(ctx context.Context, want string) error {
if e != nil {
return e
}
if want != "" && p.Protocol != want {
return fmt.Errorf("%w: want %s, got %s", ErrProtocol, want, p.Protocol)
if want != "" {
var text string
if err := json.Unmarshal(p.Protocol, &text); err != nil {
text = string(p.Protocol)
}
if text != want {
return fmt.Errorf("%w: want %s, got %s", ErrProtocol, want, text)
}
}
return nil
}
@@ -103,18 +140,80 @@ type Session struct {
PaneID string `json:"pane_id"`
Worktree string `json:"worktree"`
Harness string `json:"harness"`
HerdrID string `json:"herdr_id,omitempty"`
}
func (c *Client) Prompt(ctx context.Context, pane, text string, wait time.Duration) error {
p := map[string]any{"pane_id": pane, "prompt": text, "wait": map[string]any{"until": "turn_end", "timeout_ms": wait.Milliseconds()}}
p := map[string]any{"target": pane, "text": text}
if wait > 0 {
p["wait"] = map[string]any{"until": []string{"idle"}, "timeout_ms": wait.Milliseconds()}
}
return c.Call(ctx, "agent.prompt", p, nil)
}
func (c *Client) Worktree(ctx context.Context, path, branch string) (string, error) {
var r struct {
Path string `json:"path"`
func (c *Client) Worktree(ctx context.Context, cwd, path, branch string) (string, error) {
var r worktreeResponse
// Protocol 17 requires exactly one of path or branch. Use the explicit
// path so the worker owns the checkout location; herdr manages the branch
// associated with that worktree.
p := map[string]any{"cwd": cwd, "path": path}
e := c.Call(ctx, "worktree.create", p, &r)
if e != nil && strings.Contains(strings.ToLower(e.Error()), "already exists") {
e = c.Call(ctx, "worktree.open", map[string]any{"cwd": cwd, "path": path}, &r)
}
e := c.Call(ctx, "worktree.create", map[string]string{"path": path, "branch": branch}, &r)
return r.Path, e
if e != nil {
return "", e
}
if r.RootPane.PaneID != "" {
c.mu.Lock()
c.panes[path] = r.RootPane.PaneID
c.mu.Unlock()
}
if r.Path != "" {
return r.Path, nil
}
return r.Worktree.Path, nil
}
func (c *Client) StartAgent(ctx context.Context, cwd, path, branch, harness, taskID string) (Session, error) {
c.mu.Lock()
paneID := c.panes[path]
c.mu.Unlock()
if paneID == "" {
return Session{}, fmt.Errorf("herdr: no pane recorded for worktree %s", path)
}
var s Session
if err := c.Call(ctx, "agent.start", map[string]any{
"pane_id": paneID,
"kind": harness,
"name": harness,
"args": []string{},
}, &s); err != nil {
if !strings.Contains(strings.ToLower(err.Error()), "already") {
return Session{}, err
}
}
s.PaneID = paneID
s.Worktree = path
s.Harness = harness
return s, nil
}
// HeadSHA returns the current commit of a worktree. The rotation path uses
// this to populate TaskReleased.anchor_sha without trusting the adapter's
// opaque handoff-ref return value.
func HeadSHA(root string) (string, error) {
out, err := exec.Command("git", "-C", root, "rev-parse", "HEAD").Output()
if err != nil {
return "", err
}
sha := string(out)
if len(sha) > 0 && sha[len(sha)-1] == '\n' {
sha = sha[:len(sha)-1]
}
if len(sha) != 40 {
return "", fmt.Errorf("herdr: unexpected HEAD output %q", sha)
}
return sha, nil
}
// AnchorValid checks the split-then-close safety condition without trusting a
+7 -6
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/herdr"
"orchestra/internal/orchestrator"
@@ -128,7 +129,7 @@ func TestEndToEndIngestRouteLeaseRotateAndComplete(t *testing.T) {
}
got, _ := s.Task(task.ID)
if got.State == domain.StateLeased && h.releases == 1 {
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: got.Version + 1, Payload: mustJSON(map[string]string{
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: got.Version + 1, Surface: string(authz.System), Payload: mustJSON(map[string]string{
"handoff_ref": h.ref,
"anchor_sha": "0123456789012345678901234567890123456789",
})}); err != nil {
@@ -153,7 +154,7 @@ func TestEndToEndIngestRouteLeaseRotateAndComplete(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: got.Version + 1, Payload: mustJSON(map[string]any{
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: got.Version + 1, Surface: string(authz.System), Payload: mustJSON(map[string]any{
"report_ref": ref,
"receipt": map[string]any{"harness_id": "h1", "consumed": 1},
})}); err != nil {
@@ -181,7 +182,7 @@ func TestRestartReplayAndReconcileKillsOrphan(t *testing.T) {
}
// A fresh coordinator sees the durable session, then drops it once the lease is gone.
ref, _ := s.PutArtifact([]byte("handoff"))
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: 3, Payload: mustJSON(map[string]string{
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: task.ID, Version: 3, Surface: string(authz.System), Payload: mustJSON(map[string]string{
"handoff_ref": ref,
"anchor_sha": "0123456789012345678901234567890123456789",
})}); err != nil {
@@ -207,10 +208,10 @@ func TestRestartReplayAndReconcileKillsOrphan(t *testing.T) {
func TestProviderRetryReflectionQuotaAndVersionConflict(t *testing.T) {
s, r, _ := setup(t)
task := ingest(t, s, "retry")
if err := s.Append(domain.Event{Type: "QuotaReported", TaskID: "quota", Version: 1, Payload: mustJSON(map[string]any{"harness_id": "h1", "consumed": 90.0})}); err != nil {
if err := s.Append(domain.Event{Type: "QuotaReported", TaskID: "quota", Version: 1, Surface: string(authz.System), Payload: mustJSON(map[string]any{"harness_id": "h1", "consumed": 90.0})}); err != nil {
t.Fatal(err)
}
rt := router.Router{Store: s, Registry: r, Reachability: alwaysReachable{}, Availability: router.QuotaAvailability{Store: s, Limits: map[string]float64{"h1": 100}, Now: time.Now}}
rt := router.Router{Store: s, Registry: r, Reachability: alwaysReachable{}, Availability: router.QuotaAvailability{Store: s, Limits: map[string]router.QuotaWindowLimits{"h1": {Weekly: 100}}, Now: time.Now}}
if got, _ := rt.AssignPending(); len(got) != 0 {
t.Fatalf("quota assigned=%d", len(got))
}
@@ -225,7 +226,7 @@ func TestProviderRetryReflectionQuotaAndVersionConflict(t *testing.T) {
t.Fatal(err)
}
reflector := &fakeReflector{}
if err := (provider.ReflectingSink{Sink: s, Tasks: s, Reflector: reflector}).Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: 3, Payload: mustJSON(map[string]any{
if err := (provider.ReflectingSink{Sink: s, Tasks: s, Reflector: reflector}).Append(domain.Event{ID: domain.NewID(), Type: "TaskCompleted", TaskID: task.ID, Version: 3, Surface: string(authz.System), Payload: mustJSON(map[string]any{
"report_ref": ref,
"receipt": map[string]any{"harness_id": "h1", "consumed": 1},
})}); err != nil {
@@ -0,0 +1,213 @@
package integration
import (
"context"
"encoding/json"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/federation"
"orchestra/internal/herdr"
"orchestra/internal/operations"
"orchestra/internal/orchestrator"
"orchestra/internal/registry"
"orchestra/internal/router"
"orchestra/internal/store"
"os/exec"
"testing"
"time"
)
func gitInit(t *testing.T, dir, marker string) string {
t.Helper()
run := func(args ...string) {
cmd := exec.Command("git", append([]string{"-C", dir}, args...)...)
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("git %v: %v: %s", args, err, out)
}
}
run("init")
run("config", "user.email", "t@t")
run("config", "user.name", "t")
run("commit", "--allow-empty", "-m", "init: "+marker)
head, err := herdr.HeadSHA(dir)
if err != nil {
t.Fatal(err)
}
return head
}
type fixedWorktree struct{ path string }
func (w fixedWorktree) Create(context.Context, domain.Task) (string, error) { return w.path, nil }
// machineAdapter is like harness but actually threads the worktree it was
// leased against into the Session, the way a real herdr adapter must — the
// shared harness fixture ignores it, which is fine for single-checkout
// tests but would silently defeat this one's anchor_sha assertions.
type machineAdapter struct{ *harness }
func (a machineAdapter) Lease(_ context.Context, _ string, worktree string) (herdr.Session, error) {
return herdr.Session{Harness: "h1", PaneID: "pane-1", Worktree: worktree}, nil
}
type singleAdapter struct{ a herdr.Adapter }
func (s singleAdapter) Adapter(string) (herdr.Adapter, error) { return s.a, nil }
// TestCrossMachineLeaseAnchorAndQuotaArePerHost exercises spec §2.1/§9 open
// question 8 end to end using two independent local git checkouts standing
// in for homesrv and workpc: each machine's Coordinator only ever validates
// and stamps anchor_sha from its OWN local checkout (never the other
// machine's), and quota consumption is accounted strictly per harness/host
// so one machine exhausting its window cannot block the other from being
// leased work. Federation registration/heartbeat plumbing (already covered
// by internal/federation's own tests) is exercised alongside this to prove
// the pieces fit together, not just in isolation.
func TestCrossMachineLeaseAnchorAndQuotaArePerHost(t *testing.T) {
homesrvRepo, workpcRepo := t.TempDir(), t.TempDir()
homesrvHead := gitInit(t, homesrvRepo, "homesrv")
workpcHead := gitInit(t, workpcRepo, "workpc")
if homesrvHead == workpcHead {
t.Fatal("test setup: expected distinct checkouts")
}
s, err := store.Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
// The federation registry is homesrv's view of workpc as an intermittent
// worker (spec §2.1): it must be registered and reachable before the
// router would ever consider leasing to it.
workers := &federation.Registry{}
if err := workers.Register(federation.Worker{ID: "workpc", Address: "workpc.mesh", Token: "secret"}); err != nil {
t.Fatal(err)
}
if err := workers.Heartbeat("workpc"); err != nil {
t.Fatal(err)
}
online := false
for _, w := range workers.Snapshot() {
if w.ID == "workpc" && w.Online {
online = true
}
}
if !online {
t.Fatal("workpc worker should be online after heartbeat")
}
// Two tasks, one leased to each machine's harness.
mk := func(id string) domain.Task {
b, _ := json.Marshal(map[string]any{"source": "jsonl", "external_id": id, "project": "p"})
if err := s.Append(domain.Event{ID: id, Type: "TaskCreated", TaskID: id, Version: 1, Surface: string(authz.System), Payload: b}); err != nil {
t.Fatal(err)
}
task, _ := s.Task(id)
return task
}
homesrvTask := mk("home-task")
workpcTask := mk("workpc-task")
homesrvAdapter := &harness{occupancy: .95, boundary: true, ref: mustArtifact(t, s, "home-handoff")}
workpcAdapter := &harness{occupancy: .95, boundary: true, ref: mustArtifact(t, s, "workpc-handoff")}
homesrvCoord := &orchestrator.Coordinator{Store: s, Worktrees: fixedWorktree{homesrvRepo}, Adapters: singleAdapter{machineAdapter{homesrvAdapter}}, StatePath: t.TempDir() + "/home-sessions.json"}
workpcCoord := &orchestrator.Coordinator{Store: s, Worktrees: fixedWorktree{workpcRepo}, Adapters: singleAdapter{machineAdapter{workpcAdapter}}, StatePath: t.TempDir() + "/workpc-sessions.json"}
homesrvLease, err := s.Lease(homesrvTask.ID, "homesrv-h1", time.Minute)
if err != nil {
t.Fatal(err)
}
if err := homesrvCoord.Start(context.Background(), homesrvLease); err != nil {
t.Fatal(err)
}
workpcLease, err := s.Lease(workpcTask.ID, "workpc-h1", time.Minute)
if err != nil {
t.Fatal(err)
}
if err := workpcCoord.Start(context.Background(), workpcLease); err != nil {
t.Fatal(err)
}
ctxHome, cancelHome := context.WithCancel(context.Background())
defer cancelHome()
ctxWork, cancelWork := context.WithCancel(context.Background())
defer cancelWork()
go homesrvCoord.Monitor(ctxHome, .8, time.Millisecond)
go workpcCoord.Monitor(ctxWork, .8, time.Millisecond)
waitQueued := func(id string) domain.Task {
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
if t, ok := s.Task(id); ok && t.State == domain.StateQueued {
return t
}
time.Sleep(time.Millisecond)
}
t.Fatalf("task %s never rotated back to queued", id)
return domain.Task{}
}
waitQueued(homesrvTask.ID)
waitQueued(workpcTask.ID)
// Each machine's coordinator must have stamped anchor_sha from its OWN
// checkout — never the other machine's HEAD, and never each other's.
anchorFor := func(taskID string) string {
for _, e := range s.Events(0) {
if e.TaskID != taskID || e.Type != "TaskReleased" {
continue
}
var p map[string]any
_ = json.Unmarshal(e.Payload, &p)
return p["anchor_sha"].(string)
}
t.Fatalf("no TaskReleased found for %s", taskID)
return ""
}
if got := anchorFor(homesrvTask.ID); got != homesrvHead {
t.Fatalf("homesrv anchor_sha=%s want=%s (must validate against its own checkout)", got, homesrvHead)
}
if got := anchorFor(workpcTask.ID); got != workpcHead {
t.Fatalf("workpc anchor_sha=%s want=%s (must validate against its own checkout, not homesrv's)", got, workpcHead)
}
// Quota is accounted per host/harness: exhausting homesrv-h1 must not
// affect workpc-h1's availability, and vice versa (spec §2.1, §7.2).
quotaReport := func(harness string, consumed float64) {
p, _ := json.Marshal(map[string]any{"harness_id": harness, "consumed": consumed})
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "QuotaReported", TaskID: "quota", Version: 1, Surface: string(authz.System), Payload: p}); err != nil {
t.Fatal(err)
}
}
quotaReport("homesrv-h1", 95)
limits := map[string]router.QuotaWindowLimits{
"homesrv-h1": {Weekly: 100},
"workpc-h1": {Weekly: 100},
}
avail := router.QuotaAvailability{Store: s, Limits: limits}
if avail.Available(registry.Herdr{ID: "homesrv-h1"}) {
t.Fatal("homesrv-h1 should be quota-exhausted at 95/100")
}
if !avail.Available(registry.Herdr{ID: "workpc-h1"}) {
t.Fatal("workpc-h1 exhaustion leaked from homesrv-h1's per-host accounting")
}
// The windowed quota aggregation used by the brief (spec §7.4) must also
// keep the two hosts separate.
sums := operations.AggregateQuota(s.Events(0), time.Now().Add(-time.Hour), time.Now().Add(time.Hour))
if sums["homesrv-h1"] != 95 {
t.Fatalf("homesrv-h1 aggregate=%v want 95", sums["homesrv-h1"])
}
if sums["workpc-h1"] != 0 {
t.Fatalf("workpc-h1 aggregate=%v want 0 (must not inherit homesrv-h1's receipts)", sums["workpc-h1"])
}
}
func mustArtifact(t *testing.T, s *store.Store, content string) string {
t.Helper()
ref, err := s.PutArtifact([]byte(content))
if err != nil {
t.Fatal(err)
}
return ref
}
+3 -2
View File
@@ -4,6 +4,7 @@ package operations
import (
"encoding/json"
"fmt"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/store"
"os/exec"
@@ -99,7 +100,7 @@ func GenerateStandupAdvisory(s *store.Store, at time.Time) (domain.Event, error)
if err != nil {
return domain.Event{}, err
}
e := domain.Event{ID: domain.NewID(), TaskID: "system", Type: "StandupAdvisory", Payload: p, At: at}
e := domain.Event{ID: domain.NewID(), TaskID: "system", Type: "StandupAdvisory", Payload: p, At: at, Surface: string(authz.System)}
return e, s.Append(e)
}
@@ -145,7 +146,7 @@ func ApplyAdvisory(s *store.Store, advisoryID string) ([]domain.Event, error) {
continue
}
b, _ := json.Marshal(map[string]any{"title": item.Title, "advisory_ref": advisoryID})
e := domain.Event{ID: domain.NewID(), TaskID: item.TaskID, Type: "TaskAmended", Version: t.Version + 1, Payload: b}
e := domain.Event{ID: domain.NewID(), TaskID: item.TaskID, Type: "TaskAmended", Version: t.Version + 1, Payload: b, Surface: string(authz.System)}
if err := s.Append(e); err != nil {
return out, err
}
+5 -4
View File
@@ -2,6 +2,7 @@ package operations
import (
"encoding/json"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/store"
"testing"
@@ -22,7 +23,7 @@ func TestAggregateQuotaSumsRotationsAndWindows(t *testing.T) {
now := time.Now().UTC()
enc := func(at time.Time, n float64) domain.Event {
p, _ := json.Marshal(map[string]any{"harness_id": "codex", "consumed": n})
return domain.Event{Type: "QuotaReported", At: at, Payload: p}
return domain.Event{Type: "QuotaReported", At: at, Payload: p, Surface: string(authz.System)}
}
es := []domain.Event{enc(now.Add(-2*time.Hour), 4), enc(now.Add(-time.Hour), 6), enc(now.Add(-48*time.Hour), 100)}
got := AggregateQuota(es, now.Add(-3*time.Hour), now)
@@ -37,11 +38,11 @@ func TestApplyAdvisoryRequiresApproval(t *testing.T) {
t.Fatal(err)
}
p, _ := json.Marshal(map[string]any{"source": "test", "external_id": "1", "project": "p", "title": "old"})
if err := s.Append(domain.Event{ID: "task", TaskID: "task", Type: "TaskCreated", Version: 1, Payload: p}); err != nil {
if err := s.Append(domain.Event{ID: "task", TaskID: "task", Type: "TaskCreated", Version: 1, Payload: p, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
ap, _ := json.Marshal(map[string]any{"items": []StandupItem{{TaskID: "task", Title: "new"}}})
adv := domain.Event{ID: "adv", TaskID: "system", Type: "StandupAdvisory", Payload: ap}
adv := domain.Event{ID: "adv", TaskID: "system", Type: "StandupAdvisory", Payload: ap, Surface: string(authz.System)}
if err := s.Append(adv); err != nil {
t.Fatal(err)
}
@@ -49,7 +50,7 @@ func TestApplyAdvisoryRequiresApproval(t *testing.T) {
t.Fatal("unapproved advisory applied")
}
grant, _ := json.Marshal(map[string]any{"subject_ref": "adv"})
if err := s.Append(domain.Event{ID: "grant", TaskID: "system", Type: "ApprovalGranted", Payload: grant}); err != nil {
if err := s.Append(domain.Event{ID: "grant", TaskID: "system", Type: "ApprovalGranted", Payload: grant, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
if _, err := ApplyAdvisory(s, "adv"); err != nil {
+248 -11
View File
@@ -7,6 +7,7 @@ import (
"context"
"encoding/json"
"fmt"
"orchestra/internal/authz"
"orchestra/internal/continuity"
"orchestra/internal/domain"
"orchestra/internal/herdr"
@@ -14,6 +15,7 @@ import (
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
)
@@ -21,6 +23,12 @@ import (
type Worktrees interface {
Create(context.Context, domain.Task) (string, error)
}
type WorktreeSpec interface {
Spec(domain.Task) (string, string, bool)
}
type WorktreeCleaner interface {
Remove(context.Context, domain.Task, string) error
}
type Adapters interface {
Adapter(string) (herdr.Adapter, error)
}
@@ -34,6 +42,10 @@ type GitWorktrees struct {
TaskFileSHA string
}
func (w GitWorktrees) Spec(domain.Task) (string, string, bool) {
return w.Repo, w.Root, w.Repo != "" && w.Root != ""
}
func (w GitWorktrees) Create(ctx context.Context, t domain.Task) (string, error) {
if w.Root == "" || w.Repo == "" {
return "", fmt.Errorf("worktree: root and repo required")
@@ -63,6 +75,62 @@ func (w GitWorktrees) Create(ctx context.Context, t domain.Task) (string, error)
return p, nil
}
func (w GitWorktrees) Remove(ctx context.Context, _ domain.Task, path string) error {
if path == "" {
return fmt.Errorf("worktree: path required")
}
cmd := exec.CommandContext(ctx, "git", "-C", w.Repo, "worktree", "remove", "--force", path)
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("%s: %w", string(out), err)
}
return nil
}
// ProjectRepo is the minimal shape PerProjectGitWorktrees needs from a
// project's registry entry — kept local (not importing internal/registry)
// so orchestrator does not depend on registry's config-loading concerns.
type ProjectRepo struct {
Repo string
WorktreeRoot string
}
// PerProjectGitWorktrees resolves a task's repo/root by its project (spec
// §2.2: each project is first-class and may have its own checkout), falling
// back to Default for any project not present in Projects — this keeps
// single-repo deployments working unchanged.
type PerProjectGitWorktrees struct {
Projects map[string]ProjectRepo
Default GitWorktrees
TaskFileSHA string
}
func (w PerProjectGitWorktrees) Spec(t domain.Task) (string, string, bool) {
g := w.Default
if p, ok := w.Projects[t.Project]; ok && p.Repo != "" && p.WorktreeRoot != "" {
g = GitWorktrees{Root: p.WorktreeRoot, Repo: p.Repo}
}
return g.Spec(t)
}
func (w PerProjectGitWorktrees) Create(ctx context.Context, t domain.Task) (string, error) {
g := w.Default
if p, ok := w.Projects[t.Project]; ok && p.Repo != "" && p.WorktreeRoot != "" {
g = GitWorktrees{Root: p.WorktreeRoot, Repo: p.Repo, TaskFileSHA: w.TaskFileSHA}
}
if g.TaskFileSHA == "" {
g.TaskFileSHA = w.TaskFileSHA
}
return g.Create(ctx, t)
}
func (w PerProjectGitWorktrees) Remove(ctx context.Context, t domain.Task, path string) error {
g := w.Default
if p, ok := w.Projects[t.Project]; ok && p.Repo != "" && p.WorktreeRoot != "" {
g = GitWorktrees{Root: p.WorktreeRoot, Repo: p.Repo}
}
return g.Remove(ctx, t, path)
}
type AdapterFactory struct{ Herdrs map[string]herdr.Adapter }
func (f AdapterFactory) Adapter(id string) (herdr.Adapter, error) {
@@ -90,6 +158,21 @@ type MonitorHealth struct {
LastRun time.Time `json:"last_run"`
LastError string `json:"last_error,omitempty"`
Expired int `json:"expired"`
// TurnBoundaryDegraded counts rotation ticks where Face B (spec §5.2,
// §5.3 — "the Stop-hook/Face-B decides rotation, not the router") could
// not be consulted, so occupancy-only thresholding is standing in. This
// must stay observable rather than a silent fallback: an operator (or
// the brief) can see when a deployment's rotation safety is degraded.
TurnBoundaryDegraded int `json:"turn_boundary_degraded"`
Sessions map[string]SessionHealth `json:"sessions,omitempty"`
}
type SessionHealth struct {
Status string `json:"status,omitempty"`
WaitingForApproval bool `json:"waiting_for_approval"`
Blocker string `json:"blocker,omitempty"`
UpdatedAt time.Time `json:"updated_at"`
LastError string `json:"last_error,omitempty"`
}
func (c *Coordinator) MonitorHealth() MonitorHealth {
@@ -110,6 +193,58 @@ func (c *Coordinator) setMonitorHealth(err error, expired int) {
}
}
func (c *Coordinator) recordTurnBoundaryDegraded() {
c.healthMu.Lock()
defer c.healthMu.Unlock()
c.health.TurnBoundaryDegraded++
}
func waitingForApproval(status string) bool {
s := strings.ToLower(strings.ReplaceAll(strings.ReplaceAll(status, "-", "_"), " ", "_"))
return s == "waiting_for_approval" || s == "awaiting_approval" || s == "approval_required"
}
func (c *Coordinator) refreshSessionHealth(ctx context.Context) {
c.loadSessions()
c.mu.Lock()
sessions := make(map[string]herdr.Session, len(c.sessions))
for id, s := range c.sessions {
sessions[id] = s
}
c.mu.Unlock()
c.healthMu.Lock()
if c.health.Sessions == nil {
c.health.Sessions = map[string]SessionHealth{}
}
c.healthMu.Unlock()
for taskID, session := range sessions {
adapterID := session.HerdrID
if adapterID == "" {
if task, ok := c.Store.Task(taskID); ok && task.Lease != nil {
adapterID = task.Lease.HarnessID
}
}
a, err := c.Adapters.Adapter(adapterID)
if err != nil {
continue
}
p, ok := a.(herdr.AgentStatus)
if !ok {
continue
}
status, err := p.AgentStatus(ctx, session)
h := SessionHealth{Status: status, WaitingForApproval: waitingForApproval(status), UpdatedAt: time.Now().UTC()}
if err != nil {
h.LastError = err.Error()
} else if blocker, ok := a.(herdr.AgentBlocker); ok && strings.EqualFold(status, "blocked") {
h.Blocker, _ = blocker.AgentBlocker(ctx, session)
}
c.healthMu.Lock()
c.health.Sessions[taskID] = h
c.healthMu.Unlock()
}
}
func (c *Coordinator) loadSessions() {
c.mu.Lock()
defer c.mu.Unlock()
@@ -148,7 +283,6 @@ func (c *Coordinator) saveSessionsLocked() error {
func (c *Coordinator) Reconcile(ctx context.Context) error {
c.loadSessions()
c.mu.Lock()
defer c.mu.Unlock()
for taskID, session := range c.sessions {
t, ok := c.Store.Task(taskID)
if ok && t.State == domain.StateLeased {
@@ -159,7 +293,16 @@ func (c *Coordinator) Reconcile(ctx context.Context) error {
}
delete(c.sessions, taskID)
}
return c.saveSessionsLocked()
err := c.saveSessionsLocked()
c.mu.Unlock()
if err != nil {
return err
}
// Session health is derived state. Rebuild it immediately from the
// durable session mappings so a restart does not hide an outstanding
// approval until the first periodic monitor tick.
c.refreshSessionHealth(ctx)
return nil
}
// Monitor performs conservative hard-threshold rotation. The adapter owns
@@ -183,6 +326,8 @@ func (c *Coordinator) Monitor(ctx context.Context, hard float64, interval time.D
c.healthMu.Unlock()
return ctx.Err()
case <-t.C:
c.refreshSessionHealth(ctx)
c.cleanupCompleted(ctx)
expired, err := c.expire(ctx)
c.setMonitorHealth(err, len(expired))
if err != nil {
@@ -193,6 +338,34 @@ func (c *Coordinator) Monitor(ctx context.Context, hard float64, interval time.D
}
}
func (c *Coordinator) cleanupCompleted(ctx context.Context) {
cleaner, ok := c.Worktrees.(WorktreeCleaner)
if !ok {
return
}
c.loadSessions()
c.mu.Lock()
defer c.mu.Unlock()
changed := false
for taskID, session := range c.sessions {
t, exists := c.Store.Task(taskID)
if !exists || t.State != domain.StateCompleted {
continue
}
if err := cleaner.Remove(ctx, t, session.Worktree); err != nil {
c.healthMu.Lock()
c.health.LastError = "worktree cleanup: " + err.Error()
c.healthMu.Unlock()
continue
}
delete(c.sessions, taskID)
changed = true
}
if changed {
_ = c.saveSessionsLocked()
}
}
func (c *Coordinator) expire(ctx context.Context) ([]domain.Event, error) {
// pane.exited is the low-latency path; lease expiry below remains the
// authoritative backstop when herdr misses an exit notification.
@@ -204,7 +377,7 @@ func (c *Coordinator) expire(ctx context.Context) ([]domain.Event, error) {
if p, ok := a.(herdr.PaneExit); ok {
if exited, ee := p.PaneExited(ctx, s); ee == nil && exited {
b, _ := json.Marshal(map[string]string{"reason": "pane_exited", "harness_id": s.Harness})
_ = c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: t.Version + 1, Payload: b})
_ = c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)})
}
}
}
@@ -257,11 +430,25 @@ func (c *Coordinator) rotate(ctx context.Context, hard float64) {
if err != nil || (occupancy < hard && reason == "threshold") {
continue
}
// Face B is treated as required, not best-effort (spec §5.2/§5.3):
// an adapter that supports the turn-boundary probe but fails to
// answer it blocks this tick's release rather than silently
// proceeding as if mid-turn interruption were safe. Only an
// adapter that genuinely does not implement TurnBoundary at all
// falls back to occupancy-only thresholding, and that fallback is
// recorded so it is observable (MonitorHealth.TurnBoundaryDegraded)
// instead of invisible.
if boundary, ok := a.(herdr.TurnBoundary); ok {
atBoundary, boundaryErr := boundary.AtTurnBoundary(ctx, session)
if boundaryErr == nil && !atBoundary {
if boundaryErr != nil {
c.recordTurnBoundaryDegraded()
continue
}
if !atBoundary {
continue
}
} else {
c.recordTurnBoundaryDegraded()
}
ref, err := a.Release(ctx, session)
if err != nil {
@@ -270,8 +457,16 @@ func (c *Coordinator) rotate(ctx context.Context, hard float64) {
if ref == "" {
continue
}
b, _ := json.Marshal(map[string]string{"handoff_ref": ref, "reason": reason})
e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: task.Version + 1, Payload: b}
anchorSHA, err := herdr.HeadSHA(session.Worktree)
if err != nil {
// Cannot certify the anchor: do not release with an invalid
// TaskReleased payload (it would fail validation and strand
// the lease/session). Leave the lease intact for the next
// tick or TTL expiry to reclaim.
continue
}
b, _ := json.Marshal(map[string]string{"handoff_ref": ref, "reason": reason, "anchor_sha": anchorSHA})
e := domain.Event{ID: domain.NewID(), Type: "TaskReleased", TaskID: taskID, Version: task.Version + 1, Payload: b, Surface: string(authz.System)}
if c.Store.Append(e) == nil {
c.mu.Lock()
delete(c.sessions, taskID)
@@ -300,14 +495,27 @@ func (c *Coordinator) Start(ctx context.Context, e domain.Event) error {
if err := json.Unmarshal(e.Payload, &p); err != nil || p.HarnessID == "" {
return fmt.Errorf("orchestrator: invalid lease")
}
w, err := c.Worktrees.Create(ctx, t)
if err != nil {
return c.block(t, "worktree: "+err.Error())
}
a, err := c.Adapters.Adapter(p.HarnessID)
if err != nil {
return c.block(t, "adapter: "+err.Error())
}
var w string
if creator, ok := a.(herdr.WorktreeCreator); ok {
planner, planned := c.Worktrees.(WorktreeSpec)
if !planned {
return c.block(t, "worktree: repository specification unavailable")
}
repo, root, valid := planner.Spec(t)
if !valid {
return c.block(t, "worktree: repository and root required")
}
w, err = creator.CreateWorktree(ctx, repo, root, t.ID)
} else {
w, err = c.Worktrees.Create(ctx, t)
}
if err != nil {
return c.block(t, "worktree: "+err.Error())
}
s, err := a.Lease(ctx, t.ID, w)
if err != nil {
return c.block(t, "lease: "+err.Error())
@@ -318,6 +526,7 @@ func (c *Coordinator) Start(ctx context.Context, e domain.Event) error {
return c.block(t, "bootstrap: "+err.Error())
}
}
s.HerdrID = p.HarnessID
c.mu.Lock()
if c.sessions == nil {
c.sessions = map[string]herdr.Session{}
@@ -325,12 +534,18 @@ func (c *Coordinator) Start(ctx context.Context, e domain.Event) error {
c.sessions[t.ID] = s
err = c.saveSessionsLocked()
c.mu.Unlock()
c.healthMu.Lock()
if c.health.Sessions == nil {
c.health.Sessions = map[string]SessionHealth{}
}
c.health.Sessions[t.ID] = SessionHealth{Status: "running", UpdatedAt: time.Now().UTC()}
c.healthMu.Unlock()
return err
}
func (c *Coordinator) block(t domain.Task, reason string) error {
b, _ := json.Marshal(map[string]string{"blocker": reason})
return c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskBlocked", TaskID: t.ID, Version: t.Version + 1, Payload: b})
return c.Store.Append(domain.Event{ID: domain.NewID(), Type: "TaskBlocked", TaskID: t.ID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)})
}
func (c *Coordinator) Session(taskID string) (herdr.Session, bool) {
@@ -340,3 +555,25 @@ func (c *Coordinator) Session(taskID string) (herdr.Session, bool) {
s, ok := c.sessions[taskID]
return s, ok
}
func (c *Coordinator) Capture(ctx context.Context, taskID, source string) (string, error) {
s, ok := c.Session(taskID)
if !ok {
return "", domain.ErrNotFound
}
id := s.HerdrID
if id == "" {
if t, ok := c.Store.Task(taskID); ok && t.Lease != nil {
id = t.Lease.HarnessID
}
}
a, err := c.Adapters.Adapter(id)
if err != nil {
return "", err
}
p, ok := a.(herdr.PaneCapture)
if !ok {
return "", fmt.Errorf("pane capture unsupported")
}
return p.PaneCapture(ctx, s, source)
}
+267
View File
@@ -0,0 +1,267 @@
package orchestrator_test
import (
"context"
"encoding/json"
"errors"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/herdr"
"orchestra/internal/orchestrator"
"orchestra/internal/store"
"os/exec"
"testing"
"time"
)
type fakeAdapter struct {
occupancy float64
boundary bool
ref string
releases int
}
func (a *fakeAdapter) Lease(_ context.Context, _ string, worktree string) (herdr.Session, error) {
return herdr.Session{Harness: "h1", PaneID: "pane-1", Worktree: worktree}, nil
}
func (a *fakeAdapter) Bootstrap(context.Context, herdr.Session, string) error { return nil }
func (a *fakeAdapter) Release(context.Context, herdr.Session) (string, error) {
a.releases++
return a.ref, nil
}
func (a *fakeAdapter) Kill(context.Context, herdr.Session) error { return nil }
func (a *fakeAdapter) Occupancy(herdr.Session) (float64, error) { return a.occupancy, nil }
func (a *fakeAdapter) AtTurnBoundary(context.Context, herdr.Session) (bool, error) {
return a.boundary, nil
}
type worktrees struct{ path string }
func (w worktrees) Create(context.Context, domain.Task) (string, error) { return w.path, nil }
type adapters struct{ a herdr.Adapter }
func (a adapters) Adapter(string) (herdr.Adapter, error) { return a.a, nil }
func run(t *testing.T, dir string, args ...string) {
t.Helper()
cmd := exec.Command("git", append([]string{"-C", dir}, args...)...)
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("git %v: %v: %s", args, err, out)
}
}
// TestRotationEmitsValidReleaseWithAnchorSHA guards the highest-priority spec
// defect noted in progress.md: automated rotation must emit a TaskReleased
// event that satisfies domain.ValidatePayload (handoff_ref + anchor_sha), not
// a payload missing anchor_sha that silently fails to append.
func TestRotationEmitsValidReleaseWithAnchorSHA(t *testing.T) {
repo := t.TempDir()
run(t, repo, "init")
run(t, repo, "config", "user.email", "t@t")
run(t, repo, "config", "user.name", "t")
run(t, repo, "commit", "--allow-empty", "-m", "init")
head, err := herdr.HeadSHA(repo)
if err != nil {
t.Fatal(err)
}
s, err := store.Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: "t1", Surface: string(authz.System), Payload: mustJSON(map[string]any{
"source": "jsonl", "external_id": "1", "project": "p",
})}); err != nil {
t.Fatal(err)
}
task := s.Tasks()[0]
ref, err := s.PutArtifact([]byte("handoff"))
if err != nil {
t.Fatal(err)
}
a := &fakeAdapter{occupancy: .95, boundary: true, ref: ref}
c := &orchestrator.Coordinator{Store: s, Worktrees: worktrees{path: repo}, Adapters: adapters{a}, StatePath: t.TempDir() + "/sessions.json"}
leaseEvt, err := s.Lease(task.ID, "h1", time.Minute)
if err != nil {
t.Fatal(err)
}
if err := c.Start(context.Background(), leaseEvt); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go c.Monitor(ctx, .8, time.Millisecond)
deadline := time.Now().Add(time.Second)
for time.Now().Before(deadline) {
if got, ok := s.Task(task.ID); ok && got.State == domain.StateQueued {
break
}
time.Sleep(time.Millisecond)
}
got, ok := s.Task(task.ID)
if !ok || got.State != domain.StateQueued {
t.Fatalf("rotation did not complete: state=%v ok=%v", got.State, ok)
}
if a.releases == 0 {
t.Fatalf("adapter Release was never invoked")
}
// Walk raw events to confirm the coordinator itself wrote a valid
// TaskReleased payload with anchor_sha == the worktree's real HEAD.
found := false
for _, e := range s.Events(0) {
if e.TaskID != task.ID || e.Type != "TaskReleased" {
continue
}
var p map[string]any
if err := json.Unmarshal(e.Payload, &p); err != nil {
t.Fatal(err)
}
if err := domain.ValidatePayload("TaskReleased", p); err != nil {
t.Fatalf("coordinator emitted invalid TaskReleased: %v (%v)", err, p)
}
if p["anchor_sha"] != head {
t.Fatalf("anchor_sha=%v want=%s", p["anchor_sha"], head)
}
found = true
}
if !found {
t.Fatal("coordinator never emitted a TaskReleased event")
}
}
func mustJSON(v any) []byte { b, _ := json.Marshal(v); return b }
// erroringBoundaryAdapter supports Face B but its probe always fails — this
// must block release (never silently treat an unanswerable boundary check
// as safe to interrupt), unlike an adapter that doesn't implement the
// interface at all.
type erroringBoundaryAdapter struct{ fakeAdapter }
func (a *erroringBoundaryAdapter) AtTurnBoundary(context.Context, herdr.Session) (bool, error) {
return false, errors.New("pane.status unsupported")
}
// noBoundaryAdapter never implements herdr.TurnBoundary at all, exercising
// the genuine occupancy-only degraded fallback.
type noBoundaryAdapter struct {
occupancy float64
ref string
releases int
}
func (a *noBoundaryAdapter) Lease(_ context.Context, _ string, worktree string) (herdr.Session, error) {
return herdr.Session{Harness: "h1", PaneID: "pane-1", Worktree: worktree}, nil
}
func (a *noBoundaryAdapter) Bootstrap(context.Context, herdr.Session, string) error { return nil }
func (a *noBoundaryAdapter) Release(context.Context, herdr.Session) (string, error) {
a.releases++
return a.ref, nil
}
func (a *noBoundaryAdapter) Kill(context.Context, herdr.Session) error { return nil }
func (a *noBoundaryAdapter) Occupancy(herdr.Session) (float64, error) { return a.occupancy, nil }
func setupRotationTask(t *testing.T, repo string) (*store.Store, string, domain.Task, string) {
t.Helper()
run(t, repo, "init")
run(t, repo, "config", "user.email", "t@t")
run(t, repo, "config", "user.name", "t")
run(t, repo, "commit", "--allow-empty", "-m", "init")
head, err := herdr.HeadSHA(repo)
if err != nil {
t.Fatal(err)
}
s, err := store.Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: "t1", Surface: string(authz.System), Payload: mustJSON(map[string]any{
"source": "jsonl", "external_id": "1", "project": "p",
})}); err != nil {
t.Fatal(err)
}
task := s.Tasks()[0]
ref, err := s.PutArtifact([]byte("handoff"))
if err != nil {
t.Fatal(err)
}
return s, head, task, ref
}
// TestTurnBoundaryErrorBlocksRelease proves an adapter that implements Face B
// but cannot currently answer it (a transient herdr error) never falls
// through to an unconfirmed release — spec §5.2/§5.3 treats the boundary
// check as required, not best-effort.
func TestTurnBoundaryErrorBlocksRelease(t *testing.T) {
repo := t.TempDir()
s, _, task, ref := setupRotationTask(t, repo)
a := &erroringBoundaryAdapter{fakeAdapter{occupancy: .95, boundary: true, ref: ref}}
c := &orchestrator.Coordinator{Store: s, Worktrees: worktrees{path: repo}, Adapters: adapters{a}, StatePath: t.TempDir() + "/sessions.json"}
leaseEvt, err := s.Lease(task.ID, "h1", time.Minute)
if err != nil {
t.Fatal(err)
}
if err := c.Start(context.Background(), leaseEvt); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go c.Monitor(ctx, .8, time.Millisecond)
time.Sleep(50 * time.Millisecond)
got, _ := s.Task(task.ID)
if got.State != domain.StateLeased {
t.Fatalf("release proceeded despite an unanswerable turn-boundary check: state=%s", got.State)
}
if a.releases != 0 {
t.Fatalf("adapter.Release was called despite the boundary error, releases=%d", a.releases)
}
if c.MonitorHealth().TurnBoundaryDegraded == 0 {
t.Fatal("turn-boundary degradation was not recorded")
}
}
// TestNoTurnBoundarySupportDegradesVisibly proves an adapter that never
// implements Face B still falls back to occupancy-only thresholding (so
// existing deployments keep working) but the degradation is observable via
// MonitorHealth, not silent.
func TestNoTurnBoundarySupportDegradesVisibly(t *testing.T) {
repo := t.TempDir()
s, head, task, ref := setupRotationTask(t, repo)
a := &noBoundaryAdapter{occupancy: .95, ref: ref}
c := &orchestrator.Coordinator{Store: s, Worktrees: worktrees{path: repo}, Adapters: adapters{a}, StatePath: t.TempDir() + "/sessions.json"}
leaseEvt, err := s.Lease(task.ID, "h1", time.Minute)
if err != nil {
t.Fatal(err)
}
if err := c.Start(context.Background(), leaseEvt); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go c.Monitor(ctx, .8, time.Millisecond)
deadline := time.Now().Add(time.Second)
for time.Now().Before(deadline) {
if got, ok := s.Task(task.ID); ok && got.State == domain.StateQueued {
break
}
time.Sleep(time.Millisecond)
}
got, ok := s.Task(task.ID)
if !ok || got.State != domain.StateQueued {
t.Fatalf("rotation did not complete without Face B support: state=%v ok=%v", got.State, ok)
}
_ = head
if c.MonitorHealth().TurnBoundaryDegraded == 0 {
t.Fatal("missing Face B support was not recorded as degraded")
}
}
+63
View File
@@ -0,0 +1,63 @@
package orchestrator_test
import (
"context"
"orchestra/internal/domain"
"orchestra/internal/orchestrator"
"os"
"os/exec"
"path/filepath"
"testing"
)
func initRepo(t *testing.T, dir string) {
t.Helper()
run := func(args ...string) {
cmd := exec.Command("git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("git %v: %v: %s", args, err, out)
}
}
if err := os.MkdirAll(dir, 0755); err != nil {
t.Fatal(err)
}
run("init")
if err := os.WriteFile(filepath.Join(dir, "README"), []byte("x"), 0644); err != nil {
t.Fatal(err)
}
run("add", "README")
run("commit", "-m", "init")
}
func TestPerProjectGitWorktreesResolvesByProject(t *testing.T) {
base := t.TempDir()
repoA := filepath.Join(base, "repo-a")
repoB := filepath.Join(base, "repo-b")
initRepo(t, repoA)
initRepo(t, repoB)
w := orchestrator.PerProjectGitWorktrees{
Projects: map[string]orchestrator.ProjectRepo{
"proj-a": {Repo: repoA, WorktreeRoot: filepath.Join(base, "wt-a")},
},
Default: orchestrator.GitWorktrees{Root: filepath.Join(base, "wt-default"), Repo: repoB},
}
pathA, err := w.Create(context.Background(), domain.Task{ID: "t1", Project: "proj-a"})
if err != nil {
t.Fatalf("create for proj-a: %v", err)
}
if filepath.Dir(pathA) != filepath.Join(base, "wt-a") {
t.Fatalf("expected proj-a worktree under wt-a, got %s", pathA)
}
pathDefault, err := w.Create(context.Background(), domain.Task{ID: "t2", Project: "unconfigured-project"})
if err != nil {
t.Fatalf("create for unconfigured project: %v", err)
}
if filepath.Dir(pathDefault) != filepath.Join(base, "wt-default") {
t.Fatalf("expected unconfigured project to use default worktree root, got %s", pathDefault)
}
}
+136
View File
@@ -0,0 +1,136 @@
package provider
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"net/http"
"net/http/httptest"
"orchestra/internal/domain"
"os"
"path/filepath"
"testing"
)
func TestGiteaSourceNameDefaultsToRepo(t *testing.T) {
if got := (Gitea{Repo: "orchestra"}).SourceName(); got != "gitea" {
t.Fatalf("expected legacy unnamespaced source, got %q", got)
}
if got := (Gitea{Repo: "orchestra", Project: "correx"}).SourceName(); got != "gitea:correx" {
t.Fatalf("expected namespaced source, got %q", got)
}
}
func TestGiteaIngestWebhookTagsProject(t *testing.T) {
g := Gitea{Owner: "kami", Repo: "correx-repo", Project: "correx", WebhookSecret: "s3cret"}
body, _ := json.Marshal(map[string]any{
"action": "opened",
"issue": map[string]any{"number": 42, "title": "fix thing", "body": "", "state": "open"},
})
mac := hmac.New(sha256.New, []byte("s3cret"))
mac.Write(body)
sig := hex.EncodeToString(mac.Sum(nil))
sk := &sink{}
if err := g.IngestWebhook(body, sig, sk); err != nil {
t.Fatalf("ingest: %v", err)
}
if len(sk.events) != 1 {
t.Fatalf("expected 1 event, got %d", len(sk.events))
}
var p struct {
Project string `json:"project"`
Source string `json:"source"`
ExternalID string `json:"external_id"`
}
if err := json.Unmarshal(sk.events[0].Payload, &p); err != nil {
t.Fatalf("unmarshal payload: %v", err)
}
if p.Project != "correx" || p.Source != "gitea:correx" || p.ExternalID != "42" {
t.Fatalf("unexpected payload: %+v", p)
}
}
func TestGiteaIngestWebhookRejectsBadSignature(t *testing.T) {
g := Gitea{Owner: "kami", Repo: "r", Project: "p", WebhookSecret: "s3cret"}
if err := g.IngestWebhook([]byte(`{"action":"opened","issue":{"number":1}}`), "wrong", &sink{}); err == nil {
t.Fatal("expected signature rejection")
}
}
func TestMultiGiteaReflectDispatchesByTaskSource(t *testing.T) {
var hitCorrex, hitMaven bool
correxSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hitCorrex = true
w.WriteHeader(200)
}))
defer correxSrv.Close()
mavenSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hitMaven = true
w.WriteHeader(200)
}))
defer mavenSrv.Close()
m := MultiGitea{Sources: map[string]Gitea{
"gitea:correx": {BaseURL: correxSrv.URL, Owner: "kami", Repo: "correx-repo", Project: "correx"},
"gitea:maven": {BaseURL: mavenSrv.URL, Owner: "kami", Repo: "maven-repo", Project: "maven"},
}}
if err := m.ReflectTask(domain.Task{Source: "gitea:correx", ExternalID: "7"}, domain.Event{Type: "TaskCompleted"}); err != nil {
t.Fatalf("reflect to correx: %v", err)
}
if !hitCorrex || hitMaven {
t.Fatalf("expected only correx server hit, got correx=%v maven=%v", hitCorrex, hitMaven)
}
hitCorrex, hitMaven = false, false
if err := m.ReflectTask(domain.Task{Source: "gitea:maven", ExternalID: "3"}, domain.Event{Type: "TaskFailed"}); err != nil {
t.Fatalf("reflect to maven: %v", err)
}
if hitCorrex || !hitMaven {
t.Fatalf("expected only maven server hit, got correx=%v maven=%v", hitCorrex, hitMaven)
}
// A task from a non-Gitea source (or an unregistered Gitea project) must
// be a silent no-op, not an error.
if err := m.ReflectTask(domain.Task{Source: "jsonl"}, domain.Event{Type: "TaskCompleted"}); err != nil {
t.Fatalf("unmatched source should no-op, got %v", err)
}
}
func TestLoadGiteaConfigsValidatesAndRejectsDuplicates(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "gitea.json")
good := `[
{"project":"correx","base_url":"https://gitea.internal","owner":"kami","repo":"correx"},
{"project":"maven","base_url":"https://gitea.internal","owner":"kami","repo":"maven"}
]`
if err := os.WriteFile(path, []byte(good), 0644); err != nil {
t.Fatal(err)
}
cfgs, err := LoadGiteaConfigs(path)
if err != nil || len(cfgs) != 2 {
t.Fatalf("cfgs=%d err=%v", len(cfgs), err)
}
dupPath := filepath.Join(dir, "dup.json")
dup := `[
{"project":"correx","base_url":"https://gitea.internal","owner":"kami","repo":"correx"},
{"project":"correx","base_url":"https://gitea.internal","owner":"kami","repo":"other"}
]`
if err := os.WriteFile(dupPath, []byte(dup), 0644); err != nil {
t.Fatal(err)
}
if _, err := LoadGiteaConfigs(dupPath); err == nil {
t.Fatal("expected duplicate project rejection")
}
incompletePath := filepath.Join(dir, "incomplete.json")
if err := os.WriteFile(incompletePath, []byte(`[{"project":"x"}]`), 0644); err != nil {
t.Fatal(err)
}
if _, err := LoadGiteaConfigs(incompletePath); err == nil {
t.Fatal("expected missing-field rejection")
}
}
+87 -8
View File
@@ -18,6 +18,7 @@ import (
"sync"
"time"
"orchestra/internal/authz"
"orchestra/internal/domain"
)
@@ -136,7 +137,7 @@ func (j JSONL) Ingest(r io.Reader, sink Sink) (int, error) {
return count, fmt.Errorf("line %d: %w", line, err)
}
b, _ := json.Marshal(p)
if err := sink.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: domain.NewID(), Version: 1, Payload: b}); err != nil {
if err := sink.Append(domain.Event{ID: domain.NewID(), Type: "TaskCreated", TaskID: domain.NewID(), Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
return count, fmt.Errorf("line %d: %w", line, err)
}
count++
@@ -197,7 +198,78 @@ func (w JSONLWatcher) Run(ctx context.Context, sink Sink) error {
type Gitea struct {
BaseURL, Token, WebhookSecret, Owner, Repo string
Client *http.Client
// Project, if set, is the orchestra project id ingested tasks are
// tagged with (registry.Project.ID) and the key this source is
// dispatched under in MultiGitea. Deployments with a single Gitea repo
// may leave it empty, in which case Repo is used as both — preserving
// the historical single-source behavior.
Project string
Client *http.Client
}
// sourceName is the provider "source" every ingested TaskCreated carries,
// and the (source,external_id) idempotency/reflection key. It is namespaced
// per project so issue numbers from two different Gitea repos never
// collide in the dedup key, and so MultiGitea can route a TaskCompleted's
// reflection back to the correct repo.
func (g Gitea) sourceName() string { return g.SourceName() }
// SourceName is the exported form of sourceName, for callers (e.g.
// cmd/orchestra) that build a MultiGitea{Sources: ...} map.
func (g Gitea) SourceName() string {
if g.Project == "" {
return "gitea"
}
return "gitea:" + g.Project
}
// GiteaSourceConfig describes one Gitea repo to ingest from/reflect to.
// Load a list of these from JSON (ORCHESTRA_GITEA_CONFIG) to run more than
// one Gitea-backed project side by side — each project may have its own
// repo, owner, and credentials.
type GiteaSourceConfig struct {
Project string `json:"project"`
BaseURL string `json:"base_url"`
Owner string `json:"owner"`
Repo string `json:"repo"`
Token string `json:"token"`
WebhookSecret string `json:"webhook_secret"`
}
// LoadGiteaConfigs reads a JSON array of GiteaSourceConfig from path.
func LoadGiteaConfigs(path string) ([]GiteaSourceConfig, error) {
b, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var out []GiteaSourceConfig
if err := json.Unmarshal(b, &out); err != nil {
return nil, fmt.Errorf("gitea config: %w", err)
}
seen := map[string]bool{}
for _, c := range out {
if c.Project == "" || c.BaseURL == "" || c.Owner == "" || c.Repo == "" {
return nil, fmt.Errorf("gitea config: project, base_url, owner, and repo are required (got %+v)", c)
}
if seen[c.Project] {
return nil, fmt.Errorf("gitea config: duplicate project %q", c.Project)
}
seen[c.Project] = true
}
return out, nil
}
// MultiGitea dispatches TaskReflector reflection to whichever Gitea source
// ingested the task, keyed by Gitea.sourceName(). This lets several Gitea
// repos (one per project) share a single ReflectingSink.
type MultiGitea struct{ Sources map[string]Gitea }
func (m MultiGitea) ReflectTask(task domain.Task, e domain.Event) error {
g, ok := m.Sources[task.Source]
if !ok {
return nil
}
return g.ReflectTask(task, e)
}
type giteaIssue struct {
Number int `json:"number"`
@@ -229,7 +301,7 @@ func (g Gitea) event(issue giteaIssue, source, project string) domain.Event {
}
p := map[string]any{"source": source, "external_id": strconv.Itoa(issue.Number), "project": project, "title": issue.Title, "capability": caps}
b, _ := json.Marshal(p)
return domain.Event{ID: domain.NewID(), TaskID: domain.NewID(), Type: "TaskCreated", Version: 1, Payload: b}
return domain.Event{ID: domain.NewID(), TaskID: domain.NewID(), Type: "TaskCreated", Version: 1, Payload: b, Surface: string(authz.System)}
}
func (g Gitea) IngestWebhook(body []byte, signature string, sink Sink) error {
if !validSignature(body, signature, g.WebhookSecret) {
@@ -242,11 +314,14 @@ func (g Gitea) IngestWebhook(body []byte, signature string, sink Sink) error {
if h.Action == "closed" || h.Action == "deleted" {
return nil
}
project := g.Repo
if h.Repository.FullName != "" {
project = h.Repository.FullName
project := g.Project
if project == "" {
project = g.Repo
if h.Repository.FullName != "" {
project = h.Repository.FullName
}
}
return sink.Append(g.event(h.Issue, "gitea", project))
return sink.Append(g.event(h.Issue, g.sourceName(), project))
}
func validSignature(body []byte, got, secret string) bool {
if secret == "" || got == "" {
@@ -299,8 +374,12 @@ func (g Gitea) Poll(ctx context.Context, sink Sink) (int, error) {
if err = json.NewDecoder(resp.Body).Decode(&issues); err != nil {
return 0, err
}
project := g.Project
if project == "" {
project = g.Repo
}
for _, i := range issues {
if err := sink.Append(g.event(i, "gitea", g.Repo)); err != nil {
if err := sink.Append(g.event(i, g.sourceName(), project)); err != nil {
return 0, err
}
}
+67 -2
View File
@@ -21,6 +21,13 @@ var (
type Project struct {
ID string `json:"id"`
MachineAffinity []string `json:"machine_affinity"`
// Repo and WorktreeRoot let each project resolve its own git checkout
// (spec §2.2 — a project is first-class; nothing about the model implies
// a single shared repo across all projects). Both optional: a project
// that omits them falls back to whatever global default the deployment
// wires (single-repo deployments keep working unchanged).
Repo string `json:"repo,omitempty"`
WorktreeRoot string `json:"worktree_root,omitempty"`
}
type Machine struct {
ID string `json:"id"`
@@ -34,8 +41,17 @@ type Herdr struct {
Protocol string `json:"protocol,omitempty"`
Capabilities []string `json:"capabilities"`
Concurrency int `json:"concurrency"`
QuotaLimit float64 `json:"quota_limit,omitempty"`
// QuotaLimit is deprecated in favor of QuotaLimit5h/QuotaLimitWeekly; if
// set and QuotaLimit5h is not, it is treated as the weekly limit only
// (its historical meaning), to avoid silently inventing a 5h cap for
// existing configuration.
QuotaLimit float64 `json:"quota_limit,omitempty"`
QuotaLimit5h float64 `json:"quota_limit_5h,omitempty"`
QuotaLimitWeekly float64 `json:"quota_limit_weekly,omitempty"`
}
const defaultHerdrPort = "9245"
type Config struct {
Projects []Project `json:"projects"`
Machines []Machine `json:"machines"`
@@ -53,12 +69,58 @@ func Load(path string) (Registry, error) {
return Registry{}, err
}
var c Config
if err = json.Unmarshal(b, &c); err != nil {
if err = json.Unmarshal(stripJSONComments(b), &c); err != nil {
return Registry{}, fmt.Errorf("registry config: %w", err)
}
return New(c)
}
// stripJSONComments removes // line comments and /* */ block comments from
// JSONC input, leaving valid JSON. Comment markers inside string literals
// (respecting backslash escapes) are left untouched. This lets deployments
// annotate config.json in place instead of keeping a separate undocumented
// copy (see deploy/config.example.jsonc).
func stripJSONComments(b []byte) []byte {
out := make([]byte, 0, len(b))
inString, escaped, inLineComment, inBlockComment := false, false, false, false
for i := 0; i < len(b); i++ {
c := b[i]
switch {
case inLineComment:
if c == '\n' {
inLineComment = false
out = append(out, c)
}
case inBlockComment:
if c == '*' && i+1 < len(b) && b[i+1] == '/' {
inBlockComment = false
i++
}
case inString:
out = append(out, c)
if escaped {
escaped = false
} else if c == '\\' {
escaped = true
} else if c == '"' {
inString = false
}
case c == '"':
inString = true
out = append(out, c)
case c == '/' && i+1 < len(b) && b[i+1] == '/':
inLineComment = true
i++
case c == '/' && i+1 < len(b) && b[i+1] == '*':
inBlockComment = true
i++
default:
out = append(out, c)
}
}
return out
}
func New(c Config) (Registry, error) {
r := Registry{map[string]Project{}, map[string]Machine{}, map[string]Herdr{}}
for _, p := range c.Projects {
@@ -160,6 +222,9 @@ func (r Registry) Candidates(project string, check Reachability, timeout time.Du
addr := h.Address
if addr == "" {
addr = r.machines[h.MachineID].Address
if host, _, err := net.SplitHostPort(addr); err == nil {
addr = net.JoinHostPort(host, defaultHerdrPort)
}
}
if check == nil || check.Reachable(addr, timeout) {
out = append(out, h)
+3 -2
View File
@@ -5,6 +5,7 @@ import (
"testing"
"time"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/registry"
"orchestra/internal/store"
@@ -17,7 +18,7 @@ func TestAssignPendingUsesProjectAffinityAndCapability(t *testing.T) {
}
add := func(id, project string, caps []string) {
b, _ := json.Marshal(map[string]any{"source": "test", "external_id": id, "project": project, "capability": caps})
if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: id, Version: 1, Payload: b}); err != nil {
if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: id, Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
}
@@ -45,7 +46,7 @@ func TestAssignPendingDoesNotPreemptRunningWork(t *testing.T) {
t.Fatal(err)
}
b, _ := json.Marshal(map[string]any{"source": "test", "external_id": "queued", "project": "p", "capability": []string{}})
if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: "queued", Version: 1, Payload: b}); err != nil {
if err := s.Append(domain.Event{Type: "TaskCreated", TaskID: "queued", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
r, err := registry.New(registry.Config{Projects: []registry.Project{{ID: "p", MachineAffinity: []string{"m"}}}, Machines: []registry.Machine{{ID: "m", Address: "m:1"}}, Herdrs: []registry.Herdr{{ID: "h", MachineID: "m", Concurrency: 1}}})
+51 -25
View File
@@ -4,6 +4,7 @@ package router
import (
"encoding/json"
"errors"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/registry"
"orchestra/internal/store"
@@ -17,46 +18,71 @@ type AlwaysAvailable struct{}
func (AlwaysAvailable) Available(registry.Herdr) bool { return true }
// QuotaAvailability applies the conservative 80% rule to summed native
// receipts in the configured rolling window. Receipts are additive across
// rotations; a cumulative report must not replace earlier rotations.
// QuotaWindowLimits are the two independent caps the spec (§7.2) requires:
// the subscription pool's 5-hour rolling window and its weekly window. They
// are tracked and evaluated separately — a harness deep into its 5h window
// but fine on the week, or vice versa, must still be excluded.
type QuotaWindowLimits struct {
FiveHour float64
Weekly float64
}
const (
fiveHourWindow = 5 * time.Hour
weeklyWindow = 7 * 24 * time.Hour
// quotaConservativeFraction is the degrade-safe default from spec §7.2/§9
// item 1: since no quota pool is authoritative, treat 80% reported as
// full rather than trusting the exact number.
quotaConservativeFraction = 0.8
)
// QuotaAvailability applies the conservative 80% rule independently to the
// 5-hour rolling window and the weekly window, per harness. Receipts are
// additive across rotations; a cumulative session total must never replace
// earlier rotations' receipts (spec §5.2.1) — summing native per-report
// `consumed` deltas is what keeps this correct across rotation.
type QuotaAvailability struct {
Store *store.Store
Limits map[string]float64
Window time.Duration
Limits map[string]QuotaWindowLimits
Now func() time.Time
}
func (q QuotaAvailability) Available(h registry.Herdr) bool {
if q.Store == nil {
return false
}
limit, bounded := q.Limits[h.ID]
if !bounded || limit <= 0 {
return true
}
now := time.Now()
if q.Now != nil {
now = q.Now()
}
window := q.Window
if window <= 0 {
window = 7 * 24 * time.Hour
}
func (q QuotaAvailability) sumSince(harnessID string, since time.Time) float64 {
var consumed float64
for _, e := range q.Store.Events(0) {
if e.Type != "QuotaReported" || e.At.Before(now.Add(-window)) {
if e.Type != "QuotaReported" || e.At.Before(since) {
continue
}
var p struct {
HarnessID string `json:"harness_id"`
Consumed float64 `json:"consumed"`
}
if json.Unmarshal(e.Payload, &p) == nil && p.HarnessID == h.ID && p.Consumed >= 0 {
if json.Unmarshal(e.Payload, &p) == nil && p.HarnessID == harnessID && p.Consumed >= 0 {
consumed += p.Consumed
}
}
return consumed < limit*0.8
return consumed
}
func (q QuotaAvailability) Available(h registry.Herdr) bool {
if q.Store == nil {
return false
}
limits, bounded := q.Limits[h.ID]
if !bounded || (limits.FiveHour <= 0 && limits.Weekly <= 0) {
return true
}
now := time.Now()
if q.Now != nil {
now = q.Now()
}
if limits.FiveHour > 0 && q.sumSince(h.ID, now.Add(-fiveHourWindow)) >= limits.FiveHour*quotaConservativeFraction {
return false
}
if limits.Weekly > 0 && q.sumSince(h.ID, now.Add(-weeklyWindow)) >= limits.Weekly*quotaConservativeFraction {
return false
}
return true
}
type RetryPolicy struct {
@@ -185,6 +211,6 @@ func importance(t domain.Task, now time.Time) time.Time {
}
func (r *Router) fail(t domain.Task) (domain.Event, error) {
b, _ := json.Marshal(map[string]any{"reason": "retry_limit", "attempts": r.attempts[t.ID]})
e := domain.Event{ID: domain.NewID(), Type: "TaskFailed", TaskID: t.ID, Version: t.Version + 1, Payload: b}
e := domain.Event{ID: domain.NewID(), Type: "TaskFailed", TaskID: t.ID, Version: t.Version + 1, Payload: b, Surface: string(authz.System)}
return e, r.Store.Append(e)
}
+44 -1
View File
@@ -2,6 +2,7 @@ package router
import (
"encoding/json"
"orchestra/internal/authz"
"orchestra/internal/domain"
"orchestra/internal/registry"
"orchestra/internal/store"
@@ -28,7 +29,7 @@ func TestAssignsByAffinityCapabilityAndConcurrency(t *testing.T) {
}
makeTask := func(id string) {
b, _ := json.Marshal(map[string]any{"source": "test", "external_id": id, "project": "p", "capability": []string{"go"}})
if err := s.Append(domain.Event{ID: id, TaskID: id, Type: "TaskCreated", Version: 1, Payload: b}); err != nil {
if err := s.Append(domain.Event{ID: id, TaskID: id, Type: "TaskCreated", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
}
@@ -43,3 +44,45 @@ func TestAssignsByAffinityCapabilityAndConcurrency(t *testing.T) {
t.Fatal("no task leased")
}
}
// TestQuotaWindowsAreIndependent proves the 5-hour rolling window and the
// weekly window (spec §7.2, §9 item 1) are each conservative-80%-full gates
// on their own — a harness can be fine on one window and excluded by the
// other, and receipts outside a window must not count toward it.
func TestQuotaWindowsAreIndependent(t *testing.T) {
s, err := store.Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
now := time.Now().UTC()
report := func(at time.Time, consumed float64) {
p, _ := json.Marshal(map[string]any{"harness_id": "h1", "consumed": consumed})
if err := s.Append(domain.Event{ID: domain.NewID(), Type: "QuotaReported", TaskID: "quota", Version: 1, Surface: string(authz.System), Payload: p, At: at}); err != nil {
t.Fatal(err)
}
}
// Case 1: only weekly limit configured. A receipt older than 5h but
// within the week still counts toward the weekly gate.
report(now.Add(-6*time.Hour), 85)
weeklyOnly := QuotaAvailability{Store: s, Limits: map[string]QuotaWindowLimits{"h1": {Weekly: 100}}, Now: func() time.Time { return now }}
if weeklyOnly.Available(registry.Herdr{ID: "h1"}) {
t.Fatal("weekly window should be exhausted at 85/100 (>=80%)")
}
// Case 2: only a 5h limit configured. The same 6h-old receipt is outside
// the 5h window and must not count.
fiveHourOnly := QuotaAvailability{Store: s, Limits: map[string]QuotaWindowLimits{"h1": {FiveHour: 100}}, Now: func() time.Time { return now }}
if !fiveHourOnly.Available(registry.Herdr{ID: "h1"}) {
t.Fatal("receipt outside the 5h window incorrectly counted against it")
}
// Case 3: a fresh receipt inside the 5h window trips the 5h gate even
// though the weekly gate (fed by both receipts) also trips — both are
// independently enforced, and either failing excludes the harness.
report(now.Add(-time.Minute), 90)
both := QuotaAvailability{Store: s, Limits: map[string]QuotaWindowLimits{"h1": {FiveHour: 100, Weekly: 500}}, Now: func() time.Time { return now }}
if both.Available(registry.Herdr{ID: "h1"}) {
t.Fatal("5h window should be exhausted at 90/100 (>=80%) regardless of weekly headroom")
}
}
+16 -5
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"orchestra/internal/authz"
"orchestra/internal/domain"
"os"
"path/filepath"
@@ -45,6 +46,10 @@ func Open(dir string) (*Store, error) {
s.external[t.Source+"\x00"+t.ExternalID] = t.ID
}
snapshotSeq = snap.Seq
// Continue event numbering after the snapshot. Without restoring this
// cursor, the first append after a restart reused sequence 1 and made
// the append-only log unreplayable.
s.seq = snapshotSeq
} else if !errors.Is(readErr, os.ErrNotExist) {
return nil, readErr
}
@@ -160,9 +165,6 @@ func (s *Store) apply(e domain.Event) error {
func (s *Store) Append(e domain.Event) error {
s.mu.Lock()
defer s.mu.Unlock()
if err := domain.ValidateEvent(e); err != nil {
return err
}
if e.At.IsZero() {
e.At = time.Now().UTC()
}
@@ -172,6 +174,15 @@ func (s *Store) Append(e domain.Event) error {
if e.SchemaVersion == 0 {
e.SchemaVersion = domain.CurrentEventSchema
}
if err := domain.ValidateEvent(e); err != nil {
return err
}
// Enforced once, at the append boundary, per spec §7.1/invariant 4 — every
// producer (HTTP handler, router, coordinator, provider, federation relay)
// must declare its Surface here; there is no separate in-process bypass.
if err := authz.AuthorizeEvent(authz.Surface(e.Surface), e.Type); err != nil {
return err
}
if e.Type == "TaskCreated" {
var p map[string]any
if err := json.Unmarshal(e.Payload, &p); err != nil {
@@ -327,7 +338,7 @@ func (s *Store) Lease(id, harness string, ttl time.Duration) (domain.Event, erro
return domain.Event{}, domain.ErrConflict
}
p, _ := json.Marshal(map[string]any{"harness_id": harness, "ttl": ttl.Seconds(), "until_ns": time.Now().Add(ttl).UnixNano(), "expected_version": t.Version})
e := domain.Event{ID: id, Type: "TaskLeased", TaskID: id, Version: t.Version + 1, Payload: p}
e := domain.Event{ID: id, Type: "TaskLeased", TaskID: id, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
return e, s.Append(e)
}
@@ -336,7 +347,7 @@ func (s *Store) ExpireLeases(now time.Time) ([]domain.Event, error) {
for _, t := range s.Tasks() {
if t.State == domain.StateLeased && t.Lease != nil && !t.Lease.Until.After(now) {
p, _ := json.Marshal(map[string]any{"reason": "lease_expired", "harness_id": t.Lease.HarnessID})
e := domain.Event{ID: t.ID, Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p}
e := domain.Event{ID: t.ID, Type: "TaskReleased", TaskID: t.ID, Version: t.Version + 1, Payload: p, Surface: string(authz.System)}
if err := s.Append(e); err != nil {
return out, err
}
+38 -5
View File
@@ -6,12 +6,13 @@ import (
"path/filepath"
"testing"
"orchestra/internal/authz"
"orchestra/internal/domain"
)
func created(id string) domain.Event {
b, _ := json.Marshal(map[string]any{"source": "jsonl", "external_id": "42", "project": "demo", "capability": []string{"mechanical"}})
return domain.Event{ID: id, Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b}
return domain.Event{ID: id, Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b, Surface: string(authz.System)}
}
func TestAppendReplayAndDeduplicate(t *testing.T) {
@@ -34,10 +35,10 @@ func TestAppendReplayAndDeduplicate(t *testing.T) {
t.Fatal(err)
}
completion, _ := json.Marshal(map[string]any{"report_ref": ref, "receipt": map[string]any{"harness_id": "h", "consumed": 1}})
if err := s.Append(domain.Event{Type: "TaskCompleted", TaskID: "task-1", Version: 2, Payload: completion}); err != nil {
if err := s.Append(domain.Event{Type: "TaskCompleted", TaskID: "task-1", Version: 2, Payload: completion, Surface: string(authz.System)}); err != nil {
t.Fatal(err)
}
if err := s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Payload: json.RawMessage(`{"handoff_ref":"` + ref + `","anchor_sha":"0123456789012345678901234567890123456789"}`)}); err != domain.ErrConflict {
if err := s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Surface: string(authz.System), Payload: json.RawMessage(`{"handoff_ref":"` + ref + `","anchor_sha":"0123456789012345678901234567890123456789"}`)}); err != domain.ErrConflict {
t.Fatalf("expected conflict, got %v", err)
}
s2, err := Open(dir)
@@ -85,7 +86,7 @@ func TestLifecycleEventsRequireEvidence(t *testing.T) {
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
err := domain.ValidateEvent(domain.Event{Type: tc.typ, TaskID: "task-1", Version: 1, Payload: json.RawMessage(tc.body)})
err := domain.ValidateEvent(domain.Event{Type: tc.typ, TaskID: "task-1", Version: 1, Payload: json.RawMessage(tc.body), Surface: string(authz.System)})
if err == nil {
t.Fatal("expected lifecycle evidence validation error")
}
@@ -93,6 +94,38 @@ func TestLifecycleEventsRequireEvidence(t *testing.T) {
}
}
// TestAppendEnforcesAuthorizationAtTheBus proves authorization is checked
// once at the append boundary (spec §7.1/§1.4), not only in HTTP handlers:
// a caller writing to the store directly with a notify-only surface, or with
// no declared surface at all, is rejected exactly like an HTTP request would
// be — there is no in-process bypass for the router, coordinator, or a
// provider adapter that forgets to declare who it's acting as.
func TestAppendEnforcesAuthorizationAtTheBus(t *testing.T) {
s, err := Open(t.TempDir())
if err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(map[string]any{"source": "jsonl", "external_id": "1", "project": "demo"})
// A notify-only surface (e.g. Telegram) must never be able to create a
// task by calling the store directly, even though it bypasses HTTP.
if err := s.Append(domain.Event{ID: "e1", Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b, Surface: string(authz.Telegram)}); err == nil {
t.Fatal("notify-only surface created a task via direct store access")
}
// An internal producer that forgets to declare a surface is rejected,
// not silently trusted as the plane.
if err := s.Append(domain.Event{ID: "e2", Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b}); err == nil {
t.Fatal("event with no declared surface was accepted")
}
// The plane (router/coordinator/provider) authorizes as System and
// succeeds.
if err := s.Append(domain.Event{ID: "e3", Type: "TaskCreated", TaskID: "task-1", Version: 1, Payload: b, Surface: string(authz.System)}); err != nil {
t.Fatalf("system surface rejected: %v", err)
}
}
func TestExpectedVersionIsCheckedForEveryWriter(t *testing.T) {
s, err := Open(t.TempDir())
if err != nil {
@@ -102,7 +135,7 @@ func TestExpectedVersionIsCheckedForEveryWriter(t *testing.T) {
t.Fatal(err)
}
p := json.RawMessage(`{"reason":"rotate","expected_version":0}`)
err = s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Payload: p})
err = s.Append(domain.Event{Type: "TaskReleased", TaskID: "task-1", Version: 2, Payload: p, Surface: string(authz.System)})
if err != domain.ErrConflict {
t.Fatalf("expected CAS conflict, got %v", err)
}
+163 -179
View File
@@ -2,197 +2,181 @@
Updated: 2026-07-26
## Gaps until full implementation
## Current state
This is the canonical, exhaustive server-side gap list against `orchestra-spec (1).md`. Full implementation is not complete until every item below is closed and covered by an integration test.
This is a working Go implementation of `orchestra-spec (1).md`'s Layer 13
(substrate, harness/rotation, continuity) plus a first cut of Layer 4
(surfaces). `go build ./...` and `go test ./...` both pass. The codebase is
small (~4.6k lines across `internal/{domain,store,provider,registry,router,
herdr,orchestrator,continuity,federation,delivery,authz,operations,admin}`
and `cmd/orchestra/main.go`).
- **Execution runtime:** construct adapters for Claude, Codex, opencode, and local-model herdrs from configuration; validate herdr protocol versions with `ping`; persist session/worktree mappings; recover or reconcile them after restart; and emit lifecycle events for startup, exit, failure, release, completion, and block outcomes.
- **Worktrees and Git transport:** implement project-aware repository/worktree resolution, immutable `TASK.md` provisioning, scratch-branch WIP commits, push/pull synchronization, cross-machine checkout coordination, and synchronization/error status exposed to operations.
- **Rotation control:** implement adapter turn-boundary callbacks for all harnesses, milestone and thrash triggers, soft/hard occupancy policy, handoff creation and schema validation, anchor/TASK.md validation before close, split-then-close ordering, kill/timeout handling, and lease transfer without duplicate sessions.
- **Harness monitoring:** implement Claude stop-hook integration, Codex active-session discovery from its state database, opencode server/SSE `session.status` monitoring with message-file/stats fallback, pane-exit handling, TTL fallback, and monitor health reporting.
- **Lifecycle contracts:** replace map-based lifecycle handling with typed payloads; enforce expected-version/TTL/anchor/receipt semantics; require valid completion reports and block evidence; validate all referenced artifacts and cross-field relationships; expose handoff/report upload and amendment APIs.
- **Router availability:** implement live herdr registration/heartbeat, protocol/reachability health, concurrency accounting from actual sessions, quota headroom filtering, conservative 80% quota-full behavior, and retry/lease recovery across restarts.
- **Providers:** run JSONL and Gitea workers with cancellation, restart/error supervision, deduplication/update semantics, terminal-state reflection, webhook/poll health, and provider-to-event audit metadata.
- **Quota and standup:** implement per-harness/per-window quota projection from native session data, rolling and weekly windows, receipts summed across rotations, 3am safety behavior, scheduled standup advisories, and approval-gated application of advisories.
- **Surfaces and delivery:** implement server-side event subscriptions, Telegram/ntfy brief and alert delivery, Gitea terminal reflection, Maven gated control, approval propagation/deduplication, and artifact revalidation at every control-capable boundary.
- **Federation:** implement worker registration, heartbeats, event/lease transport, offline reclamation, remote worktree ownership, cross-machine lease correctness, and authoritative synchronization state.
- **API and operations:** add readiness probes that test dependencies, herdr/provider/project administration and diagnostics, report/handoff/amendment endpoints, structured error responses, bounded request/body handling, event cursor/subscription semantics, and complete metrics for sessions, rotations, quota, providers, and failures.
- **Durability and safety:** persist runtime state needed for crash recovery, make background loops cancellable and supervised, ensure no orphaned lease/session can survive reconciliation, and remove remaining placeholder/generated lifecycle evidence.
- **Verification:** add end-to-end tests covering ingest → route → worktree → harness → rotation → completion, restart/replay, provider retries/reflection, authorization across every surface, quota exhaustion, worker loss, and concurrent version conflicts.
Earlier revisions of this file accumulated a long, self-contradictory
chronological log — gaps were listed as open in one section and then claimed
closed in a later section, sometimes inaccurately. This revision replaces
that log with one audited snapshot. Treat prior git history of this file as
session notes, not as ground truth.
## Implementation review — 2026-07-26
### Verified fixed this pass
`go test ./...` passes, but the implementation is still a tested substrate/router prototype rather than a functioning unattended multi-harness orchestra. The following gaps were verified against `orchestra-spec (1).md` and the current code:
- **Rotation emitted an invalid `TaskReleased` (the previously reported
highest-priority defect) — now fixed.** `internal/orchestrator.Coordinator.rotate`
built the release payload as `{"handoff_ref","reason"}`, omitting the
`anchor_sha` the spec (§4, §6.2) and `domain.ValidatePayload` require
whenever `handoff_ref` is present. `store.Append` would reject it, the
error was discarded (`if c.Store.Append(e) == nil`), and the lease/session
silently never rotated — the coordinator would just retry next tick with
no visible failure. Fixed by adding `herdr.HeadSHA(worktree)` and having
`rotate` populate `anchor_sha` from the real worktree HEAD before
appending; if the anchor can't be read, rotation now correctly skips that
tick (leaving the lease intact for TTL/next-tick reclaim) instead of
emitting a payload guaranteed to fail validation.
Covered by `internal/orchestrator/rotation_test.go`
(`TestRotationEmitsValidReleaseWithAnchorSHA`), which drives the real
`Coordinator.Monitor` loop against an actual git worktree and asserts the
emitted event passes `domain.ValidatePayload` with the correct SHA — the
previous end-to-end test masked this bug by manually crafting a
replacement `TaskReleased` event after observing the (silently failed)
adapter-side release.
- **The federation worker release endpoint had the same gap.** The
`/v1/federation/workers/{id}/release` handler (cmd/orchestra/main.go)
built `TaskReleased` from a request body with only `handoff_ref`, no
`anchor_sha`. Since a remote worker is the only party with the actual
checkout (§2.1: "validate against the local checkout wherever the harness
runs"), the endpoint now requires and forwards a 40-hex-char `anchor_sha`
in the request body, rejecting the call with 400 otherwise.
- **Harness execution is wired for configured Git/Codex deployments.** The router invokes the coordinator; `ORCHESTRA_REPO` + `ORCHESTRA_WORKTREE_ROOT` enable Git worktree creation, configured herdr socket adapters, session creation, and optional bootstrap. Other harness types and monitoring callbacks remain pending.
- **Rotation is partially implemented.** The configured coordinator polls adapter occupancy, releases above `ORCHESTRA_OCCUPANCY_HARD` (default 75%), and publishes a handoff-backed `TaskReleased` event. Turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety remain pending.
- **Lifecycle API payloads are invalid.** The release, complete, and block endpoints all emit `{"source":"api"}`, while validation requires `handoff_ref` or `reason`, `report_ref`, and `blocker` respectively. The documented lifecycle endpoints therefore cannot complete successfully.
- **Provider integrations are partially wired.** JSONL watching and optional Gitea webhook/poll loops now start from environment configuration, but terminal reflection, provider health, cancellation, and delivery fan-out remain absent.
- **CAS references are not content-verified at event append.** Lifecycle events check that referenced files exist, but do not verify that the file content hashes to the supplied reference.
- **Replay bypasses event validation.** Startup replay unmarshals and applies events without validating the event envelope, payload schema, or sequence/version invariants.
- **Snapshots are written but never loaded or used for replay acceleration.** Startup always replays the complete event log.
- **Task creation projection is incomplete.** `parent`, `due`, `inherent_priority`, and `estimate` are defined in the domain model but are not projected from `TaskCreated` payloads.
- **Occupancy support is incomplete relative to the spec.** Native readers and configured hard-threshold monitoring exist, but Codex active-session discovery, opencode server/SSE plus fallback, and turn-boundary monitoring are not implemented.
- **Authorization is mostly enforced at HTTP ingress.** Lifecycle and approval handlers now call `AuthorizeEvent`; an absent surface still defaults to full-control Web, and non-HTTP/event-bus integrations remain unwired.
### Multi-repo Gitea ingestion (new)
The first pass closed the store/API defects (lifecycle defaults, CAS content verification, validated replay, snapshot loading, and projection of task metadata) and added optional Gitea webhook/poll wiring. The remaining server-side gaps are below.
- `provider.Gitea` gained an optional `Project` field and `SourceName()`
(`"gitea"` if unset, `"gitea:<project>"` if set) — the namespaced source
doubles as the `(source,external_id)` dedup key, so issue #7 in two
different repos never collides, and as the reflection dispatch key.
- New `provider.MultiGitea{Sources map[string]Gitea}` implements
`TaskReflector` by looking up `task.Source` and forwarding to the matching
Gitea instance — lets several Gitea repos (one per project) share one
`ReflectingSink`.
- New `provider.GiteaSourceConfig` + `LoadGiteaConfigs(path)` load a JSON
array of `{project,base_url,owner,repo,token,webhook_secret}`.
`main.go` reads this from `ORCHESTRA_GITEA_CONFIG` if set; each source
gets its own poll supervisor (`gitea:<project>`) and webhook path
(`/v1/providers/gitea/webhook/<project>`).
- The legacy single-repo env vars (`ORCHESTRA_GITEA_URL/TOKEN/OWNER/REPO/
WEBHOOK_SECRET`) still work unchanged when `ORCHESTRA_GITEA_CONFIG` is
unset — same unprefixed webhook path, same `project = ORCHESTRA_GITEA_REPO`
tagging, same dedup source `"gitea"` — so existing deployments and
already-configured Gitea webhooks need no changes.
- Added `internal/provider/gitea_test.go` — previously **there were zero
tests exercising the Gitea provider at all** despite progress.md's prior
claim of Gitea webhook/poll test coverage; that claim was not accurate.
New tests cover source-name namespacing, webhook signature
verification/rejection, project tagging, `MultiGitea` dispatch-by-source
(via two `httptest.Server`s, asserting only the right one is hit), and
`LoadGiteaConfigs` validation/duplicate-project rejection.
### Remaining server-side gaps
### Per-project repos (new)
- **The orchestration coordinator is operational but incomplete.** It is constructed from deployment configuration, resolves a worktree, invokes `herdr.Adapter.Lease`, bootstraps handoffs, monitors occupancy, and rotates through handoff references. Session mappings are in-memory and turn-boundary monitoring is still pending.
- **Rotation remains incomplete.** Occupancy-triggered release is wired, but adapter turn-boundary callbacks, milestone/thrash triggers, and split-then-close safety are not.
- **Harness discovery and registration are not operational.** Static herdr configuration and socket clients exist, but startup does not create adapters, ping configured herdrs, discover active Codex sessions, subscribe to opencode SSE, or run the required fallback/TTL monitoring loop.
- **Provider ingestion is only partially wired.** JSONL and Gitea are available when configured, but there is no provider lifecycle management, cancellation, error health projection, terminal-state reflection, or provider fan-out.
- **Lifecycle event contracts remain incomplete.** Validation does not enforce the spec's `expected_version`, `ttl`, `anchor_sha`, `receipt`, or optional `handoff_ref` relationships, and the HTTP API does not validate actor/surface authorization at the event construction site. Completion without a report currently creates a generated placeholder artifact rather than requiring the stop-hook/wrapper receipt described by the spec.
- **Quota and standup scheduling are not implemented.** The event types and brief fields are accepted, but there is no per-harness/window quota projection, conservative availability filter, 3am safety behavior, or scheduled standup advisory producer.
- **Brief delivery and provider reflection are not implemented.** `/v1/brief` is read-only and computes local git state, but no Telegram/ntfy delivery, Gitea terminal reflection, Maven subscription, or cross-surface approval subscriber is started by the server.
- **Federated worker behavior is not complete.** Machine affinity filtering is implemented, but there is no worker registration/heartbeat protocol, remote event transport, cross-machine worktree coordination, or server-side synchronization status beyond local git inspection.
- **The server API is narrower than the spec.** There are no explicit task amendment/report/handoff upload endpoints, event subscription/streaming endpoint, health/readiness detail for providers and herdrs, or administrative endpoints for project/machine/herdr status.
- `registry.Project` gained optional `repo`/`worktree_root` fields. Each
project can now resolve its own git checkout rather than every project
sharing one global `ORCHESTRA_REPO`/`ORCHESTRA_WORKTREE_ROOT` — matches
spec §2.2 ("projects are first-class and extensible... the binding is a
field + a config entry, not a schema change"). `main.go` builds a
`orchestrator.PerProjectGitWorktrees` from the registry, falling back to
the global default for any project that omits these fields, so
single-repo deployments are unaffected. Covered by
`internal/orchestrator/worktrees_test.go`.
The latest pass now applies `AuthorizeEvent` to lifecycle and approval writes and adds `/readyz` with router/provider configuration checks. Readiness is configuration-level only; it does not yet probe herdr/provider health.
### Closed this pass (were open gaps as of the last snapshot)
The lifecycle API contract pass now requires callers to provide explicit release evidence (`reason` or `handoff_ref`), a block `blocker`, and a completion `report_ref`. The server no longer creates generated placeholder completion artifacts or converts malformed/empty lifecycle bodies into defaults. Regression coverage validates that release, completion, and block events reject missing evidence.
- **Bus-level authorization.** `authz.AuthorizeEvent` is now enforced inside
`store.Append` itself — the single choke point every event passes through
(HTTP handlers, router, coordinator/rotation, providers, federation relay)
— not just at HTTP handlers. Event schema bumped to v2, which requires
every event to declare a `Surface`; a new `authz.System` surface (full
control) covers internal emitters (router leases/failures, coordinator
releases/blocks, standup advisory/apply). Schema v1 events on disk still
replay (tolerant reader). Covered by `internal/store/store_test.go` and
`internal/router/router_test.go` additions asserting a non-HTTP append
with no/wrong surface is rejected.
- **Dual quota windows.** `router.QuotaAvailability` now tracks a 5-hour
rolling window and a 7-day weekly window independently per harness
(`QuotaWindowLimits{FiveHour, Weekly}`), applying the conservative 80%
rule to each separately — a harness over threshold on either window is
unavailable. Replaces the old single-`Window` field. Covered by new
`router_test.go` cases for weekly-only and 5h-only exhaustion.
- **Turn-boundary detection made observable, not silently optional.**
Rotation still can't force a harness adapter to implement `TurnBoundary`
Face B, but an adapter that fails to answer it now blocks that tick's
release (never treats a failed check as "safe to proceed"), and any
adapter without the capability — or one whose check errors — increments
`MonitorHealth.TurnBoundaryDegraded`, exposed via the coordinator's health
endpoint so degraded-safety operation is visible, not silent.
- **Cross-machine lease correctness has a real test.**
`internal/integration/federation_lease_test.go`
(`TestCrossMachineLeaseAnchorAndQuotaArePerHost`) exercises a lease
claimed through the federation worker HTTP API, validates the anchor
against that worker's own local checkout (not the router's), and asserts
quota is accounted per-host. Spec §9 item 8 said "prove on the first
federated run" — this is that proof for the primitives that exist today
(registration, heartbeat, lease-claim); it does not yet run against two
real physical machines.
- **Fuzz coverage for lifecycle payload validation.**
`internal/domain/fuzz_test.go` adds `FuzzValidatePayload` and
`FuzzValidateEvent` covering all event types (including malformed nested
`receipt`/`knowledge` shapes) — asserts no panic and always a typed error
on adversarial input.
Item 1 substrate hardening pass: newly appended events use schema envelope version 1; replay rejects unsupported versions and non-object payloads; and `TaskLeased` carries an `expected_version` guard that is checked before append. Legacy envelope events remain readable for tolerant replay.
### Believed accurate from prior sessions (spot-checked, not exhaustively re-verified)
Harness registration now uses configured `harness` and `protocol` fields, pings each configured herdr before exposing it to orchestration, selects Claude/Codex/opencode adapters accordingly, and skips unavailable or unsupported deployments at startup. This is an initial discovery/health slice; active-session discovery and ongoing heartbeats remain open.
- Event log: append-only JSONL, versioned envelope (schema v1), snapshot
load/replay, CAS with content-hash verification at append.
- `domain.ValidatePayload` enforces required fields per event type,
including `expected_version`/`ttl` on `TaskLeased`, `anchor_sha` on
`TaskReleased` (now correctly emitted, see above), `report_ref`+`receipt`
on `TaskCompleted`, and `blocker` on `TaskBlocked`.
- Router: project→affinity→machine resolution, capability match, quota
availability at conservative 80% threshold, derived-importance ordering,
retry-then-`TaskFailed`.
- herdr adapters (Claude/Codex/opencode) with native occupancy readers,
optional `TurnBoundary`/`RotationSignal`/`PaneExit` capability interfaces,
bootstrap/lease/release/kill.
- Continuity: strict handoff schema/validation, CAS save/load, pickup
validation (HEAD match, dirty-file hashes, immutable `TASK.md` hash),
scratch-branch commit/push/pull helpers.
- Provider layer: JSONL watcher, Gitea webhook+poll with HMAC auth,
idempotent `(source,external_id)` dedup, terminal-state reflection,
supervised restart with backoff.
- Federation: worker registration, heartbeat/TTL offline detection, event
cursor polling/ack, lease claim endpoint.
- Authorization: bus-level capability table (notify-only / full / gated) is
applied to lifecycle and approval writes via `AuthorizeEvent`.
- Delivery: Telegram/ntfy fan-out for completion/failure/block/approval
events.
- `/readyz`, `/v1/brief`, `/v1/providers/health`, `/v1/standup` exist and
return real state (not stubs).
Added bounded `POST /v1/artifacts` CAS upload support for report/handoff evidence. It returns the verified content hash used by lifecycle events and rejects empty or oversized uploads.
## Known open gaps (named, not silently assumed done)
The coordinator now persists active task→herdr session mappings in an atomic runtime state file, reloads them after restart, reconciles them against durable task leases, kills stale recoverable sessions, and removes orphan mappings before monitoring begins.
- **Cross-machine lease correctness is proven at the primitive level, not on
real hardware.** `TestCrossMachineLeaseAnchorAndQuotaArePerHost` exercises
the federation worker HTTP API (registration, lease-claim, anchor
validation against the worker's own checkout, per-host quota) inside one
test process. Spec §9 item 8 says "prove on the first federated run" —
that means an actual homesrv/workpc pair over the real mesh, which this
repo cannot exercise by itself. Named here as the one item that needs a
live two-machine run to fully close, not more code.
- **Turn-boundary Face B still degrades to occupancy-only for adapters that
don't implement it**, by design — the spec's Face B is per-harness native
session state (Stop hook / rollout tail / SSE), which this repo can only
wire against a real running herdr+harness pair. The degradation is now
observable (`MonitorHealth.TurnBoundaryDegraded`) and blocks-on-failure
rather than silently proceeding, but whether Claude/Codex/opencode's
native hooks are wired in a live deployment is a deployment-config fact,
not something provable from source alone.
Provider supervision/reflection is now wired: Gitea webhook and polling use append-first task reflection, JSONL and Gitea loops restart with bounded backoff, and `/v1/providers/health` exposes running/error state. Provider lifecycle cancellation remains tied to process shutdown until the server gains a root cancellation context.
Quota reporting now has strict payload validation, and `router.QuotaAvailability` implements rolling-window conservative headroom filtering: a harness is considered full at 80% of its configured limit. Standup event payloads also require an items field; scheduled advisory production and approval application remain open.
Notification delivery now supports Telegram and ntfy fan-out for completion, failure, block, and approval events, with event cursors, bounded polling, and notify-only surface policy preserved. Configure `ORCHESTRA_TELEGRAM_BOT_TOKEN`/`ORCHESTRA_TELEGRAM_CHAT_ID` or `ORCHESTRA_NTFY_TOPIC` to enable it.
Rotation now honors an optional herdr turn-boundary probe (`pane.status`) before hard-threshold release. Adapters without the optional capability retain occupancy-based fallback behavior.
Federation control-plane foundations now include worker registration, heartbeat updates, TTL-based offline status, and `/v1/federation/workers` plus per-worker heartbeat endpoints. Remote event/lease transport and remote worktree ownership remain to be layered on this registry.
Configured herdr `quota_limit` values are now wired into router availability using the rolling-window 80% conservative filter; previously the quota implementation existed but was not active in server routing.
Quota/standup scheduling now treats `QuotaReported` and `StandupAdvisory` as global events, adds `/v1/standup` read/request behavior, and emits one daily advisory during the 03:00 UTC safety window. Advisory contents include queued, leased, and blocked tasks.
Harness adapters now expose optional `pane.rotation_signal` support for milestone/thrash triggers. The coordinator records the returned trigger reason in the handoff release and still requires a safe turn boundary before releasing.
The previously named remaining work is now implemented and integration-tested:
- quota receipts aggregate across rotations and rolling/weekly windows; conservative availability sums receipts;
- standup advisories have scheduled generation plus approval-gated application;
- federation has authenticated worker registration, heartbeat/offline reclamation, event cursor polling/ack, and lease claim/handoff coordination;
- provider supervision retries failed loops and terminal reflection is append-first;
- end-to-end tests cover ingest → route → lease → rotation → completion, restart/replay, orphan cleanup, provider retry/reflection, quota exhaustion, and version conflicts.
Recommended order:
1. Add the orchestration coordinator: lease → worktree → harness session → bootstrap → lifecycle events.
2. Implement rotation and turn-boundary monitoring.
3. Wire provider lifecycle management, JSONL startup, terminal reflection, and delivery integrations.
4. Add quota/standup projections and conservative availability filtering.
5. Add federated worker health/synchronization and the remaining control-plane API surface.
6. Add endpoint/contract tests for the coordinator and lifecycle receipts.
## Server implementation checklist
This is the implementation-oriented breakdown of the specification. It is a project checklist, not a replacement for the binding spec.
1. **Complete the substrate****baseline complete**
- Done: append-only JSONL event log, replay projection, task schema, optimistic versions, lifecycle events, lease TTL groundwork, CAS artifacts, sortable ULID-like IDs, event payload validation, CAS-reference validation, durable atomic snapshots, corruption errors during replay, fsync-backed event writes, and API event metadata.
- Follow-up hardening: replace the remaining map-based projection logic with generated/schema-backed payload structs and add snapshot-based replay acceleration.
2. **Provider layer****complete**
- Done: Provider/Sink contracts and replay-safe JSONL adapter.
- Done: append-only JSONL file watcher/ingester with rotation handling and bounded records.
- Done: Gitea issue adapter for webhook and open-issue polling, including label-to-capability mapping.
- Done: Gitea reflection for terminal task state, keyed by the task's stable external issue number.
- Done: constant-time HMAC webhook authentication and injectable HTTP clients for testing.
3. **Projects and machine registry****complete**
- Done: typed JSON project, machine, and herdr configuration with duplicate/reference validation.
- Done: machine-bound herdr registry with per-herdr capabilities, endpoint override, and concurrency configuration.
- Done: injectable reachability checks plus TCP reachability implementation.
- Done: hard project machine-affinity resolution; candidates are restricted to configured, reachable herdrs on allowed machines.
- Done: optional `ORCHESTRA_CONFIG` startup validation.
4. **Router and leases****complete**
- Done: manual lease/release/complete/block endpoints and lease-expiry release.
- Done: assignment on `TaskCreated` and lease release/expiry.
- Done: project-affinity, capability, reachability, availability, and concurrency filtering.
- Done: derived importance ordering, retry/backoff, and terminal `TaskFailed`.
5. **Herdr integration****complete**
- Done: Unix-socket JSON-RPC client, ping protocol check, semantic prompt/wait and worktree operations.
- Done: Claude, Codex, and opencode adapter contracts with bootstrap, release, kill, and occupancy methods.
- Done: native session usage readers and bounded current-turn occupancy calculation.
- Done: anchor validation primitive for split-then-close rotation safety.
6. **Continuity****complete**
- Done: strict JSON handoff schema/validator, including framed knowledge fields and size-safe typed fields.
- Done: CAS-backed handoff save/load with content-address verification.
- Done: pickup validation against repository HEAD, dirty-file hashes, and immutable `TASK.md` hash.
- Done: scratch-branch WIP commit helper and Markdown change notices.
7. **Authorization and surfaces****implemented**
- Done: centralized bus-level surface capabilities and optional bearer-token authentication.
- Done: full-control TUI/web policy, notify-only Telegram/ntfy policy, and gated MCP/Maven policy.
- Done: approval-request endpoint (`POST /v1/tasks/{id}/approval`) and approval event payload validation.
- Note: TUI/web, Telegram/ntfy, MCP, and Maven remain client integrations over the server's polling/event APIs; the server is the authorization boundary.
8. **Projections and operations****complete**
- Done: read-only windowed brief projection for completions, failures, blocks, approvals, quota reports, and local git sync state.
- Done: quota and standup event types are accepted by the event schema for projection/scheduling integrations.
- Done: git failures are surfaced as an unsynchronized/unavailable state.
- Done: operational projection code is covered by tests.
- Done: Prometheus-compatible task metrics and a systemd deployment unit.
## Completed
- Built the first Go server slice from `orchestra-spec (1).md`.
- Added append-only JSONL events and replay projection in `internal/store`.
- Added task creation, external-key deduplication, optimistic versions, lifecycle states, and SHA-256 CAS artifacts.
- Added HTTP endpoints on default port `9145`: health, task ingest/list, and event cursor reads.
- Added lease/release lifecycle endpoints and lease-expiry reclamation.
- Finished the item 1 provider port: `provider.Provider`/`Sink` interfaces and a replay-safe JSONL adapter.
- Finished item 2: JSONL watching, authenticated Gitea webhook/poll ingestion, and terminal-state reflection.
- Added event-type payload validation for lifecycle and amendment events.
- Unit tests pass with `go test ./...`.
- Implemented item 4 router assignment, lease-expiry polling, and retry policy.
- Implemented item 5 herdr socket integration, harness adapters, native occupancy readers, bootstrap, and anchor validation.
- Implemented item 6 continuity: validated CAS handoffs, pickup anchors/TASK.md, scratch-branch commits, and shared Markdown change notices.
## Current API additions
- `POST /v1/tasks/{id}/lease` with `{"harness_id":"...","ttl_seconds":1800}`
- `POST /v1/tasks/{id}/release`
- `POST /v1/tasks/{id}/complete`
- `POST /v1/tasks/{id}/block`
## Item 3 status
Item 3 (projects and machine registry) is implemented in `internal/registry`. Static JSON configuration is loaded and validated, projects resolve only to their explicitly configured machines, and candidate herdrs are filtered by registration and injected reachability. Set `ORCHESTRA_CONFIG` to validate a configuration file at server startup.
## Item 2 status
Item 2 (provider layer) is implemented. `internal/provider` now includes `JSONLWatcher`, `Gitea.Poll`, `Gitea.WebhookHandler`, `Gitea.IngestWebhook`, and `Gitea.ReflectTask`. Gitea ingestion remains idempotent through the store's `(source, external_id)` key. The server wiring can attach these components to deployment-specific routes and polling loops without adding provider-specific logic to the domain.
## Item 1 status
Item 1 (task schema + provider port + JSONL adapter) is implemented as the baseline slice. The event schema is still deliberately versionless and should receive an envelope/version field during item 2 without breaking tolerant readers.
## Important limitations
- This is still a Layer 1/2 prototype. Harness adapter and continuity primitives exist, but unattended orchestration, rotation, quota accounting, and delivery integrations are not server-wired.
- Surface authorization is enforced by the shared HTTP/bus policy; set `ORCHESTRA_*_TOKEN` variables to require bearer authentication per surface.
- Event payload validation currently checks required fields and primitive types; replace the remaining map-based application logic with typed payload structs before exposing the API beyond the homelab.
- Router retry counts/backoff and terminal `TaskFailed` are implemented; retry policy is currently configured in server wiring.
## Next agent: recommended order
1. Begin item 2: harden the event log and state projection with snapshots, corruption handling, and a versioned envelope.
2. Add project, machine, and herdr registries from static TOML/JSON config.
3. Implement router selection: project affinity, reachability, capability, availability, and importance ordering.
4. Add retry policy and a background lease-expiry loop.
5. Implement handoff/report schemas and CAS reference validation.
6. Integrate herdr only after the substrate/router tests are stable.
Everything else named as open in the previous snapshot (bus-level
authorization, dual 5h/weekly quota windows, fuzz coverage of lifecycle
payload validation) is now closed — see "Closed this pass" above. Broader
areas (provider layer, continuity, router matching, delivery, federation
registration) were spot-checked against the code and their tests and
matched their described behavior.